enh: interp supports using non-null prev/next values to fill #34220

Tony2h · 2026-01-08T09:01:34Z

Description

Issue(s)

resolve: Issue Link

Checklist

Please check the items in the checklist if applicable.

Is the user manual updated?
Are the test cases passed and automated?
Is there no significant decrease in test coverage?

…e notify msg

gemini-code-assist · 2026-01-08T09:02:56Z

Summary of Changes

Hello @Tony2h, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a key enhancement to the INTERP function, allowing its FILL(PREV), FILL(NEXT), and FILL(NEAR) modes to intelligently skip NULL values when the ignore_null_values parameter is active. This provides more robust and flexible data interpolation by ensuring that only meaningful data points are considered for filling gaps. The changes span core query processing logic, data serialization, and comprehensive documentation updates to reflect this new capability.

Highlights

Enhanced INTERP Functionality: The INTERP function's FILL clause now supports using non-NULL previous/next values for interpolation, taking into account the ignore_null_values parameter.
Documentation Updates: Documentation has been updated in both English and Chinese to clarify the behavior of PREV, NEXT, and NEAR fill modes with the INTERP clause, especially concerning how 'valid values' are defined when ignore_null_values is used.
Query Execution Logic Refinement: Significant internal changes have been made to the query execution and data reading mechanisms, including modifications to operator parameter handling, reader step transitions, and timestamp validation, to support the new interpolation logic.
New Test Cases: New test cases have been added to thoroughly validate the INTERP function's updated fill behavior, covering various scenarios including stream processing and scan operations.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a significant enhancement to the INTERP function, allowing it to use non-null previous/next values for filling, controlled by the ignore_null_values parameter. This is a valuable feature for time-series interpolation. The changes are extensive, touching documentation, messaging, the execution engine, and storage, and also include a new notification mechanism to optimize scans. The implementation is mostly solid, with good additions of test cases. I've found a few minor issues in the documentation and a potential memory leak in the executor logic that should be addressed.

source/libs/executor/src/scanoperator.c

+    if (pNotify->paramType != HYBRID_TYPE_SCAN_PARAM) {
+      pOperator->pOperatorGetParam = NULL;
+    }


docs/en/14-reference/03-taos-sql/20-select.md

    - `RANGE('2023-10-01T00:00:00.000', '2023-10-01T23:59:59.999', 1h)`
  - EVERY: Time interval range, with every_val being a positive value and precision options of 1n, 1u, 1a, 1s, 1m, 1h, 1d, and 1w, such as EVERY (1s).
-  - FILL: Types can be selected as NONE (unfilled), VALUE (filled with specified value), PREV (previous non NULL value), NEXT (next non NULL value), NEAR (nearest non NULL value before and after).
+  - FILL: Types can be selected as NONE (unfilled), VALUE (filled with specified value), PREV (previous valid value), NEXT (next valid value), NEAR (nearest valid value before and after), LINEAR (linear interpolation). Note: whether a NULL value is considered valid data depends on the ignore_null_values parameter of the interp function.


docs/en/14-reference/03-taos-sql/20-select.md

 - The output time range for INTERP is specified by the RANGE(timestamp1, timestamp2) field, which must satisfy timestamp1 \<= timestamp2. Here, timestamp1 is the start value of the output time range, i.e., if the conditions for interpolation are met at timestamp1, then timestamp1 is the first record output, and timestamp2 is the end value of the output time range, i.e., the timestamp of the last record output cannot be greater than timestamp2.
 - INTERP determines the number of results within the output time range based on the EVERY(time_unit) field, starting from timestamp1 and interpolating at fixed intervals of time (time_unit value), where time_unit can be time units: 1a (milliseconds), 1s (seconds), 1m (minutes), 1h (hours), 1d (days), 1w (weeks). For example, EVERY(500a) will interpolate the specified data every 500 milliseconds.
- INTERP determines how to interpolate at each time point that meets the output conditions based on the FILL field. For how to use the FILL clause, refer to [FILL Clause](../time-series-extensions/)
+- INTERP determines how to interpolate at each time point that meets the output conditions based on the FILL field. For how to use the FILL clause, refer to [FILL Clause](../time-series-extensions/). Note: The sampled data used for interpolation is not limited to the constraints of the RANGE field, but rather to all data that meets the conditions of the WHERE clause; if no WHERE clause is specified, the entire table data is used. When the parameter of the FILL clause is PREV/NEXT/NEAR, adjacent valid data will be used for interpolation. Whether NULL data is considered valid data depends on the ignore_null_values parameter of the INTERP function.


Copilot

Pull request overview

This PR enhances the INTERP function to support using non-null previous/next values for filling. The key improvement is that when ignore_null_values parameter is set to 1, the fill operations (PREV/NEXT/NEAR/LINEAR) will skip NULL values and continue searching for non-NULL data points.

Key changes:

Added notification mechanism for table scan operators to mark scanning steps as done
Enhanced timeslice operator to handle ignoring NULL values during interpolation
Added extensive test coverage for the new functionality

Reviewed changes

Copilot reviewed 33 out of 33 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
test/cases/09-DataQuerying/12-Interp/test_query_interp_fill.py	Added comprehensive test cases for interp fill with non-null values
test/cases/09-DataQuerying/12-Interp/in/interp_fill_ignore_null.in	Test input SQL statements for various fill scenarios
test/cases/09-DataQuerying/12-Interp/ans/interp_fill_ignore_null.csv	Expected test results
source/libs/executor/src/timesliceoperator.c	Core implementation of non-null value filling logic and notification mechanism
source/libs/executor/src/scanoperator.c	Added prepareTableScan to handle step done notifications
source/libs/executor/src/exchangeoperator.c	Enhanced to support notification parameters
source/dnode/vnode/src/tsdb/tsdbRead2.c	Enhanced reader to support step completion tracking
source/common/src/msg/tmsg.c	Updated message serialization for new parameter types
docs/*	Updated documentation in Chinese and English

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dapan1121 · 2026-01-09T06:10:17Z

source/common/src/msg/tmsg.c

-      TAOS_CHECK_RETURN(tEncodeI32(pEncoder, pScan->type));
+      TAOS_CHECK_RETURN(tEncodeI32(pEncoder, pScan->dynType));
+
+      // notify info


这里 notify 和 DYN_TYPE_SCAN_PARAM 参数能否共存？如果不能就各自序列化各自的，如果能paramType 的定义支持不了

dapan1121 · 2026-01-09T06:20:03Z

source/libs/executor/src/executor.c

  return code;
 }
+
+int32_t notifyTableScanTask(qTaskInfo_t tinfo, TSKEY notifyTs) {


这个函数是不是没用了？没看到调用的地方

Tony Zhang added 4 commits January 6, 2026 18:04

implement docs and tests

0690bef

impl: notify logic for interp operator

3433008

chore: formats

b6daf2c

impl timeslice optr notify logic and table scan optr openFn to prepar…

08f169c

…e notify msg

Copilot AI review requested due to automatic review settings January 8, 2026 09:01

Tony2h requested review from a team, dapan1121, guanshengliang, hzcheng and zitsen as code owners January 8, 2026 09:01

taosdata-bot bot added the team engine label Jan 8, 2026

Copilot started reviewing on behalf of Tony2h January 8, 2026 09:02 View session

gemini-code-assist bot reviewed Jan 8, 2026

View reviewed changes

Copilot AI reviewed Jan 8, 2026

View reviewed changes

enh: table scan and exchange operator notification logic for stream

e243d2e

Tony2h force-pushed the enh/3.0/interp0108 branch from 13f9bf5 to e243d2e Compare January 8, 2026 09:18

Tony Zhang added 4 commits January 9, 2026 01:26

fix: memleak and add more test cases

7601bdc

feat: initialize notification parameters in table scan operator

277c9b9

feat: add notification parameters to table scan operator initialization

f62afa6

fix: update logging to use tdSql attributes in scalar function tests

a1e1928

dapan1121 reviewed Jan 9, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

enh: interp supports using non-null prev/next values to fill #34220

enh: interp supports using non-null prev/next values to fill #34220

Tony2h commented Jan 8, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Jan 8, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

Copilot AI left a comment

Uh oh!

dapan1121 Jan 9, 2026

Uh oh!

dapan1121 Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

enh: interp supports using non-null prev/next values to fill #34220

Are you sure you want to change the base?

enh: interp supports using non-null prev/next values to fill #34220

Conversation

Tony2h commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue(s)

Checklist

Uh oh!

gemini-code-assist bot commented Jan 8, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

dapan1121 Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

dapan1121 Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Tony2h commented Jan 8, 2026 •

edited

Loading