Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: remove tumble function #4367

Closed
wants to merge 2 commits into from

Conversation

discord9
Copy link
Contributor

@discord9 discord9 commented Jul 15, 2024

I hereby agree to the terms of the GreptimeDB CLA.

Refer to a related PR or issue link (optional)

What's changed and what's your intention?

Remove tumble function which is buggy and caused many problem, the function is replaced by date_bin function instead:

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR requires documentation updates.

Summary by CodeRabbit

  • Refactor

    • Simplified test suite by removing specific test cases for tumbling window group-by aggregation.
    • Eliminated TumbleWindow related variants and methods from functions, enums, and structs.
    • Updated logic for schema handling and projection rewriting in TypedPlan.
    • Revised test cases and SQL scripts to align with updated logic.
  • Bug Fixes

    • Corrected inconsistencies in handling group expressions and output type construction.
  • Chores

    • Removed obsolete functions and logic related to TumbleWindow across multiple files.

@discord9 discord9 requested review from zhongzc, waynexia and a team as code owners July 15, 2024 07:12
Copy link
Contributor

coderabbitai bot commented Jul 15, 2024

Walkthrough

The flow module has been updated to version 0.8.2, which involves significant alterations to its aggregation and expression functionalities. Key changes include the removal of the tumble window functions, adjustments to the handling of group expressions, and various refactorings in the logic of projections and column processing. Several test cases have also been updated or removed to reflect these changes.

Changes

Files Change Summary
src/flow/src/compute/render/reduce.rs Removed the test_tumble_group_by function, which was used for aggregation with windowing functions.
src/flow/src/expr/func.rs Removed TumbleWindow, TumbleWindowFloor, and TumbleWindowCeiling variants from enums and adjusted associated logic.
src/flow/src/expr/scalar.rs Removed the expand_multi_value function in TypedExpr, which handled expansion of TumbleWindow expressions.
src/flow/src/expr/signature.rs Removed the TumbleWindow variant from the GenericFn enum.
src/flow/src/transform.rs Significant modifications to the TumbleFunction struct and methods; placeholder for custom functions added.
src/flow/src/transform/aggr.rs Refactored logic and control flow including removal of find_time_index_in_group_exprs and changes to column handling during aggregation.
src/flow/src/transform/expr.rs Removed test imports, test cases, and assertions related to ScalarFunction and TypedExpr.
src/flow/src/transform/plan.rs Simplified projection and expression logic; removed rewrite_projection_after_reduce function.
tests/cases/standalone/common/flow/... Updated SQL commands and processing logic related to creating, manipulating, and testing tables and flows.

Poem

In the meadow of code so bright,
Flow module dances, day and night.
Tumble windows bid adieu,
Aggregates and projections anew.
Bugs are fixed, tests do cheer,
Version 0.8.2 is here! 🎉


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@github-actions github-actions bot added breaking-change This pull request contains breaking changes. docs-required This change requires docs update. labels Jul 15, 2024
@github-actions github-actions bot added docs-not-required This change does not impact docs. and removed docs-required This change requires docs update. labels Jul 15, 2024
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between b8bd845 and 2100e5b.

Files selected for processing (10)
  • src/flow/src/compute/render/reduce.rs (1 hunks)
  • src/flow/src/expr/func.rs (5 hunks)
  • src/flow/src/expr/scalar.rs (1 hunks)
  • src/flow/src/expr/signature.rs (1 hunks)
  • src/flow/src/transform.rs (2 hunks)
  • src/flow/src/transform/aggr.rs (13 hunks)
  • src/flow/src/transform/expr.rs (2 hunks)
  • src/flow/src/transform/plan.rs (3 hunks)
  • tests/cases/standalone/common/flow/basic.result (1 hunks)
  • tests/cases/standalone/common/flow/basic.sql (1 hunks)
Files skipped from review due to trivial changes (3)
  • src/flow/src/compute/render/reduce.rs
  • src/flow/src/expr/signature.rs
  • src/flow/src/transform/expr.rs
Additional comments not posted (13)
tests/cases/standalone/common/flow/basic.sql (1)

Line range hint 1-24:
LGTM!

The SQL commands are correctly updated to use the date_trunc function instead of the tumble function.

tests/cases/standalone/common/flow/basic.result (1)

Line range hint 1-36:
LGTM!

The expected results are correctly updated to reflect the changes in basic.sql.

src/flow/src/transform/plan.rs (1)

Line range hint 1-81:
LGTM!

The code is correctly updated to remove references to the tumble function and related logic.

src/flow/src/transform.rs (1)

149-150: LGTM!

The code is correctly updated to remove references to the tumble function and related logic.

src/flow/src/expr/func.rs (1)

70-74: LGTM! But verify the enum usage in the codebase.

The removal of the TumbleWindow variant from the UnmaterializableFunc enum and the update to the from_str_args method are consistent with the objective of removing the tumble function and its related code.

However, ensure that all references to this variant have been removed or updated accordingly.

Verification successful

Verification Complete: No issues found.

The removal of the TumbleWindow variant from the UnmaterializableFunc enum has been successfully completed. There are no remaining references to TumbleWindow in the codebase.

  • No occurrences of UnmaterializableFunc::TumbleWindow found.
  • The UnmaterializableFunc references are related to other variants and are intact.
Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify all references to `UnmaterializableFunc::TumbleWindow` have been removed or updated.

# Test: Search for the enum variant usage. Expect: No occurrences of the variant.
rg --type rust -A 5 $'UnmaterializableFunc::TumbleWindow'

Length of output: 56


Script:

#!/bin/bash
# Description: Verify all references to `TumbleWindow` have been removed or updated.

# Test 1: Search for any references to `TumbleWindow` in the codebase. Expect: No occurrences of the variant.
rg --type rust -A 5 $'TumbleWindow'

# Test 2: Search for any references to `UnmaterializableFunc` to ensure all variants are accounted for. Expect: No references to `TumbleWindow`.
rg --type rust -A 5 $'UnmaterializableFunc'

Length of output: 9478

src/flow/src/transform/aggr.rs (8)

316-316: Verify the impact of setting time_index to None.

The time_index variable is now set to None. Ensure that this change does not affect the functionality of the TypedPlan::from_substrait_agg_rel function and related components.


Line range hint 450-550:
LGTM!

The test_df_func_basic test function appears comprehensive and covers various cases.


Line range hint 556-647:
LGTM!

The test_df_func_expr_tree test function appears comprehensive and covers various cases.


Line range hint 649-715:
LGTM!

The test_avg_group_by test function appears comprehensive and covers various cases.


Line range hint 717-775:
LGTM!

The test_avg test function appears comprehensive and covers various cases.


Line range hint 777-835:
LGTM!

The test_sum test function appears comprehensive and covers various cases.


Line range hint 837-899:
LGTM!

The test_sum_group_by test function appears comprehensive and covers various cases.


Line range hint 901-961:
LGTM!

The test_sum_add test function appears comprehensive and covers various cases.

Copy link

codecov bot commented Jul 15, 2024

Codecov Report

Attention: Patch coverage is 97.87234% with 1 line in your changes missing coverage. Please review.

Project coverage is 84.79%. Comparing base (15ac811) to head (2100e5b).
Report is 6 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4367      +/-   ##
==========================================
- Coverage   85.15%   84.79%   -0.36%     
==========================================
  Files        1063     1063              
  Lines      189987   189043     -944     
==========================================
- Hits       161782   160298    -1484     
- Misses      28205    28745     +540     

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between b8bd845 and b5da30e.

Files ignored due to path filters (1)
  • test.log is excluded by !**/*.log
Files selected for processing (11)
  • flow.ans (1 hunks)
  • src/flow/src/compute/render/reduce.rs (1 hunks)
  • src/flow/src/expr/func.rs (5 hunks)
  • src/flow/src/expr/scalar.rs (1 hunks)
  • src/flow/src/expr/signature.rs (1 hunks)
  • src/flow/src/transform.rs (2 hunks)
  • src/flow/src/transform/aggr.rs (13 hunks)
  • src/flow/src/transform/expr.rs (2 hunks)
  • src/flow/src/transform/plan.rs (3 hunks)
  • tests/cases/standalone/common/flow/basic.result (1 hunks)
  • tests/cases/standalone/common/flow/basic.sql (1 hunks)
Files skipped from review due to trivial changes (4)
  • src/flow/src/compute/render/reduce.rs
  • src/flow/src/expr/scalar.rs
  • src/flow/src/expr/signature.rs
  • src/flow/src/transform/expr.rs
Additional comments not posted (9)
tests/cases/standalone/common/flow/basic.sql (1)

Line range hint 1-18: Ensure alignment with new date_bin function.

The SQL commands for table creation, data insertion, and selection should align with the new date_bin function. Verify that the changes accurately reflect the removal of the tumble function and the introduction of the date_bin function.

tests/cases/standalone/common/flow/basic.result (1)

Line range hint 1-71: Ensure alignment with new date_bin function.

The expected results should align with the new date_bin function. Verify that the changes accurately reflect the removal of the tumble function and the introduction of the date_bin function.

flow.ans (1)

1-105: Ensure alignment with new date_bin function.

The compilation and test results should align with the new date_bin function. Verify that the changes accurately reflect the removal of the tumble function and the introduction of the date_bin function.

src/flow/src/transform/plan.rs (1)

81-81: Ensure alignment with new date_bin function.

The Rust code should align with the new date_bin function. Verify that the changes accurately reflect the removal of the tumble function and the introduction of the date_bin function.

src/flow/src/transform.rs (1)

149-150: Placeholder for custom functions.

The placeholder comment indicates that custom functions can be added here in the future.

src/flow/src/expr/func.rs (2)

70-74: Removed TumbleWindow variant from UnmaterializableFunc enum.

The removal simplifies the enum and aligns with the PR objectives.


74-79: Removed TumbleWindowFloor and TumbleWindowCeiling variants from UnaryFunc enum.

The removal simplifies the enum and aligns with the PR objectives.

src/flow/src/transform/aggr.rs (2)

312-316: Removed find_time_index_in_group_exprs function call.

The removal simplifies the function and aligns with the PR objectives.


Line range hint 450-550:
Removed test_tumble_group_by function.

The removal aligns with the removal of the tumble function and simplifies the test suite.

Copy link
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@fengjiachun fengjiachun added this pull request to the merge queue Jul 15, 2024
github-merge-queue bot pushed a commit that referenced this pull request Jul 15, 2024
* feat!: remove `tumble` function

* chore: forget to rm log file
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Jul 15, 2024
@discord9
Copy link
Contributor Author

Will wait for #4347 to merge first and rebase on it since same files are modified in both PR

@discord9
Copy link
Contributor Author

Close due to change of plan, will fix tumble function in a future version instead

@discord9 discord9 closed this Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change This pull request contains breaking changes. docs-not-required This change does not impact docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants