Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sketch for aggregation intermediate results blocked management #11943

Draft
wants to merge 112 commits into
base: main
Choose a base branch
from
Draft

Sketch for aggregation intermediate results blocked management #11943

wants to merge 112 commits into from

Commits on Aug 25, 2024

  1. re-design the sketch.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    eb74278 View commit details
    Browse the repository at this point in the history

  2. disable blocked optimization when hash agg is swtched to streaming ag… 

    …g due to spilling.
    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    728a62e View commit details
    Browse the repository at this point in the history

  3. fix style.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    a1f5e2d View commit details
    Browse the repository at this point in the history

  4. impl blocked GroupValuesRows.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    d7d22f6 View commit details
    Browse the repository at this point in the history

  5. impl simple blocked mode for Count.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    e62172a View commit details
    Browse the repository at this point in the history

  6. impl simple blocked mod for prim_op and avg.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    99cd66a View commit details
    Browse the repository at this point in the history

  7. fix init.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    520b2eb View commit details
    Browse the repository at this point in the history

  8. fix tests.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    aae2a3b View commit details
    Browse the repository at this point in the history

  9. support blocked mode in NullState.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    ed7c1b7 View commit details
    Browse the repository at this point in the history

  10. define the blocked_accumulate, so that we wont change the logic for… 

    … falt accumulators.
    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    8d9d0c0 View commit details
    Browse the repository at this point in the history

  11. impl blocked version prim_op and avg accumulators.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    cb53724 View commit details
    Browse the repository at this point in the history

  12. fix streming tests.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    cce049d View commit details
    Browse the repository at this point in the history

  13. impl blocked version count accumulator.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    89da481 View commit details
    Browse the repository at this point in the history

  14. fix the special cast that total groups num is zero.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    b2fc6d3 View commit details
    Browse the repository at this point in the history

  15. move BlockedNullState to accumulate.rs.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    fc5d05b View commit details
    Browse the repository at this point in the history

  16. fix tests.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    6092943 View commit details
    Browse the repository at this point in the history

  17. define the Blocks to replace VecDeque.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    6fcc831 View commit details
    Browse the repository at this point in the history

  18. introduce Block to accumulators.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    00cfa06 View commit details
    Browse the repository at this point in the history

  19. make Blocks more general.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    261b0f1 View commit details
    Browse the repository at this point in the history

  20. use Blocks in BlockedNullState.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    53950b9 View commit details
    Browse the repository at this point in the history

  21. use Blocks in GroupValuesRows.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    6de7fc7 View commit details
    Browse the repository at this point in the history

  22. use debug_assert instead of assert.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    1567057 View commit details
    Browse the repository at this point in the history

  23. refactor Blocks.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    a780401 View commit details
    Browse the repository at this point in the history

  24. rename CurrentBlock to NextBlock, and add more comments.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    5fc7bf5 View commit details
    Browse the repository at this point in the history

  25. minor optimization.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    e143c59 View commit details
    Browse the repository at this point in the history

  26. fix comments.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    5fb1748 View commit details
    Browse the repository at this point in the history

  27. add todos.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    564e6d3 View commit details
    Browse the repository at this point in the history

  28. reduce repeated codes.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    7e607eb View commit details
    Browse the repository at this point in the history

  29. disable blocked optimization in spilling case, and add comments.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    ec9bf21 View commit details
    Browse the repository at this point in the history

  30. add more comments and remove stale codes.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    cb8da87 View commit details
    Browse the repository at this point in the history

  31. not try to support spilling in blocked mode in currently.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    e054c8b View commit details
    Browse the repository at this point in the history

  32. improve error messages.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    478627d View commit details
    Browse the repository at this point in the history

  33. add comment for ProducingBlocks.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    ead0076 View commit details
    Browse the repository at this point in the history

  34. add comments for GroupStatesMode.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    db1adbe View commit details
    Browse the repository at this point in the history

  35. remove unused import.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    03189a7 View commit details
    Browse the repository at this point in the history

  36. fix clippy.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    49c5e5e View commit details
    Browse the repository at this point in the history

  37. fix clippy.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    50e8958 View commit details
    Browse the repository at this point in the history

  38. add comments.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    13e74b0 View commit details
    Browse the repository at this point in the history

  39. improve comments.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    f5684a0 View commit details
    Browse the repository at this point in the history

  40. eliminate the unnecessary mode check + current_mut.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    3b63eaa View commit details
    Browse the repository at this point in the history

  41. use index to replace get + unwrap.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    5602ded View commit details
    Browse the repository at this point in the history

  42. add comments for some interanl functions.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    8da7806 View commit details
    Browse the repository at this point in the history

  43. remove more unnecessary mode checks.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    e8ce09b View commit details
    Browse the repository at this point in the history

  44. fix some comments.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    48c7e4f View commit details
    Browse the repository at this point in the history

  45. improve comments about Block.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    0bbad3a View commit details
    Browse the repository at this point in the history

  46. move Blocks, BlockedIndex, some functions of Emit to `datafusio… 

    …n-functions-aggregate-common`.
    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    0ab53dc View commit details
    Browse the repository at this point in the history

  47. add test for BlockedNullState.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    78c8e82 View commit details
    Browse the repository at this point in the history

  48. fix BlockedNullState's unit test.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    f54878f View commit details
    Browse the repository at this point in the history

  49. add unit tests for blocks.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    56b0bcf View commit details
    Browse the repository at this point in the history

  50. add unit test for ensure_enough_room_for_blocked_nulls.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    94af694 View commit details
    Browse the repository at this point in the history

  51. test take needed.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    1064a72 View commit details
    Browse the repository at this point in the history

  52. fix clippy.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    2b0796a View commit details
    Browse the repository at this point in the history

  53. merge two modes to one.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    46b10b4 View commit details
    Browse the repository at this point in the history

  54. experiment.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    aef6c49 View commit details
    Browse the repository at this point in the history

  55. use function point to replace trait.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    127a6e7 View commit details
    Browse the repository at this point in the history

  56. simplify function pointer.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    9bffb4a View commit details
    Browse the repository at this point in the history

  57. init the function point during new.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    c8c0fee View commit details
    Browse the repository at this point in the history

  58. tmp.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    3e409ba View commit details
    Browse the repository at this point in the history

  59. tmp3

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    25269f4 View commit details
    Browse the repository at this point in the history

  60. tmp4

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    4e2b9bc View commit details
    Browse the repository at this point in the history

  61. tmp5.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    07deb39 View commit details
    Browse the repository at this point in the history

  62. just keep the if else.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    5101165 View commit details
    Browse the repository at this point in the history

  63. remove some unnecessary codes.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    e10a4bb View commit details
    Browse the repository at this point in the history

  64. adapt the new great comments.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    cc117d4 View commit details
    Browse the repository at this point in the history

  65. add option and disable the blocked optimization by default.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    c65b808 View commit details
    Browse the repository at this point in the history

  66. remove codes about outdated switch_to_mdoe.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    0a26de8 View commit details
    Browse the repository at this point in the history

  67. fix test.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    96f8be8 View commit details
    Browse the repository at this point in the history

  68. fix clippy.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    6535b93 View commit details
    Browse the repository at this point in the history

  69. try to eliminate more dynamic dispatch.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    781d00c View commit details
    Browse the repository at this point in the history

  70. continue to eliminate more dynamic dispatch.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    3316f8f View commit details
    Browse the repository at this point in the history

  71. test.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    8a8e799 View commit details
    Browse the repository at this point in the history

  72. fix clippy.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    921cad7 View commit details
    Browse the repository at this point in the history

  73. fix sql logic tests.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    8c5afa9 View commit details
    Browse the repository at this point in the history

  74. add options to enable blocked apporach in benmarks.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    e4dd31a View commit details
    Browse the repository at this point in the history

  75. fix fmt and clippy.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    db431cb View commit details
    Browse the repository at this point in the history

  76. fix tpch opts.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    f2e316a View commit details
    Browse the repository at this point in the history

  77. fix comments of BlockedGroupIndex.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    8b8da5e View commit details
    Browse the repository at this point in the history

  78. update config.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    21e5fdf View commit details
    Browse the repository at this point in the history

  79. fix docs.

    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    8d5cb7f View commit details
    Browse the repository at this point in the history

  80. use the right way to check if spilling enalbed, and support `emit_ear… 

    …ly_if_necessary` again.
    @Rachelint
    Rachelint committed Aug 25, 2024
    Configuration menu
    Copy the full SHA
    b33c6f9 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2024

  1. unify ensure_enough_room_for_xxx and add tests.

    @Rachelint
    Rachelint committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    fd54e1c View commit details
    Browse the repository at this point in the history

  2. fix ensure_enough_room_for_xxx.

    @Rachelint
    Rachelint committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    addcc13 View commit details
    Browse the repository at this point in the history

  3. add physical level test for blocked approach.

    @Rachelint
    Rachelint committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    5a2292d View commit details
    Browse the repository at this point in the history

  4. extract run_aggregate_test_internal for resuing later.

    @Rachelint
    Rachelint committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    207a777 View commit details
    Browse the repository at this point in the history

  5. add simple fuzz test for blocked approach.

    @Rachelint
    Rachelint committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    6a4cf5b View commit details
    Browse the repository at this point in the history

  6. Merge branch 'main' into sketch-blocked-aggr-state-management

    @Rachelint
    Rachelint committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    f1855ee View commit details
    Browse the repository at this point in the history

  7. don't support the blocked approach in bench until it compatible wit… 

    …h spilling soon.
    @Rachelint
    Rachelint committed Aug 27, 2024
    Configuration menu
    Copy the full SHA
    551690d View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. fix clippy.

    @Rachelint
    Rachelint committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    4a97d35 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into sketch-blocked-aggr-state-management

    @Rachelint
    Rachelint committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    6f877e3 View commit details
    Browse the repository at this point in the history

  3. merge main and fix compile.

    @Rachelint
    Rachelint committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    189b4c3 View commit details
    Browse the repository at this point in the history

  4. fix clippy.

    @Rachelint
    Rachelint committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    11870cb View commit details
    Browse the repository at this point in the history

  5. fix clippy.

    @Rachelint
    Rachelint committed Aug 28, 2024
    Configuration menu
    Copy the full SHA
    6ecd81b View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2024

  1. add comments to architecture about blocked approach.

    @Rachelint
    Rachelint committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    36e0791 View commit details
    Browse the repository at this point in the history

  2. fix comments.

    @Rachelint
    Rachelint committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    4426307 View commit details
    Browse the repository at this point in the history

  3. fix typo.

    @Rachelint
    Rachelint committed Aug 29, 2024
    Configuration menu
    Copy the full SHA
    886bb20 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2024

  1. fix docs.

    @Rachelint
    Rachelint committed Aug 30, 2024
    Configuration menu
    Copy the full SHA
    c2cb573 View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2024

  1. a unified and low cost way to compute the different type `BlockedGrou… 

    …pIndex`s.
    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    ef91012 View commit details
    Browse the repository at this point in the history

  2. add test to BlockedGroupIndexBuilder.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    31356d8 View commit details
    Browse the repository at this point in the history

  3. add more inlines.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    5d2ac01 View commit details
    Browse the repository at this point in the history

  4. fix clippy.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    0a7b52b View commit details
    Browse the repository at this point in the history

  5. improve config comments.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    318c650 View commit details
    Browse the repository at this point in the history

  6. remove deprecated function.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    3d82094 View commit details
    Browse the repository at this point in the history

  7. improve docs.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    b7a443a View commit details
    Browse the repository at this point in the history

  8. rename the on/off option to enable_aggregation_intermediate_states_bl… 

    …ocked_approach.
    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    0cff3be View commit details
    Browse the repository at this point in the history

  9. fix doc.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    a2d81a5 View commit details
    Browse the repository at this point in the history

  10. fix fmt and tests.

    @Rachelint
    Rachelint committed Sep 1, 2024
    Configuration menu
    Copy the full SHA
    1db8633 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2024

  1. Merge branch 'main' into sketch-blocked-aggr-state-management

    @Rachelint
    Rachelint committed Sep 2, 2024
    Configuration menu
    Copy the full SHA
    5907b8b View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2024

  1. update docs.

    @Rachelint
    Rachelint committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    6613288 View commit details
    Browse the repository at this point in the history

  2. fix fmt.

    @Rachelint
    Rachelint committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    cbafbc5 View commit details
    Browse the repository at this point in the history

  3. fix compile.

    @Rachelint
    Rachelint committed Sep 3, 2024
    Configuration menu
    Copy the full SHA
    d258ea9 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2024

  1. Merge branch 'main' into sketch-blocked-aggr-state-management

    @Rachelint
    Rachelint committed Sep 9, 2024
    Configuration menu
    Copy the full SHA
    4a48d3a View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Merge branch 'main' into sketch-blocked-aggr-state-management

    @Rachelint
    Rachelint committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7b61328 View commit details
    Browse the repository at this point in the history