Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up create_batch_from_map #338

Closed
Dandandan opened this issue May 14, 2021 · 0 comments · Fixed by #339
Closed

Speed up create_batch_from_map #338

Dandandan opened this issue May 14, 2021 · 0 comments · Fixed by #339
Labels
enhancement New feature or request

Comments

@Dandandan
Copy link
Contributor

Dandandan commented May 14, 2021

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently .to_array() is called on each scalar value that creates a single row array, which is very inefficient:

  • There is overhead for generating the array structure for each row.
  • The single-row arrays are concatenated afterwards at the end, which is slow and would be unnecessary if they are created with more values at once
  • Intermediate Vecs are generated, causing more memory usage / allocations / fragmentation.

I expect this should speed up some db-benchmark queries (group by queries with smaller groups) considerably and may decrease memory usage by quite a bit.

Describe the solution you'd like
Iterate over the values and emit arrays of batch_size elements at once.
Or as a first step just do it for all of the values (as is the case currently) - and emit smaller batches in a later PR.

To do it with batch_size there should be some state and/or the groups should be removed from the map.

Describe alternatives you've considered
n/a

Additional context
n/a

@Dandandan Dandandan added the enhancement New feature or request label May 14, 2021
@Dandandan Dandandan changed the title Speed up finalize_aggregation and create_batch_from_map Speed up create_batch_from_map May 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant