You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently .to_array() is called on each scalar value that creates a single row array, which is very inefficient:
There is overhead for generating the array structure for each row.
The single-row arrays are concatenated afterwards at the end, which is slow and would be unnecessary if they are created with more values at once
Intermediate Vecs are generated, causing more memory usage / allocations / fragmentation.
I expect this should speed up some db-benchmark queries (group by queries with smaller groups) considerably and may decrease memory usage by quite a bit.
Describe the solution you'd like
Iterate over the values and emit arrays of batch_size elements at once.
Or as a first step just do it for all of the values (as is the case currently) - and emit smaller batches in a later PR.
To do it with batch_size there should be some state and/or the groups should be removed from the map.
Describe alternatives you've considered
n/a
Additional context
n/a
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently
.to_array()
is called on each scalar value that creates a single row array, which is very inefficient:Vecs
are generated, causing more memory usage / allocations / fragmentation.I expect this should speed up some db-benchmark queries (group by queries with smaller groups) considerably and may decrease memory usage by quite a bit.
Describe the solution you'd like
Iterate over the values and emit arrays of
batch_size
elements at once.Or as a first step just do it for all of the values (as is the case currently) - and emit smaller batches in a later PR.
To do it with
batch_size
there should be some state and/or the groups should be removed from the map.Describe alternatives you've considered
n/a
Additional context
n/a
The text was updated successfully, but these errors were encountered: