-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use Specialization Instead of ScalarValue Binary Operations #6842
Labels
enhancement
New feature or request
Comments
19 tasks
tustvold
added a commit
to tustvold/arrow-datafusion
that referenced
this issue
Jul 4, 2023
tustvold
added a commit
that referenced
this issue
Jul 6, 2023
alamb
pushed a commit
to alamb/datafusion
that referenced
this issue
Jul 6, 2023
* Deprecate ScalarValue::and, ScalarValue::or (apache#6842) * Review feedback
tustvold
added a commit
to tustvold/arrow-datafusion
that referenced
this issue
Aug 21, 2023
tustvold
added a commit
to tustvold/arrow-datafusion
that referenced
this issue
Aug 21, 2023
tustvold
added a commit
to tustvold/arrow-datafusion
that referenced
this issue
Aug 21, 2023
tustvold
added a commit
that referenced
this issue
Aug 21, 2023
tustvold
added a commit
to tustvold/arrow-datafusion
that referenced
this issue
Aug 21, 2023
This was referenced Aug 21, 2023
tustvold
added a commit
to tustvold/arrow-datafusion
that referenced
this issue
Aug 22, 2023
tustvold
added a commit
that referenced
this issue
Aug 23, 2023
I think we have made substantial progress on this issue -- what is left to do? |
IIRC there are some aggregates, like first and last that are not yet specialized |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is your feature request related to a problem or challenge?
Currently a number of operations are implemented directly on ScalarValue, including:
Not only does this result in a huge amount of code, but also these operations don't behave the same way as their array counterparts.
For example:
Describe the solution you'd like
These kernels largely appear to exist for the purposes of aggregation, where the aggregated types are known statically. We should replace these uses with specialization, as done in #6800 (comment). The remaining uses should make use of the new Datum abstraction apache/arrow-rs#4393 to use the same arrow-rs kernels apache/arrow-rs#4465
Describe alternatives you've considered
No response
Additional context
#4973 tracks improving the aggregator performance
#6832 updates DF to use the Datum kernels
The text was updated successfully, but these errors were encountered: