Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

introduce StringView #436

Open
richox opened this issue Apr 7, 2024 · 1 comment
Open

introduce StringView #436

richox opened this issue Apr 7, 2024 · 1 comment
Labels
performance related Potential optimization point

Comments

@richox
Copy link
Collaborator

richox commented Apr 7, 2024

Is your feature request related to a problem? Please describe.
arrow-rs is supporting new StringViewArray type (apache/arrow-rs#5374), which is likely to achieve better performance by reducing large memcpys.

Describe the solution you'd like

  1. supports StringView basics like hashing, serialization.
  2. supports StringView in methods inside datafusion-ext-functions.
  3. test/benchmark.

Describe alternatives you've considered

Additional context

@richox richox added the performance related Potential optimization point label Apr 7, 2024
@alamb
Copy link

alamb commented Apr 15, 2024

FWIW I think the main usecases where StringView will Significantly help are:

  1. Low selectivity filters (where most values are not filtered) on string columns
  2. String manipulation functions (like substr) where the output can be calculated by changing the views rather than the underlying string values.

If you are interested and willing to help, getting StringViewArray support into the DataFusion expression evaluator (especially for filtering) would be a huge help

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance related Potential optimization point
Projects
None yet
Development

No branches or pull requests

2 participants