Merge string-view2 branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)
#11667
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add a knob to force string view in benchmark * fix sql logic test * update doc * fix ci * fix ci only test * Update benchmarks/src/util/options.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/common/src/config.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * update tests --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* add functions * add tests for hash util
* Update `string-view` branch to arrow-rs main (#10966) * Pin to arrow main * Fix clippy with latest arrow * Uncomment test that needs new arrow-rs to work * Update datafusion-cli Cargo.lock * Update Cargo.lock * tapelo * merge * update cast * consistent dep * fix ci * add more tests * make doc happy * update new implementation * fix bug * avoid unused dep * update dep * update * fix cargo check * update doc * pick up the comments change again --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
…11519) * add functions * Update `string-view` branch to arrow-rs main (#10966) * Pin to arrow main * Fix clippy with latest arrow * Uncomment test that needs new arrow-rs to work * Update datafusion-cli Cargo.lock * Update Cargo.lock * tapelo * merge * update cast * consistent dep * fix ci * avoid unused dep * update dep * update * fix cargo check * better group value view aggregation * update --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* initial support for string view regex * update tests
* Add StringView support for date_part and make_date funcs * run cargo update in datafusion-cli * cargo fmt --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* gc string view when appropriate * make clippy happy * address comments * make doc happy * update style * Add comments and tests for gc_string_view_batch * better herustic * update test * Update datafusion/physical-plan/src/coalesce_batches.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
github-actions
bot
added
documentation
|
CI appears to be failing due to #11671 |
* fix bug in return type inference * update doc * add tests --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
4 tasks
* better default block size * fix related test
This was referenced Jul 27, 2024
* use inferenced schema, don't load schema again * move config to parquet-only * update * update * better format * format * update
|
Update here is that we are on track to release arrow Then we'll need a PR to update datafusion to arrow 52.2.0 (which should be straightforward) Then I will change this PR to ready to review and we should be able to merge it into DataFusion main 🤞 |
alamb
changed the title
StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )
alamb
changed the title
StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )
* native support for character length * Update datafusion/functions/src/unicode/character_length.rs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
This was referenced Jul 27, 2024
alamb
changed the title
StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )string-view branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)
|
This PR is now ready for review (mostly I am hoping another committer will approve/merge it as I can't approve my own PR) |
16 tasks
alamb
changed the title
string-view branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)string-view2 branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)
8 tasks
|
Thanks @Dandandan -- so exciting! I plan to work with @XiangpengHao and figure out what we need to do to get this feature on by default |
|
🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Draft until arrow52.2.0is released to crates.io (expected Sat July 27)Which issue does this PR close?
Part of #10918
Closes #10921
schema_force_string_view) by default #11682Rationale for this change
We have been integrating a set of StringView changes on the
string-view2branch as they rely on un-released code in arrow-rs . Once those changes are released and DataFusion uses them we can bring this code directly to main.What changes are included in this PR?
StringViewArray#11556) - @XiangpengHaoStringViewArrayinCoalesceBatchesStream#11587) - @XiangpengHaoutf8_to_int_type#11662)--string-viewto only apply to parquet formats #11663) - @XiangpengHaoAre these changes tested?
CI
Are there any user-facing changes?
When StringView is enabled, benchmarks run significantly faster