-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add binary_view to string_view coercion #12643
Conversation
(LargeUtf8, Binary) => Some(LargeUtf8), | ||
(LargeUtf8, LargeBinary) => Some(LargeUtf8), | ||
(LargeUtf8, BinaryView) => Some(LargeUtf8), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PR #12092
(LargeUtf8, BinaryView) => Some(Utf8View)
but i think we should coercion to LargeUtf8 ? , since LargeUtf8 it can be a huge string.
(LargeUtf8, BinaryView) => Some(LargeUtf8)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think Arrow actually has the equivalent of LargeUtf8View
but arrow-rs doesn't support it yet.
I think the mapping (LargeUtf8, BinaryView) => Some(LargeUtf8)
makes sense to me for now
Without these change , hit_partitioned will failed at query 20
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @doupache -- this makes sense to me
(LargeUtf8, Binary) => Some(LargeUtf8), | ||
(LargeUtf8, LargeBinary) => Some(LargeUtf8), | ||
(LargeUtf8, BinaryView) => Some(LargeUtf8), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think Arrow actually has the equivalent of LargeUtf8View
but arrow-rs doesn't support it yet.
I think the mapping (LargeUtf8, BinaryView) => Some(LargeUtf8)
makes sense to me for now
Thanks again @doupache |
Which issue does this PR close?
Closes #12500
Rationale for this change
a step forward to enable reading StringViewArray by default from Parquet
What changes are included in this PR?
add binary_view to string_view coercion
Are these changes tested?
yes , add a slt test case
I also test manually
change
benchmarks/src/clickbench.rs
, make force_view_types = trueall 43 query completed.
Are there any user-facing changes?
No, since force_view_type default is still false.