-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support extract
on intervals
#11501
Support extract
on intervals
#11501
Conversation
apache/arrow-rs#6071 has landed. It just missed the 52.2 release of Arrow, so hopefully will be in 53. Once that is released (scheduled for September), I'll resurrect and rebase this PR and add some tests. |
@@ -81,6 +82,9 @@ impl DatePartFunc { | |||
Exact(vec![Utf8, Time32(Millisecond)]), | |||
Exact(vec![Utf8, Time64(Microsecond)]), | |||
Exact(vec![Utf8, Time64(Nanosecond)]), | |||
Exact(vec![Utf8, Interval(YearMonth)]), | |||
Exact(vec![Utf8, Interval(DayTime)]), | |||
Exact(vec![Utf8, Interval(MonthDayNano)]), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nrc can we add
Exact(vec![Utf8, Duration(Second)]),
Exact(vec![Utf8, Duration(Millisecond)]),
Exact(vec![Utf8, Duration(Microsecond)]),
Exact(vec![Utf8, Duration(Nanosecond)]),
Exact(vec![Utf8View, Duration(Second)]),
Exact(vec![Utf8View, Duration(Millisecond)]),
Exact(vec![Utf8View, Duration(Microsecond)]),
Exact(vec![Utf8View, Duration(Nanosecond)]),
here, then convert durations to Intervals — presumably MonthDayNano
is easiest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are added to this PR, Arrow PR coming soon
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Arrow PR: apache/arrow-rs#6246
Signed-off-by: Nick Cameron <nrc@ncameron.org>
After some testing, I found a couple of bugs due to the Both those bugs are fixed, but because the formatting of Intervals has changed with the Arrow update, it is difficult to write tests, so those will come after the proper Arrow update. |
Closed in favour of #12514 |
This is an early draft PR to demonstrate an Arrow change (see below for details).
Which issue does this PR close?
Closes #6327.
What changes are included in this PR?
This PR simply allows
Interval
types to be passed through to Arrow. It will require an Arrow upgrade to work since it depends on apache/arrow-rs#6071 (currently Arrow is patched for the CLI in this PR, this will only work if you have modified Arrow in a sibling folder to DataFusion, I'll remove this and update the PR when there is an Arrow release).Are these changes tested?
Tested in Arrow. Tests to come here.
Are there any user-facing changes?
The
extract
function will now work with intervals.