Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support division on ANSI interval types #5134

Merged
merged 9 commits into from
Apr 20, 2022

Conversation

res-life
Copy link
Collaborator

@res-life res-life commented Apr 2, 2022

Contributes #4151

Supports

  • Year-month interval / (byte, short, int, long, float, double)
  • Day-time interval / (byte, short, int, long, float, double)

Note: not support decimal type because of CUDF Decimal128 does not handle overflow, like:

CUDF positive decimal * positive decimal can generate negative value.
val d1 = ColumnVector.fromDecimals(new java.math.BigDecimal("123456789012345678901234567890123456.11"))
val d2 = Scalar.fromDecimal(new java.math.BigDecimal("12345678901234567890123456789012345611"))
val r = d1.mul(d2)
println(r)

123456789012345678901234567890123456.11 * 12345678901234567890123456789012345611 = -1053265448675579329726886670761974900.55

Signed-off-by: Chong Gao res_life@163.com

Signed-off-by: Chong Gao <res_life@163.com>
@res-life
Copy link
Collaborator Author

res-life commented Apr 2, 2022

Blocked by 2 Spark330 build issues:
#5123
#5133

@res-life res-life requested a review from firestarman April 2, 2022 07:57
@sameerz sameerz added the audit_3.3.0 Audit related tasks for 3.3.0 label Apr 4, 2022
@sameerz sameerz added this to the Apr 4 - Apr 15 milestone Apr 4, 2022
Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The temporary changes needed to compile on Spark 3.3 can be removed now that #5124 has been merged.

@res-life
Copy link
Collaborator Author

Waiting #5150 to merge, has some conflict on Spark33XShims.

@res-life
Copy link
Collaborator Author

Waiting #5150 to merge, has some conflict on Spark33XShims.

@res-life res-life marked this pull request as draft April 11, 2022 12:52
@res-life
Copy link
Collaborator Author

build

@res-life res-life marked this pull request as ready for review April 14, 2022 10:00
@res-life
Copy link
Collaborator Author

res-life commented Apr 14, 2022

Optimized the division code by leveraging decimal_128 type to reduce overflow check time, got about 2x speedup.

@res-life res-life requested a review from jlowe April 14, 2022 11:37
@res-life res-life changed the title Support Division on ANSI interval types Support division on ANSI interval types Apr 14, 2022
@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests are no longer passing after the recent changes, which implies they were not rerun after the changes were made. Note that since this test is Spark 3.3.0+, it's not normally run as part of premerge (that may soon change once Spark 3.3 finally ships). In the meantime, it's important to run it manually or we'll end up finding it doesn't pass on the nightly 3.3.0 runs after it already has been merged.

@res-life
Copy link
Collaborator Author

build

Copy link
Member

@jlowe jlowe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit on checking for NaN values using the dedicated predicate method, but otherwise lgtm.

@res-life
Copy link
Collaborator Author

build

@jlowe
Copy link
Member

jlowe commented Apr 19, 2022

CI failure is unrelated, appears to be the (transient?) Cloudera dependency issue. Kicking the build again.

@jlowe
Copy link
Member

jlowe commented Apr 19, 2022

build

@res-life res-life merged commit ce93b33 into NVIDIA:branch-22.06 Apr 20, 2022
@res-life res-life deleted the interval-division branch April 20, 2022 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit_3.3.0 Audit related tasks for 3.3.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants