Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ANSI mode: allow casting between numeric type and timestamp type [databricks] #5514

Merged
merged 8 commits into from
May 27, 2022
Merged

ANSI mode: allow casting between numeric type and timestamp type [databricks] #5514

merged 8 commits into from
May 27, 2022

Conversation

res-life
Copy link
Collaborator

@res-life res-life commented May 17, 2022 •
edited
Loading

Closes #5112

ANSI mode: allow casting between numeric type and timestamp type

  • Update containsNan code
    cv.contains(Nan) always return false; should use cv.isNan
    val cv = ColumnVector.fromDoubles(Double.NaN)
    val contains = cv.contains(Scalar.fromDouble(Double.NaN))
    print(contains) // always false
  • Cast float to timestamp
    ANSI
  if(NaN or Infinity)
    throw DateTimeException("Invalid input syntax")
  x = d * MICROS_PER_SECOND

  if (Math.floor(x) <= Long.Max && Math.ceil(x) >= Long.min) {
    x.toLong
  else
    SparkArithmeticException

Non-ANSI

if (d.isNaN || d.isInfinite) null else (d * MICROS_PER_SECOND).toLong
  • Cast long to timestamp
    ANSI and non-ANSI: if exceeds max seconds, round to Long.MAX or Long.MIN
  if (seconds >  Long.MAX_VALUE / 1000000L) return Long.MAX_VALUE;
  if (seconds < -(Long.MAX_VALUE / 1000000L)) return Long.MIN_VALUE;
  return d * m;
  • Update and add tests
  • Cast between decimal and timestamp, not support currently
    Spark code:
ANSI and non-ANSI
(d.toBigDecimal * MICROS_PER_SECOND).longValue
// Spark does not support ANSI actually.

Signed-off-by: Chong Gao res_life@163.com

@res-life
Copy link
Collaborator Author

WIP: need test on databricks.

@res-life res-life changed the title [WIP] ANSI mode: allow casting between numeric type and timestamp type [WIP] ANSI mode: allow casting between numeric type and timestamp type [databricks] May 18, 2022
@res-life
Copy link
Collaborator Author

build

@res-life res-life force-pushed the cast-between-integral-and-timestamp branch from caa7f78 to 3d87569 Compare May 18, 2022 05:55
@res-life
Copy link
Collaborator Author

build

ANSI mode: allow casting between numeric type and timestamp type
93804a4 
Signed-off-by: Chong Gao <res_life@163.com>
@res-life res-life force-pushed the cast-between-integral-and-timestamp branch from 3d87569 to 93804a4 Compare May 18, 2022 06:17
@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

@sameerz sameerz added the audit_3.3.0 Audit related tasks for 3.3.0 label May 18, 2022
@res-life
Copy link
Collaborator Author

build

1 similar comment
@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

@res-life res-life changed the title [WIP] ANSI mode: allow casting between numeric type and timestamp type [databricks] ANSI mode: allow casting between numeric type and timestamp type [databricks] May 24, 2022
@res-life
Copy link
Collaborator Author

build

@res-life res-life requested a review from revans2 May 25, 2022 10:04
@res-life res-life merged commit f631ab0 into NVIDIA:branch-22.06 May 27, 2022
@res-life res-life deleted the cast-between-integral-and-timestamp branch May 27, 2022 01:08
@firestarman firestarman mentioned this pull request May 27, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@revans2 revans2 revans2 approved these changes

Assignees
No one assigned
Labels
audit_3.3.0 Audit related tasks for 3.3.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] ANSI mode: allow casting between numeric type and timestamp type
3 participants
@res-life @revans2 @sameerz