Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Container decimal overflow when casting float/double to decimal #5765

Closed
sperlingxx opened this issue Jun 7, 2022 · 0 comments · Fixed by #5766
Closed

[BUG] Container decimal overflow when casting float/double to decimal #5765

sperlingxx opened this issue Jun 7, 2022 · 0 comments · Fixed by #5766
Assignees
Labels
bug Something isn't working P0 Must have for release

Comments

@sperlingxx
Copy link
Collaborator

Describe the bug
The code of casting float/double to decimal creates a container decimal for the HALF_UP round: link. However, the precision of the container decimal should have increased along with the scale. The sole increase of scale may lead to decimal overflow if the current precision already hits the max precision of cuDF decimal (9 for Decimal32; 18 for Decimal 64).

      val casted = if (targetType.getDecimalMaxPrecision == dt.scale) {
        checked.castTo(targetType)
      } else {
        val containerType = DecimalUtils.createDecimalType(dt.precision, dt.scale + 1)
        withResource(checked.castTo(containerType)) { container =>
          container.round(dt.scale, cudf.RoundMode.HALF_UP)
        }
      }

Steps/Code to reproduce bug

Seq[Double](459.288333).toDF("a").write.parquet("tmp123")
spark.read.parquet("tmp123").createOrReplaceTempView("tbl")
spark.sql("SELECT CAST(a AS decimal(9, 6)) FROM tbl") // returns -214.748364
spark.sql("SELECT CAST(CAST(a AS decimal(10, 6)) AS decimal(9, 6))  FROM tbl") // returns 459.288333
@sperlingxx sperlingxx added bug Something isn't working ? - Needs Triage Need team to review and classify labels Jun 7, 2022
@sperlingxx sperlingxx self-assigned this Jun 7, 2022
@sameerz sameerz added P0 Must have for release and removed ? - Needs Triage Need team to review and classify labels Jun 7, 2022
@sameerz sameerz added this to the Jun 6 - Jun 17 milestone Jun 7, 2022
sperlingxx added a commit that referenced this issue Jun 8, 2022
)

Fixes #5765

Fix the potential overflow when casting float/double to decimal. The overflow occurs on the container decimal for HALF_UP round.
 
Signed-off-by: sperlingxx <lovedreamf@gmail.com>
@sameerz sameerz closed this as completed Jun 8, 2022
tgravescs added a commit that referenced this issue Jun 8, 2022
* Correct the error message for test_mod_pmod_by_zero (#5781)

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

* Update the error string for test_cast_neg_to_decimal_err on 330[databricks] (#5784)

* Update the error string for test_cast_neg_to_decimal_err on 330

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

* address comments

Signed-off-by: Firestarman <firestarmanllc@gmail.com>

* Throw an exception when attempting to read columnar encrypted Parquet files on the GPU [databricks] (#5761)

* Throw useful message when parquet columnar encryption enabled

* update message

* fix message

* handle native encrypted

* move variable

* cleanup

* fix native check

* cleanup imports

* fix import order

* Sign off

Signed-off-by: Thomas Graves <tgraves@nvidia.com>

* Shim the parquet crypto exception check

Signed-off-by: Thomas Graves <tgraves@nvidia.com>

* shim 320cdh

* Add test for parquet encryption

Signed-off-by: Thomas Graves <tgraves@nvidia.com>

* fix rounds over decimal in Spark 330+ (#5786)

Passes the datatype of round-like functions directly to GPU overrides, so as to adapt different Spark versions.

Signed-off-by: sperlingxx <lovedreamf@gmail.com>

* Fix the overflow of container type when casting floats to decimal (#5766)

Fixes #5765

Fix the potential overflow when casting float/double to decimal. The overflow occurs on the container decimal for HALF_UP round.
 
Signed-off-by: sperlingxx <lovedreamf@gmail.com>

Co-authored-by: Liangcai Li <firestarmanllc@gmail.com>
Co-authored-by: Alfred Xu <lovedreamf@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P0 Must have for release
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants