You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
The code of casting float/double to decimal creates a container decimal for the HALF_UP round: link. However, the precision of the container decimal should have increased along with the scale. The sole increase of scale may lead to decimal overflow if the current precision already hits the max precision of cuDF decimal (9 for Decimal32; 18 for Decimal 64).
Seq[Double](459.288333).toDF("a").write.parquet("tmp123")
spark.read.parquet("tmp123").createOrReplaceTempView("tbl")
spark.sql("SELECT CAST(a AS decimal(9, 6)) FROM tbl") // returns -214.748364
spark.sql("SELECT CAST(CAST(a AS decimal(10, 6)) AS decimal(9, 6)) FROM tbl") // returns 459.288333
The text was updated successfully, but these errors were encountered:
)
Fixes#5765
Fix the potential overflow when casting float/double to decimal. The overflow occurs on the container decimal for HALF_UP round.
Signed-off-by: sperlingxx <lovedreamf@gmail.com>
* Correct the error message for test_mod_pmod_by_zero (#5781)
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
* Update the error string for test_cast_neg_to_decimal_err on 330[databricks] (#5784)
* Update the error string for test_cast_neg_to_decimal_err on 330
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
* address comments
Signed-off-by: Firestarman <firestarmanllc@gmail.com>
* Throw an exception when attempting to read columnar encrypted Parquet files on the GPU [databricks] (#5761)
* Throw useful message when parquet columnar encryption enabled
* update message
* fix message
* handle native encrypted
* move variable
* cleanup
* fix native check
* cleanup imports
* fix import order
* Sign off
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
* Shim the parquet crypto exception check
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
* shim 320cdh
* Add test for parquet encryption
Signed-off-by: Thomas Graves <tgraves@nvidia.com>
* fix rounds over decimal in Spark 330+ (#5786)
Passes the datatype of round-like functions directly to GPU overrides, so as to adapt different Spark versions.
Signed-off-by: sperlingxx <lovedreamf@gmail.com>
* Fix the overflow of container type when casting floats to decimal (#5766)
Fixes#5765
Fix the potential overflow when casting float/double to decimal. The overflow occurs on the container decimal for HALF_UP round.
Signed-off-by: sperlingxx <lovedreamf@gmail.com>
Co-authored-by: Liangcai Li <firestarmanllc@gmail.com>
Co-authored-by: Alfred Xu <lovedreamf@gmail.com>
Describe the bug
The code of casting float/double to decimal creates a container decimal for the HALF_UP round: link. However, the precision of the container decimal should have increased along with the scale. The sole increase of scale may lead to decimal overflow if the current precision already hits the max precision of cuDF decimal (9 for Decimal32; 18 for Decimal 64).
Steps/Code to reproduce bug
The text was updated successfully, but these errors were encountered: