Bug in Spark Decimal type to Exasol Decimal type conversion

[morazow]

Situation

Currently any Spark decimal type, even if it contains different precision and scale is mapped to Exasol default Decimal type, which by default is Decimal(18, 0).

Solution

Spark Decimal type should be correctly mapped to Exasol decimal type with exact precision and scale values.

Test code

val df = spark.sql("SELECT 1 AS A_INT, CAST(1.11 as DECIMAL(5,2)) AS A_DECIMALE")
val exasolWriteAndReadDF = exasolWriteAndReadFn(df)
exasolWriteAndReadDF.schema.map(_.dataType) shouldBe Array(LongType, DecimalType(5, 2)) 

[jpizagno]

@morazow Can I take this ticket, or is somebody already working on it? I think I am familiar with this area of the code.

[morazow]

Hey @jpizagno,

Good to hear from you! Yes sure, please feel free to work on this task. Thanks a lot!

Only to make sure that we are on the same page, the Exasol decimal type mapping should be changed here. The Spark decimal type dt should already have precision and scale values, e.g, dt.precision (or similar function).

[jpizagno]

@morazow
So I started working on a branch in my fork:

https://github.com/jpizagno/spark-exasol-connector/blob/task/%2344-bug-spark-decimal-type-to-exasol-decimal-type/src/main/scala/com/exasol/spark/util/Types.scala#L170

https://github.com/jpizagno/spark-exasol-connector/blob/task/%2344-bug-spark-decimal-type-to-exasol-decimal-type/src/test/scala/com/exasol/spark/util/TypesSuite.scala#L11

I think we also have to think about defaults. DecimalType.SYSTEM_DEFAULT, DecimalType.USER_DEFAULT, DecimalType.MAX_PRECISION. etc...

[morazow]

Hey @jpizagno,

This looks great!

Please send a pull request. I can also suggest some changes before pr. Could you please use the min on precisions, similar like inside boundedDecimal function?

val precision = math.min(decimalType.precision, MAX_EXASOL_PRECISION)

similar for the scale value also, on these lines.

Thanks for working on this! Please send a pull request once you are finished.

[morazow]

Fixed in #47.