Skip to content

Commit

Permalink
Remove NOT NULL constraint from String types. (#63)
Browse files Browse the repository at this point in the history
We create an Exasol table, if it did not exist, before saving the Spark
dataframe. The `NOT NULL` constraint was added to the create table DDL,
if the Spark schema field type is not nullable.

However, this can be problem in Exasol side. Because, Exasol puts `null`
if the string is empty for `VARCHAR` or `CLOB` column types. Therefore,
putting not null constraints fails when inserting empty strings.

This commit removes the `NOT NULL` constraints from string types even if
they are not nullable.

Fixes #60.
  • Loading branch information
morazow authored Apr 9, 2020
1 parent 5d86bdb commit ac6bf9c
Show file tree
Hide file tree
Showing 2 changed files with 21 additions and 7 deletions.
15 changes: 9 additions & 6 deletions src/main/scala/com/exasol/spark/util/Types.scala
Original file line number Diff line number Diff line change
Expand Up @@ -218,8 +218,11 @@ object Types extends Logging {
}

/**
* Returns comma separated column name and column types for Exasol table from
* Spark schema.
* Returns comma separated column name and column types for Exasol
* table from Spark schema.
*
* It skips the `NOT NULL` constraint if the Spark dataframe schema
* type is a [[org.apache.spark.sql.types.StringType$]] type.
*
* @param schema A Spark [[org.apache.spark.sql.types.StructType]] schema
* @return A comma separated column names and their types
Expand All @@ -229,10 +232,10 @@ object Types extends Logging {
.map { field =>
val fieldType = Types.exasolTypeFromSparkDataType(field.dataType)
val nameType = s"${field.name} $fieldType"
if (field.nullable) {
nameType
} else {
nameType + " NOT NULL"
field.nullable match {
case true => nameType
case false if field.dataType != StringType => nameType + " NOT NULL"
case false => nameType
}
}
.mkString(", ")
Expand Down
13 changes: 12 additions & 1 deletion src/test/scala/com/exasol/spark/util/TypesSuite.scala
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ class TypesSuite extends FunSuite with Matchers {
)
}

test("`createTableSchema` should create a comma separated column names and types") {
test("createTableSchema should create a comma separated column names and types") {
val schema: StructType = StructType(
Seq(
StructField("bool_col", BooleanType),
Expand All @@ -176,4 +176,15 @@ class TypesSuite extends FunSuite with Matchers {
assert(createTableSchema(schema) === expectedStr)
}

test("createTableSchema returns string type without not null constraint") {
val schema = StructType(
Seq(
StructField("str_col", StringType),
StructField("text_col", StringType, false)
)
)
val expectedConvertedSchema = "str_col CLOB, text_col CLOB"
assert(createTableSchema(schema) === expectedConvertedSchema)
}

}

0 comments on commit ac6bf9c

Please sign in to comment.