Changes:
- Updated spark-cassandra-connector from 2.4.2 to 2.5.0
- Updated spark-excel-connector from 0.12.4 to 0.13.1
- Updated spark-dynamodb-connector from 1.0.1 to 1.0.4
- Updated scalatest (scope test) from 3.1.0 to 3.1.2
- Updated postgresql (scope test) from 42.2.9 to 42.2.12
New Features:
- Added pipeline dependency check before starting the spark job
- Added default Spark job group and description
- Added
StructuredStreamingConnector
- Added
DeltaConnector
- Added
ZipArchiver
that can zip files/directories
Fixes
- Fixed path separator in FileConnectorSuite that cause test failure
- Fixed
Setl.hasExternalInput
that always returns false