Task/tests type conversions #26
Conversation
Add initial build matrix for spark versions 2.1.0 and 2.3.1
If Spark analyzer tells that we only need some columns, then update the Exasol query string to select only those required columns.
This is helpful in integration tests. Because sometimes, SparkContext ends the Spark application which closes the main jdbc connection. But then re-uses that SparkContext again. In this case we will be asking a closed connection for parallel sub connections.
The scalastyle settings in tests (both unit and integration) are relaxed. For example, it is a warning to use null or magic numbers in tests.
Task/column pruning
TODO: Differentiate based on datatypes, otherwise we make everything as string in comparison and `in` clauses. For example, `col = '123.5'` this should be `col = 123.5` depending on type.
For example, for equality check, `str = 'abc'` and `num = 4` (not num = '4').
Task/filter pushdown
Because the connector is going to be used in Spark environment and it comes with scala-library already included.
Adds integration tests for predicate pushdowns Closes #2
Initial infrastructure for doing artifact automated releases from travis Closes exasol#3
So they do not work if you set them in the fork of repo?
Then from maven or sbt the library can be referenced as `com.exasol` % `spark-connector` % `version`
…agno/spark-exasol-connector into task/tests-type-conversions
|
Hello @jpizagno, Great! Thanks for working on this! Going to review. |
jpizagno
force-pushed
the
task/tests-type-conversions
branch
from
9732c0f to
7754caa
Compare
…dded. refactored names of columns and methods. changed IT according to pull-request review
jpizagno
force-pushed
the
task/tests-type-conversions
branch
2 times, most recently
from
19d67c6 to
687dfcd
Compare
|
@morazow |
|
@jpizagno apart from the typo string all looks good. Please let me know once it is removed; then we can merge the pr. |
|
@morazow |
|
@jpizagno great! lgtm! |
|
@morazow You can fix this following this guide: |
|
Good morning @jpizagno , No worries! I have updated the commits. All the Best |
@morazow
Feel free to comment on anything, and we can change it.
What are your thoughts ?