Refactor user properties handling #46
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
morazow
commented
Oct 28, 2019
morazow
commented
- Adds abstract and common properties classes
- Adds Storage and KafkaConsumer properties classes
- Refactors Bucket specific implementations
- Adds unit and integration (using embeddedkafka) tests
Introduces AbstractProperties and StorageProperties classes that makes it robust and easy to maintain the UDF parameters throughout the codebase.
- Adds additional properties - Removes several old code and classes - KafkaConsumerProperties creates KafkaConsumer for scriptclasses
The [embedded-kafka-schema-registry](https://github.com/embeddedkafka/embedded-kafka-schema-registry) library provides in-memory instances of Kafka, Zookeeper and Confluent Schema Registry that makes it easy to run tests against. Furthermore, refactors and adds more unit tests.
Adds these Kafka consumer parameters that can provided to configure the UDF Kafka consumer: - fetch.max.bytes - max.partition.fetch.bytes
…afka schema
registry dependency.
This was strange error that occurs when running on Scala 2.11.12 version but not
on Scala 2.12.8 version.
The error:
```
[error] Uncaught exception when running com.exasol.cloudetl.scriptclasses.KafkaMetadataIT: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
[error] sbt.ForkMain$ForkError: java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)[Ljava/lang/Object;
[error] at kafka.cluster.EndPoint$.<init>(EndPoint.scala:32)
[error] at kafka.cluster.EndPoint$.<clinit>(EndPoint.scala)
[error] at kafka.server.Defaults$.<init>(KafkaConfig.scala:69)
[error] at kafka.server.Defaults$.<clinit>(KafkaConfig.scala)
[error] at kafka.server.KafkaConfig$.<init>(KafkaConfig.scala:854)
[error] at kafka.server.KafkaConfig$.<clinit>(KafkaConfig.scala)
[error] at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:35)
[error] at net.manub.embeddedkafka.schemaregistry.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:62)
[error] at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:58)
[error] at net.manub.embeddedkafka.schemaregistry.EmbeddedKafka$.startKafka(EmbeddedKafka.scala:62)
[error] at net.manub.embeddedkafka.schemaregistry.EmbeddedKafka$.start(EmbeddedKafka.scala:85)
[error] at com.exasol.cloudetl.kafka.KafkaIntegrationTest$class.beforeEach(KafkaIntegrationTest.scala:30)
[error] at com.exasol.cloudetl.scriptclasses.KafkaMetadataIT.beforeEach(KafkaMetadataIT.scala:16)
```
My initial gut feeling was that dependency versions do not match. That is, when
I run it with `./sbtx ++2.11.12 it:test` it downloads the Scala `2.11` version
of embedded-kafka-schema-registry but not the Scala `2.11` version of
org-apache-kafka dependency.
I checked into embedded-kafka dependencies, they look okay to me.
However, I still do not understand why this issue occurs.
My solution was to explicitly define the kafka dependency and remove the one
coming with the embedded-kafka-schema-registry.
Updated dependencies:
```scala
private val TestDependencies: Seq[ModuleID] = Seq(
...
"org.apache.kafka" %% "kafka" % "2.3.0" exclude ("org.slf4j", "slf4j-log4j12") exclude ("org.apache.kafka", "kafka-clients"),
"io.github.embeddedkafka" %% "embedded-kafka-schema-registry" % "5.3.0" exclude ("org.apacha.kafka", "kafka")
).map(_ % Test)
```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.