Kafka Consumer as Exasol IMPORT UDF #39
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is initial implementation for importing data from Kafka cluster. Currently, it only supports simple string data using String Serializers and Deserializers.
Adds the support for Schema Registry and importing Avro data using Kafka Avro deserializers. Additionally, fixes a bug with initial offset value. It should start from `-1` if no data was imported before, so that next time we start consuming from zero.
By providing the Kafka consumer keystore and truststore (JKS) files stored in Exasol BucketFS bucket, the import UDF can use them to establish secure communication with Kafka cluster. Currently, only SSL protocol is supported
Previously, Avro ENUM type was considered a complex type and could not be imported into Exasol table. This commit additionally adds support for Avro Enum type import. Enum type values will be imported as string (VARCHAR) value into Exasol table.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Kafka consumer as Exasol IMPORT UDF
Implements a Kafka consumer application as Exasol IMPORT user-defined-function (UDF).
This allows users to put SQL import statements into a cron job in order to regularly import data from Kafka topics into Exasol table.
Maintaining Kafka Topic Partition Offsets
The Kafka partition offsets are maintained inside Exasol table as additional table columns. This way, we ensure that each Kafka topic record is consumed once they are committed into Exasol table. Similarly, each time a consumer starts, they query the table for the max value of the offsets per each Kafka topic partition and start requesting records after this maximum offset from the table.
Initial fix for #40.