Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#74: an initial implementation of the Kinesis Connector Importer #77

Merged
merged 3 commits into from
Mar 27, 2020
Merged

#74: an initial implementation of the Kinesis Connector Importer #77

merged 3 commits into from
Mar 27, 2020

Conversation

AnastasiiaSergienko
Copy link
Contributor

@AnastasiiaSergienko AnastasiiaSergienko commented Mar 5, 2020 •
edited
Loading

Fixes #74

@AnastasiiaSergienko AnastasiiaSergienko added the feature Product feature label Mar 5, 2020
@AnastasiiaSergienko AnastasiiaSergienko self-assigned this Mar 5, 2020
morazow
morazow suggested changes Mar 6, 2020
View reviewed changes
docs/kinesis/kinesis_connector.md Outdated Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Outdated Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Outdated Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Show resolved Hide resolved
@morazow morazow self-requested a review March 6, 2020 08:26
morazow
morazow suggested changes Mar 9, 2020
View reviewed changes
src/main/scala/com/exasol/cloudetl/kinesis/KinesisPropertiesReader.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisPropertiesReader.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/util/JsonDeserializer.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisImportQueryGenerator.scala Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisShardsMetadataReader.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporter.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporter.scala Outdated Show resolved Hide resolved
@morazow morazow self-requested a review March 11, 2020 10:06
@morazow morazow force-pushed the feature/74_create_initial_implementation_of_kinesis_connector branch 2 times, most recently from 72c5b06 to e9b971e Compare March 16, 2020 21:48
morazow
morazow suggested changes Mar 20, 2020
View reviewed changes
src/it/scala/com/exasol/cloudetl/kinesis/KinesisAbstractIntegrationTest.scala Outdated Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisImportQueryGeneratorIT.scala Outdated Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisImportQueryGeneratorIT.scala Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisAbstractIntegrationTest.scala Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisImportQueryGeneratorIT.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisClientFactory.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisClientFactory.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisUserProperties.scala Outdated Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisAbstractIntegrationTest.scala Outdated Show resolved Hide resolved
.travis.yml Show resolved Hide resolved
morazow
morazow suggested changes Mar 23, 2020
View reviewed changes
src/main/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporter.scala Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisUserProperties.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporter.scala Outdated Show resolved Hide resolved
src/test/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporterTest.scala Outdated Show resolved Hide resolved
src/test/scala/com/exasol/cloudetl/common/AbstractPropertiesTest.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisUserProperties.scala Outdated Show resolved Hide resolved
src/test/scala/com/exasol/cloudetl/common/AbstractPropertiesTest.scala Outdated Show resolved Hide resolved
src/main/scala/com/exasol/cloudetl/kinesis/KinesisImportQueryGenerator.scala Outdated Show resolved Hide resolved
morazow
morazow suggested changes Mar 24, 2020
View reviewed changes
src/it/scala/com/exasol/cloudetl/kinesis/KinesisAbstractIntegrationTest.scala Outdated Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisAbstractIntegrationTest.scala Outdated Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisImportQueryGeneratorIT.scala Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporterIT.scala Outdated Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporterIT.scala Outdated Show resolved Hide resolved
src/it/scala/com/exasol/cloudetl/kinesis/KinesisShardsMetadataReaderIT.scala Show resolved Hide resolved
src/test/scala/com/exasol/cloudetl/kinesis/KinesisShardDataImporterTest.scala Outdated Show resolved Hide resolved
redcatbear
redcatbear requested changes Mar 25, 2020
View reviewed changes
README.md Show resolved Hide resolved
docs/developer_guide.md Outdated Show resolved Hide resolved
docs/developer_guide.md Outdated Show resolved Hide resolved
docs/developer_guide.md Outdated Show resolved Hide resolved
docs/kinesis/design.md Outdated Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Outdated Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Outdated Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Outdated Show resolved Hide resolved
docs/kinesis/kinesis_connector.md Show resolved Hide resolved
docs/kinesis/kinesis_connector.md
Provide credentials as properties and run an import query.

**Attention! Providing credentials via propeties is
deprecated and will be removed in future releases.**
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a deprecation warning to the central README too. Prominently, so that users have no excuse to not read it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will do it later

@AnastasiiaSergienko AnastasiiaSergienko marked this pull request as ready for review March 25, 2020 14:30
morazow
morazow approved these changes Mar 26, 2020
View reviewed changes
Copy link
Contributor

@morazow morazow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍
Great work! Thanks for getting this implemented and finalized!

README.md
@@ -109,7 +111,7 @@ These plugins help with project development.
| [SBT Git][sbt-git-link] | A plugin for Git integration, used to version the release jars | BSD 2-Clause License |

[travis-badge]: https://img.shields.io/travis/exasol/cloud-storage-etl-udfs/master.svg?logo=travis
[travis-link]: https://travis-ci.org/exasol/cloud-storage-etl-udfs
[travis-link]: https://travis-ci.com/exasol/cloud-storage-etl-udfs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

docs/kinesis/kinesis_connector.md
CREATE OR REPLACE JAVA SET SCRIPT KINESIS_METADATA (...)
EMITS (KINESIS_SHARD_ID VARCHAR(130), SHARD_SEQUENCE_NUMBER VARCHAR(2000)) AS
%scriptclass com.exasol.cloudetl.kinesis.KinesisShardsMetadataReader;
%jar /buckets/bfsdefault/kinesis/cloud-storage-etl-udfs-<VERSION>.jar;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will not work. The version is obtained using git describe. So every commit has a version, e.,g tag-commitNumberSince-sha.

Also, I think it will be too much. User can obtain the latest released jar from github releases.

src/it/scala/com/exasol/cloudetl/kinesis/KinesisImportQueryGeneratorIT.scala Show resolved Hide resolved
@AnastasiiaSergienko AnastasiiaSergienko force-pushed the feature/74_create_initial_implementation_of_kinesis_connector branch from c2f226f to 2d12b55 Compare March 26, 2020 09:32
AnastasiiaSergienko and others added 3 commits March 26, 2020 10:36
@AnastasiiaSergienko @morazow
Adds an implementation of the Kinesis Connector Importer.
a6a77df 
Adds an initial implementation using AWS SDK, unit tests and JSON API
for deserializing Kinesis Stream content.

Added users guide, system requirements and design documentation.

Applied pull request reviews.

Co-Authored-By: Muhammet Orazov <m.orazow@gmail.com>
@AnastasiiaSergienko @redcatbear
Adds integration tests for Kinesis.
acf1622 
Porting the project to Java 11 due to an introduction of
Exasol docker test-containers framework for integration testing.

Integration tests are based on the communication between Exasol
test-containers and Localstack with Kinesis Service.

Adds Mockito Extension because we were getting NullPointerExceptions
when mocking the final methods of a class. We learned that by default
mockito does not support mocking final methods. In order to enable it,
mockito extension should be added to the test resources.

Apply suggestions from code review.

References:

* https://github.com/exasol/exasol-testcontainers
* https://www.testcontainers.org/modules/localstack/
* https://github.com/mockito/mockito/wiki/What%27s-new-in-Mockito-2#mock-the-unmockable-opt-in-mocking-of-final-classesmethods

Co-Authored-By: Sebastian Bär <sebastian.baer@exasol.com>
@AnastasiiaSergienko
Changes JSON Jackson Scala Module dependencies to older version becau…
a5f3b6b 
…se Delta format requires older version.
@AnastasiiaSergienko AnastasiiaSergienko force-pushed the feature/74_create_initial_implementation_of_kinesis_connector branch from 2d12b55 to a5f3b6b Compare March 26, 2020 09:56
@AnastasiiaSergienko AnastasiiaSergienko merged commit 3774b32 into master Mar 27, 2020
@AnastasiiaSergienko AnastasiiaSergienko deleted the feature/74_create_initial_implementation_of_kinesis_connector branch March 27, 2020 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@redcatbear redcatbear redcatbear requested changes

@morazow morazow morazow approved these changes

Assignees

@AnastasiiaSergienko AnastasiiaSergienko

Labels
feature Product feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create an initial implementation of a Kinesis Connector
3 participants
@AnastasiiaSergienko @redcatbear @morazow