[WIP] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub #1

KhaninArtur · 2020-10-02T14:56:42Z

[Proposal] Apache Beam Template to ingest data from Apache Kafka to Google Cloud Pub/Sub. It can be used as a Dataflow Flex template in the Google Cloud Platform.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

…ecret storage with url and token

…ctored Kafka configuration

…formations from the pipeline class into the separate class

Co-authored-by: Brian Hulette <hulettbh@gmail.com>

…alueProvider for Python SDK * Support for NestedValueProvider for Python SDK * Fix typo * Update CHANGES.md * Update value_provider_test.py * Fix NestedValueProvider docstrings. (#1) * Fix isort and doc errors. (#2) * Update CHANGES.md Co-authored-by: Eugene Nikolaiev <eugene.nikolayev@gmail.com>

update

Debeziumio PoC (#7) * New DebeziumIO class. * Merge connector code * DebeziumIO and MySqlConnector integrated. * Added FormatFuntion param to Read builder on DebeziumIO. * Added arguments checker to DebeziumIO. * Add simple JSON mapper object (#1) * Add simple JSON mapper object * Fixed Mapper. * Add SqlServer connector test * Added PostgreSql Connector Test PostgreSql now works with Json mapper * Added PostgreSql Connector Test PostgreSql now works with Json mapper * Fixing MySQL schema DataException Using file instead of schema should fix it * MySQL Connector updated from 1.3.0 to 1.3.1 Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com> Co-authored-by: Carlos Dominguez <carlos.dominguez@carlos.dominguez> Co-authored-by: Carlos Domínguez <carlos.dominguez@wizeline.com> * Add debeziumio tests * Debeziumio testing json mapper (#3) * Some code refactors. Use a default DBHistory if not provided * Add basic tests for Json mapper * Debeziumio time restriction (#5) * Add simple JSON mapper object * Fixed Mapper. * Add SqlServer connector test * Added PostgreSql Connector Test PostgreSql now works with Json mapper * Added PostgreSql Connector Test PostgreSql now works with Json mapper * Fixing MySQL schema DataException Using file instead of schema should fix it * MySQL Connector updated from 1.3.0 to 1.3.1 * Some code refactors. Use a default DBHistory if not provided * Adding based-time restriction Stop polling after specified amount of time * Add basic tests for Json mapper * Adding new restriction Uses a time-based restriction * Adding optional restrcition Uses an optional time-based restriction Co-authored-by: juanitodread <juanitodread@gmail.com> Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com> * Upgrade DebeziumIO connector (#4) * Address comments (Change dependencies to testCompile, Set JsonMapper/Coder as default, refactors) (#8) * Revert file * Change dependencies to testCompile * Move Counter sample to unit test * Set JsonMapper as default mapper function * Set String Coder as default coder when using JsonMapper * Change logs from info to debug * Debeziumio javadoc (#9) * Adding javadoc * Added some titles and examples * Added SourceRecordJson doc * Added Basic Connector doc * Added KafkaSourceConsumer doc * Javadoc cleanup * Removing BasicConnector No usages of this class were found overall * Editing documentation * Debeziumio fetched records restriction (#10) * Adding javadoc * Adding restriction by number of fetched records Also adding a quick-fix for null value within SourceRecords Minor fix on both MySQL and PostgreSQL Connectors Tests * Run either by time or by number of records * Added DebeziumOffsetTrackerTest Tests both restrictions: By amount of time and by Number of records * Removing comment * DebeziumIO test for DB2. (#11) * DebeziumIO test for DB2. * DebeziumIO javadoc. * Clean code:removed commented code lines on DebeziumIOConnectorTest.java * Clean code:removing unused imports and using readAsJson(). Co-authored-by: Carlos Domínguez <74681048+carlosdominguezwl@users.noreply.github.com> * Debezium limit records (now configurable) (#12) * Adding javadoc * Records Limit is now configurable (It was fixed before) * Debeziumio dockerize (#13) * Add mysql docker container to tests * Move debezium mysql integration test to its own file * Add assertion to verify that the results contains a record. * Debeziumio readme (#15) * Adding javadoc * Adding README file * Add number of records configuration to the DebeziumIO component (#16) * Code refactors (#17) * Remove/ignore null warnings * Remove DB2 code * Remove docker dependency in DebeziumIO unit test and max number of recods to MySql integration test * Change access modifiers accordingly * Remove incomplete integration tests (Postgres and SqlServer) * Add experimenal tag * Debezium testing stoppable consumer (#18) * Add try-catch-finally, stop SourceTask at finally. * Fix warnings * stopConsumer and processedRecords local variables removed. UT for task stop use case added * Fix minor code style issue Co-authored-by: juanitodread <juanitodread@gmail.com> * Fix style issues (check, spotlessApply) (#19) Co-authored-by: Osvaldo Salinas <osvaldo.salinas@osvaldo.salinas> Co-authored-by: alejandro.maguey <alejandro.maguey@wizeline.com> Co-authored-by: osvaldo-salinas <osvaldo.salinas@wizeline.com> Co-authored-by: Carlos Dominguez <carlos.dominguez@carlos.dominguez> Co-authored-by: Carlos Domínguez <carlos.dominguez@wizeline.com> Co-authored-by: Carlos Domínguez <74681048+carlosdominguezwl@users.noreply.github.com> Co-authored-by: Alejandro Maguey <alexmaguey1@gmail.com> Co-authored-by: Hassan Reyes <hassanreyes@users.noreply.github.com> Add missing apache license to README.md Enabling integration test for DebeziumIO (#20) Rename connector package cdc=>debezium. Update doc references (#21) Fix code style on DebeziumIOMySqlConnectorIT

Adds ReadChangeStreamPartitionDoFn, which is an SDF to read partitions from change streams and process them accordingly. This component receives a change stream name, a partition, a start time and an end time to query. It then initiates a change stream query with the received parameters. Within a change stream, 3 types of records can be received: 1. A Data record 2. A Heartbeat record 3. A Child partitions record Upon receiving #1, the function updates the watermark with the record's commit timestamp and emits the record into the output PCollection. Upon receiving #2, the function updates the watermark with the record's timestamp, but it does not emit any record into the PCollection. Upon receiving #3, the function updates the watermark with the record's timestamp and writes the new child partitions into the metadata table. These partitions will be later scheduled by the DetectNewPartitions component. Once the change stream query for the element partition finishes, it marks the partition as finished in the metadata table and terminates.

Add initial test

* theme setup * Replaced ThemeProvider with ThemeSwitchNotifier * header with theme mode switcher and logo * page container with header & footer * theme mode tests * renamed the directory to tour-of-beam * compressed beam_logo.png * added missing license comments * rudimentary layout of the first screen * review comments fixes #1 * moved notifyListeners inside then * responsive todo * split into 2 simple functions * deleted redundant constants & replaced 2018 text theme with 2021 * styling refinement * home screen layout * clickable sign in text * font weights fix * removed _getBaseFontTheme function * fixed border and bg color * color fixes * difficulty component * _LastModuleBody * todo in test * footer border * fixed overflows * replaced Project prefix with Tob * replaced then with await * inferred type * started translation of the home screen * sorted translations * Complexity comments * comment fixes * home screen translations * sign in overlay * import fix * integration test does not fail * playground_components package with dismissible_overlay * missing license * removed _dots from build * widgets refinement * renamed home screen to welcome screen * deleted copyWith * _SdkButton * trailing comma & pubspec formatting * license and lints * license * removed license from .metadata * pubspec formatting * total lints update * changed from tour_of_beam to tour-of-beam in build.gradle.kts * license check * _SdkButton mimics Radio button * renamed MyApp to TourOfBeamApp * onChanged mimics Radio button Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>

* Tour of Beam frontend blank project * TOBF (ToB frontend): welcome screen (#226) * theme setup * Replaced ThemeProvider with ThemeSwitchNotifier * header with theme mode switcher and logo * page container with header & footer * theme mode tests * renamed the directory to tour-of-beam * compressed beam_logo.png * added missing license comments * rudimentary layout of the first screen * review comments fixes #1 * moved notifyListeners inside then * responsive todo * split into 2 simple functions * deleted redundant constants & replaced 2018 text theme with 2021 * styling refinement * home screen layout * clickable sign in text * font weights fix * removed _getBaseFontTheme function * fixed border and bg color * color fixes * difficulty component * _LastModuleBody * todo in test * footer border * fixed overflows * replaced Project prefix with Tob * replaced then with await * inferred type * started translation of the home screen * sorted translations * Complexity comments * comment fixes * home screen translations * sign in overlay * import fix * integration test does not fail * playground_components package with dismissible_overlay * missing license * removed _dots from build * widgets refinement * renamed home screen to welcome screen * deleted copyWith * _SdkButton * trailing comma & pubspec formatting * license and lints * license * removed license from .metadata * pubspec formatting * total lints update * changed from tour_of_beam to tour-of-beam in build.gradle.kts * license check * _SdkButton mimics Radio button * renamed MyApp to TourOfBeamApp * onChanged mimics Radio button Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> * removed whitespace from readme (issue-22583) * renamed "content" to "child" to mimic widgets * README in tour-of-beam * translation path rename, grey dot with opacity, footer link text style * report issue in github, grey dot color * table instead of row to clip the laptop image * horizontalHalves & verticalHalves * cropped laptop image * row in an expensive intrinsic height * laptop image in the bottom * ScreenBreakpoints * intrinsic height is not needed! * _WideWelcome and _NarrowWelcome * draft readme * blank line in readme * removed irrelevant info from readme * removed whitespace Co-authored-by: Alexey Inkin <alexey.inkin@akvelon.com> Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Co-authored-by: alexeyinkin <leha@inkin.ru>

* theme setup * Replaced ThemeProvider with ThemeSwitchNotifier * header with theme mode switcher and logo * page container with header & footer * theme mode tests * renamed the directory to tour-of-beam * compressed beam_logo.png * added missing license comments * rudimentary layout of the first screen * review comments fixes #1 * moved notifyListeners inside then * responsive todo * split into 2 simple functions * deleted redundant constants & replaced 2018 text theme with 2021 * styling refinement * home screen layout * clickable sign in text * font weights fix * removed _getBaseFontTheme function * fixed border and bg color * color fixes * difficulty component * _LastModuleBody * todo in test * footer border * fixed overflows * replaced Project prefix with Tob * replaced then with await * inferred type * started translation of the home screen * sorted translations * Complexity comments * comment fixes * home screen translations * sign in overlay * import fix * integration test does not fail * playground_components package with dismissible_overlay * missing license * removed _dots from build * widgets refinement * renamed home screen to welcome screen * deleted copyWith * _SdkButton * trailing comma & pubspec formatting * license and lints * license * removed license from .metadata * pubspec formatting * total lints update * changed from tour_of_beam to tour-of-beam in build.gradle.kts * license check * _SdkButton mimics Radio button * renamed MyApp to TourOfBeamApp * onChanged mimics Radio button Tour of Beam frontend blank project [Tour of Beam][Frontend][apache#22600] TourScreen layout TourScreen layout (apache#22600) common theme, constants, split view missing license flutter_gen, summary layout details content layout details no functional widgets in split view main screen todos & translation main screen todos & translation comment fixes #1 ExpansionTileWrapper SplitViewController lists in tour screen widgets comment fixes #1 (31.08) split view package in PGC fixed button overflow splitter theme color comment fixes #2 (31.08) gradlew check welcome screen overflow test (apache#22600) SDK dropdown (apache#22600) flexible complete unit OutlinedButton (apache#22600) renamed PageContainer to TobScaffold dropdown style refinement DropdownButton implicit type sdk instead of e licenses apache#22600 renamed _ShrinkedTour to _NarrowTour apache#22600 tour screen style refinement apache#22600 BeamDivider in PGC apache#22600 removed todo, added license apache#22600 built with text apache#22600 _WideWelcome with IntrinsicHeight (apache#22600) Co-Authored-By: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>

* [Tour of Beam][Frontend][apache#22600] TourScreen layout * theme setup * Replaced ThemeProvider with ThemeSwitchNotifier * header with theme mode switcher and logo * page container with header & footer * theme mode tests * renamed the directory to tour-of-beam * compressed beam_logo.png * added missing license comments * rudimentary layout of the first screen * review comments fixes #1 * moved notifyListeners inside then * responsive todo * split into 2 simple functions * deleted redundant constants & replaced 2018 text theme with 2021 * styling refinement * home screen layout * clickable sign in text * font weights fix * removed _getBaseFontTheme function * fixed border and bg color * color fixes * difficulty component * _LastModuleBody * todo in test * footer border * fixed overflows * replaced Project prefix with Tob * replaced then with await * inferred type * started translation of the home screen * sorted translations * Complexity comments * comment fixes * home screen translations * sign in overlay * import fix * integration test does not fail * playground_components package with dismissible_overlay * missing license * removed _dots from build * widgets refinement * renamed home screen to welcome screen * deleted copyWith * _SdkButton * trailing comma & pubspec formatting * license and lints * license * removed license from .metadata * pubspec formatting * total lints update * changed from tour_of_beam to tour-of-beam in build.gradle.kts * license check * _SdkButton mimics Radio button * renamed MyApp to TourOfBeamApp * onChanged mimics Radio button Tour of Beam frontend blank project [Tour of Beam][Frontend][apache#22600] TourScreen layout TourScreen layout (apache#22600) common theme, constants, split view missing license flutter_gen, summary layout details content layout details no functional widgets in split view main screen todos & translation main screen todos & translation comment fixes #1 ExpansionTileWrapper SplitViewController lists in tour screen widgets comment fixes #1 (31.08) split view package in PGC fixed button overflow splitter theme color comment fixes #2 (31.08) gradlew check welcome screen overflow test (apache#22600) SDK dropdown (apache#22600) flexible complete unit OutlinedButton (apache#22600) renamed PageContainer to TobScaffold dropdown style refinement DropdownButton implicit type sdk instead of e licenses apache#22600 renamed _ShrinkedTour to _NarrowTour apache#22600 tour screen style refinement apache#22600 BeamDivider in PGC apache#22600 removed todo, added license apache#22600 built with text apache#22600 _WideWelcome with IntrinsicHeight (apache#22600) Co-Authored-By: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> * addressing review comments apache#22600 replaced magic numbers apache#22600 comments (apache#22600) comments apache#22600 comments apache#22600 comments apache#22600 comments apache#22600 comments, flutter 3.3.0 upgrade apache#22600 renamed ActionPadding to ActionVerticalPadding apache#22600 actions formatting apache#22600 * branded sign in buttons apache#22600 * _BrandedSignInButtons apache#22600 * _Divider color apache#22600 * profile apache#22600 * moved split_view from PGC into ToB apache#22600 * indentation fix apache#22600 * split ProfileContent into widgets apache#22600 * Extract playground components to a separate package (apache#22600) * Minor fixes (apache#22600) * Address review issues (apache#22600) * Upgrade Flutter to v3.3.2 (apache#22600) * Add precommit Gradle task for playground_components, add code generation to frontend Gradle task, remove generated mocks, fix linter issues (apache#22600) * startTour button (apache#22600) * lint fixes (apache#22600) * Fix highlighting for Python and SCIO (apache#22600) Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>

* [prism] Add basic processing time queue. * Initial residual handling refactor. * Re-work teststream initilization. Remove pending element race. * touch up * rm merge duplicate * Simplify watermark hold tracking. * First successful run! * Remove duplicated test run. * Deduplicate processing time heap. * rm debug text * Remove some debug prints, cleanup. * tiny todo cleanup * ProcessingTime workming most of the time! * Some cleanup * try to get github suite to pass #1 * touch * reduce counts a bit, filter tests some. * Clean up unrelated state changes. Clean up comments somewhat. * Filter out dataflow incompatible test. * Refine processing time event comment. * Remove test touch. --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>

ilya-kozyrev and others added 8 commits September 25, 2020 15:52

add initial template and dependencies

45ef009

Added flex template creation with metadata support and instructions

321c715

added new gradle modules for templates

e88fd31

moved metadata to template location, reverted examples build.gradle

ed9be58

Moved KafkaToPubsub to template, implemented options in separate package

e2a8317

Added package-info.java to new packages

4ab8e58

Reverted build.gradle to master branch state

aeb6411

fixed JAVADOC and metadata

61bf992

ilya-kozyrev requested a review from sgrebnov October 2, 2020 15:13

Artur Khanin and others added 8 commits October 5, 2020 16:55

Added the Read Me section with a step-by-step guide

015e6e6

Update README.md

129f466

Readme fixes regarding comments

6db17ad

Update README.md

01e4987

Update README.md

b6decbd

Fixed typos in README.md

8e41d13

refactored README.md added case to run template locally

369a205

Update README.md

01db016

ilya-kozyrev changed the title ~~[WIP] Kafka to PubSub template for GCP~~ [WIP] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub Oct 7, 2020

ilya-kozyrev and others added 12 commits October 8, 2020 15:46

fix build script for dataflow in README.md

5821bdf

Added unit test and fixed metadata file

98de91f

Added Licenses and style fixes

fc1ed9a

Added support for retrieving Kafka credentials from HashiCorp Vault s…

6e159c7

…ecret storage with url and token

Updated README.md and metadata with parameters for Vault access; refa…

523f796

…ctored Kafka configuration

Style fix

e78c02c

Added description for Vault parameters in metadata

1055ec2

FIX trailing whitespaces in README.md

77defc1

FIX. Blank line contains whitespace README.md

d6ab0f6

Update README.md

8881ff3

Refactored to examples folder

a821560

Added conversion from JSON into PubsubMessage and extracted all trans…

c22f110

…formations from the pipeline class into the separate class

ilya-kozyrev and others added 3 commits December 8, 2020 22:46

Update examples/kafka-to-pubsub/README.md

dd85c93

Co-authored-by: Brian Hulette <hulettbh@gmail.com>

Merge branch 'master' into KafkaToPubsubTemplate

89adf0b

Fixed outdated import

51182eb

ramazan-yapparov and others added 6 commits December 9, 2020 17:56

Moved template to examples/complete

a5406a5

Updated paths in readme file

e7fc00e

Updated README.md and javadoc regarding comments

ac76e73

README.md stylefix

b0c4428

Added link to KafkaToPubsub example into complete/README.md

3931fba

Stylefix

21d8b36

ramazan-yapparov pushed a commit that referenced this pull request Jan 29, 2021

Merge pull request #1 from apache/master

5492d32

update

KhaninArtur closed this Oct 15, 2021

bullet03 pushed a commit that referenced this pull request May 8, 2022

Merge pull request #1 from kerrydc/add-initial-test

7a82478

Add initial test

nausharipov pushed a commit that referenced this pull request Aug 9, 2022

review comments fixes #1

d2e60ee

nausharipov pushed a commit that referenced this pull request May 22, 2023

experiment #1

055eaf1

nausharipov pushed a commit that referenced this pull request May 22, 2023

#1

a404057

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub #1

[WIP] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub #1

KhaninArtur commented Oct 2, 2020 •

edited by ilya-kozyrev

Loading

[WIP] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub #1

[WIP] Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub #1

Conversation

KhaninArtur commented Oct 2, 2020 • edited by ilya-kozyrev Loading

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

KhaninArtur commented Oct 2, 2020 •

edited by ilya-kozyrev

Loading