Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MQ V2 Sink OSS release #65

Merged
merged 13 commits into from
Aug 7, 2024
Merged

MQ V2 Sink OSS release #65

merged 13 commits into from
Aug 7, 2024

Conversation

priyankaku
Copy link
Contributor

@priyankaku priyankaku commented Jul 24, 2024 •
edited
Loading

Description

Release MQ v2 sink connector under apache 2.0 license.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • E2E
  • Unit Test
  • Integration Test

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

jhughes24816 and others added 12 commits June 8, 2023 11:06
@jhughes24816
Merge pull request ibm-messaging#1 from TechnologyGarageUKI/only-once
3db70d5 
Upstream changes from sink connector
feat: travis script to login to docker registry (ibm-messaging#25)
e5680a9 
Contributes to: event-integration/qp-planning#10883

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
fix: try running the travis job without docker login (ibm-messaging#26)
56ba356 
Contributes to: event-integration/qp-planning#10883

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
@jvulloibm @nicholasmorgan @jhughes24816
Exactly-once message delivery semantics support (ibm-messaging#9)
15d6ff4 
* fix: Move config to MQSinkConfig from MQSinkConnector

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* tests: New tests to check the duplicates

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* - Added assertj
- refactored duplicate record test + added another for connect crash.

* - Changed AuthIt tests to have specific host ports.
- Refactored testcontainer out from DuplicateIT into AbstractJMSContextIT

* feat: Make topic name and partition variables

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* Updated logging packages

* feat: Intitial change for exactly once semantics

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: handle unprovided sidequeue scenario

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Intitial change for exactly once semantics

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: checkstyle changes

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joel.hanson025@gmail.com>

* fix: Static variables should be in UPPER_CASE

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Latest changes for mq sink connector

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update config property names and consume messages as TextMessage

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* Removed a error being sent to standard out.

* feat: Test update based on feedback

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Rename JMSWriter to JMSWorker

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: map based only once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: tests for only-once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: check if the max tasks is 1 and throw an exception if not

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update changes based on feedback from PR

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Remove max task check from connector code

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson02@gmail.com>

* feat: exception testing for only-once

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* fix: Tests to check JMSRuntimeException and jsonException is only thrown

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* fix: failing tests

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* Updated MQ to remove vulnerability

* feat: Rename JMSWriter to JMSWorker

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: map based only once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: tests for only-once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: check if the max tasks is 1 and throw an exception if not

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update changes based on feedback from PR

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Remove max task check from connector code

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson02@gmail.com>

* feat: Check max task if the connector is configured with side queue

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson02@gmail.com>

* fix: rename writer to worker

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update config value names to match MQ_EXACTLY_ONCE_STATE_QUEUE

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update state queue exception handling

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Changes with the new configSupportsExactlyOnce method

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update version to 2.0.0 (ibm-messaging#14)

* feat: Update version to 1.6.0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update version to 2.0.0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: New tests for atleast once behaviour with failing conditions. (ibm-messaging#18)

* feat: New tests for only once to recreate faliure

Contributes to: mhub/qp-planning#10748

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: fix licence formatting and new line at the end

Contributes to: mhub/qp-planning#10748

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Add missing Legal disclaimer (ibm-messaging#19)

* feat: Add missing Legal disclaimer

Contributes to: mhub/qp-planning#<issue number>

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Add current year to the legal disclaimer

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Add LICENSES

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Updated to latest LICENSES

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: Move all the common test variables to a single class (ibm-messaging#20)

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* fix: Review comment fixes (ibm-messaging#21)

* fix: rename variable names

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: simplify the jsonprocessingexception failure

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Rename variable names and create method for readablity

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Move SinkRecordBuilder to utils folder

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Add missing final

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: enable exactlyOnce to be true if tasks.max is not provided

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: spelling for commited -> committed

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: early exits are traced

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Refactor handleException method

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Update version in BUG_REPORT

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Rename class SinkRecordBuilder and remove temp variables

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: tidy up in test for clarity

Contributes to: event-integration/eventstreams-planning#10774

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

Co-authored-by: Jonathan Hughes <Jonathan.Hughes@ibm.com>

* feat: Update exception handling similar to source connector (ibm-messaging#22)

* fix: Update connectionException to JMSWorkerConnectionException

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Log stack trace instead of printStackTrace

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: split out large part of jmswoker into a seperate class (ibm-messaging#23)

* refactor: split out large part of jmswoker into a seperate class

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: connect code is refactored have common method

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Move SSLContext related part to seperate class

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Rename method name to be more meaningful

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Remove unwanted imports and java docs

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: Exception handling

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: simplify the put method

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: handle exception from top and clean up

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Formatting code, sorting imports, adding new tests for exceptions (ibm-messaging#24)

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* Update kafka binary to a working version (ibm-messaging#15)

* feat: Formatting and updation of readme for exactly once configuration (ibm-messaging#27)

* feat: Formatting and updation of read for exactly once configuration

- .gitignore update to have the mq logs
- Formatted the readme file
- Added new documenation for the exactly once semantics
- New sample file for the exactly-once connector config

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Update 3.4.0 to 3.4.1

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Update README, add license declaration, remove AbstractConfig (ibm-messaging#28)

* fix: Update README, add license declaration, remove AbstractConfig

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Change DestinationBuilderIT to MessagePropertyIT

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: attempt to fix e2es (ibm-messaging#29)

* feat: attempt to fix e2es

Contributes to: event-integration/eventstreams-planning#0

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

* feat: fixed tests

Contributes to: event-integration/eventstreams-planning#0

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

* feat: revert from arm64 image from tests

Contributes to: event-integration/eventstreams-planning#0

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

* fix: Update required version from 2.6.0 to 0.11.0 (ibm-messaging#30)

Contributes to: event-integration/qp-planning#0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Update test to have meaning full variable names (ibm-messaging#31)

Contributes to: event-integration/qp-planning#0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

Co-authored-by: Joel Hanson <joel.hanson2@ibm.com>
Co-authored-by: Nick Morgan <nicholas.morgan@ibm.com>
Co-authored-by: Jonathan Hughes <Jonathan.Hughes@ibm.com>
@neeraj-laad
fix: Update packaged licenses (ibm-messaging#32)
c9f56b8 
* fix: Update packaged licenses

Update packaged licenses

Contributes to: event-integration/eventstreams-planning#11171

Signed-off-by: Neeraj Laad <neeraj.laad@uk.ibm.com>
Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com>

* fix: Include license in pom.xml

Include license in pom.xml

Contributes to: event-integration/eventstreams-planning#11171

Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com>

* fix: Move license folder under META-INF

Move license folder under META-INF

Contributes to: event-integration/eventstreams-planning#11171

Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com>
@neeraj-laad
doc: fix license in readme (ibm-messaging#33)
b4b3ce9 
fix license in readme

Contributes to: event-integration/eventstreams-planning#11171

Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com>
@dalelane
Mq sink refactor (ibm-messaging#35)
64a724b 
* feat: MQ Sink Refactor

Contributes to: event-integration/eventstreams-planning#0
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com

* feat: Test cases fixed

Contributes to: event-integration/eventstreams-planning#12532
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com

* feat: Test cases updated

Contributes to: event-integration/eventstreams-planning#12532
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com

* feat: updated the import

Contributes to: event-integration/eventstreams-planning#12532
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com

* feat: added copyright header

Contributes to: event-integration/eventstreams-planning#12532
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com

* chore: remove commented-out code

Signed-off-by: Dale Lane <dale.lane@uk.ibm.com>

---------

Signed-off-by: Dale Lane <dale.lane@uk.ibm.com>
Co-authored-by: Dale Lane <Dale.Lane@uk.ibm.com>
feat: support for writing to MQMD (ibm-messaging#36)
b5282cf 
Contributes to: event-integration/eventstreams-planning#12432

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
@vigneshibmpub
feat: Update readme with appropriate mqmd info (ibm-messaging#37)
a443099 
* feat: Update readme with appropriate mqmd info

Contributes to: event-integration/eventstreams-planning#12434

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* Apply suggestions from code review

Contributes to: event-integration/eventstreams-planning#12434

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

Co-authored-by: Vignesh Kumar Elumalai <vigneshkumar.elumalai@ibm.com>

* feat: rearrange order of docs in readme

Contributes to: event-integration/eventstreams-planning#12434

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: copyright year change and readme case fix

Contributes to: event-integration/eventstreams-planning#12434

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* change to note

* remove specifically

* Apply suggestions from code review

---------

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Co-authored-by: Vignesh Kumar Elumalai <vigneshkumar.elumalai@ibm.com>
feat: Update release to 2.1.0 (ibm-messaging#40)
b86b474 
Contributes to: event-integration/eventstreams-planning#12434

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
feat: Automate release using travis (ibm-messaging#39)
b0b28ae 
Contributes to: event-integration/eventstreams-planning#12621

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Merge remote-tracking branch 'event-integration/kafka-connect-mq-sink…
47b32ad 
…/master' into mqv2-sink-ossrelease
@priyankaku priyankaku marked this pull request as ready for review July 25, 2024 16:18
@Joel-hanson Joel-hanson self-requested a review August 6, 2024 03:42
Joel-hanson
Joel-hanson requested changes Aug 6, 2024
View reviewed changes
pom.xml Outdated Show resolved Hide resolved
pom.xml Outdated Show resolved Hide resolved
feat: updated Copyright year
ec7a4c4 
Contributes to: event-integration/eventstreams-planning#13003
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com

feat: updated  version in pom xml

Contributes to: event-integration/eventstreams-planning#13003
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com

feat: MQ Sink V2 OSS related changes

Contributes to: event-integration/eventstreams-planning#13003
Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com
@priyankaku priyankaku force-pushed the mqv2-sink-ossrelease branch from fec3967 to ec7a4c4 Compare August 7, 2024 04:55
Joel-hanson
Joel-hanson reviewed Aug 7, 2024
View reviewed changes
src/main/java/com/ibm/eventstreams/connect/mqsink/MQSinkTask.java
Comment on lines +109 to +117
* Put the records in the sink. Usually this should send the records to the sink
* asynchronously and immediately return.
* <p>
* If this operation fails, the SinkTask may throw a
* {@link org.apache.kafka.connect.errors.RetriableException} to indicate that
* the framework should attempt to retry the same call again. Other exceptions
* will cause the task to be stopped immediately.
* {@link SinkTaskContext#timeout(long)} can be used to set the maximum time
* before the batch will be retried.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticed this, all the comments are in a difficult to read manner.

Joel-hanson
Joel-hanson approved these changes Aug 7, 2024
View reviewed changes
Copy link
Contributor

@Joel-hanson Joel-hanson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@priyankaku priyankaku merged commit a187590 into ibm-messaging:main Aug 7, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Joel-hanson Joel-hanson Joel-hanson approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@priyankaku @Joel-hanson @jhughes24816 @jvulloibm @neeraj-laad