Acknowledge on message delivery from Kafka to MQ #36
Comments
|
A gentle bump! Would really appreciate any insights on this question. |
|
Hi, I'm afraid not. The Kafka producer and consumer APIs are hidden away in the Kafka Connect runtime. |
|
Thanks, @AndrewJSchofield for letting me know. |
priyankaku
pushed a commit
to priyankaku/kafka-connect-mq-sink
that referenced
this issue
Jul 24, 2024
Contributes to: event-integration/eventstreams-planning#12432 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
priyankaku
added a commit
that referenced
this issue
Aug 7, 2024
* feat: travis script to login to docker registry (#25) Contributes to: event-integration/qp-planning#10883 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: try running the travis job without docker login (#26) Contributes to: event-integration/qp-planning#10883 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * Exactly-once message delivery semantics support (#9) * fix: Move config to MQSinkConfig from MQSinkConnector Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * tests: New tests to check the duplicates Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * - Added assertj - refactored duplicate record test + added another for connect crash. * - Changed AuthIt tests to have specific host ports. - Refactored testcontainer out from DuplicateIT into AbstractJMSContextIT * feat: Make topic name and partition variables Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * Updated logging packages * feat: Intitial change for exactly once semantics Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: handle unprovided sidequeue scenario Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Intitial change for exactly once semantics Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: checkstyle changes Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joel.hanson025@gmail.com> * fix: Static variables should be in UPPER_CASE Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Latest changes for mq sink connector Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Update config property names and consume messages as TextMessage Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * Removed a error being sent to standard out. * feat: Test update based on feedback Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Rename JMSWriter to JMSWorker Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: map based only once delivery Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: tests for only-once delivery Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: check if the max tasks is 1 and throw an exception if not Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Update changes based on feedback from PR Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Remove max task check from connector code Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson02@gmail.com> * feat: exception testing for only-once Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * fix: Tests to check JMSRuntimeException and jsonException is only thrown Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * fix: failing tests Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * Updated MQ to remove vulnerability * feat: Rename JMSWriter to JMSWorker Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: map based only once delivery Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: tests for only-once delivery Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: check if the max tasks is 1 and throw an exception if not Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Update changes based on feedback from PR Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Remove max task check from connector code Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson02@gmail.com> * feat: Check max task if the connector is configured with side queue Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson02@gmail.com> * fix: rename writer to worker Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Update config value names to match MQ_EXACTLY_ONCE_STATE_QUEUE Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Update state queue exception handling Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Changes with the new configSupportsExactlyOnce method Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Update version to 2.0.0 (#14) * feat: Update version to 1.6.0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Update version to 2.0.0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: New tests for atleast once behaviour with failing conditions. (#18) * feat: New tests for only once to recreate faliure Contributes to: mhub/qp-planning#10748 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: fix licence formatting and new line at the end Contributes to: mhub/qp-planning#10748 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Add missing Legal disclaimer (#19) * feat: Add missing Legal disclaimer Contributes to: mhub/qp-planning#<issue number> Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Add current year to the legal disclaimer Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * feat: Add LICENSES Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Updated to latest LICENSES Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * refactor: Move all the common test variables to a single class (#20) Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: Joel Hanson <joelhanson025@gmail.com> * fix: Review comment fixes (#21) * fix: rename variable names Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: simplify the jsonprocessingexception failure Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Rename variable names and create method for readablity Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Move SinkRecordBuilder to utils folder Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Add missing final Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: enable exactlyOnce to be true if tasks.max is not provided Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: spelling for commited -> committed Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: early exits are traced Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Refactor handleException method Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Update version in BUG_REPORT Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Rename class SinkRecordBuilder and remove temp variables Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: tidy up in test for clarity Contributes to: event-integration/eventstreams-planning#10774 Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com> Co-authored-by: Jonathan Hughes <Jonathan.Hughes@ibm.com> * feat: Update exception handling similar to source connector (#22) * fix: Update connectionException to JMSWorkerConnectionException Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Log stack trace instead of printStackTrace Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * refactor: split out large part of jmswoker into a seperate class (#23) * refactor: split out large part of jmswoker into a seperate class Contributes to: mhub/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * refactor: connect code is refactored have common method Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Move SSLContext related part to seperate class Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Rename method name to be more meaningful Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Remove unwanted imports and java docs Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * refactor: Exception handling Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * refactor: simplify the put method Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: handle exception from top and clean up Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Formatting code, sorting imports, adding new tests for exceptions (#24) Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * Update kafka binary to a working version (#15) * feat: Formatting and updation of readme for exactly once configuration (#27) * feat: Formatting and updation of read for exactly once configuration - .gitignore update to have the mq logs - Formatted the readme file - Added new documenation for the exactly once semantics - New sample file for the exactly-once connector config Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Update 3.4.0 to 3.4.1 Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * fix: Update README, add license declaration, remove AbstractConfig (#28) * fix: Update README, add license declaration, remove AbstractConfig Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Change DestinationBuilderIT to MessagePropertyIT Contributes to: event-integration/qp-planning#10774 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: attempt to fix e2es (#29) * feat: attempt to fix e2es Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com> * feat: fixed tests Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com> * feat: revert from arm64 image from tests Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com> * fix: Update required version from 2.6.0 to 0.11.0 (#30) Contributes to: event-integration/qp-planning#0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Update test to have meaning full variable names (#31) Contributes to: event-integration/qp-planning#0 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Co-authored-by: Joel Hanson <joel.hanson2@ibm.com> Co-authored-by: Nick Morgan <nicholas.morgan@ibm.com> Co-authored-by: Jonathan Hughes <Jonathan.Hughes@ibm.com> * fix: Update packaged licenses (#32) * fix: Update packaged licenses Update packaged licenses Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: Neeraj Laad <neeraj.laad@uk.ibm.com> Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> * fix: Include license in pom.xml Include license in pom.xml Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> * fix: Move license folder under META-INF Move license folder under META-INF Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> * doc: fix license in readme (#33) fix license in readme Contributes to: event-integration/eventstreams-planning#11171 Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> * Mq sink refactor (#35) * feat: MQ Sink Refactor Contributes to: event-integration/eventstreams-planning#0 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com * feat: Test cases fixed Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com * feat: Test cases updated Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com * feat: updated the import Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com * feat: added copyright header Contributes to: event-integration/eventstreams-planning#12532 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com * chore: remove commented-out code Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> --------- Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> Co-authored-by: Dale Lane <Dale.Lane@uk.ibm.com> * feat: support for writing to MQMD (#36) Contributes to: event-integration/eventstreams-planning#12432 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Update readme with appropriate mqmd info (#37) * feat: Update readme with appropriate mqmd info Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * Apply suggestions from code review Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Co-authored-by: Vignesh Kumar Elumalai <vigneshkumar.elumalai@ibm.com> * feat: rearrange order of docs in readme Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: copyright year change and readme case fix Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * change to note * remove specifically * Apply suggestions from code review --------- Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Co-authored-by: Vignesh Kumar Elumalai <vigneshkumar.elumalai@ibm.com> * feat: Update release to 2.1.0 (#40) Contributes to: event-integration/eventstreams-planning#12434 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: Automate release using travis (#39) Contributes to: event-integration/eventstreams-planning#12621 Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> * feat: updated Copyright year Contributes to: event-integration/eventstreams-planning#13003 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: updated version in pom xml Contributes to: event-integration/eventstreams-planning#13003 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com feat: MQ Sink V2 OSS related changes Contributes to: event-integration/eventstreams-planning#13003 Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com --------- Signed-off-by: Joel Hanson <joel.hanson2@ibm.com> Signed-off-by: neeraj-laad <neeraj.laad@uk.ibm.com> Signed-off-by: Dale Lane <dale.lane@uk.ibm.com> Signed-off-by: Priyanka.K.U Priyanka.k.u@ibm.com Co-authored-by: Jonathan Hughes <Jonathan.Hughes@ibm.com> Co-authored-by: Joel Hanson <Joel.Hanson2@ibm.com> Co-authored-by: Joseph Vullo <Joseph.Vullo@ibm.com> Co-authored-by: Nick Morgan <nicholas.morgan@ibm.com> Co-authored-by: Neeraj Laad <neeraj.laad@uk.ibm.com> Co-authored-by: Priyanka K U <Priyanka.K.U@ibm.com> Co-authored-by: Dale Lane <Dale.Lane@uk.ibm.com> Co-authored-by: Vignesh Kumar Elumalai <vigneshkumar.elumalai@ibm.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi!
I am just familiarising myself with the MQ sink and source connectors. I want to thank you for this project and for all your efforts.
I have a question, is there someway I can get an acknowledge that the message has been actually sent to or delivered IBM MQ from Kafka? Maybe using the topic offset?
For example, when I do an
MQ PUTusing the mq-mqi-nodejs npm module I know for sure whether the request failed or not along with Message ID in the MQ Message Descriptor:https://github.com/ibm-messaging/mq-mqi-nodejs/blob/932f5768e48cebe11697470c3fbdc46bea8a43ba/samples/amqsput.js#L66-L71
Thanks!
The text was updated successfully, but these errors were encountered: