Exactly-once message delivery semantics support (ibm-messaging#9)

* fix: Move config to MQSinkConfig from MQSinkConnector

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* tests: New tests to check the duplicates

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* - Added assertj
- refactored duplicate record test + added another for connect crash.

* - Changed AuthIt tests to have specific host ports.
- Refactored testcontainer out from DuplicateIT into AbstractJMSContextIT

* feat: Make topic name and partition variables

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* Updated logging packages

* feat: Intitial change for exactly once semantics

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: handle unprovided sidequeue scenario

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Intitial change for exactly once semantics

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: checkstyle changes

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joel.hanson025@gmail.com>

* fix: Static variables should be in UPPER_CASE

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Latest changes for mq sink connector

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update config property names and consume messages as TextMessage

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* Removed a error being sent to standard out.

* feat: Test update based on feedback

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Rename JMSWriter to JMSWorker

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: map based only once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: tests for only-once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: check if the max tasks is 1 and throw an exception if not

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update changes based on feedback from PR

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Remove max task check from connector code

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson02@gmail.com>

* feat: exception testing for only-once

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* fix: Tests to check JMSRuntimeException and jsonException is only thrown

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* fix: failing tests

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* Updated MQ to remove vulnerability

* feat: Rename JMSWriter to JMSWorker

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: map based only once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: tests for only-once delivery

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: check if the max tasks is 1 and throw an exception if not

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update changes based on feedback from PR

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Remove max task check from connector code

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson02@gmail.com>

* feat: Check max task if the connector is configured with side queue

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson02@gmail.com>

* fix: rename writer to worker

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update config value names to match MQ_EXACTLY_ONCE_STATE_QUEUE

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update state queue exception handling

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Changes with the new configSupportsExactlyOnce method

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update version to 2.0.0 (ibm-messaging#14)

* feat: Update version to 1.6.0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Update version to 2.0.0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: New tests for atleast once behaviour with failing conditions. (ibm-messaging#18)

* feat: New tests for only once to recreate faliure

Contributes to: mhub/qp-planning#10748

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: fix licence formatting and new line at the end

Contributes to: mhub/qp-planning#10748

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Add missing Legal disclaimer (ibm-messaging#19)

* feat: Add missing Legal disclaimer

Contributes to: mhub/qp-planning#<issue number>

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Add current year to the legal disclaimer

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* feat: Add LICENSES

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Updated to latest LICENSES

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: Move all the common test variables to a single class (ibm-messaging#20)

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>
Signed-off-by: Joel Hanson <joelhanson025@gmail.com>

* fix: Review comment fixes (ibm-messaging#21)

* fix: rename variable names

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: simplify the jsonprocessingexception failure

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Rename variable names and create method for readablity

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Move SinkRecordBuilder to utils folder

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Add missing final

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: enable exactlyOnce to be true if tasks.max is not provided

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: spelling for commited -> committed

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: early exits are traced

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Refactor handleException method

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Update version in BUG_REPORT

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Rename class SinkRecordBuilder and remove temp variables

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: tidy up in test for clarity

Contributes to: event-integration/eventstreams-planning#10774

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

Co-authored-by: Jonathan Hughes <Jonathan.Hughes@ibm.com>

* feat: Update exception handling similar to source connector (ibm-messaging#22)

* fix: Update connectionException to JMSWorkerConnectionException

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Log stack trace instead of printStackTrace

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: split out large part of jmswoker into a seperate class (ibm-messaging#23)

* refactor: split out large part of jmswoker into a seperate class

Contributes to: mhub/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: connect code is refactored have common method

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Move SSLContext related part to seperate class

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Rename method name to be more meaningful

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Remove unwanted imports and java docs

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: Exception handling

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* refactor: simplify the put method

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: handle exception from top and clean up

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Formatting code, sorting imports, adding new tests for exceptions (ibm-messaging#24)

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* Update kafka binary to a working version (ibm-messaging#15)

* feat: Formatting and updation of readme for exactly once configuration (ibm-messaging#27)

* feat: Formatting and updation of read for exactly once configuration

- .gitignore update to have the mq logs
- Formatted the readme file
- Added new documenation for the exactly once semantics
- New sample file for the exactly-once connector config

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Update 3.4.0 to 3.4.1

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* fix: Update README, add license declaration, remove AbstractConfig (ibm-messaging#28)

* fix: Update README, add license declaration, remove AbstractConfig

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Change DestinationBuilderIT to MessagePropertyIT

Contributes to: event-integration/qp-planning#10774

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: attempt to fix e2es (ibm-messaging#29)

* feat: attempt to fix e2es

Contributes to: event-integration/eventstreams-planning#0

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

* feat: fixed tests

Contributes to: event-integration/eventstreams-planning#0

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

* feat: revert from arm64 image from tests

Contributes to: event-integration/eventstreams-planning#0

Signed-off-by: Jonathan Hughes <jonathan.hughes@ibm.com>

* fix: Update required version from 2.6.0 to 0.11.0 (ibm-messaging#30)

Contributes to: event-integration/qp-planning#0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

* feat: Update test to have meaning full variable names (ibm-messaging#31)

Contributes to: event-integration/qp-planning#0

Signed-off-by: Joel Hanson <joel.hanson2@ibm.com>

Co-authored-by: Joel Hanson <joel.hanson2@ibm.com>
Co-authored-by: Nick Morgan <nicholas.morgan@ibm.com>
Co-authored-by: Jonathan Hughes <Jonathan.Hughes@ibm.com>

.checkstyle/checkstyle.xml

@@ -140,4 +140,8 @@
     <!-- Filter out Checkstyle warnings that have been suppressed with the @SuppressWarnings annotation -->
     <module name="SuppressWarningsFilter"/>
 
+    <!-- new line -->
+    <module name="NewlineAtEndOfFile">
+        <property name="lineSeparator" value="lf"/>
+    </module>
 </module>

.checkstyle/intellij-checkstyle.xml

@@ -136,4 +136,8 @@
     <!-- Filter out Checkstyle warnings that have been suppressed with the @SuppressWarnings annotation -->
     <module name="SuppressWarningsFilter"/>
 
+    <!-- new line -->
+    <module name="NewlineAtEndOfFile">
+        <property name="lineSeparator" value="lf"/>
+    </module>
 </module>

.github/ISSUE_TEMPLATES/BUG_REPORT.yml

@@ -57,7 +57,8 @@ body:
       label: Version
       description: What version of our software are you running?
       options:
-        - 1.5.1 (Default)
+        - 2.0.0  (Default)
+        - 1.5.1
         - older (<1.5.1)
     validations:
       required: true

.gitignore

@@ -15,4 +15,6 @@ target/
 *.iws
 
 # Visual Studio Code
-.vscode/
+.vscode/
+
+mqjms.log.*

Dockerfile

@@ -3,7 +3,7 @@ FROM alpine as builder
 RUN apk update
 RUN apk --no-cache add curl
 
-RUN curl -L "https://downloads.apache.org/kafka/3.4.0/kafka_2.12-3.4.0.tgz" -o kafka.tgz
+RUN curl -L "https://downloads.apache.org/kafka/3.4.1/kafka_2.12-3.4.1.tgz" -o kafka.tgz
 
 RUN mkdir /opt/kafka \
     && tar -xf kafka.tgz -C /opt/kafka --strip-components=1
@@ -26,4 +26,4 @@ EXPOSE 8083
 
 USER esuser
 
-ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]
+ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]

LICENSE

@@ -175,7 +175,7 @@ Apache License
 
    END OF TERMS AND CONDITIONS
 
-   Copyright 2017 IBM Corporation
+   Copyright 2017, 2023 IBM Corporation
 
    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.

LICENSES/EVENT_AUTOMATION/LI_en

@@ -0,0 +1,57 @@
+The translated license terms can be viewed here: https://www.ibm.com/support/customer/csol/terms/?id=L-HRZF-DWHH7A&lc=en#detail-document
+
+LICENSE INFORMATION
+
+The Programs listed below are licensed under the following License Information terms and conditions in addition to the Program license terms previously agreed to by Client and IBM. If Client does not have previously agreed to license terms in effect for the Program, the International Program License Agreement (i125-3301-15) applies.
+
+Program Name (Program Number):
+IBM Event Automation 1.0.0.0 (5900-AXM)
+
+The following standard terms apply to Licensee's use of the Program.
+
+Non-Production Limitation
+
+If the Program is designated as "Non-Production", Licensee can only use the Program as part of Licensee's internal development and test environment for internal non-production activities.
+
+Components Not Used for Establishing Required Entitlements
+
+Licensee may install and use the following Program components, under the license terms, but these components are not used to determine the number of entitlements required for the Program.
+Use of the Program and Components for Development and Functional Test purposes: 
+Under Development and Functional Test use, the Program or components can only be deployed as part of Licensee's internal development, unit, and functional testing environments. In functional test environments the developer can integrate their code with that of others on shared use machines, and that code can be tested in a test harness. Functional testing is limited to ensuring that the Program can be used to connect to local or remote endpoints, and that a sample of data sent between those endpoints is processed correctly. Licensee is not authorized to use the Program or components for system test, meaning integrated code is tested as a product, processing production workloads, simulating production workloads or testing the scalability of any code, application or system.
+
+Components Used for Establishing Required Entitlements
+
+Licensee may install and use Program components other than those listed below, under the license terms, but only the listed components are used to determine the number of entitlements required for the Program.
+kafka
+georep
+mirrormaker2
+connect
+flink-main-container
+manager 
+egw
+
+Modifiable Third Party Code
+
+To the extent, if any, in the NOTICES file IBM identifies third party code as "Modifiable Third Party Code," IBM authorizes Licensee to 1) modify the Modifiable Third Party Code and 2) reverse engineer the Program modules that directly interface with the Modifiable Third Party Code provided that it is only for the purpose of debugging Licensee's modifications to such third party code. IBM's service and support obligations, if any, apply only to the unmodified Program.
+
+Separately Licensed Code
+
+Each of the components listed in the NON_IBM_LICENSE file is considered "Separately Licensed Code" licensed to Licensee under the terms of the applicable third party license agreement(s) set forth in the NON_IBM_LICENSE file(s) that accompanies the Program, and not this Agreement. Future Program updates or fixes may contain additional Separately Licensed Code. Such additional Separately Licensed Code and related licenses are listed in the applicable NON_IBM_LICENSE file that accompanies the Program update or fix. 
+
+Note: Notwithstanding any of the terms in the third party license agreement, the Agreement, or any other agreement Licensee may have with IBM, with respect to the Separately Licensed Code: 
+(a) IBM provides it to Licensee WITHOUT WARRANTIES OF ANY KIND AND DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED WARRANTIES AND CONDITIONS INCLUDING, BUT NOT LIMITED TO, THE WARRANTY OF TITLE, NON-INFRINGEMENT OR NON-INTERFERENCE, AND THE IMPLIED WARRANTIES AND CONDITIONS OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE;
+(b) IBM is not liable for any direct, indirect, incidental, special, exemplary, punitive or consequential damages including, but not limited to, lost data, lost savings, and lost profits.
+
+The following units of measure may apply to Licensee's use of the Program.
+
+Virtual Processor Core
+
+Virtual Processor Core is a unit of measure by which the Program can be licensed. A Physical Server is a physical computer that is comprised of processing units, memory, and input/output capabilities and that executes requested procedures, commands, or applications for one or more users or client devices. Where racks, blade enclosures, or other similar equipment is being employed, each separable physical device (for example, a blade or a rack-mounted device) that has the required components is considered itself a separate Physical Server. A Virtual Server is either a virtual computer created by partitioning the resources available to a Physical Server or an unpartitioned Physical Server. A Processor Core is a functional unit within a computing device that interprets and executes instructions. A Processor Core consists of at least one instruction control unit and one or more arithmetic or logic units. A Virtual Processor Core is a Processor Core in an unpartitioned Physical Server, or a virtual core assigned to a Virtual Server. Licensee must obtain entitlements for each Virtual Processor Core made available to the Program.
+
+Licensee can deploy the Program (if supported) using full capacity licensing, sub-capacity licensing (https://www.ibm.com/software/passportadvantage/subcaplicensing.html), or container licensing (https://www.ibm.com/software/passportadvantage/containerlicenses.html). See the linked websites for more information. 
+
+
+L/N: L-HRZF-DWHH7A
+D/N: L-HRZF-DWHH7A
+P/N: L-HRZF-DWHH7A
+