While you were gone... (#566)

* fix: Apply formatting, using a formatter that supports Java 17

* fix: Don't throw exception if JulieOps is not allowed to delete topics (#3)

* Don't throw exception if JuleOps is not allowed to delete topics.

* Update dependencies (#2)

* Update README.md

* fix: Apply formatting, using a formatter that supports Java 17

* chore: Update dependencies

* chore: Update dependencies

* Update container versions

* fix: Replace deprecated AssertJ method

* chore: Upgrade Confluent version in workflow

* fix: Replace deprecated methods

* fix: Replace deprecated methods

* chore: Update dependencies

* fix: Extend connector path for new version of connect

* fix: Revert ksqldb-server to a version that makes the test pass

* fix: Correct return value from exception handler

* fix: Hack around bug KAFKA-14325

* feat: Support finding topology files recursively (#4)

* fix: Clean up zookeeper (#5)

The Zookeeper dependency is required for a single test method only. This PR moves all ZK* classes to src/test, and marks the zookeeper dependency as test-coped.

Unfortunately, recent versions of zookeeper drag in a logback dependency, that appears to turn on global DEBUG logging. So this PR also exludes those logback dependencies.

Fixes #548

* fix: Allow logging of INFO+ from AdminClient (#6)

* Excludes all Log4j2Plugins.dat from the shaded jar, to avoid Log4j2 startup problems (ERROR StatusLogger Unrecognized conversion specifier...)

* Adds a log4j.properties that will be picked up by the Kafka dependencies

* Create dependabot.yml

* chore: Update plug-in versions

* chore: Update jackson and redis to remove vulnerable json and snakeyaml packages

* chore: Update plugins

* chore: Update some deps

* chore: Upgrade testcontainers

* Updates (#8)

* chore: Update some deps

* chore: Update some deps

* chore: Upgrade Confluent Platform

* docs: Fetch changelog from v4.4.1 tag to get all history

* chore: Remove unused imports and trailing whitespace (#9)

* chore: Remove unused imports

* docs: Remove trailing spaces

* Update src/test/java/com/purbon/kafka/topology/integration/MDSBaseTest.java

Co-authored-by: piotrsmolinski <piotr.smolinski.77@gmail.com>

---------

Co-authored-by: piotrsmolinski <piotr.smolinski.77@gmail.com>

.github/dependabot.yml

@@ -0,0 +1,11 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+
+version: 2
+updates:
+  - package-ecosystem: "Maven" # See documentation for possible values
+    directory: "/" # Location of package manifests
+    schedule:
+      interval: "weekly"

.github/workflows/ci-integration-test-main.yml

@@ -15,7 +15,7 @@ jobs:
       matrix:
         os: [ubuntu-latest]
         java: [11.0.x]
-        cpversion: [7.0.0]
+        cpversion: [7.5.0]
 
     runs-on: ${{ matrix.os }}
 

.gitignore

@@ -6,6 +6,7 @@ logs/*
 .cluster-state
 *.iml
 _build
+*~
 
 dependency-reduced-pom.xml
 server-api/logs/

CONTRIBUTING.md

@@ -4,7 +4,7 @@
 
 Have a problem you want the Kafka health API to solve for you?
 
-* File a ticket on [GitHub](https://github.com/purbon/kafka-topology-builder/issues). 
+* File a ticket on [GitHub](https://github.com/purbon/kafka-topology-builder/issues).
 
 ## Something Not Working? Found a Bug? or a Security Issue?
 
@@ -15,7 +15,7 @@ If you think you found a bug, it probably is a bug.
 # Contributing Documentation and Code Changes
 
 If you have a bugfix or new feature that you would like to contribute, and you think it will take
-more than a few minutes to produce the fix (ie; write code), it is worth discussing the change. 
+more than a few minutes to produce the fix (ie; write code), it is worth discussing the change.
 You can reach us via [GitHub](https://github.com/purbon/kafka-topology-builder/issues).
 
 Please note that Pull Requests without tests and documentation may not be merged. If you would like to contribute but do not have
@@ -91,11 +91,11 @@ Example:
    request](https://help.github.com/articles/using-pull-requests). In the pull
    request, describe what your changes do and mention any bugs/issues related
    to the pull request.
-   
+
 # Pull Request Guidelines
 
 The following exists as a way to set expectations for yourself and for the review process. We *want* to merge fixes and features, so let's describe how we can achieve this:
-   
+
 ## Goals
 
 * To constantly make forward progress on PRs

README.md

@@ -4,7 +4,7 @@
 
 I'm gratefuly of how many people the JulieOps project has helped during it existance, it is totally mind blowing to get more than 300 starts for a humble human like me, thanks everyone!!.
 
-Sadly this days, between my workload and personal arrangements, the project has been lacking proper mantainance and care, what honestly makes me very sad as I would love to see it grow and provide more and more people with such features, I'm a big beliver of self service and automation. 
+Sadly this days, between my workload and personal arrangements, the project has been lacking proper mantainance and care, what honestly makes me very sad as I would love to see it grow and provide more and more people with such features, I'm a big beliver of self service and automation.
 
 So, until new notice, or something change, you should take the project with care, as currently it is mostly on a long winter hibernation :-) I'm sorry for this, but I can't do more as a mostly sole mantainer.
 
@@ -13,7 +13,7 @@ Thanks again to everyone who was, is or will be involved with the project life.
 <center>
 <img src="https://media.kidadl.com/Do_Pandas_Hibernate_Why_Are_They_More_Active_During_Winter_d5daed1b94.jpg" alt="panda" width="300"/>
  </center>
- 
+
 -- Pere
 
 ### README
@@ -23,21 +23,21 @@ Thanks again to everyone who was, is or will be involved with the project life.
 <a href="https://codeclimate.com/github/purbon/kafka-topology-builder/maintainability"><img src="https://api.codeclimate.com/v1/badges/ef4bcda7d1b5fd0a4f1e/maintainability" /></a> ![CI tests](https://github.com/kafka-ops/kafka-topology-builder/workflows/CI%20tests/badge.svg?branch=master) [![Gitter](https://badges.gitter.im/kafka-topology-builder/community.svg)](https://gitter.im/kafka-topology-builder/community?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) [![Documentation Status](https://readthedocs.org/projects/julieops/badge/?version=latest)](https://julieops.readthedocs.io/?badge=latest)
 
 JulieOps helps you automate the management of your things within Apache Kafka, from Topics,
-Configuration to Metadata but as well Access Control, Schemas. 
-More items are plan, check [here](https://github.com/kafka-ops/julie/issues) for details. 
+Configuration to Metadata but as well Access Control, Schemas.
+More items are plan, check [here](https://github.com/kafka-ops/julie/issues) for details.
 
-## The motivation 
+## The motivation
 
-One of the typical questions while building an Apache Kafka infrastructure is how to handle topics, 
+One of the typical questions while building an Apache Kafka infrastructure is how to handle topics,
 configurations and the required permissions to use them (Access Control List).
 
-The JulieOps cli, in close collaboration with git and Jenkins (CI/CD) is here to help you setup an 
+The JulieOps cli, in close collaboration with git and Jenkins (CI/CD) is here to help you setup an
 organised and automated way of managing your Kafka Cluster.
- 
+
 ## Where's the docs?
 
 We recommend taking time to [read the docs](https://julieops.readthedocs.io/en/latest/).
-There's quite a bit of detailed information about GitOps, Apache Kafka and how this project can help you automate 
+There's quite a bit of detailed information about GitOps, Apache Kafka and how this project can help you automate
 the common operational tasks.
 
 ## Automating Management with CI/CD and GitOps
@@ -63,9 +63,9 @@ You might be wondering what is the usual workflow to implement this approach:
 
 Considerations:
 
-* Using webhooks, the git server (github, gitlab or bitbucket) will inform the CI/CD system changes had happened 
+* Using webhooks, the git server (github, gitlab or bitbucket) will inform the CI/CD system changes had happened
 and the need to apply them to the cluster.
-* All changes (git push) to master branch are disabled directly. 
+* All changes (git push) to master branch are disabled directly.
 Changes only can happen with a pull request. Providing a Change Management mechanism to fit into your org procedures.
 
 ## Help??
@@ -85,7 +85,7 @@ What can you achieve with this tool:
 * Automatically set access control rules for:
     * Kafka Consumers
     * Kafka Producers
-    * Kafka Connect 
+    * Kafka Connect
     * Kafka Streams applications ( microservices )
     * KSQL applications
     * Schema Registry instances
@@ -102,13 +102,13 @@ What can you achieve with this tool:
 * Manage your cluster schemas.
     - Support for Confluent Schema Registry
 
-Out of the box support for Confluent Cloud and other clouds that enable you to use the AdminClient API. 
+Out of the box support for Confluent Cloud and other clouds that enable you to use the AdminClient API.
 
 ### How can I run JulieOps directly?
 
 This tool is available in multiple formats:
 
-- As a Docker image, available from [docker hub](https://hub.docker.com/r/purbon/kafka-topology-builder) 
+- As a Docker image, available from [docker hub](https://hub.docker.com/r/purbon/kafka-topology-builder)
 - As an RPM package, for the RedHat alike distributions
 - As a DEB package, for Debian based distros
 - Directly as a fat jar (zip/tar.gz)
@@ -120,7 +120,7 @@ The latest version are available from the [releases](https://github.com/kafka-op
 
 This is how you can run the tool directly as a docker image:
 
-```bash 
+```bash
 docker run purbon/kafka-topology-builder:latest julie-ops-cli.sh  --help
 Parsing failed cause of Missing required options: topology, brokers, clientConfig
 usage: cli
@@ -142,7 +142,7 @@ usage: cli
 
 If you install the tool as rpm, you will have available in your $PATH the _julie-ops-cli.sh_.
 You can run this script with the same options observed earlier, however you will need to be using, or be in the group,
-for the user julie-kafka. 
+for the user julie-kafka.
 
 #### An example topology
 
@@ -193,10 +193,10 @@ projects:
       num.partitions: "3"
 ```
 
-more examples can be found at the [example/](example/) directory. 
+more examples can be found at the [example/](example/) directory.
 
 Also, please check, the documentation in [the docs](https://julieops.readthedocs.io/) for extra information and
-examples on managing ACLs, RBAC, Principales, Schemas and many others.  
+examples on managing ACLs, RBAC, Principales, Schemas and many others.
 
 ## Troubleshooting guides
 
@@ -213,7 +213,7 @@ Check our [contributing](CONTRIBUTING.md) doc for guidance.
 ## Building JulieOps from scratch (source code)
 
 The project is build using Java and Maven, so both are required if you aim to build the tool from scratch.
-The minimum version of Java supported is Java 8, note it soon will be deprecated here, it is only keep as supported 
+The minimum version of Java supported is Java 8, note it soon will be deprecated here, it is only keep as supported
 for very legacy environments.
 
 It is recommended to run JulieOps with Java 11 and an open JDK version.

changelog.md

@@ -1,18 +1,150 @@
 
+v4.4.1 / 2022-10-10
+===================
+
+  * Adds support for KSqlDB session variables (#544)
+  * Add sync actions for kafka generic artefacts, Connect and kSQL for now (#542)
+
+v4.3.0 / 2022-08-22
+===================
+* Implements optimized ACLs for subjects #528
+* Add Subject prefix filtering support (#525)
+* Add a config test to validate when people use empty managed prefixes as mistake (#522)
+* Ammend introduce dedicated resource filters for access control rules and extend it logic to be more precise (#521)
+* [neat] Extensions and small collateral updates for testing variables with config as JSON (#520)
+* Verify schemas deletion process successfully, complement the related test  (#516)
+* Extend the project name format to include by default the separator, this is useful to narrow down optimised acls to be more secure (#515)
+* Add autocomplete for bash (#500)
+
+v4.2.9 / 2022-08-22
+===================
+
+* Fix pagination issue when listing service accounts for confluent cloud (#531)
+* Don't print empty operations (#517)
+
+v4.2.8 / 2022-07-28
+===================
+
+* [fix] remove condition of failing if min.insync.validation == 1 and other bigger (#509)
+* [neat] Add extra validations for topics  (#508)
+
+v4.2.6 / 2022-07-27
+===================
+
+* [bug] Force special topics to be using the name topic naming convention in order
+  to avoid cases when the prefix formats break the flow (#507)
+* [testing] Extend example for rbac tls for testing (#498)
+
+v4.2.5 / 2022-05-02
+===================
+
+* Fix the acl creation in the hybrid ccloud provider (#494)
+
+v4.2.4 / 2022-04-28
+===================
+
+* [neat] make hybrid ccloud provider list acls use admin client
+
+v4.2.3 / 2022-04-27
+===================
+
+* [bug] fix a stupid missed thing when loading the hybrid provider, this code really needs to be done cleaner
+
+v4.2.2 / 2022-04-27
+===================
+
+* [Feature] Introduce the concept of an hybrid ccloud provider, to set acls via admin client and translation via api (#492)
+
+v4.2.1 / 2022-04-26
+===================
+
+* [Feature] Add feature flag to make the remote state verification backwards compatible again (#491)
+* [Feature] Allow setting log level as debug in the code (#490)
+* [Feature] allow insecure https connection when using mds, default to false
+* [Neat] add a method to simple sanitize a string that could contain empty values
+* [Feature] add a pre-flight check for valid clusterIds in your platform
+
+v4.2.0 / 2022-04-13
+===================
+
+* [Big] Fix Confluent Cloud Translation mechanism when the Service Account does not have a type prefix (default user) (#485)
+* [Feature] Introduce the concept of an AuditLog for JulieOps (#484)
+* [Feature] Add support for out of the box topics, an special topics list managed by JulieOps (#482)
+* [Feature] Add Kafka Streams applicationId as internal topics, if available (#481)
+
+v4.1.3 / 2022-04-08
+===================
+
+* Detect divergences between local state and the remote cluster current status (#478)
+* Add validators backwards compatibility (#480)
+
+v4.1.2 / 2022-04-06
+===================
+
+* [Security] Fix CWE-787 CVE-2020-36518 for jackson-databind (#476)
+* Add support for deploying packaged released to Maven Central (#473) (#475) (#477)
+* Add a JSON schema description of the topology/descriptor file sintax (#471) (#472)
+* [Test] Refactor and add tests related to Confluent Cloud service account translation feature (#468)
+* Allowing to configure the redis bucket used by JulieOps (#465)
+* [Test] Clarify S3 Backend IT test (#464)
+* [Bug] Fix RedisBackend bootstrap, NullPointerException (#462)
+* [Bug] Issue fix (456) for resolution of service account mapping (Translation of principals) (#459)
+
+v4.1.1 / 2022-02-05
+===================
+
+* Fix Confluent Cloud ACL(s) API usage, so ACL(s) are finally created properly (#444)
+* Fix config passing for topology validator for regular expressions (#443)
+* Bump log4j-api from 2.17.0 to 2.17.1 (#436)
+
+v4.1.0 / 2021-12-30
+===================
+
+* [ksqlDB] when using ACLs configure all internal topics with ALL permissions for the ksql server user (#433)
+* Bring Principal Management for Confluent Cloud out of Experimental into Production ready feature (#435)
+* Use Confluent Cloud API when integrating with the Confluent fully managed service (#431)
+* Throw an exception when an invalid plan is used (#426)
+* Add docker to the SAN to make it run in our in-house Gitlab (#421)
+* Improved execution log for topics and schemas (#383)
+
+v4.0.1 / 2021-12-20
+==================
+
+* Bump log4j to 2.17.0, prevent latest Log4j CVE, ref https://logging.apache.org/log4j/2.x/security.html (#427)
+
+v4.0.0 / 2021-12-10
+===================
+
+* neat: Adapt CI jobs and other actions to latest versions (#419)
+* port the main changelog to master
+* fix&feature: establish service accounts prefix filter a primary criteria when available to filter the list of access control rules (#418)
+* Bump log4j-api from 2.13.3 to 2.15.0 (#416)
+* fix: issue with connector and subject permissions being ignored when more than one (#415)
+* Add producer Idempotence permissions to Confluent RBAC (#414)
+* Fix: Topic config values should take precedence over plan config values (#410)
+* Quotas implementation, only quotas based in user principal (#376)
+* fix request scope to hold proper pattern type (literal, prefix) in rbac
+* add proper clusterIds for RBAC clear bindings operations, not only kafka one
+* add support in the custom roles for subject and connect attributes (#406)
+* fix incorrect rbac resource name for subjects and connectors (#405)
+* ammend order for delete in ksql to be reverse from creation (#400)
+* add Parameter for initial Kafka Backend consumer retry
+* raise initial load as done if tried at least five times
+
 v4.0.0 / 2021-12-10
 ==================
 
-  * Fix/feature: establish service accounts prefix filter a primary criteria when available to filter the list of access control rules (#418)
-  * Security fix: Bump log4j-api from 2.13.3 to 2.15.0 (#416)
-  * fix: issue with connector and subject permissions being ignored when more than one (#415)
-  * feature: Add producer Idempotence permissions to Confluent RBAC (#414)
-  * fix: Topic config values should take precedence over plan config values (#410)
-  * feature: Quotas implementation, only quotas based in user principal (#376)
-  * feature: add support in the custom roles for subject and connect attributes (#406)
-  * fix: incorrect rbac resource name for subjects and connectors (#405)
-  * fix: ammend order for delete in ksql to be reverse from creation (#400)
-  * fix: add Parameter for initial Kafka Backend consumer retry 
-  * fix: Raise initial load as done if tried at least five times for the Kafka Backend
+* Fix/feature: establish service accounts prefix filter a primary criteria when available to filter the list of access control rules (#418)
+* Security fix: Bump log4j-api from 2.13.3 to 2.15.0 (#416)
+* fix: issue with connector and subject permissions being ignored when more than one (#415)
+* feature: Add producer Idempotence permissions to Confluent RBAC (#414)
+* fix: Topic config values should take precedence over plan config values (#410)
+* feature: Quotas implementation, only quotas based in user principal (#376)
+* feature: add support in the custom roles for subject and connect attributes (#406)
+* fix: incorrect rbac resource name for subjects and connectors (#405)
+* fix: ammend order for delete in ksql to be reverse from creation (#400)
+* fix: add Parameter for initial Kafka Backend consumer retry
+* fix: Raise initial load as done if tried at least five times for the Kafka Backend
 
 v3.3.3 / 2021-11-23
 ===================