Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Default to Apache Kafka instead of Redpanda #854

Open
Tracked by #860
nscuro opened this issue Oct 17, 2023 · 3 comments
Open
Tracked by #860

Default to Apache Kafka instead of Redpanda #854

nscuro opened this issue Oct 17, 2023 · 3 comments
Assignees
@nscuro
Labels
enhancement New feature or request p2 Non-critical bugs, and features that help organizations to identify and reduce risk size/M Medium effort

Comments

@nscuro
Copy link
Member

nscuro commented Oct 17, 2023 •
edited
Loading

We chose to use Redpanda over Apache Kafka (AK) for our testing as it has a noticeably lower footprint.

However, Redpanda is licensed under BUSL which can be problematic for some organizations. Further, as Dependency-Track is an OSS project under a truly open foundation (OWASP), we should not depend on commercial solutions per default. Instead, we should default to Apache Kafka, which is Apache-2.0 licensed, just like DT itself.

Further, we may want to replace Redpanda Console with something like https://github.com/provectus/kafka-ui (for the same licensing reason).

In summary, the following needs to be done:
@nscuro nscuro added enhancement New feature or request p2 Non-critical bugs, and features that help organizations to identify and reduce risk size/M Medium effort labels Oct 17, 2023
@nscuro nscuro mentioned this issue Oct 19, 2023
Open
34 tasks
@nscuro
Copy link
Member Author

nscuro commented Mar 6, 2024 •
edited
Loading

As of v3.7.0, Kafka has an official Docker image:

@VinodAnandan VinodAnandan added the help wanted Extra attention is needed label Jun 30, 2024
@nscuro
Copy link
Member Author

nscuro commented Jul 1, 2024

As per KIP-974, Apache Kafka will ship a native image version with v3.8: https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker

That will be useful for demo setups. The KIP does say explicitly though that production usage is not yet recommended:

The GraalVM native-image based container is recommended only for development, and testing and not for production workloads.

nscuro added a commit to DependencyTrack/hyades-apiserver that referenced this issue Sep 12, 2024
@nscuro
Replace Redpanda testcontainers with Apache Kafka native
74866d3 
Relates to DependencyTrack/hyades#854

Signed-off-by: nscuro <nscuro@protonmail.com>
@nscuro nscuro mentioned this issue Sep 12, 2024
Merged
1 task
nscuro added a commit that referenced this issue Oct 2, 2024
@nscuro
Extend Compose setup and e2e tests with multiple Kafka providers
0c12e60 
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

To enable all of the above, the `create-topics.sh` script had to be modified to use official Kafka CLI tools (i.e. `kafka-topics.sh`) instead of the Redpanda CLI. The official tooling uses the Kafka API and is thus provider-independent.

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit that referenced this issue Oct 2, 2024
@nscuro
Extend Compose setup and e2e tests with multiple Kafka providers
f21dd35 
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

To enable all of the above, the `create-topics.sh` script had to be modified to use official Kafka CLI tools (i.e. `kafka-topics.sh`) instead of the Redpanda CLI. The official tooling uses the Kafka API and is thus provider-independent.

Signed-off-by: nscuro <nscuro@protonmail.com>
@nscuro nscuro mentioned this issue Oct 2, 2024
Draft
2 tasks
@nscuro nscuro assigned nscuro and unassigned VithikaS Oct 2, 2024
@nscuro nscuro reopened this Oct 2, 2024
@nscuro nscuro removed the help wanted Extra attention is needed label Oct 2, 2024
@nscuro
Copy link
Member Author

nscuro commented Oct 3, 2024

Requested addition of Apache Kafka native to Quarkus Dev Services here: quarkusio/quarkus#43272

nscuro added a commit that referenced this issue Oct 3, 2024
@nscuro
Extend Compose setup and e2e tests with multiple Kafka providers
c7004c4 
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit that referenced this issue Oct 17, 2024
@nscuro
Extend Compose setup and e2e tests with multiple Kafka providers
76cf97d 
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <nscuro@protonmail.com>
@nscuro nscuro mentioned this issue Oct 19, 2024
Open
2 tasks
nscuro added a commit that referenced this issue Oct 19, 2024
@nscuro
Extend Compose setup and e2e tests with multiple Kafka providers
062a6a5 
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro added a commit that referenced this issue Oct 19, 2024
@nscuro
Extend Compose setup and e2e tests with multiple Kafka providers
35dac0a 
Extends the Compose setup, which previously solely relied on Redpanda, with Apache Kafka and Tansu. Defaults to Apache Kafka (native) in accordance with #854. While a Tansu setup is added, Tansu can not yet be used with Hyades. It runs into errors when the API server tries to produce to it.

Extends e2e tests to support testing again different Kafka providers. Currently supported are Apache Kafka, Apache Kafka native, and Redpanda. Tansu will be added once it's functional.

Removes the `create-topics.sh` script and the `redpanda-init` container in favour of the new kafka topics init task (introduced in DependencyTrack/hyades-apiserver#943).

Signed-off-by: nscuro <nscuro@protonmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nscuro nscuro

Labels
enhancement New feature or request p2 Non-critical bugs, and features that help organizations to identify and reduce risk size/M Medium effort
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nscuro @VinodAnandan @VithikaS