-
Notifications
You must be signed in to change notification settings - Fork 120
java-openliberty: Add kafka template #775
Changes from all commits
9df5e96
4b22a93
0e3caa8
82a9638
8fc72f6
cb38ab8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
!.keep | ||
|
||
target/ | ||
|
||
### STS ### | ||
.apt_generated | ||
.classpath | ||
.factorypath | ||
.project | ||
.settings | ||
.springBeans | ||
.sts4-cache | ||
|
||
### IntelliJ IDEA ### | ||
.idea | ||
*.iws | ||
*.iml | ||
*.ipr | ||
|
||
### NetBeans ### | ||
/nbproject/private/ | ||
/nbbuild/ | ||
/dist/ | ||
/nbdist/ | ||
/.nb-gradle/ | ||
/build/ | ||
|
||
### VS Code ### | ||
.vscode/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
# Kafka template for Open Liberty | ||
|
||
This template can be used to develop Liberty applications that connect to Kafka by using MicroProfile Reactive messaging. A simple `StarterApplication` is included that enables basic production and consumption of events. | ||
|
||
|
||
## Getting Started with the StarterApplication. | ||
|
||
### 1. Create a new folder and initialize it using appsody init: | ||
|
||
|
||
``` | ||
mkdir test-appsody-kafka | ||
cd test-appsody-kafka | ||
appsody init java-openliberty kafka | ||
``` | ||
|
||
### 2. Start Kafka and ZooKeeper | ||
|
||
In order to run the `StarterApplication` you must start Kafka and ZooKeeper containers. ZooKeeper is a dependency of Kafka. Use the `docker-compose.yaml` that is provided in the template to start both containers. | ||
|
||
|
||
Start docker compose with the following command: | ||
|
||
```docker-compose up``` | ||
|
||
If you run `docker network list`, you should see a new network with the name of your project directory and the word `_default` appended. For example, `test-appsody-kafka_default`. | ||
|
||
Alternatively, if you want to connect to a Kafka broker elsewhere, edit `src/main/resources/META-INF/microprofile-config.properties` and set the value of the `mp.messaging.connector.liberty-kafka.bootstrap.servers` property to the host and port number of the your broker. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Editing the microprofile-config.properties file would indeed take effect within the container. Someone using this in "host mode" via
Not sure if you want to complicate the doc here.. but FYI at least. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For completeness..let me just note that editing There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm I think that would get a bit awkward to explain in the doc. |
||
|
||
### 3. Run the Appsody application in the new network | ||
|
||
Your Appsody application must be run in the same network as Kafka. | ||
|
||
Run the application using the following command: | ||
|
||
```appsody run --network test-appsody-kafka_default``` | ||
|
||
### 4. Produce a message to a topic | ||
|
||
Run another container in the same network: | ||
|
||
```docker run -it --network test-appsody-kafka_default strimzi/kafka:0.16.0-kafka-2.4.0 /bin/bash``` | ||
|
||
The next step is to produce a message. Use the following command to start a Kafka Producer that writes to `incomingTopic1`: | ||
|
||
```bin/kafka-console-producer.sh --broker-list kafka:9092 --topic incomingTopic1``` | ||
|
||
Enter text at the prompt to produce a message. | ||
|
||
### 5. Consume a message from a topic | ||
|
||
To view the messages, you can either look at the console log from the Appsody application or you can create a Kafka console consumer using the following command: | ||
|
||
```bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic incomingTopic1 --from-beginning``` | ||
|
||
## Deploying to Kubernetes | ||
|
||
When deploying to a Kubernetes environment, you must configure your application to connect to the Kafka broker. You can use the [Strimzi Kafka operator](https://strimzi.io/docs/quickstart/latest/) to deploy a Kafka broker in a Kubernetes cluster. | ||
|
||
To configure the connection, first run the following command: | ||
|
||
```appsody build``` | ||
|
||
This command generates `app-deploy.yaml` file. | ||
Edit the file to override the bootstrap server configuration by setting an enviornment variable as follows: | ||
|
||
``` | ||
spec: | ||
env: | ||
- name: MP_MESSAGING_CONNECTOR_LIBERTY_KAFKA_BOOTSTRAP_SERVERS | ||
value: <your-kafka-host>:9092 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I had trouble using the env var locally ..could it be this issue: OpenLiberty/open-liberty#10575 ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This definitely works when deployed. I guess setting the env var via the Maven plugin results in a slightly different property key? |
||
``` | ||
|
||
Then run the following command to deploy your application: | ||
|
||
``` | ||
appsody deploy --no-build | ||
``` |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
version: '2' | ||
services: | ||
zookeeper: | ||
image: strimzi/kafka:0.17.0-kafka-2.4.0 | ||
command: [ | ||
"sh", "-c", | ||
"bin/zookeeper-server-start.sh config/zookeeper.properties" | ||
] | ||
ports: | ||
- "2181:2181" | ||
environment: | ||
LOG_DIR: /tmp/logs | ||
|
||
kafka: | ||
image: strimzi/kafka:0.17.0-kafka-2.4.0 | ||
command: [ | ||
"sh", "-c", | ||
"bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override listener.security.protocol.map=$${KAFKA_LISTENER_SECURITY_PROTOCOL_MAP} --override inter.broker.listener.name=$${KAFKA_INTER_BROKER_LISTENER_NAME} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}" | ||
] | ||
depends_on: | ||
- zookeeper | ||
ports: | ||
- "9092:9092" | ||
- "9093:9093" | ||
expose: | ||
- "9092" | ||
- "9093" | ||
environment: | ||
LOG_DIR: "/tmp/logs" | ||
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9093 | ||
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9093 | ||
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT | ||
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE | ||
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should encourage the user to pass
MP_MESSAGING_CONNECTOR_LIBERTY_KAFKA_BOOTSTRAP_SERVERS
in as an environment variable so there is no "default" broker location.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I disagree - I think having a configuration baked in the works for local dev is a useful convenience. Saves you having to add the env var every time you
appsody run
. Also if you wanted to run the tests outside of appsody (with amvn test
) it would be awkward to set the env var. This is also the approach all of the other mpReactiveMessaging content suggests.