Skip to content
This repository has been archived by the owner on Jul 27, 2023. It is now read-only.

java-openliberty: Add kafka template #775

Merged
merged 6 commits into from
May 1, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions incubator/java-openliberty/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,22 +24,24 @@ These additional [day-2 operations](https://github.com/OpenLiberty/open-liberty-

Templates are used to create your local project and start your development. When initializing your project you will be provided with an Open Liberty template application.

### Default template

The default template provides a `pom.xml` file that references the parent POM defined by the stack and enables Liberty features that support [Eclipse MicroProfile 3.2](https://openliberty.io/docs/ref/feature/#microProfile-3.2.html). Specifically, this template includes:

### Health
#### Health

The `mpHealth` feature allows services to report their readiness and liveness status - UP if it is ready or alive and DOWN if it is not ready/alive. It publishes two corresponding endpoints to communicate the status of liveness and readiness. A service orchestrator can then use the health statuses to make decisions.

Liveness endpoint: http://localhost:9080/health/live
Readiness endpoint: http://localhost:9080/health/ready

### Metrics
#### Metrics

The `mpMetrics` feature enables MicroProfile Metrics support in Open Liberty. Note that this feature requires SSL and the configuration has been provided for you. You can monitor metrics to determine the performance and health of a service. You can also use them to pinpoint issues, collect data for capacity planning, or to decide when to scale a service to run with more or fewer resources.

Metrics endpoint: http://localhost:9080/metrics

#### Metrics Password
##### Metrics Password

Log in as the `admin` user to see both the system and application metrics in a text format. The password for this `admin` user will be generated by the container.

Expand All @@ -51,18 +53,21 @@ To get the generated password for project **my-project**, you can exec in the co

So in the above example the password value would be: `2r1aquTO3VVUVON7kCDdzno`

### OpenAPI
#### OpenAPI

The `mpOpenAPI` feature provides a set of Java interfaces and programming models that allow Java developers to natively produce OpenAPI v3 documents from their JAX-RS applications. This provides a standard interface for documenting and exposing RESTful APIs.

OpenAPI endpoints:
- http://localhost:9080/openapi (the RESTful APIs of the inventory service)
- http://localhost:9080/openapi/ui (Swagger UI of the deployed APIs)

### Junit 5
#### Junit 5

The default template uses JUnit 5. You may be used to JUnit 4, but here are some great reasons to make the switch https://developer.ibm.com/dwblog/2017/top-five-reasons-to-use-junit-5-java/

### Kafka template

The kafka template allows you to develop applications that connect to Kafka using MicroProfile Reactive Messaging. For more information, see the [kafka readme](templates/kafka/README.md).

## Getting Started

Expand Down
2 changes: 1 addition & 1 deletion incubator/java-openliberty/stack.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: Open Liberty
version: 0.2.9
version: 0.2.10
description: Eclipse MicroProfile & Jakarta EE on Open Liberty & OpenJ9 using Maven
license: Apache-2.0
language: java
Expand Down
29 changes: 29 additions & 0 deletions incubator/java-openliberty/templates/kafka/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
!.keep

target/

### STS ###
.apt_generated
.classpath
.factorypath
.project
.settings
.springBeans
.sts4-cache

### IntelliJ IDEA ###
.idea
*.iws
*.iml
*.ipr

### NetBeans ###
/nbproject/private/
/nbbuild/
/dist/
/nbdist/
/.nb-gradle/
/build/

### VS Code ###
.vscode/
78 changes: 78 additions & 0 deletions incubator/java-openliberty/templates/kafka/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Kafka template for Open Liberty

This template can be used to develop Liberty applications that connect to Kafka by using MicroProfile Reactive messaging. A simple `StarterApplication` is included that enables basic production and consumption of events.


## Getting Started with the StarterApplication.

### 1. Create a new folder and initialize it using appsody init:


```
mkdir test-appsody-kafka
cd test-appsody-kafka
appsody init java-openliberty kafka
```

### 2. Start Kafka and ZooKeeper

In order to run the `StarterApplication` you must start Kafka and ZooKeeper containers. ZooKeeper is a dependency of Kafka. Use the `docker-compose.yaml` that is provided in the template to start both containers.


Start docker compose with the following command:

```docker-compose up```

If you run `docker network list`, you should see a new network with the name of your project directory and the word `_default` appended. For example, `test-appsody-kafka_default`.

Alternatively, if you want to connect to a Kafka broker elsewhere, edit `src/main/resources/META-INF/microprofile-config.properties` and set the value of the `mp.messaging.connector.liberty-kafka.bootstrap.servers` property to the host and port number of the your broker.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should encourage the user to pass MP_MESSAGING_CONNECTOR_LIBERTY_KAFKA_BOOTSTRAP_SERVERS in as an environment variable so there is no "default" broker location.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree - I think having a configuration baked in the works for local dev is a useful convenience. Saves you having to add the env var every time you appsody run. Also if you wanted to run the tests outside of appsody (with a mvn test) it would be awkward to set the env var. This is also the approach all of the other mpReactiveMessaging content suggests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editing the microprofile-config.properties file would indeed take effect within the container. Someone using this in "host mode" via mvn liberty:dev would need to either:

  1. edit the pom.xml
  2. run like: mvn -Dliberty.var.mp.messaging.connector.liberty-kafka.bootstrap.servers=localhost:9093

Not sure if you want to complicate the doc here.. but FYI at least.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For completeness..let me just note that editing microprofile-config.properties would also take effect for the prod image built by appsody build. (If this needed to be distinct from the dev container value we could do profiles for that too).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I think that would get a bit awkward to explain in the doc.
For appsody build I think it should always be overridden in app-deploy.yaml with an env var anyway


### 3. Run the Appsody application in the new network

Your Appsody application must be run in the same network as Kafka.

Run the application using the following command:

```appsody run --network test-appsody-kafka_default```

### 4. Produce a message to a topic

Run another container in the same network:

```docker run -it --network test-appsody-kafka_default strimzi/kafka:0.16.0-kafka-2.4.0 /bin/bash```

The next step is to produce a message. Use the following command to start a Kafka Producer that writes to `incomingTopic1`:

```bin/kafka-console-producer.sh --broker-list kafka:9092 --topic incomingTopic1```

Enter text at the prompt to produce a message.

### 5. Consume a message from a topic

To view the messages, you can either look at the console log from the Appsody application or you can create a Kafka console consumer using the following command:

```bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic incomingTopic1 --from-beginning```

## Deploying to Kubernetes

When deploying to a Kubernetes environment, you must configure your application to connect to the Kafka broker. You can use the [Strimzi Kafka operator](https://strimzi.io/docs/quickstart/latest/) to deploy a Kafka broker in a Kubernetes cluster.

To configure the connection, first run the following command:

```appsody build```

This command generates `app-deploy.yaml` file.
Edit the file to override the bootstrap server configuration by setting an enviornment variable as follows:

```
spec:
env:
- name: MP_MESSAGING_CONNECTOR_LIBERTY_KAFKA_BOOTSTRAP_SERVERS
value: <your-kafka-host>:9092
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had trouble using the env var locally ..could it be this issue: OpenLiberty/open-liberty#10575 ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This definitely works when deployed. I guess setting the env var via the Maven plugin results in a slightly different property key?

```

Then run the following command to deploy your application:

```
appsody deploy --no-build
```
34 changes: 34 additions & 0 deletions incubator/java-openliberty/templates/kafka/docker-compose.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
version: '2'
services:
zookeeper:
image: strimzi/kafka:0.17.0-kafka-2.4.0
command: [
"sh", "-c",
"bin/zookeeper-server-start.sh config/zookeeper.properties"
]
ports:
- "2181:2181"
environment:
LOG_DIR: /tmp/logs

kafka:
image: strimzi/kafka:0.17.0-kafka-2.4.0
command: [
"sh", "-c",
"bin/kafka-server-start.sh config/server.properties --override listeners=$${KAFKA_LISTENERS} --override advertised.listeners=$${KAFKA_ADVERTISED_LISTENERS} --override listener.security.protocol.map=$${KAFKA_LISTENER_SECURITY_PROTOCOL_MAP} --override inter.broker.listener.name=$${KAFKA_INTER_BROKER_LISTENER_NAME} --override zookeeper.connect=$${KAFKA_ZOOKEEPER_CONNECT}"
]
depends_on:
- zookeeper
ports:
- "9092:9092"
- "9093:9093"
expose:
- "9092"
- "9093"
environment:
LOG_DIR: "/tmp/logs"
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9093
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
Loading