kafka-rest-proxy

Get your data via REST into Apache Kafka

Idea

Often in your environment you might not be allowed to directly access your kafka from web (e.g. you have a web- and application-zone, but kafa is installed in application zone and not accessible via direct-path). But still you have the need to get data coming from outside into your topics. So one easy way is: kafka-rest-proxy. It provides a simple RESTful API, forwarding your payload to kafka.

Prerequiste

1. Install Apache Kafka download and unarchive or brew install kafka

2. Start kafka

   bin/zookeeper-server-start.sh config/zookeeper.properties
   bin/kafka-server-start.sh config/server.properties
   

   or if installed via brew

   zookeeper-server-start /usr/local/etc/kafka/zookeeper.properties
   kafka-server-start /usr/local/etc/kafka/server.properties
   

3. Create a kafka topic

   Homebrew:

   kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
   

4. Check if topic is there

   Homebrew:

   kafka-topics --list --zookeeper localhost:2181
   

5. Start a console conusmer to check your posted messsages

   Homebrew:

   kafka-console-consumer --topic test --zookeeper localhost:2181
   

Usage of kafka-rest-proxy

1. Clone git repository
2. Build executable ./gradlew clean build
3. Start ./gradlew bootRun

Configure kafka-rest-proxy

Configuration file can be found in src/main/resources and is named application.yml.

• Put all Apache Kafka properties beneath node producer (see Configuration)
• For configuration of Spring Boot see Configuration

REST API

POST to any topic

curl -XPOST -H "Content-Type:application/json" -d "{\"message\": \"kafka-rest-proxy is on github.\"}" http://localhost:8080/topics/test

curl -XPOST -H "Content-Type:application/json" -d "{\"message\": \"kafka-rest-proxy is on github.\"}" http://localhost:8080/topics/test?key=key

Information endpoints

curl -XGET http://localhost:8080/topicslist  //since consumer is not thread-safe topic-list gets up-to-date approx. every 2 sec in a scheduled task.

curl -XGET http://localhost:8080/metrics

curl -XGET http://localhost:8080/topicsinfo/test

Healthcheck

curl -XGET http://localhost:8080/actuator/health

* Note: spring-boot-2.0.1, spring-kafka-2.2.0, java version 10.0.2 (2018-07-17)