Skip to content

Spring Boot, Kafka, Zookeeper, Kafdrop(kafka viewer), Kafka producer, Kafka consumer, Spring Data/JPA, MySQL, Microservices, Rest API

License

Notifications You must be signed in to change notification settings

james-kennet/kafka-project

Repository files navigation

Spring Boot, Kafka, Zookeeper, Kafdrop, Spring Data/JPA, MySQL, Rest API

Sample kafka-project

  • Single topic per individual platform.
    • Kafka as a traditional message broker.
  • Single topic to all platforms.
    • Different clientType value attribute. To be processed separately using recordFilterStrategy.

Use Cases(Address creation)

  • Improve the speed of synchronizing customer information in a typical Insurance domain.
  • In the Insurance domain, multiple platforms are able to update customer information, like addresses:
    • Website application
    • Salesforce
    • Life400 - mainframe
  • Use Kafka to receive and process all ADDRESS creation from multiple platforms and process to the downstream services.
  • Introduced address_transaction table which will be referred by the consumers.

Use Case #1

  • Kafka as traditional message broker.
    • One topic per platform
      • ADDRESS_CREATE_SALESFORCE
      • ADDRESS_CREATE_WEBSITE
      • ADDRESS_CREATE_LIFE400
    • One producer endpoint per platform/topic.
    • One consumer listener per platform/topic.

Sample High-level architecture diagram

My Image

Use Case #2

  • Kafka single topic with consumer using recordFilterStrategy.
    • One topic for all platforms with different clientType.
      • ADDRESS_CREATE_ALL_PLATFORM
    • One producer endpoint to all platforms.
    • One consumer listener per platform clientType by using recordFilterStrategy.
      • containerFactory = "allPlatformSalesforceFactory"
      • containerFactory = "allPlatformWebsiteFactory"
      • containerFactory = "allPlatformLife400Factory"

Sample High-level architecture diagram

  • My Image

Prerequisites on running this project:

  • Install Java 11(Needed by Kafkdrop - kafka message viewer).
  • Install Docker Desktop.
  • JMeter - for testing.

Tech stack and services in kafka-project.

  • kafka
  • zookeeper
  • Kafkdrop - kafka message viewer
  • kafka-producer service
  • kafka-consumer service
  • address-service - services to process consume messages
  • MySql - to save all transactions

Steps in running kafka-project.

  • Checkout the project in Github.
  • Go the project directory.
cd /<path>/kafka-project
  • Install kafka-project services.
mvn clean install
  • Start kafka-project services.
docker-compose up
  • View Kafdrop on the browser using the below URL.
http://localhost:9000

Testing kafka-project locally.

  • Copy the JMeter script to JMeter bin directory.
JMeter script - kafka-project/_external_files/kafka-producer-v1.jmx
JMeter bin - <JMeter path>/bin
cp <JMeter script> <JMeter bin>
  • Open JMeter interface. No need to run in non-GUI mode.
cd <JMeter bin>
./jmeter.sh
  • Execute JMeter script.
  • My Image

View Kafka messages using Kafdrop.

  • My Image
  • My Image
  • My Image
  • My Image

View the address records.

  • Setup MySQL client.
  • My Image
  • Verify the data.
  • My Image

About

Spring Boot, Kafka, Zookeeper, Kafdrop(kafka viewer), Kafka producer, Kafka consumer, Spring Data/JPA, MySQL, Microservices, Rest API

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published