Apache Kafka Producer and Consumer which uses Spark-Streaming and Avro Serialization written in Scala.
Steps for execution and corresponding explanation:
- Download Confluent Open Source from https://www.confluent.io/download/ (Tested on v5.0).
- Extract it and inside the directory, run the following command: bin/confluent start
- This will start Kafka, Schema Registry, Zookeeper etc.
- Run mvn clean install
- Run Prod.scala which is a Kafka Producer which generates (Key,Value)
- Key is a Random Integer and Value is a Randomly produced True or False value.
- Run Cons.scala which is a Kafka Consumer which creates a DStream for consuming data produced every 2 secs in Kafka.
- It calculates number of True and False values generated every 2 secs using map() and reduceByKey() functions.