Kafka is a distributed streaming platform that allows you to publish and subscribe to streams of records. In this guide, we will provide links to download and set up Kafka on both Windows and Linux operating systems.
You can download the latest version of Kafka for Windows from the Apache Kafka website.
To download Kafka on Linux, open a terminal and run the following command:
wget https://apache.mirror.digitalpacific.com.au/kafka/3.1.0/kafka_2.13-3.1.0.tgz
This command will download the latest version of Kafka available at the time of writing this guide.
Once you have downloaded the Kafka binaries, follow the steps below to set up Kafka on your system.
-
Extract the Kafka archive you downloaded to a directory of your choice.
-
Navigate to the Kafka directory and open the
config
folder. -
Edit the
zookeeper.properties
file and replace thedataDir
property with a directory path where you want to store ZooKeeper data. -
Start ZooKeeper by running the following command from the Kafka directory: bin\windows\zookeeper-server-start.bat config\zookeeper.properties
-
Open a new command prompt window and navigate to the Kafka directory.
-
Edit the
server.properties
file and replace thelog.dirs
property with a directory path where you want to store Kafka logs. -
Start Kafka by running the following command from the Kafka directory: bin\windows\kafka-server-start.bat config\server.properties
-
Extract the Kafka archive you downloaded to a directory of your choice.
-
Navigate to the Kafka directory and open the
config
folder. -
Edit the
zookeeper.properties
file and replace thedataDir
property with a directory path where you want to store ZooKeeper data. -
Start ZooKeeper by running the following command from the Kafka directory:bin/zookeeper-server-start.sh config/zookeeper.properties
-
Open a new terminal window and navigate to the Kafka directory.
-
Edit the
server.properties
file and replace thelog.dirs
property with a directory path where you want to store Kafka logs. -
Start Kafka by running the following command from the Kafka directory:bin/kafka-server-start.sh config/server.properties
- go to terminal where the python code producer.py is and type
python producer.py
- go to terminal where the consumer.py is and type
python consumer.py stream
Use consumer to create dataset of fake heart data (dataset.json must be present in order to run this command)
(dataset.json is automatically formed when running stream using python consumer.py stream
command)
- go to terminal where the consumer.py is and type
python consumer.py dataset