Pigeon is a UDP based message forwarding service that sits between your service and other external services and takes care of:
- Sending your messages (assumed to be via UDP) through the appropriate channel for it to reach another external service (e.g. kafka).
- Receiving your messages (or those you are interested in listening to) from differen channels (e.g. Kafka) and forwarding them to your service via UDP
The main motivation in creating Pigeon is help service developers not worry about the actual transport layer used to communicate across services in the ASED system. Service developers need only concern themselves with sending and receiving UDP messages and add a flavor of Pigeon, like Kafka based Pigeon, which would then forward all messages it receives on a specific Kafka topic to your service via UDP and send all your outgoing UDP traffic to external services via the pre-configured Kafka topics.
Currently pigeon
is able to:
-
Listen for UDP Datagrams and send a Kafka message on multiple Kafka topics
UDP --> Kafka -
Listen for Kafka messages on a specified topic and send them as UDP Datagrams
Kafka --> UDP
To run locally, create a virtualenv
with Python 3.8.3
and install from requirements.txt
:
# Clone repo
% git clone git@github.com:joelthe1/pigeon.git
% cd pigeon
# Setup virtualenv
# Change path to Python according to your system's configuration
% virtualenv -p /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 .venv
% source .venv/bin/activate
# Install dependencies
% pip install -r requirements.txt
# Peek at `Makefile` if you desire to change configurations
# Start pigeon (ensure you have a running instance of Kafka
# and that it is correctly setup in the `Makefile`
% make
# You can follow along logs from the `logs` directory
- Sending Multiple UDP Messages For sending multiple UDP messages try using nping:
# Needs `sudo` to work without errors/warnings
sudo nping --data-string="Test message" --delay=1ms --count=10 -H -N --udp --dest-port 9089 127.0.0.1
nping
also has a few optinos for specifying payload. (Try nping -h
and look for PAYLOAD OPTIONS
)
- Run and Port Forward Kafka
You also need to have Kafka running on your system (usually in
minikube
) and port forwarded on another terminal. Something like:
kubectl port-forward statefulset/ii-kafka 19092:19092
- Read Kafka Messages
To read Kafka messages sent by
pigeon
try using kafkacat:
# Read topic pigeon.test.outbound
# Run on a separate terminal window
kafkacat -C -o end -b localhost:19092 -t pigeon.test.outbound
- Send Kafka Messages
You can use
kafkacat
to send small Kafka messages too:
# Send stdin to pigeon.test.inbound topic. Sends on `enter`.
# Run on a separate terminal window
kafkacat -P -b localhost:19092 -t pigeon.test.inbound