-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
0 parents
commit f0c2781
Showing
28 changed files
with
111,977 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
example.env | ||
a | ||
*.puml | ||
*.png | ||
*.md | ||
.github | ||
.idea | ||
.gitignore | ||
docker/ | ||
server_exec | ||
client_exec | ||
*.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
name: build | ||
|
||
on: | ||
push: | ||
branches: | ||
- main | ||
pull_request: | ||
branches: | ||
- main | ||
|
||
jobs: | ||
build: | ||
name: Build | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: Set up Go | ||
uses: actions/setup-go@v2 | ||
with: | ||
go-version: '^1.22' | ||
|
||
- name: Check out code into the Go module directory | ||
uses: actions/checkout@v2 | ||
|
||
- name: Build the client | ||
run: | | ||
cd src/cmd/client | ||
go build -o clientexec | ||
- name: Check if client executable was created | ||
run: | | ||
ls src/cmd/client/clientexec | ||
- name: Build the consumerService | ||
run: | | ||
cd src/cmd/server | ||
go build -o serverexec | ||
- name: Check if consumerService executable was created | ||
run: | | ||
ls src/cmd/server/serverexec |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
name: tests | ||
|
||
on: | ||
push: | ||
branches: | ||
- main | ||
pull_request: | ||
branches: | ||
- main | ||
|
||
jobs: | ||
test: | ||
name: Run Go Tests | ||
runs-on: ubuntu-latest | ||
|
||
steps: | ||
- name: Set up Go | ||
uses: actions/setup-go@v2 | ||
with: | ||
go-version: '^1.22' | ||
|
||
- name: Check out code | ||
uses: actions/checkout@v2 | ||
|
||
- name: Get dependencies | ||
run: go mod tidy | ||
|
||
- name: Run tests | ||
run: go test -v ./... |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
# Binaries for programs and plugins | ||
*.exe | ||
*.exe~ | ||
*.dll | ||
*.so | ||
*.dylib | ||
|
||
# Test binary, built with `go test -c` | ||
*.test | ||
|
||
# Output of the go coverage tool, specifically when used with LiteIDE | ||
*.out | ||
|
||
# Dependency directories (remove the comment below to include it) | ||
# vendor/ | ||
|
||
# Go workspace file | ||
go.work | ||
|
||
.DS_Store | ||
.idea/ | ||
.env | ||
a | ||
|
||
*.json | ||
*.js | ||
node_modules/ | ||
server_exec | ||
client_exec |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
MIT License | ||
|
||
Copyright (c) 2024 Stefan Crnojević | ||
|
||
Permission is hereby granted, free of charge, to any person obtaining a copy | ||
of this software and associated documentation files (the "Software"), to deal | ||
in the Software without restriction, including without limitation the rights | ||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
copies of the Software, and to permit persons to whom the Software is | ||
furnished to do so, subject to the following conditions: | ||
|
||
The above copyright notice and this permission notice shall be included in all | ||
copies or substantial portions of the Software. | ||
|
||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
SOFTWARE. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,151 @@ | ||
|
||
[//]: # (Badges) | ||
|
||
[![Author](https://img.shields.io/badge/author-zarnoevic-blue)]() | ||
[![Language](https://img.shields.io/badge/go-1.22.0-blue)](https://github.com/zarnoevic) | ||
[![License](https://img.shields.io/github/license/zarnoevic/go-rabbitmq?color=green)](https://github.com/zarnoevic/go-rabbitmq/blob/main/LICENSE.md) | ||
[![Go Report Card](https://goreportcard.com/badge/github.com/zarnoevic/go-rabbitmq)](https://goreportcard.com/report/github.com/zarnoevic/go-rabbitmq) | ||
[![Build](https://img.shields.io/github/actions/workflow/status/zarnoevic/go-rabbitmq/build.yml?branch=main&event=push&label=build)](https://github.com/zarnoevic/go-rabbitmq/actions/workflows/build.yml) | ||
[![Tests](https://img.shields.io/github/actions/workflow/status/zarnoevic/go-rabbitmq/tests.yml?branch=main&event=push&label=tests)](https://github.com/zarnoevic/go-rabbitmq/actions/workflows/tests.yml) | ||
|
||
# Golang RabbitMQ Server-Client Framework | ||
|
||
This is a simple framework for a RabbitMQ server-client communication. | ||
The server is a consumer service that listens to a queue and processes the messages. | ||
The client is a producer service that sends messages to the queue. | ||
Each message is of a predefined format and denotes a task over an ordered mapping, whose state is kept, accessed, and updated by the server. | ||
|
||
## Results | ||
|
||
On a local Macbook Pro with 16GB of RAM and an M1 chip, the processing of 100,000 commands took `6.46` seconds, with both | ||
the client and the server instances running on the same machine, with the settings as provided in the `example.env` file. | ||
This is from the moment the first record is read from the CSV file on the client until the moment the last record is read from the | ||
RabbitMQ and processed by the server. | ||
|
||
The following image demonstrates the snapshot of the state of RabbitMQ during processing. | ||
|
||
![processing.png](processing.png) | ||
|
||
We notice that the consumption tightly follows the production, with the server processing the messages as soon as they | ||
are produced by the client, as would be expected in a real-time system. | ||
|
||
## Specification | ||
|
||
This is a visual (UML) representation of the framework. | ||
|
||
![img.png](diagram.png) | ||
|
||
|
||
## Environment setup | ||
|
||
Make your own `.env` file from the `.env.example` file. | ||
|
||
```shell | ||
cp .env.example .env | ||
``` | ||
|
||
Then modify the `.env` file to your needs. For local testing purposes, the default settings should work just fine. | ||
|
||
## Running the Project | ||
|
||
The project can be run in two ways: with docker-compose or directly on your machine. | ||
A separate docker-compose file is provided for each service, as well as a dockerfile for each service. | ||
This allows for a scalable deployment of the services, as they can be run in parallel and independently of each other, on different instances. | ||
|
||
### Running RabbitMQ | ||
|
||
Run the following command to start the service: | ||
|
||
```shell | ||
docker-compose -f rabbitClient.docker-compose.yaml --env-file .env up -d | ||
``` | ||
|
||
### Running the Client | ||
|
||
Run the following command to start the Client on your machine: | ||
|
||
```shell | ||
go run src/cmd/client/main.go | ||
``` | ||
|
||
#### With Docker | ||
|
||
```shell | ||
docker build -t client -f client.Dockerfile . | ||
docker run --network="host" --env-file .env client | ||
``` | ||
|
||
#### Scaling the Client | ||
|
||
Run the following command to start the Server in a scalable way with docker compose: | ||
|
||
```shell | ||
docker-compose -f client.docker-compose.yaml --env-file .env up -d --scale client=4 | ||
``` | ||
|
||
Change the `--scale client=4` parameter to any number of clients you'd want to run in parallel. | ||
|
||
### Running the Server | ||
|
||
Run the following command to start the Server on your machine: | ||
|
||
#### Without Docker | ||
|
||
```shell | ||
go run src/cmd/server/main.go | ||
``` | ||
|
||
#### With Docker | ||
|
||
```shell | ||
docker build -t server -f server.Dockerfile . | ||
docker run --network="host" --env-file .env server | ||
``` | ||
|
||
#### Scaling the Server | ||
|
||
Run the following command to start the Server in a scalable way with docker compose: | ||
|
||
```shell | ||
docker-compose -f server.docker-compose.yaml --env-file .env up -d --scale server=4 | ||
``` | ||
|
||
Change the `--scale server=4` parameter to any number of clients you'd want to run in parallel. | ||
|
||
## File structure | ||
|
||
A snapshot of the file structure is provided below using the `tree` command for quick reference. | ||
|
||
``` | ||
. | ||
├── .env | ||
├── go.mod | ||
├── go.sum | ||
├── client.Dockerfile | ||
├── client.docker-compose.yaml | ||
├── rabbitmq.docker-compose.yaml | ||
├── server.Dockerfile | ||
├── server.docker-compose.yaml | ||
├── resources | ||
│ ├── commands100.csv | ||
│ ├── commands100k.csv | ||
│ ├── commands10k.csv | ||
│ └── commands1k.csv | ||
└── src | ||
├── cmd | ||
│ ├── client | ||
│ │ └── main.go | ||
│ └── server | ||
│ └── main.go | ||
└── pkg | ||
├── orderedmap | ||
│ ├── orderedmap.go | ||
│ └── orderedmap_test.go | ||
├── rabbitClient | ||
│ └── rabbitClient.go | ||
└── services | ||
├── consumerService | ||
│ └── consumerService.go | ||
└── producerService | ||
└── producerService.go | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
FROM --platform=linux/amd64 golang:1.22 as builder | ||
|
||
WORKDIR /app | ||
|
||
COPY . . | ||
|
||
RUN go mod download | ||
RUN go mod tidy | ||
RUN go build -o client_exec ./src/cmd/client/ | ||
|
||
CMD ["./client_exec"] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
version: '3.8' | ||
|
||
services: | ||
client: | ||
build: | ||
context: . | ||
dockerfile: client.Dockerfile | ||
environment: | ||
RABBITMQ_USER: ${RABBITMQ_USER} | ||
RABBITMQ_PASSWORD: ${RABBITMQ_PASSWORD} | ||
RABBITMQ_AMQP_PORT: ${RABBITMQ_AMQP_PORT} | ||
RABBITMQ_AMQP_HOST: ${RABBITMQ_AMQP_HOST} | ||
COMMANDS_PATH: ${COMMANDS_PATH} | ||
network_mode: host | ||
volumes: | ||
- ./resources:/resources | ||
|
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
@startuml | ||
' https://plantuml.com/sequence-diagram | ||
|
||
title __Producer-Consumer Model__ | ||
|
||
autonumber | ||
|
||
' General Formatting | ||
skinparam responseMessageBelowArrow true | ||
skinparam lifelineStrategy solid | ||
hide footbox | ||
|
||
' Protocol Participants | ||
box System | ||
database CommandsSource as cs order 5 | ||
entity Producer as p order 10 | ||
queue RabbitMQ as rq order 40 | ||
entity Consumer as c order 70 | ||
database OrderedMap as om order 130 | ||
end box | ||
|
||
' Protocol Context & Assumptions | ||
/ note over om | ||
In-Memory store | ||
end note | ||
/ note over cs | ||
Pre-generated file | ||
end note | ||
/ note over c | ||
Server | ||
end note | ||
/ note over p | ||
Client | ||
end note | ||
|
||
|
||
' Protocol | ||
|
||
cs <-> p : Producer reads the commands | ||
p -> rq : Commands are sent to the queue | ||
loop | ||
rq <-> c : Consumer listens for commands | ||
c->om : Processes get, add, delete, getAll | ||
end | ||
|
||
||| | ||
== Protocol Continues Indefinitely == | ||
@enduml | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
RABBITMQ_USER=admin | ||
RABBITMQ_PASSWORD=i3gW0n4suKr2Ojzi78ShL4pKY3IzZn | ||
RABBITMQ_QUEUE_NAME=commands | ||
RABBITMQ_AMQP_HOST=127.0.0.1 | ||
RABBITMQ_AMQP_PORT=5672 | ||
RABBITMQ_UI_PORT=15672 | ||
|
||
COMMANDS_PATH=resources/commands100k.csv | ||
|
||
SERVER_WORKERS=100 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
module github.com/zarnoevic/go-rabbitmq | ||
|
||
go 1.22.0 | ||
|
||
require ( | ||
github.com/joho/godotenv v1.5.1 | ||
github.com/rabbitmq/amqp091-go v1.9.0 | ||
) |
Oops, something went wrong.