NOISSUE - Make MQTT Broker Configurable #1904

rodneyosodo · 2023-09-01T10:13:20Z

Signed-off-by: Rodney Osodo socials@rodneyosodo.com

What does this do?

This PR makes the MQTT broker configurable either as VerneMQ or NATS. NATS is a lightweight and highly performant message broker that offers MQTT support, making it a suitable replacement for VerneMQ in this context.

Which issue(s) does this PR fix/relate to?

No issue

List any changes that modify/break current functionality

Add docker profiles to switch between different modes. The available profiles are:
- vernemq_nats: Uses vernemq as MQTT_BROKER and nats as MESSAGE_BROKER.
- vernemq_rabbitmq: Uses vernemq as MQTT_BROKER and rabbitmq as MESSAGE_BROKER.
- nats_nats: Uses nats as both MQTT_BROKER and MESSAGE_BROKER.
- nats_rabbitmq: Uses nats as MQTT_BROKER and rabbitmq as MESSAGE_BROKER.
Integration of NATS as the MQTT broker.

Have you included tests for your changes?

Tested manually

Did you document any new/modified functionality?

absmach/magistrala-docs#159

Notes

To be merged after https://github.com/mainflux/mainflux/pull/1903 and https://github.com/mainflux/mainflux/pull/1902

NATS follows MQTT v3.1.1 specification
Limitations regarding Nats MQTT support can be found here

How to test

NATS as MQTT Broker and NATS as Message Broker

MF_MESSAGE_BROKER_TYPE=nats MF_MQTT_BROKER_TYPE=nats make run up args="-d"
MF_MESSAGE_BROKER_TYPE=nats MF_MQTT_BROKER_TYPE=nats make run_addons up args="-d"

VerneMQ as MQTT Broker and NATS as Message Broker

MF_MESSAGE_BROKER_TYPE=nats MF_MQTT_BROKER_TYPE=vernemq make run up args="-d"
MF_MESSAGE_BROKER_TYPE=nats MF_MQTT_BROKER_TYPE=vernemq make run_addons up args="-d"

NATS as MQTT Broker and RabbitMQ as Message Broker

MF_MESSAGE_BROKER_TYPE=rabbitmq MF_MQTT_BROKER_TYPE=nats make run up args="-d"
MF_MESSAGE_BROKER_TYPE=rabbitmq MF_MQTT_BROKER_TYPE=nats make run_addons up args="-d"

VerneMQ as MQTT Broker and RabbitMQ as Message Broker

MF_MESSAGE_BROKER_TYPE=rabbitmq MF_MQTT_BROKER_TYPE=vernemq make run up args="-d"
MF_MESSAGE_BROKER_TYPE=rabbitmq MF_MQTT_BROKER_TYPE=vernemq make run_addons up args="-d"

codecov · 2023-09-01T11:42:06Z

Codecov Report

Merging #1904 (e9432fa) into master (28f4965) will increase coverage by 0.00%.
Report is 19 commits behind head on master.
The diff coverage is 65.00%.

❗ Current head e9432fa differs from pull request most recent head 87c3384. Consider uploading reports for the commit 87c3384 to get more accurate results

@@           Coverage Diff           @@
##           master    #1904   +/-   ##
=======================================
  Coverage   67.14%   67.14%           
=======================================
  Files         120      120           
  Lines        9145     9144    -1     
=======================================
  Hits         6140     6140           
  Misses       2368     2368           
+ Partials      637      636    -1

Files	Coverage Δ
pkg/messaging/mqtt/pubsub.go	`75.00% <80.00%> (+1.36%)`	⬆️
mqtt/handler.go	`91.93% <71.42%> (-0.07%)`	⬇️
pkg/messaging/mqtt/publisher.go	`23.80% <33.33%> (-1.20%)`	⬇️
pkg/messaging/rabbitmq/pubsub.go	`69.04% <60.00%> (+0.37%)`	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

SammyOina · 2023-09-19T13:00:04Z

pkg/messaging/rabbitmq/pubsub_test.go

@@ -419,8 +419,6 @@ func TestPubSub(t *testing.T) {
 			assert.Equal(t, expectedMsg.Channel, receivedMsg.Channel, fmt.Sprintf("%s: expected %+v got %+v\n", tc.desc, &expectedMsg, receivedMsg))
 			assert.Equal(t, expectedMsg.Created, receivedMsg.Created, fmt.Sprintf("%s: expected %+v got %+v\n", tc.desc, &expectedMsg, receivedMsg))
 			assert.Equal(t, expectedMsg.Protocol, receivedMsg.Protocol, fmt.Sprintf("%s: expected %+v got %+v\n", tc.desc, &expectedMsg, receivedMsg))
-			assert.Equal(t, expectedMsg.Publisher, receivedMsg.Publisher, fmt.Sprintf("%s: expected %+v got %+v\n", tc.desc, &expectedMsg, receivedMsg))


why are these removed

The published message only contains channel and payload

SammyOina · 2023-09-19T13:00:43Z

mqtt/handler.go

@@ -55,19 +55,19 @@ var channelRegExp = regexp.MustCompile(`^\/?channels\/([\w\-]+)\/messages(\/[^?]

 // Event implements events.Event interface.
 type handler struct {
-	publishers []messaging.Publisher


why remove support for mutliple publishers

It was not being utilized. We only used passed one publisher.

pkg/messaging/mqtt/pubsub.go

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The Makefile has been updated to support custom Docker profiles. Previously, the Makefile only supported the default profiles for the message broker and MQTT broker. Now, the Makefile allows for custom profiles to be specified using environment variables. If the MF_BROKER_TYPE or MF_MQTT_BROKER_TYPE variables are not set, the default values "nats" and "nats" will be used, respectively. This change enables more flexibility in configuring the Docker environment for the project. The `run` target has also been modified to use the correct broker configuration file based on the MF_BROKER_TYPE variable. The sed command in the `run` target now replaces the placeholder in the docker/docker-compose.yml file with the appropriate broker configuration file. This commit improves the Makefile to support custom Docker profiles and ensures the correct broker configuration file is used when running the project. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The commit fixes an issue in the RabbitMQ pubsub implementation where the queue binding was not correctly set up. Instead of using the topic as the queue name, the commit now uses a unique client ID generated by combining the topic and subscriber ID. This ensures that each subscriber has its own dedicated queue. The commit also updates the queue binding to use the correct queue name. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The commit refactors the `edit_docker_config` function in the Makefile to improve readability and maintainability. The changes include: - Removing unnecessary conditionals related to the `rabbitmq` broker These changes ensure that the Docker configuration is correctly updated based on the specified MQTT broker type. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The MQTT_BROKER comment in the docker-compose.yml file has been updated to provide a more accurate description of its functionality. The comment now states that the MQTT_BROKER handles MQTT communication between MQTT adapters and the message broker, instead of Mainflux services. This change improves clarity and aligns with the actual purpose of the MQTT_BROKER. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The Makefile and Semaphore configuration files have been refactored to update the variable names related to the message broker type. These changes ensure consistency and clarity in the codebase by using more descriptive variable names related to the message broker type. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Update the Docker profile configuration for nats_rabbitmq by replacing the NATS URL in the .env file with the correct value. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The MQTT QoS level in the pubsub.go file was set to 1, which is the default level. However, since NATS supports up to QoS 1, I updated the QoS level comment to reflect this. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The NewPublisher function in the pkg/messaging/mqtt/publisher.go file has been refactored to accept a new parameter, qos, which represents the Quality of Service level for MQTT message publishing. This change allows for more flexibility in configuring the MQTT publisher. The NewPublisher function now has the following signature: ```go func NewPublisher(address string, qos uint8, timeout time.Duration) (messaging.Publisher, error) ``` This change ensures that the MQTT publisher can be created with the desired QoS level, enhancing the reliability and delivery guarantees of the published messages. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

The test assertions in the pubsub_test.go file were incorrect. This commit fixes the assertions to properly compare the expected and received message values. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

drasko

LGTM

rodneyosodo requested a review from a team as a code owner September 1, 2023 10:13

rodneyosodo marked this pull request as draft September 1, 2023 10:13

rodneyosodo force-pushed the noissue-replace-vernemq branch 2 times, most recently from a8c367f to aca105d Compare September 1, 2023 11:28

rodneyosodo marked this pull request as ready for review September 1, 2023 11:56

rodneyosodo marked this pull request as draft September 1, 2023 12:24

rodneyosodo force-pushed the noissue-replace-vernemq branch 4 times, most recently from da756a1 to 9ac365a Compare September 6, 2023 11:16

rodneyosodo changed the title ~~NOISSUE - Replace VerneMQ With Nats~~ NOISSUE - Make MQTT Broker Configurable Sep 6, 2023

rodneyosodo force-pushed the noissue-replace-vernemq branch 2 times, most recently from 81f7572 to 81eca50 Compare September 6, 2023 13:51

rodneyosodo marked this pull request as ready for review September 6, 2023 14:41

SammyOina requested changes Sep 19, 2023

View reviewed changes

rodneyosodo force-pushed the noissue-replace-vernemq branch from 48103f9 to e57b709 Compare September 20, 2023 10:24

rodneyosodo force-pushed the noissue-replace-vernemq branch 9 times, most recently from c60aeab to 3989996 Compare October 19, 2023 09:18

dborovcanin approved these changes Oct 19, 2023

View reviewed changes

rodneyosodo and others added 15 commits October 19, 2023 20:59

Minor changes on mqtt publisher using nats

0327eaf

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Remove vernemq dependencies

3d015f9

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Restore VerneMQ config files

f124c8a

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Fix failing tests on RabbitMQ

e662fd4

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Fix Docker profile configuration for nats_rabbitmq

a20d3dd

Update the Docker profile configuration for nats_rabbitmq by replacing the NATS URL in the .env file with the correct value. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Rename MF_BROKER_URL to MF_MESSAGE_BROKER_URL

f327f85

Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Fix MQTT QoS level in pubsub.go

77853a2

The MQTT QoS level in the pubsub.go file was set to 1, which is the default level. However, since NATS supports up to QoS 1, I updated the QoS level comment to reflect this. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Fix test assertions in pubsub_test.go

d94c6c7

The test assertions in the pubsub_test.go file were incorrect. This commit fixes the assertions to properly compare the expected and received message values. Signed-off-by: Rodney Osodo <socials@rodneyosodo.com>

Test configurable MQTT broker

87c3384

Signed-off-by: rodneyosodo <blackd0t@protonmail.com>

drasko force-pushed the noissue-replace-vernemq branch from 6aaedfb to 87c3384 Compare October 19, 2023 18:59

drasko approved these changes Oct 19, 2023

View reviewed changes

drasko merged commit 790f8a6 into absmach:master Oct 19, 2023
1 of 2 checks passed

rodneyosodo deleted the noissue-replace-vernemq branch October 22, 2024 08:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NOISSUE - Make MQTT Broker Configurable #1904

NOISSUE - Make MQTT Broker Configurable #1904

rodneyosodo commented Sep 1, 2023 •

edited

Loading

codecov bot commented Sep 1, 2023 •

edited

Loading

SammyOina Sep 19, 2023

rodneyosodo Sep 20, 2023

SammyOina Sep 19, 2023

rodneyosodo Sep 19, 2023 •

edited

Loading

drasko left a comment

NOISSUE - Make MQTT Broker Configurable #1904

NOISSUE - Make MQTT Broker Configurable #1904

Conversation

rodneyosodo commented Sep 1, 2023 • edited Loading

What does this do?

Which issue(s) does this PR fix/relate to?

List any changes that modify/break current functionality

Have you included tests for your changes?

Did you document any new/modified functionality?

Notes

How to test

codecov bot commented Sep 1, 2023 • edited Loading

Codecov Report

SammyOina Sep 19, 2023

Choose a reason for hiding this comment

rodneyosodo Sep 20, 2023

Choose a reason for hiding this comment

SammyOina Sep 19, 2023

Choose a reason for hiding this comment

rodneyosodo Sep 19, 2023 • edited Loading

Choose a reason for hiding this comment

drasko left a comment

Choose a reason for hiding this comment

rodneyosodo commented Sep 1, 2023 •

edited

Loading

codecov bot commented Sep 1, 2023 •

edited

Loading

rodneyosodo Sep 19, 2023 •

edited

Loading