MF-919 - Mainflux message updates #924

dborovcanin · 2019-10-30T17:32:08Z

What does this do?

This pull request removes RawMessage, replacing it with Message. As a consequence of this, different message types are supported using the message Transformer. The concept is similar to what used to be Normalizer service but more generic and cleaner. For more information, please read the corresponding issue (#919).

Which issue(s) does this PR fix/relate to?

Resolves #919.

List any changes that modify/break current functionality

The code is reorganized in such a way that RawMessage is renamed to Message. What used to be Message is moved to trasnformer/senml package because it represents SenML package.

Have you included tests for your changes?

Yes.

Did you document any new/modified functionality?

Yes.

codecov-io · 2019-10-31T10:10:46Z

Codecov Report

Merging #924 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #924   +/-   ##
=======================================
  Coverage   82.47%   82.47%           
=======================================
  Files          75       75           
  Lines        5244     5244           
=======================================
  Hits         4325     4325           
  Misses        659      659           
  Partials      260      260

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d7dc836...d7c505b. Read the comment docs.

drasko

This looks good to me, misses only VerneMQ changes.

Please add them and I will approve.

drasko · 2019-11-03T19:49:25Z

Makefile

@@ -2,7 +2,7 @@
 # SPDX-License-Identifier: Apache-2.0

 BUILD_DIR = build
-SERVICES = users things http ws coap lora normalizer influxdb-writer influxdb-reader mongodb-writer mongodb-reader cassandra-writer cassandra-reader postgres-writer postgres-reader cli bootstrap opcua
+SERVICES = users things http ws coap lora influxdb-writer influxdb-reader mongodb-writer mongodb-reader cassandra-writer cassandra-reader postgres-writer postgres-reader cli bootstrap opcua


If all the addons are added here, probably it would be good to add transformer here as well, so that CI can check it and build the docker (for DoeckerHub usage).

If all the addons are added here, probably it would be good to add transformer here as well, so that CI can check it and build the docker (for DoeckerHub usage).

Normalizer is removed as a service and we're now using Transformers as packages imported by Message consumers to transform messages. Since there is no standalone Transformer service, we don't have it as an add-on, either.

drasko

I would be good to rename also RawMessage record into something like Message or similar.

manuio · 2019-11-04T10:13:59Z

mqtt/verne/src/mfx_auth.erl

@@ -123,7 +123,7 @@ auth_on_publish(UserName, {_MountPoint, _ClientId} = SubscriberId, QoS, Topic, P
                contentType => ContentType,
                payload => Payload
            },
-            mfx_nats:publish(NatsSubject, message_pb:encode_msg(RawMessage, raw_message)),
+            mfx_nats:publish(NatsSubject, message_pb:encode_msg(RawMessage, message)),


We want to keep RawMessage here?

manuio · 2019-11-04T15:33:25Z

cmd/cassandra-writer/main.go

@@ -75,7 +75,7 @@ func main() {
 	defer session.Close()

 	repo := newService(session, logger)
-	norm := normalizer.New()
+	norm := senml.New()


This norm doesn't make sense anymore

anovakovic01

LGTM!

manuio

LGTM

anovakovic01

LGTM!

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

drasko

LGTM

anovakovic01

LGTM!

* Remove RawMessage Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove Normalizer Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update tests Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Replace normalizer with senml-transformer Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Rename Transformer interface and package Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update docs Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove SenML transformer service Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove SenML Protobuf support Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix readers Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix writers tests Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Refactor tests and remove normalizer Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Update docs Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Reanme Service interface to Transformer Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Use msg instead of rawmsg Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove rawMsg from Aedes code Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix VerneMQ files Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove RawMessage code Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix missing subtopic return Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Remove remaining RawMessage reference Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix formatting Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix readers and writers tests Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Rename SenML transformer variables Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com> * Fix readers and writers tests constants Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

dborovcanin added the work-in-progress label Oct 30, 2019

dborovcanin self-assigned this Oct 30, 2019

dborovcanin force-pushed the MF-919 branch from 88915f0 to ebea249 Compare October 31, 2019 10:01

dborovcanin force-pushed the MF-919 branch 5 times, most recently from 65c0c30 to 5410a92 Compare November 1, 2019 13:55

dborovcanin marked this pull request as ready for review November 1, 2019 17:53

dborovcanin requested a review from a team as a code owner November 1, 2019 17:53

dborovcanin removed the work-in-progress label Nov 1, 2019

dborovcanin force-pushed the MF-919 branch 2 times, most recently from f95165b to 27082b5 Compare November 1, 2019 17:57

drasko requested changes Nov 3, 2019

View reviewed changes

drasko requested changes Nov 4, 2019

View reviewed changes

dborovcanin force-pushed the MF-919 branch 2 times, most recently from ac5e643 to 2c32709 Compare November 4, 2019 10:02

manuio suggested changes Nov 4, 2019

View reviewed changes

anovakovic01 previously approved these changes Nov 4, 2019

View reviewed changes

dborovcanin dismissed anovakovic01’s stale review via f7a9bdf November 4, 2019 16:22

manuio previously approved these changes Nov 4, 2019

View reviewed changes

nmarcetic previously approved these changes Nov 4, 2019

View reviewed changes

anovakovic01 previously approved these changes Nov 5, 2019

View reviewed changes

dborovcanin added 5 commits November 5, 2019 10:25

Remove RawMessage

f8f304b

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Remove Normalizer

10f3bf1

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Update tests

5ca20c1

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Replace normalizer with senml-transformer

3062537

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Rename Transformer interface and package

8239c24

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

dborovcanin added 16 commits November 5, 2019 10:27

Remove SenML transformer service

93ddefc

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Remove SenML Protobuf support

deea760

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Fix readers

9e01ba5

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Fix writers tests

26be726

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Refactor tests and remove normalizer

994c5b9

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Update docs

cc848d4

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Reanme Service interface to Transformer

2703b6b

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Use msg instead of rawmsg

3a9d8b0

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Remove rawMsg from Aedes code

f6d55cd

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Fix VerneMQ files

84fe05e

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Remove RawMessage code

a081b56

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Fix missing subtopic return

106792b

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Remove remaining RawMessage reference

d1a4d13

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Fix formatting

52d90bf

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Fix readers and writers tests

1eb9d19

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

Rename SenML transformer variables

63df28b

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

dborovcanin dismissed stale reviews from anovakovic01, nmarcetic, and manuio via d8f9a2b November 5, 2019 09:38

dborovcanin force-pushed the MF-919 branch from f04786a to d8f9a2b Compare November 5, 2019 09:38

Fix readers and writers tests constants

b6d423d

Signed-off-by: Dušan Borovčanin <dusan.borovcanin@mainflux.com>

dborovcanin force-pushed the MF-919 branch from d8f9a2b to b6d423d Compare November 5, 2019 09:40

nmarcetic approved these changes Nov 5, 2019

View reviewed changes

Merge branch 'master' into MF-919

d7c505b

drasko approved these changes Nov 5, 2019

View reviewed changes

anovakovic01 approved these changes Nov 5, 2019

View reviewed changes

nmarcetic merged commit f50335a into absmach:master Nov 5, 2019

dborovcanin mentioned this pull request Nov 15, 2019

Runtime error in normalizer - CBOR SenML #950

Closed

dborovcanin deleted the MF-919 branch February 17, 2021 09:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MF-919 - Mainflux message updates #924

MF-919 - Mainflux message updates #924

dborovcanin commented Oct 30, 2019

codecov-io commented Oct 31, 2019 •

edited

Loading

drasko left a comment

drasko Nov 3, 2019

dborovcanin Nov 4, 2019

drasko left a comment •

edited

Loading

manuio Nov 4, 2019

manuio Nov 4, 2019

anovakovic01 left a comment

manuio left a comment

anovakovic01 left a comment

drasko left a comment

anovakovic01 left a comment

MF-919 - Mainflux message updates #924

MF-919 - Mainflux message updates #924

Conversation

dborovcanin commented Oct 30, 2019

What does this do?

Which issue(s) does this PR fix/relate to?

List any changes that modify/break current functionality

Have you included tests for your changes?

Did you document any new/modified functionality?

codecov-io commented Oct 31, 2019 • edited Loading

Codecov Report

drasko left a comment

Choose a reason for hiding this comment

drasko Nov 3, 2019

Choose a reason for hiding this comment

dborovcanin Nov 4, 2019

Choose a reason for hiding this comment

drasko left a comment • edited Loading

Choose a reason for hiding this comment

manuio Nov 4, 2019

Choose a reason for hiding this comment

manuio Nov 4, 2019

Choose a reason for hiding this comment

anovakovic01 left a comment

Choose a reason for hiding this comment

manuio left a comment

Choose a reason for hiding this comment

anovakovic01 left a comment

Choose a reason for hiding this comment

drasko left a comment

Choose a reason for hiding this comment

anovakovic01 left a comment

Choose a reason for hiding this comment

codecov-io commented Oct 31, 2019 •

edited

Loading

drasko left a comment •

edited

Loading