NOISSUE - Adding subtopics filtering in writer services

[jonathandreyer]

What does this do?

Adding subtopic's filtering for each writer service. It is strongly based on channel's filtering.

Which issue(s) does this PR fix/relate to?

Also resolves #925.

List any changes that modify/break current functionality

Have you included tests for your changes?

Did you document any new/modified functionality?

Update readme of each writer service with new

Notes

I have also refactored a little part of the main code of every writer services to centralize duplicate code into writer package.

[codecov-io]

Codecov Report

Merging #1072 into master will increase coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1072      +/-   ##
==========================================
+ Coverage   77.93%   77.99%   +0.06%     
==========================================
  Files          95       95              
  Lines        6743     6685      -58     
==========================================
- Hits         5255     5214      -41     
+ Misses       1174     1159      -15     
+ Partials      314      312       -2     

Impacted Files	Coverage Δ
http/api/transport.go	74.39% <0.00%> (+0.55%)	⬆️
ws/adapter.go	93.75% <0.00%> (+12.21%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70955d1...eab3574. Read the comment docs.

[manuio]

@jonathandreyer update your branch please

[drasko]

I did not yet go into details, but at the first look this looks over-complicated to me. We must brainstorm to find a simpler solution. This introduces impact, and is not something that I'd like to maintain (at least at the first look).

@jonathandreyer can you please describe us your use-case and requirement here in greater detail. From what I understood, you needed to redirect messages to 2 different DBs. Why could not you do it on the channel level?

[jonathandreyer]

I did not yet go into details, but at the first look this looks over-complicated to me. We must brainstorm to find a simpler solution. This introduces impact, and is not something that I'd like to maintain (at least at the first look).

@jonathandreyer can you please describe us your use-case and requirement here in greater detail. From what I understood, you needed to redirect messages to 2 different DBs. Why could not you do it on the channel level?

I fully understand the first point.

Sure, I will try to explain the reason. The explain that, I will take as an example the subtopics in the documentation (https://mainflux.readthedocs.io/en/latest/messaging/#subtopics). In my usage, I have two channels by device (which are provided by bootstrap service) and subtopics for each kind of sended values. I would like store numerical values (related to specific topics, like temperature) in influxDB and string values in another database (mongoDB) for more general information about device, like wake-up, shutdown, etc. With this function (filtering by subtopics), I can select which information I would like to store in which database.

[drasko]

OK, this is something what I thought.

Now second question: what about having just one config file - for example subjects.toml which would list full NATS subject to which Writer should subscribe (instead of having separate channels.toml and subtopics.toml which IMHO complicate the things, especially that it is not clear to me which is a colleration with a certain channel and its subtopics).

[jonathandreyer]

Now second question: what about having just one config file - for example subjects.toml which would list full NATS subject to which Writer should subscribe (instead of having separate channels.toml and subtopics.toml which IMHO complicate the things, especially that it is not clear to me which is a colleration with a certain channel and its subtopics).

@drasko For me, both alternatives are ok. As discussed here (https://github.com/mainflux/mainflux/pull/1072#discussion_r391604923), @manuio think is better to keep both files for retro-compatible reason and also easier to understand. With the resolution of #925, I think the version with one file is OK.

[jonathandreyer]

Hi @manuio & @drasko, I have merged both filters files into one file as explain before but now it has no effect. My knowledge in NATS is pretty poor to subscribe in only selected subjects (here: https://github.com/jonathandreyer/mainflux/blob/4c6576193344b82f067a25f22abf9d4ebbc680b5/writers/writer.go#L45). Have you an idea how proceed?

[drasko]

@jonathandreyer this starts to look good. On the first look just a minor suggestion regarding the docs, but I will analyze with more attention tomorrow.

We will be needing approvals from several maintainers, as this touches lot of code.

[manuio]

LGTM

[nmarcetic]

LGTM!

[drasko]

LGTM

[drasko]

Merged! Thanks @jonathandreyer and @manuio