Proof of Concept: MQTT database/provider

[jonnor]

NOTE: This is a quick hack, from playing with Eve and trying to apply it to embedded devices and the physical world. The code is not intended to be merged as-is, just using a PR to have the code available for the discussion.

Background

MQTT is a message broker protocol, commonly for connected embedded devices / IoT / "Internet of Things".
MQTT can be supported on client-side using a WebSocket bridge. Mosquitto, one of the most common brokers, supports this out-of-the-box (behind a configuration option). Only Node.js support is implemented here, so far.

Open questions

• Should it support multiple brokers. This would mean having to declare in a search which broker to use. Eg. [#message #incoming broker: mqtt.bitraf.no topic: 'foo/somename' ]. I'm inclined to say yes, as this makes it a generic transport, usable with 0, 1 or many brokers, with the decisions being done by the Eve programs (not the runtime).
• Should it automatically add tag based on the MQTT topic name? myorganization/temperature/office/value adds tags myorganization temperature office value. Might be nifty, but a bit automagical. Better if done via a separate Eve block, which splits the MQTT topic path, and "enriches" the records with tags?
• Should it automatically inject timestamps for the records? Or is this something done by Eve core?
• Is the logic done by the HTTP server, where the (top-level?) key/values of a request message is split and provided as separate records, desirable?
• How is error handling to be done inside a database provider?

[jonnor]

Showing the current temperature of my old hackerspace via MQTT in Eve:

Example is included in PR.

[convolvatron]

i don't know that we overloaded + in this way? pretty sure the preferred syntax here would
be 'interpolation', as in "eve/test/{{i}}". this inherits implicit string conversion from JS, but
i think there is an ongoing discussion about how much implicit string conversion makes sense

[jonnor]

This was just what I instincivally reached for as a first-day Eve programmer, and it worked, so I didn't change it. :D Agreed that string interpolation is preferred, it is easier to understand the code, and it is less magical than implicit string conversions (which I don't think are very nice at all).

[convolvatron]

what is the intent here? does the payload for each topic contain historical samples that you
want to throw away?

[jonnor]

Each message (on that topic) contains a payload with a temperature value. The sensors send such messages at a fixed messages intervals (every minute in our case, I think). And since the search is for messages, without a sort/filter I get all these messages, including things which are long in the past.

[convolvatron]

i just looked briefly at the MQTT spec, but since its a pub sub model, don't you think it would be
best if this state were modeled explicitly in the mqtt database?

its kind of sad, because the analyze phase can actually look at the eve listeners and
match them up with subscriptions in a lot of cases. but since thats a conservative and
not particularly general process, i guess (?) it would be better to keep an explicit set
of subscription objects underneath each broker

1. multiple-brokers - i think given that this is a general protocol used for lots of different instantiations it would help if this wasn't hardwired in the runtime

2. maybe? it seems like just having the native topic is enough, you can always call split and do message.tag += subtopic

3. that kind of an interesting question. my guess is that unless there is semantic value to the timestamps in the context of eve 'the current eve time' associated with the event is probably sufficient

4. i'm not sure there is a general answer here. it makes a lot of sense given that there is a very close mapping between the property/value nature of http headers and eve records. its pretty helpful to deconstruct that syntax. yeah, it could it be deferred. i think once we have a more settled way of providing layered abstractions in eve then this question just becomes the normal api design one.

5. the current line of thought, which doesn't come with sufficient experience to make it definitive is that errors should be appended to a log of records either in the session bag or in a dedicated bag for storage, later distillation, discard, framed for display to the user, or whatever. i think thats ultimately more flexible than trying to come up with some out of band 'error console'...but we haven't put the plumbing around this..so...its still just a thought

[jonnor]

@convolvatron Not sure I don't understand what you mean by "i just looked briefly at the MQTT spec, but since its a pub sub model, don't you think it would be best if this state were modeled explicitly in the mqtt database?". Which state are you referring to? The latest value from a topic? (as in the example)
I guess that could be built into the database, but there would have to be a way to get at all the messages anyway (since I might want in my Eve program to do some statistics, show values over time etc). So I figured I'd start with that ;)

[jonnor]

@convolvatron Re 3, is "current eve time" associated with a real-world time/date, so I can based on that query for things from November 14? If so, then I think that would be enough.

[jonnor]

Agreed on 5) error handling, putting in records seems like the natural way to go.

[convolvatron]

re: "current eve time" - yes, this isn't 100% cooked, but its either the real time to the best of our ability, or eventually it may be a virtual time, which is is strongly correlated to wall clock time but may also indicate a global serialization order

wrt pub sub model - again, i don't know, but the protocol spec looks like you explicitly contact the broker to subscribe to topics (and presumably the broker works with other brokers to build a distribution tree). so the state i'm referring to is 'which topics i care about'. is that a thing :) ?

[jonnor]

Yes, client must ask the broker to be subscribed to a set of topics. This is often a topic name like temperature/sensor1/value, but it can also be patterns like temperature/?/value (?=wildcard) or temperature/# (anything with this prefix).

The code now does a hacky subscribe to everything (#), and then when messages come in, dispatches based on the exact topic that was sent on. Indeed the database should allow patterns in topics, and only subscribe to these. And probably subscription this should be its own record, separate from the message.

It may make sense to then require pattern to used in subscription records (can be a specific topic name), and then provide both pattern and topic on incoming message records. Then one can search on pattern, but use topic onwards to get the concrete instance.

Afaik, there is no broker-to-broker communication specified in the MQTT protocol. But a broker like Mosquitto allows to "overlay" a set of topics from another broker into a certain prefix. So sending on 'temperature/sensor1' on broker A, may then appear on broker B under brokerA/temperature/sensor1. This is setup in the broker configuration, so not something that changes dynamically or clients can influence.
It can be useful for multi-site setups, or to inject some additional "levels" in the topic namespace without having to go and change the code in all clients (just configure them to use another broker). But only used in relatively advanced installations.

[jonnor]

It's sounding like this integration should now be possible using the newly watchers system, ref http://incidentalcomplexity.com/2017/03/08/february/
Let's see if I manage to find the time for trying that out...

[joshuafcole]

Absolutely, this has been one of my motivating examples behind the new API. It still needs a polish pass, but @cmontella is working on a survival guide to the new DSL and watcher interface.

If you have time to try this out, please let us know any thoughts you have on what works and what doesn't about it, and if you have questions feel free to ping me here or on the mailing list.