Querying and sending undefined telegram types (e.g. Holidaymodes- and Switchtimes)

[tp1de]

Following a discussion some months ago, I am still in favor to implement additional functionality on Holiday modes and Switchtimes outside of implementing this in the ems-esp firmware. E.g. in the ioBroker-Adapter.

These telegram types needs polling and the telegrams received needs to be analyzed. I do have an implementation since 12 months within my ioBroker Adapter based on a syslog server reading all raw telegrams. This is nasty but it works for read/write of this telegram types. I haven't made it public to the more than 700 users of the ioBroker adapter.

These telegrams can queried (polled) by sending raw commands (api or mqtt).
Using mqtt the response is pushed to ems-esp/response topic. Api can't read the response.
Since the ioBroker adapter just uses api I would like to implement this without the need of a mqtt broker.

I do have 2 ideas how this could be done:

1. Having the (last) response to be readable by api http get - no push needed!
2. Enabling "Raw telegrams" within Settings/Custom Entities --> new Value Type Raw with Read / Write Capabilities

@proddy @MichaelDvP Whats's your opinion about this?

[MichaelDvP]

I think both is possible, but needs some refactoring.
I'll test some options:

This will work after `system/send command, also for all other cases of mqtt response.
For the custom entities:
This works in my test for reading, but i have issues with writing for now,


But i'm not sure what is usefull. Needs some more testing and think about.

[MichaelDvP]

@tp1de Please test https://github.com/MichaelDvP/EMS-ESP32/releases/tag/test
The api command is /api/system/response. The response is always generated, the mqtt-setting is only to push it to mqtt.
For custom entities remember that each custom entity will send a ems-telegram each minute and tx-queue is 50, so more entities will not work.

[tp1de]

Thanks Michael I will test. I think that the api response is good enough.

[tp1de]

Michael I did a first test:
Holiday modes with data length of 18 bytes are working fine. Both api/mqtt response and api/mqtt request as well as custom entities read are working.
Switchtimes are not really working since their length are 84 bytes. A mqtt request and custom entities give a data length of 25 bytes. An api request does not give any reply - neither on mqtt nor on api.

BTW: the actual data values (raw) can be read by /api/custom. The details by adding the entity name. The JSON is not consistent with the custom entity definition and should contain the definition fields of the custom entity.

{
"name": "HM1",
"uom": " ",
"type": "number",
"readable": true,
"writeable": false,
"visible": true,
"value": "09 01 01 09 01 01 03 11 02 FF FF 00 00 00 00 00 00 FF"
}

[MichaelDvP]

A mqtt request and custom entities give a data length of 25 bytes.

The requests always give the telegram number of bytes you have specified from offset. Maximum length of a single read is 28 bytes for a ems- type and 26 bytes for ems+. To read longer telegrams like switchtimes you have to read in parts. Or for the custom entities make a entity for each dataset, eg: "sw1" offset 0, bytes 12, "sw2" offset 12 bytes 12, etc. That's why i warned about the tx-queue, all switchtimes and holiday entries give more than 50 entities/busreads. Using api/system/send is uncritical for queue, because you have to wait for response for each send.
The for system/send you also have to send multiple offsets to get all values.

[tp1de]

OK understood for the need of multiple telegrams .... I already thought that this might be needed. I will continue testing.

[tp1de]

Just a remark/question:

For system/send i need to use type 0169 and for system/response I get back type 0269.
Is this foreseen to be kept for ever?

[MichaelDvP]

For system/send i need to use type 0169 and for system/response I get back type 0269.
Is this foreseen to be kept for ever?

Yes, the ems+ type_id 0xFF0169 and ems-esp internally shorten to 16bit as 0x269. In a raw telegram you have to use the full 24bit address, for all ems-esp internal (like read 10 269 use the 16bit type_id. If you change the logging to raw, you also see the 24bit value in the telegram. This shortening is common, Norberts also use it, but notated as decimal value.

I added some more tags to the custom-value-info.

[tp1de]

I was aware about this but even now being part of the ems-esp community for quite a while , it's still a bit confusing requesting one type (within poll request) and getting back another one. I understood that this will stay for the future?

[tp1de]

I had another look in my syslog server coding.
When I do there a poll request with length 20 on the switchpoints, syslog shows 4 successive telegrams with offsets 0, 25, 50 and 75. Since the telegrams are successive, it's quite easy to create one long telegram. Isn't there a chance to do this similar in ems-esp?

I tested that otherwise sending 4 requests I need to introduce a wait cycle of 4 times 1 second to read the responses and conduct the whole telegram. For updating I need to send the telegram part where switchtimes where changed. (could be 4 telegrams)

[MichaelDvP]

When I do there a poll request with length 20 on the switchpoints, syslog shows 4 successive telegrams with offsets 0, 25, 50 and 75

Yes, this was a bug. The read command without a length reads the whole telegram and logs the parts to the console.
As kind of a hack api/system/send uses the read_id to get the telegram and send the response. This results in getting the last response from api will always show only the last part of the telegram (mqtt pushes every part). I've changed to clear structure to respond exactly the queried part of the telegram.

[MichaelDvP]

@tp1de Please check with my new build. I've changed the response to show the complete telegram with all parts included if the raw request is length 0x20. The log is then compatible with the previous versions. For the response all reported values are in hex (including offset).
Example: response to system send "48 90 49 00 20" (theses are the switchtimes of hc2 in my RC35, my test emsesp uses devices 48)

{
  "src": "10",
  "dest": "48",
  "type": "49",
  "offset": "00",
  "data": "01 23 00 81 21 23 20 81 41 23 40 81 61 23 60 81 81 23 80 81 A1 26 A0 81 C1 2A C0 81 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 E7 90 00 00 00 01 01 14 0C 01 14 01 01 00 01 01 00"
}

Using smaller length responses on the requested part.

[tp1de]

Thanks Michael, this is a very good adjustment. I had a chance to test it using VPN since I am not at home this week.
The send command with length 0x20 works fine. I need a read delay about 3 seconds after the send command to get a reliable read of the whole telegram with length of 84 Bytes.

[tp1de]

@MichaelDvP Would your solution work for sending the whole length telegram on changes at once?
Or do I need to send telegrams with max length of 25?

[MichaelDvP]

I've checked what is needed to accept longer raw send messages and split in valid ems-telegrams. It's possible, but we have to increase all mqtt/api/parser buffers by factor 10 to hold the input. Not sure if this is really usefull.

[tp1de]

Thanks Michael for your thoughts. Long telegrams for sending are not necessary from my point of view. The read of this telegrams is a very good improvement.

I will implement holidaymodes and switch times for RC300/310 within my ioBroker Adapter the same way as they come from km200. Are there other holidaymodes or switchtime formats I should implement?

[MichaelDvP]

The RC35/RC30 is already implemented in a simple way, only reading/writing on switchpoint at a time. Holiday/vacation is only one per hc in the same telegram, implemented as entity with text-string "dd.mm.yyy-dd.mm.yyy"
Telegrams are described here: https://emswiki.thefischer.net/doku.php?id=wiki:ems:telegramme#hk1schaltzeiten
and https://emswiki.thefischer.net/doku.php?id=wiki:ems:telegramme#schaltzeiten_allgemein
I don't think there is more needed.

BTW: a while ago i've tested to create a json output like this:
switchtime.txt
But its to long for publiishing in thermostat_data and only api reading is imho useless.

[tp1de]

Ok understood. I will rewrite my iobroker adapter within the next weeks anyhow to make it more foolproof. The iobroker km200 adapter is end of life with a couple of thousand users. If the majority will use the ems- esp adapter I don't want to be drowned by questions.

Do you know why Buderus/ Bosch has stopped selling the km interfaces? The new ones does not seem to support local LAN api anymore.
This might end in much more ems- esp gateways to be used.

[MichaelDvP]

Do you know why Buderus/ Bosch has stopped selling the km interfaces? The new ones does not seem to support local LAN api anymore.

i think the same as mostly in cloud strategie:

1. Sell cloud only products, stop other.
2. Increase cloud fee
3. repeat step 2 a few years
4. sell new products, new cloud only, stop old cloud (make old products useless).
5. continue with step 2.

[tp1de]

@MichaelDvP I am currently testing the implementation for the EMS+ holidaymodes and switchtimes within my ioBroker adapter.
This works fine so far (I just tested reading) to produce 1:1 the same content compared to the km200 gateway.

I recognized that I need to wait around 6 seconds between send and get request for stable reading.
I still use your repository (S3 bin). I recognized that this addition is not in dev or dev2.
What are planned timings for releasing this addition?

[MichaelDvP]

Yes, it takes a while to get all parts of the telegram. The timing is from busmaster, we read as fast as the busmaster sends the polls to us.

It is in dev2 in my repo and here. See commit: c47414d

[tp1de]

Yes timing varies largely. I changed the code and I am trying now every second to read the response until getting the right response matching the request.
It takes between 2 and 5 loops. That's faster then having a fixed wait time of 6 seconds.

These raw send requests just produce a lot of "notice" entries within the log. Can this be avoided / switched off?

[MichaelDvP]

I speed up the response by requesting all telegram parts within one master-poll. But it still needs some time to wait for the first poll.
Now the logging is only for read, not for response.

[tp1de]

Looks better

[tp1de]

... and it's 2 to 3 times faster 👍

[tp1de]

@MichaelDvP I just recognized that with the last change the telegram length for response of holidaymodes changes.
Requesting 18 bytes gets now a response of 19 bytes.

[MichaelDvP]

Are you requesting 18 bytes (0x12) or 0x18 = 24 bytes?

[tp1de]

Haven't touched this since a year .... yes you are right request length is 0x18 and not 0x12. Sorry .....

[tp1de]

But finally everything works very fine now. I am really happy about this request/response function. Average response time is now between 500 to 1000 msecs. Just sometimes up to 2 secs.