Adding KNX support

[Ing-Dom]

Hi there,

I'm really stuned how great this device is. 8 years I fuck around with that shitty KM200 API, and now I have all this data right to my hands. Great.

While I'm using HomeAssistant everything works, but for an even better and more direct integration into my smarthome support of KNX-IP would be great - I'm one of the founders and core devs of @OpenKNX and I could think of adding our knx stack and some funtionality into it. So the device could then communicate with KNX-IP-Routing and could be configured with ETS.

I would bring in the knx stack, and the know how to use it and also I can create the knxprod file.
But while my time is limited I would appreciate someone experienced with the architecture of this firmware so we can work together.

[MichaelDvP]

What do you need? I've tried to find some free infos how KNX works, but most is commercial.
Is there a server which requests the info from emsesp (like api or the modbus branch) or should emsesp push the changed parameters (like mqtt)? Are the datapoints numbers (register) like modbus or named?
What infos are needed for a datapoint?

[Ing-Dom]

well, KNX works in way that there are so called "group addresses" that have 16bit, are unique in one project / installation, are noted as x/y/z and normally represent some value.
Normally values are written by change or periodically, but can also be read on demand.
A device normally has configurable datapoints (so called "communication objects") which can be configured by parameters with a tool called ETS.

To add knx support I would need a udp multicast and a unicast socket, some kb remanent storage, beeing called about every 5ms and I would need read/write access to the data..

[MichaelDvP]

Thanks, but i still don't know how it works.

To add knx support I would need a udp multicast and a unicast socket, some kb remanent storage, beeing called about every 5ms and I would need read/write access to the data..

Ok, that's possible. For the udp-sockets i've found an example, that can be adapted. For the storage we have a filesystem (littleFS) with 1M on the 16MB chips for larger files and nvs 250k optimized for storing single values. Both flash storage have wear management, but we should look about writing to often.
The data can be be accessed by the devicetpe/tag/shortname, also possible to trigger an action on change of each value independent.
We have the new modbus integration in https://github.com/emsesp/EMS-ESP32/tree/feat_modbus which maps the values to modbus register numbers, see https://github.com/emsesp/EMS-ESP32/blob/feat_modbus/src/modbus_entity_parameters.cpp Maybe for KNX there is something simlar possible.

[MichaelDvP]

I've started a knx class, based on AsyncUDP with a multicast server and unicast client/server. Also a onChange routine, called by emsesp on every value changed. For the values i stick to char* for device, tag, name and value. Don't know what format knx prefer.

I see that tasmota also support knx see here and uses this library, maybe a better way?

I'm in vacation for next 3 weeks (without computer).

[Ing-Dom]

There is only one open source stack that aims to be knx compliant, and this is this one:
https://github.com/OpenKNX/knx
with the other you won't be able to program with ETS.

hardware abstraction is done in the plattforms, we already have one for esp
https://github.com/OpenKNX/knx/blob/v1/src/esp32_platform.cpp

For persistent data currently EEPROM emulation is used - is that used in EMS-ESP or is this "free" to use for the knx stack?
But there could be an extra plattform if we need to taylor it.

fortunately, we use very similar environments and also PIO so I think i can setup this and try a little bit in the next future.

How frequently is Knx::loop() called? Is there blocking stuff that causes loop not beeing called for more than a couple ms?
It is not that critical with KNX-IP but would be good to now nevertheless..

[MichaelDvP]

hardware abstraction is done in the plattforms, we already have one for esp
https://github.com/OpenKNX/knx/blob/v1/src/esp32_platform.cpp

Ok, we can use this. EEPROM uses nvs in the default partition, we have 20k there (only few bytes used), On the ESP with16M flash we have a larger second nvs, we can switch the eeprom lib if needed.

How frequently is Knx::loop() called?

normally and average <1ms loop time, but we have the dallas one-wire reads that are blocking every 5 seconds, then loop time is ~ 50 ms. Maybe better to do the loop asynchron in an own task.

[MichaelDvP]

I've changed to async loop and added the functions from https://github.com/OpenKNX/knx/blob/v1/src/esp32_platform.cpp
The eeprom seems to be a block read/write of a ram buffer, no need to use a complete eeprom library.

[Ing-Dom]

normally and average <1ms loop time, but we have the dallas one-wire reads that are blocking every 5 seconds, then loop time is ~ 50 ms. Maybe better to do the loop asynchron in an own task.

I think 50ms every few minutes would cause no problem, but if there is a dedicated task thats even better. You are using an RTOS ?
I'm not so familiar with ESP developing Im more into RP2040

I've changed to async loop and added the functions from https://github.com/OpenKNX/knx/blob/v1/src/esp32_platform.cpp The eeprom seems to be a block read/write of a ram buffer, no need to use a complete eeprom library.

Basically the stack needs a pointer where it can read/write data, and if it has finised writing, it calls commit.
Thats it. There is also a native flash support that reduces the size of the needed ram, but that is not neccessary here I think.

I tried to set it up, but got stuck with yarn. I'm developing on Win - installed node.js from the installer with also all the other tools including yarn (that took some while)..
still get this build error that yarn ist not here.
I ran that corepack command from a admin console but that changed nothing.
Is there more to do or do I have to call that corepack cmd in a specific directory?

[proddy]

what yarn errors are you getting?

[Ing-Dom]

when I build with PIO, I get the error that yarn is not found

[MichaelDvP]

corepack enabelshould do, see https://v3.yarnpkg.com/getting-started/install

[Sbried]

Do you have still any activities ongoing regarding the KNX integration in EMS-ESP? This is my key point for using EMS-ESP.
Until 2 month ago I had a Viessmann boiler connected through the Vitogate 200 KNX to my KNX installation. This covered all my requirements and ran stable over years.
I decided to switch to a heat pumpe and now I have a Buderus WLW166i. I ordered the heat pumpe with KNX gateway and all was confirmed. They delivered the heat pump but without KNX gateway - explanation: The future is the MX300 gateway (cloud connection to Buderus). But what I absolutely do not want is something connected to clouds. The sale of the KNX gateway was stopped because of the EU directive 2019/770 (contracts for supply of digital content and services) because the hardware was not designed to guarantee the update capability required by the EU directive. And a new KNX gateway has somehow priority Z in the development as the turnover is considered as low.
So, my way out of the dilemma was EMS-ESP. And now after 2-3 weeks playing around I consider EMS-ESP anyhow as the better alternative. Much more interaction with the heat pump is possible than through the KNX gateway.
Currently I talk to the heat pumpe through the API and put the results on the KNX bus, but a direct connection from EMS-ESP to KNX would be much better.

[MichaelDvP]

I don't use KNX and don't know how it works. Seems most documentation is behind paywall.
5 month ago i tried to help and build a branch https://github.com/MichaelDvP/EMS-ESP32/tree/feat_knx
based on the info i had. I never get any feedback, so now the branch is 1100 commits behind dev. But i don't spend time to keep a dead project up to date.

[Ing-Dom]

every that could shit. that is so ridiculous!

I'm defiitly interested in knx support (with ETS) for this firmware, and I'm willing to contribute with my OpenKNX knownledge.
But, time is limited and I am quite busy with my OpenKNX stuff :) so things got a little bit into background after I wasn't able to build the project my own.

at this moment, I'll give it a second try.
cloned the dev branch, installed node.js (msi installer) run corepack enable and yarn init -2 (cmd.exe with admin priv) on Win10.
Waiting for VSC to complete all the prepartions..

I assume from the infos from https://emsesp.org/ that I need to use ci_s_16M_P env for my "Gateway E32 V2 (Ethernet + WiFi Edition V2)" I purchased at BBQKees

I'm sure if you help me understanding how to develop for that project and how to integrate, we can achieve a good solution !

Question for potential KNX-User:

what are your thoughts regarding the ETS integration?
Are the entities "dynamic" or is there a list of them, with allowed values and read/write info at compile time?

PS: my build is failing:

Archiving .pio\build\ci_s_16M_P\libbfa\libeModbus.a
Archiving .pio\build\ci_s_16M_P\lib69d\libOneWire.a
Compiling .pio\build\ci_s_16M_P\libd5d\framework\UploadFileService.cpp.o
Compiling .pio\build\ci_s_16M_P\libd5d\framework\WiFiScanner.cpp.o
lib/framework/ESP8266React.cpp:3:10: fatal error: WWWData.h: No such file or directory

*****************************************************************
* Looking for WWWData.h dependency? Check our library registry!
*
* CLI  > platformio lib search "header:WWWData.h"
* Web  > https://registry.platformio.org/search?q=header:WWWData.h
*
*****************************************************************

 #include "WWWData.h" // include auto-generated static web resources
          ^~~~~~~~~~~
compilation terminated.
*** [.pio\build\ci_s_16M_P\libd5d\framework\ESP8266React.cpp.o] Error 1
========================================================================================== [FAILED] Took 177.17 seconds =========

[Ing-Dom]

@MichaelDvP
I totally understand that this branch was not kept up to date. And I appreciate the efforts you put into that feat_knx.
I just suceeded to build the dev branch, I'll continue and check if I can build the feat_knx.

Can I use my ESP32 V2 Gateway for testing and load fw over the USB port (as it is normally done for ESP devices?) Do I have to do someting else besides "upload" in VSC/PIO ?

Lets go a little bit deeper into that knx stuff. You have to know that knx devices are parametrized over a central Tool, that is called ETS. What features / parameters and "datapoints" (knx calls them group objects) a device has, is defined in s special file format. (*.knxprod).
This is essential and while this is normally only for members of the KNX Association, we from OpenKNX found a way to create them.
This knxprod is created once and must be the same for every user. So one must fit all (everything else is impractical).

If we just ignore the fact that the entities are "dynamic" one would create such a knxprodfile with every entity have a "group object" which then can be ssigned to a "group address" in the ETS so that it can be read and written over KNX. One would also add some parameters for every entity to define it it is only sent when data changes, or periodically, or when it changed more than 1% or ....
also, the datatype of this group object would be defined there (in KNX all datatypes are defined: https://support.knx.org/hc/en-us/article_attachments/15392631105682).

I currently don't know how to deal with the fact, that the entities are dynamic... and differ dependent of the connected device.

Maybe you can tell me how the EMS-ESP32 gets to know all the entities and their allowed values, dataformat and so on.. ?
Is there a list of all possible entities in the Code? Or is this "discovered" over EMS ?

[Sbried]

I currently don't know how to deal with the fact, that the entities are dynamic...

I have only one heat pump and don´t know if entities are dynamic or not. It was only an assumption as some entities make no sense for certain devices. But @MichaelDvP might know.
At least the number of entities increases due to more and more being re-engineered.

I could help with the creation of a *.knxprod if I would know the format/tool. I have knowledge about the KNX data types as well as EMS data types and I have also ETS6 and could do some tests.
We should first define a subset of entities to start with. I would appreciate if @MichaelDvP could define a standard set of entities used everywhere, if possible.

With my 2!! weeks experience with EMS I understood that the bus master (KNX has no bus master) periodically asks around who is connected to the EMS bus. And also the slaves can ask. But also here @MichaelDvP will be the better one to answer.

[MichaelDvP]

@Ing-Dom
I've merged the knx and current dev branch.
Now up to date: https://github.com/MichaelDvP/EMS-ESP32/tree/dev_knx
I'll remove the outdated feat_knx.

The entities depends on the registered devices. For ems proddy generates a list here: https://github.com/emsesp/EMS-ESP32/blob/dev/dump_entities.csv
This is also used to generate the modbus list here https://github.com/emsesp/EMS-ESP32/blob/dev/src/modbus_entity_parameters.hpp

The parameters for each entity can be read with api command, or in webpage customization, click on shortname.

[Ing-Dom]

For ems proddy generates a list here: https://github.com/emsesp/EMS-ESP32/blob/dev/dump_entities.csv

Jesus, thats a lot... - but this is the information I was searching for.

Do I understand it right - the entities of a certain device (e.g. a "Enviline/Compress 6000AW/Hybrid 3000-7000iAW/SupraEco/Geo 5xx/WLW196i/WSW196i") are fixed? So if device X is connected to the EMS-Bus, all the entities for this device are present and can be read?

[MichaelDvP]

Yes, these entities are registered if the device is identified and can be read. But depending on the individual configuration they may not have a value.

[Ing-Dom]

I suceeded in integrating the knx stack to a point where the stack is actually built :) thats a first step, some to go.

Question:
whats your favorite way to integrate the knx stack ?
I normally use lib_deps for that, but it seems not to be your way - you rather copy them into your repo? (Something I can understand, because github repos tend to vanish from time to time...)

[proddy]

lib_deps is fine. We hold local copy of libraries because we're either tweaking them a lot or we want to be able to run EMS-ESP standalone without using the PlatformIO platform. We have a GNUMake Makefile that builds and runs EMS-ESP standalone without an ESP32 microcontroller that we use for testing

[Ing-Dom]

I'm now at a point where I would like to give it a first try running it on hardware.
I have a BBQKees GW, but it is in use, so not the ideal thing to mess around - and for the first test i do not need the EMS Bus anyway.

How do you HW definition work? I found something with board_profiles but I could not figure it out how to work with that?

I would like to use it with a custom ESP32 devboard I have here (ESP32-PICO-V3-02 + LAN8720) - how do I define the specific pins?

[proddy]

If you select CUSTOM as the board profile from the WebUI you can set your own GPIOs. https://emsesp.org/Configuring/#board-profile

[Ing-Dom]

understood.. so the assumption is that WIFI is always present - which is not at this specific board I wanted to use (no antenna ..).

But - when I understand you documentation right - there is the possibility to do this setting via the console, which is also availible at UART0 ? Will try this this evening...

[proddy]

you can set the board profile via the serial console yes, but not the specific pins. I thought those ESP32 PICOs come with WiFi? Anyway you have LAN8720 so it'll just be plug&play hopefully.

[MichaelDvP]

Is the pico+lan a complete module or two modules? For 2 modules try to use thesame pins as the gateway e32v2, see here:

EMS-ESP32/src/system.cpp

Line 1779 in 8bab757

bool System::load_board_profile(std::vector<int8_t> & data, const std::string & board_profile) {

The pico has 8M flash and 2M PSRAM, if you want to program via USB yoou'll need a different env in platformio.ini:

[env:s_8M_P]
extends = espressif32_base
board = esp32dev
board_upload.flash_size = 8MB
board_build.partitions = esp32_partition_8M.csv
board_build.extra_flags = -DBOARD_HAS_PSRAM

and a esp32_partition_8M.csv:

# Name,   Type, SubType, Offset,   Size,    Flags
nvs,      data, nvs,     0x9000,   0x005000,
otadata,  data, ota,     ,         0x002000,
app0,     app,  ota_0,   ,         0x350000,
app1,     app,  ota_1,   ,         0x350000,
nvs1,     data, nvs,     ,         0x040000,
spiffs,   data, spiffs,  ,         0x200000,
coredump, data, coredump,,         0x010000,

[Ing-Dom]

thanks, I give this a try. I used the 4MB environemt but it crashed.
but its not so crucial anymore as I now test on the BBQKees Hardware.

[Ing-Dom]

Currently (as mentioned in the Discord also) I have trouble receiving multicast packages.

With a basic sketch I can receive the udp multicast packages using WifiUdp.beginMulticast().
With the exact same HW setup, but with adapted EMS-ESP32 firmware, there are no packets comming in.
But with using WifiUdp.begin() (so, normal unicast) packets are comin in, so the general network stuff works fine ! Only Multicast is troubling...

Any ideas?

[Ing-Dom]

a "erase flash" in PIO resolved the invalid mbox feature. thats weird..

Now, I can successfully receive Multicast packets !!!

Another strange thing:
I have now no more knx options in the Application Settings in the webinterface. It is not there.
I definitly changed nothing with that setting in the code.
And when I simply go to the Application setting in the webinterface, Serial console shows

abort() was called at PC 0x402000ab on core 1


Backtrace: 0x40083f21:0x3ffde7d0 0x40090545:0x3ffde7f0 0x40095d21:0x3ffde810 0x402000ab:0x3ffde890 0x402000f2:0x3ffde8b0 0x40200053:0x3ffde8d0 0x40201183:0x3ffde8f0 0x40164ca5:0x3ffde910 0x40165b8a:0x3ffde940 0x4013e88e:0x3ffdec50 0x4013f5d6:0x3ffded10 0x4013f71e:0x3ffdedc0 0x4014ee62:0x3ffdedf0 0x4014baed:0x3ffdeed0 0x4014bb61:0x3ffdef20 0x402043ed:0x3ffdef40 0x40204465:0x3ffdef70 0x40204af6:0x3ffdef90




ELF file SHA256: 37be343550ca780a

Rebooting...

but, after some waitig I get the page displayed.

I bet this is somehow linked together...

First of all, I think we need to find out WHY the knx settings are not display in my webinterface, altough I have the dev_knx branch loaded after a erase, and that branch definitely contains that setting parameters.. (and they were there once.. I have seen them !!!)

[MichaelDvP]

I have now no more knx options in the Application Settings in the webinterface.

Old browser cache? CTRL-R.

[Ing-Dom]

indeed...

[Ing-Dom]

I now have a very intersting discovery !
Only when I do a complete flash erase and only in the first run of the programm, i am able to receive multicast packets:

 *  Executing task: C:\Users\dosp\.platformio\penv\Scripts\platformio.exe device monitor --environment s_16M_P --port COM54 

--- Terminal on COM54 | 115200 8-N-1
--- Available filters and text transformations: colorize, debug, default, direct, esp32_exception_decoder, hexlify, log2file, nocontrol, printable, send_on_enter, time
--- More details at https://bit.ly/pio-monitor-filters
--- Quit: Ctrl+C | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H
readMemory
RESTORED 88 8A 2A 02 AA AA 28 88 C2 AD AA AA AA 28 A8 A0 02 0A 0A 88 80 80 A0 80 AA 8A A8 02 AA 20 28 A2 A8 0A 88 0A A2 8A 00 80 DA AD AA AA AA A8 08 AA AA 2A 00 08 00 0A A8 A0 8A AA 00 88 88 8A 20 A2 88 08 2A 02 08 A0 88 8A D2 AD AA AA AA AA A0 28 8A 02 0A 00 A0 88 28 82 A2 0A 08 22 A0 28 00 20 88 08 AA 80 AA A8 A0 22 2A AD AA AA 0A A0 00 22 2A 8A 28 88 A2 08 80 2A 8A 88 A2 02 88 A8 00 20 8A 08 80 00 08 08 AA 82 22 AD AA AA AA A2 80 A0 8A 08 2A AA 82 82 A8 8A
DataObject api changed, any data stored in flash is invalid.
expexted DataObject api version: 2, stored api version: 888A
ETS has to reprogram PA and application!
setup multicast addr: 224.0.23.12 port: 3671 ip: 192.168.178.72
result 1
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
sendBytesUniCast
E (13729) esp_netif_lwip: esp_netif_new: Failed to configure netif with config=0x3ffb20f0 (config or if_key is NULL or duplicate key)
E (13731) esp.emac: esp_eth_mac_new_esp32(602): alloc emac interrupt failed

EMS-ESP 3.7.0-dev.45: press CTRL-D to activate this serial console
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
sendBytesUniCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
sendBytesUniCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
sendBytesUniCast  
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast
sendBytesUniCast  
readBytesMultiCast
readBytesMultiCast
readBytesMultiCast

as soon as I do one reset, no more multicast packets.

can this be linked to the settings, that are only there after they were written in the first run?

[Ing-Dom]

:D I finally tracked it down.
the knx->start() was done to early in the EMSESP::start().
This was done before network was started, so setting up multicast failed. strange thing is, this seems wo work for normal unicast traffic, but not for unicast.

Now I can concentrate on the real knx stuff.