Release Schedule

[proddy]

@MichaelDvP when shall we release v3.4 to stable?

[MichaelDvP]

I think you can release v3.4.
A lot of people using the dev and there are very few complains. The enhancements for #317 and #344 can go to v3.4.1.

[proddy]

@MichaelDvP release 3.4 today or wait for #396 and/or your additional reduce mode entities?

[MichaelDvP]

Release now and we put the enhancements to new dev. This gives room for renaming and testing.
Only thing is #396, is mDNS working for you? I have no issues.
Maybe MDNS.end() do something strange if MDNS was not started? Suggestion to change to:

void WebStatusService::mDNS_start() const {
#ifndef EMSESP_STANDALONE
    static bool started = false;
    if (started) {
        MDNS.end();
        started = false;
    }
    if (!MDNS.begin(EMSESP::system_.hostname().c_str())) {
        EMSESP::logger().warning(F("Failed to start mDNS responder service"));
        return;
    }
    started               = true;

Then first start is like b7 and only on reconnects MDNS.end() is called.

[proddy]

@MichaelDvP shall we get v3.4 out after #395 is resolved?

[proddy]

Yes, i've seen it before, but forgot to feedback. It's nice to have, but excel have some problems depending on country-format and numbers are very long: actual boiler temperature,34.70000076293945,C Using Intl.NumberFormat conflicts with comma-separator. Maybe better to use semicolon instead of comma (i know csv: comma-separated-values, but semicolon is mostly excepted.)

Other small findiings: the id==0 for api attributes, maybe there is a better way to also allow api/thermostat/hc2/seltemp/value. And api/thermostat/hc2 is not working, i need to use api/thermostat/hc2/values.

How about using id value of -2 ?

[MichaelDvP]

could you do a fix for that?

I'll add a commit to the PR.

How about using id value of -2 ?

I'm testing not using the id, only check the atrribute.

    char * attribute_s = nullptr;
    // check specific attribute to fetch instead of the complete record
    char * breakp = strchr(command_s, '/');
    if (breakp) {
        *breakp = '\0';
         attribute_s = breakp + 1;
    }
...
            // if we're filtering on an attribute, go find it
            if (attribute_s) {

The parse_command_string function need some more changes, i'll make an extra PR.

[proddy]

@MichaelDvP so looks like all bugs are fixed, shall we release 3.4 today?

[MichaelDvP]

Yes. it's time to have a 3.4 release. Too many people using the old release. Write to the release note that (this is the last time) users need to reload page after update.
In 3.4.1b we should go to platform 4.3, the uart works much better than in 4.2, seems to be stable now.

[proddy]

its out!

[proddy]

@MichaelDvP shall we release 3.4.2 to main ?

[MichaelDvP]

Yes, i think it's stable and changes now should go to v3.5, Also it's easier to find translators/testers if there is a automatic build.

[proddy]

3.4.2 released to main and dev is now 3.5.0

[proddy]

@MichaelDvP I've been running my flash_optimization branch for a while now and it's stable, so planning to merge into dev. It'll be a big change. Any objections?

[MichaelDvP]

No, i've tested it, works good. Merge it.

[proddy]

it's merged. I need to adjust the rename_fw script as now the binary is called EMS-ESP-CI-3_5_0b5.bin which is misleading

[proddy]

Hi @MichaelDvP , shall we aim to get 3.5.0 out and then focus on the next set of enhancements like the scheduler?

Still to add from your fork (if you want) are:

#673 - more BusID's
#691 - Add a specific bin file for the E32 Gateway
#686 - Add min / max values on top of customization screen

and decide if you want to include:

#498 - Enhancement, provide a command to continuously read and compare typeID(s) values

[MichaelDvP]

I think norwegian translation is on the way, We should wait for it. Also for polish there were changes in web translations, We should check de (me), nl (@bbqkees), se (@mvjt) for missing translations in the index.ts.

more BusID's

Naming for 0x49.. Gateway1, 2, 3,..?

Add a specific bin file for the E32 Gateway

And automatic bins for C3, S2? I'm not sure how to handle the C3, Lolin-C3-Mini-v1 have simple LED, but Wifi-TX-power issue, Lolin-C3-Mini-v2 no issue, but the RGB-LED, For other C3 boards there is no really compatible setting. Maybe add a setting for RGB-LED in future? Lolin uses this LED also on S3 board.
I'll skip the name check for C3, S2 in upload, this was first approach, but checking the chip-ID in esp-bin header is better.
Add also S32 as extra bin as bbqkees mentioned?
Is the md5 check ok for you, or do you have better ideas?

Add min / max values on top of customization screen

What do you prefer, bracket or column? I did not get feedback from you and from @tp1de.

Enhancement, provide a command to continuously read and compare typeID(s) values

I've tried this a time ago. Working, but watch command gives more and better information, i think it's useless. This is the output of monitor command:

[proddy]

I think norwegian translation is on the way, We should wait for it.

Agree. I have asked in #675

Also for polish there were changes in web translations, We should check de (me), nl (@bbqkees), se (@mvjt) for missing translations in the index.ts.

I chase this up and check the Dutch myself.

more BusID's. Naming for 0x49.. Gateway1, 2, 3,..?

Perfect

Add a specific bin file for the E32 Gateway

I don't like the different bin files to be honest. If we can do it within the code, like you did the defines its easier and people building for C3 and S2 are power users anyway. I've updated #691 to see if we can determine if the board has an ethernet module attached.

Is the md5 check ok for you, or do you have better ideas?

MD5 is good to go. You did a nice job there.

Add min / max values on top of customization screen

Yes columns are better, you were right.

Enhancement, provide a command to continuously read and compare typeID(s) values

Then drop the monitor command. you can add it if you want, as it's harmless.

[proddy]

@MichaelDvP shall we get v3.5 out soon? Kees is already pre-loading the dev version onto new boards. A quick review

• I think all the translations are in there now, for NL and SE (index.ts) and the latest NO.
• We can leave Upgrade to ESP32 - Wemos S2/C3 mini and S3 #605 in (S2/C2 support) while we continue to optimize for the S2 RAM issues. I doubt anyone will use it right away.
• @bbqkees do you have a note on your wiki for people that upgrade from v3.4 to v3.5?

[tp1de]

I think the only thing that needs additional good explanation is how the localization affects the HA Discovery.

... and I would recommend to mention that the 3.5 flashtool needs to be used and this creates on Windows Defender error messages ...

[tp1de]

btw flashing with web tool from 3.4 to 3.5 doesn't work .... or is this solved?

[MichaelDvP]

What is "web tool"?

[tp1de]

@MichaelDvP web-ui sorry ... your flasher is needed to migrate from 3.4 ... or is this not needed anymore?

[MichaelDvP]

Web-UI update works in all v3.x.x versions if the esp32 was once partitioned with emsesp partitions. Because sometimes the tasmota-flash-tool was used for usb-flashing the partitons were to small on these emsesp for v3.5 and will give a http:500 error on upload.
Only if update via web/ota is not working usb-flashing with the emsesp-tool is recommended: https://github.com/emsesp/EMS-ESP-Flasher/releases. After flashing once with right partitions the web update is working for all versions. But i think this is discussed a hundred times now.
BTW: only esp32-C3 version has only 10KB free flash left. Flashing a loader in a factory partition scheme is usefull. See: #605, #608, and https://github.com/MichaelDvP/EMS-ESP-Loader/releases

[tp1de]

@MichaelDvP tag for hc is corrected and entities are back in ha now but it is still ww instead of dhw if I change to "German"

[MichaelDvP]

For boiler_data_ww the tag is in all languages ww, But what is different for en/de? Again: Please give an example!

[tp1de]

@MichaelDvP
OK understood it is now ww for all languages ... and it works. But it was dhw before. Is it wise to change it, so that everybody needs to change the dashboards? ... and it is dhw1 within Buderus / Bosch API as well. Keep in mind that there are systems with more than one WW-Boiler (dhw1, dhw2 etc.) The RC300/310 can control 2 WW-Boilers ..

I recognized that any changes in HA in respect to names to the entities are overwritten by the names from ems-esp. It might be an idea to introduce a settings option "empty" or "no" to the language settings for entity names to avoid this. What do you think?

[MichaelDvP]

Your'e right, it was ' dhw' and should keept compatible, i'll make a change.

I'm not using HA, but i think the idea of autoconfiguration is to autmatically set all entities, if you want to set manual, disable autoconfiguration.

[proddy]

I don't think you can change the Entity IDs in Home Assistant. You can create your own new ones of course. Also agreed dhw is better than ww

[tp1de]

don't think you can change the Entity IDs in Home Assistant

Not the id (unique_id) but the name can be edited in ha ... which I had done and it was a bit of surprise that they were overwritten.
I adapted my Javascript program to read km200 states from ioBroker to HA's Node-Red. I create the states with MQTT autodiscovery similar to ems-esp. There I can edit the names without being overwritten. I do not understand why ems-esp names are different.

But customizing / changing names in ems-esp works and this changes are used in HA as well. It might be worthwhile to explain this in the documentation.

[MichaelDvP]

@MichaelDvP shall we get v3.5 out soon? Kees is already pre-loading the dev version onto new boards. A quick review

@proddy I think wait for #747, #748 to complete, max. end of this week.
For #500 the feedback is not constructive, it will take long with much itterations to complete. Don't wait for this, move milestone to v3.5.1
Also C3/S2 milestone #605 can moved to next version. S2 does not give any advantage, C3 is nice because low price, low power and (with reduced tx power) working bus-powered. Maybe some people will use it. But they can use the dev. Actually i don't have any issue with C3.

[proddy]

@MichaelDvP all of v3.5.0 tagged issues are resolved (https://github.com/emsesp/EMS-ESP32/milestone/6). Shall we prepare a Christmas release soon? Two small things to consider

• the translations for the new HA Entity ID setting will take time (6 languages to do!) so leave it for now until we have more changes
• we did say we'd re-order the languages, sorting them and putting a divider after EN? Shall I do that or you?

And do you have minor updates you'd like to add?

[MichaelDvP]

I have the new entities for #802, #803 in my dev. I now added the sort languages to this.
As minor updates there is a missing uom (l) and reduce the log buffer at <41k free heap block, as we've seen that with 38k there were blank pages. Also added the german translations for new i18n and entities.

There are also translations missing for some entities, if we want to wait to complete all translations, we'll never release. So make the christmas release. I'll do a PR for my changes, check and decide if you want it in the release or in next beta.

[bbqkees]

Was there going to be an additional bin file for the E32 moving forward?
We discussed it a while ago but not sure if it was going to be implemented now.
Lot's of people will need to flash via USB, would be a good idea to have the E32 bin ready so it works out of the box.
I am making a more detailed guide for upgrading to 3.5.0 as well.

[MichaelDvP]

Now on empty configuration (new or factory reset) the v3.5 software tries to connect the LAN-Chip and sets E32 on success, S32 otherwise.

[bbqkees]

A yes sorry. I forgot it was this solution.

[proddy]

@MichaelDvP

platform-espressif32 5.3.0 was just released (https://github.com/platformio/platform-espressif32/releases/tag/v5.3.0)
and compared to 5.2.0 both RAM and Flash have gone up quite considerably. Shall we hardcode EMS-ESP to use 5.2.0 ?

[MichaelDvP]

Ah, this explains the 5k free flash i've seen in the latest AM200 tests, the GH uses the new platform.

Shall we hardcode EMS-ESP to use 5.2.0 ?

Maybe for v3.5, but it updates to arduino core 2.0.6 and IDF 4.4.3. For v3.6 we should update and find a solution for ram fragmentation. Maybe alloc a large buffer on start and use this for web/api outputs. But i don't know how to advice arduinoresponse to use a already alloced buffer. Free buffer, alloc arduinoresponse, and realloc buffer after send could be critical if other tasks allocs ram meanwhile.

[proddy]

yes, this is worth exploring. I think the latest 5.3.0 may also compile now with c++17 instead of c++11 by default which will also add some overhead. One thing I wanted to look at is the new ArduinoJson's shallow copy which could save on some memory when we're building up the JSON

[MichaelDvP]

What is your plan for the v3.5 release?
I have now 3 branches with additions.

• dev adds a lot of entities (AM200). Maybe critical because it uses more heap
• tr adds web TR-translations to dev, only 8k left
• blacklist adds the posibilty to remove entities from memory, could be helpfull for heatpump users with some memory related crashes.

These are all mainly feature-branches, consuming flash and heap, maybe better for next dev.
But there are also some small fixes included: renaming, limits, rounding of values, etc.
Should i make a branch with only the fixes for v3.5 to merge now?

[proddy]

Good question. I've been working on my tech branch and also made lots of small changes and fixed bugs and lost track of what I've updated. I want to get into dev soon to avoid getting too far from the mainline.

I agree with your approach, so

• you do a PR only with the critical fixes (renaming, limits, rounding etc).
• We include the PR for the PL translation updates in this
• We clean up GitHub marking the milestone to 3.5.0 for only the things that are going to be included, the rest can be tagged with 3.6.0
• We release 3.5.0
• dev becomes 3.6.0
• I'll merge my changes from the tech branch as this touches a lot in the core (emsesp.cpp, system.cpp, react routing, uuid, GH actions...)
• Then you add your features (heatpump) to this 3.6.0. We can say the theme for 3.6.0 is Heatpump support (3.5.0 was all about languages and customization).
• It's going to be a merge hell for us with a lot of manual work I'm afraid!
• I think we need to park Turkish until we've solved the memory issues

[tp1de]

I already recognized stability problems with v3.5.0-dev.15. Reboots every couple of hours. Might be already a memory problem.
I believe that stability should be fixed before releasing 3.5.
.... or am I the only one with stability problems?

EMS-ESP Version
v3.5.0-dev.15

Device (Platform / SDK)
ESP32 / v4.4.2

System Uptime
000+11:46:30.691

CPU Frequency
240 MHz

Heap (Free / Max Alloc)
128 KB / 43 KB

Flash Chip (Size / Speed)
4.096 KB / 40 MHz

Application (Used / Free)
1.962 KB / 22 KB

File System (Used / Free)
28 KB / 36 KB

[MichaelDvP]

Please note the download-date of the dev_14, it's 73 commits from first to last dev-14 (25.12.22-03.01.2023), within the change to arduinojson 6.2 with a lot of changes.

[tp1de]

Please note the download-date of the dev_14

03.01.2023 10:58

[proddy]

@MichaelDvP shall we get 3.5.0 out after (Blacklist / Remove Entities #891)? There are no more issues tagged with 3.5.0. I'm thinking this weekend....

[MichaelDvP]

I'm thinking this weekend....

Ok.

[proddy]

it's done!

[tp1de]

My findings for WPA2/WPA3 transition mode are in #1073 . I do not feel that it is working as it should be. I will go back to WPA2

[tp1de]

@MichaelDvP @bbqkees my Lilygo T7 S3 arrived today. I am bit lost for the moment and I may need your help how to flash:

• Windows does not recognize any COM Port (like for ESP32). What kind of Windows driver has to be installed?
• When driver is working, how do I have to compile? Special platformio.ini setting needed?
• How do I flash - just using your flasher Michael? Or do I need as special partition file?

Thanks for helping. Thomas

[MichaelDvP]

I've flashed it direct from vsc, platformio upload. You need to activate the bootloader: press boot and reset buttons, release reset, then release boot.
Driver is CH340, mostly it is already installed.

[bbqkees]

The LilyGO makes use of the ESP32-S3 native USB port. There is no CH340 in between anymore. I have not flashed the LilyGo but I did flash another S3 with native USB port with PIO and that went fine without pressing any buttons.

I also received the LilyGO's already. If someone likes to try it out let me know.

[bbqkees]

You may have to push some buttons to get the LilyGO into the right mode so Windows sees the COM port.
It also has a power switch, maybe yours is set to off.

[tp1de]

The LilyGO makes use of the ESP32-S3 native USB port. There is no CH340 in between anymore. I have not flashed the LilyGo but I did flash another S3 with native USB port with PIO and that went fine without pressing any buttons.

Kees how have you done it? My upload searches for a com port. What kind of settings do I need within PIO for upload? And can I use the standard 16 MB bin-file?

[bbqkees]

You need to set the env in the ini file to env:esp32s3. And then build and upload.
It's actually for the Lolin S3 but the chip on the LilyGO is the same one I believe.

[bbqkees]

Here it is:https://docs.espressif.com/projects/esp-idf/en/latest/esp32s3/api-guides/usb-otg-console.html#initial-upload-using-the-rom-download-mode-over-usb-cdc
You need to keep boot pressed while doing a reset with the rst button, then release after releasing the rst button.
Then the COM port should appear.

[tp1de]

Kees that's what I have done now a couple of times ... still no COM port.

I tried a different USB-C cable and now the COM-Port is there even without booting into download mode !

[bbqkees]

Yes I did have a look at it but I have not actually tested yet.
You need to break off 2 pins of each 10 pin header that is included with the T7 and solder it to the inner position.

[bbqkees]

I'll update the wiki soon. It has the same pinout for GND, VCC etc as the MH-ET live ESP32 board.
Only the GPIO pins may be different, so you maybe need to remap them in the settings.
https://bbqkees-electronics.nl/wiki/gateway/upgrade-the-gateway-to-an-esp32.html

[tp1de]

Thanks Kees.

[MichaelDvP]

Mapping is of pins is compatible for board profile "LiiGo S3", also mapping of valid pins (for analogs) is set for this board.

[tp1de]

Mapping is of pins is compatible for board profile "LiiGo S3"

This board pofile was choosen after first flash.

[tp1de]

First findings on Lilygo T7 S3: Memory fine but WiFi very slow even just 30 cm from my router. Maybe WiFi needs some special adjustments. WebUI is extremly slow.

[tp1de]

@MichaelDvP the s3 version does not work stable. Dashboard reacting very slow and devices are found only after a long time.
Could you explain how to flash new with your flasher? Maybe partioning is wrong by just uploading from VSC

[MichaelDvP]

Please reopen #605 and post logs there, here it is off topic. Partitioning s ok, i think you get much garbage on the uart, is the gateway connected to the bus?

[tp1de]

I can't reopen but I will post my findings in #605

[proddy]

@MichaelDvP - shall we merge dev2 into dev, leave it for a week or two and then finally release 3.6.0?

[proddy]

Ok, give me the signal when your latest dev2 is merged into the main repo, and I'll merge into dev. thanks

[MichaelDvP]

It's only the few lines of tweaking the response and the partial tr translation, but i'll make a PR to dev2. We can add the rest of tr transation also to dev, no need to wait for it.

[proddy]

@MichaelDvP shall we push v3.6.0 out this weekend? Can you think of any last minute additions or open items?

[MichaelDvP]

I don't have any last minute additions.
Open items:

• last ~ 30 translations for tr lanuage Turkish language support #1076, ask @dreixin if we should wait, or add it later to next dev.
• device names Feature: Allow device name to be customized #1174 (but this is some more work, maybe better in next dev)

I think you can push 3.6 release.

PS: i'm out of home next 2 weeks

[proddy]

3.6.0 released to stable. Hope nothing too bad happens in the next 2 weeks :)

[tp1de]

It would be nice to integrate Michaels solution for api send/request from #1212. Is this planned?

[bbqkees]

I see there are some Dutch translations missing.
Can I use the Excel file from #1076 to add them?

[MichaelDvP]

Here is the alligned file:
locale_translations.xlsx

[bbqkees]

I'll finish it today.

[bbqkees]

I translated all open Dutch items.
See the attachment.

[MichaelDvP]

See the attachment.

Can not see it.

[bbqkees]

locale_translations-KR-10082023.xlsx

[proddy]

@MichaelDvP I'll add the small MQTT enum fix #1280 and then shall we merge dev into main and get 3.6.1 out?

[MichaelDvP]

I'll also add color to networkselector and fix a typo, than you can go.

[tp1de]

Will the energy entities be included in 3.6.1?

[MichaelDvP]

Will the energy entities be included in 3.6.1?

No, but i'll make a PR to official 3.6.2-dev for more tests.

[proddy]

@MichaelDvP 3.6.2 is looking solid (#1264). Shall I release to main?

[MichaelDvP]

Yes, release it.

[proddy]

done. 3.6.2 is out. I've tagged the dev to 3.6.3 and we can experiment with Jason's platform builds.

[proddy]

3.6.5 is out. 3.7.0 is the next dev version, with a refreshed web UI

[proddy]

it's been 4 months, shall we release 3.7.0 ? @MichaelDvP

[MichaelDvP]

There were a lot of changes. Guve the users some time to test in dev.
Also it's summer, most heatings are off and users are in vaccations. We will get only little feedback.
v3.7.0 is a major update with breaking changes. We should make all breaking changes in this update.
the system/info still contains a lot of names with blanks, In #1860 you want to change to underscore (or camelCase).

I'm still working on #1886 in combination of #1862.

[proddy]

okay

[MichaelDvP]

Still have issues with command names of scheduler and custom entities,
api/scheduler/commands, show commands, etc shows scrambled names and commands not working.
Seems the string.c_str() in Command::add(... is temporary and deleted later.
So wait with 3.7.0 release until this is solved.

[MichaelDvP]

Same result ;-(

[proddy]

I think the back() goes out of scope before it call the .c_str(). Maybe make a reference first like auto ref_ce = webCustomEntity.customEntityItems.back() and then (ref_ce.name).c_str().

[MichaelDvP]

My fault, it's working in the dev.
I've tested a few days ago memory usage and changes std::list to std::vector. But vector is not fixed in memory and can be reallocated, so .c_str() points to freed memory. The change was only in my local build. (std::list and std::deque working).

[proddy]

good to know!

[MichaelDvP]

I remember the same issue with analog sensors, using analogsensor.cpp: sensors_.name does not work, but using sensors.name from settingsService work. Now i have learned that this also due to vector vs. list.

[tp1de]

Just for me to clarify: Will all changes / functions in actual dev be a part of the new stable? (just switchprograms not)

[proddy]

yes, basically we just merge 3.7.0 dev into main and change the tag and release version.