Server stops responding #11
Comments
|
@jetpax One solution is using break-outboard that has extra capacitors on the supply rail. I used the one below and added the capacitor leads in the left top in the screw connection. ( GND and VIN are next to each-other).
The log you have was similar to what I had without the capacitor. But I hope we can still look into the code and figure out why the server stops responding. Although the capacitor improved the stability of the WiFi in my case, I expect this also happens when the WiFi is really bad. In my view bad WiFi should not result in a deadlock of the server. Without the capacitor I was able to reproduce this with multiple devices connection over WiFi to the ESP32. |
|
Would it be an idea add a setting to control the WiFi TX power in the WiFi setup. If I am not mistaken there is also a powersaving option for WiFi ( Default, Off, minimum and Maximum ) |
|
@jetpax please try to stabilize the supply voltage first. ESP's are known to be very power hungry. I had similar issues with one board as well. You need to be able to supply 500mA peaks. You can hook-up a scope on your power rail and see whether it drops every few ms. The crash you observed is coming from issues with reading the temperature sensor. I haven't seen this in my tests. But I'll observe whether I can observe traces of this like @EvEggelen I don't know if this is the right approach to fix poor hardware design. |
|
@theelims Regarding adding a setting for changing the TX power. This is in that sense not related and not intended to solve this problem. To be honest I should have added this in a different ticket as feature request. My main motivation is to reduce radiation and power. When possible I reduce the WiFI power to the level that is required to function correctly ( with some margin ). I do this for my access point and so too. I see more IOT devices have this feature. Regarding the dead-lock of the server. I expect this can also happen with a stable power supply when the WiFi signal is bad. Even with my stable power supply see errors that I do not understand. Luckily they do not result in a dead-lock of the server. When I have something concrete I will make a ticket of this issue. |
|
There are a number of error messages coming from the AsnycTCP and AsnycWebServer libraries. Unfortunately they are largely unmaintained and rather buggy. I included the most stable versions and included all pull request I could gather from various forks that seemed usable. Plus I added some of my own. However, their queuing mechanism has a memory leak resulting in a crash if too much of these show up |
|
@theelims But these kind of issues related to message rate are in my view not really nice. Not sure what the exact root cause is, but based on what you indicate, I would not be amazed that if the Wifi connection is really crappy, this happens at lower rates too. As I wanted to use ESP32-sveltekit as foundation for my new project I was hoping for a stable and maintained core. Do you know if there are stable and maintained alternatives for AsnycTCP and AsnycWebServer ? |
|
I do follow the what I think best maintained forks https://github.com/OttoWinter/AsyncTCP and https://github.com/esphome/ESPAsyncWebServer Since Many big ESP open source projects use these libraries I'm not overly concerned. As long as you keep out of the danger zone of fast websocket and SSE the code should be pretty reliable. |
|
After testing I notice that the current SW is not stable. This is even with "low" amount of messages. I noticed that the WiFi connection is bad, seen As a test I compiled the original ESP8266 React project for the nodemcu-32s board and flashed it on the same HW ( functionality is simular ). So the only thing what is different is the SW on the device ( WiFi signal, power supply, board and physical location are all the same). In this case the SW is rock stable. I had 4 devices connected, went crazy with browser refreshes and let it run for hours ( with the LightState toggle in main) . I could not find any problems with that SW. I find it hard to reproduce the issues deterministic with ESP32-sveltekit. But refreshing the page, connecting with both AP and Wifi normally triggers issues ( so with 2 devices). I sometimes also notice that the number of connected devices keeps increasing while the number of connected devices is the same. Is there also code to disconnect devices ( release memory) ? Could the bad Wifi issue a scheduling problem that conflicts with WiFi reception framework ( and that this triggers other issues) ? I also noticed power consumption is very different than from ESP8266 React ( stable 130mA) while with ESP32-sveltekit I have it seen dropping to ~20mA. I would love to solve this issue. What is best to take as a next step. I can create some more logs. Would that help ? Added code for the automatic LightState toggle. Logs: Log 2 Log 3 |
|
@EvEggelen No need to do more tests. I think I know where the issue is. I make use of AsyncWebServers Server Send Messages (esp8266-react doesn't). This seems not so heavily used and still quite buggy. It will happen with Websockets as well. It's the queuing mechanism which does not release memory properly. At least once you start to stream analog sensor readings through it. I reviewed the code section many times, but couldn't find an obvious reason why it crashes. I'm tracing this bug since a while, but couldn't get a hold of it yet. |
theelims
added
bug
|
Currently I am testing the patch : esphome/ESPAsyncWebServer#24. In the comments I found the following remark
So I do not expect miracles. But so far it seems indeed an improvement. So far I did not see the error message
still pops-up :-(. Strangely enough, it does not take down the application. |
|
I came across a discussion about ESPAsyncWebserver that I found interesting, with code that supposedly fixes its shortcomings. It was written by Phil Bowles who alas has since passed away, but seems solid. Maybe worth looking at... |
|
@jetpax Thanks for posting this information. I watched all Phil videos and get the impression he did quit some work on this. Actually, he reports the exact same issues we are seeing ( instability ). If his info is valid, these ASync libs currently used in ESP32-sveltekit are so riddled with issues that I prefer not to use them in framework that has the goal to be used as a stable foundation. Also I got the impression he put a lot of effort in rewriting the whole stack and making a new stable foundation as a "bug" fix here and there will not do the job in the Async stack of libraries. Unfortunately, if he passed away ( where did you see this ), I am wondering if it is realistic to expect that his work will be maintained / matured. Did anyone ever consider using the : https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/protocols/esp_http_server.html. I did a quick test and it seems to work. I expect this version is stable and will get the maintenance when needed. So potentially this is more future proof ? |
|
Yes @theelims, I agree with you re maturity/long term support of H4, (even though Phil's work seems to have been adopted (philbowles/h4plugins@master...HamzaHajeir:h4plugins:master)) and think the espressif stack is the way to go, esp. as it has been around long enough to mature (IMO espressif tend to release early, release often) |
|
@jetpax @EvEggelen I had a look at that H4 and it basically does the queuing length fix. A similar fix is in my esphome fork included as well. But instead of limiting the queue length to a fixed number H4 takes the free heap as a measure. I'll have a more detailed look at the code differences once I'm actively working on it. It won't be a drop-in replacement as H4 depreciated websockets and json support, which are essential. That pull request from esphome you linked got me the idea to replace the whole queuing mechanism with the FreeRTOS xqueue to make it thread safe. Instead of the proposed mutex. I'll try that in the next weeks. Why Arduino instead of ESP-IDF? Why ESPAsyncWebserver instead of native ESP-IDF http server? The ESP-IDF libs are quite buggy as well. I failed to implement a Websocket Client Endoint (which I thought was straight forward) because the ESP lib kept crashing whenever I was doing something else with the payload then ESP_LOG. I postponed this to the release of the next major Arduino and ESP-IDF version which saw significant changes to that lib. Also the maintainer of ESPAsyncWebserver is an employee of espressif now. He publicly announced that he can officially work on ESPAsnycWebserver after the release of the next major version of the Arduino framework. He stated, that he'll base his new lib on ESP-IDF. Maybe time will solve this for everyone. Many big ESP32 open source projects work with ESPAsyncWebserver as stable and productive code. Support for MQTT 5.0 |
|
@theelims The good thing is that this set is really stable. Meaning I tried all kind of things with it, but did not see crashes or other problems. My perception is that the loading speed is also increased. Doing a full refresh over my wifi with really bad reception on the ESP32 took 800ms. Looking in more detail I could see that these svelte pages try to open a lot of parallel connections ( 20+) to the server. So far it seems that web browsers do not close these connections quickly. So this gives issues when trying to connect from multiple hosts simultaneously. This seems logical as LWIP is configured by default on the ESP-IDF for max 10 open sockets. When enabling features from the IDF HTTP server to manage these open connection, serving this svelte content to multiple clients is also stable. So far I could not find the SSE feature in the ESP HTTP server. I looked a bit into this, but it seems that the features to facilitate implementing this ( keep connection open) are in v5.3-dev only ( right now we have v5.1, so first v5.2 needs to go out). But websockets are there already ( going to test right now). To limited to amount of open sockets ( strain on memory) I was thinking to use one websocket only for both SSE and websockets. Basically have one single connection from the back-end and front-end. I expect that right now we have at least LWIP 2 sockets open ( one for SSE and one for the sockets) per client. Long story short. Based on what I am seeing the ESP HTTP server seems to provide stable and performant functionality. It works both on the ESP-IDF and Arduino framework ( tested briefly though ). But as I have very little experience with svelte ( planning to learn) I was hoping I could get some help to adapt the front-end to a single websocket solution. Based on the learning we see if it is possible to move the whole solution to ESP32-sveltekit. Right now I prioritizing stability over feature-set. Would you be able to jump-start me a bit on the svelte front end architecture of ESP32-sveltekit ? |
|
I think in a first step you can completely ignore the SSE. It's just adding WiFi RSSI reading on top. You might get some trash on the debug terminal, but it's not impacting the core functions. Changing that over from SSE to WS is just a few lines of code on the frontend. However, the whole back-end heavily relies on the ESPAsyncWebserver API for the REST calls. It's not just about serving the static content. I expect that you run into the real troubles there. Let me know how things progress. |
|
@EvEggelen I just stumbled across https://github.com/hoeken/PsychicHTTP which is a brand new ESPAsyncWebserver compatible library based on the ESP-IDF HTTPserver. You migh give it a try instead of rolling your own. It does not come with SSE either, but as you suggested that is easily switched over to websocket instead. |
|
@theelims Today I finished my tests. I have written new WebSocket typescript module for the frontend to enable the notification ( simulated SSE) and websocket communication over a single WebSocket. The interfaces are easy to use in new modules. I also fixed most of the warning in the front end too. The whole system seems to work well. The backend is implemented from scratch on the IDF framework with the IDF HTTPServer. Not all rest interfaces are implemented yet, but the Demo page is working ( both rest interfaces and Websockets). Also I implemented the System Status page with all the Toasts and RSSI. The MQTT and NTP and Wifi control are so far not added to the IDF backend. But the stability of stability of the whole system is good ( during my limited tests). Now I need to see if it possible to move the HTTPServer code into a class that is "compatible" with the Arduino ESP32 sveltkit code. But I guess the PsychicHttp is something close to what I have made/ plan to make. So my front-end ( to share the single socket with websockets and SSE) is still useful. I expect that my updated WWWData scripts will also help to reduce the strain stack and heap ( the old method gave an stack overflow on the IDF framework with default stack configuration). It will also only trigger a rebuild of the WWWData when the svelte source files are changed. This reduces the build time significantly on my machine during back-end development. I will look into the PsychicHttp and let you know what I find. Thanks for the tip on PsychicHttp. |
@EvEggelen Are you willing to make a pull request for that updated build script? |
|
@theelims . I am willing to try. But this means that I first need to back-port it for the current code on github. I did add the #if for Arduino and IDF, but better to test this first before committing. I also need to update a few lines in the code to make sure the new prototype of the WWWData.h is used ( to make sure all the filenames are not first wrapped in a String and then used. ) What update/change is particular interesting to you ? To make the project stable we need in my view minimally replace the HTTP server... |
|
@EvEggelen in particular I'm interested in the script being only triggered when there was a change in the frontend files. For the remaining work I'm torn between just replacing the queuing mechanisms in ESPAsyncWebserver with xqueues or a full blown re-implementation with PsychicHTTP. I think PsychicHTTP is very promising, just not yet where it needs to be. And my C++ foo is not good enough to bridge that gap. |
|
@theelims Let me try to test and commit it this weekend. Currently I am looking at at the possibility using PsychicHTTP in the project and add the new modules I created. I see 3 options. 1) create a complete new back-end on IDF 2) put my HTTP server in the backend + modules 3) use PsychicHTTP + modules. Let me evaluate the options...... Typically the things I program are very timing critical and HW accelerated. So maybe the IDF framwork is better for me. But lets see. |
|
@EvEggelen if you're capable to port the backend to ESP-IDF in any form I am very happy. Loosing SSE should be easily compensated by keeping a websocket live. With ESP-IDF I would appreciate the possibility to use TLS/SSL. Because the safety features are a laugh without SSL. I'll happily do the front end part of this. |
|
@theelims Never published code publicly on github. I assume you need to give me access rights to create a branch on this project ? At least I got an error when pushing code. |
|
@EvEggelen The right way would be that you fork this repository, create the branch there and then make a pull request to push your changes into here. |
|
@EvEggelen how are you progressing with your PsychicHTTP port? My pain threshold has been reached by ESPAsyncWebserver. Is there anything how I can support you? |
|
@theelims. I feel your pain.... In the beginning I could not imagine that these kind of problems are caused by SW that is used so broadly. Anyway, regarding the porting. Some features were missing from PsychicHTTP ( put in a request for those features) . These are now implemented. Others are also moving from ESPAsyncWebserver toward PsychicHTTP. During this process some issues were found, and to my knowledge fixed. As PsychicHTTP is not a drop in replacement, you need to do some work on the code to get it to work. I was working on it, but I stopped ( as I needed to wait for these new features). As more people are now working on porting from ESPAsyncWebserver toward PsychicHTTP it is now easier. Currently I am busy, but could re-start the work later. But honestly I as still doubting if it makes sense to move completely to IDF framework. I already gained some experience with it, and must say, the more I use it, the more it makes sense to use it. |
|
Yes, it is gaining momentum. This was so overdue. I found https://github.com/emsesp/EMS-ESP32 which has a branch that already did most of the hard work. I'll get inspired by @proddy and his progress. He did quite some drastic changes to esp8266-react though. But his HttpEndpoint file is so much more understandable then the original one. I'm confident, that it is possible to switch ESP32-SvelteKit to PsychicHTTP without much hassle. But I'll keep the multi-connection architecture for now. I've started a branch for this and will continue to work on it over the next days. What benefits do you see in going all in on ESP-IDF webserver? I'll convert the MQTT client to ESP-IDF for sure. That's on my todo list anyway. |
|
I paused my IDF ports. With my optimized versions of AsyncWebServer and AsynTCP (ESP32 only) and the switch to espMqttClient I have very low heap mem consumption and performance is 2x of what I was seeing with PsychicHttp. What you could try is using my libraries as a drop in replacement and see if that helps with your stability. |
|
@proddy I'm currently trying to get this running, but you've messed way too hard with ArduinoJSON for this to be a drop in replacement. Also, do you make use of the Event Server and Websockets? High traffic on those two is my source of troubles. The server is quite stable. I use a slightly modified version of ESPAsyncWebserver and AsnycTCP from esphome. |
|
@theelims the main branch still uses ArduinoJson 6 on the Async* libs if that is your struggle. I did not "mess" anything up, I just spent 3 years further building on Rick's excellent framework, which is outdated and has a few bugs here and there, and enhanced it with more features - added scheduler, logging, telnet, ethernet etc. And optimized the web moving from react to preact, to vite/yarn and cutting space down by 50%. You can pick up some ideas from my code if you want. Async* and esp8266-react are based on ArduinoJson 5 so that's the first thing I would fix, like removing all the reference pointers to JsonObject etc I use EventSource, but not WebSockets. The code is there in the good luck! |
|
@proddy I didn't meant to offend you with the messing around statement. I just had 20 Minutes and thought I would just swap libraries, hit compile and can see whether it'll crash in my project. Which wasn't the case because of dozens of compiler errors. I was under the impression my system uses ArduinoJSON 6. But I'll double check. Thanks for that hint. I really enjoyed Ricks work. It's an amazing foundation. And I also like your work on it. I'll certainly draw a lot of inspiration from that as well. |
This is really my bad. When porting my buildscript towards your project I forgot to include my latest fix... Sorry for that. Good that you noticed it.
So far the stability is very good with my test code. I expect to also use the resources more efficient when using that API directly. But this basically this is a complete rewrite of the back-end. Currently I am experimenting to move as much into data, bss and text sections and see what happens. As I was also considering HTTPS, the number of concurrent connections need to be reduced on the ESP. Svelte kit splits the front-end up in a lot of small files and loads them in parallel. Do know a way to reduce the number of parallel loads when using Svelte-kit ? Is this something you can configure ? @proddy |
|
@EvEggelen No problem, this happens. I found it after wondering why it wouldn't rebuild wwwdata.h despite having changed a few files. Yes, SvelteKit generates a awful lot of files. There is a long open issue regarding that topic: sveltejs/kit#3882 However, there doesn't seem to be a viable solution out there. This could become a bottle neck. For this simple demo project here I need 110 dedicated endpoints. I'm in the middle of the porting process to PsychicHTTP. We will see how it turns out in terms of memory usage. |
|
@EvEggelen I just pushed a commit to https://github.com/theelims/ESP32-sveltekit/tree/psychichttp which has everything including websocket ported over to PsychicHttp. Only did shallow testing so far, but all features appear to work. CORS pre-flight is not implemented yet. I'll do a stress testing in the coming weeks and see how it copes in my actual projects and whether it is as stable as hoped. @proddy Thank you so much for your work. Your https_36 branch helped me tremendously making this port. |
|
glad it helped @theelims . I'll keep a close eye on your stress testing (and help where I can, just shout) as I didn't instantly see the benefits of switching from AsyncWS and AsyncTCP which is showing 10KB less heap and 2x more http response times. I created a small benchmark app and used https://github.com/mcollina/autocannon to stress test and compare, and autocannon-ui to show the results. I'm hoping when IDF 5.1.2 and Arduino 3 is stable I can switch to multi-core and balance the threads over the 2 ESP32 cores. |
|
@EvEggelen Just wanted to give you a brief update on my progress. So far psychichttp has proved to be more stable and except for CORS preflight and WS authentication I have everything working. On the plus side, the RAM usage is much more favorable. With the ESPAsync libraries I couldn't load BLE libraries at the same time. With PsychicHttp this is possible and really a game changer for me. @proddy regarding MQTT I'm departing from async-mqtt-client as well. But I wanted a more solid support for SSL/TLS, especially the possibility to include the mozilla root CA cert bundle. So I went down the "Psychic" route and took the API of asnyc-mqtt-client and backed it up with the ESP-IDF MQTT client: https://github.com/theelims/PsychicMqttClient Repository is still empty as I need to push the code with examples. It also supports MQTT over WS and does not limit the message size like most other Arduino MQTT clients. And for convenience I added a |
|
Glad you got it all ported over. I'm curious to what kind of response times you'll be seeing using Psychic as I still get it to load faster than my AsyncWS libs yet. The PsyhicMqttClient would be awesome if you can get it chunking and streaming over WS, will watch out for that. Let me know if I can help in anyway. ps. I looked a your code, nice and solid. You use the xTask for the loop instead of the Arduino's loop() - was there a reason for this? |
|
@theelims Good the hear that you are making good progress. I am still tinkering with ESP IDF and see what is possible there. WS authentication is challenging with JWT ( if I am correct, the current code has nothing). You can put a token in the protocol field, but what you use today is too-large for that. I used a token in the protocol field and a cookie ( HttpOnly; path=/ws; SameSite=Strict ) for WS connect. How much heap do you now have free ? Currently I have 200+kb free while running AP and station WiFi. |
|
@proddy Just polishing over the lib with many different examples. I'll give you a note once the longer MQTT message over WS works. Why a separate task and not the Arduino loop()? Separation of concern. In your project everything is very much entangled and organically grown. In my use case I want to have a solid foundation (more like an operating system) for a series of IoT devices. So easily propagating changes of the core from one project to an other was a major design consideration. Also I don't like the main loop. I mostly use it for testing only. For productive code each major module runs it's own task and communication happens between them. @EvEggelen My code uses a search parameter to transport the JWT token. This is one of the last missing puzzle pieces for my PsychicHttp port. But JWT is meaningless without SSL. So this goes hand in hand. Let me check on the heap once my MQTT side hustle is done. |
|
@proddy Have a look at https://github.com/theelims/PsychicMqttClient . I have added docs and a lot of examples. And I tested successfully MQTT over WSS (with SSL) and very large messages. Works like a charm. Next I need to look into publishing them on the platformio registry. |
|
I'll check it out. I also saw IDF finally implemented HTTP_ANY so the HttpEndpoint can be simplified and reduce the number of endpoints (if heap is a problem for your users) |
|
@EvEggelen I got websocket with JWT authentication going. The JWT token is supplied as a search parameter (as with the old code). However, PsychichHttp has some quirks I needed to work around. I filed an issue hoeken/PsychicHttp#73 on this. I can live with it for now, but it would be nicer if this would be consistent with the ESPAsnycWebserver behavior. |
Good to see, however, it will take ages for that to trickle down into the Arduino world. |
true. I use https://github.com/tasmota/platform-espressif32/releases to test out the latest IDFs |
|
@EvEggelen It is done. Thank you very much for your contributions and discussions. |
Thanks for building this great project!
Not sure if this is related to #8, so am opening a new issue.
I did a build with webui in LittleFS, and system works but is quite unstable and server stops responding after a while.
The text was updated successfully, but these errors were encountered: