MicroPython
-
|
Hello, The the device is based on ESP32 devkit c1 v4, Chip is ESP32-D0WDQ6 (revision v1.0) When i was on micropython For comparing memory usage and so, i write this function: Then I call it JUST AFTER connecting to the wifi and do gc.collect() and JUST BEFORE calling the very first request('post') with this block of code: As said... When I run on 1.19.1 it shows... When i run on 1.20.0 it shows... When I run on 1.21.0 it shows... Suprisingly on 1.21.0 we have 128000 of GC total, instead of 111168 in previous versions... but it fails when trying to post with https. Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 6 replies
-
|
See also https://github.com/orgs/micropython/discussions/12791 (@bobveringa) which I think might be related. |
Beta Was this translation helpful? Give feedback.
-
|
I think what's going on here is a variation on #8940 and we hoped would be improved in v1.21 by the changes described in https://github.com/orgs/micropython/discussions/12316 The high-level issue is that on ESP32, the RAM is available to us and TLS (mbedtls) as disjoint regions of contiguous RAM. The old behavior was that MicroPython would allocate the largest such region and leave everything else for the IDF (wifi, ssl, socket buffers, etc). Different IDF versions resulted in a different memory layout as their internal allocations and data structures moved around (this is one major reason for differences in v1.21). The problem we have now is that MicroPython can actually allocate more RAM than it could before (as with the split heap feature, it can grow past the first contiguous region into the next one). Which means that there might be not enough left in the IDF heap for ssl buffers. The only real solution to this is to pre-allocate the SSL buffers, so you can allocate them at startup. This is something we'd like to support in v1.22, and was a big part of the motivation for introducing the SSLContext class. For a workaround... I'm not entirely sure. I think we'd need to get to bottom of why the IDF has no more ram available (i.e. is it because MicroPython has used up all available IDF heap into multiple splits). Are you able to get your program to call The best thing I can suggest is to either explicitly call But really what we need is the pre-allocated SSL buffers. @projectgus Might have more thoughts on this. |
Beta Was this translation helpful? Give feedback.
-
|
Hi,
|
Beta Was this translation helpful? Give feedback.
-
|
It looks like your application needs more than 64k of heap, and so in v1.21.0 the heap must expand to two separate pools, the initial 64k pool and then a second one. This gives your application a MicroPython heap of 128000 bytes, which is about 10000 bytes more than in v1.20.0. Unfortunately it seems that this 10000 extra MicroPython heap is taking just a bit too much from the IDF and not leaving enough for an SSL connection. @jimmo maybe this is a common enough case (esp32-og, application needs just a little more than 64k MicroPython heap, and also wants to make one SSL connection) that we should tune it so that the second pool that's added to the heap leaves enough IDF heap for one SSL buffer? Or should we just wait for pre-allocated SSL buffers? |
Beta Was this translation helpful? Give feedback.
-
|
Quick thought, but we could ease off the "greedy" doubling as we start to approach the memory limit, as well. |
Beta Was this translation helpful? Give feedback.
-
|
Should be fixed now by f6d6308: the heap is tuned so expanding it once still allows to connect to WiFi and make an SSL request. |
Beta Was this translation helpful? Give feedback.
-
|
In my application, heap fragmentation was a big issue before, so the garbage collection is already on the heavier side. During startup (when many resources are created and cleaned up) the garbage collector threshold is set at 1024, which slows down the system a bit, but significantly reduces heap fragmentation. For my use-case, I ended up finding a user c module that had a fairly large lookup table (which was nice for performance). Deleting these tables resolved my issues with SSL connections. These tables were very nice to have, so this is not ideal. I'll continue my investigation into what is happening and if any of my c user modules that allocate dynamic memory are also impacted. This is the output of |
Beta Was this translation helpful? Give feedback.
-
|
I've added a tweak here to try not to prematurely exhaust the free system heap: @abegines @bobveringa Are you able to try this version with your applications and see if it helps? |
Beta Was this translation helpful? Give feedback.
-
|
Certainly looks promising, I'll try to figure out how I can run the version on my devices. I'm a bit time constraint right now, but I'll try to make some time somewhere next week. |
Beta Was this translation helpful? Give feedback.
-
|
I have just been chasing an ENOMEM issue is my app... This got me on the right path: https://forum.micropython.org/viewtopic.php?t=3499 So in my app I called In my app I am using So on my PI Pico W the main loop runs at ~54% free... import gc
import os
import time
def df():
s = os.statvfs('//')
return ('{0} MB'.format((s[0]*s[3])/1048576))
def free(full=False, collect=False):
if collect:
gc.collect()
F = gc.mem_free()
A = gc.mem_alloc()
T = F+A
P = '{0:.2f}%'.format(F/T*100)
if not full:
return P
else:
return ('Total:{0} Free:{1} ({2})'.format(T, F, P))
while True:
print(free(True, False))
time.sleep(1)Now the odd thing is if I use the above code and print |
Beta Was this translation helpful? Give feedback.
-
Something is probably fragmenting the Python heap, to the point where a large allocation is failing with ENOMEM. Running garbage collection more frequently prevents it from fragmenting, so there's still enough contiguous space for the large allocation when needed. If you run If you're not able to restructure the application to avoid a regular large allocation, the most common way around this is to set gc.threshold() and find a value where the GC runs often enough that fragmentation no longer gets out of hand. |
Beta Was this translation helpful? Give feedback.
I think what's going on here is a variation on #8940 and we hoped would be improved in v1.21 by the changes described in https://github.com/orgs/micropython/discussions/12316
The high-level issue is that on ESP32, the RAM is available to us and TLS (mbedtls) as disjoint regions of contiguous RAM. The old behavior was that MicroPython would allocate the largest such region and leave everything else for the IDF (wifi, ssl, socket buffers, etc). Different IDF versions resulted in a different memory layout as their internal allocations and data structures moved around (this is one major reason for differences in v1.21).
The problem we have now is that MicroPython can actually allocate more RA…