Not enough HEAP on idf for sending http Requests

[ds-digid]

Checks

• I agree to follow the MicroPython Code of Conduct to ensure a safe and respectful space for everyone.

• I've searched for existing issues matching this bug, and didn't find any.

Port, board and/or hardware

LILYGO_TTGO_LORA32

MicroPython version

LILYGO_TTGO_LORA32-20240222-v1.22.2.bin

Reproduction

I have a program that fetches data from a serial interface and send it via http to a server.
As you can see the amount of free heap for the idf is very limitted. When I try to send data via http the memory will drop.

This code is put before calling the http request function and right after the receiption of the answer.

mem = esp32.idf_heap_info(esp32.HEAP_DATA)
print(f"SEND: {mem[5]}")

Output before sending:
SEND: (113840, 38844, 36864, 480)

Output on receive:
RECEIVE: (113840, 4032, 1856, 480)

As 4032 bytes are far to small for the next send process the device crashes.

Is there a way to extend the heap memory for the idf?

Expected behaviour

No response

Observed behaviour

2024-03-01T13:17:47 - CRITICAL - - Task exception wasn't retrieved ; Details: Wifi Internal Error ; IoT-Device version 0.3.2 - Exception:
2024-03-01T13:17:47 - CRITICAL - - handle_exception - Resetting device
Traceback (most recent call last):
File "asyncio/core.py", line 1, in run_until_complete
File "communication/microwifimanager/async_wifi.py", line 158, in _run
File "communication/microwifimanager/async_wifi.py", line 138, in _connect
OSError: Wifi Internal Error
ets Jun 8 2016 00:22:57

This fails obviosly because of too short memory on the idf heap.

Additional Information

No, I've provided everything above.

[projectgus]

Hi @ds-digid,

Recent versions of Micropython will automatically grow the Python heap as needed, meaning at some point during the request step the ESP32 would otherwise run out of memory for Python so it's expanded the heap (taking from IDF). So the best approach is to figure out which Python code requires a lot of memory and find ways to use less, or less at the one time. If the HTTP response is large then this depend on how it's handled, although not possible to make suggestions without seeing the code.

If there isn't anything obvious, one candidate may be compiling large .py files on demand. You can freeze packages into the firmware or use mpy-cross to produce .mpy files on the host and copy these over instead of the .py files.

[ds-digid]

Hi @projectgus,
thank you for your response. I checked this out. Python does not run out of heap. There is no problem on this side. I am still precompiling all my files with mpy-cross and work only with .mpy files on the device.
The http response is around ~400 Bytes. My expactation is that this should not be a great problem.
As the same code runs with Micropython version 1.21.0 my guess is, that it is a problem of the underlining idf implementation.

[ds-digid]

For more information: I use a custom async request library for http requests:

import asyncio
import gc
import json

from exceptions import ConnectionError

gc.collect()

# TODO: Remove unneccessary comments for RELEASE


class Response:
    def __init__(self, f, length=None):
        self.raw = f
        self.encoding = "utf-8"
        self._cached = None
        self._length = int(length)

    async def close(self):
        if self.raw:
            self.raw.close()
            await self.raw.wait_closed()
            self.raw = None
        self._cached = None

    @property
    async def content(self):
        if self._cached is None:
            try:
                if self._length is None:
                    self._cached = await self.raw.read()
                else:
                    self._cached = await self.raw.read(self._length)
            finally:
                self.raw.close()
                await self.raw.wait_closed()
                self.raw = None
        return self._cached

    @property
    async def text(self):
        return str(await self.content, self.encoding)

    async def json(self):
        return json.loads(await self.content)


async def request(
    method,
    url,
    data=None,
    loc_json=None,
    headers={},
    stream=None,
    auth=None,
    hostname=None,
    timeout=None,
    parse_headers=True,
):
    redirect = None  # redirection url, None means no redirection
    writer = None
    chunked_data = (
        data and getattr(data, "__iter__", None) and not getattr(data, "__len__", None)
    )

    if auth is not None:
        import ubinascii

        username, password = auth
        formated = b"{}:{}".format(username, password)
        formated = str(ubinascii.b2a_base64(formated)[:-1], "ascii")
        headers["Authorization"] = "Basic {}".format(formated)

    try:
        proto, dummy, host, path = url.split("/", 3)
    except ValueError:
        proto, dummy, host = url.split("/", 2)
        path = ""
    if proto == "http:":
        port = 80
    elif proto == "https:":
        port = 443
    else:
        raise ValueError("Unsupported protocol: " + proto)

    if ":" in host:
        host, port = host.split(":", 1)
        port = int(port)

    # ai = usocket.getaddrinfo(host, port, 0, usocket.SOCK_STREAM)
    # ai = ai[0]
    # print(proto, host, port)
    resp_d = None
    if parse_headers is not False:
        resp_d = {}

    # s = usocket.socket(ai[0], usocket.SOCK_STREAM, ai[2])

    # if timeout is not None:
    #     # Note: settimeout is not supported on all platforms, will raise
    #     # an AttributeError if not available.
    #     s.settimeout(timeout)

    try:
        # s.connect(ai[-1])
        # host ai[-1][0]
        sslctx = None
        if not hostname:
            hostname = host
        if proto == "https:":
            import ssl

            sslctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
            try:
                import certifi.digicert
                import certifi.isrg

                sslctx.load_verify_locations(
                    cadata=certifi.isrg.CACERTS + certifi.digicert.CACERTS
                )

            except Exception:
                sslctx.check_hostname = False
                sslctx.verify_mode = ssl.CERT_NONE

        # reader, writer = await asyncio.open_connection(host, port, ssl=sslctx)
        reader, writer = await asyncio.open_connection(host, port)
        writer.write(b"%s /%s HTTP/1.0\r\n" % (method, path))
        await writer.drain()
        if "Host" not in headers:
            writer.write(b"Host: %s\r\n" % host)
            await writer.drain()
        # Iterate over keys to avoid tuple alloc
        for k in headers:
            writer.write(k)
            writer.write(b": ")
            writer.write(headers[k])
            writer.write(b"\r\n")
            await writer.drain()
        if loc_json is not None:
            assert data is None

            data = json.dumps(loc_json)
            writer.write(b"Content-Type: application/json\r\n")
            await writer.drain()
        if data:
            if chunked_data:
                writer.write(b"Transfer-Encoding: chunked\r\n")
            else:
                writer.write(b"Content-Length: %d\r\n" % len(data))
            await writer.drain()
        writer.write(b"Connection: close\r\n\r\n")
        await writer.drain()
        if data:
            if chunked_data:
                for chunk in data:
                    writer.write(b"%x\r\n" % len(chunk))
                    writer.write(chunk)
                    writer.write(b"\r\n")
                    await writer.drain()
                writer.write("0\r\n\r\n")
                await writer.drain()
            else:
                writer.write(data)
                await writer.drain()
                gc.collect()

        l = await reader.readline()
        # print(l)
        l = l.split(None, 2)
        if len(l) < 2:
            # Invalid response
            raise ValueError("HTTP error: BadStatusLine:\n%s" % l)
        status = int(l[1])
        reason = ""
        if len(l) > 2:
            reason = l[2].rstrip()
        while True:
            l = await reader.readline()
            if not l or l == b"\r\n":
                break
            # print(l)
            if l.startswith(b"Transfer-Encoding:"):
                if b"chunked" in l:
                    raise ValueError("Unsupported " + str(l, "utf-8"))
            elif l.startswith(b"Location:") and not 200 <= status <= 299:
                if status in [301, 302, 303, 307, 308]:
                    redirect = str(l[10:-2], "utf-8")
                else:
                    raise NotImplementedError("Redirect %d not yet supported" % status)
            if parse_headers is False:
                pass
            elif parse_headers is True:
                l = str(l, "utf-8")
                k, v = l.split(":", 1)
                resp_d[k] = v.strip()
                # print(f"header: {k} : {v.strip()}")
            else:
                parse_headers(l, resp_d)
    except OSError as error:
        if writer is not None:
            writer.close()
            await writer.wait_closed()
        # raise connection error
        raise ConnectionError(error)

    if redirect:
        writer.close()
        await writer.wait_closed()
        if status in [301, 302, 303]:
            return await request("GET", redirect, None, None, headers, stream)
        else:
            return await request(method, redirect, data, json, headers, stream)
    else:
        length = resp_d.get("Content-Length")
        if length is None:
            length = 0
        resp = Response(reader, length)
        resp.status_code = status
        resp.reason = reason
        if resp_d is not None:
            resp.headers = resp_d
            return resp


async def head(url, **kw):
    try:
        return await request("HEAD", url, **kw)
    except Exception:
        raise


async def get(url, **kw):
    try:
        return await request("GET", url, **kw)
    except Exception:
        raise


async def post(url, **kw):
    try:
        return await request("POST", url, **kw)
    except Exception:
        raise


async def put(url, **kw):
    try:
        return await request("PUT", url, **kw)
    except Exception:
        raise


async def patch(url, **kw):
    try:
        return await request("PATCH", url, **kw)
    except Exception:
        raise


async def delete(url, **kw):
    try:
        return await request("DELETE", url, **kw)
    except Exception:
        raise

[Carglglz]

@ds-digid there is aiohttp now, which I made to replace this "custom async request" library, so try it and see if the problem persists. 👍🏼

[projectgus]

Python does not run out of heap. There is no problem on this side.

Hmm, OK. The point is that instead of running out of heap, Python will claim more memory from IDF - so you don't get any Python MemoryError, but you do get lower free IDF heap instead.

To rule this out as a problem, call micropython.mem_info() at the same two times you call idf_heap_info().

[ds-digid]

@Carglglz: I have tried the aiohttp library but the results are the same.

Here are some more informations with mem_info() on the specific points.

• First Block: A complete POST Request

• Second Block: Malfunction with the attempt to reconnect wifi on error. The REQUEST log is placed inside the request library where I catch the exception.

stack: 3168 out of 15360
GC: total: 111936, used: 103312, free: 8624, max new split: 36864
No. of 1-blocks: 1361, 2-blocks: 346, max blk sz: 282, max free sz: 67
POST: (113840, 39060, 36864, 4)
-------------------------------------
RECEIVE: (113840, 38992, 36864, 4)
stack: 3168 out of 15360
GC: total: 111936, used: 106336, free: 5600, max new split: 36864
 No. of 1-blocks: 1426, 2-blocks: 360, max blk sz: 282, max free sz: 64
+++++++++++++++++++++++++++++++++++++
2024-03-06T07:24:02+01:00 - HEARTBEAT status: 304
stack: 3168 out of 15360
GC: total: 111936, used: 106640, free: 5296, max new split: 36864
 No. of 1-blocks: 1433, 2-blocks: 361, max blk sz: 282, max free sz: 64

stack: 3168 out of 15360
GC: total: 111936, used: 106640, free: 5296, max new split: 36864
No. of 1-blocks: 1433, 2-blocks: 361, max blk sz: 282, max free sz: 64
POST: (113840, 39196, 36864, 4)
-------------------------------------
stack: 3680 out of 15360
GC: total: 147904, used: 105984, free: 41920, max new split: 1664
 No. of 1-blocks: 1397, 2-blocks: 349, max blk sz: 282, max free sz: 814
REQUEST: (113840, 3248, 1664, 4)
#######################################
2024-03-06T06:24:29 - ERROR -  - CommMgr - uart_http_requests - No Connection: [Errno 113] ECONNABORTED
Communication failure
Connecting to WiFi
Checking WiFi stability for 3s
Connecting to WiFi
2024-03-06T06:24:43 - CRITICAL -  - Task exception wasn't retrieved ; Details: Wifi Internal Error ; IoT-Device version 0.3.3 - Exception:
stack: 3392 out of 15360
GC: total: 147904, used: 103648, free: 44256, max new split: 1664
 No. of 1-blocks: 1361, 2-blocks: 346, max blk sz: 282, max free sz: 814
2024-03-06T06:24:43 - CRITICAL -  - handle_exception - Resetting device
Traceback (most recent call last):
  File "asyncio/core.py", line 1, in run_until_complete
  File "communication/microwifimanager/async_wifi.py", line 155, in _run
  File "communication/microwifimanager/async_wifi.py", line 129, in _connect
OSError: Wifi Internal Error
ets Jun  8 2016 00:22:57

[ds-digid]

Btw, is there any chance to print out the installed idf version?

[robert-hh]

import platform
print(platform.platform())

[ds-digid]

import platform
print(platform.platform())

Thank you

[projectgus]

Thanks @ds-digid .

From the numbers for the failed sequence above, between "before" and "after" snapshots MicroPython finds it has run out of heap and grows the total Python heap from 111936 to 147904 to avoid failure (+35968). This drops the available IDF memory from 39196 to 3248, causing the Wi-Fi internal failure after that.

Do you have an idea of what is using 106640 bytes of Python heap in the "before" snapshot? If you can find the cause of that, before the request even starts, and get this usage number down, then the later failure will go away.

[ds-digid]

@projectgus
The 106640 bytes come mainly from the imports at startup. Micropython seems not very efficient there. Some parts take an incredible amount of Heap memory.
For example:
importing the datetime library (previously installed via mip) takes ~20kB of heap memory. (I need this library because micropython builtin time module lacks compatibility with unix timestamps)
After all imports from all submodules in my system (With very often calls of gc.collect() to reduce memory consumption):

$Init System
stack: 2288 out of 15360
GC: total: 127936, used: 82176, free: 45760, max new split: 17408
 No. of 1-blocks: 935, 2-blocks: 280, max blk sz: 282, max free sz: 2740

another example:

class SCMsg:
    def __init__(self):
        self.deviceID = None
        self.msg_trackID = None

        self.bioSensId = None
        self.fwVersion = None

        self.devModel = None
        self.devState = None
        self.sensState = None
        self.error_number = None
        self.battery_status = None

        self.vref = None
        self.vcc = None
        self.sens_vground = None

        self.humidity = None
        self.temperature = None

        self.timestamp = None
        self.se1 = None
        self.se2 = None
        self.diff = None
        self.measurement_mode = None

        self.x_val_gyro = None
        self.y_val_gyro = None
        self.z_val_gyro = None
        self.or_xy_gyro = None
        self.or_z_gyro = None

        self.error_message = None

        self.has_status_msg = False
        self.has_health_msg = False
        self.has_rawdata_msg = False
        self.has_productionparameters_msg = False


class Wifi:
    def __init__(self):
        self.ssid = None
        self.password = None
        self.startwebserver = None


class HTTPRequest:
    """
    Represents an HTTP request

    Attributes:
        type (str): The type of the request
        url (str): The URL of the request
        json (dict): The JSON payload of the request
        status_code (int): The status code of the response
    """

    def __init__(self):
        self.type = None
        self.url = None
        self.json = None
        self.status_code = None
        self.request_type = None

The imports of this 3 classes alone take ~4kB of RAM. This is just insane in my view. There is not that much going on...

So that there is not that much free heap space for the running program.

[projectgus]

The 106640 bytes come mainly from the imports at startup. Micropython seems not very efficient there. Some parts take an incredible amount of Heap memory. For example: importing the datetime library (previously installed via mip) takes ~20kB of heap memory. (I need this library because micropython builtin time module lacks compatibility with unix timestamps) After all imports from all submodules in my system (With very often calls of gc.collect() to reduce memory consumption):

Right, thanks for the extra info. There's a few things you can do here, depending on how deep you want to go. It seems like your current setup almost has enough RAM, so with luck you don't need to change much to get it all working smoothly:

1. Write some workaround functions instead of pulling in the full datetime module. If all you need is to convert between Unix and Micropython Epoch time stamps, this can be done by adding or subtracting a constant.
2. Make a custom build that freezes the main dependencies into the firmware. They will use a lot less RAM compared to loading from the filesystem. Unfortunately this requires a MicroPython build environment. There is an open PR to add a feature allowing bytecode can be loaded via mpremote without using RAM, but not available yet.
3. Rewrite your code to reduce memory usage. One thing that can help is shortening names of variables, functions, etc. Each name is only stored in RAM once (unless frozen), but if you have a lot of unique long names then it can add up.

There are some similar discussions in other threads which may have other tips, for example.

The imports of this 3 classes alone take ~4kB of RAM. This is just insane in my view. There is not that much going on...

It can be a bit tricky to accurately measure this kind of thing. In particular, the first time you import something from the filesystem then there may be some additional filesystem initialisation that happens in the background and uses some RAM.

I compiled those classes down to a 790 byte .mpy file with mpy-cross (mostly strings for names), and as best as I can tell importing it uses 1760 bytes of RAM on ESP32 (although some of this is probably space for strings that isn't all used, so the next import can use some of that too.)

To accurately measure I also created a 0 byte file empty.py and compiled it to empty.mpy, to take out any initialisation costs from the first import.

Full session log - with and without empty.mpy - if you're interested

MPY: soft reboot
MicroPython v1.23.0-preview.34.g5785c6658d.dirty on 2024-01-10; Generic ESP32 module with ESP32
Type "help()" for more information.
>>> import micropython, gc
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 56000, used: 1152, free: 54848, max new split: 110592
 No. of 1-blocks: 20, 2-blocks: 6, max blk sz: 18, max free sz: 3406
>>> import scmsg
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 56000, used: 3440, free: 52560, max new split: 110592
 No. of 1-blocks: 41, 2-blocks: 15, max blk sz: 19, max free sz: 3260
>>> 
MPY: soft reboot
MicroPython v1.23.0-preview.34.g5785c6658d.dirty on 2024-01-10; Generic ESP32 module with ESP32
Type "help()" for more information.
>>> import micropython, gc
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 56000, used: 1152, free: 54848, max new split: 110592
 No. of 1-blocks: 20, 2-blocks: 6, max blk sz: 18, max free sz: 3406
>>> import empty
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 56000, used: 1856, free: 54144, max new split: 110592
 No. of 1-blocks: 27, 2-blocks: 7, max blk sz: 18, max free sz: 3339
>>> import scmsg
>>> gc.collect()
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 56000, used: 3616, free: 52384, max new split: 110592
 No. of 1-blocks: 47, 2-blocks: 16, max blk sz: 19, max free sz: 3241

[ds-digid]

Thank you for your explanations. I wrote a small workaround for the datetime module, now everything works much better. The PR looks very promissing to me. I hope this will find its way to the main branch very soon :)

In your example: importing an empty file will result in 700 Byte heap memory usage by the system if I get things right?

[projectgus]

In your example: importing an empty file will result in 700 Byte heap memory usage by the system if I get things right?

Yes but not every time, this is the first time after boot that something has accessed the file system, loaded a module that isn't frozen into the firmware, etc. So most of this is a one-off memory cost from setting these things up, not a memory cost for each module that is imported.

It'd be interesting to make an empty2.mpy and see how much memory it uses as the second module. I haven't done this, but pretty confident it will be much much less.