stream response memory leak

[odysseusmax]

Long story short

I'm using aiohttp for downloading files. From my experience I found that the chunked stream requests are causing memory leaks. I'm working with large files and i have very limited memory. When i make a request, it completes the request succesfully, but the response is loaded into memory and not released causing memory overflow.

Expected behaviour

Load chunked responses and release it to avoid memory leak.

Actual behaviour

Loads response content as chunks and not releasing.

Steps to reproduce

This is what i use.

url = 'some direct download link'
filename = 'some file name'
async with aiohttp.ClientSession() as session:
                async with session.get(url) as r:
                    if(r.status < 400):
                          with open(filename, 'wb') as fd:
                                        async for chunk in r.content.iter_any():
                                            fd.write(chunk)

Your environment

using aiohttp-client
script running on heroku
latest aiohttp (as of now)

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #1656 (Memory leak), #3631 (Chunked stream memory leak), #271 (Memory leak in request), #1029 (Memory leak when doing https request), and #133 (Memory leak when doing https request).

[socketpair]

Seems, this is #3631. Please recheck. If so, please close issue. If not, please provide us a way to reproduce the bug.

[socketpair]

@asvetlov interesting, this bug was added a long time before I have started to inspect streams code :)

[odysseusmax]

Seems, this is #3631. Please recheck. If so, please close issue. If not, please provide us a way to reproduce the bug.

The code i use is provided here. The issue you mentioned is with post method, for me its with get method.

[socketpair]

@odysseusmax you did not provide exact URL. The issue I have provided is not connected with specific method. It is connected with reading a chunked response. So, GET is affected too.

[odysseusmax]

@socketpair any link that points to some file greater than 1GB

[socketpair]

@odysseusmax if so, I consider this bug is fixed. Please re-test on latest 4.0 or 3.5 branch.

[odysseusmax]

@odysseusmax if so, I consider this bug is fixed. Please re-test on latest 4.0 or 3.5 branch.

okey will test and respond

[socketpair]

Reopen please if required.

[odysseusmax]

Issue is solved in 4.0

[socketpair]

@odysseusmax actually in latest 3.5 too.