Received "Response payload is not completed" when reading response #4581
Comments
|
CC: @socketpair @asvetlov as the TODO was added in 5c4cb82 |
|
I'm curious if it's somehow related to #4549. |
Change the |
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. This should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have CIBA disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # scripts/run-test-plan.py
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # scripts/run-test-plan.py
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783
|
Hi there, I've received this error too. My use case is a reverse-proxy in front of gunicorn. The other reports of this instance on this repo appear to be poorly behaved HTTP servers, which I think is a separate issue. My code streams the proxy body back to upstream, using code roughly like: response = web.StreamingResponse(headers=response_headers, status=status)
async for chunk in gunicorn_response.content.iter_chunked(64 * 1_024):
await response.write(chunk)
await response.write_eof()
return responseWhat seems to be happening is that gunicorn returns a Connection: close header and then there is a race condition between aiohttp reading the body and the connection being closed. If I get the data out of aiohttp in time, it works, but sometimes this stack trace is triggered. Investigation:
|
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # .gitlab-ci.yml # .gitlab-ci/run-tests.sh # scripts/run-test-plan.py # test/Dockerfile
Use python's asyncio / await / async to run modules within one test plan in parallel if they don't have an alias, and to run plans in parallel where they use no / different aliases. The parallelism of tests within a plan should speed up the CIBA tests and OIDC test modules where we use dcr can hence can proceed without any alias. Unfortunately the CIBA tests had to have parallelism disabled because Authlete's CIBA simulated authentication device doesn't seem to cope with multiple parallel. Reduces the time to run oidcc-basic-certification-test-plan from 313 seconds to 63 seconds on my local machine - most of which comes from the test that sleeps for 30 seconds before reusing an auth code. The ability to run plans with different aliases in parallel means we can run some of the FAPI tests in parallel with other tests. There may be further potential to speed this up by tweaking the aliases and redirect urls we use in (say) half of the FAPI tests. There was some hassle with python modules/node, I tried switching away from alpine for the reasons given here, where we were seeing issues installing the aiohttp module: https://pythonspeed.com/articles/alpine-docker-python/ but ending up switching back because the version of node (10) in Debian buster is ancient and doesn't support some of the crypto Fillip's client needs. Luckily the aiohttp module is one of the ones that can relatively easily be made to work on alpine. We add a retrying http client, as it seems either the parallelisation or something about the asyncio http client ends up giving weird http errors - I think we're running into the same bug as here: aio-libs/aiohttp#4581 Switching to https://www.python-httpx.org might be an option to avoid that, but that's still in beta. part of #783 # Conflicts: # .gitlab-ci.yml # .gitlab-ci/run-tests.sh # scripts/run-test-plan.py # test/Dockerfile
|
we have this happening between an aiohttp 3.8.1 client + server in production, trying to reproduce |
|
I've just experienced the same issue via s3fs, downloading a few hundred files from S3. Python: 3.7.12 Traceback (most recent call last):
File "dataset_io.py", line 336, in download
filesystem.get(self._location, self._download_location, recursive=True)
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
return sync(self.loop, func, *args, **kwargs)
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
raise return_result
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
result[0] = await coro
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 529, in _get
coros, batch_size=batch_size, callback=callback
File "/.venv/lib/python3.7/site-packages/fsspec/asyn.py", line 249, in _run_coros_in_chunks
await asyncio.gather(*chunk, return_exceptions=return_exceptions),
File "/root/.pyenv/versions/3.7.12/lib/python3.7/asyncio/tasks.py", line 414, in wait_for
return await fut
File "/.venv/lib/python3.7/site-packages/s3fs/core.py", line 1002, in _get_file
chunk = await body.read(2 ** 16)
File "/.venv/lib/python3.7/site-packages/aiobotocore/response.py", line 53, in read
chunk = await self.__wrapped__.read(amt if amt is not None else -1)
File "/.venv/lib/python3.7/site-packages/aiohttp/streams.py", line 385, in read
await self._wait("read")
File "/.venv/lib/python3.7/site-packages/aiohttp/streams.py", line 304, in _wait
await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completedSorry I don't have much more information here, other than the fact that this is occurring on Python 3.7, whereas #4549 has been described as occurring on Python 3.8 only. |
|
I got this error on Python 3.9 with aiohttp 3.8.1 on Windows. I haven't been able to reproduce. My server failed around the same time as the error. The |
|
I have the same issue with Python 3.10 |
This comment was marked as duplicate.
This comment was marked as duplicate.
I am also getting that error using iter_chunked when downloading files from S3 |
Instead, pass through the upstream Connection header, like we do with Upgrade: already. This helps to avoid aio-libs/aiohttp#4581
This is conceptually broken -- the browser should decide when it wants to upgrade a connection to websocket, and we should just pass that through (like we do with `Upgrade:` already). This is also what the real 3scale does, so more faithfully reproduces its behaviour. This also helps to avoid aio-libs/aiohttp#4581
This is conceptually broken -- the browser should decide when it wants to upgrade a connection to websocket, and we should just pass that through (like we do with `Upgrade:` already). This is also what the real 3scale does, so more faithfully reproduces its behaviour. This also helps to avoid aio-libs/aiohttp#4581
…r retrying (#14545) The treatment of `ClientPayloadError` as a sometimes transient error was originally made in response to [an existing issue](aio-libs/aiohttp#4581) in aiohttp that can cause transient errors on the client that are difficult to distinguish from a real broken server. What's in `main` matched exactly on the error message, but that error message has [since changed](aio-libs/aiohttp@dc38630) to include more information, breaking our transient error handling. This change relaxes the requirement of the error response string to fix transient error handling for our current version of `aiohttp`. I wish I had a better approach. `ClientPayloadError` can also be thrown in the case of malformed data, so I am reticent to treat it as always transient, but we could perhaps make it a `limited_retries_error` and avoid inspecting the error message.
|
Sorry - was there an official fix for this? I am able to reproduce when utilizing Basically: File "path/Library/Caches/pypoetry/virtualenvs/parlay-pricing-service-OsJas3qw-py3.11/lib/python3.11/site-packages/de_common/aws/s3.py", line 229, in get_object
body = await response["Body"].read()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "path/Library/Caches/pypoetry/virtualenvs/parlay-pricing-service-OsJas3qw-py3.11/lib/python3.11/site-packages/aiobotocore/response.py", line 56, in read
chunk = await self.__wrapped__.content.read(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "path/Library/Caches/pypoetry/virtualenvs/parlay-pricing-service-OsJas3qw-py3.11/lib/python3.11/site-packages/aiohttp/streams.py", line 383, in read
block = await self.readany()
^^^^^^^^^^^^^^^^^^^^
File "path/Library/Caches/pypoetry/virtualenvs/parlay-pricing-service-OsJas3qw-py3.11/lib/python3.11/site-packages/aiohttp/streams.py", line 405, in readany
await self._wait("readany")
File "path/Library/Caches/pypoetry/virtualenvs/parlay-pricing-service-OsJas3qw-py3.11/lib/python3.11/site-packages/aiohttp/streams.py", line 312, in _wait
await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <ContentLengthError: 400, message='Not enough data for satisfy content length header.'>Anyone have any suggestions or thoughts? I'm using |
|
I think I have a reliably reproducible example that is derived from @gtedesco-r7's example Python 3.12 A |
|
Was there traction on this (or one of the other incredibly similar errors)? I suspect I'm getting some form of throttle that is causing things to get weird after spending a while reading all sorts of random posts from folks. I can't seem to get a better error message no matter how I try, probably because one of the libraries in the middle eats it or something. I'm downloading a 124.5MB file from Google Cloud using the gcloud-aio library and fastapi with a streaming response. And I intermittently get this exception. My download takes roughly 4-5 seconds when working correctly. But if I try it a few times in a row (say like 5 or 6), it errors out instead of working. aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed: <ContentLengthError: 400, message='Not enough data for satisfy content length header.'>. SSLError(1, '[SSL] record layer failure (_ssl.c:2580)')I suspect this is a me problem in some manner, but I can't seem to figure out what issue I'm trying to fix. Here's my fastapi endpoint: async def download(request: Request) -> StreamingResponse:
async def generate_response():
# we need to make a new session inside this generator
# because the one on the request context is closed when we return
# the streaming response, and then it goes byebye and fails
# the streaming download out with a partial file error.
async with aiohttp.ClientSession(raise_for_status=True) as blob_session:
new_storage_client = Storage(session=blob_session)
async with await new_storage_client.download_stream(bucket, file) as stream:
yield await stream.read()
return StreamingResponse(generate_response()) |
|
For our situation, we've figured out that yeah, it was legitimately the connection being broken from the server side (I.e. No clear bug from aiohttp). We had Nginx as a load balancer/reverse proxy in front of our servers, and whenever a reload happens, the workers would get killed some time after, and any long running requests would get forcefully terminated (and correctly reported as an error). |
Dreamsorcerer
added
the
needs-info
|
I can give it a shot this weekend when I'll have the ability to let it run for an extended period (assuming my reattempt at a long travel day tomorrow doesn't go as poorly as my attempt on Wednesday and I actually get to my destination) |
|
Hey! I’m encountering a similar issue where the connection appears to be terminated on the server, resulting in a Would simply retrying the request on the client side be sufficient to handle this error, or would it be necessary to recreate the aiohttp session (or implement any special error-handling logic)? Will aiohttp attempt to reconnect to the server automatically upon a retry? |
|
The project you've referenced above appears to be using 3.10.3, the fix I mentioned above is in 3.10.4. If you're still experiencing problems on the latest release, you'll have to try and figure out a reproducer or some way to log the raw chunks. However, you should verify whether you are using the compiled C parser (e.g. wheels from pip) or the Python parser (which can be forced with AIOHTTP_NO_EXTENSIONS=1 envvar) first. If the server is sending malformed responses, then the behaviour appears correct, if you send another request to retry, then aiohttp should close the bad connection and create a new connection for the next request. If the responses are correct, then we'd need to find the cause of the issue here. |
|
@Dreamsorcerer I'm afraid I'm at capacity these days, so I'll defer to @bdraco for now. |
|
I started running the test client again. I'll leave it for a few hours and report back
|
|
No longer reproducible on 3.10.5. I'd say this is fixed |
|
I'm getting this error with the following: male_doctor = 'Arzt' # works
FEMALE_DOCTOR = 'Ärztin' # THROWS
async with aiohttp.ClientSession(
connector=aiohttp.TCPConnector(ttl_dns_cache=300, enable_cleanup_closed=True),
cookie_jar=aiohttp.DummyCookieJar(),
raise_for_status=True,
timeout=aiohttp.ClientTimeout(sock_connect=10)
) as session:
async with session.get('https://www.linguee.com/english-german/search',
params={'query': FEMALE_DOCTOR}) as rsp:
txt = await rsp.text()Server return slightly different responses to a successful and an unsuccessful queries. Artz has Ärztin doesn't have May be a server problem, but I have no control over it, and browser and curl manage to parse the response successfully. I'm on Linux. Stacktrace: |
|
The server closes the connection without completing the message. If we wanted to support something like this, we'd need some way to signal to the user that the message is probably not complete. i.e. There is no way for us to know if what we received is a valid message or only part of one (or, strictly speaking, it is only part of a message). |
|
I don't see it working with curl though: |
Had the same response for both Arzt and Ärztin. When I said curl is working I meant that there is no difference between a throwing and a non-throwing response as far as curl is concerned. I haven't played with command line options, probably there was some way to decode the binary output. Apparently they've fixed the server: can't reproduce a problem anymore. curl also print HTML to the console.
Would be appreciated, apparently the problem pop-ups from time to time. May be change |
Huh, I didn't test curl with the other one. But, I was pretty sure that it was producing binary output because the message was incomplete. The binary output was the raw brotli encoded bytes, i.e. curl didn't attempt to decompress the response.
If you're using a .read() variant that loads the whole body into memory anyway, then that could possibly work. If someone wants to try and implement that, we'll review it. But, given it is literally invalid HTTP behaviour and very rarely encountered, I doubt any maintainers will volunteer to work on it. |
|
In case there are sad people like me who still encounter this problem. If you are also using nginx as a reverse proxy and somewhere in the config you have By making this |
and others
🐞 Describe the bug
A
ClientPayloadError:Response payload is not completedexception is raised when reading the response of a GET request from the GitHub REST API (this should not be especially relevant). The response seems correct and usingcurlsucceeds all the time.💡 To Reproduce
Do lots of requests against an endpoint of the GitHub API (it happens from time to time).
💡 Expected behavior
The response is correctly parsed.
📋 Logs/tracebacks
Some contextual information at the time of the exception at
aiohttp/streams.py", line 358, in read:As you can see the blocks contain the whole payload (the full JSON) but
aiohttpstill complains about the payload not being completed.📋 Your version of the Python
📋 Your version of the aiohttp/yarl/multidict distributions
📋 Additional context
I suspect this has to be with this comment:
aiohttp/aiohttp/streams.py
Lines 352 to 356 in 1740c99
as this is where is failing and the situation seems that is the one described in the
TODO.The text was updated successfully, but these errors were encountered: