ServerDisconnectedError with get compress=True

[kapkirl]

Long story short

While using aiohttp found a lot of ServerDisconnectedError messages, while simple curl fetched same target with no problems. So after small research I made some scripts to reproduce behavior. And looks like session can't finish get request with compress=True.

Expected behaviour

Sessions get method finishes properly. Is there way I can use compression in such conditions?

Actual behaviour

Using compress=True in get request - produces ServerDisconnectedError.

    ....
   connector = TCPConnector(force_close=False)
    async with ClientSession(connector=connector) as session:
        while not tq.empty():
            num = tq.get()
            try:
                async with session.get(url=url, compress=True, allow_redirects=True) as response:
                    txt = await response.read()
    .....

I was trying to force closing connection by force_close=True in connector. But that caused creation of new hanging connections (tracking them with watch -n 0.1 'netstat -n | grep :8080 | wc').

When using compress=False. Everything works fine.

Steps to reproduce

server.py

import asyncio
from datetime import datetime
from aiohttp import web
import random

random.seed(1)

html_body = b'Hello world!!!'

async def hello(request):
    n = datetime.now().isoformat()
    delay = random.randint(0, 0)
    await asyncio.sleep(delay)
    headers = {"content_type": "text/html", "delay": str(delay), 'request_catched': str(n)}
    response = web.Response(body=html_body, headers=headers)
    return response

app = web.Application()
app.router.add_route("GET", "/{name}", hello)
web.run_app(app)

client.py

import sys
import traceback
import asyncio
from aiohttp import ClientSession, TCPConnector
import queue
import datetime

max_tasks = 10
err_count = 0
loop = asyncio.get_event_loop()
tq = queue.Queue()
rq = queue.Queue()


async def fetch(worker_id):
    global err_count
    url = "http://localhost:8080/hello"
    connector = TCPConnector(force_close=False)
    async with ClientSession(connector=connector) as session:
        while not tq.empty():
            num = tq.get()

            try:
                async with session.get(url=url, compress=True, allow_redirects=True) as response:
                    txt = await response.read()
                    sys.stdout.write("[{}]: {}: OK: {}: {}. result: {}\n".format(
                        worker_id, datetime.datetime.now(), num, response.url, txt))
                    rq.put(num)
                    sys.stdout.write("{}\n{}\n".format(response.cookies, response.raw_headers))
                    sys.stdout.write("{}\n".format('-' * 10))

            except Exception as e0:
                sys.stdout.write('[{}]: {}: Exception: {}\n'.format(worker_id, datetime.datetime.now(), e0))
                traceback.print_exc()

                tq.put(num)
                err_count += 1


async def control():
    global err_count
    dif = 0
    prev_size = 0
    qsize = rq.qsize()
    while qsize < max_tasks:
        await asyncio.sleep(1)
        qsize = rq.qsize()
        sys.stdout.write('{}: qsize: {}; dif: {}; err_count: {}\n'.format(
           datetime.datetime.now(), qsize, qsize - prev_size, err_count
        ))
        err_count = 0
        prev_size = qsize
    loop.stop()

for t in range(max_tasks):
    tq.put(t)

asyncio.ensure_future(control())
workers_count = 1
for i in range(workers_count):
    asyncio.ensure_future(fetch(i))
loop.run_forever()
loop.close()

Start running server.py, and then client.py. Got such logs from client.

[0]: 2018-11-29 14:36:04.136142: OK: 0: http://localhost:8080/hello. result: b'Hello world!!!'
....
got no results, and after a minute ....
....

    async with session.get(url=url, compress=True, allow_redirects=True) as response:
  File "/usr/local/lib/python3.6/site-packages/aiohttp/client.py", line 855, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.6/site-packages/aiohttp/client.py", line 391, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 757, in start
    message, payload = await self._protocol.read()
  File "/usr/local/lib/python3.6/site-packages/aiohttp/streams.py", line 543, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: None

after this message again OK result, and then same error.

Your environment

Ubuntu 14.04
Python 3.6.1
aiohttp 3.4.4

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #117 (HTTP GET + compress=True = 501), #2198 (Document encoded=True for yarl.URL), #3364 (HTTP response not released with raise_for_status=True), #394 (ServerDisconnectedError is trigger happy), and #2859 (Deprecate Application(debug=True) in favor of DEBUG global flag).

[kapkirl]

Any suggestions on this subject?

[Dreamsorcerer]

Not sure why the reproducer above is so complicated. I can still reproduce this issue with:

Server

from aiohttp import web

async def hello(request):
    return web.Response(text="hello")

app = web.Application()
app.router.add_route("GET", "/", hello)
web.run_app(app)

Client

import asyncio
from aiohttp import ClientSession

async def fetch():
    async with ClientSession() as session:
        for _ in range(10):
            async with session.get("http://localhost:8080/", compress=True) as response:
                txt = await response.read()

asyncio.run(fetch())

Client now reports ConnectionResetError: Cannot write to closing transport
Which appears to be correct, because the server is reporting:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 355, in data_received
    messages, upgraded, tail = self._request_parser.feed_data(data)
  File "aiohttp/_http_parser.pyx", line 557, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadStatusLine: 400, message:
  Invalid method encountered:

    b'x\x9c\x03'
      ^

Curiously, if I use a TCPConnector(force_close=True), that resolves the client side exceptions, but I still get the BadStatusLine on several requests. Something must be messing up the decompression.

[Dreamsorcerer]

Actually, docs say:

Set to True if request has to be compressed with deflate encoding. If compress can not be combined with a Content-Encoding and Content-Length headers.

So, this option is incorrectly compressing the request without any headers for the server to know this.
This is presumably for some kind of legacy thing and shouldn't be used generally.

We should atleast rename the parameter in v4 to make this a lot clearer.