ClientSession benchmark only works with aiohttp.TCPConnector(force_close=True)

[aparamon]

Long story short

Simple benchmark test fails on the defaults but works with connector=aiohttp.TCPConnector(force_close=True).

Steps to reproduce

1. docker run -p 80:80 httpd
2. python3 test.py

import asyncio
import aiohttp

async def test():
    async def fetch():
        async with aiohttp.ClientSession() as client:
            for i in range(3):
                async with client.get('http://localhost') as r:
                    assert r.status == 200
                    await r.text()

    tasks = tuple(fetch() for i in range(1000))
    await asyncio.gather(*tasks)

asyncio.run(test())

Traceback (most recent call last):
  File "test-hl.py", line 15, in <module>
    asyncio.run(test())
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "test-hl.py", line 13, in test
    await asyncio.gather(*tasks)
  File "test-hl.py", line 8, in fetch
    async with client.get('http://localhost') as r:
  File "/usr/local/lib/python3.7/dist-packages/aiohttp/client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.7/dist-packages/aiohttp/client.py", line 497, in _request
    await resp.start(conn)
  File "/usr/local/lib/python3.7/dist-packages/aiohttp/client_reqrep.py", line 844, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/usr/local/lib/python3.7/dist-packages/aiohttp/streams.py", line 588, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: None

3. Change
   async with aiohttp.ClientSession() as client:
   to
   async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(force_close=True)) as client:
4. python3 test.py
   Script runs to completion in several seconds.

It's expected that the default setup (with keep-alive enabled) is at least as stable as with force_close=True for this benchmark.

Your environment

Windows 10 and Debian Gnu/Linux, Python 3.7, aiohttp 3.5.2.

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #2224 (aiohttp.TCPConnector()), #2867 (aiohttp.ClientSession reconnect ), #975 (aiohttp.ClientSession raises ServerDisconnectedError), #1812 (Working with aiohttp.ClientSession from a command line), and #3371 (wrong cookies when i use aiohttp.ClientSession(cookies=cookies)).

[MtzwOsk]

I have similar problem with my script below. python 3.7.0 and aiohttp-3.5.4

import aiohttp
import asyncio
import math


from bs4 import BeautifulSoup as bs

BEGIN_COUNTER: int = 11
TABLE_PAGINATION: int = 10

query_params = {
    'func': 'find-b',
    'request': 'Romeo i Julia',  # TITLE
    'find_code': 'WTI',
    'adjacent': 'Y',
    'local_base': 'SROBK',
}


async def fetch(session, url):
    print(id(session))
    async with session.get(url) as response:
        print(response.status)
        return await response.text()
    

async def fetch_with_params(session, url, params):
    print(id(session))
    async with session.get(url, params=params) as response:
        print(response.status)
        return await response.text()


def create_soup(html):
    return bs(html, 'html.parser')


def get_number_of_tables(soup):
    books_number = int(list(filter(None, soup.find(class_='text3').text.split(' ')))[6])  # math.ceil
    return math.ceil(books_number/TABLE_PAGINATION)


async def main():
    async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(force_close=True)) as session:
        # base_request = [asyncio.ensure_future(fetch_with_params(session, 'https://aleph.koszykowa.pl/F', params=query_params))]
        # html = await asyncio.gather(*base_request)
        html = await asyncio.wait_for(fetch_with_params(session, 'https://aleph.koszykowa.pl/F', params=query_params), timeout=3.0)
        # html = await fetch_with_params(session, 'https://aleph.koszykowa.pl/F', params=query_params)
        soup = create_soup(html)
        tables_number = get_number_of_tables(soup)
        urls = [f'https://aleph.koszykowa.pl/F?func=short-jump&jump={BEGIN_COUNTER+10*number}' for number in range(tables_number)]
        tables_request = [asyncio.ensure_future(fetch(session, url)) for url in urls]
        result = asyncio.gather(*tables_request)
        return await result

if __name__ == '__main__':
    test = asyncio.run(main())
    

Traceback (most recent call last):
  File "/home/mtzw/projects/get_book/aio.py", line 56, in <module>
    test = asyncio.run(main())
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 568, in run_until_complete
    return future.result()
  File "/home/mtzw/projects/get_book/aio.py", line 53, in main
    return await result
  File "/home/mtzw/projects/get_book/aio.py", line 21, in fetch
    async with session.get(url) as response:
  File "/home/mtzw/projects/get_book/env/lib/python3.7/site-packages/aiohttp/client.py", line 1005, in __aenter__
    self._resp = await self._coro
  File "/home/mtzw/projects/get_book/env/lib/python3.7/site-packages/aiohttp/client.py", line 497, in _request
    await resp.start(conn)
  File "/home/mtzw/projects/get_book/env/lib/python3.7/site-packages/aiohttp/client_reqrep.py", line 844, in start
    message, payload = await self._protocol.read()  # type: ignore  # noqa
  File "/home/mtzw/projects/get_book/env/lib/python3.7/site-packages/aiohttp/streams.py", line 588, in read
    await self._waiter
aiohttp.client_exceptions.ServerDisconnectedError: None

[pynkpanther]

we just hit the same bug using aiohttp 3.7.4.

• our workarount was to always create a new session for each request but should result in the same as force_close=True
• we first noticed the bug when we switched the destination webserver from kraken proxy to self implemented proxy using uvicorn.
• uvicorn uses default keep-alive of 5 seconds. i guess the kraken proxy is more optimized for this kind of stuff...
• nevertheless, if the destination server closing the connection is in fact the issue, then i would expect aiohttp to do proper error handling since in most cases i might not know the destination server implementation

[Hanaasagi]

It looks like aiohttp take a disconnected connection from the connection pool. But if the server close the connection first, I remember that the socket fd will become readable and return '' when call read. I'm not sure connection will be auto removed in this case.

[Dreamsorcerer]

This may be caused by (badly implemented) servers which close a connection immediately after sending a response with a keep-alive header. In this case, aiohttp (rightly) assumes the connection will remain open and releases it to the connection pool. It then gets picked up for another request before the server close event comes through, resulting in an error when the new request is made.

To work around this, we now have retry logic for safe/idempotent methods (e.g. GET), so these common errors shouldn't occur anymore. Can you retest with 3.10?