How can one create a persistent ClientSession

[raees-khan]

The official aiohttp doc is ambiguous, it says that creating a session per request is a very bad idea, but apparently doesn't tell how one can create a persistent session. There are plenty of examples all across the internet on how one can make concurrent requests, but there is unfortunately no example reusing the same session across all requests.

One would argue that the following code reuses the session which is partially true. Partially because every time I call main(), a new session will be created.

async def fetch(session, url):
    async with session.get(url) as resp:
        return await resp.text()

async def main():
    urls = ['https://httpstat.us/200'] * 10
    async with aiohttp.ClientSession() as session:
         tasks = [asyncio.create_task(fetch(session, url)) for url in urls]
         await asyncio.gather(*tasks)

asyncio.run(main())

The following example in doc is quite confusing:

session = aiohttp.ClientSession()
async with session.get('...'):
    # ...
await session.close()

Here's what I call a persistent session in its true sense because no matter where you call fetch from or how many times you call it, the same session will be reused, but this apparently doesn't work

class Session:
    def __init__(self):
        self.session = aiohttp.ClientSession()
    
    async def fetch(self, url):
        async with self.session.get(url) as resp:
            return await resp.text()

This code doesn't work and RuntimeError: Timeout context manager should be used inside a task exception is caught. If you pass timeout=None to self.session.get it fixes the RuntimeError, but you get other warnings about Unclosed Session. I've learnt that you shouldn't create a session outside a coroutine and was suggested something like async def create_session(): return aiohttp.ClientSession which brings me back to square on how would I persist it.

I believe the author should provide examples of creating persisting session.

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #2036 (ClientSession created in aiohttp.request() remains unclosed), #329 (Unittests for ClientSession), #2451 (Documentation for ClientSession), #3072 (Why is creating a ClientSession outside of an event loop "dangerous"?), and #857 (Can one ClientSession be used for multiple simultaneous websocket connections?).

[kornicameister]

@cpython-rocks Do this via cleanup_ctx.

app.cleanup_ctx.append(persisten_session)

async def persistent_session(app):
   app['MY_PERSISTENT_SESSION'] = session = aiohttp.ClientSession()
   yield
   await session.close()

async def my_request_handler(request) ->
   async with request.app['MY_PERSISTEN_SESSION'].get() as resp:
     ....

alternatively you can use on_startup and on_shutdown signals if running below Python 3.7. However I so much prefer async generator syntax 'cause everything is kept withing a single coroutine. I've introduced that in my project and it actually had been using 3 different sessions (one for internal cluster communication, 2nd for cloud communication and the 3rd was used to talk with some backup services).

Everything works nice so far and it's been 3 months since we've deployed that.

[raees-khan]

Thanks @kornicameister for your suggestion, but the documentation is ambiguous and needs an example on how should one create a persistent session. Just saying X is recommended and Y is a bad idea is not enough unless the doc also tells how to correctly implement X. There are many users that are struggling with this persistent session part.

[kornicameister]

I suppose you can drop a PR with this example or I can. Would actually wait for @asvetlov or someone else to confirm if that's the way or just a way.

[asvetlov]

the provided example is correct

[kornicameister]

@asvetlov @cpython-rocks I've added #3703. I hope everything will be in order there.

[shifr]

@kornicameister @asvetlov I’m a little confused now.

I usually use on_startup/on_shutdown for db/client-session setup/teardown but it looks like cleanup_ctx is a more convenient way to do that, am I right?
Could you please give some examples/best practices of the use cases for on_startup/on_shutdown

[asvetlov]

cleanup_ctx was added a couple of years later.

P.S.
@shifr please don't use a random old already closed issue to ask a completely unrelated question.
The "New issue" button exists for it; I hope it is pretty visible in GitHub UI.