Document a way to set up aiohttp for CPU-bound, logically-synchronous apps

[underyx]

Long story short

My company uses aiohttp for over 100 microservices. Sometimes we have services that are logically not asynchronous, and instead do lots of CPU work for each request. We still like to use aiohttp for these, though; it has a nice, familiar API, and we have lots of resources (libraries, documentation, etc.) for aiohttp that we want to keep using.

We're not sure how to optimize the deployment for this case, though! It seems that aiohttp provides us no way of limiting concurrency and we can't find a way to set gunicorn up with the aiohttp worker so that it keeps a backlog of requests to avoid overloading workers and grinding the whole system to a halt. We tried creating a middleware that responds with HTTP 503 if a concurrency limit is reached, but requests are very unevenly distributed by gunicorn and without a backlog system from which workers pick up requests when idle, we end up with way too many 503s.

I think it would be nice to have some notes in the server deployment docs about how to optimally operate a CPU-bound app 🙃

Your environment

• kubernetes nginx-ingress-controller
• gunicorn 19.9.0
• aiohttp server 3.5.4

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #3454 ([Question] - Eventloop busy with CPU bound operations), #2638 (There is no way to set limit size for response.read()), #3274 (1K Request can't be finished in a simultaneous way in aiohttp), #2224 (aiohttp.TCPConnector()), and #2320 (aiohttp-jinja2).

[asvetlov]

The question is too broad.
You have CPU bound tasks (I hope you run them in executor).
These tasks overload the server.
What do you want to achieve?

[underyx]

You have CPU bound tasks (I hope you run them in executor).

No, I don't think so. I'm not familiar with the concept of executor, and https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor does not really explain. Could you point me to the correct docs please?

What do you want to achieve?

I would like a request backlog that aiohttp workers take requests from one by one instead of all the workers just trying to process unlimited requests (they can take more while the request body is being read asynchronously) and therefore oversubscribing. Setting a backend connection limit on HAProxy to 1 or 2 might be the solution: https://github.com/jcmoraisjr/haproxy-ingress#connection

[asvetlov]

https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor is a good starter for learning about running CPU-bound tasks

Setting a connection limit on reverse proxy is a viable solution