make HTTP routes async

[technillogue]

this doesn't change any logic, but as far as I understand sync routes use threads instead of the starlette event loop: https://fastapi.tiangolo.com/async/#path-operation-functions

[nickstenning]

I believe this may not be as harmless as it looks.

There's a --threads argument to cog.server.http and I believe that it functions as a concurrency control for the synchronous prediction create endpoint.

At least some people rely on this behaviour so they can use the network socket's accept queue as a mechanism to queue up work. If this breaks that behaviour (i.e. those requests are immediately accepted and we serve a 409 instead) that may break those folks' expectations.

Is this change a requirement or just a nice-to-have for the batching/concurrent predictions work?

[technillogue]

Do we have any specific examples of someone actually using that, or is hypothetical?

I think it will be required somewhere down the line, I put it up front here because it can be reviewed as an independent PR

[nickstenning]

Yes, in that if I remember correctly, we broke it previously while working on Cog and had to put it back.

[technillogue]

we could stick a timeout or something where the endpoint waits for the worker for a similar time as what the tcp timeout would have been? do you remember what exactly the use case was and why they didn't want to use retries instead?

[nickstenning]

I think it would be nice if we could have a test for this behaviour, but I'm not going to block this on that. I've tested it locally and it works well!