Fire and forget http requests

[jean-david-ruvini]

I am trying to send a large number (hundreds or thousands) of HTTP POST requests in a fire-and-forget manner, meaning I don't want to wait for the responses of these requests.

I am creating a session with aiohttp.ClientSession() and sending the requests using asyncio.create_task(session.myfunc(url)).

However, I believe that the session object cannot be closed until all the requests have completed, which means the calling function needs to wait before exiting and closing the session. This seems to defeat the purpose of fire-and-forget.

What am I missing?

Thanks in advance for any help

[Dreamsorcerer]

create_task() cannot be used for fire-and-forget (pretty sure it highlights this in the docs these days).

Instead, use aiojobs or something similar. You can then await the scheduler's wait_and_close() before exiting the ClientSession.

[Dreamsorcerer]

https://aiojobs.readthedocs.io/en/stable/#usage-example

[wisecuesdev]

Thank you very much for the pointer.

However, it seems that wait_and_close "wait for currently scheduled tasks to finish". Which is not exactly what we are trying to accomplish. The post requests can take a long time to execute and we don't want the caller to wait for a response (which I guess is what "tasks to finish" implies). Instead, we want to exit has soon as all the data has been sent.

[Dreamsorcerer]

How do you know the requests were sent successfully, if you haven't received a response?

If you really want that, then just cancel all the request tasks after initiating them? Or use the tracing API to figure out when the request has been sent and cancel then.

[wisecuesdev]

We can later verify if the processing was successful and retry it.
Yes, we really want to send hundreds of thousands of post requests but cannot wait for each one of them to return a response.
We cannot cancel them unless there is a high probability that the data was sent and that the server has started working on it. We are not sure how to accomplish that. Maybe operating at the socket level?
Another option is to change the server and have it return a response and process the data aync. But that's more complicated.
Thanks!