-
Notifications
You must be signed in to change notification settings - Fork 105
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Handle asynchronous cancellations for HTTPConnection instances #785
Comments
The issue appears to be HTTPConnections, which do not have the._connection attribute. Exampleimport trio
import httpcore
TIMEOUT = {"read": 5, "write": 5, "connect": 5, "pool": 5}
async def send_request(client, seen_response):
response = await client.request("GET", "http://127.0.0.1:8000", extensions={"timeout": TIMEOUT})
# Print the first response that we see and set the "seen_response" flag.
if not seen_response.is_set():
seen_response.set()
async def main():
async with httpcore.AsyncConnectionPool() as client:
while 1:
try:
async with trio.open_nursery() as nursery:
# This event is used to signal the first response we recieve.
seen_response = trio.Event()
# Kick off one hundred HTTP requests.
for idx in range(100):
nursery.start_soon(send_request, client, seen_response)
# As soon as we see a response we can cancel out of the nursery,
# rather than allowing all tasks to complete.
await seen_response.wait()
nursery.cancel_scope.cancel()
except BaseException:
print(client.connections[0].has_expired())
print(client.connections[0].is_available())
return
trio.run(main) OUTPUT
This is the section of code where connection pools determine whether or not the connection can be used for the request. httpcore/httpcore/_async/connection_pool.py Lines 161 to 167 in b649bb0
It appears that these types of connections are not releasing the connection pool. How to make these kinds of connections. ?This problem appears to occur if the newly created connection is cancelled before entering this try block. httpcore/httpcore/_async/connection.py Lines 75 to 99 in b649bb0
I'm not sure this is what we're looking for; what do you think, @tomchristie ? |
The simplified version of the reproducible code for this issue. import httpcore
pool = httpcore.ConnectionPool()
for j in range(10):
pool._pool.append(
httpcore.HTTPConnection(httpcore.Origin(b"https", b"www.google.com", 443))
)
response = pool.request(
"GET", "https://www.google.com", extensions={"timeout": {"pool": 5}}
) # timeout error |
@tomchristie Can you give me some advice here? Should I open a PR that resolves such cases? |
The gist of the race condition is that when a web request is cancelled within a specific time window, it will degrade the state of the connection pool, leaving it unable to issue further requests. It also seems to create a deadlock.
It can be reproduced with the following code:
This should produce a
httpcore.PoolTimeout
error on the second or third iteration, after which the loop enters into a deadlock.The text was updated successfully, but these errors were encountered: