Timeout issues with httpx.AsyncClient on a large number of requests (around 100k domains)

[loic-bellinger]

Hello,

I am encountering a problem with httpx.AsyncClient when making requests to a large number of different domains and subdomains (about 100,000). Many of these requests fail with timeouts, but when I reduce the number of requests to a smaller subset, they succeed without issues (status code 200).

Context:
I am using httpx.AsyncClient with a basic configuration and a typical pattern for handling requests:

async with self.client:
    tasks = (self.check_single_domain(domain) for domain in domains)
    return await asyncio.gather(*tasks)

async def check_single_domain(self, domain: str) -> list[DomainCheckResult]:
    tasks = (
        self._get(domain, f"{protocol}://{subdomain}{domain}")
        for protocol in self.PROTOCOLS
        for subdomain in self.SUBDOMAINS
    )
    return await asyncio.gather(*tasks)

async def _get(self, domain: str, url: str) -> DomainCheckResult:
    try:
        response = await self.client.get(url)
        response.raise_for_status()
    except (httpx.HTTPError, httpx.InvalidURL, httpx.CookieConflict, httpx.StreamError) as exc:
        # Specific error handling
        ...
    except Exception as exc:
        # Generic error handling
        ...
    else:
        return something

What I’ve already tried:

• Playing with httpx.AsyncHTTPTransport, httpx.Limits, and httpx.Timeout: I tried adjusting these parameters to better manage the load, but this did not resolve the timeout issues.
• Windows as a potential cause: I’m wondering if this behavior could be related to my operating system (Windows), although I’m unsure what impact this might have (I'm on Python 3.12 and httpx 0.27.2 by the way).
• Retry and backoff: I could implement a retry or exponential backoff mechanism, but this would only treat the symptom without addressing the root cause of the problem.
• Suitability of AsyncClient: Given that each domain is queried no more than 4 times, I’m questioning whether AsyncClient is actually well-suited for this use case. It might not offer significant advantages in this context.
• I changed my code to use asyncio.Semaphore. It seems it improved the issue a lot - but it remains nonetheless. I thought that setting max_connections was nearly equivalent to using Semaphore - but it seems that's not the case. I'd very appreciate some information on that point.

async with self.semaphore:
    response = await self.client.get(url)

Potential causes I’m considering:

• System limits: Could Windows be limiting the number of simultaneous connections? Should I adjust some system-level settings?
• HTTP keep-alive or network congestion: Could the issue be related to how httpx manages connections or keeps them open?

I would greatly appreciate any suggestions on how to diagnose and fix this issue. Are there any specific configurations I should explore, or advice on how to better handle this load with httpx.AsyncClient?

Thanks for your help!

[loic-bellinger]

I wasn't entirely convinced about the efficiency of using a single AsyncClient to test a large number of different domains. Since I’m only making 4 requests per domain (that may change though), I did some more tests and refactored my code to create a new AsyncClient for each domain, and limited the number of concurrent requests to 1000 with a semaphore. The limits parameters of the clients are httpx.Limits(max_connections=4, max_keepalive_connections=4)

Here is an excerpt from the updated code, where the semaphore is initialized when the class is instantiated:

self.semaphore = asyncio.Semaphore(1000)

Next, here is the check_multiple_domains method:

async def check_multiple_domains(self, domains: pl.Series) -> list[list[DomainCheckResult]]:
    tasks = (self.check_single_domain(domain) for domain in domains)
    return await asyncio.gather(*tasks)

In check_single_domain, I create a new AsyncClient for each domain:

async def check_single_domain(self, domain: str) -> list[DomainCheckResult]:
    async with httpx.AsyncClient(
            follow_redirects=True,
            verify=False,
            headers=self.HEADERS,
            timeout=self.TIMEOUT,
            limits=self.LIMITS
    ) as client:
        tasks = (
            self._get(domain, client, f"{protocol}://{subdomain}{domain}")
            for protocol in self.PROTOCOLS
            for subdomain in self.SUBDOMAINS
        )
        return await asyncio.gather(*tasks)

Finally, here is the _get method that use the semaphore and performs the HTTP requests:

async def _get(self, domain: str, client: httpx.AsyncClient, url: str) -> DomainCheckResult:
    try:
        async with self.semaphore:
            response = await client.get(url)
            response.raise_for_status()
    except:
        # Exception handling

It seems that does improve things a little and the code seems to run faster. I would be happy to hear your feedback on this approach.

[loic-bellinger]

The errors I frequently face are: httpx.ConnectTimeout (timed out while connecting to the host) and httpx.ConnectError (failed to establish a connection). I'm mostly using the following timeout settings:

httpx.Timeout(connect=30.0, read=30.0, write=30.0, pool=None)

For connection limits, I've experimented with multiple configurations. E.g:

httpx.Limits(max_connections=100)
httpx.Limits(max_connections=None, max_keepalive_connections=100)

I also attempted to mitigate the issue by adding a transport with retries (httpx.AsyncHTTPTransport(retries=2)). However, this approach did not seem to improve the situation significantly. On this matter, I'm wondering: if I'm using a custom AsyncHTTPTransport (for retries), do I need to discard the limits parameter in the AsyncClient instantiation and pass it to the transport instead? My understanding is that the limits specified in AsyncHTTPTransport might override the limits passed directly to the AsyncClient.

So instead of doing

async with httpx.AsyncClient(
    follow_redirects=True,
    headers=self.HEADERS,
    timeout=self.TIMEOUT,
    limits=self.LIMITS,
)

I should do

async with httpx.AsyncClient(
    follow_redirects=True,
    headers=self.HEADERS,
    timeout=self.TIMEOUT,
    transport=httpx.AsyncHTTPTransport(retries=1, limits=self.LIMITS)
)

Finally, I'm using verify=False in my AsyncClient parameters.

[tomchristie]

I'm not sure @loic-bellinger?

If you're getting ConnectTimeout then perhaps you're running into ISP-level throttling?
(Or else just hitting servers that aren't responding.)

max_connections=100

I guess you might want to try bumping that up to 1000, since that's what you're using with your semaphore-limiting case.

Something I'd really like us to have would be an AiohttpTransport() class that bridges to using aiohttp for sending requests. It'd be really helpful to have an alternative backend implementation to help us triangulate in on fiddly networking issues.

[loic-bellinger]

So I rewrote my code to use aiohttp, and it seems to provide more detailed information about DNS-related issues. I discovered that many of my requests were failing due to "Timeout while contacting DNS servers" errors, especially when making a large number of requests in a short time. Interestingly, the number of these errors seems to decrease when I slow down the rate of my requests.

This leads me to believe that the issue might be related to DNS server rate-limiting or some kind of policy on their end. I’ll continue debugging and investigating to confirm whether that’s the case or if there might be other factors at play.

[loic-bellinger]

I continued investigating. After ruling out errors potentially caused by the operating system, I can confirm that it was indeed a rate-limiting policy imposed by the DNS servers. Aiohttp allows specifying a list of nameservers (via AsyncResolver passed to TCPConnector), which helps bypass the machine's DNS resolution settings and distribute the load across multiple servers.

This experience made me realize that httpx could improve by offering clearer error messages, especially for DNS-related issues, and by allowing users to specify custom DNS servers for resolution.