Expose server_hostname override for verification purposes

[rb-determined-ai]

It seems that all of the backends support a hostname argument to _open_tcp_stream, and that hostname is also used as the server_hostname for verification purposes in all cases.

It can be really useful to override the server_hostname in enterprise environments where you might be on the private side of a server that has a valid certificate (example.com), but which you can only reach via an internal address (10.1.2.3).

Is there any chance of seeing a feature where the server_hostname can be overridden manually? Ideally it would be nice to have it at the httpx level, even if it was through a custom transport.

[tomchristie]

So... is this the same as using the IP in the URL, but setting an explicit host header?

r = httpx.get("https://10.1.2.3/...", headers={"Host": "example.com"})

[rb-determined-ai]

That could work maybe. Would an explicit Host header override the hostname from the url in all cases?

[rb-determined-ai]

Thinking more about it, hijacking the Host header for tls verification purposes does seem a little bit like an unexpected side-effect. Most low-level TLS libraries I'm used to take a server_hostname or verification_hostname or certificate_hostname or something very explicitly, if you choose to override it.

[tomchristie]

Ah, okay I can see that there's two separate things going on here...

• Forcible seperation of "Use this IP" vs. "Use this hostname for the host header".
• Explicitly specifying "Use this hostname for the TLS verification".

Some useful context for us here might by gained by digging into if other Python HTTP libraries allow for this functionality, and if so, what API they provide onto it. (Notably urllib3/requests/urllib3,aiohttp.)

[rb-determined-ai]

Sure! We've done a little of this work for the libraries we've been using.

With requests, they offer a plugin-based strategy for overriding certain behaviors, which includes letting us configure how urllib3 is going to validate the certificate. See our HTTPAdapter class. I think this is roughly equivalent to the custom Transport experience in httpx.

We also use a less-well-known websocket framework called lomond in a few places. They expose the actual call to ssl_context.wrap_socket via a plugin interface. See our CustomSSLWebsocketSession class for an example.

This all works for us right now but we are in the process of a big inversion of control between our platform and the user code we run on our platform, so we are investigating ways to do things like asynchronous background uploads. It is pretty hard to do right with requests but httpx would make that super easy.

[bwelling]

For requests, HostHeaderSSLAdapter in requests_toolbelt (https://toolbelt.readthedocs.io/en/latest/adapters.html#hostheaderssladapter) also provides this functionality.

[rb-determined-ai]

@tomchristie would a PR be welcome? I think it seems fairly clear at the httpcore layer what would be done about this.