Do we really need the proxy "mode" argument?

[tomchristie]

We currently have an argument on httpxProxy() that allows for explicitly forcing between forwarded HTTP requests and tunnelled HTTPS request, regardless of the actual target scheme.

So, normally...

# Requests to `http` URLs will be forwarded via "http://localhost:8080"
# Requests to `https` URLs will be tunnelled via "http://localhost:8080"
client = httpx.Client(proxies=httpx.Proxy("http://localhost:8080"))

However this can be overridden, for example...

# All requests will be tunnelled
client = httpx.Client(proxies=httpx.Proxy("http://localhost:8080", mode="TUNNEL_ONLY"))

But its not really clear to me if we actually want this at all.

• Is this even actually useful / meaningful?
• Does it actually work?
• Is anyone using it?

Looking through the requests documentation and API reference, I can't immediately see any equivalent support for it outside of digging into urllib3 directly, so I don't think that we need it in order to meet our aims of requests-parity.

I'd also assume that eg. tunnelling a request to an http URL just doesn't make sense at all. Why would we start an SSL connection if the end URL is HTTP?

Perhaps this is a corner bit of API that we've accidentally ended up with that we could drop?

Note: I'm asking this because while doing work on rejigging httpcore and thinking about SOCKs support it wasn't very clear to me how the mode flag could fit in neatly with that. If we don't have it, then there's a really simple map from http -> ForwardConnection, https -> TunnelConnection, socks4 -> SOCKS4Connection etc... but things just seem more grungy if there's a special casing for the http and https cases.

[SteveAlexander]

I wish I'd seen this before the 0.19 release!

client = httpx.Client(proxies=httpx.Proxy("http://localhost:8080", mode="TUNNEL_ONLY"))

This is exactly what I'm using.

The reason for using this is that I'm using a commercial proxying service (https://brightdata.com/), and I couldn't get https-https proxying working using the 0.18 version of httpx. I had it working previously in version 0.7.6, before the big refactor, but during the refactor things changed so that this did not work any more. (This was a discussion we had at the time around getting proxy response headers, rather than discarding them: #523. After the refactor, this approach no longer worked, but also https-https proxies using connect no longer worked, at least not with the service I'm using).

I could only get the proxy service working at all with 0.18 by specifying TUNNEL_ONLY and connecting to the proxy using HTTP, because #1434

Now it looks like with 0.19, I can no longer use this workaround.

Would you be willing to reinstate this feature until issue #523 is implemented?

(side note: I have a fork that I'm using that implements an option to merge proxy response headers with the regular response headers, which works for me. I was planning at some point soon to raise this as a discussion point for httpx / httpcore.
https://github.com/SteveAlexander/httpcore https://github.com/SteveAlexander/httpx )

[tomchristie]

Thanks for this Steve - some interesting stuff there.
I do have a better idea now how we might support making the headers from the connect visible, so perhaps that'll be the best route for us to go down. To be determined.
I'm on holiday this coming week, but this looks worth digging into and resolving one way or another.

[SteveAlexander]

I appreciate the reply, Tom.

I hope you have a great holiday, look forward to discussing after you're back.

[SteveAlexander]

btw, overall, as a "power user" of httpx and httpcore (and trio, I guess), I'm enjoying the design direction it's going in! Just tripping up on this rather niche use-case.