Configure FORWARD/TUNNEL mode for proxies.

[karpetrosyan]

Regarding the proxies, i have two questions.

• Why our create_connection method looks like this?

def create_connection(self, origin: Origin) -> ConnectionInterface:
    if origin.scheme == b"http":
        return ForwardHTTPConnection(
            proxy_origin=self._proxy_url.origin,
            proxy_headers=self._proxy_headers,
            remote_origin=origin,
            keepalive_expiry=self._keepalive_expiry,
            network_backend=self._network_backend,
        )
    return TunnelHTTPConnection(
        proxy_origin=self._proxy_url.origin,
        proxy_headers=self._proxy_headers,
        remote_origin=origin,
        ssl_context=self._ssl_context,
        keepalive_expiry=self._keepalive_expiry,
        http1=self._http1,
        http2=self._http2,
        network_backend=self._network_backend,
    )

• What if we want to forward HTTPS requests?

When using mitmproxy I must perform some magic in the "httpcore" source code to enable HTTPS forwarding because I can't use tunneling, which will attempt to upgrade the stream to tls, when the proxy server for analyzing the HTTPS traffic is running on localhost.

Of course, by adding a self-signed certificate to my mitmproxy, I can fix these issues, but why can't we just make it configurable?

[karpetrosyan]

This is how to reproduce the problem.

$ mitmweb
Web server listening at http://127.0.0.1:8081/

import httpcore

pool = httpcore.HTTPProxy(proxy_url='http://127.0.0.1:8080')
resp = pool.request("GET", "https://www.google.com")
print(resp)
# tunneling: httpcore.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:992)
# forwarding: <Response [200]>

[tomchristie]

Okay, so let's talk this through so we can get the documentation and the API correct...

Our existing behaviour is to forward requests to HTTP endpoints (the content of these is visible to the proxy) and to tunnel requests to requests to HTTPS endpoints. (the content of these is not visible to the proxy).

Some proxy configuration such as mitmproxy require all requests to be forwarded. The end-user implication is that you need to be aware that in this configuration the proxy has visibility onto the HTTPS content.

1. Does it seem to you that I've described all that correctly?
2. Would a use_forwarding_for_https=<bool> flag be a sufficent API for supporting this use-case, or do we need to be able to support both force-forwarding and force-tunneling?

Related comment: encode/httpx#1783 (comment)

[karpetrosyan]

1. Yes
2. HTTP proxy servers do not follow any rules regarding whether they should support forwarding, tunneling, or both; they simply implement what they want. I believe we should support both cases because our end users may be able to work with a proxy server that only requires tunneling.
   We might consider supporting both cases or not if it was difficult to implement, but in the current case, there is no difference between supporting one case and supporting both cases; if we can easily support both cases and give our end-user the option to use httpcore or httpx with his extraordinary proxy in the future, we should probably do it.

[T-256]

they simply implement what they want.

That's absolutely true, I encountered HTTP proxy servers those was accepting only tunneled connections (using CONNECT method). then even for HTTP in HTTP(S) we should able to force it to tunneling.

[T-256]

2. Would a use_forwarding_for_https=<bool> flag be a sufficent API for supporting this use-case, or do we need to be able to support both force-forwarding and force-tunneling?

For API if you are going to choose first, I suggest add use_tunneling_for_http=<bool> too.
but if you are going for second one, there is not forcing for all scenarios. forwarding is default for http destinations as well as tunneling is default for https destinations. So we can have: proxy_mode: "FORWARD_HTTPS" | "TUNNEL_HTTP".

Also another option that may reduce the confusion is using parameters with default values:

httpcore.HTTPProxy(
    http_dest: httpcore.PROXY_MODE = httpcore.PROXY_MODE.FORWARD,
    https_dest: httpcore.PROXY_MODE = httpcore.PROXY_MODE.TUNNEL,
)

(names can be improved)

[karpetrosyan]

I believe we should add something like "proxy_mode," which was removed from httpx in encode/httpx#1795.