[upd] httpx v0.21.2 --> v0.23.3 & httpx-socks v0.7.2 --> v0.7.5

[return42]

Dependabot alerts but did not update the httpx dependencies.

@dalf what do you think, can upgrade httpx packages?

I test this PR on my instance https://darmarit.org/searx/ .. so far I could not detect any issues.

Related: https://github.com/searxng/searxng/security/dependabot/4

[dalf]

Most probably, this code needs to be updated:

searxng/searx/network/client.py

Lines 63 to 126 in 2761147

	class AsyncHTTPTransportNoHttp(httpx.AsyncHTTPTransport):
	"""Block HTTP request"""
	async def handle_async_request(self, request):
	raise httpx.UnsupportedProtocol('HTTP protocol is disabled')
	class AsyncProxyTransportFixed(AsyncProxyTransport):
	"""Fix httpx_socks.AsyncProxyTransport
	Map python_socks exceptions to httpx.ProxyError exceptions
	"""
	async def handle_async_request(self, request):
	try:
	return await super().handle_async_request(request)
	except ProxyConnectionError as e:
	raise httpx.ProxyError("ProxyConnectionError: " + e.strerror, request=request) from e
	except ProxyTimeoutError as e:
	raise httpx.ProxyError("ProxyTimeoutError: " + e.args[0], request=request) from e
	except ProxyError as e:
	raise httpx.ProxyError("ProxyError: " + e.args[0], request=request) from e
	def get_transport_for_socks_proxy(verify, http2, local_address, proxy_url, limit, retries):
	# support socks5h (requests compatibility):
	# https://requests.readthedocs.io/en/master/user/advanced/#socks
	# socks5:// hostname is resolved on client side
	# socks5h:// hostname is resolved on proxy side
	rdns = False
	socks5h = 'socks5h://'
	if proxy_url.startswith(socks5h):
	proxy_url = 'socks5://' + proxy_url[len(socks5h) :]
	rdns = True
	proxy_type, proxy_host, proxy_port, proxy_username, proxy_password = parse_proxy_url(proxy_url)
	verify = get_sslcontexts(proxy_url, None, verify, True, http2) if verify is True else verify
	return AsyncProxyTransportFixed(
	proxy_type=proxy_type,
	proxy_host=proxy_host,
	proxy_port=proxy_port,
	username=proxy_username,
	password=proxy_password,
	rdns=rdns,
	loop=get_loop(),
	verify=verify,
	http2=http2,
	local_address=local_address,
	limits=limit,
	retries=retries,
	)
	def get_transport(verify, http2, local_address, proxy_url, limit, retries):
	verify = get_sslcontexts(None, None, verify, True, http2) if verify is True else verify
	return httpx.AsyncHTTPTransport(
	# pylint: disable=protected-access
	verify=verify,
	http2=http2,
	limits=limit,
	proxy=httpx._config.Proxy(proxy_url) if proxy_url else None,
	local_address=local_address,
	retries=retries,
	)

Without diving into the httpx code, a simple check is to upgrade, start a with mitmproxy for example and configure SearXNG to use it. If the requests are sent through the proxy that's good. Note that socks4, socks5, http and https proxies need to be tested. And then other parameters need to be checked.

I was hoping the version 0.24 to be released sooner.

I will have a look at it tomorrow.

[return42]

I was hoping the version 0.24 to be released sooner.

From my side there is no need to hurry ... should we wait? --> encode/httpx#2534 (comment)

Note that socks4, socks5, http and https proxies need to be tested. And then other parameters need to be checked. .. I will have a look at it tomorrow.

Thanks a lot, I would be overwhelmed with this for now / wouldn't know how to assemble a setup for a test :-o

[return42]

copy @dalf's comment from return42@d86ec70#commitcomment-107513541 to here ..

The important thing to test is the proxy configuration (http and socks). When I have tried with mitmproxy and verify: ~/.mitmproxy/mitmproxy-ca-cert.cer I had an error. It worked before.

[dalf]

Another note: in new version httpx supports socks and http proxy but not https proxy.
For https proxy, httpx-socks is still required.

See:

• Implement SOCKS v4, v5 Proxy encode/httpx#203
• HTTPS proxies support encode/httpx#1434 (comment)
• SSL + NordVPN Proxy  encode/httpx#2632

[mrpaulblack]

Closing in favor of #2644; Closing.