URL.host returns Punycode instead of Unicode for some URLs

[loic-bellinger]

Description

The URL.host property does not decode IDNA hostnames into Unicode, which contradicts the specification. According to the httpx documentation, the host should always be returned as a string, normalized to lowercase, with IDNA hosts decoded into Unicode.

Step to reproduce

from urllib.parse import urlparse
from httpx import URL

test_url = "https://www.égalité-femmes-hommes.gouv.fr"
print(URL(test_url).host)  # Expected: "www.égalité-femmes-hommes.gouv.fr", but returns: "www.xn--galit-femmes-hommes-9ybf.gouv.fr"
print(urlparse(test_url).hostname)  # returns: "www.égalité-femmes-hommes.gouv.fr". idna.decode() also returns this.

Expected behavior

The URL.host property should return the Unicode version of the host, in this case: www.égalité-femmes-hommes.gouv.fr.

Actual behavior

The URL.host property returns the Punycode-encoded version of the host: www.xn--galit-femmes-hommes-9ybf.gouv.fr.

Potential fix
It seems the issue arises in this part of the httpx code:

@property
def host(self) -> str: 
    host: str = self._uri_reference.host

    if host.startswith("xn--"):
        host = idna.decode(host)

    return host

The use of startswith("xn--") checks only for Punycode-encoded hosts that begin with this prefix. However, it should handle cases where IDNA encoding is used more comprehensively.

Replacing host.startswith("xn--") with something like if "xn--" in host might handle a broader set of cases?

Environment

httpx version: 0.27.2
Python version: 3.12.x
OS: Linux/Windows

[tomchristie]

The URL.host property should return the Unicode version of the host

I don't think the documentation currently say very much at all about the subtleties of the URL parameters, or if the user should expect the .host parameter to represent the IDNA-normalised ascii host that's connected to or the unicode string?

(Yes it should - there's quite an involved set of documentation work around the details here)

[loic-bellinger]

Docstring states that URL.host returns "The URL host as a string. Always normalized to lowercase, with IDNA hosts decoded into unicode".

I might need to provide a bit more context. I believe it's important to clarify whether .host returns Unicode, as opposed to non-Unicode, because this can lead to issues, notably when interpreting logs. For instance, if I'm testing the availability of a list of domains and subdomains, and I allow redirects, it makes sense to log response.url.host. This allows for comparisons with the original host (particularly if we don't have access to the original request.url.host). Additionally, it's easier for a human to read.