Release 2.9.900 (#150)

2.9.900 (2024-09-24)
====================

- Fixed a rare issue where HTTPS record is misinterpreted, thus leading
to a missed preemptive HTTP/3 negotiation.
- Restored support for older-and-deprecated ``PySocks`` if installed and
``python-socks`` is absent for synchronous support of SOCKS proxies.
- Added support for HTTP Trailers across HTTP/1, HTTP/2 and HTTP/3
responses. We added the property ``trailers`` in ``HTTPResponse`` to
reflect that.
- Fixed unclosed resource warning for socket created in asynchronous
mode.
- Added support for Upgrading to HTTP/2 (If coming from HTTP/1) via
Alt-Svc. Whether it's h2c (http/2 over cleartext) or h2.
- Largely improve download speed on the QUIC layer by increasing
automatically the blocksize to the largest value allowed on UDP (value
taken from sysconf).
- Fixed the test suite outcome if no support for HTTP/3 exist in current
environment.

CHANGES.rst

@@ -1,3 +1,14 @@
+2.9.900 (2024-09-24)
+====================
+
+- Fixed a rare issue where HTTPS record is misinterpreted, thus leading to a missed preemptive HTTP/3 negotiation.
+- Restored support for older-and-deprecated ``PySocks`` if installed and ``python-socks`` is absent for synchronous support of SOCKS proxies.
+- Added support for HTTP Trailers across HTTP/1, HTTP/2 and HTTP/3 responses. We added the property ``trailers`` in ``HTTPResponse`` to reflect that.
+- Fixed unclosed resource warning for socket created in asynchronous mode.
+- Added support for Upgrading to HTTP/2 (If coming from HTTP/1) via Alt-Svc. Whether it's h2c (http/2 over cleartext) or h2.
+- Largely improve download speed on the QUIC layer by increasing automatically the blocksize to the largest value allowed on UDP (value taken from sysconf).
+- Fixed the test suite outcome if no support for HTTP/3 exist in current environment.
+
 2.8.907 (2024-08-20)
 ====================
 

README.md

@@ -30,10 +30,12 @@
 - Support for Python/PyPy 3.7+, no compromise.
 - HTTP/1.1, HTTP/2 and HTTP/3 support.
 - Proxy support for HTTP and SOCKS.
+- Post-Quantum Security with QUIC.
 - Detailed connection inspection.
 - HTTP/2 with prior knowledge.
 - Multiplexed connection.
 - Mirrored Sync & Async.
+- Trailer Headers.
 - Amazingly Fast.
 
 urllib3.future is powerful and easy to use:
@@ -210,6 +212,9 @@ guaranteed by upstream is also carefully watched down there. See the CI/pipeline
 
 In addition to that, we enforced key integration tests to watch how urllib3-future act with some critical projects.
 
+Top-priorities issues are those impacting users with the "shadowing" part. Meaning, if a user is suffering
+an error or something that ends up causing an undesirable outcome from a third-party library that leverage urllib3.
+
 - **OS Package Managers**
 
 Fellow OS package maintainers, you cannot _just_ build and ship this package to your package registry.

docker-compose.win.yaml

@@ -1,6 +1,6 @@
 services:
   proxy:
-    image: traefik:v3.1-windowsservercore-ltsc2022
+    image: traefik:v3.1.4-windowsservercore-ltsc2022
     restart: unless-stopped
     healthcheck:
       test: [ "CMD", "traefik" ,"healthcheck", "--ping" ]

docker-compose.yaml

@@ -1,6 +1,6 @@
 services:
   proxy:
-    image: traefik:v3.1
+    image: traefik:v3.1.4
     restart: unless-stopped
     healthcheck:
       test: [ "CMD", "traefik" ,"healthcheck", "--ping" ]
@@ -61,7 +61,7 @@ services:
       - --log.level=INFO
 
   httpbin:
-    image: mccutchen/go-httpbin:v2.14.0
+    image: mccutchen/go-httpbin:v2.15.0
     restart: unless-stopped
     depends_on:
       proxy:

docs/advanced-usage.rst

@@ -1493,3 +1493,27 @@ Here is a simple example::
     with urllib3.PoolManager(disabled_svn={urllib3.HttpVersion.h11}) as pm:
         r = pm.urlopen("GET", "http://my-internal.svc.local/")
 
+
+HTTP Trailers
+-------------
+
+.. note:: Available since version 2.9+
+
+HTTP response may contain one or several trailer headers. Those special headers are received
+after the reception of the body. Before this, those headers were unreachable and dropped silently.
+
+Quoted from Mozilla MDN: "The Trailer response header allows the sender to include additional fields
+at the end of chunked messages in order to supply metadata that might be dynamically generated while the
+message body is sent, such as a message integrity check, digital signature, or post-processing status."
+
+Here is a simple example::
+
+    import urllib3
+
+    with urllib3.PoolManager() as pm:
+        r = pm.urlopen("GET", "https://example.test")
+        print(r.trailers)
+
+.. note:: The property ``trailers`` return either ``None`` or a fully constructed ``HTTPHeaderDict``.
+
+.. warning:: ``None`` means we did not receive trailer headers (yet). If ``preload_content`` is set to False, you will need to consume the entire body before reaching the ``trailers`` property.

noxfile.py

@@ -171,7 +171,7 @@ def traefik_boot(session: nox.Session) -> typing.Generator[None, None, None]:
                     ),
                     timeout=1.0,
                 )
-            except (HTTPError, URLError, RemoteDisconnected) as e:
+            except (HTTPError, URLError, RemoteDisconnected, TimeoutError) as e:
                 i += 1
                 time.sleep(1)
                 session.log(f"Waiting for the Traefik server: {e}...")

setup.cfg

@@ -2,3 +2,5 @@
 ignore = E501, E203, W503, W504, E721
 exclude=./docs/conf.py
 max-line-length=99
+per-file-ignores =
+    src/urllib3/contrib/socks.py:F811

src/urllib3/_async/response.py

@@ -196,6 +196,20 @@ async def drain_conn(self) -> None:  # type: ignore[override]
         except (HTTPError, OSError, BaseSSLError):
             pass
 
+    @property
+    def trailers(self) -> HTTPHeaderDict | None:
+        """
+        Retrieve post-response (trailing headers) if any.
+        This WILL return None if no HTTP Trailer Headers have been received.
+        """
+        if self._fp is None:
+            return None
+
+        if hasattr(self._fp, "trailers"):
+            return self._fp.trailers
+
+        return None
+
     async def json(self) -> typing.Any:
         """
         Parses the body of the HTTP response as JSON.

src/urllib3/_constant.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import socket
 import typing
 from enum import IntEnum
 
@@ -219,9 +220,25 @@ def __new__(
 
 # Default value for `blocksize` - a new parameter introduced to
 # http.client.HTTPConnection & http.client.HTTPSConnection in Python 3.7
-# The maximum TCP packet size is 65535 octets. But Python seems to buffer packets, so
-# passing a "very-high" value does improve responsiveness.
-DEFAULT_BLOCKSIZE: int = 65535 * 2
+# The maximum TCP packet size is 65535 octets. But OSes can have a recv buffer greater than 65k, so
+# passing the highest value does improve responsiveness.
+# udp usually is set to 212992
+# and tcp usually is set to 131072 (16384 * 8) or (65535 * 2)
+try:
+    # dynamically retrieve the kernel rcvbuf size.
+    stream = socket.socket(type=socket.SOCK_STREAM)
+    dgram = socket.socket(type=socket.SOCK_DGRAM)
+
+    DEFAULT_BLOCKSIZE: int = stream.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)
+    UDP_DEFAULT_BLOCKSIZE: int = dgram.getsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF)
+
+    stream.close()
+    dgram.close()
+except OSError:
+    DEFAULT_BLOCKSIZE = 131072
+    UDP_DEFAULT_BLOCKSIZE = 212992
+
+TCP_DEFAULT_BLOCKSIZE: int = DEFAULT_BLOCKSIZE
 
 # Mozilla TLS recommendations for ciphers
 # General-purpose servers with a variety of clients, recommended for almost all systems.

src/urllib3/_version.py

@@ -1,4 +1,4 @@
 # This file is protected via CODEOWNERS
 from __future__ import annotations
 
-__version__ = "2.8.907"
+__version__ = "2.9.900"

src/urllib3/backend/_async/_base.py

@@ -12,7 +12,8 @@ class AsyncLowLevelResponse:
     basic response object. So that we don't have to change urllib3 tested behaviors."""
 
     __internal_read_st: typing.Callable[
-        [int | None, int | None], typing.Awaitable[tuple[bytes, bool]]
+        [int | None, int | None],
+        typing.Awaitable[tuple[bytes, bool, HTTPHeaderDict | None]],
     ] | None
 
     def __init__(
@@ -23,7 +24,8 @@ def __init__(
         reason: str,
         headers: HTTPHeaderDict,
         body: typing.Callable[
-            [int | None, int | None], typing.Awaitable[tuple[bytes, bool]]
+            [int | None, int | None],
+            typing.Awaitable[tuple[bytes, bool, HTTPHeaderDict | None]],
         ]
         | None,
         *,
@@ -71,6 +73,8 @@ def __init__(
         self.__buffer_excess: bytes = b""
         self.__promise: ResponsePromise | None = None
 
+        self.trailers: HTTPHeaderDict | None = None
+
     @property
     def fp(self) -> typing.NoReturn:
         raise RuntimeError(
@@ -110,7 +114,9 @@ async def read(self, __size: int | None = None) -> bytes:
             return b""  # Defensive: This is unreachable, this case is already covered higher in the stack.
 
         if self._eot is False:
-            data, self._eot = await self.__internal_read_st(__size, self._stream_id)
+            data, self._eot, self.trailers = await self.__internal_read_st(
+                __size, self._stream_id
+            )
 
             # that's awkward, but rather no choice. the state machine
             # consume and render event regardless of your amt !