⚗️ Try compatible fork Niquests to supercharge HTTPie

httpie · Jan 7, 2024 · 454fed7 · 454fed7
1 parent e52a60e
commit 454fed7
Show file tree

Hide file tree

Showing 52 changed files with 1,244 additions and 309 deletions.
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
@@ -25,8 +25,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, macos-latest, windows-latest]
-        python-version: [3.7, 3.8, 3.9, "3.10"]
-        pyopenssl: [0, 1]
+        python-version: [3.7, 3.8, 3.9, "3.10", "3.11", "3.12"]
     runs-on: ${{ matrix.os }}
     steps:
       - uses: actions/checkout@v3
@@ -39,12 +38,8 @@ jobs:
           python -m pip install --upgrade pip wheel
           python -m pip install --upgrade '.[dev]'
           python -m pytest --verbose ./httpie ./tests
-        env:
-          HTTPIE_TEST_WITH_PYOPENSSL: ${{ matrix.pyopenssl }}
       - name: Linux & Mac setup
         if: matrix.os != 'windows-latest'
         run: |
           make install
           make test
-        env:
-          HTTPIE_TEST_WITH_PYOPENSSL: ${{ matrix.pyopenssl }}
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,9 +3,21 @@
 This document records all notable changes to [HTTPie](https://httpie.io).
 This project adheres to [Semantic Versioning](https://semver.org/).
 
-## [3.3.0-dev](https://github.com/httpie/cli/compare/3.2.2...master) (unreleased)
-
-- Make it possible to [unset](https://httpie.io/docs/cli/default-request-headers) the `User-Agent`, `Accept-Encoding`, and `Host` request headers. ([#1502](https://github.com/httpie/cli/issues/1502))
+## [4.0.0.b1](https://github.com/httpie/cli/compare/3.2.2...master) (unreleased)
+
+- Make it possible to [unset](https://httpie.io/docs/cli/default-request-headers) the `User-Agent`, and `Accept-Encoding` headers. ([#1502](https://github.com/httpie/cli/issues/1502))
+- Dependency on requests was changed in favor of compatible niquests. ([#1531](https://github.com/httpie/cli/pull/1531))
+- Added support for HTTP/2, and HTTP/3 protocols. ([#523](https://github.com/httpie/cli/issues/523)) ([#692](https://github.com/httpie/cli/issues/692)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Added request metadata for the TLS certificate, negotiated version with cipher, and his revocation status and the remote peer IP address. ([#1495](https://github.com/httpie/cli/issues/1495)) ([#1023](https://github.com/httpie/cli/issues/1023)) ([#826](https://github.com/httpie/cli/issues/826)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Added support to load the operating system trust store for the peer certificate validation. ([#480](https://github.com/httpie/cli/issues/480)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. ([#1023](https://github.com/httpie/cli/issues/1023)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Added support for using alternative DNS resolver using `--resolver`. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. ([#99](https://github.com/httpie/cli/issues/99)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Added support for binding to a specific network adapter with `--interface`. ([#1422](https://github.com/httpie/cli/issues/1422)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Added support for forcing either IPv4 or IPv6 to reach the remote HTTP server with `-6` or `-4`. ([#94](https://github.com/httpie/cli/issues/94)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Removed support for pyopenssl. ([#1531](https://github.com/httpie/cli/pull/1531))
+- Dropped dependency on `requests_toolbelt` in favor of directly including `MultipartEncoder` into HTTPie due to its direct dependency to requests. ([#1531](https://github.com/httpie/cli/pull/1531))
+- Dropped dependency on `multidict` in favor of implementing an internal one due to often missing pre-built wheels. ([#1522](https://github.com/httpie/cli/issues/1522)) ([#1531](https://github.com/httpie/cli/pull/1531))
+- Fixed the case when multiple headers where concatenated in the response output. ([#1413](https://github.com/httpie/cli/issues/1413)) ([#1531](https://github.com/httpie/cli/pull/1531))
 
 ## [3.2.2](https://github.com/httpie/cli/compare/3.2.1...3.2.2) (2022-05-19)
 

diff --git a/docs/README.md b/docs/README.md
@@ -1562,9 +1562,9 @@ be printed via several options:
 |---------------------------:|----------------------------------------------------------------------------------------------------|
 |            `--headers, -h` | Only the response headers are printed                                                              |
 |               `--body, -b` | Only the response body is printed                                                                  |
-|               `--meta, -m` | Only the [response metadata](#response-meta) is printed                                            |
+|               `--meta, -m` | Only the [request, response metadata](#response-meta) are printed                                  |
 |            `--verbose, -v` | Print the whole HTTP exchange (request and response). This option also enables `--all` (see below) |
-| `--verbose --verbose, -vv` | Just like `-v`, but also include the response metadata.                                            |
+| `--verbose --verbose, -vv` | Just like `-v`, but also include the request, and response metadata.                               |
 |              `--print, -p` | Selects parts of the HTTP exchange                                                                 |
 |              `--quiet, -q` | Don’t print anything to `stdout` and `stderr`                                                      |
 
@@ -1573,13 +1573,13 @@ be printed via several options:
 All the other [output options](#output-options) are under the hood just shortcuts for the more powerful `--print, -p`.
 It accepts a string of characters each of which represents a specific part of the HTTP exchange:
 
-| Character | Stands for                      |
-|----------:|---------------------------------|
-|       `H` | request headers                 |
-|       `B` | request body                    |
-|       `h` | response headers                |
-|       `b` | response body                   |
-|       `m` | [response meta](#response-meta) |
+| Character | Stands for                               |
+|----------:|------------------------------------------|
+|       `H` | request headers                          |
+|       `B` | request body                             |
+|       `h` | response headers                         |
+|       `b` | response body                            |
+|       `m` | [request, response meta](#response-meta) |
 
 Print request and response headers:
 
@@ -1592,27 +1592,49 @@ $ http --print=Hh PUT pie.dev/put hello=world
 The response metadata section currently includes the total time elapsed. It’s the number of seconds between opening the network connection and downloading the last byte of response the body.
 
 
-To _only_ show the response metadata, use `--meta, -m` (analogically to `--headers, -h` and `--body, -b`):
+To _only_ show the request, and response metadata, use `--meta, -m` (analogically to `--headers, -h` and `--body, -b`):
 
 ```bash
 $ http --meta pie.dev/delay/1
 ```
 
 ```console
-Elapsed time: 1.099171542s
+Connected to: 2a06:98c1:3120::2 port 443
+Connection secured using: TLSv1.3 with AES-256-GCM-SHA384
+Server certificate: commonName="pie.dev"; DNS="*.pie.dev"; DNS="pie.dev"
+Certificate validity: "Nov 11 01:14:24 2023 UTC" to "Feb 09 01:14:23 2024 UTC"
+Issuer: countryName="US"; organizationName="Let's Encrypt"; commonName="E1"
+Revocation status: Good
+
+Elapsed DNS: 0.11338s
+Elapsed established connection: 3.8e-05s
+Elapsed TLS handshake: 0.057503s
+Elapsed emitting request: 0.000275s
+Elapsed time: 0.292854214s
 ```
 
 The [extra verbose `-vv` output](#extra-verbose-output) includes the meta section by default. You can also show it in combination with other parts of the exchange via [`--print=m`](#what-parts-of-the-http-exchange-should-be-printed). For example, here we print it together with the response headers:
 
 ```bash
-$ http --print=hm pie.dev/get
+$ https --print=hm pie.dev/get
 ```
 
 ```http
-HTTP/1.1 200 OK
+Connected to: 2a06:98c1:3120::2 port 443
+Connection secured using: TLSv1.3 with AES-256-GCM-SHA384
+Server certificate: commonName="pie.dev"; DNS="*.pie.dev"; DNS="pie.dev"
+Certificate validity: "Nov 11 01:14:24 2023 UTC" to "Feb 09 01:14:23 2024 UTC"
+Issuer: countryName="US"; organizationName="Let's Encrypt"; commonName="E1"
+Revocation status: Good
+
+HTTP/2 200 OK
 Content-Type: application/json
 
-Elapsed time: 0.077538375s
+Elapsed DNS: 0.11338s
+Elapsed established connection: 3.8e-05s
+Elapsed TLS handshake: 0.057503s
+Elapsed emitting request: 0.000275s
+Elapsed time: 0.292854214s
 ```
 
 
@@ -1626,19 +1648,19 @@ If you [use `--style` with one of the Pie themes](#colors-and-formatting), you
 `--verbose` can often be useful for debugging the request and generating documentation examples:
 
 ```bash
-$ http --verbose PUT pie.dev/put hello=world
-PUT /put HTTP/1.1
+$ https --verbose PUT pie.dev/put hello=world
+PUT /put HTTP/2
 Accept: application/json, */*;q=0.5
 Accept-Encoding: gzip, deflate
 Content-Type: application/json
 Host: pie.dev
-User-Agent: HTTPie/0.2.7dev
+User-Agent: HTTPie/4.0.0
 
 {
     "hello": "world"
 }
 
-HTTP/1.1 200 OK
+HTTP/2 200 OK
 Connection: keep-alive
 Content-Length: 477
 Content-Type: application/json
@@ -1652,10 +1674,10 @@ Server: gunicorn/0.13.4
 
 #### Extra verbose output
 
-If you run HTTPie with `-vv` or `--verbose --verbose`, then it would also display the [response metadata](#response-meta).
+If you run HTTPie with `-vv` or `--verbose --verbose`, then it would also display the [response and request metadata](#response-meta).
 
 ```bash
-# Just like the above, but with additional columns like the total elapsed time
+# Just like the above, but with additional columns like the total elapsed time, remote peer connection informations
 $ http -vv pie.dev/get
 ```
 
@@ -1833,6 +1855,73 @@ $ http --chunked pie.dev/post @files/data.xml
 $ cat files/data.xml | http --chunked pie.dev/post
 ```
 
+## Disable HTTP/2, or HTTP/3
+
+You can at your own discretion toggle on and off HTTP/2, or/and HTTP/3.
+
+```bash
+$ https --disable-http2 PUT pie.dev/put hello=world
+```
+
+```bash
+$ https --disable-http3 PUT pie.dev/put hello=world
+```
+
+## Force HTTP/3
+
+By opposition to the previous section, you can force the HTTP/3 negotiation.
+
+```bash
+$ https --http3 pie.dev/get
+```
+
+By default, HTTPie cannot negotiate HTTP/3 without a first HTTP/1.1, or HTTP/2 successful response unless the
+remote host specified a DNS HTTPS record that indicate its support.
+
+The remote server yield its support for HTTP/3 in the Alt-Svc header, if present HTTPie will issue
+the successive requests via HTTP/3. You may use that argument in case the remote peer does not support
+either HTTP/1.1 or HTTP/2.
+
+## Custom DNS resolver
+
+### Using DNS url
+
+You can specify one or many custom DNS resolvers using the `--resolver` flag. They will be tested in
+presented order to resolver given hostname.
+
+```bash
+$ https --resolver "doh+cloudflare://" pie.dev/get
+```
+
+To know more about DNS url and supported protocols, visit [Niquests documentation](https://niquests.readthedocs.io/en/stable/user/quickstart.html#dns-resolution).
+
+### Forcing hostname to resolve with a manual entry
+
+It is possible to fake DNS resolution using a virtual resolver. We'll make use of the `--resolver` flag
+using the `in-memory` provider.
+
+```bash
+$ https --resolver "in-memory://default/?hosts=pie.dev:10.10.4.1" pie.dev/get
+```
+
+In that example, `pie.dev` will resolve to `10.10.4.1`. The TLS HELLO / SNI will be set with host = `pie.dev`.
+
+## Attach to a specific network adapter
+
+In order to bind emitted request from a specific network adapter you can use the `--interface` flag.
+
+```bash
+$ https --interface 172.17.0.1 pie.dev/get
+```
+
+## Enforcing IPv4 or IPv6
+
+Since HTTPie 4, you may pass the flags `--ipv4, -4` or `--ipv6, -6` to enforce connecting to an IPv4 or IPv6 address.
+
+```bash
+$ https -4 pie.dev/get
+```
+
 ## Compressed request body
 
 You can use the `--compress, -x` flag to instruct HTTPie to use `Content-Encoding: deflate` and compress the request data:
@@ -2556,7 +2645,7 @@ HTTPie has the following community channels:
 
 Under the hood, HTTPie uses these two amazing libraries:
 
-- [Requests](https://requests.readthedocs.io/en/latest/) — Python HTTP library for humans
+- [Niquests](https://niquests.readthedocs.io/en/latest/) — Python HTTP library for humans
 - [Pygments](https://pygments.org/) — Python syntax highlighter
 
 #### HTTPie friends

diff --git a/docs/contributors/fetch.py b/docs/contributors/fetch.py
@@ -1,7 +1,7 @@
 """
 Generate the contributors database.
 
-FIXME: replace `requests` calls with the HTTPie API, when available.
+FIXME: replace `niquests` calls with the HTTPie API, when available.
 """
 import json
 import os
@@ -14,7 +14,7 @@
 from time import sleep
 from typing import Any, Dict, Optional, Set
 
-import requests
+import niquests
 
 FullNames = Set[str]
 GitHubLogins = Set[str]
@@ -197,10 +197,10 @@ def fetch(url: str, params: Optional[Dict[str, str]] = None) -> UserInfo:
     }
     for retry in range(1, 6):
         debug(f'[{retry}/5]', f'{url = }', f'{params = }')
-        with requests.get(url, params=params, headers=headers) as req:
+        with niquests.get(url, params=params, headers=headers) as req:
             try:
                 req.raise_for_status()
-            except requests.exceptions.HTTPError as exc:
+            except niquests.exceptions.HTTPError as exc:
                 if exc.response.status_code == 403:
                     # 403 Client Error: rate limit exceeded for url: ...
                     now = int(datetime.utcnow().timestamp())

diff --git a/httpie/__init__.py b/httpie/__init__.py
@@ -3,7 +3,7 @@
 
 """
 
-__version__ = '3.2.2'
-__date__ = '2022-05-06'
+__version__ = '4.0.0.b1'
+__date__ = '2024-01-01'
 __author__ = 'Jakub Roztocil'
 __licence__ = 'BSD'
diff --git a/httpie/adapters.py b/httpie/adapters.py
@@ -1,5 +1,5 @@
 from httpie.cli.dicts import HTTPHeadersDict
-from requests.adapters import HTTPAdapter
+from niquests.adapters import HTTPAdapter
 
 
 class HTTPieHTTPAdapter(HTTPAdapter):

diff --git a/httpie/cli/argparser.py b/httpie/cli/argparser.py
@@ -7,7 +7,7 @@
 from textwrap import dedent
 from urllib.parse import urlsplit
 
-from requests.utils import get_netrc_auth
+from niquests.utils import get_netrc_auth
 
 from .argtypes import (
     AuthCredentials, SSLCredentials, KeyValueArgType,

diff --git a/httpie/cli/definition.py b/httpie/cli/definition.py
@@ -726,6 +726,20 @@ def format_auth_help(auth_plugins_mapping, *, isolation_mode: bool = False):
 
     """,
 )
+network.add_argument(
+    '--ipv6',
+    '-6',
+    default=False,
+    action='store_true',
+    short_help='Force using a IPv6 address to reach the remote peer.'
+)
+network.add_argument(
+    '--ipv4',
+    '-4',
+    default=False,
+    action='store_true',
+    short_help='Force using a IPv4 address to reach the remote peer.'
+)
 network.add_argument(
     '--follow',
     '-F',
@@ -802,6 +816,52 @@ def format_auth_help(auth_plugins_mapping, *, isolation_mode: bool = False):
         'The Transfer-Encoding header is set to chunked.'
     )
 )
+network.add_argument(
+    "--disable-http2",
+    default=False,
+    action="store_true",
+    short_help="Disable the HTTP/2 protocol."
+)
+network.add_argument(
+    "--disable-http3",
+    default=False,
+    action="store_true",
+    short_help="Disable the HTTP/3 over QUIC protocol."
+)
+network.add_argument(
+    "--http3",
+    default=False,
+    dest="force_http3",
+    action="store_true",
+    short_help="Use the HTTP/3 protocol for the request.",
+    help="""
+    By default, HTTPie cannot negotiate HTTP/3 without a first HTTP/1.1, or HTTP/2 successful response unless the
+    remote host specified a DNS HTTPS record that indicate its support.
+
+    The remote server yield its support for HTTP/3 in the Alt-Svc header, if present HTTPie will issue
+    the successive requests via HTTP/3. You may use that argument in case the remote peer does not support
+    either HTTP/1.1 or HTTP/2.
+
+    """
+)
+network.add_argument(
+    "--resolver",
+    default=[],
+    action='append',
+    short_help="Specify a DNS resolver url to resolve hostname.",
+    help="""
+    By default, HTTPie use the system DNS through Python standard library.
+    You can specify an alternative DNS server to be used. (e.g. doh://cloudflare-dns.com or doh://google.dns).
+    You can specify multiple resolvers with different protocols. The environment
+    variable $NIQUESTS_DNS_URL is supported as well.
+
+    """
+)
+network.add_argument(
+    "--interface",
+    default=None,
+    short_help="Bind to a specific network interface.",
+)
 
 #######################################################################
 # SSL