Make it possible to request a keyring provider: auto, disabled, `…

…import` or `subprocess`

Refactored `_get_index_url()` to get integration tests for the subprocess backend working.

Keyring support via the 'subprocess' provider can only retrieve a password, not a username-password combo. The username therefor MUST come from the URL.
If the URL obtained from the index does not contain a username then the username from a matching index is used. `_get_index_url()` does that matching.

The problem this refactoring solves is that the URL where a wheel or sdist can be downloaded from does not always start with the index url. Azure DevOps Artifacts Feeds are an example since it replaces the friendly name of the Feed with the GUID of the Feed. Causing `url.startswith(prefix)` to evaluate as `False`.

The new behaviour is to return the index which matches the netloc and has the longest common prefix of the `path` property of the value returned by `urllib.parse.urlsplit()`. The behaviour for resolving ties is unspecified.

.pre-commit-config.yaml

@@ -47,7 +47,7 @@ repos:
     additional_dependencies: [
         'keyring==23.0.1',
         'nox==2021.6.12',
-        'pytest==7.1.1',
+        'pytest',
         'types-docutils==0.18.3',
         'types-setuptools==57.4.14',
         'types-freezegun==1.1.9',

docs/html/topics/authentication.md

@@ -66,12 +66,118 @@ man pages][netrc-docs].
 ## Keyring Support
 
 pip supports loading credentials stored in your keyring using the
-{pypi}`keyring` library.
+{pypi}`keyring` library which can be enabled py passing `--keyring-provider`
+with a value of `auto`, `disabled`, `import` or `subprocess`. The default value
+is `auto`. `auto` will respect `--no-input` and not query keyring at all if that
+option is used. The `auto` provider will use the `import` provider if the
+`keyring` module can be imported. If that is not the case it will use the
+`subprocess` provider if a `keyring` executable can be found. Otherwise, the
+`disabled` provider will be used.
+
+### Configuring Pip
+Passing this as a command line argument will work, but is not how the majority
+of this feature's users will use it. They instead will want to overwrite the
+default of `disabled` in the global, user of site configuration file:
+```bash
+$ pip config set --global global.keyring-provider subprocess
+
+# A different user on the same system which has PYTHONPATH configured and and
+# wanting to use keyring installed that way could then run
+$ pip config set --user global.keyring-provider import
+
+# For a specific virtual environment you might want to use disable it again
+# because you will only be using PyPI and the private repo (and mirror)
+# requires 2FA with a keycard and a pincode
+$ pip config set --site global.index https://pypi.org/simple
+$ pip config set --site global.keyring-provider disabled
+
+# configuring it via environment variable is also possible
+$ export PIP_KEYRING_PROVIDER=disabled
+```
+
+### Installing and using the keyring python module
+
+Setting it to `import` tries to communicate with `keyring` by importing it
+and using its Python api.
 
 ```bash
-$ pip install keyring  # install keyring from PyPI
+# install keyring from PyPI
+$ pip install keyring --index-url https://pypi.org/simple
 $ echo "your-password" | keyring set pypi.company.com your-username
-$ pip install your-package --index-url https://pypi.company.com/
+$ pip install your-package --keyring-provider import --index-url https://pypi.company.com/
+```
+
+### Installing and using the keyring cli application
+
+Setting it to `subprocess` will look for a `keyring` executable on the PATH
+if one can be found that is different from the `keyring` installation `import`
+would be using.
+
+The cli requires a username, therefore you MUST put a username in the url.
+See the example below or the basic HTTP authentication section at the top of
+this page.
+
+```bash
+# install keyring from PyPI using pipx, which we assume if installed properly
+# you can also create a venv somewhere and add it to the PATH yourself instead
+$ pipx install keyring --index-url https://pypi.org/simple
+
+# install the keyring backend for Azure DevOps for example
+# VssSessionToken is the username you MUST use for this backend
+$ pipx inject keyring artifacts-keyring --index-url https://pypi.org/simple
+
+# or the one for Google Artifact Registry
+$ pipx inject keyring keyrings.google-artifactregistry-auth --index-url https://pypi.org/simple
+$ gcloud auth login
+
+$ pip install your-package --keyring-provider subprocess --index-url https://username@pypi.example.com/
+```
+
+### Here be dragons
+
+The `auto` provider is conservative and does not query keyring at all when
+`--no-input` is used because the keyring might require user interaction such as
+prompting the user on the console. Third party tools frequently call Pip for
+you and do indeed pass `--no-input` as they are well-behaved and don't have
+much information to work with. (Keyring does have an api to request a backend
+that does not require user input.) You have more information about your system,
+however!
+
+You can force keyring usage by requesting a keyring provider other than `auto`
+(or `disabled`). Leaving `import` and `subprocess`. You do this by passing
+`--keyring-provider import` or one of the following methods:
+
+```bash
+# via config file, possibly with --user, --global or --site
+$ pip config set global.keyring-provider subprocess
+# or via environment variable
+$ export PIP_KEYRING_PROVIDER=import
+```
+
+```{warning}
+Be careful when doing this since it could cause tools such as Pipx and Pipenv
+to appear to hang. They show their own progress indicator while hiding output
+from the subprocess in which they run Pip. You won't know whether the keyring
+backend is waiting the user input or not in such situations.
+```
+
+Pip is conservative and does not query keyring at all when `--no-input` is used
+because the keyring might require user interaction such as prompting the user
+on the console. You can force keyring usage by passing `--force-keyring` or one
+of the following:
+
+```bash
+# possibly with --user, --global or --site
+$ pip config set global.force-keyring true
+# or
+$ export PIP_FORCE_KEYRING=1
+```
+
+```{warning}
+Be careful when doing this since it could cause tools such as Pipx and Pipenv
+to appear to hang. They show their own progress indicator while hiding output
+from the subprocess in which they run Pip. You won't know whether the keyring
+backend is waiting the user input or not in such situations.
 ```
 
 Note that `keyring` (the Python package) needs to be installed separately from

news/8719.feature.rst

@@ -0,0 +1 @@
+Add ``--keyring-provider`` flag. See the Authentication page in the documentation for more info.

src/pip/_internal/cli/cmdoptions.py

@@ -244,6 +244,19 @@ class PipOption(Option):
     help="Disable prompting for input.",
 )
 
+keyring_provider: Callable[..., Option] = partial(
+    Option,
+    "--keyring-provider",
+    dest="keyring_provider",
+    choices=["auto", "disabled", "import", "subprocess"],
+    default="disabled",
+    help=(
+        "Enable the credential lookup via the keyring library if user input is allowed."
+        " Specify which mechanism to use [disabled, import, subprocess]."
+        " (default: disabled)"
+    ),
+)
+
 proxy: Callable[..., Option] = partial(
     Option,
     "--proxy",
@@ -1019,6 +1032,7 @@ def check_list_path_option(options: Values) -> None:
         quiet,
         log,
         no_input,
+        keyring_provider,
         proxy,
         retries,
         timeout,

src/pip/_internal/cli/req_command.py

@@ -151,6 +151,14 @@ def _build_session(
 
         # Determine if we can prompt the user for authentication or not
         session.auth.prompting = not options.no_input
+        session.auth.keyring_provider = options.keyring_provider
+        # We won't use keyring when --no-input is passed unless
+        # a specific provider is requested because it might require
+        # user interaction
+        session.auth.use_keyring = (
+            session.auth.prompting
+            or options.keyring_provider not in ["disabled", "auto"]
+        )
 
         return session