Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent - <> _ normalization makes actions/cache fail #10910

Closed
1 task done
Conchylicultor opened this issue Feb 15, 2022 · 4 comments
Closed
1 task done

Inconsistent - <> _ normalization makes actions/cache fail #10910

Conchylicultor opened this issue Feb 15, 2022 · 4 comments
Labels
resolution: duplicate Duplicate of an existing issue/PR

Comments

@Conchylicultor
Copy link

Conchylicultor commented Feb 15, 2022 •
edited
Loading

Description

I'm encountering another variation of #10757.

In my requirement .txt I install:

etils[array-types] @ git+git://github.com/google/etils

In my github action, when installing with pip install -e ., this work fine. However, when trying to cache in github action:

    - uses: actions/setup-python@v2
      with:
        python-version: 3.7

    # The cache is refreshed everyday.
    - uses: actions/cache@v2
      with:
        path: ${{ env.pythonLocation }}

    - run: pip install -e .  # <<< FAIL when restoring the cache

Raises:

  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line [27], in requires
    "%s has no such extra feature %r" % (self, ext)
pip._vendor.pkg_resources.UnknownExtra: etils 0.4.0 has no such extra feature 'array_types'

Somewhere, array-types got normalised to array_types, which makes the installation fail.

Expected behavior

pip should be internally consistent when normalizing - <> _ (e.g. - should be used every where).
Currently pip sometimes normalize - -> _ which create failures during instalations.

pip version

22.0.3

Python version

3.7

OS

Linux

How to Reproduce

I could only reproduce the issue on Github.
See: https://github.com/google-research/jax3d/runs/5198336972?check_suite_focus=true

Output

Run pip install --upgrade -e .
  pip install --upgrade -e .
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.7.1[2](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:2)/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/[3](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:3).7.12/x6[4](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:4)/lib
Obtaining file:///home/runner/work/jax3d/jax3d
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Checking if build backend supports build_editable: started
  Checking if build backend supports build_editable: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
ERROR: Exception:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2741, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'array_types'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
    status = run_func(*args)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/cli/req_command.py", line 20[5](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:5), in wrapper
    return func(self, options, args)
  File "/opt/hostedtoolcache/Python/3.7.12/x[6](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:6)4/lib/python3.[7](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:7)/site-packages/pip/_internal/commands/install.py", line 340, in run
    reqs, check_supported_wheels=not options.target_dir
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 75, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 50[8](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:8), in collect_root_requirements
    requested_extras=(),
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 473, in _make_requirement_from_install_req
    version=None,
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 1[9](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:9)3, in _make_candidate_from_link
    version=version,
  File "/opt/hostedtoolcache/Python/3.7.[12](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:12)/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 319, in __init__
    version=version,
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line [15](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:15)8, in __init__
    self.dist = self._prepare()
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
    dist = self._prepare_distribution()
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 323, in _prepare_distribution
    return self._factory.preparer.prepare_editable_requirement(self._ireq)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 6[16](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:16), in prepare_editable_requirement
    req.check_if_exists(self.use_user_site)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 390, in check_if_exists
    existing_dist = get_default_environment().get_distribution(self.req.name)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_internal/metadata/pkg_resources.py", line [24](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:24)9, in get_distribution
    self._ws.require(name)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 900, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 794, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line [27](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:27)[44](https://github.com/google-research/jax3d/runs/5198086942?check_suite_focus=true#step:6:44), in requires
    "%s has no such extra feature %r" % (self, ext)
pip._vendor.pkg_resources.UnknownExtra: etils 0.4.0 has no such extra feature 'array_types'
Error: Process completed with exit code 2.

Code of Conduct
@Conchylicultor Conchylicultor added S: needs triage Issues/PRs that need to be triaged type: bug A confirmed bug or unintended behavior labels Feb 15, 2022
@jakirkham
Copy link
Contributor

Maybe this was fixed in PR: #12044

@hmc-cs-mdrissi
Copy link

I recently ran in this error while using latest pip so that PR did not resolve this. I don't think this issue really lies with pip but instead pkg_resources (which I think comes from setuptools). While error trace says pip/_vendor/pkg_resources/__init__.py the vendor is key part. I've encountered the error from pkg_resources alone where process was,

  1. pip install library with a transitive dependency on etils[array-types] that makes a setuptools script (script in this sense).
  2. When script is then run it triggers pkg_resources validation check that fails with error message of etils 0.4.0 has no such extra feature 'array_types'

So I'd recommend moving this ticket to setuptools, improve pkg_resources to lookup extras in a way that handles normalization, and then update pip vendored pkg_resources.

@Grub4K Grub4K mentioned this issue Apr 14, 2024
Closed
10 tasks
@Grub4K
Copy link

Grub4K commented Apr 14, 2024 •
edited
Loading

I think yt-dlp/yt-dlp#9682 was caused by the same issue. Reproduction steps:

Note that I was able to reproduce this using pip version 23.2.1 only. 24.0 correctly handled that situation:

$ .venv/bin/pip install 'yt_dlp-2024.4.9-py3-none-any.whl[curl_cffi]'
Processing ./yt_dlp-2024.4.9-py3-none-any.whl (from yt-dlp==2024.4.9)
Requirement already satisfied: brotli in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (1.1.0)
Requirement already satisfied: certifi in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (2024.2.2)
Requirement already satisfied: mutagen in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (1.47.0)
Requirement already satisfied: pycryptodomex in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (3.20.0)
Requirement already satisfied: requests<3,>=2.31.0 in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (2.31.0)
Requirement already satisfied: urllib3<3,>=1.26.17 in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (2.2.1)
Requirement already satisfied: websockets>=12.0 in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (12.0)
Requirement already satisfied: curl-cffi==0.5.10 in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9->yt-dlp==2024.4.9) (0.5.10)
Requirement already satisfied: cffi>=1.12.0 in ./.venv/lib/python3.11/site-packages (from curl-cffi==0.5.10->yt-dlp==2024.4.9->yt-dlp==2024.4.9) (1.16.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.11/site-packages (from requests<3,>=2.31.0->yt-dlp==2024.4.9->yt-dlp==2024.4.9) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.11/site-packages (from requests<3,>=2.31.0->yt-dlp==2024.4.9->yt-dlp==2024.4.9) (3.7)
Requirement already satisfied: pycparser in ./.venv/lib/python3.11/site-packages (from cffi>=1.12.0->curl-cffi==0.5.10->yt-dlp==2024.4.9->yt-dlp==2024.4.9) (2.22)
yt-dlp is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.

$ .venv/bin/python -m pip install 'pip==23.2.1'
Collecting pip==23.2.1
  Using cached pip-23.2.1-py3-none-any.whl.metadata (4.2 kB)
Using cached pip-23.2.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
Successfully installed pip-23.2.1

[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: python -m pip install --upgrade pip

$ .venv/bin/pip install 'yt_dlp-2024.4.9-py3-none-any.whl[curl_cffi]'
Processing ./yt_dlp-2024.4.9-py3-none-any.whl
WARNING: yt-dlp 2024.4.9 does not provide the extra 'curl_cffi'
Requirement already satisfied: brotli in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9) (1.1.0)
Requirement already satisfied: certifi in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9) (2024.2.2)
Requirement already satisfied: mutagen in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9) (1.47.0)
Requirement already satisfied: pycryptodomex in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9) (3.20.0)
Requirement already satisfied: requests<3,>=2.31.0 in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9) (2.31.0)
Requirement already satisfied: urllib3<3,>=1.26.17 in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9) (2.2.1)
Requirement already satisfied: websockets>=12.0 in ./.venv/lib/python3.11/site-packages (from yt-dlp==2024.4.9) (12.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./.venv/lib/python3.11/site-packages (from requests<3,>=2.31.0->yt-dlp==2024.4.9) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.11/site-packages (from requests<3,>=2.31.0->yt-dlp==2024.4.9) (3.7)
yt-dlp is already installed with the same version as the provided wheel. Use --force-reinstall to force an installation of the wheel.
WARNING: There was an error checking the latest version of pip.

@pradyunsg
Copy link
Member

Consolidating into #11445

@pradyunsg pradyunsg added resolution: duplicate Duplicate of an existing issue/PR and removed type: bug A confirmed bug or unintended behavior S: needs triage Issues/PRs that need to be triaged labels Apr 14, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 15, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
resolution: duplicate Duplicate of an existing issue/PR
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@jakirkham @pradyunsg @Conchylicultor @hmc-cs-mdrissi @Grub4K