Poetry refuses to install package with correct hash

[MartinWallgren]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian Buster

• Poetry version: 1.1.9

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/MartinWallgren/65ffceae6b597698602afdcdae927d7f

Issue

Poetry refuses to install a package even though the checksum is correct. Looking at the output it seems as if the cheksum stored in the lock file is md5 and the checksum used during installation is sha256.

Both sha256:3ae5020d5eddabcb57db9211e3f1a46ebafa28cb31cdeb4a497189041757bb7b and md5:75dbe554e7838a35e3a5836887cf9efc are valid checksums for this package according to our index (artifactory).

❯ poetry install
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing axis-json-log-formatter (0.1.0): Failed

  RuntimeError

  Retrieved digest for link axis_json_log_formatter-0.1.0.tar.gz(sha256:3ae5020d5eddabcb57db9211e3f1a46ebafa28cb31cdeb4a497189041757bb7b) not in poetry.lock metadata ['md5:75dbe554e7838a35e3a5836887cf9efc']

  at ~/.poetry/lib/poetry/installation/chooser.py:115 in _get_links
      111│
      112│         if links and not selected_links:
      113│             raise RuntimeError(
      114│                 "Retrieved digest for link {}({}) not in poetry.lock metadata {}".format(
    → 115│                     link.filename, h, hashes
      116│                 )
      117│             )
      118│
      119│         return selected_links

[MartinWallgren]

I have tried to re-create the lock file without any difference in behavior.

[danygielow]

This seems to be a bug in poetry-core 1.0.5. With poetry-core 1.0.4 packages with md5 hashes are installed normally.

[orShap]

@danygielow looks like when trying to install poetry 1.1.8 in docker (with poetry-core 1.0.4 locked in poetry.lock file)
it do installs 1.1.8 but somehow with poetry-core 1.0.5

#7 [ 4/10] RUN pip3 install pip==21.0.1 poetry==1.1.8
#7 sha256:8f42120a417b77040a77cf6cf1e2daeec7c2225b895246de3f5b5ecf08cc05e3
#7 2.256 Collecting pip==21.0.1
#7 2.276   Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
#7 2.560 Collecting poetry==1.1.8
#7 2.564   Downloading poetry-1.1.8-py2.py3-none-any.whl (173 kB)
#7 2.694 Collecting packaging<21.0,>=20.4
#7 2.698   Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
#7 2.730 Collecting crashtest<0.4.0,>=0.3.0
#7 2.734   Downloading crashtest-0.3.1-py3-none-any.whl (7.0 kB)
#7 2.847 Collecting requests<3.0,>=2.18
#7 2.852   Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
#7 3.007 Collecting keyring<22.0.0,>=21.2.0
#7 3.011   Downloading keyring-21.8.0-py3-none-any.whl (32 kB)
#7 3.052 Collecting pexpect<5.0.0,>=4.7.0
#7 3.056   Downloading pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
#7 3.127 Collecting poetry-core<1.1.0,>=1.0.4
#7 3.132   Downloading poetry_core-1.0.5-py2.py3-none-any.whl (424 kB)
...

1.1.8 poetry.lock file locks 1.0.4 https://github.com/python-poetry/poetry/blob/1.1.8/poetry.lock#L626

[danygielow]

@orShap The locked versions in poetry.lock are only used for poetry install. If you install poetry via pip the version constraints in pyproject.toml are used. In this case its >=1.0.4, <1.1 because it uses a tilde requirement.
https://github.com/python-poetry/poetry/blob/1.1.8/pyproject.toml#L27

You could try:
pip3 install pip==21.0.1 poetry==1.1.8 poetry-core==1.0.4

[hydra-zim]

according to pep-0503 , we should support all the hash functions in hashlib module in the Python standard library (currently md5, sha1, sha224, sha256, sha384, sha512)

[orShap]

@orShap The locked versions in poetry.lock are only used for poetry install. If you install poetry via pip the version constraints in pyproject.toml are used. In this case its >=1.0.4, <1.1 because it uses a tilde requirement.
https://github.com/python-poetry/poetry/blob/1.1.8/pyproject.toml#L27

You could try:
pip3 install pip==21.0.1 poetry==1.1.8 poetry-core==1.0.4

thanks! you're right

[sevaho]

Having a similar issue for private hosted pypi after upgrading to 1.1.9:

  • Updating wg-py-models (3.4.13 -> 3.4.15): Pending...
  • Updating wg-py-models (3.4.13 -> 3.4.15): Failed

  AttributeError

  'Link' object has no attribute 'is_absolute'

  at ~/.local/lib/python3.9/site-packages/poetry/core/packages/file_dependency.py:33 in __init__
       29│         self._path = path
       30│         self._base = base or Path.cwd()
       31│         self._full_path = path
       32│ 
    →  33│         if not self._path.is_absolute():
       34│             try:
       35│                 self._full_path = self._base.joinpath(self._path).resolve()
       36│             except FileNotFoundError:
       37│                 raise ValueError("Directory {} does not exist".format(self._path))

Could this be linked?

Can be mitigated by following the comment of @danygielow

@orShap The locked versions in poetry.lock are only used for poetry install. If you install poetry via pip the version constraints in pyproject.toml are used. In this case its >=1.0.4, <1.1 because it uses a tilde requirement.
https://github.com/python-poetry/poetry/blob/1.1.8/pyproject.toml#L27

You could try:
pip3 install pip==21.0.1 poetry==1.1.8 poetry-core==1.0.4

[mbelang]

Hitting the same issue but only for packages from our internal private Gemfury pypi repo. Poetry comes with standard CircleCI images and I do not have the control over the version that is installed there. Thanks to this issue, our entire stack is currently broken.

Update:
The proposed fix didn't work for me, I had to do poetry self update 1.1.8

[hydra-zim]

This seems to be a bug in poetry-core 1.0.5. With poetry-core 1.0.4 packages with md5 hashes are installed normally.

It's not the bug in poetry-core 1.0.5, the problem always exists, but it happened that Package.clone omits files of Package, so the branch in executor.py#L673 is always passed.

After python-poetry/poetry-core@3f718c5 , Package.clone use deepclone instead now, so the issue occurs

[bravefencermusashi]

Hello everyone similar problem here,

I have a dependency to a private package published through a private repository. That is what I have in my poetry.lock file:

my-package = [
    {file = "my-package-0.0.2-py3-none-any.whl", hash = "md5:3edd58d7d248531000f1e6671cf04c18"},
]

With poetry 1.1.8 everything is fine and the dependency can be installed with a poetry install, but with poetry 1.1.9 the installation (poetry install) fails with:

RuntimeError

 Invalid hash for my-package (0.0.2) using archive my-package-0.0.2-py3-none-any.whl

 at ~/.poetry/lib/poetry/installation/executor.py:619 in _download_link
     615│                     Path(archive.path) if isinstance(archive, Link) else archive,
     616│                 ).hash()
     617│             )
     618│             if archive_hash not in {f["hash"] for f in package.files}:
   → 619│                 raise RuntimeError(
     620│                     "Invalid hash for {} using archive {}".format(package, archive.name)
     621│                 )
     622│ 
     623│         return archive

I have cleared the poetry cache to make the tests
There is only one version of the "my-package" which is 0.0.2 (the one I want to install)
I have checked that the md5 is correct (lock vs real)
my repo is pypiserver 1.4.2
I'm using python 3.8.8

I had to rollback to 1.1.8 to make it work

[adawalli]

If you want a way to downgrade the "jailed" version of poetry core in a one-liner

source $(dirname $(readlink -f $(which poetry)))/activate \
  && pip install poetry-core==1.0.4 \
  && deactivate

[moinerus]

rolled back to an old build image with poetry==1.1.8 would love to see this fixed asap so we can keep up with latest.

[pietrodn]

I can't reproduce the issue. All of the hashes in my poetry.lock are SHA256, not MD5.
I've tried on existing virtualenvs as well as on fresh ones.

However some of my colleagues on the very same private Python index can reproduce the issue as for some reason they have MD5 hashes in the lockfile.

[pietrodn]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround.

Try:

poetry cache clear . --all
rm poetry.lock
poetry install

[adawalli]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround.

Try:

poetry cache clear . --all
rm poetry.lock
poetry install

beware: This will update all of the packages to the latest the resolver can resolve.

[qbedard]

It seems that clearing the Poetry cache (and deleting the generated poetry.lock file) makes Poetry 1.1.9 use SHA256 instead of MD5, providing a workaround.

Try:

poetry cache clear . --all
rm poetry.lock
poetry install

Nexus here, and this did not work for us. Followed these steps exactly and had the exact same result as before. Still getting md5 hashes. poetry 1.1.9, poetry-core 1.0.5.

[wcn00]

We (waveapps.com) have this problem too. All packages from our private repo fail to install with the same error as above. The above workaround does not work for me.
However I can confirm that it is the packages using an md5 hash in the lock file are the ones that will not install.

[adawalli]

We (waveapps.com) have this problem too. All packages from our private repo fail to install with the same error as above. The above workaround does not work for me.
However I can confirm that it is the packages using an md5 hash in the lock file are the ones that will not install.

Why doesn't the following work for you?

source $(dirname $(readlink -f $(which poetry)))/activate \
  && pip install poetry-core==1.0.4 \
  && deactivate

[qbedard]

I should clarify that pinning poetry-core to 1.0.4 does work for me. @wcn00 Have you tried this yet (per @adawalli)?

[wcn00]

We (waveapps.com) have this problem too. All packages from our private repo fail to install with the same error as above. The above workaround does not work for me.
However I can confirm that it is the packages using an md5 hash in the lock file are the ones that will not install.

Why doesn't the following work for you?

source $(dirname $(readlink -f $(which poetry)))/activate \
  && pip install poetry-core==1.0.4 \
  && deactivate

That will work. However doing this means that we have to package and build dozens of packages to get started. This problem just started happening TODAY. It means that every build that pulls one of our custom python libs (most of them) will fail.

[adawalli]

Yes, the same problem just started affecting us today as well. Thankfully, I ship poetry in a base image to all of my other images, so the fix was in one place. The poetry installer doesn't allow you to lock in core unfortunately. I haven't tried pip installing poetry, but I really like to "jail" poetry, which is why I really like the installer script....

[neersighted]

(new issue is #6301)

[cglacet]

Got a similar issue (poetry 1.3.2) today on numpy:

• Updating numpy (1.21.1 /Users/cglacet/Library/Caches/pypoetry/artifacts/46/00/22/7edac5aaf3cb24c4b0d07e36cf6dd2bfd8de15c759075430fd7e428f42/numpy-1.21.1.zip -> 1.24.2): Failed

  RuntimeError

  Hash for numpy (1.24.2) from archive numpy-1.24.2-cp310-cp310-macosx_10_9_x86_64.whl not found in known hashes (was: sha256:ee472695253c34bc93406ce542ee02b2f0e9a3e32c9f51b0089bbb7bb9218e71)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:673 in _validate_archive_hash
      669│         archive_hash: str = "sha256:" + get_file_hash(archive)
      670│         known_hashes = {f["hash"] for f in package.files}
      671│
      672│         if archive_hash not in known_hashes:
    → 673│             raise RuntimeError(
      674│                 f"Hash for {package} from archive {archive.name} not found in"
      675│                 f" known hashes (was: {archive_hash})"
      676│             )
      677│

I've tried both:

poetry cache clear pypi:numpy:1.21.1
poetry cache clear pypi:numpy:1.24.2

But got the same error.

On the other hand it works with numpy 1.24.1. Any idea?

[alirezanobakht13]

Got a similar issue (poetry 1.3.2) today on numpy:

• Updating numpy (1.21.1 /Users/cglacet/Library/Caches/pypoetry/artifacts/46/00/22/7edac5aaf3cb24c4b0d07e36cf6dd2bfd8de15c759075430fd7e428f42/numpy-1.21.1.zip -> 1.24.2): Failed

  RuntimeError

  Hash for numpy (1.24.2) from archive numpy-1.24.2-cp310-cp310-macosx_10_9_x86_64.whl not found in known hashes (was: sha256:ee472695253c34bc93406ce542ee02b2f0e9a3e32c9f51b0089bbb7bb9218e71)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:673 in _validate_archive_hash
      669│         archive_hash: str = "sha256:" + get_file_hash(archive)
      670│         known_hashes = {f["hash"] for f in package.files}
      671│
      672│         if archive_hash not in known_hashes:
    → 673│             raise RuntimeError(
      674│                 f"Hash for {package} from archive {archive.name} not found in"
      675│                 f" known hashes (was: {archive_hash})"
      676│             )
      677│

I've tried both:

poetry cache clear pypi:numpy:1.21.1
poetry cache clear pypi:numpy:1.24.2

But got the same error.

On the other hand it works with numpy 1.24.1. Any idea?

I'm facing this error installing torchrl using poetry add torchrl

Hash for torch (2.0.1) from archive torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl not found in known hashes (was: sha256:84ca53b9bbd6e9b177706ddd65964f9a449a07cb0a17191ac6ec60365f04116f)

None of above approaches worked for me:(

[rzuckerm]

@alirezanobakht13 Please try this:

rm -rf ~/.cache/pypoetry

[detkov]

Recently had a similar issue with the bitsandbytes==0.38.0.

What helped:

rm -r $POETRY_CACHE_DIR/artifacts
rm -r $POETRY_CACHE_DIR/cache
poetry cache clear . --all
rm poetry.lock
poetry install

Just clearing the cache isn't working.

[tsdaemon]

Same issue for poetry 1.5.1. Happens in CI pipelines on docker build action

[dorschw]

Happens randomly for us in CI pipelines as well quite often (multiple times a day), with random libraries. Retrying usually fixes it, but it's pretty annoying to have to do that. We may have to wrap it with a retry block

example

[SylwekFr]

still there I agree, cannot install some packages from for example Sonar Nexus Repository Manage cause it compare sha256 to md5

[plathrop-mule]

Why is this closed? The issue is clearly still present.

[omerbe-nym]

Same issue for poetry 1.5.1. Happens in our CI:

   Hash for jedi (0.19.0) from archive jedi-0.19.0-py2.py3-none-any.whl not found in known hashes (was: sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855)
 
   at /opt/poetry/venv/lib/python3.7/site-packages/poetry/installation/executor.py:819 in _validate_archive_hash
       815│         known_hashes = {f["hash"] for f in package.files if f["file"] == archive.name}
       816│ 
       817│         if archive_hash not in known_hashes:
       818│             raise RuntimeError(
     → 819│                 f"Hash for {package} from archive {archive.name} not found in"
       820│                 f" known hashes (was: {archive_hash})"
       821│             )
       822│ 
       823│         return archive_hash

[SkynetParadigm]

Add numpy 1.25.2 hash to the list when installing Pandas ^2.0.3 dependencies.

  Hash for numpy (1.25.2) from archive numpy-1.25.2-cp311-cp311-win_amd64.whl not found in known hashes (was: sha256:6d53cdc02dd39b839a28cd61176028dbf9b5a9db03183465fc97ed9929b819f0)

  at ~\.local\pipx\venvs\poetry\Lib\site-packages\poetry\installation\executor.py:818 in _validate_archive_hash
      814│         archive_hash: str = "sha256:" + get_file_hash(archive)
      815│         known_hashes = {f["hash"] for f in package.files if f["file"] == archive.name}
      816│
      817│         if archive_hash not in known_hashes:
    → 818│             raise RuntimeError(
      819│                 f"Hash for {package} from archive {archive.name} not found in"
      820│                 f" known hashes (was: {archive_hash})"
      821│             )
      822│

[miraclecorp]

does anyone know how to resolve it/??

[AmitArie]

In OSX,
Executing: rm -rf artifacts cache while in ~/Library/Caches/pypoetry did the trick.

[Sanix-Darker]

Could be great to have a way to force a hash from the pyproject.toml directly instead of requesting endless ways of updating it...
i tried deleting the ~/.cache/pyproject + poetry cache clear pypi --all and still ending with the same hash than before on poetry lock --no-update command !

Am talking about something like :

mylib = { git="https://github.com/sanix-darker/mylib", branch="fix/this-and-that", hash="this-hash" }

for example.

Am on Poetry (version 1.6.1) | OS : ubuntu 20.04.

btw, was just wondering... is there any roadmap track for this idea ? (could be really great tbh).

---

EDIT:
To make my poetry getting the good hash i had to run this monstrosity :

CLEAN_CMD="rm -rf ~/.cache/pypoetry/ && poetry cache clear pypi --all"
NOT_THIS_HASH="hash-i-want-to-get-rid-of"

for i in {1..1000}; do
    $(CLEAN_CMD)

    grep -q "$NOT_THIS_HASH" ./poetry.lock && \
        echo "Hash still there, ah s**t, there we go again..." || \
        { echo "Looks updated, please check"; break; }
    sleep 1;
done;

---

EDIT2:
I just found that, you can update manually the poetry.lock file for your desired lib and it just WORKs ? 😬
Am talking about resolved_reference field... (in my case it is git deps...

type = "git"
url = "https://github.com/sanix-darker/mylib"
reference = "fix/this"
- resolved_reference = "previous-hash"
+ resolved_reference = "current-hash"

[joelricci]

The thing that seems to work for me these days whenever I get a checksum error is:

poetry cache clear . --all and then updating the lock file.

It's a bit of a pain having to lose the entire cache though.

(Poetry version 1.3.2 on macOS)

[codeskyblue]

Poetry version 1.7.1:
This problem still exists. It took me a lot of time to find out why the hash does not match.
This problem only happens when using my own PyPI index (Artifactory).

Poetry uses requests to download packages.
(Reference: Poetry Source Code)
The issue is that requests will decode the file when the server response header contains Content-Encoding: gzip.

So when you try to download the file https://pypi-own-mirror.com/simple/colored/colored-1.4.4.tar.gz, you expect to download colored-1.4.4.tar.gz, but it actually downloads colored-1.4.4.tar (without .gz), which causes the hash to not match.

There are two ways to fix this issue

1. remove content-encoding from server response
2. change poetry source code to disable requests auto decode download file

for example (ref https://stackoverflow.com/questions/18364193/requests-disable-auto-decoding)

import requests

r = requests.get(url, stream=True)
with open(local_filename, 'wb') as f:
    for chunk in r.raw.stream(1024, decode_content=False):
        if chunk:
            f.write(chunk)

Pull request: #8701

[dimbleby]

per that pull request, your server is doing it wrong. poetry asked for colored-1.4.4.tar.gz, however the server is returning colored-1.4.4.tar (and optimizing bandwidth by sending a compressed version of that file over the wire). These are not the same thing.

[Jmennius]

For Nexus there is an easy solution to refresh hash indexes to sha256, see comment.

[ftcister]

same issue happens with torch-scatter and torch-spline-conv, when i run poetry install in my local (mac osx) runs perfectly, but in the workflows of my repo fails.

Installing torch-cluster (1.6.0)
  • Installing torch-geometric (1.7.2)
  • Installing torch-scatter (2.0.9)
  • Installing torch-sparse (0.6.15)

  RuntimeError

  Hash for torch-scatter (2.0.9) from archive torch_scatter-2.0.9-cp38-cp38-linux_x86_64.whl not found in known hashes (was: sha256:665c755a840f6890ddc967a79622aaeb2d598de7177e6feac2781a0488cd28c2)

  at ~/.local/venv/lib/python3.8/site-packages/poetry/installation/executor.py:818 in _validate_archive_hash
      814│         archive_hash: str = "sha256:" + get_file_hash(archive)
      815│         known_hashes = {f["hash"] for f in package.files if f["file"] == archive.name}
      816│ 
      817│         if archive_hash not in known_hashes:
    → 818│             raise RuntimeError(
      819│                 f"Hash for {package} from archive {archive.name} not found in"
      820│                 f" known hashes (was: {archive_hash})"
      821│             )
      822│ 

  • Installing torch-spline-conv (1.2.1)
  • Installing torchaudio (0.11.0)
  • Installing torchvision (0.12.0)

  RuntimeError

  Hash for torch-spline-conv (1.2.1) from archive torch_spline_conv-1.2.1-cp38-cp38-linux_x86_64.whl not found in known hashes (was: sha256:86cb0a75fc4c983a0d8069dc54af1aadd33ba5b9c343e1b2981b88273b710bb8)

  at ~/.local/venv/lib/python3.8/site-packages/poetry/installation/executor.py:818 in _validate_archive_hash
      814│         archive_hash: str = "sha256:" + get_file_hash(archive)
      815│         known_hashes = {f["hash"] for f in package.files if f["file"] == archive.name}
      816│ 
      817│         if archive_hash not in known_hashes:
    → 818│             raise RuntimeError(
      819│                 f"Hash for {package} from archive {archive.name} not found in"
      820│                 f" known hashes (was: {archive_hash})"
      821│             )
      822│ 

Any solution?

[RajendraP]

I think we will have to wait for new release of poetry with the fix in #8701

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.