Blitz API failing during the installation process of raspiblitz v1.8: "ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE."

[qlrd]

Describe the bug

During the blitz_apiinstallation process, performed by the build_sdcard.sh script from the raspiblitz project, I came across the following error, in a generalized format for several packages:

ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    <MODULE_NAME>==<VERSION> from https://www.piwheels.org/simple/<MODULE_NAME>/<WHEEL_FILE>.whl#sha256=<HASH_ERROR_FROM_PIWHEELS> (from -r requirements.txt (<SOME_LINE>)):
        Expected sha256 <HASH_PYPI_1>
        Expected     or <HASH_ANOTHER_INDEX>
             Got        <HASH_ERROR_FROM_PIWHEELS>

To Reproduce

Steps to reproduce the behavior:

1. Build Raspiblitz SDCard to version 1.8 with fatpack option enabled
2. Wait until Blitz API installation process occur (message "# INSTALL Web API ...")
3. Got the mentioned error

Expected behavior

The pip install -r requirements.txt command used to install blitz_api call on many --hash:sh256:<some_hash> command options to check every .whl file consistency. Every package must have a correct hash provided by a index (pypi or piwheels).

Desktop:

• OS: Linux
• LN implementation: LND
• Platform: Raspiblitz 1.8 in raspiberry pi 4

Additional context

The raspiblitz project uses https://www.piwheels.org/simple/ index for install packages, therefore, some hashes described in requirements.txt will be different from those described in pypi's index.The requirements.txt file must be updated to indicate the correct index piwheels hashes.

In this way I indicate the following update done manually, for verification of the developers and a later PR if it is approved.

[ChuckNorrison]

updated my requirements file manually here
ChuckNorrison@92a26ac

[fusion44]

Thanks for the PR!

@qlrd how did you update the requirements file? Just by running poetry export --output requirements.txt?

[qlrd]

| @qlrd how did you update the requirements file? Just by running poetry export --output requirements.txt?

@fusion44, I had hard coded one hash at a time. I was updating my raspiblitz and encountered this issue.

So I hunted down where the error was and, concluding that it wasn't raspiblitz, but blitz_api, updated the hashes, one by one, according to the outputs indicated by the command pip install -r requirements.txt.

Each error provided a proper link in the form https://www.piwheels.org/simple/<MODULE_NAME>/<WHEEL_FILE>.whl#sha256=<HASH_FROM_PIWHEELS>by the command, what I added was the HASH_FROM_PIWHEELS.

[qlrd]

Can I close?

[fusion44]

@qlrd No, please don't yet. It is not clear to me yet why this happened.

Please try poetry export --output requirements.txt and install with this generated requirements file. If it still fails then something else is off. We should not have to manually edit the requirements file.

[qlrd]

Please try poetry export --output requirements.txt and install with this generated requirements file. If it still fails then something else is off. We should not have to manually edit the requirements file.

@fusion44, i will try fast as i can

[qlrd]

@fusion44, i executed poetry update && poetry export --output requirements.txt && pip install -r requirements.txt and the output is:

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Ignoring cffi: markers 'python_version >= "3.9" and python_version < "4.0" and implementation_name == "pypy"' don't match your environment
Ignoring colorama: markers 'python_version >= "3.9" and python_version < "4.0" and platform_system == "Windows"' don't match your environment
Ignoring py: markers 'python_version >= "3.9" and python_version < "4.0" and implementation_name == "pypy"' don't match your environment
Ignoring pycparser: markers 'python_version >= "3.9" and python_version < "4.0" and implementation_name == "pypy"' don't match your environment
Collecting aiohttp==3.8.1
  Using cached aiohttp-3.8.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB)
Collecting aiojobs==1.1.0
  Using cached https://www.piwheels.org/simple/aiojobs/aiojobs-1.1.0-py3-none-any.whl (12 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    aiojobs==1.1.0 from https://www.piwheels.org/simple/aiojobs/aiojobs-1.1.0-py3-none-any.whl#sha256=acdfa7df718c7273522d237b25644651bf6eef834f71f1bf0e3bb579d54a2daa (from -r requirements.txt (line 74)):
        Expected sha256 2080af76fda924bf2a60446f9b4435b11bb2418315c82664b07c2bb369b595d3
        Expected     or f0c43dfc4de359052a67df309b593732c32028deef1bb787c7a363da29accde3
             Got        acdfa7df718c7273522d237b25644651bf6eef834f71f1bf0e3bb579d54a2daa

[rootzoll]

On last sd card build test (building main branch) I also got this error and install failed:

  Downloading https://www.piwheels.org/simple/ordered-set/ordered_set-4.1.0-py3-none-any.whl (7.6 kB)
Collecting packaging==21.3
  Using cached https://www.piwheels.org/simple/packaging/packaging-21.3-py3-none-any.whl (40 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    packaging==21.3 from https://www.piwheels.org/simple/packaging/packaging-21.3-py3-none-any.whl#sha256=36bb3108f8e33f88e52e7985b4068e2585b67e5b7b3a0fa2817f9852bc39d2ef (from -r requirements.txt (line 498)):
        Expected sha256 dd47c42927d89ab911e606518907cc2d3a1f38bbd026385970643f9c5b8ecfeb
        Expected     or ef103e05f519cdc783ae24ea4e2e0f508a9c99b2d4969652eed6a2e1ea5bd522
             Got        36bb3108f8e33f88e52e7985b4068e2585b67e5b7b3a0fa2817f9852bc39d2ef

error='pip install failed'

[ChuckNorrison]

Downloaded packaging 21.3 with wget as recommended here

wget https://www.piwheels.org/simple/packaging/packaging-21.3-py3-none-any.whl

then check the hash of the file with pip:

pip hash packaging-21.3-py3-none-any.whl 
packaging-21.3-py3-none-any.whl:
--hash=sha256:36bb3108f8e33f88e52e7985b4068e2585b67e5b7b3a0fa2817f9852bc39d2ef

The hash in requirements file does not match, but it does some time ago and package version did not change, its still 21.3. piwheels issues are full with this behaviour.

tested on Raspiblitz and my Desktop-PC which runs different wheel versions, but they get the same hash

Desktop-PC wheel version:

pip list | grep wheel
wheel                  0.37.1

RaspiBlitz wheel version:

pip list | grep wheel
wheel                    0.34.2

[notice] A new release of pip available: 22.3 -> 22.3.1
[notice] To update, run: python3 -m pip install --upgrade pip

[rootzoll]

I tested now with resetting the main branch to release tag v0.5.0-beta (that build worked before) ... but now its failes with:

Collecting aiosignal==1.2.0
  Downloading https://www.piwheels.org/simple/aiosignal/aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    aiosignal==1.2.0 from https://www.piwheels.org/simple/aiosignal/aiosignal-1.2.0-py3-none-any.whl#sha256=795a51b23e074d030519301177892f0c5ebc4384ddf49fabadeca87aac8462e6 (from -r requirements.txt (line 80)):
        Expected sha256 26e62109036cd181df6e6ad646f91f0dcfd05fe16d0cb924138ff2ab75d64e3a
        Expected     or 78ed67db6c7b7ced4f98e495e572106d5c432a93e1ddd1bf475e1dc05f5b7df2
             Got        795a51b23e074d030519301177892f0c5ebc4384ddf49fabadeca87aac8462e6

error='pip install failed'

pi@raspberrypi:/home/blitzapi/blitz_api $ git log --name-status HEAD^..HEAD
commit f282e3af8d351d5dec742d74d8678f59db5f6c4d (HEAD -> main, tag: v0.5.0-beta)
Author: fusion44 <some.fusion@gmail.com>
Date:   Mon Jul 25 18:07:35 2022 +0200

    fix: add missing imports

M       app/main.py

[qlrd]

@rootzoll here the latest aiosignal==1.2.0 passes (with the last raspiblitz 1.8.0 with fatpack enabled).
But another failure occurs with the package wrapt:

Cannot uninstall 'wrapt'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall

Locally, I "resolved" this with the following command:

pi@[IP]> sudo su blitzapi
blitzapi@[IP]> cd /home/blitzapi/blitz_api
blitzapi@[IP]:~/blitz_api> pip install wrapt --upgrade --ignore-installed
blitzapi@[IP]> exit
pi@[IP]> sudo /home/admin/config.scripts/blitz.web.api.sh on

But the result was:

...
[a lot of installation...]
...
# Update Web API CONFIG (Sun 20 Nov 16:46:49 GMT 2022)
# init secret ...
# CONFIG Web API ... still in setup, skip bitcoin & lightning
# '.env' config updates - blitzapi maybe needs to be restarted

[Unit]
Description=BlitzBackendAPI
Wants=network.target
After=network.target mnt-hdd.mount

[Service]
WorkingDirectory=/home/blitzapi/blitz_api
# before every start update the config with latest credentials/settings
ExecStartPre=-/home/admin/config.scripts/blitz.web.api.sh update-config
ExecStart=/usr/bin/python -m uvicorn app.main:app --port 11111 --host=0.0.0.0 --root-path /api
User=blitzapi
Group=blitzapi
Type=simple
Restart=always
StandardOutput=journal
StandardError=journal
RestartSec=60

# Hardening
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Created symlink /etc/systemd/system/multi-user.target.wants/blitzapi.service → /etc/systemd/system/blitzapi.service.
ERROR: Couldn't determine iptables version
# the API is now running on port 11111 & doc available under:
# http:///api/docs
# check for systemd:  sudo systemctl status blitzapi
# check for logs:     sudo journalctl -f -u blitzapi
# blitz.conf.sh set blitzapi on
# FAIL: missing config file: /mnt/hdd/raspiblitz.conf

[rootzoll]

Cannot uninstall 'wrapt'.

The wrapt problem I fixt alrteady (see brach sdcard-fix on raspiblitz repo) with adding warpt to the globals pip installs: https://github.com/rootzoll/raspiblitz/blob/67d2f7591310bfc8c389fe5fe4576a7d4f0bdb51/build_sdcard.sh#L324 and the mako problem by adding python3-mako to the apt-get: https://github.com/rootzoll/raspiblitz/blob/67d2f7591310bfc8c389fe5fe4576a7d4f0bdb51/build_sdcard.sh#L308

here the latest aiosignal==1.2.0 passes

hmm .. why is that not passing on my side? @fusion44 I mean I use the tag v0.5.0-beta that even compiled before. Why now the error?

Tested also with the old RaspiOS base image ... running now into the same error.

[fusion44]

That is strange. The aiosignal package is not a direct dependency. Can you check if it is installed from OS side maybe instead via pip?

[rootzoll]

@fusion44 Good news ... thanks to a commit by @ChuckNorrison iits now working on sd card build by compiling the backend in its own venv and also running it then in it. From my side thats OK for now.

[fusion44]

That was quick. Glad to hear that.

[fusion44]

We should look into using Poetry to build the image. I think it should make the process more robust but would introduce another thing on the base system.

[rootzoll]

You mean the whole sd card image? or just the backend?

[fusion44]

The API part only. I don't think it is applicable for many other apps we usually install on the Blitz.

[fusion44]

I'll close this for now. If the error persists, please open a new issue.