auditwheel 5.1.0: policy.json not installed when dist-info created first / when running from git tarball

[hroncok]

When I run the tests over installed auditwheel 4.0.0 from source, It fails with:

==================================== ERRORS ====================================
___________________ ERROR collecting tests/unit/test_main.py ___________________
tests/unit/test_main.py:4: in <module>
    from auditwheel.main import main
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/main.py:11: in <module>
    from . import main_repair
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/main_repair.py:4: in <module>
    from .policy import (load_policies, get_policy_by_name, get_policy_name,
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/__init__.py:59: in <module>
    with open(join(dirname(abspath(__file__)), 'policy.json')) as f:
E   FileNotFoundError: [Errno 2] No such file or directory: '/builddir/build/BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/policy.json'
__________________ ERROR collecting tests/unit/test_policy.py __________________
tests/unit/test_policy.py:5: in <module>
    from auditwheel.policy import get_arch_name, get_policy_name, \
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/__init__.py:59: in <module>
    with open(join(dirname(abspath(__file__)), 'policy.json')) as f:
E   FileNotFoundError: [Errno 2] No such file or directory: '/builddir/build/BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/policy.json'
__________________ ERROR collecting tests/unit/test_repair.py __________________
tests/unit/test_repair.py:5: in <module>
    from auditwheel.repair import append_rpath_within_wheel
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/repair.py:17: in <module>
    from .policy import get_replace_platforms
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/__init__.py:59: in <module>
    with open(join(dirname(abspath(__file__)), 'policy.json')) as f:
E   FileNotFoundError: [Errno 2] No such file or directory: '/builddir/build/BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/policy.json'
________________ ERROR collecting tests/unit/test_wheel_abi.py _________________
tests/unit/test_wheel_abi.py:5: in <module>
    from auditwheel import wheel_abi
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/wheel_abi.py:17: in <module>
    from .policy import (lddtree_external_references, versioned_symbols_policy,
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/__init__.py:59: in <module>
    with open(join(dirname(abspath(__file__)), 'policy.json')) as f:
E   FileNotFoundError: [Errno 2] No such file or directory: '/builddir/build/BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/policy/policy.json'
=============================== warnings summary ===============================
../../BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/patcher.py:2
  /builddir/build/BUILDROOT/auditwheel-4.0.0-1.fc35.x86_64/usr/lib/python3.10/site-packages/auditwheel/patcher.py:2: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
    from distutils.spawn import find_executable

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
ERROR tests/unit/test_main.py - FileNotFoundError: [Errno 2] No such file or ...
ERROR tests/unit/test_policy.py - FileNotFoundError: [Errno 2] No such file o...
ERROR tests/unit/test_repair.py - FileNotFoundError: [Errno 2] No such file o...
ERROR tests/unit/test_wheel_abi.py - FileNotFoundError: [Errno 2] No such fil...
!!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
========================= 1 warning, 4 errors in 0.33s =========================

The policy.json file does not appear to be installed.

I was wondering why the official wheel has it.

My observations so far are:

• when I build the wheel from sdist directly, policy.json is installed
• when I run prepare_metadata_for_build_wheel hook first, it deletes the egg-info a creates dist-info, policy.json is not installed
• when I build the wheel from git directly, policy.json is installed
• when I build the wheel from git archive with export PBR_VERSION=4.0.0, policy.json is not installed

Is there a way to make policy.json always installed? Should it be listed somewhere explicitly as a data file?

[hroncok]

With 5.0.0, this no longer happens. Not sure if fixed or just flaky.

[mayeut]

most things changed with 5.0.0 w.r.t. the way auditwheel is built/layout/tested so I'd say it's been fixed in the process.

[hroncok]

With 5.1.0, this strikes back again.

[hroncok]

Installing from a downloaded git tarball (because sdist does not have tests):

$ python3.10 -m venv __venv__
$ . __venv__/bin/activate
$ export SETUPTOOLS_SCM_PRETEND_VERSION_FOR_AUDITWHEEL=5.1.0
$ pip install .
Processing /home/churchyard/rpmbuild/fedora-scm/auditwheel/auditwheel-5.1.0
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pyelftools>=0.24
  Using cached pyelftools-0.27-py2.py3-none-any.whl (151 kB)
Building wheels for collected packages: auditwheel
  Building wheel for auditwheel (PEP 517) ... done
  Created wheel for auditwheel: filename=auditwheel-5.1.0-py3-none-any.whl size=49074 sha256=5f818e9e6765ab5a1a03cdabbe5da1b60e4797a61affe48d28954983f091c472
  Stored in directory: /home/churchyard/.cache/pip/wheels/89/73/ca/836ce0ce0483975c7a8e3d5eb072ed710f20031c21c28ff4e8
Successfully built auditwheel
Installing collected packages: pyelftools, auditwheel
Successfully installed auditwheel-5.1.0 pyelftools-0.27
$ ls __venv__/lib/python3.10/site-packages/auditwheel/policy/
__init__.py  __pycache__  external_references.py  versioned_symbols.py

[mayeut]

Installing from a downloaded git tarball

OK, I think that the files might just be ignored because they're not tracked by SCM in the git tarball.
Might tweak the manifest to make this possible.

It might have been a bit much to remove the tests from the SDist (there's been some discussion about this on discuss as a generic thing) but as they're not needed for installation and quite large, they've been removed.

I'll fix the issue for installation from a git tarball for sure.
I guess there are really few people rebuilding from sources these days and, as I'm guessing from the logs and your resume, you're probably repackaging this for Fedora. I have only one question left, as a "repackager", do you prefer to build from an SDist on PyPI or from the source repo ? (I guess I have the answer now that I looked how it was done for 5.0.0 https://src.fedoraproject.org/rpms/auditwheel/blob/rawhide/f/auditwheel.spec)

I will can re-add tests in the SDist as well then.

[hroncok]

as I'm guessing from the logs and your resume, you're probably repackaging this for Fedora

Indeed. I've always tried to provide a Fedora-less reporducers for you that I forgot to mention that.

do you prefer to build from an SDist on PyPI or from the source repo

I have no preference, expect that with setuptools_scm, I need to manually set SETUPTOOLS_SCM_PRETEND_VERSION_FOR_AUDITWHEEL when building from git tarball which is a bit annoying.

Thank you!

[mayeut]

Let's re-add tests to the SDist also then.

[mayeut]

hopefully, this is now fixed in 5.1.1.