Prevent missing package_data in CI

[jpy-git]

Current problem

This is an improvement to prevent issue like #6028 (Fixed by #6029) from occurring again.

Desired solution

2 options:

• Don't use editable installs for testing: Editable install just uses an egg-link to link the code to site-packages and therefore simulate the package being installed. The risk is that non-python files within the code aren't automatically included in package_data and therefore, whilst present in editable install, aren't present in actual installs. This can either be via tox or switching pip install -e . with pip install .. Looking at tests.yaml it's not immediately obvious to me where the package is getting installed so hopefully a maintainer can point this out 😄

• have a separate check to assert that certain files are within distributions:

# build wheel/source
python setup.py sdist bdist_wheel
# check file appears in them
tar --list -f dist/pylint-2.14.0.dev0.tar.gz | grep "testing_pylintrc" | wc -l
unzip -l dist/pylint-2.14.0.dev0-py3-none-any.whl | grep "testing_pylintrc" | wc -l

My preference is option 1 since option 2 requires maintaining a separate list of files to check.

Additional context

No response

[Pierre-Sassoulas]

We can launch the test with tox, but pytest alone is faster which is probably why we don't use tox in github action (locally you also have to --recreate or remember to reinstall if you're not using editable install). To be fair I don't think we have non python files to include in the packaging often enough to warrant a check or having to remember that we need to reinstall so the code is up to date.

There's another solution: it's to generate the default value at runtime in the testutil if the file do not exists. As those tests class are used only by downstream library maintainer this file is very seldom required so not adding it to the packaging at all will be better 99,95% of the time.

[cdce8p]

I would argue against using tox. The current caching mechanism is pretty optimized, not sure the same can be archived with tox. In the end, I don't expect that to be a frequent problem. So we might also be fine doing nothing.

[jpy-git]

Yeah it's not essential, only made the issue in case you wanted extra options for checking but doesn't sound worth it 👍

There's another solution: it's to generate the default value at runtime in the testutil if the file do not exists.

We can discuss this on #6029