[BUG] Ubuntu 22.04 and Debian Unstable, install fails with "KeyError: 'headers'"

[hswong3i]

setuptools version

61.3.0

Python version

3.10.3+

OS

Ubuntu 22.04 Beta and Debian Unstable

Additional environment information

Running DEB packaging with dh-python.

Description

Since 60.0.0+, running debuild -uc -us require additional https://github.com/pypa/distutils/blob/debian-patch/_distutils_system_mod.py introduce by pypa/distutils#68.

For my OBS packaging for setuptools 61.3.0 (https://build.opensuse.org/package/show/home:alvistack/pypa-setuptools-61.3.0 and alvistack@d0d8d95) this logic working perfectly for:

• Ubuntu 18.04, 20.04, 21.10, or
• Debian 10, 11, Testing

But failed for:

• Ubuntu 22.04 Beta (Python 3.10.3), or
• Debian Unstable (Python 3.10.4)

Similar as #2956, again if modify the build script as below, build now OK:

 override_dh_auto_install:
-       dh_auto_install --destdir=debian/tmp
+       SETUPTOOLS_USE_DISTUTILS=stdlib dh_auto_install --destdir=debian/tmp

Expected behavior

It works for Ubuntu 21.10:

sudo podman run -ti --rm ubuntu:21.10 bash

apt update
apt full-upgrade -y
apt install -y curl wget git vim bash-completion
apt install -y python3 python3-dev dh-python build-essential debhelper devscripts fdupes
git clone https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools/
git checkout alvistack/v61.3.0
git clean -xdf
tar zcvf ../python-setuptools_61.3.0.orig.tar.gz --exclude=.git .
debuild -uc -us

Also works for Debian Testing:

sudo podman run -ti --rm debian:testing bash

apt update
apt full-upgrade -y
apt install -y curl wget git vim bash-completion
apt install -y python3 python3-dev dh-python build-essential debhelper devscripts fdupes
git clone https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools/
git checkout alvistack/v61.3.0
git clean -xdf
tar zcvf ../python-setuptools_61.3.0.orig.tar.gz --exclude=.git .
debuild -uc -us

How to Reproduce

Failed for Ubuntu 22.04:

sudo podman run -ti --rm ubuntu:22.04 bash

apt update
apt full-upgrade -y
apt install -y curl wget git vim bash-completion
apt install -y python3 python3-dev dh-python build-essential debhelper devscripts fdupes
git clone https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools/
git checkout alvistack/v61.3.0
git clean -xdf
tar zcvf ../python-setuptools_61.3.0.orig.tar.gz --exclude=.git .
debuild -uc -us

Also failed for Debian Unstable:

sudo podman run -ti --rm debian:sid bash

apt update
apt full-upgrade -y
apt install -y curl wget git vim bash-completion
apt install -y python3 python3-dev dh-python build-essential debhelper devscripts fdupes
git clone https://github.com/alvistack/pypa-setuptools.git
cd pypa-setuptools/
git checkout alvistack/v61.3.0
git clean -xdf
tar zcvf ../python-setuptools_61.3.0.orig.tar.gz --exclude=.git .
debuild -uc -us

Output

   debian/rules override_dh_auto_install
make[1]: Entering directory '/pypa-setuptools'
dh_auto_install --destdir=debian/tmp
I: pybuild base:237: /usr/bin/python3 setup.py install --root /pypa-setuptools/debian/tmp 
running install
/pypa-setuptools/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
Traceback (most recent call last):
  File "/pypa-setuptools/setup.py", line 87, in <module>
    dist = setuptools.setup(**setup_params)
  File "/pypa-setuptools/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/pypa-setuptools/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/pypa-setuptools/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/pypa-setuptools/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/pypa-setuptools/setuptools/dist.py", line 1214, in run_command
    super().run_command(command)
  File "/pypa-setuptools/setuptools/_distutils/dist.py", line 985, in run_command
    cmd_obj.ensure_finalized()
  File "/pypa-setuptools/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
    self.finalize_options()
  File "/pypa-setuptools/setup.py", line 66, in finalize_options
    install.finalize_options(self)
  File "/pypa-setuptools/setuptools/command/install.py", line 45, in finalize_options
    orig.install.finalize_options(self)
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 380, in finalize_options
    self.finalize_unix()
  File "/pypa-setuptools/_distutils_system_mod.py", line 42, in finalize_unix
    self.select_scheme("deb_system")
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 586, in select_scheme
    _select_scheme(self, name)
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 123, in _select_scheme
    scheme = _inject_headers(name, _load_scheme(_resolve_scheme(name)))
  File "/pypa-setuptools/setuptools/_distutils/command/install.py", line 161, in _inject_headers
    scheme.setdefault('headers', fallback['headers'])
KeyError: 'headers'
E: pybuild pybuild:367: install: plugin distutils failed with: exit code=1: /usr/bin/python3 setup.py install --root /pypa-setuptools/debian/tmp 
dh_auto_install: error: pybuild --install -i python{version} -p 3.10 --dest-dir /pypa-setuptools/debian/tmp returned exit code 13
make[1]: *** [debian/rules:6: override_dh_auto_install] Error 25
make[1]: Leaving directory '/pypa-setuptools'
make: *** [debian/rules:17: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui failed

[hswong3i]

@jaraco could that due to https://github.com/pypa/distutils/blob/debian-patch/_distutils_system_mod.py not Python 3.10 compatible?

[hswong3i]

A quick dirty workaround from https://sources.debian.org/src/python3.10/3.10.4-1/debian/patches/distutils-install-layout.diff/#L76-L98 temporary fix the issue for Ubuntu 22.04 and Debian Unstable:

diff --git a/setuptools/_distutils/command/install.py b/setuptools/_distutils/command/install.py
index a38cddcd..fd9a0943 100644
--- a/setuptools/_distutils/command/install.py
+++ b/setuptools/_distutils/command/install.py
@@ -64,6 +64,44 @@ INSTALL_SCHEMES = {
         'scripts': '{base}/Scripts',
         'data': '{base}',
     },
+    'unix_local': {
+        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
+        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
+        'purelib': '{base}/local/lib/python{py_version_short}/dist-packages',
+        'platlib':
+            '{platbase}/local/{platlibdir}/python{py_version_short}/dist-packages',
+        'include': '{installed_base}/include/python{py_version_short}{abiflags}',
+        'headers': '{base}/local/include/python{py_version_short}{abiflags}',
+        'platinclude':
+            '{installed_platbase}/include/python{py_version_short}{abiflags}',
+        'scripts': '{base}/local/bin',
+        'data': '{base}/local',
+        },
+    'deb_system': {
+        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
+        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
+        'purelib': '{base}/lib/python3/dist-packages',
+        'platlib': '{platbase}/{platlibdir}/python3/dist-packages',
+        'include': '{installed_base}/include/python{py_version_short}{abiflags}',
+        'headers': '{installed_base}/include/python{py_version_short}{abiflags}',
+        'platinclude':
+            '{installed_platbase}/include/python{py_version_short}{abiflags}',
+        'scripts': '{base}/bin',
+        'data': '{base}',
+        },
+    'posix_local': {
+        'stdlib': '{installed_base}/lib/python{py_version_short}',
+        'platstdlib': '{platbase}/lib/python{py_version_short}',
+        'purelib': '{base}/local/lib/python{py_version_short}/dist-packages',
+        'platlib': '{platbase}/local/lib/python{py_version_short}/dist-packages',
+        'include':
+            '{installed_base}/include/python{py_version_short}{abiflags}',
+        'headers': '{base}/local/include/python{py_version_short}{abiflags}',
+        'platinclude':
+            '{installed_platbase}/include/python{py_version_short}{abiflags}',
+        'scripts': '{base}/local/bin',
+        'data': '{base}/local',
+        },
 }
 
 # user site schemes