Setuptools 68.2+ produces metadata with extra that’s read incorrectly by importlib.metadata

[uranusjr]

Work in progress…

[sanderr]

You've probably already seen this, but just in case you haven't, I expect this is related to the changes in pypa/setuptools#3903.

[uranusjr]

This only seems to fail on Python 3.11 (with only setuptools 68.2), but nothing in the setuptools change seems to jump out. There are many things involved though so I’m not sure yet.

[uranusjr]

Bisecting brought me to pypa/setuptools@807ab8a, a part of pypa/setuptools#3904 instead.

From what I can tell, the commits in 3904 and 3903 are pretty closely coupled though, so the actual logic change might as well happened in that PR you mentioned, but was enabled in that specific commit. No idea.

This only happens on 3.11 because that’s the cutoff point pip switches to use impotlib.metadata (instead of pkg_resources) to read package metadata, and apparently the new metadata format setuptools now generates is interpreted differently by the two different libraries. I haven’t figured out why that’s the case yet.

This can be validated by running the test (with setuptools 68.2, on Python 3.11+) with environment variable _PIP_USE_IMPORTLIB_METADATA set to false, which forces pip to use pkg_resources. The test would pass.

[uranusjr]

Hah, found the culprit. Our importlib.metadata backend sanitises Provides-Extras values:

pip/src/pip/_internal/metadata/importlib/_dists.py

Lines 210 to 213 in 3e232ce

	def iter_provided_extras(self) -> Iterable[str]:
	return (
	safe_extra(extra) for extra in self.metadata.get_all("Provides-Extra", [])
	)

But the pkg_resources backend does not:

pip/src/pip/_internal/metadata/pkg_resources.py

Lines 218 to 219 in 3e232ce

	def iter_provided_extras(self) -> Iterable[str]:
	return self._dist.extras

I think some change included in 68.2 changes how extra values are processed and causes the raw extra values in METADATA to become non-normalised. When the user passes in a non-normalised extra, this makes the importlib.metadata backend to not able to pick it up. We probably need to overhaul how extras are handled, which we should do anyway to better implement PEP 685.

[pradyunsg]

So... #12002 which I completely forgot to re-review? :)

[uranusjr]

Actually I think that PR plus one more change would be needed; the PR only changes the resolution logic, but the extra would still be read wrong when pip tries to actually install the resolved candidates.

[uranusjr]

I just merged a change that should skip the failing test for now so it doesn’t block everyone’s workflow on other stuff.