Python 3.12-dev & Windows: zipimporter has no attribute find_module

[kdeldycke]

Description

The vendored version of pkg_resources in pip fails with Python 3.12-dev on Windows.

Expected behavior

No response

pip version

22.3.1

Python version

3.12.0-alpha.3 (x64)

OS

Windows 2022 & 2019

How to Reproduce

Tl;Dr, you just have to run in Windows with Python 3.12:

$ pip download --no-binary=:all: --no-deps --dest ./local-download/ pygments==2.14.0

---

The long version is that I am emulating the CLI above in my code, by mimicking some of pip's internals.

That way I can download the source distribution of the Pygments version I depends on, and inspect its test data files which are not distributed as part of the wheel.

This construction is working perfectly on the full matrix of:

• Distributions:
  • ubuntu-22.04
  • ubuntu-20.04
  • macos-12
  • macos-11
  • windows-2022
  • windows-2019
• Python versions:
  • 3.7
  • 3.8
  • 3.9
  • 3.10
  • 3.11
  • 3.12-dev

But not on the combination of 3.12-dev with either windows-2022 or windows-2019.

See the result of this matrix execution here, and the exact trace here.

Output

C:\(...)\pip\_internal\cli\req_command.py:247: in wrapper
    return func(self, options, args)
C:\(...)\pip\_internal\commands\download.py:93: in run
    session = self.get_default_session(options)
C:\(...)\pip\_internal\cli\req_command.py:98: in get_default_session
    self._session = self.enter_context(self._build_session(options))
C:\(...)\pip\_internal\cli\req_command.py:125: in _build_session
    session = PipSession(
C:\(...)\pip\_internal\network\session.py:343: in __init__
    self.headers["User-Agent"] = user_agent()
C:\(...)\pip\_internal\network\session.py:175: in user_agent
    setuptools_dist = get_default_environment().get_distribution("setuptools")
C:\(...)\pip\_internal\metadata\importlib\_envs.py:188: in get_distribution
    return next(matches, None)
C:\(...)\pip\_internal\metadata\importlib\_envs.py:183: in <genexpr>
    matches = (
C:\(...)\pip\_internal\metadata\base.py:612: in iter_all_distributions
    for dist in self._iter_distributions():
C:\(...)\pip\_internal\metadata\importlib\_envs.py:176: in _iter_distributions
    for dist in finder.find_eggs(location):
C:\(...)\pip\_internal\metadata\importlib\_envs.py:144: in find_eggs
    yield from self._find_eggs_in_dir(location)
C:\(...)\pip\_internal\metadata\importlib\_envs.py:111: in _find_eggs_in_dir
    from pip._vendor.pkg_resources import find_distributions
C:\(...)\pip\_vendor\pkg_resources\__init__.py:3251: in <module>
    @_call_aside
C:\(...)\pip\_vendor\pkg_resources\__init__.py:3235: in _call_aside
    f(*args, **kwargs)
C:\(...)\pip\_vendor\pkg_resources\__init__.py:3277: in _initialize_master_working_set
    tuple(
C:\(...)\pip\_vendor\pkg_resources\__init__.py:3278: in <genexpr>
    dist.activate(replace=False)
C:\(...)\pip\_vendor\pkg_resources\__init__.py:2785: in activate
    declare_namespace(pkg)
C:\(...)\pip\_vendor\pkg_resources\__init__.py:2284: in declare_namespace
    _handle_ns(packageName, path_item)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def _handle_ns(packageName, path_item):
        """Ensure that named package includes a subpath of path_item (if needed)"""
    
        importer = get_importer(path_item)
        if importer is None:
            return None
    
        # capture warnings due to #1111
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
>           loader = importer.find_module(packageName)
E           AttributeError: 'zipimporter' object has no attribute 'find_module'. Did you mean: 'load_module'?

C:\(...)\pip\_vendor\pkg_resources\__init__.py:2201: AttributeError

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[kdeldycke]

See the general deprecation issues of importlib APIs for Python 3.12:

• Remove more deprecated importlib APIs from Python 3.12 python/cpython#98040
• Meta issue for cleaning up import system cruft python/cpython#97850
• gh-97850: Deprecate find_loader and get_loader in pkgutil python/cpython#98520

[pfmoore]

Thanks for reporting this. Normally we'd pick up the fix when we update our vendored copy of pkg_resources. But we pin to a very old version of setuptools, so that won't happen automatically - this pin was added in #9170 but there's no information I can find as to why we don't keep pkg_resources up to date. @pradyunsg do you remember what the issue was there?

So we need to do something at our end to address this, once the setuptools fix is landed.

[uranusjr]

On 3.12 we shouldn’t use pkg_resources, but importlib.metadata. So I’m guessing something made a distribution-wise override (by vendor-patching or environment variable). See also #11661.

pip/src/pip/_internal/metadata/__init__.py

Lines 29 to 49 in ba38c33

	def _should_use_importlib_metadata() -> bool:
	"""Whether to use the ``importlib.metadata`` or ``pkg_resources`` backend.
	By default, pip uses ``importlib.metadata`` on Python 3.11+, and
	``pkg_resourcess`` otherwise. This can be overridden by a couple of ways:
	* If environment variable ``_PIP_USE_IMPORTLIB_METADATA`` is set, it
	dictates whether ``importlib.metadata`` is used, regardless of Python
	version.
	* On Python 3.11+, Python distributors can patch ``importlib.metadata``
	to add a global constant ``_PIP_USE_IMPORTLIB_METADATA = False``. This
	makes pip use ``pkg_resources`` (unless the user set the aforementioned
	environment variable to *True*).
	"""
	with contextlib.suppress(KeyError, ValueError):
	return bool(strtobool(os.environ["_PIP_USE_IMPORTLIB_METADATA"]))
	if sys.version_info < (3, 11):
	return False
	import importlib.metadata
	return bool(getattr(importlib.metadata, "_PIP_USE_IMPORTLIB_METADATA", True))

[uranusjr]

Duplicate of #11501

[kdeldycke]

Thanks @uranusjr for confirming the vendored pkg_resources is indeed imposed by the setuptools dependency. I stumble upon #11501 looking at the issue backlog but wasn't certain it was the right place to add my own observations. I should have mentioned it. Thanks again for making things clear! :)

[uranusjr]

@kdeldycke It would be a good idea to investigate why your copy of pip is trying to import pkg_resources on 3.12. It shouldn’t unless someone tinkered with the installation.