5.0.0 is causing problems in Python 3.7. Can't import celery until I downgrade to 4.13.0.

[coredumperror]

I can't run the import statement from celery import Celery in Python 3.7 with importlib-metadata 5.0.0 installed. I get:

>>> from celery import Celery
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'Celery' from 'celery' (/Users/rrollins/.pyenv/versions/temp-3.7/lib/python3.7/site-packages/celery/__init__.py)

This makes no sense. That file totally has a Celery object in it... kindof. It's brought in by an unusual construction at the bottom of https://github.com/celery/celery/blob/master/celery/__init__.py, which may be what's confusing importlib-metadata 5.

Minimally necessary install to trigger this is to make a virtualenv from Python 3.7.10, activate it, and run pip install celery. This will automatically install importlib-metadata 5.0.0.

Then just start python and run from celery import Celery. It'll crash with the above traceback. Downgrade to importlib-metadata==4.13.0 and it won't crash.

This also affects 3.7.13, so it probably affects all other point releases of 3.7. We use 3.7.10 because that's what the AmazonLinux docker image provides.

Oddly enough, installing celery on Python 3.8 doesn't result in importlib-metadata being installed automatically. The import works fine without it installed. And manually installing 5.0 doesn't break the import like it does in 3.7. So this seems to be a Python 3.7-specific issue.

[TkTech]

Can confirm this was a very confusing bug that hit us today, and we tracked it down to an update of importlib-metadata from 4.12.0 to 5.0.0. From a quick check it seems to be this line, v4.13.0...v5.0.0#diff-bf79a43449f7a7e1e76063e303fbdd35bec7eb50f2e9ddba26e3048def32ed06R856, but it seems odd this would impact importing celery.

[sonerayberk]

Same issue.

[Safihre]

Yep same here! Breaks Python 3.7 compatibility.
Took a long time to figure out....

[longwuyuan]

This broke every single airflow cluster that did a pull :-(

[Safihre]

See pinned issue #409.
@jaraco can you confirm if this really is Python 3.7 specific problem or the problem is in the packages we use that need updating?

[jaraco]

See celery/celery#7783 where celery has been tracking the issue.

confirm if this really is Python 3.7 specific problem or the problem is in the packages we use that need updating?

The issue isn't specific to Python 3.7. It will affect any consumers of importlib_metadata 5 or later that use entry_points() and rely on the dict-like interface it previously returned. In the case of Celery, it did only affect Python 3.7 and earlier because it relied on stdlib's importlib.metadata on Python 3.8 and later.

There are many projects that only rely on importlib_metadata for Python 3.7 and earlier and rely on importlib.metadata in the stdlib for 3.8 and later. These projects are also at risk, because in Python 3.12, importlib.metadata gets the same behavior. Only on importlib_metadata 3.6 and Python 3.10 or later is the SelectableGroups compatibility interface supplied, so anyone using importlib.metadata on Python 3.8 or 3.9 must use the deprecated interface.

That's why I recommend to either:

• depend on importlib_metadata >= 3.6 on Python < 3.10
• depend on backports.entry_points_selectable

Other options that I don't recommend include:

• update the code to adapt different interfaces on Python 3.8 and 3.9
• Bypass the entry_points function entirely and use more raw interfaces on Distribution objects.

A temporary workaround is to pin to importlib_metadata < 5 (or force-install that version).

I recognize that as consumers of Celery, this issue feels out of your control, which is why I'm glad to see that Celery has addressed the issue (though best I can tell, the fix isn't released yet).

Reliance on this deprecated behavior has been marked as Deprecated for over a year. I'm curious to see - is there something unique to celery (or other projects) that made it difficult for these downstream projects to detect and respond to the warnings?

I can confirm that in at least one run, celery did emit the warnings:

    =============================== warnings summary ===============================
  .tox/3.7-integration-redis/lib/python3.7/site-packages/kombu/utils/compat.py:85
    /home/runner/work/celery/celery/.tox/3.7-integration-redis/lib/python3.7/site-packages/kombu/utils/compat.py:85: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
      entry_points = importlib_metadata.entry_points().get(namespace, [])
  
  t/integration/test_canvas.py: 150 warnings
  t/integration/test_inspect.py: 5 warnings
  t/integration/test_security.py: 1 warning
  t/integration/test_tasks.py: 40 warnings
    /home/runner/work/celery/celery/celery/utils/imports.py:144: DeprecationWarning: SelectableGroups dict interface is deprecated. Use select.
      for ep in entry_points().get(namespace, []):

Things that celery and other projects can do to detect these incoming incompatible changes:

• When testing, treat warnings like errors. This approach means more maintenance burden as any new warning will result in a broken CI. On the other hand, it creates a strong signal that action is required. The errors can be triaged, registered in a bug tracker, and suppressed, and the maintainers can prioritize the fix as needed.
• Detect warnings when testing interactively. Tools like pytest will change the result from green to yellow if any warnings are issued. Anything but a green result indicates that action is required.
• Create a separate workflow or alerting system for detecting and reporting warnings.

I regret the disruption caused here. If there's something more that importlib_metadata can do, please let me know.

[Safihre]

@jaraco Thanks for the details.
Just wanted to let you know that in our runs, we haven't seen anything.
For example a run where I have set importlib_metadata<5.0.0.
The problem for us was in tavern which relies on stevedore.

[jaraco]

@jaraco Thanks for the details. Just wanted to let you know that in our runs, we haven't seen anything. For example a run where I have set importlib_metadata<5.0.0. The problem for us was in tavern which relies on stevedore.

I looked briefly and I concur - it appears as if no warning was issued for sabnzbd/sabnzbd and I don't see anything that would have disabled the warnings (e.g. -W ignore). I think it would be a useful exercise to investigate why the tests failed to report the deprecation warning, as it may help me understand a condition where such a warning doesn't reach the intended audience, or it may help projects like yours avoid this situation for other deprecations.

Is it possible that stevedore has updated the underlying usage and that's why the warnings aren't emitted any longer? Are you able to run with -W error to force any deprecation warnings to raise as Exceptions to find a traceback showing the codepath that's reached?

[jaraco]

I suspect the way tavern.run invokes pytest.main directly may be implicated in masking any warnings that occur within that run.

[Lawouach]

Hi @jaraco

I failed to detect this issue with the Chaos Toolkit but an user reported it recently.

import click
import importlib_metadata
from click_plugins import with_plugins

@click.group
def cli():
    pass


with_plugins(importlib_metadata.entry_points().get("chaostoolkit.cli_plugins"))(cli)

This throws an error with importlib_metadata >= 5

So I switched to:

import click
import importlib_metadata
from click_plugins import with_plugins

@click.group
def cli():
    pass

with_plugins(importlib_metadata.entry_points(group="chaostoolkit.cli_plugins"))(cli)

This now works.

But I'm confused by your statement above. What's the right path forward? Should I use this fix and force importlib_metadata >= 6 ? Or should I stick to older versions as other projects will likely not accept this newer version when installing?

Basically, I'm not clear what's the official position.

Cheers,

[jaraco]

You don't need to force importlib_metadata 6. >=3.6 is the minimum version for selectable groups (the latter usage). If you can't rely on that version (released Feb 21), there is backports.entry_points_selectable that provides the preferred interface for older importlib_metadata and Python releases.