Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extension module is already cached error with mypyc on Python 3.13 #17748

Closed
hauntsaninja opened this issue Sep 8, 2024 · 6 comments
Closed
Labels
bug mypy got something wrong topic-mypyc mypyc bugs

Comments

@hauntsaninja
Copy link
Collaborator

hauntsaninja commented Sep 8, 2024

@cdce8p already bisected this upstream to python/cpython#118532

To Reproduce

# Set up using something like:
# python3.13 -m pip install setuptools
# python3.13 -m pip install -e .

printf '
from non_native import pure_python_const
class CanRemoveThisClass:
    pass
' > native.py

printf '
import native  # circular import
pure_python_const = 10
' > non_native.py

rm -rf build
rm *.so
python3.13 -m mypyc native.py
python3.13 -c 'import native'

Using a debug build of Python, you get:

Assertion failed: (value == NULL), function _extensions_cache_set, file import.c, line 1325.
zsh: abort      python3.13 -c 'import native

Using a non-debug build, you get:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import native
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 921, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 819, in module_from_spec
  File "<frozen importlib._bootstrap>", line 782, in _init_module_attrs
SystemError: extension module 'native' is already cached

mypyc transpiles Python code to an extension module that uses the C API. In the above, we compile native.py into an extension module and form an import cycle with a normal pure Python module

The C code mypyc generates is visible in the build directory

@hauntsaninja hauntsaninja added the bug mypy got something wrong label Sep 8, 2024
@hauntsaninja
Copy link
Collaborator Author

cc @ericsnowcurrently in case you happen to have time and kindness to take a look and offer any advice :-)

@hauntsaninja
Copy link
Collaborator Author

hauntsaninja commented Sep 8, 2024

Here's a repro that doesn't involve mypyc. I'll read more about multi-phase init tomorrow, maybe imports in single init is not allowed or something. mypyc does globals, which I think multi-phase init doesn't like, would be a big change

printf '
#define PY_SSIZE_T_CLEAN
#include <Python.h>

static struct PyModuleDef nativemodule = {
    PyModuleDef_HEAD_INIT,
    .m_name = "native",
};

PyObject* module = NULL;

PyMODINIT_FUNC PyInit_native(void) {
    if (module) {
        Py_INCREF(module);
        return module;
    }
    module = PyModule_Create(&nativemodule);
    assert(module);

    Py_XDECREF(PyImport_ImportModule("non_native"));
    PySys_WriteStdout("hello from native\\n");

    return module;
}
' > native.c

printf 'import native  # circular import' > non_native.py

printf '
from setuptools import setup, Extension

setup(name="native", ext_modules=[Extension("native", sources=["native.c"])])
' > setup.py

python setup.py build_ext --inplace

python -c 'import native'

@JukkaL
Copy link
Collaborator

JukkaL commented Sep 9, 2024

My understanding is that multi-phase init would solve the issue. I also encountered this when running tests on the release candidates, but I haven't had time to look into it in detail yet (and probably won't have time in the next week or two at least, unfortunately). Several mypyc tests fail because of this issue.

We've been relying on circular imports working without multi-phase init for years now, even if they haven't been officially supported by the Python C API.

Just to be clear, this also blocks compiling mypy on 3.13 release candidates, since mypy contains (many) import cycles.

@hauntsaninja
Copy link
Collaborator Author

I opened python/cpython#123880 upstream

@hauntsaninja
Copy link
Collaborator Author

This was fixed upstream, we should be all set for 3.13.0

@hauntsaninja
Copy link
Collaborator Author

Would be nice to make a release, I opened #17815

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mypy got something wrong topic-mypyc mypyc bugs
Projects
None yet
Development

No branches or pull requests

3 participants