-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Segfault in multithreaded Cython code in memoryview __dealloc__ in scikit-learn #50
Comments
Thanks, I can reproduce the issue and see the problem (a global cache of locks in Cython). I should have a fix soon. |
Hi @ogrisel - the fix is taking longer than I originally expected. The Cython issue was easy enough to track down and fix, but it also exposed issues in the NumPy fork and nogil Python. |
I've fixed issues in:
I have not ported the fix to NumPy 1.22.3 yet. You'll need to rebuild nogil Python from source and install the updated packages.
The bugs were:
|
I've fixed the NumPy 1.22.3 as well now.
That's all the bugs I found. I think pytest passes now. Please let me know if you run into any other issues. |
Yes I had already had noticed but did not mention it because I knew it was on the scikit-learn side and was planning to investigate. Thanks for the patch, I will open a PR to get it upstream. |
I confirm the fixes seem to work as intended: all scikit-learn tests now pass with the latest nogil versions of CPython / Cython / numpy / scipy. However I might have observed a memory leak when running some tests under I will investigate an open a dedicated issue if needed. Thanks again for the quick feedback. |
Here is the original reproducer:
Note that in this code, joblib is using simple Python-level threads instead of Python worker processes.
with either of the following GDB backtraces:
at another run I got the following backtrace:
So in both cases this is occurring in the
__dealloc__
method of a Cython managed memoryview but from 2 different Cython files in the scikit-learn source code.Note: I installed the
nogil
wheels for numpy, scipy and Cython with pip:If you want I can try to spend time to craft a minimal reproducer using only Cython (and probably numpy) without scikit-learn.
The text was updated successfully, but these errors were encountered: