Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange bug using ProcessPoolExecutor on Mac an Linux #399

Closed
joelvbernier opened this issue Feb 11, 2022 · 1 comment · Fixed by #398
Closed

Strange bug using ProcessPoolExecutor on Mac an Linux #399

joelvbernier opened this issue Feb 11, 2022 · 1 comment · Fixed by #398
Assignees
Labels
bug Something isn't working high priority move these to the top of the stack

Comments

@joelvbernier
Copy link
Member

Was getting a weird error when trying to run a script that used a ProcessPoolExecutor with mp_context='fork' from 'spawn' processes. I tracked down at least one sneaky 'spawn' to the calculation of pixel_angles in hexrd.instrument. The fix, at least to start, is to force context to 'fork' where supported, and explicitly set the context in all instances of ProcessPoolExecutor.

@psavery
Copy link
Collaborator

psavery commented Feb 11, 2022

Yeah, currently, Linux defaults to fork but Mac and Windows default to spawn.

I'm guessing this issue was on Mac? The multiprocessing docs indicate that spawn should be used for Mac. If that's the case, I think we should always be able to leave everything to their defaults?

Changed in version 3.8: On macOS, the spawn start method is now the default.
The fork start method should be considered unsafe as it can lead to crashes
of the subprocess. See **[bpo-33725](https://bugs.python.org/issue33725).**

https://docs.python.org/3/library/multiprocessing.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working high priority move these to the top of the stack
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants