Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: using thread pool on macOS #1861

Merged
merged 1 commit into from
Apr 17, 2024
Merged

Conversation

linzhp
Copy link
Contributor

@linzhp linzhp commented Apr 17, 2024

The python parser uses ProcessPoolExecutor, which is problematic on macOS when it is distributed as a zip file, leading to errors like:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/var/folders/fw/vythc6112ygfsvky8mdb5p580000gn/T/Bazel.runfiles_esxfeg_v/runfiles/python3_aarch64-apple-darwin/lib/python3.9/multiprocessing/resource_tracker.py", line 24, in <module>
    from . import spawn
  File "/var/folders/fw/vythc6112ygfsvky8mdb5p580000gn/T/Bazel.runfiles_esxfeg_v/runfiles/python3_aarch64-apple-darwin/lib/python3.9/multiprocessing/spawn.py", line 13, in <module>
    import runpy
  File "/var/folders/fw/vythc6112ygfsvky8mdb5p580000gn/T/Bazel.runfiles_esxfeg_v/runfiles/python3_aarch64-apple-darwin/lib/python3.9/runpy.py", line 19, in <module>
    from pkgutil import read_code, get_importer
ModuleNotFoundError: No module named 'pkgutil'

According to "Contexts and start methods" section of the documentation:

On macOS, the spawn start method is now the default. The fork start method should be considered unsafe as it can lead to crashes of the subprocess as macOS system libraries may start threads.

meanwhile:

The 'spawn' and 'forkserver' start methods generally cannot be used with “frozen” executables (i.e., binaries produced by packages like PyInstaller and cx_Freeze) on POSIX systems.

This means there is no way to start a ProcessPoolExecutor when the Python zip file is running on macOS. This PR switches it to ThreadPoolExecutor instead.

@linzhp linzhp requested a review from f0rmiga as a code owner April 17, 2024 18:38
@f0rmiga f0rmiga added this pull request to the merge queue Apr 17, 2024
Merged via the queue into bazelbuild:main with commit afae3f0 Apr 17, 2024
4 checks passed
@linzhp linzhp deleted the thread_pool branch May 19, 2024 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants