Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable frozen modules by default, add a toggle #1213

Merged
merged 1 commit into from
Feb 15, 2024

Conversation

krassowski
Copy link
Member

@krassowski krassowski commented Feb 14, 2024

Fixes #1198

Frozen modules were extended to modules imported by cPython on startup and enabled by default with the intend of improving startup performance by up to 15% (see python/cpython#28335)

I would suggest we disable frozen modules by default because the performance difference in startup time is ~1% for ipykernel (4 to 5ms compared to total startup time of ~430ms) and this prevents user from getting difficult to diagnose errors during debugging. Of note the same 5ms makes for ~3% of plain Python startup time in 3.11.

Local micro benchmark (1000 repeats each) on Python 3.11 and 3.12.2 (using hyperfine).

Plain Python startup:

Command Mean [ms] Min [ms] Max [ms] Relative
python3.11 -Xfrozen_modules=on -c 'pass' 155.3 ± 16.8 138.5 315.1 1.00
python3.11 -Xfrozen_modules=off -c 'pass' 159.5 ± 18.8 140.9 371.2 1.03 ± 0.16
python3.12 -Xfrozen_modules=on -c 'pass' 138.2 ± 13.3 124.9 338.8 1.00
python3.12 -Xfrozen_modules=off -c 'pass' 141.6 ± 11.5 128.5 281.9 1.02 ± 0.13
hyperfine --runs 1000 "python -Xfrozen_modules=on -c 'pass'" "python -Xfrozen_modules=off -c 'pass'" --export-markdown x_frozen_on_off_plain_python_pass_1000.md

ipykernel import:

Command Mean [ms] Min [ms] Max [ms] Rel.
python3.11 -Xfrozen_modules=on -c 'import ipykernel' 428.0±33.0 382.5 648.2 1.00
python3.11 -Xfrozen_modules=off -c 'import ipykernel' 432.4±33.3 388.5 633.5 1.01 ± 0.11
python3.12 -Xfrozen_modules=on -c 'import ipykernel' 468.0±30.5 423.7 750.6 1.00
python3.12 -Xfrozen_modules=off -c 'import ipykernel' 472.4±29.3 425.8 664.5 1.01 ± 0.09
hyperfine --runs 1000 "PYDEVD_DISABLE_FILE_VALIDATION=1 python -Xfrozen_modules=on -c 'import ipykernel'" "python -Xfrozen_modules=on -c 'import ipykernel'" "python -Xfrozen_modules=off -c 'import ipykernel'" --export-markdown x_frozen_on_off_ipykernel_import_1000.md
Command Mean [ms] Min [ms] Max [ms] Relative
PYDEVD_DISABLE_FILE_VALIDATION=1 python3.11 -Xfrozen_modules=on -c 'import ipykernel' 427.0 ± 30.9 383.4 642.7 1.00
PYDEVD_DISABLE_FILE_VALIDATION=1 python3.12 -Xfrozen_modules=on -c 'import ipykernel' 471.1 ± 36.1 425.3 751.1 1.01 ± 0.10

Users can still re-enable freezing modules by passing --frozen_modules argument to the python -m ipykernel install

Existing users will need to regenerate their kernelspec to get the benefit of this change (or add -Xfrozen_modules=off manually).

@krassowski
Copy link
Member Author

I think the downstream test failure is due to a flaky test as it was passing on my fork.

@blink1073 blink1073 added the bug label Feb 15, 2024
Copy link
Contributor

@blink1073 blink1073 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@blink1073 blink1073 enabled auto-merge (squash) February 15, 2024 02:24
@blink1073 blink1073 merged commit eddd3e6 into ipython:main Feb 15, 2024
32 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

python 3.11 debugpy error messages
2 participants