-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images #12027
base: master
Are you sure you want to change the base?
Commits on Jun 4, 2024
-
[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images
The changes introduced here upgrade Python from 3.8 to 3.10.14 inside the base-builder and base-runner images. ### base-builder changes: Prior to these changes, base-builder compiled Python 3.8 from source using sources downloaded from the official release servers at https://www.python.org/ftp/python/. This updates the compiled version to 3.10.14 (the latest 3.10 release) instead. ### base-runner changes: Prior to these changes, base-runner installed Python 3.8 from the default apt repository provided by the Ubuntu 20.04 image it's based on. These apt repositories do not have a version of Python 3.10 available by default. This updates the base-runner to instead use a multi-stage build to copy the same Python interpreter compiled by the base-builder image into the runner image, which ensures both Python versions remain in-sync while saving build time by re-using a pre-built version. ## Motivation - Code coverage does not work on Python projects that use Python 3.10+ syntax, and will not work until this or similar changes are landed (see google#11419) - Upgrading the base-image to use Ubuntu 22.04 (which provides more recent Python versions via apt) has been stated as being unlikely to happen any time soon (see google#3290) - Many OSS-Fuzz integrated Python projects no longer support Python 3.8 and have resorted to implementing ad-hoc workarounds to upgrade to newer Python versions, including installing Python from the Dead Snakes PPA. - This leads to fragmentation and hard to debug issues. Maintenance is easier when everyone is using the same version without issue. - With [Python 3.8 reaching end of life soon (in 2024-10)][python- versions-EOL], it is likely that more Python projects will begin dropping support for 3.8, further increasing the number of broken builds and ad-hoc workarounds. - Previous attempts at upgrading Python have stalled. ## Known & Expected Issues Several project Dockerfiles and build scripts contain hard coded references to python3.8 file system paths, and many more have implanted ad-hoc workarounds to upgrade to newer Python versions than 3.8 (typically 3.9.) Additional changes are required to each of these projects to ensure they successfully build after this upgrade to Python 3.10. ### Fuzz Introspector Caveat Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is not expected to introduce any new issues, it was not tested on these changes and may require additional work. ## Possible Areas of Improvement Using the base-builder image in a multi-stage build to copy the pre- compiled Python into base-runner is effective, but feels like a workaround that may be introducing tech debt. A cleaner approach would be to extract the Python compilation into a discrete base image similar to how `base-clang` works, and use that as the multi-stage builder in images that need it. --- Fixes: - google#11419 Supersedes: - google#9532 - google#11420 [python-versions-EOL]: https://devguide.python.org/versions/
Configuration menu - View commit details
-
Copy full SHA for a9fc958 - Browse repository at this point
Copy the full SHA a9fc958View commit details
Commits on Jun 21, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0f004a4 - Browse repository at this point
Copy the full SHA 0f004a4View commit details
Commits on Jun 22, 2024
-
Fix coverage builds on Python 3.10
`MarkupSafe` is a transitive dependency through `code_coverage`'s Jinja2 requirement. The previously pinned version, `MarkupSafe==0.23`, is incompatible with Python 3.10 raising the following error: ``` ImportError: cannot import name 'Mapping' from 'collections' ``` Upgrading MarkupSafe to a compatible version requires `code_coverage`'s Jinja2 requirement to be bumped from Jinja2==2.10 to 2.10.3 The `sed` change introduced here is not ideal, but is required until the upstream requirement is bumped. At that point, the `sed` should become a no-op.
Configuration menu - View commit details
-
Copy full SHA for e1a6e9f - Browse repository at this point
Copy the full SHA e1a6e9fView commit details
Commits on Aug 7, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8624580 - Browse repository at this point
Copy the full SHA 8624580View commit details
Commits on Aug 13, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 72b1206 - Browse repository at this point
Copy the full SHA 72b1206View commit details -
Fix compatibility issue with pysecsan hooks in Python 3.10
Updated the hook_pre_exec_eval function in command_injection.py to accept additional arguments (*args, **kwargs). This resolves a TypeError encountered in Python 3.10 where the function was called with more arguments than expected. The change ensures compatibility with Python 3.10 by aligning the function signature with the arguments passed by the add_hook mechanism. Also replaces the deprecated `importlib.find_loader` methoc call with the recommended ` importlib.util.find_spec` alternative. These changes were tested by running the "proof-of-exploit" examples, the pyscan tests in this project, and by running `check_build` on several projects (such as `black`) that enable Pyscan.
Configuration menu - View commit details
-
Copy full SHA for 3e3fa52 - Browse repository at this point
Copy the full SHA 3e3fa52View commit details -
Upgrade Python Dependencies in Base Builder for Python 3.10
Atheris: Among many useful patches, the Python 3.10 compatability fixes in v2.2.2 are of particular note. See https://github.com/google/atheris/releases/tag/2.2.2 Pyinstaller: Dependency collection was improved significantly between Pyintstaller v5 and v6, in both the core library, and the more recent `pyinstaller-hooks-contrib` package it ships with. Pyinstaller versions 3.9.0 & 3.10.0 are particularly noteworthy. 3.9.0 includes updates for scipy, numpy 2.0.0, and Django to fix compatibility issues. 3.10.0 implements support for setuptools >= 71.0.0 and its new approach to vendoring its dependencies. See: https://setuptools.pypa.io/en/latest/history.html Setuptools: Many projects expect a more recent version of setuptools than was previously installed, including the pyscanner sanatizer from this repo: `infra/base-images/base-builder/sanitizers/pysecsan/`
Configuration menu - View commit details
-
Copy full SHA for e6fc52c - Browse repository at this point
Copy the full SHA e6fc52cView commit details -
Replace deprecated setup.py install method with direct pip equivalent
Fixes `SetuptoolsDeprecationWarning` warnings during Pyscan installation. See: - https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html - pypa/setuptools#917
Configuration menu - View commit details
-
Copy full SHA for 9a666fc - Browse repository at this point
Copy the full SHA 9a666fcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 15ddae5 - Browse repository at this point
Copy the full SHA 15ddae5View commit details
Commits on Aug 20, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 6e04b72 - Browse repository at this point
Copy the full SHA 6e04b72View commit details -
Bump Python Version from 3.8 to 3.10 in GitHub Actions
Ensures the CI actions use the same Python version as OSS-Fuzz images.
Configuration menu - View commit details
-
Copy full SHA for 8b056dc - Browse repository at this point
Copy the full SHA 8b056dcView commit details -
Specify Python Version as Strings
Versions with multiple digits after the forst "." in the version number must be quoted strings, otherwise the GH action runner does not read the whole version number and actions fail with an error similar to: > Error: The version '3.1' with architecture 'x64' was not found
Configuration menu - View commit details
-
Copy full SHA for c4957f5 - Browse repository at this point
Copy the full SHA c4957f5View commit details -
Bump google-github-actions/setup-gcloud from v0 to v2
Also upgraddes the Cloud SDK version to the latest availiable to attempt to avoid a python 3.10 compat issue:" module 'collections' has no attribute 'MutableMapping'" tracked here: https://issuetracker.google.com/issues/202172882 This also resolves an error in the GH actions prompting for upgrade: > The v0 series of google-github-actions/setup-gcloud is no longer > maintained. It will not receive updates, improvements, > or security patches.
Configuration menu - View commit details
-
Copy full SHA for 26a5c01 - Browse repository at this point
Copy the full SHA 26a5c01View commit details -
Revert change to atheris version constraint
The `>=` was unintentionally changed to `==` in commit: e6fc52c This reverts that change.
Configuration menu - View commit details
-
Copy full SHA for 98bec38 - Browse repository at this point
Copy the full SHA 98bec38View commit details -
Use
python -m pip
to invoke pip install commandfor consistentcy with pip commands in other files
Configuration menu - View commit details
-
Copy full SHA for de241d9 - Browse repository at this point
Copy the full SHA de241d9View commit details
Commits on Aug 21, 2024
-
Revert CI Python Version Bumps
The issue these attempted to solve appear to be related to GH Action caching and not the python version, meanwhile upgrading python in these actions introduces additional issues that would need to be addressed. - Revert "Bump Python Version from 3.8 to 3.10 in GitHub Actions" from commit 8b056dc. - Revert "Specify Python Version as Strings" from commit c4957f5. - Revert "Bump google-github-actions/setup-gcloud from v0 to v2" from commit 26a5c01.
Configuration menu - View commit details
-
Copy full SHA for 0ea6b5c - Browse repository at this point
Copy the full SHA 0ea6b5cView commit details -
Revert "Use
python -m pip
to invoke pip install command"This reverts commit de241d9.
Configuration menu - View commit details
-
Copy full SHA for 72c5ae9 - Browse repository at this point
Copy the full SHA 72c5ae9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2c3f94d - Browse repository at this point
Copy the full SHA 2c3f94dView commit details -
Configuration menu - View commit details
-
Copy full SHA for cd86abc - Browse repository at this point
Copy the full SHA cd86abcView commit details
Commits on Aug 22, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 9db721e - Browse repository at this point
Copy the full SHA 9db721eView commit details
Commits on Aug 28, 2024
-
Configuration menu - View commit details
-
Copy full SHA for b076c35 - Browse repository at this point
Copy the full SHA b076c35View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4f6e007 - Browse repository at this point
Copy the full SHA 4f6e007View commit details
Commits on Oct 4, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 83c8ec1 - Browse repository at this point
Copy the full SHA 83c8ec1View commit details