Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images #12027

Open
wants to merge 23 commits into
base: master
Choose a base branch
from

Commits on Jun 4, 2024

  1. [infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images

    The changes introduced here upgrade Python from 3.8 to 3.10.14 inside
    the base-builder and base-runner images.
    
     ### base-builder changes:
    
    Prior to these changes, base-builder compiled Python 3.8 from source
    using sources downloaded from the official release servers at
    https://www.python.org/ftp/python/. This updates the compiled version
    to 3.10.14 (the latest 3.10 release) instead.
    
     ### base-runner changes:
    
    Prior to these changes, base-runner installed Python 3.8 from the
    default apt repository provided by the Ubuntu 20.04 image it's based
    on. These apt repositories do not have a version of Python 3.10
    available by default. This updates the base-runner to instead use a
    multi-stage build to copy the same Python interpreter compiled by the
    base-builder image into the runner image, which ensures both Python
    versions remain in-sync while saving build time by re-using a pre-built
    version.
    
     ## Motivation
    
    - Code coverage does not work on Python projects that use Python 3.10+
      syntax, and will not work until this or similar changes are landed
      (see google#11419)
    - Upgrading the base-image to use Ubuntu 22.04 (which provides more
      recent Python versions via apt) has been stated as being unlikely to
      happen any time soon (see google#3290)
    - Many OSS-Fuzz integrated Python projects no longer support Python 3.8
      and have resorted to implementing ad-hoc workarounds to upgrade to
      newer Python versions, including installing Python from the Dead
      Snakes PPA.
      - This leads to fragmentation and hard to debug issues. Maintenance
        is easier when everyone is using the same version without issue.
    - With [Python 3.8 reaching end of life soon (in 2024-10)][python-
      versions-EOL], it is likely that more Python projects will begin
      dropping support for 3.8, further increasing the number of broken
      builds and ad-hoc workarounds.
    - Previous attempts at upgrading Python have stalled.
    
     ## Known & Expected Issues
    
    Several project Dockerfiles and build scripts contain hard coded
    references to python3.8 file system paths, and many more have implanted
    ad-hoc workarounds to upgrade to newer Python versions than 3.8
    (typically 3.9.) Additional changes are required to each of these
    projects to ensure they successfully build after this upgrade to Python
    3.10.
    
     ### Fuzz Introspector Caveat
    
    Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is
    not expected to introduce any new issues, it was not tested on these
    changes and may require additional work.
    
     ## Possible Areas of Improvement
    
    Using the base-builder image in a multi-stage build to copy the pre-
    compiled Python into base-runner is effective, but feels like a
    workaround that may be introducing tech debt. A cleaner approach would
    be to extract the Python compilation into a discrete base image similar
    to how `base-clang` works, and use that as the multi-stage builder in
    images that need it.
    
    ---
    
    Fixes:
    - google#11419
    
    Supersedes:
    - google#9532
    - google#11420
    
    [python-versions-EOL]: https://devguide.python.org/versions/
    DaveLak committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    a9fc958 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. Configuration menu
    Copy the full SHA
    0f004a4 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. Fix coverage builds on Python 3.10

    `MarkupSafe` is a transitive dependency through `code_coverage`'s
    Jinja2 requirement. The previously pinned version, `MarkupSafe==0.23`,
    is incompatible with Python 3.10 raising the following error:
    
    ```
    ImportError: cannot import name 'Mapping' from 'collections'
    ```
    
    Upgrading MarkupSafe to a compatible version requires `code_coverage`'s
    Jinja2 requirement to be bumped from Jinja2==2.10 to 2.10.3
    
    The `sed` change introduced here is not ideal, but is required until the
    upstream requirement is bumped. At that point, the `sed` should become a
    no-op.
    DaveLak committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    e1a6e9f View commit details
    Browse the repository at this point in the history

Commits on Aug 7, 2024

  1. Configuration menu
    Copy the full SHA
    8624580 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. Configuration menu
    Copy the full SHA
    72b1206 View commit details
    Browse the repository at this point in the history
  2. Fix compatibility issue with pysecsan hooks in Python 3.10

    Updated the hook_pre_exec_eval function in command_injection.py to
    accept additional arguments (*args, **kwargs). This resolves a TypeError
    encountered in Python 3.10 where the function was called with more
    arguments than expected.
    
    The change ensures compatibility with Python 3.10 by aligning the
    function signature with the arguments passed by the add_hook mechanism.
    
    Also replaces the deprecated `importlib.find_loader` methoc call with
    the recommended ` importlib.util.find_spec` alternative.
    
    These changes were tested by running the "proof-of-exploit" examples,
    the pyscan tests in this project, and by running `check_build` on
    several projects (such as `black`) that enable Pyscan.
    DaveLak committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    3e3fa52 View commit details
    Browse the repository at this point in the history
  3. Upgrade Python Dependencies in Base Builder for Python 3.10

    Atheris: Among many useful patches, the Python 3.10 compatability fixes
    in v2.2.2 are of particular note.
    See https://github.com/google/atheris/releases/tag/2.2.2
    
    Pyinstaller: Dependency collection was improved significantly between
    Pyintstaller v5 and v6, in both the core library, and the more recent
    `pyinstaller-hooks-contrib` package it ships with.
    
    Pyinstaller versions 3.9.0 & 3.10.0 are particularly noteworthy.
    3.9.0 includes updates for scipy, numpy 2.0.0, and Django to fix
    compatibility issues.
    3.10.0 implements support for setuptools >= 71.0.0 and its new approach
    to vendoring its dependencies.
    See: https://setuptools.pypa.io/en/latest/history.html
    
    Setuptools: Many projects expect a more recent version of setuptools
    than was previously installed, including the pyscanner sanatizer from
    this repo:
    `infra/base-images/base-builder/sanitizers/pysecsan/`
    DaveLak committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    e6fc52c View commit details
    Browse the repository at this point in the history
  4. Replace deprecated setup.py install method with direct pip equivalent

    Fixes `SetuptoolsDeprecationWarning` warnings during Pyscan installation.
    See:
    - https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html
    - pypa/setuptools#917
    DaveLak committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    9a666fc View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    15ddae5 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Configuration menu
    Copy the full SHA
    6e04b72 View commit details
    Browse the repository at this point in the history
  2. Bump Python Version from 3.8 to 3.10 in GitHub Actions

    Ensures the CI actions use the same Python version as OSS-Fuzz images.
    DaveLak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    8b056dc View commit details
    Browse the repository at this point in the history
  3. Specify Python Version as Strings

    Versions with multiple digits after the forst "." in the version number
    must be quoted strings, otherwise the GH action runner does not read the
    whole version number and actions fail with an error similar to:
    > Error: The version '3.1' with architecture 'x64' was not found
    DaveLak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    c4957f5 View commit details
    Browse the repository at this point in the history
  4. Bump google-github-actions/setup-gcloud from v0 to v2

    Also upgraddes the Cloud SDK version to the latest availiable to attempt
    to avoid a python 3.10 compat issue:" module 'collections' has no
    attribute 'MutableMapping'" tracked here:
    https://issuetracker.google.com/issues/202172882
    
    This also resolves an error in the GH actions prompting for upgrade:
    > The v0 series of google-github-actions/setup-gcloud is no longer
    > maintained. It will not receive updates, improvements,
    > or security patches.
    DaveLak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    26a5c01 View commit details
    Browse the repository at this point in the history
  5. Revert change to atheris version constraint

    The `>=` was unintentionally changed to `==` in commit:
    e6fc52c
    
    This reverts that change.
    DaveLak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    98bec38 View commit details
    Browse the repository at this point in the history
  6. Use python -m pip to invoke pip install command

    for consistentcy with pip commands in other files
    DaveLak committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    de241d9 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. Revert CI Python Version Bumps

    The issue these attempted to solve appear to be related to GH Action caching and not the python version, meanwhile upgrading python in these actions introduces additional issues that would need to be addressed.
    
    - Revert "Bump Python Version from 3.8 to 3.10 in GitHub Actions" from commit 8b056dc.
    
    - Revert "Specify Python Version as Strings" from commit c4957f5.
    
    - Revert "Bump  google-github-actions/setup-gcloud from v0 to v2" from commit 26a5c01.
    DaveLak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    0ea6b5c View commit details
    Browse the repository at this point in the history
  2. Revert "Use python -m pip to invoke pip install command"

    This reverts commit de241d9.
    DaveLak committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    72c5ae9 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2c3f94d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    cd86abc View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Configuration menu
    Copy the full SHA
    9db721e View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2024

  1. Configuration menu
    Copy the full SHA
    b076c35 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4f6e007 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Configuration menu
    Copy the full SHA
    83c8ec1 View commit details
    Browse the repository at this point in the history