From a9fc958763ac134ae35bf276354857a0915a34db Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 4 Jun 2024 02:23:48 -0400 Subject: [PATCH 01/12] [infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images The changes introduced here upgrade Python from 3.8 to 3.10.14 inside the base-builder and base-runner images. ### base-builder changes: Prior to these changes, base-builder compiled Python 3.8 from source using sources downloaded from the official release servers at https://www.python.org/ftp/python/. This updates the compiled version to 3.10.14 (the latest 3.10 release) instead. ### base-runner changes: Prior to these changes, base-runner installed Python 3.8 from the default apt repository provided by the Ubuntu 20.04 image it's based on. These apt repositories do not have a version of Python 3.10 available by default. This updates the base-runner to instead use a multi-stage build to copy the same Python interpreter compiled by the base-builder image into the runner image, which ensures both Python versions remain in-sync while saving build time by re-using a pre-built version. ## Motivation - Code coverage does not work on Python projects that use Python 3.10+ syntax, and will not work until this or similar changes are landed (see #11419) - Upgrading the base-image to use Ubuntu 22.04 (which provides more recent Python versions via apt) has been stated as being unlikely to happen any time soon (see #3290) - Many OSS-Fuzz integrated Python projects no longer support Python 3.8 and have resorted to implementing ad-hoc workarounds to upgrade to newer Python versions, including installing Python from the Dead Snakes PPA. - This leads to fragmentation and hard to debug issues. Maintenance is easier when everyone is using the same version without issue. - With [Python 3.8 reaching end of life soon (in 2024-10)][python- versions-EOL], it is likely that more Python projects will begin dropping support for 3.8, further increasing the number of broken builds and ad-hoc workarounds. - Previous attempts at upgrading Python have stalled. ## Known & Expected Issues Several project Dockerfiles and build scripts contain hard coded references to python3.8 file system paths, and many more have implanted ad-hoc workarounds to upgrade to newer Python versions than 3.8 (typically 3.9.) Additional changes are required to each of these projects to ensure they successfully build after this upgrade to Python 3.10. ### Fuzz Introspector Caveat Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is not expected to introduce any new issues, it was not tested on these changes and may require additional work. ## Possible Areas of Improvement Using the base-builder image in a multi-stage build to copy the pre- compiled Python into base-runner is effective, but feels like a workaround that may be introducing tech debt. A cleaner approach would be to extract the Python compilation into a discrete base image similar to how `base-clang` works, and use that as the multi-stage builder in images that need it. --- Fixes: - #11419 Supersedes: - #9532 - #11420 [python-versions-EOL]: https://devguide.python.org/versions/ --- infra/base-images/base-builder/Dockerfile | 16 +++++++++------- infra/base-images/base-runner/Dockerfile | 17 +++++++++++++++++ infra/base-images/base-runner/install_deps.sh | 4 +--- 3 files changed, 27 insertions(+), 10 deletions(-) diff --git a/infra/base-images/base-builder/Dockerfile b/infra/base-images/base-builder/Dockerfile index 8dcbdce6cc5c..0ab8942650cc 100644 --- a/infra/base-images/base-builder/Dockerfile +++ b/infra/base-images/base-builder/Dockerfile @@ -19,9 +19,9 @@ FROM gcr.io/oss-fuzz-base/base-clang COPY install_deps.sh / RUN /install_deps.sh && rm /install_deps.sh -# Build and install latest Python 3 (3.8.3). -ENV PYTHON_VERSION 3.8.3 -RUN export PYTHON_DEPS="\ +# Build and install latest Python 3.10. +ENV PYTHON_VERSION 3.10.14 +RUN PYTHON_DEPS="\ zlib1g-dev \ libncurses5-dev \ libgdbm-dev \ @@ -39,12 +39,14 @@ RUN export PYTHON_DEPS="\ tar -xvf Python-$PYTHON_VERSION.tar.xz && \ cd Python-$PYTHON_VERSION && \ ./configure --enable-optimizations --enable-shared && \ - make -j install && \ + make -j$(nproc) install && \ ldconfig && \ - ln -s /usr/bin/python3 /usr/bin/python && \ + ln -s /usr/local/bin/python3 /usr/local/bin/python && \ cd .. && \ rm -r /tmp/Python-$PYTHON_VERSION.tar.xz /tmp/Python-$PYTHON_VERSION && \ - rm -rf /usr/local/lib/python3.8/test && \ + rm -rf /usr/local/lib/python${PYTHON_VERSION%.*}/test && \ + python3 -m ensurepip && \ + python3 -m pip install --upgrade pip && \ apt-get remove -y $PYTHON_DEPS # https://github.com/google/oss-fuzz/issues/3888 # Install six for Bazel rules. @@ -177,4 +179,4 @@ COPY llvmsymbol.diff $SRC COPY detect_repo.py /opt/cifuzz/ COPY bazel.bazelrc /root/.bazelrc -CMD ["compile"] \ No newline at end of file +CMD ["compile"] diff --git a/infra/base-images/base-runner/Dockerfile b/infra/base-images/base-runner/Dockerfile index 234ac9a01096..951d5b781865 100755 --- a/infra/base-images/base-runner/Dockerfile +++ b/infra/base-images/base-runner/Dockerfile @@ -24,6 +24,11 @@ RUN cargo install rustfilt # Using multi-stage build to copy some LLVM binaries needed in the runner image. FROM gcr.io/oss-fuzz-base/base-clang AS base-clang +# The base builder image compiles a specific Python version. Using a multi-stage build +# to copy that same Python interpreter into the runner image saves build time and keeps +# the Python versions in sync. +FROM gcr.io/oss-fuzz-base/base-builder AS base-builder + # Real image that will be used later. FROM gcr.io/oss-fuzz-base/base-image @@ -35,6 +40,18 @@ COPY --from=base-clang /usr/local/bin/llvm-cov \ /usr/local/bin/llvm-symbolizer \ /usr/local/bin/ +# Copy the pre-compiled Python binaries and libraries +COPY --from=base-builder /usr/local/bin/python3.10 /usr/local/bin/python3.10 +COPY --from=base-builder /usr/local/lib/libpython3.10.so.1.0 /usr/local/lib/libpython3.10.so.1.0 +COPY --from=base-builder /usr/local/include/python3.10 /usr/local/include/python3.10 +COPY --from=base-builder /usr/local/lib/python3.10 /usr/local/lib/python3.10 +COPY --from=base-builder /usr/local/bin/pip3 /usr/local/bin/pip3 + +# Create symbolic links to ensure compatibility +RUN ldconfig && \ + ln -s /usr/local/bin/python3.10 /usr/local/bin/python3 && \ + ln -s /usr/local/bin/python3.10 /usr/local/bin/python + COPY install_deps.sh / RUN /install_deps.sh && rm /install_deps.sh diff --git a/infra/base-images/base-runner/install_deps.sh b/infra/base-images/base-runner/install_deps.sh index ef12cde0061a..fc0569b339ad 100755 --- a/infra/base-images/base-runner/install_deps.sh +++ b/infra/base-images/base-runner/install_deps.sh @@ -20,12 +20,10 @@ apt-get update && apt-get install -y \ binutils \ file \ + ca-certificates \ fonts-dejavu \ git \ libcap2 \ - python3 \ - python3-pip \ - python3-setuptools \ rsync \ unzip \ wget \ From e1a6e9fe59e29563f0a5dfbaeecd9d8c71762fdd Mon Sep 17 00:00:00 2001 From: David Lakin Date: Sat, 22 Jun 2024 02:37:07 -0400 Subject: [PATCH 02/12] Fix coverage builds on Python 3.10 `MarkupSafe` is a transitive dependency through `code_coverage`'s Jinja2 requirement. The previously pinned version, `MarkupSafe==0.23`, is incompatible with Python 3.10 raising the following error: ``` ImportError: cannot import name 'Mapping' from 'collections' ``` Upgrading MarkupSafe to a compatible version requires `code_coverage`'s Jinja2 requirement to be bumped from Jinja2==2.10 to 2.10.3 The `sed` change introduced here is not ideal, but is required until the upstream requirement is bumped. At that point, the `sed` should become a no-op. --- infra/base-images/base-runner/Dockerfile | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/infra/base-images/base-runner/Dockerfile b/infra/base-images/base-runner/Dockerfile index 951d5b781865..021a7f42f32d 100755 --- a/infra/base-images/base-runner/Dockerfile +++ b/infra/base-images/base-runner/Dockerfile @@ -62,8 +62,11 @@ RUN git clone https://chromium.googlesource.com/chromium/src/tools/code_coverage cd /opt/code_coverage && \ git checkout edba4873b5e8a390e977a64c522db2df18a8b27d && \ pip3 install wheel && \ + # If version "Jinja2==2.10" is in requirements.txt, bump it to a patch version that + # supports upgrading its MarkupSafe dependency to a Python 3.10 compatible release: + sed -i 's/Jinja2==2.10/Jinja2==2.10.3/' requirements.txt && \ pip3 install -r requirements.txt && \ - pip3 install MarkupSafe==0.23 && \ + pip3 install MarkupSafe==2.0.1 && \ pip3 install coverage==6.3.2 # Default environment options for various sanitizers. From 3e3fa52a81cfe1e459f2430c4bce98fb8ab23834 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Mon, 12 Aug 2024 20:08:09 -0400 Subject: [PATCH 03/12] Fix compatibility issue with pysecsan hooks in Python 3.10 Updated the hook_pre_exec_eval function in command_injection.py to accept additional arguments (*args, **kwargs). This resolves a TypeError encountered in Python 3.10 where the function was called with more arguments than expected. The change ensures compatibility with Python 3.10 by aligning the function signature with the arguments passed by the add_hook mechanism. Also replaces the deprecated `importlib.find_loader` methoc call with the recommended ` importlib.util.find_spec` alternative. These changes were tested by running the "proof-of-exploit" examples, the pyscan tests in this project, and by running `check_build` on several projects (such as `black`) that enable Pyscan. --- .../sanitizers/pysecsan/pysecsan/command_injection.py | 2 +- .../base-builder/sanitizers/pysecsan/pysecsan/sanlib.py | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py index a6c6d09b4c2b..1d010047588f 100644 --- a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py +++ b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/command_injection.py @@ -98,7 +98,7 @@ def hook_pre_exec_os_system(cmd): 'Command injection') -def hook_pre_exec_eval(cmd): +def hook_pre_exec_eval(cmd, *args, **kwargs): """Hook for eval. Experimental atm.""" res = check_code_injection_match(cmd, check_unquoted=True) if res is not None: diff --git a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py index 4647f47ffd5d..eead5c3979e5 100644 --- a/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py +++ b/infra/base-images/base-builder/sanitizers/pysecsan/pysecsan/sanlib.py @@ -22,7 +22,7 @@ import functools import subprocess import traceback -import importlib +import importlib.util from typing import Any, Callable, Optional from pysecsan import command_injection, redos, yaml_deserialization @@ -54,7 +54,7 @@ def sanitizer_log_always(msg, log_prefix=True): def is_module_present(mod_name): """Identify if module is importable.""" # pylint: disable=deprecated-method - return importlib.find_loader(mod_name) is not None + return importlib.util.find_spec(mod_name) is not None def _log_bug(bug_title): From e6fc52c3da5801b87d0b7b0950a2880db3aeb13b Mon Sep 17 00:00:00 2001 From: David Lakin Date: Mon, 12 Aug 2024 23:33:22 -0400 Subject: [PATCH 04/12] Upgrade Python Dependencies in Base Builder for Python 3.10 Atheris: Among many useful patches, the Python 3.10 compatability fixes in v2.2.2 are of particular note. See https://github.com/google/atheris/releases/tag/2.2.2 Pyinstaller: Dependency collection was improved significantly between Pyintstaller v5 and v6, in both the core library, and the more recent `pyinstaller-hooks-contrib` package it ships with. Pyinstaller versions 3.9.0 & 3.10.0 are particularly noteworthy. 3.9.0 includes updates for scipy, numpy 2.0.0, and Django to fix compatibility issues. 3.10.0 implements support for setuptools >= 71.0.0 and its new approach to vendoring its dependencies. See: https://setuptools.pypa.io/en/latest/history.html Setuptools: Many projects expect a more recent version of setuptools than was previously installed, including the pyscanner sanatizer from this repo: `infra/base-images/base-builder/sanitizers/pysecsan/` --- infra/base-images/base-builder/install_python.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/infra/base-images/base-builder/install_python.sh b/infra/base-images/base-builder/install_python.sh index 01403eae3c66..1238f26f0e5a 100755 --- a/infra/base-images/base-builder/install_python.sh +++ b/infra/base-images/base-builder/install_python.sh @@ -19,5 +19,5 @@ echo "ATHERIS INSTALL" unset CFLAGS CXXFLAGS # PYI_STATIC_ZLIB=1 is needed for installing pyinstaller 5.0 export PYI_STATIC_ZLIB=1 -LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris>=2.1.1" "pyinstaller==5.0.1" "setuptools==42.0.2" "coverage==6.3.2" +LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris==2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" rm -rf /tmp/* From 9a666fc51bf5c750e65cbd0ae1baea4d97890997 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Mon, 12 Aug 2024 23:44:26 -0400 Subject: [PATCH 05/12] Replace deprecated setup.py install method with direct pip equivalent Fixes `SetuptoolsDeprecationWarning` warnings during Pyscan installation. See: - https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html - https://github.com/pypa/setuptools/issues/917 --- infra/base-images/base-builder/compile_python_fuzzer | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/infra/base-images/base-builder/compile_python_fuzzer b/infra/base-images/base-builder/compile_python_fuzzer index 40629a007abd..a36c05f3d1a2 100755 --- a/infra/base-images/base-builder/compile_python_fuzzer +++ b/infra/base-images/base-builder/compile_python_fuzzer @@ -32,7 +32,7 @@ if [[ $SANITIZER = *introspector* ]]; then # we enter the virtual environment in the following lines because we need # to use the same python environment that installed the fuzzer dependencies. python3 /fuzz-introspector/frontends/python/prepare_fuzz_imports.py $fuzzer_path isossfuzz - + # We must ensure python3.9, this is because we use certain # AST logic from there. # The below should probably be refined @@ -84,7 +84,7 @@ then if [[ ! -d "/pysecsan" ]]; then pushd /usr/local/lib/sanitizers/pysecsan - python3 setup.py install + python3 -m pip install . popd fi From 8b056dc6c83f5b2de323ef9d31d2602ebf147fa6 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 20 Aug 2024 13:18:21 -0400 Subject: [PATCH 06/12] Bump Python Version from 3.8 to 3.10 in GitHub Actions Ensures the CI actions use the same Python version as OSS-Fuzz images. --- .github/workflows/infra_tests.yml | 2 +- .github/workflows/pr_helper.yml | 4 ++-- .github/workflows/presubmit.yml | 2 +- .github/workflows/project_tests.yml | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/workflows/infra_tests.yml b/.github/workflows/infra_tests.yml index 8197b66b2a4c..be13feeca58e 100644 --- a/.github/workflows/infra_tests.yml +++ b/.github/workflows/infra_tests.yml @@ -27,7 +27,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.8 + python-version: 3.10 - name: Install dependencies run: | diff --git a/.github/workflows/pr_helper.yml b/.github/workflows/pr_helper.yml index ddd14ef7ffb4..fdc748329804 100644 --- a/.github/workflows/pr_helper.yml +++ b/.github/workflows/pr_helper.yml @@ -20,7 +20,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.8 + python-version: 3.10 cache: pip cache-dependency-path: | infra/ci/requirements.txt @@ -41,7 +41,7 @@ jobs: env: GITHUBTOKEN: ${{secrets.GITHUB_TOKEN}} PRAUTHOR: ${{ github.event.pull_request.user.login }} - PRNUMBER: ${{ github.event.pull_request.number }} + PRNUMBER: ${{ github.event.pull_request.number }} run: python infra/pr_helper.py - name: Leave comments diff --git a/.github/workflows/presubmit.yml b/.github/workflows/presubmit.yml index 0537201f8809..42eee27966a1 100644 --- a/.github/workflows/presubmit.yml +++ b/.github/workflows/presubmit.yml @@ -27,7 +27,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.8 + python-version: 3.10 cache: pip cache-dependency-path: | infra/ci/requirements.txt diff --git a/.github/workflows/project_tests.yml b/.github/workflows/project_tests.yml index 6caf21280c48..5a98e32e895c 100644 --- a/.github/workflows/project_tests.yml +++ b/.github/workflows/project_tests.yml @@ -76,7 +76,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.8 + python-version: 3.10 cache: pip cache-dependency-path: | infra/ci/requirements.txt From c4957f515fc05936123bd220d229fd18c78bda65 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 20 Aug 2024 13:24:39 -0400 Subject: [PATCH 07/12] Specify Python Version as Strings Versions with multiple digits after the forst "." in the version number must be quoted strings, otherwise the GH action runner does not read the whole version number and actions fail with an error similar to: > Error: The version '3.1' with architecture 'x64' was not found --- .github/workflows/infra_tests.yml | 2 +- .github/workflows/pr_helper.yml | 2 +- .github/workflows/presubmit.yml | 2 +- .github/workflows/project_tests.yml | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/infra_tests.yml b/.github/workflows/infra_tests.yml index be13feeca58e..be3430606198 100644 --- a/.github/workflows/infra_tests.yml +++ b/.github/workflows/infra_tests.yml @@ -27,7 +27,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.10 + python-version: '3.10' - name: Install dependencies run: | diff --git a/.github/workflows/pr_helper.yml b/.github/workflows/pr_helper.yml index fdc748329804..c7b321c40cf7 100644 --- a/.github/workflows/pr_helper.yml +++ b/.github/workflows/pr_helper.yml @@ -20,7 +20,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.10 + python-version: '3.10' cache: pip cache-dependency-path: | infra/ci/requirements.txt diff --git a/.github/workflows/presubmit.yml b/.github/workflows/presubmit.yml index 42eee27966a1..b56966fd2b6a 100644 --- a/.github/workflows/presubmit.yml +++ b/.github/workflows/presubmit.yml @@ -27,7 +27,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.10 + python-version: '3.10' cache: pip cache-dependency-path: | infra/ci/requirements.txt diff --git a/.github/workflows/project_tests.yml b/.github/workflows/project_tests.yml index 5a98e32e895c..5b6f954b6052 100644 --- a/.github/workflows/project_tests.yml +++ b/.github/workflows/project_tests.yml @@ -76,7 +76,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: 3.10 + python-version: '3.10' cache: pip cache-dependency-path: | infra/ci/requirements.txt From 26a5c0125558023034e1d338162871c065dba89b Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 20 Aug 2024 14:04:21 -0400 Subject: [PATCH 08/12] Bump google-github-actions/setup-gcloud from v0 to v2 Also upgraddes the Cloud SDK version to the latest availiable to attempt to avoid a python 3.10 compat issue:" module 'collections' has no attribute 'MutableMapping'" tracked here: https://issuetracker.google.com/issues/202172882 This also resolves an error in the GH actions prompting for upgrade: > The v0 series of google-github-actions/setup-gcloud is no longer > maintained. It will not receive updates, improvements, > or security patches. --- .github/workflows/infra_tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/infra_tests.yml b/.github/workflows/infra_tests.yml index be3430606198..2a3e1215dda6 100644 --- a/.github/workflows/infra_tests.yml +++ b/.github/workflows/infra_tests.yml @@ -36,9 +36,9 @@ jobs: sudo env "PATH=$PATH" pip install -r infra/build/functions/requirements.txt sudo env "PATH=$PATH" pip install -r infra/cifuzz/requirements.txt - - uses: google-github-actions/setup-gcloud@v0 + - uses: google-github-actions/setup-gcloud@v2 with: - version: '298.0.0' + version: '489.0.0' - run: | sudo env "PATH=$PATH" gcloud components install beta cloud-datastore-emulator From 98bec3849def52e16c4b23d887b2332bd54ace57 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 20 Aug 2024 14:11:07 -0400 Subject: [PATCH 09/12] Revert change to atheris version constraint The `>=` was unintentionally changed to `==` in commit: e6fc52c3da5801b87d0b7b0950a2880db3aeb13b This reverts that change. --- infra/base-images/base-builder/install_python.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/infra/base-images/base-builder/install_python.sh b/infra/base-images/base-builder/install_python.sh index 1238f26f0e5a..0d5fcb96450a 100755 --- a/infra/base-images/base-builder/install_python.sh +++ b/infra/base-images/base-builder/install_python.sh @@ -19,5 +19,5 @@ echo "ATHERIS INSTALL" unset CFLAGS CXXFLAGS # PYI_STATIC_ZLIB=1 is needed for installing pyinstaller 5.0 export PYI_STATIC_ZLIB=1 -LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris==2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" +LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris>=2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" rm -rf /tmp/* From de241d97deab4f1a98e6da138b9172c38a1a4fe3 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 20 Aug 2024 14:13:24 -0400 Subject: [PATCH 10/12] Use `python -m pip` to invoke pip install command for consistentcy with pip commands in other files --- infra/base-images/base-builder/install_python.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/infra/base-images/base-builder/install_python.sh b/infra/base-images/base-builder/install_python.sh index 0d5fcb96450a..e96791c13f6e 100755 --- a/infra/base-images/base-builder/install_python.sh +++ b/infra/base-images/base-builder/install_python.sh @@ -19,5 +19,5 @@ echo "ATHERIS INSTALL" unset CFLAGS CXXFLAGS # PYI_STATIC_ZLIB=1 is needed for installing pyinstaller 5.0 export PYI_STATIC_ZLIB=1 -LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris>=2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" +LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) python -m pip install -v --no-cache-dir "atheris>=2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" rm -rf /tmp/* From 0ea6b5cdccef35246a5e57cd1d76cadebdc90aa1 Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 20 Aug 2024 21:33:34 -0400 Subject: [PATCH 11/12] Revert CI Python Version Bumps The issue these attempted to solve appear to be related to GH Action caching and not the python version, meanwhile upgrading python in these actions introduces additional issues that would need to be addressed. - Revert "Bump Python Version from 3.8 to 3.10 in GitHub Actions" from commit 8b056dc6c83f5b2de323ef9d31d2602ebf147fa6. - Revert "Specify Python Version as Strings" from commit c4957f515fc05936123bd220d229fd18c78bda65. - Revert "Bump google-github-actions/setup-gcloud from v0 to v2" from commit 26a5c0125558023034e1d338162871c065dba89b. --- .github/workflows/infra_tests.yml | 6 +++--- .github/workflows/pr_helper.yml | 4 ++-- .github/workflows/presubmit.yml | 2 +- .github/workflows/project_tests.yml | 2 +- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/.github/workflows/infra_tests.yml b/.github/workflows/infra_tests.yml index 2a3e1215dda6..8197b66b2a4c 100644 --- a/.github/workflows/infra_tests.yml +++ b/.github/workflows/infra_tests.yml @@ -27,7 +27,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: '3.10' + python-version: 3.8 - name: Install dependencies run: | @@ -36,9 +36,9 @@ jobs: sudo env "PATH=$PATH" pip install -r infra/build/functions/requirements.txt sudo env "PATH=$PATH" pip install -r infra/cifuzz/requirements.txt - - uses: google-github-actions/setup-gcloud@v2 + - uses: google-github-actions/setup-gcloud@v0 with: - version: '489.0.0' + version: '298.0.0' - run: | sudo env "PATH=$PATH" gcloud components install beta cloud-datastore-emulator diff --git a/.github/workflows/pr_helper.yml b/.github/workflows/pr_helper.yml index c7b321c40cf7..ddd14ef7ffb4 100644 --- a/.github/workflows/pr_helper.yml +++ b/.github/workflows/pr_helper.yml @@ -20,7 +20,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: '3.10' + python-version: 3.8 cache: pip cache-dependency-path: | infra/ci/requirements.txt @@ -41,7 +41,7 @@ jobs: env: GITHUBTOKEN: ${{secrets.GITHUB_TOKEN}} PRAUTHOR: ${{ github.event.pull_request.user.login }} - PRNUMBER: ${{ github.event.pull_request.number }} + PRNUMBER: ${{ github.event.pull_request.number }} run: python infra/pr_helper.py - name: Leave comments diff --git a/.github/workflows/presubmit.yml b/.github/workflows/presubmit.yml index b56966fd2b6a..0537201f8809 100644 --- a/.github/workflows/presubmit.yml +++ b/.github/workflows/presubmit.yml @@ -27,7 +27,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: '3.10' + python-version: 3.8 cache: pip cache-dependency-path: | infra/ci/requirements.txt diff --git a/.github/workflows/project_tests.yml b/.github/workflows/project_tests.yml index 5b6f954b6052..6caf21280c48 100644 --- a/.github/workflows/project_tests.yml +++ b/.github/workflows/project_tests.yml @@ -76,7 +76,7 @@ jobs: - name: Setup python environment uses: actions/setup-python@v5 with: - python-version: '3.10' + python-version: 3.8 cache: pip cache-dependency-path: | infra/ci/requirements.txt From 72c5ae9f16a48dda7f31d72364ed31cbe8c8ea3d Mon Sep 17 00:00:00 2001 From: David Lakin Date: Tue, 20 Aug 2024 21:33:57 -0400 Subject: [PATCH 12/12] Revert "Use `python -m pip` to invoke pip install command" This reverts commit de241d97deab4f1a98e6da138b9172c38a1a4fe3. --- infra/base-images/base-builder/install_python.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/infra/base-images/base-builder/install_python.sh b/infra/base-images/base-builder/install_python.sh index e96791c13f6e..0d5fcb96450a 100755 --- a/infra/base-images/base-builder/install_python.sh +++ b/infra/base-images/base-builder/install_python.sh @@ -19,5 +19,5 @@ echo "ATHERIS INSTALL" unset CFLAGS CXXFLAGS # PYI_STATIC_ZLIB=1 is needed for installing pyinstaller 5.0 export PYI_STATIC_ZLIB=1 -LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) python -m pip install -v --no-cache-dir "atheris>=2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" +LIBFUZZER_LIB=$( echo /usr/local/lib/clang/*/lib/x86_64-unknown-linux-gnu/libclang_rt.fuzzer_no_main.a ) pip3 install -v --no-cache-dir "atheris>=2.3.0" "pyinstaller==6.10.0" "setuptools==72.1.0" "coverage==6.3.2" rm -rf /tmp/*