Use pre-compiled Python from official Docker image #7934

jeffwidman · 2023-08-31T00:00:06Z

Our Python docker builds are slow... a typical CI test run for Python takes ~21 minutes, of which the first ~10 minutes are spent building the image.

This results in slower local development, slow CI test suites, and slow deployments.

The main culprit is pyenv install which under the covers downloads the python source and then compiles it locally. Profiling showed that the download was quick, so even though pyenv supports aria2c, there's not much to be gained there. Unfortunately, a quick look at the pyenv issue tracker showed there's no way to to pass pre-compiled artifacts to pyenv.

For a long time we've bandied about the idea of switching from pyenv to downloading pre-compiled Pythons.However, we use pyenv local + pyenv exec throughout our Ruby code for switching to different Python versions. So we thought that it'd take a week or more to fully migrate away from pyenv.

Today I had to rebuild the python image multiple times, and got so annoyed that I decided to poke at it a bit.

It turns out that pyenv is simply a shim layer, and as long as /usr/local/.pyenv/versions/<x.y.z>/bin exists, it will happily pass commands to anything in that folder.

So I was able to come up with an intermediate solution that speeds the builds up drastically without requiring a large code refactor.

Running this locally results in the Python download/install/build step going from ~500 seconds all the way down to ~33 seconds, a savings of nearly 8 minutes. Given that a full CI run of the python test suite previously took ~21 minutes, this cuts it by 1/3.

Related but pulling the pre-compiled Python from a different source:

Speed up Python docker builds using pre-compiled python #7928

jeffwidman · 2023-08-31T00:10:31Z

This seems like a better way forward than #7928 for the reasons explained here:

Speed up Python docker builds using pre-compiled python #7928 (comment)

In particular, I was surprised by the 15s speedup in build times compared to that.

jeffwidman · 2023-08-31T00:19:47Z

python/Dockerfile

+## 3.8
+# Docker doesn't support parametrizing `COPY --from:python:$PY_1_23-bookworm`, so work around it using an alias.
+# TODO: If upstream adds support for Ubuntu, use that instead of Debian as the base suffix: https://github.com/docker-library/python/pull/791
+FROM python:$PY_3_8-bookworm as upstream-python-3.8


@phillipuniverse will be happy to see that this is fully parametrized so it doesn't have to be updated when we bump to latest python patch releases. 😉

Target latest Python versions - 3.11.5, 3.10.13, 3.9.18, 3.8.18 #7914 (comment)

Nice!! I am indeed happy to see this!

Also, good notes overall on optimizing the Docker build with pyenv. I have struggled with long build times myself in other projects for the same reasons!

python/Dockerfile

python/lib/dependabot/python/language_version_manager.rb

Nishnha · 2023-08-31T00:41:59Z

python/Dockerfile

+# `pip` and other scripts need their shebangs rewritten for the new location
+RUN find $PYTHON_INSTALL_LOCATION/bin -type f -exec sed -i "1s|^#!/usr/local/bin/python|#!$PYTHON_INSTALL_LOCATION/bin/python|" {} +
+# Ensure pyenv works and it's the python version we expect
+RUN PYENV_VERSION=$PY_3_8 pyenv exec python --version | grep "Python $PY_3_8" || exit 1


What happens if this command fails and returns exit 1? Does the whole Docker build fail? and would we be able to notice that it's failing?

Yes!

IMO we should be doing this test on main regardless of whether we go with this approach or not, as a safeguard against copy/pasting a change from one version to another and forgetting to update the version number... although I suppose in that case having it parametrized kinda defeats the purpose 😁.

But I figured this guards against at least some breakage, or if the upstream python docker image or pyenv changed behavior / file structure in some way...

as a safeguard against copy/pasting a change from one version to another and forgetting to update the version number... although I suppose in that case having it parametrized kinda defeats the purpose 😁.

Agreed, this is a good idea. It would also reveal if pyenv changes where/how it recognizes valid python installs

But I figured this guards against at least some breakage, or if the upstream python docker image or pyenv changed behavior / file structure in some way...

Yeah I think this method of installing python packages could be seen as temperamental if upstream changes their behavior (especially with swapping out the shebang path), but we can always revert to using pyenv if we need to.

Yeah, I wasn't super excited about needing to mung the shebang path, and for compiling ourselves we could even tune the compiler for our use cases, but since our use case is mostly CI / ephemeral jobs that run just a few lines of Python, rather than a long-lived service, it makes a lot more sense to trade faster build times for slower runtimes since it's likely a few ms runtime improvement at most...

Nishnha

I really like this approach of using the official docker images. The faster CI build step is a huge win 🏅

Also, merging this PR would mean #7891 is no longer necessary.

Your comments helped a lot in understanding the changes, thanks! You may want to open this as a separate issue though

TODO: Now that switched from pyenv install which compiled from source to downloading / copying a pre-compiled python we could entirely drop pyenv if we change our ruby code that calls pyenv exec to track which version of python to call and uses the full python paths.

Our Python docker builds are slow... a typical CI test run for Python takes ~21 minutes, of which the first ~10 minutes are spent building the image. This results in slower local development, slow CI test suites, and slow deployments. The main culprit is `pyenv install` which under the covers downloads the python source and then compiles it locally. Profiling showed that the download was quick, so even though `pyenv` supports `aria2c`, there's not much to be gained there. Unfortunately, a quick look at the `pyenv` issue tracker showed [there's no way to to pass pre-compiled artifacts to `pyenv`](https://github.com/orgs/pyenv/discussions/1872). For a long time we've bandied about the idea of switching from `pyenv` to downloading pre-compiled Pythons.However, we use `pyenv local` + `pyenv exec` throughout our Ruby code for switching to different Python versions. So we thought that it'd take a week or more to fully migrate away from `pyenv`. Today I had to rebuild the python image multiple times, and got so annoyed that I decided to poke at it a bit. It turns out that `pyenv` is simply a shim layer, and as long as `/usr/local/.pyenv/versions/<x.y.z>/bin` exists, it will happily pass commands to anything in that folder. So I was able to come up with an intermediate solution that speeds the builds up drastically without requiring a large code refactor. Running this locally results in the Python download/install/build step going from ~500 seconds all the way down to ~33 seconds, a savings of nearly 8 minutes. Given that a full CI run of the python test suite previously took ~21 minutes, this cuts it by 1/3.

jeffwidman requested a review from a team as a code owner August 31, 2023 00:00

github-actions bot added the L: python label Aug 31, 2023

jeffwidman mentioned this pull request Aug 31, 2023

Speed up Python docker builds using pre-compiled python #7928

Closed

jeffwidman force-pushed the use-pre-built-python-from-official-docker-image branch from 6429875 to 4d45fd0 Compare August 31, 2023 00:03

jeffwidman force-pushed the use-pre-built-python-from-official-docker-image branch from 4d45fd0 to b97b5dd Compare August 31, 2023 00:12

jeffwidman commented Aug 31, 2023

View reviewed changes

python/Dockerfile Show resolved Hide resolved

Nishnha reviewed Aug 31, 2023

View reviewed changes

python/Dockerfile Show resolved Hide resolved

Nishnha reviewed Aug 31, 2023

View reviewed changes

python/lib/dependabot/python/language_version_manager.rb Show resolved Hide resolved

jeffwidman mentioned this pull request Aug 31, 2023

Collapse run command experiment #7935

Closed

Nishnha reviewed Aug 31, 2023

View reviewed changes

Nishnha approved these changes Aug 31, 2023

View reviewed changes

jeffwidman mentioned this pull request Aug 31, 2023

Produce smaller Python builds #7891

Closed

jeffwidman force-pushed the use-pre-built-python-from-official-docker-image branch from b97b5dd to c2c420d Compare August 31, 2023 04:35

jeffwidman merged commit 7136328 into main Aug 31, 2023
79 checks passed

jeffwidman deleted the use-pre-built-python-from-official-docker-image branch August 31, 2023 06:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use pre-compiled Python from official Docker image #7934

Use pre-compiled Python from official Docker image #7934

jeffwidman commented Aug 31, 2023 •

edited

Loading

jeffwidman commented Aug 31, 2023

jeffwidman Aug 31, 2023

phillipuniverse Aug 31, 2023

Nishnha Aug 31, 2023 •

edited

Loading

jeffwidman Aug 31, 2023 •

edited

Loading

Nishnha Aug 31, 2023 •

edited

Loading

jeffwidman Aug 31, 2023

Nishnha left a comment

Use pre-compiled Python from official Docker image #7934

Use pre-compiled Python from official Docker image #7934

Conversation

jeffwidman commented Aug 31, 2023 • edited Loading

jeffwidman commented Aug 31, 2023

jeffwidman Aug 31, 2023

Choose a reason for hiding this comment

phillipuniverse Aug 31, 2023

Choose a reason for hiding this comment

Nishnha Aug 31, 2023 • edited Loading

Choose a reason for hiding this comment

jeffwidman Aug 31, 2023 • edited Loading

Choose a reason for hiding this comment

Nishnha Aug 31, 2023 • edited Loading

Choose a reason for hiding this comment

jeffwidman Aug 31, 2023

Choose a reason for hiding this comment

Nishnha left a comment

Choose a reason for hiding this comment

jeffwidman commented Aug 31, 2023 •

edited

Loading

Nishnha Aug 31, 2023 •

edited

Loading

jeffwidman Aug 31, 2023 •

edited

Loading

Nishnha Aug 31, 2023 •

edited

Loading