Provide configurable python versions in base images

[newswangerd]

Is your feature request related to a problem? Please describe.
Right now all of the pulp base images use python 3.8. AAP increases the minimum python version with each release. This has been causing us problems in CI for some time, but has recently come to a head with the latest galaxy ng release which will depend on some shared ansible libraries that (currently) require a minimum of python 3.9. Because of this, we've had to increase the minimum python version in galaxy_ng to 3.9 (ansible/galaxy_ng#2048) and as a result the vanilla pulp-ci-centos base image no longer works with galaxy_ng.

Describe the solution you'd like
This is not the first or last time we will need to bump our minimum python version (as mentioned above this will happen with each AAP release). I would like some way to configure the python version at build time in the base images. OCI Env (https://github.com/pulp/oci_env/blob/main/base/Dockerfile#L12-L14) has the option to specify a different python version as a docker build arg, and I would like to have the same option in the base images. The base pulp images don't have to work the same way that OCI Env is set up right now (using the switch_python), since this is somewhat of a hacky approach. Rather, the base image should just install the python version that's provided in the build arg, rather than trying to change it after the fact.

Describe alternatives you've considered
Being able to run galaxy_ng using newer versions of python is a hard requirement for us.

[dkliban]

The base image is pulp-ci-centos and it's current tags are here[0]. If we start building base images for Python 3.8 and 3.9, we could publish those as pulp-ci-centos:latest-python38 and pulp-ci-centos:latest-python39. Is this tagging scheme acceptable? Or should we split these into separate repositories?

[0] https://quay.io/repository/pulp/pulp-ci-centos?tab=tags

[mdellweg]

... we could publish those as pulp-ci-centos:latest-python38 and pulp-ci-centos:latest-python39.

What do we get from the "latest" part? We do not version them in any way, do we? Also we used python36 as a tag in the past.

https://github.com/pulp/plugin_template/blob/main/templates/github/.github/workflows/scripts/install.sh.j2#L100

[dkliban]

We do version pulp-ci-centos by pulpcore version. However, I don't think that is needed. This image doesn't have any Pulp application code in it. So it would make sense to have two additional tags pulp-ci-centos:python38 and pulp-ci-centos:python39.

[mikedep333]

@dkliban The reason for that is that the containerfile differs based on branch, each tied to the pulpcore version. Mostly differences in C dependencies. Usually we just add new stuff in newer branches, but we do occasionally replace or remove stuff.

[dkliban]

@mikedep333 That makes sense. So then we should use more verbose tags that include the pulpcore version. @mdellweg what do you think about that?

[mdellweg]

So: pulp-ci-centos:3.45-python39.
But we only use these in ci when we realize, we need to. Most of the time we use lastest (and to me that word is only a placeholder for "nothing special").
Am I right?

[dkliban]

Yes, that's right. The word comes from the name of the branch where the Containerfile is coming from.

[newswangerd]

Just keep in mind that the process for building containers for different versions of python should be easier in the future. We'll have to make the jump to python 3.11 soon as well.