[SPARK-46020][INFRA] Add Python 3.12 to Infra docker image

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to add Python 3.12 to Infra docker images.

Note that Python 3.12 has a breaking change in the installation.

• distutils module itself is removed at Python 3.12 via PEP-632 in favor of packaging package.
• Apache Spark 4.0.0 is ready for Python 3.12 via SPARK-45390 by removing distutils usages
  • [SPARK-45390][PYTHON] Remove distutils usage #43192
• However, some 3rd party packages are not ready for Python 3.12. So, this PR skips those kind of packages.

Why are the changes needed?

This PR is a preparation to add a daily Python 3.12 GitHub Action job later for Apache Spark 4.0.0.

As of today, Apache Spark 4.0.0 has Python 3.8 ~ Python 3.11 test coverage.

• Python 3.9 (Main)
  • https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml
• PyPy3.8, Python 3.10, Python 3.11 (Daily)
  • https://github.com/apache/spark/actions/workflows/build_python.yml

Does this PR introduce any user-facing change?

No.

How was this patch tested?

$ docker run -it --rm ghcr.io/dongjoon-hyun/apache-spark-ci-image:master-6939290578 python3.12 --version
Python 3.12.0

$ docker run -it --rm ghcr.io/dongjoon-hyun/apache-spark-ci-image:master-6939290578 python3.12 -m pip freeze
alembic==1.12.1
blinker==1.7.0
certifi==2019.11.28
chardet==3.0.4
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
contourpy==1.2.0
coverage==7.3.2
cycler==0.12.1
databricks-cli==0.18.0
dbus-python==1.2.16
distro-info==0.23+ubuntu1.1
docker==6.1.3
entrypoints==0.4
et-xmlfile==1.1.0
Flask==3.0.0
fonttools==4.45.0
gitdb==4.0.11
GitPython==3.1.40
googleapis-common-protos==1.56.4
greenlet==3.0.1
gunicorn==21.2.0
idna==2.8
importlib-metadata==6.8.0
itsdangerous==2.1.2
Jinja2==3.1.2
joblib==1.3.2
kiwisolver==1.4.5
lxml==4.9.3
Mako==1.3.0
Markdown==3.5.1
MarkupSafe==2.1.3
matplotlib==3.8.2
mlflow==2.8.1
numpy==1.26.2
oauthlib==3.2.2
openpyxl==3.1.2
packaging==23.2
pandas==2.1.3
Pillow==10.1.0
plotly==5.18.0
protobuf==4.25.1
pyarrow==14.0.1
PyGObject==3.36.0
PyJWT==2.8.0
pyparsing==3.1.1
python-apt==2.0.1+ubuntu0.20.4.1
python-dateutil==2.8.2
pytz==2023.3.post1
PyYAML==6.0.1
querystring-parser==1.2.4
requests==2.31.0
requests-unixsocket==0.2.0
scikit-learn==1.3.2
scipy==1.11.4
setuptools==45.2.0
six==1.14.0
smmap==5.0.1
SQLAlchemy==2.0.23
sqlparse==0.4.4
tabulate==0.9.0
tenacity==8.2.3
threadpoolctl==3.2.0
typing_extensions==4.8.0
tzdata==2023.3
unattended-upgrades==0.1
unittest-xml-reporting==3.2.0
urllib3==2.1.0
websocket-client==1.6.4
Werkzeug==3.0.1
wheel==0.34.2
zipp==3.17.0

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

Could you review this PR, @HyukjinKwon ?

[dongjoon-hyun]

Thank you, @HyukjinKwon !

[dongjoon-hyun]

Merged to master.