Add more details about the Python build in sys.version

[vstinner]

Currently, it's not easy to guess how Python was built just by looking at python -VV output or sys.version.

I propose to enhance sys.version to include more details about how Python was configured for the build:

• debug or release build
• "shared" or "framework" build
• assertions enabled or not
• address/memory/undefined behavior sanitizer
• etc.

It should help me to more quickly identify if a buildbot was built in debug mode or release mode.

It should help to see how Python was optimized: with or without LTO and PGO optimizations?

It should help to quickly have an idea of which ABI is used: the "pystats" ABI is different than the "debug" than the "release" ABI.

Linked PRs

• gh-100086: Add build details to sys.version #100087
• gh-100086: Add build info to test.libregrtest #100093

[ned-deily]

I don't think this change should be made. It potentially affects all users of Python, for example, the REPL users, but is of no benefit to 99% of them. All of this information is readily available via multiple other mechanisms, in particular, via python3 -m test.pythoninfo or via sysconfig.get_config_vars(). And as implemented in the PR, the information is truncated when displayed due to the string format limitation in getversion.c and can make the header unnecessarily long:

Python 3.12.0a3+ (heads/py_build_str-dirty:94b93efeed, Dec 7 2022, 16:23:18, release framework bu) [Clang 14.0.0 (clang-1400.0.29.202)] on darwin

[vstinner]

One motivation for me is that many users don't use Python binaries from python.org or from their Linux packaging system, but build their own binary with their custom recipe and it's not trivial to guess how the binary was built. Was it properly optimized with LTO+PGO? Was it built in debug mode?

For a long time, the most popular Docker recipe to build Python used ./configure && make: no LTO, no PGO.

Today, I cannot recall if the macOS binary is optimized with LTO or not.

Recently, @JulienPalard shared his recipe to build a custom Python with few lines of shell code: compile-python.sh. I was surprised to see --with-pydebug in the middle of the script: why building Python in debug mode? Some users may use it without paying attention to this configure flag.

in particular, via python3 -m test.pythoninfo or via sysconfig.get_config_vars()

Well, these APIs are not great :-(

test.pythoninfo is designed to be used on the command line, it doesn't have a simple API to query info about the current Python.

sysconfig.get_config_vars() is hard to use: you have to guess which variable is the best to get compiler flags: make your choice in this long list, it can depend on how Python was built, in case of doubt you might have to check multiple variables. Moreover, once you have compiler options, you have to parse this string which is not trivial (I hate parsing shell). For LTO, do you have to check compiler flags or linker flags, hum?

[vstinner]

It potentially affects all users of Python, for example, the REPL users, but is of no benefit to 99% of them.

Ah? I'm not the 99%, I build Python on a daily basis and work on Python built in various ways. I know that :-)

I saw this "debug build" vs "release build" as an useful information, but maybe it should be omitted for the common case: "release build" string? What about "lto+pgo": is it useful for most users?

On my Fedora 37, sys.version is Python 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux.

String length:

• Python version: 8% (6 characters)
• space + Build info: 39% (30 characters)
• space + compiler name and version: 53% (41 characters)
• Total: 77 characters

To make this string shorter, the compiler part should be shorter: [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux.

[vstinner]

At the beginning, I started by modifying test.libregrtest to add some "build details" in the first lines of the test runner:

== CPython 3.12.0a2+ (heads/main-dirty:9dc787ea96, Dec 6 2022, 11:51:12) [GCC 9.4.0]
== Linux-5.15.0-1019-azure-x86_64-with-glibc2.31 little-endian
== cwd: /opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node-pthreads/build/build_oot/host/build/test_python_829492æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
Using random seed 7151121
0:00:00 load avg: 7.01 Run tests in parallel using 2 child processes (timeout: 15 min, worker timeout: 20 min)

These lines don't say if Python was built in debug mode or not. The test.pythonfo data is not far in CIs, but it's annoying to have to reach the test.pythoninfo step, unfold it, scroll (this is especially painful in the buildbot web UI), just to check if Python was built in debug mode or not.

The first line is coming from the code:

        print("==", platform.python_implementation(), *sys.version.split())

Since sys.version is used, I thought that maybe putting the info directly in sys.version can benefit to more users.

[vstinner]

About the REPL, python2 and pypy put the info on two lines which makes it more readable IMO:

$ python2
Python 2.7.18 (default, Aug 22 2022, 00:00:00) 
[GCC 12.2.1 20220819 (Red Hat 12.2.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

and

$ pypy3
Python 3.9.12 (2fc6706f5902, Oct 12 2022, 21:40:58)
[PyPy 7.3.9 with GCC 12.2.1 20220819 (Red Hat 12.2.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>> 

The single line format (sys.version) is also used by the python -VV command:

$ python3 -VV
Python 3.11.0 (main, Oct 24 2022, 00:00:00) [GCC 12.2.1 20220819 (Red Hat 12.2.1-2)]

[vstinner]

I wrote PR #100093 which only changes the Python test runner (libregrtest), so it doesn't change the Python REPL nor python -VV. It doesn't change sys.version, so the platform module doesn't need to be updated.

[vstinner]

Fixed by: 3c89202