python: manylinux packaging jobs have been failing since the new version of cibuildwheel was released

[raulcd]

What happened?

The manylinux packaging jobs have been failing for the last 3 days, see example.

The previous success was using:

cibuildwheel version 2.20.0
...
Starting container image quay.io/pypa/manylinux2014_x86_64:2024.08.03-1...

the failures started since:

cibuildwheel version 2.21.0
...
Starting container image quay.io/pypa/manylinux2014_x86_64:2024.09.09-0...

Stack Trace

 + docker version -f '{{json .}}'
Traceback (most recent call last):
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/__main__.py", line 49, in main
    main_inner(global_options)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/__main__.py", line 204, in main_inner
    build_in_directory(args)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/__main__.py", line 352, in build_in_directory
    platform_module.build(options, tmp_path)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/linux.py", line 459, in build
    with OCIContainer(
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/oci_container.py", line 201, in __enter__
    _check_engine_version(self.engine)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/oci_container.py", line 121, in _check_engine_version
    raise OCIEngineTooOldError() from None
cibuildwheel.errors.OCIEngineTooOldError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/cp39-cp39/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/python/cp39-cp39/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/__main__.py", line 422, in <module>
    main()
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/__main__.py", line 51, in main
    message = e.args[0]
IndexError: tuple index out of range

How can we reproduce the bug?

No response

Environment/Setup

No response

[raulcd]

This commit was added to 2.21.1 which got released yesterday: pypa/cibuildwheel@0787a44

[raulcd]

cibuildwheel 2.21.1 doesn't fix the issue: https://github.com/apache/arrow-adbc/actions/runs/10894035189/job/30251871078#step:6:3140

[lidavidm]

It sounds intentional.

Would installing a newer Docker CLI in the CI environment help or would we need GitHub to upgrade it on the underlying image?

[lidavidm]

Interestingly glancing through the original PR they appear to have tested several CI environments...just not GitHub (unless I missed it)

[raulcd]

Unless I am mistaken it seems to be the version of docker we install here:
https://github.com/apache/arrow-adbc/actions/runs/10857739997/job/30134687179#step:6:229
from: https://github.com/apache/arrow-adbc/blob/main/ci/docker/python-wheel-manylinux.dockerfile#L29

[lidavidm]

Hmm, maybe I should unlink that image from upstream then

[paleolimbot]

I am not sure I can help with the existing system here, but if something closer to nanoarrow's setup could work I would be happy to take a stab at it:

https://github.com/apache/arrow-nanoarrow/blob/499d865c862606c69d8dca1a15fc0cf542696599/.github/workflows/python-wheels.yaml#L89-L132

I imagine the trick is efficiently building the driver once and reusing it for each Python version but very possible I am being naive here!

[lidavidm]

I wanted to try a newer manylinux, but that appears to run into containers/podman#10802

[lidavidm]

Traceback (most recent call last):
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/cibuildwheel/oci_container.py", line 112, in _check_engine_version
    client_api_version = Version(version_info["Client"]["APIVersion"])
  File "/opt/python/cp39-cp39/lib/python3.9/site-packages/packaging/version.py", line 202, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: '4.9.4-rhel'

[lidavidm]

...I think I'm going to pin cibuildwheel for now since their check is not quite up to par...

[lidavidm]

I filed pypa/cibuildwheel#2020