Configuration in pyproject.toml

[henryiii]

I noticed while reading #179 (comment) that there was a thought at some point to drive (most) of cibuildwheel from a common config file. I'd like to play with an outline of what that might look like, and get some feedback and design discussion going on it (not for immediate implementation, but maybe a 2.0 idea?)

First, the name. pyproject.toml would be the natural name, since it's supposed to be the one place to control Python package configuration. Since we already have environment variable overrides, I don't think we need a .cibuildwheel.toml or similar "alternate" config file for users who just don't want to use pyproject.toml.

Second, the sections. We are suppose to put things in tool.cibuildwheel.*, so I'd propose four sections: tool.cibuildwheel.global, tool.cibuildwheel.windows, tool.cibuildwheel.linux, and tool.cibuildwheel.macos, with the same structure as the current CIBW_* and CIBW_<PLATFORM>_* options.

Third, the names. I would power everything though the current interface, so each variable CIBW_<SOME_NAME> would have a toml setting with <some-name>, lower case and with dashes.

Fourth, precedence. I would always let environment variables override toml variables, since you might have to override an existing setting, or you might need a per-job setting for something like archs.

Fifth, scope. We can easily add opt in/out, so should we evenly add everything that is a current environment variable, or should some settings be opted out?

Sixth, special case for CIBW_BUILD? Unlike all the other variables, BUILD, and to a lesser extent, SKIP, and possibly ARCHS are for driving the current job, not only describing "what can be done". If you want to split the built to avoid the short total job time (on things like Travis) or utilize the 10+ parallel builds on Azure/GHA, you have to use these variables in a matrix to do it. If you want to run two jobs with the same OS, you have to use one of these to not get exactly the same result (probably becoming more important with multiple arches and macOS ARM). So should we do something special here? Maybe selection by variables could be applied on top of configuration versions (but then there would be no way to add a skipped job without changing the pyproject.toml). Or maybe we could keep these out of the configuration file entirely; that's expecially natural for BUILD/SKIP, which are not even valid in the per-OS part anyway. But for simple packages that do fit within a single job's time, they might want to only use the file?

Benefits:
Most of the configuration could be completely independent of the CI, so you wouldn't have to depend on the method a CI provides for global environment variables, and could likely switch CI's more easily. If you support 2+ CIs, you no longer have to specify things multiple times.

Cons:
Splits the config into a separate file, forcing users to look in two places to find config. But this is a common problem for lots of packages (pytest, in fact, pretty much everything except black).

[Czaki]

Environment variables should still overwrite config files. For example, I prefer a separate job for x86_64 and i686 for imagecodecs package (because of the time of build)

[henryiii]

Fully agree, that's point 4. Also, if you are careful, BUILD (and maybe SKIP/ARCHS) tend to drive individual builds, everything else likely can be put in a file, but BUILD will often be unique per matrix job.

[henryiii]

Also, you are not always building your own package or sometimes the build is just in a separate repository, so you need overrides in those cases too.

[joerick]

So, yes this has been something I've been toying with in my head for a while. But the main reason I've not actually done it is because I haven't seen a situation where it obviously better then env vars! When I looks at cibuildwheel configs around GitHub (search GitHub for CIBW_SKIP or similar) I see a lot of people using it alongside other CI features, like matrixes, or a test step before. And I think that if you're playing with your CI config, you're better of configuring cibuildwheel there. Mixing the two is probably a recipe for confusion.

I'm forgetting something... Ah, I remember! Local running. You keep all your config in pyproject.toml , and then do cibuildwheel --platform linux to test it without pushing to CI.

Hmm. Is that worth it? It does cost us in simplicity. Documenting options gets harder. Our options suddenly have two names. Our examples have two syntaxes. We can no longer say simple things like 'cibuildwheel is configured with environment variables'.

Maybe there's another upside to this that I've forgotten? (Aside from neatness and less ALL_CAPS)

[henryiii]

Might become less important over time, but sharing config between Travis & GHA (for example) is another use case. In general "driver" code should be in the CI config, and static details about what you support or need could (should?) be in the pyproject.toml

It costs almost nothing at all in complexity for our code, but as you are correctly pointing out, the complexity cost is in docs/examples/usage/tests. There are lots of programs with multiple configs (pytest, isort, pretty much everything but black), so I don't think it's insurmountable. Is it a good idea, though? That I don't know. Looking at some examples or variables might help.

Just having python-requires might be half way there. ;)

[chrisantonellis]

Also, sharing config between different "builders". I think defining the configuration in a builder-agnostic (not meant for cibuildhwheel specifically) allows other tools to more readily make use of this data, for example: a large organization that prefers to build and link against internally compiled system libraries.

imagine a package consumer who wants to know how a package is built so they can build it themselves, with their own tooling / automation etc. from this perspective, the configuration looks like a "recipe" and the user is most interested in swapping out python dependency versions, system dependency versions etc

the above user story is what motivates the design in this post: #564

[manylinux_build_specification.manylinux2014]
extra_base_system_repositories = ["http://foo.com/packages/"]
system_dependencies = ["foo-1.0.0", "bar-1.0.0"]
python_dependencies = ["foo==1.0.0", "bar==1.0.0"]
environment_variables = ["FOO=BAR"]
steps = [
  "./scripts/build_and_upload.sh --my_option"
]

[henryiii]

So let's run through the options and see what might be useful in a pyproject.toml:

• CIBW_PLATFORM: No

• CIBW_BUILD, CIBW_SKIP: Tricky one. I'd say no for BUILD, though listing unsupported systems in SKIP might be. I'd still say these are "driver" configs and probably not static ones.

• CIBW_ARCHS*: Probably not.

• CIBW_ENVIRONMENT*: Possibly.

• CIBW_PROJECT_REQUIRES_PYTHON*: This one is the most useful static value - but is already specifiable statically, so don't need a second way to write it most of the time.

• CIBW_BEFORE_*: With pyproject.toml, you often don't need these. Again, Possibly. Depends if they have static things you always need. The fact these are split by OS helps for static variables.

• CIBW_REPAIR_WHEEL_COMMAND*: Yes, this is likely static

• CIBW_MANYLINUX_*_IMAGE: Most of the time, yes. (In boost-histogram, I have two sets of manylinux builds, so in that case, it would be no)

• CIBW_DEPENDENCY_VERSIONS*: Sure, this could be a static choice.

• CIBW_TEST_COMMAND*: Before CIBW_TEST_SKIP or CIBW_TEST_SELECT #529 , this was how you skipped tests (all at once) if a test config was not supported. Now, this can be a static description of how to test. Yes. :)

• CIBW_*_TEST*: Yes, generally extras and such for testing are description.

• CIBW_TEST_SKIP*: This also could be static (similar to SKIP, this is more about listing what you currently do not support), or it could be dynamic (as new releases come out, maybe you can add testing without changing the source).

• CIBW_BUILD_VERBOSITY*: This one would be weird in a config, but I have been known to add -vv to pytest.ini...

That's not a "slam dunk" either way.

[henryiii]

This is a thread looking at some examples. Starting with the largest users (by number of stars), scikit-learn:

CIBW_ENVIRONMENT: OMP_NUM_THREADS=2
                  OPENBLAS_NUM_THREADS=2
                  SKLEARN_SKIP_NETWORK_TESTS=1
                  SKLEARN_BUILD_PARALLEL=3
                  MACOSX_DEPLOYMENT_TARGET=10.13
CIBW_BUILD: cp${{ matrix.python }}-${{ matrix.platform_id }}
CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: bash build_tools/github/repair_windows_wheels.sh {wheel} {dest_dir} ${{ matrix.bitness }}
CIBW_BEFORE_TEST_WINDOWS: bash build_tools/github/build_minimal_windows_image.sh ${{ matrix.python }} ${{ matrix.bitness }}
CIBW_TEST_REQUIRES: pytest pandas threadpoolctl
CIBW_TEST_COMMAND: bash {project}/build_tools/github/test_wheels.sh
CIBW_TEST_COMMAND_WINDOWS: bash {project}/build_tools/github/test_windows_wheels.sh ${{ matrix.python }} ${{ matrix.bitness }}

Hmm, didn't realize that was valid YAML with the multiline setting, nice. Okay, this is what you could pull out:

[tool.cibuildwheel.global]
test-requires = "pytest pandas threadpoolctl"
test-command = "bash {project}/build_tools/github/test_wheels.sh"
environment = [
    "OMP_NUM_THREADS=2",
    "OPENBLAS_NUM_THREADS=2",
    "SKLEARN_SKIP_NETWORK_TESTS=1",
    "SKLEARN_BUILD_PARALLEL=3",
    "MACOSX_DEPLOYMENT_TARGET=10.13",
]

This would have been much better if they didn't have so many commands with ${{ matrix.python }} and ${{ matrix.bitness }} in them, but putting the environment in here is actually rather nice. But if OMP_NUM_THREADS=2 (for example) depended on the CI system, it's no longer very helpful.

Score from -5 to 5

Helpfulness Score: 1 (or -1 if environment was not pulled out)

[henryiii]

psutil:

      CIBW_TEST_COMMAND:
        PYTHONWARNINGS=always PYTHONUNBUFFERED=1 PSUTIL_TESTING=1 PSUTIL_DEBUG=1 python {project}/psutil/tests/runner.py &&
        PYTHONWARNINGS=always PYTHONUNBUFFERED=1 PSUTIL_TESTING=1 PSUTIL_DEBUG=1 python {project}/psutil/tests/test_memleaks.py
      CIBW_TEST_EXTRAS: test
      CIBW_SKIP: cp35-* pp*

This one is completely static, even the skip (though that one I'd probably leave off, and it would be easier with python requires). Would be just fine in a config file.

Helpfulness score: 4.

[henryiii]

Twisted: Used for windows only - and there's no config. At all. Pure defaults.

Helpfulness score: irrelevant, no config settings.

[henryiii]

Jumping down the list, for one of my projects, boost-histogram, test-extras and test-command would be useful, but only with the addition of CIBW_TEST_SKIP and/or test-skip (not sure whether I'd put that a ci file or pyproject.toml. The manylinux images would be useful eventually, but not until I dropped the dual modified-manylinux1 and normal manylinux2010 builds.

Helpfulness score: 2

[henryiii]

And, including another Scikit-HEP project that I didn't write, iMinuit:

CIBW_BEFORE_BUILD: "python -m pip install numpy --only-binary=:all:"
CIBW_TEST_REQUIRES: pytest
CIBW_TEST_COMMAND: python -m pytest {package}/tests

# Later, in matrix:
env:
      CIBW_BUILD: ${{ matrix.py }}-*

I don't actually know what the before-build is doing there, actually, this has a pyproject.toml. It really should be an environment PIP_ONLY_BINARY=all or however you spell it. Again, a good candidate for static specifications, only the CIBW_BUILD is needed in the matrix.

Helpfulness score: 4

[henryiii]

In working with scikit-build/cmake-python-distributions#141 , specifically scikit-build/cmake-python-distributions#139 (comment) this would have been fantastic. Helpfulness score: 5.

[henryiii]

Suggestion: how about refactoring loading values into a "config" style? We don't need to actually depend on a library for this (like confuse, which is the same idea but for YAML), but the structure might gain us a little cleanliness instead of losing a little.

The idea would be that there would be a cibuildwheel/resources/default.toml file that has all the default values in it. If we add a stage after loading the defaults where any matching keys found in pyproject.toml update the loaded keys, that would provide the user override ability discussed above. We could also load from .cibuildhwheel.toml if it exists. Finally, we load from environment variables if those exist.

# Usage
config_options = ConfigOptions(platform)
config_options.get_var("CIBW_TEST_COMMAND", platform_variants=True)

# Design
class ConfigOptions:
    def __init__(self, platform: str) -> None:
        self.platform = platform
        # Open defaults.toml and load tool.cibuildwheel.global, then update with tool.cibuildwheel.<platform>
        # Open pyproject.toml and update with tool.cibuildwheel.global, then update with tool.cibuildwheel.<platform>
        # Open .cibuildwheel.toml and update with tool.cibuildwheel.global, then update with tool.cibuildwheel.<platform>
        self.config_dict = ... # result of above
    def get_var(self, name: str, platform_variants: bool = False) -> Optional[str]:
        # Get and return envvar for name or f"{name}_{self.platform)" if platform_variants if a var exists
        key = name.removesuffix("CIBW_").lower().replace("_", "-") # Fix for Python < 3.9
        return self.config_dict[key]

I haven't thought this through completely yet, like if a "None" value should be stored in the defaults.

[chrisantonellis]

Crossposting for visibility: #564

I had the same idea and am suggesting a higher level approach for configuration that would live in pyproject.toml

[henryiii]

So it's about time to revisit this. Is there interest in a higher level approach? I'm pretty sure pip and build wouldn't be interested in dealing with the non-python parts, so it likely would be just us; not sure how important it is to build a standard, but would be happy to work with one if that's best.

One important consideration that is even more important now that manylinux_2_24 is out and musllinux is planned would be the handling of linux variants. Let's say CIBW_BEFORE_ALL_LINUX requires a package (or system_dependencies in #564). Up until now, that was likely a yum install. Now, it is an apt install if building for manylinux_2_24, or a apk add if building for musllinux (and very likely has different package names, etc). How should we handle this?

Edit: I notice that #564 had the manylinux version included in the section, which would likely solve this issue.

[henryiii]

Could we provide a {packageman} variable that would expand to apt, yum, or apk? Though I think you will still be stuck with an if in most cases, since packages don't have the same names (often split over -devel vs. -dev, etc). Not even sure if having the expansion would be that helpful. And it would be tricky to add, probably, since you'd have to perform the same sort of if inside the container, since it could be any one of the three.

[henryiii]

Edit: and add vs. install. Yes, probably best just do do the full if else.

[joerick]

Yeah, maybe you're right, something like $PM or {packageman} seems like a false abstraction, they are different tools with different interfaces.

Maybe a better solution would be docs example with a nice multiline string, like

CIBW_BEFORE_BUILD_LINUX: |
    if which -s apt; then
        apt install thing1 thing2
    elif which -s yum; then
        yum install thing1 thing2
    else
        apk add thing1 thing2
    fi

or in toml, it would be

before-build = """
if which -s apt; then
apt install thing1 thing2
elif which -s yum; then
yum install thing1 thing2
else
apk add thing1 thing2
fi
"""

I'm pretty sure the multiline thing should work, anyway.

---

As a side note, we should also add some multiline examples of CIBW_ENVIRONMENT, as I've seen some pretty gnarly configs out there, but I think that CIBW_ENVIRONMENT: >- should do the right thing.

[Czaki]

I good understand that environment variables will overwrite toml file values?

[henryiii]

Yes, that's the idea in #684.

[joerick]

🎉 This feature was merged and released (as an alpha). Check out v2.0.0a3. Docs here: https://cibuildwheel.readthedocs.io/en/latest/options/#setting-options