Unifying PLATFORM/ARCHS/BUILD/SKIP ?

[joerick]

I realised when working on #484 that if we had a clever way to set the default of CIBW_BUILD or to compose build/skip somehow, we might not even need CIBW_ARCHS at all. Really CIBW_ARCHS is a way to override some 'default' of build identifiers - it happens before CIBW_BUILD/CIBW_SKIP is evaluated. And we also have CIBW_PLATFORM, controlling the baseline assumption of platform before that.

Could there be some way to unify some of this? Perhaps an option like CIBW_SELECT, which had some kind of mini-language which specified both built-in (but overridable) platform/arch concerns and user-specified deselect/explicit select instructions? Alas I don't have any solid ideas for this, just a vague notion that there might be a unifying concept that might be interesting to discuss :P

[joerick]

More thoughts on this...

• Is it kind of strange that you can do CIBW_BUILD="cp36-manylinux_x86_64" cibuildwheel and nothing will be built - you have to do CIBW_BUILD="cp36-manylinux_x86_64" CIBW_PLATFORM=linux cibuildwheel - you were giving an explicit instruction but you had to know about PLATFORM before that would work.

• Composing rules is kinda strange... like, you can't reselect something that was previously deselected. I quite like how in .gitignore, you can deselect something, and then later reselect it with a more specific rule. Like:

  # ignore user config files
  /.editor/*.config
  # except the template file
  ^/.editor/template.config
  

[joerick]

I suppose another analogy is CSS rules. They have this notion of 'precedence' where user-agent rules are the least important, then your rules begin to be applied in order of how specific they are to a specific element.

That gives me a rough idea...

  CIBW_CONFIG: >
    * {
      test_command: 'pytest'
    }
    implementation:pypy {
      # disable PyPy builds
      enabled: false
    }
    host_platform:linux, wheel_arch:aarch64 {
       # enable aarch64 builds, but change the test_command there
       enabled: true;
       test_command: 'pytest --skip-long-tests';
    }

ah... maybe it's a little overwraught. Probably we don't want to compete with the CI's matrix functionality too much.

[henryiii]

I wouldn't go that far. I think PLATFORM/ARCH have a clear purpose - they describe the system you are willing to run on. Maybe we could just extend BUILD to support exclusions? It would be evaluated left to right, and a rule starting with ^ would be excluded. That would make SKIP unnecessary; any skip rule would be expressible as * ^rule1 ^rule2.

[YannickJadoul]

I think we should then probably turn this into a new option, CIBW_SELECT or so, to be backwards compatible?

[henryiii]

It could be backwards compatible this way unless you added ^cp27-* to the start (since that would suddenly start dropping Python 2.7; honestly, not a bad idea?). Though applying a selector after a filter could be problematic. ^cp27-* *-macosx_universal2 would match cp27-*-macosx_universal2, and turn it back on? (probably is not an actual match, but you get the idea?). We'd need a clear set of rules for how these interact.

[henryiii]

We could call the next version 2.0 to let people know that identifiers may need to be updated; would be a good opportunity to update defaults, like Python 2 & 3.5 being deselected by default.

[YannickJadoul]

(Already nitpicking on something that's not yet further than design phase, but, to avoid confusion with regexes, should we pick !cp27-* or ~cp27-*?)

[YannickJadoul]

^cp27-* *-macosx_universal2 would match cp27-*-macosx_universal2, and turn it back on? (probably is not an actual match, but you get the idea?). We'd need a clear set of rules for how these interact.

Yeah, exactly, that's what I'm a bit afraid of. CIBW_BUILD - CIBW_SKIP is (strangely enough) quite intuitive.

[henryiii]

I’m not a big fan of it (have had issues matching what I want - more of an issue with the yaml part though) but there’s a mini language here for file matching: https://github.com/actions/labeler

[henryiii]

The language is here: https://github.com/isaacs/minimatch

[YannickJadoul]

That's cool, but it's JS :-/ Do we need something as complex as this?

[henryiii]

I don’t think so. :)

[YannickJadoul]

CIBW_PLATFORM is only necessary in very rare occasions, though. Normally, you'd just expect that to be automatically determined. So from that perspective, I'm not too worried about @joerick's CIBW_BUILD="cp36-manylinux_x86_64" cibuildwheel example.
But maybe we wouldn't go with CIBW_PLATFORM, if we redesigned things from scratch, and maybe it is a bit of a historical/technical accident?

I am still a fan of a reduction of CIBW_ARCHS, though. As far as I'm concerned, just having a flag that says "yes, I know emulation is slow", and then selecting builds with CIBW_BUILD/CIBW_SKIP would work.

About the mini-language: that's going to be harder/annoying to cram into environment variables in a CI config, though? Now, everything normally fits onto one line. Making that more complex enforce making files, etc... :-/

On @henryiii's thing about merging CIBW_SKIP and CIBW_BUILD: I expect most often either SKIP or BUILD are used, but not together? Also, merging them into one might cause confusion on precedence (I quite like the current simple skip - build rule) ?

[YannickJadoul]

But maybe that's not a discussion for here. Apologies for hijacking this discussion, if this was off-topic ;-)

[YannickJadoul]

Also, the current system does not include cross-compiling. MacOS is cross compiling (well, sometimes, universal is a weird beast), and we might really want Linux cross compiling support someday. Something to keep in mind (not claiming to have a solution).

I see no tension there, actually. Even if you say "please allow emulation", you could still let cibuildwheel prefer cross-compilation if that's found?

[henryiii]

It needs to be selectable. You might prefer emulation if your package doesn't support cross-compilation because it has an entry point, for example. I don't think it needs to be fine grained, though. Maybe we can add an extension mechanism and cross compiling would be a plugin? Just want to keep in mind that eventually we might need more.

[YannickJadoul]

Yep. At any rate, I don't see how using --arch for both Linux emulation and mac cross-compilation (our current goals) would help here, and why this couldn't be done with CIBW_BUILD/CIBW_SKIP on the identifiers?

[henryiii]

The problem is upgrading to cibuildwheel should not suddenly cause macOS builds to produce three wheels, universal2, x86_64, and arm64. And even worse, it's in an operating system-dependent way; upgrading to macOS 11 might cause the issue later when you didn't even touch cibuildwheel. Maybe it could add one of the two by default; but that means your code needs to support ARM; and I don't think that's a given. If you bundle a shared library, you need to update those to match, for example. But at the least, it certainly should not produce all three! They need to be clearly opt-in.

It also should not suddenly start emulating, which is why we added --arch. The idea here is we could use the same mechanism to fix the macOS new-wheel problem too.

[henryiii]

Proposal (just thinking out loud here):

1. Add "all" to --archs, which turns on all arches, emulated, cross-compiled, whatever is possible, and then selectors are used to filter/pick.
2. Extend CIBW_BUILD with ^ skipping, applied left to right. If you place all skips to the right, that's the same as CIBW_SKIP today.
3. Items on the left with ^ gives a new "deselected by default" behavior, so we can deselect "uncommon" arch's right off the bat in the future (2.0).
4. We deprecate CIBW_SKIP, as it can be applied as a single expression now, dropping it in 2.0 to avoid confusing about how to do things.

For cibuildwheel 2.0, we add CPython 2.7 and 3.5 (if we still support it at that point) to the skip-by-default. This could be used to deselect either univeral2 & arm, or x86 and arm (leaving universal2 on by default) for 2.0 (before 2.0, we do not include univeral2 and arm in the "native/default" archs, only in "all", to avoid disruption).

So, for someone upgrading to 1.8, they don't need to do anything; they get identical behavior. But they can start using ^ instead of CIBW_SKIP (we could print a message). If they want to use new architectures, they can use --all then start being careful with their build selectors to not suddenly pick up new arch's that are unexpected, or they can set the ones they support via CIBW_ARCHS_LINUX. macOS would be highly recommended to set CIBW_ARCHS_MACOS="x86_64 universal2", maybe even printing a diagnostic warning (I don't really trust anyone to read logs, similar logic in #515).

When upgrading to 2.0, we start CIBW_BUILD with ^?p27-*. Users will have to be a little more careful; adding a selector like *-stuff-* would match on cp27-* and enable it, but I think this would still be a little nicer than being stuck with Python 2 by default. Maybe we could change the default to Universal2 here as well, filtering the other two so users would have to enable them manually via selector.

PS: Most packages that drop Python 2 support also drop pypy2 support, like NumPy. So, regardless of the fact PyPy isn't expecting it to go anywhere soon, library support is dwindling and it should likely not be in the default. In fact, we could even make pypy opt-in in this scheme....)

I haven't checked to see if we support {,} in selectors with the same meaning as on the shell, but that might be a good idea to help keep them compact.

[henryiii]

I think we've been throwing around rule ideas, but they are not all applicable. I think this last one is not that much more than the current BUILD starts with * unless given rule.

Although now, you can say "I never want to build Python 2!", put it in CIBW_SKIP, then select a subset later?

That's nice, actually, so a bit of a trade on possible issue vs. potentially useful.

Honestly, many of the example libraries I've checked just put this in a matrix, with CIBW_BUILD: cp36* and so for each supported version, and that's about it.

[joerick]

Hehe, you guys have been having fun with this one!

As I'm interpreting this, we still have --platform and --archs, but we now have a way to deselect things by default. Nice! My thoughts on this approach are that we'd have to figure out how we clearly communicate the default 'left side of BUILD'. The danger here is that if the default is kinda complicated, we're lumping that complexity onto every user. Right now, the default build is '*', and the default skip is '' - with this, things wouldn't be so simple.

I suppose, in practice, instead of setting a prefix for users, we could just document that 'Python 2.7 is disabled by default', and that it must be explicitly enabled to build. So maybe we'd have a table of build identifiers in the docs, and the cp27/pp27 ones are dimmed, meaning they must be explicitly selected before they build.

Another thought that comes to me is that to build 'only' a single config becomes a little confusing with this system. Currently, you can do CIBW_BUILD=cp38-macosx_x86_64 cibuildwheel to build just that one, with this new option, you'd have to do CIBW_BUILD="!* cp38-macosx_x86_64" cibuildwheel - i.e. maintain this 'set' idea when setting the option. That's a critique of the idea generally, but it could be mitigated by calling the option CIBW_SELECT, and deprecating both BUILD/SKIP.

(@henryiii you listed a few examples above, but I think some of them are wrong, as anything that builds 'only' Python 3.6 would need to be prefixed by !*)

I'm also a little concerned it might be a little bit clever? If the mental model really is to think of a set of build selectors moving left to right, it's probably too much.

p.s. I did indeed mean !, instead of ^ earlier. FWIW I think ! is more recognisable as 'not'.

[henryiii]

Not with either of the rules I outlined. They were completely backward compatible with today’s syntax. Unless you interact with a default. The first set of rules were nicer in that case, I believe, but more complex in implementation. I can try it later today.

[henryiii]

Being backward compatible also means extra complexity is opt-in.

[YannickJadoul]

The confusion about this reasonably simple example is (to me) showing there's still something lacking/unintuitive in this proposal.

I'm also a little concerned it might be a little bit clever? If the mental model really is to think of a set of build selectors moving left to right, it's probably too much.

And this, yes. Though I guess we'll also get used to that?

[YannickJadoul]

How crazy is this (counter-?, extended?) proposal, and what use cases did I miss?

1. Extending @henryiii's 1., replace the (unreleased) --arch ... flag, just add --emulate as an option, and let the selection mechanism of build identifiers do the job. Rationale: if someone adds --emulate, they will know to update CIBW_BUILD/CIBW_SKIP?
2. Keep CIBW_BUILD/CIBW_SKIP, but from cibuildwheel 2.0 (or whenever we want to drop Python 2 by default), require --py2 as a flag to "enable" these build identifiers. Importantly, this flag can always be added, because cp27-* still matches in CIBW_BUILD and CIBW_SKIP: it just changes the default!
   The same could be done for --pypy, if we decide that shouldn't be on by default (not a fan, but a different discussion). Again, the key is that this just changes the default.
3. Our universal2-wheel case could be solved with a --universal2 flag. By default, you only have cp39-macosx_x86_64 and cp39-macos_arm64, but --universal2 enables cp39-macos_universal2 to be selected/unselected by BUILD/SKIP.
   So if you want to build universal2 but not arm64, you'd pass --universal and add macos_arm64 to CIBW_SKIP.

[YannickJadoul]

I don't see the issue with 2 or 3 clearly defined, independent and orthogonal options either, though.

Adding a BUILD="*_x86_64" would suddenly start picking up python2, for example, but the ARCH's give us a way out of that, and the correct expression is BUILD="?p3*_x86_64"

But then you're suddenly pushing things to CIBW_ARCH, which is just as well an option that takes on another meaning. CIBW_ARCH was meant for special cases (emulation or cross-compilation, not turned on by default), not to replace/complement CIBW_BUILD, right!?

I really don't see the advantage of causing more confusion, there. Somehow, these defaults for BUILD and their sequential interaction seem to be too implicit, for me, and will be circumvented by BUILD="* ..." after users recover from the surprise that something was selected/unselected they didn't expect.

[henryiii]

But then you're suddenly pushing

Only new things (specifically, only the new universal2 arch). So, the note would be:

Version 1.8.0: New universal2 arch support on macOS 11! If you don't want to produce Universal2 arches, set CIBW_ARCHS_MACOS to x86_64 to restore classic behavior. If you want to produce only non-universal wheels, set CIBW_ARCHS_MACOS to x86_64 arm64. And, if you only want universal2 wheels (keeping in mind they require a brand new pip), set CIBW_ARCHS_MACOS to universal2.

Everything else is the same. You now can filter with CIBW_ARCHS for 64/32 bit on windows/linux, if you want to, but that's just a consequence of the design. This to be is controlling "exceptional/overall" decisions, while build selectors are about picking a python version or removing pypy.

[henryiii]

and will be circumvented by BUILD="* ..."

I was referring to the current design here, not the "smart" one we were discussing - currently "* anything" would select everything.

[YannickJadoul]

Oops, sorry, got that completely wrong! Seems I need some sleep :-)

[henryiii]

I'm still not sure I like adding universal2 based on macOS version before it's common (or at least out of beta on GHA); I was thinking of it more as a "2.0" sort of thing. Someone with macos-latest would randomly double the wheels and build time when macos-latest becomes default. But I guess someone who would do that (guiltily looks away) would deserve that.

And helping spur the rollout is a big deal, so I am okay with it.

[YannickJadoul]

Idea to approach this differently: can we set up a complex but realistic example of a build selection, and see how we'd like to write a config for it?

[henryiii]

Here's my proposal compared to the classic syntax, using ! for the negation. I didn't add {}, as it's missing from the current fnmatch only impl. too. Excuse my JS, I don't use it all that often. Couldn't we stick this in the docs, whatever we go with?

https://jsfiddle.net/henryiii/btuon0hy/1/show

I love #511 :)

import toml

t = toml.load("cibuildwheel/resources/build-platforms.toml")
bp = t["tool"]["cibw"]["build-platforms"]
print([v["identifier"] for n in bp for v in bp[n]])

[YannickJadoul]

That's pretty cool, yes!

Is it me, or is something going wrong with !cp27-* *-manylinux_*, though? I wasn't expecting e.g. cp35-macosx_x86_64 to still be selected after that?

Note: I don't know why I forgot before, but we do already have --print-build-identifiers (which will match exactly what's implemented, so we don't need to keep two versions). Of course, we can't really have that as interactive version on the website, but users can install an interactive version locally, already.

[henryiii]

I think having it right in the docs would be much more useful and used than installing it locally. We almost discourage that, since it installs to global locations if you try to run it.

That is correct. You did "all except cp27", then you added many linux. The exclusion goes at the end if you want classic "skip" style.

[YannickJadoul]

We almost discourage that, since it installs to global locations if you try to run it.

That's kind of true, yes, though that's mostly about running and not about installing. You also shouldn't be able to run it by accident (since platform doesn't get set). But you're right that there's not a lot of reason for users to have it installed.

That is correct.

Ugh, you're right, yes; like that, it works. (honest mistake, btw; not trying to prove any point, but I'm just not used to reading it, it seems :-) )

[henryiii]

I wonder how close this is? https://facelessuser.github.io/wcmatch/fnmatch/

[YannickJadoul]

Exclusion patterns are allowed as well. When exclusion patterns are used in conjunction with inclusion patterns, a file will be considered matched if one of the inclusion patterns match and none of the exclusion patterns match. If an exclusion pattern is given without any inclusion patterns, the pattern will match nothing. Exclusion patterns are meant to filter other patterns, not match anything by themselves.

Isn't that our current CIBW_BUILD - CIBW_SKIP rule? There doesn't seem to be any ordering there.

That could make sense to me, though, adding !pattern as an alternative to adding them to CIBW_SKIP. Just not sure if I like the idea of CIBW_SKIP being deprecated, but it would provide the possibility of having the same as now, but in a single environment variable.

[henryiii]

Yes, I believe it is.

[henryiii]

Looking over all the different options and different projects, I feel that the special-rule CIBW_BUILD, while not that hard to implement, is too hard to reason with; most users are not straining at the edge of what can be done with the current syntax already. It could be very powerful if a user bothered to learn it, but users don't and should be bothered to learn a custom mini match syntax just to use cibuildwheel. Users tend to do the simplest thing here.

I think one reason that many projects already split by python version is that GHA builds 10 jobs in parallel (officially anyway, it's more like 15+ in practice so far), so you can finish the wheels in a fraction of the time if you split it up more (and for larger projects, less likely to get the 6 hour limit).

So how about, with reasoning, basically iterating @YannickJadoul's proposal:

1. We continue with CIBW_ARCHS as we are now; "native" is the default, and native (could) include universal2 for macOS, rather like it already includes 32bit for 64bit on the other OSs. I think most users will find this ideal for enabling or playing with the new architectures, since they are using things like cp36* now in build and that doesn't have to change unless they want to split based on it, see the final comments. macOS 11 considers "universal2" to be native.
2. We add a CIBW_MIN_PYTHON flag, with the usual variants (and maybe command line flag? Don't know what the policy is for that). If it is not set, we read the value from PEP 621's location if set (eventually), and for now use 2.7 (27?) if not set. For CIBW 2.0, we can change the default to 3.6 (36?), but users could get at least 2.7 back by manually specifying this. This is mostly so we can keep the defaults "nice", while also supporting the full range that we can support. From 2.0 on, we raise the default minimum to always match the official Python EOL.
3. (Optional) We upgrade to using the wcmatch library; this gives {} syntax, which I love using on the shell (partially because I'm a fish user and tab completion works correctly when using it, so it's perfect for moves and such). cp{27,35}* seems like it should work. Ve can activate the BRACE and NEGATE flags; this is simpler than my example above (basically just BUILD + SKIP), and is already well documented so we don't have to try to teach people how to use it, we can just point at this. Eventually, if this works well, we could push users away from the SKIP flag.
4. (Optional) We could add a little javascript selection in the docs as shown above. I just think that's very handy and will be "flashy" :)

ARCH/MIN_PYTHON are about telling cibuildwheel what you are interested in supporting, and is something we can adjust over time to "ideal" defaults, unless a user overrides them, in which case they specifically want that thing. With SKIP, which is in this category sort of, user overrides blend CPython/PyPy, Python version, and OS-ARCH, so we can't control each default separately without the new variables. BUILD is more about telling it what "this" job is trying to build out of the total allowed set.

[YannickJadoul]

This all sounds pretty good to me! I think this is distilling the best of different proposals and visions. A few notes and thoughts:

I'm a fan of the {}-syntax. I think it will solve some of the more complex things from @henryiii's first proposal, while still keeping this "monotonic" property rather than applying things left-to-right. Possible complication about the implementation: {cp36,cp37}-*-{manylinux,macos}* suddenly results in 4 combinations, I'd expect? Also, what about wildcards inside of braces?

I can also see the use of ! inside CIBW_BUILD as alternative to putting it inside CIBW_SKIP (note: we need to figure out how this combines with the braces, but this seems reasonably straightforward?). I'm not sure I want to push users away from CIBW_SKIP, though. This could still be useful to e.g. set a global CIBW_SKIP ("I never want to build Python 2.7!") and set CIBW_BUILD inside the different matrix entries. But time will tell how users actually set up things (maybe CIBW_SKIP really goes out of fashion?), so I wouldn't worry too much about making a decision about this, yet. :-)

CIBW_MIN_PYTHON: except for 2.7, we have just been dropping old versions of Python anyway, a few months after their EOL (which makes it possible to drop workarounds, etc). So I'm wondering if that's going to be very useful on the long term?
Cfr. PyPy 2, I think that when the time's there, we should create a new discussion and drag in some of the PyPy contributors, and ask them what they prefer and what's going on? In the end, it's their call when they want to drop support or encourage bulding wheels for PyPy 2? Now that pip is officially dropping 2.7, I assume/hope things will start getting more clear about the future of PyPy 2 as well, so with a bit of luck, by cibuildwheel 2.0, this is already less of an issue?

Beware of Python 3.10, btw, when using 27, 35, etc? (Not sure what the version and ABI tags are planning to do, though; that's probably relevant to make decisions about this.)

ARCH/MIN_PYTHON are about telling cibuildwheel what you are interested in supporting, and is something we can adjust over time to "ideal" defaults, unless a user overrides them, in which case they specifically want that thing. With SKIP, which is in this category sort of, user overrides blend CPython/PyPy, Python version, and OS-ARCH, so we can't control each default separately without the new variables. BUILD is more about telling it what "this" job is trying to build out of the total allowed set.

I love this description/summary. I think I completely agree with these levels of description! Thanks for this clarity, Henry!

[YannickJadoul]

If you do something really unusual, you can always set the minimum required Python to 2.7 and then filter CPython. This is CIBW_MIN_PYTHON, not CIBW_MIN_CPYTHON, after all - it's referring to the language, not the implementation.

Agree on this, though, yes.

[YannickJadoul]

Finally, the main concern about the JS thing I have is that we need a way to keep this in check with the Python code. But I suppose it's been ages since we touched CIBW_BUILD/CIBW_SKIP, so we should be fine, here :-)

[henryiii]

except for 2.7, we have just been dropping old versions of Python anyway, a few months after their EOL

I think ideally we should match manylinux on what we support, but match Python for what is default (via this flag). I'm hoping we drop 3.5 the day manylinux does (it's on their radar). But MIN_VERSION should have already been bumped to 3.6 on September 13, 2020, based on EOL. 11 months from now we could bump it to 3.7. https://endoflife.date/python (Unrelated: CentOS 6 ended a month ago. The fact they are ending CentOS 8 before 7 is even sadder when you see it listed like this.)

Beware of Python 3.10, btw, when using 27, 35, etc?

Great use case for brackets! cp{39,310}-*. :) For the other stuff, I'd just do whatever wcmatch does. I'm sure they've thought about it more than I have. :) Wildcards have their original meaning inside square braces, I do know that. ([*] matches a literal star). If we did implement it by hand, you just expand anything in brackets by commas and recompile, and yes, it's an outer product. But I like libraries. :)

For PyPy2, I'm happy to keep support it, but I think it should always be via CIBW_MIN_PYTHON=2.7 as that's what it is. Things like NumPy not building PyPy2 wheels is already sealing it's fate as a default choice, I think...

[YannickJadoul]

I think ideally we should match manylinux on what we support, but match Python for what is default (via this flag). I'm hoping we drop 3.5 the day manylinux does (it's on their radar).

Well, yes, we can't really do anything else, because once manylinux drops support for a Python version, we can't update the pinned image, so we need to drop support in new cibuildwheel versions.

Beware of Python 3.10, btw, when using 27, 35, etc?

Great use case for brackets! cp{39,310}-*. :) For the other stuff, I'd just do whatever wcmatch does. I'm sure they've thought about it more than I have. :) Wildcards have their original meaning inside square braces, I do know that. ([*] matches a literal star). If we did implement it by hand, you just expand anything in brackets by commas and recompile, and yes, it's an outer product. But I like libraries. :)

Yes, agreed. I just meant that 310 is just annoyingly ambiguous. But nothing we can do about that if cp310 is the version tag...

For PyPy2, I'm happy to keep support it, but I think it should always be via CIBW_MIN_PYTHON=2.7 as that's what it is. Things like NumPy not building PyPy2 wheels is already sealing it's fate as a default choice, I think...

Agreed, here, yes.