changes to master should be gated on tier-1 compilers building libcore correctly for the targets they support

[gnzlbg]

#62932 #62896 are regressions on tier-1 rustc nightly-x86_64-unknown-linux-gnu builds failing to build libcore for some of the targets that they are supposed to support cross-compiling to.

Changes to master should be gated on not introducing regressions here. Testing this is easy and takes very little CI time: once the toolchain has been build, just loop over all the targets that the rustc binary supports trying to build libcore with it. Building libcore for ~50 targets takes a couple of minutes.

The libc crate does that for stable and nightly toolchains here (build.sh). It might be possible to adapt that to rustc's CI.

Regressing on being able to build libcore for some target is not something that should land on beta, much less on stable Rust.

[cuviper]

Isn't that already tested by the fact that dist-* builders produce cross-compiled artifacts? I suppose that's only from an x86_64-linux host though, not all tier-1 hosts -- is this what you mean?

[cuviper]

I don't mean to discount the regressions you mention. I'm just pointing out that CI already does have some cross-compilation that would gate all branches. There may be some additional factor that affects those regressions differently, like the use of xargo.

[gnzlbg]

What I mean is that with a nightly-x86_64-unknown-linux-gnu rustc on a x86_64-unknown-linux HOST you can cross compile to dozens of targets, and for many targets the first step is building libcore for that target.

Isn't that already tested by the fact that dist-* builders produce cross-compiled artifacts?

No, because cross-compiled artifacts are not produced by each HOST for all possible TARGETS that can be cross-compiled to from that host.

I suppose that's only from an x86_64-linux host though, not all tier-1 hosts -- is this what you mean?

The dist builders that cross compile on the x86_64-linux hosts only cross compile artifacts for some of the supported TARGETS, e.g., the ones that ship a rust-std component via rustup. For all other targets AFAICT nothing is done, even though when building code for them the first thing that happens is that libcore must be built.

There may be some additional factor that affects those regressions differently, like the use of xargo.

What xargo does is build a sysroot for the target, starting by building libcore, and then if requested building liballoc, libstd, and its dependencies. The regressions mentioned occur because libcore fails to build. If libcore doesn't build, the target is pretty much useless.

[nikic]

The dist builders that cross compile on the x86_64-linux hosts only cross compile artifacts for some of the supported TARGETS, e.g., the ones that ship a rust-std component via rustup. For all other targets AFAICT nothing is done, even though when building code for them the first thing that happens is that libcore must be built.

rust-std is (afaik) not necessarily std, it can also just be core/alloc, and those are built for pretty much all targets, including the ones in the referenced issues. If you check the dist-various-1 build logs, there's a Building stage2 std artifacts (x86_64-unknown-linux-gnu -> thumbv6m-none-eabi) at some point, which in that case will only build core/alloc, not std, and later dist that.

[gnzlbg]

If that is the case, then I have no idea how those components have stopped to build.

[pnkfelix]

As far as I can tell this is a question for the infrastructure team. (If not, then I guess its T-compiler, but either way, it needs a team assigned.)

Nominating for prioritization.

[aidanhs]

So, I think there are two pieces here:

1. libcore fails to compile for thumbv6m-none-eabi #62932 relates to cross compiling using xargo - the only blessed way of building libcore is via x.py, which I believe is tested. Outside of that is generally best effort and subject to breakage
2. Segfault compiling libc on armv7-unknown-linux-gnueabihf #62896 relates to using an arm-host compiler to build libc (also doesn't mention libcore) - we don't test this compiler in CI because we don't have ARM hardware, so indeed it is subject to random bugs in LLVM

I don't think there's anything to do related to libcore on tier1 compilers left to do under this issue - let me know if I've got something wrong.

[gnzlbg]

Each Rust compiler is a cross-compiler. We do test, for some cross-compilers, that they are able to cross-compile to some targets, e.g. the Linux x86_64-unknown-linux-gnu compiles libcore, etc. for all targets for which we do ship these components with rustup.

AFAICT, we don't test that each cross-compiler can actually cross-compile to all the targets it support, e.g., we don't test anywhere that the i686-unknown-linux-gnu can cross-compile libcore for a powerpc64le-unknown-linux-gnu target, or that x86_64-unknown-linux-gnu can cross-compile libcore for those targets for which we do not ship libcore with rustup, but that the cross-compiler supports.

[aidanhs]

I can believe that we don't test all 528 (8 tier 1, 66 tier 2) possible build combinations 😄 though I note that this is a different issue again, i.e. it would be item 3 from "1. support xargo, 2. check validity of built cross-compiled artifacts, 3. attempt to build all possible cross-compilation targets"

On this third point, the logic is likely that the backend process should remain identical across architectures and one run through should be enough - indeed, in the case of the 2 in the list above, the problem would have presumably manifested from any host system.

All that said - if there is a combination that is particularly troublesome then it may be worth testing (assuming the right hardware is available).

[gnzlbg]

Again, the issue here is that the tier-1 cross-compiler for x86_64-unknown-Linux-gnu hosts cannot compile libcore for all the targets that it advertises as supported on stable Rust. You closed this issue because you believe that we test that x86_64-unknown-Linux-gnu compiles libcore for all targets it advertises as supported. If that were the case, it would be impossible for the errors that we are seeing on the libc side to happen. I tried to find a ci script here where that’s tested and couldn’t find any - we only produce artifacts for some of the targets that we advertise as supported, and we mostly only test this for x86_64-unknown-linux-gnu hosts, and not for any of the other tier-1 hosts. If that’s incorrect then please point me to the ci script where that is tested. If this is a wontfix then I did not understand the reason. rustc —target=i686-unknown-Linux-gnu is a stable option that I can pass to any tier-1 toolchain to cross-compile to a tier-1 target. If that stopping working isn’t a regression then it should IMO become an unstable option so that people are warned on their workflows that it can stop working anytime.

[Mark-Simulacrum]

We can't test everything. I think in this case, it doesn't make sense to spend our CI time testing this specific scenario. Ideally, perhaps we would, but it's sort of out of scope for our testing infrastructure at this point IMO. Of course, hearing about specific cases where this happens is good, and we can try and fix those as they come up - usually, that means the normal process: file an issue on this repo, and it'll get tagged with the regression labels and such as appropriate, and compiler team will decide how much we need to invest resources into fixing.

To be clear, I understand that it is not especially time consuming to test i686 -> x86_64 cross compilation of libcore. It might even make sense to add that specific cross compilation to one of our shorter test builders. However, the ask of "let's test all tier-1 targets for cross compilation to all tier 1 platforms" is not feasible for our CI.

[gnzlbg]

So it is unfeasible to spend 4 min per build job to test that cross-compilation targets that we advertise as supported actually work?

[gnzlbg]

As in, my 6 year old dual core laptop compiled libcore in 5 seconds. A tier-1 Rust toolchain with support for 50 targets would need to compile libcore 50 times. Assuming a 6 year old laptop CPU, but with a single core, that’s 250 seconds or ~4 min. I can only guess about how long this takes using gnu parallel on a 4 core CPU on Azure... but we might be talking about on the order of 1 minute or so.

[Mark-Simulacrum]

It is unclear to me exactly what you're proposing to test. However, I would personally be against this sort of general belief that we should be testing everything possible on our CI -- that's my read of this issue. To me it doesn't really matter that this specific ask might be relatively easy/low on resources. Ultimately these sorts of bugs will need to get caught in beta/nightly before they land on stable, which seems like a reasonable ask.

[nikic]

Reminder that the two bugs mentioned in the description would not be caught by what is proposed here: For thumbv6m-none-eabi we already build libcore and that succeeded -- xargo just uses different compilation option. The armv7-unknown-linux-gnueabihf issue only occurs when using it as a host compiler, not during cross-compilation.