Phase out LLVM versions older than 3.9.

[eddyb]

See #45225 (comment) for an old bug rearing its ugly head. cc @rust-lang/infra

[alexcrichton]

AFAIK we have no reason to stick to LLVM 3.7 other than "it's what we stuck to awhile ago". Along those lines I think it'd be fine to bump the minimum version to 3.9, it's already at least 2 releases old anyway.

I think so long as the builder with 3.7 pre-installed is updated to 3.9 we should be good to go!

[petrochenkov]

cc @sanxiyn, due to #37912, any reasons to keep compatibility now?

[Mark-Simulacrum]

It'd be good to reach out to distros as well, in case any of them need the LLVM 3.7 dependency, but I don't know how best to do that.

[aidanhs]

Looking at some popular distros and versions of LLVM available:

• Ubuntu 14.04, the oldest non-EOL version of Ubuntu (https://www.ubuntu.com/info/release-end-of-life), has LLVM 3.3/3.4/3.5/3.6/3.7/3.8/3.9 in default repositories. So they can just install the 3.9 package if they've been using one of 3.7/3.8.
• Centos 6(/RHEL 6) has no llvm in the default repositories and 3.4 in the EPEL repositories. So they're already unable to use a packaged version of LLVM.
• Centos 7(/RHEL 7) has llvm 3.4 in the default repositories and 3.9 in the EPEL repositories. So they already have to use EPEL and that version is fine.

So no reason to stick to LLVM 3.7 from that quick look.

[est31]

Which clang/gcc version does llvm 3.9 require? Is it newer or older than the one provided by centos 6? Also ping @infinity0 @cuviper

[cuviper]

I think EL6's gcc is already too old for even LLVM 3.7. For internal builds I've been using devtoolset-gcc and just the bundled rust-llvm.

For EPEL7 I use 3.9, as you said, and same for Fedora 25. F26+ and the coming rust-toolset for RHEL7 are all on 4.0. So a minimum of 3.9 is fine for me.

[infinity0]

Sounds good to me - the Debian and Ubuntu rustc package already uses the same specific version of LLVM as what rustc uses, just minus most of the rust patches. (As opposed to the default Debian system version.)

TBH I was not even aware that Rust supported LLVM 3.7. Quite often some tests break when I run the build against an unpatched-but-otherwise-same-version of LLVM that rust uses - currently 4.0 - sometimes on tier 1 platforms. I just noticed src/ci/ where there is one job to test builds against an Ubuntu system LLVM 3.7. I'm very surprised that this is even green, but given that you've had this for a while could I submit a PR to add another job that tests against Ubuntu's system LLVM 4.0?

[aidanhs]

@infinity0 my personal perspective is that tier 1 platforms implicitly include the LLVM version they're built with as part of the platform - I don't know if we want to gate on lots of LLVM versions on different distros given that the submodule is the only recommended one. Making sure things work on one vanilla LLVM with reasonably broad support (3.9 appears to be the inclination) seems like a reasonable compromise for the sake of convenience?

[infinity0]

Whose convenience, though? At the moment it's testing Ubuntu LLVM 3.7 from 16.04 and (if I understand correctly) the proposal is to change this to 3.9. But this doesn't match any distribution's current configuration - Ubuntu 16.04 already has LLVM 4.0 so the rustc package (which it takes from Debian) would eventually just use that instead of 3.9.

It seems very strange to gate merges on tests passing in Ubuntu's vanilla LLVM 3.9, when they don't necessarily pass on vanilla LLVM 4.0 which is the base of rust's fork, which is available on the newest (and not-so newest) suites of both Debian and Ubuntu.

CI testing with vanilla LLVM 3.9, would only help Ubuntu maintainers backport Debian unstable packages back to Ubuntu 14.04, and it would only help it in a very minor way because: (1) They build and test on 6 architectures. (2) They'd have to patch the Debian package to make it use an older LLVM version - not hard but a "paper-cut" time consumer. (3) The Debian package uses extra ./configure settings not present in this rust CI test, so the test results might still fail when they come to it. Then (4) if you're serious about this particular case, you'd have to continue supporting LLVM 3.9 all the way to early 2019, otherwise it doesn't really save them any work - i.e. they would still end up having to backport LLVM 4.0 to 14.04 when you drop support for LLVM 3.9.

Fedora currently ignore test failures, which I'm considering for Debian as well because it's too time-consuming to keep a clean sheet, and this logic would eventually make its way into Ubuntu. I believe all other distributions either don't run tests, or also ignore test failures. All of these factors IMO makes this whole "gating PR merges on LLVM 3.9 test success" a bit pointless.

To make it not pointless, it would be good to have some level of co-ordination between us all on which vanilla LLVM versions we should target and which tests failures (that occur on vanilla LLVM but pass on rust's fork) should be seen as unacceptable vs acceptable to ship. Debian/Ubuntu's LLVM is quite open to accepting backported patches that help rust - I've done this several times already. I'd guess other distributions would be similarly OK with these patches, if Rust formally states that they are "urgent". It's just really really hard to tell from a maintainer's perspective because I'm not super-involved with the details, and so my time incentives are pushing me to just simply ignoring test failures altogether.

[infinity0]

So as a proposal, maybe (a) add the ability for min-llvm-version to distinguish between vanilla LLVM and rust's fork, e.g. // min-llvm-version 4.0+rust and (b) prioritise vanilla-but-current LLVM versions in CI tests because that is what most distributions are targeting, and where the "main work" is done. Then if resources allow, perhaps (c) have a second CI test for an older version but only gate it on the build succeeding and ignore any test failures - because that is what distributions are doing in practise anyways.

[eddyb]

I don't understand why the extreme complication or why Ubuntu in particular matters.

[eddyb]

I proposed 3.9 because that's the first version which doesn't miscompile rustc itself aftee my changes. We need to test that minimum version, so we allocate one configuration to it (i.e. the one currently testing 3.7).

That's it. Nothing else in our policy changes. If I had any say, I'd suggest "just" dropping support for non-bundled LLVM as this sort of thing keeps happening, but some distros have minor (size?) concerns so it's not that easy.

[cuviper]

The docker image is based on Ubuntu - that's all that makes it special here.

I think all we really need is to make sure that the stated minimum LLVM works, for some definition of "works" that may or may not include tests. It's nicer if we do ensure testing too, of course. Using Ubuntu's LLVM to represent "vanilla" LLVM would be OK (is that what we use now?), but could carry its own quirks. An upstream vanilla LLVM is even better for this purpose.

Beyond that, I think it's fine to place the rest of the burden on distro packagers. That's our job as system integrators, to make sure packages work well together, especially when they deviate from an official upstream environment.

[cuviper]

(I don't think it's worth debating distro unbundling here.)

[petrochenkov]

@eddyb

I'd suggest "just" dropping support for non-bundled LLVM

Some people may have custom LLVM forks (e.g. for custom backends) which are typically based on some release of vanilla LLVM and not our bundled version.

[eddyb]

@petrochenkov We were actually sync'ing our LLVM updated with emscripten for a while (maybe we still are?), as an interesting data point. But yeah, sure, that's a valid niche usecase.

[arielb1]

The tests that break when an unpatched LLVM is used test for the absence of a few miscompilation bugs we have encountered. For each such bug, we make sure it is patched on upstream LLVM and backport the fix to our LLVM version. Most fixes don't appear to be very Rust-specific - they can be encountered through clang too.

I don't believe either us or a distro should regard a compiler as "supported" when it miscompiles code under semi-common situations, so distros should try to keep up on LLVM patches. Maybe we should try to get these patches in LLVM minor versions too, but there's still a desync between the Rust and LLVM release schedules.

In terms of testing, we just want to test the oldest version of LLVM to make sure our bindings aren't broken.

[shepmaster]

We were actually sync'ing our LLVM updated with emscripten for a while (maybe we still are?)

Yes, and that causes friction when upgrading LLVM, as we cannot "just" update LLVM anymore, we are blocked on emscripten to update first. This is also annoying for other forks (e.g. AVR) as we find we sometimes need to combine Rust's fork, our own changes as well as upstream LLVM, a trifecta of fun.

[infinity0]

There are other things in the LLVM fork like fastcomp and, at least in the past, various other optimisation patches. It would be easier for distros to include the critical patches in our LLVM packages if Rust distinguishes them from the non-critical ones I just mentioned. (Of course it is better if they made their way into a minor release, but it is good already that they are submitted upstream and do make their way into major releases.)

I'm not sure about @cuviper, but it's not my full-time job to package rustc for Debian - and a large part of this helps Ubuntu - so making the process more efficient than it is, is a major goal for me. I don't have time or knowledge to interpret all of rust's LLVM commit messages and recommend them for adoption by the Debian LLVM maintainer in advance; the only practical way for me is to run the rustc tests against the Debian LLVM package and see which tests fail. It's also not obvious to me which tests are critical ("miscompilation") or not. (Fedora ignore all test failures, which I'm seriously considering at this point.) So it would be good if the classification into critical vs non-critical, was done by Rust in an easily-visible way.

I don't understand why the extreme complication or why Ubuntu in particular matters.

The point was to explain why the current CI tests do not really help anyone. One could target some fanciful "abstract user" that might theoretically want to compile against Ubuntu LLVM 3.9 but also wants to link in LLVM statically and all the other default rustc settings that distributions don't use; but I think realistically it would be more useful to test against settings that distributions are actually using - if the main reason of "supporting LLVM 3.9" was to support distributions.

Now there is the concern that adding different settings to CI would slow down developers. However I've seen a fair few PRs in the past, to fix broken non-default settings, that would have been caught earlier by these sorts of CI tests that I'm proposing. (For example #42543, but I can dig others out if people want. In fact @brson even mentions extra CI tests there, which so far hasn't been done.)

we just want to test the oldest version of LLVM to make sure our bindings aren't broken.

That makes sense too, in which case the test for older versions could be relaxed to ignore test failures. Tagging critical LLVM fixes clearly would IMO help packagers more, than gating merges on tests passing on old nearly-unused LLVM versions.

[aidanhs]

Whose convenience, though?

To take a single personal perspective - when I make small changes to rustc and want to make sure my changes compile and a couple of relevant tests pass, I don't want to compile LLVM myself and am fine with the resulting rustc being 'unsupported'.

However I've seen a fair few PRs in the past, to fix broken non-default settings, that would have been caught earlier by these sorts of CI tests that I'm proposing.

Perhaps we're getting off track of the original proposal to upgrade LLVM to 3.9, given that what you're suggesting involves adding more builders (which presents additional difficulties and considerations). It might be worth raising another issue to talk about that, and just leave this one for the conservative bump to 3.9?

[cuviper]

@infinity0

I'm not sure about @cuviper, but it's not my full-time job to package rustc for Debian - and a large part of this helps Ubuntu - so making the process more efficient than it is, is a major goal for me.

Sorry, I don't mean to trivialize the effort required. I just worry about pushing too much of distro-level concerns back onto the Rust community.

I don't have time or knowledge to interpret all of rust's LLVM commit messages and recommend them for adoption by the Debian LLVM maintainer in advance; the only practical way for me is to run the rustc tests against the Debian LLVM package and see which tests fail.

Can you ask your LLVM maintainer to watch pull requests on the rust-lang/llvm repo? I'm sure that person is busy too, but it's actually pretty low traffic.

(Fedora ignore all test failures, which I'm seriously considering at this point.)

To be clear, I do try to pay attention to test failures, otherwise I'd save half the build time by not running them at all. It's just that I don't let failure block the rpm build. Instead I inspect the test results manually, but I admit I'm not always as thorough as I should be across all arches.

[infinity0]

Understood @ both of the above, sorry for hijacking this thread. I'll think of a better way of bringing this up elsewhere / some other time.