ci: Test on development snapshots of GCC and Clang

[real-or-random]

This adds development snapshots of GCC and Clang to our Dockerfile, and -- as a prototype -- checks if CI likes this. I will later add a commit that actually defines some separate jobs.

It's not clear yet if it's a good idea to have these in a Dockerfile, or if this should be a separate Dockerfile, or no Docker CI environment at all. The advantage of the Docker CI environment is that the built Docker images are cached. But that's also the disadvantage. We probably want to rebuild the Docker image from time to time, and we can't do this automatically. The images are cached based on the contents of the Dockerfile. I think it's possible to nuke the cache from the web interface, but we'd need to do this on a regular basis then.

Moreover, it's not clear to me yet if these should be run as part of PRs and on master (as currently), or as cron/nightly builds or something else. I lean towards adding a few jobs to the current strategy (PRs+master) because I think moving some builds to nightly is an orthogonal question.

[sipa]

See #1316 for a fix for one of the issues detected in the CI run here.

[real-or-random]

@sipa Do you have any idea about the other failure? https://cirrus-ci.com/task/4554836482457600?logs=test#L252 I have no idea what GCC is trying to tell us here... I don't know which ret variable it is referring to? Maybe that's a real false positive, or at least a bug in the diagnostic message? I mean, it's an unreleased compiler in the end...

[sipa]

@real-or-random I can reproduce it locally, and can silence the warning by memset'ing the precj array in the function, but don't understand the comment otherwise. Note that it's in a mode where the precompute binary is being compiled with -DVERIFY, so it may include magnitude/normalize fields.

[real-or-random]

I think it's possible to nuke the cache from the web interface, but we'd need to do this on a regular basis then.

According to my experiments, clearing the cache is possible. (Only using a workaround, see cirruslabs/cirrus-ci-docs#544, but I doubt that's a problem.) I guess that's good enough for starters. We could do it every few weeks, or easier, in every IRC meeting. What do you think?

[real-or-random]

@real-or-random I can reproduce it locally, and can silence the warning by memset'ing the precj array in the function, but don't understand the comment otherwise. Note that it's in a mode where the precompute binary is being compiled with -DVERIFY, so it may include magnitude/normalize fields.

After some deeper digging, I've managed to find a commit that improves our code and makes GCC happy here at the same time.

[hebasto]

Time to rebase? :)

[real-or-random]

Ready for review.

Moreover, it's not clear to me yet if these should be run as part of PRs and on master (as currently), or as cron/nightly builds or something else. I lean towards adding a few jobs to the current strategy (PRs+master) because I think moving some builds to nightly is an orthogonal question.

What I did now is something else:

• Run all (14) Linux x86_64 jobs not only on gcc and clang, but additionally with gcc-snapshot and clang-snapshot (this gives us 28 additional jobs)
• Instead of running the same 14 jobs on macOS jobs for both gcc and clang, reduce the macOS jobs to a more "curated set" of 8 jobs (this removes 28-8 = 20 jobs)

In total, the number of jobs is increased only by 8, and it is 89 after this PR. My rationale here is that I think it makes sense to test many jobs on snapshots, and I didn't want to increase the number of jobs > 100.

Open to other suggestions and bikeshedding here. However, we should reconsider the macOS jobs anyway now that we can get Linux on native ARM64 too (#1163). The Linux tests are better (we get Dockerfile support, valgrind works, etc...) It still makes sense to test building on macOS, and it makes also sense to run a few tests there given that Apple's clang is a bit modified. But given that our actual code has almost zero interaction with the OS, I'd prefer to run just a handful jobs on macOS, and see if we can get more ARM64 testing on Linux instead. For example, we could move some of the x86_64 jobs to ARM64.

[hebasto]

What the process is suggested to update compiler snapshots in the Docker image?

UPD. Asking because the PR description seems a bit vague about that.

[real-or-random]

What the process is suggested to update compiler snapshots in the Docker image?

Oh right, I should have pointed this out. As I said above, the image is cached based on a hash of the Dockerfile. So unless we change the Dockerfile,

As I wrote here, it's possible for a maintainer to trigger a manual rebuild by clicking "re-run" on a Docker task. My suggestion is simply to do this manually, say in every (or every second) IRC meeting.

A similar alternative is that we have some kind of version number in the Dockerfile that we increment manually. We could again do this every two weeks or every four weeks. This is a bit inelegant because it needs a (trivial) PR every time we update. But the advantage of this is that we can postpone merging that PR if the update breaks...

I'm open to suggestions to automate this, but I currently don't see a nice way to force this using Cirrus CI features.

[hebasto]

We could again do this every two weeks or every four weeks.

Maybe do this when new packages are available? For instance, the last time gcc snapshot was updated was on 2023-04-19.

[real-or-random]

Maybe do this when new packages are available? For instance, the last time gcc snapshot was updated was on 2023-04-19.

Right, for GCC. The Clang packages are usually just a few days old: https://apt.llvm.org/, so I think something between "three months" and "a few days" is appropriate.

[jonasnick]

What I did now is something else:

sgtm

In total, the number of jobs is increased only by 8, and it is 89 after this PR.

As far as I can see, it's 91 now.

[jonasnick]

This installs gcc version 13.0.1 20230416. There are newer gcc snapshot sources available from the official mirrors, e.g., gcc-14-20230625. I suppose you didn't pick this because we don't want to get into the business of building the compiler (for now)?

[real-or-random]

Hm, yeah, it seems that this build is already ~2.5 months old.

I suppose you didn't pick this because we don't want to get into the business of building the compiler (for now)?

Indeed.

The gcc team publishes development snapshots in source form (what you linked to). https://jwakely.github.io/pkg-gcc-latest/ picks these up and provides binary packages (what we install here). I wonder if there's a regular cadence. Asked here: jwakely/pkg-gcc-latest#14

[jwakely]

It's supposed to be every week.

[real-or-random]

GCC 14 snapshot binaries are available now.

[hebasto]

re: #1313 (comment)

I suppose you didn't pick this because we don't want to get into the business of building the compiler (for now)?

Suggesting to do not rely on the https://github.com/jwakely/pkg-gcc-latest project at all.

Here is a proof of concept of building gcc from the source:

• CI builds: https://cirrus-ci.com/build/5250711244505088
• hebasto@73f6886:

--- a/ci/linux-debian.Dockerfile
+++ b/ci/linux-debian.Dockerfile
@@ -26,9 +26,15 @@ RUN apt-get update && apt-get install --no-install-recommends -y \
 
 WORKDIR /root
 
-# Install gcc snapshot
-RUN wget --progress=dot:giga --content-disposition https://kayari.org/gcc-latest/gcc-latest.deb && \
-    dpkg -i gcc-latest_*.deb && \
+# Build and install gcc snapshot
+ARG GCC_SNAPSHOT_VERSION=gcc-14-20230702
+RUN wget --progress=dot:giga --content-disposition https://gcc.gnu.org/pub/gcc/snapshots/LATEST-14/${GCC_SNAPSHOT_VERSION}.tar.xz && \
+    tar xf ${GCC_SNAPSHOT_VERSION}.tar.xz && \
+    apt-get update && apt-get install --no-install-recommends -y libgmp-dev libmpfr-dev libmpc-dev flex && \
+    mkdir gcc-build && cd gcc-build && \
+    ../${GCC_SNAPSHOT_VERSION}/configure --prefix=/opt/gcc-latest --enable-languages=c --disable-bootstrap --disable-multilib --without-isl && \
+    make -j $(nproc) && \
+    make install && \
     ln -s /opt/gcc-latest/bin/gcc /usr/bin/gcc-snapshot
 
 # Install clang snapshot

One of the benefits is building for C language only.

[real-or-random]

I suppose you didn't pick this because we don't want to get into the business of building the compiler (for now)?

Suggesting to do not rely on the jwakely/pkg-gcc-latest project at all.

Hm, yeah. I assumed it's more hassle. If it's just these 6 lines and runs in ~30 min more, then we should indeed consider this.

--without-isl

AFAIU this makes some more optimizations possible, which I think are only enabled at -O3. But I'm not sure and who knows if some of these will later be enabled in -O2. Is it difficult to build with ISL support?

[jwakely]

AFAIU this makes some more optimizations possible, which I think are only enabled at -O3.

No, they're not enabled by -O3, they're only enabled if you explicitly request them (and if GCC was built with ISL).

But I'm not sure and who knows if some of these will later be enabled in -O2.

They won't be.

Is it difficult to build with ISL support?

No, but the additional optimization passes it enables are not used by default and do not improve performance of most code anyway.

[hebasto]

Agree with #1313 (comment).

To be precise, ISL enables optimization options as follows:

• -ftree-loop-linear
• -floop-strip-mine
• -floop-block
• -fgraphite-identity

None of them are managed by the -O<N> options.

[real-or-random]

Force-pushed and included @hebasto's suggestion.

[jwakely]

This will break at the weekend, when the LATEST-14 symlink no longer points to the 20230702 snapshot. It will point to a different date, and so the name of the tarball will be different.

[jwakely]

You could replace the LATEST-14 directory with 14-20230702 which will not be invalidated every week, but then your CI will just be using the same snapshot forever.

[thesamesam]

Or just use the ones built for this purpose 😄

[jwakely]

It also seems quite wasteful to keep building the same snapshots again and again, but that is on brand for bitcoin.

[real-or-random]

This will break at the weekend, when the LATEST-14 symlink no longer points to the 20230702 snapshot. It will point to a different date, and so the name of the tarball will be different.

Oh sure...

You could replace the LATEST-14 directory with 14-20230702 which will not be invalidated every week, but then your CI will just be using the same snapshot forever.

Indeed. Hm. The ability to pin a specific snapshot is probably useful. This means we'd need to update the dockerfile regularly (e.g., once per month) with a PR. I think that's okay. It's just a bit inconsistent if we don't do the same for the clang snapshots...

For now, I simply replaced this with a version that downloads the latest snapshot. (Using wget's recursion feature that recurses into downloading links.) I think any approach will be good enough as an initial attempt. We'll need to see how often we run into problems.

Or just use the ones built for this purpose smile

You mean those at https://jwakely.github.io/pkg-gcc-latest/ ?

[real-or-random]

It also seems quite wasteful to keep building the same snapshots again and again,

Note that our CI infrastructure builds this once as a docker image, and then every CI jobs runs in a new container using this prebuilt image, which has the GCC snapshot preinstalled then. The docker image is only rebuilt if we update the Dockerfile, or otherwise trigger a rebuild manually. That's a pretty neat feature of Cirrus CI, see https://cirrus-ci.org/guide/docker-builder-vm/#dockerfile-as-a-ci-environment

[hebasto]

ACK 53980df.

https://api.cirrus-ci.com/v1/task/6462209514012672/logs/test.log:

...
+ gcc-snapshot -v
Using built-in specs.
COLLECT_GCC=gcc-snapshot
COLLECT_LTO_WRAPPER=/opt/gcc-snapshot/libexec/gcc/x86_64-pc-linux-gnu/14.0.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: ../gcc-14-20230702/configure --prefix=/opt/gcc-snapshot --enable-languages=c --disable-bootstrap --disable-multilib --without-isl
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 14.0.0 20230702 (experimental) (GCC) 
...

https://api.cirrus-ci.com/v1/task/5054834630459392/logs/test.log:

...
+ clang-snapshot -v
Debian clang version 17.0.0 (++20230624112259+eaaacc3c651e-1~exp1~20230624112316.43)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc-cross/i686-linux-gnu/12
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
...

[real-or-random]

Added checking of SHA512 checksum for the GCC download.

@hebasto sorry to invalidate your ACK,

[hebasto]

re-ACK 981e5be

Added checking of SHA512 checksum for the GCC download.

Nice.

[jonasnick]

ACK 981e5be