How to build the images reliably?

[mhever]

We have nightly pipelines set up to build the latest stable releases for Ubuntu and Win flavors, every night Later on we use this night image as base image and build our customization on top of this blueprint image - building the win base image takes like 6 hours, then our customization takes only like 20 mins.

Now, this process fails too many times, and is in general unreliable. Basically, we have two set of problems:

• there are unreliable third party dependencies, like temurin downloads. Sometimes the download fails, sometimes it works. Usually, if the downloads fail, a restart of the packer build won't solve the issue, those download errors appear to be clustered.
• second kind of problem is some kind of external dependency change, blocking the building completely. Recent examples: kotlin download, or random SHA1 thumprint changes. For the latest kotlin one, there was a quite quick fix, that was merged to master, and that is wonderful, but until the next stable release, we can't build the latest stable release. Usually, we can emergency fix the build by doing a "local patch", for example using the latest release, but pull in the kotlin installer script from master.

This process is quite painful for us, too much time and effort spent on babysitting the latest stable build. Does anyone have any tips on dealing with these issues more efficiently?

[zioalex]

Same problem here.
Would be great to have a more reliable process to get stable release.

[mikhailkoliada]

Hey! While concern is understandable there are only 2 ways of mitigating this:

1. Cut the parts you do not need, for example SHA sum checking (then, beware that in a case of issues you may face we may not help you as we can not predict all the installation flaws, custom modifications, etc)
2. Keep up building images using only the main branch as we do not support fully idempotent builds and only support the cutting edge state of the repository, we guarantee that main is buildable and we provide fixes there as soon as we see things go broken, so it is always a good idea to subscribe to the runner-images repo notifications and perform an automatic git pull / git rebase before running ny custom build.

[crohr]

Most painful issue is indeed the Java temurin downloads. Easiest solution to make the builds stable is to just remove all but one version (or even all of them, and rely on the setup-java action). This has greatly improved stuff for me, since it looks like the artifactory repo might rate-limit downloads or just plain break every now and then.

I'm a bit confused about why your main build takes 6h though. On AWS it takes at most 45min.

Edit regarding timings: ok I see you are talking about the Windows image. No idea it was that slow!