Run multi-arch builds on merges to main and release nightly builds

[hiddeco]

Based on a discussion in fluxcd/image-reflector-controller#151

What

1. Run multi-arch container builds on merges to main
2. Push our nightly builds as an "early release"

Why

Running multi-arch builds for every pull request is too aggressively burning CI minutes, but running them "nightly" for conformance may be "too late". Running them on merges to main should give a relatively prompt responsive if anything (32-bit) related is wrong with the thing you just merged, while heavily reducing the feedback time for pull requests.

In addition, people do not have an easy way to access "early releases" at the moment except for when we manually trigger a "release candidate". Based on maintainer experience with fluxcd/flux and fluxcd/helm-operator, this could be solved for most people by releasing a "nightly" build every day with the state of main.

Where

• fluxcd/source-controller
• fluxcd/kustomize-controller
• fluxcd/notification-controller
• fluxcd/helm-controller
• fluxcd/image-automation-controller
• fluxcd/image-reflector-controller

[stefanprodan]

For flux v1 we've created a separate image repository for the nightly builds, I think we need to do the same here for each controller.

[hiddeco]

There is another divergence with Flux v1 in that we create a "nightly" for every (merge) commit there, instead of running it nightly.

This means that there is something else we need to keep in mind: if there is no activity on the repository, the commit will not have changed and we can thus not blindly push e.g. fluxcd/<controller>-nightly:<commit sha>. Instead, we probably want to use something like fluxcd/<controller>-nightly:<commit sha>-<build id>.

[squaremo]

Could we publish nightlies like flux 1, i.e., on every merge to main? For some subprojects I'd expect less than one merge a day, and for some it might be a handful, but I'd expect on average it would be about the same rate overall.

[hiddeco]

The thing you would miss then are upstream dependency changes that result in a failing build, this can for example happen for the source-controller and/or image-automation-controller that have version selectors for some but not all packages, and a more complex build process in general.

[squaremo]

this can for example happen for the source-controller and/or image-automation-controller that have version selectors for some but not all packages

We should probably lock those down so they only get changed explicitly, and use -mod=readonly in builds.

[stefanprodan]

@squaremo we fail a build if go.sum changes

[hiddeco]

Not talking about Go Module dependencies, but the ones in the container, e.g. https://github.com/fluxcd/source-controller/blob/main/Dockerfile#L4

[squaremo]

Not talking about Go Module dependencies, but the ones in the container, e.g. https://github.com/fluxcd/source-controller/blob/main/Dockerfile#L4

Ah of course. Could we lock those down, or is it not practical or possible?

[squaremo]

we fail a build if go.sum changes

Yes true; it would be simpler to use -mod=readonly though ...

[stefanprodan]

-mod=readonly does not catches issues like stale CRD yamls or docs, what we do now is run make tidy, make crds, make docs, etc and if anything changed on disk we fail the build.

[hiddeco]

Ah of course. Could we lock those down, or is it not practical or possible?

We could lock those down, but we would then need to invent another mechanism to automatically update them (using e.g. Dependabot or Renovate).

Renovate would have my preference as it is capable of bundling updates, while Dependabot will create a pull request for literally every single dependency that has a newer version.