[WIP] [DO NOT MERGE]: Bump Alpine version to 3.20

[rene]

Warning

This PR is still working in progress. It impacts basically all components of the system, so it requires an extreme careful review

Description

This PR updates Alpine version from 3.16 to 3.20. The general changes introduced by this PR are:

• Bumps pkg/alpine-base to Alpine 3.20
• Bumps pkg/alpine to Alpine 3.20
• Required packages are added to the package list (pkg/alpine/mirrors/3.20/)
• Patches not required anymore are removed
• A few patches required to build components with Alpine 3.20 are introduced
• Stop to build programs on Dockerfile that are now provided on Alpine 3.20 (as much as possible)

Important

Developers familiar with one or more packages under pkg/ : PLEASE, review the Dockerfile and the changes introduced by this PR on it. Test locally is also really and much appreciated!

TODO

Packages not yet updated:

• pkg/gpt-tools/Dockerfile
• pkg/vtpm/Dockerfile
• pkg/pillar/Dockerfile

Issue: #3994

[OhmSpectator]

I didn't look through carefully, but I already appreciate the way the commits are organized...

[rene]

I didn't look through carefully, but I already appreciate the way the commits are organized...

Thanks @OhmSpectator , probably this PR will be broke into more PRs when it gets ready.... let's see...

[OhmSpectator]

I'm curious if we do not use the Alpine package directly but via ledge/eve-alpine in most of the packages, will it make sense to use some tags in all the Dockerfiles?
So we can use something like FROM lfedge/eve-alpine:12.08 and define this 12.08 somewhere else with a proper hash...

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 17.51%. Comparing base (5894895) to head (97ea17b).
Report is 35 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #4092   +/-   ##
=======================================
  Coverage   17.51%   17.51%           
=======================================
  Files           3        3           
  Lines         805      805           
=======================================
  Hits          141      141           
  Misses        629      629           
  Partials       35       35           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[deitch]

I think we should take the opportunity to take a look at how we do apk package management in eve. The current system might be the best we can have, but at least we should consider alternatives.

Our current system looks something like this:

• eve-alpine-base is the base, which is FROM alpine (or busybox, doesn't matter), i.e. the upstream specific version
• eve-alpine first is FROM eve-alpine-base to get any packages it needs
• when we make changes to eve-alpine, e.g. adding packages or updating versions, it is then FROM eve-alpine@<previous-hash>

This leads to a chain: eve-alpine@10 is FROM eve-alpine@9 is FROM eve-alpine@8 etc.

The reason for this is two:

• the lack of guarantee that an upstream package won't go away, or a specific desired version
• the non-reproducible alpine package management. You can run apk add foo, or even foo@1.2.3-r5 and get completely different binaries

eve-alpine acts as our own caching system, guaranteeing both package availability and version reproducibility.

The downside is that it is complex. It is difficult to grasp ("eve-alpine is FROM eve-alpine?? What does that even mean?"), and depends on a long chain of earlier tags to rebuild.

Is there a way to make this simpler? Could we have eve-alpine simply be FROM alpine, where we intelligently update only the packages we want to update? eve-alpine continues to serve as the caching mechanism, but it is a single current image, easily reproducible, and manages the apk packages intelligently?

That is likely to require some more intelligent usage of apk packages and storage in eve-alpine. It may require enough that we do not benefit from the system, and it would be easier to stick with the current chain. That is fine, as long as we consciously choose to go down the path after evaluating alternatives.

Some things to think about.

At the eve-alpine build stage, how would we:

• cache packages?
• indicate "cache package foo"?
• indicate "cache package foo version 1.2.3"?
• indicate "cache a newer version of foo but keep the original one around too"?
• indicate "replace foo 1.2.3 in the cache with whatever is on the package distribution, even though it changed the underlying binaries"? Would we ever do this? What if there is a CVE that needs to be addressed?

At the usage stage (i.e. when we run apk add inside pkg/*/Dockerfile, i.e. installing from cache), how would we:

• indicate "add package foo"?
• indicate "add package foo of a specific version"?
• prevent unintended updates of specific binaries, e.g. if we ever replace a version (above) or if we add a new version?

For the latter, we probably can just enforce that package names in pkg/*/Dockerfile always must include a version. But what happens when we update eve-alpine?

[milan-zededa]

What version of modemmanager are we getting from Alpine 20?
Our mmagent expects fairly new version 1.22
What I like about building as opposed to using already built packages, is how easy it is to try a new version or a patch.

[milan-zededa]

Also note that in our build of MM we are disabling many features which are not needed or relevant to EVE. Such as sustemd or polkit (which btw. consumes lots of memory).

[rene]

@milan-zededa Alpine 3.20 provides ModemManager 1.22.0, exactly the same that was being built... Indeed, its shipped with full features, which includes polkit but also all the plugins for different modem brands... In this case I think it's ok to build MM in order to remove features not relevant for EVE, but we should avoid build packages by ourselves as much as we can, at one hand can be easy to test a patch, but at the other hand it can causes several other issues, specially when we want to upgrade something, library dependencies, differences for newer compilers... it's being really painful to manage the Alpine update for packages that we build stuff: some doesn't compile with newer gcc, some needs newer packages for dependencies on 3.20.... Alpine 3.20 is providing lots of new packages not supported before, let's take advantage of that....

Let's build only what is really necessary.... for wwan case I will keep the libraries from Alpine and build ModemManager... is that ok for you?

[milan-zededa]

Let's build only what is really necessary.... for wwan case I will keep the libraries from Alpine and build ModemManager... is that ok for you?

I think that's OK

[rene]

There are more packages that needs to be fixed (pkg/ipxe, pkg/uefi among others) in order to build with newer gcc... @famleebob is taking care of this update, I will drop this PR so he can push a new one with merged changes...