Support easy multi-arch build à la docker buildx

[Romain-Geissler-1A]

FEATURE REQUEST

Hi,

I am right now exploring how to easily create multi-arch builds in an easy way. Docker buildx seems to implement things which make the full thing rather simplified, and by checking the current man-pages of podman/buildah, it doesn't seem as easy.

1. One cool feature of buildx is that --platform is not a single platform, but a list of platform. Basically this allows to run one single "docker buildx build" command and in the end have a full manifest list of all the architecture you are interested about. Apparently the way this is currently done with buildah/podman is that you have to manually run all build for each arch individually (and handle parallelism yourself), then create the manifest and append all these arch images all by yourself. The buildx alternative seems like a syntactic sugar that is appealing and it would be great if buildah would support it as well.

2. In addition to having a list of supported platform, the other cool thing of "docker buildx build" is that thanks to a bit of builder configuration (see for example here https://medium.com/nttlabs/buildx-multiarch-2c6c2df00ca2 the "Option 3: Remote mode" paragraph) buildx can transparently execute each builds for each arch on a docker daemon running natively on this arch. Again, all this is done in a single command with just a bit of configuration. I tried to see if anything similar (automatic choose of remote daemon/remote builders based on arch) was possible with podman/buildah, but it doesn't seem so. Is this a feature that would make sense to see landing eventually ?

Cheers,
Romain

[cevich]

checking the current man-pages of podman/buildah, it doesn't seem as easy.

Having just completed a proof-of-concept github action that replaces multi-arch builds we're using buildx for, I completely agree with your sentiments. While I don't think squeezing in kitchen-sink-tool (like buildx) into buildah or podman is a good fit, there is certainly work needed on documentation and (perhaps) short-cut options to make life easier. Especially for/if somebody were interested in developing a buildx-like-tool for buildah or podman.

Besides better documentation, off-hand I'm imagining having like a --add-to-manifest option to buildah bud to help eliminate that extra step. Another idea could be transparent image->manifest conversion if the destination for a buildah bud --add-to-manifest=<FQIN> is a regular image. Would that help address some of your concerns? Do you have ideas for other short-cuts that could make the process easier?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@nalin has been working to improve our multi-arch support. Some buildx support has made its way into buildah already

[cevich]

Fantastic! I will very likely need help making use of these for containers/automation#84

I noticed there's now a build --manifest (great!), what else was added to make parallel multi-arch easier?

[cevich]

@rhatdan @nalind more specific question: Would you expect this to work yet (assuming QEMU is setup properly)?

for arch in amd64 s390x ppc64le arm64; do
    buildah bud --arch=$arch --manifest example.com/foo/bar:latest /path/to/Containerfile &
done
wait $(jobs -rp)
buildah push example.com/foo/bar:latest

(I'm remembering a race where the different arch's base images would frequently name-clash on pull)

[nalind]

The names of base images will still be reassigned as they get re-pulled for a given architecture, but as of 1.22 (or rather containers/common 0.42) that shouldn't break anything. The main branch expands --platform to accept a list, which should simplify this further in 1.23.

[cevich]

The main branch expands --platform to accept a list, which should simplify this further in 1.23.

Oh! So I can just do like buildah bud --platform "[some/thing, foo/bar]" --manifest example.com/foo/bar:latest /path/to/Containerfile and walk away for some coffee?

Wow, that would actually be really, REALLY cool!

[nalind]

It wouldn't be a JSON list, so the [] would be unexpected. Otherwise, yes.

[cevich]

Gosh that will be a HUGELY useful feature, thanks! Will it do the builds in parallel? Even if not...still very useful and more reliable than me trying to do it all in bash 😕

[nalind]

The --jobs option should still work correctly, yes.

[cevich]

Dang Nalin! There simply MUST be something buildah can't do...Next you'll tell me there's a --doughnuts option enabled n 1.23! 😆

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@cevich @nalind @Romain-Geissler-1A Where are we on this Issue? Do we have enough features to close it?

[cevich]

IMHO the --jobs option kind of takes the place of request 2. I'm not sure it's worth the code-bloat and effort to implement the buildah-farm concept. The --platform list is definitely working so request 1 is handled also. Yep, I say we can probably close this as complete-ish.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[arpitjindal97]

Multiarch isn't working for me

$ buildah bud --platform linux/arm64/v8 -t arpit .
STEP 1: FROM alpine:latest
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Getting image source signatures
Copying blob 552d1f2373af done
Copying config bb3de5531c done
Writing manifest to image destination
Storing signatures
STEP 2: RUN apk update
exec container process `/bin/sh`: Exec format error
error building at STEP "RUN apk update": error while running runtime: exit status 1

Dockerfile

FROM alpine:latest

RUN apk update

Buildah Version

$ buildah version
Version:         1.19.6
Go Version:      go1.15.9
Image Spec:      1.0.1
Runtime Spec:    1.0.2-dev
CNI Spec:        0.4.0
libcni Version:
image Version:   5.10.3
Git Commit:
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64

[nalind]

1.19 is from before --platform handled multiple values, but the Exec format error message indicates that your system can't run arm64 binaries. You either need to be on an arm64 system for that to work, or you need to have emulation set up so that your kernel will invoke an emulator to run binaries that aren't built for your native processor. If you're on Fedora, installing the qemu-user-static package should take care of that.

[cevich]

@arpitjindal97 the exec container process /bin/sh: Exec format error indicates the host kernel (linux/amd64) doesn't know how to run binaries for linux/arm64/v8. You need to...oof, clashing-posts...yeah, what Nalin said 😄

[rhatdan]

Do you have qemu-user-static package installed?

[arpitjindal97]

Thanks @rhatdan installing qemu-user-static package worked for me on my Linux machine.

Is there any easy way to install the same on MacOS ?

[rhatdan]

I think you can do
$ sudo rpm-ostree install qemu-user-static
inside of the machine.
https://developers.redhat.com/blog/2020/03/12/how-to-customize-fedora-coreos-for-dedicated-workloads-with-ostree#extending_ignition_files

[rhatdan]

@baude we might want to do this automatically in the ignition script, if we are already adding packages.