"fork/exec /bin/sh: exec format error" and "operation not permitted error" when trying to build an arm64 image

[Julian-Marco]

We're using kaniko inside an Openshift cluster. Specifically, I would like to use it to build an image based on the following dockerfile:

https://github.com/microsoft/onnxruntime/blob/master/dockerfiles/Dockerfile.jetson

This image itself uses arm64v8/ubuntu:18.04 as a base image, so I suppose that this is where the errors are coming from. To be more precise, when I use v1.8.1 or v1.8.0, kaniko throws the following error:

error building image: error building stage: failed to get filesystem from image: failed to write "security.capability" attribute to "/usr/lib/aarch64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper": operation not permitted

And when I use v1.7.0 (or older), I receive the following error:

error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: exec format error

When I build the image locally with Docker (on a Windows machine), it works just fine though. Inspired by issue #1587, I tried using the respective flags, but it doesn't work no matter what kind of arguments I try. My setup basically looks like this:

container:
      image: gcr.io/kaniko-project/executor:latest
      securityContext: 
        runAsUser: 0
      openshift.io/scc: privileged
      command: [/kaniko/executor]
      args: ["--dockerfile=Dockerfile",
         "--cache=true",
         "--context=***",
         "--destination=***",
         "--build-arg='opts='GOARCH=arm64''",
         "--customPlatform=windows/arm64"]

I also tried it with --customPlatform=linux/arm64/v8 and what not, but the results are always the same. I also read that there is a specific arm64 image for kaniko, but using this just throws another error:

Error: fork/exec /kaniko/executor: exec format error
fork/exec /kaniko/executor: exec format error

So how do I use kaniko to build the desired image? Is that even possible?

[Julian-Marco]

So the error arises due to the different types of architectures (arm architecture vs amd architecture), and this doesn't seem to be solvable with kaniko. For now, we're using qemu as an emulator + podman, something like this:

podman run --rm --privileged multiarch/qemu-user-static --reset -p yes
podman build -f Dockerfile .

This works for building the image.

[ricardorqr]

So the error arises due to the different types of architectures (arm architecture vs amd architecture), and this doesn't seem to be solvable with kaniko. For now, we're using qemu as an emulator + podman, something like this:

podman run --rm --privileged multiarch/qemu-user-static --reset -p yes podman build -f Dockerfile .

This works for building the image.

How have you solved this problem?