Kaniko excecutor is wrongly tagging the architecture metadata on ARM image.

[ericbl]

I am working on an dockerized service built for both ARM64 and AMD64 devices.
My Gitlab pipeline is thus creating 2 separated images, one for each architecture.

My service is a dotnet core (3.1) component.
My dockerfile only specifies the base image (linux alpine) and copy my .NET lib on it:

FROM $BASE_IMAGE
COPY --from=build /out .

The base image is Linux Alpine 3.13, either the x86_64 one for AMD target or the aarch64 one for ARM64 target.
The build step is also using the $DOTNET_RUNTIME. either linux-musl-x64 or linux-musl-arm64:
RUN dotnet publish myServiceHost.csproj -c Release -o out -r $DOTNET_RUNTIME --self-contained true

as from my gitlab script:

My build stage is using gcr.io/kaniko-project/executor:debug as requested by Gitlab.

image:
    name: gcr.io/kaniko-project/executor:debug #$ARCH
 script:    
 - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $IMAGE:$BUILD_IMAGE_TAG
       --target=build
       --cache=true
       --build-arg BASE_IMAGE=$BASE_IMAGE
       --build-arg DOTNET_RUNTIME=$DOTNET_RUNTIME

The Gitlab pipeline is using AWS runners, on AMD/Intel machines.

Kaniko properly create the target images: on my ARM device, the service is a fully ARM image with ARM binaries.
BUT the metadata are wrong, and thus causing many isssue with Docker Swarm

# docker image inspect <my-image-id> | grep "Architecture"
"Architecture": "amd64",

It should be ARM64 !

If I enter the running with
# docker exec -it <container-id> sh
and then
# uname -a
it reads aarch64, as defined in the base Alpine image.

BEFORE moving to Kaniko, my pipeline was using docker pull/push commands, also using AMD/Intel AWS runners.
ARM image built through docker push command is still tagged as ARM64 architecture.

The issue here is only the Kaniko Excecutor overwriting the architecture metadata of the final image.

How is kaniko setting the "Architecture" tag of the docker image ?
Is it related to the runner used to build it?
Or which parameters do I need in my kaniko script to set it to ARM64 ?

The issue is likely related to #786 and #1526

[ericbl]

as mentioned in the related #786 and #1526, I added
--build-arg opts="CGO_ENABLED=0 GOARCH=$ARCH"
in my pipeline were ARCH is arm64 or amd64.

# docker image inspect <my-image-id> | grep "Architecture"
still see it as
"Architecture": "amd64",
for the arm64 image :(

I also tried to use an ARM64 runner on AWS, but my pipeline fails because of /bin/sh dotnet on ARM

Running: [/bin/sh -c dotnet publish ./src/Host/Host.csproj -c Release -o out -r $DOTNET_RUNTIME --self-contained true] 
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: exec format error

[ericbl]

Also related is the missing support of pushing manifest #1550
and obviously the missing ARM64 build of kaniko excecutor:debug required for Gitlab: #1488

Recent build 1.4 and 1.5 are not yet available on Google image registry

[ericbl]

so what is the recommended approach to build and target ARM64 images?

A) use the standard gcr.io/kaniko-project/executor:debug on AMD/Intel runners and expect them to tag properly ARM64 images ?
or
B) use a ARM64 version of kaniko executor and use a ARM64 runner for it?

[ericbl]

I edited the initial message and title since the issue here is really only about tagging the metadata.

My current work around is to still have the docker pull/push commands for my ARM64 target, waiting for a proper fix with Kaniko

[ericbl]

any news for building ARM64 images with Kaniko?

[LeventeLantos]

@tejal29 Are you guys planning to fix this any time soon?

[azamora-zsc]

I've done some tests and by adding the flag:
--customPlatform=linux/arm64/v8
you get "Architecture": "amd64" in your image. I have to do more tests but this worked for me.

[ericbl]

does
--customPlatform=linux/arm64/v8
replace or is additional to
--build-arg opts="CGO_ENABLED=0 GOARCH=$ARCH"
?

edit: seems to be a full replacement.

However, the docs says:

Allows to build with another default platform than the host, similarly to docker build --platform xxx the value has to be on the form --customPlatform=linux/arm, with acceptable values listed here: GOOS/GOARCH.
It's also possible specifying CPU variants adding it as a third parameter (like --customPlatform=linux/arm/v5). Currently CPU variants are only known to be used for the ARM architecture as listed here: GOARM

This is a is a bit misleading since
https://gist.github.com/asukakenji/f15ba7e588ac42795f421b48b8aede63
lists only the 'arm64' and not 'linux/arm64/v8'

[azamora-zsc]

I used something like this locally:

$ docker run --rm \                                                                                                                                                       
    -v "$PWD":/workspace \
    gcr.io/kaniko-project/executor:latest \
    --dockerfile /workspace/Dockerfile \
    --destination=test:image --tarPath=/workspace/image.tar \
    --context dir:///workspace/ \
    --build-arg opts='GOARCH=arm64' \
    --no-push \
  --customPlatform=linux/arm64/v8

IMO, both are needed. By only using build-arg opts='GOARCH=arm64' I didn't get the right value in the Architecture label inside the image. By also adding --customPlatform=linux/arm64/v8 I got the expected result.

[mickkael]

It is different. The builds arg will make the binaries compatible with Arm or other architecture requested. Custom platform will download the base image for the architecture in parameters and put architecture metadata in the output image as per the parameter.

…

On Thu, 2 Sep 2021, 15:59 Eric, ***@***.***> wrote: does --customPlatform=linux/arm64/v8 replace or is additional to --build-arg opts="CGO_ENABLED=0 GOARCH=$ARCH" ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEWXDHPUEISSHF5M5DT3WSLT74VEVANCNFSM4XZ5CWYA> .

[ericbl]

--build-arg opts='GOARCH=arm64' --customPlatform=linux/arm64/v8

is giving me good results and therefore I can close the issue.