How to cross-compile

[luxas]

Now with docker based setup it's very easy to cross-compile:

GOARCH=arm docker-image:canary

DOCKER_RUN_OPTS="-e GOARCH=arm" build/run-gulp-in-docker.sh docker-image:canary

This produces the image, but it's tagged as normal amd64 ones.
I tried to modify the gulp code, but I just don't get it working.

We want a docker-image:cross task, that depends on build-frontend and docker-file. Then it should call a function that builds the backend for the right arch. It's possible to refactor backend:prod into an exported function, but we can't call the tasks backend:prod is depending on (package-backend-source and clean-dist).

So because gulp doesn't support calling tasks directly (I would like to loop arches in docker-image:cross and for each call backend:prod with an argument and then build the docker image) I have no idea how to do this in gulp code.

It gets even worse when it should handle push-to-gcr too.

I can easily solve this with a bash script, though
@cheld @bryk What do you think about this?
It's easily done by hand, but it's obviously better if it's in code

Example bash script:

#!/bin/bash

# The set of platforms to cross-compile dashboard for
KUBE_CROSS_PLATFORMS=(
    "amd64"
    "arm"
    "arm64"
    "ppc64le"
)

# The image dashboard is pushed to. The same value exists in build/conf.js
KUBE_DASHBOARD_IMAGE="gcr.io/google_containers/kubernetes-dashboard"
TAG=${1:-"canary"}

# Only do this once
gulp build-frontend
gulp docker-file

# Loop all platforms, build the docker image and tag them right
for platform in ${KUBE_CROSS_PLATFORMS[@]}; do
    GOARCH=${arch} gulp docker-image:no-frontend
    docker tag ${KUBE_DASHBOARD_IMAGE}:canary ${KUBE_DASHBOARD_IMAGE}-${arch}:${TAG}
    docker rmi ${KUBE_DASHBOARD_IMAGE}:canary
done

# Tag the amd64 image without arch suffix for compability
docker tag ${KUBE_DASHBOARD_IMAGE}-amd64:${TAG} ${KUBE_DASHBOARD_IMAGE}:${TAG}

Where the docker-image:no-frontend just depends on backend:prod. If we choose this solution, we could wrap this script in a build-image:cross-{canary,release} task

PS: About the naming, I suppose we're gonna have gcr.io/google_containers/kubernetes-dashboard-ARCH:VERSION and maybe have two amd64 versions: kubernetes-dashboard and kubernetes-dashboard-amd64

[cheld]

The gulp task suffixes are use case oriented. Basically, :prod is for CI and nothing is for development. So, please do not use suffixes like 'no-frontend' or 'cross'. It would get very confusing. Cross compilation will be done only in CI so simply give the task the suffix :prod

The docker tasks are currently not integrated in the build process. In general, it would be nice to build the container deploy it to the cluster and then run the integration tests. I am wondering about the cross-compilation...how do we want to test the cross-compiled containers...

About gulp: I would not try to implement a loop. The number or architectures is limited. I would do the following: I would create a task for each platform like docker-build-arm, then copy & paste code to all tasks, and finally remove code duplication through refactoring. As last step I would create a dependency from docker-build:prod -> [ docker-build-arm,....]. This way all image builds are executed in parallel.

[bryk]

The gulp task suffixes are use case oriented. Basically, :prod is for CI and nothing is for development. So, please do not use suffixes like 'no-frontend' or 'cross'. It would get very confusing. Cross compilation will be done only in CI so simply give the task the suffix :prod

I dont agree. There's no need to run cross compilation on CI. Theres no benefit from this. The CI can potentailly cross compile, but cannot run the cross compiled code, as it is single architecture.

The cross compilation would be used when releasing new version. I.e., when calling gulp push-to-gcr:{release,canary}.

Am I correct?

[bryk]

@cheld @luxas
How about rearranging our dist/ folder so that it is divided by architecture. So there would be dist/amd64, dist/arm, etc. With that, we could have build and build:cross (name TBD). The first would build for current architecture, the other would iterate over architectures and build for all of them. This would require having :cross and default versions of a few tasks and moving some logic from tasks to functions that the tasks would call.
What do you think?

[luxas]

First off, cross-compilation should not be run on CI. Of course, the downside is that one might introduce only-working-on-amd64 go code without a CI failure, but that's a rare scenario and I don't think it will happen.

I have one idea that might work. If every (or at least some) module exported their functions, then another module deploy_cross.js or something could import buildBackendProd and it's dep package-backend-source, docker-image:cross could loop the arch list in conf.js and call cleanDist, packageBackendSource, buildBackendProd, dockerImage{Canary,Release} for every arch.

This would result in some refactoring though, so I'm not sure if we should do it right now (before the initial release). What has to be done though, is rename the default image from kubernetes-dashboard to kubernetes-dashboard-amd64. Can you rename it before merging to k8s core? We do not need to have cross compilation finished for that.

And now in the beginning, I am sure everyone is happy if you push the -arm, -arm64 and -ppc64le when you're doing the initial release. I may add to the docs how to cross-compile (as I said above, just set GOARCH)

BTW, I have already added dashboard as an experimental addon to kubernetes-on-arm

[bryk]

Can you give me the bash script that would iterate over all archs and call gulp docker-image?

[luxas]

#!/bin/bash

# The set of platforms to cross-compile dashboard for
KUBE_CROSS_PLATFORMS=(
    "amd64"
    "arm"
    "arm64"
    "ppc64le"
)

# The image dashboard is pushed to. The same value exists in build/conf.js
KUBE_DASHBOARD_IMAGE="gcr.io/google_containers/kubernetes-dashboard"

# Loop all platforms, build the docker image and tag them right
for arch in ${KUBE_CROSS_PLATFORMS[@]}; do
    GOARCH=${arch} gulp docker-image:canary
    docker tag ${KUBE_DASHBOARD_IMAGE}:canary ${KUBE_DASHBOARD_IMAGE}-${arch}:canary
    docker rmi ${KUBE_DASHBOARD_IMAGE}:canary
done

That will produce images for all platforms. You just have to push them after that.

[bryk]

I've got the following errors when cross-compiling. Do you know what they are?

[08:26:00] Starting 'backend:prod'...
# syscall
/usr/lib/google-golang/src/syscall/zsyscall_linux_ppc64le.go:375: Exit redeclared in this block
    previous declaration at /usr/lib/google-golang/src/syscall/syscall.go:101

[luxas]

Thanks for pushing an arm image :)
Which version of go do you have? I think it should work with go1.5.3
Maybe it's best to run this inside the docker container?
docker run -it -v /var/run/docker.sock:/var/run/docker.sock kubernetes-dashboard-build-image /bin/bash and execute the script from there...

If you like I can send a PR with this temporary build/cross.sh script...

[bryk]

I'll try to run this in the docker image next time. Maybe by base system misses some libraries.

[bryk]

If you like I can send a PR with this temporary build/cross.sh script...

No need to rush on this. Let's do the proper way, i.e., fragment the build artifacts into separate directories for each arch and make gulp take care of this :)