Add pre-compiled ARM binaries for releases #1379
Comments
|
What is the status on this ? |
|
@Niraj-Fonseka its currently in our icebox under the Operations project. Unfortunately its not a priority for the team right now |
|
@jama-pivotal I'm also interested in this, and open to building my own binaries, but I can't find any documentation anywhere about how to build the set of concourse binaries. Could you please point me to documentation? |
|
@brownjohnf we have some getting started with contributing under our quickstart guide for engineer contributors: https://github.com/concourse/concourse/blob/master/CONTRIBUTING.md Not sure if that has enough specific detail to get you started though... /cc @vito to see if he has any more resources |
|
I also tried to take a look at that and to be honest I struggled quite a bit. I looked both at the existing pipelines at https://github.com/concourse/pipelines to run with my own concourse setup and tried to build things locally on my machine (and various combinations thereof). And didn't get very far... I have a local concourse setup I can experiment with, it just seems to duplicate the whole pipeline touches on so many custom things (versioning in s3, all the bosh stuff maybe?) that shouldn't be necessary just to build and yet they seem so integrated. So any help would really be appreciated. I might be able to help out with various things in that context since we do have a new requirement for building arm stuff here. The concrete thing I need really is a concourse binary built with GOARCH=arm. |
tl;drThere are armv7 and aarch64 builds available that work with caveats. BuildingI had a go on this and I went down a deep rabbit hole but I finally managed to cross compile both an So I took the path of building everything from scratch using a bash script that bootstraps a concourse binary. The bash script more or less follows what the pipeline does. The idea is that this stage0 concourse binary can then be used on an actual arm machine to run the normal build pipeline and produce the concourse binary. There are some build issues with both armv7 and aarch64 of which I think I fixed all of them and some of them are already upstreamed: concourse/baggageclaim#11 You can find the bootstrap repo here https://github.com/resin-io/concourse-arm. To cross compile it all you need is a linux system with a working Go and Docker installation. You can also find pre-compiled binaries in the Github releases page. RunningI haven't yet tested the arm64 version but I believe it will work without issues. Unfortunately there are still some runtime issues for armv7 builds which are due to a bug of golang's syscall module. If you attempt to run concourse on 32bit hardware with user namespaces enabled you'll get: This is because the entries in A potential way forward (fixing go aside) would be to use runc's libraries that has already fixed this issue opencontainers/runc#1819 . It's still the wrong datatype but at least it doesn't overflow. Resource typesThe official concourse binaries currently ship with the following resource types embedded:
Of those I only cross-compiled:
This means that the current binaries are not yet able to run the full concourse pipeline and build itself since it uses more resource types but you can definitely use it if your pipelines don't need those resource types. Depending on the resource type, cross compiling could be as simple as switching the base image. I expect bosh to be the most tricky one. UpstreamingCurrently the build process changes the projects in a way that breaks normal amd64 builds. So there is work left to be done to have a multi-arch build process that can be upstreamed. |
|
@petrosagg Thanks for looking into this! |
|
Yeah, thanks a lot. I'll try to set this up myself and will report back on how it goes :-) |
|
I just pushed a couple of PRs for the int overflow issue concourse/baggageclaim#12 and published a new arm binary as v3.14.1-rc2 :) https://github.com/resin-io/concourse-arm/releases/tag/v3.14.1-rc2 It now initialises correctly and I can load the UI. Haven't done further testing or actual builds yet |
|
I managed to successfully run an aarch64 worker (before your update). Now I just need to figure out how to bootstrap the aarch64 image used to build another aarch64 image in my pipeline :-) |
|
@neumayer nice! What do you mean by bootstrapping the aarch64 image? We (resin) build the cross platform base images that I used for this so if you have any questions about how the cross-compilation works or how to run them natively I can help you |
|
The actual task I'm trying to solve is to make concourse build (and publish) docker images for me. This is a bit new to me but I'll keep you posted once I get that working (and especially if I don't get it working :-)). Thanks for the offer, I just need to find the time to read up on the multi arch docker thing a bit first. |
|
Hi! I have some updates on my multi arch journey :-) One thing I thought would be great for this whole thing is to build all the images involved multi-arch aware. I.e. the images I want to use in concourse should exist for both amd64 and aarch64 architectures and this is easily accomplished by using alpine based images. Wherever they are built the architecture is properly propagated and the right repos are set up and the packages for the right architecture are installed. I don't know how this works for other base images (I do expect some issues here though, and at some point I think it'll be inevitable to have one Dockerfile per architecture or lots of if statements in the Dockerfiles). I build a modified dind image that supports multi arch which in turn can be used to build other docker images. It's a bit of a chicken and egg problem, but once the initial image is bootstrapped you're ready to go. Then I made a concourse pipeline to use this image with to tasks, one for amd64 (i.e. the normal job) building a xxx:amd64 image and one that has a tag specified so the right worker is used (aarch64 is the tag I use for the aarch64 workers) to build a xxx:aarch64 image. There's a third task building a manifest and pushing the manifest for both images (xxx -> xxx:amd64, xxx:aarch64). I'm quite new to the multi-arch stuff in docker, but it seems to me that the only viable way forward is to add architecture tags to all images anyway (so the manifests can refer to them later). There's one thing I noticed: when concourse pulls the image for the aarch64 task, i.e. the one that is run on the aarch64 host, it seems to ask for the amd64 image explicitly (via the architecture setting in docker). So my assumption is that the code somewhere has some custom logic that falls back to the amd64 architecture rather than propagating the architecture of the worker in a generic way. I'll double check that if I find the time, it's just a theory so far. When I pull the image without tags from inside the build container the right image is pulled (via the docker binary). The easy workaround for now is to add aarch64 tags to both the task and to the image used in the task. Ideally this would be integrated in the docker image resource somehow, but I don't see how the manifest stuff can fit there, maybe a docker-manifest resource. If both docker-image resource and docker-manifest resources had the same set of architecture tags for which they can check this might work nicely (the docker-image resource has a multi-arch flag that adds these tags and the manifest resource uses the outputs and builds manifests for all known architectures). I'll try to post updates when I have time to look for the architecture propagation code. |
|
I have a short update on my previous ramblings about the architecture propagation code being buggy. That is not true. It works as intended, I had mistagged my images. It happens a lot with building multi-arch images :-), it seems to be the main challenge to keep things consistent. Anyway. I've been using this rogue worker for aarch64 for almost three months from the binary and so far it's been very stable and worked well together with the concourse-web instance (4.0.0). So is there any chance of producing official arm 64 binaries? And is there anything I can do to make that easier (even with my limited understanding of the concourse build process)? |
|
Hey @neumayer, I followed your journey with great interest. Currently I think of using concourse on my raspberry pi 3. So stumbled over this issue. Can you give use a short description or mini tutorial how you achieved your goal? Perhaps we could manage to create a PR to integrate the changes into the build pipeline. |
|
I just used @petrosagg's binary. |
|
Ok, thanks for the info. I will look into https://github.com/resin-io/concourse-arm and try to get to the point of using it on my raspi. |
|
The question of #arm64 builds of Concourse came up yesterday at Dockercon, I am looking forward to seeing this support from my point of view of the @WorksOnArm project. |
|
Just here to give my +1 for having arm build. |
|
I took another look after concourse 5 came out and was hoping the new build structure would be easier to deal with when it comes to arm. Which was true, creating the binary was super easy, just since the resource images are packaged separately now those of course were missing. And I did not investigate how they are built now. At least that's what I remember it's been a couple of months I last looked at this. But at the latest when the worker protocol changes I'll have to take another look. |
|
I've created a repository to build This repository helps you build both the web and worker Bundled resources
|
|
Hey there @robinhuiser . Sorry if this is a dumb question, but I've tried to run that docker image and I keep getting the error message "error: Please specify one command of: generate-key, land-worker, migrate, quickstart, retire-worker, web or worker". Is it even possible to run Concourse on M1 AArch64/Arm64? |
|
Hi @sahmeder - I have tested the arm64 images on Raspberry Pi only, but I would be happy to help out. Could you please share your deployment (docker-compose, Kubernetes manifests) so I can be of assistance? |
|
Hey @robinhuiser Thanks so much for responding :) What I did was run the rdclda/concourse docker image that you created here. I was getting the errror message that my iptables were failing and someone created a github issue here where using the rdclda/concourse image fixed that issue for them. I tried to run a container and use your image which is when I got the "error: Please specify one command of: generate-key, land-worker, migrate, quickstart, retire-worker, web or worker". I did some more digging and it looks like this message happens when the docker run privilege isn't set to true, but when I checked the docker-compose file, it's showing that it's set to privilege. I've been scratching my head trying to get this figured out but I'm at a lost so far lol. |
|
Hi @sahmeder - I will replay your issue later tonight, need access to an M1 MacBook or Raspberry Pi which I do not have now. |
|
@robinhuiser No rush :) |
|
I wanted to add that I tried installing it via a Ubuntu CLI through multipass and got the same results as well. |
|
@sahmeder - did you try to use the provided docker-compose.yaml file in the Docker registry linked Git repo? version: '3'
services:
concourse-db:
image: postgres
environment:
POSTGRES_DB: concourse
POSTGRES_PASSWORD: concourse_pass
POSTGRES_USER: concourse_user
PGDATA: /database
concourse:
image: rdclda/concourse:7.5.0
command: quickstart
privileged: true
depends_on: [concourse-db]
ports: ["8080:8080"]
environment:
CONCOURSE_POSTGRES_HOST: concourse-db
CONCOURSE_POSTGRES_USER: concourse_user
CONCOURSE_POSTGRES_PASSWORD: concourse_pass
CONCOURSE_POSTGRES_DATABASE: concourse
# replace this with your external IP address
CONCOURSE_EXTERNAL_URL: http://10.0.19.18:8080
CONCOURSE_ADD_LOCAL_USER: test:test
CONCOURSE_MAIN_TEAM_LOCAL_USER: test
# instead of relying on the default "detect"
CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay
CONCOURSE_CLIENT_SECRET: Y29uY291cnNlLXdlYgo=
CONCOURSE_TSA_CLIENT_SECRET: Y29uY291cnNlLXdvcmtlcgo=
CONCOURSE_X_FRAME_OPTIONS: allow
CONCOURSE_CONTENT_SECURITY_POLICY: "*"
CONCOURSE_CLUSTER_NAME: arm64
CONCOURSE_WORKER_CONTAINERD_DNS_SERVER: "8.8.8.8"
CONCOURSE_WORKER_RUNTIME: "containerd"As can be seen from the code snippet above - you do need to provide an argument to the Docker image - in this case (running web and worker in single container) this is |
|
Hey @robinhuiser ! I was able to get it up and running. So what I was doing incorrectly was instead of zipping the concourse-arm64 folder and running docker-compose up within that folder locally, I was pulling the docker image from the docker registry. A little new to docker and containerization so please excuse my ignorance :) |
|
Hello @robinhuiser, have you experienced something like concourse/concourse-docker#79 ? |
|
I would love to run concourse with ansible on raspberry pi’s k3s or native |
|
@DonBower - please check out the prebuilt Docker images for Raspberry Pi - they can be found on Docker Hub rdclda/concourse. |
|
Daily driving a rpi4 at the office. Would definitely like this. Specifically the fly binary. |
|
+1 for arm64-support. Maybe it is worth mentioning that GCP by now also seems to offer arm64-VMs. I could build and deploy a multi-platform (x86_64 and arm64) instance of concourse using @robinhuiser 's build-scripts basically "out-of-the-box", so it seems there is not that much missing (except, maybe, some adjustments in upstream helm-chart). I figure, (native) multiarch support might become more relevant w/ arm gaining popularity (after all, this is why I setup multi-platform concourse). |
|
+1 for arm64; I'd like to use this in my home lab ARM k8s cluster. |
|
+1 apple silicon MacBooks |
|
bump |
|
Hi all, thank you for your interest and patience on this topic. With the multi-arch feature now available in oci-build-task, after running a spike on Concourse CI to build concourse/dev with multi-arch, the path is clear for us to have the ARM release: that we need the following dependencies to be available in ARM. The dependencies:
The blocker:
The plan:
Please let us know your thoughts or ideas. Thank you! |
for context see concourse/concourse#1379 (comment) Signed-off-by: Rui Yang <ruiya@vmware.com>
|
@xtremerui I see in https://github.com/cloudfoundry/garden-runc-release/releases that gdn now has arm builds since 1.22.9 That removes at least one blocker. Is your second point of the above mentioned plan still valid? In other words, is performance still a concern or would it be possible to start making arm releases of Concourse? I'd really like to get this off the ground as it were.. |
|
@mvdkleijn yes it is now happening, slowly... The concourse/dev image supports ARM platform already. So running Concourse docker compose on a M1 machine should work flawlessly. I can see the compilation time of Concourse binary in the image takes less time significantly. That might allow our CI to adapt ARM faster i.e. without separating the build task. As the ARM support propagating through our CI, eventually we should be able to release Concourse in ARM. However there is no timeline for that yet. |
|
Tried dev:latest. on hello world job with arm64v8/alpine image get error Full worker env docker-compose for worker log for job pipeline |
|
Hello, what's the status on supporting arm64 for darwin? |
|
@NorseGaud there is really no timeline yet. |
|
+1 for arm64 for new Macs please I have scripts which auto-determine the latest GitHub release, OS + architecture and download binaries, hit this on my new Mac: |
and others
Hey guys,
Feature Request
I wanted to use concourse ci workers on my raspberry pi 3 for doing QA with x86 firmware.
The QA should be really cheap < 100$ so that people can attach their own test stand.
Would it be possible to deliver pre-compiled ARM binaries for releases ?
Best Regards, Zaolin
Edit from @taylorsilva April 2022
No plans for official ARM images yet because it's hard to add this to our process. AWS is the only cloud provider with ARM instances and we do everything on GCP currently. Adding a single ARM instance is therefore really hard. This will be more feasible once more cloud providers add ARM instances to their offerings.
Currently this comment, farther down in this issue, is your best option: #1379 (comment)
The text was updated successfully, but these errors were encountered: