ARM64v8 - Raspberry Pi compatible docker image

[arslanakhtar61]

I want to deploy fusionAuth onto ARM64v8 - Raspberry Pi 4. So far the docker images for fusionAuth only support amd64 architecture. Is there a way to deploy fusionAuth docker image on Raspberry Pi 4 k8s cluster?

[mooreds]

I don't see why not, as long as you can run java, and I believe there is support for that architecture. Please feel free to fork and experiment with this.

Jerry Hopper took a look at "aarch64 support, for use on ARM single-board-computers": jerryhopper/fusionauth-install@b10349b

so it looks like you might need to change the java install scripts as well.

[mooreds]

Here's more:

DISCLAIMER, this is my first build: i have to double-test everything, but here is where to find the arm64-docker images.
the Java for Arm64 container (JDKv14):
https://hub.docker.com/repository/docker/botnyx/fusionauth-java-aarch64
and the FusionAuth container build with that java :  https://hub.docker.com/repository/docker/botnyx/fusionauth-app-aarch64

[arslanakhtar61]

@mooreds Thankyou very mush for the effort to build amd64v8 images. I tried to run the images on my raspberry pi 4 but it's throwing an error

$ uname -a 
 Linux ubuntu 5.4.0-1011-raspi #11-Ubuntu SMP Fri May 8 07:43:33 UTC 2020 aarch64 aarch64 
 aarch64 GNU/Linux

#simple enough to just make it run - ignoring sql-database configs
$ docker run \
    -p 9011:9011 \
    --name fusionauth-app \
    botnyx/fusionauth-app-aarch64:latest

standard_init_linux.go:211: exec user process caused "exec format error"

It's throwing the following error in docker logs fusionauth-app
standard_init_linux.go:211: exec user process caused "exec format error"

[mooreds]

Hmmm. I'm not sure what's going on. Unfortunately I can't provide much support for ARM64 images at this time, was just relaying a message from one of our community members who is working on this.

[jerryhopper]

My apologies for this @arslanakhtar61 , but pull the image again, and it works.
there was a problem, since dockerhub doesnt build arm64 (and i lack enough knowledge).

So i now build the image again on my arm64 sbc, and pushed that to dockerhub, and its working now. The downside is that its not automatically updated when FA releases a new version.

I hope to fix automated builds in the future.

[jerryhopper]

Meanwhile, i found a guide to build arm64 containers on dockerhub (https://www.balena.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/).
ill be trying to achieve this, so i can get some automated builds up and running

[arslanakhtar61]

@jerryhopper Thanks alot for the effort.

You could also use the new docker buildx with DOCKER_CLI_EXPERIMENTAL=enabled to build images for multiarch using your x86_x64 architecture machine.
example

 docker buildx build -t botnyx/fusionauth-java-aarch64:latest --platform linux/arm64 .

 docker buildx build -t botnyx/fusionauth-app-aarch64:latest --platform linux/arm64 .

Also I just pulled the latest images and when I try to run using

#simple enough to just make it run - ignoring sql-database configs
$ docker run \
    -p 9011:9011 \
    --name fusionauth-app \
    botnyx/fusionauth-app-aarch64:latest

I get the following error:
tty: ignoring all arguments
/dev/pts/0

Any suggestions on what is going wrong?

[jerryhopper]

Bear with me: its my first aarch64 image, and everything is very expirimental.

The problem: I cant get a Alpine container to work correctly. - So i did some testing with other images.

Ive used a Ubuntu image which seem to work instantly. pull the updated images!

root@arm64docker:~/fusionauth-containers/docker/fusionauth/fusionauth-app# uname -a
Linux arm64docker 5.4.43-sunxi64 #20.05.2 SMP Tue Jun 2 17:20:17 CEST 2020 aarch64 GNU/Linux
root@arm64docker:~/fusionauth-containers/docker/fusionauth/fusionauth-app# docker ps
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS              PORTS                    NAMES
fd969082c6b4        botnyx/fusionauth-app-aarch64:latest   "/usr/local/fusionau…"   7 minutes ago       Up 7 minutes        0.0.0.0:9011->9011/tcp   fusionauth-app

unfortunatly, the image is quite big.

[jerryhopper]

@arslanakhtar61 please try the latest containers: botnyx/fusionauth-app-aarch64:latest

DIGEST:sha256:18eb63e5f60e52d3bcd5b784961560f4e29b2ec1d953274732253f40ae4de824

This release uses a adoptopenjdk container, which runs ubuntu. its 267MB big.
I updated the dockerhub-repo, using tags as version.

thanks for your interest in my little fusionauth pet-project.

[arslanakhtar61]

@jerryhopper Great Job!
It's working. Thank you very much.

@mooreds, @jerryhopper can we create like automatic builds using perhaps docker buildx build to publish the latest aarch64 version of docker image (to docker hub botnyx/ maybe) along with the regular amd64 whenever a new version of fusionauth-app is available? Any plan for this?

Will there be an official docker image for aarch64? Especially now when we have arm based AWS instance - EC2.

We can also work on minimizing the size of the docker image in the future. But at least for now, it's working on my raspberry pi 4. Thanks

[jerryhopper]

I will look into that automatic building. for now - i have a nanopi-neo2 dedicated to building arm64 images.
About minimizing the image: i already tried to keep it as small as possible, but if it can be improved - or better - run on alpine - it would be awesome.

Regarding the 'official' image, thats totally up to the FusionAuth staff. Im just a big fan of fusionauth

[robotdan]

Depends on what we mean by "official". :-)

@arslanakhtar61 or @jerryhopper assuming this is just for fun - but if there is a real business case to run FusionAuth in this embedded mode, perhaps we would be interested in publishing the image.

At a minimum, we could think about accept a PR for the Dockerfile under a contrib directory and let people build it themselves.

[jerryhopper]

For what its worth, after some more fiddling, i produced a image of 108mb in size, and setup a dedicated automated builder. I've used debian-buster as base for this container.

This essentially means there will be a automated build every time there is a new official dockerimage was released.

The automated arm64 builds can be found here :
https://hub.docker.com/repository/docker/jerryhopper/fusionauth-app-arm64/tags?page=1

[arslanakhtar61]

Great Job @jerryhopper. Thanks a lot for the effort. I will now start pulling fusionauth images for my raspberry pi 4 arm64 board from your dockerhub repo.

[robotdan]

yes, thanks for your contribution @jerryhopper !
https://github.com/FusionAuth/fusionauth-containers#community-contributed-docker-images

[arslanakhtar61]

@jerryhopper

I am trying to replace the botnyx/fusionauth-app-aarch64:latest docker image with your jerryhopper/fusionauth-app-arm64:latest image but am getting the following error.

/usr/local/fusionauth/fusionauth-app/apache-tomcat/bin/catalina.sh: 38: /usr/local/fusionauth/fusionauth-app/apache-tomcat/bin/setenv.sh: Syntax error: "(" unexpected

[jerryhopper]

Please use the new repo, its a WAY smaller image - (and its on my personal account) https://hub.docker.com/repository/docker/jerryhopper/fusionauth-app-arm64

…

On Tue, Jun 30, 2020 at 11:32 AM Muhammad Arslan AKHTAR < ***@***.***> wrote: @jerryhopper <https://github.com/jerryhopper> I am trying to replace the botnyx/fusionauth-app-aarch64:latest docker image with your jerryhopper/fusionauth-app-arm64:latest image but am getting the following error. /usr/local/fusionauth/fusionauth-app/apache-tomcat/bin/catalina.sh: 38: /usr/local/fusionauth/fusionauth-app/apache-tomcat/bin/setenv.sh: Syntax error: "(" unexpected — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#49 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAI3NGJINMFEJFI4MMFY47LRZGWKXANCNFSM4NS2GFSQ> .

-- 8 mythes over omgaan met hardcore techies <http://computerworld.nl/projectmanagement/75977-8-mythes-over-omgaan-met-hardcore-techies> Skype : jerryhopper Google/mail/talk : hopper.jerry@gmail.com Google+ : https://plus.google.com/102634396402324749838/ Twitter : http://twitter.com/jerry_hopper/ Facebook : http://www.facebook.com/jerry.hopper.9 <http://nl-nl.facebook.com/people/Jerry-Hopper/527420824> www : http://www.helpivanmartin.org/ www : http://www.sahraniradio.com/ *"Ignorance is bliss" * - The lack of knowlege to a situation. Usually once the whole truth is revealed you realize you were happier being clueless.

[arslanakhtar61]

Yes, the error is coming out from your docker hub repo jerryhopper/fusionauth-app-arm64:latest image.

[jerryhopper]

im investigating..

[jerryhopper]

@arslanakhtar61 - please use the repo's issue-tracker for future issues please. jerryhopper/fusionauth_app_arm64#1

Thanks.

[jerryhopper]

@arslanakhtar61 i was fiddling with buildx, and i think you might be interested in this repo : https://hub.docker.com/repository/docker/jerryhopper/fusionauth-app/tags?page=1

[arslanakhtar61]

@jerryhopper Great Work! I will try to run this new multi-arch docker image build with buildx. Thanks

[ceefour]

Although the original use case is for Raspberry Pi, this would apply to AWS Graviton2 as well, which I'm already using for another app.

[jerryhopper]

Although the original use case is for Raspberry Pi, this would apply to AWS Graviton2 as well, which I'm already using for another app.

Thats great to hear! i'll mention this in the readme!

i have a question though: Are you running FA with or without a Elastic engine?

[ceefour]

@jerryhopper

Although the original use case is for Raspberry Pi, this would apply to AWS Graviton2 as well, which I'm already using for another app.

Thats great to hear! i'll mention this in the readme!

Oh and also:

• Apple Silicon Arm Macs, which is using the same architecture.
• Azure & Oracle Cloud have been evaluating Arm, and with AWS Graviton's aggressive push on this, I'd be surprised if they are sitting still

This docker build will be usable for more use cases.

i have a question though: Are you running FA with or without a Elastic engine?

I was using Elastic since it was required at that time, after 1.18 I disabled Elastic engine because currently the users are still small. I'm using FusionAuth as part of migration from previous system (Firebase), and doing in stages, after final stage migration is done I expect about 300k users and beyond, so if it gets slow then I'll use Elastic again.

[robotdan]

Will FusionAuth be the first IAM platform to run on "Apple Silicon" !? 😛

[ceefour]

I hate to break bad news but community-wise, another product did: https://www.itix.fr/blog/secure-raspberry-pi-keycloak-gatekeeper/ , https://collabnix.com/building-arm-based-docker-images-on-docker-desktop-made-possible-using-buildx/

I also searched for another obvious competitor of FusionAuth, and no Arm package so far in the ubuntu focal repository.

But what I can is for Keycloak, it's still a community build (pretty much like FusionAuth right now with @jerryhopper's excellent work).

If FusionAuth will support Arm64 officially that'd be awesome definitely! Both for those early adopters of Arm-powered MacBook, and also those running on servers (and those on Raspberry Pi, as @arslanakhtar61's original request).

So I have to slightly disagree of @robotdan's "a real business case to run FusionAuth in this embedded mode" 😉 because Arm64 is not just for Raspi but also these additional platforms.

Update: Just found out that Netflix now uses AWS Graviton2 instances.

[robotdan]

I hate to break bad news but community-wise, another product did.

Sad. 😔 Well, we could be 2nd, or 3rd!

Thanks for the details @ceefour !

So I have to slightly disagree of @robotdan's "a real business case to run FusionAuth in this embedded mode" 😉 because Arm64 is not just for Raspi but also these additional platforms.

Fair enough!

For what it is worth, my definition of "a real business case" is if we have a client that wants to pay for support and our hosting and requires it to run on ARM.

For our FusionAuth Cloud, in theory the client doesn't care what it is running on and they just need a URL.

For our clients that need support and then want (or need) to run it on ARM to support their use case, that could be a real business case for FusionAuth. We have not seen this to date. However, we are building out entity management / IoT support (FusionAuth/fusionauth-issues#881) , and perhaps this may expose some new use cases where it could make sense to run this on ARM.

With regard to "official" support, from your perspective, what does this entail?

These arm based platforms are not my expertise, so I'll gladly take feedback from @ceefour @jerryhopper @arslanakhtar61 and others smarter than me on what you think it would look like to make these images official.

My hesitation with "owning" this is that we don't have infrastructure or expertise to know if the build works. So perhaps there is some middle ground where we could publish the images and "own" the docker files in that they could exist in the FusionAuth containers repo. We could then (at least for now) mark them as community maintained and see how it goes.

Comments, thoughts welcome.

[jerryhopper]

"With regard to "official" support, from your perspective, what does this entail?"
For me personally, it ONLY entails that there is one dockerhub-repo for fusionauth, which can be used for all architectures.

In short: docker pull fusionauth/fusionauth-app should just work on all platforms

Here is my current state of mind on the subject:

Running FA on Arm64 is my personal learning project to master automated building multi-arch docker images, and if my findings help the eventual realisation of FA official arm support - that's cool.

About 'Real Tech Support from FA' for these extra architectures - i dont really need it. As there is so little difference from the original x64 container, and its only the underlying OS and Java that differs. In case of problems, i always try to replicate them on x64 - and then escalate the problem to FA support if the problem also exists in the x64 version.

Buildx is a Experimental feature
Im building using docker buildx - which is a experimental feature in docker - and im building this manually in a x64 VM.
Ultimatly the builds should be automated and tested, which is something im looking into. The testing of the containers obviously requires seperated environments to run in, and for now i have some (manually operated) Qemu(*) instances specificly for this.
So, eventually (but very slowly) i will get to the point that everything is automated and tested.

What is fundamentally different in my builds and the official FA docker images?
To make the multiarch docker files - i needed to create 1 dockerfile that contains all specifics for all architectures. i learned that 'importing' the JAVA container like FA does in its dockerfiles - just doesnt work with buildX (its that - or im missing something)

(community-build) FusionAuth Multiarch roadmap

• Automated building when new FA version is released.
• Automated multiplatform testing
• Make all of the above working using a dockerhub builder-node

Personally i think it's time to start thinking about 'official images' only when :

• Buildx reaches stable level, and is distributed in the docker package in a default installation.
• Dockerfiles support multiarch importing (see above).
• Some kind of automated multi-architecture testing is achieved.
• A version of Java is available that matches FA's choice of Os and Java. ( There is no stable openjdk14 for arm64 Alpine )

I fully understand @robotdan hesitations about adopting these architectures due to a lack of proper knowledge and/or (hardware) resources. If i would be in @robotdan 's shoes , the lack of automated testing would be my main reason to not support arm64

[ceefour]

Thank you @robotdan ! Minor things:

• typo: should be Graviton
• typo: should be Raspberry
• this one is subjective, certainly some people do use Raspberry Pi as a hobby, but the Pi is also used in various "serious" work. I myself use Pi in my PhD as part of IoT sensor network. This is up to you, but if you'd like a replacement word for "hobby" , maybe "embedded" or "IoT"

[robotdan]

Thanks @ceefour I'm embarrassed I didn't double check my spelling. 😔 I'll update to fix shortly.

[robotdan]

Fixed!

[jerryhopper]

for what its worth to anyone: I've setup automation on the gitrepository, using a github-runner. See /.github/workflow/main.yml for details.
https://github.com/jerryhopper/fusionauth-app-multiarch

[robotdan]

@voidmain -> https://github.com/jerryhopper/fusionauth-app-multiarch/blob/master/.github/workflows/main.yml

[ColinM9991]

Thank you for your persistence @jerryhopper. I've recently stumbled across this as I'm also building some Docker images on RPi ARM64. Your Actions yaml file, with a few tweaks, did the trick for me.

[ceefour]

OMG! OMG! OMG!

Evidence: https://hub.docker.com/layers/fusionauth/fusionauth-app/1.32.1_multi/images/sha256-da71d5be2bc849ef9d4e1205508fa34994973d75798a47cab9a8484ee2592ff7?context=explore

So this is official?

Thank youuuuuuuuu @robotdan and @jerryhopper !!!

[ceefour]

Today is I think a major milestone for FusionAuth, definitely deserves a release note.

I've been running FusionAuth in Fargate (amd64) so far and it's been rock-solid, and relatively easy to deploy & manage.

AWS is (understandably) pushing its own Arm processors, and now available in Fargate:

Announcing AWS Graviton2 Support for AWS Fargate – Get up to 40% Better Price-Performance for Your Serverless Containers

When I'm not using Fargate, I use k3s in cloud servers (k3s is also great for IoT, edge devices, etc. which are usually Arm-based).

So I'm excited how this empowers users/sysadmins on choices of deployment (and cost).

[ceefour]

Example usage with Pulumi:

const taskDef = new aws.ecs.TaskDefinition(pulumi.getProject(), {
  family: `${pulumi.getProject()}-${pulumi.getStack()}`,
  executionRoleArn: pulumi.output(ecsTaskExecutionRole).arn,
  containerDefinitions: containerDefinitions.apply(JSON.stringify),
  volumes: [],
  memory: "1024",
  requiresCompatibilities: ["FARGATE"],
  networkMode: "awsvpc",
  cpu: "256",
  runtimePlatform: {
    cpuArchitecture: "ARM64",
  },
  tags: {
    "Environment": "prod",
    "tenant_id": "lovia",
  },
});

[ceefour]

I just switched a FusionAuth instance to arm64 and the transition was so smooth I couldn't even tell whether it's actually running the arm64 version. Definitely not here:

Amazon ECS Web UI doesn't show anything either at runtime. The only clue is the Task Definition, and even that in JSON, it doesn't anywhere else:

@robotdan Do you think it's a good idea to put the arch in System > About ?

Second: Since "About" also mentioned the entire Database version string... I guess wouldn't hurt to also put the entire JVM string as well?

[ceefour]

The downside at this point is that AWS seems to be overbooked with Graviton.

I previously used FARGATE_SPOT capacity provider in ap-southeast-1a and had no capacity issues.
After switching to ARM64 it cannot provision due to undercapacity. But changing to FARGATE capacity provider works, with higher bill obviously.

But this is awesome! Choice!

[robotdan]

@jerryhopper I too get this error when trying to build the arm v7 image.

#5 166.0 java.nio.file.DirectoryIteratorException: java.nio.file.FileSystemException: /tmp/openjdk/jmods: Value too large for defined data type

But I am building on an intel MacBook. We will be getting our M1 MacBooks shortly, so I can retry on that platform. But we currently release on an x86 Linux server. However, we could probably spin up an arm based build agent to release additional image architectures .

@ceefour

So this is official?

Hopefully soon. Just ensuring it works, but I hope to add this to the release process and all future images will be published form arm64 and amd64. And hopefully linux/arm/v7 if we can get it to work.

@ceefour in your above screen shots, can you confirm you are running FusionAuth on Graviton? And yes, that is a great suggestion to add the architecture type to the about panel.

[robotdan]

Arch type for the about panel -> FusionAuth/fusionauth-issues#1531

[robotdan]

@ceefour how about this?

That is essentially just a string made up from Java properties os.name, os.version and os.arch. I think this arch value will be the Java arch, and not necessarily the underlying OS arch type. In most cases they should be the same, but not necessarily. So I may still if I can reliably also get the underlying operating system arch.

[jerryhopper]

a quick note from my side: i will look into the armv7 issue tonight - it previously runned here on a qemu, i will test/try again on real armv7 hardware instead of a emu.

[ceefour]

@robotdan that is awesome! 🔥

[jerryhopper]

@jerryhopper I too get this error when trying to build the arm v7 image.

#5 166.0 java.nio.file.DirectoryIteratorException: java.nio.file.FileSystemException: /tmp/openjdk/jmods: Value too large for defined data type

But I am building on an intel MacBook. We will be getting our M1 MacBooks shortly, so I can retry on that platform. But we currently release on an x86 Linux server. However, we could probably spin up an arm based build agent to release additional image architectures .

As far as i know, jlink doesnt seem to import/copy the jmods for some shady reason.
i have tried but somehow dont get this error. it seems to hit some kind of limit. (32/64bit?) Also im not quite familiar with your build-enviroment. (and i dont have a mac) - but i will look into it.

Wild guess: maybe specifying the arch during build helps?

FROM ubuntu:focal as build
to
FROM --platform=$BUILDPLATFORM ubuntu:focal as build

Building using github actions runs fine... you might want to have a look at my build-log - i have invited you to my gitproject- So you could have a look at the 'Actions/buildlogs' overthere.

On a sidenote: amv7 is merely a 'gimmick' ... the memory limitation doesnt makes it the best choice for running FA :)

[jerryhopper]

Maybe it is an idea to use Github Actions? - i created a workflow for that, which you can find in this branch: https://github.com/jerryhopper/fusionauth-containers/tree/jerryhopper--multi-build-githubworkflow

i had to slice the command in the dockerfile, because for some weird reason it fails if i dont split the command.
Also, i had to expliciltly declare "FROM --platform=$BUILDPLATFORM ubuntu:focal as build" in the Dockerfile.

[nodesocket]

We also would like a arm64 version, specifically back ported to tag 1.24.0. 🙏🏻

Looks to be a work in progress only using the latest version:

[robotdan]

Hmm, that may be possible. I’ll take a look.

[robotdan]

Done and done.

https://hub.docker.com/layers/fusionauth/fusionauth-app/1.24.0/images/sha256-fb023e73f644c114636914ab7ea1ccaeb525726cc8dcda106c01dea94469f638?context=repo

Any specific versions you want in between 1.24.0 and current? I suppose we could just republish all of them as well.

[nodesocket]

@robotdan just saw that it looks like you pushed up 1.24.0 with arm64 support. First of all, thank you so much. 🙏🏻 👨🏻‍🎤

Shouldn't that tag also have linux/amd64 as well though for Intel? Or does the original tag push with amd64 still get pulled in that case?

Thanks again 💯

[robotdan]

Shouldn't that tag also have linux/amd64 as well though for Intel? Or does the original tag push with amd64 still get pulled in that case?

Yes.. was in the middle of testing a few things. Refresh and it should show all 3 architectures now.

Thanks again 💯

You're welcome! Thanks for using FusionAuth!

[robotdan]

I think I'll close this one out. I will published (or have published depending upon when you read this) images back to 1.24.0 for linux/amd64, linux/arm64 and linux/arm/v7.

Going forward all releases will be published with at least those three architectures, perhaps linux/ppc64le and linux/s390x for good measure. I don't want to leave our IBM Power and IBM z friends out.

Anyone have an opinion an y additional architectures? I see there is a linux/arm/v5 - not sure if anyone would use that? I'm not smart enough to know what that is used for, so if you want it let us know.

[robotdan]

Thank you so much to everyone who has provided suggestions here, helped us understand the need, for contributing examples, PRs, etc. @jerryhopper @ceefour @nodesocket @arslanakhtar61 @ColinM9991 .

We plan to put out a press release for our multi-arch support. We may add some community member quotes if anyone is interested in contributing one. If interested, use the contact form on the website and indicate @robotdan sent you for a quote. 😎

[robotdan]

I only released the latest patch in each point version.
https://github.com/FusionAuth/fusionauth-containers/actions

If anyone needs another version for arm64 let me know.