Migration left over from publicK8s to arm64

[smerle33]

Service(s)

Azure

Summary

following #3619
we still have :

• artifact-caching-proxy
• datadog
• falco
• get-jenkins-io (mirrorbits has no arm image ... yet)
• keycloak
• ldap
• updates-jenkins-io (mirrorbits has no arm image provided ... yet)

Reproduction steps

No response

[dduportal]

Starting with LDAP:

• The image seems old and should be updated to meet our usual "update/release" system
• Then we should be able to build it for arm64

[dduportal]

Starting with LDAP:

* The image seems old and should be updated to meet our usual "update/release" system

Todo:

• Rename jenkins-infra/ldap to jenkins-infra/docker-ldap to fulfill convention

• Add the jenkins-infra/docker-ldap repository to infra.ci to start building / releasing the container image

• Set the principal branch of jenkins-infra/docker-ldap to main - (see [INFRA-3125] Migrate jenkins-infra repositories from branch "master" to "main" #2671 (comment))

• Open a PR in enkins-infra/docker-ldap with the required elements for the first build (Jenkinsfile_k8s, release drafter setup, hadolint fixes). The checks should be OK before merging. - chore: migrate CI/CD to infra.ci.jenkins.io docker-ldap#33

• Ensure the first release is made (eventually add PRs/hotfixes). DoD: both GH release (and git tag) + DockerHub tag

  • https://hub.docker.com/r/jenkinsciinfra/ldap/tags?page=1&name=1.0.1

  • https://github.com/jenkins-infra/docker-ldap/releases/tag/1.0.1

  • The release 1.0.0 was created manually through GH release on the latest tag, and the associated image was deployed to DockerHub from the current production images

    skopeo copy --all docker://docker.io/jenkinsciinfra/ldap@sha256:d662790b4a5f1ca979c186241005e816ff86ccedf8161d9f11df43501f077f31 docker://docker.io/jenkinsciinfra/ldap:cron-1.0.0
    
    skopeo copy --all docker://docker.io/jenkinsciinfra/ldap@sha256:a1518d683d3098be912a684892d10f909dd6c5bf2a65dce3f477a2caab73ef22 docker://docker.io/jenkinsciinfra/ldap:1.0.0

• Update jenkins-infra/kubernetes-management to use the new image and track it

  • Inital deployment of 1.0.1: feat(publick8s) pin LDAP images to version 1.0.1 kubernetes-management#4881
  • Bump Helm chart to specify versions: feat(ldap) pin image version to the tag 1.0.1 helm-charts#1031
  • Deploy new helm-chart version + cleanup pinned image version in jenkins-infra/kubernetes-management: Bump ldap helm chart version to 1.0.0 kubernetes-management#4882

• (eventually) add updatecli tracking

  • Manifest updated in feat(ldap) pin image version to the tag 1.0.1 helm-charts#1031
  • Automated PR: Bump ldap slapd and crond docker images to 1.0.2 helm-charts#1032 => released ldap-1.0.2
  • Deployed to production: Bump ldap helm chart version to 1.0.1 kubernetes-management#4883

• Remove the Docker LDAP job from trusted.ci and ci.jenkins.io (were already deleted)

Then:

* Then we should be able to build it for arm64

[dduportal]

Update:

• LDAP "pinned" version released and deployed (see above comment)
• WiP on arm64:
  • Build arm64 image: feat: add arm64 images docker-ldap#35
  • Helm chart image bump: Bump ldap slapd and crond docker images to 1.1.0 helm-charts#1033
  • Kubernetes deployment of the new chart allowing to switch to arm64 image when needed: Bump ldap helm chart version to 1.0.2 kubernetes-management#4884
  • The migration to arm64 will require to migrate a copy of the data in the same availability "Zone" as the arm64 node pool in the same way we did for weekly.ci

[dduportal]

Important note: these migrations are triggering SNAT exhaustion problem, which we though was fixed earlier this week. We have to suspend tentatives until we've fixed it (even temporarily): #3908

[dduportal]

Update:

• LDAP is now starting with arm64. But authentications have weird behaviors: I'm not seen as "Admin" in accountapp anymore and ci.jenkins.io auth. says "auth error". No logs difference on LDAP side between x86 and arm64 except the message mdb_equality_candidates: (member) not indexed only present on x86 (when it is working).
• Keycloak them image is being updated (see https://github.com/jenkins-infra/docker-keycloak-theme PR) but need a bit of auth. work to properly deploy on DockerHub.

[dduportal]

Update:

• About mirrorbits, as the arm64 most probably will appear with a 1.x version, we should resume @lemeurherve's work in feat: build mirrorbits with Golang 1.23.0 in the image to get an arm64 binary docker-mirrorbits#18 where we could build the image on both amd64 and arm64, with custom build of the 0.5.1 tag of mirrorbits
  • Fallback is to schedule the 4 container into the system pool x86 nodes so we can get rid of the current x86 node pool and pack services

[dduportal]

Update:

We see a visible cost decrease thanks to:

• the ACP migration to arm64 (x86 node pool is now 2 node instead of 3):

• The data disk migration to ZRS + optimizations:

---

[dduportal]

Update:

• keycloak is now running on arm64 nodes 🥳
  • Initial PR: feat(publick8s/keycloak) migrate to arm64 kubernetes-management#5401
  • Required an hotfix as I forgot the tolerations: jenkins-infra/kubernetes-management@0cbf481. Checked with a kubernetes-management full run after hotfix was applied

[smerle33]

Update:

* LDAP is now starting with `arm64`. But authentications have weird behaviors: I'm not seen as "Admin" in accountapp anymore and ci.jenkins.io auth. says "auth error". No logs difference on LDAP side between x86 and `arm64` except the message `mdb_equality_candidates: (member) not indexed` only present on x86 (when it is working).

seems to say that the indexing is happening on x86 and not on arm64 (cveda/cveda_databank#1)

for reminder: we got a mock ldap in our repositories : https://github.com/jenkins-infra/mock-ldap

I did try to launch the ldap 1.1.1 on my ARM M1 machine and got :

Status: Downloaded newer image for jenkinsciinfra/ldap:1.1.1
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

[dduportal]

Update:

• LDAP image has been fixed and is built for both arm64 and x86. It allows debugging locally on a Silicon Mac
• We might want to run the x86_64 leftover workload on the system pool (which is already x86) to improve packing of resources.

[dduportal]

Update: We are the proud owners of an arm64 image of mirrorbits \o/ Let's move mirrorbits to arm64!

[dduportal]

Update: We are the proud owners of an arm64 image of mirrorbits \o/ Let's move mirrorbits to arm64!

Update:

• We now only have LDAP running x86 \o/
• The x86 node (non system) nodepool has been scaled down from 2 to 1 node, while the arm64 stays at 3 nodes