Proposal for application in publick8s to migrate to arm64

[smerle33]

Service(s)

Azure

Summary

Work in progress: determining candidates to migrate on the arm node pool

existing deployments on publick8s :

• accountapp
• cert-manager
• cert-manager-cainjector
• cert-manager-webhook
• coredns (kube-system)
• coredns-autoscaler (kube-system)
• datadog-cluster-agent
• incrementals-publisher
• javadoc
• jenkinsio
• jenkinsio-zh
• konnectivity-agent (kube-system)
• metrics-server (kube-system)
• mirrorbits
• mirrorbits-files
• plugin-health-scoring
• plugin-site-backend
• plugin-site-frontend
• plugin-site-issues
• private-nginx-ingress-ingress-nginx-controller
• private-nginx-ingress-ingress-nginx-defaultbackend
• public-nginx-ingress-ingress-nginx-controller
• public-nginx-ingress-ingress-nginx-defaultbackend
• rating
• reports
• uplink
• wiki

Progress

#3619 (comment)

[lemeurherve]

Update

plugin-site-api

Arm64 image published from infra.ci.jenkins.io, chart updated, ready to migrate.

plugin-site-issues

Arm64 image published, ready to migrate.

Next steps

We can proceed to plugin-site components migration to arm64.

Then we'll migrate weekly.ci.jenkins.io to arm64, the corresponding arm64 image is already published.

[lemeurherve]

Last plugin-site helm chart version including the tag with an arm64 variation deployed on publick8s cluster.
Only remain the helmfile release changes to migrate its components to arm64.

[smerle33]

plugin-site and plugin-site-issues migration to arm64 done.

Migration plugin-site and plugin-site-issues post mortem:

I did forget to update the charts version in the PR : jenkins-infra/kubernetes-management#4683
I created the PR jenkins-infra/kubernetes-management#4684 to fix it but the helm engine locked the update with: Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

The solution was to rollback :
first we list the releases : helm ls --namespace plugin-site
then we revert: helm rollback --namespace plugin-site plugin-site and helm rollback --namespace plugin-site plugin-site-issues
we can then launch a new kubernetes-management build in the infra.ci.jenkins.io

In order to avoid this kind of problem, I think that I need to first check the opened PR related to this release (we had jenkins-infra/kubernetes-management#4671) and to better plan the migration to make sure that more than one can focus and check the PR.

This had no impact on production as kubernetes was able to hold the upgrade as it was not successful.

[dduportal]

Before closing this PR, the following services will have to be migrated to arm64:

• httpd (in both mirrorbits and mirrorbits-parent releases)
• rsyncd (in mirrorbits-parent release)
• weekly.ci.jenkins.io

Next step after this issue (to be continued and detailed):

• infra.ci.jenkins.io on arm64 (controller and agents) #3823
• ci.jenkins.io and ARM64 #3824

[smerle33]

mirrorbits-parent : httpd and rsyncd are now on arm64:

[smerle33]

first attempt to move WEEKLY.CI.JENKINS.IO to arm64 was a failure, but probably due to my impatience
Next attempt will involved before merging the PR:

• manual scaling +1 arm node
• manually change the statefulset to downsize to 0 to help the volume migration

[smerle33]

first attempt to move WEEKLY.CI.JENKINS.IO to arm64 was a failure, but probably due to my impatience Next attempt will involved before merging the PR:

* manual scaling +1 arm node

* manually change the statefulset to downsize to 0 to help the volume migration

More testing and investigation brought us to discovering a zone incompatibility. ARM VM for the node pool are only available on zone 1 (useast2-1) while our others nodepool are located in zone 3 (useast2-3).
The problem remain on the volumes that are not able to be mounted from one zone to another.
We decided the following plan :

• take a snapshot of the actual weekly.ci.jenkins.io volume and create a new volume on ZRS mode as the price is almost the same (https://azure.microsoft.com/en-us/pricing/details/managed-disks)

• create a PV and PVC using that volume on the ZRS multizone
• use the migrate option of the helm file for weekly (https://github.com/jenkinsci/helm-charts/blob/a052ad9f6fc0f95e67802edc7cef9ab9c3572bee/charts/jenkins/values.yaml#L860-L864) by specifying a pvc to use. https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/

[smerle33]

We created a new class : jenkins-infra/azure#526 to use ZRS storage.
and use a temporary PV/PVC on this volume in order to use it as a source for the migration

it looks like that there was a bug with the CSI volume clone, it failed with the following error on the new PVC created by the cloning system :

Warning  ProvisioningFailed    23s (x6 over 54s)  disk.csi.azure.com_csi-azuredisk-controller-68cfbf9cc6-vknhs_e4bc448f-ce5d-4e2c-9af5-2f47223a8443  failed to provision volume with StorageClass "managed-csi-premium-zrs-retain": rpc error: code = Internal desc = sourceResourceID(/subscriptions/redacted/resourceGroups/mc_publick8s_publick8s-endless-ghoul_eastus2/providers/Microsoft.Compute/disks//subscriptions/redacted/resourcegroups/MC_publick8s_publick8s-endless-ghoul_eastus2/providers/Microsoft.Compute/disks/jenkins-weekly-snap) is invalid, correct format: .*/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Compute/disks/(.+)   

we changed the volumeHandle of the source PV from /subscriptions/redacted/resourcegroups/MC_publick8s_publick8s-endless-ghoul_eastus2/providers/Microsoft.Compute/disks/jenkins-weekly-snapto jenkins-weekly-snap (we exploited the csi clone bug)

Everything went well: WEEKLY.CI.JENKINS.IO now runs on ARM64 🚀