Upgrade to Kubernetes 1.25

[dduportal]

Previous upgrade: #3387

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% Upgrade to Kubernetes 1.23 #3053
2. 92% Upgrade to Kubernetes 1.22 #2930
3. 92% Upgrade to Kubernetes 1.21 #2866
4. 77% [INFRA-3118] Upgrade to Kubernetes 1.20 #2664

[dduportal]

Updating kubectl:

• Update the updatecli version tracking - chore(updatecli) track kubectl 1.25.x (instead of 1.24.x) docker-helmfile#300
• Updatecli automated update to the latest 1.25.x - Bump kubectl version to 1.25.10 docker-helmfile#312
• Deploy to jenkins-infra/kubernetes-management - Bump jenkins-infra helmfile docker image to 2.5.98 kubernetes-management#3962

[dduportal]

Since we have disabled doks due to Digital Ocean outage the 21 June 2023, we are taking the opportunity to upgrade both Digital Ocean clusters to 1.25 before putting back DigitalOcean clusters back to use.

Task list for both DigitalOcean clusters

• No need to announce publicly as we already have an open incident

• Changelog DO-specific: https://docs.digitalocean.com/products/kubernetes/details/changelog/. Other than the components upgrades (expected), 2 majors points:

  • Kubernetes (vanilla) general API removal for 1.25 (of deprecated APIs since 1.19/1.12): https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.25.md#urgent-upgrade-notes
  • DigitalOcean Control plane migration: https://docs.digitalocean.com/products/kubernetes/how-to/upgrade-cluster/#new-control-plane

• Preparation based on the changelog:

  • No blockers
  • The detected deprecated APIs (using pluto) are only on DO-internal components (cilium-operatore and kube-dns). Recycling the worker nodes upgrades to new API: no action needed as the cluster upgrade will take care of it
  • There will be control-plane interruption so we're disabling management of API objects (no more helmfile until migration is complete => feat(doks/doks-public) disabling during upgrade to 1.25 kubernetes-management#4099
  • Since there will be control-plane interruption, given DigitalOcean gaves us credits only for testing the HA control plane, it looks a good incentive to enable it as part of the migration.

• Upgrade both clusters to use HA control planes - feat(doks,doks-public) enable HA for Kubernetes control-planes digitalocean#117

• Upgrade doks using Terraform - feat(doks): upgrade to kubernetes 1.25 digitalocean#121

• Upgrade doks-public using Terraform - feat(doks-public) bump Kubernetes to 1.25.x digitalocean#119

• enable kubernetes-management and check it's working - feat(doks/doks-public) disabling during upgrade to 1.25 kubernetes-management#4099

• Add doks back to ci.jenkins.io and test a pod with ACP - Revert "fix(ci.jenkins.io): disable digital ocean during their network outage" jenkins-infra#2918

• Close status and report here - close DO incident status#330

[Gaoithe]

There is an outage. http://get.jenkins.io/ 13:29:49 UTC Friday, 7 July 2023 ..
Okay, I see you are aware of outage.
Good luck, hope you can fix and recover it without extreme stress, thank you!

[dduportal]

All the public services should be back. We are working on finishing the 1.25 post upgrade steps and we'll publish a post-mortem next week.

[dduportal]

Sub-tasks left beforer closing this issue:

• Postmortem: write it down - blog: add an article about the 2023-07-07 outage jenkins.io#6532
• Postmortem: Add a lock to the public IP to avoid deletion when the clusters are deleted (which transitively removes the "automatic" resource group where the public_ip must be) - feat(locks) add locks on all public IPs that should not change azure#433
• Postmortem: Put back javadoc to arm64 nodepool - hotifx(publick8s) put back javadoc workload to the arm64small nodepool kubernetes-management#4160
• Postmortem: put "as code" the setup of the infraciadmin service account used to administrate clusters (instead of a hidden script on my laptop...) - Tracked in Kubernetes clusters: define infraciadmin SVC account as code #3679
• Prepare the Kubernetes 1.26 issue with all the changes (and non changes) from this release - Upgrade to Kubernetes 1.26 #3683

[timja]

which transitively removes the "automatic" resource group where the public_ip must be

not true, set this label:
service.beta.kubernetes.io/azure-load-balancer-resource-group: myNetworkResourceGroup

https://learn.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address

[lemeurherve]

which transitively removes the "automatic" resource group where the public_ip must be

not true, set this label: service.beta.kubernetes.io/azure-load-balancer-resource-group: myNetworkResourceGroup

https://learn.microsoft.com/en-us/azure/aks/static-ip#create-a-service-using-the-static-ip-address

Thanks! We'll create a test IP in a resource group to check if we can safely move IPs to another resource group without recreating them, then we'll move prod IPs in a dedicated resource group (instead of the cluster node resource group) and add the label to the concerned services.

[dduportal]

Last step: #3683

[dduportal]

😱 Forgot the 1.25 logo:

Ref. https://kubernetes.io/blog/2022/08/23/kubernetes-v1-25-release/