Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to Kubernetes 1.23 #3053

Closed
23 tasks done
lemeurherve opened this issue Jul 13, 2022 · 7 comments
Closed
23 tasks done

Upgrade to Kubernetes 1.23 #3053

lemeurherve opened this issue Jul 13, 2022 · 7 comments
Assignees
@lemeurherve @smerle33
Labels
kubernetes
Milestone
infra-team-sync-2...

Comments

@lemeurherve
Copy link
Member

lemeurherve commented Jul 13, 2022 •
edited by smerle33
Loading

Previous upgrade: #2930

  • Upgrade kubectl within docker-helmfile

  • Upgrade DOKS (doks and doks-public)

    • Disable kubernetes management (disable job on infra.ci.jenkins.io)
    • Bump Kubernetes version in digitalocean /variables.tf
    • Ensure cluster is re-created by terraform in jenkins-infra/digitalocean
    • Re-Enable kubernetes management (and update kubeconfigs in the infra.ci's SOPS credentials (private repo), if kubernetes management cannot connect)
    • Add the credential of the service account jenkins-agents in ci.jenkins.io if needed (manual step) doks only not doks-public

  • Upgrade EKS

    • Open datetime announcement on status.jenkins.io
    • Disable cluster in ci.jenkins.io configuration as code
    • Upgrade cluster through Terraform (control plane + node groups)
    • Upgrade add-ons if needed
      • If done via the UI, select the checkbox "replace existing configuration"
      • For vpc-cni select the inheritance from node.
    • Ensure the upgraded cluster is still OK (check kubernetes-management) ⚠️ ensure that the security group eks-cluster-sg-<cluster name>-<random ID> that is managed directly by EKS doesn't contain the tag kubernetes.io/cluster/ (the tag is provided by our terraform)
    • Enable it again in ci.jenkins.io config as code

  • Upgrade AKS

    • Check previous AKS upgrades and create one for 1.23
    • Open datetime announcement on status.jenkins.io
    • Upgrade temp-privatek8s first. ⚠️ it will put infra.ci out of service so might need manual steps to re-boostrap the kubernetes-management job
    • Upgrade prodpublick8s
    • Close announcement
@lemeurherve lemeurherve added this to the infra-team-sync-next milestone Jul 13, 2022
@github-actions
Copy link

github-actions bot commented Jul 13, 2022 •
edited
Loading

Take a look at these similar issues to see if there isn't already a response to your problem:

  1. 92% Upgrade to Kubernetes 1.21 #2866
  2. 77% [INFRA-3118] Upgrade to Kubernetes 1.20 #2664

@lemeurherve
Copy link
Member Author

lemeurherve commented Oct 10, 2022 •
edited
Loading

According to DigitalOcean supported Kubernetes release page, the version v1.22 will be "end of support" on the 28th of October, and automatic upgrades will begin 27th of November.

And for Azure, it will be EOL when the version 1.25 will be GA, planned for November.

For AWS, Amazon EKS guarantees support for at least four production-ready versions of Kubernetes at any given time.

To keep in mind for the next milestones.

smerle33 added a commit to jenkins-infra/docker-helmfile that referenced this issue Oct 12, 2022
@smerle33
feat(kubectl): update kubectl to 1.23
d4932da 
as per jenkins-infra/helpdesk#3053 upgrading the client kubectl
@smerle33 smerle33 mentioned this issue Oct 12, 2022
Merged
smerle33 added a commit to jenkins-infra/digitalocean that referenced this issue Oct 12, 2022
@smerle33
feat(kub 1.23): upgrade kubernetes to 1.23
8bd0476 
as per jenkins-infra/helpdesk#3053
@smerle33 smerle33 mentioned this issue Oct 12, 2022
Merged
@smerle33
Copy link
Contributor

smerle33 commented Oct 12, 2022 •
edited
Loading

Here is a link to hackdays notes for the changelog from 1.22 : https://hackmd.io/z0sMiMMpReenvVvheoZCMA

A link to the dedicated changelog for DigitalOcean : https://docs.digitalocean.com/products/kubernetes/details/changelog/#1-23-x

@dduportal
Copy link
Contributor

dduportal added a commit to jenkins-infra/digitalocean that referenced this issue Oct 20, 2022
@smerle33 @dduportal
feat(doks-public): upgrade kubernetes to 1.23 (#78)
ca24a08 
* feat(kub 1.23): upgrade kubernetes to 1.23

as per jenkins-infra/helpdesk#3053

* feat(doks/doks-public) split kubernetes versions per cluster: 1.22 for doks and 1.23 for doks-public

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

* fixup

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

* fixup

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>

Signed-off-by: Damien Duportal <damien.duportal@gmail.com>
Co-authored-by: Damien Duportal <damien.duportal@gmail.com>
This was referenced Oct 24, 2022
Merged
Merged
@dduportal dduportal mentioned this issue Oct 24, 2022
Merged
@lemeurherve lemeurherve mentioned this issue Oct 24, 2022
Merged
@dduportal dduportal mentioned this issue Oct 24, 2022
Merged
This was referenced Oct 24, 2022
Merged
Merged
@dduportal dduportal mentioned this issue Oct 25, 2022
Merged
@dduportal
Copy link
Contributor

dduportal commented Oct 26, 2022 •
edited by smerle33
Loading

Note related to AKS clusters: as per https://github.com/Azure/AKS/blob/master/CHANGELOG.md#release-2022-10-17, there are 2 changes we have to look for, after the upgrade, about our Windows node pools:

  • Switching to containerd
  • Ensuring that the labels/taints/tolerations are not deprecated

@dduportal dduportal mentioned this issue Oct 26, 2022
Merged
@smerle33
Copy link
Contributor

AKS upgrade checklist : https://github.com/jenkins-infra/documentation/blob/main/maintenance/kubernetes/aks-1.23.md

@lemeurherve lemeurherve mentioned this issue Feb 14, 2023
Closed
32 tasks
@github-actions github-actions bot mentioned this issue May 12, 2023
Closed
@github-actions github-actions bot mentioned this issue Jul 26, 2023
Closed
7 tasks
@lemeurherve
Copy link
Member Author

Forgot the most important, the release logo:

image

@github-actions github-actions bot mentioned this issue Feb 12, 2024
Closed
8 tasks
@github-actions github-actions bot mentioned this issue Jun 18, 2024
Closed
13 tasks
@github-actions github-actions bot mentioned this issue Jul 4, 2024
Closed
12 tasks
@github-actions github-actions bot mentioned this issue Aug 23, 2024
Open
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lemeurherve lemeurherve

@smerle33 smerle33

Labels
kubernetes
Projects
None yet
Milestone
infra-team-sync-2022-11-08
Development

No branches or pull requests
3 participants
@dduportal @lemeurherve @smerle33