Ubuntu 22.04 upgrade campaign

[dduportal]

In jenkinsci/packaging#317, there has been a work to use Ubuntu 22.04 Jammy Jellyfish the latest LTS.

This issue tracks the tasks and work to get all our resources to use Ubuntu 22.04, which might have consequences (as per #2978):

• openssl
• curl
• git
• createrepo
• etc.

List of resources that would need an update:

• jenkins-infra/packer-images: agent templates (used by our Jenkins controllers). Using Ubuntu 20.04 for now.
  • Easy to change (full IAC)
  • We have to be careful as it might break some ci.jenkins.io use cases.
• jenkins-infra/jenkins-infra: VMs managed by puppet, such as pkg.jenkins.io
  • We need to validate on a test machine that our current version of puppet works with Ubuntu 22.04 (not only the agent, but we also have a risk of some modules not working at runtime). @olblak had some surprises when migrating archives.jenkins.io to oracle (from "Ubuntu 18.04, Intel CPU, on AWS EC2" to "Ubuntu 20.04, ARM64 CPU, on Oracle cloud) last year.
  • Machine per machine, we have to communicate, and backup to allow rollbacks
• Container images:
  • Besides jenkins-infra/docker-packaging, it does not seem we have other base images on ubuntu. To be double check by someone else than me.

[dduportal]

For information @smerle33 caught that the puppet and puppet-agent packages are not yet available to 22.04 while working on #2649 .

There is an official issue for Puppet OpenSource tracking the support: https://tickets.puppetlabs.com/browse/PA-4233.
As for today, there is a "nightly" build only: https://groups.google.com/g/puppet-users/c/3IWiwwG9Ydo/m/Tzh7xrKwAAAJ

[dduportal]

For info, AKS swicthes its nodes to Ubuntu 22.04 starting with Kubernetes 1.25 as per https://github.com/Azure/AKS/releases/tag/2022-10-02

[dduportal]

Ubuntu 22.04 is now used for the Linux VM and container agents for our controllers (except a few edge cases):

• PR in the packer-image templates: breaking: bump from Ubuntu 20.04 to Ubuntu 22.04 packer-images#299
• Release: https://github.com/jenkins-infra/packer-images/releases/tag/1.0.0
• Deployment for the VM agents (Azure +EC2) of infra.ci.jenkins.io: chore: upgrade packer images to Ubuntu 22.04 jenkins-infra#2740
• Deployment for the VM agents (Azure +EC2) and container of ci.jenkins.io: chore: upgrade packer images to Ubuntu 22.04 jenkins-infra#2740

[dduportal]

Ubuntu 22.04 is now used for the Linux VM and container agents for our controllers (except a few edge cases):

* PR in the packer-image templates: [breaking: bump from Ubuntu 20.04 to Ubuntu 22.04 packer-images#299](https://github.com/jenkins-infra/packer-images/pull/299)

* Release: https://github.com/jenkins-infra/packer-images/releases/tag/1.0.0

* Deployment for the VM agents (Azure +EC2) of infra.ci.jenkins.io: [chore: upgrade packer images to Ubuntu 22.04 jenkins-infra#2740](https://github.com/jenkins-infra/jenkins-infra/pull/2740)

* Deployment for the VM agents (Azure +EC2) and container of ci.jenkins.io: [chore: upgrade packer images to Ubuntu 22.04 jenkins-infra#2740](https://github.com/jenkins-infra/jenkins-infra/pull/2740)

Note that Ubuntu 22.04 created some issues in jenkinsci/packaging#388, but fixed.

[dduportal]

• Migrate trusted.ci.jenkins.io from AWS to Azure #3486 for trusted.ci 3 machines
• Check the Ubuntu versions (if any) on AKS, and eventually DigitalOcean
• docker-openvpn => uses 18.04 as base image

[dduportal]

Update of cert.ci finished with success. Had to update puppet configuration with jenkins-infra/jenkins-infra#2898

[dduportal]

Next candidates:

• VM for census.jenkins.io (in AWS, Ubuntu 18.04)
• VM for usage.jenkins.io (in AWS, Ubuntu 18.04)
• VM for archives.jenkins.io (in Oracle, Ubuntu 20.04)

[dduportal]

Next candidates:

* VM for `census.jenkins.io` (in AWS, Ubuntu 18.04)

* VM for `usage.jenkins.io` (in AWS, Ubuntu 18.04)

* VM for `archives.jenkins.io` (in Oracle, Ubuntu 20.04)

These 3 VMs have been upgraded to 22.04 with success

[dduportal]

Remaining Ubuntu 18.04 to migrate to 22.04 (only puppet master stays in 20.04 for now):

• vpn.jenkins.io (easy to do)
• s390x permanent agent of ci.jenkins.io
• ci.jenkins.io, tracked in [ci.jenkins.io] Use a new VM instance type #3535
• publick8s and privatek8s AKS Linux worker nodes, tracked in Upgrade to Kubernetes 1.25 #3582
• updates.jenkins.io (pkg.origin.jenkins.io) hard to do as a lot of packages used to generates packages are tied to ubuntu 22.04. Gotta sync with release team to see the next steps:
  • Install Docker and use it with an Ubuntu 18.04 image on short term (easier to test and upgrade without triggering a full release)
  • Implement [INFRA-3100] Migrate updates.jenkins.io to another Cloud #2649 somehow
  • Something else?

[dduportal]

Post mortem of vpn.jenkins.io's upgrade to Ubuntu 22.04 (18.04 -> 20.04) which broke the VPN to ci.jenkins.io (SSH) and cert.ci.jenkins.io (HTTPS) for a few hours:

• Disabled Puppet ✅

• Updated all packages, cleanup of remnant packages (apt remove --purge of all items from dpkg -l | grep '^rc' ) and reboot as a "cleanup" operation ✅

• Upgrade to focal (20.04) + reboot ✅

• Upgrade to jammy (22.04) + reboot ❌

  • Update itself went well, VPN still "working" (can connect) but no more routing: imposible to reach cert.ci.jenkins.io in HTTPS or ci.jenkins.io on SSH (ci.jenkins.io in HTTPS still working as it uses public IP)

• After 2 hours of outage, found the 2 root issues, both caused by iptables new version in Ubuntu 22.04 which is stricter with rules order and definitions (which is good!) 🔧

  • VPN tun0 Network was misconfigured: sometimes the netwmask was 255.255.255.0 and sometimes it was 255.255.254.0
  • the default policy for the chain FORWARD (to cross networks with nat rules) was DROP instead of ACCEPT: all packets were dropped

• Ensured puppet code is using Ubuntu 22.04 (except for ci.jenkins.io and pkg.origin.jenkins.io) before enabling it again ✅

  • feat: make Ubuntu 22.04 the default jenkins-infra#2920
  • tests(Vagrant) fix vagrant for ubuntu 22.04 jenkins-infra#2923

• Fixes applied manually to validate and then persisted in the code with the following PRs ✅ :

  • fix(default) updates and corrects network definitions docker-openvpn#283 deployed with Bump OpenVPN Docker image version to 2.0.29 jenkins-infra#2921
  • fix(openvpn) add ports 22 and 5432 to iptables nat rules + ensure default forward policy is accept jenkins-infra#2922 for the policy chain + explicit firewall rule for SSH 22 (bonus: added 5432 for future database forwarding)

• Enabled Puppet + last reboot before final check ✅

[dduportal]

ci.jenkins.io is now running with Ubuntu jammy ✅ (ref. #3535)

[dduportal]

Removing this issue from any milestone as it required the following items to be finished:

• Update (or removal) of the AWS VM "pkg.origin.jenkins.io/updates.jenkins.io" is tracked in the 2 following issues:
  • PKG: [pkg.jenkins.io] migrate the pkg.origin.jenkins.io service from AWS VM to Azure publick8s #3705
  • UC: [INFRA-3100] Migrate updates.jenkins.io to another Cloud #2649
• Update of the Puppet Master requires Puppet 7/8 upgrade in the Community edition: Puppet Upgrade Campaign to Community Edition (7.x or 8.x) #3058