[ci.jenkins.io] Migrate ci.jenkins.io EKS clusters out from CloudBees AWS account

[dduportal]

Service(s)

AWS, Azure, ci.jenkins.io, sponsors

Summary

Today, ci.jenkins.io utilizes 2 EKS clusters to spin up ephemeral agents (for plugin and BOM builds). These clusters are hosted in a CloudBees-sponsored account (historically used to host a lot of Jenkins services).

We want to move these clusters out of CloudBees AWS to ensure non CloudBees Jenkins contributors can manage it and to use credits from other sponsors as AWS, DigitalOcean and Azure gave us credits to be used.

Initial working path (destination: AWS sponsored account)

AWS is sponsoring the Jenkins project with $60.000 for 2024, which are applied to a fresh new AWS account.

We want to migrate the 2 clusters used by ci.jenkins.io into this new AWS account:

• Moving out from CloudBees-owned AWS account allows non CloudBees employees to help managing these resources
• Consuming these credits is key to ensure we can continue sponsor on long term

Updated working path

As discussed during the 2 previous infra SIG meetings, we have around 28k$ credits on the Azure sponsored account which expires end of August 2024 (was May 2024 but @MarkEWaite asked for extension of this deadline ❤️ ), while both DigitalOcean and AWS (non CloudBees) accounts have credits until January 2025.

=> As such, let's start by using a Kubernetes cluster in Azure (sponsored) AKS to use these credits until end of summer before moving to the new AWS account

---

Notes 📖

A few elements for planning these migrations:

• This is a good opportunity to re-assess the naming convention we used for jenkins-infra/aws project:

  • cik8s and eks-public for instance...

• The terraform module for EKS has a major upgrade version currently waiting (20.x): Bump version of the Terraform module "eks" to 20.11.0 aws#517 . It features breaking changes around the management of the EKS configmap. Upgrading the module by using the new version on fresh new cluster would avoid a tedious migration of existing ones...

  • Note: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/UPGRADE-20.0.md#%EF%B8%8F-upcoming-changes-planned-in-v210-%EF%B8%8F is important to have in mind!

• We have an upcoming Kubernetes 1.27 upgrade: it will most probably be applied to AWS cluster before, but we have to keep it in mind

• We'll have to define at least 2 different AWS providers in the Terraform project to allow management of both accounts at the same time: https://build5nines.com/terraform-deploy-to-multiple-aws-accounts-in-single-project/ (we already have this kind of pattern with Azure)

Reproduction steps

No response

[dduportal]

First things first: connected to the account with the jenkins-infra-team account (and its shared TOTP for 2FA) and was able to confirm we have the $60,000 credits:

[dduportal]

Update: proposal to boostrap the AWS account. To be discussed and validated during the next weekly team meeting.

• Root account:

  • Should only be used for initial permission bootstrap, to add/remove Jenkins Infra team members or for account-level (exceptional) actions such as payment, credits management, etc.
  • It has a shared password and a shared TOTP (MFA) encrypted using our SOPS system (GPG keys).

• Each Jenkins Infra team member ("OPS") will have a nominative AWS account with mandatory password and MFA, no API access (only Web Console) and only the permission to assume a role based on their "trust" level.

• The following roles are proposed:

  • infra-admin: allows management of usual resources (EC2, EKS, S3, etc.) but also access (read only) to billing
  • infra-user: allows management of usual resources (EC2, EKS, S3, etc.)
  • infra-read: allows access (read-only) of usual resources (EC2, EKS, S3, etc.)

• The infrastructure as code (jenkins-infra/aws, Terraform project) will have 2 IAM users, and each one will only be able to assume a role.

• The "Assume Role" means AWS STS will be used to generate 1 hour valid token (e.g. whether Web Console or API is used, the credential is only valid 1 hour). It will require additional commands for end users or Terraform but it will avoid keeping APi keys unchanged for months (years?).

• We won't use the AWS IAM Identity Center as it is overkill (we only have one AWS account with just a few resources).

• We won't deploy stuff outside of a base region (eventually 2), in a single AZ per region (no HA: it fails, then it fails).

• The scope of resources must only be ephemeral workloads. Ideally for ci.jenkins.io: public services so the workloads are considered unsafe and untrusted by default (so no mix up with other controllers such as infra.ci.jenkins.io).

[dduportal]

Update:

Stop managing these clusters

jenkins-infra/kubernetes-management#5243

Remove ci.jenkins.io configurations for these clusters

jenkins-infra/jenkins-infra#3442

Delete these clusters from clouds (jenkins-infra/aws and jenkins-infra/digitalocean)

• DigitalOcean:
  • cleanup: empty doks and doks-public Kubernetes clusters digitalocean#198
  • cleanup: delete doks and doks-public Kubernetes clusters digitalocean#199
• AWS:
  • cleanup: empty cik8s and eks-public EKS clusters aws#555
  • cleanup: remove management and leftovers of cik8s and eks-public EKS clusters aws#554

=> also, forgot to disable monitors, reminded by @smerle33 and done in jenkins-infra/datadog#250

[dduportal]

Update: this issue is closable:

• Cleanup is finished:

  • Removed leftovers DNS records pointing to the doks-public cluster (for ACP and PoC of the new update center)
    • cleanup(DNS records) remove Digital Ocean Kubernetes cluster DNS records azure-net#240
    • cleanup(updates.jenkins.io) removes the (unused) digitalocean.updates.jenkins.io CNAME record azure#707
  • Updated documentation and runbooks to remove references of these 2 clusters:
    • docs(ci) remove cik8s and doks clusters references documentation#31
    • https://github.com/jenkins-infra/runbooks/pull/83
    • cleanup(jenkins-jobs) remove references to cik8s deleted cluster helm-charts#1166

• Confirmation that the AWS incurred costs are decreasing since the 17 May 2024: