feat: add nightly testing for rke2 #808

noahpb · 2024-09-23T16:24:27Z

Description

This pull request introduces a new nightly test of uds-core on RKE2. The pipeline runs in parallel alongside our existing EKS nightly tests. Included in this PR are additional IaC resources to deploy RKE2, an additional bundle for RKE2 and updated task files/workflows.

Details:

Changes test-infra directory layout to support multiple k8s distributions (merges test-infra/buckets-iac and test-infra/rds-iac into test-infra/aws/eks and adds test-infra/aws/rke2)
Adds IaC for a minimal deployment of RKE2 under test-infra/aws/rke2
Introduces new test bundle for uds-core on RKE2 (.github/bundles/rke2), changes .github/bundles directory layout to support additional distros (moves .github/bundles/*.yaml to .github/bundles/eks)
Introduces new workflow for nightly tests of uds-core on rke2 (.github/workflows/test-rke2.yaml)
Updates workflows and task files to support testing across multiple K8s distros.

Callouts:

uses existing vpc and subnets in ci account
the rke2_version var will be used to query the latest ami for the given minor version
uses rhel amis built from uds-rke2-image-builder repo
includes support for irsa
no cluster-autoscaler deployed, currently hardcoded to provision 4 m5.2xlarge nodes
does not deploy any external dependencies apart from s3 buckets for loki and velero

Example pipeline run with new RKE2 tests can be seen here.

Related Issue

Fixes #726

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Other (security config, docs update, etc)

Checklist before merging

Test, docs, adr added or updated as needed
Contributor Guide followed

make shared terraform more dry make irsa conditional when provisioning s3

update paths in eks iac maru task

…a module in storage module, make all dynamic based on var

tasks/create.yaml

.github/workflows/nightly-testing.yaml

.github/test-infra/aws/rke2/terraform.tfvars

mjnagel

Few last comments here, LGTM overall. Tracking a few potential follow-on pieces of work:

Swap EKS to TF deploy so that we can reuse more and parallelize more IAC for EKS
Add RDS to RKE2 (probably?) to support HA deployment of Keycloak/Grafana
Run validations against all nightly clusters (may need some rework of metallb depending on how tests are run)
Modify PR triggers

.github/workflows/test-eks.yaml

.github/workflows/test-rke2.yaml

noahpb · 2024-10-07T20:52:38Z

Moving to draft temporarily while I validate irsa functionality.

🤖 I have created a release *beep* *boop* --- ## [0.29.0](v0.28.0...v0.29.0) (2024-10-11) ### Features * add base and identity layers ([#853](#853)) ([b3f532a](b3f532a)) * add logging functional layer ([#861](#861)) ([c1a67b9](c1a67b9)) * add metrics-server functional layer ([#865](#865)) ([290367a](290367a)) * add monitoring layer ([#872](#872)) ([5ecb040](5ecb040)) * add nightly testing for rke2 ([#808](#808)) ([c401419](c401419)) * add service accounts options to sso ([#852](#852)) ([1029162](1029162)) * backup and restore layer, ui layer, runtime security layer ([#862](#862)) ([b1d8015](b1d8015)) * grafana-ha ([#838](#838)) ([d532d76](d532d76)) ### Bug Fixes * broken readme link ([#899](#899)) ([6e47b11](6e47b11)) * **ci:** switch to larger runners to resolve ci disk space issues ([#882](#882)) ([1af0401](1af0401)) * snapshot ci version modification and tasks for publish ([#877](#877)) ([f01e5bd](f01e5bd)) * support for anywhere network policies in cilium ([#884](#884)) ([5df0737](5df0737)) ### Miscellaneous * cleanup license parsing for github ([#881](#881)) ([43c98ce](43c98ce)) * **deps:** update chainctl action to v0.2.3 ([#864](#864)) ([d782b59](d782b59)) * **deps:** update checkout action to v4.2.0 ([#825](#825)) ([29d1c98](29d1c98)) * **deps:** update dependency defenseunicorns/lula to v0.8.0 ([#841](#841)) ([fe36150](fe36150)) * **deps:** update githubactions ([#866](#866)) ([44f8ea5](44f8ea5)) * **deps:** update grafana to 11.2.1 ([#836](#836)) ([11383c1](11383c1)) * **deps:** update grafana to v11.2.2 ([#867](#867)) ([06ed2c3](06ed2c3)) * **deps:** update loki nginx image to v1.27.2 ([#894](#894)) ([df7d427](df7d427)) * **deps:** update loki to v3.2.0 ([#791](#791)) ([d3c60b5](d3c60b5)) * **deps:** update metrics-server chart to v3.12.2 ([#873](#873)) ([e2e61ce](e2e61ce)) * **deps:** update pepr to v0.37.1 ([#843](#843)) ([68abcb2](68abcb2)) * **deps:** update pepr to v0.37.2 ([#850](#850)) ([b51f659](b51f659)) * **deps:** update prometheus operator to 0.77.1 ([#819](#819)) ([0864b33](0864b33)) * **deps:** update prometheus-stack ([#855](#855)) ([c791c24](c791c24)) * **deps:** update prometheus-stack helm-charts to v64.0.0 ([#849](#849)) ([50a2588](50a2588)) * **deps:** update runtime to v0.6.0 ([#897](#897)) ([89ae6e2](89ae6e2)) * **deps:** update support-deps ([#890](#890)) ([26ea612](26ea612)) * **deps:** update test-infra ([#875](#875)) ([583f07c](583f07c)) * **deps:** update test-infra to v6.9.0 ([#848](#848)) ([ef9d317](ef9d317)) * **deps:** update uds to v0.17.0 ([#859](#859)) ([1489fef](1489fef)) * **deps:** update zarf to v0.41.0 ([#857](#857)) ([a390c3d](a390c3d)) * **docs:** update doc structure for site refresh ([#895](#895)) ([1946a9a](1946a9a)) * fix broken link in docs ([#845](#845)) ([3078a5b](3078a5b)) * fix license header references ([#901](#901)) ([cf38b82](cf38b82)) * handle upgrade path for functional layers, add doc for usage ([#896](#896)) ([70d6b1b](70d6b1b)) * regroup 'support dependencies' in renovate config ([#885](#885)) ([640d859](640d859)) * update license ([#878](#878)) ([b086170](b086170)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

## Description This pull request introduces a new nightly test of uds-core on RKE2. The pipeline runs in parallel alongside our existing EKS nightly tests. Included in this PR are additional IaC resources to deploy RKE2, an additional bundle for RKE2 and updated task files/workflows. Details: - Changes `test-infra` directory layout to support multiple k8s distributions (merges `test-infra/buckets-iac` and `test-infra/rds-iac` into `test-infra/aws/eks` and adds `test-infra/aws/rke2`) - Adds IaC for a minimal deployment of RKE2 under `test-infra/aws/rke2` - Introduces new test bundle for uds-core on RKE2 (`.github/bundles/rke2`), changes `.github/bundles` directory layout to support additional distros (moves `.github/bundles/*.yaml` to `.github/bundles/eks`) - Introduces new workflow for nightly tests of uds-core on rke2 (`.github/workflows/test-rke2.yaml`) - Updates workflows and task files to support testing across multiple K8s distros. Callouts: - uses existing vpc and subnets in ci account - the `rke2_version` var will be used to query the latest ami for the given minor version - uses rhel amis built from uds-rke2-image-builder repo - includes support for irsa - no cluster-autoscaler deployed, currently hardcoded to provision 4 `m5.2xlarge` nodes - does not deploy any external dependencies apart from s3 buckets for loki and velero Example pipeline run with new RKE2 tests can be seen [here](https://github.com/defenseunicorns/uds-core/actions/runs/11150004170/). ## Related Issue Fixes #726 ## Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Other (security config, docs update, etc) ## Checklist before merging - [ ] Test, docs, adr added or updated as needed - [ ] [Contributor Guide](https://github.com/defenseunicorns/uds-template-capability/blob/main/CONTRIBUTING.md) followed

🤖 I have created a release *beep* *boop* --- ## [0.29.0](v0.28.0...v0.29.0) (2024-10-11) ### Features * add base and identity layers ([#853](#853)) ([b3f532a](b3f532a)) * add logging functional layer ([#861](#861)) ([c1a67b9](c1a67b9)) * add metrics-server functional layer ([#865](#865)) ([290367a](290367a)) * add monitoring layer ([#872](#872)) ([5ecb040](5ecb040)) * add nightly testing for rke2 ([#808](#808)) ([c401419](c401419)) * add service accounts options to sso ([#852](#852)) ([1029162](1029162)) * backup and restore layer, ui layer, runtime security layer ([#862](#862)) ([b1d8015](b1d8015)) * grafana-ha ([#838](#838)) ([d532d76](d532d76)) ### Bug Fixes * broken readme link ([#899](#899)) ([6e47b11](6e47b11)) * **ci:** switch to larger runners to resolve ci disk space issues ([#882](#882)) ([1af0401](1af0401)) * snapshot ci version modification and tasks for publish ([#877](#877)) ([f01e5bd](f01e5bd)) * support for anywhere network policies in cilium ([#884](#884)) ([5df0737](5df0737)) ### Miscellaneous * cleanup license parsing for github ([#881](#881)) ([43c98ce](43c98ce)) * **deps:** update chainctl action to v0.2.3 ([#864](#864)) ([d782b59](d782b59)) * **deps:** update checkout action to v4.2.0 ([#825](#825)) ([29d1c98](29d1c98)) * **deps:** update dependency defenseunicorns/lula to v0.8.0 ([#841](#841)) ([fe36150](fe36150)) * **deps:** update githubactions ([#866](#866)) ([44f8ea5](44f8ea5)) * **deps:** update grafana to 11.2.1 ([#836](#836)) ([11383c1](11383c1)) * **deps:** update grafana to v11.2.2 ([#867](#867)) ([06ed2c3](06ed2c3)) * **deps:** update loki nginx image to v1.27.2 ([#894](#894)) ([df7d427](df7d427)) * **deps:** update loki to v3.2.0 ([#791](#791)) ([d3c60b5](d3c60b5)) * **deps:** update metrics-server chart to v3.12.2 ([#873](#873)) ([e2e61ce](e2e61ce)) * **deps:** update pepr to v0.37.1 ([#843](#843)) ([68abcb2](68abcb2)) * **deps:** update pepr to v0.37.2 ([#850](#850)) ([b51f659](b51f659)) * **deps:** update prometheus operator to 0.77.1 ([#819](#819)) ([0864b33](0864b33)) * **deps:** update prometheus-stack ([#855](#855)) ([c791c24](c791c24)) * **deps:** update prometheus-stack helm-charts to v64.0.0 ([#849](#849)) ([50a2588](50a2588)) * **deps:** update runtime to v0.6.0 ([#897](#897)) ([89ae6e2](89ae6e2)) * **deps:** update support-deps ([#890](#890)) ([26ea612](26ea612)) * **deps:** update test-infra ([#875](#875)) ([583f07c](583f07c)) * **deps:** update test-infra to v6.9.0 ([#848](#848)) ([ef9d317](ef9d317)) * **deps:** update uds to v0.17.0 ([#859](#859)) ([1489fef](1489fef)) * **deps:** update zarf to v0.41.0 ([#857](#857)) ([a390c3d](a390c3d)) * **docs:** update doc structure for site refresh ([#895](#895)) ([1946a9a](1946a9a)) * fix broken link in docs ([#845](#845)) ([3078a5b](3078a5b)) * fix license header references ([#901](#901)) ([cf38b82](cf38b82)) * handle upgrade path for functional layers, add doc for usage ([#896](#896)) ([70d6b1b](70d6b1b)) * regroup 'support dependencies' in renovate config ([#885](#885)) ([640d859](640d859)) * update license ([#878](#878)) ([b086170](b086170)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

noahpb and others added 30 commits September 23, 2024 12:17

initial refactor of iac to support multiple distributions

8bf03a1

make shared terraform more dry make irsa conditional when provisioning s3

Merge branch 'main' into feat/rke2-nightly

36fbc55

wrote changes to wrong file [ci skip]

49aa767

initial commit of minimal rke2 terraform

baf0766

update paths in eks iac maru task

update gitignore

621e611

[ci skip] making things cleaner

76dee10

update nightly workflow trigger paths

00bc448

[ci skip] rm old iac dir

5cb28b7

fix eks tf

60c352e

pass permissions_boundary_name as input

31dcd7c

add PermissionsBoundary default tag

7ed495b

update outputs

5b97455

correct tf outputs

217772f

hopefully this is the last one

f996e98

Merge branch 'main' into feat/rke2-nightly

a174607

fix: refactor the storage module for sharing across distros, nest irs…

2a2bbd0

…a module in storage module, make all dynamic based on var

revert to old tf for eks s3

b4a78c4

tf format

feeddcb

no longer use shared s3 module

e3120c7

fix userdata, get aws ccm working

8138799

required ira and iam resources for the rke2

f0be709

add rke2 storage module, update outputs

36399e5

add statestore module, required for rke2 oidc config

cdacf28

add a module for rke2 s3 and irsa resources

05b3020

update iac task to support mutiple distros

12b1772

fix: incorrect var names in eks uds core bundle

4dc8b0b

add metallb manifest

6c92752

add newlineto outputs.tf

abdfc1c

Merge branch 'main' into feat/rke2-nightly

4870b6a

update bundles

c3430a8

mjnagel reviewed Oct 3, 2024

View reviewed changes

tasks/create.yaml Outdated Show resolved Hide resolved

noahpb added 2 commits October 3, 2024 10:53

build path fixes

69255b7

dont use distro-specific dir for core package

5d586ea

mjnagel reviewed Oct 3, 2024

View reviewed changes

.github/workflows/nightly-testing.yaml Show resolved Hide resolved

mjnagel mentioned this pull request Oct 3, 2024

Separate our PR triggers for RKE2/EKS #856

Closed

mjnagel reviewed Oct 3, 2024

View reviewed changes

.github/test-infra/aws/rke2/terraform.tfvars Outdated Show resolved Hide resolved

mjnagel reviewed Oct 3, 2024

View reviewed changes

.github/workflows/test-eks.yaml Outdated Show resolved Hide resolved

.github/workflows/test-rke2.yaml Outdated Show resolved Hide resolved

.github/workflows/test-rke2.yaml Outdated Show resolved Hide resolved

noahpb added 6 commits October 3, 2024 15:20

rm unneeded step

fc4e575

no --set K8S_DISTRO on pkg create

da498e9

dont need -n in echo

8293422

use most recent patch version ami

062a88b

rm var declaration

dd649ca

add missing .image_id

9384322

noahpb changed the title ~~feat: add nightly testing for rke2~~ draft: feat: add nightly testing for rke2 Oct 7, 2024

noahpb marked this pull request as draft October 7, 2024 20:52

noahpb changed the title ~~draft: feat: add nightly testing for rke2~~ feat: add nightly testing for rke2 Oct 7, 2024

noahpb added 4 commits October 7, 2024 19:02

irsa fixes

6fdefab

Merge branch 'main' into feat/rke2-nightly

d7dfdac

[ci skip] linting fix

fc26a87

tf fmt

ceb3a87

noahpb marked this pull request as ready for review October 8, 2024 12:34

Merge branch 'main' into feat/rke2-nightly

c5cd790

mjnagel approved these changes Oct 8, 2024

View reviewed changes

mjnagel merged commit c401419 into main Oct 8, 2024
18 checks passed

mjnagel deleted the feat/rke2-nightly branch October 8, 2024 14:23

github-actions bot mentioned this pull request Oct 8, 2024

chore(main): release 0.29.0 #842

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add nightly testing for rke2 #808

feat: add nightly testing for rke2 #808

noahpb commented Sep 23, 2024 •

edited

Loading

mjnagel left a comment

noahpb commented Oct 7, 2024

feat: add nightly testing for rke2 #808

feat: add nightly testing for rke2 #808

Conversation

noahpb commented Sep 23, 2024 • edited Loading

Description

Related Issue

Type of change

Checklist before merging

mjnagel left a comment

Choose a reason for hiding this comment

noahpb commented Oct 7, 2024

noahpb commented Sep 23, 2024 •

edited

Loading