-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
E2E test optimization for multi-distro support #319
Merged
Merged
Changes from 26 commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
d3e65b2
Initial attempt at reworking e2e tests
YrrepNoj 865e70f
adding the rest of the old e2e tests
YrrepNoj a439a6a
wip: cleaning up tests
YrrepNoj 1c7e083
wip k3d, silly go mod
YrrepNoj fb82c4b
dear diary, did I do good?
jeff-mccoy d04a07b
change k3d version for testing to 5.2.1
YrrepNoj 777dd64
cleanup e2e test for multi distro testing
YrrepNoj 81df250
add unique github action for testing each e2e distro
YrrepNoj 953d480
turns out its useful to build dependencies
YrrepNoj 7390004
fix setup for k3s testing
YrrepNoj fc8f9e0
try to make k3s run as root during e2e pileine
YrrepNoj 718a372
make sure we clean up kubeconfig after e2e test
YrrepNoj 2083465
run k3s e2e test as root
YrrepNoj 3bbde2d
wip, extend time and break apart build steps
YrrepNoj 3ae1afe
doanlod dependencies early for k3s
YrrepNoj bf77ced
remove old e2e content
YrrepNoj 107f10f
misc test code cleanup
YrrepNoj e604547
more test code cleanup
YrrepNoj e4874b7
add more docs to e2e README
YrrepNoj 9a9e710
update key for registry1 e2e login
YrrepNoj d9b7a6c
Re-add cloud e2e test for the game example
YrrepNoj c13b70a
add k3s e2e workflow
YrrepNoj d077518
remove chat based e2e test
YrrepNoj e2bfd36
cleanup of e2e test code
YrrepNoj f6fc96c
Merge branch 'master' into 100-e2e-test-refresh
jeff-mccoy f915f8a
adr for e2e testing
YrrepNoj 7aa6cd7
Merge branch 'master' into 100-e2e-test-refresh
jeff-mccoy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| name: e2e-k3d | ||
| on: | ||
| - pull_request | ||
|
|
||
| jobs: | ||
| build-and-test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Install GoLang | ||
| uses: actions/setup-go@v2 | ||
| with: | ||
| go-version: 1.16.x | ||
| - name: Checkout Repo | ||
| uses: actions/checkout@v2 | ||
| - name: Build CLI | ||
| run: make build-cli-linux | ||
| - name: Make Packages | ||
| run: make init-package package-example-game package-example-data-injection package-example-gitops-data | ||
| - name: Run Tests | ||
| run: TESTDISTRO=k3d make test-e2e |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| name: e2e-k3s | ||
| on: | ||
| - pull_request | ||
|
|
||
| jobs: | ||
| build-and-test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Install GoLang | ||
| uses: actions/setup-go@v2 | ||
| with: | ||
| go-version: 1.16.x | ||
| - name: Checkout Repo | ||
| uses: actions/checkout@v2 | ||
| - name: Build CLI | ||
| run: make build-cli-linux | ||
| - name: Make Packages | ||
| run: make init-package package-example-game package-example-data-injection package-example-gitops-data | ||
| - name: Run Tests | ||
| # NOTE: "PATH=$PATH" preserves the default user $PATH. This is needed to maintain the version of go installed in a previous step | ||
| run: sudo env "PATH=$PATH" TESTDISTRO=k3s make test-e2e |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| name: e2e-kind | ||
| on: | ||
| - pull_request | ||
|
|
||
| jobs: | ||
| build-and-test: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Install GoLang | ||
| uses: actions/setup-go@v2 | ||
| with: | ||
| go-version: 1.16.x | ||
| - name: Checkout Repo | ||
| uses: actions/checkout@v2 | ||
| - name: Build CLI | ||
| run: make build-cli-linux | ||
| - name: Make Packages | ||
| run: make init-package package-example-game package-example-data-injection package-example-gitops-data | ||
| - name: Run Tests | ||
| run: TESTDISTRO=kind make test-e2e |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| # 2. Moving E2E Tests Away From Terratest | ||
|
|
||
| Date: 2022-03-04 | ||
|
|
||
| ## Status | ||
|
|
||
| Accepted | ||
|
|
||
| ## Context | ||
|
|
||
| In previous releases of Zarf, the creation of the initialization package at the core of many of our E2E tests required repository secrets to login to registry1. Since this is an open-source project, anyone could submit a change to one of our GitHub workflows that could steal our secrets. In order to protect our secrets from any bad-actors we used [peter-evans/slash-command-dispatch@v2](https://github.com/peter-evans/slash-command-dispatch) so that only a maintainer would have the ability to run the E2E tests when a PR is submitted for review. | ||
|
|
||
| In the current version of Zarf (v0.15) images from registry1 are no longer needed to create the zarf-init.tar.zst. This means, given our current span of E2E tests, we no longer need to use repository secrets when running tests. This gives us the ability to reassess the way we do our E2E testing. | ||
|
|
||
| When considering how to handle the tests, some of the important additions we were considering were: | ||
| 1. Ability to test against different kubernetes distributions | ||
| 2. Ability to test against different linux distributions | ||
| 3. Ability to run (at least some of) the E2E tests locally without relying on an ec2 instance - for quicker feedback loops when developing new features | ||
|
|
||
| ## Decision | ||
|
|
||
| The previous E2E test code was not extensible enough to be reused to test Zarf against different kubernetes distributions. The test suite was refactored so that we could write a setup and teardown function for each kubernetes distribution we wanted to verify against and the test suite was then responsible for cycling through the different distributions. This gives us the ability to test multiple kubernetes distributions against the same exact test cases. | ||
|
|
||
| The individual test cases were also rewritten to not rely on terratest running a bash command over ssh. Instead, the test uses the locally built Zarf binary and example packages to validate expected behavior. This approach works both on local dev machines (linux/mac) and on the Ubuntu GitHub Runner that gets triggered when a pull request is created. This also has the positive side effect of not needing to wait several minutes for an ec2 instance to spin up for testing. | ||
|
|
||
| Since we no longer need repository secrets to run the E2E tests, we removed the requirement for a maintainer to use a `/test all` chatops command to dispatch the tests. Instead, there is a new test workflow defined for each kubernetes distribution we are verifying against and the tests get run automatically whenever a PR is created or updated. | ||
|
|
||
| When looking back at the list of 'important additions' we were considering above. All three are addressed with this approach. Testing against different kubernetes distributions is as simple as defining how to create and destroy the cluster. All of the test cases are runnable locally and then because of that testing on a linux distribution is possible by just switching over to another machine and running the same `make test-e2e` there. This also gives us the ability to test against cloud distributions like EKS! All you need is a valid kubeconfig and running `go test ./...` in the `./test/e2e` directory will run all of the test cases against the EKS cluster. | ||
|
|
||
| ## Consequences | ||
|
|
||
| While it was not something we were doing before, testing directly on the GitHub Runner instead of using Terratest to test on an ec2 instance means that when we get around to adding automated testing of Zarf against different linux distrobutions we will want to have more discussions on if we want to use self-hosted runners of different OS's or if we want to go back to Terratest to stand up ec2 instances with different AMIs. | ||
|
|
||
| In the future, we will likely want to write E2E tests that use images that require repository secrets to access. When that happens we will want to bring back some form of 'maintainer action' to initiate the test workflow. Going back to [peter-evans/slash-command-dispatch@v2](https://github.com/peter-evans/slash-command-dispatch) might be the right answer for that but there will need to be more discussion to make sure the team agrees that's the best solution first. | ||
|
|
||
| As the amount of E2E tests we have grows, the longer it will take to get results back on each PR. Duhh! Paralyzing the tests on a single host will be difficult (but not impossible) because it means more logic will be needed in the test suite to make sure the host has enough resources to handle multiple clusters being run in parallel. The simpler solution would be to break out each of our tests into a new GitHub Workflow. This will easily mitigate the issue of tests taking long to run but also give us a lot more yaml to maintain. Either solution is valid but more team discussions will be needed as we get closer to needing it. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You actually don't have to do
zarf init, when you run a package deploy or evenzarf zarf-init.tar.zst, it should do the exact same thing now. Init is nothing more than a fixed shortcut right now to use the init package in the same directory.