Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run automated tests with more than one cluster at a time #2897

Open
pebrc opened this issue Apr 17, 2020 · 1 comment
Open

Run automated tests with more than one cluster at a time #2897

pebrc opened this issue Apr 17, 2020 · 1 comment
Labels
:ci Things related to Continuous Integration, automation and releases >test Related to unit/integration/e2e tests

Comments

@pebrc
Copy link
Collaborator

pebrc commented Apr 17, 2020

We are currently e2e testing typically with only one cluster at a time (exception: remote cluster test). Scale testing has so far been a manual effort (#357 )

While a full blown automated scale test is maybe a lot of work for an unclear return, there is value in testing in automated fashion that ECK can orchestrate more than one cluster at a time. We should therefore:

  • design a test case that runs multiple clusters (10/100?) at a time
    • define success criteria/metrics for this test case (no errors, but should we also for example measure reconciliation throughput or memory footprint of the operator?)
    • define test steps (just a simple deploy, deploy + rolling upgrade?)
  • implement the test case/job
    • it should be runnable independently from our regular e2e test suite given the additional resource requirements (e.g. via separate go build constraints)
    • it needs a CI job/environment that creates a cluster appropriately sized for the larger test case
    • we need to decide when/how to run it. Could be for example part of a pipeline we run after a release candidate build has finished to test for regressions
@pebrc pebrc added :ci Things related to Continuous Integration, automation and releases >test Related to unit/integration/e2e tests labels Apr 17, 2020
@pebrc
Copy link
Collaborator Author

pebrc commented Apr 17, 2020

Potential duplicate of #1121

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:ci Things related to Continuous Integration, automation and releases >test Related to unit/integration/e2e tests
Projects
None yet
Development

No branches or pull requests

1 participant