enhancement(tests): Kubernetes E2E test framework #2702
Conversation
|
Great PR! Please pay attention to the following items before merging: Files matching
This is an automatically generated QA checklist based on modified files |
MOZGIII
force-pushed
the
k8s-test-framework
branch
from
674d950 to
d81b5dd
Compare
MOZGIII
changed the title
MOZGIII
changed the title
MOZGIII
force-pushed
the
k8s-test-framework
branch
2 times, most recently
from
1c96467 to
8c792a6
Compare
MOZGIII
force-pushed
the
k8s-test-framework
branch
from
8c792a6 to
01e81f4
Compare
MOZGIII
force-pushed
the
k8s-impl
branch
3 times, most recently
from
aa1c6c7 to
8fdfa92
Compare
MOZGIII
force-pushed
the
k8s-test-framework
branch
from
01e81f4 to
b83d8fc
Compare
MOZGIII
force-pushed
the
k8s-test-framework
branch
2 times, most recently
from
0a041fa to
c4baf90
Compare
MOZGIII
force-pushed
the
k8s-impl
branch
2 times, most recently
from
a3a882e to
2fe4731
Compare
MOZGIII
force-pushed
the
k8s-test-framework
branch
from
88b06a9 to
c63e8ef
Compare
MOZGIII
force-pushed
the
k8s-test-framework
branch
4 times, most recently
from
55827ad to
74ab186
Compare
MOZGIII
force-pushed
the
k8s-test-framework
branch
4 times, most recently
from
7387876 to
5217151
Compare
MOZGIII
force-pushed
the
k8s-impl
branch
2 times, most recently
from
cb14221 to
c0f2131
Compare
|
You're hitting a |
|
Ok. :) Consider this unblocked then. |
Signed-off-by: MOZGIII <mike-n@narod.ru>
|
Another issue: kubernetes/minikube#8840 |
|
@MOZGIII just checking in. What's remaining here? @Hoverbear given your review, do you approve or still need changes made? |
|
I'll see if minikube has been updated and try to rerun. |
|
So I'd like |
|
We can autodetect it. See #2702 (comment) However, the command will still not be really "usable" as the rest of the test commands: those are optimized for local run, while E2E is targeted towards optimizing the CI run use case - the reason for that is it's main goal is to test release builds, rather than debug. The overall theme of this test requiring manual involvement from the user remains, and it's sort of by design. |
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
|
Okay so we need to wait for minikube to update before we merge? It still seems like
Can you explain why you think it would be more surprising to a user that a test run is using their docker credentials to push images? As opposed to use a local cache on the minikube they'll be testing against? |
|
I think ec3f9c5 solves the |
|
I think everything's sorted out. Are we ready to merge?
It's slightly more complicated. |
I don't think it worth it. We can just keep using a working version. There are relevant TODOs in place, and issues at upstream are opened. |
Signed-off-by: MOZGIII <mike-n@narod.ru>
|
Ok! |
|
So I noted as a ZFS user minikube can't work. So I had to use microk8s which does work very well if you do not have |
|
@MOZGIII I wonder if we can make |
It doesn't work on mac, and has some other limitations that make it poorly suitable to be the default. |
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
Signed-off-by: MOZGIII <mike-n@narod.ru>
|
Ok! |
* Correct test-integration-kubernetes at Makefile Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix the tag overwrite logic at scripts/deploy-kubernetes-test.sh Signed-off-by: MOZGIII <mike-n@narod.ru> * Make scripts/test-integration-kubernetes.sh more tweakable Signed-off-by: MOZGIII <mike-n@narod.ru> * Reorder namespace and global config deletion command The idea is namespace removal takes the longest, so we'd rather leave it hanging than config deletion. Then is user gets tired of waiting and sends a SIGINT we don't leave the global config dangling - just the namespace removal, which will complete in the background. So it's just a user experience improvement. Signed-off-by: MOZGIII <mike-n@narod.ru> * Add kubernetes-test-framework Signed-off-by: MOZGIII <mike-n@narod.ru> * Implement a first PoC kubernetes test Signed-off-by: MOZGIII <mike-n@narod.ru> * K8s integration test is really an e2e test, rename accordingly Signed-off-by: MOZGIII <mike-n@narod.ru> * Do not even publish container image at CI since we use "none" minikube driver Signed-off-by: MOZGIII <mike-n@narod.ru> * Isolate kubernetes e2e tests via requried-features Signed-off-by: MOZGIII <mike-n@narod.ru> * Add lock to the test framework Signed-off-by: MOZGIII <mike-n@narod.ru> * Add some test cases to k8s e2e tests Signed-off-by: MOZGIII <mike-n@narod.ru> * Add the ability to use quick debug builds in e2e tests Useful to speed up the development cycles Signed-off-by: MOZGIII <mike-n@narod.ru> * Use a single thread for test Signed-off-by: MOZGIII <mike-n@narod.ru> * Made test framework async Signed-off-by: MOZGIII <mike-n@narod.ru> * Allow specifying scope Signed-off-by: MOZGIII <mike-n@narod.ru> * Correct arguments preparation for cargo test at scripts/test-e2e-kubernetes.sh Signed-off-by: MOZGIII <mike-n@narod.ru> * Get rid of $(RUN) at test-e2e-kubernetes target at Makefile Signed-off-by: MOZGIII <mike-n@narod.ru> * Set LOG at distribution/kubernetes/vector-namespaced.yaml Signed-off-by: MOZGIII <mike-n@narod.ru> * Add a test to validate the pods are properly excluded This tool a while to implement, and required that we make framework async. Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix a typo Signed-off-by: MOZGIII <mike-n@narod.ru> * Add test to assert we properly collect logs from multiple namespaces Signed-off-by: MOZGIII <mike-n@narod.ru> * Polish the test framework API Signed-off-by: MOZGIII <mike-n@narod.ru> * Add E2E tests section to the contribution guide Signed-off-by: MOZGIII <mike-n@narod.ru> * Kubernetes E2E tests are no longer experimental, should work consistently Signed-off-by: MOZGIII <mike-n@narod.ru> * Add kubernetes version to the test name Signed-off-by: MOZGIII <mike-n@narod.ru> * Bump minikube Signed-off-by: MOZGIII <mike-n@narod.ru> * Bump kubernetes releases Signed-off-by: MOZGIII <mike-n@narod.ru> * Use minikube cache instead of manually moving image around Signed-off-by: MOZGIII <mike-n@narod.ru> * Test against multiple container runtimes Signed-off-by: MOZGIII <mike-n@narod.ru> * Remove unused repeating_echo_cmd Signed-off-by: MOZGIII <mike-n@narod.ru> * Display timeout Signed-off-by: MOZGIII <mike-n@narod.ru> * Shorter title Signed-off-by: MOZGIII <mike-n@narod.ru> * Switch to docker driver at minikube Signed-off-by: MOZGIII <mike-n@narod.ru> * Remove the no_newline_at_eol test Turns out, this test was invalid. The root cause with this is that, in essence, Kubernetes expects logs to consist of line, with line being defined as in POSIX - a sequence of characters *ending with \n*. Thus it's *not valid* to emit a log line without the terminating newline symbol in Kubernetes. One effect of this is that when using the CRI log format, lines won't be considered complete until we emit a newline character arrives - and the additional content before the newline will be added to the log line that's missing the newline. Given all of the above, there's no reason for this test to exist. The reason it was added was the behaviour detail of the docker log driver, but it's a mere implementation detail, and it we should abstract from it. The original statement of the test is also ill-posed, cause, as explained above, it's non-partial messages (and, generally speaking, any message) that doesn't end with newline isn't a valid log line in the first place. Signed-off-by: MOZGIII <mike-n@narod.ru> * Increase timeout to rollout vector to 30s Signed-off-by: MOZGIII <mike-n@narod.ru> * Temporarily disable crio Signed-off-by: MOZGIII <mike-n@narod.ru> * Apply workaround for CRIO Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix clippy Signed-off-by: MOZGIII <mike-n@narod.ru> * Unset log level in skaffold dev config to fallback to the one set in container Signed-off-by: MOZGIII <mike-n@narod.ru> * Add exec_tail to the test framework Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix a typo at the comment Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix the typos and styling at the crate doccomment Signed-off-by: MOZGIII <mike-n@narod.ru> * Bump k8s versions for E2E tests at CI Signed-off-by: MOZGIII <mike-n@narod.ru> * Rename template params to pascal case Signed-off-by: MOZGIII <mike-n@narod.ru> * Remove Drop from ResourceFile Signed-off-by: MOZGIII <mike-n@narod.ru> * Proper authors Signed-off-by: MOZGIII <mike-n@narod.ru> * Rename crate to k8s-test-framework More in-line with the naming patterns of the rest of the k8s-related crates. Signed-off-by: MOZGIII <mike-n@narod.ru> * Correct kubectl comment at the interface Signed-off-by: MOZGIII <mike-n@narod.ru> * Bumped k8s and minikube versions at CI Signed-off-by: MOZGIII <mike-n@narod.ru> * Add a comment explaining the timeout at pod filtering test Signed-off-by: MOZGIII <mike-n@narod.ru> * Rollback minikube to 0.11.0 Signed-off-by: MOZGIII <mike-n@narod.ru> * Update CONTRIBUTING.md Co-authored-by: Ana Hobden <operator@hoverbear.org> Signed-off-by: MOZGIII <mike-n@narod.ru> * Update distribution/kubernetes/vector-namespaced.yaml Co-authored-by: Ana Hobden <operator@hoverbear.org> Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix an error at CONTRIBUTING.md Signed-off-by: MOZGIII <mike-n@narod.ru> * Remove a trivial line from the doc Signed-off-by: MOZGIII <mike-n@narod.ru> * Do second attemtp to start up minikube if the first one failed Signed-off-by: MOZGIII <mike-n@narod.ru> * Print minikube logs if it fails to start Signed-off-by: MOZGIII <mike-n@narod.ru> * Provide a default for CONTAINER_IMAGE_REPO if USE_MINIKUBE_CACHE is set Signed-off-by: MOZGIII <mike-n@narod.ru> * Update the CONTRIBUTING.md for CONTAINER_IMAGE_REPO default if USE_MINIKUBE_CACHE is set Signed-off-by: MOZGIII <mike-n@narod.ru> * Increase all rollout/wait timeouts to one minute Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix syntax error around minikube start command Signed-off-by: MOZGIII <mike-n@narod.ru> * Rollback k8s v1.16.13 to v1.16.12 at CI Signed-off-by: MOZGIII <mike-n@narod.ru> * Add minikube cache autodetection Signed-off-by: MOZGIII <mike-n@narod.ru> * Document USE_MINIKUBE_CACHE=auto mode Signed-off-by: MOZGIII <mike-n@narod.ru> * Add a note on minikube bug to CONTRIBUTING.md Signed-off-by: MOZGIII <mike-n@narod.ru> * Add a note on minikube on ZFS to CONTRIBUTING.md Signed-off-by: MOZGIII <mike-n@narod.ru> * Fix the doc comment at scripts/deploy-kubernetes-test.sh Signed-off-by: MOZGIII <mike-n@narod.ru> * Apply a workaround for kubectl from snap Signed-off-by: MOZGIII <mike-n@narod.ru> * Extract and reuse scripts/skaffold-dockerignore.sh Signed-off-by: MOZGIII <mike-n@narod.ru> Co-authored-by: Ana Hobden <operator@hoverbear.org> Signed-off-by: Brian Menges <brian.menges@anaplan.com>
Will be rebased after #2653 is merged.🎉This PR adds E2E (end-to-end) test for our Kubernetes integration.
It also renames
kubernetes-integration-testsintokubernetes-e2e-tests, and introduces a concept of an E2E test itself - a kind of test where we build a full Vector binary, deploy it and assert it's observable behavior. Very similar conceptually to the test harness, but it's part of the main Vector tree.Check out CI jobs to see it in action, and take a look at updated
CONRTIBUTING.mdfor how to run tests locally.Closes #2632.