-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster #17656
Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster #17656
Conversation
Pinging @elastic/integrations-platforms (Team:Platforms) |
@exekias @jsoriano @ChrsMark Whats your thoughts on running the intergration tests for Kubernetes using kind? Feels like to me this works rather well and we could add more detailed integration tests this way. Kind also allows mounting of volumes, could possible use this with filebeat to access the log path and have filebeat connect to kubernetes API server to test autodiscovery. At the moment this works as long as |
Been looking at the failed test it seems that
|
Added more tests, got almost all metrcsets tested except for |
Thank you for starting this! I understand that this will work in any linux box with Docker installed? If that's the case this sounds like a nice approach to me! any thoughts @jsoriano? In the past we discussed treating this similar to the cloud provider integration tests, but this looks simpler.
This is probably related to some ECS mapping, where we put some info about the container under ECS. |
@blakerouse do you know if this will allow us to access container logs? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whats your thoughts on running the intergration tests for Kubernetes using kind?
Using kind sounds good to me.
I would prefer to avoid adding more python code, but our kubernetes tests are already in python and this PR improves them, and we also have unit tests based on real responses, so let's continue like this.
Btw, it may be also good to add some environment variable that can be used to run the tests using an existing cluster. For example think on making that if |
Thanks for the feedback! I am still working on this because currently the testing is all over the place. Travis tests with minikube (by deploying the manifests, without even checking if they actually deployed), Jenkins tests with kind (by deploying the manifests, without even checking if they actually deployed), and the Kubernetes python integration tests that where disabled and didn't work. My goal is to get it all working with Kind and across multiple versions of Kubernetes. @jsoriano Will update to use @exekias I believe it will allow use to get access to the logs because you can bind mount into the k8s node with kind allowing the host to get the logs from the containers. If we transition to building the images and deploying those into the cluster, then it definitely could always access them. |
I would avoid building + pushing the image if we don't really need to, that's something we will test in e2e tests anyway (ie helm chart tests are doing this already). |
If we are going to avoid that and save those tests for e2e, then I will not add the I also think we should remove the @exekias wydt? |
I understand the use of KUBECONFIG aims to be able to launch tests against a running you may have (when developing). KUBECONFIG should work also when running from outside the cluster, isn't it?
I would hope we can keep tests separted by concern, for instance, Filebeat & Metricbeat tests are different, or Kubernetes module vs autodiscover. They belong in different places in the code. As we move towards the agent & integrations living somewhere else, I would expect this split to happen anyway at some point, so it's probably better to keep things separated today. That said, I don't see anything wrong with sharing the code to launch kind across the board, for instance |
To expose |
@exekias @jsoriano I have actually updated the PR to run the tests with This means we can easily run integration tests against GKE, AKS, EKS, or others without having to adjust the integration tests, as the tests can assume they are running inside of a kubernetes pod that has all rights to the cluster itself. |
a2b0742
to
a442bee
Compare
I also forgot to mention that this currently does not test all metricsets in the kubernetes module. The branch has gotten so big that I wanted to stop adding and just get it working and merged before adding more, as the branch is already huge! But after this is merged I will be adding more tests to cover all metricsets in kubernetes, add in field coverage, add tests for filebeat and heartbeat, add tests for autodiscover for metricbeat, filebeat, and heartbeat. |
Thank you for digging deeper, this is looking really promising! Just a few comments:
|
@exekias I am currently working on splitting the mage work out into its own PR. It does not require |
#17799 splits out the mage work, once that merges I will rebase and cleanup this PR. |
Thanks! I will take a look. |
Does this work with the Kubernetes built into the Docker for Mac install? I think a lot of us develop on Mac with this setup so if that just worked it would be nice. |
@andrewkroh Yes as long as you have |
8510353
to
431afe2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Leaving a few comments. I didn't get to every file (it's big), but I think Jaime did.
dev-tools/mage/gotest.go
Outdated
// Set MODULE because only want that modules tests to run inside the testing environment. | ||
os.Setenv("MODULE", fi.Name()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a way you can pass the module value to wherever it is being used without modifying this process's environment? Haven't looked at the code yet, but like maybe pass it into the runner and then directly add it to what ever process needs it. It's just kind of hard to track what's using it and could cause issues if you ever try to do things in parallel.
💔 Build FailedExpand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
Okay all comments have either been answered or resolved. Anyway I can get a +1 to get this landed. Blocking me adding even more kubernetes tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
…p a kubernetes cluster (elastic#17656) * Use kind to bring up a local kubernetes cluster and then run the integration tests against the kind cluster. * Run mage update. * Add more tests. * Fix more tests, install kind in prepare-tests. * Switch to running kubernetes integration tests inside of Kubernetes. * Use golang 1.13.9. * Fix for other beats. * Run mage fmt. * Don't run kubernetes integration tests if not inside kubernetes environment. * Fix metricbeat to use mage and the makefile shim for mage. * Improve the error message when kind or kubectl is not available. * Refactor the integration tests into a more module system. * Fix go vet. * Setup travis to use kind. * Run kubernetes integration tests in Jenkins. * Fix filebeat magefile. * Fix travis and Jenkins. * Check requirements of the test runner before actually running the tests. * Add return on parsebool error. * Don't return err on missing requirements for tester. * Run make update. * Move the kubernetes items to its own module, import that module only by metricbeat. (cherry picked from commit db29874)
I have seen after updating #17480 with master that with this change integration tests for modules without docker compose or kubernetes files are not executed, not so serious because it only affects some system tests, and cloud tests, that are not merged yet. I will prepare a fix. |
@jsoriano Could you file a bug about what is not working? Be happy to fix it. |
@blakerouse I opened a PR with a temptative fix #18125, let me know what you think. |
@blakerouse The Metricbeat generator test is failing in master. Do you think it's related to your changes?
|
… a pod and use kind to setup a kubernetes cluster (#18111) * Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster (#17656) * Use kind to bring up a local kubernetes cluster and then run the integration tests against the kind cluster. * Run mage update. * Add more tests. * Fix more tests, install kind in prepare-tests. * Switch to running kubernetes integration tests inside of Kubernetes. * Use golang 1.13.9. * Fix for other beats. * Run mage fmt. * Don't run kubernetes integration tests if not inside kubernetes environment. * Fix metricbeat to use mage and the makefile shim for mage. * Improve the error message when kind or kubectl is not available. * Refactor the integration tests into a more module system. * Fix go vet. * Setup travis to use kind. * Run kubernetes integration tests in Jenkins. * Fix filebeat magefile. * Fix travis and Jenkins. * Check requirements of the test runner before actually running the tests. * Add return on parsebool error. * Don't return err on missing requirements for tester. * Run make update. * Move the kubernetes items to its own module, import that module only by metricbeat. (cherry picked from commit db29874) * Update fields.
@andrewkroh I am working on this here - #18086 The latest run I got with that MP showed something different. |
What does this PR do?
Refactors how
mage goIntegTest
run for metricbeat. Before each module was run inside of a docker container usingdocker-compose
. Now depending on the module it will either run in docker or kubernetes. At the moment only thekubernetes
module runs inside of kubernetes.Each module is determined if it should run inside of docker or inside of kubernetes. If the module has a
docker-compose.yml
file then it will run in docker, if the module has akubernetes.yml
then it will run inside of kubernetes.The docker runner stuff is the same so nothing really changes there, but the kubernetes runner is what is new. It works by executing the tests inside of a Pod running inside of the kubernetes.
If
KUBECONFIG
is defined in the environment then the runner just executes the tests against that defined kubernetes. If its not defined then kind will be used to setup a cluster and run the tests against that cluster.Running the tests inside of kubernetes is handled by
KubeRemote
which uses the kubernetes client-go to setup all the requirements to get it running. Below breaks down the steps that are taken to get it running:sshd-rsync
and the test executer container that starts once init container exits. The init container exits after the first SSH connection is made and disconnects.rsync
is ran locally to sync the beats repo to the init container sync directory which was mounted as anemptyDir
and shared between the init container and exec container.kubernetes
.Why is it important?
Currently all of the Kubernetes integration tests are skipped, this fixes them so they can actually be ran inside of a real Kubernetes cluster.
Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesCHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
How to test this PR locally
Related issues