Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster #17656

Merged
merged 22 commits into from
Apr 29, 2020

Conversation

blakerouse
Copy link
Contributor

@blakerouse blakerouse commented Apr 9, 2020

What does this PR do?

Refactors how mage goIntegTest run for metricbeat. Before each module was run inside of a docker container using docker-compose. Now depending on the module it will either run in docker or kubernetes. At the moment only the kubernetes module runs inside of kubernetes.

Each module is determined if it should run inside of docker or inside of kubernetes. If the module has a docker-compose.yml file then it will run in docker, if the module has a kubernetes.yml then it will run inside of kubernetes.

The docker runner stuff is the same so nothing really changes there, but the kubernetes runner is what is new. It works by executing the tests inside of a Pod running inside of the kubernetes.

If KUBECONFIG is defined in the environment then the runner just executes the tests against that defined kubernetes. If its not defined then kind will be used to setup a cluster and run the tests against that cluster.

Running the tests inside of kubernetes is handled by KubeRemote which uses the kubernetes client-go to setup all the requirements to get it running. Below breaks down the steps that are taken to get it running:

  1. Generate a SSH private/public key.
  2. Add public key as secret to kubernetes.
  3. Add service account, cluster role, cluster role binding to give full access to all of kubernetes.
  4. Deploy a Pod with an init container running sshd-rsync and the test executer container that starts once init container exits. The init container exits after the first SSH connection is made and disconnects.
  5. While the init container is running port-forwarding is used to forward the SSH port to the local system.
  6. rsync is ran locally to sync the beats repo to the init container sync directory which was mounted as an emptyDir and shared between the init container and exec container.
  7. Init container exits and the execute container starts the test for the specific module in this case kubernetes.
  8. Then the logs are tailed to the local terminal while the test runner is running.
  9. Once the test is done all the added resources are cleaned up.

Why is it important?

Currently all of the Kubernetes integration tests are skipped, this fixes them so they can actually be ran inside of a real Kubernetes cluster.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Ensure that Kubernetes integration tests are ran.

How to test this PR locally

MODULE="kubernetes" mage goIntegTest

Related issues

@blakerouse blakerouse added enhancement Team:Platforms Label for the Integrations - Platforms team labels Apr 9, 2020
@blakerouse blakerouse requested a review from a team April 9, 2020 22:25
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@blakerouse
Copy link
Contributor Author

@exekias @jsoriano @ChrsMark Whats your thoughts on running the intergration tests for Kubernetes using kind?

Feels like to me this works rather well and we could add more detailed integration tests this way. Kind also allows mounting of volumes, could possible use this with filebeat to access the log path and have filebeat connect to kubernetes API server to test autodiscovery.

At the moment this works as long as kind and kubectl is installed. There is still 1 test failing, but its not related to the actual running cluster, seems just like a test issue (still looking into it, thats why its in draft).

@blakerouse
Copy link
Contributor Author

Been looking at the failed test it seems that state_container metricset is added a top level key of container to the event. It has both kubernetes.container and container, is that correct? I would think not unless it has to do with ECS, maybe?

(Pdb) sorted(self.de_dot(KUBERNETES_FIELDS).keys())
['@timestamp', 'agent', 'ecs', 'event', 'host', 'kubernetes', 'metricset', 'service']
(Pdb) sorted(evt.keys())
['@timestamp', 'agent', 'container', 'ecs', 'event', 'host', 'kubernetes', 'metricset', 'service']

@blakerouse
Copy link
Contributor Author

Added more tests, got almost all metrcsets tested except for event, volume, state_resourcequota.

@exekias
Copy link
Contributor

exekias commented Apr 14, 2020

Thank you for starting this! I understand that this will work in any linux box with Docker installed? If that's the case this sounds like a nice approach to me!

any thoughts @jsoriano? In the past we discussed treating this similar to the cloud provider integration tests, but this looks simpler.

Been looking at the failed test it seems that state_container metricset is added a top level key of container to the event. It has both kubernetes.container and container, is that correct? I would think not unless it has to do with ECS, maybe?

(Pdb) sorted(self.de_dot(KUBERNETES_FIELDS).keys())
['@timestamp', 'agent', 'ecs', 'event', 'host', 'kubernetes', 'metricset', 'service']
(Pdb) sorted(evt.keys())
['@timestamp', 'agent', 'container', 'ecs', 'event', 'host', 'kubernetes', 'metricset', 'service']

This is probably related to some ECS mapping, where we put some info about the container under ECS.

@exekias
Copy link
Contributor

exekias commented Apr 14, 2020

@blakerouse do you know if this will allow us to access container logs?

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whats your thoughts on running the intergration tests for Kubernetes using kind?

Using kind sounds good to me.

I would prefer to avoid adding more python code, but our kubernetes tests are already in python and this PR improves them, and we also have unit tests based on real responses, so let's continue like this.

libbeat/tests/system/beat/kind.py Outdated Show resolved Hide resolved
metricbeat/module/kubernetes/test_kubernetes.py Outdated Show resolved Hide resolved
metricbeat/module/kubernetes/test_kubernetes.py Outdated Show resolved Hide resolved
metricbeat/Makefile Outdated Show resolved Hide resolved
@jsoriano
Copy link
Member

Btw, it may be also good to add some environment variable that can be used to run the tests using an existing cluster. For example think on making that if KUBECONFIG environment variable is set, the test doesn't try to start a cluster, and instead it uses the config in KUBECONFIG.
This would allow us to run the same tests against different Kubernetes providers.

@blakerouse
Copy link
Contributor Author

Thanks for the feedback! I am still working on this because currently the testing is all over the place.

Travis tests with minikube (by deploying the manifests, without even checking if they actually deployed), Jenkins tests with kind (by deploying the manifests, without even checking if they actually deployed), and the Kubernetes python integration tests that where disabled and didn't work.

My goal is to get it all working with Kind and across multiple versions of Kubernetes.

@jsoriano Will update to use TEST_TAGS, I was wanting a way to selectively run these tests and this is perfect! As for the KUBECONFIG idea the current issue would be that the tests would most likely fail because it needs the cluster on the local system to actually pass the tests. It also doesn't currently build docker images to run inside of Kind, it runs metricbeat outside of the cluster on the host. More work will need to be done to build the images and run them inside of the cluster. (I will see how hard this would be, because it would allow us to do what your suggesting, which would allow use to test inside of other cloud providers easily.

@exekias I believe it will allow use to get access to the logs because you can bind mount into the k8s node with kind allowing the host to get the logs from the containers. If we transition to building the images and deploying those into the cluster, then it definitely could always access them.

@exekias
Copy link
Contributor

exekias commented Apr 14, 2020

I would avoid building + pushing the image if we don't really need to, that's something we will test in e2e tests anyway (ie helm chart tests are doing this already).

@blakerouse
Copy link
Contributor Author

If we are going to avoid that and save those tests for e2e, then I will not add the KUBECONFIG check as the tests will require the usage of kind.

I also think we should remove the test target in deploy/kubernetes/Makefile and add the test inside of the kubernetes metricbeat integration tests. That way its all in once place and not divided up. I think the test really should just verify the schema of the manifest, because the images will not be built.

@exekias wydt?

@exekias
Copy link
Contributor

exekias commented Apr 14, 2020

If we are going to avoid that and save those tests for e2e, then I will not add the KUBECONFIG check as the tests will require the usage of kind.

I understand the use of KUBECONFIG aims to be able to launch tests against a running you may have (when developing). KUBECONFIG should work also when running from outside the cluster, isn't it?

I also think we should remove the test target in deploy/kubernetes/Makefile and add the test inside of the kubernetes metricbeat integration tests. That way its all in once place and not divided up. I think the test really should just verify the schema of the manifest, because the images will not be built.

I would hope we can keep tests separted by concern, for instance, Filebeat & Metricbeat tests are different, or Kubernetes module vs autodiscover. They belong in different places in the code. As we move towards the agent & integrations living somewhere else, I would expect this split to happen anyway at some point, so it's probably better to keep things separated today.

That said, I don't see anything wrong with sharing the code to launch kind across the board, for instance

@blakerouse
Copy link
Contributor Author

I understand the use of KUBECONFIG aims to be able to launch tests against a running you may have (when developing). KUBECONFIG should work also when running from outside the cluster, isn't it?

To expose kube-state-metrics with kind the code uses a service with a nodeport and kind create cluster that maps that port to the local system. Without this specifics the kube-state-metrics tests will fail. I could make it work with KUBECONFIG and skip the kube-state-metrics if using that mode.

@blakerouse
Copy link
Contributor Author

@exekias @jsoriano I have actually updated the PR to run the tests with goIntegTest as it seemed the future was to move all tests to be ran with go instead of python. This meant changing the integration test runner quite a bit to allow this, but it enables us to add lots of kubernetes tests without having to worry about what was used to create the cluster (the runner will use Kind, if no cluster is defined).

This means we can easily run integration tests against GKE, AKS, EKS, or others without having to adjust the integration tests, as the tests can assume they are running inside of a kubernetes pod that has all rights to the cluster itself.

@blakerouse blakerouse changed the title Use kind to run Kubernetes integration tests Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster Apr 16, 2020
@blakerouse blakerouse force-pushed the k8s-integration-use-kind branch from a2b0742 to a442bee Compare April 16, 2020 16:40
@blakerouse
Copy link
Contributor Author

I also forgot to mention that this currently does not test all metricsets in the kubernetes module. The branch has gotten so big that I wanted to stop adding and just get it working and merged before adding more, as the branch is already huge!

But after this is merged I will be adding more tests to cover all metricsets in kubernetes, add in field coverage, add tests for filebeat and heartbeat, add tests for autodiscover for metricbeat, filebeat, and heartbeat.

dev-tools/make/mage.mk Outdated Show resolved Hide resolved
@exekias
Copy link
Contributor

exekias commented Apr 17, 2020

Thank you for digging deeper, this is looking really promising! Just a few comments:

  • I understand that by now this would only work when you use MODULE=kubernetes, right? to be clear, this would be good as we can handle that from the CI perspective :)
  • It would be nice to split the concerns of this PR, for instance moving the refactor on Makefile and mage-install to a new one, I think @andrewkroh has some plans about that

@blakerouse
Copy link
Contributor Author

@exekias I am currently working on splitting the mage work out into its own PR.

It does not require MODULE=kubernetes it will run the kubernetes tests when running mage goIntegTest as long as either KUBECONFIG or kind is present.

@blakerouse
Copy link
Contributor Author

#17799 splits out the mage work, once that merges I will rebase and cleanup this PR.

@jsoriano
Copy link
Member

#17799 splits out the mage work, once that merges I will rebase and cleanup this PR.

Thanks! I will take a look.

@andrewkroh
Copy link
Member

Does this work with the Kubernetes built into the Docker for Mac install? I think a lot of us develop on Mac with this setup so if that just worked it would be nice.

@blakerouse
Copy link
Contributor Author

@andrewkroh Yes as long as you have docker, kind and kubectl installed on your mac it jsut works. I developed this and tested it on a mac.

@blakerouse blakerouse force-pushed the k8s-integration-use-kind branch from 8510353 to 431afe2 Compare April 28, 2020 23:00
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving a few comments. I didn't get to every file (it's big), but I think Jaime did.

Comment on lines 157 to 159
// Set MODULE because only want that modules tests to run inside the testing environment.
os.Setenv("MODULE", fi.Name())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way you can pass the module value to wherever it is being used without modifying this process's environment? Haven't looked at the code yet, but like maybe pass it into the runner and then directly add it to what ever process needs it. It's just kind of hard to track what's using it and could cause issues if you ever try to do things in parallel.

dev-tools/mage/integtest_kubernetes.go Outdated Show resolved Hide resolved
dev-tools/mage/kuberemote.go Show resolved Hide resolved
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 29, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 2
Passed 7478
Skipped 1166
Total 8646

Test errors

Expand to view the tests failures

  • Name: Build and Test / Metricbeat x-pack / test_server – test_statsd.Test

    • Status: FAILED
    • Age: 1
    • Duration: 5.869
    • Error Details: 'ascii' codec can't decode byte 0xe2 in position 702041: ordinal not in range(128)
  • Name: Build and Test / Auditbeat x-pack / test_dns_long_request – test_system_socket.Test

    • Status: FAILED
    • Age: 1
    • Duration: 31.175
    • Error Details: The events in: [
      {
      "@timestamp": "2020-04-29T20:23:39.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 64,
      "client.ip": "127.25.215.226",
      "client.packets": 1,
      "client.port": 49127,
      "destination.bytes": 72,
      "destination.ip": "127.4.84.143",
      "destination.packets": 1,
      "destination.port": 53,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 27594,
      "event.end": "2020-04-29T20:23:36.668Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:36.668Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 136,
      "network.community_id": "1:/DyDIkfzsDVLElnNigHa6ErKYSA=",
      "network.direction": "inbound",
      "network.packets": 2,
      "network.transport": "udp",
      "network.type": "ipv4",
      "process.args": [
      "/tmp/python-env/build/ve/docker/bin/python3",
      "/tmp/python-env/build/ve/docker/bin/nosetests",
      "--process-timeout=90",
      "--with-timer",
      "--with-xunit",
      "--xunit-file=build/TEST-python-integration.xml",
      "tests/system/test_metricsets.py",
      "tests/system/test_system_socket.py"
      ],
      "process.created": "2020-04-29T20:23:00.829Z",
      "process.executable": "/usr/bin/python3.7",
      "process.name": "nosetests",
      "process.pid": 49391,
      "server.bytes": 72,
      "server.ip": "127.4.84.143",
      "server.packets": 1,
      "server.port": 53,
      "service.type": "system",
      "source.bytes": 64,
      "source.ip": "127.25.215.226",
      "source.packets": 1,
      "source.port": 49127,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c940d5400",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:39.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 56,
      "client.ip": "127.25.215.226",
      "client.packets": 1,
      "client.port": 49127,
      "destination.bytes": 80,
      "destination.ip": "127.4.84.143",
      "destination.packets": 1,
      "destination.port": 53,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 179125,
      "event.end": "2020-04-29T20:23:36.668Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:36.668Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 136,
      "network.community_id": "1:/DyDIkfzsDVLElnNigHa6ErKYSA=",
      "network.direction": "outbound",
      "network.packets": 2,
      "network.transport": "udp",
      "network.type": "ipv4",
      "process.args": [
      "/tmp/python-env/build/ve/docker/bin/python3",
      "/tmp/python-env/build/ve/docker/bin/nosetests",
      "--process-timeout=90",
      "--with-timer",
      "--with-xunit",
      "--xunit-file=build/TEST-python-integration.xml",
      "tests/system/test_metricsets.py",
      "tests/system/test_system_socket.py"
      ],
      "process.created": "2020-04-29T20:23:00.829Z",
      "process.executable": "/usr/bin/python3.7",
      "process.name": "nosetests",
      "process.pid": 49391,
      "server.bytes": 80,
      "server.ip": "127.4.84.143",
      "server.packets": 1,
      "server.port": 53,
      "service.type": "system",
      "source.bytes": 56,
      "source.ip": "127.25.215.226",
      "source.packets": 1,
      "source.port": 49127,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c940d5800",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:39.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 9297,
      "client.ip": "10.224.1.141",
      "client.packets": 3,
      "client.port": 37106,
      "destination.bytes": 99,
      "destination.ip": "35.184.73.24",
      "destination.packets": 2,
      "destination.port": 5001,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 4036662,
      "event.end": "2020-04-29T20:23:36.805Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:36.801Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 9396,
      "network.community_id": "1:Nfiwum8Qm2MqZs3lPx7daUv88t0=",
      "network.direction": "unknown",
      "network.packets": 5,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/auditbeat",
      "-e",
      "-c",
      "/etc/auditbeat/auditbeat.yml"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/auditbeat-7.3.0-linux-x86_64/auditbeat",
      "process.name": "auditbeat",
      "process.pid": 2402,
      "server.bytes": 99,
      "server.ip": "35.184.73.24",
      "server.packets": 2,
      "server.port": 5001,
      "service.type": "system",
      "source.bytes": 9297,
      "source.ip": "10.224.1.141",
      "source.packets": 3,
      "source.port": 37106,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa92000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:42.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 2162,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 37106,
      "destination.bytes": 99,
      "destination.ip": "35.184.73.24",
      "destination.packets": 2,
      "destination.port": 5001,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2227669,
      "event.end": "2020-04-29T20:23:40.175Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:40.173Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2261,
      "network.community_id": "1:Nfiwum8Qm2MqZs3lPx7daUv88t0=",
      "network.direction": "unknown",
      "network.packets": 4,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/auditbeat",
      "-e",
      "-c",
      "/etc/auditbeat/auditbeat.yml"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/auditbeat-7.3.0-linux-x86_64/auditbeat",
      "process.name": "auditbeat",
      "process.pid": 2402,
      "server.bytes": 99,
      "server.ip": "35.184.73.24",
      "server.packets": 2,
      "server.port": 5001,
      "service.type": "system",
      "source.bytes": 2162,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 37106,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa92000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:43.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 43456,
      "client.ip": "10.224.1.141",
      "client.packets": 65,
      "client.port": 33484,
      "destination.bytes": 70732,
      "destination.ip": "10.224.0.26",
      "destination.packets": 46,
      "destination.port": 49187,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 5806455355,
      "event.end": "2020-04-29T20:23:42.544Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:36.737Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "1158",
      "host.name": "60920b1008e7",
      "network.bytes": 114188,
      "network.community_id": "1:JFIcjp9WW3QoHSvELTlbh3gEb8A=",
      "network.direction": "unknown",
      "network.packets": 111,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "java",
      "-Djava.util.logging.config.file=/var/lib/jenkins/logging.properties",
      "-javaagent:/opt/jolokia/jolokia-agent.jar=config=/opt/jolokia/jolokia.properties",
      "-Dorg.jenkinsci.plugins.gitclient.Git.timeOut=10",
      "-jar",
      "/usr/local/share/jenkins/swarm-client-3.17.jar",
      "-description",
      "Ubuntu 16.04 - 2 CPU Cores / 3539 MB Memory",
      "-disableClientsUniqueId",
      "-executors",
      "1",
      "-fsroot",
      "/var/lib/jenkins",
      "-master",
      "https://beats-ci.elastic.co/",
      "-maxRetryInterval",
      "160",
      "-retryBackOffStrategy",
      "linear",
      "-mode",
      "exclusive",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-username",
      "local-admin",
      "-password",
      "1101471da3a313cbf3999ee6888d412a7e",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-master",
      "https://beats-ci.elastic.co",
      "-username",
      "local-admin",
      "-password",
      "11af84f32a633296a49297085d82b8279d",
      "-labels",
      "docker",
      "-labels",
      "immutable",
      "-labels",
      "linux",
      "-labels",
      "linux-immutable",
      "-labels",
      "swarm",
      "-labels",
      "ubuntu",
      "-labels",
      "ubuntu-16.04",
      "-labels",
      "ubuntu-16",
      "-labels",
      "virtual",
      "-labels",
      "x86_64",
      "-description",
      "MachineType:n1-highmem-4,DiskSize:150"
      ],
      "process.created": "2020-04-29T20:05:30.530Z",
      "process.executable": "/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java",
      "process.name": "java",
      "process.pid": 2411,
      "server.bytes": 70732,
      "server.ip": "10.224.0.26",
      "server.packets": 46,
      "server.port": 49187,
      "service.type": "system",
      "source.bytes": 43456,
      "source.ip": "10.224.1.141",
      "source.packets": 65,
      "source.port": 33484,
      "system.audit.socket.egid": 1158,
      "system.audit.socket.euid": 1157,
      "system.audit.socket.gid": 1158,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c811bc400",
      "system.audit.socket.uid": 1157,
      "user.id": "1157"
      },
      {
      "@timestamp": "2020-04-29T20:23:43.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 2208,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 37106,
      "destination.bytes": 99,
      "destination.ip": "35.184.73.24",
      "destination.packets": 2,
      "destination.port": 5001,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2331171,
      "event.end": "2020-04-29T20:23:43.175Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:43.172Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2307,
      "network.community_id": "1:Nfiwum8Qm2MqZs3lPx7daUv88t0=",
      "network.direction": "unknown",
      "network.packets": 4,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/auditbeat",
      "-e",
      "-c",
      "/etc/auditbeat/auditbeat.yml"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/auditbeat-7.3.0-linux-x86_64/auditbeat",
      "process.name": "auditbeat",
      "process.pid": 2402,
      "server.bytes": 99,
      "server.ip": "35.184.73.24",
      "server.packets": 2,
      "server.port": 5001,
      "service.type": "system",
      "source.bytes": 2208,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 37106,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa92000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:44.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 1198,
      "client.ip": "127.0.0.1",
      "client.packets": 4,
      "client.port": 48060,
      "destination.bytes": 1565,
      "destination.ip": "127.0.0.1",
      "destination.packets": 3,
      "destination.port": 8778,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2019789,
      "event.end": "2020-04-29T20:23:42.370Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:42.368Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2763,
      "network.community_id": "1:yW5Cl7d24rDDfdaQejZ+p7MWst4=",
      "network.direction": "unknown",
      "network.packets": 7,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/metricbeat",
      "-e",
      "-c",
      "/etc/metricbeat/metricbeat.yml",
      "--path.home",
      "/etc/metricbeat"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/metricbeat-7.3.2-linux-x86_64/metricbeat",
      "process.name": "metricbeat",
      "process.pid": 2401,
      "server.bytes": 1565,
      "server.ip": "127.0.0.1",
      "server.packets": 3,
      "server.port": 8778,
      "service.type": "system",
      "source.bytes": 1198,
      "source.ip": "127.0.0.1",
      "source.packets": 4,
      "source.port": 48060,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c95483000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:44.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 1625,
      "client.ip": "127.0.0.1",
      "client.packets": 3,
      "client.port": 8778,
      "destination.bytes": 1118,
      "destination.ip": "127.0.0.1",
      "destination.packets": 4,
      "destination.port": 48060,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 1946869,
      "event.end": "2020-04-29T20:23:42.370Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:42.368Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2743,
      "network.community_id": "1:yW5Cl7d24rDDfdaQejZ+p7MWst4=",
      "network.direction": "unknown",
      "network.packets": 7,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/metricbeat",
      "-e",
      "-c",
      "/etc/metricbeat/metricbeat.yml",
      "--path.home",
      "/etc/metricbeat"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/metricbeat-7.3.2-linux-x86_64/metricbeat",
      "process.name": "metricbeat",
      "process.pid": 2401,
      "server.bytes": 1118,
      "server.ip": "127.0.0.1",
      "server.packets": 4,
      "server.port": 48060,
      "service.type": "system",
      "source.bytes": 1625,
      "source.ip": "127.0.0.1",
      "source.packets": 3,
      "source.port": 8778,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c956e9980",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:45.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 87744,
      "client.ip": "10.224.1.141",
      "client.packets": 33,
      "client.port": 32824,
      "destination.bytes": 1481,
      "destination.ip": "13.56.146.199",
      "destination.packets": 19,
      "destination.port": 9243,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 162399230,
      "event.end": "2020-04-29T20:23:43.187Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:43.025Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 89225,
      "network.community_id": "1:VJOfnDgg0v6f+slKZKLaXnihIVA=",
      "network.direction": "unknown",
      "network.packets": 52,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/metricbeat",
      "-e",
      "-c",
      "/etc/metricbeat/metricbeat.yml",
      "--path.home",
      "/etc/metricbeat"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/metricbeat-7.3.2-linux-x86_64/metricbeat",
      "process.name": "metricbeat",
      "process.pid": 2401,
      "server.bytes": 1481,
      "server.ip": "13.56.146.199",
      "server.packets": 19,
      "server.port": 9243,
      "service.type": "system",
      "source.bytes": 87744,
      "source.ip": "10.224.1.141",
      "source.packets": 33,
      "source.port": 32824,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa93800",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:45.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 80,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 41536,
      "destination.bytes": 644,
      "destination.ip": "169.254.169.254",
      "destination.packets": 3,
      "destination.port": 80,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 854090,
      "event.end": "2020-04-29T20:23:43.290Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:43.289Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 724,
      "network.community_id": "1:JKkWKipWC0nVjunXACKNNGYYH0U=",
      "network.direction": "unknown",
      "network.packets": 5,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/bin/python3",
      "/usr/bin/google_network_daemon"
      ],
      "process.created": "2020-04-29T20:05:28.690Z",
      "process.executable": "/usr/bin/python3.5",
      "process.name": "google_network_",
      "process.pid": 1962,
      "server.bytes": 644,
      "server.ip": "169.254.169.254",
      "server.packets": 3,
      "server.port": 80,
      "service.type": "system",
      "source.bytes": 80,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 41536,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c95487000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:45.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 426,
      "client.ip": "10.224.1.141",
      "client.packets": 3,
      "client.port": 41546,
      "destination.bytes": 68,
      "destination.ip": "169.254.169.254",
      "destination.packets": 3,
      "destination.port": 80,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 593972,
      "event.end": "2020-04-29T20:23:43.295Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:43.294Z",
      "flow.complete": true,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 494,
      "network.community_id": "1:SA1Y1JKGZ/pb06I0hWJSqUfPkKM=",
      "network.direction": "outbound",
      "network.packets": 6,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/bin/python3",
      "/usr/bin/google_network_daemon"
      ],
      "process.created": "2020-04-29T20:05:28.690Z",
      "process.executable": "/usr/bin/python3.5",
      "process.name": "google_network_",
      "process.pid": 1962,
      "server.bytes": 68,
      "server.ip": "169.254.169.254",
      "server.packets": 3,
      "server.port": 80,
      "service.type": "system",
      "source.bytes": 426,
      "source.ip": "10.224.1.141",
      "source.packets": 3,
      "source.port": 41546,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa90800",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:45.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 80,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 41532,
      "destination.bytes": 6903,
      "destination.ip": "169.254.169.254",
      "destination.packets": 3,
      "destination.port": 80,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 902650,
      "event.end": "2020-04-29T20:23:43.472Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:43.471Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 6983,
      "network.community_id": "1:KaYheRuGzGdj6QL/tZVQCnLcLV8=",
      "network.direction": "unknown",
      "network.packets": 5,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/bin/python3",
      "/usr/bin/google_accounts_daemon"
      ],
      "process.created": "2020-04-29T20:05:28.700Z",
      "process.executable": "/usr/bin/python3.5",
      "process.name": "google_accounts",
      "process.pid": 1971,
      "server.bytes": 6903,
      "server.ip": "169.254.169.254",
      "server.packets": 3,
      "server.port": 80,
      "service.type": "system",
      "source.bytes": 80,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 41532,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa94000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:45.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 398,
      "client.ip": "10.224.1.141",
      "client.packets": 3,
      "client.port": 41548,
      "destination.bytes": 68,
      "destination.ip": "169.254.169.254",
      "destination.packets": 3,
      "destination.port": 80,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 705711,
      "event.end": "2020-04-29T20:23:43.501Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:43.500Z",
      "flow.complete": true,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 466,
      "network.community_id": "1:BWzzJU1XCiosMt4clu2XGEnWtoM=",
      "network.direction": "outbound",
      "network.packets": 6,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/bin/python3",
      "/usr/bin/google_accounts_daemon"
      ],
      "process.created": "2020-04-29T20:05:28.700Z",
      "process.executable": "/usr/bin/python3.5",
      "process.name": "google_accounts",
      "process.pid": 1971,
      "server.bytes": 68,
      "server.ip": "169.254.169.254",
      "server.packets": 3,
      "server.port": 80,
      "service.type": "system",
      "source.bytes": 398,
      "source.ip": "10.224.1.141",
      "source.packets": 3,
      "source.port": 41548,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c95484000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:48.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 132,
      "client.ip": "127.107.7.14",
      "client.packets": 3,
      "client.port": 44171,
      "destination.bytes": 234,
      "destination.ip": "127.101.220.29",
      "destination.packets": 4,
      "destination.port": 51907,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 455343,
      "event.end": "2020-04-29T20:23:46.678Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:46.678Z",
      "flow.complete": true,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 366,
      "network.community_id": "1:o7s0k3FICweDqv/DtTzLLbOj/7k=",
      "network.direction": "inbound",
      "network.packets": 7,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/bin/python3",
      "/usr/bin/google_network_daemon"
      ],
      "process.created": "2020-04-29T20:05:28.690Z",
      "process.executable": "/usr/bin/python3.5",
      "process.name": "google_network_",
      "process.pid": 49391,
      "server.bytes": 234,
      "server.ip": "127.101.220.29",
      "server.packets": 4,
      "server.port": 51907,
      "service.type": "system",
      "source.bytes": 132,
      "source.ip": "127.107.7.14",
      "source.packets": 3,
      "source.port": 44171,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c95487000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:49.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 304,
      "client.ip": "127.107.7.14",
      "client.packets": 5,
      "client.port": 44171,
      "destination.bytes": 194,
      "destination.domain": "elastic.co",
      "destination.ip": "127.101.220.29",
      "destination.packets": 5,
      "destination.port": 51907,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 711192,
      "event.end": "2020-04-29T20:23:46.678Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:46.678Z",
      "flow.complete": true,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 498,
      "network.community_id": "1:o7s0k3FICweDqv/DtTzLLbOj/7k=",
      "network.direction": "outbound",
      "network.packets": 10,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/tmp/python-env/build/ve/docker/bin/python3",
      "/tmp/python-env/build/ve/docker/bin/nosetests",
      "--process-timeout=90",
      "--with-timer",
      "--with-xunit",
      "--xunit-file=build/TEST-python-integration.xml",
      "tests/system/test_metricsets.py",
      "tests/system/test_system_socket.py"
      ],
      "process.created": "2020-04-29T20:23:00.829Z",
      "process.executable": "/usr/bin/python3.7",
      "process.name": "nosetests",
      "process.pid": 49391,
      "server.bytes": 194,
      "server.domain": "elastic.co",
      "server.ip": "127.101.220.29",
      "server.packets": 5,
      "server.port": 51907,
      "service.type": "system",
      "source.bytes": 304,
      "source.ip": "127.107.7.14",
      "source.packets": 5,
      "source.port": 44171,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c95480000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:49.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 15760,
      "client.ip": "10.224.1.141",
      "client.packets": 24,
      "client.port": 33484,
      "destination.bytes": 26608,
      "destination.ip": "10.224.0.26",
      "destination.packets": 17,
      "destination.port": 49187,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2687250851,
      "event.end": "2020-04-29T20:23:47.453Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:44.766Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "1158",
      "host.name": "60920b1008e7",
      "network.bytes": 42368,
      "network.community_id": "1:JFIcjp9WW3QoHSvELTlbh3gEb8A=",
      "network.direction": "unknown",
      "network.packets": 41,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "java",
      "-Djava.util.logging.config.file=/var/lib/jenkins/logging.properties",
      "-javaagent:/opt/jolokia/jolokia-agent.jar=config=/opt/jolokia/jolokia.properties",
      "-Dorg.jenkinsci.plugins.gitclient.Git.timeOut=10",
      "-jar",
      "/usr/local/share/jenkins/swarm-client-3.17.jar",
      "-description",
      "Ubuntu 16.04 - 2 CPU Cores / 3539 MB Memory",
      "-disableClientsUniqueId",
      "-executors",
      "1",
      "-fsroot",
      "/var/lib/jenkins",
      "-master",
      "https://beats-ci.elastic.co/",
      "-maxRetryInterval",
      "160",
      "-retryBackOffStrategy",
      "linear",
      "-mode",
      "exclusive",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-username",
      "local-admin",
      "-password",
      "1101471da3a313cbf3999ee6888d412a7e",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-master",
      "https://beats-ci.elastic.co",
      "-username",
      "local-admin",
      "-password",
      "11af84f32a633296a49297085d82b8279d",
      "-labels",
      "docker",
      "-labels",
      "immutable",
      "-labels",
      "linux",
      "-labels",
      "linux-immutable",
      "-labels",
      "swarm",
      "-labels",
      "ubuntu",
      "-labels",
      "ubuntu-16.04",
      "-labels",
      "ubuntu-16",
      "-labels",
      "virtual",
      "-labels",
      "x86_64",
      "-description",
      "MachineType:n1-highmem-4,DiskSize:150"
      ],
      "process.created": "2020-04-29T20:05:30.530Z",
      "process.executable": "/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java",
      "process.name": "java",
      "process.pid": 2411,
      "server.bytes": 26608,
      "server.ip": "10.224.0.26",
      "server.packets": 17,
      "server.port": 49187,
      "service.type": "system",
      "source.bytes": 15760,
      "source.ip": "10.224.1.141",
      "source.packets": 24,
      "source.port": 33484,
      "system.audit.socket.egid": 1158,
      "system.audit.socket.euid": 1157,
      "system.audit.socket.gid": 1158,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c811bc400",
      "system.audit.socket.uid": 1157,
      "user.id": "1157"
      },
      {
      "@timestamp": "2020-04-29T20:23:49.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 14343,
      "client.ip": "10.224.1.141",
      "client.packets": 9,
      "client.port": 37106,
      "destination.bytes": 396,
      "destination.ip": "35.184.73.24",
      "destination.packets": 8,
      "destination.port": 5001,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 4888005800,
      "event.end": "2020-04-29T20:23:49.182Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:44.294Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 14739,
      "network.community_id": "1:Nfiwum8Qm2MqZs3lPx7daUv88t0=",
      "network.direction": "unknown",
      "network.packets": 17,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/auditbeat",
      "-e",
      "-c",
      "/etc/auditbeat/auditbeat.yml"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/auditbeat-7.3.0-linux-x86_64/auditbeat",
      "process.name": "auditbeat",
      "process.pid": 2402,
      "server.bytes": 396,
      "server.ip": "35.184.73.24",
      "server.packets": 8,
      "server.port": 5001,
      "service.type": "system",
      "source.bytes": 14343,
      "source.ip": "10.224.1.141",
      "source.packets": 9,
      "source.port": 37106,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa92000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:53.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 7828,
      "client.ip": "10.224.1.141",
      "client.packets": 11,
      "client.port": 33484,
      "destination.bytes": 13320,
      "destination.ip": "10.224.0.26",
      "destination.packets": 9,
      "destination.port": 49187,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 9598609,
      "event.end": "2020-04-29T20:23:50.662Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:50.652Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "1158",
      "host.name": "60920b1008e7",
      "network.bytes": 21148,
      "network.community_id": "1:JFIcjp9WW3QoHSvELTlbh3gEb8A=",
      "network.direction": "unknown",
      "network.packets": 20,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "java",
      "-Djava.util.logging.config.file=/var/lib/jenkins/logging.properties",
      "-javaagent:/opt/jolokia/jolokia-agent.jar=config=/opt/jolokia/jolokia.properties",
      "-Dorg.jenkinsci.plugins.gitclient.Git.timeOut=10",
      "-jar",
      "/usr/local/share/jenkins/swarm-client-3.17.jar",
      "-description",
      "Ubuntu 16.04 - 2 CPU Cores / 3539 MB Memory",
      "-disableClientsUniqueId",
      "-executors",
      "1",
      "-fsroot",
      "/var/lib/jenkins",
      "-master",
      "https://beats-ci.elastic.co/",
      "-maxRetryInterval",
      "160",
      "-retryBackOffStrategy",
      "linear",
      "-mode",
      "exclusive",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-username",
      "local-admin",
      "-password",
      "1101471da3a313cbf3999ee6888d412a7e",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-master",
      "https://beats-ci.elastic.co",
      "-username",
      "local-admin",
      "-password",
      "11af84f32a633296a49297085d82b8279d",
      "-labels",
      "docker",
      "-labels",
      "immutable",
      "-labels",
      "linux",
      "-labels",
      "linux-immutable",
      "-labels",
      "swarm",
      "-labels",
      "ubuntu",
      "-labels",
      "ubuntu-16.04",
      "-labels",
      "ubuntu-16",
      "-labels",
      "virtual",
      "-labels",
      "x86_64",
      "-description",
      "MachineType:n1-highmem-4,DiskSize:150"
      ],
      "process.created": "2020-04-29T20:05:30.530Z",
      "process.executable": "/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java",
      "process.name": "java",
      "process.pid": 2411,
      "server.bytes": 13320,
      "server.ip": "10.224.0.26",
      "server.packets": 9,
      "server.port": 49187,
      "service.type": "system",
      "source.bytes": 7828,
      "source.ip": "10.224.1.141",
      "source.packets": 11,
      "source.port": 33484,
      "system.audit.socket.egid": 1158,
      "system.audit.socket.euid": 1157,
      "system.audit.socket.gid": 1158,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c811bc400",
      "system.audit.socket.uid": 1157,
      "user.id": "1157"
      },
      {
      "@timestamp": "2020-04-29T20:23:54.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 2194,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 37106,
      "destination.bytes": 99,
      "destination.ip": "35.184.73.24",
      "destination.packets": 2,
      "destination.port": 5001,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2025316,
      "event.end": "2020-04-29T20:23:52.183Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:52.181Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2293,
      "network.community_id": "1:Nfiwum8Qm2MqZs3lPx7daUv88t0=",
      "network.direction": "unknown",
      "network.packets": 4,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/auditbeat",
      "-e",
      "-c",
      "/etc/auditbeat/auditbeat.yml"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/auditbeat-7.3.0-linux-x86_64/auditbeat",
      "process.name": "auditbeat",
      "process.pid": 2402,
      "server.bytes": 99,
      "server.ip": "35.184.73.24",
      "server.packets": 2,
      "server.port": 5001,
      "service.type": "system",
      "source.bytes": 2194,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 37106,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa92000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:54.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 1198,
      "client.ip": "127.0.0.1",
      "client.packets": 4,
      "client.port": 48060,
      "destination.bytes": 1566,
      "destination.ip": "127.0.0.1",
      "destination.packets": 3,
      "destination.port": 8778,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2101324,
      "event.end": "2020-04-29T20:23:52.369Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:52.367Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2764,
      "network.community_id": "1:yW5Cl7d24rDDfdaQejZ+p7MWst4=",
      "network.direction": "unknown",
      "network.packets": 7,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/metricbeat",
      "-e",
      "-c",
      "/etc/metricbeat/metricbeat.yml",
      "--path.home",
      "/etc/metricbeat"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/metricbeat-7.3.2-linux-x86_64/metricbeat",
      "process.name": "metricbeat",
      "process.pid": 2401,
      "server.bytes": 1566,
      "server.ip": "127.0.0.1",
      "server.packets": 3,
      "server.port": 8778,
      "service.type": "system",
      "source.bytes": 1198,
      "source.ip": "127.0.0.1",
      "source.packets": 4,
      "source.port": 48060,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c95483000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:54.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 1626,
      "client.ip": "127.0.0.1",
      "client.packets": 3,
      "client.port": 8778,
      "destination.bytes": 1118,
      "destination.ip": "127.0.0.1",
      "destination.packets": 4,
      "destination.port": 48060,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2032312,
      "event.end": "2020-04-29T20:23:52.369Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:52.367Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2744,
      "network.community_id": "1:yW5Cl7d24rDDfdaQejZ+p7MWst4=",
      "network.direction": "unknown",
      "network.packets": 7,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/metricbeat",
      "-e",
      "-c",
      "/etc/metricbeat/metricbeat.yml",
      "--path.home",
      "/etc/metricbeat"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/metricbeat-7.3.2-linux-x86_64/metricbeat",
      "process.name": "metricbeat",
      "process.pid": 2401,
      "server.bytes": 1118,
      "server.ip": "127.0.0.1",
      "server.packets": 4,
      "server.port": 48060,
      "service.type": "system",
      "source.bytes": 1626,
      "source.ip": "127.0.0.1",
      "source.packets": 3,
      "source.port": 8778,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c956e9980",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:55.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 2262,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 32824,
      "destination.bytes": 689,
      "destination.ip": "13.56.146.199",
      "destination.packets": 2,
      "destination.port": 9243,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 58397212,
      "event.end": "2020-04-29T20:23:53.429Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:53.370Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2951,
      "network.community_id": "1:VJOfnDgg0v6f+slKZKLaXnihIVA=",
      "network.direction": "unknown",
      "network.packets": 4,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/metricbeat",
      "-e",
      "-c",
      "/etc/metricbeat/metricbeat.yml",
      "--path.home",
      "/etc/metricbeat"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/metricbeat-7.3.2-linux-x86_64/metricbeat",
      "process.name": "metricbeat",
      "process.pid": 2401,
      "server.bytes": 689,
      "server.ip": "13.56.146.199",
      "server.packets": 2,
      "server.port": 9243,
      "service.type": "system",
      "source.bytes": 2262,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 32824,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa93800",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:56.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 52,
      "client.ip": "10.224.1.141",
      "client.packets": 1,
      "client.port": 36798,
      "destination.bytes": 32,
      "destination.ip": "52.52.242.223",
      "destination.packets": 1,
      "destination.port": 9243,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 40297573,
      "event.end": "2020-04-29T20:23:54.232Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:54.192Z",
      "flow.complete": false,
      "flow.final": true,
      "host.name": "60920b1008e7",
      "network.bytes": 84,
      "network.community_id": "1:cCxKAYFYmElLx6Mv2+ikKEcIZ9E=",
      "network.direction": "unknown",
      "network.packets": 2,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "server.bytes": 32,
      "server.ip": "52.52.242.223",
      "server.packets": 1,
      "server.port": 9243,
      "service.type": "system",
      "source.bytes": 52,
      "source.ip": "10.224.1.141",
      "source.packets": 1,
      "source.port": 36798,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c95485000"
      },
      {
      "@timestamp": "2020-04-29T20:23:56.567Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 527412,
      "client.ip": "10.224.1.141",
      "client.packets": 149,
      "client.port": 33484,
      "destination.bytes": 14760,
      "destination.ip": "10.224.0.26",
      "destination.packets": 54,
      "destination.port": 49187,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 65418548,
      "event.end": "2020-04-29T20:23:54.565Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:54.499Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "1158",
      "host.name": "60920b1008e7",
      "network.bytes": 542172,
      "network.community_id": "1:JFIcjp9WW3QoHSvELTlbh3gEb8A=",
      "network.direction": "unknown",
      "network.packets": 203,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "java",
      "-Djava.util.logging.config.file=/var/lib/jenkins/logging.properties",
      "-javaagent:/opt/jolokia/jolokia-agent.jar=config=/opt/jolokia/jolokia.properties",
      "-Dorg.jenkinsci.plugins.gitclient.Git.timeOut=10",
      "-jar",
      "/usr/local/share/jenkins/swarm-client-3.17.jar",
      "-description",
      "Ubuntu 16.04 - 2 CPU Cores / 3539 MB Memory",
      "-disableClientsUniqueId",
      "-executors",
      "1",
      "-fsroot",
      "/var/lib/jenkins",
      "-master",
      "https://beats-ci.elastic.co/",
      "-maxRetryInterval",
      "160",
      "-retryBackOffStrategy",
      "linear",
      "-mode",
      "exclusive",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-username",
      "local-admin",
      "-password",
      "1101471da3a313cbf3999ee6888d412a7e",
      "-name",
      "beats-ci-immutable-ubuntu-1604-1588190696301853669",
      "-master",
      "https://beats-ci.elastic.co",
      "-username",
      "local-admin",
      "-password",
      "11af84f32a633296a49297085d82b8279d",
      "-labels",
      "docker",
      "-labels",
      "immutable",
      "-labels",
      "linux",
      "-labels",
      "linux-immutable",
      "-labels",
      "swarm",
      "-labels",
      "ubuntu",
      "-labels",
      "ubuntu-16.04",
      "-labels",
      "ubuntu-16",
      "-labels",
      "virtual",
      "-labels",
      "x86_64",
      "-description",
      "MachineType:n1-highmem-4,DiskSize:150"
      ],
      "process.created": "2020-04-29T20:05:30.530Z",
      "process.executable": "/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java",
      "process.name": "java",
      "process.pid": 2411,
      "server.bytes": 14760,
      "server.ip": "10.224.0.26",
      "server.packets": 54,
      "server.port": 49187,
      "service.type": "system",
      "source.bytes": 527412,
      "source.ip": "10.224.1.141",
      "source.packets": 149,
      "source.port": 33484,
      "system.audit.socket.egid": 1158,
      "system.audit.socket.euid": 1157,
      "system.audit.socket.gid": 1158,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c811bc400",
      "system.audit.socket.uid": 1157,
      "user.id": "1157"
      },
      {
      "@timestamp": "2020-04-29T20:23:57.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 2227,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 37106,
      "destination.bytes": 99,
      "destination.ip": "35.184.73.24",
      "destination.packets": 2,
      "destination.port": 5001,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2226986,
      "event.end": "2020-04-29T20:23:55.185Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:55.183Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2326,
      "network.community_id": "1:Nfiwum8Qm2MqZs3lPx7daUv88t0=",
      "network.direction": "unknown",
      "network.packets": 4,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/auditbeat",
      "-e",
      "-c",
      "/etc/auditbeat/auditbeat.yml"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/auditbeat-7.3.0-linux-x86_64/auditbeat",
      "process.name": "auditbeat",
      "process.pid": 2402,
      "server.bytes": 99,
      "server.ip": "35.184.73.24",
      "server.packets": 2,
      "server.port": 5001,
      "service.type": "system",
      "source.bytes": 2227,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 37106,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa92000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      },
      {
      "@timestamp": "2020-04-29T20:23:58.566Z",
      "agent.ephemeral_id": "0fa5b388-af46-4632-b27f-ef351acaa865",
      "agent.hostname": "60920b1008e7",
      "agent.id": "f4462e67-d5d7-48fd-8f10-12224691a01c",
      "agent.type": "auditbeat",
      "agent.version": "8.0.0",
      "client.bytes": 2201,
      "client.ip": "10.224.1.141",
      "client.packets": 2,
      "client.port": 37106,
      "destination.bytes": 99,
      "destination.ip": "35.184.73.24",
      "destination.packets": 2,
      "destination.port": 5001,
      "ecs.version": "1.5.0",
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.duration": 2384914,
      "event.end": "2020-04-29T20:23:58.189Z",
      "event.kind": "event",
      "event.module": "system",
      "event.start": "2020-04-29T20:23:58.187Z",
      "flow.complete": false,
      "flow.final": true,
      "group.id": "0",
      "group.name": "root",
      "host.name": "60920b1008e7",
      "network.bytes": 2300,
      "network.community_id": "1:Nfiwum8Qm2MqZs3lPx7daUv88t0=",
      "network.direction": "unknown",
      "network.packets": 4,
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.args": [
      "/usr/local/bin/auditbeat",
      "-e",
      "-c",
      "/etc/auditbeat/auditbeat.yml"
      ],
      "process.created": "2020-04-29T20:05:30.520Z",
      "process.executable": "/opt/auditbeat-7.3.0-linux-x86_64/auditbeat",
      "process.name": "auditbeat",
      "process.pid": 2402,
      "server.bytes": 99,
      "server.ip": "35.184.73.24",
      "server.packets": 2,
      "server.port": 5001,
      "service.type": "system",
      "source.bytes": 2201,
      "source.ip": "10.224.1.141",
      "source.packets": 2,
      "source.port": 37106,
      "system.audit.socket.egid": 0,
      "system.audit.socket.euid": 0,
      "system.audit.socket.gid": 0,
      "system.audit.socket.kernel_sock_address": "0xffff8f7c9fa92000",
      "system.audit.socket.uid": 0,
      "user.id": "0",
      "user.name": "root"
      }
      ] don't match the condition: the documents contain {
      "agent.type": "auditbeat",
      "client.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 30
      },
      "client.ip": "127.25.215.226",
      "client.packets": 1,
      "client.port": 49127,
      "destination.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 30
      },
      "destination.ip": "127.4.84.143",
      "destination.packets": 1,
      "destination.port": 53,
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.kind": "event",
      "event.module": "system",
      "network.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 60
      },
      "network.direction": "inbound",
      "network.packets": 2,
      "network.transport": "udp",
      "network.type": "ipv4",
      "process.pid": 49391,
      "server.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 30
      },
      "server.ip": "127.4.84.143",
      "server.packets": 1,
      "server.port": 53,
      "source.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 30
      },
      "source.ip": "127.25.215.226",
      "source.packets": 1,
      "source.port": 49127,
      "user.id": "0"
      },
      {
      "agent.type": "auditbeat",
      "client.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 80
      },
      "client.ip": "127.107.7.14",
      "client.packets": {
      "operator": "",
      "type": "comparison",
      "value": 2
      },
      "client.port": 44171,
      "destination.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 2
      },
      "destination.domain": "elastic.co",
      "destination.ip": "127.101.220.29",
      "destination.packets": {
      "operator": "",
      "type": "comparison",
      "value": 2
      },
      "destination.port": 51907,
      "event.action": "network_flow",
      "event.category": "network_traffic",
      "event.dataset": "socket",
      "event.kind": "event",
      "event.module": "system",
      "network.direction": "inbound",
      "network.packets": {
      "operator": "",
      "type": "comparison",
      "value": 5
      },
      "network.transport": "tcp",
      "network.type": "ipv4",
      "process.pid": 49391,
      "server.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 2
      },
      "server.domain": "elastic.co",
      "server.ip": "127.101.220.29",
      "server.packets": {
      "operator": "",
      "type": "comparison",
      "value": 2
      },
      "server.port": 51907,
      "service.type": "system",
      "source.bytes": {
      "operator": "",
      "type": "comparison",
      "value": 80
      },
      "source.ip": "127.107.7.14",
      "source.packets": {
      "operator": "",
      "type": "comparison",
      "value": 2
      },
      "source.port": 44171
      }

Steps errors

Expand to view the steps failures

  • Name: Mage update build test

    • Description: mage update build test

    • Result: FAILURE

    • Duration: 12 min 19 sec<

    • Start Time: 2020-04-29T20:13:06.619+0000

  • Name: Mage build test

    • Description: mage build test

    • Result: FAILURE

    • Duration: 9 min 53 sec<

    • Start Time: 2020-04-29T20:12:57.335+0000

  • Name: Make -C generator/_templates/metricbeat test

    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 2 min 25 sec<

    • Start Time: 2020-04-29T20:13:06.867+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-29T20:49:38.308Z]   OS/Arch:          linux/amd64
[2020-04-29T20:49:38.308Z]   Experimental:     false
[2020-04-29T20:49:38.308Z]  containerd:
[2020-04-29T20:49:38.308Z]   Version:          1.2.13
[2020-04-29T20:49:38.308Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2020-04-29T20:49:38.308Z]  runc:
[2020-04-29T20:49:38.308Z]   Version:          1.0.0-rc10
[2020-04-29T20:49:38.308Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-04-29T20:49:38.308Z]  docker-init:
[2020-04-29T20:49:38.308Z]   Version:          0.18.0
[2020-04-29T20:49:38.308Z]   GitCommit:        fec3683
[2020-04-29T20:49:42.045Z] Recording test results
[2020-04-29T20:49:42.046Z] Archiving artifacts
[2020-04-29T20:49:42.572Z] None of the test reports contained any result
[2020-04-29T20:49:42.584Z] Archiving artifacts
[2020-04-29T20:49:42.857Z] java.lang.InterruptedException: no matches found within 10000
[2020-04-29T20:49:42.857Z] 	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2826)
[2020-04-29T20:49:42.857Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
[2020-04-29T20:49:42.857Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2686)
[2020-04-29T20:49:42.857Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3073)
[2020-04-29T20:49:42.858Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1604-1588190696302007193.c.elastic-ci-prod.internal/10.224.0.125:46938
[2020-04-29T20:49:42.858Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
[2020-04-29T20:49:42.858Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2020-04-29T20:49:42.858Z] 		at hudson.remoting.Channel.call(Channel.java:998)
[2020-04-29T20:49:42.858Z] 		at hudson.FilePath.act(FilePath.java:1069)
[2020-04-29T20:49:42.858Z] 		at hudson.FilePath.act(FilePath.java:1058)
[2020-04-29T20:49:42.858Z] 		at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-04-29T20:49:42.858Z] 		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-04-29T20:49:42.858Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-04-29T20:49:42.858Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-04-29T20:49:42.858Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-04-29T20:49:42.858Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-04-29T20:49:42.858Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-29T20:49:42.858Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-29T20:49:42.858Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-29T20:49:42.858Z] Caused: hudson.FilePath$TunneledInterruptedException
[2020-04-29T20:49:42.858Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3075)
[2020-04-29T20:49:42.858Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-04-29T20:49:42.858Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-04-29T20:49:42.858Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-04-29T20:49:42.858Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-04-29T20:49:42.858Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-29T20:49:42.858Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-29T20:49:42.858Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-29T20:49:42.858Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-04-29T20:49:42.858Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-04-29T20:49:42.858Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-04-29T20:49:42.858Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-04-29T20:49:42.858Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-04-29T20:49:42.858Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-04-29T20:49:42.858Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-04-29T20:49:42.858Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-04-29T20:49:42.858Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-04-29T20:49:42.858Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-04-29T20:49:42.858Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-29T20:49:42.858Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-29T20:49:42.858Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-29T20:49:42.858Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-04-29T20:49:42.858Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-04-29T20:49:43.260Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-29T20:49:43.341Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-29T20:49:43.521Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.521Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.521Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.521Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.521Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.521Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.521Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.521Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T20:49:43.521Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-29T20:49:43.607Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-29T20:49:43.607Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-29T20:49:45.283Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-17656
[2020-04-29T20:49:45.398Z] [INFO] getVaultSecret: Getting secrets
[2020-04-29T20:49:45.463Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-29T20:49:46.093Z] + chmod 755 generate-build-data.sh
[2020-04-29T20:49:46.093Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17656/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17656/runs/26 FAILURE 4031907
[2020-04-29T20:49:46.644Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17656/runs/26/steps/?limit=10000 -o steps-info.json
[2020-04-29T20:49:47.194Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-17656/runs/26/tests/?status=FAILED -o tests-errors.json

@blakerouse
Copy link
Contributor Author

Okay all comments have either been answered or resolved. Anyway I can get a +1 to get this landed. Blocking me adding even more kubernetes tests.

Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@blakerouse blakerouse merged commit db29874 into elastic:master Apr 29, 2020
@blakerouse blakerouse deleted the k8s-integration-use-kind branch April 29, 2020 20:54
blakerouse added a commit to blakerouse/beats that referenced this pull request Apr 29, 2020
…p a kubernetes cluster (elastic#17656)

* Use kind to bring up a local kubernetes cluster and then run the integration tests against the kind cluster.

* Run mage update.

* Add more tests.

* Fix more tests, install kind in prepare-tests.

* Switch to running kubernetes integration tests inside of Kubernetes.

* Use golang 1.13.9.

* Fix for other beats.

* Run mage fmt.

* Don't run kubernetes integration tests if not inside kubernetes environment.

* Fix metricbeat to use mage and the makefile shim for mage.

* Improve the error message when kind or kubectl is not available.

* Refactor the integration tests into a more module system.

* Fix go vet.

* Setup travis to use kind.

* Run kubernetes integration tests in Jenkins.

* Fix filebeat magefile.

* Fix travis and Jenkins.

* Check requirements of the test runner before actually running the tests.

* Add return on parsebool error.

* Don't return err on missing requirements for tester.

* Run make update.

* Move the kubernetes items to its own module, import that module only by metricbeat.

(cherry picked from commit db29874)
@jsoriano
Copy link
Member

jsoriano commented Apr 30, 2020

I have seen after updating #17480 with master that with this change integration tests for modules without docker compose or kubernetes files are not executed, not so serious because it only affects some system tests, and cloud tests, that are not merged yet. I will prepare a fix.

@blakerouse
Copy link
Contributor Author

@jsoriano Could you file a bug about what is not working? Be happy to fix it.

@jsoriano
Copy link
Member

@jsoriano Could you file a bug about what is not working? Be happy to fix it.

@blakerouse I opened a PR with a temptative fix #18125, let me know what you think.

@andrewkroh
Copy link
Member

@blakerouse The Metricbeat generator test is failing in master. Do you think it's related to your changes?

https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats-beats-mbp/detail/master/1571/pipeline/573

[2020-04-30T16:41:41.466Z] beatpath/testmetricbeat imports

[2020-04-30T16:41:41.466Z] 	github.com/elastic/beats/v7/dev-tools/mage imports

[2020-04-30T16:41:41.466Z] 	k8s.io/client-go/kubernetes imports

[2020-04-30T16:41:41.466Z] 	k8s.io/client-go/discovery imports

[2020-04-30T16:41:41.466Z] 	github.com/googleapis/gnostic/OpenAPIv2: module github.com/googleapis/gnostic@latest found (v0.4.1), but does not contain package github.com/googleapis/gnostic/OpenAPIv2

blakerouse added a commit that referenced this pull request Apr 30, 2020
… a pod and use kind to setup a kubernetes cluster (#18111)

* Run kubernetes integration tests inside of a pod and use kind to setup a kubernetes cluster (#17656)

* Use kind to bring up a local kubernetes cluster and then run the integration tests against the kind cluster.

* Run mage update.

* Add more tests.

* Fix more tests, install kind in prepare-tests.

* Switch to running kubernetes integration tests inside of Kubernetes.

* Use golang 1.13.9.

* Fix for other beats.

* Run mage fmt.

* Don't run kubernetes integration tests if not inside kubernetes environment.

* Fix metricbeat to use mage and the makefile shim for mage.

* Improve the error message when kind or kubectl is not available.

* Refactor the integration tests into a more module system.

* Fix go vet.

* Setup travis to use kind.

* Run kubernetes integration tests in Jenkins.

* Fix filebeat magefile.

* Fix travis and Jenkins.

* Check requirements of the test runner before actually running the tests.

* Add return on parsebool error.

* Don't return err on missing requirements for tester.

* Run make update.

* Move the kubernetes items to its own module, import that module only by metricbeat.

(cherry picked from commit db29874)

* Update fields.
@blakerouse
Copy link
Contributor Author

@andrewkroh I am working on this here - #18086

The latest run I got with that MP showed something different.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Team:Platforms Label for the Integrations - Platforms team v7.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants