Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add leader election for autodiscover #20281

Merged
merged 22 commits into from
Aug 10, 2020

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Jul 28, 2020

What does this PR do?

Implements leader election as part of kubernetes autodiscover provider. With this, when the beat is elected as leader it will start cluster scope metricsets.

Under the hood, all leader candidates will try to gain the lock on Lease object (https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#lease-v1-coordination-k8s-io). When the lock is gained then the configured cluster wide mericsets will get started.

Why is it important?

To get rid of Deployment procedure for the singleton Metricbeat instance which is needed for the cluster scope metricsets.
Sample configuration that will start state_node metricset when a leadership is gained:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      scope: cluster
      node: ${NODE_NAME}
      unique: true
      identifier: leaderelectionmetricbeat
      templates:
        - config:
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_node

Then we can monitor the lease to check which provider holds the lock:

kubectl describe lease beats-cluster-leader
Name:         beats-cluster-leader
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  coordination.k8s.io/v1
Kind:         Lease
Metadata:
  Creation Timestamp:  2020-07-30T14:48:16Z
  Resource Version:    357033
  Self Link:           /apis/coordination.k8s.io/v1/namespaces/default/leases/beats-cluster-leader
  UID:                 ef255352-3d30-42f9-b27a-3a3e5dcb80dd
Spec:
  Acquire Time:            2020-07-30T14:51:00.592028Z
  Holder Identity:         beats-leader-leaderelectionmetricbeat
  Lease Duration Seconds:  15
  Lease Transitions:       2
  Renew Time:              2020-07-30T14:51:00.597852Z
Events:                    <none>

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

It is recommended to testing this along with #20512 for easiness of configuration/setup etc

  1. Prepare a multinode k8s cluster (for instance on gke)
  2. Deploy Metricbeat as a Daemonset on the cluster with leader election config:
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      scope: cluster
      node: ${NODE_NAME}
      unique: true
      leader_lease: leaderelectionmetricbeat
      templates:
        - config:
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_node
  1. Make sure that events for state_node are being collected from only one metricbeat instance. You can the in the logs and verify that only one metricbeat has gained the lock so far.

  2. Monitor the lease object to keep track of the lock holder: watch kubectl describe lease beats-cluster-leader

  3. Stop the Metricbeat instance that currently holds the lock and make sure that another one takes over and we keep getting events from state-node. Make sure that the lease changed holder (logs will also let us know that a new metricset is getting started)

  4. Also test without setting identifier and check that it takes the default value properly like metricbeat-cluster-leader.

  5. Check with old autodiscover (template based/hints based) for possible regressions.

Related issues

Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added enhancement in progress Pull request is currently in progress. [zube]: In Progress autodiscovery Team:Platforms Label for the Integrations - Platforms team labels Jul 28, 2020
@ChrsMark ChrsMark self-assigned this Jul 28, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 28, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 28, 2020

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

Pipeline View Test View Changes Artifacts

Expand to view the summary

Build stats

  • Build Cause: [jsoriano commented: jenkins run the tests please]

  • Start Time: 2020-08-08T08:53:30.466+0000

  • Duration: 124 min 9 sec

Test stats 🧪

Test Results
Failed 0
Passed 14331
Skipped 1379
Total 15710

Log output

Expand to view the last 100 lines of log output

[2020-08-08T10:57:06.770Z] c1e54eec4b57: Verifying Checksum
[2020-08-08T10:57:06.770Z] c1e54eec4b57: Download complete
[2020-08-08T10:57:06.770Z] c1e54eec4b57: Pull complete
[2020-08-08T10:57:06.770Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-08-08T10:57:06.770Z] Status: Downloaded newer image for alpine:3.4
[2020-08-08T10:57:08.999Z] + python .ci/scripts/pre_archive_test.py
[2020-08-08T10:57:08.999Z] Copy ./x-pack/libbeat/build into build/x-pack/libbeat/build
[2020-08-08T10:57:09.012Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/build
[2020-08-08T10:57:09.037Z] Recording test results
[2020-08-08T10:57:10.627Z] Stashed 2 file(s)
[2020-08-08T10:57:10.638Z] Archiving artifacts
[2020-08-08T10:57:11.020Z] + python .ci/scripts/search_system_tests.py
[2020-08-08T10:57:11.066Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2020-08-08T10:57:12.792Z] Failed in branch Libbeat x-pack
[2020-08-08T10:57:12.922Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats
[2020-08-08T10:57:13.232Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-08-08T10:57:13.247Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Lint
[2020-08-08T10:57:13.331Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-08-08T10:57:13.408Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-08-08T10:57:13.487Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Winlogbeat-oss
[2020-08-08T10:57:13.565Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Dockerlogbeat
[2020-08-08T10:57:13.645Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Journalbeat-oss
[2020-08-08T10:57:13.721Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-08-08T10:57:13.797Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Packetbeat-Linux
[2020-08-08T10:57:13.874Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-08-08T10:57:13.952Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-08-08T10:57:14.032Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-08-08T10:57:14.115Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Heartbeat-oss
[2020-08-08T10:57:14.192Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-08-08T10:57:14.268Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-08-08T10:57:14.344Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-08-08T10:57:14.420Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-08-08T10:57:14.497Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-08-08T10:57:14.578Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-08-08T10:57:14.653Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-08-08T10:57:14.730Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-Windows
[2020-08-08T10:57:14.805Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-08-08T10:57:14.881Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-08-08T10:57:14.957Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-Windows
[2020-08-08T10:57:15.041Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Heartbeat-Windows
[2020-08-08T10:57:15.124Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Functionbeat-Windows
[2020-08-08T10:57:15.205Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Packetbeat-Windows
[2020-08-08T10:57:15.287Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-08-08T10:57:15.367Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-oss
[2020-08-08T10:57:15.452Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-x-pack
[2020-08-08T10:57:15.538Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Libbeat-oss
[2020-08-08T10:57:15.625Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Libbeat-crosscompile
[2020-08-08T10:57:15.704Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Libbeat-stress-tests
[2020-08-08T10:57:15.782Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-08-08T10:57:15.862Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-08-08T10:57:15.944Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-08-08T10:57:16.021Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Libbeat-x-pack
[2020-08-08T10:57:16.380Z] + cat
[2020-08-08T10:57:16.380Z] + /usr/local/bin/runbld ./runbld-script
[2020-08-08T10:57:16.380Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-08-08T10:57:22.984Z] runbld>>> runbld started
[2020-08-08T10:57:22.984Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-08-08T10:57:23.926Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-20281' in order of occurrence in the config (last value wins).
[2020-08-08T10:57:25.308Z] runbld>>> Debug logging enabled.
[2020-08-08T10:57:25.308Z] runbld>>> Storing result
[2020-08-08T10:57:25.568Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-08-08T10:57:25.568Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200808105725-ABEDAB75
[2020-08-08T10:57:25.568Z] runbld>>> Adding system facts.
[2020-08-08T10:57:26.509Z] runbld>>> Adding vcs info for the latest commit:  da6b30921a1ab8949ae86b8e18229d0d86b37b2d
[2020-08-08T10:57:26.509Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-08-08T10:57:26.509Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-08-08T10:57:26.509Z] + echo 'Processing JUnit reports with runbld...'
[2020-08-08T10:57:26.509Z] Processing JUnit reports with runbld...
[2020-08-08T10:57:27.081Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-08-08T10:57:27.081Z] runbld>>> DURATION: 37ms
[2020-08-08T10:57:27.081Z] runbld>>> STDOUT: 40 bytes
[2020-08-08T10:57:27.081Z] runbld>>> STDERR: 49 bytes
[2020-08-08T10:57:27.081Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-08-08T10:57:27.081Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats
[2020-08-08T10:57:28.031Z] runbld>>> Storing build metadata: 
[2020-08-08T10:57:28.031Z] runbld>>> Adding test report.
[2020-08-08T10:57:28.031Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats
[2020-08-08T10:57:28.975Z] runbld>>> Found 117 test output files
[2020-08-08T10:57:29.547Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-08-08T10:57:29.547Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-08-08T10:57:29.547Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-08-08T10:57:29.808Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-08-08T10:57:29.808Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-08-08T10:57:29.808Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-08-08T10:57:29.808Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-08-08T10:57:31.721Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 15560 Skipped: 1142
[2020-08-08T10:57:31.721Z] runbld>>> Storing result
[2020-08-08T10:57:31.721Z] runbld>>> FAILURES: 0
[2020-08-08T10:57:31.721Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-08-08T10:57:31.721Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200808105725-ABEDAB75
[2020-08-08T10:57:31.981Z] runbld>>> Email notification disabled by environment variable.
[2020-08-08T10:57:31.981Z] runbld>>> Slack notification disabled by environment variable.
[2020-08-08T10:57:37.976Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20281
[2020-08-08T10:57:38.249Z] [INFO] getVaultSecret: Getting secrets
[2020-08-08T10:57:38.355Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-08T10:57:39.425Z] + chmod 755 generate-build-data.sh
[2020-08-08T10:57:39.426Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/runs/23 ABORTED 7448698
[2020-08-08T10:57:39.426Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/runs/23/steps/?limit=10000 -o steps-info.json
[2020-08-08T10:57:41.287Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/runs/23/tests/?status=FAILED -o tests-errors.json
[2020-08-08T10:57:42.198Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/runs/23/log/ -o pipeline-log.txt

ChrsMark added 4 commits July 30, 2020 15:54
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member Author

ChrsMark commented Jul 31, 2020

Tested this on a 3-node cluster on GKE. Seems to work:

  1. Only one of the 3 Metricbeats ships events for state_node metricset.
  2. Killing the lock holder another takes over
    (see more testing info on testing note in the PR's description)

I will open it for review by now.

@ChrsMark ChrsMark requested review from jsoriano and exekias July 31, 2020 12:06
@ChrsMark ChrsMark marked this pull request as ready for review July 31, 2020 12:07
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark changed the title [WIP] Add leader election for autodiscover Add leader election for autodiscover Aug 3, 2020
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark
Copy link
Member Author

ChrsMark commented Aug 5, 2020

Thanks for the review and the (offline) suggestions @jsoriano!
Note that documentation section is missing but I plan to add it soonish since we have almost conclude to the implementation details.

Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@jsoriano jsoriano added needs_backport PR is waiting to be backported to other branches. test-plan Add this PR to be manual test plan v7.10.0 labels Aug 6, 2020
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, only some small details.


// Start for EventManager interface.
func (p *leaderElectionManager) Start() {
ctx, cancel := context.WithCancel(context.Background())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit.

Suggested change
ctx, cancel := context.WithCancel(context.Background())
ctx, cancel := context.WithCancel(context.TODO())

func (p *leaderElectionManager) startLeaderElector(ctx context.Context, lec leaderelection.LeaderElectionConfig) {
le, err := leaderelection.NewLeaderElector(lec)
if err != nil {
p.logger.Errorf("leader election lock GAINED, id %v", err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't look like an error 🙂 Log it at the info or debug level?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually an error..but with wrong error message. This piece is "copied" from RunOrDie (https://github.com/kubernetes/client-go/blob/36233866f1c7c0ad3bdac1fc466cb5de3746cfa2/tools/leaderelection/leaderelection.go#L214). See the watchdog related comment for more info.

if err != nil {
p.logger.Errorf("leader election lock GAINED, id %v", err)
}
if lec.WatchDog != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this watchdog? we are never setting it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was mostly because I tried to "copy" the RunOrDie of https://github.com/kubernetes/client-go/blob/36233866f1c7c0ad3bdac1fc466cb5de3746cfa2/tools/leaderelection/leaderelection.go#L219, because of an async problem described at kubernetes/client-go#837 .

Good catch I will remove it, it's redundant.

@@ -367,6 +367,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Added the `max_cached_sessions` option to the script processor. {pull}19562[19562]
- Add support for DNS over TLS for the dns_processor. {pull}19321[19321]
- Set index.max_docvalue_fields_search in index template to increase value to 200 fields. {issue}20215[20215]
- Add leader election for autodiscover. {pull}20281[20281]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Add leader election for autodiscover. {pull}20281[20281]
- Add leader election for Kubernetes autodiscover. {pull}20281[20281]

libbeat/docs/shared-autodiscover.asciidoc Show resolved Hide resolved
Signed-off-by: chrismark <chrismarkou92@gmail.com>
@jsoriano
Copy link
Member

jsoriano commented Aug 8, 2020

jenkins run the tests please

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but let's wait to have a greener build.

@elasticmachine
Copy link
Collaborator

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20281 updated]

  • Start Time: 2020-08-10T06:21:27.285+0000

  • Duration: 123 min 50 sec

Test stats 🧪

Test Results
Failed 2
Passed 15585
Skipped 1860
Total 17447

Test errors

Expand to view the tests failures

  • Name: Build and Test / Libbeat / Libbeat oss / TestReplaceClientWorker – pipeline

    • Age: 1
    • Duration: 20.94
    • Error Details: Failed
  • Name: Build and Test / Libbeat / Libbeat oss / TestReplaceClientWorker/network_client – pipeline

    • Age: 1
    • Duration: 20.27
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: Mage build test
    • Description: mage build test

    • Duration: 4 min 35 sec

    • Start Time: 2020-08-10T06:45:40.166+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-08-10T08:23:44.120Z] Recording test results
[2020-08-10T08:23:45.195Z] Stashed 2 file(s)
[2020-08-10T08:23:45.205Z] Archiving artifacts
[2020-08-10T08:23:45.592Z] + python .ci/scripts/search_system_tests.py
[2020-08-10T08:23:45.636Z] [INFO] system-tests=''. If no empty then let's create a tarball
[2020-08-10T08:23:47.195Z] Failed in branch Libbeat x-pack
[2020-08-10T08:23:47.329Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats
[2020-08-10T08:23:47.634Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-08-10T08:23:47.648Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Lint
[2020-08-10T08:23:47.738Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-08-10T08:23:47.816Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-08-10T08:23:47.894Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-08-10T08:23:47.979Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-08-10T08:23:48.064Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Winlogbeat-oss
[2020-08-10T08:23:48.147Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Dockerlogbeat
[2020-08-10T08:23:48.224Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-08-10T08:23:48.306Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Journalbeat-oss
[2020-08-10T08:23:48.384Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-08-10T08:23:48.463Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-08-10T08:23:48.547Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-08-10T08:23:48.632Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Libbeat-oss
[2020-08-10T08:23:48.711Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-08-10T08:23:48.791Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Packetbeat-Linux
[2020-08-10T08:23:48.871Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-08-10T08:23:48.948Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-08-10T08:23:49.173Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Heartbeat-oss
[2020-08-10T08:23:49.257Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-08-10T08:23:49.343Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-08-10T08:23:49.425Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-08-10T08:23:49.506Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-08-10T08:23:49.583Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-08-10T08:23:49.678Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-08-10T08:23:49.762Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-08-10T08:23:49.859Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-08-10T08:23:49.937Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-08-10T08:23:50.016Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-Windows
[2020-08-10T08:23:50.095Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-08-10T08:23:50.176Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-08-10T08:23:50.250Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-08-10T08:23:50.332Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-08-10T08:23:50.418Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-Windows
[2020-08-10T08:23:50.495Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-08-10T08:23:50.571Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-x-pack
[2020-08-10T08:23:50.661Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Functionbeat-Windows
[2020-08-10T08:23:50.738Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Heartbeat-Windows
[2020-08-10T08:23:50.815Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-oss
[2020-08-10T08:23:50.893Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Packetbeat-Windows
[2020-08-10T08:23:50.971Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-08-10T08:23:51.047Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-08-10T08:23:51.125Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-08-10T08:23:51.201Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-08-10T08:23:51.278Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-08-10T08:23:51.355Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-08-10T08:23:51.439Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Libbeat-x-pack
[2020-08-10T08:23:51.794Z] + cat
[2020-08-10T08:23:51.794Z] + /usr/local/bin/runbld ./runbld-script
[2020-08-10T08:23:51.794Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-08-10T08:23:59.925Z] runbld>>> runbld started
[2020-08-10T08:23:59.925Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-08-10T08:24:00.861Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-20281' in order of occurrence in the config (last value wins).
[2020-08-10T08:24:02.238Z] runbld>>> Debug logging enabled.
[2020-08-10T08:24:02.238Z] runbld>>> Storing result
[2020-08-10T08:24:02.497Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-08-10T08:24:02.497Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200810082402-2483CA1D
[2020-08-10T08:24:02.497Z] runbld>>> Adding system facts.
[2020-08-10T08:24:03.910Z] runbld>>> Adding vcs info for the latest commit:  df4033e3513534111d0e86af3e6f6daf55b31c4d
[2020-08-10T08:24:03.910Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-08-10T08:24:03.910Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-08-10T08:24:03.910Z] Processing JUnit reports with runbld...
[2020-08-10T08:24:03.910Z] + echo 'Processing JUnit reports with runbld...'
[2020-08-10T08:24:04.179Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-08-10T08:24:04.179Z] runbld>>> DURATION: 56ms
[2020-08-10T08:24:04.179Z] runbld>>> STDOUT: 40 bytes
[2020-08-10T08:24:04.179Z] runbld>>> STDERR: 49 bytes
[2020-08-10T08:24:04.179Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-08-10T08:24:04.179Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats
[2020-08-10T08:24:05.563Z] runbld>>> Storing build metadata: 
[2020-08-10T08:24:05.563Z] runbld>>> Adding test report.
[2020-08-10T08:24:05.563Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats
[2020-08-10T08:24:06.130Z] runbld>>> Found 132 test output files
[2020-08-10T08:24:07.507Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-08-10T08:24:07.507Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-08-10T08:24:07.507Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-08-10T08:24:07.507Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-08-10T08:24:07.507Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-08-10T08:24:07.507Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-08-10T08:24:07.507Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20281/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-08-10T08:24:10.039Z] runbld>>> Test output logs contained: Errors: 0 Failures: 2 Tests: 17297 Skipped: 1599
[2020-08-10T08:24:10.039Z] runbld>>> Storing result
[2020-08-10T08:24:10.039Z] runbld>>> FAILURES: 2
[2020-08-10T08:24:10.609Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-08-10T08:24:10.609Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200810082402-2483CA1D
[2020-08-10T08:24:10.609Z] runbld>>> Email notification disabled by environment variable.
[2020-08-10T08:24:10.609Z] runbld>>> Slack notification disabled by environment variable.
[2020-08-10T08:24:16.314Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20281
[2020-08-10T08:24:16.530Z] [INFO] getVaultSecret: Getting secrets
[2020-08-10T08:24:16.610Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-10T08:24:17.384Z] + chmod 755 generate-build-data.sh
[2020-08-10T08:24:17.384Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/runs/25 FAILURE 7369837
[2020-08-10T08:24:17.384Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20281/runs/25/steps/?limit=10000 -o steps-info.json

@ChrsMark
Copy link
Member Author

@jsoriano I think the only issue with the tests is https://travis-ci.org/github/elastic/beats/jobs/716456468#L1590 which is also failing in general currently. I think we can proceed with merging, wdyt?

@jsoriano
Copy link
Member

@jsoriano I think the only issue with the tests is https://travis-ci.org/github/elastic/beats/jobs/716456468#L1590 which is also failing in general currently. I think we can proceed with merging, wdyt?

Yes, I think the issue in CI is not related.

@ChrsMark ChrsMark merged commit 9ab9b97 into elastic:master Aug 10, 2020
ChrsMark added a commit to ChrsMark/beats that referenced this pull request Aug 10, 2020
@ChrsMark ChrsMark removed the needs_backport PR is waiting to be backported to other branches. label Aug 10, 2020
ChrsMark added a commit that referenced this pull request Aug 11, 2020
@andresrc andresrc added the test-plan-added This PR has been added to the test plan label Oct 3, 2020
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
jorgelbg added a commit to jorgelbg/beats that referenced this pull request Nov 16, 2020
After PR elastic#20281 was merged, the signature of the `autodiscover.ProviderBuilder` changed to accept an additional name (string) parameter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
autodiscovery enhancement in progress Pull request is currently in progress. review Team:Platforms Label for the Integrations - Platforms team test-plan Add this PR to be manual test plan test-plan-added This PR has been added to the test plan v7.10.0 [zube]: In Progress [zube]: In Review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants