Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[filebeat] Fix shutdown tracking in s3 input #21380

Merged
merged 1 commit into from
Sep 29, 2020
Merged

Conversation

faec
Copy link
Contributor

@faec faec commented Sep 29, 2020

This is just a short fix -- the s3 input's wait group (used to wait for shutdown) currently calls wg.Done() immediately after the (asynchronous) call site, so the wg.Wait() on shutdown is always a no-op. This PR moves the wg.Done() call to the end of the worker goroutine, which is probably what was intended.

@faec faec added bug Filebeat Filebeat Team:Integrations Label for the Integrations team labels Sep 29, 2020
@faec faec requested a review from kaiyan-sheng September 29, 2020 13:37
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Sep 29, 2020
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for fixing this!

@kaiyan-sheng kaiyan-sheng added the needs_backport PR is waiting to be backported to other branches. label Sep 29, 2020
@elasticmachine
Copy link
Collaborator

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21380 opened]

  • Start Time: 2020-09-29T13:39:22.607+0000

  • Duration: 82 min 2 sec

Test stats 🧪

Test Results
Failed 3
Passed 20302
Skipped 1855
Total 22160

Test errors

Expand to view the tests failures

  • Name: Build&Test / metricbeat-goIntegTest / TestFetch – dbstats

    • Age: 1
    • Duration: 300.12
    • Error Details: Failed
  • Name: Build&Test / metricbeat-goIntegTest / TestFetch – metrics

    • Age: 1
    • Duration: 300.12
    • Error Details: Failed
  • Name: Build&Test / metricbeat-goIntegTest / TestFetch – replstatus

    • Age: 1
    • Duration: 300.12
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: mage goIntegTest

    • Description: mage goIntegTest

    • Duration: 41 min 41 sec

    • Start Time: 2020-09-29T14:08:31.836+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 1

    • Duration: 0 min 1 sec

    • Start Time: 2020-09-29T14:49:33.255+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-09-29T15:00:53.548Z] + rm source.tgz
[2020-09-29T15:00:53.677Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats
[2020-09-29T15:00:53.715Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/uncategorized-1601388230156
[2020-09-29T15:00:53.816Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/libbeat-stress-tests-1601388505599
[2020-09-29T15:00:53.916Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/winlogbeat-crosscompile-1601388584678
[2020-09-29T15:00:54.015Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-elastic-agent-build-1601388594142
[2020-09-29T15:00:54.117Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/auditbeat-crosscompile-1601388622961
[2020-09-29T15:00:54.219Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/journalbeat-unitTest-1601388634011
[2020-09-29T15:00:54.316Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-dockerlogbeat-build-1601388665456
[2020-09-29T15:00:54.416Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-functionbeat-build-1601388706772
[2020-09-29T15:00:54.624Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/libbeat-crosscompile-1601388791200
[2020-09-29T15:00:54.727Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/packetbeat-build-1601388811287
[2020-09-29T15:00:54.830Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-elastic-agent-windows-windows-2019-1601388859782
[2020-09-29T15:00:54.943Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-unitTest-1601388873289
[2020-09-29T15:00:55.052Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/auditbeat-windows-windows-2019-1601388961950
[2020-09-29T15:00:55.158Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-auditbeat-build-1601388967498
[2020-09-29T15:00:55.259Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/heartbeat-build-1601388969288
[2020-09-29T15:00:55.359Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019-1601389005086
[2020-09-29T15:00:55.460Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/auditbeat-build-1601389028058
[2020-09-29T15:00:55.560Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-winlogbeat-build-windows-2019-1601389032223
[2020-09-29T15:00:55.661Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/heartbeat-windows-windows-2019-1601389035621
[2020-09-29T15:00:55.759Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-libbeat-build-1601389059883
[2020-09-29T15:00:55.856Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/winlogbeat-windows-windows-2019-1601389074699
[2020-09-29T15:00:55.953Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019-1601389106669
[2020-09-29T15:00:56.055Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019-1601389134815
[2020-09-29T15:00:56.151Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/generator-metricbeat-test-1601389155612
[2020-09-29T15:00:56.249Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-crosscompile-1601389180642
[2020-09-29T15:00:56.351Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/packetbeat-windows-windows-2019-1601389218106
[2020-09-29T15:00:56.450Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/generator-beat-test-1601389240697
[2020-09-29T15:00:56.551Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/filebeat-windows-windows-2019-1601389240912
[2020-09-29T15:00:56.657Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-windows-windows-2019-1601389371118
[2020-09-29T15:00:56.770Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019-1601389399235
[2020-09-29T15:00:56.879Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/generator-macos-metricbeat-macosx-1601389461349
[2020-09-29T15:00:56.990Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/filebeat-macos-macosx-1601389640252
[2020-09-29T15:00:57.097Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/generator-macos-beat-macosx-1601389668508
[2020-09-29T15:00:57.207Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/heartbeat-macos-macosx-1601389677551
[2020-09-29T15:00:57.314Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/auditbeat-macos-macosx-1601389760049
[2020-09-29T15:00:57.411Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-elastic-agent-macos-macosx-1601389821028
[2020-09-29T15:00:57.525Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-auditbeat-macos-macosx-1601389838440
[2020-09-29T15:00:57.630Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/packetbeat-macos-macosx-1601389889308
[2020-09-29T15:00:57.735Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-filebeat-macos-macosx-1601389927365
[2020-09-29T15:00:57.845Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-macos-macosx-1601390132840
[2020-09-29T15:00:57.952Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/libbeat-build-1601390190138
[2020-09-29T15:00:58.063Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/filebeat-build-1601390194834
[2020-09-29T15:00:58.165Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-filebeat-build-1601390359185
[2020-09-29T15:00:58.265Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-functionbeat-macos-macosx-1601390464652
[2020-09-29T15:00:58.363Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-pythonIntegTest-1601390774286
[2020-09-29T15:00:58.472Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-goIntegTest-1601390964391
[2020-09-29T15:00:58.578Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-build-1601391330248
[2020-09-29T15:00:58.679Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-macos-macosx-1601391482482
[2020-09-29T15:00:59.128Z] + cat
[2020-09-29T15:00:59.128Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-09-29T15:00:59.128Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-29T15:01:05.731Z] runbld>>> runbld started
[2020-09-29T15:01:05.731Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-29T15:01:07.655Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-29T15:01:07.655Z] runbld>>> Matches in the system config:
[2020-09-29T15:01:07.655Z] runbld>>> - Matched ^elastic\+beats
[2020-09-29T15:01:07.655Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-29T15:01:09.045Z] runbld>>> Debug logging enabled.
[2020-09-29T15:01:09.045Z] runbld>>> Storing result
[2020-09-29T15:01:09.309Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-29T15:01:09.309Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200929150108-DB8BF9D9
[2020-09-29T15:01:09.309Z] runbld>>> Adding system facts.
[2020-09-29T15:01:10.699Z] runbld>>> Adding vcs info for the latest commit:  55c4f24058ecad8dbccfdcd302e1f09a2873e8dc
[2020-09-29T15:01:10.699Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-29T15:01:10.699Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-29T15:01:10.699Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-29T15:01:10.699Z] Processing JUnit reports with runbld...
[2020-09-29T15:01:10.963Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-29T15:01:10.963Z] runbld>>> DURATION: 31ms
[2020-09-29T15:01:10.963Z] runbld>>> STDOUT: 40 bytes
[2020-09-29T15:01:10.963Z] runbld>>> STDERR: 49 bytes
[2020-09-29T15:01:10.963Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-29T15:01:10.963Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21380
[2020-09-29T15:01:12.356Z] runbld>>> Storing build metadata: 
[2020-09-29T15:01:12.356Z] runbld>>> Adding test report.
[2020-09-29T15:01:12.356Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats
[2020-09-29T15:01:12.931Z] runbld>>> Found 142 test output files
[2020-09-29T15:01:14.889Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-goIntegTest-1601390964391/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-29T15:01:14.889Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/metricbeat-goIntegTest-1601390964391/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-29T15:01:14.889Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-build-1601391330248/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-09-29T15:01:14.889Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-build-1601391330248/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-09-29T15:01:14.889Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-build-1601391330248/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-09-29T15:01:14.889Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-build-1601391330248/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-09-29T15:01:14.889Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21380/src/github.com/elastic/beats/x-pack-metricbeat-build-1601391330248/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-09-29T15:01:16.285Z] runbld>>> Test output logs contained: Errors: 0 Failures: 3 Tests: 22160 Skipped: 1564
[2020-09-29T15:01:16.547Z] runbld>>> Storing result
[2020-09-29T15:01:16.547Z] runbld>>> FAILURES: 3
[2020-09-29T15:01:17.493Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-29T15:01:17.494Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200929150108-DB8BF9D9
[2020-09-29T15:01:17.494Z] runbld>>> Email notification disabled by environment variable.
[2020-09-29T15:01:17.494Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-29T15:01:23.903Z] Running on worker-395930 in /var/lib/jenkins/workspace/Beats_beats_PR-21380
[2020-09-29T15:01:23.977Z] [INFO] getVaultSecret: Getting secrets
[2020-09-29T15:01:24.074Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-29T15:01:26.339Z] + chmod 755 generate-build-data.sh
[2020-09-29T15:01:26.339Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21380/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21380/runs/1 FAILURE 4922263
[2020-09-29T15:01:26.339Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21380/runs/1/steps/?limit=10000 -o steps-info.json
[2020-09-29T15:01:31.673Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21380/runs/1/tests/?status=FAILED -o tests-errors.json

@faec faec merged commit d8d35a1 into elastic:master Sep 29, 2020
@faec faec deleted the s3waitgroup branch September 29, 2020 17:00
faec added a commit to faec/beats that referenced this pull request Sep 29, 2020
@faec faec added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 29, 2020
@faec faec added the v7.9.3 label Sep 29, 2020
faec added a commit to faec/beats that referenced this pull request Sep 29, 2020
faec added a commit that referenced this pull request Sep 30, 2020
faec added a commit that referenced this pull request Sep 30, 2020
v1v added a commit to v1v/beats that referenced this pull request Sep 30, 2020
…ci-build-label-support

* upstream/master:
  [JJBB] Set shallow cloning to 10 (elastic#21409)
  docs: add link to release notes for 7.9.2 (elastic#21405) (elastic#21419)
  docs: Prepare Changelog for 7.9.2 (elastic#21229) (elastic#21403)
  fix: mark flaky tests (elastic#21300)
  fix: use a fixed version of setuptools (elastic#21393)
  Move Kubernetes events metricset to its own block in reference config (elastic#21407)
  [libbeat] Enable WriteAheadLimit in the disk queue (elastic#21391)
  docs: fix apt/yum formatting (elastic#21362)
  Fix shutdown tracking in s3 input (elastic#21380)
  [libbeat] Fix position writing in the disk queue
  Add UBI 8 image to the dependencies report (elastic#21374)
  Fix debug message to show actual SQS message ID (elastic#20614)
  [Elastic Agent] Rename *ConfigChange to PolicyChange (elastic#20779)
  [Elastic Agent] Add install/uninstall sub-command (elastic#21206)
  [Filebeat][httpjson] Make httpjson use cursor input when using date cursor (elastic#20751)
  feat: prepare release pipelines (elastic#21238)
  Add IP validation to Security module (elastic#21325)
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Filebeat Filebeat Team:Integrations Label for the Integrations team v7.9.3 v7.10.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants