Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Fix indentation problems in processor examples #18115

Merged
merged 5 commits into from
May 4, 2020

Conversation

dedemorton
Copy link
Contributor

@dedemorton dedemorton commented Apr 30, 2020

Closes #16696

Makes docs and reference.yml files consistent with how processors are defined in the beats.yml files. This change makes it easier for users to copy/paste and avoids the situation where they might create invalid yaml.

Also fixes a few other minor issues I found while going through the processor docs.

TODO:

  • There are a couple of places in the config files where we don't use spaces. They need to be fixed in a separate PR. (Added in Prettify processors section in docker yml files #18218)
  • The following processors have not docs: add_formatted_index, checks, timeseries, and util. Need to follow up. Confirmed with Noemi that they are internal only.

@dedemorton dedemorton added docs review needs_backport PR is waiting to be backported to other branches. Team:Integrations Label for the Integrations team labels Apr 30, 2020
@dedemorton dedemorton requested review from a team as code owners April 30, 2020 03:36
@dedemorton dedemorton self-assigned this Apr 30, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 30, 2020
@dedemorton dedemorton removed the needs_team Indicates that the issue/PR needs a Team:* label label Apr 30, 2020
@elasticmachine
Copy link
Collaborator

💔 Build Failed

Pipeline View Test View Changes Artifacts preview stats

Expand to view the summary

Build stats

Test stats 🧪

Test Results
Failed 2
Passed 6809
Skipped 1049
Total 7860

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat x-pack / runTest – nose.failure.Failure

    • Status: FAILED
    • Age: 1
    • Duration: 0
    • Error Details: Cannot subscript an existing Union. Use Union[u, t] instead.
  • Name: Build and Test / Metricbeat x-pack / test_server – test_statsd.Test

    • Status: FAILED
    • Age: 1
    • Duration: 5.611
    • Error Details: 'ascii' codec can't decode byte 0xe2 in position 702041: ordinal not in range(128)

Steps errors

Expand to view the steps failures

  • Name: Mage update build test

    • Description: mage update build test

    • Result: FAILURE

    • Duration: 6 min 45 sec<

    • Start Time: 2020-04-30T04:03:47.097+0000

  • Name: Report to Codecov

    • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

    • Result: FAILURE

    • Duration: 2 min 22 sec<

    • Start Time: 2020-04-30T04:45:58.963+0000

  • Name: Mage build test

    • Description: mage build test

    • Result: FAILURE

    • Duration: 9 min 57 sec<

    • Start Time: 2020-04-30T04:07:52.341+0000

  • Name: Make -C generator/_templates/metricbeat test

    • Description: make -C generator/_templates/metricbeat test

    • Result: FAILURE

    • Duration: 1 min 0 sec<

    • Start Time: 2020-04-30T04:09:01.534+0000

Log output

Expand to view the last 100 lines of log output

[2020-04-30T04:49:07.676Z] + script/fix_permissions.sh /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18115
[2020-04-30T04:49:07.676Z] Client: Docker Engine - Community
[2020-04-30T04:49:07.676Z]  Version:           19.03.8
[2020-04-30T04:49:07.676Z]  API version:       1.40
[2020-04-30T04:49:07.676Z]  Go version:        go1.12.17
[2020-04-30T04:49:07.676Z]  Git commit:        afacb8b7f0
[2020-04-30T04:49:07.677Z]  Built:             Wed Mar 11 01:25:58 2020
[2020-04-30T04:49:07.677Z]  OS/Arch:           linux/amd64
[2020-04-30T04:49:07.677Z]  Experimental:      false
[2020-04-30T04:49:07.677Z] 
[2020-04-30T04:49:07.677Z] Server: Docker Engine - Community
[2020-04-30T04:49:07.677Z]  Engine:
[2020-04-30T04:49:07.677Z]   Version:          19.03.8
[2020-04-30T04:49:07.677Z]   API version:      1.40 (minimum version 1.12)
[2020-04-30T04:49:07.677Z]   Go version:       go1.12.17
[2020-04-30T04:49:07.677Z]   Git commit:       afacb8b7f0
[2020-04-30T04:49:07.677Z]   Built:            Wed Mar 11 01:24:30 2020
[2020-04-30T04:49:07.677Z]   OS/Arch:          linux/amd64
[2020-04-30T04:49:07.677Z]   Experimental:     false
[2020-04-30T04:49:07.677Z]  containerd:
[2020-04-30T04:49:07.677Z]   Version:          1.2.13
[2020-04-30T04:49:07.677Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2020-04-30T04:49:07.677Z]  runc:
[2020-04-30T04:49:07.677Z]   Version:          1.0.0-rc10
[2020-04-30T04:49:07.677Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-04-30T04:49:07.677Z]  docker-init:
[2020-04-30T04:49:07.677Z]   Version:          0.18.0
[2020-04-30T04:49:07.677Z]   GitCommit:        fec3683
[2020-04-30T04:49:14.447Z] Recording test results
[2020-04-30T04:49:15.186Z] None of the test reports contained any result
[2020-04-30T04:49:15.206Z] Archiving artifacts
[2020-04-30T04:49:15.467Z] java.lang.InterruptedException: no matches found within 10000
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2826)
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2686)
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3073)
[2020-04-30T04:49:15.467Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1604-1588219314770873603.c.elastic-ci-prod.internal/10.224.0.85:35720
[2020-04-30T04:49:15.467Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
[2020-04-30T04:49:15.467Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2020-04-30T04:49:15.467Z] 		at hudson.remoting.Channel.call(Channel.java:998)
[2020-04-30T04:49:15.467Z] 		at hudson.FilePath.act(FilePath.java:1069)
[2020-04-30T04:49:15.467Z] 		at hudson.FilePath.act(FilePath.java:1058)
[2020-04-30T04:49:15.467Z] 		at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-04-30T04:49:15.467Z] 		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-04-30T04:49:15.467Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-04-30T04:49:15.467Z] 		at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-04-30T04:49:15.467Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-04-30T04:49:15.467Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-04-30T04:49:15.467Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-30T04:49:15.467Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-30T04:49:15.467Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-30T04:49:15.467Z] Caused: hudson.FilePath$TunneledInterruptedException
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3075)
[2020-04-30T04:49:15.467Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-04-30T04:49:15.467Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-04-30T04:49:15.467Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-04-30T04:49:15.467Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-04-30T04:49:15.467Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-30T04:49:15.467Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-30T04:49:15.467Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-30T04:49:15.467Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-04-30T04:49:15.467Z] Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath.act(FilePath.java:1071)
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath.act(FilePath.java:1058)
[2020-04-30T04:49:15.467Z] 	at hudson.FilePath.validateAntFileMask(FilePath.java:2684)
[2020-04-30T04:49:15.467Z] 	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:265)
[2020-04-30T04:49:15.467Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
[2020-04-30T04:49:15.467Z] 	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
[2020-04-30T04:49:15.467Z] 	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-04-30T04:49:15.467Z] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-04-30T04:49:15.467Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-04-30T04:49:15.467Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-04-30T04:49:15.467Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-04-30T04:49:15.467Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-04-30T04:49:15.467Z] No artifacts found that match the file pattern "**/build/TEST*.out". Configuration error?
[2020-04-30T04:49:15.918Z] + curl -sSLo codecov https://codecov.io/bash
[2020-04-30T04:49:16.180Z] + FILE=auditbeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f auditbeat/build/coverage/full.cov ]
[2020-04-30T04:49:16.180Z] + FILE=filebeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f filebeat/build/coverage/full.cov ]
[2020-04-30T04:49:16.180Z] + FILE=heartbeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f heartbeat/build/coverage/full.cov ]
[2020-04-30T04:49:16.180Z] + FILE=libbeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f libbeat/build/coverage/full.cov ]
[2020-04-30T04:49:16.180Z] + FILE=metricbeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f metricbeat/build/coverage/full.cov ]
[2020-04-30T04:49:16.180Z] + FILE=packetbeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-04-30T04:49:16.180Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-04-30T04:49:16.180Z] + FILE=journalbeat/build/coverage/full.cov
[2020-04-30T04:49:16.180Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-04-30T04:49:19.244Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-18115
[2020-04-30T04:49:19.533Z] [INFO] getVaultSecret: Getting secrets
[2020-04-30T04:49:19.602Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-04-30T04:49:20.484Z] + chmod 755 generate-build-data.sh
[2020-04-30T04:49:20.484Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18115/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18115/runs/1 FAILURE 4339882
[2020-04-30T04:49:21.035Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18115/runs/1/steps/?limit=10000 -o steps-info.json
[2020-04-30T04:49:21.586Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18115/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-04-30T04:49:21.586Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-18115/runs/1/log/ -o pipeline-log.txt

@kvch
Copy link
Contributor

kvch commented Apr 30, 2020

Are you sure these configuration snippets in the *beat.reference.yml files are incorrect? I've been using this indentation for years and it never caused me any issues. Also, I've just tested locally copy_fields and it worked for me.

@dedemorton
Copy link
Contributor Author

@kvch They original snippets do work, but they aren't "tidy" and and can't be pasted directly into an existing processors section without causing an error. I know this seems trivial, but invalid yaml files seem to trip people up quite a bit, and things like whitespace aren't always obvious.

So our existing config files look like this:

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

If you paste one of our processor snippets into this config without adjusting the indentation, you get: Exiting: error loading config file: yaml: line 148: did not find expected key. I think it's easy to go to that line and figure out the problem, but for someone who is learning, maybe it's not.

So in #16696, I suggested that we fix the regular config files (filebeat.yml etc) rather that all the docs, but @exekias pointed out that we use spaces everywhere else so it would be awkward to not use them under processors.

I wasn't expecting there to be so many changes but since all the reference yaml files are generated from the same source, it wasn't as much work as it seems.

@dedemorton dedemorton merged commit 1e837c5 into elastic:master May 4, 2020
@dedemorton dedemorton deleted the issue#16696 branch May 4, 2020 17:56
dedemorton added a commit to dedemorton/beats that referenced this pull request May 4, 2020
dedemorton added a commit to dedemorton/beats that referenced this pull request May 4, 2020
@dedemorton dedemorton removed the needs_backport PR is waiting to be backported to other branches. label May 5, 2020
@dedemorton dedemorton added the needs_backport PR is waiting to be backported to other branches. label May 5, 2020
@dedemorton dedemorton removed the needs_backport PR is waiting to be backported to other branches. label May 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs review Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Change spacing in processor examples
3 participants