Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ingest Manager] New Agent structure #20307

Closed
wants to merge 16 commits into from

Conversation

michalpristas
Copy link
Contributor

@michalpristas michalpristas commented Jul 29, 2020

What does this PR do?

This PR does a multiple things.

First thing, it alignes structure of resulting package with the proposal defined in issue #20048

For this to work there were a few changes needed.
First one is having two executables one is at root another at /data/v7.x.x-abc/ directory.

The one in root serves just as proxy and executes binary in versioned dir and pass all variables there.
This is achieved by pre-run check, check reads the .build_hash.txt file if it's there it redirects control to the binary in versioned directory. if it's not there it continues as usual.

Draft, because more tests needs to be done

Test on:

  • darwin standalone
  • linux standalone
  • windows standalone
  • darwin fleet
  • linux fleet
  • windows fleet

Why is it important?

Step 1 for upgradable agent

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@michalpristas michalpristas self-assigned this Jul 29, 2020
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 29, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 29, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #20307 updated]

  • Start Time: 2020-07-30T13:00:22.276+0000

  • Duration: 84 min 59 sec

Test stats 🧪

Test Results
Failed 13
Passed 14944
Skipped 1901
Total 16858

Test errors

Expand to view the tests failures

  • Name: Build and Test / Auditbeat x-pack / test_connected_udp_ipv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.116
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_connected_udp_ipv6 – test_system_socket.Test

    • Age: 1
    • Duration: 60.113
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_enrichment – test_system_socket.Test

    • Age: 1
    • Duration: 60.121
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_long_request – test_system_socket.Test

    • Age: 1
    • Duration: 60.13
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_udp_ipv6 – test_system_socket.Test

    • Age: 1
    • Duration: 60.104
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_dns_unidirectional_udp – test_system_socket.Test

    • Age: 1
    • Duration: 60.105
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_multi_udp_upv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.107
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_no_dns_enrichment – test_system_socket.Test

    • Age: 1
    • Duration: 60.112
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_tcp_ipv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.108
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_tcp_ipv6_disabled – test_system_socket.Test

    • Age: 1
    • Duration: 60.111
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_udp_ipv4 – test_system_socket.Test

    • Age: 1
    • Duration: 60.113
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_udp_ipv6 – test_system_socket.Test

    • Age: 1
    • Duration: 60.109
    • Error Details: Expected exit code to be 0, but it was 1
  • Name: Build and Test / Auditbeat x-pack / test_udp_ipv6_disabled – test_system_socket.Test

    • Age: 1
    • Duration: 60.123
    • Error Details: Expected exit code to be 0, but it was 1

Steps errors

Expand to view the steps failures

  • Name: Mage build test

    • Description: mage build test

    • Duration: 9 min 34 sec

    • Start Time: 2020-07-30T13:25:08.898+0000

    • log

  • Name: Mage update build test

    • Description: mage update build test

    • Duration: 19 min 22 sec

    • Start Time: 2020-07-30T13:25:09.242+0000

    • log

  • Name: Mage build test

    • Description: mage build test

    • Duration: 7 min 9 sec

    • Start Time: 2020-07-30T13:25:00.809+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-07-30T14:23:27.154Z]  OS/Arch:           darwin/amd64
[2020-07-30T14:23:27.154Z]  Experimental:      false
[2020-07-30T14:23:27.154Z] Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
[2020-07-30T14:23:27.154Z] It requires Docker daemon to be installed and running
[2020-07-30T14:23:50.384Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats
[2020-07-30T14:23:50.699Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-30T14:23:50.713Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Lint
[2020-07-30T14:23:50.809Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-30T14:23:50.901Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-30T14:23:50.985Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-30T14:23:51.075Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-30T14:23:51.177Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-30T14:23:51.267Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-30T14:23:51.371Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-30T14:23:51.458Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-30T14:23:51.553Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-30T14:23:51.658Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-30T14:23:51.757Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-30T14:23:51.858Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Packetbeat-Linux
[2020-07-30T14:23:51.950Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-30T14:23:52.066Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-30T14:23:52.158Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-30T14:23:52.239Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-30T14:23:52.325Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-30T14:23:52.414Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Libbeat-oss
[2020-07-30T14:23:52.505Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-30T14:23:52.613Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-30T14:23:52.721Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-30T14:23:52.824Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Filebeat-oss
[2020-07-30T14:23:52.921Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-30T14:23:53.012Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-30T14:23:53.105Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-30T14:23:53.196Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-30T14:23:53.292Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-30T14:23:53.395Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-30T14:23:53.491Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-07-30T14:23:53.583Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-30T14:23:53.674Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-30T14:23:53.782Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-30T14:23:53.893Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-30T14:23:53.999Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-30T14:23:54.092Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-07-30T14:23:54.196Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Heartbeat-Windows
[2020-07-30T14:23:54.383Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-30T14:23:54.474Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Packetbeat-Mac-OS-X
[2020-07-30T14:23:54.574Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-30T14:23:54.672Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests
[2020-07-30T14:23:54.766Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-OSS-Python-Integration-tests
[2020-07-30T14:23:54.870Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Packetbeat-Windows
[2020-07-30T14:23:54.960Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-07-30T14:23:55.062Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-30T14:23:55.176Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-07-30T14:23:55.572Z] + cat
[2020-07-30T14:23:55.572Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-30T14:23:55.572Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-30T14:24:02.179Z] runbld>>> runbld started
[2020-07-30T14:24:02.179Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-30T14:24:03.567Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-20307' in order of occurrence in the config (last value wins).
[2020-07-30T14:24:04.957Z] runbld>>> Debug logging enabled.
[2020-07-30T14:24:04.957Z] runbld>>> Storing result
[2020-07-30T14:24:05.219Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-30T14:24:05.219Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200730142404-27B9E46E
[2020-07-30T14:24:05.219Z] runbld>>> Adding system facts.
[2020-07-30T14:24:06.167Z] runbld>>> Adding vcs info for the latest commit:  be29caccff464a9eea6532b0b76599c86e821ac9
[2020-07-30T14:24:06.167Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-30T14:24:06.167Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-30T14:24:06.167Z] Processing JUnit reports with runbld...
[2020-07-30T14:24:06.167Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-30T14:24:06.740Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-30T14:24:06.740Z] runbld>>> DURATION: 33ms
[2020-07-30T14:24:06.740Z] runbld>>> STDOUT: 40 bytes
[2020-07-30T14:24:06.740Z] runbld>>> STDERR: 49 bytes
[2020-07-30T14:24:06.740Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-30T14:24:06.740Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats
[2020-07-30T14:24:07.689Z] runbld>>> Storing build metadata: 
[2020-07-30T14:24:07.689Z] runbld>>> Adding test report.
[2020-07-30T14:24:07.689Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats
[2020-07-30T14:24:08.635Z] runbld>>> Found 133 test output files
[2020-07-30T14:24:09.208Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-30T14:24:09.208Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-30T14:24:09.208Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-30T14:24:09.208Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-30T14:24:09.208Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-30T14:24:09.209Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-30T14:24:09.471Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-20307/src/github.com/elastic/beats/Metricbeat-OSS-Go-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-30T14:24:11.400Z] runbld>>> Test output logs contained: Errors: 0 Failures: 13 Tests: 16708 Skipped: 1641
[2020-07-30T14:24:11.400Z] runbld>>> Storing result
[2020-07-30T14:24:11.400Z] runbld>>> FAILURES: 13
[2020-07-30T14:24:13.955Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-30T14:24:13.955Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200730142404-27B9E46E
[2020-07-30T14:24:14.216Z] runbld>>> Email notification disabled by environment variable.
[2020-07-30T14:24:14.216Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-30T14:24:20.385Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20307
[2020-07-30T14:24:20.654Z] [INFO] getVaultSecret: Getting secrets
[2020-07-30T14:24:20.722Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-30T14:24:21.653Z] + chmod 755 generate-build-data.sh
[2020-07-30T14:24:21.653Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20307/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20307/runs/10 FAILURE 5039116
[2020-07-30T14:24:21.653Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20307/runs/10/steps/?limit=10000 -o steps-info.json
[2020-07-30T14:24:24.137Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20307/runs/10/tests/?status=FAILED -o tests-errors.json
[2020-07-30T14:24:25.481Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20307/runs/10/log/ -o pipeline-log.txt

"--path.config", paths.Config(),
}
rm := reexec.Manager(rexLogger, reexecPath)
rm.ReExec(argsOverrides...)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is really going to work on Windows. being that Windows doesn't support proper re-exec.

Running from Powershell will show this weird behavior, where calling elastic-agent run -e will exit back to the shell but the logs keep printing to the terminal.

Running as a service I looks like it would just be an infinite loop. It would keep restarting the service, and the service would then just keep re-exec, because this same binary would be pointed at by the service.

This is a good approach on Unix based systems, but on Windows I don't believe it will work.

What about the symlink idea? That seems like it would work on all OS types, including Windows: https://golang.org/pkg/syscall/?GOOS=windows#CreateSymbolicLink

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i will need to test it on windows, symlink is ok but needs to be tested as well. we need to create symlink at the build time or provide some activation script, start script or something similar.
i havent tried but i think that creating a symlink during release would not result in something working when you unpack it locally.

@michalpristas
Copy link
Contributor Author

michalpristas commented Jul 30, 2020

image

Succesfully running

  • vagrant == linux64
  • vagrant10 = windows 10
  • Michals-Macbook... = darwin
  • windows-as-a-service == windows managed service

@michalpristas michalpristas force-pushed the agent-new-structure branch 3 times, most recently from 03f6559 to a54ed0a Compare July 30, 2020 12:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Ingest Management:beta2 Group issues for ingest management beta2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants