Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Elastic Agent] Allow embedding of certificate #21179

Merged
merged 9 commits into from
Sep 28, 2020

Conversation

ph
Copy link
Contributor

@ph ph commented Sep 18, 2020

This PR allow to embed Certificate authorities directly in the yaml
configuration. This is useful in the context of fleet where distributing
file to the remote host is not possible. The format of the string need
to be in PEM.

This changes how we handle TLS Options internally and should enable embedding of CAs, Certificate or Private key for everything that uses the common TLSConfig.

Example: Certificate Authorities

enabled: true
verification_mode: null
certificate: null
key: null
key_passphrase: null
certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
cipher_suites: null
curve_types: null
supported_protocols: null

Example: Certificate and Key

enabled: true
verification_mode: null
certificate: |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDXHufGPycpCOfI
    sjl6cRn8NP4DLxdIVEAHFK0jMRDup32UQOPW+DleEsFpgN9/ebi9ngdjQfMvKnUP
    Zrl1HTwVhOJfazGeoJn7vdDeQebhJfeDXHwX2DiotXyUPYu1ioU45UZDAoAZFj5F
    KJLwWRUbfEbRe8yO+wUhKKxxkApPbfw+wUtBicn1RIX7W1nBRABt1UXKDIRe5FM2
    MKfqhEqK4hUWC3g1r+vGTrxu3qFpzz7L2UrRFRIpo7yuTUhEhEGvcVsiTppTil4Z
    HcprXFHf5158elEwhYJ5IM0nU1leNQiOgemifbLwkyNkLqCKth8V/4sezr1tYblZ
    nMh1cclBAgMBAAECggEBAKdP5jyOicqknoG9/G564RcDsDyRt64NuO7I6hBg7SZx
    Jn7UKWDdFuFP/RYtoabn6QOxkVVlydp5Typ3Xu7zmfOyss479Q/HIXxmmbkD0Kp0
    eRm2KN3y0b6FySsS40KDRjKGQCuGGlNotW3crMw6vOvvsLTlcKgUHF054UVCHoK/
    Piz7igkDU7NjvJeha53vXL4hIjb10UtJNaGPxIyFLYRZdRPyyBJX7Yt3w8dgz8WM
    epOPu0dq3bUrY3WQXcxKZo6sQjE1h7kdl4TNji5jaFlvD01Y8LnyG0oThOzf0tve
    Gaw+kuy17gTGZGMIfGVcdeb+SlioXMAAfOps+mNIwTECgYEA/gTO8W0hgYpOQJzn
    BpWkic3LAoBXWNpvsQkkC3uba8Fcps7iiEzotXGfwYcb5Ewf5O3Lrz1EwLj7GTW8
    VNhB3gb7bGOvuwI/6vYk2/dwo84bwW9qRWP5hqPhNZ2AWl8kxmZgHns6WTTxpkRU
    zrfZ5eUrBDWjRU2R8uppgRImsxMCgYEA2MxuL/C/Ko0d7XsSX1kM4JHJiGpQDvb5
    GUrlKjP/qVyUysNF92B9xAZZHxxfPWpdfGGBynhw7X6s+YeIoxTzFPZVV9hlkpAA
    5igma0n8ZpZEqzttjVdpOQZK8o/Oni/Q2S10WGftQOOGw5Is8+LY30XnLvHBJhO7
    TKMurJ4KCNsCgYAe5TDSVmaj3dGEtFC5EUxQ4nHVnQyCpxa8npL+vor5wSvmsfUF
    hO0s3GQE4sz2qHecnXuPldEd66HGwC1m2GKygYDk/v7prO1fQ47aHi9aDQB9N3Li
    e7Vmtdn3bm+lDjtn0h3Qt0YygWj+wwLZnazn9EaWHXv9OuEMfYxVgYKpdwKBgEze
    Zy8+WDm5IWRjn8cI5wT1DBT/RPWZYgcyxABrwXmGZwdhp3wnzU/kxFLAl5BKF22T
    kRZ+D+RVZvVutebE9c937BiilJkb0AXLNJwT9pdVLnHcN2LHHHronUhV7vetkop+
    kGMMLlY0lkLfoGq1AxpfSbIea9KZam6o6VKxEnPDAoGAFDCJm+ZtsJK9nE5GEMav
    NHy+PwkYsHhbrPl4dgStTNXLenJLIJ+Ke0Pcld4ZPfYdSyu/Tv4rNswZBNpNsW9K
    0NwJlyMBfayoPNcJKXrH/csJY7hbKviAHr1eYy9/8OL0dHf85FV+9uY5YndLcsDc
    nygO9KTJuUiBrLr0AHEnqko=
    -----END PRIVATE KEY-----
key_passphrase: null
certificate_authorities:
cipher_suites: null
curve_types: null
supported_protocols: null

Related to: #19504

What does this PR do?

Why is it important?

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 18, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ingest-management (Team:Ingest Management)

@ph
Copy link
Contributor Author

ph commented Sep 18, 2020

To the reviewers if we agree on the way I will add documentation to this, also the way is implemented its compatible with libbeat's keystore, so if user want to add certificate or keys in the keystore it should work because keystore values are replaced before any TLS struct is created.

@ph ph requested review from urso, blakerouse and ruflin September 18, 2020 19:30
@ph ph self-assigned this Sep 18, 2020
@ph
Copy link
Contributor Author

ph commented Sep 18, 2020

I could have did it without using a reader. But I've figured it could be a bit more flexible for other parts of the code.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Sep 18, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #21179 updated]

  • Start Time: 2020-09-24T18:12:25.139+0000

  • Duration: 63 min 32 sec

Test stats 🧪

Test Results
Failed 1
Passed 15615
Skipped 1269
Total 16885

Test errors

Expand to view the tests failures

  • Name: Build&Test / libbeat-build / TestClientPublishEventKerberosAware – elasticsearch

    • Age: 8
    • Duration: 2.35
    • Error Details: Failed

Steps errors

Expand to view the steps failures

  • Name: mage build test

    • Description: mage build test

    • Duration: 17 min 24 sec

    • Start Time: 2020-09-24T18:39:01.726+0000

    • log

  • Name: Notifies GitHub of the status of a Pull Request

    • Description: script returned exit code 1

    • Duration: 0 min 2 sec

    • Start Time: 2020-09-24T18:56:40.603+0000

    • log

  • Name: Terraform Apply on x-pack/metricbeat/module/aws

    • Description:

    • Duration: 0 min 2 sec

    • Start Time: 2020-09-24T18:39:13.191+0000

    • log

  • Name: Terraform Apply on x-pack/metricbeat/module/aws

    • Description:

    • Duration: 0 min 2 sec

    • Start Time: 2020-09-24T18:39:21.411+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/packetbeat-windows-windows-2019/packetbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/packetbeat-windows-windows-2019/packetbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/metricbeat-windows-windows-2019/metricbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/metricbeat-windows-windows-2019/metricbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-python-integration.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-go-integration.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-auditbeat-build/x-pack/auditbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/packetbeat-build/packetbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/packetbeat-build/packetbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-python-integration.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-go-integration.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-libbeat-build/x-pack/libbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019/x-pack/auditbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.261Z] ./src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019/x-pack/auditbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019/x-pack/metricbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019/x-pack/metricbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/filebeat-windows-windows-2019/filebeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/filebeat-windows-windows-2019/filebeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/libbeat-build/libbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/libbeat-build/libbeat/build/TEST-go-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/libbeat-build/libbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-python-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-go-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-filebeat-build/x-pack/filebeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/heartbeat-windows-windows-2019/heartbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/heartbeat-windows-windows-2019/heartbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-python-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-go-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/heartbeat-build/heartbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/journalbeat-unitTest/journalbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/journalbeat-unitTest/journalbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-python-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/auditbeat-build/auditbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-dockerlogbeat-build/x-pack/dockerlogbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-python-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-go-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/filebeat-build/filebeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019/x-pack/filebeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/metricbeat-unitTest/metricbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/metricbeat-unitTest/metricbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019/x-pack/functionbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/x-pack-functionbeat-build/x-pack/functionbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/metricbeat-pythonIntegTest/metricbeat/build/TEST-python-integration.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-python-unit.xml
[2020-09-24T19:14:34.262Z] ./src/github.com/elastic/beats/auditbeat-windows-windows-2019/auditbeat/build/TEST-go-unit.xml
[2020-09-24T19:14:34.262Z] + cat
[2020-09-24T19:14:34.262Z] + /usr/local/bin/runbld ./runbld-script --job-name elastic+beats+pull-request
[2020-09-24T19:14:34.262Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-09-24T19:14:40.865Z] runbld>>> runbld started
[2020-09-24T19:14:40.865Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-09-24T19:14:43.418Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-09-24T19:14:43.418Z] runbld>>> Matches in the system config:
[2020-09-24T19:14:43.418Z] runbld>>> - Matched ^elastic\+beats
[2020-09-24T19:14:43.418Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-09-24T19:14:44.364Z] runbld>>> Debug logging enabled.
[2020-09-24T19:14:44.364Z] runbld>>> Storing result
[2020-09-24T19:14:44.630Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-09-24T19:14:44.630Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200924191444-338BAE0F
[2020-09-24T19:14:44.630Z] runbld>>> Adding system facts.
[2020-09-24T19:14:46.022Z] runbld>>> Adding vcs info for the latest commit:  bf266c8104f78936fc956971e7b18d1fcc476f20
[2020-09-24T19:14:46.022Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-09-24T19:14:46.022Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-09-24T19:14:46.022Z] + echo 'Processing JUnit reports with runbld...'
[2020-09-24T19:14:46.022Z] Processing JUnit reports with runbld...
[2020-09-24T19:14:46.284Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-09-24T19:14:46.284Z] runbld>>> DURATION: 46ms
[2020-09-24T19:14:46.284Z] runbld>>> STDOUT: 40 bytes
[2020-09-24T19:14:46.284Z] runbld>>> STDERR: 49 bytes
[2020-09-24T19:14:46.284Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-09-24T19:14:46.284Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-21179
[2020-09-24T19:14:47.241Z] runbld>>> Storing build metadata: 
[2020-09-24T19:14:47.241Z] runbld>>> Adding test report.
[2020-09-24T19:14:47.241Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-21179/src/github.com/elastic/beats
[2020-09-24T19:14:48.194Z] runbld>>> Found 99 test output files
[2020-09-24T19:14:48.456Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21179/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-graphite.xml
[2020-09-24T19:14:48.456Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-21179/src/github.com/elastic/beats/metricbeat-goIntegTest/metricbeat/build/TEST-go-integration-windows.xml
[2020-09-24T19:14:51.008Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 16885 Skipped: 1050
[2020-09-24T19:14:51.008Z] runbld>>> Storing result
[2020-09-24T19:14:51.008Z] runbld>>> FAILURES: 1
[2020-09-24T19:14:51.270Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-09-24T19:14:51.270Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20200924191444-338BAE0F
[2020-09-24T19:14:51.531Z] runbld>>> Email notification disabled by environment variable.
[2020-09-24T19:14:51.531Z] runbld>>> Slack notification disabled by environment variable.
[2020-09-24T19:14:56.995Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-21179
[2020-09-24T19:14:57.072Z] [INFO] getVaultSecret: Getting secrets
[2020-09-24T19:14:57.151Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-09-24T19:14:57.759Z] + chmod 755 generate-build-data.sh
[2020-09-24T19:14:57.759Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21179/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21179/runs/10 FAILURE 3752352
[2020-09-24T19:14:57.759Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21179/runs/10/steps/?limit=10000 -o steps-info.json
[2020-09-24T19:15:02.954Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-21179/runs/10/tests/?status=FAILED -o tests-errors.json

@urso
Copy link

urso commented Sep 18, 2020

+100 on the change.

My only concern is logging and error message of the keys. These are somewhat confidential data and if possible should not leak into logs by accident. Maybe it helps to also only print a substring of last N characters if we really need to log.

Does this format work with the keystore?

@ruflin
Copy link
Contributor

ruflin commented Sep 21, 2020

++ on moving forward with this.

@ph
Copy link
Contributor Author

ph commented Sep 21, 2020

@urso Good point for the logging, I was also uneasy with it. will try to make a change.

Concerning the keystore, I think it should work but maybe the CLI subcommand will require a change to correctly keep the format of the certificate.

@ruflin @urso I'll do the change today, I want to move quickly on this so we can have a fix in the UI to support it at least for the ES outputs this will reduce a lot of our discuss posts.

@ph
Copy link
Contributor Author

ph commented Sep 21, 2020

@urso Concerning the keystore, the following work:

 cat /tmp/certificate.crt | ./filebeat keystore add mycert --force --stdin
Successfully updated the keystore
winterfell~/go/src/github.com/elastic/beats/filebeat(elastic-agent/embed-ca-in-yaml|✚1) % ./filebeat keystore list
mycert

@ph
Copy link
Contributor Author

ph commented Sep 21, 2020

Minus #21179 (comment)

I think its ready for a round of review, documentation was added too.

@ph ph force-pushed the elastic-agent/embed-ca-in-yaml branch from a9f284f to f49fdd1 Compare September 21, 2020 14:56
@ph
Copy link
Contributor Author

ph commented Sep 21, 2020

@ruflin @blakerouse lets get that reviewed so we can unblock the yaml box in the elasticsearch output.

@ph
Copy link
Contributor Author

ph commented Sep 21, 2020

jenkins test this please

@ph
Copy link
Contributor Author

ph commented Sep 21, 2020

I am confused by the errors, doesn't seems related. But I will retrigger it.

@ph ph added the review label Sep 21, 2020
Copy link
Contributor

@blakerouse blakerouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, with the '-' change.

@ph ph closed this Sep 21, 2020
@ph ph reopened this Sep 21, 2020
Copy link
Contributor

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The part I'm stumbling over is that a single config can either contain the path or the certificate itself. From a "magic" perspective it is very nice that it just works but it is kind of surprising that a single config option can be used in 2 very different ways. I'm good moving forward with it.

libbeat/common/transport/tlscommon/tls.go Outdated Show resolved Hide resolved
Copy link
Contributor

@michalpristas michalpristas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 this looks really goood

@ph
Copy link
Contributor Author

ph commented Sep 22, 2020

The part I'm stumbling over is that a single config can either contain the path or the certificate itself. From a "magic" perspective it is very nice that it just works but it is kind of surprising that a single config option can be used in 2 very different ways. I'm good moving forward with it.

Agree, but the PEM format is well defined, so I doub't it will be an issue. I have added "mixed" test just to make sure we don't break it.

@ph
Copy link
Contributor Author

ph commented Sep 22, 2020

taking a look at "TestClientPublishEventKerberosAware" failure

ph added 9 commits September 24, 2020 14:10
This PR allow to embed Certificate authorities directly in the yaml
configuration. This is useful in the context of fleet where distributing
file to the remote host is not possible. The format of the string need
to be in PEM.

Example: Certificate Authorities
```yaml
enabled: true
verification_mode: null
certificate: null
key: null
key_passphrase: null
certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
cipher_suites: null
curve_types: null
supported_protocols: null
```

```Certificate and Key
enabled: true
verification_mode: null
certificate: |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDXHufGPycpCOfI
    sjl6cRn8NP4DLxdIVEAHFK0jMRDup32UQOPW+DleEsFpgN9/ebi9ngdjQfMvKnUP
    Zrl1HTwVhOJfazGeoJn7vdDeQebhJfeDXHwX2DiotXyUPYu1ioU45UZDAoAZFj5F
    KJLwWRUbfEbRe8yO+wUhKKxxkApPbfw+wUtBicn1RIX7W1nBRABt1UXKDIRe5FM2
    MKfqhEqK4hUWC3g1r+vGTrxu3qFpzz7L2UrRFRIpo7yuTUhEhEGvcVsiTppTil4Z
    HcprXFHf5158elEwhYJ5IM0nU1leNQiOgemifbLwkyNkLqCKth8V/4sezr1tYblZ
    nMh1cclBAgMBAAECggEBAKdP5jyOicqknoG9/G564RcDsDyRt64NuO7I6hBg7SZx
    Jn7UKWDdFuFP/RYtoabn6QOxkVVlydp5Typ3Xu7zmfOyss479Q/HIXxmmbkD0Kp0
    eRm2KN3y0b6FySsS40KDRjKGQCuGGlNotW3crMw6vOvvsLTlcKgUHF054UVCHoK/
    Piz7igkDU7NjvJeha53vXL4hIjb10UtJNaGPxIyFLYRZdRPyyBJX7Yt3w8dgz8WM
    epOPu0dq3bUrY3WQXcxKZo6sQjE1h7kdl4TNji5jaFlvD01Y8LnyG0oThOzf0tve
    Gaw+kuy17gTGZGMIfGVcdeb+SlioXMAAfOps+mNIwTECgYEA/gTO8W0hgYpOQJzn
    BpWkic3LAoBXWNpvsQkkC3uba8Fcps7iiEzotXGfwYcb5Ewf5O3Lrz1EwLj7GTW8
    VNhB3gb7bGOvuwI/6vYk2/dwo84bwW9qRWP5hqPhNZ2AWl8kxmZgHns6WTTxpkRU
    zrfZ5eUrBDWjRU2R8uppgRImsxMCgYEA2MxuL/C/Ko0d7XsSX1kM4JHJiGpQDvb5
    GUrlKjP/qVyUysNF92B9xAZZHxxfPWpdfGGBynhw7X6s+YeIoxTzFPZVV9hlkpAA
    5igma0n8ZpZEqzttjVdpOQZK8o/Oni/Q2S10WGftQOOGw5Is8+LY30XnLvHBJhO7
    TKMurJ4KCNsCgYAe5TDSVmaj3dGEtFC5EUxQ4nHVnQyCpxa8npL+vor5wSvmsfUF
    hO0s3GQE4sz2qHecnXuPldEd66HGwC1m2GKygYDk/v7prO1fQ47aHi9aDQB9N3Li
    e7Vmtdn3bm+lDjtn0h3Qt0YygWj+wwLZnazn9EaWHXv9OuEMfYxVgYKpdwKBgEze
    Zy8+WDm5IWRjn8cI5wT1DBT/RPWZYgcyxABrwXmGZwdhp3wnzU/kxFLAl5BKF22T
    kRZ+D+RVZvVutebE9c937BiilJkb0AXLNJwT9pdVLnHcN2LHHHronUhV7vetkop+
    kGMMLlY0lkLfoGq1AxpfSbIea9KZam6o6VKxEnPDAoGAFDCJm+ZtsJK9nE5GEMav
    NHy+PwkYsHhbrPl4dgStTNXLenJLIJ+Ke0Pcld4ZPfYdSyu/Tv4rNswZBNpNsW9K
    0NwJlyMBfayoPNcJKXrH/csJY7hbKviAHr1eYy9/8OL0dHf85FV+9uY5YndLcsDc
    nygO9KTJuUiBrLr0AHEnqko=
    -----END PRIVATE KEY-----
key_passphrase: null
certificate_authorities:
cipher_suites: null
curve_types: null
supported_protocols: null
```

Related to: elastic#19504
@ph ph force-pushed the elastic-agent/embed-ca-in-yaml branch from 8fe38ae to bf266c8 Compare September 24, 2020 18:11
@ph ph merged commit 3f017df into elastic:master Sep 28, 2020
@ph
Copy link
Contributor Author

ph commented Sep 28, 2020

Ok, I've created an empty PR and received the kerberos failure is not from this PR.

@ph ph added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 28, 2020
ph added a commit to ph/beats that referenced this pull request Sep 28, 2020
* [Elastic Agent] Allow embedding of certificate

This PR allow to embed Certificate authorities directly in the yaml
configuration. This is useful in the context of fleet where distributing
file to the remote host is not possible. The format of the string need
to be in PEM.

Example: Certificate Authorities
```yaml
enabled: true
verification_mode: null
certificate: null
key: null
key_passphrase: null
certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
cipher_suites: null
curve_types: null
supported_protocols: null
```

```Certificate and Key
enabled: true
verification_mode: null
certificate: |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDXHufGPycpCOfI
    sjl6cRn8NP4DLxdIVEAHFK0jMRDup32UQOPW+DleEsFpgN9/ebi9ngdjQfMvKnUP
    Zrl1HTwVhOJfazGeoJn7vdDeQebhJfeDXHwX2DiotXyUPYu1ioU45UZDAoAZFj5F
    KJLwWRUbfEbRe8yO+wUhKKxxkApPbfw+wUtBicn1RIX7W1nBRABt1UXKDIRe5FM2
    MKfqhEqK4hUWC3g1r+vGTrxu3qFpzz7L2UrRFRIpo7yuTUhEhEGvcVsiTppTil4Z
    HcprXFHf5158elEwhYJ5IM0nU1leNQiOgemifbLwkyNkLqCKth8V/4sezr1tYblZ
    nMh1cclBAgMBAAECggEBAKdP5jyOicqknoG9/G564RcDsDyRt64NuO7I6hBg7SZx
    Jn7UKWDdFuFP/RYtoabn6QOxkVVlydp5Typ3Xu7zmfOyss479Q/HIXxmmbkD0Kp0
    eRm2KN3y0b6FySsS40KDRjKGQCuGGlNotW3crMw6vOvvsLTlcKgUHF054UVCHoK/
    Piz7igkDU7NjvJeha53vXL4hIjb10UtJNaGPxIyFLYRZdRPyyBJX7Yt3w8dgz8WM
    epOPu0dq3bUrY3WQXcxKZo6sQjE1h7kdl4TNji5jaFlvD01Y8LnyG0oThOzf0tve
    Gaw+kuy17gTGZGMIfGVcdeb+SlioXMAAfOps+mNIwTECgYEA/gTO8W0hgYpOQJzn
    BpWkic3LAoBXWNpvsQkkC3uba8Fcps7iiEzotXGfwYcb5Ewf5O3Lrz1EwLj7GTW8
    VNhB3gb7bGOvuwI/6vYk2/dwo84bwW9qRWP5hqPhNZ2AWl8kxmZgHns6WTTxpkRU
    zrfZ5eUrBDWjRU2R8uppgRImsxMCgYEA2MxuL/C/Ko0d7XsSX1kM4JHJiGpQDvb5
    GUrlKjP/qVyUysNF92B9xAZZHxxfPWpdfGGBynhw7X6s+YeIoxTzFPZVV9hlkpAA
    5igma0n8ZpZEqzttjVdpOQZK8o/Oni/Q2S10WGftQOOGw5Is8+LY30XnLvHBJhO7
    TKMurJ4KCNsCgYAe5TDSVmaj3dGEtFC5EUxQ4nHVnQyCpxa8npL+vor5wSvmsfUF
    hO0s3GQE4sz2qHecnXuPldEd66HGwC1m2GKygYDk/v7prO1fQ47aHi9aDQB9N3Li
    e7Vmtdn3bm+lDjtn0h3Qt0YygWj+wwLZnazn9EaWHXv9OuEMfYxVgYKpdwKBgEze
    Zy8+WDm5IWRjn8cI5wT1DBT/RPWZYgcyxABrwXmGZwdhp3wnzU/kxFLAl5BKF22T
    kRZ+D+RVZvVutebE9c937BiilJkb0AXLNJwT9pdVLnHcN2LHHHronUhV7vetkop+
    kGMMLlY0lkLfoGq1AxpfSbIea9KZam6o6VKxEnPDAoGAFDCJm+ZtsJK9nE5GEMav
    NHy+PwkYsHhbrPl4dgStTNXLenJLIJ+Ke0Pcld4ZPfYdSyu/Tv4rNswZBNpNsW9K
    0NwJlyMBfayoPNcJKXrH/csJY7hbKviAHr1eYy9/8OL0dHf85FV+9uY5YndLcsDc
    nygO9KTJuUiBrLr0AHEnqko=
    -----END PRIVATE KEY-----
key_passphrase: null
certificate_authorities:
cipher_suites: null
curve_types: null
supported_protocols: null
```

Related to: elastic#19504

(cherry picked from commit 3f017df)
v1v added a commit to v1v/beats that referenced this pull request Sep 29, 2020
* upstream/master:
  feat: prepare release pipelines (elastic#21238)
  Add IP validation to Security module (elastic#21325)
  Fixes for new 7.10 rsa2elk datasets (elastic#21240)
  o365input: Restart after fatal error (elastic#21258)
  Fix panic in cgroups monitoring (elastic#21355)
  Handle multiple upstreams in ingress-controller (elastic#21215)
  [CI] Fix runbld when workspace does not exist (elastic#21350)
  [Filebeat] Fix checkpoint (elastic#21344)
  [CI] Archive build reasons (elastic#21347)
  Add dashboard for pubsub metricset in googlecloud module (elastic#21326)
  [Elastic Agent] Allow embedding of certificate (elastic#21179)
  Adds a default for failure_cache.min_ttl (elastic#21085)
  [libbeat] Disk queue implementation (elastic#21176)
ph added a commit that referenced this pull request Sep 29, 2020
* [Elastic Agent] Allow embedding of certificate

This PR allow to embed Certificate authorities directly in the yaml
configuration. This is useful in the context of fleet where distributing
file to the remote host is not possible. The format of the string need
to be in PEM.

Example: Certificate Authorities
```yaml
enabled: true
verification_mode: null
certificate: null
key: null
key_passphrase: null
certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
cipher_suites: null
curve_types: null
supported_protocols: null
```

```Certificate and Key
enabled: true
verification_mode: null
certificate: |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----
key: |
    -----BEGIN PRIVATE KEY-----
    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDXHufGPycpCOfI
    sjl6cRn8NP4DLxdIVEAHFK0jMRDup32UQOPW+DleEsFpgN9/ebi9ngdjQfMvKnUP
    Zrl1HTwVhOJfazGeoJn7vdDeQebhJfeDXHwX2DiotXyUPYu1ioU45UZDAoAZFj5F
    KJLwWRUbfEbRe8yO+wUhKKxxkApPbfw+wUtBicn1RIX7W1nBRABt1UXKDIRe5FM2
    MKfqhEqK4hUWC3g1r+vGTrxu3qFpzz7L2UrRFRIpo7yuTUhEhEGvcVsiTppTil4Z
    HcprXFHf5158elEwhYJ5IM0nU1leNQiOgemifbLwkyNkLqCKth8V/4sezr1tYblZ
    nMh1cclBAgMBAAECggEBAKdP5jyOicqknoG9/G564RcDsDyRt64NuO7I6hBg7SZx
    Jn7UKWDdFuFP/RYtoabn6QOxkVVlydp5Typ3Xu7zmfOyss479Q/HIXxmmbkD0Kp0
    eRm2KN3y0b6FySsS40KDRjKGQCuGGlNotW3crMw6vOvvsLTlcKgUHF054UVCHoK/
    Piz7igkDU7NjvJeha53vXL4hIjb10UtJNaGPxIyFLYRZdRPyyBJX7Yt3w8dgz8WM
    epOPu0dq3bUrY3WQXcxKZo6sQjE1h7kdl4TNji5jaFlvD01Y8LnyG0oThOzf0tve
    Gaw+kuy17gTGZGMIfGVcdeb+SlioXMAAfOps+mNIwTECgYEA/gTO8W0hgYpOQJzn
    BpWkic3LAoBXWNpvsQkkC3uba8Fcps7iiEzotXGfwYcb5Ewf5O3Lrz1EwLj7GTW8
    VNhB3gb7bGOvuwI/6vYk2/dwo84bwW9qRWP5hqPhNZ2AWl8kxmZgHns6WTTxpkRU
    zrfZ5eUrBDWjRU2R8uppgRImsxMCgYEA2MxuL/C/Ko0d7XsSX1kM4JHJiGpQDvb5
    GUrlKjP/qVyUysNF92B9xAZZHxxfPWpdfGGBynhw7X6s+YeIoxTzFPZVV9hlkpAA
    5igma0n8ZpZEqzttjVdpOQZK8o/Oni/Q2S10WGftQOOGw5Is8+LY30XnLvHBJhO7
    TKMurJ4KCNsCgYAe5TDSVmaj3dGEtFC5EUxQ4nHVnQyCpxa8npL+vor5wSvmsfUF
    hO0s3GQE4sz2qHecnXuPldEd66HGwC1m2GKygYDk/v7prO1fQ47aHi9aDQB9N3Li
    e7Vmtdn3bm+lDjtn0h3Qt0YygWj+wwLZnazn9EaWHXv9OuEMfYxVgYKpdwKBgEze
    Zy8+WDm5IWRjn8cI5wT1DBT/RPWZYgcyxABrwXmGZwdhp3wnzU/kxFLAl5BKF22T
    kRZ+D+RVZvVutebE9c937BiilJkb0AXLNJwT9pdVLnHcN2LHHHronUhV7vetkop+
    kGMMLlY0lkLfoGq1AxpfSbIea9KZam6o6VKxEnPDAoGAFDCJm+ZtsJK9nE5GEMav
    NHy+PwkYsHhbrPl4dgStTNXLenJLIJ+Ke0Pcld4ZPfYdSyu/Tv4rNswZBNpNsW9K
    0NwJlyMBfayoPNcJKXrH/csJY7hbKviAHr1eYy9/8OL0dHf85FV+9uY5YndLcsDc
    nygO9KTJuUiBrLr0AHEnqko=
    -----END PRIVATE KEY-----
key_passphrase: null
certificate_authorities:
cipher_suites: null
curve_types: null
supported_protocols: null
```

Related to: #19504

(cherry picked from commit 3f017df)
@ferullo
Copy link

ferullo commented Dec 3, 2020

Will this be in 7.11? Is there an end to end way via Kibana that we can make use of this to make sure Endpoint ingests this data format and works?

@ph
Copy link
Contributor Author

ph commented Dec 3, 2020

@ferullo It's in 7.10.

I haven't thought that it would affect you, adding support for it should be trivial, assume that certificate_authorities and certificate can use this format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants