[Ingest Manager] Add a raw YML box when configuring the Elasticsearch output. (CA, Certificate) #75913

ph · 2020-08-25T19:10:16Z

Use case:

It would be nice to be able to configure the Elasticsearch output to use custom options like certificate_authorities or custom params. This would solve the issue of the Elastic Agent and self-signed certificate.

Solution:

Add a raw YAML box to the elasticsearch output settings, it will be a free form field for the users.

We should ensure that a user cannot set the api_key there because these keys are handled by fleet.

This can be done before having a complete self signed in fleet experience see #73483

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-08-25T19:10:18Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ph · 2020-08-31T15:58:42Z

@neptunian if you can take a look?

ph · 2020-09-16T11:33:47Z

@jfsiii Can you take this over?

ph · 2020-09-17T14:03:29Z

@ruflin @jfsiii After talking to @crowens Could we just retrieve the CA from the Kibana configuration and send it to the elastic agent? This assumes that the beats can understand the raw CA.

ruflin · 2020-09-18T07:06:09Z

@ph If we do that it would assume all the cert files on the Agent machine are in the same location as on the server where Kibana is hosted? So not sure this would work.

In any case, I would treat this as 2 separate features. I think the YAML box is a quick way to make us move forward and if we can have the Kibana CA as default set, even better.

blakerouse · 2020-09-18T13:28:32Z

@ruflin I think in this case it would the the actual certificate data in the YAML. Something we need to add support for Agent side.

I don't even know if it's worth adding the ability to set it to a path. It will be very hard for that path to be the same path across all platforms. I would recommand installing the certificate as a system certificate so Agent will look in the correct location for each system.

ph · 2020-09-18T19:26:08Z

@ruflin Its something I've discussed a long time ago, here a way to embed certificate authorities, certificate or keys in anything that support TLS options. elastic/beats#21179

ph · 2020-09-21T12:07:55Z

I don't even know if it's worth adding the ability to set it to a path. It will be very hard for that path to be the same path across all platforms. I would recommand installing the certificate as a system certificate so Agent will look in the correct location for each system.

IIRC this is not the default behavior on windows though? Also I think its better if we don't mess around with the system certificate.

ph · 2020-09-21T12:53:24Z

@ruflin @jfsiii For adding the YAML box in the elasticsearch output options, A user would need to add the following in the yaml box.

certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----

YAML string is seriously a pain and the format of the certificate does need to match, newline and everything, maybe its okay to do that in 7.10 to at least enable that workflow.

ph · 2020-09-23T15:42:24Z

This box will be editable per Agent policy since "output" are linked to a policy.

@ruflin I had a chat with John over zoom, we should probably have some "rejected" keys so they don't mess up what fleet is actually using. I see theses keys as problematics:

api_key
username
password
index
hosts

step 1, make it work,
step 2: add the reject or accept list.

@ruflin should we be more strict and have an "accept" list instead, WDYT? With great power comes great responsibility.

See the https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

blakerouse · 2020-09-23T15:48:21Z

@ph Maybe the entire YAMl block is just to much, just input fields for the certificates and keys is what should be added instead?

ph · 2020-09-23T15:56:16Z

@blakerouse actually we need three?

certificates_authorities (list) (could probably be fetched from Kibana?)
certificate
key

Concerning 1, Beats accepts a list today, but seriously is that really used in the wild, I am not even sure there is a use case for it?

ruflin · 2020-09-24T06:28:01Z

My suggestion would be:

Offer raw YAML to have a quick solution for a common problem
Add specific fields + still have the extended yaml option for expert
Add validation for fields

The part I didn't get how the raw yaml is going to mess up with the Fleet part? You mean if the api_key for example is used? Which one does win in the case of yaml, first or last? If that is what you are getting at, I would think it is ok for a first iteration.

ph · 2020-09-28T19:43:19Z

I've merged the dependent PR on the beats side elastic/beats#21179

ph · 2020-09-28T19:44:36Z

The part I didn't get how the raw yaml is going to mess up with the Fleet part? You mean if the api_key for example is used? Which one does win in the case of yaml, first or last? If that is what you are getting at, I would think it is ok for a first iteration.

Yes, I meant that would be unexpected behavior, I presume that the logic in fleet adding the output information would win. @nchaulet

ph · 2020-09-28T19:45:14Z

I am +1 for solving the case with a simple first implementation like 1 like ruflin suggested.

nchaulet · 2020-09-28T19:46:14Z

Yes, I meant that would be unexpected behavior, I presume that the logic in fleet adding the output information would win.

Yes we add the API key just before sending the config to the agent so this would replace any previously set property

jfsiii · 2020-10-01T16:24:32Z

I've opened a PR at #79019

ferullo · 2020-10-01T16:46:04Z

@k-g-elastic @EricDavisX once this is merged we should add testing this to the routine tests for Endpoint.

ghost · 2020-10-06T12:28:49Z

Hi Eric

We have created a testcase https://elastic.testrail.io/index.php?/cases/view/34210 for this ticket.

We will validate this ticket once the changes are merged,

nicpenning · 2020-11-14T18:37:26Z

@ruflin @jfsiii For adding the YAML box in the elasticsearch output options, A user would need to add the following in the yaml box.

certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----

YAML string is seriously a pain and the format of the certificate does need to match, newline and everything, maybe its okay to do that in 7.10 to at least enable that workflow.

Is it strange that in 7.10 I had to have both this format and the ca.crt file for the agent to successfully send logs?

Note: I replaced my cert with your cert here to show what I am talking about. This is what I have in the new output settings.

# YAML settings here will be added to the Elasticsearch output section of each policy
ssl.certificate_authorities: ["C:\\Program Files\\Elastic\\Agent\\ca.crt"]
certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
    sxSmbIUfc2SGJGCJD4I=
    -----END CERTIFICATE-----

ph added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 25, 2020

ph changed the title ~~[Ingest Manager] Add a raw YML box when configuring the Elasticsearch output.~~ [Ingest Manager] Add a raw YML box when configuring the Elasticsearch output. (CA, Certificate) Sep 2, 2020

ph assigned neptunian Sep 8, 2020

ph assigned jfsiii and unassigned neptunian Sep 16, 2020

jfsiii mentioned this issue Sep 30, 2020

[Ingest Manager] Configure Elasticsearch output with YAML in global output settings #79019

Merged

1 task

EricDavisX mentioned this issue Oct 5, 2020

[Fleet, Agent & Endpoint] support raw yaml input for global config (for Elasticsearch) elastic/e2e-testing#350

Closed

ph closed this as completed Oct 7, 2020

liu-xiao-guo mentioned this issue Mar 4, 2021

Agent stays in 'updating' on self-managed cluster (works when using cloud for stack) elastic/beats#24274

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Add a raw YML box when configuring the Elasticsearch output. (CA, Certificate) #75913

[Ingest Manager] Add a raw YML box when configuring the Elasticsearch output. (CA, Certificate) #75913

ph commented Aug 25, 2020 •

edited

Loading

elasticmachine commented Aug 25, 2020

ph commented Aug 31, 2020

ph commented Sep 16, 2020

ph commented Sep 17, 2020

ruflin commented Sep 18, 2020

blakerouse commented Sep 18, 2020

ph commented Sep 18, 2020

ph commented Sep 21, 2020

ph commented Sep 21, 2020 •

edited

Loading

ph commented Sep 23, 2020 •

edited

Loading

blakerouse commented Sep 23, 2020

ph commented Sep 23, 2020

ruflin commented Sep 24, 2020

ph commented Sep 28, 2020

ph commented Sep 28, 2020

ph commented Sep 28, 2020

nchaulet commented Sep 28, 2020

jfsiii commented Oct 1, 2020

ferullo commented Oct 1, 2020

ghost commented Oct 6, 2020

nicpenning commented Nov 14, 2020

[Ingest Manager] Add a raw YML box when configuring the Elasticsearch output. (CA, Certificate) #75913

[Ingest Manager] Add a raw YML box when configuring the Elasticsearch output. (CA, Certificate) #75913

Comments

ph commented Aug 25, 2020 • edited Loading

elasticmachine commented Aug 25, 2020

ph commented Aug 31, 2020

ph commented Sep 16, 2020

ph commented Sep 17, 2020

ruflin commented Sep 18, 2020

blakerouse commented Sep 18, 2020

ph commented Sep 18, 2020

ph commented Sep 21, 2020

ph commented Sep 21, 2020 • edited Loading

ph commented Sep 23, 2020 • edited Loading

blakerouse commented Sep 23, 2020

ph commented Sep 23, 2020

ruflin commented Sep 24, 2020

ph commented Sep 28, 2020

ph commented Sep 28, 2020

ph commented Sep 28, 2020

nchaulet commented Sep 28, 2020

jfsiii commented Oct 1, 2020

ferullo commented Oct 1, 2020

ghost commented Oct 6, 2020

nicpenning commented Nov 14, 2020

ph commented Aug 25, 2020 •

edited

Loading

ph commented Sep 21, 2020 •

edited

Loading

ph commented Sep 23, 2020 •

edited

Loading