Support running Agent as a non-root

[naemono]

relates #6599

This adds documentation to allow a user to run a DaemonSet to automatically adjust Agent hostPath permissions and allow the user to not run the Agent as runAsUser: 0.

This also only updates the CA store in Agent if version is < 7.14.0, which also removes the need to run Agent+fleet as root, which was verified locally. The minimum version of Agent+Fleet we allow is 7.14.0.

Changes

• We only set the Elasticsearch CA to be trusted by the pod's CA store when the pod is set to run as root. The documentation discusses and gives examples on how to run Agent+Fleet as non-root and the Kibana changes required for the Elasticsearch CA to be trusted.

TODO

• Test the update-ca-trust requirement for running with Fleet.
• Update the Agent+Fleet documentation
• End-to-End test using Daemonset + Agent + Fleet running as non-root

[naemono]

buildkite test this -f p=gke,t=TestFleetKubernetesNonRootIntegrationRecipe -m s=8.7.0,s=7.17.8

[naemono]

buildkite test this -f p=gke,t=TestFleetKubernetesNonRootIntegrationRecipe -m s=8.7.0,s=7.17.8

[naemono]

buildkite test this -f p=gke,t=TestFleetKubernetesNonRootIntegrationRecipe -m s=8.7.0,s=7.17.8

[barkbay]

Almost LGTM. Overall it feels complicated and hacky (especially runningAsRoot) but I'm not sure there is a good alternative.

[naemono]

Almost LGTM. Overall it feels complicated and hacky (especially runningAsRoot) but I'm not sure there is a good alternative.

I don't disagree here. This has been a complicated issue since it's inception. We initially tried to "just make it work" within the eck controller, but thought that the permissions needed were a bit too much from a security standpoint, so decided to try and document how this could be done. Unfortunately this was the only solution found when working with the Fleet team. Honestly I'm not sure at this point if leaving the root requirement in place is a simpler solution than documenting the complexities of this non-root solution, but I'd love others input into this question.

[naemono]

buildkite test this -f p=gke,t=TestFleetKubernetesNonRootIntegrationRecipe -m s=8.9.0,s=7.17.8

Testing a final time after removing is_default* and adding PolicyID to ensure completeness.

[naemono]

buildkite test this -f p=gke,t=TestFleetKubernetesNonRootIntegrationRecipe -m s=8.9.0,s=7.17.8

Hopefully last one after reverting the addition of a # comment to the yaml.

[naemono]

Somehow the last change to remove is_default and adding PolicyID broke the e2e tests.

The es output isn't being propagated agent => metricbeat

elasticsearch url: http://localhost:9200

Something to do with this:

    xpack.fleet.outputs:
    - id: eck-fleet-agent-output-elasticsearch
      name: eck-elasticsearch
      type: elasticsearch
      hosts: ["https://elasticsearch-es-http.default.svc:9200"]
      ssl:
        certificate_authorities: ["/mnt/elastic-internal/elasticsearch-association/default/elasticsearch/certs/ca.crt"]

I'll investigate and resolve and update.

[naemono]

buildkite test this -f p=gke,t=TestFleetKubernetesNonRootIntegrationRecipe -m s=8.9.0,s=7.17.8

[naemono]

Ok, is_default: true is required on the xpack.fleet.outputs, which was causing the error. The e2e tests appear to be passing now.

@pebrc @thbkrkr @barkbay This is ready for the next set of eyes when you all have the time. Thanks.

[pebrc]

LGTM just a few more nits.

[barkbay]

LGTM

[dbalijapalli0264]

Hello Elastic team,
Is this change backward compatible too? meaning can we run it on ECK operator 2.9? or should we have ECK operator 2.10?