agent-fleet.asciidoc

:page_id: elastic-agent-fleet
:agent_recipes: https://raw.githubusercontent.com/elastic/cloud-on-k8s/{eck_release_branch}/config/recipes/elastic-agent
ifdef::env-github[]
****
link:https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-{page_id}.html[View this document on the Elastic website]
****
endif::[]
[id="{p}-{page_id}"]
= Run Fleet-managed Elastic Agent on ECK

This section describes how to configure and deploy Elastic Agent in link:https://www.elastic.co/guide/en/fleet/current/elastic-agent-installation.html[Fleet-managed] mode with ECK. Check the link:k8s-elastic-agent.html[Standalone section] if you want to run Elastic Agent in the link:https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html[standalone mode].

* <<{p}-elastic-agent-fleet-quickstart,Quickstart>>
* <<{p}-elastic-agent-fleet-configuration,Configuration>>
* <<{p}-elastic-agent-fleet-configuration-examples,Configuration Examples>>
* <<{p}-elastic-agent-fleet-known-limitations,Known Limitations>>

[id="{p}-elastic-agent-fleet-quickstart"]
== Quickstart

. To deploy Fleet Server, Elastic Agents, Elasticsearch, and Kibana, apply the following specification:
+
[source,yaml,subs="attributes,callouts,+macros"]
----
cat $$<<$$EOF | kubectl apply -f -
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-quickstart
  namespace: default
spec:
  version: {version}
  kibanaRef:
    name: kibana-quickstart
  elasticsearchRefs:
  - name: elasticsearch-quickstart
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0 <1>
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-quickstart
  namespace: default
spec:
  version: {version}
  kibanaRef:
    name: kibana-quickstart
  fleetServerRef:
    name: fleet-server-quickstart
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0 <1>
        volumes:
        - name: agent-data
          emptyDir: {}
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-quickstart
  namespace: default
spec:
  version: {version}
  count: 1
  elasticsearchRef:
    name: elasticsearch-quickstart
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-quickstart-es-http.default.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-quickstart-agent-http.default.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-quickstart
  namespace: default
spec:
  version: {version}
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - pods
  - nodes
  - namespaces
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
- apiGroups: ["apps"]
  resources:
  - replicasets
  verbs:
  - list
  - watch
- apiGroups: ["batch"]
  resources:
  - jobs
  verbs:
  - list
  - watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
- kind: ServiceAccount
  name: elastic-agent
  namespace: default
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
EOF
----
+
<1> The root user is required to persist state in a hostPath volume and to trust the Elasticsearch CA in Fleet mode. See <<{p}_storing_local_state_in_host_path_volume>> for options to not run the Agent container as root.
+
Check <<{p}-elastic-agent-fleet-configuration-examples>> for more ready-to-use manifests.

ECK automatically configures secure connections between all components. Fleet will be set up, and all agents are enrolled in the default policy.

. Monitor the status of Fleet Server and Elastic Agent.

+
[source,sh]
----
kubectl get agent
----
+
[source,sh,subs="attributes"]
----
NAME                       HEALTH   AVAILABLE   EXPECTED   VERSION      AGE
elastic-agent-quickstart   green    3           3          {version}    14s
fleet-server-quickstart    green    1           1          {version}    19s

----

. List all the Pods belonging to a given Elastic Agent specification.
+
[source,sh]
----
kubectl get pods --selector='agent.k8s.elastic.co/name=elastic-agent-quickstart'
----
+
[source,sh]
----
NAME                                   READY   STATUS    RESTARTS   AGE
elastic-agent-quickstart-agent-t49fd   1/1     Running   0          54s
elastic-agent-quickstart-agent-xbcxr   1/1     Running   0          54s
elastic-agent-quickstart-agent-zqp55   1/1     Running   0          54s
----

. Access logs for one of the Pods.
+
[source,sh]
----
kubectl logs -f elastic-agent-quickstart-agent-xbcxr
----

. Configure the policy used by Elastic Agents. Check link:https://www.elastic.co/guide/en/fleet/current/agent-policy.html[Elastic Agent policies] for more details.

[id="{p}-elastic-agent-fleet-configuration"]
== Configuration

Fleet-managed Elastic Agents must connect to Fleet Server to receive their configurations. You can deploy Fleet Server instances using ECKs Agent CRD with the appropriate configuration, as shown in <<{p}-elastic-agent-fleet-configuration-fleet-mode-and-fleet-server,Fleet mode and Fleet Server>>.

To know more about Fleet architecture and related components, check the Fleet link:https://www.elastic.co/guide/en/fleet/current/fleet-server.html[documentation].

[id="{p}-elastic-agent-fleet-configuration-fleet-mode-and-fleet-server"]
=== Fleet mode and Fleet Server
To run both Fleet Server and Elastic Agent in Fleet-managed mode, set the `mode` configuration element to `fleet`.

[source,yaml,subs="attributes,+macros"]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  mode: fleet
----

To run Fleet Server, set the `fleetServerEnabled` configuration element to `true`, as shown in this example: 

[source,yaml,subs="attributes,+macros"]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  mode: fleet
  fleetServerEnabled: true
----
You can leave the default value `false` for any other case.

[id="{p}-elastic-agent-fleet-configuration-required-kibana-configuration"]
=== Configure Kibana

To have Fleet running properly, the following settings must be correctly set in the Kibana configuration:

[source,yaml,subs="attributes,+macros"]
----
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-sample-es-http.default.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-sample-agent-http.default.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        is_default: true
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
----

*  `xpack.fleet.agents.elasticsearch.hosts` must point to the Elasticsearch cluster where Elastic Agents should send data. For ECK-managed Elasticsearch clusters ECK creates a Service accessible through `https://ES_RESOURCE_NAME-es-http.ES_RESOURCE_NAMESPACE.svc:9200` URL, where `ES_RESOURCE_NAME` is the name of Elasticsearch resource and `ES_RESOURCE_NAMESPACE` is the namespace it was deployed within. See <<{p}_storing_local_state_in_host_path_volume>> for details on adjusting this field when running agent as non-root as it becomes required.

*  `xpack.fleet.agents.fleet_server.hosts` must point to Fleet Server that Elastic Agents should connect to. For ECK-managed Fleet Server instances, ECK creates a Service accessible through `https://FS_RESOURCE_NAME-agent-http.FS_RESOURCE_NAMESPACE.svc:8220` URL, where `FS_RESOURCE_NAME` is the name of Elastic Agent resource with Fleet Server enabled and `FS_RESOURCE_NAMESPACE` is the namespace it was deployed in.

*  `xpack.fleet.packages` are required packages to enable Fleet Server and Elastic Agents to enroll. 

*  `xpack.fleet.agentPolicies` policies are needed for Fleet Server and Elastic Agents to enroll to, check https://www.elastic.co/guide/en/fleet/current/agent-policy.html for more information.

[id="{p}-elastic-agent-fleet-configuration-setting-referenced-resources"]
=== Set referenced resources

Both Fleet Server and Elastic Agent in Fleet mode can be automatically set up with Fleet by ECK. The ECK operator can set up Fleet in Kibana (which otherwise requires manual steps) and enroll Fleet Server in the default Fleet Server policy. Elastic Agent can be automatically enrolled in the default Elastic Agent policy. To allow ECK to set this up, provide a reference to a ECK-managed Kibana through the `kibanaRef` configuration element.

[source,yaml,subs="attributes,+macros"]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  kibanaRef:
    name: kibana
----

ECK can also facilitate the connection between Elastic Agents and a ECK-managed Fleet Server. To allow ECK to set this up, provide a reference to Fleet Server through the `fleetServerRef` configuration element.

[source,yaml,subs="attributes,+macros"]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  fleetServerRef:
    name: fleet-server-sample
----


Set the `elasticsearchRefs` element in your Fleet Server to point to the Elasticsearch cluster that will manage Fleet. Leave `elasticsearchRefs` empty or unset it for any Elastic Agent running in Fleet mode as the Elasticsearch cluster to target will come from Kibana's `xpack.fleet.agents.elasticsearch.hosts` configuration element.

NOTE: Currently, Elastic Agent in Fleet mode supports only a single output, so only a single Elasticsearch cluster can be referenced.

[source,yaml,subs="attributes,+macros"]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  elasticsearchRefs:
  - name: elasticsearch-sample
----

By default, every reference targets all instances in your Elasticsearch, Kibana and Fleet Server deployments, respectively. If you want to direct traffic to specific instances, refer to <<{p}-traffic-splitting>> for more information and examples.

[id="{p}-elastic-agent-fleet-configuration-custom-configuration"]
=== Customize Elastic Agent configuration

In contrast to Elastic Agents in standalone mode, the configuration is managed through Fleet, and it cannot be defined through `config` or `configRef` elements.

[id="{p}-elastic-agent-fleet-configuration-upgrade-specification"]
=== Upgrade the Elastic Agent specification

You can upgrade the Elastic Agent version or change settings by editing the YAML specification file. ECK applies the changes by performing a rolling restart of the Agent's Pods. Depending on the settings that you used, ECK will set up Fleet in Kibana, enrolls the agent in Fleet, or restarts Elastic Agent on certificate rollover.

[id="{p}-elastic-agent-fleet-configuration-chose-the-deployment-model"]
=== Choose the deployment model

Depending on the use case, Elastic Agent may need to be deployed as a link:https://kubernetes.io/docs/concepts/workloads/controllers/deployment/[Deployment] or a link:https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/[DaemonSet]. To choose how to deploy your Elastic Agents, provide a `podTemplate` element under the `deployment` or the `daemonSet` element in the specification. If you choose the `deployment` option, you can additionally specify the link:https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#strategy[strategy] used to replace old Pods with new ones.

Similarly, you can set the link:https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/[update strategy] when deploying as a DaemonSet. This allows you to control the rollout speed for new configuration by modifying the `maxUnavailable` setting:

[source,yaml,subs="attributes,+macros"]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  version: {version}
  daemonSet:
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 3
...
----

Refer to <<{p}-compute-resources-beats-agent>> for more information on how to use the Pod template to adjust the resources given to Elastic Agent.

[id="{p}-elastic-agent-fleet-configuration-role-based-access-control"]
=== Role Based Access Control for Elastic Agent

Some Elastic Agent features, such as the link:https://epr.elastic.co/package/kubernetes/0.2.8/[Kubernetes integration], require that Agent Pods interact with Kubernetes APIs. This functionality requires specific permissions. Standard Kubernetes link:https://kubernetes.io/docs/reference/access-authn-authz/rbac/[RBAC] rules apply. For example, to allow API interactions:

[source,yaml,subs="attributes,+macros"]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
spec:
  version: {version}
  elasticsearchRefs:
  - name: elasticsearch-sample
  daemonSet:
    podTemplate:
      spec:
        automountServiceAccountToken: true
        serviceAccountName: elastic-agent
...
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  - nodes/metrics
  - nodes/proxy
  - nodes/stats
  - events
  verbs:
  - get
  - watch
  - list
- nonResourceURLs:
  - /metrics
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
- kind: ServiceAccount
  name: elastic-agent
  namespace: default
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
----

[id="{p}-elastic-agent-fleet-configuration-deploying-in-secured-clusters"]
=== Deploy Elastic Agent in secured clusters

To deploy Elastic Agent in clusters with the Pod Security Policy admission controller enabled, or in <<{p}-openshift-agent,OpenShift>> clusters, you might need to grant additional permissions to the Service Account used by the Elastic Agent Pods. Those Service Accounts must be bound to a Role or ClusterRole that has `use` permission for the required Pod Security Policy or Security Context Constraints. Different Elastic Agent integrations might require different settings set in their PSP/link:{p}-openshift-agent.html[SCC].

[id="{p}-elastic-agent-fleet-configuration-customize-fleet-server-service"]
=== Customize Fleet Server Service

By default, ECK creates a Service for Fleet Server that Elastic Agents can connect through. You can customize it using the `http` configuration element. Check more information on how to link:k8s-services.html[make changes] to the Service and link:k8s-tls-certificates.html[customize] the TLS configuration.

[id="{p}-elastic-agent-control-fleet-policy-selection"]
=== Control Fleet policy selection

ECK uses the default policy to enroll Elastic Agents in Fleet and the default Fleet Server policy to enroll Fleet Server. A different policy can be chosen by using the `policyID` attribute in the Elastic Agent resource:
[source,yaml]
----

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
spec:
  policyID: my-custom-policy
...
----

Please note that the environment variables related to policy selection mentioned in the Elastic Agent link:https://www.elastic.co/guide/en/fleet/current/agent-environment-variables.html[docs] like `FLEET_SERVER_POLICY_ID` will be managed by the ECK operator.


[id="{p}-elastic-agent-running-as-a-non-root-user"]
// tag::configuration-example-elastic-agent-running-as-a-non-root-user[]
=== Running as a non-root user

In order to run Elastic Agent as a non-root user you must choose how you want to persist data to the Agent's volume.

1. Run Elastic Agent with an `emptyDir` volume. This has the downside of not persisting data between restarts of the Elastic Agent which can duplicate work done by the previous running Agent.
2. Run Elastic Agent with a `hostPath` volume in addition to a `DaemonSet` running as `root` that sets up permissions for the `agent` user.

In addition to these decisions, if you are running Elastic Agent in Fleet mode as a non-root user, you must configure `certificate_authorities.ssl` in each `xpack.fleet.outputs` to trust the CA of the Elasticsearch Cluster.

To run Elastic Agent with an `emptyDir` volume.

[source,yaml]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
spec:
  deployment:
    podTemplate:
      spec:
        securityContext: <1>
          fsGroup: 1000
        volumes:
        - name: agent-data
          emptyDir: {}
...
----
<1> Gid 1000 is the default group at which the Agent container runs. Adjust as necessary if `runAsGroup` has been modified.

To run Elastic Agent with a `hostPath` volume and a `DaemonSet` to maintain permissions.

[source,yaml]
----
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-sample
  namespace: elastic-apps
spec:
  mode: fleet
  fleetServerEnabled: true
  deployment: {}
...
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-sample
  namespace: elastic-apps
spec:
  daemonSet: {}
...
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: manage-agent-hostpath-permissions
  namespace: elastic-apps
spec:
  selector:
    matchLabels:
      name: manage-agent-hostpath-permissions
  template:
    metadata:
      labels:
        name: manage-agent-hostpath-permissions
    spec:
      # serviceAccountName: elastic-agent <1>
      volumes:
        - hostPath:
            path: /var/lib/elastic-agent
            type: DirectoryOrCreate
          name: "agent-data"
      initContainers:
        - name: manage-agent-hostpath-permissions
          # image: registry.access.redhat.com/ubi8/ubi-minimal:latest <2>
          image: docker.io/bash:5.2.15
          resources:
            limits:
              cpu: 100m
              memory: 32Mi
          securityContext:
            # privileged: true <3>
            runAsUser: 0
          volumeMounts:
            - mountPath: /var/lib/elastic-agent
              name: agent-data
          command:
          - 'bash'
          - '-e'
          - '-c'
          - |-
            # Adjust this with /var/lib/elastic-agent/YOUR-NAMESPACE/YOUR-AGENT-NAME/state
            # Multiple directories are supported for the fleet-server + agent use case.
            dirs=(
              "/var/lib/elastic-agent/default/elastic-agent/state"
              "/var/lib/elastic-agent/default/fleet-server/state"
              )
            for dir in ${dirs[@]}; do
              mkdir -p "${dir}"
              # chcon is only required when running an an SELinux-enabled/OpenShift environment.
              # chcon -Rt svirt_sandbox_file_t "${dir}"
              chmod g+rw "${dir}"
              # Gid 1000 is the default group at which the Agent container runs. Adjust as necessary if `runAsGroup` has been modified.
              chgrp 1000 "${dir}"
              if [ -n "$(ls -A ${dir} 2>/dev/null)" ]
              then
                # Gid 1000 is the default group at which the Agent container runs. Adjust as necessary if `runAsGroup` has been modified.
                chgrp 1000 "${dir}"/*
                chmod g+rw "${dir}"/*
              fi
            done
      containers:
        - name: sleep
          image: gcr.io/google-containers/pause-amd64:3.2
----
<1> This is only required when running in an SElinux-enabled/OpenShift environment. Ensure this user has been added to the privileged security context constraints (SCC) in the correct namespace. `oc adm policy add-scc-to-user privileged -z elastic-agent -n elastic-apps`
<2> UBI is only required when needing the `chcon` binary when running in an SELinux-enabled/OpenShift environment. If that is not required then the following smaller image can be used instead: `docker.io/bash:5.2.15`
<3> Privileged is only required when running in an SElinux-enabled/OpenShift environment.

When running Agent in fleet mode as a non-root user Kibana must be configured in order to properly accept the CA of the Elasticsearch cluster.

[source,yaml]
----
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  config:
    # xpack.fleet.agents.elasticsearch.hosts: <1>
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-sample-agent-http.default.svc:8220"]
    xpack.fleet.outputs:
    - id: eck-fleet-agent-output-elasticsearch
      is_default: true
      name: eck-elasticsearch
      type: elasticsearch
      hosts:
      - "https://elasticsearch-sample-es-http.default.svc:9200" <2>
      ssl:
        certificate_authorities: ["/mnt/elastic-internal/elasticsearch-association/default/elasticsearch-sample/certs/ca.crt"] <3>
----

<1> This entry must not exist when running agent in fleet mode as a non-root user.
<2> Note that the correct URL for Elasticsearch is `https://ELASTICSEARCH_NAME-es-http.YOUR-NAMESPACE.svc:9200`
<3> Note that the correct path for Elasticsearch `certificate_authorities` is `/mnt/elastic-internal/elasticsearch-association/YOUR-NAMESPACE/ELASTICSEARCH-NAME/certs/ca.crt`

// end::configuration-example-elastic-agent-running-as-a-non-root-user[]

[id="{p}-elastic-agent-fleet-configuration-examples"]
== Configuration Examples

This section contains manifests that illustrate common use cases, and can be your starting point in exploring Elastic Agent deployed with ECK. These manifests are self-contained and work out-of-the-box on any non-secured Kubernetes cluster. They all contain a three-node Elasticsearch cluster, a single Kibana instance and a single Fleet Server instance.

CAUTION: The examples in this section are for illustration purposes only and should not be considered to be production-ready. Some of these examples use the `node.store.allow_mmap: false` setting which has performance implications and should be tuned for production workloads, as described in <<{p}-virtual-memory>>.


=== System and Kubernetes integrations

[source,sh,subs="attributes"]
----
kubectl apply -f {agent_recipes}/fleet-kubernetes-integration.yaml
----
Deploys Elastic Agent as a DaemonSet in Fleet mode with System and Kubernetes integrations enabled. System integration collects syslog logs, auth logs and system metrics (for CPU, I/O, filesystem, memory, network, process and others). Kubernetes integrations collects API server, Container, Event, Node, Pod, Volume and system metrics.

=== System and Kubernetes integrations running as non-root

[source,sh,subs="attributes"]
----
kubectl apply -f {agent_recipes}/fleet-kubernetes-integration-nonroot.yaml
----
The provided example is functionally identical to the previous section but runs the Elastic Agent processes (both the Elastic Agent running as the Fleet server and the Elastic Agent connected to Fleet) as a non-root user by utilizing a DaemonSet to ensure directory and file permissions.

NOTE: The DaemonSet itself must run as root to set up permissions and ECK >= 2.10.0 is required.

=== Custom logs integration with autodiscover

[source,sh,subs="attributes"]
----
kubectl apply -f {agent_recipes}/fleet-custom-logs-integration.yaml
----

Deploys Elastic Agent as a DaemonSet in Fleet mode with Custom Logs integration enabled. Collects logs from all Pods in the `default` namespace using autodiscover feature.


=== APM integration

[source,sh,subs="attributes"]
----
kubectl apply -f {agent_recipes}/fleet-apm-integration.yaml
----

Deploys single instance Elastic Agent Deployment in Fleet mode with APM integration enabled.

[id="{p}-elastic-agent-fleet-known-limitations"]
== Known limitations

=== Running as root and within a single namespace (ECK < 2.10.0 and Agent < 7.14.0)
Until version 7.14.0 and ECK version 2.10.0, Elastic Agent in Fleet mode has to run as root and in the same namespace as the Elasticsearch cluster it connects to. 

This was due to configuration limitations in Fleet/Elastic Agent. ECK needed to establish trust between Elastic Agents and Elasticsearch. ECK was only able to fetch the required Elasticsearch CA correctly if both resources are in the same namespace.
As of Elastic Stack version 7.14.0 and ECK version 2.10.0 it is also possible to run Elastic Agent and Fleet as a non-root user. See <<{p}_storing_local_state_in_host_path_volume>> for instructions.
To establish trust, the Pod needs to update the CA store through a call to `update-ca-trust` before Elastic Agent runs. To call it successfully, the Pod needs to run with elevated privileges.

=== Running Endpoint Security integration
Running Endpoint Security link:https://www.elastic.co/guide/en/security/current/install-endpoint.html[integration] is not yet supported in containerized environments, like Kubernetes. This is not an ECK limitation, but the limitation of the integration itself. Note that you can use ECK to deploy Elasticsearch, Kibana and Fleet Server, and add Endpoint Security integration to your policies if Elastic Agents running those policies are deployed in non-containerized environments.

=== Fleet Server initialization fails on minikube when CNI is disabled
When deployed with ECK, the Fleet Server Pod makes an HTTP call to itself during Fleet initialization using its Service. Since a link:https://github.com/kubernetes/minikube/issues/1568[Pod cannot reach itself through its Service on minikube] when CNI is disabled, the call hangs until the connection times out and the Pod enters a crash loop.

Solution: enable CNI when starting minikube: `minikube start --cni=true`.

// tag::elastic-agent-fleet-known-limitations-local-state[]

=== Storing local state in host path volume
Elastic Agent managed by ECK stores local state in a host path volume by default. This ensures that integrations run by the agent can continue their work without duplicating work that has already been done after the Pod has been recreated for example because of a Pod configuration change. Multiple replicas of an agent, for example Fleet Servers, can not be deployed on the same underlying Kubernetes node as they would try to use the same host path. There are 2 options for managing this feature:

1. If local state storage in `hostPath` volumes is not desired this can be turned off by configuring an `emptyDir` volume instead.
2. If local state storage is still desired but running the Agent container as root is not allowed, then you can run a `DaemonSet` that adjusts the permissions for the Agent local state on each Node prior to running Elastic Agent. Note that this `DaemonSet` must be `runAsUser: 0` and possibly `privileged: true`. Also note the Kibana changes required to trust the Elasticsearch CA when running in fleet mode.

Full configuration examples exist in  <<{p}-elastic-agent-running-as-a-non-root-user>>.

// end::elastic-agent-fleet-known-limitations-local-state[]