Add StatefulSet as a deployment option for Elastic Agent

[pkoutsovasilis]

This PR enriches the possible types of deploying agent by adding support for statefulSet. A use-case, that this PR adds support for, is deploying ksm with sharding (requires statefulset) with agent containers that extract only kube-state metrics, e.g. as being done here. Having this feature will also benefit this one elastic/elastic-agent#3847.

[barkbay]

A use-case, that this PR adds support for, is deploying ksm with sharding (requires statefulset) with agent containers that extract only kube-state metrics,

Do you think it would be possible to add an example in config/recipes/elastic-agent?

Also I believe GetPodTemplate() is not tested, but it's not from your PR. It could be interesting to check somewhere that the expected "native" resource (daemonset, deployment or statefulet) is created using the expected template.

[pkoutsovasilis]

A use-case, that this PR adds support for, is deploying ksm with sharding (requires statefulset) with agent containers that extract only kube-state metrics,

Do you think it would be possible to add an example in config/recipes/elastic-agent?

Also I believe GetPodTemplate() is not tested, but it's not from your PR. It could be interesting to check somewhere that the expected "native" resource (daemonset, deployment or statefulet) is created using the expected template.

@barkbay I added a recipe for ksm-sharding specifically that utilises the introduced statefulset type of the agent. Would you like me to try to tackle GetPodTemplate() in this PR?

[barkbay]

LGTM

Sample does not seem to be working as expected.

[barkbay]

I did a test using the sample and Agent is stuck in a red state:

k get es,agent,sts,pods                  
NAME                                                       HEALTH   NODES   VERSION   PHASE   AGE
elasticsearch.elasticsearch.k8s.elastic.co/elasticsearch   green    3       8.10.4    Ready   118s

NAME                                       HEALTH   AVAILABLE   EXPECTED   VERSION   AGE
agent.agent.k8s.elastic.co/elastic-agent   red                  1          8.10.4    119s

NAME                                        READY   AGE
statefulset.apps/elastic-agent-agent        1/1     116s
statefulset.apps/elasticsearch-es-default   3/3     117s

NAME                             READY   STATUS    RESTARTS   AGE
pod/elastic-agent-agent-0        2/2     Running   0          116s
pod/elasticsearch-es-default-0   1/1     Running   0          116s
pod/elasticsearch-es-default-1   1/1     Running   0          116s
pod/elasticsearch-es-default-2   1/1     Running   0          116s
pod/kibana-kb-7d57f7cf77-qvkxj   1/1     Running   0          116s

I have the following error warning in Agent logs (k logs pod/elastic-agent-agent-0 -c agent):

{
    "log.level": "warn",
    "@timestamp": "2023-12-19T07:30:40.848Z",
    "message": "Cannot index event publisher.Event ... (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:2047] failed to parse: data stream timestamp field [@timestamp] is missing\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"data stream timestamp field [@timestamp] is missing\"}}, dropping event!",
    "component": {
        "binary": "filebeat",
        "dataset": "elastic_agent.filebeat",
        "id": "filestream-monitoring",
        "type": "filestream"
    },
    "log": {
        "source": "filestream-monitoring"
    },
    "log.origin": {
        "file.line": 446,
        "file.name": "elasticsearch/client.go"
    },
    "service.name": "filebeat",
    "ecs.version": "1.6.0",
    "log.logger": "elasticsearch"
}

Also ecs.version seems to be duplicated in the original document.

[barkbay]

I did a test using the sample and Agent is stuck in a red state:

Actually Agent became green once I restarted the ECK Pod:

NAME                                       HEALTH   AVAILABLE   EXPECTED   VERSION   AGE
agent.agent.k8s.elastic.co/elastic-agent   green    1           1          8.10.4    15m

I think we may have a bug in Agent where reconciliations don't always happen when they should (in case of state change for example) 😕 .

[barkbay]

Re LGTM