Enable runAsNonRoot=true for Beats >= 8.8.x

[naemono]

Since elastic/beats#35272 was merged, and backported into upcoming 8.8 Beats release, we should be able to enable runAsNonRoot=true for beats now > 8.8.x.

Testing Done

With manifest at 8.8.0-SNAPSHOT

---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: testing
spec:
  version: 8.8.0-SNAPSHOT
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: monitoring
  nodeSets:
    - name: masters
      count: 1
      config:
        node.roles: ["master", "data"]
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 100Mi
                  cpu: 0.1
                limits:
                  memory: 1000Mi
                  cpu: 1
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: monitoring
spec:
  version: 8.8.0-SNAPSHOT
  nodeSets:
    - name: masters
      count: 1
      config:
        node.roles: ["master", "data"]
        node.store.allow_mmap: false
      podTemplate:
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 100Mi
                  cpu: 0.1
                limits:
                  memory: 1000Mi
                  cpu: 1

❯ kc get sts -n default testing-es-masters -o yaml | yq e '.spec.template.spec.containers[]|select(.name == "metricbeat")|.securityContext' -
allowPrivilegeEscalation: false
capabilities:
  drop:
    - ALL
privileged: false
readOnlyRootFilesystem: true
runAsNonRoot: true

❯ kc get pod -n default testing-es-masters-0 -o yaml | yq e '.status.containerStatuses[]|select(.name=="metricbeat")' -
containerID: containerd://fff7cca18a4c3a17564277b66ae106d7d3865fb915d1857f83c82d0697b53f41
image: docker.elastic.co/beats/metricbeat:8.8.0-SNAPSHOT
imageID: docker.elastic.co/beats/metricbeat@sha256:d3c4b47bbed4aa2eab5ad5cbdc5757d85d5ae90e8960ec4ed05cd7e1ebc0be42
lastState: {}
name: metricbeat
ready: true
restartCount: 0
started: true
state:
  running:
    startedAt: "2023-05-10T14:38:23Z"

With same manifest with 8.7.0

❯ kc get sts -n default testing-es-masters -o yaml | yq e '.spec.template.spec.containers[]|select(.name == "metricbeat")|.securityContext' -
allowPrivilegeEscalation: false
capabilities:
  drop:
    - ALL
privileged: false
readOnlyRootFilesystem: true

❯ kc get pods -n default testing-es-masters-0 -o yaml | yq e '.status.containerStatuses[]|select(.name=="metricbeat")' -
containerID: containerd://5779b2e36e829ccbbef9e75bf7da43f1f7a2d8e710223e582d0162458c4b62c0
image: docker.elastic.co/beats/metricbeat:8.7.0
imageID: docker.elastic.co/beats/metricbeat@sha256:6ba26079ed2631c1a52e2464620571220432173f4196d28d6c16e1a81407b56f
lastState: {}
name: metricbeat
ready: true
restartCount: 0
started: true
state:
  running:
    startedAt: "2023-05-10T15:04:07Z"

TODO: e2e tests verification

• verifying e2e tests for es on 8.8.0-SNAPSHOT
• verifying e2e tests for es on 8.7.0

[naemono]

buildkite test this -f p=gke,E2E_TAGS=es -m s=8.7.0,s=7.17.10,s=8.8.0-SNAPSHOT

[naemono]

buildkite test this -f p=gke,E2E_TAGS=es -m s=8.7.0,s=7.17.10,s=8.8.0-SNAPSHOT

[naemono]

8.7.0 and 7.17.0 failed in BK because of invalid logic in e2e tests, which I've fixed, and TestESStackMonitoring succeeded locally with both versions, but BK isn't running because of a GH outage. I'll trigger this again when things are green.

[naemono]

E2E tests succeeded @barkbay : https://buildkite.com/elastic/cloud-on-k8s-operator/builds/3552

[barkbay]

LGTM