Filebeat/autodiscover ES_data_should_pass_validations fails in 8.9.0-SNAPSHOT

[thbkrkr]

cloud-on-k8s-operator-nightly/builds/217@stack-8.9.0-SNAPSHOT

• TestFilebeatDefaultConfig/ES_data_should_pass_validations
• TestBeatSecureSettings/ES_data_should_pass_validations
• TestBeatConfigRef/ES_data_should_pass_validations

--- FAIL: TestFilebeatDefaultConfig (1887.47s)
    --- FAIL: TestFilebeatDefaultConfig/ES_data_should_pass_validations (1800.00s)
                Error Trace:    /go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94
                Error:          Received unexpected error:
                                hit count should be more than 0 for /*beat*/_search?q=agent.type:filebeat

--- FAIL: TestBeatSecureSettings (1888.35s)
    --- FAIL: TestBeatSecureSettings/ES_data_should_pass_validations (1800.00s)
                Error Trace:    /go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94
                Error:          Received unexpected error:
                                hit count should be more than 0 for /*beat*/_search?q=agent.type:filebeat

--- FAIL: TestBeatConfigRef (1896.88s)
    --- FAIL: TestBeatConfigRef/ES_data_should_pass_validations (1800.00s)
                Error Trace:    /go/src/github.com/elastic/cloud-on-k8s/test/e2e/test/utils.go:94
                Error:          Received unexpected error:
                                hit count should be more than 0 for /*beat*/_search?q=agent.type:filebeat

[thbkrkr]

Running TestFilebeatDefaultConfig locally I see missing RBAC permissions to use Filebeat with autodiscover, not sure if this is the root cause:

E0627 12:15:14.396909       7 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: 
  Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: 
  User "system:serviceaccount:e2e-mercury:test-fb-default-cfg-d26k-sa" cannot list resource "replicasets" in
  API group "apps" at the cluster scope

After fixing it, I got:

E0627 12:30:27.699959       7 reflector.go:138] pkg/mod/k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: 
  Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden:
  User "system:serviceaccount:e2e-mercury:test-fb-default-cfg-n9tn-sa" cannot list resource "jobs" in 
  API group "batch" at the cluster scope

After fixing it, it looks good.

[thbkrkr]

I asked the Beats team to confirm if this change is normal.

[thbkrkr]

I got confirmed that the change is normal. These permissions are required since a very long time.

However, a recent change makes that when the permissions are not set, the errors are propagated and that is why we only see them now.

• Add replicaset and job metaGenerator based on watcher elastic-agent-autodiscover#44
• Use replicaset and Job MetaGen based on watchers beats#35483
• add_resource_metadata.cronjob overloads the memory usage elastic-agent-autodiscover#31

Going back in time and following the movement of the source code, I think these permissions are required since:

• 7.12 for ReplicaSets: Add deployment name in pod's meta beats#23610
• 8.2 for Jobs: Add cronjob metadata by default beats#30637

We need to update our manifests with Filebeat using autodiscover in config/recipes/beats.

[mmentges]

Ran into this issue after upgrading our ECK Test environment from 8.6.2 to 8.9.0 on an Azure Redhat Openshift Cluster 4.11

failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:elastic:filebeat" cannot list resource "replicasets" in API group "apps" at the cluster scope
failed to list *v1.Job: jobs.batch is forbidden: User "system:serviceaccount:elastic:filebeat" cannot list resource "jobs" in API group "batch" at the cluster scope

Modified the cluster role, otherwise filebeat stopped working completely

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  verbs:
  - get
  - watch
  - list
 ###New Rules below
- apiGroups: ["apps"]
  resources: 
  - replicasets
  verbs: 
  - list
  - watch
- apiGroups: ["batch"]
  resources: 
  - jobs
  verbs: 
  - list
  - watch

[thbkrkr]

We also need to update the doc https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-elastic-agent-fleet-quickstart.html.

[barkbay]

I'm closing this one as I think this particular issue is fixed (feel free to reopen if I'm wrong)
New failures are tracked in #7172