Add more metrics collected by stack monitoring

[kvalliyurnatt]

This PR adds some of the missing metricsets to the stack monitoring configuration
fixes: #7277

[naemono]

I think it's good, but we may want to note if there's a version requirement, see: #7277 (comment)

Also, run the full e2e test suite on this to make sure all versions are ok with this change.... I suspect older versions may fail.

[naemono]

buildkite test this -f p=gke -m s=7.17.8,s=8.15.0,s=8.16.0-SNAPSHOT

[naemono]

Also, run the full e2e test suite on this to make sure all versions are ok with this change.... I suspect older versions may fail.

Update: I kicked them off for you....

[pebrc]

Yes I think we need a version gate for this.

[pebrc]

Just to capture a discussion we had out of band: the change required here needs to amend the Metricbeat template for the ECK stack monitoring feature. We can also add it to the recipe as done here but the template is the main part and we can use conditionals etc there to make it version dependent

cloud-on-k8s/pkg/controller/elasticsearch/stackmon/metricbeat.tpl.yml

Lines 4 to 14 in bf5c373

	metricsets:
	- ccr
	- cluster_stats
	- enrich
	- index
	- index_recovery
	- index_summary
	- ml_job
	- node_stats
	- pending_tasks
	- shard

[kvalliyurnatt]

Added a version gate for the ingest pipeline and also added a unit test for it

[kvalliyurnatt]

buildkite test this -f p=gke -m s=7.17.8,s=8.15.0,s=8.16.0-SNAPSHOT

[kvalliyurnatt]

I tested this locally on my dev cluster
Deployed an Elasticsearch version of 8.3.3

metricbeat:
  modules:
    - hosts:
        - https://localhost:9200/
      metricsets:
        - ccr
        - cluster_stats
        - enrich
        - index
        - index_recovery
        - index_summary
        - ml_job
        - node
        - node_stats
        - pending_tasks
        - shard
        - pending_tasks
      module: elasticsearch
      password: ""
      period: 10s
      ssl:
        certificate_authorities:
          - /mnt/elastic-internal/kb-monitoring/default/source/certs/ca.crt
        enabled: true
        verification_mode: certificate
      username: elastic-internal-monitoring
      xpack:
        enabled: true
output:
  elasticsearch:
    hosts:
      - https://destination-es-http.default.svc:9200/
    password: ""
    ssl:
      certificate_authorities:
        - /mnt/elastic-internal/es-monitoring-association/default/destination/certs/ca.crt
      verification_mode: certificate
    username: default-source-default-destination-beat-es-mon-user
processors:
  - add_cloud_metadata: null
  - add_host_metadata: null

no ingest pipeline in the metric set

deployed Elasticsearch 8.7.0, ingest pipeline present in the metricset

metricbeat:
  modules:
    - hosts:
        - https://localhost:9200/
      metricsets:
        - ccr
        - cluster_stats
        - enrich
        - index
        - index_recovery
        - index_summary
        - ml_job
        - node
        - node_stats
        - pending_tasks
        - shard
        - ingest_pipeline
        - pending_tasks
      module: elasticsearch
      password: ""
      period: 10s
      ssl:
        certificate_authorities:
          - /mnt/elastic-internal/kb-monitoring/default/source/certs/ca.crt
        enabled: true
        verification_mode: certificate
      username: elastic-internal-monitoring
      xpack:
        enabled: true
output:
  elasticsearch:
    hosts:
      - https://destination-es-http.default.svc:9200/
    password: ""
    ssl:
      certificate_authorities:
        - /mnt/elastic-internal/es-monitoring-association/default/destination/certs/ca.crt
      verification_mode: certificate
    username: default-source-default-destination-beat-es-mon-user
processors:
  - add_cloud_metadata: null
  - add_host_metadata: null

[naemono]

looks good to me.

[thbkrkr]

Oh! (I'm so surprised I missed this initially.)

[thbkrkr]

LGTM