[Metricbeat] gcp: add GKE metricset

[endorama]

What does this PR do?

Add a lightweight module to pull GCP GKE metrics.

Metrics gathered are all GA metrics as of today available for Kubernetes on GCP.

GCP Docs: https://cloud.google.com/monitoring/api/metrics_kubernetes

Why is it important?

Implement GKE integration as requested in elastic/integrations#816

Dates

• Development Start date: Tue, Aug 17 2021
• Feature Freeze date: Wed, Aug 18 2021
• Release date: Tue, Sep 21 2021

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

You can build and run the metricbeat binary from this PR to collect GKE metrics. You may use this configuration for gcp.yml module:

- module: gcp
  metricsets:
    - gke
  zone: "us-"
  project_id: "<fill>"
  credentials_file_path: "<fill>"
  exclude_labels: false
  period: 1m

Related issues

• Closes [Google Cloud] Add GKE integration integrations#816
• Requires [Metricbeat] gcp: allow service metric prefix override #26960
• Requires [Metricbeat] gcp: refactor service based metric filter creation #26973
• Requires [Metricbeat] gcp: use ServiceMetrixPrefix in field extractor too #27286
• Relates [Metricbeat] gcp: fields do not respect naming best practices #27231
  NOTE: due to this issue the fields do not align with Beats naming convention or ECS, to keep the behaviour across Metricsets consistent. Once this is addressed I'll tackle migrating all Metricsets to the correct format.

Use cases

Feature: GKE metric collection

  GKE is the dedicated Kubernetes distribution from GCP. It exposes a set of dedicated 
  metrics useful for Kubernetes operators and developers.

  Scenario: Collect GKE specific metrics
	Given that gke metricset collection has been enabled and configured
    When running metricbeat
	Then I receive GKE specific metrics in Elasticsearch

Screenshots

Logs

Metricbeat Module / Dataset release checklist

This checklist is intended for Devs which create or update a module to make sure modules are consistent.

Modules

For a metricset to go GA, the following criterias should be met:

gcp module is currently in beta.

Metricbeat module

• Example data.json exists and an automated way to generate it exists (go test -data)
• Test environment in Docker exist for integration tests

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-08-17T21:06:04.733+0000

• Duration: 101 min 29 sec

• Commit: 9dbbbaa

Test stats 🧪

Test	Results
Failed	0
Passed	9495
Skipped	2519
Total	12014

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	9495
Skipped	2519
Total	12014

[mtojek]

This is a good start, keep pushing the missing items. You can copy the TODO list from the template: https://github.com/elastic/beats/issues/new?assignees=&labels=&template=module-checklist.md

I think that the next step is to make it working and passing lint.

[mtojek]

To review available metrics, you can sync with @sorantis .

[mtojek]

I think we used pct for percentages. See:

  - name: memory.broker.pct
    description: The percentage of the memory limit used.
    type: scaled_float
    format: percent

Sometimes you need an additional Javascript processor to convert to the right format:

processors:
  - script:
      lang: javascript
      source: >
        function process(event) {
          var broker_memory_broker_pct = event.Get("activemq.broker.memory.broker.pct")
          if (broker_memory_broker_pct != null) {
            event.Put("activemq.broker.memory.broker.pct", broker_memory_broker_pct / 100.0)
          }

          var broker_memory_temp_pct = event.Get("activemq.broker.memory.temp.pct")
          if (broker_memory_temp_pct != null) {
            event.Put("activemq.broker.memory.temp.pct", broker_memory_temp_pct / 100.0)
          }

          var broker_memory_store_pct = event.Get("activemq.broker.memory.store.pct")
          if (broker_memory_store_pct != null) {
            event.Put("activemq.broker.memory.store.pct", broker_memory_store_pct / 100.0)
          }
        }

(based on the ActiveMQ module.

[mtojek]

Hm.. empty name?

[mtojek]

Try to follow the Beats naming convention: https://www.elastic.co/guide/en/beats/devguide/current/event-conventions.html

[endorama]

Should be max_bytes or allocatable.bytes?

[mtojek]

In this case - memory.allocable.bytes. But this comment applies to all fields you submitted.

[endorama]

:) Will update all occurrences

[kaiyan-sheng]

I see this PR is still in draft, just want to bring elastic/ecs#1441 to your attention. As a part of the inventory schema work, we are planning to add some additional container metrics. It would be good if you can adopt these fields in this new metricset 🙂

[endorama]

To keep consistency with the GCP documentation and implementation, I'll rename gke to kubernetes. It's the name used in the GCP docs so I think the consistency is a must here.

[endorama]

Found a bug in #26960, fix split in #27286

[endorama]

Added GKE overview dashboard. Preview:

[endorama]

#27416 has been merged

Rebasing onto master and triggering a new CI check.

[kaiyan-sheng]

LGTM! Feel free to ship the dashboard separately.

[kaiyan-sheng]

When shipping a dashboard, please also add a screenshot into this folder under Metricbeat. Also reference the dashboard screenshot in the documentation.

[endorama]

I removed the dashboard... Now waiting on the CI