Fix Azure Monitor support for multiple aggregation types

[zmoog]

Proposed commit message

Fixes two bugs related to supporting multiple aggregation types.

1. Use aggregation type in MetricRegistry in the cache key.
2. Create one KeyValuePoint for each aggregation type

(1) Use aggregation type in MetricRegistry in the cache key

The MetricRegistry.NeedsUpdate checks if a metric definition needs an update. If there are multiple metrics config with no dimensions:

  resources:
  - resource_query: "resourceType eq 'Microsoft.EventHub/Namespaces'"
    metrics:
    - aggregations:
        - Maximum
      ignore_unsupported: true
      name:
        - ActiveConnections
      namespace: Microsoft.EventHub/Namespaces
    - aggregations:
        - Minimum
      ignore_unsupported: true
      name:
        - ActiveConnections
      namespace: Microsoft.EventHub/Namespaces
    - aggregations:
        - Average
      ignore_unsupported: true
      name:
        - ActiveConnections
      namespace: Microsoft.EventHub/Namespaces

All three metric configs end up having the same key, ending up skipping the metric collection for the second and third configs.

By adding the aggregation type in the cache key, we can correctly keep track of metric updates.

(2) Create one KeyValuePoint for each aggregation type

Add support for multiple aggregation types in the mapToKeyValuePoints() function to restore support for configs like the following:

  resources:
  - resource_query: "resourceType eq 'Microsoft.EventHub/Namespaces'"
    metrics:
    - namespace: Microsoft.EventHub/Namespaces
      name:
        - ActiveConnections
      aggregations:
        - Average
        - Maximum
        - Minimum
      ignore_unsupported: true
      timegrain: PT1M

I missed this feature in a previous change. Unfortunately, no existing modules or integrations use multiple aggregation types, and no tests exist.

We are adding more tests targeting to cover all existing use cases soon.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Cover all multiple aggregation type use cases with tests

How to test this PR locally

Related issues

• Closes Azure Monitor metricset only collect one aggregation type at a time #39192

Use cases

The PR must support multiple aggregation types in two different variants:

• multiple aggregation types in one metric group
• one aggregation type in multiple groups with the same metric

Here are two variants of the metric ActiveConnections with three aggregation types: Maximum, Minimum, and Avergate:

- module: azure
  metricsets:
    - monitor
  enabled: true
  period: 60s
  client_id: '<redacted>'
  client_secret: '<redacted>'
  tenant_id: '<redacted>'
  subscription_id: '<redacted>'
  refresh_list_interval: 600s
  resources:
  - resource_query: "resourceType eq 'Microsoft.EventHub/Namespaces'"
    metrics:
    - aggregations:
        - Maximum
      ignore_unsupported: true
      name:
        - ActiveConnections
      namespace: Microsoft.EventHub/Namespaces
    - aggregations:
        - Minimum
      ignore_unsupported: true
      name:
        - ActiveConnections
      namespace: Microsoft.EventHub/Namespaces
    - aggregations:
        - Average
      ignore_unsupported: true
      name:
        - ActiveConnections
      namespace: Microsoft.EventHub/Namespaces

This variant should be preferred since in needs less API calls to fetch the same data:

- module: azure
  metricsets:
    - monitor
  enabled: true
  period: 60s
  client_id: '<redacted>'
  client_secret: '<redacted>'
  tenant_id: '<redacted>'
  subscription_id: '<redacted>'
  refresh_list_interval: 600s
  resources:
  - resource_query: "resourceType eq 'Microsoft.EventHub/Namespaces'"
    metrics:
    - namespace: Microsoft.EventHub/Namespaces
      name:
        - ActiveConnections
      aggregations:
        - Average
        - Maximum
        - Minimum
      ignore_unsupported: true
      timegrain: PT1M

Screenshots

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 79 min 38 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[gizas]

Wondering if you first save the value of armmonitor.MetricUnit("Count") to a variable and reference here the variable will eliminate this lint error

[zmoog]

Oh, I think you're right.

[gizas]

You can just add some description what each parameter is here used for
Maybe you can update the GetMetricValues description

[zmoog]

Yeah, good point. I'm updating the GetMetricValues description in the interface.

[zmoog]

I noticed there is an unused method in the interface. I'll update the docs for the other methods and remove the unused ones in a cleanup PR.

[gizas]

LGTM and the tests run successfully on myside