Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow namespace: "*" for cloudwatch metricset #14971

Merged
merged 6 commits into from
Dec 15, 2019
Merged

Allow namespace: "*" for cloudwatch metricset #14971

merged 6 commits into from
Dec 15, 2019

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Dec 5, 2019

I tried to run cloudwatch metricset with namespace: "*" and ListMetrics API did what we expected: it returned all metrics exists in that AWS account and was able to use to obtain statistical data.

- module: aws
  period: 300s
  credential_profile_name: elastic-test
  metricsets:
    - cloudwatch
  metrics:
    - namespace: "*"
      statistic: ["Average"]

Here is what cloudwatch metricset collected:
Screen Shot 2019-12-05 at 1 42 46 PM

The problem is: statistic and period are defined for all metrics. For example, AWS/Usage metrics should be collected every 1 minute instead of every 5 minutes(data loss in this case). Also some metrics like AWS/Billing, should be collected every 6 hours or 12 hours instead of every 5 minutes(introduced extra cost in this case).

The limitations are documented in doc under cloudwatch metricset.
closes #14965

How to test this

In your AWS account, please make sure the services are from more than one namespace.
Enable aws module in Metricbeat and change configuration file aws.yml to be:

- module: aws
  period: 300s
  credential_profile_name: elastic-test
  metricsets:
    - cloudwatch
  metrics:
    - namespace: "*"
      statistic: ["Average"]

Then start Metricbeat. You should be able to see metrics from all existing services from all different namespaces(with only *.avg metrics).

@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review December 6, 2019 20:54
@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner December 6, 2019 20:55
@kaiyan-sheng kaiyan-sheng added Metricbeat Metricbeat Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan needs_backport PR is waiting to be backported to other branches. review labels Dec 6, 2019
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Does it mean we can deprecate all other modules built upon CloudWatch?

@kaiyan-sheng kaiyan-sheng merged commit 83b309d into elastic:master Dec 15, 2019
@kaiyan-sheng kaiyan-sheng deleted the use_wildcard_namespace branch December 15, 2019 17:23
@kaiyan-sheng kaiyan-sheng added v7.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Dec 16, 2019
kaiyan-sheng added a commit that referenced this pull request Dec 16, 2019
…set (#15119)

* Allow namespace: "*" for cloudwatch metricset (#14971)

* Allow namespace: "*" for cloudwatch metricset

(cherry picked from commit 83b309d)

* update changelog
@kaiyan-sheng kaiyan-sheng removed their assignment Jan 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Metricbeat Metricbeat review Team:Integrations Label for the Integrations team test-plan Add this PR to be manual test plan test-plan-ok This PR passed manual testing v7.6.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Check AWS ListMetrics API call without giving namespace
5 participants