Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Metricbeat] Add vpc metricset for aws module #16111

Merged
merged 6 commits into from
Feb 27, 2020
Merged

[Metricbeat] Add vpc metricset for aws module #16111

merged 6 commits into from
Feb 27, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Feb 5, 2020 •
edited
Loading

What does this PR do?

This PR is to add support for collecting vpc related monitoring metrics from CloudWatch. This includes AWS/NATGateway, AWS/VPN and AWS/TransitGateway namespaces.

Why is it important?

Users can use this vpc metricset to collect metrics related to their VPCs, for example: bytes received/sent through VPN tunnel, total number of concurrent active TCP connections through NAT gateway, and number of bytes sent/received from transit gateway.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works

Author's Checklist

  • generate data.json
  • dashboard

How to test this PR locally

Make sure the AWS account you use for testing has VPC related metrics first.
Enable vpc metricset and then change aws.yml to:

- module: aws
  period: 1m
  credential_profile_name: elastic-test
  metricsets:
    - vpc

Related issues

#14854

@kaiyan-sheng kaiyan-sheng self-assigned this Feb 5, 2020
@kaiyan-sheng kaiyan-sheng added enhancement Metricbeat Metricbeat needs_backport PR is waiting to be backported to other branches. Team:Platforms Label for the Integrations - Platforms team labels Feb 5, 2020
@kaiyan-sheng
Copy link
Contributor Author

Question: there are three namespaces included in this metricset, should they be split into 3 separate metricsets?
My opinion is they are all VPC related, we can keep them together. But the concern is, by looking at the output events, metricset.name is vpc. But actual metrics are aws.natgateway.metrics.*, aws.transitgateway.metrics.* and aws.vpn.metrics.*.

@kaiyan-sheng kaiyan-sheng marked this pull request as ready for review February 11, 2020 16:03
@kaiyan-sheng kaiyan-sheng requested a review from a team as a code owner February 11, 2020 16:03
@mtojek
Copy link
Contributor

mtojek commented Feb 14, 2020

Question: there are three namespaces included in this metricset, should they be split into 3 separate metricsets?
My opinion is they are all VPC related, we can keep them together. But the concern is, by looking at the output events, metricset.name is vpc. But actual metrics are aws.natgateway.metrics.*, aws.transitgateway.metrics.* and aws.vpn.metrics.*.

I looked into the AWS docs to see if there're mentioned under a common VPC umbrella or separately. Is it possible to add a common key to these metrics? e.g. aws.vpc.vpn.metrics.*, aws.vpc.natgateway.metrics.*

mtojek
mtojek approved these changes Feb 14, 2020
View reviewed changes
x-pack/metricbeat/module/aws/vpc/vpc_test.go
import (
"os"

"github.com/elastic/beats/metricbeat/mb"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't that be together with other beats imports?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other two beats import are for registering input module and metricset 😄 So I left them separate with a comment.

@kaiyan-sheng
Copy link
Contributor Author

I looked into the AWS docs to see if there're mentioned under a common VPC umbrella or separately. Is it possible to add a common key to these metrics? e.g. aws.vpc.vpn.metrics.*, aws.vpc.natgateway.metrics.*

Thanks for your review!! Here is the AWS doc that mentioned these three namespaces under VPC: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/aws-services-cloudwatch-metrics.html
Hmm since this metricset is using light module to create, vpn/natgateway/transitgateway are from the given namespace. Similarly, for ELBs, metrics are started with aws.elb, aws.applicationelb and aws.networkelb.

@kaiyan-sheng kaiyan-sheng merged commit 96100df into elastic:master Feb 27, 2020
@kaiyan-sheng kaiyan-sheng deleted the vpc_mb branch February 27, 2020 22:05
@kaiyan-sheng kaiyan-sheng mentioned this pull request Feb 27, 2020
Merged
7 tasks
@kaiyan-sheng kaiyan-sheng added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Feb 27, 2020
blakerouse pushed a commit to blakerouse/beats that referenced this pull request Feb 28, 2020
@kaiyan-sheng @blakerouse
[Metricbeat] Add vpc metricset for aws module (elastic#16111)
6e79791 
* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway
kaiyan-sheng added a commit that referenced this pull request Feb 28, 2020
@kaiyan-sheng
[Metricbeat] Add vpc metricset for aws module (#16111) (#16677)
2f48b2d 
* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway

(cherry picked from commit 96100df)
blakerouse added a commit that referenced this pull request Mar 2, 2020
@blakerouse @ycombinator
@dedemorton @ph @faec @kaiyan-sheng @jsoriano @mtojek
Add cloudfoundry module to metricbeat (#16671)
07b03a7 
* Split up parts into different metricsets.

* Add value and container metricsets.

* Add documentation.

* Revert metricbeat.yml.

* Run make update.

* Fix documentation add changelog.

* Fix bad change in azure module.

* Run make update.

* Improve reference config.

* Document Kibana Space ID setting (#16622)

* Document Kibana Space ID setting

* Update reference YML files

* Update libbeat/docs/shared-kibana-config.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Updating reference YML files

* Adding link to doc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

* Remove the special pins type for `ca_sha256` (#16615)

We have changed the type from a special custom type to a slice of
string. This reduce the number of exposed types and make it easier in
general to work with in the tests.

* Remove feature.Describer and enhance Details (#16658)

* Make memqueue.Broker internal (now memqueue.broker) (#16667)

* [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584)

* [Metricbeat] Add vpc metricset for aws module (#16111)

* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway

* Update cisco.asciidoc (#16434)

* Update cisco.asciidoc

Making Elasticsearch explicit in the instructions

* Update docs source in the module dir

Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

* Fix wording in dev guide (#16497)

* Added NetBeat (#12612)

* Close files earlier when generating fields files (#16681)

Fields generator keeps all open files while generating the global
fields.yml file. This is not needed and may lead to "too many open
files" errors in some environments.

* [Metricbeat] Add new module for Redis Enterprise (#16482)

Add metricsets: node, proxy

* Add doppler_address, uaa_address, rlp_address to config.reference.yml.

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Loek van Gool <loek.vangool@elastic.co>
Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
blakerouse added a commit to blakerouse/beats that referenced this pull request Mar 3, 2020
@blakerouse
Add cloudfoundry module to metricbeat (elastic#16671)
44e4375 
* Split up parts into different metricsets.

* Add value and container metricsets.

* Add documentation.

* Revert metricbeat.yml.

* Run make update.

* Fix documentation add changelog.

* Fix bad change in azure module.

* Run make update.

* Improve reference config.

* Document Kibana Space ID setting (elastic#16622)

* Document Kibana Space ID setting

* Update reference YML files

* Update libbeat/docs/shared-kibana-config.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Updating reference YML files

* Adding link to doc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

* Remove the special pins type for `ca_sha256` (elastic#16615)

We have changed the type from a special custom type to a slice of
string. This reduce the number of exposed types and make it easier in
general to work with in the tests.

* Remove feature.Describer and enhance Details (elastic#16658)

* Make memqueue.Broker internal (now memqueue.broker) (elastic#16667)

* [docs] Add skeleton files for 7.7 release highlights and breaking changes (elastic#16584)

* [Metricbeat] Add vpc metricset for aws module (elastic#16111)

* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway

* Update cisco.asciidoc (elastic#16434)

* Update cisco.asciidoc

Making Elasticsearch explicit in the instructions

* Update docs source in the module dir

Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

* Fix wording in dev guide (elastic#16497)

* Added NetBeat (elastic#12612)

* Close files earlier when generating fields files (elastic#16681)

Fields generator keeps all open files while generating the global
fields.yml file. This is not needed and may lead to "too many open
files" errors in some environments.

* [Metricbeat] Add new module for Redis Enterprise (elastic#16482)

Add metricsets: node, proxy

* Add doppler_address, uaa_address, rlp_address to config.reference.yml.

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Loek van Gool <loek.vangool@elastic.co>
Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
(cherry picked from commit 07b03a7)
blakerouse added a commit that referenced this pull request Mar 3, 2020
@blakerouse
Cherry-pick #16671 to 7.x: Add cloudfoundry module to metricbeat (#16741
5f7c5a6 
)

* Add cloudfoundry module to metricbeat (#16671)

* Split up parts into different metricsets.

* Add value and container metricsets.

* Add documentation.

* Revert metricbeat.yml.

* Run make update.

* Fix documentation add changelog.

* Fix bad change in azure module.

* Run make update.

* Improve reference config.

* Document Kibana Space ID setting (#16622)

* Document Kibana Space ID setting

* Update reference YML files

* Update libbeat/docs/shared-kibana-config.asciidoc

Co-Authored-By: DeDe Morton <dede.morton@elastic.co>

* Updating reference YML files

* Adding link to doc

Co-authored-by: DeDe Morton <dede.morton@elastic.co>

* Remove the special pins type for `ca_sha256` (#16615)

We have changed the type from a special custom type to a slice of
string. This reduce the number of exposed types and make it easier in
general to work with in the tests.

* Remove feature.Describer and enhance Details (#16658)

* Make memqueue.Broker internal (now memqueue.broker) (#16667)

* [docs] Add skeleton files for 7.7 release highlights and breaking changes (#16584)

* [Metricbeat] Add vpc metricset for aws module (#16111)

* Add vpn into vpc metricset
* Add NATGateway and TrasitGateway

* Update cisco.asciidoc (#16434)

* Update cisco.asciidoc

Making Elasticsearch explicit in the instructions

* Update docs source in the module dir

Co-authored-by: Loek van Gool <loek.vangool@elastic.co>

* Fix wording in dev guide (#16497)

* Added NetBeat (#12612)

* Close files earlier when generating fields files (#16681)

Fields generator keeps all open files while generating the global
fields.yml file. This is not needed and may lead to "too many open
files" errors in some environments.

* [Metricbeat] Add new module for Redis Enterprise (#16482)

Add metricsets: node, proxy

* Add doppler_address, uaa_address, rlp_address to config.reference.yml.

Co-authored-by: Shaunak Kashyap <ycombinator@gmail.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Pier-Hugues Pellerin <phpellerin@gmail.com>
Co-authored-by: Steffen Siering <steffen.siering@elastic.co>
Co-authored-by: Fae Charlton <fae.charlton@elastic.co>
Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
Co-authored-by: Loek van Gool <loek.vangool@elastic.co>
Co-authored-by: Harold Schreckengost <harold@haroldmschreckengost.com>
Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Marcin Tojek <mtojek@users.noreply.github.com>
(cherry picked from commit 07b03a7)

* Fix changelog.

* Run mage fmt.

* Fix imports to have v7.

* Run mage fmt update.

* Fix changelog.
@kaiyan-sheng
Copy link
Contributor Author

This will be tested by #16892.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mtojek mtojek mtojek approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
enhancement Metricbeat Metricbeat Team:Platforms Label for the Integrations - Platforms team v7.7.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@kaiyan-sheng @mtojek @andresrc