Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat] Add googlecloud compute overview dashboard #16819

Merged
merged 13 commits into from
Mar 18, 2020
Merged
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add `redisenterprise` module. {pull}16482[16482] {issue}15269[15269]
- Align fields to ECS and add more tests for the azure module. {issue}16024[16024] {pull}16754[16754]
- Add PubSub metricset to Google Cloud Platform module {pull}15536[15536]
- Add overview dashboard for googlecloud compute. {issue}16534[16534] {pull}16819[16819]
kaiyan-sheng marked this conversation as resolved.
Show resolved Hide resolved

*Packetbeat*

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions metricbeat/docs/modules/googlecloud.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,24 @@ Google Cloud Platform pricing depends of the number of requests you do to their

For example, if Compute Metricset fetches 14 metrics (which is the number of metrics fetched in the early beta version). Each of those metrics will attempt an API call to Compute API to retrieve also their metadata. Because you have 20 different instances, the total number of API calls that will be done on each refresh period are: 14 metrics + 20 instances = 34 API requests every 5 minutes if that is your current Period. 9792 API requests per day with one zone. If you add 2 zones more with the same amount of instances you'll have 19584 API requests per day (9792 on each zone) or around 587520 per month for the Compute Metricset. This maths must be done for each different Metricset with slight variations.

[float]
== Metricsets
Currently, we have `compute`, `loadbalancing`, and `pubsub` metricset in
`googlecloud` module.

[float]
=== `compute`
Compute Metricset to fetch metrics from https://cloud.google.com/compute/[Compute Engine]
Virtual Machines in Google Cloud Platform. The `compute` Metricset contains some of the
metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[Stackdriver API].
Extra labels and metadata are also extracted using the https://cloud.google.com/compute/docs/reference/rest/v1/instances/get[Compute API].
This is enough to get most of the info associated with a metric like compute
labels and metadata and metric specific Labels.

The compute metricset comes with a predefined dashboard:

image::./images/metricbeat-googlecloud-compute-overview.png[]


[float]
=== Example configuration
Expand Down
2 changes: 1 addition & 1 deletion metricbeat/docs/modules_list.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ This file is generated! See scripts/mage/docs_collector.go
|<<metricbeat-module-golang,Golang>> |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.2+| .2+| |<<metricbeat-metricset-golang-expvar,expvar>>
|<<metricbeat-metricset-golang-heap,heap>>
|<<metricbeat-module-googlecloud,Google Cloud Platform>> beta[] |image:./images/icon-no.png[No prebuilt dashboards] |
|<<metricbeat-module-googlecloud,Google Cloud Platform>> beta[] |image:./images/icon-yes.png[Prebuilt dashboards are available] |
.3+| .3+| |<<metricbeat-metricset-googlecloud-compute,compute>> beta[]
|<<metricbeat-metricset-googlecloud-loadbalancing,loadbalancing>> beta[]
|<<metricbeat-metricset-googlecloud-pubsub,pubsub>> beta[]
Expand Down
18 changes: 18 additions & 0 deletions x-pack/metricbeat/module/googlecloud/_meta/docs.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -59,3 +59,21 @@ A recommended `period` value between fetches is between 5 and 10 minutes, depend
Google Cloud Platform pricing depends of the number of requests you do to their API's. Here you have some information that you can use to make an estimation of the pricing you should expect. For example, imagine that you have a Compute Metricset activated and you don't want to exclude labels. You have a total of 20 instances running in a particular GCP project, region and zone.

For example, if Compute Metricset fetches 14 metrics (which is the number of metrics fetched in the early beta version). Each of those metrics will attempt an API call to Compute API to retrieve also their metadata. Because you have 20 different instances, the total number of API calls that will be done on each refresh period are: 14 metrics + 20 instances = 34 API requests every 5 minutes if that is your current Period. 9792 API requests per day with one zone. If you add 2 zones more with the same amount of instances you'll have 19584 API requests per day (9792 on each zone) or around 587520 per month for the Compute Metricset. This maths must be done for each different Metricset with slight variations.

[float]
== Metricsets
Currently, we have `compute`, `loadbalancing`, and `pubsub` metricset in
`googlecloud` module.

[float]
=== `compute`
Compute Metricset to fetch metrics from https://cloud.google.com/compute/[Compute Engine]
Virtual Machines in Google Cloud Platform. The `compute` Metricset contains some of the
kaiyan-sheng marked this conversation as resolved.
Show resolved Hide resolved
metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-compute[Stackdriver API].
Extra labels and metadata are also extracted using the https://cloud.google.com/compute/docs/reference/rest/v1/instances/get[Compute API].
This is enough to get most of the info associated with a metric like compute
labels and metadata and metric specific Labels.

The compute metricset comes with a predefined dashboard:
kaiyan-sheng marked this conversation as resolved.
Show resolved Hide resolved

image::./images/metricbeat-googlecloud-compute-overview.png[]
Loading