[Metricbeat] add more metric collection into pubsub metricset

CHANGELOG.next.asciidoc

@@ -151,6 +151,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix vsphere VM dashboard host aggregation visualizations. {pull}17555[17555]
 - Fix azure storage dashboards. {pull}17590[17590]
 - Metricbeat no longer needs to be started strictly after Logstash for `logstash-xpack` module to report correct data. {issue}17261[17261] {pull}17497[17497]
+- Fix pubsub metricset to collect all GA stage metrics from gcp stackdriver. {issue}17154[17154] {pull}17600[17600]
 - Add privileged option so as mb to access data dir in Openshift. {pull}17606[17606]
 - Fix "ID" event generator of Google Cloud module {issue}17160[17160] {pull}17608[17608]
 

x-pack/metricbeat/module/googlecloud/pubsub/_meta/data.json

@@ -14,14 +14,34 @@
     "googlecloud": {
         "labels": {
             "resource": {
-                "topic_id": "test-ks"
+                "subscription_id": "test-ks"
             }
         },
         "pubsub": {
+            "snapshot": {
+                "backlog_bytes": 19,
+                "backlog_bytes_by_region": 19,
+                "num_messages": 4,
+                "num_messages_by_region": 4,
+                "oldest_message_age": 69319,
+                "oldest_message_age_by_region": 69319
+            },
             "subscription": {
-                "backlog_bytes": 20,
-                "num_undelivered_messages": 4,
-                "oldest_unacked_message_age": 435752
+                "backlog_bytes": 0,
+                "num_undelivered_messages": 0,
+                "oldest_retained_acked_message_age": 0,
+                "oldest_retained_acked_message_age_by_region": 0,
+                "oldest_unacked_message_age": 0,
+                "oldest_unacked_message_age_by_region": 69277,
+                "retained_acked_bytes": 0,
+                "retained_acked_bytes_by_region": 0,
+                "unacked_bytes_by_region": 19
+            },
+            "topic": {
+                "oldest_retained_acked_message_age_by_region": 0,
+                "oldest_unacked_message_age_by_region": 69319,
+                "retained_acked_bytes_by_region": 0,
+                "unacked_bytes_by_region": 76
             }
         }
     },

x-pack/metricbeat/module/googlecloud/pubsub/_meta/docs.asciidoc

@@ -1,25 +1,61 @@
 PubSub Metricset to fetch metrics from https://cloud.google.com/pubsub/[Pub/Sub] topics and subscriptions in Google Cloud Platform.
 
-The `pubsub` Metricset contains all metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub[Stackdriver API]. The field names have been left untouched for people already familiar with them.
+The `pubsub` Metricset contains all GA stage metrics exported from the https://cloud.google.com/monitoring/api/metrics_gcp#gcp-pubsub[Stackdriver API]. The field names have been left untouched for people already familiar with them.
 
 No special permissions are needed apart from the ones detailed in the module section of the docs.
 
+[float]
+==== Snapshot Metrics
+- `pubsub.snapshot.backlog_bytes`: Total byte size of the messages retained in a snapshot.
+- `pubsub.snapshot.backlog_bytes_by_region`: Total byte size of the messages retained in a snapshot, broken down by Cloud region.
+- `pubsub.snapshot.config_updates_count`: Cumulative count of configuration changes, grouped by operation type and result.
+- `pubsub.snapshot.num_messages`: Number of messages retained in a snapshot. Sampled every 60 seconds.
+- `pubsub.snapshot.num_messages_by_region`: Number of messages retained in a snapshot, broken down by Cloud region.
+- `pubsub.snapshot.oldest_message_age`: Age (in seconds) of the oldest message retained in a snapshot.
+- `pubsub.snapshot.oldest_message_age_by_region`: Age (in seconds) of the oldest message retained in a snapshot, broken down by Cloud region.
+
+[float]
+==== Subscription Metrics
 - `pubsub.subscription.ack_message_count`: Cumulative count of messages acknowledged by Acknowledge requests, grouped by delivery type.
 - `pubsub.subscription.backlog_bytes`: Total byte size of the unacknowledged messages (a.k.a. backlog messages) in a subscription.
+- `pubsub.subscription.byte_cost`: Cumulative cost of operations, measured in bytes. This is used to measure quota utilization.
+- `pubsub.subscription.config_updates_count`: Cumulative count of configuration changes for each subscription, grouped by operation type and result.
+- `pubsub.subscription.dead_letter_message_count`: Cumulative count of messages published to dead letter topic, grouped by result.
+- `pubsub.subscription.mod_ack_deadline_message_count`: Cumulative count of messages whose deadline was updated by ModifyAckDeadline requests, grouped by delivery type.
+- `pubsub.subscription.mod_ack_deadline_message_operation_count`: Cumulative count of ModifyAckDeadline message operations, grouped by result.
+- `pubsub.subscription.mod_ack_deadline_request_count`: Cumulative count of ModifyAckDeadline requests, grouped by result.
 - `pubsub.subscription.num_outstanding_messages`: Number of messages delivered to a subscription's push endpoint, but not yet acknowledged.
 - `pubsub.subscription.num_undelivered_messages`: Number of unacknowledged messages (a.k.a. backlog messages) in a subscription.
+- `pubsub.subscription.oldest_retained_acked_message_age`: Age (in seconds) of the oldest acknowledged message retained in a subscription.
+- `pubsub.subscription.oldest_retained_acked_message_age_by_region`: Age (in seconds) of the oldest acknowledged message retained in a subscription, broken down by Cloud region.
 - `pubsub.subscription.oldest_unacked_message_age`: Age (in seconds) of the oldest unacknowledged message (a.k.a. backlog message) in a subscription.
+- `pubsub.subscription.oldest_unacked_message_age_by_region`: Age (in seconds) of the oldest unacknowledged message in a subscription, broken down by Cloud region.
 - `pubsub.subscription.pull_ack_message_operation_count`: Cumulative count of acknowledge message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
 - `pubsub.subscription.pull_ack_request_count`: Cumulative count of acknowledge requests, grouped by result.
 - `pubsub.subscription.pull_message_operation_count`: Cumulative count of pull message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
 - `pubsub.subscription.pull_request_count`: Cumulative count of pull requests, grouped by result.
 - `pubsub.subscription.push_request_count`: Cumulative count of push attempts, grouped by result. Unlike pulls, the push server implementation does not batch user messages. So each request only contains one user message. The push server retries on errors, so a given user message can appear multiple times.
 - `pubsub.subscription.push_request_latencies`: Distribution of push request latencies (in microseconds), grouped by result.
+- `pubsub.subscription.retained_acked_bytes`: otal byte size of the acknowledged messages retained in a subscription.
+- `pubsub.subscription.retained_acked_bytes_by_region`: Total byte size of the acknowledged messages retained in a subscription, broken down by Cloud region.
+- `pubsub.subscription.seek_request_count`: Cumulative count of seek attempts, grouped by result.
 - `pubsub.subscription.sent_message_count`: Cumulative count of messages sent by Cloud Pub/Sub to subscriber clients, grouped by delivery type.
 - `pubsub.subscription.streaming_pull_ack_message_operation_count`: Cumulative count of StreamingPull acknowledge message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
 - `pubsub.subscription.streaming_pull_ack_request_count`: Cumulative count of streaming pull requests with non-empty acknowledge ids, grouped by result.
 - `pubsub.subscription.streaming_pull_message_operation_count`: Cumulative count of streaming pull message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric <code>subscription/mod_ack_deadline_message_operation_count
+- `pubsub.subscription.streaming_pull_mod_ack_deadline_message_operation_count`: Cumulative count of StreamingPull ModifyAckDeadline operations, grouped by result.
+- `pubsub.subscription.streaming_pull_mod_ack_deadline_request_count`: Cumulative count of streaming pull requests with non-empty ModifyAckDeadline fields, grouped by result.
 - `pubsub.subscription.streaming_pull_response_count`: Cumulative count of streaming pull responses, grouped by result.
-- `pubsub.topic.streaming_pull_response_count`: Cumulative count of streaming pull responses, grouped by result.
+- `pubsub.subscription.unacked_bytes_by_region`: Total byte size of the unacknowledged messages in a subscription, broken down by Cloud region.
+
+[float]
+==== Topic Metrics
+- `pubsub.topic.byte_cost`: Cost of operations, measured in bytes. This is used to measure utilization for quotas.
+- `pubsub.topic.config_updates_count`: Cumulative count of configuration changes, grouped by operation type and result.
+- `pubsub.topic.message_sizes`: Distribution of publish message sizes (in bytes).
+- `pubsub.topic.oldest_retained_acked_message_age_by_region`: Age (in seconds) of the oldest acknowledged message retained in a topic, broken down by Cloud region.
+- `pubsub.topic.oldest_unacked_message_age_by_region`: Age (in seconds) of the oldest unacknowledged message in a topic, broken down by Cloud region.
+- `pubsub.topic.retained_acked_bytes_by_region`: Total byte size of the acknowledged messages retained in a topic, broken down by Cloud region.
 - `pubsub.topic.send_message_operation_count`: Cumulative count of publish message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count.
 - `pubsub.topic.send_request_count`: Cumulative count of publish requests, grouped by result.
+- `pubsub.topic.unacked_bytes_by_region`: Total byte size of the unacknowledged messages in a topic, broken down by Cloud region.