Add module for GCP VPC flow logs (#12747)

This module ingests Google Cloud VPC flow logs that have been exported by Stackdriver to a Google Cloud Pub/Sub topic sink.

The published LogEntry objects have this format: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry

The structured data in the flow logs have this format: https://cloud.google.com/vpc/docs/using-flow-logs#record_format

CHANGELOG.next.asciidoc

@@ -223,6 +223,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add support for client addresses with port in Apache error logs {pull}12695[12695]
 - Add `google-pubsub` input type for consuming messages from a Google Cloud Pub/Sub topic subscription. {pull}12746[12746]
 - Add module for ingesting Cisco IOS logs over syslog. {pull}12748[12748]
+- Add module for ingesting Google Cloud VPC flow logs. {pull}12747[12747]
 
 *Heartbeat*
 

filebeat/_meta/fields.common.yml

@@ -150,4 +150,3 @@
       type: keyword
       description: >
         Name of organization associated with the autonomous system.
-

filebeat/docs/fields.asciidoc

@@ -22,6 +22,7 @@ grouped in the following categories:
 * <<exported-fields-ecs>>
 * <<exported-fields-elasticsearch>>
 * <<exported-fields-envoyproxy>>
+* <<exported-fields-googlecloud>>
 * <<exported-fields-haproxy>>
 * <<exported-fields-host-processor>>
 * <<exported-fields-icinga>>
@@ -5512,6 +5513,195 @@ type: keyword
 Envoy proxy type, tcp or http
 
 
+type: keyword
+
+--
+
+[[exported-fields-googlecloud]]
+== Google Cloud fields
+
+Module for handling logs from Google Cloud.
+
+
+
+[float]
+=== googlecloud
+
+Fields from Google Cloud logs.
+
+
+
+[float]
+=== vpcflow
+
+Fields for Google Cloud VPC flow logs.
+
+
+
+*`googlecloud.vpcflow.reporter`*::
++
+--
+The side which reported the flow. Can be either 'SRC' or 'DEST'.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.rtt.ms`*::
++
+--
+Latency as measured (for TCP flows only) during the time interval. This is the time elapsed between sending a SEQ and receiving a corresponding ACK and it contains the network RTT as well as the application related delay.
+
+
+type: long
+
+--
+
+[float]
+=== destination.instance
+
+If the destination of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
+
+
+
+*`googlecloud.vpcflow.destination.instance.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.destination.instance.region`*::
++
+--
+Region of the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.destination.instance.zone`*::
++
+--
+Zone of the VM.
+
+
+type: keyword
+
+--
+
+[float]
+=== destination.vpc
+
+If the destination of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
+
+
+
+*`googlecloud.vpcflow.destination.vpc.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.destination.vpc.vpc_name`*::
++
+--
+VPC on which the VM is operating.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.destination.vpc.subnetwork_name`*::
++
+--
+Subnetwork on which the VM is operating.
+
+
+type: keyword
+
+--
+
+[float]
+=== source.instance
+
+If the source of the connection was a VM located on the same VPC, this field is populated with VM instance details. In a Shared VPC configuration, project_id corresponds to the project that owns the instance, usually the service project.
+
+
+
+*`googlecloud.vpcflow.source.instance.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.source.instance.region`*::
++
+--
+Region of the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.source.instance.zone`*::
++
+--
+Zone of the VM.
+
+
+type: keyword
+
+--
+
+[float]
+=== source.vpc
+
+If the source of the connection was a VM located on the same VPC, this field is populated with VPC network details. In a Shared VPC configuration, project_id corresponds to that of the host project.
+
+
+
+*`googlecloud.vpcflow.source.vpc.project_id`*::
++
+--
+ID of the project containing the VM.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.source.vpc.vpc_name`*::
++
+--
+VPC on which the VM is operating.
+
+
+type: keyword
+
+--
+
+*`googlecloud.vpcflow.source.vpc.subnetwork_name`*::
++
+--
+Subnetwork on which the VM is operating.
+
+
 type: keyword
 
 --

filebeat/docs/modules/googlecloud.asciidoc

@@ -0,0 +1,75 @@
+////
+This file is generated! See scripts/docs_collector.py
+////
+
+[[filebeat-module-googlecloud]]
+[role="xpack"]
+
+:modulename: googlecloud
+:has-dashboards: false
+
+== Google Cloud module
+
+beta[]
+
+This is a module for Google Cloud logs. It supports reading VPC flow logs that
+have been exported from Stackdriver to a Google Pub/Sub topic sink.
+
+include::../include/what-happens.asciidoc[]
+
+include::../include/running-modules.asciidoc[]
+
+include::../include/configuring-intro.asciidoc[]
+
+:fileset_ex: vpcflow
+
+include::../include/config-option-intro.asciidoc[]
+
+[float]
+==== `vpcflow` fileset settings
+
+Example config:
+
+[source,yaml]
+----
+- module: googleclcoud
+  vpcflow:
+    enabled: true
+    var.project_id: my-gcp-project-id
+    var.topic: googlecloud-vpc-flowlogs
+    var.subscription_name: filebeat-googlecloud-vpc-flowlogs-sub
+    var.credentials_file: ${path.config}/gcp-service-account-xyz.json
+----
+
+include::../include/var-paths.asciidoc[]
+
+*`var.project_id`*::
+
+Google Cloud project ID.
+
+*`var.topic`*::
+
+Google Cloud Pub/Sub topic name.
+
+*`var.subscription_name`*::
+
+Google Cloud Pub/Sub topic subscription name. If the subscription does not
+exist it will be created.
+
+*`var.credentials_file`*::
+
+Path to a JSON file containing the credentials and key used to subscribe.
+
+:has-dashboards!:
+
+:fileset_ex!:
+
+:modulename!:
+
+
+[float]
+=== Fields
+
+For a description of each field in the module, see the
+<<exported-fields-googlecloud,exported fields>> section.
+

filebeat/docs/modules_list.asciidoc

@@ -9,6 +9,7 @@ This file is generated! See scripts/docs_collector.py
   * <<filebeat-module-coredns>>
   * <<filebeat-module-elasticsearch>>
   * <<filebeat-module-envoyproxy>>
+  * <<filebeat-module-googlecloud>>
   * <<filebeat-module-haproxy>>
   * <<filebeat-module-icinga>>
   * <<filebeat-module-iis>>
@@ -43,6 +44,7 @@ include::modules/cisco.asciidoc[]
 include::modules/coredns.asciidoc[]
 include::modules/elasticsearch.asciidoc[]
 include::modules/envoyproxy.asciidoc[]
+include::modules/googlecloud.asciidoc[]
 include::modules/haproxy.asciidoc[]
 include::modules/icinga.asciidoc[]
 include::modules/iis.asciidoc[]

filebeat/fileset/fileset.go

@@ -41,6 +41,7 @@ import (
 	"github.com/elastic/beats/libbeat/common/cfgwarn"
 	"github.com/elastic/beats/libbeat/logp"
 	mlimporter "github.com/elastic/beats/libbeat/ml-importer"
+	"github.com/elastic/beats/libbeat/paths"
 )
 
 // Fileset struct is the representation of a fileset.
@@ -352,6 +353,20 @@ func (fs *Fileset) getInputConfig() (*common.Config, error) {
 		return nil, fmt.Errorf("Error reading input config: %v", err)
 	}
 
+	// Additional default settings, that must be available for variable expansion.
+	defaults := common.MustNewConfigFrom(map[string]interface{}{
+		"path": map[string]interface{}{
+			"home":   paths.Paths.Home,
+			"config": "${path.home}",
+			"data":   fmt.Sprint("${path.home}", string(os.PathSeparator), "data"),
+			"logs":   fmt.Sprint("${path.home}", string(os.PathSeparator), "logs"),
+		},
+	})
+
+	if err := cfg.Merge(defaults); err != nil {
+		return nil, err
+	}
+
 	// overrides
 	if len(fs.fcfg.Input) > 0 {
 		overrides, err := common.NewConfigFrom(fs.fcfg.Input)

x-pack/filebeat/filebeat.reference.yml

@@ -167,6 +167,26 @@ filebeat.modules:
     # Filebeat will choose the paths depending on your OS.
     #var.paths:
 
+#----------------------------- Google Cloud Module -----------------------------
+- module: googlecloud
+  vpcflow:
+    enabled: true
+
+    # Google Cloud project ID.
+    var.project_id: my-gcp-project-id
+
+    # Google Pub/Sub topic containing VPC flow logs. Stackdriver must be
+    # configured to use this topic as a sink for VPC flow logs.
+    var.topic: googlecloud-vpc-flowlogs
+
+    # Google Pub/Sub subscription for the topic. Filebeat will create this
+    # subscription if it does not exist.
+    var.subscription_name: filebeat-googlecloud-vpc-flowlogs-sub
+
+    # Credentials file for the service account with authorization to read from
+    # the subscription.
+    var.credentials_file: ${path.config}/gcp-service-account-xyz.json
+
 #------------------------------- Haproxy Module -------------------------------
 - module: haproxy
   # All logs

x-pack/filebeat/module/googlecloud/_meta/config.yml

@@ -0,0 +1,18 @@
+- module: googlecloud
+  vpcflow:
+    enabled: true
+
+    # Google Cloud project ID.
+    var.project_id: my-gcp-project-id
+
+    # Google Pub/Sub topic containing VPC flow logs. Stackdriver must be
+    # configured to use this topic as a sink for VPC flow logs.
+    var.topic: googlecloud-vpc-flowlogs
+
+    # Google Pub/Sub subscription for the topic. Filebeat will create this
+    # subscription if it does not exist.
+    var.subscription_name: filebeat-googlecloud-vpc-flowlogs-sub
+
+    # Credentials file for the service account with authorization to read from
+    # the subscription.
+    var.credentials_file: ${path.config}/gcp-service-account-xyz.json