Add module for GCP VPC flow logs #12747
Conversation
|
Pinging @elastic/secops |
andrewkroh
force-pushed
the
feature/fb/google-vpc-flowlogs-module
branch
from
a89ffa5 to
83027d7
Compare
This module ingests Google Cloud VPC flow logs that have been exported by Stackdriver to a Google Cloud Pub/Sub topic sink. The published LogEntry objects have this format: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry The structured data in the flow logs have this format: https://cloud.google.com/vpc/docs/using-flow-logs#record_format
andrewkroh
force-pushed
the
feature/fb/google-vpc-flowlogs-module
branch
from
83027d7 to
e27540b
Compare
|
This is smaller now and easier to review. I rebased it to remove all of other merged PRs that this was built upon. |
|
|
||
| *`var.credentials_file`*:: | ||
|
|
||
| Path to a JSON file containing the credentials and key used to subscribe. |
There was a problem hiding this comment.
Not for this PR, but eventually we'll want to support the secrets store for this?
There was a problem hiding this comment.
Yeah, the underlying input supports the keystore via credentials_json, where you can embed the whole json blob as string/bytes in the config this way the keystore is supported. But I didn't expose the param in the module, but probably should in the future.
|
|
||
| # IP Autonomous System (AS) Lookup | ||
| - geoip: | ||
| database_file: GeoLite2-ASN.mmdb |
There was a problem hiding this comment.
Do these come by default with the maxmind installation? Will it degrade somewhat gracefully if the files are not found?
There was a problem hiding this comment.
They come with Elasticsearch by default. (source)
The database filename in the geoip config directory. The ingest-geoip module ships with the GeoLite2-City.mmdb, GeoLite2-Country.mmdb and GeoLite2-ASN.mmdb files.
I assume the pipeline would fail if someone removed them.
This module ingests Google Cloud VPC flow logs that have been exported by Stackdriver to a Google Cloud Pub/Sub topic sink.
The published LogEntry objects have this format: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
The structured data in the flow logs have this format: https://cloud.google.com/vpc/docs/using-flow-logs#record_format