Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add EC2, GCE, or DigitalOcean metadata to events #2728

Merged
merged 1 commit into from
Oct 13, 2016
Merged

Add EC2, GCE, or DigitalOcean metadata to events #2728

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ https://github.com/elastic/beats/compare/v5.0.0-rc1...master[Check the HEAD diff

*Affecting all Beats*
- Add beat.version fields to all events.
- Add add_cloud_metadata processor for collecting cloud provider metadata. {pull}2728[2728]

*Metricbeat*

Expand Down
52 changes: 52 additions & 0 deletions filebeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This document describes the fields that are exported by Filebeat. They are
grouped in the following categories:

* <<exported-fields-beat>>
* <<exported-fields-cloud>>
* <<exported-fields-log>>

--
Expand Down Expand Up @@ -69,6 +70,57 @@ type: dict
Contains user configurable fields.


[[exported-fields-cloud]]
== Cloud Provider Metadata Fields

Metadata from cloud providers added by the add_cloud_metadata processor.



[float]
=== meta.cloud.provider

example: ec2

Name of the cloud provider. Possible values are ec2, gce, or digitalocean.


[float]
=== meta.cloud.instance_id

Instance ID of the host machine.


[float]
=== meta.cloud.machine_type

example: t2.medium

Machine type of the host machine.


[float]
=== meta.cloud.availability_zone

example: us-east-1c

Availability zone in which this host is running.


[float]
=== meta.cloud.project_id

example: project-x

Name of the project in Google Cloud.


[float]
=== meta.cloud.region

Region in which this host is running.


[[exported-fields-log]]
== Log File Content Fields

Expand Down
2 changes: 1 addition & 1 deletion filebeat/etc/kibana/index-pattern/filebeat.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"fields": "[{\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"beat.name\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"beat.hostname\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"beat.version\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"@timestamp\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"date\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"tags\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"fields\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"source\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"offset\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"number\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": false, \"name\": \"message\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"type\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"input_type\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}]",
"fields": "[{\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"beat.name\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"beat.hostname\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"beat.version\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"@timestamp\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"date\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"tags\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"fields\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"meta.cloud.provider\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"meta.cloud.instance_id\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"meta.cloud.machine_type\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"meta.cloud.availability_zone\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"meta.cloud.project_id\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"meta.cloud.region\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"source\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"offset\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"number\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": false, \"name\": \"message\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"type\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}, {\"count\": 0, \"analyzed\": false, \"aggregatable\": true, \"name\": \"input_type\", \"searchable\": true, \"indexed\": true, \"doc_values\": true, \"type\": \"string\", \"scripted\": false}]",
"fieldFormatMap": "{\"@timestamp\": {\"id\": \"date\"}}",
"timeFieldName": "@timestamp",
"title": "filebeat-*"
Expand Down
38 changes: 23 additions & 15 deletions filebeat/filebeat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,7 +235,7 @@ filebeat.prospectors:
# Default is 0, not waiting.
#filebeat.shutdown_timeout: 0

#================================ General =====================================
#================================ General ======================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
Expand Down Expand Up @@ -269,19 +269,21 @@ filebeat.prospectors:
# default is the number of logical CPUs available in the system.
#max_procs:

#================================ Processors =====================================
#================================ Processors ===================================

# Processors are used to reduce the number of fields in the exported event or to
# enhance the event with external meta data. This section defines a list of processors
# that are applied one by one and the first one receives the initial event:
# enhance the event with external metadata. This section defines a list of
# processors that are applied one by one and the first one receives the initial
# event:
#
# event -> filter1 -> event1 -> filter2 ->event2 ...
#
# Supported processors: drop_fields, drop_event, include_fields
# The supported processors are drop_fields, drop_event, include_fields, and
# add_cloud_metadata.
#
# For example, you can use the following processors to keep
# the fields that contain CPU load percentages, but remove the fields that
# contain CPU ticks values:
# For example, you can use the following processors to keep the fields that
# contain CPU load percentages, but remove the fields that contain CPU ticks
# values:
#
#processors:
#- include_fields:
Expand All @@ -297,13 +299,19 @@ filebeat.prospectors:
# equals:
# http.code: 200
#
# The following example enriches each event with metadata from the cloud
# provider about the host machine. It works on EC2, GCE, and DigitalOcean.
#
#processors:
#- add_cloud_metadata:
#

#================================ Outputs =====================================
#================================ Outputs ======================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
# Boolean flag to enable or disable the output module.
#enabled: true
Expand Down Expand Up @@ -417,7 +425,7 @@ output.elasticsearch:
#ssl.curve_types: []


#----------------------------- Logstash output --------------------------------
#----------------------------- Logstash output ---------------------------------
#output.logstash:
# Boolean flag to enable or disable the output module.
#enabled: true
Expand Down Expand Up @@ -480,7 +488,7 @@ output.elasticsearch:
# Configure curve types for ECDHE based cipher suites
#ssl.curve_types: []

#------------------------------- Kafka output ---------------------------------
#------------------------------- Kafka output ----------------------------------
#output.kafka:
# Boolean flag to enable or disable the output module.
#enabled: true
Expand Down Expand Up @@ -615,7 +623,7 @@ output.elasticsearch:
# Configure curve types for ECDHE based cipher suites
#ssl.curve_types: []

#------------------------------- Redis output ---------------------------------
#------------------------------- Redis output ----------------------------------
#output.redis:
# Boolean flag to enable or disable the output module.
#enabled: true
Expand Down Expand Up @@ -713,7 +721,7 @@ output.elasticsearch:
#ssl.curve_types: []


#------------------------------- File output ----------------------------------
#------------------------------- File output -----------------------------------
#output.file:
# Boolean flag to enable or disable the output module.
#enabled: true
Expand Down Expand Up @@ -771,7 +779,7 @@ output.elasticsearch:
# the default for the logs path is a logs subdirectory inside the home path.
#path.logs: ${path.home}/logs

#================================ Logging =====================================
#================================ Logging ======================================
# There are three options for the log output: syslog, file, stderr.
# Under Windows systems, the log files are per default sent to the file output,
# under all other system per default to syslog.
Expand Down
38 changes: 38 additions & 0 deletions filebeat/filebeat.template-es2x.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,44 @@
},
"type": "string"
},
"meta": {
"properties": {
"cloud": {
"properties": {
"availability_zone": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"instance_id": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"machine_type": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"project_id": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"provider": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
},
"region": {
"ignore_above": 1024,
"index": "not_analyzed",
"type": "string"
}
}
}
}
},
"offset": {
"type": "long"
},
Expand Down
32 changes: 32 additions & 0 deletions filebeat/filebeat.template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,38 @@
"norms": false,
"type": "text"
},
"meta": {
"properties": {
"cloud": {
"properties": {
"availability_zone": {
"ignore_above": 1024,
"type": "keyword"
},
"instance_id": {
"ignore_above": 1024,
"type": "keyword"
},
"machine_type": {
"ignore_above": 1024,
"type": "keyword"
},
"project_id": {
"ignore_above": 1024,
"type": "keyword"
},
"provider": {
"ignore_above": 1024,
"type": "keyword"
},
"region": {
"ignore_above": 1024,
"type": "keyword"
}
}
}
}
},
"offset": {
"type": "long"
},
Expand Down
52 changes: 52 additions & 0 deletions heartbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ This document describes the fields that are exported by Heartbeat. They are
grouped in the following categories:

* <<exported-fields-beat>>
* <<exported-fields-cloud>>
* <<exported-fields-common>>

--
Expand Down Expand Up @@ -69,6 +70,57 @@ type: dict
Contains user configurable fields.


[[exported-fields-cloud]]
== Cloud Provider Metadata Fields

Metadata from cloud providers added by the add_cloud_metadata processor.



[float]
=== meta.cloud.provider

example: ec2

Name of the cloud provider. Possible values are ec2, gce, or digitalocean.


[float]
=== meta.cloud.instance_id

Instance ID of the host machine.


[float]
=== meta.cloud.machine_type

example: t2.medium

Machine type of the host machine.


[float]
=== meta.cloud.availability_zone

example: us-east-1c

Availability zone in which this host is running.


[float]
=== meta.cloud.project_id

example: project-x

Name of the project in Google Cloud.


[float]
=== meta.cloud.region

Region in which this host is running.


[[exported-fields-common]]
== Common monitoring fields Fields

Expand Down
Loading