Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[libbeat] AWS ELB Monitoring #12401

Merged
merged 78 commits into from
Jul 30, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
72894c5
Unify heartbeat jobs and tasks for simplicity
andrewvc Nov 14, 2018
ab4a73d
AWS Autodisco
andrewvc Oct 17, 2018
d0bdd60
checkpoint
andrewvc Nov 15, 2018
2c69797
Everything is a job
andrewvc Nov 16, 2018
5e223ba
Improve job types
andrewvc Nov 16, 2018
c54f02d
Checkpoint
andrewvc Nov 16, 2018
53b53e0
Checkpoint
andrewvc Nov 16, 2018
f83d586
CHECKPOINT
andrewvc Nov 16, 2018
3735992
checkpoint
andrewvc Nov 19, 2018
3a27a98
Merge remote-tracking branch 'origin/master' into auto-elb
andrewvc Jun 3, 2019
885eda8
Fix govendor
andrewvc Jun 3, 2019
900b2fd
Fix govendor
andrewvc Jun 3, 2019
9694836
Updates
andrewvc Jun 3, 2019
e5db213
Autodisco sorta works
andrewvc Jun 4, 2019
338afdc
moar
andrewvc Jun 6, 2019
d7b7982
Actually handle additional listeners
andrewvc Jun 6, 2019
2173a3e
More cleanup
andrewvc Jun 6, 2019
48285fa
Support custom access key
andrewvc Jun 7, 2019
64b2d60
Moar
andrewvc Jun 7, 2019
7de2b77
Improve asciidocs
andrewvc Jun 7, 2019
eda8ef2
Revert heartbeat yml
andrewvc Jun 7, 2019
9e7fd30
Fix authentication_failure event.type for security module (#12393)
andrewkroh Jun 3, 2019
381e161
Hardcode namespace for prometheus metrics (#12400)
jsoriano Jun 3, 2019
e98297f
Add documentation for Winlogbeat modules (#12361)
andrewkroh Jun 3, 2019
5298de2
Fix references to `beat.timezone` in docs (#12399)
jsoriano Jun 3, 2019
4c7dab9
Parse the timezone in PostgreSQL logs as part of the date (#12338)
jsoriano Jun 3, 2019
9b62b34
Timeouts on calls to docker api (#12310)
marqc Jun 4, 2019
b43d8cf
Add operating system from vsphere to 'os' field (#12391)
jsoriano Jun 4, 2019
bc22929
Un-skip flaky tests (#12388)
ycombinator Jun 4, 2019
023a46c
Load configuration from modules.d when running setup subcommand (#12340)
kvch Jun 4, 2019
c401af9
[Metricbeat] Decrease timeout time of compose.EnsureUp functions (#10…
sayden Jun 4, 2019
31f72d0
Mark Cisco and PANW modules as beta (#12426)
adriansr Jun 4, 2019
07de1e3
Validate stack modules' metricsets when xpack.enabled = true (#12386)
ycombinator Jun 4, 2019
4e5f001
Fix a typo in kubernetes deploy readme files (#12413)
Jun 4, 2019
05b42a6
Ensure Pod UID is always present in `pod` metricset (#12345)
Jun 4, 2019
26c87ba
[dev guide] Add docs on optional schema arguments (#12371)
fearful-symmetry Jun 5, 2019
f9ad369
Update filebeat EnvoyProxy module README file to reflect the changes …
Jun 5, 2019
1e9ecca
Update CoreDNS module README file to reflect the changes to autodisco…
Jun 5, 2019
1f00ab3
Revert "Enforce presence of Certificate Authorities, Certificate file…
ph Jun 5, 2019
4fc44ee
Govendor ECS 1.0.1 (#12317)
Jun 5, 2019
06d3f08
Don't duplicate @timestamp to beat.Event's Fields (#12449)
andrewkroh Jun 5, 2019
a23a8b7
migrate the system module to reporterV2 with error return, part one (…
fearful-symmetry Jun 6, 2019
338fabb
Remove `convert_timezone` option from filebeat system module (#12410)
jsoriano Jun 6, 2019
03752cd
Export Kibana Client's Transport and Version (#12422)
jalvz Jun 7, 2019
89cab2f
Fix 11975 - change Kubernetes container cpu cores fields to float (#1…
easkay Jun 7, 2019
0f883d1
[Heartbeat] Fix Continuation Dispatch / mode: all pings (#12458)
andrewvc Jun 7, 2019
c449d3c
Remove ECS from aws dashboard names (#12471)
kaiyan-sheng Jun 7, 2019
e5b8a8c
A variety of improvements
andrewvc Jun 7, 2019
e0202ec
Add ELB Listener fields
andrewvc Jun 12, 2019
1b078e4
Merge remote-tracking branch 'origin/master' into auto-elb
andrewvc Jun 12, 2019
462d0de
Revert accidental notice change
andrewvc Jun 12, 2019
ceedb5b
Incorporate PR feedback
andrewvc Jun 13, 2019
dce76e4
Incorporate PR feedback
andrewvc Jun 13, 2019
f5d8a69
Improve fields
andrewvc Jun 13, 2019
11ab929
Update all vendored imports
andrewvc Jun 14, 2019
0a1ba5a
Merge remote-tracking branch 'origin/master' into auto-elb
andrewvc Jun 17, 2019
d8e7230
Reset vendor to master
andrewvc Jun 17, 2019
555fd32
Merge remote-tracking branch 'origin/master' into auto-elb
andrewvc Jul 23, 2019
adf4d23
Update
andrewvc Jul 23, 2019
3817d9b
Update
andrewvc Jul 23, 2019
c279606
Incorporate PR feedback
andrewvc Jul 24, 2019
7242238
Merge remote-tracking branch 'origin/master' into auto-elb
andrewvc Jul 24, 2019
0d075ec
Incorporate PR feedback
andrewvc Jul 24, 2019
282cec2
Add timeout equal to interval
andrewvc Jul 24, 2019
7056db5
Incorporate PR feedback
andrewvc Jul 24, 2019
563e85d
Various improvements
andrewvc Jul 25, 2019
d35e091
Improve docs
andrewvc Jul 25, 2019
a5340f1
Fix whitespace
andrewvc Jul 25, 2019
ee5fdde
Merge remote-tracking branch 'origin/master' into auto-elb
andrewvc Jul 25, 2019
aa019c3
Fix test race
andrewvc Jul 29, 2019
bf2904e
Fix detected races
andrewvc Jul 29, 2019
7a6f151
Merge remote-tracking branch 'origin/master' into auto-elb
andrewvc Jul 29, 2019
431c5f6
Fix detected races
andrewvc Jul 29, 2019
9a53417
Make timeouts more lax
andrewvc Jul 29, 2019
0af6bd4
Fix docs
andrewvc Jul 29, 2019
f8683a0
Fix docs
andrewvc Jul 29, 2019
e120361
Just make the docs work for now
andrewvc Jul 30, 2019
abfb978
Don't leave in broken docs
andrewvc Jul 30, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions heartbeat/docs/autodiscover-aws-elb-config.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{beatname_uc} supports templates for modules:

["source","yaml",subs="attributes"]
-------------------------------------------------------------------------------------
heartbeat.autodiscover:
providers:
- type: aws_elb
period: 1m
regions: ["us-east-1", "us-east-2"]
# If you don't wish to use env vars or shared credentials files explicitly put credentials here.
#access_key_id: my-access-key
#secret_access_key: my-secret-access-key
#session_token: my-session-token
templates:
- condition:
equals.port: 8080
config:
- type: tcp
hosts: ["${data.host}:${data.port}"]
schedule: "@every 5s"
timeout: 1s
-------------------------------------------------------------------------------------

This configuration launches a `tcp` monitor for all ELBs that have a declared port.

This autodiscover provider takes our standard AWS credentials options.

2 changes: 2 additions & 0 deletions heartbeat/docs/configuring-howto.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,9 @@ include::{libbeat-dir}/docs/loggingconfig.asciidoc[]
include::{libbeat-dir}/docs/shared-env-vars.asciidoc[]
:standalone!:

:autodiscoverAWSELB:
include::{libbeat-dir}/docs/shared-autodiscover.asciidoc[]
:autodiscoverAWSELB!:

:standalone:
include::{libbeat-dir}/docs/yaml.asciidoc[]
Expand Down
49 changes: 45 additions & 4 deletions libbeat/docs/shared-autodiscover.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,10 @@ start/stop events. This ensures you don't need to worry about state, but only de
[float]
===== Docker

The Docker autodiscover provider watches for Docker containers to start and stop. These are the available fields on every
The Docker autodiscover provider watches for Docker containers to start and stop.


These are the available fields during within config templating. The `docker.*` fields will be available on each emitted event.
event:

* host
Expand Down Expand Up @@ -113,8 +116,9 @@ endif::[]
[float]
===== Kubernetes

The Kubernetes autodiscover provider watches for Kubernetes pods to start, update, and stop. These are the available fields on every
event:
The Kubernetes autodiscover provider watches for Kubernetes pods to start, update, and stop.

These are the available fields during within config templating. The `kubernetes.*` fields will be available on each emitted event.

* host
* port (if exposed)
Expand Down Expand Up @@ -231,7 +235,7 @@ and the Jolokia agents has to be allowed. Also notice that this multicast
address is in the 239.0.0.0/8 range, that is reserved for private use within an
organization, so it can only be used in private networks.

These are the available fields on every event:
These are the available fields during within config templating. The `jolokia.*` fields will be available on each emitted event.

* jolokia.agent.id
* jolokia.agent.version
Expand All @@ -257,6 +261,43 @@ the ones used for discovery probes, they have these settings:
include::../../{beatname_lc}/docs/autodiscover-jolokia-config.asciidoc[]
endif::autodiscoverJolokia[]

ifdef::autodiscoverAWSELB[]
[float]
===== Amazon ELBs
andrewvc marked this conversation as resolved.
Show resolved Hide resolved

*Note: This provider is experimental*

The Amazon ELB autodiscover provider discovers https://aws.amazon.com/elasticloadbalancing/[ELBs] and their listeners. This is useful when you don't want to connect
directly to a service, but rather to the ELB fronting a pool of services.

This provider will yield one config block per ELB Listener. So, if you have one ELB exposing both ports 80 and 443, it
will generate two configs, one for each port. Keep in mind that the beat will de-duplicate configs. So, if the generated
configs are the same only one will actually run.

This provider will load AWS credentials using the standard AWS environment variables and shared credentials files see https://docs.aws.amazon.com/general/latest/gr/aws-access-keys-best-practices.html[Best Practices for Managing AWS Access Keys] for more information. If you do not wish to use these, you may explicitly set the `access_key_id` and `secret_access_key` variables.

These are the available fields during within config templating. The `elb_listener.*` fields will be available on each emitted event.

* host
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not directly but I thought worth mentioning: This is not following ECS but is the same in other autodiscovery providers. I wonder if long term this could become an issue if we would also ingest some autodiscovery data for later querying.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The host and port fields aren't inserted into the event. Only the elb_listener fields are. The fields listed here are what are present in the templating phase of the config. I'll add info here to clarify.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm aware that it's not added to the event, but I'm still thinking if we should potentially align to prevent confusion? @exekias ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I think there is some confusion here. Those fields do contain their own host and port fields. Additionally, if using heartbeat you'd probably use url.port and url.domain anyway.

* port
* elb_listener.listener_arn
andrewvc marked this conversation as resolved.
Show resolved Hide resolved
* elb_listener.load_balancer_arn
* elb_listener.protocol
* elb_listener.type
* elb_listener.scheme
* elb_listener.availability_zones
* elb_listener.created
* elb_listener.state
* elb_listener.ip_address_type
* elb_listener.security_groups
* elb_listener.vpc_id
* elb_listener.ssl_policy

include::../../{beatname_lc}/docs/autodiscover-aws-elb-config.asciidoc[]

endif::autodiscoverAWSELB[]


ifdef::autodiscoverHints[]
[[configuration-autodiscover-hints]]
=== Hints based autodiscover
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading