Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable cgroup metric collection by default #3519

Merged
merged 2 commits into from
Feb 5, 2017
Merged

Enable cgroup metric collection by default #3519

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
61af83e
Enable cgroup metric collection by default
andrewkroh Feb 2, 2017
d26e275
Update system test to check cgroup fields on linux
andrewkroh Feb 3, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@ https://github.com/elastic/beats/compare/v5.1.1...master[Check the HEAD diff]
*Affecting all Beats*

*Metricbeat*
- Linux cgroup metrics are now enabled by default for the system process
metricset. The configuration option for the feature was renamed from
`cgroups` to `process.cgroups.enabled`. {pull}3519[3519]

*Packetbeat*

Expand Down
1 change: 0 additions & 1 deletion metricbeat/docs/fields.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6237,7 +6237,6 @@ The hard limit on the number of file descriptors opened by the process. The hard
[float]
== cgroup Fields

experimental[]
Metrics and limits from the cgroup of which the task is a member. cgroup metrics are reported when the process has membership in a non-root cgroup. These metrics are only available on Linux.


Expand Down
22 changes: 11 additions & 11 deletions metricbeat/docs/metricbeat-in-a-container.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
Elastic does not provide any official container images for Metricbeat. The
examples on this page assume you are using your own Metricbeat container image.

When executing Metricbeat in a container, there are some important
When executing Metricbeat in a container, there are some important
things to be aware of if you want to monitor the host machine or other
containers. Let's walk-through some examples using Docker as our container
orchestration tool.
Expand All @@ -18,14 +18,14 @@ work properly inside of a container. This enables Metricbeat to monitor the
host machine from within the container.

[source,sh]
---
----
sudo docker run \
--volume=/proc:/hostfs/proc:ro \ <1>
--volume=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro \ <2>
--volume=/:/hostfs:ro \ <3>
--net=host <4>
my/metricbeat:latest -system.hostfs=/hostfs
---
----

<1> Metricbeat's <<metricbeat-module-system,system module>> collects much of its data through the Linux proc
filesystem, which is normally located at `/proc`. Because containers
Expand All @@ -34,10 +34,10 @@ container's `/proc` is different than the host's `/proc`. To account for this, y
can mount the host's `/proc` filesystem inside of the container and tell
Metricbeat to look inside the `/hostfs` directory when looking for `/proc` by
using the `-system.hostfs=/hostfs` CLI flag.
<2> If you have enabled cgroup reporting (an experimental feature) from the
<<metricbeat-metricset-system-process,system process metricset>>, then you need to mount the host's cgroup mountpoints
within the container. They need to be mounted inside the directory specified by
the `-system.hostfs` CLI flag.
<2> If cgroup reporting is enabled for the
<<metricbeat-metricset-system-process,system process metricset>>, then you need
to mount the host's cgroup mountpoints within the container. They need to be
mounted inside the directory specified by the `-system.hostfs` CLI flag.
<3> If you want to be able to monitor filesystems from the host by using the
<<metricbeat-metricset-system-filesystem,system filesystem metricset>>, then those filesystems need to be mounted inside
of the container. They can be mounted at any location.
Expand All @@ -55,12 +55,12 @@ Next let's look at an example of monitoring a containerized service from a
Metricbeat container.

[source,sh]
---
----
sudo docker run \
--link some-mysql:mysql \ <1>
-e MYSQL_PASSWORD=secret \ <2>
my/metricbeat:latest
---
----

<1> Linking the containers enables Metricbeat access the exposed ports of the
mysql container, and it makes the hostname `mysql` resolvable to Metricbeat.
Expand All @@ -71,14 +71,14 @@ variables or as command line flags to Metricbeat (see the `-E` CLI flag in <<com
The mysql module configuration would look like this:

[source,yaml]
---
----
metricbeat.modules:
- module: mysql
metricsets: ["status"]
hosts: ["mysql:3306"] <1>
username: root
password: ${MYSQL_PASSWORD} <2>
---
----

<1> The `mysql` hostname will resolve to the `some-mysql` container's address.
<2> The `MYSQL_PASSWORD` variable will be evaluated at startup. If the variable
Expand Down
9 changes: 5 additions & 4 deletions metricbeat/docs/modules/system.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,17 +26,18 @@ metricbeat.modules:
metricsets: ["process"]
processes: ['.*']
----
*`cgroups`*:: When the `process` metricset is enabled, you can use the boolean
`cgroups` option to enable the experimental cgroup metrics on Linux.
*`process.cgroups.enabled`*:: When the `process` metricset is enabled, you can
use this boolean configuration option to disable cgroup metrics. By default
cgroup metrics collection is enabled.
+
The following example config enables cgroups metrics on Linux.
The following example config disables cgroup metrics on Linux.
+
[source,yaml]
----
metricbeat.modules:
- module: system
metricsets: ["process"]
cgroups: true
process.cgroups.enabled: false
----
*`cpu_ticks`*:: When the `cpu` or `core` metricset is enabled, you can specify `cpu_ticks: true` to report CPU ticks in addition to CPU percentages stats. For example:
+
Expand Down
4 changes: 2 additions & 2 deletions metricbeat/metricbeat.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ metricbeat.modules:
# if true, exports the CPU usage in ticks, together with the percentage values
#cpu_ticks: false

# EXPERIMENTAL: cgroups can be enabled for the process metricset.
#cgroups: false
# Enable collection of cgroup metrics from processes on Linux.
#process.cgroups.enabled: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add it to the short config as well as it is now a "default" feature?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather leave it out of the short config. I'd actually be in favor of removing more stuff from the short config. I feel like there's a lot of "noise" in the files and we could have short comment at the top about "see xbeat.full.yml for a commented example with all options" and then remove everything that not necessary. I like the readability of something like this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andrewkroh Sounds like something we should discuss in general as it would also apply to all other beats. I kind of like the docs as it helps new users (I hope) but it is much more noisy, agree.


# A list of regular expressions used to whitelist environment variables
# reported with the process metricset's events. Defaults to empty.
Expand Down
4 changes: 2 additions & 2 deletions metricbeat/module/system/_meta/config.full.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@
# if true, exports the CPU usage in ticks, together with the percentage values
#cpu_ticks: false

# EXPERIMENTAL: cgroups can be enabled for the process metricset.
#cgroups: false
# Enable collection of cgroup metrics from processes on Linux.
#process.cgroups.enabled: true

# A list of regular expressions used to whitelist environment variables
# reported with the process metricset's events. Defaults to empty.
Expand Down
9 changes: 5 additions & 4 deletions metricbeat/module/system/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,18 @@ metricbeat.modules:
metricsets: ["process"]
processes: ['.*']
----
*`cgroups`*:: When the `process` metricset is enabled, you can use the boolean
`cgroups` option to enable the experimental cgroup metrics on Linux.
*`process.cgroups.enabled`*:: When the `process` metricset is enabled, you can
use this boolean configuration option to disable cgroup metrics. By default
cgroup metrics collection is enabled.
+
The following example config enables cgroups metrics on Linux.
The following example config disables cgroup metrics on Linux.
+
[source,yaml]
----
metricbeat.modules:
- module: system
metricsets: ["process"]
cgroups: true
process.cgroups.enabled: false
----
*`cpu_ticks`*:: When the `cpu` or `core` metricset is enabled, you can specify `cpu_ticks: true` to report CPU ticks in addition to CPU percentages stats. For example:
+
Expand Down
7 changes: 3 additions & 4 deletions metricbeat/module/system/process/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,7 @@ This metricset is available on:
[float]
=== Control Group (cgroup) Metrics

On Linux this metricset can collect metrics from any cgroups that the process
is a member of. This feature is currently experimental, and it is not enabled by
default so you must add `cgroup: true` to the system module configuration to
enable it.
On Linux this metricset will collect metrics from any cgroups that the process
is a member of. This feature is enabled by default and can be disabled by adding
`process.cgroup.enabled: false` to the system module configuration.

2 changes: 0 additions & 2 deletions metricbeat/module/system/process/_meta/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,6 @@
- name: cgroup
type: group
description: >
experimental[]

Metrics and limits from the cgroup of which the task is a member.
cgroup metrics are reported when the process has membership in a
non-root cgroup. These metrics are only available on Linux.
Expand Down
11 changes: 5 additions & 6 deletions metricbeat/module/system/process/process.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,12 +34,11 @@ type MetricSet struct {
// New creates and returns a new MetricSet.
func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
config := struct {
Procs []string `config:"processes"` // collect all processes by default
Cgroups bool `config:"cgroups"`
Procs []string `config:"processes"`
Cgroups *bool `config:"process.cgroups.enabled"`
EnvWhitelist []string `config:"process.env.whitelist"`
}{
Procs: []string{".*"},
Cgroups: false,
Procs: []string{".*"}, // collect all processes by default
}
if err := base.Module().UnpackConfig(&config); err != nil {
return nil, err
Expand All @@ -63,8 +62,8 @@ func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
return nil, fmt.Errorf("unexpected module type")
}

if config.Cgroups {
logp.Warn("EXPERIMENTAL: Cgroup is enabled for the system.process MetricSet.")
if config.Cgroups == nil || *config.Cgroups {
debugf("process cgroup data collection is enabled")
m.cgroup, err = cgroup.NewReader(systemModule.HostFS, true)
if err != nil {
return nil, errors.Wrap(err, "error initializing cgroup reader")
Expand Down
16 changes: 8 additions & 8 deletions metricbeat/module/system/socket/_meta/docs.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@ The system `socket` metricset reports an event for each new TCP socket that it
sees. It does this by polling the kernel periodically to get a dump of all
sockets. You set the polling interval by configuring the `period` option.
Specifying a short polling interval with this metricset is important to avoid
missing short-lived connections. For example:
missing short-lived connections. For example:

[source,yaml]
---
----
metricbeat.modules:
- module: system
metricsets: [cpu, memory]
- module: system
- module: system
metricsets: [socket] <1>
period: 1s
---
----

<1> You can configure the `socket` metricset separately to specify a different
`period` value than the other metricsets.
`period` value than the other metricsets.

The metricset reports the process that has the socket open. In order to provide
this information, Metricbeat must be running as root. Root access is also
Expand All @@ -30,16 +30,16 @@ required to read the file descriptor information of other processes.
You can configure the metricset to perform a reverse lookup on the remote IP,
and the returned hostname will be added to the event and cached. If a hostname
is found, then the eTLD+1 (effective top-level domain plus one level) value will
also be added to the event. Reverse lookups are disabled by default.
also be added to the event. Reverse lookups are disabled by default.

The following example shows the full configuration for the metricset along with
the defaults.

[source,yaml]
---
----
- module: system
metricsets: [socket]
socket.reverse_lookup.enabled: false
socket.reverse_lookup.success_ttl: 60s
socket.reverse_lookup.failure_ttl: 60s
---
----
6 changes: 5 additions & 1 deletion metricbeat/tests/system/test_system.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,9 @@
# cmdline is also part of the system process fields, but it may not be present
# for some kernel level processes. fd is also part of the system process, but
# is not available on all OSes and requires root to read for all processes.
# cgroup is only available on linux.
SYSTEM_PROCESS_FIELDS = ["cpu", "memory", "name", "pid", "ppid", "pgid",
"state", "username"]
"state", "username", "cgroup"]


class SystemTest(metricbeat.BaseTest):
Expand Down Expand Up @@ -325,6 +326,9 @@ def test_process(self):
"""
Test system/process output.
"""
if not sys.platform.startswith("linux") and "cgroup" in SYSTEM_PROCESS_FIELDS:
SYSTEM_PROCESS_FIELDS.remove("cgroup")

self.render_config_template(modules=[{
"name": "system",
"metricsets": ["process"],
Expand Down