Metricbeat6.0 system module doesn't capture kernel process in Centos7 #5700
Comments
|
I just tried this out in a docker centos:7 container. Initially I set up the module manually with only the Not sure if this is related. @maozi07 Can you share the config you are using? Could you also share some of the documents that metricbeat puts out? |
|
Hi @ruflin docker doesn't have any kernel process. I only collect system metric configure like this：
What else can I share or do you need？ |
|
@maozi07 Does metricbeat on your end not collect the processes at all or is just the command line off? If it does collect the events, could you share one of the json documents that you are seeing? |
|
@ruflin I think it doesn't collect the process at all，because I haven't find any data using cmdline/process name from ES about the process. |
|
As a data point, I am seeing events for these processes using 5.5.0. I believe there is no {
"@timestamp": "2017-11-28T11:01:17.780Z",
"beat": {
"hostname": "xxx",
"name": "xxx",
"version": "5.5.0"
},
"meta": {
"cloud": {
"instance_id": "xxx",
"provider": "xxx",
"region": "xxx"
}
},
"metricset": {
"module": "system",
"name": "process",
"rtt": 61001
},
"system": {
"process": {
"cpu": {
"start_time": "2015-10-27T11:49:47.000Z",
"total": {
"pct": 0
}
},
"fd": {
"limit": {
"hard": 4096,
"soft": 1024
},
"open": 0
},
"memory": {
"rss": {
"bytes": 0,
"pct": 0
},
"share": 0,
"size": 0
},
"name": "ksoftirqd/0",
"pgid": 0,
"pid": 4,
"ppid": 2,
"state": "sleeping",
"username": "root"
}
},
"type": "metricsets"
} |
|
@andrewkroh I'm using 6.0.0, and I tried to query |
|
@ruflin @andrewkroh Hi ,I found some log maybe useful to you when I tracked other problem.
|
|
That's very helpful. Those kernel processes don't have an exe on disk so the /proc/[PID]/exe symlink is broken. So Metricbeat should ignore the ENOENT error and continue processing data from the process. I think the code here should also check for beats/metricbeat/module/system/process/helper.go Lines 66 to 69 in 75a1fc8 |
|
I've hit this also. This is for all pid=2..n EDIT: I can confirm commenting out process related stat sets causes no further error messages: - module: system
period: 15s
metricsets:
- cpu
- load
- memory
- network
#- process
#- process_summary
#- core
#- diskio
#- socket
processes: ['.*']
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
- module: system
period: 15m
metricsets:
- uptime |
andrewkroh
added
help wanted
On Linux, kernel processes don't have an exe on disk so the `/proc/[PID]/exe` symlink is broken. When Metricbeat tries to read the symlink it gets an error and it skips the whole process. This adds a check for ENOENT and ignores the error which allows kernel processes to be reported by Metricbeat. This is the error that it fixes.
2018-01-30T20:31:02.512Z DEBUG [processes] process/process.go:443 Skip process pid=12113: error getting process exe for pid=12113: readlink /proc/12113/exe: no such file or directory
Fixes elastic#5700
On Linux, kernel processes don't have an exe on disk so the `/proc/[PID]/exe` symlink is broken. When Metricbeat tries to read the symlink it gets an error and it skips the whole process. This adds a check for ENOENT and ignores the error which allows kernel processes to be reported by Metricbeat. This is the error that it fixes.
2018-01-30T20:31:02.512Z DEBUG [processes] process/process.go:443 Skip process pid=12113: error getting process exe for pid=12113: readlink /proc/12113/exe: no such file or directory
Fixes #5700
|
Thank you! |
On Linux, kernel processes don't have an exe on disk so the `/proc/[PID]/exe` symlink is broken. When Metricbeat tries to read the symlink it gets an error and it skips the whole process. This adds a check for ENOENT and ignores the error which allows kernel processes to be reported by Metricbeat. This is the error that it fixes.
2018-01-30T20:31:02.512Z DEBUG [processes] process/process.go:443 Skip process pid=12113: error getting process exe for pid=12113: readlink /proc/12113/exe: no such file or directory
Fixes elastic#5700
(cherry picked from commit 6bde7d8)
On Linux, kernel processes don't have an exe on disk so the `/proc/[PID]/exe` symlink is broken. When Metricbeat tries to read the symlink it gets an error and it skips the whole process. This adds a check for ENOENT and ignores the error which allows kernel processes to be reported by Metricbeat. This is the error that it fixes.
2018-01-30T20:31:02.512Z DEBUG [processes] process/process.go:443 Skip process pid=12113: error getting process exe for pid=12113: readlink /proc/12113/exe: no such file or directory
Fixes elastic#5700
(cherry picked from commit ee38051)
metricbeat system module doesn't capture the system process there cmdline always like this
[xxxx]and I enabled debug log found nothing useful, there nothing about these process
It's all kernel process like this:
https://discuss.elastic.co/t/metricbeat6-0-system-module-doesnt-capture-linux-system-process/108654
The text was updated successfully, but these errors were encountered: