Faulty values of system_p and user_p fields.

[pawel3ak]

Hi All,

I've observed that topbeats get wrong values in my servers - especially fields system_p and user_p in cpu sections are wrong in my opionion. Despite of running "stress" process which generates enormous high cpu usage this statistics have zero-values. What can be a root cause of such situation? Below information about software/hardware and steps to reproduction. Thanks in advance for any hints.

Version: topbeat version 1.2.0 (amd64)
Operating System: Linux RedHat 2.6.32-573.18.1.el6.x86_64 #1 SMP Wed Jan 6 11:20:49 EST 2016 x86_64 x86_64 x86_64 GNU/Linux, 8-core machine.

Steps to Reproduce:
* type in CLI: stress -c 6 // predicted usage of cpu resources 75 %
* run topbeat as follows: topbeat -c topbeat.yml -e -d "*"

example output after above steps:

[...]
{
  "@timestamp": "2016-05-11T10:11:09.491Z",
  "beat": {
    "hostname": "ltxld-mvmgr-025",
    "name": "ltxld-mvmgr-025"
  },
  "count": 1,
  "proc": {
    "cpu": {
      "user": 0,
      "user_p": 0,
      "system": 34250,
      "total": 34250,
      "start_time": "Mar14"
    },
    "mem": {
      "size": 0,
      "rss": 0,
      "rss_p": 0,
      "share": 0
    },
    "name": "jbd2/dm-0-8",
    "pid": 459,
    "ppid": 2,
    "state": "sleeping"
  },
  "type": "process"
}
[...]  //some sleeping process informations

{
  "@timestamp": "2016-05-11T10:11:09.492Z",
  "beat": {
    "hostname": "ltxld-mvmgr-025",
    "name": "ltxld-mvmgr-025"
  },
  "count": 1,
  "proc": {
    "cmdline": "stress -c 6",
    "cpu": {
      "user": 0,
      "user_p": 0,
      "system": 0,
      "total": 0,
      "start_time": "04:06"
    },
    "mem": {
      "size": 6672384,
      "rss": 577536,
      "rss_p": 0,
      "share": 491520
    },
    "name": "stress",
    "pid": 887,
    "ppid": 838,
    "state": "sleeping"
  },
  "type": "process"
}

[ruflin]

Not sure if this will help, but could you try running topbeat with sudo?

[pawel3ak]

Thanks for response @ruflin :) But it brings no effects. Some records are ok, but most of them has wrong value - as before;/

[ruflin]

Can you provide a gist with more output in it? Means multiple results for example for the cmdlin stress inside. A full debug log would be useful.

[pawel3ak]

@ruflin - I attache a file with redirected output of topbeats. I hope that You will have some ideas what can be root cause of such situation.
topbeats.txt

[andrewkroh]

For the first example you posted: user = 0, so zero milliseconds were spent in userspace so user_p = 0. Topbeat doesn't export a total_p or system_p metric for processes, though the code could be modified to do so.

Next example: This process is sleeping and it is not consuming any CPU time so all values are zero. The stress command spawns processes so maybe you are looking at the parent here and not one of the workers. Look for other processes where the ppid = 887 in this case.

Attached file: There is a stress process with pid = 6790, this is the parent and it uses no CPU. It spawns multiple workers and those workers are consuming a lot of CPU according to Topbeat.