cAdvisor does not return CPU metrics for 1.4 #11199

mwringe · 2016-10-03T20:10:52Z

cAdvisor is returning empty cpu usage for origin 1.4 (built from master, commit ffdeb1b)

  "stats": [
   {
    "timestamp": "2016-10-03T15:53:03.359772024-04:00",
    "cpu": {
     "usage": {
      "total": 0,
      "user": 0,
      "system": 0
     },

Version

commit ffdeb1b

but I suspect this also affect the v1.4.0-alpha.0

In the v1.3.0 versions, cpu metrics are collected

Steps To Reproduce

build origin from git master
access the cAdvisor endpoint for container and verify that it has a zero value for cpu.

eg:
curl -k -H "Authorization: Bearer oc whoami -t" -X POST -d '{"num_stats":1}' https://127.0.0.1:10250/stats/${PROJECT_NAME}/${POD_NAME}/${POD_ID}/${CONTAINER_NAME}

Alternatively, install origin metrics (https://docs.openshift.org/latest/install_config/cluster_metrics.html) and verify that the cpu graphs are empty.

Current Result

cpu metrics are all zero

Expected Result

cpu metrics to not be zero

The text was updated successfully, but these errors were encountered:

mwringe · 2016-10-03T20:11:45Z

@spadgett the root cause is cAdvisor in the 1.4 version of OpenShift returing 0 values for the cpu usage

mwringe · 2016-10-03T20:14:57Z

@derekwaynecarr any idea here? This breaks the console graphing of cpu usage in 1.4

spadgett · 2016-10-03T20:15:24Z

/cc @jwforres

derekwaynecarr · 2016-10-03T20:20:15Z

Is openshift running as a container?
What version of docker?
What systemd version (i.e. systemctl --version)

If openshift is running as a container, this will not work until the following is merged upstream and we cherry-pick it into origin 1.4: kubernetes/kubernetes#33806

mwringe · 2016-10-03T20:21:35Z

docker version: 1.10.3
os: Fedora 24
systemd: 229

OpenShift is running directly on the machine, not in a docker container.

I know that cAdvisor doesn't necessarily work well with a newer systemd, was hoping that would have been resolved by now

derekwaynecarr · 2016-10-03T20:28:55Z

systemd 229 changed the cgroup hierarchy and put pid 1 in init.scope.
this caused a bug in runc using cgroup driver for systemd.
this bug is fixed in projectatomic/docker-1.12 but it has not yet been published for rpm install on F24.

mwringe · 2016-10-03T20:51:50Z

I am now running the fedora docker 1.12 rpms which should have this fix, and yet I still don't see any difference here with the cpu usage, its still zero

derekwaynecarr · 2016-10-04T14:11:00Z

@sjenning -- ptal

sjenning · 2016-10-04T17:53:20Z

i've been able to recreate. debugging now.

sjenning · 2016-10-04T19:22:26Z

could be related to this 9607748

sjenning · 2016-10-04T19:27:36Z

yes, reverting 9607748 fixes the issue

sjenning · 2016-10-04T20:20:49Z

Ok gathering information:

opencontainers/runc PR that allows for all cgroup mount points:
opencontainers/runc#1049

PR that vendors this into cAdvisor:
google/cadvisor#1476

PR that vendors new cAdvisor into Kubernetes:
kubernetes/kubernetes#33806

This isn't vendored into any version of projectatomic/docker right now.

Hack commit to get cAdvisor work at all containerized on RHEL for openshift/origin:
9607748

IIUC, the fix for this issue is to vendor the fixed cAdvisor into origin and revert the hack commit.

smarterclayton · 2016-10-04T21:37:13Z

Yes

sjenning · 2016-10-04T22:26:27Z

Just out of morbid curiosity, I wanted to find out why cpuacct,cpu is inverted for RHEL7. Turns out that the list of cgroup subsystems is a statically defined enum in the kernel:
https://github.com/torvalds/linux/blob/master/include/linux/cgroup_subsys.h.

For v3.10 (RHEL7) that subsystem enum is converted into a linked list which inserts at the head, inverting the enum order.

In v3.15, the support for adding cgroup subsystems as modules was dropped and the code was greatly simplified, removing the need for the linked list and reversing the order for users of for_each_subsys

torvalds/linux@3ed80a62b

Upstream for_each_subsys
https://github.com/torvalds/linux/blob/master/kernel/cgroup.c#L557

v3.10 for_each_subsys
https://github.com/torvalds/linux/blob/v3.10/kernel/cgroup.c#L268

proc_cgroup_show uses for_each_subsys to create the output for /proc/pid/cgroup

So... mystery solved!

smarterclayton · 2016-10-04T22:35:51Z

Gets out knife. Stabs self. Chooses another profession.

On Tue, Oct 4, 2016 at 6:26 PM, Seth Jennings notifications@github.com
wrote:

Just out of morbid curiosity, I wanted to find out why cpuacct,cpu is
inverted for RHEL7. Turns out that the list of cgroup subsystems is a
statically defined enum in the kernel:
https://github.com/torvalds/linux/blob/master/include/
linux/cgroup_subsys.h.

For v3.10 (RHEL7) that subsystem list is converted into a linked list
which inserts at the head, inverting the list.

In v3.14, the support for adding cgroup subsystems as modules was dropped
and the code was greatly simplified, removing the need for the linked list
and reversing the order for users of for_each_subsys

torvalds/linux@3ed80a6
torvalds/linux@3ed80a62b

Upstream for_each_subsys
https://github.com/torvalds/linux/blob/master/kernel/cgroup.c#L557

v3.10 for_each_subsys
https://github.com/torvalds/linux/blob/v3.10/kernel/cgroup.c#L268

proc_cgroup_show uses for_each_subsys to create the output for
/proc//cgroup

So... mystery solved!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#11199 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p0oALk_zNjmK3JdKpwO6Jl-RnIhxks5qwtKVgaJpZM4KM_RL
.

derekwaynecarr · 2016-10-06T02:57:56Z

Linked lists should only support append!

On Tuesday, October 4, 2016, Clayton Coleman notifications@github.com
wrote:

Gets out knife. Stabs self. Chooses another profession.

On Tue, Oct 4, 2016 at 6:26 PM, Seth Jennings <notifications@github.com
javascript:_e(%7B%7D,'cvml','notifications@github.com');>
wrote:

Just out of morbid curiosity, I wanted to find out why cpuacct,cpu is
inverted for RHEL7. Turns out that the list of cgroup subsystems is a
statically defined enum in the kernel:
https://github.com/torvalds/linux/blob/master/include/
linux/cgroup_subsys.h.

For v3.10 (RHEL7) that subsystem list is converted into a linked list
which inserts at the head, inverting the list.

In v3.14, the support for adding cgroup subsystems as modules was dropped
and the code was greatly simplified, removing the need for the linked
list
and reversing the order for users of for_each_subsys

torvalds/linux@3ed80a6
torvalds/linux@3ed80a62b

Upstream for_each_subsys
https://github.com/torvalds/linux/blob/master/kernel/cgroup.c#L557

v3.10 for_each_subsys
https://github.com/torvalds/linux/blob/v3.10/kernel/cgroup.c#L268

proc_cgroup_show uses for_each_subsys to create the output for
/proc//cgroup

So... mystery solved!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#11199 (comment)
,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABG_p0oALk_
zNjmK3JdKpwO6Jl-RnIhxks5qwtKVgaJpZM4KM_RL>
.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#11199 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF8dbDiVaiUTjxdJrAXXqvD9MB1AEIX3ks5qwtTNgaJpZM4KM_RL
.

sjenning · 2016-10-31T18:41:56Z

This is fixed by #11642

mwringe · 2016-11-01T13:11:30Z

Bugzilla opened for this: https://bugzilla.redhat.com/show_bug.cgi?id=1390502

ncdc · 2016-11-02T19:23:26Z

#11642 was superseded by #11709 which has merged, so this can be closed now

danmcp added priority/P1 component/kubernetes labels Oct 4, 2016

danmcp assigned derekwaynecarr Oct 4, 2016

sjenning self-assigned this Oct 4, 2016

derekwaynecarr mentioned this issue Oct 28, 2016

Kube 1.4 cherry picks #11642

Closed

ncdc closed this as completed Nov 2, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cAdvisor does not return CPU metrics for 1.4 #11199

cAdvisor does not return CPU metrics for 1.4 #11199

mwringe commented Oct 3, 2016

mwringe commented Oct 3, 2016

mwringe commented Oct 3, 2016

spadgett commented Oct 3, 2016

derekwaynecarr commented Oct 3, 2016 •

edited

Loading

mwringe commented Oct 3, 2016

derekwaynecarr commented Oct 3, 2016

mwringe commented Oct 3, 2016

derekwaynecarr commented Oct 4, 2016

sjenning commented Oct 4, 2016

sjenning commented Oct 4, 2016

sjenning commented Oct 4, 2016

sjenning commented Oct 4, 2016 •

edited

Loading

smarterclayton commented Oct 4, 2016

sjenning commented Oct 4, 2016 •

edited

Loading

smarterclayton commented Oct 4, 2016

derekwaynecarr commented Oct 6, 2016

sjenning commented Oct 31, 2016

mwringe commented Nov 1, 2016

ncdc commented Nov 2, 2016

cAdvisor does not return CPU metrics for 1.4 #11199

cAdvisor does not return CPU metrics for 1.4 #11199

Comments

mwringe commented Oct 3, 2016

Version

Steps To Reproduce

Current Result

Expected Result

mwringe commented Oct 3, 2016

mwringe commented Oct 3, 2016

spadgett commented Oct 3, 2016

derekwaynecarr commented Oct 3, 2016 • edited Loading

mwringe commented Oct 3, 2016

derekwaynecarr commented Oct 3, 2016

mwringe commented Oct 3, 2016

derekwaynecarr commented Oct 4, 2016

sjenning commented Oct 4, 2016

sjenning commented Oct 4, 2016

sjenning commented Oct 4, 2016

sjenning commented Oct 4, 2016 • edited Loading

smarterclayton commented Oct 4, 2016

sjenning commented Oct 4, 2016 • edited Loading

smarterclayton commented Oct 4, 2016

derekwaynecarr commented Oct 6, 2016

sjenning commented Oct 31, 2016

mwringe commented Nov 1, 2016

ncdc commented Nov 2, 2016

derekwaynecarr commented Oct 3, 2016 •

edited

Loading

sjenning commented Oct 4, 2016 •

edited

Loading

sjenning commented Oct 4, 2016 •

edited

Loading