cAdvisor running in docker ignore isolcpus

[shohamd4]

Hi,

I'm running cAdvisor in docker on machines with high CPU load.
I've configured the server kernel with isolcpus=1,2,3

I'm running cAdvisor with the following params:

docker run \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --volume=/dev/disk/:/dev/disk:ro \
  --publish=8080:8080 \
  --detach=true \
  --name=cadvisor \
  --privileged \
  --device=/dev/kmsg \
  gcr.io/cadvisor/cadvisor:v0.37.0

I've noticed that cAdvisor still runs on all cpus (not only cpu 0).
If I run cAdvisor with docker --cpuset-cpus=0, then cAdvisor doesn't show info on the other cpus.
Also if I enforce cpu affinity with taskset (without the --cpuset-cpus flag) - cAdvisor will show me stats only on the allowed cpus.

What is the correct way to run cAdvisor on a specific cpu?
BTW, I can see that setting -max_procs to 1 doesn't affect the number of processes that cAdvisor uses.

Thanks,
Doron

[iwankgb]

@shohamd4 there is something clearly wrong with handling isolcpus and tasksets. I will take a look into this.

[iwankgb]

Environment: 4-core aarch64 system. isolcpus=1-3. alexeiled/stress-ng docker container running on CPUs 2,3 (--cpuset-cpus 2,3).
With #2744:

• Topology information is displayed correctly despite of isolcpus and taskset used simultaneously.
• CPU usage is displayed for one CPU only. Overall CPU usage seems to be accounted for (no taskset used):

"cpu": {
  "usage": {
   "total": 567239332835,
   "per_cpu_usage": [
     0
    ],
    "user": 567560000000,
    "system": 80000000
  }
}  

• cAdvisor seems to consume up to 80% of CPU 0.

[iwankgb]

And the culprit is:

cadvisor/container/libcontainer/handler.go

Line 815 in 730e7df

if err := unix.SchedGetaffinity(0, &availableCPUs); err != nil {

According to the manual:

sched_getaffinity() writes the affinity mask of the process whose ID is pid into the cpu_set_t structure pointed to by mask. The cpusetsize argument specifies the size (in bytes) of mask. If pid is zero, then the mask of the calling process is returned.

I believe that this function and any filtering that uses it must be removed: sched_getaffinity() does not return number of CPUs in the system. Such information can be obtained from sysfs but I don't think it is relevant in this case.

[iwankgb]

@shohamd4 you can try code from #2805 - it should fix your problem.

[shohamd4]

@iwankgb - Thank you very much for the quick fix, I'll try it on Monday.

[shohamd4]

Hi, looks great, do you plan to push it into an official release?
Thank you very much!
Doron

[iwankgb]

As soon as it gets merged :)