Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make bazel cpu counter container aware #5042

Closed
hanwen opened this issue Apr 18, 2018 · 3 comments
Closed

make bazel cpu counter container aware #5042

hanwen opened this issue Apr 18, 2018 · 3 comments
Assignees
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Local-Exec Issues and PRs for the Execution (Local) team type: feature request

Comments

@hanwen
Copy link
Contributor

hanwen commented Apr 18, 2018

https://github.com/bazelbuild/bazel/blob/827007729889126cd7e07a08116f9bcb82f819a9/src/main/java/com/google/devtools/build/lib/actions/LocalHostResourceManagerLinux.java

reads the number of CPUs from /proc/cpuinfo. AFAIK, Docker doesn't virtualize /proc/ , so if you try to control resource usage on a machine with Docker CPU limits, bazel will disregard this an thrash the machine.

@c-parsons c-parsons added type: feature request P2 We'll consider working on this in future. (Assignee optional) category: sandboxing labels Apr 18, 2018
@janakdr janakdr assigned philwo and unassigned janakdr Apr 18, 2018
@janakdr
Copy link
Contributor

janakdr commented Apr 18, 2018

Giving to philwo@, who's much more of an expert in this area than I am.

@philwo
Copy link
Member

philwo commented May 3, 2018

Note to self:

We can get the CPUs available inside a standard Docker container via:
/sys/fs/cgroup/cpu/cpu.cfs_quota_us divided by /sys/fs/cgroup/cpu/cpu.cfs_quota_us/cpu.cfs_period_us

We can get the memory limits via:
/sys/fs/cgroup/memory/memory.limit_in_bytes

@jin jin added team-Local-Exec Issues and PRs for the Execution (Local) team and removed category: sandboxing labels Mar 1, 2021
@jwbee
Copy link

jwbee commented Oct 19, 2022

Mostly just subscribing to this ... but I have to say the way bazel subverts the Linux CPU mask is quite annoying. It starts a number of parallel tasks equal to the number of logical processors in the machine, yes, that's weird, but it's not as weird as the fact that it runs on all CPUs regardless of the fact that I run it under taskset -c 10 bazel test .... My intention is to isolate it on one CPU but it just goes right ahead and uses all CPUs anyway. It must be something about the way the sandbox is started, because with plain old fork the cpumask would be inherited by descendants all the way down.

The only effective workaround I know is to create a cgroup without access to other CPUs and run the build inside that.

fweikert pushed a commit to fweikert/bazel that referenced this issue May 25, 2023
As of JDK 11, `Runtime#availableProcessors` on Linux is aware of cgroup resource limits and thus behaves as expected in containers, whereas `/proc/cpuinfo` still reports the number of logical processors available to the host.

Unfortunately, the JVM-internal `os::total_memory` function, which similarly takes cgroup limits into account when computing the available RAM, does not seem to be accessible from Java.

https://hg.openjdk.java.net/jdk/hs/rev/7f22774a5f42#l6.178

Fixes bazelbuild#5042

Closes bazelbuild#16512.

RELNOTES[INC]: Bazel's local CPU resource on Linux is now container aware. Use `--local_cpu_resources`, `--loading_phase_threads` or `--jobs` to override.

PiperOrigin-RevId: 515032721
Change-Id: I103ee25920d6908545ce6bf03d9c42c604c8589b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Local-Exec Issues and PRs for the Execution (Local) team type: feature request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants