Default local resources should respect cgroup limits on Linux

[nelhage]

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

Bazel, by default, looks at available RAM on the system to set local_resources defaults, so as to best-use the resources of the machine.

Unfortunately, inside a Docker container or other cgroup environment, the system-wide memory statisics (/proc/meminfo, the output of free, etc) reflect the memory usage of the host, not the container.

bazel should make a best-effort attempt to find the effective cgroup memory controller limits, and use those.

If possible, provide a minimal example to reproduce the problem:

I ran into this in a CircleCI build; All builds would fail with

Server terminated abruptly (error code: 14, error message: '', log file: '/root/.cache/bazel/_bazel_root/f85b6fb5740e6e8c7efea142eec4b6e8/server/jvm.out')

until I added build --local_resources=4096,4,1.0 to my .bazelrc.

Circle's build containers report 60G of RAM, but are cgroup-limited to 4G, so building any large application on Circle ought reproduce the issue.

Environment info

• Operating System:
  Linux; Tested on Ubuntu 16.04

• Bazel version (output of bazel info release):
  release 0.5.4

• If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):

Have you found anything relevant by searching the web?

(e.g. StackOverflow answers,
GitHub issues,
email threads on the bazel-discuss Google group)

There are a number of reports online of people puzzling with bazel OOM-ing. It's hard to know how many of them root-cause to this issue, but almost certainly some of them do, since container environments are increasingly popular these days.

• Bazel fails during analysis phase - "Server terminated abruptly" #3020
• Compiling Error with empty error message tensorflow/tensorflow#9940
• Remote Worker failures on bazel build #3251

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

https://fabiokung.com/2014/03/13/memory-inside-linux-containers/ has some notes on how to detect memory availability inside containers.

[Steve-Munday]

I can still reproduce this issue with our large app on Circle CI using Bazel 0.10.0

I configured Circle to use their "large" container which includes 8GB of RAM, and set my bazel.rc file to use --local_resources=4096,4.0,1.0 and it still dies intermittently. See my related comment on #4616

[philwo]

Yes. Bazel's resource management is based on guesstimates as to how much RAM / CPU / IO a certain action will need. If you have a container with just 4GB RAM and no swap space, stuff will horribly die, but so would a physical machine with 4GB RAM, no allowed memory overcommit and no swap space (which is what it looks like CircleCI is doing here).

I think the only way to fix this is to limit the parallelism, e.g. try to run with --jobs=4 and adjust up or down as needed.

If you have an idea how Bazel could solve this better, I'd be very happy to hear about it :)

[Steve-Munday]

Thanks @philwo that's a good suggestion. I have added it and will continue to monitor results. So far so good. For resource limitation I am using this in Circle CI (in a container with 8 GB of RAM):

  resource_class: large
  environment:
    JVM_OPTS: -Xmx6144m

and using this in tools/bazel.rc:

build --local_resources=6144,4,1.0 --jobs=4

[jmmv]

It seems that it's possible to query a container's own memory limit via /sys/fs/cgroup/memory/memory.limit_in_bytes, so we should be checking that first, if present, to calculate total RAM, and then falling back to the existing system-wide mechanism.

I think there is another open bug about CPU limits in this same context, and those details are also available via /sys/fs/cgroup/cpuset/.... Will dedup against this if/when I find it.

[ulfjack]

Ideally, it'll work with both cgroups v1 and v2.

[dmivankov]

This issue is still present even in bazel 4.x.
There are also newer options are called --local_cpu_resources and --local_ram_resources but default limits still are coming from /proc/cpuinfo and /proc/meminfo on Linux

[philwo]

Bazel 4.x uses OpenJDK 11, which has the container resource detection improvements from https://bugs.openjdk.java.net/browse/JDK-8146115 and thus everything "should" just work out of the box if we'd just use the Java APIs to get the available RAM and CPU. 🤔

However, it looks like we're trying to be too smart and parse /proc/cpuinfo and /proc/meminfo ourselves.

I'll try to find someone who can look into this so we can fix it.

[jwnimmer-tri]

I think there is another open bug about CPU limits in this same context, and those details are also available via /sys/fs/cgroup/cpuset/.... Will dedup against this if/when I find it.

FYI that might be #5042.

[dmivankov]

Interesting thing to investigate is also bazel server jvm resource consumption vs actions resources

• is bazel jvm current cpu/mem consumption subtracted from local_{cpu,ram}_resources
• is bazel jvm mem ever/often released to OS

[karlhepler]

Is this still an issue in Bazel 5+?

[jfirebaugh]

Yes, this is still an issue. We run Bazel 5.2.0 on CircleCI at Figma and have encountered this.

[jfirebaugh]

Here's our workaround:

if [ -f /sys/fs/cgroup/memory/memory.limit_in_bytes ]; then
  echo "build --local_ram_resources=$(($(cat /sys/fs/cgroup/memory/memory.limit_in_bytes) / 1048576))" >>$HOME/.bazelrc
fi
if [ -f /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]; then
  echo "build --local_cpu_resources=$(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) / $(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us)))" >>$HOME/.bazelrc
fi

[larsrc-google]

Still an issue.

[larsrc-google]

No, I do think #16512 fixes this.

[fmeum]

@larsrc-google My PR doesn't fix this for RAM limits, only CPU shares.

[larsrc-google]

The VM itself does use the cgroup memory limit for the heap, but obviously that doesn't help us if we can't access that value. We could potentially use the cgroups handing I introduced recently, then take the lower of the available values. It would need a bit of tweaking to accept being in read-only mode, but not too much.

[iancha1992]

A fix for this issue has been included in Bazel 7.1.0 RC1. Please test out the release candidate and report any issues as soon as possible. Thanks!