Bazel consumes too much memory and hangs in docker container

[alexeagle]

Description of the problem / feature request / question:

I have a sass_binary rule from https://github.com/bazelbuild/rules_sass
And build in this docker image on CircleCI
https://github.com/alexeagle/ngcontainer/blob/master/Dockerfile

Most of the time the build hangs while compiling C++, but sometimes it succeeds.

If possible, provide a minimal example to reproduce the problem:

https://circleci.com/gh/alexeagle/angular-bazel-example/1
is a (somewhat) minimal example.
Source: https://github.com/alexeagle/angular-bazel-example

Environment info

• Operating System:
  Debian linux (jessie) (in docker)

• Bazel version (output of bazel info release):
  Build label: 0.5.1
  Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
  Build time: Tue Jun 6 10:34:11 2017 (1496745251)
  Build timestamp: 1496745251
  Build timestamp as int: 1496745251

Have you found anything relevant by searching the web?

(e.g. StackOverflow answers,
GitHub issues,
email threads on the bazel-discuss Google group)

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

[alexeagle]

Confirmed this also happens on release 0.5.4

[alexeagle]

Another sample, got a different behavior, instead of hanging it fails with an error:

ERROR: /root/.cache/bazel/_bazel_root/049ed6f058a98cba95753eee774cab7a/external/sassc/BUILD.bazel:4:1: C++ compilation of rule '@sassc//:sassc' failed (Exit 1)

external/libsass/src/sass.cpp:41:1: fatal error: error writing to /tmp/ccLTBo6L.s: Cannot allocate memory
 }
 ^
compilation terminated.
The bug is not reproducible, so it is likely a hardware or OS problem.

[alexeagle]

Hi, this prevents Angular bazel apps from using CircleCI, which is our recommendation.
It also means the canonical demo app
https://github.com/alexeagle/angular-bazel-example
has a flaky build status, which looks bad.

Can someone triage this? I might be able to fix it if I could tell what is wrong.
/cc @davidstanke

[alexeagle]

Ping!

[alexeagle]

@gregmagolan tracked it down (thanks!)

Under the docker container, the amount of memory available to a process is limited
https://docs.docker.com/engine/admin/resource_constraints/

Bazel should determine the amount of available RAM (you hint at this in https://docs.bazel.build/versions/master/user-manual.html#flag--ram_utilization_factor )
but that doesn't account for the cgroup limitation.
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt

(Edited by @laszlocsomor on 2018/06/14: fix broken URL)

[djudd-stripe]

Underlying issues might be #3886 plus #2946?

[laszlocsomor]

/cc: @jhfield @ulfjack @ericfelly

Summary: #3645 (comment)

Who should work on this?

[ulfjack]

We have no idea what's going on there, and I don't think we can debug this without seeing jvm.out. In order to get better debug information in the future, we will also need to add code to bazel client to cat jvm.out.

[ulfjack]

This isn't the first time that not having jvm.out is preventing us from making progress.

[ittaiz]

I think I opened an issue a while back to add documentation on “what should you report” when seeing serious bazel issues (I think I was talking about reproducibility).

…

On Thu, 14 Jun 2018 at 10:51 Ulf Adams ***@***.***> wrote: This isn't the first time that not having jvm.out is preventing us from making progress. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3645 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIF87V8AeAGft38-2Bnc2n13YKwsEuks5t8hYRgaJpZM4PICpo> .

[Globegitter]

I have run into similar issues and the jvm.out has nothing relevant, just the some warnings that I believe are always there.

[meteorcloudy]

@ulfjack This issue is still bothering Angular. Is it possible to make Bazel auto-tune the memory usage (eg. adjusting the parallel job number) according to the RAM size?

[fmeum]

@sgowroji Not stale, has a Bazel team assignee.

[larsrc-google]

I think #16512 does fix this.

[fmeum]

@larsrc-google Please reopen, my fix only applied to CPU shares, not RAM.

[larsrc-google]

The memory usage in general is hard to control - Bazel needs to hold your build graph in memory. The amount of parallel actions can be controlled with --jobs, e.g. --jobs=HOST_RAM*0.01, together with --local_ram_resources.