CentOS 6.7 building with custom GCC

[stephenroller]

Hi all,

Thanks for your hard work on the project.

I've been trying to get TensorFlow working on a cluster which runs on CentOS 6.7 with a custom GCC. I can't use the binary releases because of the older version of glibc, which we do not have the option of upgrading (cluster supports hundreds of other users who are infinitely more important than me). So the route to victory here is Compile Bazel -> Compile Tensorflow -> Profit.

This issue seemed to be resolved in #760, but here I am. I'm using HEAD, but the also tried 0.2.2b, but I haven't tried 0.1.1.

The new cc_configure.bzl seems to successfully load my environment variables (woot!) and set the proper options for part of ./compile.sh. I get past the bootstrapping part (where we historically usually see complaints about GLIB being too old), and then make it to the Building Bazel with Bazel step.

INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    ./compile.sh compile /path/to/bazel
🍃  Building Bazel from scratch......
🍃  Building Bazel with Bazel.
WARNING: /work/01813/roller/maverick/packages/bazel/src/bazel/src/main/native/BUILD:46:16: in includes attribute of cc_binary rule //src/main/native:libunix.so: '.' resolves to 'src/main/native' not in 'third_party'. This will be an error in the future.
WARNING: /work/01813/roller/maverick/packages/bazel/src/bazel/src/main/protobuf/BUILD:37:1: in includes attribute of cc_library rule //src/main/protobuf:command_server_cc_proto: '.' resolves to 'src/main/protobuf' not in 'third_party'. This will be an error in the future. Since this rule was created by the macro 'cc_grpc_library', the error might have been caused by the macro implementation in /work/01813/roller/maverick/packages/bazel/src/bazel/tools/build_rules/genproto.bzl:104:18.
INFO: Found 1 target...
>>>>> # //src:java-version [action 'Executing genrule //src:java-version']
(cd /tmp/bazel.HerjB3ym/out/bazel && \
  exec env - \
    PATH=/opt/apps/gcc/4.9.1/bin:/opt/apps/git/2.7.0/bin:/usr/local/bin:/usr/bin:/bin \
    TMPDIR=/tmp \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh;
          VERSION_LINE=$(cat bazel-out/local-fastbuild/bin/src/java_toolchain_content | grep target_version);
          JAVA_VERSION=$(echo ${VERSION_LINE} | sed '\''s/.*value=\"\([^\"]\)\".*/\1/'\'');
          if [ -z "${JAVA_VERSION}" ]; then
            echo "1.8" >bazel-out/local-fastbuild/genfiles/src/java.version  # Java 8 is the default
          elif [[ "${JAVA_VERSION}" =~ ^[0-9]+$ ]]; then
            echo "1.${JAVA_VERSION}" >bazel-out/local-fastbuild/genfiles/src/java.version  # Add 1. before 7 or 8
          else
            echo "${JAVA_VERSION}" >bazel-out/local-fastbuild/genfiles/src/java.version
          fi
          ')
>>>>> # //src/main/tools:process-tools [action 'Compiling src/main/tools/process-tools.c']
(cd /tmp/bazel.HerjB3ym/out/bazel && \
  exec env - \
    PATH=/opt/apps/gcc/4.9.1/bin:/opt/apps/git/2.7.0/bin:/usr/local/bin:/usr/bin:/bin \
    TMPDIR=/tmp \
  /opt/apps/gcc/4.9.1/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -B/opt/apps/gcc/4.9.1/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 '-std=c99' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local-fastbuild/bin/src/main/tools/_objs/process-tools/src/main/tools/process-tools.pic.o' -MD -MF bazel-out/local-fastbuild/bin/src/main/tools/_objs/process-tools/src/main/tools/process-tools.pic.d -fPIC -c src/main/tools/process-tools.c -o bazel-out/local-fastbuild/bin/src/main/tools/_objs/process-tools/src/main/tools/process-tools.pic.o)
ERROR: /work/01813/roller/maverick/packages/bazel/src/bazel/src/main/tools/BUILD:11:1: C++ compilation of rule '//src/main/tools:process-tools' failed: gcc failed: error executing command
  (cd /tmp/bazel.HerjB3ym/out/bazel && \
  exec env - \
    PATH=/opt/apps/gcc/4.9.1/bin:/opt/apps/git/2.7.0/bin:/usr/local/bin:/usr/bin:/bin \
    TMPDIR=/tmp \
  /opt/apps/gcc/4.9.1/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -B/opt/apps/gcc/4.9.1/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -iquote . -iquote bazel-out/local-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/local-fastbuild/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 '-std=c99' -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/local-fastbuild/bin/src/main/tools/_objs/process-tools/src/main/tools/process-tools.pic.o' -MD -MF bazel-out/local-fastbuild/bin/src/main/tools/_objs/process-tools/src/main/tools/process-tools.pic.d -fPIC -c src/main/tools/process-tools.c -o bazel-out/local-fastbuild/bin/src/main/tools/_objs/process-tools/src/main/tools/process-tools.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
/opt/apps/gcc/4.9.1/libexec/gcc/x86_64-unknown-linux-gnu/4.9.1/cc1: error while loading shared libraries: libcloog-isl.so.4: cannot open shared object file: No such file or directory
Target //src:bazel failed to build
INFO: Elapsed time: 2.332s, Critical Path: 0.01s

Running the failed command also fails, with the same error message. However, if I drop the "exec env - PATH" bit, then GCC sees the LD_LIBRARY_PATH and naturally finds the elusive libcloog-isl.so.4. So somewhere along the building Bazel with Bazel step, the options inferred in cc_configure.bzl are getting lost.

Does anyone have any suggestions?

(Just to make sure i'm not insane, the following confirms libcloog-isl does exist...)

$ ls /opt/apps/gcc/4.9.1/lib | grep cloog
cloog-isl
libcloog-isl.a
libcloog-isl.la
libcloog-isl.so
libcloog-isl.so.4
libcloog-isl.so.4.0.0

Environment

PATH=/opt/apps/gcc/4.9.1/bin:/opt/apps/git/2.7.0/bin:/usr/local/bin:/usr/bin:/bin
CC=/opt/apps/gcc/4.9.1/bin/gcc
CXX=/opt/apps/gcc/4.9.1/bin/cpp
CPLUS_INCLUDE_PATH=/opt/apps/gcc/4.9.1/lib/gcc/x86_64-unknown-linux-gnu/4.9.1/include/:/opt/apps/gcc/4.9.1/lib/gcc/x86_64-unknown-linux-gnu/4.9.1/include-fixed:/opt/apps/gcc/4.9.1/lib/gcc/x86_64-unknown-linux-gnu/4.9.1/include-fixed
LD_LIBRARY_PATH=/opt/apps/gcc/4.9.1/lib:/opt/apps/gcc/4.9.1/lib64
EXTRA_BAZEL_ARGS="-s --verbose_failures --ignore_unsupported_sandboxing --genrule_strategy=standalone --spawn_strategy=standalone --jobs=1"

[stephenroller]

Ahh shucks. It seems that GCC's cc1 and cc1plus couldn't find the necessary libcloog-isl library. An unfortunate result of the way the admin's have set up so users have multiple GCC compilers available, requiring that LD_LIBRARY_PATH be set to use them (which is automatically set by some magic shell scripts of theirs).

I ended up writing some sad little shell wrappers to call gcc, cpp, cc1 and cc1plus with the necessary LD_LIBRARY_PATH set, and this resolved the issue.

A more principled solution would have been to recompile GCC from scratch in my own directory.

[damienmg]

fwiw it looks as the same issue seen in #1152

[GekTo]

@stephenroller I meet the same error, but with the libmpc library, could you please give more details of how you solve the problem?

[stephenroller]

Yeah this is real hackish, but I ended up making 5 bash scripts: gcc, g++, cpp, cc1 and cc1plus. All of them were:

#/bin/bash
LD_LIBRARY_FLAGS=/necessary/paths:/here /path/to/real/gcc "${@}"

These were all put inside a hackbin/ folder inside bazel source, then added $BAZELSRC/hackbin to my $PATH, and updated CROSSTOOL appropriately. Then the compiler called these bash scripts which set environmental flags needed.

Really, really bad hack. But it got it to compile.

Edit: Fixed the lack of $@, thanks

[GekTo]

thanks, it works for me. Create those file and chmod +x for them, and should have "$@" at the end of the line.
Sadly, the compilation now terminates when compiling for zlib:

>>>>> # //third_party/zlib:zlib [action 'Compiling third_party/zlib/gzlib.c [for host]']
(cd /tmp/bazel.R37YRumo/out/bazel- && \
  exec env - \
    PATH=/HOME/sysu_sc_ll/WORKSPACE/tf-rh/bazel-/hackbin/:/WORK/app/Python/2.7.9-icc15/bin:/WORK/app/gcc/4.9.2/bin:/WORK/app/java/jdk1.8.0_11/bin:/HOME/sysu_sc_ll/env/bin:/WORK/app/modules/3.2.10/Modules/3.2.10/bin:/usr/lib/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/bin:/HOME/sysu_sc_ll/software/matlab/bin:/HOME/sysu_sc_ll/.local/bin:/HOME/sysu_sc_ll/bin:/HOME/sysu_sc_ll/bin:/WORK/app/osenv/ln1/usr/local/mpi3-dynamic/bin:/WORK/app/osenv/ln1/usr/lib64/qt-3.3/bin:/WORK/app/osenv/ln1/usr/kerberos/sbin:/WORK/app/osenv/ln1/usr/kerberos/bin:/WORK/app/osenv/ln1/usr/local/bin:/WORK/app/osenv/ln1/bin:/WORK/app/osenv/ln1/usr/bin:/WORK/app/osenv/ln1/usr/local/sbin:/WORK/app/osenv/ln1/sbin:/WORK/app/osenv/ln1/usr/libexec/gcc/x86_64-redhat-linux/3.4.6 \
  /HOME/sysu_sc_ll/WORKSPACE/tf-rh/bazel-/hackbin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -B/HOME/sysu_sc_ll/WORKSPACE/tf-rh/bazel-/hackbin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -g0 -iquote . -iquote bazel-out/host/genfiles -iquote external/bazel_tools -iquote bazel-out/host/genfiles/external/bazel_tools -isystem third_party/zlib -isystem bazel-out/host/genfiles/third_party/zlib -isystem external/bazel_tools/tools/cpp/gcc3 -w -no-canonical-prefixes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-frandom-seed=bazel-out/host/bin/third_party/zlib/_objs/zlib/third_party/zlib/gzlib.o' -MD -MF bazel-out/host/bin/third_party/zlib/_objs/zlib/third_party/zlib/gzlib.d -c third_party/zlib/gzlib.c -o bazel-out/host/bin/third_party/zlib/_objs/zlib/third_party/zlib/gzlib.o)
ERROR: /HOME/sysu_sc_ll/WORKSPACE/tf-rh/bazel-/third_party/zlib/BUILD:9:1: error while parsing .d file: /tmp/bazel.R37YRumo/out/bazel-/bazel-out/host/bin/third_party/zlib/_objs/zlib/third_party/zlib/adler32.d (No such file or directory).

@damienmg @stephenroller , how should i handle it ?

So strange， the problem is fixed now.
While waiting for help, I tried to build and test with the GRPC, which build the protobuf again. And then I add those bins into PATH and those lib to the LD_LIBRARY_PATH. And then wake up this morning, things just went right, and I get bazel compiled.