[sw, bazel] Compilation of uart_smoketest fails

[glaserf]

Description

When trying to run to uart_smoketest as per the Getting Started command, after (quite) a while I get the following, similar errors:

./bazelisk.sh test --test_output=streamed --sandbox_debug --disk_cache=~/bazel_cache //sw/device/tests:uart_smoketest_sim_verilator

ERROR: /home/glaserf/[..]/opentitan/hw/top_earlgrey/sw/autogen/BUILD:9:11: Compiling hw/top_earlgrey/sw/autogen/top_earlgrey.c failed: (Exit 1): linux-sandbox failed: error executing command 
  (cd /home/glaserf/.cache/bazel/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/97/execroot/lowrisc_opentitan && \
  exec env - \
    BAZEL_CXXOPTS='-std=gnu++14' \
    PATH=/home/glaserf/.cache/bazelisk/downloads/bazelbuild/bazel-5.1.1-linux-x86_64/bin:/home/glaserf/bin/miniconda3/condabin:/home/glaserf/ot_tools/verilator/4.210/bin:/home/glaserf/.local/bin:/home/glaserf/bin/miniconda3/bin:/home/glaserf/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sepp/bin:/usr/local/AppImage/bin:/usr/local/anaconda3/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/glaserf/.cache/bazel/_bazel_glaserf/install/d81761ab5244f5f4735b9254de6662ba/linux-sandbox -t 15 -w /home/glaserf/.cache/bazel/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/97/execroot/lowrisc_opentitan -w /tmp -w /dev/shm -D -- /usr/lib64/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF bazel-out/k8-fastbuild-ST-e846b08c7501/bin/hw/top_earlgrey/sw/autogen/_objs/top_earlgrey/top_earlgrey.pic.d '-frandom-seed=bazel-out/k8-fastbuild-ST-e846b08c7501/bin/hw/top_earlgrey/sw/autogen/_objs/top_earlgrey/top_earlgrey.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild-ST-e846b08c7501/bin '-std=gnu11' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c hw/top_earlgrey/sw/autogen/top_earlgrey.c -o bazel-out/k8-fastbuild-ST-e846b08c7501/bin/hw/top_earlgrey/sw/autogen/_objs/top_earlgrey/top_earlgrey.pic.o)

ERROR: /home/glaserf/[..]/opentitan/sw/device/lib/base/BUILD:71:11: Compiling sw/device/lib/base/bitfield.c failed: (Exit 1): linux-sandbox failed: error executing command 
  (cd /home/glaserf/.cache/bazel/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/92/execroot/lowrisc_opentitan && \
  exec env - \
    BAZEL_CXXOPTS='-std=gnu++14' \
    PATH=/home/glaserf/.cache/bazelisk/downloads/bazelbuild/bazel-5.1.1-linux-x86_64/bin:/home/glaserf/bin/miniconda3/condabin:/home/glaserf/ot_tools/verilator/4.210/bin:/home/glaserf/.local/bin:/home/glaserf/bin/miniconda3/bin:/home/glaserf/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sepp/bin:/usr/local/AppImage/bin:/usr/local/anaconda3/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/glaserf/.cache/bazel/_bazel_glaserf/install/d81761ab5244f5f4735b9254de6662ba/linux-sandbox -t 15 -w /home/glaserf/.cache/bazel/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/92/execroot/lowrisc_opentitan -w /tmp -w /dev/shm -D -- /usr/lib64/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF bazel-out/k8-fastbuild-ST-e846b08c7501/bin/sw/device/lib/base/_objs/bitfield/bitfield.pic.d '-frandom-seed=bazel-out/k8-fastbuild-ST-e846b08c7501/bin/sw/device/lib/base/_objs/bitfield/bitfield.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild-ST-e846b08c7501/bin '-std=gnu11' -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c sw/device/lib/base/bitfield.c -o bazel-out/k8-fastbuild-ST-e846b08c7501/bin/sw/device/lib/base/_objs/bitfield/bitfield.pic.o)

I already added the --sandbox_debug toggle but honestly can't make much sense out of the error messages above. Any help would be greatly appreciated.

[glaserf]

I was able to narrow the problem down a bit by manually executing the failing command in the respective directory. Then, the error I get is

1687945598.037190615: src/main/tools/linux-sandbox-pid1.cc:292: writable: /scratch/glaserf/bazel_build/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/24/execroot/lowrisc_opentitan
src/main/tools/linux-sandbox-pid1.cc:295: "mount(/scratch/glaserf/bazel_build/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/24/execroot/lowrisc_opentitan, /scratch/glaserf/bazel_build/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/24/execroot/lowrisc_opentitan, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory

The problem seems to be somewhere internally in Bazels build directory structure. When I do

ls /scratch/glaserf/bazel_build/_bazel_glaserf/70f1faabbc86ee8cc3125d2e06c49463/sandbox/linux-sandbox/

I get

1/  10/ 11/ 12/ 13/ 14/ 2/  3/  4/  5/  6/  7/  8/  9/

but not the request 24 subdirectory. Any hints for where this might come from and how to resolve it?

[a-will]

Any chance you are using NFS for bazel's outputs? bazel would have warned that this leads to surprising failures.

[glaserf]

Thanks for the hint towards NFS directories.

This was indeed the case in the beginning, but after figuring out how to change the output directory via $TEST_TMPDIR from here (the also documented --output_user_root throws an unknown argument error), the failure mode remains the same (includes the issue above).

Additionally, I switched from CentOS 7 to Alma Linux 8, again with no success.

The issue above seems to be resolved by deleting the output directory and rebuilding from scratch, not exactly sure how.
Next, Bazel complained about a non-writable .ccache directory, which I had to make writable from the sandbox as per here (manually add --sandbox_writable_path=$HOME/.ccache).

Now I am stuck at

src/main/tools/linux-sandbox-pid1.cc:478: "execvp(srec_cat, 0x164bba0)": No such file or directory

which I guess is a result from srec_cat not being installed on my machine - a tool to manipulagte EPROM files?

[a-will]

Yes, missing srecord:

opentitan/yum-requirements.txt

Line 45 in 43ad38b

srecord

If you're back to Alma Linux 8, you'll have to roll your own installation of that, since the package wasn't built for EPEL 8. One option would be to build the RPM yourself (assumes you have rpmdevtools and all the dependencies for the package build installed):

# Get the srecord build from Fedora
git clone https://src.fedoraproject.org/rpms/srecord.git
# Download the sources
spectool -g -R srecord/srecord.spec
# Build the RPM
rpmbuild -bb srecord/srecord.spec
# Install the RPM (actual path may depend on where your RPM dev tree is--I forget)
dnf localinstall -y ./rpmbuild/RPMS/x86_64/srecord-1.64-24.el8.x86_64.rpm

We use srec_cat to generate the Verilog memory files for simulation.

[glaserf]

Thanks for the clarification of srecord. In the end this was a consequence of me being on a managed machine.
I was able to compile srecord from sources and install in my home, along with updated versions of cmake and doxygen.

Additionally, I had to manually re-export CC="/usr/bin/gcc" and export CXX="/usr/bin/g++" as per here, after which I finally got

INFO: Elapsed time: 911.747s, Critical Path: 879.66s
INFO: 592 processes: 91 internal, 498 linux-sandbox, 3 local.
INFO: Build completed successfully, 592 total actions
//sw/device/tests:uart_smoketest_sim_verilator                           PASSED in 40.6s

INFO: Build completed successfully, 592 total actions

So, using my updated bazel or bazelisk.sh, respectively, command which looks like this:
./bazelisk.sh test --test_output=streamed --disk_cache=/scratch/glaserf/bazel_cache --sandbox_writable_path=$HOME/.ccache //sw/device/tests:uart_smoketest_sim_verilator along with setting $TEST_TMPDIR to avoid running on NFS and ensuring all dependencies are on required versions or newer, did the trick.

Thanks for the help!