execvp: catkin_generated/env_cached.sh: Text file busy

[kopp]

System Info

• Operating System: Linux ABTZ0E9E 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 2.7.6
• Version of catkin_tools:

% catkin --version
catkin_tools 0.4.4 (C) 2014-2017 Open Source Robotics Foundation
catkin_tools is released under the Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)
---
Using Python 2.7.6 (default, Oct 26 2016, 20:30:19) [GCC 4.8.4]
% dpkg -l | grep catkin                       
ii  python-catkin-pkg                                     0.3.1-1                                             all          catkin package library
ii  python-catkin-pkg-modules                             0.3.1-1                                             all          catkin package library
ii  python-catkin-tools                                   0.4.4-1                                             all          Command line tools for working with catkin.
ii  ros-jade-catkin                                       0.6.18-0trusty-20160318-200207-0700                 amd64        Low-level build system macros and infrastructure for ROS.

• ROS Distro: `jade``

Build / Run Issue

• happens with catkin build, did not test with catkin_make*, see below.

Expected Behavior

If I type catkin build, my packages should build.

Actual Behavior

Most of the time, it actually works fine, but in some cases I get the following strange error when building a catkin package (called PACKAGE here). If I just do a catkin build again, it typically works. It also happens for different packages with no clear structure.

I typically get this output:

Errors     << PACKAGE:make workspace/logs_current/PACKAGE/build.make.319.log
make[2]: execvp: catkin_generated/env_cached.sh: Text file busy
make[2]: *** [CMakeFiles/_PACKAGE_generate_messages_check_deps_PredictionProbability] Error 127
make[1]: *** [CMakeFiles/_PACKAGE_generate_messages_check_deps_PredictionProbability.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [all] Error 2
cd workspace/build_current/PACKAGE; catkin build --get-env PACKAGE | catkin env -si  /usr/bin/make --jobserver-fds=6,7 -j; cd -


% cat workspace/logs_current/PACKAGE/build.make.319.log                                              :(
make[2]: execvp: catkin_generated/env_cached.sh: Text file busy
make[2]: *** [CMakeFiles/_PACKAGE_generate_messages_check_deps_PredictionProbability] Error 127
make[1]: *** [CMakeFiles/_PACKAGE_generate_messages_check_deps_PredictionProbability.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[  0%] Built target geometry_msgs_generate_messages_py
[  0%] Built target _PACKAGE_generate_messages_check_deps_DynamicWorld
[  0%] Built target _PACKAGE_generate_messages_check_deps_ServerSpeedLimits
make: *** [all] Error 2

Steps to Reproduce the Issue

Unfortunately, it's hard to reproduce, because it happens sporadically. Repeat the following until you see the error:

catkin build

:-(

[mikepurvis]

What is the output of catkin config in this scenario? Are you using the merged or linked develspace?

[kopp]

I'm soo sorry for the late reply. My catkin config is nothing special (I guess); linked and using ccache (see below). I did not capture catkin config from directly before one of these issues, because they happen too sporadically, but I don't think that it would change :-)

% catkin config
-----------------------------------------------------------------------------------------------------------
Profile:                     current
Extending:        [explicit] /opt/ros/jade
Workspace:                   /home/k/workspace
-----------------------------------------------------------------------------------------------------------
Source Space:       [exists] /home/k/workspace/src
Log Space:          [exists] /home/k/workspace/logs_current
Build Space:        [exists] /home/k/workspace/build_current
Devel Space:        [exists] /home/k/workspace/devel_current
Install Space:      [unused] /home/k/workspace/install_current
DESTDIR:            [unused] None
-----------------------------------------------------------------------------------------------------------
Devel Space Layout:          linked
Install Space Layout:        None
-----------------------------------------------------------------------------------------------------------
Additional CMake Args:       -DCMAKE_CXX_COMPILER=/usr/lib/ccache/c++ -DCMAKE_C_COMPILER=/usr/lib/ccache/cc
Additional Make Args:        None
Additional catkin Make Args: None
Internal Make Job Server:    True
Cache Job Environments:      False
-----------------------------------------------------------------------------------------------------------
Whitelisted Packages:        None
Blacklisted Packages:        supervisor_msgs
-----------------------------------------------------------------------------------------------------------
Workspace configuration appears valid.
-----------------------------------------------------------------------------------------------------------

[mikepurvis]

Your description sounds very much like the installspace mutexing problem (#378, #391). I would have thought a linked develspace would be safe from that, but perhaps not. In any case, the fix in #391 only mutexes writes to the installspace, since that's my use case. So at the very least, you could try switching your config to that, eg:

catkin config --isolate-devel --install --merge-install

If you want to play with reproducing these kinds of issues, feel free to rip off my little script for repeatedly building a small workspace in an attempt to trigger the problem, in #378 (comment).

@jbohren is the architect of the linked develspace, so he'd know better how this is all supposed to be handled in there.