sporadic UnicodeDecodeError errors during symlink stage

[rhaschke]

System Info

• Operating System: Linux idas 4.4.0-34-generic Make compiler warnings visible #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Python Version: Python 2.7.12
• Version of catkin_tools: 2442e1f
• ROS Distro: Kinetic

Build / Run Issue

• [-] Works with catkin build
• [-] Works with catkin build -p1

Actual Behavior

Sporadically, catkin fails building during symlink stage with the following error:

Finished   << srdfdom:make                                                                                                                                                             
Starting   >> srdfdom:symlink                                                                                                                                                          
_______________________________________________________________________________________________________________________________________________________________________________________
Errors     << srdfdom:symlink /vol/sandbox/ros/logs/srdfdom/build.symlink.018.log                                                                                                      
Linked: (/vol/sandbox/ros/devel/.private/srdfdom/lib/libsrdfdom.so, /vol/sandbox/ros/devel/lib/libsrdfdom.so)
Stage `symlink` failed with arguments:
  package: {'package_format': 1, 'exports': [], 'buildtool_export_depends': [], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x7feb24561b90>, <catkin_pkg.package.Dependency object at 0x7feb24561bf0>, <catkin_pkg.package.Dependency object at 0x7feb24561c50>, <catkin_pkg.package.Dependency object at 0x7feb24561cb0>, <catkin_pkg.package.Dependency object at 0x7feb24561d10>], 'name': 'srdfdom', 'filename': '/vol/sandbox/ros/src.moveit/srdfdom/package.xml', 'replaces': [], 'version_abi': None, 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x7feb24561d70>, <catkin_pkg.package.Dependency object at 0x7feb24561dd0>, <catkin_pkg.package.Dependency object at 0x7feb24561e30>, <catkin_pkg.package.Dependency object at 0x7feb24561e90>, <catkin_pkg.package.Dependency object at 0x7feb24561ef0>], 'doc_depends': [], 'test_depends': [<catkin_pkg.package.Dependency object at 0x7feb24561f50>], 'maintainers': [<catkin_pkg.package.Person object at 0x7feb24562b10>, <catkin_pkg.package.Person object at 0x7feb24562b90>], 'version': '0.3.1', 'build_depends': [<catkin_pkg.package.Dependency object at 0x7feb24561fb0>, <catkin_pkg.package.Dependency object at 0x7feb24563050>, <catkin_pkg.package.Dependency object at 0x7feb245630b0>, <catkin_pkg.package.Dependency object at 0x7feb24563110>, <catkin_pkg.package.Dependency object at 0x7feb24563170>, <catkin_pkg.package.Dependency object at 0x7feb245631d0>], 'urls': [<catkin_pkg.package.Url object at 0x7feb24562d90>, <catkin_pkg.package.Url object at 0x7feb24562e10>, <catkin_pkg.package.Url object at 0x7feb24562e90>], 'authors': [<catkin_pkg.package.Person object at 0x7feb24562f10>, <catkin_pkg.package.Person object at 0x7feb24562f90>], 'licenses': ['BSD'], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x7feb24563230>], 'conflicts': [], 'description': u'Parser for Semantic Robot Description Format (SRDF).'}
  metadata_path: /vol/sandbox/ros/.catkin_tools/profiles/moveit
  devel_manifest_path: /vol/sandbox/ros/.catkin_tools/profiles/moveit/packages/srdfdom
  prebuild: False
  package_path: srdfdom
  dest_devel_path: /vol/sandbox/ros/devel
  source_devel_path: /vol/sandbox/ros/devel/.private/srdfdom
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/catkin_tools/execution/executor.py", line 146, in async_job
    event_queue))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 1: ordinal not in range(128)

Steps to Reproduce the Issue

I did observed that behaviour only twice out of 50. Removing the build dir of the failing package doesn't change anything. However, cleaning everything and rebuilding from scratch fixed the issue. However, that's frustrating if the build of the workspace takes 30min.

[gavanderhoorn]

However, that's frustrating if the build of the workspace takes 30min.

A bit off-topic (and obviously not an answer), but I've been using ccache for a long time now, exactly because of this. A full MoveIt rebuild with the cache is a matter of minutes.

[ahoarau]

same here with a different package. +100 for ccache and distcc !

[rhaschke]

I tracked the bug down to the computation of md5 hashes on utf-8 encoded file content. I guess, performing utf8 encoding is wrong here, because it's not given that the files are utf-8 encoded. Removing utf8 encoding solves the issue.
I encountered the problem on a .pyc file that was probably generated during catkin run_tests. I guess those shouldn't be symlinked at all.

[rhaschke]

This relates to #311 and to #368. However, #368 doesn't resolve the underlying issue here.

[rhaschke]

Duplicate of #311. Fix proposed in #399 and #368.