docker on v1.12.2: StartHost failed during initscript (fix_cgroup, SIGPIPE)

[tstromberg]

This only recovered because --delete-on-failure was set:

! StartHost failed, but will try again: creating host: create: creating: create kic node: container name ...

As best as I can tell, the initscript crashed here:

2020-08-11T17:59:28.714339506Z  ++ cut '-d ' -f 4
2020-08-11T17:59:28.715572766Z  ++ head -n 1
2020-08-11T17:59:28.717012478Z  ++ echo '535 534 0:29 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:138 master:18 - cgroup cpuset rw,cpuset
2020-08-11T17:59:28.717032922Z  536 534 0:30 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/cpu rw,nosuid,nodev,noexec,relatime shared:139 master:19 - cgroup cpu rw,cpu
2020-08-11T17:59:28.717036987Z  537 534 0:31 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/cpuacct rw,nosuid,nodev,noexec,relatime shared:140 master:20 - cgroup cpuacct rw,cpuacct
2020-08-11T17:59:28.717040276Z  538 534 0:32 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:141 master:21 - cgroup blkio rw,blkio
2020-08-11T17:59:28.717053926Z  539 534 0:33 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:142 master:22 - cgroup memory rw,memory
2020-08-11T17:59:28.717057990Z  540 534 0:34 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:143 master:23 - cgroup devices rw,devices
2020-08-11T17:59:28.717061284Z  541 534 0:35 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:144 master:24 - cgroup freezer rw,freezer
2020-08-11T17:59:28.717064460Z  542 534 0:36 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/net_cls rw,nosuid,nodev,noexec,relatime shared:145 master:25 - cgroup net_cls rw,net_cls
2020-08-11T17:59:28.717067647Z  543 534 0:37 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:146 master:26 - cgroup perf_event rw,perf_event
2020-08-11T17:59:28.717070944Z  544 534 0:38 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/net_prio rw,nosuid,nodev,noexec,relatime shared:147 master:27 - cgroup net_prio rw,net_prio
2020-08-11T17:59:28.717074699Z  545 534 0:39 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:148 master:28 - cgroup hugetlb rw,hugetlb
2020-08-11T17:59:28.717079332Z  546 534 0:40 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:149 master:29 - cgroup pids rw,pids
2020-08-11T17:59:28.717082749Z  548 534 0:42 /docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206 /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:151 master:31 - cgroup cgroup rw,name=systemd'
2020-08-11T17:59:28.718217347Z  + docker_cgroup=/docker/562a56986a84b3cd38d6a32ac43fdfcc8ad4d2473acf2839cbf549273f35c206

Here is the full crash log:

crash.log

[tstromberg]

Here is the place of fail, as best as I can tell:

docker_cgroup=$(echo "${docker_cgroup_mounts}" | head -n 1 | cut -d' ' -f 4)

I suspect a race condition here.