ProcessRunner timeouts in highly parallel Mono invocations

[atruskie]

Expected behaviour

Process runner should capture exit codes and SIGCHLD signals in highly parallel scenarios on all platforms.

Actual behaviour

The ProcessRunner times out when running external executables in highly parallel scenarios. The sub-processes themselves seem to complete successfully but nevertheless timeout. This behaviour only happens when AP.exe is run on the Mono platform.

Steps to reproduce the behaviour

• Platform: Mono (Mono JIT compiler version 5.10.1.47)
• OS: SUSE Linux Enterprise Server 12 SP2
• Container: debian-stretch built from https://github.com/QutEcoacoustics/baw-private/commit/b90ce912bb9decb66148859d3201e6cbd30464cc#diff-d46c280462c00168367ff40b78615769

Any other details

This problem has already been fixed. This issue is being filed to document the fix.

It appears the problem is some kind of resource exhaustion that means the SIGCHLD signal get's swallowed. This means Mono does not unblock from the .Wait call and the sub-process (whether it exited successfully, exited with an error, or has legitimately timed out) will always timeout. This issue is a known issue with the Mono platform:

• https://bugzilla.xamarin.com/show_bug.cgi?id=27246
• (may be related) [Unix] w32process-unix.c mono_w32process_signal_finished breaks all child process status queries mono/mono#6200
• (may be related) Process.WaitForExit() gets slower the larger your open file limit (ulimit -n) is on Linux mono/mono#6555

A series of experimental patches were made (essentially guessing at how to bypass the bug because simulating resource exhaustion was hard). An experimental analysis that always failed was run repeatedly, using incremental patches and builds, on a 48-core machine. Eventually, the patches resulted in a build that worked twice in a row and I determined the problem was fixed.

The logs from the processes:
bhutan_test.zip

And the patches that fixed this issue:

• 59b1152
• 72cc826
• 75bc53b
• 2ffac42
• 5b82297