Retry fork/exec errors when running hook

[triarius]

A customer reported that their jobs were unable to run a hook because the hook wrapper script was locked.

The hooks should have collision avoidant names, and they should have closed by the agent worker goroutine that created them prior to execution, which is in the same goroutine. So it's a bit of a mystery how the script wrapper could remain open.

We could just add more retry to work around this, but it would like to find the cause. So, I've done two things in this PR:

• Retry executing the hook if we get a fork/exec error
• If the retry is exhausted, log the error and try to determine what process has the file open

That latter will only work if:

• the process that has the file open is long-lived enough for the error handling to catch it with the file open, and
• the agent has permission to read the open file descriptors of the other processes

So we can't be certain what the other process is, but at least we should be able to prove or disprove that it is the agent itself.

Detecting other processes only works on Linux because it uses the procfs. For other Unixes, we can use /dev/fd to check that the agent process does not have the file open. I'm not sure if this will work if the two goroutines are scheduled on different OS threads, however.

[triarius]

@moskyb after chatting with @pda, we decided to feed back errors from failing to close files that are opened in NewScriptWrapper. If the files are kept open by the agent, that's the likely cause.