add shutdown method to CommandRunner

[tdyas]

The Docker command runner needs to cleanly remove cached containers when it shuts down. The removal API must be called in an async context, however, so the shutdown cannot be performed from drop handler since that is a non-async context.

Add a shutdown method to CommandRunner and implement it for all relevant command runners. For the Docker command runner, this means that the shutdown logic for ContainerCache will be invoked in an async context.

[ci skip-build-wheels]

[stuhood]

Thanks!

[stuhood]

Thanks!

[stuhood]

@tdyas : For posterity, here is an example of the hang that was located via #16927: https://gist.github.com/stuhood/d05d4fcc2b2fd7e464fa397c362eb894

Looking at it again, the thing that jumps out at me is that the thing that is being Droped is actually a PySession... and then what I think is happening is that it is triggering Drop for the Session -> Core -> CommandRunner, etc.

So... it's possible that this is because Drop for our Python wrapper types doesn't release the GIL...? Or perhaps you see something more specific in the stack, where we're hanging trying to tear something down?

[stuhood]

Or perhaps you see something more specific in the stack, where we're hanging trying to tear something down?

Oh. Is it possible that we're failing to connect to docker? It looks like we do that regardless of whether we have any container ids to cancel.

That could probably also skip tearing anything down if the DockerOnceCell had never been successfully initialized...

[tdyas]

Oh. Is it possible that we're failing to connect to docker? It looks like we do that regardless of whether we have any container ids to cancel.

That could probably also skip tearing anything down if the DockerOnceCell had never been successfully initialized...

I'll put that fix in now as a separate PR since this one already landed.

[tdyas]

#16951