[BUG] Research associating MemoryCleaner to Spark's ShutdownHookManager

[abellina]

This task could live in cuDF but I am putting it here since it is Spark related. We can add another issue in cuDF if we want to proceed.

This issue: #4669 brought to light a shutdown race between JVM's shutdown hooks and Spark's orderly ShutdownHookManager (also triggered by the JVM). Since the JVM doesn't guarantee the order of execution, the MemoryCleaner in cuDF could think there is a leak, which it checks when shutting down, when it really is a false positive since the SparkContext hasn't been stopped (so ExecutorPlugin is still alive, and so is the RapidsBufferCatalog).

Ideally we could add the MemoryCleaner leak check as the lowest priority (last) hook that Spark will trigger, to prevent false positives.

This is only visible when the ref count debug is turned on, so it is not an issue for production jobs, and hence it is low priority.

[res-life]

@abellina We can close this issue via #5854, please double-check.

[abellina]

Yes this is the same thing @res-life. Thanks!