Automatic cluster state export upon test timeout

[fjetter]

We are frequently bothered by unit tests timing out. Most of the timeout are not easily reproducible and are hard to debug. In the past, these timeouts could often be linked to a known issue, deadlocking the cluster. Often this deadlock can be investigated by inspecting the cluster state.

For a manual extraction of the cluster state, I once wrote #5068 which is trying to create a clean, serializable representation of the cluster.

Upon test timeout, this cluster dump could be persisted and archived as an artifact of the GH actions runner.

For instance, if gen_cluster fixture is used with a client, a timeout exception could be handled and the state is persisted, see

distributed/distributed/utils_test.py

Lines 954 to 964 in 6a0217e

	try:
	future = func(*args, *outer_args, **kwargs)
	future = asyncio.wait_for(future, timeout)
	result = await future
	if s.validate:
	s.validate_state()
	finally:
	if client and c.status not in ("closing", "closed"):
	await c._close(fast=s.status == Status.closed)
	await end_cluster(s, workers)
	await asyncio.wait_for(cleanup_global_workers(), 1)

[jrbourbeau]

Closed via #5470