Generalize UCX errors on connect() and correct pytest fixtures

[pentschev]

Various different errors may happen when trying to connect to a remote endpoint. By generalizing the exception we catch, we leave the responsibility of raising the appropriate error depending on known communication issues to Distributed.

Additionally moved UCX pytest fixture to utils_test.ucx_loop and the argument added to all pytest functions that do UCX testing, which was not being properly used.

Closes #6429

• Tests added / passed
• Passes pre-commit run --all-files

[charlesbluca]

Looks like things timed out on distributed/tests/test_worker.py::test_protocol_from_scheduler_address[Worker]?

[quasiben]

rerun tests

[quasiben]

with #6428 in we can rerun and hopefully resolve UCX issues

[quasiben]

rerun tests

[quasiben]

rerun tests

[jacobtomlinson]

This looks good. Does this change mean we need to bump minimum versions anywhere?

[pentschev]

I can reproduce the CI hang locally but only if running the entire gpuCI test set, (i.e., not when running on stributed/tests/test_worker.py::test_protocol_from_scheduler_address alone). I'm investigating.

[pentschev]

This looks good. Does this change mean we need to bump minimum versions anywhere?

No, it just has been generalized so that the exception is (hopefully) always an OSError. It's possible that the version assumption was incorrect in the past from conda packages vs local builds for the same reasons I mentioned in #6429 (comment) , but I'm not sure about that.

[github-actions]

Unit Test Results

       15 files  ±  0         15 suites  ±0   6h 7m 23s ⏱️ - 26m 20s
  2 812 tests +  2    2 727 ✔️  - 1    82 💤 +  3  2 ❌  - 1  1 🔥 +1 
20 848 runs  +15  19 900 ✔️ +3  944 💤 +12  3 ❌  - 1  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit e95fb9f. ± Comparison against base commit d32f4b0.

♻️ This comment has been updated with latest results.

[quasiben]

The failures here are for known failing test test_stress_scatter_death . @fjetter as this does add new functions to utils_test.py would you care to review ? If not, this is ready to be merged in

[pentschev]

To add to @quasiben's comment above, here is a more detailed list of failing tests:

• distributed/dashboard/tests/test_scheduler_bokeh.py::test_basic
• distributed/dashboard/tests/test_worker_bokeh.py::test_services_kwargs
• distributed/dashboard/tests/test_scheduler_bokeh.py::test_memory_by_key
• distributed/tests/test_stress.py::test_stress_scatter_death (3 instances)

I believe none of them are related to this PR, but please let me know if you think they are related somehow.

[jacobtomlinson]

test_stress_scatter_death is a known issue but I'm just going to restart CI to see if the others pop up again.

[pentschev]

Failed tests now are:

• distributed/tests/test_stress.py::test_stress_scatter_death (all 3 instances);
• distributed/diagnostics/tests/test_progress.py::test_group_timing.

I still seems that neither of the errors are related to this PR, but let me know if there's reason to believe otherwise.

[jacobtomlinson]

Yeah I agree they are unrelated. Thanks for handling this @pentschev.