Refactor find_missing and refresh_who_has

[crusaderky]

• Contributes to Migrate ensure_communicating transitions to new WorkerState event mechanism #5896
• Blocked by and incorporates update_who_has can remove workers #6342

Additional changes to #6342 are visible at crusaderky#3

Refactor Worker->Scheduler queries to fetch who_has, from RPC to bulk comms. This removes one of the two async commands in gather_dep (the other being the actual data request) and is propaedeutic to its refactoring.

Fix minor bug where:

1. all workers that hold a replica of a key are busy
2. gather_dep triggers a query to scheduler.who_has to get more workers
3. the scheduler replies with new workers
4. nothing happens until the next call to _ensure_communicating, which will happen at the latest 150ms later (when a worker gets out of busy state).

[github-actions]

Unit Test Results

       15 files  ±  0         15 suites  ±0   6h 27m 49s ⏱️ - 6m 5s
  2 826 tests +  7    2 745 ✔️ +  9    81 💤 +1  0 ❌  - 3 
20 945 runs  +49  20 002 ✔️ +44  943 💤 +8  0 ❌  - 3 

Results for commit 1bc0283. ± Comparison against base commit 5feb171.

♻️ This comment has been updated with latest results.

[crusaderky]

As the git tree has become messy, until #6342 is merged the diff of this PR is visible at crusaderky#3

[crusaderky]

@fjetter this has been updated to reflect the new update_who_has.
Diff here: crusaderky#3

[crusaderky]

Ironically, with this PR some changes that we just introduced in the update_who_has PR can now be reverted: df488f0

[fjetter]

+1 LGTM

Just for reference, there is a discussion going on that the entire find-missing mechanism can likely be removed entirely. I'm fine with this refactoring that moves us closer to a more robust architecture see #6445

[fjetter]

I assume you're logging this because it is flooding the log otherwise?

[crusaderky]

Correct. Plus, it's really not interesting to see in the log even when relevant. Note that the response from the scheduler is logged.