Work stealing cpu load with high numbers of tasks

[mrocklin]

I was speaking with @bnaul . He mentioned that he noticed the scheduler at 100% CPU usage when it should have been in a somewhat quiet state. He had on the order of 100,000 tasks and it was spending all of its time in work stealing, in particular the maybe_move_task function. Something there may not be O(1).

[fjetter]

I can't see anything that's not O(1) at first glance. We changed one thing about move_task_request recently that slows it down but is constant time. We introduced a UUID generation in there

https://github.com/dask/distributed/blame/main/distributed/stealing.py#L221-L223

Note: this is the only place where we're abusing the stimulus_id for anything functional

• Resolve work stealing deadlock caused by race in move_task_confirm #5379 (this was the deadlock)
• Ensure uniqueness of steal stimulus ID #5620 (here we introduced the UUID since the simple time caused duplicates)

I opened #6179 to remove the UUID

[mrocklin]

I think that we should just remember to run a larger workload and track profiling here. It also might be O(1) in number of tasks, but O(n) in number of workers, for example.

…

On Fri, Apr 22, 2022 at 12:52 PM Florian Jetter ***@***.***> wrote: I can't see anything that's not O(1) at first glance. We changed one thing about move_task_request recently that slows it down but is constant time. We introduced a UUID generation in there https://github.com/dask/distributed/blame/main/distributed/stealing.py#L221-L223 Note: this is the *only* place where we're abusing the stimulus_id for anything functional - #5379 <#5379> (this was the deadlock) - #5620 <#5620> (here we introduced the UUID since the simple time caused duplicates) I opened #6179 <#6179> to remove the UUID — Reply to this email directly, view it on GitHub <#6178 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTF6YIMRB76UTS7KAIDVGLRO5ANCNFSM5UC4YLKQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[gjoseph92]

If there's a reproducible workload, pyspy_on_scheduler could easily give a definitive answer to the question.

https://github.com/gjoseph92/scheduler-profilers