vine, wq: ramp down mode #3485
Merged
vine, wq: ramp down mode #3485
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A queue is hungry if the number of waiting tasks is less than (number of workers) * hungry_minimum_factor.
When enabled (aq.tune("ramp-down-heuristic", 1)), and the number of waiting tasks is less than the number of
connected workers, then tasks are allocated all the free resources of a
worker. This avoids tasks terminated with resource exhaustion when there
are more resources than tasks.
If monitoring is not enabled, or if it just set to measure, then this
heuristic has no effect. By default it is disabled.
|
Nice! I like it when a simple idea has a big impact. Does "ramp down" also apply at any other time (not the end) when tasks are fewer than workers? |
|
Yes, if activated, then are less tasks than workers, then tasks get all free resources of a worker. Ideally whatever is managing vine would activate this mode when it knows the workflow is about to end. (E.g. in topcoffea, when all the processing tasks are done and we only have accumulation tasks.) It can be seen as if the queue is starving (e.g. more than hungry), then be aggressive allocating resources for the tasks. |
dthain
approved these changes
Sep 11, 2023
|
RTM |
Jbrocket
pushed a commit
to Jbrocket/cctools
that referenced
this pull request
Sep 11, 2023
* wq: adds hungry minimum factor tune parameter
A queue is hungry if the number of waiting tasks is less than (number of
workers) * hungry_minimum_factor.
* vine: hungry-minimum-factor
* adds ramp-down mode
When enabled (aq.tune("ramp-down-heuristic", 1)), and the number of waiting tasks is less than the number of
connected workers, then tasks are allocated all the free resources of a
worker. This avoids tasks terminated with resource exhaustion when there
are more resources than tasks.
If monitoring is not enabled, or if it just set to measure, then this
heuristic has no effect. By default it is disabled.
* vine: ramp down mode
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds
m.tune("ramp-down-heuristic", 1). When this is enabled and there are more workers than ready tasks, then tasks are allocated all the free resources of a worker. When looking for workers by files or by time, ties are broken by worst fit. This mode avoids unnecessary resource exhaustion when there are more workers than tasks. If the monitoring watchdog is not enabled, then this heuristic has no effect. By default it is disabled (set to 0).It also adds
m.tune("hungry-minimum-factor", 2). A manager is hungry if the number of waiting tasks is less than then number of workers times this factor. By default is set to 2 to avoid ramp-down mode to trigger when the workflow is not really terminating.With this mode enabled I was able to run the processing+accumulation part of topEFT using the old wq executor in 40 minutes, as the expensive final accumulations are never retried. Together with other advances in taskvine, 30 min processing time should be within reach.