[NO MRG] Use line_profiler to measure time in transitions #4265
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In order to identify any slow parts within the transition methods themselves, this leverages line_profiler and decorates the relevant methods. Should help us get a better picture about what in these methods may take a long time and therefore where effort is best directed.
This should filter out processes that merely import `distributed` or use other functionality that is not actively profiled from the results. As a result should make the results easier to parse.
When looking at the profiling information from transition functions, these particular functions stood out as taking 10% or more of that time. So this decorates them with the profiler so we can get a better idea of what they are doing.
Decorating `_background_send` before the coroutine decorator resulted in some issues and decorating after doesn't tell us what we want to know. So try using a `with` block instead to avoid fiddling with the coroutine decorator.
Also skip profiling `send`. We already understand the `wait` is what takes the bulk of the time.
This is what we need for `line_profiler` to work correctly on Cython compiled modules.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to identify any slow parts within the transition methods themselves, this leverages line_profiler and decorates the relevant methods. Should help us get a better picture about what in these methods may take a long time and therefore where effort is best directed.