Annotation of some comm related methods in the Scheduler #4341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Provides type annotation for `stream_comms` in the form of a locally stored variable in `worker_send`. This should allow the Python C API to used for lookups when the scheduler is Cythonized.
Instead of checking for `"key"` in `msg`, just use `.get(...)` to try and retrieve the `key`. If it fails, we get `None`, which we can check and ignore quickly. If not, we get the value and can proceed to use it. Should avoid needing to check for the presence of `"key"` in `msg` twice.
Should allow more efficient access of `tasks` in Cython as it is known to be a `dict`.
Should allow more efficient access of `client_comms` in Cython as it is known to be a `dict`.
This ends up being a bit faster than catching the error (especially in Cython/C where it knows this error can just be cleared instead of handled). Plus the comparison to `None` is fast in both (in Cython/C this is a simple pointer value comparison similar to comparing to `NULL`).
Should allow more efficient access of `tasks` in Cython as it is known to be a `dict`.
This ends up being a bit faster than catching the error (especially in Cython/C where it knows this error can just be cleared instead of handled). Plus the comparison to `None` is fast in both (in Cython/C this is a simple pointer value comparison similar to comparing to `NULL`).
jakirkham
commented
Dec 9, 2020
| @@ -4201,7 +4201,7 @@ def update_data( | |||
| self.client_desires_keys(keys=list(who_has), client=client) | |||
|
|
|||
| def report_on_key(self, key: str = None, ts: TaskState = None, client: str = None): | |||
| assert (key is None) + (ts is None) == 1, (key, ts) | |||
| assert (key is None) != (ts is None), (key, ts) | |||
There was a problem hiding this comment.
Found another case in the scheduler like this, addressing in PR ( #4342 ).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This does some annotation around comm related methods in the scheduler:
report_on_key,report, andworker_send. Should help speed through the intermediate code needed to send out a message.