Fix performance related to dynamic scheduler scaling #2751
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As part of performance testing Wallaroo using multiple workers,
@JONBRWN discovered a regression in both throughput and latency.
He tracked the issue down the commit that re-enabled dynamic
scheduler scaling (fc80968).
NOTE: This performance issue did not exist for singler worker
runs of Wallaroo.
Some head scratching and testing led to the current commit to
resolve the multi-worker performance issue. My best guess is that
before this change the
steal
loop was dependent on a memoryaccess to determine if dynamic scheduler scaling needed to
suspend a thread or not as its initial check. This would lead to
somewhat erratic behavior where some times the
steal
loop wouldtake long while other times it wouldn't depending on how long the
memory load took. This had a follow-on impact on actor execution
because of ASIO messages because they wouldn't be picked up off
of the queue for work as quickly as they could be due to the
extra memory accesses.
This commit changes the ordering of some operations to ensure
that there is more consistent memory accesses for the loop
resulting in more consistent actor actor execution for ASIO
messages resolving the multi-worker performance issue that
@JONBRWN discovered.