Skip to content

Commit

Permalink
Fix multi-worker performance related to dynamic scheduler scaling
Browse files Browse the repository at this point in the history
As part of performance testing Wallaroo using multiple workers,
@JONBRWN discovered a regression in both throughput and latency.
He tracked the issue down the commit that re-enabled dynamic
scheduler scaling (fc80968).
NOTE: This performance issue did not exist for singler worker
runs of Wallaroo.

Some head scratching and testing led to the current commit to
resolve the multi-worker performance issue. My best guess is that
before this change the `steal` loop was dependent on a memory
access to determine if dynamic scheduler scaling needed to
suspend a thread or not as its initial check. This would lead to
somewhat erratic behavior where some times the `steal` loop would
take long while other times it wouldn't depending on how long the
memory load took. This had a follow-on impact on actor execution
because of ASIO messages because they wouldn't be picked up off
of the queue for work as quickly as they could be due to the
extra memory accesses.

This commit changes the ordering of some operations to ensure
that there is more consistent memory accesses for the loop
resulting in more consistent actor actor execution for ASIO
messages resolving the multi-worker performance issue that
@JONBRWN discovered.
  • Loading branch information
dipinhora authored and SeanTAllen committed Jun 6, 2018
1 parent f5a43b0 commit 5e35604
Showing 1 changed file with 4 additions and 7 deletions.
11 changes: 4 additions & 7 deletions src/libponyrt/sched/scheduler.c
Original file line number Diff line number Diff line change
Expand Up @@ -543,8 +543,8 @@ static pony_actor_t* perhaps_suspend_scheduler(
// if we're the highest active scheduler thread
// and there are more active schedulers than the minimum requested
// and we're not terminating
if ((sched == &scheduler[current_active_scheduler_count - 1])
&& (current_active_scheduler_count > min_scheduler_count)
if ((current_active_scheduler_count > min_scheduler_count)
&& (sched == &scheduler[current_active_scheduler_count - 1])
&& (!sched->terminate)
#if defined(USE_SCHEDULER_SCALING_PTHREADS)
// try to acquire mutex if using pthreads
Expand Down Expand Up @@ -677,10 +677,10 @@ static pony_actor_t* steal(scheduler_t* sched)
// By waiting 1 millisecond before sending a block message, we are going to
// delay quiescence by a small amount of time but also optimize work
// stealing for generating far fewer block/unblock messages.
uint32_t current_active_scheduler_count = get_active_scheduler_count();

if (!block_sent)
{
uint32_t current_active_scheduler_count = get_active_scheduler_count();

// make sure thread scaling order is still valid. we should never be
// active if the active_scheduler_count isn't larger than our index.
pony_assert(current_active_scheduler_count > (uint32_t)sched->index);
Expand Down Expand Up @@ -718,9 +718,6 @@ static pony_actor_t* steal(scheduler_t* sched)
// if we do suspend, we'll send a unblock message first to ensure cnf/ack
// cycle works as expected

// get active scheduler count
uint32_t current_active_scheduler_count = get_active_scheduler_count();

// make sure thread scaling order is still valid. we should never be
// active if the active_scheduler_count isn't larger than our index.
pony_assert(current_active_scheduler_count > (uint32_t)sched->index);
Expand Down

0 comments on commit 5e35604

Please sign in to comment.