Fix deadlocks in condition_variable
#689
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Only hold lock while swapping the queue of waiting threads into a local queue. Retake lock if there's an error and the queue has to be spliced back into the member variable queue.
This change was only introduced to never yield while holding a lock in condition_variable::notify_all. However, that was relaxed to not hold a lock while resuming threads so we can again allow yielding in set_thread_state.
|
bors try |
tryBuild failed: |
msimberg
force-pushed
the
condition-variable-deadlocks
branch
from
73f49cc to
17028a4
Compare
|
bors try |
|
bors try- |
msimberg
force-pushed
the
condition-variable-deadlocks
branch
from
17028a4 to
449c0ad
Compare
|
bors try |
tryBuild failed: |
msimberg
force-pushed
the
condition-variable-deadlocks
branch
from
449c0ad to
a46a7ce
Compare
|
There were zero hangs in |
|
bors try |
|
bors merge |
bors bot
added a commit
that referenced
this pull request
May 17, 2023
689: Fix deadlocks in `condition_variable` r=msimberg a=msimberg This fixes deadlocks that started appearing after #672 which disabled yielding for `spinlock`. (One of) the deadlock(s) was the following scenario: | thread 1 | thread 2 | |-|-| | wait_until | | | take lock | | | add self to cv queue | | | release lock | | | timed suspend | | | | notify_all | | | take lock | | timed resume | attempt to set thread 1 to pending | | attempt to take lock | fail because thread 1 is active | | spin trying to take lock | spin waiting for thread 1 to not be active | | deadlock | deadlock | This PR changes `notify_all` to not hold the lock while resuming threads that need to be woken up. I see no reason to keep the lock for that time since there is anyway a delay between setting a thread to `pending` and the thread actually being run by a worker thread, with the latter _not_ happening under a lock already right now. This PR just relaxes that constraint further. It also significantly reduces the time the lock is held in `notify_all`. I'm quite sure this change is safe but we'll need to continue looking out for failures in CI in case I've missed something. I've also reverted the change in `set_thread_state` to never yield from #672. Since the lock in `notify_all` is no longer held while resuming threads it's again safe to yield in `set_thread_state`. I think spurious wakeups were probably possible before this change, but if they weren't they're now definitely possible with pika's `condition_variable`. Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
|
Build failed: |
|
bors merge |
bors bot
added a commit
that referenced
this pull request
May 17, 2023
689: Fix deadlocks in `condition_variable` r=msimberg a=msimberg This fixes deadlocks that started appearing after #672 which disabled yielding for `spinlock`. (One of) the deadlock(s) was the following scenario: | thread 1 | thread 2 | |-|-| | wait_until | | | take lock | | | add self to cv queue | | | release lock | | | timed suspend | | | | notify_all | | | take lock | | timed resume | attempt to set thread 1 to pending | | attempt to take lock | fail because thread 1 is active | | spin trying to take lock | spin waiting for thread 1 to not be active | | deadlock | deadlock | This PR changes `notify_all` to not hold the lock while resuming threads that need to be woken up. I see no reason to keep the lock for that time since there is anyway a delay between setting a thread to `pending` and the thread actually being run by a worker thread, with the latter _not_ happening under a lock already right now. This PR just relaxes that constraint further. It also significantly reduces the time the lock is held in `notify_all`. I'm quite sure this change is safe but we'll need to continue looking out for failures in CI in case I've missed something. I've also reverted the change in `set_thread_state` to never yield from #672. Since the lock in `notify_all` is no longer held while resuming threads it's again safe to yield in `set_thread_state`. I think spurious wakeups were probably possible before this change, but if they weren't they're now definitely possible with pika's `condition_variable`. Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
|
Build failed: |
|
This also happened: https://cdash.cscs.ch/test/83546786. More investigation needed after this is merged. bors merge |
bors bot
added a commit
that referenced
this pull request
May 17, 2023
689: Fix deadlocks in `condition_variable` r=msimberg a=msimberg This fixes deadlocks that started appearing after #672 which disabled yielding for `spinlock`. (One of) the deadlock(s) was the following scenario: | thread 1 | thread 2 | |-|-| | wait_until | | | take lock | | | add self to cv queue | | | release lock | | | timed suspend | | | | notify_all | | | take lock | | timed resume | attempt to set thread 1 to pending | | attempt to take lock | fail because thread 1 is active | | spin trying to take lock | spin waiting for thread 1 to not be active | | deadlock | deadlock | This PR changes `notify_all` to not hold the lock while resuming threads that need to be woken up. I see no reason to keep the lock for that time since there is anyway a delay between setting a thread to `pending` and the thread actually being run by a worker thread, with the latter _not_ happening under a lock already right now. This PR just relaxes that constraint further. It also significantly reduces the time the lock is held in `notify_all`. I'm quite sure this change is safe but we'll need to continue looking out for failures in CI in case I've missed something. I've also reverted the change in `set_thread_state` to never yield from #672. Since the lock in `notify_all` is no longer held while resuming threads it's again safe to yield in `set_thread_state`. I think spurious wakeups were probably possible before this change, but if they weren't they're now definitely possible with pika's `condition_variable`. Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
|
Build failed: |
|
bors merge |
bors bot
added a commit
that referenced
this pull request
May 17, 2023
689: Fix deadlocks in `condition_variable` r=msimberg a=msimberg This fixes deadlocks that started appearing after #672 which disabled yielding for `spinlock`. (One of) the deadlock(s) was the following scenario: | thread 1 | thread 2 | |-|-| | wait_until | | | take lock | | | add self to cv queue | | | release lock | | | timed suspend | | | | notify_all | | | take lock | | timed resume | attempt to set thread 1 to pending | | attempt to take lock | fail because thread 1 is active | | spin trying to take lock | spin waiting for thread 1 to not be active | | deadlock | deadlock | This PR changes `notify_all` to not hold the lock while resuming threads that need to be woken up. I see no reason to keep the lock for that time since there is anyway a delay between setting a thread to `pending` and the thread actually being run by a worker thread, with the latter _not_ happening under a lock already right now. This PR just relaxes that constraint further. It also significantly reduces the time the lock is held in `notify_all`. I'm quite sure this change is safe but we'll need to continue looking out for failures in CI in case I've missed something. I've also reverted the change in `set_thread_state` to never yield from #672. Since the lock in `notify_all` is no longer held while resuming threads it's again safe to yield in `set_thread_state`. I think spurious wakeups were probably possible before this change, but if they weren't they're now definitely possible with pika's `condition_variable`. Co-authored-by: Mikael Simberg <mikael.simberg@iki.fi>
|
Build failed: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes deadlocks that started appearing after #672 which disabled yielding for
spinlock.(One of) the deadlock(s) was the following scenario:
This PR changes
notify_allto not hold the lock while resuming threads that need to be woken up. I see no reason to keep the lock for that time since there is anyway a delay between setting a thread topendingand the thread actually being run by a worker thread, with the latter not happening under a lock already right now. This PR just relaxes that constraint further. It also significantly reduces the time the lock is held innotify_all. I'm quite sure this change is safe but we'll need to continue looking out for failures in CI in case I've missed something.I've also reverted the change in
set_thread_stateto never yield from #672. Since the lock innotify_allis no longer held while resuming threads it's again safe to yield inset_thread_state.I think spurious wakeups were probably possible before this change, but if they weren't they're now definitely possible with pika's
condition_variable.