Keep only the latest value in the health channel #14087
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rkapka
changed the title
| @@ -87,12 +87,6 @@ func TestNodeHealth_Concurrency(t *testing.T) { | |||
| // Number of goroutines to spawn for both reading and writing | |||
| numGoroutines := 6 | |||
|
|
|||
| go func() { | |||
There was a problem hiding this comment.
This is the regression test to show that CheckHealth is non-blocking
prestonvanloon
added a commit
that referenced
this pull request
Jun 10, 2024
* Increase health tracker channel buffer size * keep only the latest value * Make health test blocking as a regression test for PR #14807 * Fix new race conditions in the MockHealthClient --------- Co-authored-by: Preston Van Loon <preston@pvl.dev>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Bug fix
What does this PR do? Why is it needed?
#14033 introduced a subtle race condition. The channel has a buffer size of 1. Previously we unlocked before sending to the channel. This means that if we tried to send a second value, sending to the channel blocked but any other function that required the lock could take it. In the current version this other function will need to wait for the lock. And that's what happened with the REST VC. I don't know how exactly it got to this, but apparently a second value was sent to the channel before reading the first one, which captured the lock. Another function tried to obtain the lock, and reading the first value from the channel depended on this function returning.
To fix this, we introduce a switch statement that always replaces the existing value in the channel with a new one.