-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix desync after errCatchupTooManyRetries #5939
Fix desync after errCatchupTooManyRetries #5939
Conversation
Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
@@ -8383,7 +8388,7 @@ RETRY: | |||
} | |||
|
|||
numRetries++ | |||
if numRetries >= maxRetries { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was an off-by-one. maxRetries=3
, but there would only be max 2 retries.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Story of my life ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -8383,7 +8388,7 @@ RETRY: | |||
} | |||
|
|||
numRetries++ | |||
if numRetries >= maxRetries { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Story of my life ;)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Previously a related case of RAFT state being deleted was fixed, when running into `errCatchupTooManyRetries`: #5939 After hitting this we shutdown and retry.. but if we have not elected a leader yet we'd hit `"catchup for stream '%s > %s' aborted, no leader"`, which then would again throw away RAFT state. This PR proposes a fix for that case. Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
Looked into the issue reported in Antithesis of non-monotonic sequence / pub ack sequence moving back due to stream desync.
When looking at these logs:
In
monitorStream
we call intoapplyStreamEntries
which calls intoprocessSnapshot
when about a snapshot. If after reachingmaxRetries
catchup remains stalled, the RAFT data is deleted, which meansn.pindex=0
.Then when a leader election comes around this server with missing RAFT data would allow an outdated server that misses data to become leader. This is reproduced in the test.
mset.resetClusteredState(errCatchupTooManyRetries)
n.pindex=0
on the reset server, it grants leader to the outdated server. Resulting in desync.This PR proposes to not fully delete the RAFT state when we are not able to reach the leader during the processing of a snapshot. Which ensures the outdated server does NOT get selected as a leader and it gets correctly caught up to contain the data it missed.
Signed-off-by: Maurice van Veen github@mauricevanveen.com