Allow raft to recover state after non-graceful shutdown in non-archive mode #860
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ync up post non-disgraceful shutdown
Continue in loop instead of if/else.
…l' into raft-enable-gcmode-full
prd-fox
previously approved these changes
Oct 18, 2019
prd-fox
changed the title
|
FYI I've been performing some investigation to reproduce a scenario where disk filling up was causing raft issues. I discovered that when disk space ran out, raft crashed and corrupted the logs, in the same way as when it's killed non-gracefully. I re-tested using your branch (3 times) and can confirm that your code changes fix the issue for my scenario too. |
trung
approved these changes
Nov 29, 2019
There was a problem hiding this comment.
Changes include:
- Revert set gcmode=archive always for Raft #727
- Replay blocks from snapshot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
System information
Geth version: v2.2.4
OS & Version: Windows/Linux/OSX
Expected behaviour
In case of non graceful shutdown of a node running with Raft consensus, when the node joins back the network it should be able to sync up with rest of network.
Actual behaviour
Post non graceful shutdown of the node, the node is not able sync up with the network. Logs show the following error:
Steps to reproduce the behaviour
Bring up the network with geth nodes running in --gcmode full
Do a non-graceful shutdown of one of the nodes kill -9 and bring back the node
Closes #814, fixes #807