Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix node storage corruption automatically #704

Open
pasqu4le opened this issue Aug 22, 2023 · 0 comments
Open

Fix node storage corruption automatically #704

pasqu4le opened this issue Aug 22, 2023 · 0 comments
Assignees
@krendelhoff2

Comments

@pasqu4le
Copy link
Contributor

Clarification and motivation

octez-node can sometime exit with leaving a corrupted state on disk, which happens more frequently if it's interrupted abruptly.

The node can sometimes fix the state by itself, but it can also fail to do so, preventing it from starting.

This affects also nodes launched with our systemd services, which can fail to start for this reason.

It should be possible to automatically detect the failure of the node to fix its corrupted state and consequently perform a cleanup of the node storage and a re-sync of it.

Acceptance criteria

When our tezos-node service is started with a corrupted state that octez-node cannot fix by itself, then the service fixes it instead.
PruStephan added a commit that referenced this issue Dec 9, 2023
@PruStephan
[#704] Add env variable
a05c86d
PruStephan added a commit that referenced this issue Dec 9, 2023
@PruStephan
[#704] Add corruption recovery script
817641a 
Problem: Currently the only way to fix the problem
with node corruption is to manually delete the directory
and load snapshot anew.

Solution: Add script to do all of this automatically
@PruStephan PruStephan mentioned this issue Dec 9, 2023
Closed
3 tasks
PruStephan added a commit that referenced this issue Dec 9, 2023
@PruStephan
[#704] Add corruption recovery script
8e35bb8 
Problem: Currently the only way to fix the problem
with node corruption is to manually delete the directory
and load snapshot anew.

Solution: Add script to do all of this automatically
PruStephan added a commit that referenced this issue Dec 9, 2023
@PruStephan
[#704] Add corruption recovery script
9bf2ea1 
Problem: Currently the only way to fix the problem
with node corruption is to manually delete the directory
and load snapshot anew.

Solution: Add script to do all of this automatically
PruStephan added a commit that referenced this issue Dec 9, 2023
@PruStephan
[#704] Add corruption recovery script
a78f852 
Problem: Currently the only way to fix the problem
with node corruption is to manually delete the directory
and load snapshot anew.

Solution: Add script to do all of this automatically
PruStephan added a commit that referenced this issue Dec 9, 2023
@PruStephan
[#704] Add corruption recovery script
aced221 
Problem: Currently the only way to fix the problem
with node corruption is to manually delete the directory
and load snapshot anew.

Solution: Add script to do all of this automatically
PruStephan added a commit that referenced this issue Dec 14, 2023
@PruStephan
fixup! [#704] Add env variable
22fe622
PruStephan added a commit that referenced this issue Dec 14, 2023
@PruStephan
fixup! [#704] Add env variable
9499993
PruStephan added a commit that referenced this issue Dec 18, 2023
@PruStephan
fixup! [#704] Add env variable
10ae4fd
PruStephan added a commit that referenced this issue Dec 18, 2023
@PruStephan
fixup! [#704] Add env variable
d74780d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@krendelhoff2 krendelhoff2

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[#704] Add script to automatically fix node corruption serokell/tezos-packaging
3 participants
@PruStephan @pasqu4le @krendelhoff2