Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: prune everything #11177

Merged
merged 9 commits into from
Feb 23, 2022
Merged

refactor: prune everything #11177

merged 9 commits into from
Feb 23, 2022

Conversation

alexanderbez
Copy link
Contributor

@alexanderbez alexanderbez commented Feb 12, 2022 •
edited
Loading

Description

Closes: #10352

  • Tweak the everything pruning setting to keep the previous state in addition to the current state.
  • Disable (return an error) if pruning=everything AND state sync is enabled

Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • included the correct type prefix in the PR title
  • added ! to the type prefix if API or client breaking change
  • targeted the correct branch (see PR Targeting)
  • provided a link to the relevant issue or specification
  • followed the guidelines for building modules
  • included the necessary unit and integration tests
  • added a changelog entry to CHANGELOG.md
  • included comments for documenting Go code
  • updated the relevant documentation or specification
  • reviewed "Files changed" and left comments if necessary
  • confirmed all CI checks have passed

Reviewers Checklist

All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.

I have...

  • confirmed the correct type prefix in the PR title
  • confirmed ! in the type prefix if API or client breaking change
  • confirmed all author checklist items have been addressed
  • reviewed state machine logic
  • reviewed API design and naming
  • reviewed documentation is accurate
  • reviewed tests and test coverage
  • manually tested (if applicable)

@tac0turtle
Copy link
Member

I don't think this solves the issue. If I understand it correctly you are assuming it takes less time to snapshot than it does to proceed two blocks. I think in some cases this would work but for chains with large amounts of current state you would still get data base corruption.

If this is the case I would suggest using snapshot-interval the same way we had keep-every, except after the snapshot is saved we prune that height.

@alexanderbez
Copy link
Contributor Author

@marbar3778 this PR has nothing to do with snapshotting. It has to do with fixing the off by on error, i.e. the failed to load at height X error you see when using pruning=everything.

@tac0turtle
Copy link
Member

Yes I know. But only keeping the last two blocks will only fix some on the issues is what I'm saying. If the snapshot takes longer than two blocks the app will try to prune a version that has an active reader on it.

@alexanderbez
Copy link
Contributor Author

Yes I know. But only keeping the last two blocks will only fix some on the issues is what I'm saying. If the snapshot takes longer than two blocks the app will try to prune a version that has an active reader on it.

I see. I mean it would be ideal to not have to conflate the two things. The issue you state is currently also an issue. What do you propose exactly?

@alexanderbez alexanderbez marked this pull request as ready for review February 22, 2022 17:38
tac0turtle
tac0turtle approved these changes Feb 22, 2022
View reviewed changes
Copy link
Member

@tac0turtle tac0turtle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alexanderbez
Copy link
Contributor Author

alexanderbez commented Feb 23, 2022 •
edited
Loading

@marbar3778 @AmauryM wdyt about backporting this? There technically aren't any API breaking changes. The only thing that would happen now is app.toml would fail validation if a user enables state sync snapshotting and has pruning=everything, which is a no no anyway (hence the change).

ValarDragon
ValarDragon approved these changes Feb 23, 2022
View reviewed changes
Copy link
Contributor

@ValarDragon ValarDragon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change! Im in favor of backporting this

@alexanderbez alexanderbez merged commit 75bcf47 into master Feb 23, 2022
@alexanderbez alexanderbez deleted the bez/10352-fix-prune-everything branch February 23, 2022 20:32
mergify bot pushed a commit that referenced this pull request Feb 23, 2022
@alexanderbez
refactor: prune everything (#11177)
bc49d4f 
(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go
@mergify mergify bot mentioned this pull request Feb 23, 2022
Merged
mergify bot pushed a commit that referenced this pull request Feb 23, 2022
@alexanderbez
refactor: prune everything (#11177)
358c9fc 
(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go
@mergify mergify bot mentioned this pull request Feb 23, 2022
Merged
mergify bot pushed a commit that referenced this pull request Feb 23, 2022
@alexanderbez
refactor: prune everything (#11177)
de6bf77 
(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/config.go
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go
@mergify mergify bot mentioned this pull request Feb 23, 2022
Closed
tac0turtle pushed a commit that referenced this pull request Feb 23, 2022
@mergify @alexanderbez
refactor: prune everything (backport #11177) (#11258)
febcb92 
* refactor: prune everything (#11177)

(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go

* updates

* updates

* updates

* updates

* updates

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
alexanderbez pushed a commit that referenced this pull request Feb 24, 2022
@mergify
refactor: prune everything (backport #11177) (#11259)
6268a93
alexanderbez pushed a commit that referenced this pull request Feb 24, 2022
@mergify
chore: fix changelog entry for #11177 (backport #11263) (#11264)
d69a7b6
alexanderbez pushed a commit that referenced this pull request Feb 24, 2022
@mergify
chore: fix changelog entry for #11177 (backport #11263) (#11265)
9f7a530
@faddat faddat mentioned this pull request Feb 28, 2022
Closed
8 tasks
zakir-code pushed a commit to PundiAI/cosmos-sdk that referenced this pull request Apr 8, 2022
@mergify @alexanderbez
refactor: prune everything (backport cosmos#11177) (cosmos#11258)
c7efdc2 
* refactor: prune everything (cosmos#11177)

(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go

* updates

* updates

* updates

* updates

* updates

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
JimLarson pushed a commit to agoric-labs/cosmos-sdk that referenced this pull request Jul 7, 2022
@mergify @alexanderbez @JimLarson
refactor: prune everything (backport cosmos#11177) (cosmos#11258)
b1ecdfa 
* refactor: prune everything (cosmos#11177)

(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go

* updates

* updates

* updates

* updates

* updates

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
JimLarson pushed a commit to agoric-labs/cosmos-sdk that referenced this pull request Jul 7, 2022
@mergify @JimLarson
chore: fix changelog entry for cosmos#11177 (backport cosmos#11263) (c…
873d2b9 
…osmos#11264)
randy75828 pushed a commit to Switcheo/cosmos-sdk that referenced this pull request Aug 10, 2022
@mergify @alexanderbez @randy75828
refactor: prune everything (backport cosmos#11177) (cosmos#11258)
02b13e3 
* refactor: prune everything (cosmos#11177)

(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go

* updates

* updates

* updates

* updates

* updates

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
randy75828 pushed a commit to Switcheo/cosmos-sdk that referenced this pull request Aug 10, 2022
@mergify @randy75828
chore: fix changelog entry for cosmos#11177 (backport cosmos#11263) (c…
6a703a5 
…osmos#11264)
randy75828 pushed a commit to Switcheo/cosmos-sdk that referenced this pull request Aug 10, 2022
@mergify @alexanderbez @randy75828
refactor: prune everything (backport cosmos#11177) (cosmos#11258)
ee2ca14 
* refactor: prune everything (cosmos#11177)

(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go

* updates

* updates

* updates

* updates

* updates

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
randy75828 pushed a commit to Switcheo/cosmos-sdk that referenced this pull request Aug 10, 2022
@mergify @randy75828
chore: fix changelog entry for cosmos#11177 (backport cosmos#11263) (c…
a933763 
…osmos#11264)
JeancarloBarrios pushed a commit to agoric-labs/cosmos-sdk that referenced this pull request Sep 28, 2024
@mergify @alexanderbez @JeancarloBarrios
refactor: prune everything (backport cosmos#11177) (cosmos#11258)
bc9627b 
* refactor: prune everything (cosmos#11177)

(cherry picked from commit 75bcf47)

# Conflicts:
#	CHANGELOG.md
#	server/config/toml.go
#	server/start.go
#	store/rootmulti/store.go
#	store/types/pruning.go
#	store/v2/multi/store_test.go

* updates

* updates

* updates

* updates

* updates

Co-authored-by: Aleksandr Bezobchuk <alexanderbez@users.noreply.github.com>
Co-authored-by: Aleksandr Bezobchuk <aleks.bezobchuk@gmail.com>
JeancarloBarrios pushed a commit to agoric-labs/cosmos-sdk that referenced this pull request Sep 28, 2024
@mergify @JeancarloBarrios
chore: fix changelog entry for cosmos#11177 (backport cosmos#11263) (c…
bc3c5c0 
…osmos#11264)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tac0turtle tac0turtle tac0turtle approved these changes

@ValarDragon ValarDragon ValarDragon approved these changes

@aaronc aaronc Awaiting requested review from aaronc

@robert-zaremba robert-zaremba Awaiting requested review from robert-zaremba

@amaury1093 amaury1093 Awaiting requested review from amaury1093

Assignees
No one assigned
Labels
C:Store
Projects
No open projects
Cosmos SDK: Core Development
Archived in project
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

pruning=everything causes db corruption
3 participants
@alexanderbez @tac0turtle @ValarDragon