feat!: Add hooks to allow app modules to add things to state-sync (backport #10961) #11267

mergify · 2022-02-24T16:17:48Z

This is an automatic backport of pull request #10961 done by Mergify.
Cherry-pick of 7e18e9f has failed:

On branch mergify/bp/release/v0.45.x/pr-10961
Your branch is up to date with 'origin/release/v0.45.x'.

You are currently cherry-picking commit 7e18e9f1b.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   baseapp/baseapp.go
	modified:   baseapp/options.go
	modified:   proto/cosmos/base/snapshots/v1beta1/snapshot.proto
	deleted:    proto/cosmos/base/store/v1beta1/snapshot.proto
	renamed:    snapshots/util.go -> snapshots/chunk.go
	renamed:    snapshots/util_test.go -> snapshots/chunk_test.go
	modified:   snapshots/manager_test.go
	new file:   snapshots/stream.go
	modified:   snapshots/types/snapshot.pb.go
	modified:   snapshots/types/snapshotter.go
	new file:   snapshots/types/util.go
	new file:   store/rootmulti/snapshot_test.go
	modified:   store/rootmulti/store.go
	modified:   store/types/snapshot.pb.go

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   api/cosmos/base/snapshots/v1beta1/snapshot.pulsar.go
	both modified:   server/mock/store.go
	both modified:   snapshots/helpers_test.go
	both modified:   snapshots/manager.go
	both modified:   store/rootmulti/store_test.go
	deleted by us:   store/v2/multi/store.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

@Mergifyio refresh will re-evaluate the rules
@Mergifyio rebase will rebase this PR on its base branch
@Mergifyio update will merge the base branch into this PR
@Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

look at your merge queues
generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

…0961) ## Description Closes: #7340 - Support registering multiple snapshotters in snapshot manager. - Append the extension snapshotters to existing snapshot stream. ~TODO: testing.~ - existing tests are fixed --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7e18e9f) # Conflicts: # api/cosmos/base/snapshots/v1beta1/snapshot.pulsar.go # server/mock/store.go # snapshots/helpers_test.go # snapshots/manager.go # store/rootmulti/store_test.go # store/v2/multi/store.go

proto/cosmos/base/snapshots/v1beta1/snapshot.proto

tac0turtle

This has the potential to fragment a network for state sync. I am fine with this fragmentation mainly because the fragmentation is between state sync working and not. Its something to keep in mind for chains that upgrade and use wasm.

We should mention this in the changelog possibly?

yihuang · 2022-02-25T01:28:30Z

This has the potential to fragment a network for state sync. I am fine with this fragmentation mainly because the fragmentation is between state sync working and not. Its something to keep in mind for chains that upgrade and use wasm.

We should mention this in the changelog possibly?

do you mean when wasm start register extension snapshotter, it'll be breaking?

tac0turtle · 2022-02-25T10:09:23Z

do you mean when wasm start register extension snapshotter, it'll be breaking?

yea, if its done in a minor release some nodes will provide invalid snapshots. maybe something for wasm to add. If snapshot has empty metadata dev it

yihuang · 2022-02-25T10:34:40Z

do you mean when wasm start register extension snapshotter, it'll be breaking?

yea, if its done in a minor release some nodes will provide invalid snapshots. maybe something for wasm to add. If snapshot has empty metadata dev it

maybe wasm can bump a major version when adding extension snapshotter?

ValarDragon

Im in favor of this backport!

yihuang · 2022-02-28T06:51:07Z

Can wasmd team do a real-world test with this patch?

yihuang · 2022-02-28T07:08:33Z

#11286
let's wait for this fix merged, and add it.

amaury1093 · 2022-02-28T16:57:14Z

snapshots/manager.go

@@ -55,13 +61,29 @@ type Manager struct {
 }

 // NewManager creates a new manager.
-func NewManager(store *Store, target types.Snapshotter) *Manager {
+func NewManager(store *Store, multistore types.Snapshotter, extensions map[string]types.ExtensionSnapshotter) *Manager {


Can we also add changelogs for all API-breaking changes?

Can we also add changelogs for all API-breaking changes?

should we add a new constructor to avoid breaking API, in this backport PR?

I've added a new constructor and added an improvement changelog.
~~I just noticed I forget to add changelog for the original PR, will add in another PR.~~ done

…ed (#11286) ## Description Solution: - return the next unknown item and add a unit test to ensure that. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable)

yihuang · 2022-03-01T00:45:13Z

I've cherry-picked the bug fix of #11293.

tac0turtle · 2022-03-02T11:18:18Z

merging this then will coordinate with wasmd on the upgrade

…ckport cosmos#10961) (cosmos#11267) * feat!: Add hooks to allow app modules to add things to state-sync (cosmos#10961) Closes: cosmos#7340 - Support registering multiple snapshotters in snapshot manager. - Append the extension snapshotters to existing snapshot stream. ~TODO: testing.~ - existing tests are fixed --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7e18e9f) * fix conflicts * avoid api breakage * changelog * fix: rootmulti's Restore don't return the next unknown item as expected (cosmos#11286) Solution: - return the next unknown item and add a unit test to ensure that. --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) Co-authored-by: yihuang <huang@crypto.com>

…-sync #237 (#238) * feat!: Add hooks to allow app modules to add things to state-sync (backport cosmos#10961) (cosmos#11267) * feat!: Add hooks to allow app modules to add things to state-sync (cosmos#10961) Closes: cosmos#7340 - Support registering multiple snapshotters in snapshot manager. - Append the extension snapshotters to existing snapshot stream. ~TODO: testing.~ - existing tests are fixed --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7e18e9f) * fix conflicts * avoid api breakage * changelog * fix: rootmulti's Restore don't return the next unknown item as expected (cosmos#11286) Solution: - return the next unknown item and add a unit test to ensure that. --- *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) Co-authored-by: yihuang <huang@crypto.com> * remove TestTraceConcurrency Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: yihuang <huang@crypto.com>

…ckport cosmos#10961) (cosmos#11267) * feat!: Add hooks to allow app modules to add things to state-sync (cosmos#10961) ## Description Closes: cosmos#7340 - Support registering multiple snapshotters in snapshot manager. - Append the extension snapshotters to existing snapshot stream. ~TODO: testing.~ - existing tests are fixed --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) (cherry picked from commit 7e18e9f) # Conflicts: # api/cosmos/base/snapshots/v1beta1/snapshot.pulsar.go # server/mock/store.go # snapshots/helpers_test.go # snapshots/manager.go # store/rootmulti/store_test.go # store/v2/multi/store.go * fix conflicts * avoid api breakage * changelog * fix: rootmulti's Restore don't return the next unknown item as expected (cosmos#11286) ## Description Solution: - return the next unknown item and add a unit test to ensure that. --- ### Author Checklist *All items are required. Please add a note to the item if the item is not applicable and please add links to any relevant follow up issues.* I have... - [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] added `!` to the type prefix if API or client breaking change - [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting)) - [ ] provided a link to the relevant issue or specification - [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules) - [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing) - [ ] added a changelog entry to `CHANGELOG.md` - [ ] included comments for [documenting Go code](https://blog.golang.org/godoc) - [ ] updated the relevant documentation or specification - [ ] reviewed "Files changed" and left comments if necessary - [ ] confirmed all CI checks have passed ### Reviewers Checklist *All items are required. Please add a note if the item is not applicable and please add your handle next to the items reviewed if you only reviewed selected items.* I have... - [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title - [ ] confirmed `!` in the type prefix if API or client breaking change - [ ] confirmed all author checklist items have been addressed - [ ] reviewed state machine logic - [ ] reviewed API design and naming - [ ] reviewed documentation is accurate - [ ] reviewed tests and test coverage - [ ] manually tested (if applicable) Co-authored-by: yihuang <huang@crypto.com>

mergify bot requested review from aaronc and alexanderbez as code owners February 24, 2022 16:17

mergify bot added the conflicts label Feb 24, 2022

fix conflicts

91159d4

tac0turtle reviewed Feb 24, 2022

View reviewed changes

proto/cosmos/base/snapshots/v1beta1/snapshot.proto Show resolved Hide resolved

tac0turtle reviewed Feb 24, 2022

View reviewed changes

tac0turtle removed the conflicts label Feb 27, 2022

tac0turtle approved these changes Feb 27, 2022

View reviewed changes

ValarDragon approved these changes Feb 27, 2022

View reviewed changes

yihuang marked this pull request as draft February 28, 2022 07:08

amaury1093 reviewed Feb 28, 2022

View reviewed changes

yihuang added 3 commits March 1, 2022 08:39

avoid api breakage

57e1e68

changelog

04b0fcd

yihuang marked this pull request as ready for review March 1, 2022 00:45

yihuang mentioned this pull request Mar 1, 2022

fix: rootmulti's Restore don't return the next unknown item as expected (backport #11286) #11293

Closed

tac0turtle approved these changes Mar 2, 2022

View reviewed changes

tac0turtle merged commit 811cb8a into release/v0.45.x Mar 2, 2022

tac0turtle deleted the mergify/bp/release/v0.45.x/pr-10961 branch March 2, 2022 11:18

This was referenced May 17, 2022

chore: update wasmd to v0.26.0 osmosis-labs/osmosis#1516

Closed

[v9 RC Blocker] Upgrade Main to Cosmwasm v0.27.0.rc3 osmosis-labs/osmosis#1502

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: Add hooks to allow app modules to add things to state-sync (backport #10961) #11267

feat!: Add hooks to allow app modules to add things to state-sync (backport #10961) #11267

mergify bot commented Feb 24, 2022

tac0turtle left a comment

yihuang commented Feb 25, 2022

tac0turtle commented Feb 25, 2022

yihuang commented Feb 25, 2022

ValarDragon left a comment

yihuang commented Feb 28, 2022 •

edited

Loading

yihuang commented Feb 28, 2022

amaury1093 Feb 28, 2022

yihuang Mar 1, 2022 •

edited

Loading

yihuang Mar 1, 2022 •

edited

Loading

yihuang commented Mar 1, 2022 •

edited

Loading

tac0turtle commented Mar 2, 2022

feat!: Add hooks to allow app modules to add things to state-sync (backport #10961) #11267

feat!: Add hooks to allow app modules to add things to state-sync (backport #10961) #11267

Conversation

mergify bot commented Feb 24, 2022

tac0turtle left a comment

Choose a reason for hiding this comment

yihuang commented Feb 25, 2022

tac0turtle commented Feb 25, 2022

yihuang commented Feb 25, 2022

ValarDragon left a comment

Choose a reason for hiding this comment

yihuang commented Feb 28, 2022 • edited Loading

yihuang commented Feb 28, 2022

amaury1093 Feb 28, 2022

Choose a reason for hiding this comment

yihuang Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

yihuang Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

yihuang commented Mar 1, 2022 • edited Loading

tac0turtle commented Mar 2, 2022

yihuang commented Feb 28, 2022 •

edited

Loading

yihuang Mar 1, 2022 •

edited

Loading

yihuang Mar 1, 2022 •

edited

Loading

yihuang commented Mar 1, 2022 •

edited

Loading