Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store/v2): implement the feature to upgrade the store keys #20453

Merged
merged 46 commits into from
Aug 5, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
34f301a
init
cool-develope May 25, 2024
e22fa63
Merge branch 'main' into store/upgrade
cool-develope May 29, 2024
deb1cef
wip
cool-develope May 31, 2024
422c2f1
Merge branch 'main' into store/upgrade
cool-develope May 31, 2024
de5188c
add tests
cool-develope Jun 3, 2024
787c2f1
Merge branch 'main' into store/upgrade
cool-develope Jun 3, 2024
cc5803d
lint
cool-develope Jun 3, 2024
982234c
feat(store/v2): Implement the feature to upgrade `storeKeys` on the s…
cool-develope Jun 11, 2024
52a0148
Merge branch 'main' into store/upgrade
cool-develope Jun 11, 2024
c434513
address comments
cool-develope Jun 12, 2024
01f58fe
Merge branch 'main' into store/upgrade
cool-develope Jun 12, 2024
0a395e0
error handle
cool-develope Jun 12, 2024
9b91446
Merge branch 'main' into store/upgrade
cool-develope Jun 19, 2024
fee3fd7
comments
cool-develope Jun 19, 2024
9ddef3c
Merge branch 'main' into store/upgrade
cool-develope Jun 24, 2024
cc04a80
Merge branch 'main' into store/upgrade
cool-develope Jun 27, 2024
5231c52
comments
cool-develope Jun 27, 2024
37079b9
Merge branch 'main' into store/upgrade
cool-develope Jun 27, 2024
e23e66a
comments
cool-develope Jun 28, 2024
5249bcd
Merge branch 'main' into store/upgrade
cool-develope Jul 1, 2024
b74f833
Merge branch 'main' into store/upgrade
cool-develope Jul 3, 2024
dd22733
wrap up
cool-develope Jul 3, 2024
6f23f8e
Merge branch 'main' into store/upgrade
cool-develope Jul 8, 2024
0b11f85
feat(store/v2): Removing old store keys by pruning (#20927)
cool-develope Jul 24, 2024
ad0310d
remove rename feature
cool-develope Jul 24, 2024
5e5b340
Merge branch 'main' into store/upgrade
cool-develope Jul 24, 2024
2af4c06
wip
cool-develope Jul 24, 2024
7240ced
fixing
cool-develope Jul 24, 2024
4309308
minor fix
cool-develope Jul 24, 2024
6d81e39
minor cleanup
cool-develope Jul 24, 2024
e99168d
linting
cool-develope Jul 24, 2024
77d0533
Merge branch 'main' into store/upgrade
cool-develope Jul 25, 2024
f70c4fa
Merge branch 'main' into store/upgrade
cool-develope Jul 29, 2024
2d341ae
go mod tidy
cool-develope Jul 29, 2024
6ae8def
docs
cool-develope Jul 30, 2024
805f697
Merge branch 'main' into store/upgrade
cool-develope Jul 30, 2024
d3c6e87
revert go.mod update
cool-develope Jul 30, 2024
2e1d53b
go mod update
cool-develope Jul 31, 2024
be3e5af
Merge branch 'main' into store/upgrade
cool-develope Jul 31, 2024
0948f11
iavl version match
cool-develope Aug 1, 2024
74b31e0
Merge branch 'main' into store/upgrade
cool-develope Aug 1, 2024
66d86ec
Update store/v2/README.md
cool-develope Aug 1, 2024
f50b207
go mod update
cool-develope Aug 1, 2024
bca11e4
remove mountTreeFn
cool-develope Aug 5, 2024
32fbcfe
Merge branch 'main' into store/upgrade
cool-develope Aug 5, 2024
65bce60
linting
cool-develope Aug 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 30 additions & 6 deletions store/v2/commitment/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"fmt"
"io"
"math"
"sort"

protoio "github.com/cosmos/gogoproto/io"

Expand Down Expand Up @@ -120,7 +121,7 @@ func (c *CommitStore) GetLatestVersion() (uint64, error) {
return version, nil
}

func (c *CommitStore) LoadVersion(targetVersion uint64) error {
func (c *CommitStore) LoadVersion(targetVersion uint64, upgrades *corestore.StoreUpgrades) error {
// Rollback the metadata to the target version.
latestVersion, err := c.GetLatestVersion()
if err != nil {
Expand All @@ -139,20 +140,43 @@ func (c *CommitStore) LoadVersion(targetVersion uint64) error {
}
}

for _, tree := range c.multiTrees {
storesKeys := make([]string, 0, len(c.multiTrees))
cool-develope marked this conversation as resolved.
Show resolved Hide resolved
for storeKey := range c.multiTrees {
alpe marked this conversation as resolved.
Show resolved Hide resolved
storesKeys = append(storesKeys, storeKey)
}

if upgrades != nil {
// deterministic iteration order for upgrades
// (as the underlying store may change and
// upgrades make store changes where the execution order may matter)
sort.Slice(storesKeys, func(i, j int) bool {
return storesKeys[i] < storesKeys[j]
})
}

for _, storeKey := range storesKeys {
tree := c.multiTrees[storeKey]
alpe marked this conversation as resolved.
Show resolved Hide resolved

// If it has been added, set the initial version.
if upgrades.IsAdded(storeKey) || upgrades.RenamedFrom(storeKey) != "" {
cool-develope marked this conversation as resolved.
Show resolved Hide resolved
if err := tree.SetInitialVersion(targetVersion + 1); err != nil {
alpe marked this conversation as resolved.
Show resolved Hide resolved
return err
}
}

if err := tree.LoadVersion(targetVersion); err != nil {
return err
}
}

alpe marked this conversation as resolved.
Show resolved Hide resolved
// If the target version is greater than the latest version, it is the snapshot
// restore case, we should create a new commit info for the target version.
var cInfo *proof.CommitInfo
if targetVersion > latestVersion {
cInfo = c.WorkingCommitInfo(targetVersion)
cInfo := c.WorkingCommitInfo(targetVersion)
return c.flushCommitInfo(targetVersion, cInfo)
}

return c.flushCommitInfo(targetVersion, cInfo)
return nil
cool-develope marked this conversation as resolved.
Show resolved Hide resolved
}

func (c *CommitStore) GetCommitInfo(version uint64) (*proof.CommitInfo, error) {
Expand Down Expand Up @@ -470,7 +494,7 @@ loop:
}
}

return snapshotItem, c.LoadVersion(version)
return snapshotItem, c.LoadVersion(version, nil)
}

func (c *CommitStore) Close() (ferr error) {
Expand Down
4 changes: 2 additions & 2 deletions store/v2/database.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ type Committer interface {
GetLatestVersion() (uint64, error)

// LoadVersion loads the tree at the given version.
LoadVersion(targetVersion uint64) error
LoadVersion(targetVersion uint64, upgrades *corestore.StoreUpgrades) error
cool-develope marked this conversation as resolved.
Show resolved Hide resolved
cool-develope marked this conversation as resolved.
Show resolved Hide resolved

// Commit commits the working tree to the database.
Commit(version uint64) (*proof.CommitInfo, error)
Expand All @@ -56,7 +56,7 @@ type Committer interface {
// Once migration is complete, this method should be removed and/or not used.
Get(storeKey []byte, version uint64, key []byte) ([]byte, error)

// SetInitialVersion sets the initial version of the tree.
// SetInitialVersion sets the initial version of the committer.
SetInitialVersion(version uint64) error

// GetCommitInfo returns the CommitInfo for the given version.
Expand Down
26 changes: 22 additions & 4 deletions store/v2/root/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@
return err
}

return s.loadVersion(lv)
return s.loadVersion(lv, nil)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The methods LoadLatestVersion and LoadVersion correctly delegate to loadVersion, maintaining a clean and DRY codebase. However, consider adding error handling or logging within loadVersion to provide more detailed diagnostics in case of failures, especially given the complexity introduced with the upgrade logic.

Also applies to: 236-236

}

func (s *Store) LoadVersion(version uint64) error {
Expand All @@ -224,13 +224,31 @@
defer s.telemetry.MeasureSince(now, "root_store", "load_version")
}

return s.loadVersion(version)
return s.loadVersion(version, nil)
}

func (s *Store) loadVersion(v uint64) error {
// LoadVersionAndUpgrade implements the UpgradeableRootStore interface.
//
// NOTE: It cannot be called while the store is migrating.
func (s *Store) LoadVersionAndUpgrade(version uint64, upgrades *corestore.StoreUpgrades) error {
if s.telemetry != nil {
now := time.Now()
Fixed Show fixed Hide fixed
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use telemetry.Now() instead

defer s.telemetry.MeasureSince(now, "root_store", "load_version_and_upgrade")
}

if s.isMigrating {
return fmt.Errorf("cannot upgrade while migrating")
}

// TODO: Implement the pruning logic for the renamed/deleted store keys.

return s.loadVersion(version, upgrades)
}
cool-develope marked this conversation as resolved.
Show resolved Hide resolved

func (s *Store) loadVersion(v uint64, upgrades *corestore.StoreUpgrades) error {
s.logger.Debug("loading version", "version", v)

if err := s.stateCommitment.LoadVersion(v); err != nil {
if err := s.stateCommitment.LoadVersion(v, upgrades); err != nil {
cool-develope marked this conversation as resolved.
Show resolved Hide resolved
return fmt.Errorf("failed to load SC version %d: %w", v, err)
}

Expand Down
Loading