Skip to content

Commit

Permalink
feat: add config iavl-lazy-loading to enable lazy loading of iavl s…
Browse files Browse the repository at this point in the history
…tore (#14189)

Co-authored-by: Marko <marbar3778@yahoo.com>
(cherry picked from commit b266522)

# Conflicts:
#	CHANGELOG.md
#	go.mod
#	go.sum
#	simapp/go.mod
#	simapp/go.sum
#	store/go.mod
#	store/go.sum
#	tests/go.mod
#	tests/go.sum
#	x/nft/go.mod
#	x/nft/go.sum
#	x/upgrade/go.mod
#	x/upgrade/go.sum
  • Loading branch information
yihuang authored and mergify[bot] committed Feb 1, 2023
1 parent 5f4ddc6 commit c71dc1c
Show file tree
Hide file tree
Showing 23 changed files with 3,898 additions and 1 deletion.
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,16 @@ Ref: https://keepachangelog.com/en/1.0.0/
* (x/auth) [#13210](https://github.com/cosmos/cosmos-sdk/pull/13210) Add `Query/AccountInfo` endpoint for simplified access to basic account info.
* (x/consensus) [#12905](https://github.com/cosmos/cosmos-sdk/pull/12905) Create a new `x/consensus` module that is now responsible for maintaining Tendermint consensus parameters instead of `x/param`. Legacy types remain in order to facilitate parameter migration from the deprecated `x/params`. App developers should ensure that they execute `baseapp.MigrateParams` during their chain upgrade. These legacy types will be removed in a future release.
* (client/tx) [#13670](https://github.com/cosmos/cosmos-sdk/pull/13670) Add validation in `BuildUnsignedTx` to prevent simple inclusion of valid mnemonics
<<<<<<< HEAD
=======
* [#13473](https://github.com/cosmos/cosmos-sdk/pull/13473) ADR-038: Go plugin system proposal
* [#14356](https://github.com/cosmos/cosmos-sdk/pull/14356) Add `events.GetAttributes` and `event.GetAttribute` methods to simplify the retrieval of an attribute from event(s).
* [#14472](https://github.com/cosmos/cosmos-sdk/pull/14356) The recommended metadata format for x/gov and x/group proposals now uses an array of strings (instead of a single string) for the `authors` field.
* (mempool) [#14484](https://github.com/cosmos/cosmos-sdk/pull/14484) Add priority nonce mempool option for transaction replacement.
* (client) [#14509](https://github.com/cosmos/cosmos-sdk/pull/#14509) Added `AddKeyringFlags` function.
* (x/gov,cli) [#14718](https://github.com/cosmos/cosmos-sdk/pull/14718) Added `AddGovPropFlagsToCmd` and `ReadGovPropFlags` functions.
* (store) [#14189](https://github.com/cosmos/cosmos-sdk/pull/14189) Add config `iavl-lazy-loading` to enable lazy loading of iavl store, to improve start up time of archive nodes, add method `SetLazyLoading` to `CommitMultiStore` interface.
>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
### Improvements

Expand Down
5 changes: 5 additions & 0 deletions baseapp/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,11 @@ func SetIAVLDisableFastNode(disable bool) func(*BaseApp) {
return func(bapp *BaseApp) { bapp.cms.SetIAVLDisableFastNode(disable) }
}

// SetIAVLLazyLoading enables/disables lazy loading of the IAVL store.
func SetIAVLLazyLoading(lazyLoading bool) func(*BaseApp) {
return func(bapp *BaseApp) { bapp.cms.SetLazyLoading(lazyLoading) }
}

// SetInterBlockCache provides a BaseApp option function that sets the
// inter-block cache.
func SetInterBlockCache(cache sdk.MultiStorePersistentCache) func(*BaseApp) {
Expand Down
8 changes: 8 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,15 @@ require (
)

replace (
<<<<<<< HEAD
github.com/99designs/keyring => github.com/cosmos/keyring v1.2.0
=======
cosmossdk.io/collections => ./collections
cosmossdk.io/store => ./store
github.com/99designs/keyring => github.com/cosmos/keyring v1.2.0
github.com/cosmos/iavl => github.com/cosmos/iavl v0.20.0-alpha.2

>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
// dgrijalva/jwt-go is deprecated and doesn't receive security updates.
// TODO: remove it: https://github.com/cosmos/cosmos-sdk/issues/13134
github.com/dgrijalva/jwt-go => github.com/golang-jwt/jwt/v4 v4.4.2
Expand Down
12 changes: 12 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,15 @@ cosmossdk.io/errors v1.0.0-beta.7 h1:gypHW76pTQGVnHKo6QBkb4yFOJjC+sUGRc5Al3Odj1w
cosmossdk.io/errors v1.0.0-beta.7/go.mod h1:mz6FQMJRku4bY7aqS/Gwfcmr/ue91roMEKAmDUDpBfE=
cosmossdk.io/math v1.0.0-beta.4 h1:JtKedVLGzA0vv84xjYmZ75RKG35Kf2WwcFu8IjRkIIw=
cosmossdk.io/math v1.0.0-beta.4/go.mod h1:An0MllWJY6PxibUpnwGk8jOm+a/qIxlKmL5Zyp9NnaM=
<<<<<<< HEAD
cosmossdk.io/tools/rosetta v0.2.0 h1:Ae499UiZ9yPNCXvjOBO/R9I1pksCJfxoqWauEZgA/gs=
cosmossdk.io/tools/rosetta v0.2.0/go.mod h1:3mn8QuE2wLUdTi77/gbDXdFqXZdBdiBJhgAWUTSXPv8=
=======
cosmossdk.io/x/tx v0.1.0 h1:uyyYVjG22B+jf54N803Z99Y1uPvfuNP3K1YShoCHYL8=
cosmossdk.io/x/tx v0.1.0/go.mod h1:qsDv7e1fSftkF16kpSAk+7ROOojyj+SC0Mz3ukI52EQ=
cosmossdk.io/x/upgrade v0.0.0-20230127052425-54c8e1568335 h1:Ke74O79cHxXhPiVWk9D+TkXsgSbjuXwunDwddj1XlsY=
cosmossdk.io/x/upgrade v0.0.0-20230127052425-54c8e1568335/go.mod h1:H+VCBqiCBedKNSYuUEhIeq0lgRQunTOHmGa0u+DLLDY=
>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
filippo.io/edwards25519 v1.0.0-rc.1 h1:m0VOOB23frXZvAOK44usCgLWvtsxIoMCTBGJZlpmGfU=
filippo.io/edwards25519 v1.0.0-rc.1/go.mod h1:N1IkdkCkiLB6tki+MYJoSx2JTY9NUlxZE7eHn5EwJns=
Expand Down Expand Up @@ -176,8 +183,13 @@ github.com/cosmos/gogoproto v1.4.4 h1:nVAsgLlAf5jeN0fV7hRlkZvf768zU+dy4pG+hxc2P3
github.com/cosmos/gogoproto v1.4.4/go.mod h1:/yl6/nLwsZcZ2JY3OrqjRqvqCG9InUMcXRfRjQiF9DU=
github.com/cosmos/gorocksdb v1.2.0 h1:d0l3jJG8M4hBouIZq0mDUHZ+zjOx044J3nGRskwTb4Y=
github.com/cosmos/gorocksdb v1.2.0/go.mod h1:aaKvKItm514hKfNJpUJXnnOWeBnk2GL4+Qw9NHizILw=
<<<<<<< HEAD
github.com/cosmos/iavl v0.19.5-rc.1 h1:4PjF2PdScyPbN1WbXpiQU21YtyonnrMU31xN74g8Rkg=
github.com/cosmos/iavl v0.19.5-rc.1/go.mod h1:X9PKD3J0iFxdmgNLa7b2LYWdsGd90ToV5cAONApkEPw=
=======
github.com/cosmos/iavl v0.20.0-alpha.2 h1:IIUIOqPG1LC2JGiQVeNxAi+Fon/M8TXMBAN0XsUSqq8=
github.com/cosmos/iavl v0.20.0-alpha.2/go.mod h1:25YJYzilTErJ2mKfNB3xyWL9IsCwEQdNzdIutg2mh3U=
>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
github.com/cosmos/keyring v1.2.0 h1:8C1lBP9xhImmIabyXW4c3vFjjLiBdGCmfLUfeZlV1Yo=
github.com/cosmos/keyring v1.2.0/go.mod h1:fc+wB5KTk9wQ9sDx0kFXB3A0MaeGHM9AwRStKOQ5vOA=
github.com/cosmos/ledger-cosmos-go v0.12.1 h1:sMBxza5p/rNK/06nBSNmsI/WDqI0pVJFVNihy1Y984w=
Expand Down
4 changes: 4 additions & 0 deletions server/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,9 @@ type BaseConfig struct {
// IAVLDisableFastNode enables or disables the fast sync node.
IAVLDisableFastNode bool `mapstructure:"iavl-disable-fastnode"`

// IAVLLazyLoading enable/disable the lazy loading of iavl store.
IAVLLazyLoading bool `mapstructure:"iavl-lazy-loading"`

// AppDBBackend defines the type of Database to use for the application and snapshots databases.
// An empty string indicates that the Tendermint config's DBBackend value should be used.
AppDBBackend string `mapstructure:"app-db-backend"`
Expand Down Expand Up @@ -296,6 +299,7 @@ func DefaultConfig() *Config {
IndexEvents: make([]string, 0),
IAVLCacheSize: 781250,
IAVLDisableFastNode: false,
IAVLLazyLoading: false,
AppDBBackend: "",
},
Telemetry: telemetry.Config{
Expand Down
4 changes: 4 additions & 0 deletions server/config/toml.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,10 @@ iavl-cache-size = {{ .BaseConfig.IAVLCacheSize }}
# Default is false.
iavl-disable-fastnode = {{ .BaseConfig.IAVLDisableFastNode }}
# IAVLLazyLoading enable/disable the lazy loading of iavl store.
# Default is false.
iavl-lazy-loading = {{ .BaseConfig.IAVLLazyLoading }}
# AppDBBackend defines the database backend type to use for the application and snapshots DBs.
# An empty string indicates that a fallback will be used.
# First fallback is the deprecated compile-time types.DBBackend value.
Expand Down
4 changes: 4 additions & 0 deletions server/mock/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,10 @@ func (ms multiStore) SetIAVLDisableFastNode(disable bool) {
panic("not implemented")
}

func (ms multiStore) SetLazyLoading(bool) {
panic("not implemented")
}

func (ms multiStore) SetInitialVersion(version int64) error {
panic("not implemented")
}
Expand Down
1 change: 1 addition & 0 deletions server/start.go
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ const (
FlagMinRetainBlocks = "min-retain-blocks"
FlagIAVLCacheSize = "iavl-cache-size"
FlagDisableIAVLFastNode = "iavl-disable-fastnode"
FlagIAVLLazyLoading = "iavl-lazy-loading"

// state sync-related flags
FlagStateSyncSnapshotInterval = "state-sync.snapshot-interval"
Expand Down
1 change: 1 addition & 0 deletions server/util.go
Original file line number Diff line number Diff line change
Expand Up @@ -467,5 +467,6 @@ func DefaultBaseappOptions(appOpts types.AppOptions) []func(*baseapp.BaseApp) {
mempool.SenderNonceMaxTxOpt(cast.ToInt(appOpts.Get(FlagMempoolMaxTxs))),
),
),
baseapp.SetIAVLLazyLoading(cast.ToBool(appOpts.Get(FlagIAVLLazyLoading))),
}
}
18 changes: 18 additions & 0 deletions simapp/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,13 @@ require (
github.com/cosmos/gogogateway v1.2.0 // indirect
github.com/cosmos/gogoproto v1.4.4 // indirect
github.com/cosmos/gorocksdb v1.2.0 // indirect
<<<<<<< HEAD
github.com/cosmos/iavl v0.19.5-rc.1 // indirect
github.com/cosmos/ledger-cosmos-go v0.12.1 // indirect
=======
github.com/cosmos/iavl v0.20.0-alpha1.0.20230129210330-66c167f401c7 // indirect
github.com/cosmos/ledger-cosmos-go v0.13.0 // indirect
>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
github.com/cosmos/rosetta-sdk-go v0.9.0 // indirect
github.com/creachadair/taskgroup v0.3.2 // indirect
github.com/danieljoos/wincred v1.1.2 // indirect
Expand Down Expand Up @@ -160,11 +165,24 @@ require (
sigs.k8s.io/yaml v1.3.0 // indirect
)

<<<<<<< HEAD
=======
// TODO tag all extracted modules after SDK refactor
replace (
cosmossdk.io/store => ../store
cosmossdk.io/x/evidence => ../x/evidence
cosmossdk.io/x/feegrant => ../x/feegrant
cosmossdk.io/x/nft => ../x/nft
cosmossdk.io/x/upgrade => ../x/upgrade
)

>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
replace (
// use cosmos fork of keyring
github.com/99designs/keyring => github.com/cosmos/keyring v1.2.0
// Simapp always use the latest version of the cosmos-sdk
github.com/cosmos/cosmos-sdk => ../.
github.com/cosmos/iavl => github.com/cosmos/iavl v0.20.0-alpha.2
// Fix upstream GHSA-h395-qcrw-5vmq vulnerability.
// TODO Remove it: https://github.com/cosmos/cosmos-sdk/issues/10409
github.com/gin-gonic/gin => github.com/gin-gonic/gin v1.8.1
Expand Down
10 changes: 10 additions & 0 deletions simapp/go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,11 @@ cosmossdk.io/errors v1.0.0-beta.7 h1:gypHW76pTQGVnHKo6QBkb4yFOJjC+sUGRc5Al3Odj1w
cosmossdk.io/errors v1.0.0-beta.7/go.mod h1:mz6FQMJRku4bY7aqS/Gwfcmr/ue91roMEKAmDUDpBfE=
cosmossdk.io/math v1.0.0-beta.4 h1:JtKedVLGzA0vv84xjYmZ75RKG35Kf2WwcFu8IjRkIIw=
cosmossdk.io/math v1.0.0-beta.4/go.mod h1:An0MllWJY6PxibUpnwGk8jOm+a/qIxlKmL5Zyp9NnaM=
<<<<<<< HEAD
=======
cosmossdk.io/tools/confix v0.0.0-20230120150717-4f6f6c00021f h1:LMXqH69KBG/R8w18sooHtoUZ0+5hcc99m6OjBiooDAo=
cosmossdk.io/tools/confix v0.0.0-20230120150717-4f6f6c00021f/go.mod h1:/apC5+JHM2A72kUY3z+55FWdIn/2ai2mTAYtSBDY4Lo=
>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
cosmossdk.io/tools/rosetta v0.2.0 h1:Ae499UiZ9yPNCXvjOBO/R9I1pksCJfxoqWauEZgA/gs=
cosmossdk.io/tools/rosetta v0.2.0/go.mod h1:3mn8QuE2wLUdTi77/gbDXdFqXZdBdiBJhgAWUTSXPv8=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
Expand Down Expand Up @@ -174,8 +179,13 @@ github.com/cosmos/gogoproto v1.4.4 h1:nVAsgLlAf5jeN0fV7hRlkZvf768zU+dy4pG+hxc2P3
github.com/cosmos/gogoproto v1.4.4/go.mod h1:/yl6/nLwsZcZ2JY3OrqjRqvqCG9InUMcXRfRjQiF9DU=
github.com/cosmos/gorocksdb v1.2.0 h1:d0l3jJG8M4hBouIZq0mDUHZ+zjOx044J3nGRskwTb4Y=
github.com/cosmos/gorocksdb v1.2.0/go.mod h1:aaKvKItm514hKfNJpUJXnnOWeBnk2GL4+Qw9NHizILw=
<<<<<<< HEAD
github.com/cosmos/iavl v0.19.5-rc.1 h1:4PjF2PdScyPbN1WbXpiQU21YtyonnrMU31xN74g8Rkg=
github.com/cosmos/iavl v0.19.5-rc.1/go.mod h1:X9PKD3J0iFxdmgNLa7b2LYWdsGd90ToV5cAONApkEPw=
=======
github.com/cosmos/iavl v0.20.0-alpha.2 h1:IIUIOqPG1LC2JGiQVeNxAi+Fon/M8TXMBAN0XsUSqq8=
github.com/cosmos/iavl v0.20.0-alpha.2/go.mod h1:25YJYzilTErJ2mKfNB3xyWL9IsCwEQdNzdIutg2mh3U=
>>>>>>> b266522c8 (feat: add config `iavl-lazy-loading` to enable lazy loading of iavl store (#14189))
github.com/cosmos/keyring v1.2.0 h1:8C1lBP9xhImmIabyXW4c3vFjjLiBdGCmfLUfeZlV1Yo=
github.com/cosmos/keyring v1.2.0/go.mod h1:fc+wB5KTk9wQ9sDx0kFXB3A0MaeGHM9AwRStKOQ5vOA=
github.com/cosmos/ledger-cosmos-go v0.12.1 h1:sMBxza5p/rNK/06nBSNmsI/WDqI0pVJFVNihy1Y984w=
Expand Down
67 changes: 67 additions & 0 deletions store/go.mod
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
module cosmossdk.io/store

go 1.19

require (
cosmossdk.io/errors v1.0.0-beta.7
github.com/armon/go-metrics v0.4.1
github.com/confio/ics23/go v0.9.0
github.com/cosmos/cosmos-db v0.0.0-20230119180254-161cf3632b7c
github.com/cosmos/gogoproto v1.4.4
github.com/cosmos/iavl v0.20.0-alpha.2
github.com/golang/mock v1.6.0
github.com/golang/protobuf v1.5.2 // indirect
github.com/hashicorp/golang-lru v0.5.5-0.20210104140557-80c98217689d
github.com/spf13/cast v1.5.0
github.com/stretchr/testify v1.8.1
github.com/tendermint/tendermint v0.37.0-rc2
github.com/tidwall/btree v1.6.0
golang.org/x/exp v0.0.0-20230118134722-a68e582fa157
google.golang.org/genproto v0.0.0-20230125152338-dcaf20b6aeaa // indirect
google.golang.org/grpc v1.52.3 // indirect
google.golang.org/protobuf v1.28.1 // indirect
gotest.tools/v3 v3.4.0
)

require (
github.com/DataDog/zstd v1.4.5 // indirect
github.com/HdrHistogram/hdrhistogram-go v1.1.2 // indirect
github.com/btcsuite/btcd/btcec/v2 v2.3.2 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/cockroachdb/errors v1.9.1 // indirect
github.com/cockroachdb/logtags v0.0.0-20230118201751-21c54148d20b // indirect
github.com/cockroachdb/pebble v0.0.0-20220817183557-09c6e030a677 // indirect
github.com/cockroachdb/redact v1.1.3 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/decred/dcrd/dcrec/secp256k1/v4 v4.1.0 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/getsentry/sentry-go v0.17.0 // indirect
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.5.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
github.com/klauspost/compress v1.15.12 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/kr/text v0.2.0 // indirect
github.com/linxGnu/grocksdb v1.7.10 // indirect
github.com/nxadm/tail v1.4.8 // indirect
github.com/petermattis/goid v0.0.0-20180202154549-b0b1615b78e5 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
github.com/rogpeppe/go-internal v1.9.0 // indirect
github.com/sasha-s/go-deadlock v0.3.1 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20210819022825-2ae1ddf74ef7 // indirect
golang.org/x/crypto v0.5.0 // indirect
golang.org/x/net v0.5.0 // indirect
golang.org/x/sys v0.4.0 // indirect
golang.org/x/text v0.6.0 // indirect
golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

// Fix upstream GHSA-h395-qcrw-5vmq vulnerability.
// TODO Remove it: https://github.com/cosmos/cosmos-sdk/issues/10409
replace github.com/gin-gonic/gin => github.com/gin-gonic/gin v1.8.1
Loading

0 comments on commit c71dc1c

Please sign in to comment.