feat(wasmer): get and cache state version in instance context #2747
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qdm12
changed the base branch from
development
to
qdm12/runtime/version-structs-refactor
This was referenced Aug 11, 2022
qdm12
force-pushed
the
qdm12/runtime/version-structs-refactor
branch
from
b5dd0cd
to
11e5200
Compare
qdm12
force-pushed
the
qdm12/wasmer/persist-version
branch
from
051b304
to
411556f
Compare
qdm12
force-pushed
the
qdm12/runtime/version-structs-refactor
branch
2 times, most recently
from
6486321
to
3386e46
Compare
qdm12
force-pushed
the
qdm12/wasmer/persist-version
branch
from
411556f
to
2b18bf8
Compare
qdm12
commented
Aug 16, 2022
lib/runtime/wasmer/test_helpers.go
Outdated
Comment on lines
60
to
62
| versions = &testVersions{ | ||
| stateVersion: 0, | ||
| } |
There was a problem hiding this comment.
When shall we bump this to 1 for the host api runtime? When we support the v1 state trie (finally) or would you see another milestone to bump this up for the host api test runtime? 🤔
There was a problem hiding this comment.
From elizabeth 👍
i guess, do you want to test the imported trie functions w the different versions? if so you might need the two diff versions (or just override it)
qdm12
requested review from
noot,
edwardmack,
timwu20,
EclesioMeloJunior,
jimjbrettj and
kishansagathiya
as code owners
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## development #2747 +/- ##
===============================================
- Coverage 63.12% 63.03% -0.09%
===============================================
Files 213 213
Lines 27034 27050 +16
===============================================
- Hits 17064 17051 -13
- Misses 8420 8446 +26
- Partials 1550 1553 +3 |
Base automatically changed from
qdm12/runtime/version-structs-refactor
to
development
August 16, 2022 12:09
qdm12
force-pushed
the
qdm12/wasmer/persist-version
branch
from
2b18bf8
to
33097de
Compare
qdm12
force-pushed
the
qdm12/wasmer/persist-version
branch
from
33097de
to
15da099
Compare
- Set when creating instance - Update when updating runtime code
- So it can be used by Version calls and be cheap for external packages
qdm12
force-pushed
the
qdm12/wasmer/persist-version
branch
from
15da099
to
90d8326
Compare
qdm12
force-pushed
the
qdm12/wasmer/persist-version
branch
from
90d8326
to
20a41ee
Compare
edwardmack
approved these changes
Aug 24, 2022
rrtti
pushed a commit
to polkadot-fellows/seeding
that referenced
this pull request
Sep 29, 2022
I am Quentin Mc Gaw, a software engineer working the Go Polkadot host **Gossamer**. I have been working full time on Gossamer since October 2021, mostly on the state trie and storage. I have also made a [few minor pull requests](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12) to the Polkadot specification repository. I am requesting to join the Fellowship at rank 1. ## Main contributions ### Gossamer - Fix memory leaks - Trie encoding buffer pools usage fixed [#2009](ChainSafe/gossamer#2009) - Fix state map of tries memory leak [#2286](ChainSafe/gossamer#2286) - Fix sync benchmark [#2234](ChainSafe/gossamer#2234) - Trie proof fixes ([#2604](ChainSafe/gossamer#2604), [#2661](ChainSafe/gossamer#2661)) - Fix end to end tests orchestration ([#2470](ChainSafe/gossamer#2470), [#2452](ChainSafe/gossamer#2452), [#2385](ChainSafe/gossamer#2385), [#2370](ChainSafe/gossamer#2370)) - State trie statistics ([#2378](ChainSafe/gossamer#2378), [#2310](ChainSafe/gossamer#2310), [#2272](ChainSafe/gossamer#2272)) - State trie fixes and improvements - Only deep copy nodes when mutation is certain [#2352](ChainSafe/gossamer#2352) and [#2223](ChainSafe/gossamer#2223) - Only deep copy necessary fields of a node [#2384](ChainSafe/gossamer#2384) - Use Merkle values for database keys instead of always hash [#2725](ChainSafe/gossamer#2725) - Opportunistic parallel Merkle value commputing [#2081](ChainSafe/gossamer#2081) - Grandpa capped number of tracked messages ([#2490](ChainSafe/gossamer#2490), [#2485](ChainSafe/gossamer#2485)) - Add pprof HTTP service for profiling [#1991](ChainSafe/gossamer#1991) Ongoing work: - State trie lazy loading and caching - State trie v1 support ([#2736](ChainSafe/gossamer#2736), [#2747](ChainSafe/gossamer#2747), [#2687](ChainSafe/gossamer#2687), [#2686](ChainSafe/gossamer#2686), [#2685](ChainSafe/gossamer#2685), [#2673](ChainSafe/gossamer#2673), [#2611](ChainSafe/gossamer#2611), [#2530](ChainSafe/gossamer#2530)) ### Polkadot specification ➡️ [Pull requests from qdm12](https://github.com/w3f/polkadot-spec/pulls?q=is%3Apr+is%3Aclosed+author%3Aqdm12)
github-actions bot
pushed a commit
that referenced
this pull request
Nov 23, 2022
# [0.7.0](v0.6.0...v0.7.0) (2022-11-23) ### Bug Fixes * **chain:** update ed25519 addresses in dev/gssmr genesis files ([#2225](#2225)) ([5f47d8b](5f47d8b)) * **ci:** caching of Go caches ([#2451](#2451)) ([ce3c10c](ce3c10c)) * **ci:** codecov.yml configuration ([#2698](#2698)) ([d4fc383](d4fc383)) * **ci:** comment skip code for required workflows ([#2312](#2312)) ([45dce9b](45dce9b)) * **ci:** copyright workflow to exit if different files ([#2487](#2487)) ([89c32ae](89c32ae)) * **ci:** deepsource toml configuration ([#2744](#2744)) ([86a70de](86a70de)) * **ci:** embed v0.9.20 runtime, update test suite, and ci workflows ([#2543](#2543)) ([0fff418](0fff418)), closes [#2419](#2419) [#2561](#2561) [#2572](#2572) [#2581](#2581) [#2671](#2671) * **ci:** fix staging Dockerfile ([#2474](#2474)) ([ae04b80](ae04b80)) * **ci:** mocks checking fixes ([#2274](#2274)) ([d1308e0](d1308e0)) * **ci:** run devnet module unit tests ([#2756](#2756)) ([f635c59](f635c59)) * **ci:** run golangci-lint on integration tests ([#2275](#2275)) ([3ae3401](3ae3401)) * **cmd:** allow --genesis flag to be passed to base command ([#2427](#2427)) ([7f5b5aa](7f5b5aa)) * **cmd:** avoid nil pointer dereference ([#2578](#2578)) ([f2cdfea](f2cdfea)) * **config:** temporary fix for pprof enabled setting precedence ([#2786](#2786)) ([d4d6262](d4d6262)) * **core:** fix txn pool for latest runtime ([#2809](#2809)) ([1551e66](1551e66)) * **deps:** upgrade chaindb to remove badger logs ([#2738](#2738)) ([e0c5706](e0c5706)) * **devnet:** Fix build workflow for devnet ([#2125](#2125)) ([0375fc2](0375fc2)) * **Dockerfile:** remove script entrypoint ([#2707](#2707)) ([abd161b](abd161b)) * **dot/core:** `RuntimeInstance` interface `Version` signature ([#2783](#2783)) ([7d66ec0](7d66ec0)) * **dot/core:** fix the race condition in TrieState ([#2499](#2499)) ([804069c](804069c)), closes [#2402](#2402) * **dot/digest:** BABE NextEpochData and NextConfigData should be set on finalization ([#2339](#2339)) ([e991cc8](e991cc8)) * **dot/digest:** verify if next epoch already contains some definition ([#2472](#2472)) ([a2ac6c2](a2ac6c2)) * **dot/netwok:** check for duplicate message earlier ([#2435](#2435)) ([d62503f](d62503f)) * **dot/network:** change BlockRequestMessage number from uint64 to uint32 ([8105cd4](8105cd4)) * **dot/network:** close notifications streams ([#2093](#2093)) ([de6e7c9](de6e7c9)), closes [#2046](#2046) * **dot/network:** fixing errMissingHandshakeMutex ([#2303](#2303)) ([eb07a53](eb07a53)) * **dot/network:** memory improvement for network buffers ([#2233](#2233)) ([fd9b70d](fd9b70d)) * **dot/network:** public IP address logging ([#2140](#2140)) ([9e21587](9e21587)) * **dot/network:** re-add nil mutex check for disconnected peer ([#2408](#2408)) ([9b39bd1](9b39bd1)) * **dot/network:** remove `defer cancel()` inside loop ([#2248](#2248)) ([9e360a5](9e360a5)) * **dot/network:** resize bytes slice buffer if needed ([#2291](#2291)) ([8db8b2a](8db8b2a)) * **dot/peerset:** fix sending on closed channel race condition when dropping peer ([#2573](#2573)) ([2fa5d8a](2fa5d8a)) * **dot/peerset:** remove race conditions from `peerset` package ([#2267](#2267)) ([df09d45](df09d45)) * **dot/rpc/modules:** grandpa.proveFinality update parameters, fix bug ([#2576](#2576)) ([e7749cf](e7749cf)) * **dot/rpc/modules:** rpc.state.queryStorage fixed ([#2565](#2565)) ([1ec0d47](1ec0d47)) * **dot/rpc:** include unsafe flags to be considered by RPC layer ([#2483](#2483)) ([3822257](3822257)) * **dot/state/epoch, lib/babe:** enable block production through epochs without rely on finalization ([#2593](#2593)) ([a0a1804](a0a1804)) * **dot/state:** actually prune finalized tries from memory ([#2196](#2196)) ([e4bc375](e4bc375)) * **dot/state:** change map of tries implementation to have working garbage collection ([#2206](#2206)) ([fada46b](fada46b)) * **dot/state:** inject mutex protected tries to states ([#2287](#2287)) ([67a9bbb](67a9bbb)) * **dot/subscription:** check websocket message from untrusted data ([#2527](#2527)) ([1f20d98](1f20d98)) * **dot/subscription:** unsafe type casting from untrusted input ([#2529](#2529)) ([1015733](1015733)) * **dot/sync, dot/rpc:** implement HighestBlock ([#2195](#2195)) ([f8d8657](f8d8657)) * **dot/sync:** cleanup logs; don't log case where we fail to get parent while processing ([#2188](#2188)) ([cb360ab](cb360ab)) * **dot/sync:** fix "block with unknown header is ready" error ([#2191](#2191)) ([483466f](483466f)) * **dot/sync:** fix `Test_lockQueue_threadSafety` ([#2605](#2605)) ([223cfbb](223cfbb)) * **dot/sync:** Fix flaky tests `Test_chainSync_logSyncSpeed` and `Test_chainSync_start` ([#2610](#2610)) ([7e1014b](7e1014b)) * **dot/sync:** Gossip `BlockAnnounceMessage` only after successfully imported ([#2885](#2885)) ([69031a6](69031a6)) * **dot/sync:** remove block announcement in `bootstrap` sync mode ([#2906](#2906)) ([2b4c257](2b4c257)) * **dot/sync:** remove max size limit from ascending block requests ([#2256](#2256)) ([e287d7e](e287d7e)) * **dot/sync:** sync benchmark ([#2234](#2234)) ([2f3aef8](2f3aef8)) * **dot/telemetry:** telemetry hashes to be in the hexadecimal format ([#2194](#2194)) ([9b48106](9b48106)) * **dot:** database close error checks ([#2948](#2948)) ([bdb0eea](bdb0eea)) * **dot:** no error logged for init check ([#2502](#2502)) ([2971325](2971325)) * ensure we convert the `uint` type ([#2626](#2626)) ([792e53f](792e53f)) * fix logger mutex locking in `.New` method ([#2114](#2114)) ([e7207ed](e7207ed)) * **internal/log:** log level `DoNotChange` ([#2672](#2672)) ([0008b59](0008b59)) * **levels-logged:** Fix log levels logging at start ([#2236](#2236)) ([a90a6e0](a90a6e0)) * **lib/babe:** check if authority index is in the `authorities` range ([#2601](#2601)) ([1072888](1072888)) * **lib/babe:** ensure the slot time is correct before build a block ([#2648](#2648)) ([78c03b6](78c03b6)) * **lib/babe:** epoch context error wrapping ([#2484](#2484)) ([c053dea](c053dea)) * **lib/babe:** Unrestricted Loop When Building Blocks (GSR-19) ([#2632](#2632)) ([139ad89](139ad89)) * **lib/blocktree:** reimplement `BestBlockHash` to take into account primary blocks in fork choice rule ([#2254](#2254)) ([1a368e2](1a368e2)) * **lib/grandpa:** avoid spamming round messages ([#2688](#2688)) ([b0042b8](b0042b8)) * **lib/grandpa:** capped number of tracked commit messages ([#2490](#2490)) ([47c23e6](47c23e6)) * **lib/grandpa:** capped number of tracked vote messages ([#2485](#2485)) ([d2ee47e](d2ee47e)), closes [#1531](#1531) * **lib/grandpa:** check equivocatory votes count ([#2497](#2497)) ([014629d](014629d)), closes [#2401](#2401) * **lib/grandpa:** Duplicate votes is GRANDPA are counted as equivocatory votes (GSR-11) ([#2624](#2624)) ([422e7b3](422e7b3)) * **lib/grandpa:** Storing Justification Allows Extra Bytes (GSR-13) ([#2618](#2618)) ([0fcde63](0fcde63)) * **lib/grandpa:** update grandpa protocol ID ([#2678](#2678)) ([3be75b2](3be75b2)) * **lib/grandpa:** various finality fixes, improves cross-client finality ([#2368](#2368)) ([c04d185](c04d185)) * **lib/grandpa:** verify equivocatory votes in grandpa justifications ([#2486](#2486)) ([368f8b6](368f8b6)) * **lib/runtime:** avoid caching version in runtime instance ([#2425](#2425)) ([7ab31f0](7ab31f0)) * **lib/runtime:** stub v0.9.17 host API functions ([#2420](#2420)) ([6a7b223](6a7b223)) * **lib/trie:** `handleDeletion` generation propagation ([24c303d](24c303d)) * **lib/trie:** `PopulateMerkleValues` functionality changes and fixes ([#2871](#2871)) ([7131290](7131290)) * **lib/trie:** Check for root in EncodeAndHash ([#2359](#2359)) ([087db89](087db89)) * **lib/trie:** Make sure writing and reading a trie to disk gives the same trie and cover more store/load child trie related test cases ([#2302](#2302)) ([7cd4118](7cd4118)) * **lib/trie:** prepare trie nodes for mutation only when needed ([#2834](#2834)) ([26868df](26868df)) * **lib/trie:** remove map deletion at `loadProof` ([#2259](#2259)) ([fbd13d2](fbd13d2)) * **lint:** fix issues found by golangcilint 1.47.3 ([#2715](#2715)) ([5765e67](5765e67)) * **mocks:** add missing `//go:generate` for mocks ([#2273](#2273)) ([f4f7465](f4f7465)) * **pprof:** pprofserver flag changed to boolean ([#2205](#2205)) ([be00a69](be00a69)) * **staging:** revise datadog-agent start process ([#2935](#2935)) ([36ce37d](36ce37d)) * **state/epoch:** assign epoch 1 when block number is 0 ([#2592](#2592)) ([e5c8cf5](e5c8cf5)) * **state/grandpa:** track changes across forks ([#2519](#2519)) ([3ab76bc](3ab76bc)) * **tests:** `TestAuthorModule_HasSessionKeys_Integration` ([#2932](#2932)) ([8d809aa](8d809aa)) * **tests:** fix block body regex in `TestChainRPC` ([#2805](#2805)) ([b0680f8](b0680f8)) * **tests:** Fix RFC3339 regex for log unit tests ([9caea2a](9caea2a)) * **tests:** Fix wasmer flaky sorts ([#2643](#2643)) ([7eede9a](7eede9a)) * **tests:** handle node crash during waiting ([#2691](#2691)) ([843bd50](843bd50)) * **tests:** update block body regex in `TestChainRPC` ([#2674](#2674)) ([055e5c3](055e5c3)) * **trie:** decode inline child nodes ([#2369](#2369)) ([9efde47](9efde47)) * **trie:** descendants count for clear prefix ([#2606](#2606)) ([1826896](1826896)) * **trie:** disallow empty byte slice node values ([#2927](#2927)) ([d769d1c](d769d1c)) * **trie:** equality differentiate nil and empty storage values ([#2969](#2969)) ([72a08ec](72a08ec)) * **trie:** no in-memory caching of node encoding ([#2919](#2919)) ([856780b](856780b)) * **trie:** Panic when deleting nonexistent keys from trie (GSR-10) ([#2609](#2609)) ([7886318](7886318)) * **trie:** remove encoding buffers pool ([#2929](#2929)) ([f4074cc](f4074cc)) * **trie:** use cached Merkle values for root hash ([#2943](#2943)) ([ec2549a](ec2549a)) * **trie:** use direct Merkle value for database keys ([#2725](#2725)) ([1a3c3ae](1a3c3ae)) * upgrade auto-generated mocks ([#2910](#2910)) ([a2975a5](a2975a5)) * **wasmer:** error logs for signature verification ([#2752](#2752)) ([363c080](363c080)) * **wasmer:** fix flaky sort in `Test_ext_crypto_sr25519_public_keys_version_1` ([#2607](#2607)) ([c061b35](c061b35)) ### Features * **build:** add `github.com/breml/rootcerts` ([#2695](#2695)) ([c74a5b0](c74a5b0)) * **build:** binary built-in timezone data ([#2697](#2697)) ([fdd5bda](fdd5bda)) * **chain:** use always the raw genesis file ([#2775](#2775)) ([dd2fbc9](dd2fbc9)) * **ci:** update mockery from `2.10` to `2.14` ([#2642](#2642)) ([d2c42b8](d2c42b8)) * **cross-client:** create docker-compose.yml for local devnet ([#2282](#2282)) ([8abbd87](8abbd87)) * detect chain directory dynamically ([#2292](#2292)) ([85c466c](85c466c)) * **devnet:** add substrate docker images to dockerfile ([#2263](#2263)) ([b7b2a66](b7b2a66)) * **devnet:** continuous integration `gssmr` devnet on AWS ECS ([#2096](#2096)) ([d096d44](d096d44)) * **docker:** docker-compose.yml to run Gossamer, Prometheus and Grafana ([#2706](#2706)) ([c5dda51](c5dda51)) * **dot/network:** add mismatched genesis peer reporting ([#2265](#2265)) ([a1d7269](a1d7269)) * **dot/state:** `gossamer_storage_tries_cached_total` gauge metric ([#2272](#2272)) ([625cbcf](625cbcf)) * **e2e:** build Gossamer on any test run ([#2608](#2608)) ([f97e0ef](f97e0ef)) * **go:** upgrade Go from 1.17 to 1.18 ([#2379](#2379)) ([d85a1db](d85a1db)) * include nested varying data type on neighbor messages ([#2722](#2722)) ([426569a](426569a)) * **lib/babe:** implement secondary slot block production ([#2260](#2260)) ([fcb81a3](fcb81a3)) * **lib/runtime:** support Substrate WASM compression ([#2213](#2213)) ([fd60061](fd60061)) * **lib/trie:** atomic tracked merkle values ([#2876](#2876)) ([1c4174c](1c4174c)) * **lib/trie:** clear fields when node is dirty ([#2297](#2297)) ([1162828](1162828)) * **lib/trie:** only copy nodes when mutation is certain ([#2352](#2352)) ([86624cf](86624cf)) * **lib/trie:** opportunistic parallel hashing ([#2081](#2081)) ([790dfb5](790dfb5)) * **metrics:** replace metrics port with address (breaking change) ([#2382](#2382)) ([d2ec68d](d2ec68d)) * **pkg/scale:** add `Encoder` with `Encode` method ([#2741](#2741)) ([af5c63f](af5c63f)) * **pkg/scale:** add use of pkg/error Wrap for error handling ([#2708](#2708)) ([08c4281](08c4281)) * **pkg/scale:** encoding and decoding of maps in scale ([#2894](#2894)) ([405db51](405db51)), closes [#2796](#2796) * **pkg/scale:** support for custom `VaryingDataType` types ([#2612](#2612)) ([914a747](914a747)) * remove uneeded runtime prefix logs ([#2110](#2110)) ([8bd05d1](8bd05d1)) * remove unused code ([#2677](#2677)) ([b3698d7](b3698d7)) * **scale:** add range checks to decodeUint function ([#2683](#2683)) ([ac700f8](ac700f8)) * **trie:** decode all inlined node variants ([#2611](#2611)) ([b09eb07](b09eb07)) * **trie:** export `LoadFromProof` ([#2455](#2455)) ([0b4f33d](0b4f33d)) * **trie:** faster header decoding ([#2649](#2649)) ([d9460e3](d9460e3)) * **trie:** finer deep copy of nodes ([#2384](#2384)) ([bd6d8e4](bd6d8e4)) * **trie:** tracking of number of descendant nodes for each node ([#2378](#2378)) ([dfcdd3c](dfcdd3c)) * **trie:** use scale encoder ([#2930](#2930)) ([e3dc108](e3dc108)) * **wasmer/crypto:** move sig verifier to crypto pkg ([#2057](#2057)) ([dc8bbef](dc8bbef)) * **wasmer:** Add `SetTestVersion` method to `Config` struct ([#2823](#2823)) ([e5c9336](e5c9336)) * **wasmer:** get and cache state version in instance context ([#2747](#2747)) ([3fd63db](3fd63db))
|
🎉 This PR is included in version 0.7.0 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Note this caches the state version for usages in several import functions (see #2664)
Core_versionnot there in test runtimesTests
go test -tags integration github.com/ChainSafe/gossamer/lib/runtime/...Issues
One more step towards #2418
Primary Reviewer
@timwu20