Migration testing runtime API/Bot #8038

kianenigma · 2021-02-03T16:35:08Z

A quick and dirty, but functioning attempt at making an end to end runtime api that can be triggered by a bot to test all the OnRuntimeUpgrade implementations inside a runtime, including two newly added hooks for some pre-and-post checks. I have one example of how these can work here: paritytech/polkadot@146f1e2#diff-db5840c3c03c740de25aed56b9257559c224a04619c079c94cddbb23f9a855a4R961

Given that it works now, I see this as a leaner and more minimalistic approach compared to #7665, but we can replace parts of this PR with test-runner once it is compatible.

My vision with this work is:

Each substrate PR can request the OnRuntimeUpgrade of a particular pallet to be executed against the live state of any chain: \migration_bot run target:pallet-staking source:remote('http://remote-host:9933').
Prior to each release, we run the same command once more on the polkadot node: \migration_bot run target:all cache('./path-to-cache-file.bin').

Current branch works fine, just needs a clean-up and configurability + making a few things generic (e.g. I assume Hash = H256 which is not good). In its current state, running cargo run -- dry-run --target all -lremote-ext=debug will yield something like:

2021-02-03 16:11:16.421   INFO main staking: 💸 new validator set of size 1 has been elected via ElectionCompute::OnChain for era 0
2021-02-03 16:11:16.443   INFO main remote-ext: scraping keypairs from remote node http://localhost:9933 @ 0x36b8402ce0037f61795cfa50a5615c3b0a30bd2dc2b5c2171faae3622886de12
2021-02-03 16:11:16.443   INFO main remote-ext: downloading data for all modules.
2021-02-03 16:12:18.371   INFO main remote-ext: injecting a total of 328567 keys
2021-02-03 16:15:27.709   INFO main node_runtime: !!! DRYRUN MIGRATION UP AHEAD !!!
2021-02-03 16:15:50.996   INFO main node_runtime: !!! DRYRUN MIGRATION DONE !!!

You can try the same, if you have a locally running node.

If this line of work is deemed okay, I will continue with cleaning the code, and making a companion PR that adds the same subcommand to polkadot as well.

CC @apopiak @seunlanlege @bkchr @shawntabrizi

kianenigma · 2021-02-03T16:39:06Z

utils/frame/dry-run-runtime-upgrade/cli/src/lib.rs

+		let heap_pages = Some(1024);
+		let executor = NativeExecutor::<ExecDispatch>::new(wasm_method, heap_pages, 2);
+
+		let ext = remote_externalities::Builder::new().inject(&[(code_key, code)]).build().await;


this .inject here is where the whole crux of the work happens: inject the real code (i.e. the one that is currently in this repo) into the remote fake state. I should double-check that this is what we want.

client/state-db/src/lib.rs

kianenigma · 2021-02-11T11:45:15Z

This is more or less ready for review. A note about naming: I first called everything in this PR along the lines of DryRuntRuntimeUpgrade and such. But this is too specific. I foresee that in the future it will be useful (for me, at least) to add new cli commands and APIs that run the OnInitialize, or a particular Call of a new (not deployed) pallet over the state of a particular chain (or a range of blocks, updating as we go through). So I had to make it a bit more generalized, and I thought the Try prefix, such as TryFrom, TryInto and such are cool enough to go with. It is also much easier to type than dry-run-runtime-upgrade :D so now everything is called try-runtime and TryRuntime. Not perfect, but okay for now.

To test it out, try either of:

cargo remote run -- --release --features try-runtime --manifest-path bin/node/cli/Cargo.toml -- try-runtime --state file://Kusama,0xdd772d86d5cfd2be9a11dd504866c7878ee071aefe02ff7db3d749f98bf890b8,.bin --target Staking

or

cargo run --release --features try-runtime --manifest-path bin/node/cli/Cargo.toml -- try-runtime --state file://Kusama,0xdd772d86d5cfd2be9a11dd504866c7878ee071aefe02ff7db3d749f98bf890b8,.bin --target Staking

and put this file into your substrate root https://drive.google.com/file/d/1WMzYzQREBSlNfPMXcRT33Qq1SKJNqjzf/view?usp=sharing

kianenigma · 2021-02-11T13:39:14Z

utils/frame/try-runtime/remote-externalities/src/lib.rs

+type KeyPair = (StorageKey, StorageData);
+type Number = u32;
+type Hash = sp_core::H256;
+// TODO: make these two generic.


I would be happy to do this in a future PR. Given that we want to use this initially for known chains, I think it is an overkill for now.

kianenigma · 2021-02-11T13:39:59Z

utils/frame/try-runtime/remote-externalities/src/lib.rs

+	}
+
+	#[async_std::test]
+	async fn can_load_cache() {


that binary file (20k) that I bushed will allow us to have at least this one test for this crate.. not sure what to do about it.

bin/node/runtime/src/lib.rs

frame/executive/src/lib.rs

frame/support/src/traits.rs

frame/try-runtime/src/lib.rs

utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

utils/frame/try-runtime/cli/src/lib.rs

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

kianenigma

I would highly appreciate a rather nimble review on this as I can use it quite a lot, and not needing to merge this branch into other works is a great luxury.

Note that I think there are lots of improvements to be made, this PR is just setting the foundation. All in all, I envision the TryRuntime api (all gated by try-runtime feature) to expose a lot of testing functionalities that would allow a runtime to be deeply diagnosed as a black box.

…' of github.com:paritytech/substrate into kiz-finally-finally-finally-finally-migration-testing-2

arkpar · 2021-02-15T10:19:18Z

TryRuntimeCmd stuff looks good to me

gui1117

what is utils/frame/try-runtime/remote-externalities/proxy_test ?

gui1117 · 2021-02-18T15:25:45Z

bin/node/cli/src/command.rs

+					config.task_executor.clone(),
+					registry,
+					config.telemetry_span.clone(),
+				).unwrap();


out of curiosity, why async here ? instead of sync like we do for benchmarks ?

remote-externalities is working with async/await, my assumption was that in such case you need to use the async_runner 🤔

gui1117 · 2021-02-18T15:26:24Z

client/service/src/task_manager/mod.rs

@@ -273,7 +273,7 @@ pub struct TaskManager {
 impl TaskManager {
 	/// If a Prometheus registry is passed, it will be used to report statistics about the
 	/// service tasks.
-	pub(super) fn new(
+	pub fn new(


this looks like a structural change but I don't know that much about the client to say.

I think it is fine but yeah might be worth asking, maybe @cecton?

Normally you don't need to instantiate this but I guess in your case it makes sense if you want to benefit from the clean shutdown of the TaskManager 🤔 ... so I'm not opposed to it. @expenses maybe has another opinion on this?

On the other hand I'm worried by the tokio runtime created later on. You are creating and running 2 tokio runtimes.

utils/frame/try-runtime/remote-externalities/Cargo.toml

primitives/core/src/hexdisplay.rs

kianenigma · 2021-02-18T17:49:47Z

what is utils/frame/try-runtime/remote-externalities/proxy_test ?

#8038 (comment)

…ally-finally-finally-finally-migration-testing-2

shawntabrizi

I am pretty happy with this to start with. I expect it may change a bit as it gets used more, but since it is isolated under its own feature flag, seems super reasonable to merge soon.

utils/frame/try-runtime/remote-externalities/src/lib.rs

bin/node/cli/src/command.rs

…ally-finally-finally-finally-migration-testing-2

kianenigma · 2021-02-19T13:00:41Z

bot merge

ghost · 2021-02-19T13:00:45Z

Waiting for commit status.

ghost · 2021-02-19T13:03:35Z

Checks failed; merge aborted.

…ally-finally-finally-finally-migration-testing-2

kianenigma · 2021-02-19T14:52:03Z

bot merge

ghost · 2021-02-19T14:52:08Z

Trying merge.

bkchr

I don't understand why this was merged with tons of unwraps in non-testing code. (Non-testing is everything that is not behind #[cfg(test)]

bkchr · 2021-02-19T15:04:26Z

utils/frame/remote-externalities/src/lib.rs

+			trace!(target: LOG_TARGET, "rpc: finalized_head");
+			let client: sc_rpc_api::chain::ChainClient<Number, Hash, (), ()> =
+				jsonrpc_core_client::transports::http::connect(&uri).wait().unwrap();
+			Ok(client.finalized_head().wait().unwrap())


bkchr · 2021-02-19T15:04:30Z

utils/frame/remote-externalities/src/lib.rs

+		rt.block_on::<_, _, ()>(futures::lazy(move || {
+			trace!(target: LOG_TARGET, "rpc: finalized_head");
+			let client: sc_rpc_api::chain::ChainClient<Number, Hash, (), ()> =
+				jsonrpc_core_client::transports::http::connect(&uri).wait().unwrap();


bkchr · 2021-02-19T15:05:10Z

utils/frame/remote-externalities/src/lib.rs

+		rt.block_on::<_, _, ()>(futures::lazy(move || {
+			trace!(target: LOG_TARGET, "rpc: storage_pairs: {:?} / {:?}", prefix, at);
+			let client: sc_rpc_api::state::StateClient<Hash> =
+				jsonrpc_core_client::transports::http::connect(&uri).wait().unwrap();


And more and more

bkchr · 2021-02-19T15:05:45Z

utils/frame/remote-externalities/src/lib.rs

+	/// Relay the request to `state_getPairs` rpc endpoint.
+	///
+	/// Note that this is an unsafe RPC.
+	async fn rpc_get_pairs(&self, prefix: StorageKey, at: Hash) -> Vec<KeyPair> {


And why do we create a tokio runtime inside an async function!?

bin/node/cli/src/command.rs

kianenigma · 2021-02-19T15:44:12Z

Sorry, will fix the unwraps in #8123 😄 the async runtime is discussed here: #8038 (comment)

@kianenigma

# Membership Request Hi, I am Kian Paimani, known as @kianenigma. I have been working on Polkadot/Kusama through Parity since February 2019 and I can categorize my main contributions to Polkadot's ecosystem as follows: 1. Maintaining and developing the staking sub-system. 2. General FRAME development, especially testing and quality assurance. 3. Polkadot-native side-projects. 4. Education > My first contribution to Polkadot is also indeed related to staking: paritytech/substrate#1915 ### Staking system I joke as the Polkadot staking to be both my blessing and my curse over the years. I started working on it since the first days that I joined this ecosystem and the work [is ongoing ever since](https://github.com/orgs/paritytech/projects/33/views/9). In the past, I focused on making sure that the staking system is secure and to some extent scalable. More recently, I coordinated the (imminent) launch of Nomination Pools. Nowadays I also put an extra effort on making sure that this sub-system of Polkadot is *sustainable*, through code refactor and educating other core developers. Lastly, I have been the main author of the [Polkadot staking newsletter](https://gist.github.com/kianenigma/aa835946455b9a3f167821b9d05ba376), which is my main attempt at making the entire complexity and development of this part of the protocol transparent to the end-users. I expect myself to contribute *directly* to the staking system for at least another ~12, if not more, and afterwards having the role of an advisor. Some notable contributions: - paritytech/substrate#4517 - paritytech/substrate#7910 - paritytech/substrate#6242 - paritytech/substrate#9415 - paritytech/polkadot#3141 - paritytech/substrate#11212 - paritytech/substrate#12129 ### FRAME Historically, I have contributed a variety of domains in FRAME, namely: - Early version of the weight system paritytech/substrate#3816 paritytech/substrate#3157 - Early version of the transaction fee system - Primitive arithmetic types paritytech/substrate#3456 - Council election pallet paritytech/substrate#3364 Many of which were, admittedly, a PoC at most, if not considered "poor". I am happy that nowadays many of the above have been refactored and are being maintained by new domain experts. These days, I put most of my FRAME focus on testing and quality assurance. Through my work in the staking system, I have had to deal with the high sensitivity and liveness requirement of protocol development first hand (I believe I had to do among the [very first storage migrations](paritytech/substrate#3948) in Kusama) and consequently I felt the need to make better testing facilities, all of which have been formulated in https://forum.polkadot.network/t/testing-complex-frame-pallets-discussion-tools/356. Some relevant PRs: - paritytech/substrate#8038 - paritytech/substrate#9788 - paritytech/substrate#10174 Regardless of wearing the staking hat, I plan to remain a direct contributor to FRAME, namely because I consider it to be an important requirements of successfully delivering more features to Polkadot's ecosystem. ### Polkadot-Native Side Projects I have started multiple small, mostly non-RUST projects in the polkadot ecosystem that I am very happy about, and I plan to continue doing so. I have not yet found the time to make a "polished product" out of any of these, but I hope that I can help foster our community such that someday a team will do so. I consider my role, for the time being, to *put ideas out there* through these side projects. - https://github.com/substrate-portfolio/polkadot-portfolio/ - https://github.com/kianenigma/polkadot-basic-notification/ - https://github.com/paritytech/polkadot-scripts/ - https://github.com/paritytech/substrate-debug-kit/ ### Education Lastly, aside from having had a number of educational talks over the years (all of which [are listed](https://hello.kianenigma.nl/talks/) in my personal website), I am a big enthusiast of the newly formed Polkadot Blockchain Academy. I have [been an instructor](https://singular.app/collectibles/statemine/16/2) in the first cohort, and continue to contribute for as long and as much as I can, whilst still attending to the former 3 duties. --- With all of that being said and done, I consider myself at the beginning of the path to Dan 4, but happy to start at a lower one as well.

@kianenigma

# Membership Request Hi, I am Kian Paimani, known as @kianenigma. I have been working on Polkadot/Kusama through Parity since February 2019 and I can categorize my main contributions to Polkadot's ecosystem as follows: 1. Maintaining and developing the staking sub-system. 2. General FRAME development, especially testing and quality assurance. 3. Polkadot-native side-projects. 4. Education > My first contribution to Polkadot is also indeed related to staking: paritytech/substrate#1915 ### Staking system I joke as the Polkadot staking to be both my blessing and my curse over the years. I started working on it since the first days that I joined this ecosystem and the work [is ongoing ever since](https://github.com/orgs/paritytech/projects/33/views/9). In the past, I focused on making sure that the staking system is secure and to some extent scalable. More recently, I coordinated the (imminent) launch of Nomination Pools. Nowadays I also put an extra effort on making sure that this sub-system of Polkadot is *sustainable*, through code refactor and educating other core developers. Lastly, I have been the main author of the [Polkadot staking newsletter](https://gist.github.com/kianenigma/aa835946455b9a3f167821b9d05ba376), which is my main attempt at making the entire complexity and development of this part of the protocol transparent to the end-users. I expect myself to contribute *directly* to the staking system for at least another ~12, if not more, and afterwards having the role of an advisor. Some notable contributions: - paritytech/substrate#4517 - paritytech/substrate#7910 - paritytech/substrate#6242 - paritytech/substrate#9415 - paritytech/polkadot#3141 - paritytech/substrate#11212 - paritytech/substrate#12129 ### FRAME Historically, I have contributed a variety of domains in FRAME, namely: - Early version of the weight system paritytech/substrate#3816 paritytech/substrate#3157 - Early version of the transaction fee system - Primitive arithmetic types paritytech/substrate#3456 - Council election pallet paritytech/substrate#3364 Many of which were, admittedly, a PoC at most, if not considered "poor". I am happy that nowadays many of the above have been refactored and are being maintained by new domain experts. These days, I put most of my FRAME focus on testing and quality assurance. Through my work in the staking system, I have had to deal with the high sensitivity and liveness requirement of protocol development first hand (I believe I had to do among the [very first storage migrations](paritytech/substrate#3948) in Kusama) and consequently I felt the need to make better testing facilities, all of which have been formulated in https://forum.polkadot.network/t/testing-complex-frame-pallets-discussion-tools/356. Some relevant PRs: - paritytech/substrate#8038 - paritytech/substrate#9788 - paritytech/substrate#10174 Regardless of wearing the staking hat, I plan to remain a direct contributor to FRAME, namely because I consider it to be an important requirements of successfully delivering more features to Polkadot's ecosystem. ### Polkadot-Native Side Projects I have started multiple small, mostly non-RUST projects in the polkadot ecosystem that I am very happy about, and I plan to continue doing so. I have not yet found the time to make a "polished product" out of any of these, but I hope that I can help foster our community such that someday a team will do so. I consider my role, for the time being, to *put ideas out there* through these side projects. - https://github.com/substrate-portfolio/polkadot-portfolio/ - https://github.com/kianenigma/polkadot-basic-notification/ - https://github.com/paritytech/polkadot-scripts/ - https://github.com/paritytech/substrate-debug-kit/ ### Education Lastly, aside from having had a number of educational talks over the years (all of which [are listed](https://hello.kianenigma.nl/talks/) in my personal website), I am a big enthusiast of the newly formed Polkadot Blockchain Academy. I have [been an instructor](https://singular.app/collectibles/statemine/16/2) in the first cohort, and continue to contribute for as long and as much as I can, whilst still attending to the former 3 duties. --- With all of that being said and done, I consider myself at the beginning of the path to Dan 4, but happy to start at a lower one as well. Co-authored-by: Bastian Köcher <git@kchr.de>

A clean new attempt

4c0abc5

github-actions bot added the A3-in_progress Pull request is in progress. No review needed at this stage. label Feb 3, 2021

kianenigma changed the title ~~A clean new attempt~~ Migration testing runtime API/Bot Feb 3, 2021

kianenigma commented Feb 3, 2021

View reviewed changes

client/state-db/src/lib.rs Outdated Show resolved Hide resolved

kianenigma added 3 commits February 10, 2021 09:39

Checkpoint to move remote.

7f9b5b8

A lot of dependency wiring to make it feature gated.

4883812

bad macro, bad macro.

be549b4

kianenigma marked this pull request as ready for review February 11, 2021 10:57

Master.into()

8e97733

kianenigma requested review from bkchr, seunlanlege, apopiak and arkpar February 11, 2021 11:45

kianenigma commented Feb 11, 2021

View reviewed changes

Undo the DB mess.

d84dad4

apopiak reviewed Feb 11, 2021

View reviewed changes

Update frame/support/src/traits.rs

aeb7a0e

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

kianenigma commented Feb 11, 2021

View reviewed changes

utils/frame/try-runtime/cli/src/lib.rs Outdated Show resolved Hide resolved

Apply suggestions from code review

d968f58

Co-authored-by: Alexander Popiak <alexander.popiak@parity.io>

kianenigma commented Feb 13, 2021

View reviewed changes

kianenigma added 2 commits February 13, 2021 19:58

unbreak the build

ce4128b

Merge branch 'kiz-finally-finally-finally-finally-migration-testing-2…

ee8ae08

…' of github.com:paritytech/substrate into kiz-finally-finally-finally-finally-migration-testing-2

Remove allowing one pallet at a time.

c8ba546

kianenigma requested review from shawntabrizi, bkchr and gui1117 February 18, 2021 12:39

gui1117 reviewed Feb 18, 2021

View reviewed changes

kianenigma added 2 commits February 18, 2021 17:59

More grumbles.

2f9ad0e

Merge branch 'master' of github.com:paritytech/substrate into kiz-fin…

6db195c

…ally-finally-finally-finally-migration-testing-2

shawntabrizi approved these changes Feb 18, 2021

View reviewed changes

cecton suggested changes Feb 19, 2021

View reviewed changes

utils/frame/try-runtime/remote-externalities/src/lib.rs Outdated Show resolved Hide resolved

bin/node/cli/src/command.rs Show resolved Hide resolved

cecton approved these changes Feb 19, 2021

View reviewed changes

kianenigma added 2 commits February 19, 2021 11:31

relocate remote-ext

b8ab620

Merge branch 'master' of github.com:paritytech/substrate into kiz-fin…

be5210d

…ally-finally-finally-finally-migration-testing-2

kianenigma added 2 commits February 19, 2021 13:08

Fix build

b5e394b

Merge branch 'master' of github.com:paritytech/substrate into kiz-fin…

deb03d4

…ally-finally-finally-finally-migration-testing-2

ghost merged commit 394c52e into master Feb 19, 2021

ghost deleted the kiz-finally-finally-finally-finally-migration-testing-2 branch February 19, 2021 14:52

bkchr reviewed Feb 19, 2021

View reviewed changes

niklasad1 mentioned this pull request Mar 4, 2021

refactor(remote ext): use jsonrpsee #8105

Merged

hackfisher mentioned this pull request Mar 15, 2021

Storage Name Checking Script darwinia-network/darwinia-common#314

Closed

xlc mentioned this pull request Apr 27, 2021

Support try-runtime AcalaNetwork/Acala#957

Closed

kianenigma mentioned this pull request Sep 27, 2022

Member Request polkadot-fellows/seeding#14

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration testing runtime API/Bot #8038

Migration testing runtime API/Bot #8038

kianenigma commented Feb 3, 2021 •

edited

Loading

kianenigma Feb 3, 2021

kianenigma commented Feb 11, 2021 •

edited

Loading

kianenigma Feb 11, 2021

kianenigma Feb 11, 2021

kianenigma left a comment

arkpar commented Feb 15, 2021

gui1117 left a comment

gui1117 Feb 18, 2021

kianenigma Feb 18, 2021 •

edited

Loading

gui1117 Feb 18, 2021 •

edited

Loading

kianenigma Feb 18, 2021

cecton Feb 19, 2021

kianenigma commented Feb 18, 2021

shawntabrizi left a comment

kianenigma commented Feb 19, 2021

ghost commented Feb 19, 2021

ghost commented Feb 19, 2021

kianenigma commented Feb 19, 2021

ghost commented Feb 19, 2021

bkchr left a comment

bkchr Feb 19, 2021

bkchr Feb 19, 2021

bkchr Feb 19, 2021

bkchr Feb 19, 2021

kianenigma commented Feb 19, 2021

Migration testing runtime API/Bot #8038

Migration testing runtime API/Bot #8038

Conversation

kianenigma commented Feb 3, 2021 • edited Loading

Choose a reason for hiding this comment

kianenigma commented Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma left a comment

Choose a reason for hiding this comment

arkpar commented Feb 15, 2021

gui1117 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma Feb 18, 2021 • edited Loading

Choose a reason for hiding this comment

gui1117 Feb 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma commented Feb 18, 2021

shawntabrizi left a comment

Choose a reason for hiding this comment

kianenigma commented Feb 19, 2021

ghost commented Feb 19, 2021

ghost commented Feb 19, 2021

kianenigma commented Feb 19, 2021

ghost commented Feb 19, 2021

bkchr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma commented Feb 19, 2021

kianenigma commented Feb 3, 2021 •

edited

Loading

kianenigma commented Feb 11, 2021 •

edited

Loading

kianenigma Feb 18, 2021 •

edited

Loading

gui1117 Feb 18, 2021 •

edited

Loading