Refactored block body database scheme #10779

arkpar · 2022-02-02T14:44:42Z

This PR changes the way block bodies are stored in the database. It is backward compatible with the current plain body storage, but breaks compatibility with existing storage chains. Since there are no actual storage chains deployed, this is non-issue.

Motivation:

Simplified user configuration. Storage mode is automatic and the --storage-chain CLI flag is removed.
Extensibility. This new format allows to implement full extrinsic index without breaking compatibility. This should make it possible to E.g. implement a RPC function to retrieve any extrinsic by hash or serve it over IPFS.
Better alternative to changes in Remark storage #10698

Summary of changes:

Currently there are two incompatible storage scheme selected with with the --storage-chain flag. When the flag is not specified, block bodies are stored as blobs in the the BODY column and indexed extrinsic are not allowed. In storage chain mode each extrinsic is split into indexed portion (which may be empty) and the header, that contains the rest of the extrinsic. A list of headers is inserted into BODY column. Indexed parts are inserted into TRANSACTION column with reference counting.

With this PR there's just single mode.
Blocks that don't contain a single indexed extrinsic are saved in the BODY column, maintaining compatible with current format.
Block that have indexed content are split. The headers are written to the BODY_INDEX column and indexed parts are inserted into TRANSACTIONS.

polkadot companion: paritytech/polkadot#4837
cumulus companion: paritytech/cumulus#949

cheme

Looks like a really good change.
IIReadC this change set, the storage chain mode was merely enforcing fast sync, and removing this explicit mode makes sense. (also having rc on the transaction column 👍 )
Got a few small question after first reading (in comments).

Cargo.lock

client/db/src/lib.rs

…x-storage

bkchr · 2022-02-17T11:08:24Z

Blocks that don't contain a single indexed extrinsic are saved in the BODY column, maintaining compatible with current format.
Block that have indexed content are split. The headers are written to the BODY_INDEX column and indexed parts are inserted into TRANSACTIONS.

Without having reviewed the pr. Why do we not just have a HEADER and a TRANSACTIONS column? To stay backwards compatible to the old db format? Or to also reduce the latency when reading a block without indexed transactions?

arkpar · 2022-02-17T11:14:55Z

@bkchr HEADER is for block headers. TRANSACTIONS is for indexed extrinsic data. The two are not related.
You probably mean why the need to have both BODY and TRANSACTIONS?
That is for backwards compatibility indeed. Data encoded in BODY, BODY_INDEX and TRANSACTIONS has each a separate corresponding rust type. Rather than mixing multiple types into a single column I think it is more clear to have them separated.

bkchr · 2022-02-17T11:22:55Z

@bkchr HEADER is for block headers.

Ahh makes sense 🤦 I was confused because of "The headers are written to the BODY_INDEX column". Do you mean something different here with "headers"?

That is for backwards compatibility indeed.

Why can we not "unify" this? Aka write a migration to move everything over to "one" format. Like 2 columns, BODY and TRANSACTIONS. While BODY only holds "references" (probably hashes) of the transactions?

arkpar · 2022-02-17T11:33:45Z

Right, headers refer to extrinsic header. Each extrinsic may have an indexed "transaction" portion. The rest is considered to be a header.

Current format makes more sense when the blockchain does not have that many indexed extrinsics. Storing each extrinsic as a separate database entry will increase space requirements for e.g. kusama by quite a bit. That's why the new encoding scheme is only used for blocks that actually have indexed content. The new encoding format actually allows for each individual extrinsic to be either embedded in the BODY_INDEX or refer to a hash in TRANSACTIONS. It still uses a bit more space than simply putting all extrinsics together in BODY.

bkchr · 2022-03-03T14:43:31Z

client/db/src/parity_db.rs

 		},
 	}

+	if upgrade {
+		log::info!("Upgrading metadata");


Do we really want this to be an info log?

If yes, we should probably make the message more expressive.

bkchr · 2022-03-03T14:45:01Z

client/db/src/parity_db.rs

@@ -38,20 +38,22 @@ pub fn open<H: Clone + AsRef<[u8]>>(
 	path: &std::path::Path,
 	db_type: DatabaseType,
 	create: bool,
+	upgrade: bool,


I would propose some simple enum:

enum Upgrade { Yes, No }

to make it a little bit more expressive.

bkchr · 2022-03-03T20:53:55Z

client/db/src/lib.rs

 			},
+			Err(err) =>
+				return Err(sp_blockchain::Error::Backend(format!(


Suggested change

return Err(sp_blockchain::Error::Backend(format!(

Err(sp_blockchain::Error::Backend(format!(

…x-storage

ordian · 2022-03-11T16:17:33Z

This bumps db version implying one-way db migration. Does that imply E2-databasemigration label?

* Refactored tx storage database scheme * Bump parity-db * fmt * Fix handling invalid index size + test * Removed superflous result * Minor changes * fmt

* Make some UI test expectations crisper (#10791) * Make some UI test expectations crisper * Update frame/support/test/tests/construct_runtime_ui/undefined_genesis_config_part.rs * Update UI test expectations * Missing newlines * More whitespace issues * contracts: `is_contract(address)` and `caller_is_origin()` are added to API (#10789) * is_contract() and caller_is_origin() added to Ext API * is_contract() exposed in wasm runtime.rs * + test for is_contract() * + seal_is_contract benchmark * caller_is_origin() exposed to wasm/runtime.rs and covered by a test * + seal_caller_is_origin benchmark * Update frame/contracts/src/exec.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Update frame/contracts/src/exec.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Update frame/contracts/src/exec.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Update frame/contracts/src/wasm/runtime.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Update frame/contracts/src/wasm/runtime.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Update frame/contracts/src/wasm/runtime.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Update frame/contracts/src/exec.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * identation fix for benchmark macroses; test cosmetic improvement * benchmark fix * + is_contract() wasm test * + caller_is_origin() wasm test * Apply suggestions from code review Co-authored-by: Alexander Theißen <alex.theissen@me.com> * is_contract() to borrow param instead of taking ownership * phrasing improved Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com> * fixed wasm tests according to @athei feedback * dead code warnings suppressed by unstable-interface attributes * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com> Co-authored-by: Parity Bot <admin@parity.io> * maybe_total_supply in pallet-assets (#10799) * maybe_total_supply in pallet-assets * Update frame/assets/src/functions.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Referenda and Conviction Voting pallets (#10195) * Initial draft of new referendum state machine. * Docs * Fixes * Fixes * Add conviction-voting pallet * Basic build * Building * Some TODOs * Tests building * Add missing file * Basic lifecycle test * Add couple of tests * Another test * More tests * Fixes * Fixes * Formatting * Fixes * Tests * Fixes * Fixes * More tests * Formatting * First few benchmarks * First few benchmarks * Defered queue servicing * More testing * Benchmarks * Fiddly benchmark * Final nudge benchmarks * Formatting * Formatting * Finished up benchmarks * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_referenda --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/referenda/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Events finished * Missing file * No GenesisConfig for Referenda * Formatting * Docs * Docs * Docs * Per-class conviction voting * New test & mock utils * More tests * Tests * Tests finished 🎉 * Benchmarking stuff * Fixes * Test harness * Test harness * Benchmarks for Conviction=Voting * Benchmarking pipeline complete * Docs * Formatting * Remove unneeded warning * Fix UI tests * cargo run --quiet --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_conviction_voting --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/conviction-voting/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Docs * Update frame/conviction-voting/src/vote.rs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * update sp-runtime version * MEL Fixes for Referenda and Conviction Voting (#10725) * free maxencodedlen * more maxencodedlen * more MEL * more mel * disable storage info * More Referenda Patches (#10760) * basic fixes * fix benchmarking * fix license * prevent panic in curve math * fmt * bump crate versions * Update mock.rs Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * More efficient WASM instance memory decommit on macos (#10801) * More efficient WASM instance memory decommit on macos * Apply suggestions from code review Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Updated error message Co-authored-by: Alexander Theißen <alex.theissen@me.com> * make inner field of IdentityFields pub (#10773) * Corrected description of clear_attribute extrinsic (#10729) * block validators after X sessions, update mock runtime for test * sp-maybe-compressed-blob: reduce boilerplate code (#10814) Signed-off-by: koushiro <koushiro.cqx@gmail.com> * `seal_delegate_call` api function (support for library contracts) (#10617) * seal_call_code implementation - tests - benchmark * Addressing @xgreenx's comments * Fix test-linux-stable-int * Rename seal_call_code to seal_delegate_call * Pass value unchanged into lib contract * Address @athei's comments - whitespace .wat issues - wrong/missing .wat comments - redundant .wat calls/declarations - change order of functions (seal_delegate_call right after seal_call) in decls, tests, benchmark - fix comments, move doc comments to enum variants - remove unnecessary empty lines - rename runtime cost DelegateCall to DelegateCallBase - do not set CallFlags::ALLOW_REENTRY for delegate_call * Do not pass CallFlags::ALLOWS_REENTRY for delegate_call * Update comment for seal_delegate_call and CallFlags * Addressing @athei's comments (minor) * Allow reentry for a new frame after delegate_call (revert) * Same seal_caller and seal_value_transferred for lib contract - test - refactor frame args due to review - logic for seal_caller (please review) * Put caller on frame for delegate_call, minor fixes * Update comment for delegate_call * Addressing @athei's comments * Update weights generated by benchmark * Improve comments * Address @HCastano's comments * Update weights, thanks @joao-paulo-parity * Improve InvalidCallFlags error comment * Add Storage Info to Various Pallets (#10810) * atomic swap * bounties * bounties fmt * gilt * indices * nicks * randomness-collective-flip * recovery * reuse maxapprovals * Update tests.rs * Update frame/randomness-collective-flip/src/lib.rs Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * use the correct bound * complete recovery * use `bounded_vec` macro * Update tests.rs * transaction payment * uniques * mmr * example offchain worker * beefy-mmr * Update frame/recovery/src/lib.rs Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use BoundedVec instead of a type-parameterized BoundedString * cargo fmt * Update frame/atomic-swap/src/lib.rs * use config const * Update lib.rs * update mel_bound * fmt Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Keith Yeung <kungfukeith11@gmail.com> * fixed regression in inline docs (#10819) * try-runtime-cli: Add to docs (#10822) * try-runtime-cli: Add to docs of pre/post hooks * Add note about --ws-max-out-buffer-capacity 1000 * Bump serde from 1.0.132 to 1.0.136 (#10816) Bumps [serde](https://github.com/serde-rs/serde) from 1.0.132 to 1.0.136. - [Release notes](https://github.com/serde-rs/serde/releases) - [Commits](https://github.com/serde-rs/serde/compare/v1.0.132...v1.0.136) --- updated-dependencies: - dependency-name: serde dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * uodate tests, ipfs tests failing * --dev implies --tmp (#10828) * Add a new host function for reporting fatal errors; make WASM backtraces readable when printing out errors (#10741) * Add a new host function for reporting fatal errors * Fix one of the wasmtime executor tests * Have `#[runtime_interface(wasm_only)]` actually mean WASM-only, and not no_std-only * Print out errors through `Display` instead of `Debug` * Switch one more trait to require `Error` for its error instead of only `Debug` * Align to review comments * specify ipfs version, update tests * Add db params for export-state CLI command (#10830) * update tests, cleanup * Allow `SetBalance` to handle error when trying to kill acount with reference counter. (#10826) * bug found * fix logic * a little simpler * add test * UncheckedExtrinsic: Harden decode and clarify `EXTRINSIC_FORMAT_VERSION` (#10829) * UncheckedExtrinsic: Harden decode and clarify `EXTRINSIC_FORMAT_VERSION` * Apply suggestions from code review * fix typo * pallet-staking: Add extrinsic `force_apply_min_commission` (#10786) * pallet-staking: Add extrinsic `force_apply_min_commission` * Add benchmarks * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Bound iteration by max_validator_count * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Only apply to 1 validator * Update doc comments * Uncomment tests * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Accept signed origins * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Remove contains_key check * Add test for try_mutate_exists * Impove try_mutate_exists docs * Delete redundant try_mutate_exists tests; * Delete residual from removed test * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Return an error when the stash does not exist * Update try_mutate_exist doc wording * Update frame/staking/src/pallet/mod.rs * Apply suggestions from code review Co-authored-by: Parity Bot <admin@parity.io> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Upgradable contracts using `set_code` function (#10690) * poc logic * set_code_hash impl, tests, benchmark * Address @xgreenx's comments * Move func defs closer to set_storage * Check if code exists - increment/decrement codehash refcount * Document error for non-existing code hash * Revert unrelated change * Changes due to @athei's review * Fix error handling - comment errors: ReturnCodes - update mock ext implementation - return Error::CodeNotFound when no code for such hash * Emit ContractCodeUpdated when setting new code_hash * Address @athei's comments * Move related defs to the bottom * Minor comment update Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Improve docs * Improve docs * Update frame/contracts/src/wasm/runtime.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Refactor set_code_hash test * Minor change to benchmark Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Minor change to benchmark Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Minor comment refactor Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Address @HCastano's comments * Update seal_set_code_hash comment Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com> * Move set_code_hash after delegate_call * Move function to the bottom * Moved and changed banchmark, added verify block * Bring back previous benchmark * Remove skip_meta for seal_set_code_hash * Bring back skip_meta for seal_set_storage_per_new_kb * Apply weights Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Hernando Castano <HCastano@users.noreply.github.com> * Separate wasmi and wasmer sandbox implementations into their own modules (#10563) * Moves wasmi specific `ImportResolver` and `MemoryTransfer` impls to submodule * Splits context store environmental, moves impl `Externals` to wasmi backend * Adds wasmer sandbox backend stub module * Move sandbox impl code to backend specific modules * Moves wasmi stuff * Fixes value conversion * Makes it all compile * Remove `with_context_store` * Moves `WasmerBackend` to the impl * Reformat the source * Moves wasmer MemoryWrapper * Reformats the source * Fixes mutability * Moves backend impls to a submodule * Fix visibility * Reformat the source * Feature gate wasmer backend module * Moves wasmi memory allocation to backend module * Rename WasmerBackend to Backend * Refactor dispatch result decoding, get rid of Wasmi types in common sandbox code * Reformat the source * Remove redundant prefixes in backend functions * Remove wasmer-sandbox from default features * Post-review changes * Add conversion soundness proof * Remove redundant prefix * Removes now redundant clone_inner * Add `Error::SandboxBackend`, refactor invoke result * Fix comments * Rename `Error::SandboxBackend` to `Sandbox` * Simplifies logic in `wasmer_backend::invoke` * Fixes memory management * Show Network ID when creating and inspecting (#10838) Signed-off-by: Antonio Yang <yanganto@gmail.com> * Reduce overhead of generating network event metrics (#10839) * Fix reentrancy of FrozenBalance::died hook (#10473) * assets: execute `died` hook outside of mutate Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo> * assets: extend tests for `died` hook Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo> * assets: update doc of FrozenBalance::died Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo> * assets: review fixes - fix cases where `died` should not have been called - use `Option<DeadConsequence>` instead of `DeadConsequence` Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo> * assets: update comment in mock.rs Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo> * assets: return `Remove` in dead_account The return value is ignored in the only case that it is produced by a call, but having it this way makes it more understandable. Signed-off-by: Oliver Tale-Yazdi <oliver@tasty.limo> * allow trailing comma (#10841) * build-script-utils: allow reading the git commit hash from env var (#10845) * build-script-utils: allow reading the git commit hash from env var * build-script-utils: make the env var name substrate specific * Don't create DB leaves when syncing historic blocks (#10844) * Don't create DB leaves when syncing historic blocks * Changed leaves check and added test * fmt * cleanup, use unsigned tx when appropriate * add weights * fix broken link (#10846) * subkey: Support `--version` cli command (#10853) * subkey: Support `--version` cli command * FMT :facepalm: * Remove `u32_trait` (#10850) * Remove `u32_trait` This trait only existed because there wasn't any const generic support at time of creation. However, we now have support for it :) * FMT * wasm-builder: Support latest nightly (#10837) * wasm-builder: Support latest nightly With latest nightly, aka rust version 1.60+ namespaced features are added. This changes the handling of optional dependencies. We currently have features that enable optional dependencies when `std` is enabled. This was before no problem, but now the wasm-builder detects them as enabled. To support the transition period until 1.60 is released as stable, this pr adds an heuristic to not enable these optional crates in the wasm build when they are enabled in the `std` feature. This heuristic fails when someones enables these optional dependencies from the outside as well as via the `std` feature, however we hope that no one is doing this at the moment. When namespaced features are enabled, these dependencies needs to be enabled using `dep:dependency-name` to solve this properly. https://doc.rust-lang.org/cargo/reference/unstable.html#namespaced-features * Remove accidentally added features * update tests, remove unneeded params * min authorities check before removal * contracts: Fix `seal_call` weights (#10796) * Fix call weights * Fix instantiate benchmark * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Remove stale and superflous comments * `decrement_refcount` should be infallible * Don't hardcode increment_refcount, decrement_refcount * Rename CopyIn/CopyOut * Fix warning in tests Co-authored-by: Parity Bot <admin@parity.io> * Enable the `parking_lot` feature for `tokio` (#10855) * [ci] fix publish-rustdocs (#10858) * Fix beefy mock ecdsa keys (#10854) Compressed ECDSA keys requires to have 0x02 or 0x03 as their first byte in order to allow public key recovery. Nevertheless the test was working because of the `unwrap_or_default()` at the end of the conversion routine (i.e. the invalid keys were converted to an empty vector). * Mark sync_state_genSyncSpec JSON-RPC as safe (#10832) * Mark sync_state_genSyncSpec JSON-RPC as safe * Note that parameter is unused * Ideally I'd wait for compilation to be finished before pushing, but it's really taking a long time * Remove deny_unsafe parameter * Remove unused dependency * Reduce CPU overhead of gossip (#10859) * sp-core-hashing: use the `digest::Digest` trait to handle the hashing function uniformly (#10835) * sp-core-hashing: use sha3 instead of tiny-keccak Signed-off-by: koushiro <koushiro.cqx@gmail.com> * use blake2 instead of blake2-rfc Signed-off-by: koushiro <koushiro.cqx@gmail.com> * improve som hashing of sp-core and sp-api-proc-macro Signed-off-by: koushiro <koushiro.cqx@gmail.com> * Some nits Signed-off-by: koushiro <koushiro.cqx@gmail.com> * cargo fmt Signed-off-by: koushiro <koushiro.cqx@gmail.com> * Use fully qualified sytnax for `retain_mut` (#10865) * add llvm (#10864) * refactor election score (#10834) * refactor election score * Test for ord * remove reference * vec -> slice * change iter to iter_by_significance * improve doc * fix typo * add explanation about [u128; 3] * consolidate threshold and epsilon * random fixes * rename * remove Into * make iter_by_sig private * remove vec * Fix tests * Measure per byte and not kb for certain benchmarks (#10863) * Tidy Democracy (#10867) * add test * Assorted refactorings * complete test * saturating math * final check * use `default` Co-authored-by: Gav Wood <gavin@parity.io> * cleanup storage maps, cleanup unused imports * sc-executor: Improve logging (#10869) Improves the logging by switching to `tracing` for a better log output. Besides that, it also adds a trace for the function being executed. * fix test issues * Track allowed requests for state/warp sync (#10843) * Track allowed requests for state/warp sync * Added missing allowed_requests resets * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * fmt Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Slots: Ensure that a race betwen finalized and best number is taken care off (#10876) * Remove old lock file (#10879) * Further reduce the CPU overhead of networking metrics (#10875) * Simplify `num_connected_peers` * Track requested peer counts * Revert "Track requested peer counts" This reverts commit 9f1c8704353df6afc17ed7e9f4ab8d8e29466ae4. * Remove `substrate_sub_libp2p_peerset_num_requested` metric * Remove two unused functions that I forgot to get rid of in previous commit * Introduce `BoundedVec::iter_mut` (#10884) * Introduce iters into BoundedVec * Fix * Remove unneeded funcs * Update frame/support/src/storage/bounded_vec.rs * Update frame/support/src/storage/bounded_vec.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Slots: Log total proposing duration as milliseconds (#10886) Parachains have currently a total proposing time of 500ms, so it this currently always prints `0`. While actually the value is not `0` ;) * staking: Clarify reward calc docs (#10890) * tracing: Adds `init_for_tests` (#10893) This function is useful for tests. It will enable `TRACE` logging and also uses the libtest aware writer. * make submissions pub (#10899) * contracts: Allow stack height metering to be disabled (#10877) * Allow stack height metering to be disabled * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_contracts --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/contracts/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: Parity Bot <admin@parity.io> * Rename Uniques Error::Unknown to something more sensible (#10895) * Rename Uniques Error::Unknown to something more sensible * Typos * Typos * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix tests Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * state-machine: Move all functionality from trie backend to the essence (#10904) * state-machine: Move all functionality from trie backend to the essence This is required for some future changes of me and it also makes more sense to have all the functionality inside the essence. Besides that it changes the child root cache to directly cache the hash. * Update primitives/state-machine/src/trie_backend_essence.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * FMT Co-authored-by: cheme <emericchevalier.pro@gmail.com> * staking: Remove `SessionInterface` supertrait (#10901) * consensus-slots: cleanup SlotDuration config (#10878) * consensus-slots: cleanup the SlotDuration config * fix tests * address review comments * add contracts pallet + rpc to runtime * Replace libsecp256k1 with k256 in FRAME related code (#10883) * Replace libsecp256k1 with k256 in beefy-mmr * Port of FRAME `contracts` benchmarking from `libsecp256k1` to `k256` * Newtype to allow `Pcg32` rng usage with `k256` in contracts benchmarks * Use `sp-io::crypto` to generate dummy keys in `contracts` bechmarks * More compact code * Cargo fmt * Build `sp-keystore` only for dev profile * Move public key generation back to the `map` * Clean obsolete BABE's weight data (#10748) * Clean obsolete BABE weight data * Take out test assertion from check closure * Optimize metadata access using `HeaderMetadata` trait * Apply suggestions from code review * Introduce finalize and import pre-commit synchronous actions * Do not hold locks between internal methods calls * Remove unused generic bound * Apply suggestions from code review * Register BABE's pre-commit actions on `block_import` instead of `start_babe` * PreCommit actions should be `Fn` instead of `FnMut` * More robust safenet in case of malformed finality notifications Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Factor DB weights out into their own files (#10908) * Factor DB weights out into their own files Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix CI Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Weights in own mod Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Get rid of unnecessary use of `async-std` in non-test code (#10891) * No longer generate specs with consensus_engine field (#10346) * No longer generate specs with consensus_engine field * #[allow(unused)] * Upgrading parity-scale-codec to v3 (#10825) * Upgraded dependencies * Adapting code to scale v3 * Empty commit to trigger CI * Triggering CI * Fixing UI test * Remove superfluous dev-dep added by #9228 * Cryout for CI * sc-cli: Fix bugs after switching to clap3 (#10920) * sc-cli: Fix bugs after switching to clap3 Before switching to clap3 we support cli options like `--reserved-nodes A B` and after you needed to pass `--reserved-nodes` cli option multiple times `--reserved-nodes A --reserved-nodes B`. This is fixed by setting `multiple_occurrences(true)` option. This also done for all the other `Vec` cli options in `sc-cli`. Besides that `--sync` wasn't supporting case insensitive parsing of the value. This is now also supported. For both regressions a test is added. Besides that the pr removes all the `rename_all = PascalCase` attributes, because they are not needed. All other `ArgEnum`s were checked and all are already using `ignore_case(true)`. * Bring back `PascalCase`, because otherwise it falls back to `kebab-case`... * Storage benchmarking (#10897) * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP: DB benchmarking Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * WIP Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Simplify code Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove old files Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove old files Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Minimize changes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add license Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove dependencies Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Extend template Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Linter Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Linter Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Beauty fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove default Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add feature Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove seed Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * CI wakeup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Review fixes Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Adding doc Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Adding doc Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Improve template Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Do not expose columns Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix ColumnId Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Nicer template prints Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Cleanup Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fix json path Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Simplify `bench_write` logic Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Invert tx before the second commit Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * add transfer assets extrinsic * add chain ext * Minor improvements to `bounded_vec` and `defensive`. (#10873) * Fix a few things in bounded_vec * add test for try_extend * Update frame/support/src/storage/bounded_vec.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * some review comments * use swap * remove clone * use pop instead of truncate * remove warn * review comments * Update frame/support/src/storage/bounded_vec.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * fix rustdoc * fix links * undo link Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Replace libsecp256k1 with secp256k1 (#10798) * Replace libsecp256k1 with secp256k1 * Wipe ecdsa secret key from memory on drop * Some comments for a known issue * Safer core crypto primitives `from_slice` constructor Previous version panics if slice lenght is not the expected one. * Unit test fix * Enable use of global secp256k1 context * Better comments for ecdsa `Pair` drop * Replace `libsecp256k1` with `seco256k1` in `beefy-mmr` Used to convert ecdsa public key to ETH address * Replace `libsecp256k1` with `secp256k1` in FRAME `contracts`benchmarks * Temporary rollback of `beefy-mmr` to libsecp256k1 Check for detected build issues * Cargo fmt * Rollback of FRAME `contracts` benchmarks to `libsecp256k1` * Rollback for unrelated changes * Typo fix * Add comments for deprecated `ecdsa_verify` and `secp256k1_ecdsa_recover` * Integrate try-runtime into substrate node template (#10909) * [10892-integrate-try-runtime-into-node-template] - Integrated try-runtime into node template * [10892-integrate-try-runtime-into-node-template] Added match arms for try-runtime in command.rs * [10892-integrate-try-runtime-into-node-template] Added match arms for try-runtime in command.rs * Added feature flag for try-runtime in node-template/node and enabled try-runtime for node-template/runtime * Added missing type annotations for try-runtime SubCommand in node-template * Added missing type annotations for try-runtime SubCommand in node-template * Implemented frame_try_runtime::TryRuntime<Block> for the node-template Runtime * doc corrections (#10936) * #10576: generic utility to unsubscribe from broadcast upon drop of the rx-side. (#10708) * #10576: refactor `sc-utils::notification` and `sc-client-api::notifications`, so that they use common subscribe/unsubscribe routines * Add some docs. Reorganise `sc-utils::notification` * `sc-clent-api::notifications` and `sc-utils::notification` — ensure the SubscriptionGuard is dropped before the Rx-channel * `sc-utils::pubsub::SubscriptionGuard` make it a bit more ergonomic. Let the `Rx` to be put inside of the `SubscriptionGuard`, so that the latter shall guarantee the order: - first unsubscribe; - then drop the `Rx`. * Being less zealous with splitting the modules into little pieces * rework pubsub: the concrete usage should only define a good registry type * sc-client-api::notifications: make it comply with the reworked pubsub * cargo fmt * make sc-client-api tests work * Address the review notes * cargo fmt * Describe the behaviour of pubsub registry * Doc-comments for module `sc-utils::pubsub` * Fix: it used to send notifications regardless of the filter setup during subscription * `sc-client-api::StorageNotifications` the API does not have to require mut-self-reference. As a result `sc-service::Client` does not have to wrap its `storage_notifications` into a Mutex. * cargo fmt * Several changes addressing the notes by @bckhr. - Remove the `impl Default for StorageNotifications<Block>`; - no need for groupping the `remove_from` and `listen_from` into a separate `helpers` module; - remove unnecessary import `use registry::SubscribeOp`. * Add a doc-comment to the `sc-client::notifications::SubscribeOp` * As per @bkchr note on the unproven assertion: behave gracefully upon receiving a duplicate subscription-ID. * sc-utils::pubsub: log when a registry yields an ID that does point to an existing sink * `sc-utils::notifications`: payload materialized lazily * Update Cargo.lock (after adding `log` as a dependency to the `sc-utils`) * `sc-client-api::notifications`: introduce a struct (instead of a type def) for the notification message * Get rid of `sc-utils::pubsub::Channel` trait (instead just use the `sc-utils::mpsc`) * The SubsID is no more generic: the fact it is a `Copy` is known — no need to pass it by ref * sc-utils::pubsub internals do not have to be generic over the channel type * Rename Hub::dispatch into Hub::send * That method was unnecessary (`SubscriberSink::render_notification`) * cargo fmt * No need for a separate UnsubscribeGuard type * Ditch the type-def of SubsID in the sc-utils::pubsub, instead — just use the crate::id_sequence::SeqID * Return the <Registry as Dispatch>::Ret when sending an item * Make the `Hub<M, R>::lock_registry(...)` method more ergonomic * cargo doc links * cargo doc links * Use a simpler name for the type * cargo doc links * Derive `Default` rather than implement it * Derive `Default` rather than implement it * Remove an unnecessary usage of type_name * Define a more cautious order between sinks.remove->registry.unsubscribe and registry.subscribe->sinks.insert * Hub: lock_registry_for_tests->map_registry_for_tests — a safer choice for a public API * Replace Mutex over the shared Registry with a ReentrableMutex+RefCell * sc-utils::pubsub: add tests for a panicking registry * Add the missing copyright headers * Arc<Vec<_>> -> Arc<[_]> * Bring back MaxNominations as a metadata constant (#10947) * chain ext: parse params and call transfer func * Bump futures from 0.3.16 to 0.3.19 (#10930) Bumps [futures](https://github.com/rust-lang/futures-rs) from 0.3.16 to 0.3.19. - [Release notes](https://github.com/rust-lang/futures-rs/releases) - [Changelog](https://github.com/rust-lang/futures-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/rust-lang/futures-rs/compare/0.3.16...0.3.19) --- updated-dependencies: - dependency-name: futures dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Clean up extra_constant renaming. (#10935) Co-authored-by: Xiankun Cheng <xiankuncheng@Xiankuns-MacBook-Pro-2.local> * Bump smallvec from 1.7.0 to 1.8.0 (#10949) Bumps [smallvec](https://github.com/servo/rust-smallvec) from 1.7.0 to 1.8.0. - [Release notes](https://github.com/servo/rust-smallvec/releases) - [Commits](https://github.com/servo/rust-smallvec/compare/v1.7.0...v1.8.0) --- updated-dependencies: - dependency-name: smallvec dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * new pallet: whitelist pallet (#10159) * pallet whitelist * refactor a bit * fmt * address audit * improve tests * return Ok + refund * add test for dispatching failing * add dispatch_whitelisted_call_with_preimage * fmt * better name * Consume all data on decode Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Add error docs Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Remove phantom data Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Use rust 2021 Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Update crate features Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make compile Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_whitelist --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/whitelist/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Bump Preimage max size Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_whitelist --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/whitelist/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Parity Bot <admin@parity.io> * Release primitive crates (#10937) * cargo set-version --bump major -p pallet-contracts-primitives * cargo set-version --bump major -p sp-core * cargo set-version --bump major -p sp-runtime-interface * cargo set-version --bump major -p sp-wasm-interface * cargo set-version --bump major -p sp-runtime * cargo set-version --bump major -p sp-storage * cargo set-version --bump major -p sp-rpc * cargo set-version --bump major -p sp-io * cargo set-version --bump major -p sp-trie * cargo set-version -p sp-state-machine -- 0.12.0 * cargo set-version -p sp-externalities -- 0.12.0 * cargo set-version -p sp-keystore -- 0.12.0 * cargo set-version --bump major -p sp-keyring * cargo set-version --bump major -p sp-version * cargo set-version --bump major -p sp-tracing * cargo set-version --bump major -p sp-application-crypto * cargo set-version --bump major -p sp-arithmetic * cargo unleash version bump-major -p sp-runtime-interface-proc-macro * Add codec max-encoded-len feature to sp-arithmetic * cargo unleash version bump-major -p sp-core-hashing-proc-macro * Update new whitelist pallet to new primitive versions (#10953) * sp-trie: Switch to thiserror and some other small cleanups (#10954) * sp-trie: Switch to thiserror and some other small cleanups * Add some extra method for converting a compact proof to a memory db * Only maintain at most 1 `UnlockChunk` per era (#10670) * Only maintain at most 1 `UnlockChunk` per era * Bound `unlocking` * Run cargo +nightly-2021-10-29 fmt * Make benchmarks stuff compile * Update frame/staking/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Remove DerefMut; Implement neccesary methods directly * Doc comments for new BoundedVec methods * Fix benchmarks * wip bonded_vec macro * Correct rust doc * Apply suggestions from code review Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update staking::Config impls * Add MaxUnlockingChunks to more places * Use defensive saturating add * FMT Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * staking: Expose `MaxUnlockingChunks` in metadata as a const (#10958) * Allow trailing commas for `bounded_vec!` (#10959) * SimpleSlotWorker: Do not implement `SlotWorker` for all types implementing `SimpleSlotWorker` (#10934) Because Rust currently doesn't support specialization, it prevents users from implementing `SlotWorker` for their own types. This pr solves this by removing the generic implementation of `SlotWorker` for `SimpleSlotWorker` and providing some wrapper type for that. * Bump digest from 0.10.2 to 0.10.3 (#10960) Bumps [digest](https://github.com/RustCrypto/traits) from 0.10.2 to 0.10.3. - [Release notes](https://github.com/RustCrypto/traits/releases) - [Commits](https://github.com/RustCrypto/traits/compare/digest-v0.10.2...digest-v0.10.3) --- updated-dependencies: - dependency-name: digest dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump scale-info from 2.0.0 to 2.0.1 (#10965) Bumps [scale-info](https://github.com/paritytech/scale-info) from 2.0.0 to 2.0.1. - [Release notes](https://github.com/paritytech/scale-info/releases) - [Changelog](https://github.com/paritytech/scale-info/blob/master/CHANGELOG.md) - [Commits](https://github.com/paritytech/scale-info/commits) --- updated-dependencies: - dependency-name: scale-info dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add pr-custom-review config (#10968) * Revise how staking configurations are set (#10955) * Revise how staking configurations are set fixes #10938 * Fix and add additional tests * Format * Formatting * Add doc Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/staking/src/tests.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Format * Fix build * Update weights.rs * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_staking --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/staking/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Parity Bot <admin@parity.io> * Bump libc from 0.2.112 to 0.2.119 (#10967) Bumps [libc](https://github.com/rust-lang/libc) from 0.2.112 to 0.2.119. - [Release notes](https://github.com/rust-lang/libc/releases) - [Commits](https://github.com/rust-lang/libc/compare/0.2.112...0.2.119) --- updated-dependencies: - dependency-name: libc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump parity-db from 0.3.5 to 0.3.8 (#10970) Bumps [parity-db](https://github.com/paritytech/parity-db) from 0.3.5 to 0.3.8. - [Release notes](https://github.com/paritytech/parity-db/releases) - [Commits](https://github.com/paritytech/parity-db/commits/v0.3.8) --- updated-dependencies: - dependency-name: parity-db dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [EPM pallet]: remove `number of signed submissions` (#10945) * [EPM pallet]: remove `number of signed submissions` Closing #9229 * fix tests * remove needless assert * Update frame/election-provider-multi-phase/src/lib.rs * cargo fmt Signed-off-by: Niklas <niklasadolfsson1@gmail.com> * fix grumbles * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_election_provider_multi_phase --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/election-provider-multi-phase/src/weights.rs --template=./.maintain/frame-weight-template.hbs * drop `num_signed_submissions` in WeightInfo too * fix build Co-authored-by: Parity Bot <admin@parity.io> * Refactored block body database scheme (#10779) * Refactored tx storage database scheme * Bump parity-db * fmt * Fix handling invalid index size + test * Removed superflous result * Minor changes * fmt * Deprecate "paritydb-experimental" CLI in favour or "paritydb" (#10975) * Deprecate paritydb-experimental * Updated comment * [contracts] Stabilize "seal0" `is_contract` and `caller_is_origin` (#10971) * stabilize `seal_is_contract` * stabilize `seal_caller_is_origin` * Trie version migration pallet (#10073) * starting * Updated from other branch. * setting flag * flag in storage struct * fix flagging to access and insert. * added todo to fix * also missing serialize meta to storage proof * extract meta. * Isolate old trie layout. * failing test that requires storing in meta when old hash scheme is used. * old hash compatibility * Db migrate. * runing tests with both states when interesting. * fix chain spec test with serde default. * export state (missing trie function). * Pending using new branch, lacking genericity on layout resolution. * extract and set global meta * Update to branch 4 * fix iterator with root flag (no longer insert node). * fix trie root hashing of root * complete basic backend. * Remove old_hash meta from proof that do not use inner_hashing. * fix trie test for empty (force layout on empty deltas). * Root update fix. * debug on meta * Use trie key iteration that do not include value in proofs. * switch default test ext to use inner hash. * small integration test, and fix tx cache mgmt in ext. test failing * Proof scenario at state-machine level. * trace for db upgrade * try different param * act more like iter_from. * Bigger batches. * Update trie dependency. * drafting codec changes and refact * before removing unused branch no value alt hashing. more work todo rename all flag var to alt_hash, and remove extrinsic replace by storage query at every storage_root call. * alt hashing only for branch with value. * fix trie tests * Hash of value include the encoded size. * removing fields(broken) * fix trie_stream to also include value length in inner hash. * triedbmut only using alt type if inner hashing. * trie_stream to also only use alt hashing type when actually alt hashing. * Refactor meta state, logic should work with change of trie treshold. * Remove NoMeta variant. * Remove state_hashed trigger specific functions. * pending switching to using threshold, new storage root api does not make much sense. * refactoring to use state from backend (not possible payload changes). * Applying from previous state * Remove default from storage, genesis need a special build. * rem empty space * Catch problem: when using triedb with default: we should not revert nodes: otherwhise thing as trie codec cannot decode-encode without changing state. * fix compilation * Right logic to avoid switch on reencode when default layout. * Clean up some todos * remove trie meta from root upstream * update upstream and fix benches. * split some long lines. * UPdate trie crate to work with new design. * Finish update to refactored upstream. * update to latest triedb changes. * Clean up. * fix executor test. * rust fmt from master. * rust format. * rustfmt * fix * start host function driven versioning * update state-machine part * still need access to state version from runtime * state hash in mem: wrong * direction likely correct, but passing call to code exec for genesis init seem awkward. * state version serialize in runtime, wrong approach, just initialize it with no threshold for core api < 4 seems more proper. * stateversion from runtime version (core api >= 4). * update trie, fix tests * unused import * clean some TODOs * Require RuntimeVersionOf for executor * use RuntimeVersionOf to resolve genesis state version. * update runtime version test * fix state-machine tests * TODO * Use runtime version from storage wasm with fast sync. * rustfmt * fmt * fix test * revert useless changes. * clean some unused changes * fmt * removing useless trait function. * remove remaining reference to state_hash * fix some imports * Follow chain state version management. * trie update, fix and constant threshold for trie layouts. * update deps * Update to latest trie pr changes. * fix benches * Verify proof requires right layout. * update trie_root * Update trie deps to latest * Update to latest trie versioning * Removing patch * update lock * extrinsic for sc-service-test using layout v0. * Adding RuntimeVersionOf to CallExecutor works. * fmt * error when resolving version and no wasm in storage. * use existing utils to instantiate runtime code. * migration pallet * Patch to delay runtime switch. * Revert "Patch to delay runtime switch." This reverts commit d35f273b7d67b1b85a9e72973cab13c5c156c1d3. * fix test * fix child migration calls. * useless closure * remove remaining state_hash variables. * Fix and add more tests * Remove outdated comment * useless inner hash * fmt * remote tests * finally ksm works * batches are broken * clean the benchmarks * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> * Update frame/state-trie-migration/src/lib.rs * brand new version * fix build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Update primitives/storage/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * fmt and opt-in feature to apply state change. * feature gate core version, use new test feature for node and test node * Use a 'State' api version instead of Core one. * fix merge of test function * use blake macro. * Fix state api (require declaring the api in runtime). * Opt out feature, fix macro for io to select a given version instead of latest. * run test nodes on new state. * fix * new test structure * new testing stuff from emeric * Add commit_all, still not working * Fix all tests * add comment * we have PoV tracking baby * document stuff, but proof size is still wrong * FUCK YEAH * a big batch of review comments * add more tests * tweak test * update config * some remote-ext stuff * delete some of the old stuff * sync more files with master to minimize the diff * Fix all tests * make signed migration a bit more relaxed * add witness check to signed submissions * allow custom migration to also go above limit * Fix these pesky tests * ==== removal of the unsigned stuff ==== * Make all tests work again * separate the tests from the logic so it can be reused easier * fix overall build * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Update frame/state-trie-migration/src/lib.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Slightly better termination * some final tweaks * Fix tests * Restrict access to signed migrations * address most of the review comments * fix defensive * New simplified code * Fix weights * fmt * Update frame/state-trie-migration/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * make the tests correctly fail * Fix build * Fix build * try and fix the benchmarks * fix build * Fix cargo file * Fix runtime deposit * make rustdoc happy * cargo run --quiet --profile=production --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_state_trie_migration --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/state-trie-migration/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Joshy Orndorff <JoshOrndorff@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Parity Bot <admin@parity.io> * BEEFY and GRANDPA protocol names should use full genesis hash (#10974) std::fmt::Display shows formats as reduced hash (e.g. 0xb0a8…dafe) Use hex::encode to format full hash. Signed-off-by: acatangiu <adrian@parity.io> * Move weight constants to own mod (#10980) * Move block+ext weights to own mod Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Unused import Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * add block usage logs to system pallet (#10940) * add block usage logs to system pallet * add Debug * use % instead of default Debug impl * change formatting * revert * update runtime storage, update chain extension * cleanup chain ext * Fix the undeterministic storage proof recorded for the same execution (#10915) * Add a test case for the determinism of recorded proof * Replace HashMap with BTreeMap for the actual proof records * cargo +nightly fmt --all * Store the trie nodes in BTreeSet for StorageProof * Nit * Revert the BTreeMap changes and sort when converting to storage proof * Remove PartialEq from StorageProof * Remove unnecessary change * Add `compare` method to StorageProof * FMT * Dummy change to trigger CI * Use `BTreeSet` for StorageProof and keep using `Vec` for CompactProof * Update comment on `iter_nodes` * Revert `PartialEq` removal * create iris ledger pallet * ledger funcs in chain ext * rename Iris -> IrisAssets, update readmes * Minor Uniques pallet improvements and XCM v3 preparations (#10896) * Introduce Helper to Uniques for benchmark stuff * Fixes * Formatting * Featuregate the Helper, include ContainsPair * Introduce & use EnsureOriginWithArg * Benchmarking * Docs * More ContainsBoth helpers * Formatting * Formatting * Fixes Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * election provider support: Update some test only types (#10983) * Feedback from @XLC for Referenda Pallet (#10991) * feedback from @xlc * english * fmt Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Make bags-list generic over node value and instantiable (#10997) * make instantiable * update * cargo fmt * Clean up * bags-list: Make it generic over node value * Respond to some feedback * Apply suggestions from code review Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Add back default impl for weight update worst case * Update to Score in more places' * Use VoteWeight, not u64 to reduce test diff * FMT * FullCodec implies Codec * formatting * Fixup bags list remote test Co-authored-by: doordashcon <jesse.chejieh@gmail.com> Co-authored-by: Doordashcon <90750465+Doordashcon@users.noreply.github.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * sc-finality-grandpa: use the #[from] attriute to remove boilerplate code (#11003) Signed-off-by: koushiro <koushiro.cqx@gmail.com> * contracts: Add test to verify unique trie ids (#10914) * Add test to verify unique trie ids * Rename trie_seed to nonce * Rename AccountCounter -> Nonce * fmt * sp-core: `full_crypto` doesn't imply `std` (#11006) * sp-core: `full_crypto` doesn't imply `std` This pr changes the feature set of `secp256k1` to not use `global-context` when only the `full_crypto` is enabled. It will be slower when the `std` feature is not enabled as the context always needs to be recreated, but that is fine. * Update client/cli/src/arg_enums.rs Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Davide Galassi <davxy@datawok.net> * Remove unused `parent_hash` in `OverlayedChanges::into_storage_changes` (#11011) Ref https://github.com/paritytech/substrate/pull/10922#issuecomment-1064258443 CC @cheme * iris ledger pallet -> transfer not yet functioning * sp-api: Don't be dirty (#11015) Ensure that the sp api macros don't use functions without providing the full path to the function. This hygiene ensures that we don't actually try to call a method of an imported trait for example. * unlock and transfer works * Update clap to the latest version (#11017) * Update clap to the latest version Besides that it also removes some `structopt` leftovers from some docs. * Fix compile errors * More fixes * Move scripts used in CI to the new location (#11008) Move scripts used in CI to the new location - **./scripts/ci/** * Move github scripts * Move more files * Move ci scripts and fix dependencies * Update docs/node-template-release.md Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com> * Remove Cargo.lock * Apply suggestions from code review Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> * Make more paths uniform Co-authored-by: João Paulo Silva de Souza <77391175+joao-paulo-parity@users.noreply.github.com> Co-authored-by: Denis Pisarev <denis.pisarev@parity.io> * Move `sp-npos-elections-solution-type` to `frame-election-provider-support` (#11016) * Move `sp-npos-elections-solution-type` to `frame-election-provider-support` First stab at it, will need to amend some more stuff * Fixing tests * Fixing tests * Fixing cargo.toml for std configuration * fmt * Committing suggested changes renaming, and re exporting macro. * Removing unneeded imports * shell.nix: Update to a newer nightly (#11028) * update chain ext * update naming, formatting, start tests * Update lockfile (#11035) Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io> * Additional `benchmark-storage` flags (#11004) * Fix typos * Enable overwriting handlebars template * Optionally name json output or disable json altogether * Don't write to json by default * Include block id in handlebars output * Include warmups for write benchmarks * PR comments * Drop unnecessary file extension * Use more appropriate types * Use more appropriate error message * More use of more appropriate types * Rework write benchmark warmups * Run same benchmark for both read and write * SharedData: Update locks to mention possible deadlocks (#11034) * SharedData: Update locks to mention possible deadlocks * Update `Cargo.lock` * Stabilize `seal_delegate_call` (#11037) * add tests * re-enumerate chain ext func ids * Bump names from 0.12.0 to 0.13.0 (#11047) Bumps [names](https://github.com/fnichol/names) from 0.12.0 to 0.13.0. - [Release notes](https://github.com/fnichol/names/releases) - [Changelog](https://github.com/fnichol/names/blob/main/CHANGELOG.md) - [Commits](https://github.com/fnichol/names/compare/v0.12.0...v0.13.0) --- updated-dependencies: - dependency-name: names dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * State migration rpc (#10981) * setting flag * flag in storage struct * fix flagging to access and insert. * added todo to fix * also missing serialize meta to storage proof * extract meta. * Isolate old trie layout. * failing test that requires storing in meta when old hash scheme is used. * old hash compatibility * Db migrate. * runing tests with both states when interesting. * fix chain spec test with serde default. * export state (missing trie function). * Pending using new branch, lacking genericity on layout resolution. * extract and set global meta * Update to branch 4 * fix iterator with root flag (no longer insert node). * fix trie root hashing of root * complete basic backend. * Remove old_hash meta from proof that do not use inner_hashing. * fix trie test for empty (force layout on empty deltas). * Root update fix. * debug on meta * Use trie key iteration that do not include value in proofs. * switch default test ext to use inner hash. * small integration test, and fix tx cache mgmt in ext. test failing * Proof scenario at state-machine level. * trace for db upgrade * try different param * act more like iter_from. * Bigger batches. * Update trie dependency. * drafting codec changes and refact * before removing unused branch no value alt hashing. more work todo rename all flag var to alt_hash, and remove extrinsic replace by storage query at every storage_root call. * alt hashing only for branch with value. * fix trie tests * Hash of value include the encoded size. * removing fields(broken) * fix trie_stream to also include value length in inner hash. * triedbmut only using alt type if inner hashing. * trie_stream to also only use alt hashing type when actually alt hashing. * Refactor meta state, logic should work with change of trie treshold. * Remove NoMeta variant. * Remove state_hashed trigger specific functions. * pending switching to using threshold, new storage root api does not make much sense. * refactoring to use state from backend (not possible payload changes). * Applying from previous state * Remove default from storage, genesis need a special build. * rem empty space * Catch problem: when using triedb with default: we should not revert nodes: otherwhise thing as trie codec cannot decode-encode without changing state. * fix compilation * Right …

* Refactored tx storage database scheme * Bump parity-db * fmt * Fix handling invalid index size + test * Removed superflous result * Minor changes * fmt

Refactored tx storage database scheme

5e77168

arkpar requested review from bkchr and cheme February 2, 2022 14:44

github-actions bot added the A0-please_review Pull request needs code review. label Feb 2, 2022

arkpar added B5-clientnoteworthy C1-low PR touches the given topic and has a low impact on builders. D3-trivial 🧸 PR contains trivial changes in a runtime directory that do not require an audit labels Feb 2, 2022

arkpar added 2 commits February 2, 2022 16:04

Bump parity-db

60a9a69

fmt

5e8bfa5

This was referenced Feb 2, 2022

Companion PR for storage changes paritytech/polkadot#4837

Merged

Companion PR for storage changes paritytech/cumulus#949

Merged

cheme reviewed Feb 3, 2022

View reviewed changes

Cargo.lock Outdated Show resolved Hide resolved

client/db/src/lib.rs Outdated Show resolved Hide resolved

client/db/src/lib.rs Show resolved Hide resolved

client/db/src/lib.rs Show resolved Hide resolved

client/db/src/lib.rs Outdated Show resolved Hide resolved

Fix handling invalid index size + test

d123355

cheme approved these changes Feb 3, 2022

View reviewed changes

client/db/src/lib.rs Outdated Show resolved Hide resolved

arkpar added 3 commits February 3, 2022 16:18

Removed superflous result

ef13720

Merge branch 'master' of github.com:paritytech/substrate into a-new-t…

4f570cf

…x-storage

Merge branch 'master' of github.com:paritytech/substrate into a-new-t…

d393c92

…x-storage

arkpar mentioned this pull request Feb 19, 2022

Remark storage #10698

Merged

bkchr approved these changes Mar 3, 2022

View reviewed changes

arkpar added 3 commits March 4, 2022 10:07

Merge branch 'master' of github.com:paritytech/substrate into a-new-t…

cda5604

…x-storage

Minor changes

8e0961b

fmt

7c8e0d5

arkpar merged commit 9b5e654 into master Mar 4, 2022

arkpar deleted the a-new-tx-storage branch March 4, 2022 10:30

arkpar added the E2-databasemigration label Mar 11, 2022

librelois mentioned this pull request Mar 21, 2022

Update substrate/polkadot/cumulus from v0.9.17 to v0.9.18 moonbeam-foundation/moonbeam#1357

Closed

Dengjianping mentioned this pull request Apr 19, 2022

Bump deps to v0.9.18 Manta-Network/Manta#481

Merged

10 tasks

wischli mentioned this pull request Apr 22, 2022

feat: upgrade to Polkadot v0.9.19 KILTprotocol/kilt-node#357

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactored block body database scheme #10779

Refactored block body database scheme #10779

arkpar commented Feb 2, 2022 •

edited

Loading

cheme left a comment

bkchr commented Feb 17, 2022

arkpar commented Feb 17, 2022

bkchr commented Feb 17, 2022

arkpar commented Feb 17, 2022

bkchr Mar 3, 2022

bkchr Mar 3, 2022

bkchr Mar 3, 2022

ordian commented Mar 11, 2022

	return Err(sp_blockchain::Error::Backend(format!(
	Err(sp_blockchain::Error::Backend(format!(

Refactored block body database scheme #10779

Refactored block body database scheme #10779

Conversation

arkpar commented Feb 2, 2022 • edited Loading

Motivation:

Summary of changes:

cheme left a comment

Choose a reason for hiding this comment

bkchr commented Feb 17, 2022

arkpar commented Feb 17, 2022

bkchr commented Feb 17, 2022

arkpar commented Feb 17, 2022

bkchr Mar 3, 2022

Choose a reason for hiding this comment

bkchr Mar 3, 2022

Choose a reason for hiding this comment

bkchr Mar 3, 2022

Choose a reason for hiding this comment

ordian commented Mar 11, 2022

arkpar commented Feb 2, 2022 •

edited

Loading