contracts: Lazy storage removal #7740

athei · 2020-12-16T17:13:25Z

This PR adds a mechanism that moves the removal of contract storage to the on_initialize function. What that means is that when removing a contract due to eviction or deliberate termination its metadata is removed and is no longer accessible but the potentially heavy weight deletion of the child trie holding all the data is delayed. Because the deletion will happen earliest at the next block we need to store all the tries that should be deleted in on-chain storage as a queue. The maximum size of this queue is configurable and prevents the storage from growing indefinitely:

substrate/frame/contracts/src/lib.rs

Lines 323 to 324 in 2722cfc

    
           /// The maximum number of tries that can be queued for deletion. 
        
           type DeletionQueueDepth: Get<u32>;

This also means that seal_terminate and claim_surcharge are able to fail when this queue is full. There are no other implicit ways to delete a contract anymore (See the first commit in this PR).

The amount of weight that is dedicated to storage removal per block can be configured, too:

substrate/frame/contracts/src/lib.rs

Lines 326 to 327 in 2722cfc

    
           /// The maximum amount of weight that can be consumed per block for lazy trie removal. 
        
           type DeletionWeightLimit: Get<Weight>;

Tries that are too big to be removed within the configured limit will be split up and removed over the course of multiple blocks.

fixes #6594

We don't want to trigger contract eviction automatically when a contract is called. This is because those changes can be reverted due to how storage transactions are used at the moment. More Information: #6439 (comment) It can be re-introduced once the linked issue is resolved. In the meantime `claim_surcharge` must be called to evict a contract.

…ontract

It being the only dispatchable that is private is an oversight.

frame/contracts/src/storage.rs

frame/contracts/src/tests.rs

gui1117 · 2020-12-29T11:25:33Z

bin/node/runtime/src/lib.rs

@@ -714,6 +715,12 @@ parameter_types! {
 	pub const MaxDepth: u32 = 32;
 	pub const StorageSizeOffset: u32 = 8;
 	pub const MaxValueSize: u32 = 16 * 1024;
+	pub DeletionWeightLimit: Weight = Perbill::from_percent(20) *
+		RuntimeBlockWeights::get().max_block;


I wonder how this can collide with staking implementation of OffchainSolutionWeightLimit.

Basically staking offchain worker will use the maximum weight available for normal extrinsic. This maximum is computed using AVERAGE_ON_INITIALIZE_RATIO. If the on_initialize weight is more than AVERAGE_ON_INITIALIZE_RATIO for all the time where offchain worker needs to submit election result. Then a less good on-chain election will happen (I'm not sure how much this should be avoided).

So having underestimated AVERAGE_ON_INITIALIZE_RATIO under the offchain election is open can be a problem maybe, WDYT @kianenigma

Having multiple on_initialize functions that want to take some non-substantial amounts of weight from the block are a problem. They need to be coordinated somehow. Moving the lazy removal from on_initialize to some potential on_idle could be a solution.

That said, I am open to change those numbers here to whatever works.

I guess one way could be to have DeletionWeightLimitRatio < AVERAGE_ON_INITIALIZE_RATIO. so that even if the chain is busy removing lots of tries, election is still fine.

But I agree on_idle could be helpful, maybe staking could notify on_idle that it needs some space somehow

I guess one way could be to have DeletionWeightLimitRatio < AVERAGE_ON_INITIALIZE_RATIO. so that even if the chain is busy removing lots of tries, election is still fine.

But this assumes that this would be the only on_initialize in the block which it isn't. The AVERAGE_ON_INITIALIZE_RATIO shared between all of them.

Don't we already have this problem because of this code already:

substrate/bin/node/runtime/src/lib.rs

Lines 818 to 821 in 0fc8329

parameter_types! {

pub OffencesWeightSoftLimit: Weight = Perbill::from_percent(60) *

RuntimeBlockWeights::get().max_block;

}

I tuned it down to AVERAGE_ON_INITIALIZE_RATIO. With that setting we get the following throughput of the lazy removal:

Lazy deletion throughput per block (empty queue, full queue): 1954, 1564

In order to increase this throughput in the future we should look into on_idle or off chain workers.

The current OffchainSolutionWeightLimit is a bit tricky because it essential to the system to work properly, yet it is an unsigned transaction, so it is always likely that sum of on_initialize causes it to fail, in which case we do the notorious on-chain election. This will be fixed with #7319 which is far from being shipped yet, but almost ready. Once this is merged, we do hope to do most of the solution processing in on_initialize as well. In which case, the competition becomes fair. Something like staking, because it is critical to the system, would stay in on_initialize, and this can move to lazy removal.

A little brain dump about that: Let's say that a block has 100 weight units. Usually, 10 is consumed in on_initialize, and 40 in extrinsic, leaving 50 unused. Spreading this into the on_idle cases resembleas a thread/process scheduling situation. Two important notes come to mind:

Can we enforce the weight of each on_idle {} call? No, the runtime should have trust within it and the on_idle call of a module should simply return the weight that it has consumed, similar to on_initialize.

How can we ensure that the idle-scheduling is fair? This can be tricky but I guess a number of policies can be used, and ideally we can have it configurable. Naively, we can start by running the on_idle of each pallet until there is no weight left. Else, we can do it in a ring fashion, where if pallet at indices {0..3} got to execute their on_idle on this block, on the next block we start from {4..}.

Shouldn't we discuss this in the on_idle issue? Or do you think the lazy deletion would essentially break elections and therefore we need to implement on_idle first?

I added the following issue to keep track of this issue: #7817

That said, I will merge in the current form because:

This configuration is only for the dev node and neither kusama nor polkadot have it deployed

This is only an issue when contracts are heavily utilized the time an offchain election happens

Having an on-chain election will not break the chain

Decouple Staking and Election #7319 will most likely be merged before it gets important

on_idle will most likely be available before it gets important

I don't have any objections with merging this as it is. Thanks for making the issue!

About

Or do you think the lazy deletion would essentially break elections and therefore we need to implement on_idle first?

Yes, it could break it, but I don't think it is a showstopper because afaik there are not mainnets with both pallets. Just once you are setting the value of maximum weight that you can consume for this operation in node/runtime/src/lib, for the sake of prosperity, document that fact that the value shoudl be chosen with care and with care.

bin/node/runtime/src/lib.rs

gui1117

looks good to me, ~~with the additional test suggested by Shawn,~~ and not sure how to fix AVERAGE_ON_INITIALIZE usage.

frame/contracts/src/exec.rs

frame/contracts/src/lib.rs

kianenigma · 2021-01-04T09:18:48Z

Having just read the description, I guess maybe this can use something like #4064?

athei · 2021-01-04T09:24:07Z

Yes we already discussed that it could profit from it. Until then I tuned down the weight consumption so it should be OK. The on_idle would be very useful to improve on deletion throughput.

kianenigma · 2021-01-04T09:25:09Z

ahh right is is mentioned multiple times :D 🤦‍♂️

athei · 2021-01-04T12:13:04Z

bot merge

ghost · 2021-01-04T12:13:09Z

Waiting for commit status.

ghost · 2021-01-04T12:30:01Z

Merge failed: "Required status check \"continuous-integration/gitlab-check-labels\" is expected."

athei · 2021-01-04T12:35:51Z

bot merge

ghost · 2021-01-04T12:35:56Z

Trying merge.

* Do not evict a contract from within a call stack We don't want to trigger contract eviction automatically when a contract is called. This is because those changes can be reverted due to how storage transactions are used at the moment. More Information: paritytech#6439 (comment) It can be re-introduced once the linked issue is resolved. In the meantime `claim_surcharge` must be called to evict a contract. * Lazily delete storage in on_initialize instead of when removing the contract * Add missing documentation of new error * Make Module::claim_surcharge public It being the only dispatchable that is private is an oversight. * review: Add final newline * review: Simplify assert statement * Add test that checks that partial remove of a contract works * Premote warning to error * Added missing docs for seal_terminate * Lazy deletion should only take AVERAGE_ON_INITIALIZE_RATIO of the block * Added informational about the lazy deletion throughput * Avoid lazy deletion in case the block is already full * Prevent queue decoding in case of an already full block * Add test that checks that on_initialize honors block limits

* Features needed for reserve-backed stablecoins (#7152) * Features needed for reserve-backed stablecoins * Builds & tests. * Double map for an efficient destroy. * Update frame/assets/src/lib.rs Co-authored-by: Nikolay Volf <nikvolf@gmail.com> * ED/zombie-count/refs Feature: ED/minimum balance enforcement Feature: enforce zombie count Feature: allow system-alive accounts to exist, but add reference * Update frame/assets/src/lib.rs Co-authored-by: Nikolay Volf <nikvolf@gmail.com> * Update frame/assets/Cargo.toml Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com> * Docs * Some tests * More tests * Allow for max_zombies to be adjusted * Test for set_max_zombies * Tests and a couple of fixes * First few benchmarks * Benchmarks. * Fix error message in test * Fixes * Fixes * Fixes * cargo run --release --features runtime-benchmarks --manifest-path bin/node/cli/Cargo.toml -- benchmark --chain dev --steps 50 --repeat 20 --extrinsic * --execution=wasm --wasm-execution=compiled --output ./bin/node/runtime/src/weights --header ./HEADER --pallet pallet_assets * Update frame/assets/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Fixes * Fixes * Fixes * cargo run --release --features runtime-benchmarks --manifest-path bin/node/cli/Cargo.toml -- benchmark --chain dev --steps 50 --repeat 20 --extrinsic * --execution=wasm --wasm-execution=compiled --output ./bin/node/runtime/src/weights --header ./HEADER --pallet pallet_assets * Fixes * Update default weight * Add proper verification to benchmarks * minor improvements to tests * Update frame/assets/src/benchmarking.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Fix * New weights system * fix compile * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_assets --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/assets/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: Nikolay Volf <nikvolf@gmail.com> Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com> Co-authored-by: Parity Benchmarking Bot <admin@parity.io> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Add Standard Error to Weight Template (#7652) * Add standard error to weight template * fix test compile * Adds support for tuples in runtime-interface (#7672) * Fix two potential connection poisonings (#7677) * Fix #7629 (#7680) * Fix #7629 * Update client/network/src/service.rs Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Max Inden <mail@max-inden.de> * Update Cargo.toml (#7627) On behalf of dependabot, because we use 1.4.x in it clashes with this. * Make sure the companion CI test fails on any error (#7683) * implement more convertion on NumberOrHex (#7682) * Streamline frame_system weight parametrization (#6629) * Basic weights builder. * Fixing WiP * Make the tests work. * Fix weights in node/runtime. * WiP. * Update pallets with new weights parameters. * Validate returns a Result now. * Count mandatory weight separately. * DRY * BREAKING: Updating state root, because of the left-over weight-tracking stuff * Update tests affected by Mandatory tracking. * Fixing tests. * Fix defaults for simple_max * Update frame/system/src/weights.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Rework the API a bit. * Fix compilation & tests. * Apply suggestions from code review Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Add extra docs & rename few things. * Fix whitespace in ASCII art. * Update frame/system/src/limits.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Fix max_extrinsic calculations. * Fix conflicts. * Fix compilation. * Fix new code. * re-remove generic asset * Fix usage. * Update state root. * Update proxy. * Fix tests. * Move weights validity to integrity_test * Remove redundant BlockWeights. * Add all/non_mandatory comment * Add test. * Remove fn block_weights * Make the macro prettier. * Fix some docs. * Make max_total behave more predictabily. * Add BlockWeights to metadata. * fix balances test * Fix utility test. Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Benjamin Kampmann <ben@gnunicorn.org> Co-authored-by: thiolliere <gui.thiolliere@gmail.com> * client: remove duplicate implementation of block_status (#7694) * Pin md link checker's version (#7697) * CI: pin md link checker's version to the previous release * CI: add a whitelist for allowed GH actions pins * CI: try master tip [skip ci] * CI: return to a previous markdown-link-check dependency version [skip ci] * CI: substitute %20 for a literal space [skip ci] * CI: substitute %20 for a literal space * Remove waterfall CI trigger (#7519) * fix (CI): fix the triggered project name * fix (CI): debug run * fix (CI): debug run 2 * fix (CI): revert debug2 * Revert "fix (CI): debug run" This reverts commit 96866a953da797ef6e0689374eecc0d8cba4627d. * CI: remove the trigger * add an upgrade_keys method for pallet-session (#7688) * add an upgrade_keys method for pallet-session * test the upgrade_keys function * Allow capping the amount of work performed when deleting a child trie (#7671) * Allow Backend::for_keys_in_child_storage to be aborted by the closure * Ext::kill_child_storage now takes an upper limit for backend deletion * Add Storage::storage_kill_limited() runtime interface * review: Use a new version of kill_storage instead of a new interface * review: Simplify boolean expression Co-authored-by: cheme <emericchevalier.pro@gmail.com> * review: Rename for_keys_in_child_storage Co-authored-by: cheme <emericchevalier.pro@gmail.com> * benchmarks: Fix panic in case of a missing model (#7698) Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Expand remote keystore interface to allow for hybrid mode (#7628) * update to latest master * updates on docs, license, meta * hide ssrs behind feature flag * implement remaining functions on the server * sign server line length fix * fix tests * fixup in-memory-keystore * adding failsafe * skipping ecdsa test for now * remote keystore param * remote sign urls made available * integrating keystore remotes features * don't forget the dependency * remove old cruft * reset local keystore * applying suggestions * Switch to single remote, minor grumbles * minor grumbles, docs * fork-tree: expose silent failures of is_descendent_of (#7695) * fork-tree: expose silent failures of is_descendent_of * fork-tree: use mem::take instead of mem::swap * Merkle Mountain Range pallet (#7312) * Add MMR pallet. * WiP * Working on testing. * WiP - test * Tests passing. * Add proof generation. * Generate and verify proofs. * Allow verification of older proofs. * Move stuff to a module. * Split MMR stuff to it's own module. * Add docs. * Make parent hash optional. * LeafData failed approach. * Finally implement Compact stuff. * Compact encoding WiP * Implement remaining pieces. * Fix tests * Add docs to compact. * Implement for tuples. * Fix documentation. * Fix warnings and address review suggestion. * Update frame/merkle-mountain-range/src/primitives.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Address review grumbles. * Removing missing crate. * Fix test. * Add some docs and test. * Add multiple instances. * Cargo.toml sync. * Fix no_std compilation. * More no_std stuff. * Rename MMR struct. * Addressing other grumbles. * Fix test. * Remove format for no_std compat. * Add test for MMR pallet. * Fix std feature. * Update versions. * Add to node/runtime. * Add hook to insert digest. * Make primitives public. * Update lib.rs tech spec/typos etc * Use WeightInfo and benchmarks. * Fix test. * Fix benchmarks. * Trait -> Config. * Fix typo. * Fix tests. Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Addie Wagenknecht <addie@nortd.com> * frame/staking/README: Fix broken link (#7703) * *: Update to libp2p v0.32.0 (#7696) * *: Update to libp2p v0.32.0 * Cargo.lock: Update async-tls to 0.10.2 * client/network/request_response: Adjust to new request response events * client/network/request_response.rs: Clean up silently failing responses * client/network/discovery: Lazily instantiate mdns * client/network/discovery: Exclude MdnsWrapper for target_os unknown * client/network/discovery: Fix indentation * client/network/request-response: Use LruCache to track pending resp time * client/network/request_responses: Fix early connection closed error * client/network/request-response: Replace debug_assert with debug * client/network/request-response: Fix typo * client/network/request-response: Don't emit event on send_response fail * client/network/request-response: Revert waker.wake_by_ref() * client/network/request-resp: Make duration in InboundRequest optional * client/network/req-resp: Don't emit two events for busy builder When a response builder is busy incoming requests are dropped. Previously this was reported both via a `ResponseFailure::Busy` and a `ReponseFailure::Network(InboundFailure::Omisssion)` event. With this commit the former is removed, leaving only the latter in place. * *: Update to libp2p v0.32.2 (#7708) Version v0.32.2 pins async-tls to the semver compliant v0.11.0 instead of the yanked async-tls v0.10.2. * kilt&polimec ss58 prefix (#7547) * kilt ss58 address * polimec ss58 prefix * Fix docs (#7710) * fix docs * Update frame/merkle-mountain-range/src/lib.rs Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Update md link checker (#7709) * CI: ignore %20 for md link checker * CI: update to a new action release that fixes %20 issue * Fix PerThing::from_percent. (#7701) * Fix overflow in per_things::from_percent. * Fix test * Fix the whole thing.. :| * Support Multiple Instances with Benchmarks (#7669) * Support multiple instances with benchmarks * fix tests * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * docs * fix output * Update lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Add keccak-512 to host functions. (#7531) * Update debug.rs (#7721) * Avoid too high values. (#7716) * Bump hyper from 0.12.35 to 0.13.9 (#7569) * Bump hyper from 0.12.35 to 0.13.9 Bumps [hyper](https://github.com/hyperium/hyper) from 0.12.35 to 0.13.9. - [Release notes](https://github.com/hyperium/hyper/releases) - [Changelog](https://github.com/hyperium/hyper/blob/master/CHANGELOG.md) - [Commits](https://github.com/hyperium/hyper/compare/v0.12.35...v0.13.9) Signed-off-by: dependabot[bot] <support@github.com> * Update bin/node/rpc-client/Cargo.toml * Update Cargo.lock Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> * Prevent dust in staking by disallowing cheap bond_extra (#7718) * prevent bond_extra to cause staking actve lower than ed * prevent bond_extra to cause staking actve lower than ed * Check in post conditions. * check rebond as well. * also change withdraw_unbonded. * Fix build * change check format. * Apply suggestions from code review Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Default block announce validation should reject additional data (#7726) If we are using the default block announce validator and receive additional data alongside the block announcement, we should disconnect this peer. * Fix wrong order of values in log line (#7728) * Fix polkadot companion test (#7732) * remove duplicate accounts in chain-spec. (#7725) * remove duplicate accounts in chain-spec. * Fix build * Enforce in balances modules. * Apply suggestions from code review * Update frame/balances/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> * Participating in Council Governance is Free for First Time Voters and Successful Closing (#7661) * wk2049 | D5 | addition-motions first free vote for members | p1 * Update frame/collective/src/lib.rs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * wk2049 | D6 | addition-motions first free vote for members | p2 * Update frame/collective/src/lib.rs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Update frame/collective/src/lib.rs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * wk2049 | D7 | addition-motions first free vote for members | p3 * Update frame/collective/src/lib.rs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * wk2049 | D7 | addition-motions first free vote for members | p4 * wk2049 | D7 | addition-motions first free vote for members | p6 * Update frame/collective/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/collective/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/collective/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/collective/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/collective/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/collective/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/collective/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * wk2050 | D3 | addition-motions first free vote for members | p7 * wk2050 | D3 | addition-motions first free vote for members | p8 * update comment Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Bump zeroize from 1.1.1 to 1.2.0 (#7731) Bumps [zeroize](https://github.com/iqlusioninc/crates) from 1.1.1 to 1.2.0. - [Release notes](https://github.com/iqlusioninc/crates/releases) - [Commits](https://github.com/iqlusioninc/crates/compare/zeroize/v1.1.1...zeroize/v1.2.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Leave some header related info for inspection (#7727) * Leave some system data for inspection There is not much benefit in being active when removing this data. It's actively harmful when one tries to read the block number in runtime APIs in the context of a block. * Update the expected root hash This is excepted since now we persist new members. * Revert extrinsics_root to `take` It's going away in one of the following PRs anyway * Update the state root once again * Update the comment on the storage items that are left in the storage Excluding ExtrinsicsRoot since it's going away * Issue 7143 | Refactor Treasury Pallet into Bounties, Tips, and Proposals (#7536) * wk2046 | D5 | treasury refactor to bounties and tips * wk2046 | D5 | treasury refactor to bounties and tips | p2 * fix test compilation, ignoring events * initialize treasury in genesis * wk2046 | D7 | treasury refactor | fix bounties test build issues * wk2047 | D1 | treasury refactor | tips pallet bringup * wk2047 | D2 | treasury refactor | bounties pallet | unit test bringup * wk2047 | D2 | treasury refactor | bounties pallet | unit test bringup | p2 * wk2047 | D2 | treasury refactor | pallet-tips| test_last_reward_migration | test failure - analysis * wk2047 | D3 | treasury refactor | pallet-tips| test_last_reward_migration | test failure - fix * wk2047 | D3 | treasury refactor | pallet-bounties | on_initialize() fix * wk2047 | D3 | treasury refactor | pallet-bounties | on_initialize() fix | p2 * wk2047 | D4 | treasury refactor | pallet-bounties + pallet-treasury | spend_fund runtime hooks * wk2047 | D4 | treasury refactor | pallet-bounties + pallet-treasury | spend_fund runtime hooks | p2 * wk2047 | D4 | treasury refactor | pallet-bounties + pallet-treasury | spend_fund runtime hooks | p3 * wk2047 | D5 | treasury refactor | pallet-bounties + pallet-treasury | spend_fund runtime hooks | p4 * wk2047 | D6 | treasury refactor | review comments fix * some fixes * fix bounties instantiable * remove instantiable from tips and bounties * fix compile for benchmarks * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_treasury --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/treasury/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Update lib.rs * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_tips --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/tips/src/weights.rs --template=./.maintain/frame-weight-template.hbs * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bounties --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bounties/src/weights.rs --template=./.maintain/frame-weight-template.hbs * add back `on_initialize_bounties` * patch up bounties benchmarks * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_bounties --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/bounties/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Remove development TODO from public doc comment (#7500) * refactor subtrait/elevated trait as not needed (#7497) * Fix comments of indices pallet events (#7511) Arguments for IndexAssigned and IndexFrozen were inverted in comments. * Skip slot lenience on first block in BABE (#7515) The genesis header doesn't have the BABE pre-digest and we insert `0` as slot number. The slot lenience calculation will return the maximum in this situation. Besides returning the maximum which is not bad at all, it also prints some a debug message that can be confusing in the first moment. To prevent printing this debug message, we now just return early when we see that the parent block is the genesis block. * slots: incrementally backoff claiming slots if finality lags behind (#7186) * babe: backoff authoring blocks when finality lags * babe: move backoff authoring params to default constructor * babe: deduplicate the test a bit * babe: set backoff constants in service * babe: use better names for backoff authoring block parameters * babe: remove last unwrap * babe: slight style tweak * babe: fix comment * slots: move backoff block authorship logic to SimpleSlotWorker * aura: append SlotInfo in on_slot * slots: use the correct types for parameters * slots: fix review comments * aura: add missing backoff authoring blocks parameters * slots: add comments for default values * slots: add additional checks in test * slots: update implementation for new master * slots: revert the change to SlotInfo * Fix review comments * slots: rework unit tests for backing off claiming slots * slots: add test for asymptotic behaviour for slot claims * slots: address review comments * slots: add test for max_interval * slots: add assertion for intervals between between claimed slots * slots: remove rustfmt directive * slots: another attempt at explaining authoring_rate * slots: up unfinalized_slack to 50 by default * slots: add tests for time to reach max_interval * slots: fix typo in comments * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * slots: additional tweaks to comments and info calls * slots: rename to BackoffAuthoringOnFinalizedHeadLagging * slots: make the backing off strategy generic * Apply suggestions from code review Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * slots: implement backoff trait for () for simplicity * slots: move logging inside backing off function to make it more specific * aura: add missing function parameter Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Export app-crypto specific keystore functions (#7489) * Export app-crypto specific keystore functions * Also add back the insert function * Switch KeystoreContainer to an enum * Only export the bare minimal for LocalKeystore and fix service compile * fix: should return Arc * Add docs stating that functions only available in local keystore * Remove insert and generate functions * fix: generate function should be available in test * Add keypair function to trait * Revert "Add keypair function to trait" This reverts commit ad921b09ca73d3c09298e3a51b562ef8e0067781. * Add note for local_keystore function in service * Update doc for the --chain flag (#7520) * contracts: Add missing instruction to the `Schedule` (#7527) * Don't log with colors when we are writing to a tty (#7525) * Don't log with colors when we are writing to a tty This fixes a regression that was introduced by the switch to tracing. Before we killed all colors before writing to a tty, this pr brings the behaviour back. * Remove accidentally added crate * Review feedback * More feedback * Update client/cli/src/logging.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Update client/cli/src/logging.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * MemoryId -> u32 (#7534) * Enable local addresses in DHT when chain type == `Local` | `Development` (#7538) * Enable local addresses in DHT when chain type == `Local` | `Development` This pr changes when to add local addresses to DHT. Instead of only checking if `--discover-local` and `--dev` are present, we now also check if the chain type is `Local` or `Development`. * Update the docs! * Update tiny-bip39 to v0.8 (#7539) It would improve secret zeroization due to https://github.com/maciejhirsz/tiny-bip39/pull/22, and would also remove one of the points where we depend on `failure` crate, which is deprecated (see https://github.com/rust-lang-nursery/failure/pull/347) * make LocalCallExecutor public (#7528) * Fix some weirdness in `offchain_worker` (#7541) We call `offchain_worker` with the state of the imported block and pass the header of this block. However in the runtime we call all `offchain_worker` functions with the number of the parent block. Besides that we also pass all digests and not only the pre runtime digests. In the context where the offchain worker is executed we have all digests, so there is no real reason to only pass pre runtime digests. Another fix is that we also insert the hash of the current header into the block hash map. * Use inbound peerslot slots when a substream is received, rather than a connection (#7464) * Use inbound peerslot slots when a substream is received, rather than a connection * Refactor PeerState * Some bugfixes * Fix warnings so that CI runs, gmlrlblbl * Bugfixes * Update docs * Apply suggestions from code review Co-authored-by: Roman Borschel <romanb@users.noreply.github.com> * Clean up Banned state * Refactor connections state * Fix possibility of Enabled with no Opening or Open connection * Line width * Add some debug_asserts! and fix TODO * Refactor legacy handler * Rewrite group.rs entirely [part 1] * Rewrite group.rs entirely [part 2] * Remove faulty assertion Because of the asynchronous nature of the behaviour <-> handler communications, it is possible to receive notifications while in the Closing state * Don't poll the legacy substream is not Open * Tolerate when not all substreams are accepted * Remove TODOs * Dummy commit to make CI log interesting things * Try race condition fix * Revert "Try race condition fix" This reverts commit 0675c659d06195c30f8c5bc13e2d88141d57a3ba. * Correctly rebuild pending_opening * Minor tweaks * Printlns for CI debugging * Revert "Printlns for CI debugging" This reverts commit e7852a231f4fc418898767aaa27c9a4358e12e8b. * Revert "Dummy commit to make CI log interesting things" This reverts commit 259ddd74088e53e7c6a9b0a62a8d1573a0063ce3. * mv group.rs ../handler.rs * Apply suggestions from code review Co-authored-by: Max Inden <mail@max-inden.de> * Banned => Backoff * Mention the actual PeerStates * OpenDesired -> OpenDesiredByRemote * OpeningThenClosing * Add doc links to PeerState * Simplify increment logic * One more debug_assert * debug_assert! * OpenDesiredByRemote * Update client/network/src/protocol/generic_proto/behaviour.rs Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Roman Borschel <romanb@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> * *: Update to libp2p v0.30.0 (#7508) * *: Update to libp2p v0.30.0 * Cargo.lock: Update * *: Update to libp2p v0.30.1 * make ClientConfig public (#7544) * sc-basic-authorship: remove useless dependencies (#7550) Signed-off-by: koushiro <koushiro.cqx@gmail.com> * Add slashing events to elections-phragmen. (#7543) * Add slashing events to elections-phragmen. * Fix build * Apply suggestions from code review * Update frame/elections-phragmen/src/lib.rs * Update frame/elections-phragmen/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Remove necessity to pass ConsensusEngineId when registering notifications protocol (#7549) * Remove necessity to pass ConsensusEngineId when registering notifications protocol * Line width * Fix tests protocol name * Other renames * Doc update * Change issue in TODO * sc-cli: replace bip39 with tiny-bip39 (#7551) Signed-off-by: koushiro <koushiro.cqx@gmail.com> * Add extra docs to on_initialize (#7552) * Add some extra on_initialize docs. * Address review comments. * More Extensible Multiaddress Format (#7380) * More extensible multiaddress format * update name * Don't depend on indices to define multiaddress type * Use MultiAddress in Node Template too! * reduce traits, fix build * support multiple `StaticLookup` * bump tx version * feedback * Fix weight template to remove ugliness in rust doc (#7565) fixed weight template * Cargo.lock: Run cargo update (#7553) * Cargo.lock: Run cargo update * Cargo.lock: Downgrade cc to v1.0.62 * Cargo.lock: Revert wasm-* updates * .github: Add dependabot config and thus enable dependabot (#7509) * .github: Add dependabot config and thus enable dependabot * Update .github/dependabot.yml Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> * Thread-local parameter_types for testing. (#7542) * Thread-local parameter_types for testing. * Better docs. * Some minors * Merge'em * Update frame/support/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Align more to basti's trick * Update frame/support/src/lib.rs * Update frame/support/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> * Bump wasm-bindgen-test from 0.3.12 to 0.3.17 (#7567) * Bump wasm-bindgen-test from 0.3.12 to 0.3.17 Bumps [wasm-bindgen-test](https://github.com/rustwasm/wasm-bindgen) from 0.3.12 to 0.3.17. - [Release notes](https://github.com/rustwasm/wasm-bindgen/releases) - [Changelog](https://github.com/rustwasm/wasm-bindgen/blob/master/CHANGELOG.md) - [Commits](https://github.com/rustwasm/wasm-bindgen/commits) Signed-off-by: dependabot[bot] <support@github.com> * Update wasm-bindgen pin to 0.2.68 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> * wk2047 | D6 | treasury refactor | review comments fix | p2 * wk2048 | D1 | treasury refactor | review comments fix | p3 * Update bin/node/runtime/src/lib.rs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Update bin/node/runtime/src/lib.rs Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * wk2048 | D3 | treasury refactor | review comments fix | p4 * wk2048 | D3 | treasury refactor | review comments fix | p5 * wk2048 | D4 | treasury refactor | review comments fix | removal of deadcode | p6 * remove broken link * wk2048 | D5 | treasury refactor | review comments fix | bountise doc string | p7 * wk2048 | D5 | treasury refactor | review comments fix | p8 * docs and formatting * Update frame/tips/src/benchmarking.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * formatting nits * Trait -> Config * trait -> config in benchmarks * clean up weight docs * Trait -> Config in Runtime * fix test build * try to fix polkadot build check * fix traits * Update lib.rs * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Apply suggestions from code review Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * fix trait location * nits * uncomment on_initialize for bounties benchmarks * update weights Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Parity Benchmarking Bot <admin@parity.io> Co-authored-by: Caio <c410.f3r@gmail.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> Co-authored-by: Antoine Le Calvez <alecalve@users.noreply.github.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com> Co-authored-by: Wei Tang <wei@that.world> Co-authored-by: Sergei Shulepov <sergei@parity.io> Co-authored-by: Alexander Theißen <alex.theissen@me.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: Pierre Krieger <pierre.krieger1708@gmail.com> Co-authored-by: Kirill Pimenov <kirill@parity.io> Co-authored-by: Andrew Plaza <aplaza@liquidthink.net> Co-authored-by: Roman Borschel <romanb@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Qinxuan Chen <koushiro.cqx@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Tomasz Drwięga <tomusdrw@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Addie Wagenknecht <addie@nortd.com> * CI: buildah to build images (#7741) * CI: buildah to build images * CI: simplify variables * CI: whitespaces * CI: secure login * optimize arithmetic cargo path (#7745) * optimize runtime-interface cargo path (#7729) * delete primitives in Cargo in primitives (#7751) * delete client in Cargo in client (#7752) * prevent too many genesis council members (#7749) * Prevent too many genesis members in elections-phragmen. * Fix test. * reformat.' * Update common block in sync after importing blocks of a peer, please read UPDATE (#7733) * Update common block in sync after importing blocks of a peer This updates the sync code to update the common block of a peer, after we have imported blocks from this peer. This fixes a bug for when we are connected to one or more nodes that are doing a full sync as our node. Nodes in full sync will not announce new blocks, as we don't send import notifications on full sync. The problem as now that we were connected to some peer that reported some low number as its best and we tried to sync these blocks. But, as we did not update the common block of this peer, we would sync these blocks over and over again. Being captured in some time warp. The solution to this problem is that we increase the common number as we import blocks from this peer. * Test * Test name.. * Fix test * Cleanup some code and write some new regression test * Implement the ancestor search * Check that the common number is smaller than the last finalized block * Update client/network/src/protocol/sync.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Update client/network/src/protocol/sync.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Update client/network/src/protocol/sync.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Change the way we build the status messages * Start some new test... * Finish test * Rename test * Update client/network/src/protocol.rs Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> Co-authored-by: André Silva <123550+andresilva@users.noreply.github.com> * Remove unnecessary parameter from sync (#7761) * Improve `InvalidTransaction::AncientBirthBlock` doc comment (#7743) * Improve `InvalidTransaction::AncientBirthBlock` doc comment Update Remove stray parentheses Tabs not spaces Improve whoops * Update primitives/runtime/src/transaction_validity.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Possible causes section; Expand example Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Switch back to the crates.io release of trybuild (#7764) * Switch back to the crates.io release of trybuild My fix was merged on upstream and this release contains it. So, no more reason to keep the git dependency. * The lock file... * No longer put keys back manually after a failed restoration (#7747) This is no longer necessary with storage transactions. * Fix UI Tests (#7722) * Fix UI tests * make updates. * Revert "make updates." This reverts commit 3f41979c12e86acb75fd086675ac09659dde533e. * Try to fix companion build * Use some depth * tests: fix UI tests on the latest nightly * tests: fix polkadot companion test, depth * chore: update deps * chore: update deps * chore: sp-io Co-authored-by: kianenigma <kian.peymani@gmail.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Denis P <denis.pisarev@parity.io> * update substrate to sc in docs (#7767) * Allow Staking tests to run with session length other than 1 (#7719) * fix periodic session * Allow staking tests to run with session lengths other than 1. * Update frame/staking/src/mock.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Fix all tests with session length 5. * Test for active != current * Better doc * Update frame/staking/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * also set the timestamp properly. * trigger CI * Revert "trigger CI" This reverts commit 0f254944cdad848aa6e63bd8a618db95447a8e68. * Update frame/staking/src/lib.rs Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * optimize biguint div closure (#7754) * optimize biguint div closure * optimize biguint sub and fix note * change and add biguint split test * add biguint div_unit test * update biguint sub v to v1 * add biguint shift_check * Make it possible to calculate the storage root as often as you want (#7714) * Make it possible to calculate the storage as often as you want So, until now each Substrate based blockchain has calculated the storage root once, at the end of the block. Now there is Frontier that wants to calculate some intermediate storage root. However this failed on block import. The problem with that was the extrinsics root. When building the block we stored `Default::default()` as extrinsics root, because yeah, we don't know the extrinsics root before finishing the block. At the end this extrinsics root was then calculated. But on block import we passed the already known extrinsics root. This was no problem, as we removed this value at the end of the block. However when you all the storage root in between, that changes the storage root between block building and block import. This pr changes this behavior. It removes the `ExtrinsicsRoot` storage entry and also doesn't pass it anymore to `System::initialize`. By doing it, we remove the difference in the storage and fix the storage root mismatch. * Fix bug with incorrectly calculating the extrinscs root * Review feedback * Typo fix: eror => error (#7773) * Adds `try_get` for `StorageMap` and `StorageDoubleMap` (#7774) * Adds `try_get` for `StorageMap` and `StorageDoubleMap` * Switch to Value as return type * Add account prefix for Valiu (#7776) * Feat support weight test (#7769) * update support weights_are_correct * add support polynomial_does_not_underflow * Add `pallet` attribute macro to declare pallets (#6877) * rename system Config to system Trait. command used: ``` find frame/ bin/ test-utils/ utils/ -name *.rs -exec sed -i 's/system::Trait>::/system::Config>::/g' {} \; find frame/ bin/ test-utils/ utils/ -name *.rs -exec sed -i 's/impl frame_system::Trait for /impl frame_system::Config for /g' {} \; find frame/ bin/ test-utils/ utils/ -name *.rs -exec sed -i 's/impl system::Trait for /impl system::Config for /g' {} \; ``` plus some manual ones especially for frame-support tests and frame-system * make construct_runtime handle Pallet and Module pallets can now be implemented on struct named Pallet or Module, both definition are valid. This is because next macro will generate only Pallet placeholder. * introduce pallet attribute macro currently just with tests, frame_system and other example hasn't been upgraded * allow to print some upgrade helper from decl_storage * Improved error msg, typo. Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Improved error msg, typo. Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Improved error message on unexpected attributes + ui test * add test for transactional * various typo * some tips when spans are lost * allow pallet to depend on other pallet instances * make event type metadata consistent with call and constant * error messages * ignore doc example * fix pallet upgrade template * fixup * fix doc * fix indentation * Apply suggestions code formatting Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * some renames + fix compilation * remove unsupported genesis config type alias * merge fixup * fix ui tests * additional doc * implement StorageInstance with new syntax * fix line width * fix doc: because pallet doc goes below reexport doc * Update frame/support/procedural/src/pallet/parse/event.rs Co-authored-by: Andrew Jones <ascjones@gmail.com> * Update frame/system/src/lib.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * Update frame/support/test/tests/pallet_ui.rs Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * improve doc as suggested * revert construct_runtime Pallet part. This revert the changes on construct_runtime. Now construct_runtime is unchanged and instead pallet macro create a type alias `type Module<..> = Pallet<..>` to be used by construct_runtime * refactor with less intricated code * fix ui test with new image * fix ui tests * add minor tests Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Andrew Jones <ascjones@gmail.com> Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com> * add a `current_epoch` to BabeApi (#7789) * add a `current_epoch` to BabeApi * add current_epoch_start back again * fix node-runtime * bump spec version * Update tests.rs (#7784) Fix testing for number of slashing spans. * add a slot-duration getter to babe config (#7793) * Fix base-path handling in `key insert` (#7775) This fixes the handling of base-path when using `key insert`. Before the base-path wasn't setup correctly, as done when starting a node. This resulted in putting the keys into the wrong directory. This pr fixes this by creating the correct base-path/config dir for the keystore. Besides that it also removes the insert command from `subkey` as it doesn't make that much sense. If requested, we could bring it back later. * Lazily evaluate error string (#7802) * contracts: Change `define_env!` to expect a `Result<T, DispatchError>` for every function (#7762) * Make host functions return TrapReason This avoids the need to manually store any trap reasons to the `Runtime` from the host function. This adds the following benefits: * It properly composes with the upcoming chain extensions * Missing to set a trap value is now a compile error * review: Remove superflous .into() * Hex Balance deserialize for contracts_call RPC (#7807) * Hex Balance deserialize for contracts_call RPC * Avoid temporary conversion into u128 Co-authored-by: Alexander Theißen <alex.theissen@me.com> * Don't allow self proxies (#7803) * Allow council to slash treasury tip (#7753) * wk2051 | D4 |Allow council to slash treasury tip | p1 * Update frame/tips/src/lib.rs Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> * wk2051 | D5 |Allow council to slash treasury tip | p2 * wk2051 | D5 |Allow council to slash treasury tip | p3 * wk2051 | D5 |Allow council to slash treasury tip | p4 * wk2051 | D5 |Allow council to slash treasury tip | p5 * random change * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_tips --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/tips/src/weights.rs --template=./.maintain/frame-weight-template.hbs * fix typo * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/tests.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * wk2052 | D1 | Allow council to slash treasury tip | p6 Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Parity Benchmarking Bot <admin@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Add CLI argument to disable log color output (#7795) * Use path instead of ident (#7809) * Add proper `commit_all` to `TestExternalities` (#7808) * Add proper `commit_all` to `TestExternalities` This pr adds a propoer `commit_all` function to `TestExternalities` to commit all changes from the overlay to the internal backend. Besides that it fixes some bugs with handling empty dbs when calculating a delta storage root. It also changes the way data is added to the in memory backend. * Update primitives/state-machine/src/testing.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> * Don't allow self proxies (#7803) * Allow council to slash treasury tip (#7753) * wk2051 | D4 |Allow council to slash treasury tip | p1 * Update frame/tips/src/lib.rs Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> * wk2051 | D5 |Allow council to slash treasury tip | p2 * wk2051 | D5 |Allow council to slash treasury tip | p3 * wk2051 | D5 |Allow council to slash treasury tip | p4 * wk2051 | D5 |Allow council to slash treasury tip | p5 * random change * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_tips --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/tips/src/weights.rs --template=./.maintain/frame-weight-template.hbs * fix typo * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/lib.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Update frame/tips/src/tests.rs Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * wk2052 | D1 | Allow council to slash treasury tip | p6 Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: Parity Benchmarking Bot <admin@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> * Review feedback * Review feedback * Update docs * More docs * Make it private * Use `None` * Use apply transaction * Update primitives/state-machine/src/testing.rs Co-authored-by: cheme <emericchevalier.pro@gmail.com> Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> Co-authored-by: RK <r.raajey@gmail.com> Co-authored-by: Xiliang Chen <xlchen1291@gmail.com> Co-authored-by: Parity Benchmarking Bot <admin@parity.io> Co-authored-by: Kian Paimani <5588131+kianenigma@users.noreply.github.com> Co-authored-by: Guillaume Thiolliere <gui.thiolliere@gmail.com> * Define ss58 prefix inside the runtime (#7810) * Add SS58Prefix type to the frame_system config trait * Remove unused chain_id runtime interface * remove some unecessary bound (#7813) * Fix ss58check test when executed with other tests (#7815) There was a bug that could make other ss58 tests fail when being executed with this one in parallel. This test changes the default ss58 version and if other tests are run at the time the default version is changed, they would fail. To fix this problem, we now run the actual test as a new process. * Cleanup some warnings (#7816) * client: cleanup redundant semicolon warnings * grandpa: remove usage of deprecated compare_and_swap * Happy new year (#7814) * Happy new year Updates the copyright years and fixes wrong license headers. * Fix the template * Split HEADER into HEADER-APACHE & HEADER-GPL * contracts: Allow runtime authors to define a chain extension (#7548) * Make host functions return TrapReason This avoids the need to manually store any trap reasons to the `Runtime` from the host function. This adds the following benefits: * It properly composes with the upcoming chain extensions * Missing to set a trap value is now a compile error * Add chain extension The chain extension is a way for the contract author to add new host functions for contracts to call. * Add tests for chain extensions * Fix regression in set_rent.wat fixture Not all offsets where properly updated when changing the fixtures for the new salt on instantiate. * Pre-charge a weight amount based off the specified length * Improve fn write docs * Renamed state to phantom * Fix typo * *: Update to libp2p v0.33.0 (#7759) * *: Update to libp2p v0.33.0 * client/network: Consistently track request arrival time With https://github.com/libp2p/rust-libp2p/pull/1886/ one is guaranteed to receive either a `ResponseSent` or a `InboundFailure` event for each received inbound request via `RequestResponseEvent::Message`. Given this guarantee there is no need to track arrival times in a best-effort manner and thus there is no need to use a LRU cache for arrival times. * client/offchain: Adjust to PeerId API changes * contracts: Lazy storage removal (#7740) * Do not evict a contract from within a call stack We don't want to trigger contract eviction automatically when a contract is called. This is because those changes can be reverted due to how storage transactions are used at the moment. More Information: https://github.com/paritytech/substrate/issues/6439#issuecomment-648754324 It can be re-introduced once the linked issue is resolved. In the meantime `claim_surcharge` must be called to evict a contract. * Lazily delete storage in on_initialize instead of when removing the contract * Add missing documentation of new error * Make Module::claim_surcharge public It being the only dispatchable that is private is an oversight. * review: Add final newline * review: Simplify assert statement * Add test that checks that partial remove of a contract works * Premote warning to error * Added missing docs for seal_terminate * Lazy deletion should only take AVERAGE_ON_INITIALIZE_RATIO of the block * Added informational about the lazy deletion throughput * Avoid lazy deletion in case the block is already full * Prevent queue decoding in case of an already full block * Add test that checks that on_initialize honors block limits * fix template (#7823) * rename HEADER files so that they are consistent with LICENSE filenames (#7825) * contracts: Prevent contracts from allocating a too large buffer (#7818) * Prevent contracts from allocating a too large buffer * Fix possible integer overflow * Improve error message on where clause on pallet error (#7821) * improve error message on where clause on pallet error * Revert "improve error message on where clause on pallet error" This reverts commit 5a3cc38976813fccef3357833553ce30f5b988ea. * Revert "Revert "improve error message on where clause on pallet error"" This reverts commit e3b3fca6bc4fa89816f80dbcb82dc4536a9b2549. * Feat sp keystore (#7826) * delete not used VRFTranscriptValue * specification variable naming * minor fix (#7828) * Participation Lottery Pallet (#7221) * Basic design * start adding tests * finish tests * clean up crates * use call index for match * finish benchmarks * add to runtime * fix * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs * more efficient storage * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Update lib.rs * Update bin/node/runtime/src/lib.rs * trait -> config * add repeating lottery * new benchmarks * fix build * move trait for warning * feedback from @xlc * add stop_repeat * fix * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs * Support static calls * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs * fix test * add loop to mitigate modulo bias * Update weights for worst case scenario loop * Initialize pot with ED * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_lottery --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/lottery/src/weights.rs --template=./.maintain/frame-weight-template.hbs Co-authored-by: Parity Benchmarking Bot <admin@parity.io> * client/network: Use request response for block requests (#7478) * client/network: Add scaffolding for finality req to use req resp #sc * client/network/src/finality_requests: Remove * client/network/src/behaviour: Pass request id down to sync * client/network: Use request response for block requests * client/network: Move handler logic into *_*_handler.rs * client/network: Track ongoing finality requests in protocol.rs * client/network: Remove commented out finalization initialization * client/network: Add docs for request handlers * client/network/finality_request_handler: Log errors * client/network/block_request_handler: Log errors * client/network: Format * client/network: Handle block request failure * protocols/network: Fix tests * client/network/src/behaviour: Handle request sending errors * client/network: Move response handling into custom method * client/network/protocol: Handle block response errors * client/network/protocol: Remove tracking of obsolete requests * client/network/protocol: Remove block request start time tracking This will be handled generically via request-responses. * client/network/protocol: Refactor on_*_request_started * client/network: Pass protocol config instead of protocol name * client/network: Pass protocol config in tests * client/network/config: Document request response configs * client/network/src/_request_handler: Document protocol config gen * client/network/src/protocol: Document Peer request values * client/network: Rework request response to always use oneshot * client/network: Unified metric reporting for all request protocols * client/network: Move protobuf parsing into protocol.rs * client/network/src/protocol: Return pending events after poll * client/network: Improve error handling and documentation * client/network/behaviour: Remove outdated error types * Update client/network/src/block_request_handler.rs Co-authored-by: Ashley <ashley.ruglys@gmail.com> * Update client/network/src/finality_request_handler.rs Co-authored-by: Ashley <ashley.ruglys@gmail.com> * client/network/protocol: Reduce reputation on timeout * client/network/protocol: Refine reputation changes * client/network/block_request_handler: Set and explain queue length * client/service: Deny block requests when light client * client/service: Fix role matching * client: Enforce line width * client/network/request_responses: Fix unit tests * client/network: Expose time to build response via metrics * client/network/request_responses: Fix early connection closed error * client/network/protocol: Fix line length * client/network/protocol: Disconnect on most request failures * client/network/protocol: Disconnect peer when oneshot is canceled * client/network/protocol: Disconnect peer even when connection closed * client/network/protocol: Remove debugging log line * client/network/request_response: Use Clone::clone for error * client/network/request_response: Remove outdated comment With libp2p v0.33.0 libp2p-request-response properly sends inbound failures on connections being closed. Co-authored-by: Addie Wagenknecht <addie@nortd.com> Co-authored-by: Ashley <ashley.ruglys@gmail.com> * fix : remove `_{ }` syntax from benchmark macro (#7822) * commented use of common * hack to pass tests * another hack * remove all commented code * fix the easy tests * temp hack * follow through comma hack until better solution * patch macro * missed one * update benchmarks * update docs * fix docs * removed too much * fix changes Co-authored-by: Shawn Tabrizi <shawntabrizi@gmail.com> * Improve spans of pallet macro (#7830) * fix spans * convert name to snake case * Fix master build (#7837) * Fix master build * Use correct copyright year * babe: expose next epoch data (#7829) * babe: expose next epoch data * babe: add runtime api for next_epoch * babe: avoid reading next authorities from storage unnecessarily * babe: add notes about epoch duration constraints * babe: guard against overflow * babe: add test for fetching current and next epoch data * contracts: Add configurable per-storage item cost (#7819) * Rework rent parameters * No need for empty_pair_count any longer * Parameterize runtime * upgrade a few dependencies (#7831) * upgrade a few dependencies * make it compile at the expense of duplicate deps * fix web-wasm and a warning * introduce activate-wasm-bindgen-features crate * Revert "introduce activate-wasm-bindgen-features crate" This reverts commit 5a6e41e683f8a4844c0a735dcd08caabb2313f11. * add getrandom feature to sc-consensus-aura * CI: remove squash and fix buildah push (#7841) * Fix incorrect use of syn::exports (#7838) * Fix incorrect use of syn::exports Instead of using `syn::exports` we should import the trait from the quote crate directly. * Use own macro for test cases to fix compilation with latest syn * Fix test * Subkey should not import the entire world. (#7845) There is no reason for subkey to import the default Substrate node to support a feature that would only be usable for the Substrate node. Subkey itself should be more the default key management binary for Substrate related chains. If certain chains require some special functionality, they can easily stick together their own "my-chain-key". * Rework priority groups, take 2 (#7700) * Rework priority groups * Broken tests fix * Fix warning causing CI to fail * [Hack] Try restore backwards-compatibility * Fix peerset bug * Doc fixes and clean up * Error on state mismatch * Try debug CI * CI debugging * [CI debug] Can I please see this line * Revert "[CI debug] Can I please see this line" This reverts commit 4b7cf7c1511f579cd818b21d46bd11642dfac5cb. * Revert "CI debugging" This reverts commit 9011f1f564b860386dc7dd6ffa9fc34ea7107623. * Fix error! which isn't actually an error * Fix Ok() returned when actually Err() * Tweaks and fixes * Fix build * Peerset bugfix * [Debug] Try outbound GrandPa slots * Another bugfix * Revert "[Debug] Try outbound GrandPa slots" This reverts commit d175b9208c088faad77d9f0ce36ff6f48bd92dd3. * [Debug] Try outbound GrandPa slots * Apply suggestions from code review Co-authored-by: Max Inden <mail@max-inden.de> * Use consts for hardcoded peersets * Revert "Try debug CI" This reverts commit 62c4ad5e79c03d561c714a008022ecac463a597e. * Renames * Line widths * Add doc Co-authored-by: Max Inden <mail@max-inden.de> * Better Handle Dead Accounts in Balances (#7843) * Don't mutate storage when account is dead and should stay dead * cargo run --release --features=runtime-benchmarks --manifest-path=bin/node/cli/Cargo.toml -- benchmark --chain=dev --steps=50 --repeat=20 --pallet=pallet_balances --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./frame/balances/src/weights.rs --template=./.maintain/frame-weight-template.hbs * more concrete storage noop Co-authored-by: Parity Benchmarking Bot <admin@parity.io> * bump fs-swap (#7834) * UniArts reserve SS58 address id 38 (#7651) * UniArts reserve SS58 address id 45 * Update ss58-registry.json Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> Co-authored-by: Xiang Li <bigxiang@gmail.com> Co-authored-by: joe petrowski <25483142+joepetrowski@users.noreply.github.com> * Update to futures 0.3.9 (#7854) * Store dispatch info of calls locally in weight calculation (#7849) * utility * sudo * more * recovery * better formatting * client/network: Re-enable light_client_handler.rs unit tests (#7853) * Fix max log level (#7851) With the switch to tracing we did not set the `max_log_level` anymore. This resulted in a performance degradation as logging did not early exited and thus, `trace` logs were at least resolved every time. This pr fixes it by ensuring that we set the correct max log level. * Fix missing network for uniarts (#7859) * contracts: Collect rent for the first block during deployment (#7847) * Pay first rent during instantiation * Fix and add new tests * Do not increment trie id counter on failure * Merge 2.0.1 backport branch into mainline master (#7842) * Backport paritytech/substrate#7381 * Bring back genesis storage build in aura/timestamp To not change spec version, see https://github.com/paritytech/substrate/pull/7686#discussion_r540032743 * Backport paritytech/substrate#7238 * Backport paritytech/substrate#7395 * Bump impl_version * Fix UI tests and bump trybuild dep See https://github.com/rust-lang/rust/pull/73996 Backports: https://github.com/paritytech/substrate/pull/7764 https://github.com/paritytech/substrate/pull/7656 * Partially backport paritytech/substrate#7838 * Release frame-support with a dep compilation fix * Bump patch level for remaining crates This is done because at the time of writing cargo-unleash does not fully support partial workspace publishing and mixes both local and crates.io versions of the packages, leading to errors in the release check workflow. * Backport paritytech/substrate#7854 ...to fix compilation error when using futures-* v0.3.9. * Adding Changelog entry for patch release Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Benjamin Kampmann <ben@parity.io> * Bump cargo-unleash to latest alpha release (#7867) * Bump sha2 from 0.8.2 to 0.9.2 (#7643) * Bump sha2 from 0.8.2 to 0.9.2 Bumps [sha2](https://github.com/RustCrypto/hashes) from 0.8.2 to 0.9.2. - [Release notes](https://github.com/RustCrypto/hashes/releases) - [Commits](https://github.com/RustCrypto/hashes/compare/sha2-v0.8.2...streebog-v0.9.2) Signed-off-by: dependabot[bot] <support@github.com> * Fix compilation error Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> * bumpd minor version (#7873) * Add Prometheus alerts if unbounded channels are too large (#7866) * Add Prometheus alerts if unbounded channels are too large * Tweaks * Bump retain_mut from 0.1.1 to 0.1.2 (#7869) Bumps [retain_mut](https://github.com/upsuper/retain_mut) from 0.1.1 to 0.1.2. - [Release notes](https://github.com/upsuper/retain_mut/releases) - [Commits](https://github.com/upsuper/retain_mut/compare/v0.1.1...v0.1.2) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * babe: initialize next authorities on genesis (#7872) * babe: initialize next authorities on genesis * babe: add test for genesis authorities * Update serde and parity-multiaddr, to fix master CI (#7877) * Add ss58 version prefix for CORD (from Dhiway) (#7862) * Add ss58 version prefix for CORD * Add ss58 version pref…

athei added 2 commits December 16, 2020 13:03

Lazily delete storage in on_initialize instead of when removing the c…

6456a36

…ontract

athei added A0-please_review Pull request needs code review. B3-apinoteworthy C1-low PR touches the given topic and has a low impact on builders. labels Dec 16, 2020

athei requested review from ascjones, NikVolf and gui1117 December 16, 2020 17:13

paritytech-ci added the D2-breaksapi label Dec 16, 2020

athei added 2 commits December 17, 2020 10:57

Add missing documentation of new error

a702a46

Make Module::claim_surcharge public

aa03aa7

It being the only dispatchable that is private is an oversight.

shawntabrizi reviewed Dec 28, 2020

View reviewed changes

frame/contracts/src/storage.rs Outdated Show resolved Hide resolved

shawntabrizi reviewed Dec 28, 2020

View reviewed changes

frame/contracts/src/storage.rs Outdated Show resolved Hide resolved

shawntabrizi reviewed Dec 28, 2020

View reviewed changes

frame/contracts/src/tests.rs Outdated Show resolved Hide resolved

shawntabrizi reviewed Dec 28, 2020

View reviewed changes

frame/contracts/src/tests.rs Show resolved Hide resolved

athei added 3 commits December 29, 2020 12:36

Merge branch 'master' into at-lazy-contract-removal

9eb3a9f

review: Add final newline

e9be4d6

review: Simplify assert statement

d0ccc05

gui1117 reviewed Dec 29, 2020

View reviewed changes

Merge branch 'master' into at-lazy-contract-removal

765c21a

gui1117 approved these changes Dec 30, 2020

View reviewed changes

gui1117 reviewed Dec 30, 2020

View reviewed changes

frame/contracts/src/exec.rs Show resolved Hide resolved

athei added 4 commits December 30, 2020 23:18

Merge branch 'master' into at-lazy-contract-removal

76a54f8

Add test that checks that partial remove of a contract works

efa81bb

Premote warning to error

91850a0

Added missing docs for seal_terminate

91c2d76

gui1117 reviewed Dec 31, 2020

View reviewed changes

frame/contracts/src/lib.rs Outdated Show resolved Hide resolved

athei added 2 commits December 31, 2020 19:33

Lazy deletion should only take AVERAGE_ON_INITIALIZE_RATIO of the block

8f2fc0c

Added informational about the lazy deletion throughput

98d3f28

Avoid lazy deletion in case the block is already full

d5a820c

shawntabrizi approved these changes Jan 1, 2021

View reviewed changes

athei added 3 commits January 2, 2021 17:43

Prevent queue decoding in case of an already full block

57737ff

Add test that checks that on_initialize honors block limits

ed4c7e8

Merge branch 'master' into at-lazy-contract-removal

736dba6

Merge branch 'master' into at-lazy-contract-removal

5c24ce0

athei mentioned this pull request Jan 4, 2021

contracts: Move lazy removal to on_idle #7817

Closed

ghost merged commit fcdba68 into master Jan 4, 2021

ghost deleted the at-lazy-contract-removal branch January 4, 2021 12:35

athei mentioned this pull request Mar 22, 2021

Subtrie API for contracts #5215

Closed

athei mentioned this pull request Sep 27, 2022

Member Request polkadot-fellows/seeding#17

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

contracts: Lazy storage removal #7740

contracts: Lazy storage removal #7740

athei commented Dec 16, 2020

gui1117 Dec 29, 2020

athei Dec 29, 2020

athei Dec 30, 2020

gui1117 Dec 30, 2020

athei Dec 31, 2020

athei Jan 1, 2021

kianenigma Jan 4, 2021

athei Jan 4, 2021

athei Jan 4, 2021

kianenigma Jan 8, 2021 •

edited

Loading

gui1117 left a comment •

edited

Loading

kianenigma commented Jan 4, 2021

athei commented Jan 4, 2021

kianenigma commented Jan 4, 2021

athei commented Jan 4, 2021

ghost commented Jan 4, 2021

ghost commented Jan 4, 2021

athei commented Jan 4, 2021

ghost commented Jan 4, 2021

	/// The maximum number of tries that can be queued for deletion.
	type DeletionQueueDepth: Get<u32>;

	/// The maximum amount of weight that can be consumed per block for lazy trie removal.
	type DeletionWeightLimit: Get<Weight>;

	parameter_types! {
	pub OffencesWeightSoftLimit: Weight = Perbill::from_percent(60) *
	RuntimeBlockWeights::get().max_block;
	}

contracts: Lazy storage removal #7740

contracts: Lazy storage removal #7740

Conversation

athei commented Dec 16, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kianenigma Jan 8, 2021 • edited Loading

Choose a reason for hiding this comment

gui1117 left a comment • edited Loading

Choose a reason for hiding this comment

kianenigma commented Jan 4, 2021

athei commented Jan 4, 2021

kianenigma commented Jan 4, 2021

athei commented Jan 4, 2021

ghost commented Jan 4, 2021

ghost commented Jan 4, 2021

athei commented Jan 4, 2021

ghost commented Jan 4, 2021

kianenigma Jan 8, 2021 •

edited

Loading

gui1117 left a comment •

edited

Loading