Skip to content

Commit

Permalink
General Message Queue Pallet (paritytech#12485)
Browse files Browse the repository at this point in the history 
* The message queue

* Make fully generic

* Refactor

* Docs

* Refactor

* Use iter not slice

* Per-origin queues

* Multi-queue processing

* Introduce MaxReady

* Remove MaxReady in favour of ready ring

* Cleanups

* ReadyRing and tests

* Stale page reaping

* from_components -> from_parts

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Move WeightCounter to sp_weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add MockedWeightInfo

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Deploy to kitchensink

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use WeightCounter

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Small fixes and logging

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add service_page

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Typo

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Move service_page below service_queue

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add service_message

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use correct weight function

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Overweight execution

* Refactor

* Missing file

* Fix WeightCounter usage in scheduler

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix peek_index

Take into account that decoding from a mutable slice modifies it.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add tests and bench service_page_item

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add debug_info

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add no-progress check to service_queues

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add more benches

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Bound from_message and try_append_message

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add PageReaped event

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Rename BookStateOf and BookStateFor

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update tests and remove logging

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove redundant per-message origins; add footprint() and sweep_queue()

* Move testing stuff to mock.rs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add integration test

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix no-progress check

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix debug_info

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup merge and tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix footprint tracking

* Introduce

* Formatting

* OverweightEnqueued event, auto-servicing config item

* Update tests and benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Provide change handler

* Add missing BookStateFor::insert and call QueueChangeHandler

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Docs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update benchmarks and weights

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More tests...

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Use weight metering functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* weightInfo::process_message_payload is gone

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add defensive_saturating_accrue

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Rename WeightCounter to WeightMeter

Ctr+Shift+H should do the trick.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test on_initialize

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add module docs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove origin from MaxMessageLen

The message origin is not encoded into the heap and does
therefore not influence the max message length anymore.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add BoundedVec::as_slice

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test Page::{from_message, try_append_message}

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fixup docs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Docs

* Do nothing in sweep_queue if the queue does not exist

... otherwise it inserts default values into the storage.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test ring (un)knitting

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Upgrade stress-test

Change the test to not assume that all queued messages will be
processed in the next block but split it over multiple.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More tests...

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Beauty fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Rename BoundedVec::as_slice to as_bounded_slice

Conflicts with deref().as_slice() otherwise.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix imports

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove ReadyRing struct

Was used for testing only. Instead use 'fn assert_ring' which also
check the service head and backlinks.

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Beauty fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix stale page watermark

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix test feature and clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* QueueChanged handler is called correctly

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update benches

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Abstract testing functions

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Cleanup

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Clippy

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Simplify tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make stuff compile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Extend overweight execution benchmark

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove TODOs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Test service queue with faulty MessageProcessor

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* fmt

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Update pallet ui tests to 1.65

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* More docs

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Review doc fixes

Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Add weight_limit to extrinsic weight of execute_overweight

* Correctly return unused weight

* Return actual weight consumed in do_execute_overweight

* Review fixes

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Set version 7.0.0-dev

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make it compile

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Switch message_size to u64

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Switch message_count to u64

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Fix benchmarks

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Make CI green

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Docs

* Update tests

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* ".git/.scripts/bench-bot.sh" pallet dev pallet_message_queue

* Dont mention README.md in the Cargo.toml

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>

* Remove reference to readme

Signed-off-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: Oliver Tale-Yazdi <oliver.tale-yazdi@parity.io>
Co-authored-by: parity-processbot <>
Co-authored-by: Robert Klotzner <eskimor@users.noreply.github.com>
Co-authored-by: Keith Yeung <kungfukeith11@gmail.com>
  • Loading branch information
@gavofyork @ggwpez
@eskimor @KiChjang @ark0f
4 people authored and ark0f committed Feb 27, 2023
1 parent 5b9b5b8 commit dcbd710
Show file tree
Hide file tree
Showing 20 changed files with 3,883 additions and 15 deletions.
40 changes: 32 additions & 8 deletions Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ members = [
"frame/offences/benchmarking",
"frame/preimage",
"frame/proxy",
"frame/message-queue",
"frame/nomination-pools",
"frame/nomination-pools/fuzzer",
"frame/nomination-pools/benchmarking",
Expand Down
4 changes: 4 additions & 0 deletions bin/node/runtime/Cargo.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ pallet-indices = { version = "4.0.0-dev", default-features = false, path = "../.
pallet-identity = { version = "4.0.0-dev", default-features = false, path = "../../../frame/identity" }
pallet-lottery = { version = "4.0.0-dev", default-features = false, path = "../../../frame/lottery" }
pallet-membership = { version = "4.0.0-dev", default-features = false, path = "../../../frame/membership" }
pallet-message-queue = { version = "7.0.0-dev", default-features = false, path = "../../../frame/message-queue" }
pallet-mmr = { version = "4.0.0-dev", default-features = false, path = "../../../frame/merkle-mountain-range" }
pallet-multisig = { version = "4.0.0-dev", default-features = false, path = "../../../frame/multisig" }
pallet-nomination-pools = { version = "1.0.0", default-features = false, path = "../../../frame/nomination-pools"}
Expand Down Expand Up @@ -150,6 +151,7 @@ std = [
"sp-inherents/std",
"pallet-lottery/std",
"pallet-membership/std",
"pallet-message-queue/std",
"pallet-mmr/std",
"pallet-multisig/std",
"pallet-nomination-pools/std",
Expand Down Expand Up @@ -229,6 +231,7 @@ runtime-benchmarks = [
"pallet-indices/runtime-benchmarks",
"pallet-lottery/runtime-benchmarks",
"pallet-membership/runtime-benchmarks",
"pallet-message-queue/runtime-benchmarks",
"pallet-mmr/runtime-benchmarks",
"pallet-multisig/runtime-benchmarks",
"pallet-nomination-pools-benchmarking/runtime-benchmarks",
Expand Down Expand Up @@ -282,6 +285,7 @@ try-runtime = [
"pallet-identity/try-runtime",
"pallet-lottery/try-runtime",
"pallet-membership/try-runtime",
"pallet-message-queue/try-runtime",
"pallet-mmr/try-runtime",
"pallet-multisig/try-runtime",
"pallet-nomination-pools/try-runtime",
Expand Down
21 changes: 21 additions & 0 deletions bin/node/runtime/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1135,6 +1135,25 @@ impl pallet_bounties::Config for Runtime {
type ChildBountyManager = ChildBounties;
}

parameter_types! {
/// Allocate at most 20% of each block for message processing.
///
/// Is set to 20% since the scheduler can already consume a maximum of 80%.
pub MessageQueueServiceWeight: Option<Weight> = Some(Perbill::from_percent(20) * RuntimeBlockWeights::get().max_block);
}

impl pallet_message_queue::Config for Runtime {
type RuntimeEvent = RuntimeEvent;
type WeightInfo = ();
/// NOTE: Always set this to `NoopMessageProcessor` for benchmarking.
type MessageProcessor = pallet_message_queue::mock_helpers::NoopMessageProcessor;
type Size = u32;
type QueueChangeHandler = ();
type HeapSize = ConstU32<{ 64 * 1024 }>;
type MaxStale = ConstU32<128>;
type ServiceWeight = MessageQueueServiceWeight;
}

parameter_types! {
pub const ChildBountyValueMinimum: Balance = 1 * DOLLARS;
}
Expand Down Expand Up @@ -1699,6 +1718,7 @@ construct_runtime!(
RankedPolls: pallet_referenda::<Instance2>,
RankedCollective: pallet_ranked_collective,
FastUnstake: pallet_fast_unstake,
MessageQueue: pallet_message_queue,
}
);

Expand Down Expand Up @@ -1793,6 +1813,7 @@ mod benches {
[pallet_indices, Indices]
[pallet_lottery, Lottery]
[pallet_membership, TechnicalMembership]
[pallet_message_queue, MessageQueue]
[pallet_mmr, Mmr]
[pallet_multisig, Multisig]
[pallet_nomination_pools, NominationPoolsBench::<Runtime>]
Expand Down
53 changes: 53 additions & 0 deletions frame/message-queue/Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
[package]
authors = ["Parity Technologies <admin@parity.io>"]
edition = "2021"
name = "pallet-message-queue"
version = "7.0.0-dev"
license = "Apache-2.0"
homepage = "https://substrate.io"
repository = "https://github.com/paritytech/substrate/"
description = "FRAME pallet to queue and process messages"

[dependencies]
codec = { package = "parity-scale-codec", version = "3.0.0", default-features = false, features = ["derive"] }
scale-info = { version = "2.1.2", default-features = false, features = ["derive"] }
serde = { version = "1.0.137", optional = true, features = ["derive"] }
log = { version = "0.4.17", default-features = false }

sp-core = { version = "7.0.0", default-features = false, path = "../../primitives/core" }
sp-io = { version = "7.0.0", default-features = false, path = "../../primitives/io" }
sp-runtime = { version = "7.0.0", default-features = false, path = "../../primitives/runtime" }
sp-std = { version = "5.0.0", default-features = false, path = "../../primitives/std" }
sp-arithmetic = { version = "6.0.0", default-features = false, path = "../../primitives/arithmetic" }
sp-weights = { version = "4.0.0", default-features = false, path = "../../primitives/weights" }

frame-benchmarking = { version = "4.0.0-dev", default-features = false, optional = true, path = "../benchmarking" }
frame-support = { version = "4.0.0-dev", default-features = false, path = "../support" }
frame-system = { version = "4.0.0-dev", default-features = false, path = "../system" }

[dev-dependencies]
sp-tracing = { version = "6.0.0", path = "../../primitives/tracing" }
rand = "0.8.5"
rand_distr = "0.4.3"

[features]
default = ["std"]
std = [
"codec/std",
"scale-info/std",
"sp-core/std",
"sp-io/std",
"sp-runtime/std",
"sp-std/std",
"sp-arithmetic/std",
"sp-weights/std",
"frame-benchmarking?/std",
"frame-support/std",
"frame-system/std",
]
runtime-benchmarks = [
"frame-benchmarking/runtime-benchmarks",
"frame-support/runtime-benchmarks",
"frame-system/runtime-benchmarks",
]
try-runtime = ["frame-support/try-runtime"]
Loading

0 comments on commit dcbd710

Please sign in to comment.