Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce subsystem benchmarking tool #2528

Merged
merged 56 commits into from
Dec 14, 2023
Merged

Conversation

sandreim
Copy link
Contributor

@sandreim sandreim commented Nov 28, 2023

This tool makes it easy to run parachain consensus stress/performance testing on your development machine or in CI.

Motivation

The parachain consensus node implementation spans across many modules which we call subsystems. Each subsystem is responsible for a small part of logic of the parachain consensus pipeline, but in general the most load and performance issues are localized in just a few core subsystems like availability-recovery, approval-voting or dispute-coordinator. In the absence of such a tool, we would run large test nets to load/stress test these parts of the system. Setting up and making sense of the amount of data produced by such a large test is very expensive, hard to orchestrate and is a huge development time sink.

PR contents

  • CLI tool
  • Data Availability Read test
  • reusable mockups and components needed so far
  • Documentation on how to get started

Data Availability Read test

An overseer is built with using a real availability-recovery susbsytem instance while dependent subsystems like av-store, network-bridge and runtime-api are mocked. The network bridge will emulate all the network peers and their answering to requests.

The test is going to be run for a number of blocks. For each block it will generate send a “RecoverAvailableData” request for an arbitrary number of candidates. We wait for the subsystem to respond to all requests before moving to the next block.
At the same time we collect the usual subsystem metrics and task CPU metrics and show some nice progress reports while running.

Here is how the CLI looks like:

[2023-11-28T13:06:27Z INFO  subsystem_bench::core::display] n_validators = 1000, n_cores = 20, pov_size = 5120 - 5120, error = 3, latency = Some(PeerLatency { min_latency: 1ms, max_latency: 100ms })
[2023-11-28T13:06:27Z INFO  subsystem-bench::availability] Generating template candidate index=0 pov_size=5242880
[2023-11-28T13:06:27Z INFO  subsystem-bench::availability] Created test environment.
[2023-11-28T13:06:27Z INFO  subsystem-bench::availability] Pre-generating 60 candidates.
[2023-11-28T13:06:30Z INFO  subsystem-bench::core] Initializing network emulation for 1000 peers.
[2023-11-28T13:06:30Z INFO  subsystem-bench::availability] Current block 1/3
[2023-11-28T13:06:30Z INFO  substrate_prometheus_endpoint] 〽️ Prometheus exporter started at 127.0.0.1:9999
[2023-11-28T13:06:30Z INFO  subsystem_bench::availability] 20 recoveries pending
[2023-11-28T13:06:37Z INFO  subsystem_bench::availability] Block time 6262ms
[2023-11-28T13:06:37Z INFO  subsystem-bench::availability] Sleeping till end of block (0ms)
[2023-11-28T13:06:37Z INFO  subsystem-bench::availability] Current block 2/3
[2023-11-28T13:06:37Z INFO  subsystem_bench::availability] 20 recoveries pending
[2023-11-28T13:06:43Z INFO  subsystem_bench::availability] Block time 6369ms
[2023-11-28T13:06:43Z INFO  subsystem-bench::availability] Sleeping till end of block (0ms)
[2023-11-28T13:06:43Z INFO  subsystem-bench::availability] Current block 3/3
[2023-11-28T13:06:43Z INFO  subsystem_bench::availability] 20 recoveries pending
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] Block time 6194ms
[2023-11-28T13:06:49Z INFO  subsystem-bench::availability] Sleeping till end of block (0ms)
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] All blocks processed in 18829ms
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] Throughput: 102400 KiB/block
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] Block time: 6276 ms
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] 
    
    Total received from network: 415 MiB
    Total sent to network: 724 KiB
    Total subsystem CPU usage 24.00s
    CPU usage per block 8.00s
    Total test environment CPU usage 0.15s
    CPU usage per block 0.05s

Prometheus/Grafana stack in action

Screenshot 2023-11-28 at 15 11 10 Screenshot 2023-11-28 at 15 12 01 Screenshot 2023-11-28 at 15 12 38

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
@sandreim sandreim merged commit 8a6e9ef into master Dec 14, 2023
114 of 116 checks passed
@sandreim sandreim deleted the sandreim/subsystem-bench branch December 14, 2023 10:57
paritytech-ci pushed a commit that referenced this pull request Dec 14, 2023
This tool makes it easy to run parachain consensus stress/performance
testing on your development machine or in CI.

## Motivation
The parachain consensus node implementation spans across many modules
which we call subsystems. Each subsystem is responsible for a small part
of logic of the parachain consensus pipeline, but in general the most
load and performance issues are localized in just a few core subsystems
like `availability-recovery`, `approval-voting` or
`dispute-coordinator`. In the absence of such a tool, we would run large
test nets to load/stress test these parts of the system. Setting up and
making sense of the amount of data produced by such a large test is very
expensive, hard to orchestrate and is a huge development time sink.

## PR contents
- CLI tool 
- Data Availability Read test
- reusable mockups and components needed so far
- Documentation on how to get started

### Data Availability Read test

An overseer is built with using a real `availability-recovery` susbsytem
instance while dependent subsystems like `av-store`, `network-bridge`
and `runtime-api` are mocked. The network bridge will emulate all the
network peers and their answering to requests.

The test is going to be run for a number of blocks. For each block it
will generate send a “RecoverAvailableData” request for an arbitrary
number of candidates. We wait for the subsystem to respond to all
requests before moving to the next block.
At the same time we collect the usual subsystem metrics and task CPU
metrics and show some nice progress reports while running.

### Here is how the CLI looks like:

```
[2023-11-28T13:06:27Z INFO  subsystem_bench::core::display] n_validators = 1000, n_cores = 20, pov_size = 5120 - 5120, error = 3, latency = Some(PeerLatency { min_latency: 1ms, max_latency: 100ms })
[2023-11-28T13:06:27Z INFO  subsystem-bench::availability] Generating template candidate index=0 pov_size=5242880
[2023-11-28T13:06:27Z INFO  subsystem-bench::availability] Created test environment.
[2023-11-28T13:06:27Z INFO  subsystem-bench::availability] Pre-generating 60 candidates.
[2023-11-28T13:06:30Z INFO  subsystem-bench::core] Initializing network emulation for 1000 peers.
[2023-11-28T13:06:30Z INFO  subsystem-bench::availability] Current block 1/3
[2023-11-28T13:06:30Z INFO  substrate_prometheus_endpoint] 〽️ Prometheus exporter started at 127.0.0.1:9999
[2023-11-28T13:06:30Z INFO  subsystem_bench::availability] 20 recoveries pending
[2023-11-28T13:06:37Z INFO  subsystem_bench::availability] Block time 6262ms
[2023-11-28T13:06:37Z INFO  subsystem-bench::availability] Sleeping till end of block (0ms)
[2023-11-28T13:06:37Z INFO  subsystem-bench::availability] Current block 2/3
[2023-11-28T13:06:37Z INFO  subsystem_bench::availability] 20 recoveries pending
[2023-11-28T13:06:43Z INFO  subsystem_bench::availability] Block time 6369ms
[2023-11-28T13:06:43Z INFO  subsystem-bench::availability] Sleeping till end of block (0ms)
[2023-11-28T13:06:43Z INFO  subsystem-bench::availability] Current block 3/3
[2023-11-28T13:06:43Z INFO  subsystem_bench::availability] 20 recoveries pending
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] Block time 6194ms
[2023-11-28T13:06:49Z INFO  subsystem-bench::availability] Sleeping till end of block (0ms)
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] All blocks processed in 18829ms
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] Throughput: 102400 KiB/block
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] Block time: 6276 ms
[2023-11-28T13:06:49Z INFO  subsystem_bench::availability] 
    
    Total received from network: 415 MiB
    Total sent to network: 724 KiB
    Total subsystem CPU usage 24.00s
    CPU usage per block 8.00s
    Total test environment CPU usage 0.15s
    CPU usage per block 0.05s
```

### Prometheus/Grafana stack in action
<img width="1246" alt="Screenshot 2023-11-28 at 15 11 10"
src="https://github.com/paritytech/polkadot-sdk/assets/54316454/eaa47422-4a5e-4a3a-aaef-14ca644c1574">
<img width="1246" alt="Screenshot 2023-11-28 at 15 12 01"
src="https://github.com/paritytech/polkadot-sdk/assets/54316454/237329d6-1710-4c27-8f67-5fb11d7f66ea">
<img width="1246" alt="Screenshot 2023-11-28 at 15 12 38"
src="https://github.com/paritytech/polkadot-sdk/assets/54316454/a07119e8-c9f1-4810-a1b3-f1b7b01cf357">

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
bkontur added a commit that referenced this pull request Dec 18, 2023
68d8650 Bump thiserror from 1.0.50 to 1.0.51
009c989 remove no longer valid check from the ensure_weights_are_correct (#2740)
94c44a7 Added Rococo BH <> Rococo Bulletin bridge (#2724)
5fe0f2f Bump tokio from 1.34.0 to 1.35.0
25f8251 Grafana update stuff (#2733)
06fbe8b Improved `ExportXcm::validate` implementation for BridgeHubs - step 1 (#2727)
390e836 Select header that will be fully refunded in on-demand batch finality relay (#2729)
ce701dd separate constants for average and worst case relay headers (#2728)
09215c5 Backport from `polkadot-sdk` + bump (#2725)
6327261 Bump serde from 1.0.192 to 1.0.193
fff9ddd Bump sysinfo from 0.29.10 to 0.29.11
4be99fe Monitoring and alerts for Rococo/Westend (#2710)
67a683a Bump ed25519-dalek from 2.0.0 to 2.1.0
8e0e794 quick and dirty fix for the `wait -p` and older distros (#2712)
3ab6562 Add withdraw reserve assets to zombienet tests (#2711)
c2c409b increase init timeouts in zombienet tests (#2706)
a8c60b4 fix lane id and bridged chain id (#2705)
9ac0f26 removed bp-asset-hub-kusama and bp-asset-hub-polkadot (#2703)
4916475 Some fixes for zombienet tests (polkadot-staging) (#2704)
6f9a147 zombienet from Wococo to Westend (#2699)
3ba7910 Porting changes from polkadot-sdk to polkadot-staging - before update subtree with removed wococo stuff (#2696)
653448f Remove Woococo related stuff (#2692)
03aaab2 Gitspiegel polkadot staging (#2695)
702a4c1 Drop Rialto <> Millau bridges (#2663) (#2694)
6a63b5f Start version guards for the ED loop (#2678)
896b9a9 typo (#2690)
671d27c Bump serde from 1.0.190 to 1.0.192
991b229 Bump clap from 4.4.7 to 4.4.8
ec267ec Bump env_logger from 0.10.0 to 0.10.1
592e407 Bump tokio from 1.33.0 to 1.34.0
c49ce3d Bump serde_json from 1.0.107 to 1.0.108
04b3319 Update subxt-codegen version (#2674)
03f9804 backport #2139 (#2673)
49245dd removed unused PARACHAINS_FINALITY_PALLET_NAME constant (#2670)
658a3f5 BHR/BHWE spec_version according to the `polkadot-sdk` (#2668)
7666b94 Nit from `polkadot-sdk` (#2665)
b5c43bb Adjusted constant because for measuring we used mistakenly rococo constants (#2664)
062449d Add Rococo<>Westend bridge support/relay (#2647)
55eb44e Add basic zombienet test to be used in the future (#2649) (#2660)
93b6b3f Bump clap from 4.4.6 to 4.4.7
4c01ab0 Bump futures from 0.3.28 to 0.3.29
a31a6c0 Bump tempfile from 3.8.0 to 3.8.1
bcdfe83 Bump serde from 1.0.189 to 1.0.190
f7433b0 Port #2648 to polkadot-staging (#2651)
3896738 Bump scale-info from 2.9.0 to 2.10.0
12d62c5 Bump thiserror from 1.0.49 to 1.0.50
1d78aa1 Backport from `polkadot-sdk` with actual master (#2633)
ab4de94 Grandpa justifications: Avoid duplicate vote ancestries (#2634) (#2635)
465562a add missing crate descriptions (#2629)
28d3680 Bump fixed-hash
67528c4 Bump serde from 1.0.188 to 1.0.189
d450c47 Bump time from 0.3.29 to 0.3.30
6a19f83 Bump async-trait from 0.1.73 to 0.1.74
a92d213 Millau, Rialto: accept equivocation reports (#2614) (#2617)
a61f777 Bump tokio from 1.32.0 to 1.33.0
0052f64 Bump subxt from 0.32.0 to 0.32.1
ccc849d Bump num-traits from 0.2.16 to 0.2.17
22f2752 apply late suggestions for #2600 (#2603)
0320172 actualize check_obsolete_call comment (#2601)
5cbbd25 Reject transactions if bridge pallets are halted (#2600)
ca4dfe3 Bump subxt from 0.31.0 to 0.32.0
8bf7b58 Bump clap from 4.4.4 to 4.4.6
88b0b99 Bump thiserror from 1.0.48 to 1.0.49
263833b https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3833103 (#2589)
4f44968 Backport changes from polkadot-sdk (#2588)
7200ed1 fiox overflow when computing priority boost (#2587)
e02cbd3 Bump time from 0.3.28 to 0.3.29
a097dd2 Bump clap from 4.4.3 to 4.4.4
801ce88 Merge bulletin chain changes into polkadot staging (#2574)
a3803ce Add unit tests for the equivocation detection loop (#2571)
26dfc31 Bump clap from 4.4.2 to 4.4.3
66a8beb Bump serde_json from 1.0.106 to 1.0.107
18c50da Bump trie-db from 0.27.1 to 0.28.0
4c4fa92 Equivocation detection loop: Reorganize block checking logic as state machine (#2555) (#2557)
6bd317a Bump serde_json from 1.0.105 to 1.0.106
a7e6bfd Backport for polkadot-sdk#1446 (#2546)
d9f8050 Bump sysinfo from 0.29.9 to 0.29.10
901f44c Bump thiserror from 1.0.47 to 1.0.48
82eeb50 Bump sysinfo from 0.29.8 to 0.29.9
a0c934b Bump strum from 0.24.1 to 0.25.0
1064fbf Bump subxt from 0.28.0 to 0.31.0
e50398d bridges subtree fixes (#2528)
99af075 Markdown linter (#1309) (#2526)
733ff0f `polkadot-staging` branch: Use polkadot-sdk dependencies (#2524)
e8a59f1 Fix benchmark with new XCM::V3 `MAX_INSTRUCTIONS_TO_DECODE` (#2514)
62b185d Backport `polkadot-sdk` changes to `polkadot-staging` (#2518)
d9658f4 Fix equivocation detection containers startup (#2516) (#2517)
d65db28 Backport: building images from locally built binaries (#2513)
5fdbaf4 Start the equivocation detection loop from the complex relayer (#2507) (#2512)
7fbb67d Backport: Implement basic equivocations detection loop (#2375)
cb7efe2 Manually update deps in polkadot staging (#2371)
d17981f #2351 to polkadot-staging (#2359)

git-subtree-dir: bridges
git-subtree-split: 68d8650
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Jan 19, 2024
2755d51 [polkadot-staging] Backport polkadot-sdk changes + update refs (paritytech#2798)
dfec8bc [`polkadot-staging`] Separate `send_message()` validations (paritytech#2795)
c3e469e Bump thiserror from 1.0.55 to 1.0.56
8fb8fdd Port safe commits from master to polkadot-staging (paritytech#2761)
6e526c3 [polkadot-staging] Bump sysinfo from 0.29.11 to 0.30.5 (paritytech#2790)
c3436b2 Bump anyhow from 1.0.78 to 1.0.79
d618416 Bump async-trait from 0.1.76 to 0.1.77
d3f97a4 Bump serde_json from 1.0.109 to 1.0.111
688281f Bump thiserror from 1.0.53 to 1.0.55
d26dd51 get rid of LaneMessageVerifier (paritytech#2168) (paritytech#2764)
e189156 Bump serde_json from 1.0.108 to 1.0.109
e816e13 Bump tempfile from 3.8.1 to 3.9.0
1cd5db7 Bump async-trait from 0.1.75 to 0.1.76
d9cf492 Bump anyhow from 1.0.76 to 1.0.78
eabed3f Bump thiserror from 1.0.51 to 1.0.53
5011481 Bump async-trait from 0.1.74 to 0.1.75
64e635d Bump tokio from 1.35.0 to 1.35.1
c68b2a2 Bump futures from 0.3.29 to 0.3.30
114ad74 Bump anyhow from 1.0.75 to 1.0.76
afd1787 Bump time from 0.3.30 to 0.3.31
ea3d0cd Pokladot sdk backport to staging (paritytech#2746)
68d8650 Bump thiserror from 1.0.50 to 1.0.51
009c989 remove no longer valid check from the ensure_weights_are_correct (paritytech#2740)
94c44a7 Added Rococo BH <> Rococo Bulletin bridge (paritytech#2724)
5fe0f2f Bump tokio from 1.34.0 to 1.35.0
25f8251 Grafana update stuff (paritytech#2733)
06fbe8b Improved `ExportXcm::validate` implementation for BridgeHubs - step 1 (paritytech#2727)
390e836 Select header that will be fully refunded in on-demand batch finality relay (paritytech#2729)
ce701dd separate constants for average and worst case relay headers (paritytech#2728)
09215c5 Backport from `polkadot-sdk` + bump (paritytech#2725)
6327261 Bump serde from 1.0.192 to 1.0.193
fff9ddd Bump sysinfo from 0.29.10 to 0.29.11
4be99fe Monitoring and alerts for Rococo/Westend (paritytech#2710)
67a683a Bump ed25519-dalek from 2.0.0 to 2.1.0
8e0e794 quick and dirty fix for the `wait -p` and older distros (paritytech#2712)
3ab6562 Add withdraw reserve assets to zombienet tests (paritytech#2711)
c2c409b increase init timeouts in zombienet tests (paritytech#2706)
a8c60b4 fix lane id and bridged chain id (paritytech#2705)
9ac0f26 removed bp-asset-hub-kusama and bp-asset-hub-polkadot (paritytech#2703)
4916475 Some fixes for zombienet tests (polkadot-staging) (paritytech#2704)
6f9a147 zombienet from Wococo to Westend (paritytech#2699)
3ba7910 Porting changes from polkadot-sdk to polkadot-staging - before update subtree with removed wococo stuff (paritytech#2696)
653448f Remove Woococo related stuff (paritytech#2692)
03aaab2 Gitspiegel polkadot staging (paritytech#2695)
702a4c1 Drop Rialto <> Millau bridges (paritytech#2663) (paritytech#2694)
6a63b5f Start version guards for the ED loop (paritytech#2678)
896b9a9 typo (paritytech#2690)
671d27c Bump serde from 1.0.190 to 1.0.192
991b229 Bump clap from 4.4.7 to 4.4.8
ec267ec Bump env_logger from 0.10.0 to 0.10.1
592e407 Bump tokio from 1.33.0 to 1.34.0
c49ce3d Bump serde_json from 1.0.107 to 1.0.108
04b3319 Update subxt-codegen version (paritytech#2674)
03f9804 backport paritytech#2139 (paritytech#2673)
49245dd removed unused PARACHAINS_FINALITY_PALLET_NAME constant (paritytech#2670)
658a3f5 BHR/BHWE spec_version according to the `polkadot-sdk` (paritytech#2668)
7666b94 Nit from `polkadot-sdk` (paritytech#2665)
b5c43bb Adjusted constant because for measuring we used mistakenly rococo constants (paritytech#2664)
062449d Add Rococo<>Westend bridge support/relay (paritytech#2647)
55eb44e Add basic zombienet test to be used in the future (paritytech#2649) (paritytech#2660)
93b6b3f Bump clap from 4.4.6 to 4.4.7
4c01ab0 Bump futures from 0.3.28 to 0.3.29
a31a6c0 Bump tempfile from 3.8.0 to 3.8.1
bcdfe83 Bump serde from 1.0.189 to 1.0.190
f7433b0 Port paritytech#2648 to polkadot-staging (paritytech#2651)
3896738 Bump scale-info from 2.9.0 to 2.10.0
12d62c5 Bump thiserror from 1.0.49 to 1.0.50
1d78aa1 Backport from `polkadot-sdk` with actual master (paritytech#2633)
ab4de94 Grandpa justifications: Avoid duplicate vote ancestries (paritytech#2634) (paritytech#2635)
465562a add missing crate descriptions (paritytech#2629)
28d3680 Bump fixed-hash
67528c4 Bump serde from 1.0.188 to 1.0.189
d450c47 Bump time from 0.3.29 to 0.3.30
6a19f83 Bump async-trait from 0.1.73 to 0.1.74
a92d213 Millau, Rialto: accept equivocation reports (paritytech#2614) (paritytech#2617)
a61f777 Bump tokio from 1.32.0 to 1.33.0
0052f64 Bump subxt from 0.32.0 to 0.32.1
ccc849d Bump num-traits from 0.2.16 to 0.2.17
22f2752 apply late suggestions for paritytech#2600 (paritytech#2603)
0320172 actualize check_obsolete_call comment (paritytech#2601)
5cbbd25 Reject transactions if bridge pallets are halted (paritytech#2600)
ca4dfe3 Bump subxt from 0.31.0 to 0.32.0
8bf7b58 Bump clap from 4.4.4 to 4.4.6
88b0b99 Bump thiserror from 1.0.48 to 1.0.49
263833b https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3833103 (paritytech#2589)
4f44968 Backport changes from polkadot-sdk (paritytech#2588)
7200ed1 fiox overflow when computing priority boost (paritytech#2587)
e02cbd3 Bump time from 0.3.28 to 0.3.29
a097dd2 Bump clap from 4.4.3 to 4.4.4
801ce88 Merge bulletin chain changes into polkadot staging (paritytech#2574)
a3803ce Add unit tests for the equivocation detection loop (paritytech#2571)
26dfc31 Bump clap from 4.4.2 to 4.4.3
66a8beb Bump serde_json from 1.0.106 to 1.0.107
18c50da Bump trie-db from 0.27.1 to 0.28.0
4c4fa92 Equivocation detection loop: Reorganize block checking logic as state machine (paritytech#2555) (paritytech#2557)
6bd317a Bump serde_json from 1.0.105 to 1.0.106
a7e6bfd Backport for polkadot-sdk#1446 (paritytech#2546)
d9f8050 Bump sysinfo from 0.29.9 to 0.29.10
901f44c Bump thiserror from 1.0.47 to 1.0.48
82eeb50 Bump sysinfo from 0.29.8 to 0.29.9
a0c934b Bump strum from 0.24.1 to 0.25.0
1064fbf Bump subxt from 0.28.0 to 0.31.0
e50398d bridges subtree fixes (paritytech#2528)
99af075 Markdown linter (paritytech#1309) (paritytech#2526)
733ff0f `polkadot-staging` branch: Use polkadot-sdk dependencies (paritytech#2524)
e8a59f1 Fix benchmark with new XCM::V3 `MAX_INSTRUCTIONS_TO_DECODE` (paritytech#2514)
62b185d Backport `polkadot-sdk` changes to `polkadot-staging` (paritytech#2518)
d9658f4 Fix equivocation detection containers startup (paritytech#2516) (paritytech#2517)
d65db28 Backport: building images from locally built binaries (paritytech#2513)
5fdbaf4 Start the equivocation detection loop from the complex relayer (paritytech#2507) (paritytech#2512)
7fbb67d Backport: Implement basic equivocations detection loop (paritytech#2375)
cb7efe2 Manually update deps in polkadot staging (paritytech#2371)
d17981f paritytech#2351 to polkadot-staging (paritytech#2359)

git-subtree-dir: bridges
git-subtree-split: 2755d51
github-merge-queue bot pushed a commit that referenced this pull request Feb 5, 2024
## Summary
Built on top of the tooling and ideas introduced in
#2528, this PR introduces
a synthetic benchmark for measuring and assessing the performance
characteristics of the approval-voting and approval-distribution
subsystems.

Currently this allows, us to simulate the behaviours of these systems
based on the following dimensions:
```
TestConfiguration:
# Test 1
- objective: !ApprovalsTest
    last_considered_tranche: 89
    min_coalesce: 1
    max_coalesce: 6
    enable_assignments_v2: true
    send_till_tranche: 60
    stop_when_approved: false
    coalesce_tranche_diff: 12
    workdir_prefix: "/tmp"
    num_no_shows_per_candidate: 0
    approval_distribution_expected_tof: 6.0
    approval_distribution_cpu_ms: 3.0
    approval_voting_cpu_ms: 4.30
  n_validators: 500
  n_cores: 100
  n_included_candidates: 100
  min_pov_size: 1120
  max_pov_size: 5120
  peer_bandwidth: 524288000000
  bandwidth: 524288000000
  latency:
    min_latency:
      secs: 0
      nanos: 1000000
    max_latency:
      secs: 0
      nanos: 100000000
  error: 0
  num_blocks: 10
```

## The approach
1. We build a real overseer with the real implementations for
approval-voting and approval-distribution subsystems.
2. For a given network size, for each validator we pre-computed all
potential assignments and approvals it would send, because this a
computation heavy operation this will be cached on a file on disk and be
re-used if the generation parameters don't change.
3. The messages will be sent accordingly to the configured parameters
and those are split into 3 main benchmarking scenarios.

## Benchmarking scenarios

### Best case scenario *approvals_throughput_best_case.yaml*
It send to the approval-distribution only the minimum required tranche
to gathered the needed_approvals, so that a candidate is approved.

### Behaviour in the presence of no-shows *approvals_no_shows.yaml*
It sends the tranche needed to approve a candidate when we have a
maximum of *num_no_shows_per_candidate* tranches with no-shows for each
candidate.

### Maximum throughput *approvals_throughput.yaml*
It sends all the tranches for each block and measures the used CPU and
necessary network bandwidth. by the approval-voting and
approval-distribution subsystem.

## How to run it
```
cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/approvals_throughput.yaml
```

## Evaluating performance
### Use the real subsystems metrics
If you follow the steps in
https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-grafana
for installing locally prometheus and grafana, all real metrics for the
`approval-distribution`, `approval-voting` and overseer are available.
E.g:
<img width="2149" alt="Screenshot 2023-12-05 at 11 07 46"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/cb8ae2dd-178b-4922-bfa4-dc37e572ed38">

<img width="2551" alt="Screenshot 2023-12-05 at 11 09 42"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/8b4542ba-88b9-46f9-9b70-cc345366081b">

<img width="2154" alt="Screenshot 2023-12-05 at 11 10 15"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/b8874d8d-632e-443a-9840-14ad8e90c54f">

<img width="2535" alt="Screenshot 2023-12-05 at 11 10 52"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/779a439f-fd18-4985-bb80-85d5afad78e2">

### Profile with pyroscope
1. Setup pyroscope following the steps in
https://github.com/paritytech/polkadot-sdk/tree/master/polkadot/node/subsystem-bench#install-pyroscope,
then run any of the benchmark scenario with `--profile` as the
arguments.
2. Open the pyroscope dashboard in grafana, e.g:
<img width="2544" alt="Screenshot 2024-01-09 at 17 09 58"
src="https://github.com/paritytech/polkadot-sdk/assets/49718502/58f50c99-a910-4d20-951a-8b16639303d9">



### Useful  logs
1. Network bandwidth requirements:
```
Payload bytes received from peers: 503993 KiB total, 50399 KiB/block
Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block
```

2. Cpu usage by the approval-distribution/approval-voting subsystems.
```
approval-distribution CPU usage 84.061s
approval-distribution CPU usage per block 8.406s
approval-voting CPU usage 96.532s
approval-voting CPU usage per block 9.653s
```

3. Time passed until a given block is approved
```
 Chain selection approved  after 3500 ms hash=0x0101010101010101010101010101010101010101010101010101010101010101
Chain selection approved  after 4500 ms hash=0x0202020202020202020202020202020202020202020202020202020202020202
```

### Using benchmark to quantify improvements from
#1178 +
#1191

Using a versi-node we compare the scenarios where all new optimisations
are disabled with a scenarios where tranche0 assignments are sent in a
single message and a conservative simulation where the coalescing of
approvals gives us just 50% reduction in the number of messages we send.

Overall, what we see is a speedup of around 30-40% in the time it takes
to process the necessary messages and a 30-40% reduction in the
necessary bandwidth.

#### Best case scenario comparison(minimum required tranches sent).
Unoptimised
```
    Number of blocks: 10
    Payload bytes received from peers: 53289 KiB total, 5328 KiB/block
    Payload bytes sent to peers: 52489 KiB total, 5248 KiB/block
    approval-distribution CPU usage 6.732s
    approval-distribution CPU usage per block 0.673s
    approval-voting CPU usage 9.523s
    approval-voting CPU usage per block 0.952s
```

vs Optimisation enabled
```
   Number of blocks: 10
   Payload bytes received from peers: 32141 KiB total, 3214 KiB/block
   Payload bytes sent to peers: 37314 KiB total, 3731 KiB/block
   approval-distribution CPU usage 4.658s
   approval-distribution CPU usage per block 0.466s
   approval-voting CPU usage 6.236s
   approval-voting CPU usage per block 0.624s
```

#### Worst case all tranches sent, very unlikely happens when sharding
breaks.

Unoptimised
```
   Number of blocks: 10
   Payload bytes received from peers: 746393 KiB total, 74639 KiB/block
   Payload bytes sent to peers: 729151 KiB total, 72915 KiB/block
   approval-distribution CPU usage 118.681s
   approval-distribution CPU usage per block 11.868s
   approval-voting CPU usage 124.118s
   approval-voting CPU usage per block 12.412s
```

vs optimised
```
    Number of blocks: 10
    Payload bytes received from peers: 503993 KiB total, 50399 KiB/block
    Payload bytes sent to peers: 629971 KiB total, 62997 KiB/block
    approval-distribution CPU usage 84.061s
    approval-distribution CPU usage per block 8.406s
    approval-voting CPU usage 96.532s
    approval-voting CPU usage per block 9.653s
```


## TODOs
[x] Polish implementation.
[x] Use what we have so far to evaluate
#1191 before merging.
[x] List of features and additional dimensions we want to use for
benchmarking.
[x] Run benchmark on hardware similar with versi and kusama nodes.
[ ] Add benchmark to be run in CI for catching regression in
performance.
[ ] Rebase on latest changes for network emulation.

---------

Signed-off-by: Andrei Sandu <andrei-mihail@parity.io>
Signed-off-by: Alexandru Gheorghe <alexandru.gheorghe@parity.io>
Co-authored-by: Andrei Sandu <andrei-mihail@parity.io>
Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com>
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Mar 26, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Mar 26, 2024
b9acdab polkadot-sdk backport leftovers (paritytech#2896)
1022b6d Relayer v1.2.1 (paritytech#2895)
47b4c48 relayer waits until chain spec version matches the configured in Client constructor/reconnect (paritytech#2894)
a6bac6b Move relay clients to separate folder (paritytech#2893)
4bc73d8 [Backport from `polkadot-sdk`] Move chain definitions to separate folder (paritytech#2892)
8e58eb9 Bump async-trait from 0.1.78 to 0.1.79
28c459b Backport changes from `polkadot-sdk/master` (paritytech#2887)
2a76cbb Move generic CLI logic to different crate (paritytech#2885)
9cb8a2c fixed dependency in runtime-codegen (paritytech#2886)
62372e7 Bump async-trait from 0.1.77 to 0.1.78
61e865b Bump anyhow from 1.0.80 to 1.0.81
d9c5e41 Bump scale-info from 2.10.0 to 2.11.0
9be3e0b Bump thiserror from 1.0.57 to 1.0.58
88900bd backport changes from polkadot-sdk (paritytech#2872)
5c923e0 Bump strum from 0.26.1 to 0.26.2
7e22cc3 Bump sysinfo from 0.30.6 to 0.30.7
3f3d836 Bump env_logger from 0.11.2 to 0.11.3
8b903c6 Prepare P<> K relay release: 1.2.0 (paritytech#2863)
1b0ea65 Document: how to make relayer releases (paritytech#2860)
af8480e Some remaining fixes (mostly insubstantial) for relay (paritytech#2859)
503f64b Bump log from 0.4.20 to 0.4.21
7ee64ec Bump tempfile from 3.10.0 to 3.10.1
38254ac Bump sysinfo from 0.30.5 to 0.30.6
ccf18d6 [Do Not Merge until RBH/WBH upgrade] Relayers use new submit_finality_proof_ex call in Rococo <> Westend bridge (paritytech#2827)
f1c1487 Running relayer: documentation (paritytech#2829)
9f89c7e Bump anyhow from 1.0.79 to 1.0.80
5317ccb Bump serde from 1.0.196 to 1.0.197
d218c7a Bump serde_json from 1.0.113 to 1.0.114
bf7b81a Bump env_logger from 0.11.1 to 0.11.2
8e7b471 Fix relay submitting extra parachain headers during reorg (paritytech#2839)
8a0ed23 Bump thiserror from 1.0.56 to 1.0.57
915628a Bump ed25519-dalek from 2.1.0 to 2.1.1
6c1ed87 Bump tempfile from 3.9.0 to 3.10.0
723f38f Bump num-traits from 0.2.17 to 0.2.18
409922b fix typo (paritytech#2826)
555866b Backport latest PRs from polkadot sdk (paritytech#2824)
d099107 Pass finality proof verification context to the call builder (paritytech#2823)
b3528d2 Add submit_finality_proof_ex call to the GRANDPA pallet (paritytech#2821)
577d2e6 Bump time from 0.3.31 to 0.3.34
b04e82c Bump tokio from 1.35.1 to 1.36.0
b266084 Bump serde_json from 1.0.112 to 1.0.113
dfb0f90 fix clippy
49d1d5e Bump strum from 0.25.0 to 0.26.1
7d92922 Bump serde from 1.0.195 to 1.0.196
2605273 Bump serde_json from 1.0.111 to 1.0.112
b5fa75b Bump env_logger from 0.11.0 to 0.11.1
f116c29 Bump env_logger from 0.10.1 to 0.11.0
2755d51 [polkadot-staging] Backport polkadot-sdk changes + update refs (paritytech#2798)
dfec8bc [`polkadot-staging`] Separate `send_message()` validations (paritytech#2795)
c3e469e Bump thiserror from 1.0.55 to 1.0.56
8fb8fdd Port safe commits from master to polkadot-staging (paritytech#2761)
6e526c3 [polkadot-staging] Bump sysinfo from 0.29.11 to 0.30.5 (paritytech#2790)
c3436b2 Bump anyhow from 1.0.78 to 1.0.79
d618416 Bump async-trait from 0.1.76 to 0.1.77
d3f97a4 Bump serde_json from 1.0.109 to 1.0.111
688281f Bump thiserror from 1.0.53 to 1.0.55
d26dd51 get rid of LaneMessageVerifier (paritytech#2168) (paritytech#2764)
e189156 Bump serde_json from 1.0.108 to 1.0.109
e816e13 Bump tempfile from 3.8.1 to 3.9.0
1cd5db7 Bump async-trait from 0.1.75 to 0.1.76
d9cf492 Bump anyhow from 1.0.76 to 1.0.78
eabed3f Bump thiserror from 1.0.51 to 1.0.53
5011481 Bump async-trait from 0.1.74 to 0.1.75
64e635d Bump tokio from 1.35.0 to 1.35.1
c68b2a2 Bump futures from 0.3.29 to 0.3.30
114ad74 Bump anyhow from 1.0.75 to 1.0.76
afd1787 Bump time from 0.3.30 to 0.3.31
ea3d0cd Pokladot sdk backport to staging (paritytech#2746)
68d8650 Bump thiserror from 1.0.50 to 1.0.51
009c989 remove no longer valid check from the ensure_weights_are_correct (paritytech#2740)
94c44a7 Added Rococo BH <> Rococo Bulletin bridge (paritytech#2724)
5fe0f2f Bump tokio from 1.34.0 to 1.35.0
25f8251 Grafana update stuff (paritytech#2733)
06fbe8b Improved `ExportXcm::validate` implementation for BridgeHubs - step 1 (paritytech#2727)
390e836 Select header that will be fully refunded in on-demand batch finality relay (paritytech#2729)
ce701dd separate constants for average and worst case relay headers (paritytech#2728)
09215c5 Backport from `polkadot-sdk` + bump (paritytech#2725)
6327261 Bump serde from 1.0.192 to 1.0.193
fff9ddd Bump sysinfo from 0.29.10 to 0.29.11
4be99fe Monitoring and alerts for Rococo/Westend (paritytech#2710)
67a683a Bump ed25519-dalek from 2.0.0 to 2.1.0
8e0e794 quick and dirty fix for the `wait -p` and older distros (paritytech#2712)
3ab6562 Add withdraw reserve assets to zombienet tests (paritytech#2711)
c2c409b increase init timeouts in zombienet tests (paritytech#2706)
a8c60b4 fix lane id and bridged chain id (paritytech#2705)
9ac0f26 removed bp-asset-hub-kusama and bp-asset-hub-polkadot (paritytech#2703)
4916475 Some fixes for zombienet tests (polkadot-staging) (paritytech#2704)
6f9a147 zombienet from Wococo to Westend (paritytech#2699)
3ba7910 Porting changes from polkadot-sdk to polkadot-staging - before update subtree with removed wococo stuff (paritytech#2696)
653448f Remove Woococo related stuff (paritytech#2692)
03aaab2 Gitspiegel polkadot staging (paritytech#2695)
702a4c1 Drop Rialto <> Millau bridges (paritytech#2663) (paritytech#2694)
6a63b5f Start version guards for the ED loop (paritytech#2678)
896b9a9 typo (paritytech#2690)
671d27c Bump serde from 1.0.190 to 1.0.192
991b229 Bump clap from 4.4.7 to 4.4.8
ec267ec Bump env_logger from 0.10.0 to 0.10.1
592e407 Bump tokio from 1.33.0 to 1.34.0
c49ce3d Bump serde_json from 1.0.107 to 1.0.108
04b3319 Update subxt-codegen version (paritytech#2674)
03f9804 backport paritytech#2139 (paritytech#2673)
49245dd removed unused PARACHAINS_FINALITY_PALLET_NAME constant (paritytech#2670)
658a3f5 BHR/BHWE spec_version according to the `polkadot-sdk` (paritytech#2668)
7666b94 Nit from `polkadot-sdk` (paritytech#2665)
b5c43bb Adjusted constant because for measuring we used mistakenly rococo constants (paritytech#2664)
062449d Add Rococo<>Westend bridge support/relay (paritytech#2647)
55eb44e Add basic zombienet test to be used in the future (paritytech#2649) (paritytech#2660)
93b6b3f Bump clap from 4.4.6 to 4.4.7
4c01ab0 Bump futures from 0.3.28 to 0.3.29
a31a6c0 Bump tempfile from 3.8.0 to 3.8.1
bcdfe83 Bump serde from 1.0.189 to 1.0.190
f7433b0 Port paritytech#2648 to polkadot-staging (paritytech#2651)
3896738 Bump scale-info from 2.9.0 to 2.10.0
12d62c5 Bump thiserror from 1.0.49 to 1.0.50
1d78aa1 Backport from `polkadot-sdk` with actual master (paritytech#2633)
ab4de94 Grandpa justifications: Avoid duplicate vote ancestries (paritytech#2634) (paritytech#2635)
465562a add missing crate descriptions (paritytech#2629)
28d3680 Bump fixed-hash
67528c4 Bump serde from 1.0.188 to 1.0.189
d450c47 Bump time from 0.3.29 to 0.3.30
6a19f83 Bump async-trait from 0.1.73 to 0.1.74
a92d213 Millau, Rialto: accept equivocation reports (paritytech#2614) (paritytech#2617)
a61f777 Bump tokio from 1.32.0 to 1.33.0
0052f64 Bump subxt from 0.32.0 to 0.32.1
ccc849d Bump num-traits from 0.2.16 to 0.2.17
22f2752 apply late suggestions for paritytech#2600 (paritytech#2603)
0320172 actualize check_obsolete_call comment (paritytech#2601)
5cbbd25 Reject transactions if bridge pallets are halted (paritytech#2600)
ca4dfe3 Bump subxt from 0.31.0 to 0.32.0
8bf7b58 Bump clap from 4.4.4 to 4.4.6
88b0b99 Bump thiserror from 1.0.48 to 1.0.49
263833b https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3833103 (paritytech#2589)
4f44968 Backport changes from polkadot-sdk (paritytech#2588)
7200ed1 fiox overflow when computing priority boost (paritytech#2587)
e02cbd3 Bump time from 0.3.28 to 0.3.29
a097dd2 Bump clap from 4.4.3 to 4.4.4
801ce88 Merge bulletin chain changes into polkadot staging (paritytech#2574)
a3803ce Add unit tests for the equivocation detection loop (paritytech#2571)
26dfc31 Bump clap from 4.4.2 to 4.4.3
66a8beb Bump serde_json from 1.0.106 to 1.0.107
18c50da Bump trie-db from 0.27.1 to 0.28.0
4c4fa92 Equivocation detection loop: Reorganize block checking logic as state machine (paritytech#2555) (paritytech#2557)
6bd317a Bump serde_json from 1.0.105 to 1.0.106
a7e6bfd Backport for polkadot-sdk#1446 (paritytech#2546)
d9f8050 Bump sysinfo from 0.29.9 to 0.29.10
901f44c Bump thiserror from 1.0.47 to 1.0.48
82eeb50 Bump sysinfo from 0.29.8 to 0.29.9
a0c934b Bump strum from 0.24.1 to 0.25.0
1064fbf Bump subxt from 0.28.0 to 0.31.0
e50398d bridges subtree fixes (paritytech#2528)
99af075 Markdown linter (paritytech#1309) (paritytech#2526)
733ff0f `polkadot-staging` branch: Use polkadot-sdk dependencies (paritytech#2524)
e8a59f1 Fix benchmark with new XCM::V3 `MAX_INSTRUCTIONS_TO_DECODE` (paritytech#2514)
62b185d Backport `polkadot-sdk` changes to `polkadot-staging` (paritytech#2518)
d9658f4 Fix equivocation detection containers startup (paritytech#2516) (paritytech#2517)
d65db28 Backport: building images from locally built binaries (paritytech#2513)
5fdbaf4 Start the equivocation detection loop from the complex relayer (paritytech#2507) (paritytech#2512)
7fbb67d Backport: Implement basic equivocations detection loop (paritytech#2375)
cb7efe2 Manually update deps in polkadot staging (paritytech#2371)
d17981f paritytech#2351 to polkadot-staging (paritytech#2359)

git-subtree-dir: bridges
git-subtree-split: b9acdab
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Mar 27, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 2, 2024
bea13eab99a ckb-merkle-mountain-range -> 0.5.2 (paritytech#2911)
8c4c99d11b3 Address migration comments (paritytech#2910)
34817d81593 Some relayer improvments (paritytech#2902)
f7f983c4f8f Bump tokio from 1.36.0 to 1.37.0
051d6ed6446 Bump scale-info from 2.11.0 to 2.11.1
35474455e9a Bump serde_json from 1.0.114 to 1.0.115
956601369a5 Fix polkadot-sdk CI failures (paritytech#2901)
1e4fd28e99c Leftover (paritytech#2900)
a7a47eaebe6 Backport changes from polkadot-sdk (paritytech#2899)
e4e1ea60cfb Remove test dependecies on specific relay clients (paritytech#2898)
b9acdab polkadot-sdk backport leftovers (paritytech#2896)
1022b6d Relayer v1.2.1 (paritytech#2895)
47b4c48 relayer waits until chain spec version matches the configured in Client constructor/reconnect (paritytech#2894)
a6bac6b Move relay clients to separate folder (paritytech#2893)
4bc73d8 [Backport from `polkadot-sdk`] Move chain definitions to separate folder (paritytech#2892)
8e58eb9 Bump async-trait from 0.1.78 to 0.1.79
28c459b Backport changes from `polkadot-sdk/master` (paritytech#2887)
2a76cbb Move generic CLI logic to different crate (paritytech#2885)
9cb8a2c fixed dependency in runtime-codegen (paritytech#2886)
62372e7 Bump async-trait from 0.1.77 to 0.1.78
61e865b Bump anyhow from 1.0.80 to 1.0.81
d9c5e41 Bump scale-info from 2.10.0 to 2.11.0
9be3e0b Bump thiserror from 1.0.57 to 1.0.58
88900bd backport changes from polkadot-sdk (paritytech#2872)
5c923e0 Bump strum from 0.26.1 to 0.26.2
7e22cc3 Bump sysinfo from 0.30.6 to 0.30.7
3f3d836 Bump env_logger from 0.11.2 to 0.11.3
8b903c6 Prepare P<> K relay release: 1.2.0 (paritytech#2863)
1b0ea65 Document: how to make relayer releases (paritytech#2860)
af8480e Some remaining fixes (mostly insubstantial) for relay (paritytech#2859)
503f64b Bump log from 0.4.20 to 0.4.21
7ee64ec Bump tempfile from 3.10.0 to 3.10.1
38254ac Bump sysinfo from 0.30.5 to 0.30.6
ccf18d6 [Do Not Merge until RBH/WBH upgrade] Relayers use new submit_finality_proof_ex call in Rococo <> Westend bridge (paritytech#2827)
f1c1487 Running relayer: documentation (paritytech#2829)
9f89c7e Bump anyhow from 1.0.79 to 1.0.80
5317ccb Bump serde from 1.0.196 to 1.0.197
d218c7a Bump serde_json from 1.0.113 to 1.0.114
bf7b81a Bump env_logger from 0.11.1 to 0.11.2
8e7b471 Fix relay submitting extra parachain headers during reorg (paritytech#2839)
8a0ed23 Bump thiserror from 1.0.56 to 1.0.57
915628a Bump ed25519-dalek from 2.1.0 to 2.1.1
6c1ed87 Bump tempfile from 3.9.0 to 3.10.0
723f38f Bump num-traits from 0.2.17 to 0.2.18
409922b fix typo (paritytech#2826)
555866b Backport latest PRs from polkadot sdk (paritytech#2824)
d099107 Pass finality proof verification context to the call builder (paritytech#2823)
b3528d2 Add submit_finality_proof_ex call to the GRANDPA pallet (paritytech#2821)
577d2e6 Bump time from 0.3.31 to 0.3.34
b04e82c Bump tokio from 1.35.1 to 1.36.0
b266084 Bump serde_json from 1.0.112 to 1.0.113
dfb0f90 fix clippy
49d1d5e Bump strum from 0.25.0 to 0.26.1
7d92922 Bump serde from 1.0.195 to 1.0.196
2605273 Bump serde_json from 1.0.111 to 1.0.112
b5fa75b Bump env_logger from 0.11.0 to 0.11.1
f116c29 Bump env_logger from 0.10.1 to 0.11.0
2755d51 [polkadot-staging] Backport polkadot-sdk changes + update refs (paritytech#2798)
dfec8bc [`polkadot-staging`] Separate `send_message()` validations (paritytech#2795)
c3e469e Bump thiserror from 1.0.55 to 1.0.56
8fb8fdd Port safe commits from master to polkadot-staging (paritytech#2761)
6e526c3 [polkadot-staging] Bump sysinfo from 0.29.11 to 0.30.5 (paritytech#2790)
c3436b2 Bump anyhow from 1.0.78 to 1.0.79
d618416 Bump async-trait from 0.1.76 to 0.1.77
d3f97a4 Bump serde_json from 1.0.109 to 1.0.111
688281f Bump thiserror from 1.0.53 to 1.0.55
d26dd51 get rid of LaneMessageVerifier (paritytech#2168) (paritytech#2764)
e189156 Bump serde_json from 1.0.108 to 1.0.109
e816e13 Bump tempfile from 3.8.1 to 3.9.0
1cd5db7 Bump async-trait from 0.1.75 to 0.1.76
d9cf492 Bump anyhow from 1.0.76 to 1.0.78
eabed3f Bump thiserror from 1.0.51 to 1.0.53
5011481 Bump async-trait from 0.1.74 to 0.1.75
64e635d Bump tokio from 1.35.0 to 1.35.1
c68b2a2 Bump futures from 0.3.29 to 0.3.30
114ad74 Bump anyhow from 1.0.75 to 1.0.76
afd1787 Bump time from 0.3.30 to 0.3.31
ea3d0cd Pokladot sdk backport to staging (paritytech#2746)
68d8650 Bump thiserror from 1.0.50 to 1.0.51
009c989 remove no longer valid check from the ensure_weights_are_correct (paritytech#2740)
94c44a7 Added Rococo BH <> Rococo Bulletin bridge (paritytech#2724)
5fe0f2f Bump tokio from 1.34.0 to 1.35.0
25f8251 Grafana update stuff (paritytech#2733)
06fbe8b Improved `ExportXcm::validate` implementation for BridgeHubs - step 1 (paritytech#2727)
390e836 Select header that will be fully refunded in on-demand batch finality relay (paritytech#2729)
ce701dd separate constants for average and worst case relay headers (paritytech#2728)
09215c5 Backport from `polkadot-sdk` + bump (paritytech#2725)
6327261 Bump serde from 1.0.192 to 1.0.193
fff9ddd Bump sysinfo from 0.29.10 to 0.29.11
4be99fe Monitoring and alerts for Rococo/Westend (paritytech#2710)
67a683a Bump ed25519-dalek from 2.0.0 to 2.1.0
8e0e794 quick and dirty fix for the `wait -p` and older distros (paritytech#2712)
3ab6562 Add withdraw reserve assets to zombienet tests (paritytech#2711)
c2c409b increase init timeouts in zombienet tests (paritytech#2706)
a8c60b4 fix lane id and bridged chain id (paritytech#2705)
9ac0f26 removed bp-asset-hub-kusama and bp-asset-hub-polkadot (paritytech#2703)
4916475 Some fixes for zombienet tests (polkadot-staging) (paritytech#2704)
6f9a147 zombienet from Wococo to Westend (paritytech#2699)
3ba7910 Porting changes from polkadot-sdk to polkadot-staging - before update subtree with removed wococo stuff (paritytech#2696)
653448f Remove Woococo related stuff (paritytech#2692)
03aaab2 Gitspiegel polkadot staging (paritytech#2695)
702a4c1 Drop Rialto <> Millau bridges (paritytech#2663) (paritytech#2694)
6a63b5f Start version guards for the ED loop (paritytech#2678)
896b9a9 typo (paritytech#2690)
671d27c Bump serde from 1.0.190 to 1.0.192
991b229 Bump clap from 4.4.7 to 4.4.8
ec267ec Bump env_logger from 0.10.0 to 0.10.1
592e407 Bump tokio from 1.33.0 to 1.34.0
c49ce3d Bump serde_json from 1.0.107 to 1.0.108
04b3319 Update subxt-codegen version (paritytech#2674)
03f9804 backport paritytech#2139 (paritytech#2673)
49245dd removed unused PARACHAINS_FINALITY_PALLET_NAME constant (paritytech#2670)
658a3f5 BHR/BHWE spec_version according to the `polkadot-sdk` (paritytech#2668)
7666b94 Nit from `polkadot-sdk` (paritytech#2665)
b5c43bb Adjusted constant because for measuring we used mistakenly rococo constants (paritytech#2664)
062449d Add Rococo<>Westend bridge support/relay (paritytech#2647)
55eb44e Add basic zombienet test to be used in the future (paritytech#2649) (paritytech#2660)
93b6b3f Bump clap from 4.4.6 to 4.4.7
4c01ab0 Bump futures from 0.3.28 to 0.3.29
a31a6c0 Bump tempfile from 3.8.0 to 3.8.1
bcdfe83 Bump serde from 1.0.189 to 1.0.190
f7433b0 Port paritytech#2648 to polkadot-staging (paritytech#2651)
3896738 Bump scale-info from 2.9.0 to 2.10.0
12d62c5 Bump thiserror from 1.0.49 to 1.0.50
1d78aa1 Backport from `polkadot-sdk` with actual master (paritytech#2633)
ab4de94 Grandpa justifications: Avoid duplicate vote ancestries (paritytech#2634) (paritytech#2635)
465562a add missing crate descriptions (paritytech#2629)
28d3680 Bump fixed-hash
67528c4 Bump serde from 1.0.188 to 1.0.189
d450c47 Bump time from 0.3.29 to 0.3.30
6a19f83 Bump async-trait from 0.1.73 to 0.1.74
a92d213 Millau, Rialto: accept equivocation reports (paritytech#2614) (paritytech#2617)
a61f777 Bump tokio from 1.32.0 to 1.33.0
0052f64 Bump subxt from 0.32.0 to 0.32.1
ccc849d Bump num-traits from 0.2.16 to 0.2.17
22f2752 apply late suggestions for paritytech#2600 (paritytech#2603)
0320172 actualize check_obsolete_call comment (paritytech#2601)
5cbbd25 Reject transactions if bridge pallets are halted (paritytech#2600)
ca4dfe3 Bump subxt from 0.31.0 to 0.32.0
8bf7b58 Bump clap from 4.4.4 to 4.4.6
88b0b99 Bump thiserror from 1.0.48 to 1.0.49
263833b https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/3833103 (paritytech#2589)
4f44968 Backport changes from polkadot-sdk (paritytech#2588)
7200ed1 fiox overflow when computing priority boost (paritytech#2587)
e02cbd3 Bump time from 0.3.28 to 0.3.29
a097dd2 Bump clap from 4.4.3 to 4.4.4
801ce88 Merge bulletin chain changes into polkadot staging (paritytech#2574)
a3803ce Add unit tests for the equivocation detection loop (paritytech#2571)
26dfc31 Bump clap from 4.4.2 to 4.4.3
66a8beb Bump serde_json from 1.0.106 to 1.0.107
18c50da Bump trie-db from 0.27.1 to 0.28.0
4c4fa92 Equivocation detection loop: Reorganize block checking logic as state machine (paritytech#2555) (paritytech#2557)
6bd317a Bump serde_json from 1.0.105 to 1.0.106
a7e6bfd Backport for polkadot-sdk#1446 (paritytech#2546)
d9f8050 Bump sysinfo from 0.29.9 to 0.29.10
901f44c Bump thiserror from 1.0.47 to 1.0.48
82eeb50 Bump sysinfo from 0.29.8 to 0.29.9
a0c934b Bump strum from 0.24.1 to 0.25.0
1064fbf Bump subxt from 0.28.0 to 0.31.0
e50398d bridges subtree fixes (paritytech#2528)
99af075 Markdown linter (paritytech#1309) (paritytech#2526)
733ff0f `polkadot-staging` branch: Use polkadot-sdk dependencies (paritytech#2524)
e8a59f1 Fix benchmark with new XCM::V3 `MAX_INSTRUCTIONS_TO_DECODE` (paritytech#2514)
62b185d Backport `polkadot-sdk` changes to `polkadot-staging` (paritytech#2518)
d9658f4 Fix equivocation detection containers startup (paritytech#2516) (paritytech#2517)
d65db28 Backport: building images from locally built binaries (paritytech#2513)
5fdbaf4 Start the equivocation detection loop from the complex relayer (paritytech#2507) (paritytech#2512)
7fbb67d Backport: Implement basic equivocations detection loop (paritytech#2375)
cb7efe2 Manually update deps in polkadot staging (paritytech#2371)
d17981f paritytech#2351 to polkadot-staging (paritytech#2359)

git-subtree-dir: bridges
git-subtree-split: bea13eab99af22f943b06397324c14fb2d7149e1
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 8, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 9, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 10, 2024
serban300 added a commit to serban300/polkadot-sdk that referenced this pull request Apr 10, 2024
bkchr pushed a commit that referenced this pull request Apr 10, 2024
@Polkadot-Forum
Copy link

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/what-are-subsystem-benchmarks/8212/1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
R0-silent Changes should not be mentioned in any release notes T0-node This PR/Issue is related to the topic “node”. T10-tests This PR/Issue is related to tests. T12-benchmarks This PR/Issue is related to benchmarking and weights.
Projects
Status: Completed
Development

Successfully merging this pull request may close these issues.

5 participants