Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

collator-side: send parent head data #3521

Merged
merged 30 commits into from
Mar 19, 2024
Merged

collator-side: send parent head data #3521

merged 30 commits into from
Mar 19, 2024

Conversation

ordian
Copy link
Member

@ordian ordian commented Feb 29, 2024 •
edited
Loading

On top of #3302.

We want the validators to upgrade first before we add changes to the collation side to send the new variants, which is why this part is extracted into a separate PR.

The detection of when to send the parent head is based on the core assignments at the relay parent of the candidate. We probably want to make it more flexible in the future, but for now, it will work for a simple use case when a para always has multiple cores assigned to it.

This feature is currently gated by elastic-scaling-experimental feature flag.

@ordian
add new collation advertisement
504b919
@ordian
remove v3 in favor of enum variant
efceb5e
@ordian
get rid of with_elastic_scaling
94ba49b
@ordian
no maybes
211edb7
@ordian
Merge branch 'master' into ao-collator-parent-head-data
739e63c 
* master: (41 commits)
  Add Coretime to Westend (#3319)
  removed `pallet::getter` from `pallet-sudo` (#3370)
  gossip-support: add unittests for update authorities (#3258)
  [FRAME Core] remove unnecessary overrides while using derive_impl for frame_system (#3317)
  Update coretime-westend bootnodes (#3380)
  `im-online` removal cleanup: remove off-chain storage (#2290)
  Bump the known_good_semver group with 1 update (#3379)
  Fix documentation dead link (#3372)
  Make `sp-keystore` `no_std`-compatible and fix the `build-runtimes-polkavm` CI job (#3363)
  Remove unused `im-online` weights (#3373)
  Ensure referenda `TracksInfo` is sorted (#3325)
  rpc server: add rate limiting middleware (#3301)
  do not block finality for "disabled" disputes (#3358)
  fix(zombienet): docker `img` version to use in merge queues for bridges (#3337)
  Various nits and alignments for SP testnets found during bumping `polkadot-fellows` repo (#3359)
  Add broker pallet to `coretime-westend` (#3272)
  remove recursion limit (#3348)
  Update subkey README.md (#3355)
  Bump the known_good_semver group with 6 updates (#3347)
  Document LocalKeystore insert method (#3336)
  ...
@ordian
prdoc
3320447
@ordian
sanity check the provided parent head data hash
10a3e9c
@ordian
address review: use enum for parent head-data
b7466c2
@ordian
test validator sanity check parent_head_data hash
1ff7a54
@ordian
Merge remote-tracking branch 'origin/master' into ao-collator-parent-…
8b70d0f 
…head-data

* origin/master: (51 commits)
  Runtime Upgrade ref docs and Single Block Migration example pallet  (#1554)
  Collator overseer builder unification (#3335)
  Introduce storage attr macro `#[disable_try_decode_storage]` and set it on `System::Events` and `ParachainSystem::HostConfiguration` (#3454)
  Add Polkadotters bootnoders per IBP application (#3423)
  Add documentation around FRAME Origin (#3362)
  Bridge zombienet tests: Check amount received at destination (#3490)
  Snowbridge benchmark tests fix (#3424)
  fix(zombienet): increase timeout in download artifacts (#3376)
  Cleanup String::from_utf8 (#3446)
  [prdoc] Validate crate names (#3467)
  Limit max execution time for `test-linux-stable` CI jobs (#3483)
  Introduce Notification block pinning limit (#2935)
  frame-support: Improve error reporting when having too many pallets (#3478)
  add Encointer as trusted teleporter for Westend (#3411)
  [pallet-xcm] Adjust benchmarks (teleport_assets/reserve_transfer_assets) not relying on ED (#3464)
  Add more debug logs to understand if statement-distribution misbehaving (#3419)
  Remove redundant parachains assigner pallet. (#3457)
  Use generic hash for runtime wasm in resolve_state_version_from_wasm (#3447)
  Runtime: allow backing multiple candidates of same parachain on different cores  (#3231)
  Bridge zombienet tests: move all "framework" files under one folder (#3462)
  ...
@ordian
collator-side: send parent head data
bd2ee35 
When elastic scaling is detected (at the relay parent state),
meaning more than one is assigned to the para at the time,
the collator will send the collation variant with parent head data.
@ordian
test parent head data is sent
560da60
@ordian ordian mentioned this pull request Feb 29, 2024
Merged
2 tasks
@ordian ordian marked this pull request as ready for review February 29, 2024 11:08
@ordian ordian added the T9-cumulus This PR/Issue is related to cumulus. label Mar 1, 2024
@ordian
prdoc
e2751f5
@ordian
Merge remote-tracking branch 'origin/master' into ao-collator-parent-…
f44d58e 
…head-data

* origin/master:
  Fix call enum's metadata regression (#3513)
  Enable elastic scaling node feature in local testnets genesis (#3509)
  update development setup in sdk-docs (#3506)
  Fix accidental no-shows on node restart (#3277)
  Remove `AssignmentProviderConfig` and use parameters from `HostConfiguration` instead (#3181)
  [Deprecation] Remove sp_weights::OldWeight (#3491)
  Fixup multi-collator parachain transition to async backing (#3510)
  Multi-Block-Migrations, `poll` hook and new System callbacks (#1781)
  Snowbridge - Extract Ethereum Chain ID (#3501)
  PVF: re-preparing artifact on failed runtime construction (#3187)
  Add documentation around FRAME Offchain workers (#3463)
  [prdoc] Optional SemVer bumps and Docs (#3441)
  rpc-v2/tx/tests: Add transaction broadcast tests and check propagated tx status (#3193)
@ordian
Merge branch 'ao-collator-parent-head-data' into ao-send-parent-head-…
366b101 
…data

* ao-collator-parent-head-data:
  Fix call enum's metadata regression (#3513)
  Enable elastic scaling node feature in local testnets genesis (#3509)
  update development setup in sdk-docs (#3506)
  Fix accidental no-shows on node restart (#3277)
  Remove `AssignmentProviderConfig` and use parameters from `HostConfiguration` instead (#3181)
  [Deprecation] Remove sp_weights::OldWeight (#3491)
  Fixup multi-collator parachain transition to async backing (#3510)
  Multi-Block-Migrations, `poll` hook and new System callbacks (#1781)
  Snowbridge - Extract Ethereum Chain ID (#3501)
  PVF: re-preparing artifact on failed runtime construction (#3187)
  Add documentation around FRAME Offchain workers (#3463)
  [prdoc] Optional SemVer bumps and Docs (#3441)
  rpc-v2/tx/tests: Add transaction broadcast tests and check propagated tx status (#3193)
sandreim
sandreim approved these changes Mar 1, 2024
View reviewed changes
Copy link
Contributor

@sandreim sandreim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@ordian
Merge branch 'master' into ao-collator-parent-head-data
8910798 
* master:
  Finish documenting `#[pallet::xxx]` macros  (#2638)
  Remove `as frame_system::DefaultConfig` from the required syntax in `derive_impl` (#3505)
  provisioner: allow multiple cores assigned to the same para (#3233)
  subsystem-bench: add regression tests for availability read and write (#3311)
  make SelfParaId a metadata constant (#3517)
  Fix crash of synced parachain node run with `--sync=warp` (#3523)
  [Backport] Node version and spec_version bumps and ordering of the prdoc files from 1.8.0 (#3508)
  Add `claim_assets` extrinsic to `pallet-xcm` (#3403)
@ordian
add a comment (review)
7322058
@ordian
Merge branch 'ao-collator-parent-head-data' into ao-send-parent-head-…
6109444 
…data

* ao-collator-parent-head-data:
  add a comment (review)
  Finish documenting `#[pallet::xxx]` macros  (#2638)
  Remove `as frame_system::DefaultConfig` from the required syntax in `derive_impl` (#3505)
  provisioner: allow multiple cores assigned to the same para (#3233)
  subsystem-bench: add regression tests for availability read and write (#3311)
  make SelfParaId a metadata constant (#3517)
  Fix crash of synced parachain node run with `--sync=warp` (#3523)
  [Backport] Node version and spec_version bumps and ordering of the prdoc files from 1.8.0 (#3508)
  Add `claim_assets` extrinsic to `pallet-xcm` (#3403)
Base automatically changed from ao-collator-parent-head-data to master March 15, 2024 20:06
@ordian
Merge branch 'master' into ao-send-parent-head-data
227ff93 
* master: (65 commits)
  collator protocol changes for elastic scaling (validator side) (#3302)
  Contracts use polkavm workspace deps (#3715)
  Add elastic scaling support in ParaInherent BenchBuilder  (#3690)
  Removes `as [disambiguation_path]` from `derive_impl` usage (#3652)
  fix(paseo-spec): New Paseo Bootnodes (#3674)
  Improve Penpal runtime + emulated tests (#3543)
  Staking ledger bonding fixes (#3639)
  DescribeAllTerminal for HashedDescription (#3349)
  Increase timeout for assertions (#3680)
  Add subsystems regression tests to CI (#3527)
  Always print connectivity report (#3677)
  Revert "FRAME: Create `TransactionExtension` as a replacement for `SignedExtension` (#2280)" (#3665)
  authority-discovery: Add log for debugging DHT authority records (#3668)
  Construct Runtime v2  (#1378)
  Support for `keyring` in runtimes (#2044)
  Add api-name in `cannot query the runtime API version` warning (#3653)
  Add a PolkaVM-based executor (#3458)
  Adds default config for assets pallet (#3637)
  Bump handlebars from 4.3.7 to 5.1.0 (#3248)
  [Collator Selection] Fix weight refund for `set_candidacy_bond` (#3643)
  ...
@ordian ordian linked an issue Mar 16, 2024 that may be closed by this pull request
collator protocol: Provide parent head data #3128
Closed
@ordian ordian mentioned this pull request Mar 16, 2024
Closed
@alindima
Copy link
Contributor

I guess this should be marked as draft to prevent it from being merged and released until validators are upgraded to understand this protocol message?
From my understanding, the only way of currently preventing this is by not ordering multiple cores for the same paraid (but this is already happening in the wild). Alternatively, we could add a knob for enabling this, like a command-line flag

@ordian ordian marked this pull request as draft March 18, 2024 13:22
sandreim
sandreim reviewed Mar 19, 2024
View reviewed changes
Copy link
Contributor

@sandreim sandreim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ordian I think we can safely merge this and not block testing, please see my comment.

polkadot/node/network/collator-protocol/src/collator_side/mod.rs Show resolved Hide resolved
muraca and others added 7 commits March 19, 2024 11:54
@muraca @ordian
removed pallet::getter usage from cumulus pallets (#3471)
4f4ef35 
Part of #3326 

@ggwpez @kianenigma @shawntabrizi

polkadot address: 12poSUQPtcF1HUPQGY3zZu2P8emuW9YnsPduA4XG3oCEfJVp

---------

Signed-off-by: Matteo Muraca <mmuraca247@gmail.com>
@dependabot @ordian
Bump the known_good_semver group with 2 updates (#3726)
867edcd 
Bumps the known_good_semver group with 2 updates:
[serde_yaml](https://github.com/dtolnay/serde-yaml) and
[syn](https://github.com/dtolnay/syn).

Updates `serde_yaml` from 0.9.32 to 0.9.33
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/serde-yaml/releases">serde_yaml's
releases</a>.</em></p>
<blockquote>
<h2>0.9.33</h2>
<ul>
<li>Fix quadratic parse time for YAML containing deeply nested flow
collections (<a
href="https://redirect.github.com/dtolnay/unsafe-libyaml/issues/26">dtolnay/unsafe-libyaml#26</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/serde-yaml/commit/f4c9ed92385c827a677dae60c2b5a894e24709f0"><code>f4c9ed9</code></a>
Release 0.9.33</li>
<li><a
href="https://github.com/dtolnay/serde-yaml/commit/b4edaee907373ee69c6389687a35fced63d8addf"><code>b4edaee</code></a>
Pull in yaml_parser_fetch_more_tokens fix from libyaml</li>
<li><a
href="https://github.com/dtolnay/serde-yaml/commit/8a5542ced61ae21d0772e504fac01bd1dbfaaa6b"><code>8a5542c</code></a>
Resolve non_local_definitions warning in test</li>
<li>See full diff in <a
href="https://github.com/dtolnay/serde-yaml/compare/0.9.32...0.9.33">compare
view</a></li>
</ul>
</details>
<br />

Updates `syn` from 2.0.52 to 2.0.53
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/dtolnay/syn/releases">syn's
releases</a>.</em></p>
<blockquote>
<h2>2.0.53</h2>
<ul>
<li>Implement Copy, Clone, and ToTokens for syn::parse::Nothing (<a
href="https://redirect.github.com/dtolnay/syn/issues/1597">#1597</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/dtolnay/syn/commit/32dcf8df303c7181d95325f159ffbb1870a73e70"><code>32dcf8d</code></a>
Release 2.0.53</li>
<li><a
href="https://github.com/dtolnay/syn/commit/fd1f2aa9979c557e5b95c85f35b8949b8059153f"><code>fd1f2aa</code></a>
Merge pull request <a
href="https://redirect.github.com/dtolnay/syn/issues/1597">#1597</a>
from dtolnay/copyprintnothing</li>
<li><a
href="https://github.com/dtolnay/syn/commit/4f6c0528d63874495ce23d197a898cd63cb04d60"><code>4f6c052</code></a>
Implement ToTokens for syn::parse::Nothing</li>
<li><a
href="https://github.com/dtolnay/syn/commit/3f37543794e5d43952399847a2ecbc1eb39f609f"><code>3f37543</code></a>
Implement Copy for syn::parse::Nothing</li>
<li><a
href="https://github.com/dtolnay/syn/commit/36a412217d5217fab5657ab03ebcf29102d87a64"><code>36a4122</code></a>
Update test suite to nightly-2024-03-16</li>
<li><a
href="https://github.com/dtolnay/syn/commit/bd931069f5adb81168e7e6186ead0663cc605257"><code>bd93106</code></a>
Revert &quot;Temporarily disable nightly testing due to libLLVM link
issue&quot;</li>
<li><a
href="https://github.com/dtolnay/syn/commit/06166a77b7d465b0e4c45e1a2162f15031d7f722"><code>06166a7</code></a>
Update test suite to nightly-2024-03-09</li>
<li><a
href="https://github.com/dtolnay/syn/commit/ed545e75d8fdcab5078789ad25fc0efadc14e59b"><code>ed545e7</code></a>
Work around doc_markdown lint in test_precedence</li>
<li><a
href="https://github.com/dtolnay/syn/commit/7aef1edbba28afe1bb6f9ddc0b5095dcd5135849"><code>7aef1ed</code></a>
Temporarily disable nightly testing due to libLLVM link issue</li>
<li><a
href="https://github.com/dtolnay/syn/commit/556b10bf201264172e4b840f8cf1be38b8d75738"><code>556b10b</code></a>
Update test suite to nightly-2024-03-06</li>
<li>Additional commits viewable in <a
href="https://github.com/dtolnay/syn/compare/2.0.52...2.0.53">compare
view</a></li>
</ul>
</details>
<br />


Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@JuaniRios @bkontur @ordian
Add HRMP notification handlers to the xcm-executor (#3696)
03f5577 
Currently the xcm-executor returns an `Unimplemented` error if it
receives any HRMP-related instruction.
What I propose here, which is what we are currently doing in our forked
executor at polimec, is to introduce a trait implemented by the executor
which will handle those instructions.

This way, if parachains want to keep the default behavior, they just use
`()` and it will return unimplemented, but they can also implement their
own logic to establish HRMP channels with other chains in an automated
fashion, without requiring to go through governance.

Our implementation is mentioned in the [polkadot HRMP
docs](https://arc.net/l/quote/hduiivbu), and it was suggested to us to
submit a PR to add these changes to polkadot-sdk.

---------

Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: command-bot <>
@bkchr @ordian
Make availability-recovery-regression-bench a benchmark (#3741)
4697b7b 
Closes: #3704
@ordian
gate collator changes under elastic-scaling-experimental
d40a2a6
@ordian
Merge branch 'master' into ao-send-parent-head-data
6d34dcb 
* master:
  Make `availability-recovery-regression-bench` a benchmark (#3741)
  Add HRMP notification handlers to the xcm-executor (#3696)
  Bump the known_good_semver group with 2 updates (#3726)
  removed `pallet::getter` usage from cumulus pallets (#3471)
@ordian
taplo format --config .config/taplo.toml
ee7ccdf
@paritytech-cicd-pr
Copy link

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5569259

@ordian ordian marked this pull request as ready for review March 19, 2024 13:17
sandreim
sandreim approved these changes Mar 19, 2024
View reviewed changes
Copy link
Contributor

@sandreim sandreim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ordian !

alindima
alindima approved these changes Mar 19, 2024
View reviewed changes
Copy link
Contributor

@alindima alindima left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good

@sandreim sandreim added this pull request to the merge queue Mar 19, 2024
Merged via the queue into master with commit 5fd72a1 Mar 19, 2024
137 of 140 checks passed
@sandreim sandreim deleted the ao-send-parent-head-data branch March 19, 2024 15:23
dharjeezy pushed a commit to dharjeezy/polkadot-sdk that referenced this pull request Mar 24, 2024
@ordian @muraca
@dependabot @JuaniRios @bkontur @bkchr @dharjeezy
collator-side: send parent head data (paritytech#3521)
01e9452 
On top of paritytech#3302.

We want the validators to upgrade first before we add changes to the
collation side to send the new variants, which is why this part is
extracted into a separate PR.

The detection of when to send the parent head is based on the core
assignments at the relay parent of the candidate. We probably want to
make it more flexible in the future, but for now, it will work for a
simple use case when a para always has multiple cores assigned to it.

---------

Signed-off-by: Matteo Muraca <mmuraca247@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Matteo Muraca <56828990+muraca@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Juan Ignacio Rios <54085674+JuaniRios@users.noreply.github.com>
Co-authored-by: Branislav Kontur <bkontur@gmail.com>
Co-authored-by: Bastian Köcher <git@kchr.de>
This was referenced Jun 5, 2024
Closed
Closed
Closed
Closed
@TDemeco TDemeco mentioned this pull request Sep 27, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sandreim sandreim sandreim approved these changes

@AndreiEres AndreiEres AndreiEres approved these changes

@alindima alindima alindima approved these changes

Assignees
No one assigned
Labels
T9-cumulus This PR/Issue is related to cumulus.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

collator protocol: Provide parent head data
8 participants
@ordian @alindima @paritytech-cicd-pr @AndreiEres @sandreim @muraca @JuaniRios @bkchr