Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data columns initial sync: Refactor #14522

Merged
merged 1 commit into from
Oct 15, 2024
Merged

Data columns initial sync: Refactor #14522

merged 1 commit into from
Oct 15, 2024

Conversation

nalepae
Copy link
Contributor

@nalepae nalepae commented Oct 9, 2024

This pull requests rework the way data columns sidecars are retrieved from peers.

Before this PR:
If in bwb we needed to retrieve columns [1, 3] for block [12, 13] and columns [8,9] for block 20, then, we sent a data column sidecar by range request corresponding to columns [1, 3, 8, 9] for blocks [12, 13, 14, 15, 16, 17, 18, 19, 20].

So,

  • columns [8, 9] for blocks [12, 13],
  • columns [1, 3] for blocks 20, and
  • columns [1, 3, 8, 9] for blocks [14, 15, 16, 17, 18, 19]

were returned by our peer for strictly no usage at all.
==> This occurs a useless bandwidth usage and a higher initial sync time (especially in small devnets) due to our internal rate limiting.

With this PR:
We carefully craft columns by range requests to ask peers to return exactly the columns we need.
(We take into account missed blocks and blocks without commitments, to minimise the number of generated requests.)

Remaining tasks regarding initial syncing:

  • Do not verify KZG proofs, inclusion proofs etc... for finalized blocks
  • Do run KZG proofs verification by batch
  • Manage the case where an orphaned block has been requested, and no peers is able to serve data columns for it
  • Use all peers for columns, and not only super nodes

@nalepae nalepae added the peerDAS label Oct 9, 2024
@nalepae nalepae force-pushed the initial-sync-rework branch 4 times, most recently from 7e7b57d to ec43b0b Compare October 11, 2024 11:01
@nalepae nalepae force-pushed the initial-sync-rework branch from ec43b0b to 4a48941 Compare October 11, 2024 11:07
@nalepae nalepae marked this pull request as ready for review October 11, 2024 11:17
@nalepae nalepae requested a review from a team as a code owner October 11, 2024 11:17
@nalepae nalepae requested review from potuz, nisdas and james-prysm and removed request for a team October 11, 2024 11:17
@nalepae nalepae merged commit a82a170 into peerDAS Oct 15, 2024
15 of 16 checks passed
@nalepae nalepae deleted the initial-sync-rework branch October 15, 2024 10:08
nalepae added a commit that referenced this pull request Oct 23, 2024
nalepae added a commit that referenced this pull request Oct 23, 2024
nalepae added a commit that referenced this pull request Oct 23, 2024
nalepae added a commit that referenced this pull request Oct 24, 2024
nalepae added a commit that referenced this pull request Nov 20, 2024
nalepae added a commit that referenced this pull request Nov 22, 2024
nalepae added a commit that referenced this pull request Nov 22, 2024
nalepae added a commit that referenced this pull request Nov 25, 2024
nalepae added a commit that referenced this pull request Nov 25, 2024
nalepae added a commit that referenced this pull request Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants