Add custody by root request to sync #6275

dapplion · 2024-08-16T11:22:38Z

Issue Addressed

Part of

Build on / depends on

Add data columns by root sync request #6274

Proposed Changes

Adds a new higher order request to fetch all custody columns of a given block root.

It's a stateful request that may trigger one or more data_columns_by_root internally with features:

per-column retry
peer load balancing / prioritization of peer
don't re-request peers that return failures for some time

jimmygchen · 2024-08-21T02:29:14Z

This was automatically closed by mergify as the upstream branch was merged. I'll reopen this PR against unstable.

…est-custody-by-root

jimmygchen · 2024-08-21T10:15:57Z

The refactoring on these PRs are making merging back to das really difficult, can we please avoid refactoring on these PRs, and instead do it against das branch?

Some of the new function signatures require reworking parts of the existing code on das branch, so it would be easier if the refactor happens directly on das branch, otherwise we'd have to do the rework as part of the merge back to das which is really painful.

jimmygchen · 2024-08-21T12:40:20Z

beacon_node/lighthouse_network/src/types/globals.rs

+                DataColumnSubnetId::from_column_index::<E>(column_index as usize, spec),
+            ))
+            .cloned()
+            .collect::<Vec<_>>()


Why not take the latest change from dasbranch and just use good_custody_subnet_peer?
This undos most the changes from #6218 - i understand selecting peers from entire peers is not the right solution, but even this doesn't fix it and just revert back to the previous version.

Also happy if you'd like to change how it works on the das branch but i think it's better to target rework on the das branch before we finish the merge to unstable.

jimmygchen · 2024-08-21T12:40:50Z

beacon_node/lighthouse_network/src/peer_manager/peerdb.rs

+            peer.insert_subnet(Subnet::DataColumn(subnet.into()));
+        }
+
+        peer_id


Is all this change required? Why not just take the latest change from das branch? The change in #6218 fixes the custody lookup tests.

jimmygchen · 2024-08-21T12:47:21Z

beacon_node/network/src/sync/network_context/custody_by_root.rs

+
+const FAILED_PEERS_CACHE_EXPIRY_SECONDS: u64 = 5;
+
+type DataColumnSidecarVec<E> = Vec<Arc<DataColumnSidecar<E>>>;


For consistency:

Suggested change

type DataColumnSidecarVec<E> = Vec<Arc<DataColumnSidecar<E>>>;

type DataColumnSidecarList<E> = Vec<Arc<DataColumnSidecar<E>>>;

Also updated on the das branch custody.rs

jimmygchen · 2024-08-21T12:48:52Z

beacon_node/network/src/sync/network_context/custody_by_root.rs

+
+pub struct ActiveCustodyRequest<T: BeaconChainTypes> {
+    block_root: Hash256,
+    requester: SingleLookupReqId,


What's the reason to change this from CustodyId? (on das branch)

jimmygchen · 2024-08-21T12:49:37Z

beacon_node/network/src/sync/network_context/custody_by_root.rs

+    indices: Vec<ColumnIndex>,
+}
+
+type CustodyRequestResult<E> = Result<Option<DataColumnSidecarVec<E>>, Error>;


Are we dumping the peer group?

das branch:

type CustodyRequestResult<E> = Result<Option<(DataColumnSidecarList<E>, PeerGroup)>, Error>;

jimmygchen · 2024-08-21T12:50:35Z

beacon_node/network/src/sync/network_context/custody_by_root.rs

+                        column_request.on_download_success(
+                            req_id,
+                            peer_id,
+                            // Safe to cast, self.column_requests only contains indexes for columns we must custody


This comment can be removed as we're no longer casting, although I'm not sure why we removed the CustodyDataColumn type

jimmygchen · 2024-08-21T12:57:49Z

beacon_node/network/src/sync/block_lookups/tests.rs

+                .network_globals
+                .custody_peers_for_column(i, &self.harness.spec);
+            println!("peers column {} {:?}", i, peers);
+        }


Is this just for debugging? it doesn't actually assert anything but just prints

jimmygchen · 2024-08-21T13:03:29Z

beacon_node/network/src/sync/manager.rs

@@ -965,15 +966,31 @@ impl<T: BeaconChainTypes> SyncManager<T> {
    fn on_data_columns_by_root_response(
        &mut self,
        req_id: DataColumnsByRootRequestId,
-        _requester: SingleLookupReqId,
+        requester: SingleLookupReqId,


This parameter type isn't compatible with the das branch, and if we want to change this type i think we should change it there instead, as it would affect other part of the code (e.g. sampling) and make merging back and conga line difficult

jimmygchen · 2024-08-21T13:05:29Z

beacon_node/network/src/sync/manager.rs

+                        // TODO(das): this is not the correct peer to attribute to a response. It's
+                        // just the last peer of multiple potential requests. We should switch to a
+                        // concept of "peer group" to attribute fault to the correct peer.
+                        peer_id,


We already have this on das branch, is there a reason why you don't want it included in this PR? (this would diverge with das branch)

jimmygchen

Nice, another 900+ lines down 🎉
I've went through the code and I don't see anything impacting mainnet code path, so we should be good to go once comments are addressed and CI passes.

jimmygchen · 2024-08-28T06:13:04Z

@dapplion Is there any refactor in this PR that you'd like to merge to unstable? or shall we close this one?

dapplion · 2024-08-28T08:35:58Z

I ported a change to

Add reason of NoRequestNeeded to lookup sync #6317

else nothing relevant so I'll close

dapplion added 2 commits August 16, 2024 12:24

Add data columns by root sync request

f2abdd4

Add custody by root request to sync

01e897b

dapplion added ready-for-review The code is ready for review das Data Availability Sampling das-unstable-merge labels Aug 16, 2024

dapplion requested review from jimmygchen and realbigsean August 16, 2024 11:22

dapplion added the do-not-merge label Aug 16, 2024

dapplion added 2 commits August 20, 2024 16:01

Add custody happy path test

cc92d31

WIP fix custody test

6707b24

mergify bot deleted the branch sigp:unstable August 21, 2024 00:12

mergify bot closed this Aug 21, 2024

Merge branch 'unstable' into custody-by-root-base

63555dc

jimmygchen reopened this Aug 21, 2024

jimmygchen changed the base branch from peerdas-request-data-columns-by-root to unstable August 21, 2024 02:38

jimmygchen changed the base branch from unstable to peerdas-request-data-columns-by-root August 21, 2024 02:40

Merge branch 'peerdas-request-data-columns-by-root' into peerdas-requ…

5be3755

…est-custody-by-root

jimmygchen changed the base branch from peerdas-request-data-columns-by-root to unstable August 21, 2024 02:41

jimmygchen removed the do-not-merge label Aug 21, 2024

jimmygchen added do-not-merge and removed ready-for-review The code is ready for review labels Aug 21, 2024

dapplion added work-in-progress PR is a work-in-progress and removed do-not-merge labels Aug 21, 2024

Fix test

052845a

dapplion added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Aug 21, 2024

jimmygchen reviewed Aug 21, 2024

View reviewed changes

jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Aug 21, 2024

jimmygchen requested changes Aug 21, 2024

View reviewed changes

jimmygchen removed the das-unstable-merge label Aug 28, 2024

dapplion mentioned this pull request Aug 28, 2024

Add reason of NoRequestNeeded to lookup sync #6317

Merged

dapplion closed this Aug 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add custody by root request to sync #6275

Add custody by root request to sync #6275

dapplion commented Aug 16, 2024 •

edited

Loading

jimmygchen commented Aug 21, 2024

jimmygchen commented Aug 21, 2024

jimmygchen Aug 21, 2024 •

edited

Loading

jimmygchen Aug 21, 2024 •

edited

Loading

jimmygchen Aug 21, 2024

jimmygchen Aug 21, 2024

jimmygchen Aug 21, 2024

jimmygchen Aug 21, 2024

jimmygchen Aug 21, 2024

jimmygchen Aug 21, 2024

jimmygchen Aug 21, 2024 •

edited

Loading

jimmygchen Aug 21, 2024 •

edited

Loading

jimmygchen left a comment

jimmygchen commented Aug 28, 2024

dapplion commented Aug 28, 2024 •

edited

Loading


		const FAILED_PEERS_CACHE_EXPIRY_SECONDS: u64 = 5;

		type DataColumnSidecarVec<E> = Vec<Arc<DataColumnSidecar<E>>>;

	type DataColumnSidecarVec<E> = Vec<Arc<DataColumnSidecar<E>>>;
	type DataColumnSidecarList<E> = Vec<Arc<DataColumnSidecar<E>>>;

Add custody by root request to sync #6275

Add custody by root request to sync #6275

Conversation

dapplion commented Aug 16, 2024 • edited Loading

Issue Addressed

Proposed Changes

jimmygchen commented Aug 21, 2024

jimmygchen commented Aug 21, 2024

jimmygchen Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

jimmygchen Aug 21, 2024 • edited Loading

Choose a reason for hiding this comment

jimmygchen left a comment

Choose a reason for hiding this comment

jimmygchen commented Aug 28, 2024

dapplion commented Aug 28, 2024 • edited Loading

dapplion commented Aug 16, 2024 •

edited

Loading

jimmygchen Aug 21, 2024 •

edited

Loading

jimmygchen Aug 21, 2024 •

edited

Loading

jimmygchen Aug 21, 2024 •

edited

Loading

jimmygchen Aug 21, 2024 •

edited

Loading

dapplion commented Aug 28, 2024 •

edited

Loading