Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PeerDAS: Implement / use data column feed from database. #14062

Merged
merged 4 commits into from
Jun 4, 2024
Merged

PeerDAS: Implement / use data column feed from database. #14062

merged 4 commits into from
Jun 4, 2024

Conversation

nalepae
Copy link
Contributor

@nalepae nalepae commented May 30, 2024 •
edited
Loading

Please read commit by commit.

When receiving a dataColumnByRoot requests, this pull request replaces the 100 ms polling by a event system, produced by the database.

In the following scenario:

  1. We custody all the columns
  2. When a peer sends us a dataColumnByRoot, it wants to retrieve the following columns: [20 58 83 91 96 102 116 123]
  3. However, column 123 is not (yet) available
  4. Once the column is available, we move on
...
["2024-05-30 15:22:13"]  DEBUG sync: "Received data column sidecar by root request" custodied=[0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127] custodiedCount=128 handler="data_column_sidecars_by_root" requested=[20 58 83 91 96 102 116 123] requestedCount=8
...
["2024-05-30 15:22:13"]  DEBUG sync: "Peer requested data column sidecar by root not found in db, waiting for it to be available" handler="data_column_sidecars_by_root" index=123 root=0x86234822f22f0c658c26ed771cb52050eadf8d5ce0a0030877c8aa80b77e8bf8
...
["2024-05-30 15:22:13"]  DEBUG sync: "Data column sidecar by root is now available in the db" handler="data_column_sidecars_by_root" index=123 root=0x86234822f22f0c658c26ed771cb52050eadf8d5ce0a0030877c8aa80b77e8bf8

@nalepae nalepae requested a review from a team as a code owner May 30, 2024 15:35
@nalepae nalepae requested review from kasey, rauljordan and nisdas and removed request for a team May 30, 2024 15:35
@nalepae nalepae force-pushed the peerdas-data-column-feed branch from a85e960 to 7ff1dc5 Compare May 30, 2024 15:38
@nalepae nalepae force-pushed the peerdas-data-column-feed branch from 7ff1dc5 to a14700e Compare May 30, 2024 17:30
nisdas
nisdas reviewed May 31, 2024
View reviewed changes
beacon-chain/sync/rpc_data_column_sidecars_by_root.go Outdated
}
// Subscribe to the data column channel for this root.
rootChannel := s.cfg.blobStorage.DataColumnNotifier.ForRoot(requestedRoot)
defer s.cfg.blobStorage.DataColumnNotifier.Delete(requestedRoot)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have multiple requests which are waiting for it, is it safe to delete the notifier before all requests have been fulfilled ?

Copy link
Contributor Author

@nalepae nalepae May 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Will try to use event feed. It seems the right tool to use in this case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7d25e08.

@nalepae nalepae merged commit 54f2d91 into peerDAS Jun 4, 2024
13 of 16 checks passed
@nalepae nalepae deleted the peerdas-data-column-feed branch June 4, 2024 07:52
nalepae added a commit that referenced this pull request Jun 12, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
f5c46cf 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Jul 17, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
ab30599 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Jul 17, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
d3ca616 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nisdas pushed a commit that referenced this pull request Jul 18, 2024
@nalepae @nisdas
PeerDAS: Implement / use data column feed from database. (#14062)
ecf4195 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Jul 18, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
5fd9f13 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Jul 29, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
747de1a 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nisdas pushed a commit that referenced this pull request Aug 14, 2024
@nalepae @nisdas
PeerDAS: Implement / use data column feed from database. (#14062)
ae4bcaf 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nisdas pushed a commit that referenced this pull request Aug 15, 2024
@nalepae @nisdas
PeerDAS: Implement / use data column feed from database. (#14062)
515f855 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nisdas pushed a commit that referenced this pull request Aug 20, 2024
@nalepae @nisdas
PeerDAS: Implement / use data column feed from database. (#14062)
85f9820 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Aug 27, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
b2076e3 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Aug 28, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
4c2caa9 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Aug 28, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
98bfe2b 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Oct 7, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
e69e815 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Oct 7, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
16ac600 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Oct 8, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
53d43f5 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Oct 23, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
e3ac36f 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Oct 24, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
e7f4807 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Nov 20, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
a8c9b91 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Nov 22, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
2711e6f 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Nov 22, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
804b1bd 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Nov 25, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
aaac368 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Nov 25, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
36ce1e2 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
nalepae added a commit that referenced this pull request Nov 27, 2024
@nalepae
PeerDAS: Implement / use data column feed from database. (#14062)
42f4c0f 
* Remove some `_` identifiers.

* Blob storage: Implement a notifier system for data columns.

* `dataColumnSidecarByRootRPCHandler`: Remove ugly `time.Sleep(100 * time.Millisecond)`.

* Address Nishant's comment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nisdas nisdas nisdas approved these changes

@kasey kasey Awaiting requested review from kasey kasey is a code owner automatically assigned from prysmaticlabs/core-team

@rauljordan rauljordan Awaiting requested review from rauljordan rauljordan is a code owner automatically assigned from prysmaticlabs/core-team

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@nalepae @nisdas