Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

share/getters: Consider removing / not using IPLDGetter for requests of historic data #3653

Open
renaynay opened this issue Aug 12, 2024 · 0 comments
Labels
area:ipld IPLD plugin area:shares Shares and samples shrex For issues and PRs related to shrex protocol

Comments

@renaynay
Copy link
Member

Related to #3630

If we pursue the solution suggested in the above linked issue:

For the described data existence pre-checks, FN/BNs always report like they have the data, without doing lookups into inverted index.

This means that FNs + BNs of either type (archival or pruned) would always respond that they have the data regardless of whether they do or not. Archival nodes absolutely will have the data as they sync and retain all blocks, but pruned nodes will not have any data that is older than historic, so if a request for shares / EDS that is historic falls back to IPLDGetter, pruned nodes in the network will answer that they have the data even if they do not, so it would be misleading and could result in more roundtrips for the LN (e.g. requesting CID from a pruned node for historic shares by namespace when that data has already been deleted from the node, so LN has to go and ask another node).

The benefits of the solution quoted above are to reduce DAS' impact on the CPU of BNs + FNs in the network. DAS only occurs in the sampling window which is mostly what IPLDGetter is needed for (specifically, the GetShare request).

So when it comes to shares by namespace or EDS requests, the ShrexGetter should be sufficient to handle that as it's lighter-weight and the happy path. This would require some improvements in the robustness of the peer manager for shrex, but I think it's possible and can be done fairly easily (needs to be tested though of course).

FWIW, I would even consider removing the IPLDGetter as a fallback for requests for shares by namespace and EDS within the sampling window as well, but I think turning IPLDGetter off incrementally (for historic requests as they're less frequent and lower volume) is a first step approach.

@renaynay renaynay added area:shares Shares and samples area:ipld IPLD plugin shrex For issues and PRs related to shrex protocol labels Aug 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:ipld IPLD plugin area:shares Shares and samples shrex For issues and PRs related to shrex protocol
Projects
None yet
Development

No branches or pull requests

1 participant