Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

95% of DHT find provider requests are trying to find relays #694

Closed
Stebalien opened this issue Jul 30, 2019 · 13 comments
Closed

95% of DHT find provider requests are trying to find relays #694

Stebalien opened this issue Jul 30, 2019 · 13 comments
Assignees

Comments

@Stebalien
Copy link
Member

Stebalien commented Jul 30, 2019

When combined with the issues outlined in libp2p/go-libp2p-kad-dht#146, this is spamming the network. Worse, we're doing this constantly.

Every 15 minutes, if we're connected to fewer than 5 relays, we search for relays 5 3 times until we give up, waiting 30s between each search.

That means:

  1. 30 seconds searching, 30 seconds waiting. 5 times (5 minutes).
  2. 10 minutes waiting.

That means that a full third of the network (assuming they have auto-relay enabled) is asking for this key at any given time unless they happen to find 5 3 stable nodes.


Key issues:

  • Requiring 5 is probably too many.
  • This module makes no effort to differentiate between stable/unstable relays. This means that crappy relays with low connection limits could get picked repeatedly, causing relay users to constantly have to try to find more relays.
@Stebalien
Copy link
Member Author

For now, let's just hard-code a set of relays and turn this off.

@vyzo
Copy link
Contributor

vyzo commented Jul 31, 2019

It tries to find 3 relays, but we can reduce this to 1 or 2.

@vyzo
Copy link
Contributor

vyzo commented Jul 31, 2019

And yeah, it should try to estimate relay stability to pick good relays.

@Stebalien
Copy link
Member Author

It tries to find 3 relays, but we can reduce this to 1 or 2.

Ah, got it. It just tries 5 times. I'm pretty sure one of the issues here is that there are groups running relays with low connection limits. That's why we need to make sure we pick a stable relay.

In the mean-time, I'd like to switch to hard-coded relays.

@vyzo
Copy link
Contributor

vyzo commented Jul 31, 2019

I don't think hard-coded relays are a very good idea, we can't just route all the traffic to our 3 medium relays and call it day. Who else is a stable relay operator?
I think it's better to try and estimate relay stability and pick accordingly.

@Stebalien
Copy link
Member Author

The current issue is both stability and libp2p/go-libp2p-kad-dht#146. I'm not kidding when I say that 95% of the DHT requests are trying to find relays.

@vyzo
Copy link
Contributor

vyzo commented Jul 31, 2019

This is a problem indeed, but hard-coding relays doesn't seem like the right solution. What relays to hard code? We don't have operating guidelines or known relay operators we can contact, and we (as pl) can't run enough relays to support the entire network.

@Stebalien
Copy link
Member Author

I completely agree. I just want to hard-code them for now until we fix the DHT. Right now, we have a nasty feedback loop where a significant portion of the relays' bandwidth is likely waisted by these queries.

@aschmahmann
Copy link
Collaborator

aschmahmann commented Jul 31, 2019

@Stebalien @vyzo is the problem simply that the DHT discovery queries for floodsub:/libp2p/relay are taking too long/never ending? If so it really doesn't seem super unreasonable to limit the number of required relays since probably if we find one we'll find multiple. If we're concerned about DHT churn fragmenting the list of relays we can potentially up the publishing on the limited number of relays.

Another approach might be to have a pubsub channel where the relays are published. Hard coding a few peers in the pubsub channel for relay advertisements should be really easy and not eat too many resources.

@raulk
Copy link
Member

raulk commented Aug 1, 2019

This search needs to be adaptive. We should start with target N (proposal: 2) and we increment or decrement the target adaptively based on how many relays we found in the last round, bounded by a min=1 and max=5.

If the network can't give you 5 results because they are no 5 relays globally, you are constantly asking the impossible. You need to scale back.

@aschmahmann using pubsub makes sense if the peer anyway supports pubsub. But I wouldn't want to make autorelay (part of the lower stack of libp2p protocols) dependent on a higher-level protocol.

Can we use the routing-helpers to attach pubsub-based and DHT-based discovery in daisy chain?

@aschmahmann
Copy link
Collaborator

aschmahmann commented Aug 1, 2019

But I wouldn't want to make autorelay (part of the lower stack of libp2p protocols) dependent on a higher-level protocol.

For sure, and using interfaces like Discovery or ContentRouting make sense to make this swappable as you suggest. However, it's worth noting that using the DHT as Discovery for a total list of values is currently pretty wonky since AFAIK the DHT doesn't cap the number of Provider Records per key, which can lead to some peers getting hammered with requests in addition to if multiple peers contain parts of the total value you basically end up with the issues that IPNS over DHT suffers from.

Can we use the routing-helpers to attach pubsub-based and DHT-based discovery in daisy chain?

@raulk Yes and No.

Yes, if you use the parallel or tiered routers and call SearchValue then you call both the DHT and PubSub in parallel every time you do a request. We could then leverage stateful validators to return the union of relays returned from the DHT and PubSub.

No, if you use the approach above without also doing something like your adaptive scaling (or just setting the number of relays to be small) we still end up doing many DHT requests in addition to listening to the pubsub channel (e.g. if we keep looking for 5 relays and they just aren't there). However, we could create a new utility/wrapper that has some linear or exponential backoff to wrap the DHT requests. Additionally, to distribute a persistent piece of data (e.g. a list of relays) over pubsub you need, at the very least, some chunks of the changes from libp2p/go-libp2p-pubsub-router#33 (e.g. repeated rebroadcasting is necessary like in go-ds-crdt, however, in the above PR which relies on libp2p/go-libp2p-pubsub#190 we are more bandwidth efficient and lower latency).

Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 17, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Stebalien added a commit to ipfs/kubo that referenced this issue Jan 18, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
@raulk
Copy link
Member

raulk commented May 20, 2020

I believe we have fixed this problem. If not, do you mind reopening, @Stebalien?

@raulk raulk closed this as completed May 20, 2020
@Stebalien
Copy link
Member Author

We have. Thanks for closing.

ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 6, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 6, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 6, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
ralendor pushed a commit to ralendor/go-ipfs that referenced this issue Jun 8, 2020
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
hannahhoward pushed a commit to filecoin-project/kubo-api-client that referenced this issue Jun 19, 2023
Unfortunately, we don't currently have any way to pick out good relays from bad.
That means we keep searching, trying bad relays, searching some more, trying
_the same relays_, etc. until we randomly find 3 good stable relays. In
practice, this means we just keep searching forever and keep thrashing the DHT.

see libp2p/go-libp2p#694
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants