libtorrent 2.0.x is much too aggressive in retrying connections to known seeds

[reardonia]

libtorrent version (or branch): noted in 2.0.5 and above

platform/architecture:
all

please describe what symptom you see, what you would expect to see instead and
how to reproduce it.

Have tested with Deluge and qBittorrent using libtorrent 2.0.5 and above.

(This is coming from interop testing with Transmission 4+)

libtorrent is entirely too aggressive in retrying known seeds. When an LT2 client is a seed, it tries every peer it can find once peer minute. Even when learning that the other peer is a seed (via FastExt "have-all" message BEP-6, or LTEP "upload_only" message BEP-21), it never throttles back or stops contacting peers that are known seeds.

Further, if the torrent is private, and therefore no PEX peer-exchange can occur, when LT2 client is seed, it should stopped contacting known-seeds entirely.

Please consider that this has become signficant enough that the private trackers I work on are considering banning LT2 clients because of the excessive peer-seed traffic generated.

LT1.2 clients do not seem to exhibit this overly aggressive behavior. EDIT: LT1.2.19 also seems to evidence this behavior.

[reardonia]

the private trackers I work on are considering banning LT2 clients because of the excessive peer-seed traffic generated.

Just wondering, what does a tracker have to do with what two peers are doing under the blanket? In what way does the tracker suffer from this? Total domination of pompous private tracker admins over their users is always disastrous for the tracker.

Hey, I'll take the feedback that I came across as petulant, but really this issue was raised by our users, not admins. Our main tracker focuses on and rewards long-terms seeds, and this becomes very problematic when the background chatter among seeds is too large. For comparison, chatters amongst Transmission-based seeds is 1/200th (measured at packet level) of chatter from LT seeds. That is an enormous delta and becomes prohibitive when seeds are hosting 1000's to 10000's torrents.

[Seeker2]

The seeds retrying seeds problem is far worse than first appears, if using uTP:
#3542 (comment)

It can flood out other connections for seeds...

...even appearing to cause massive file losses and/or hard drive damage for other BitTorrent clients on basically every shared torrent they have in common with libtorrent-based clients:
"Disconnect: Peer error: The system cannot find the file specified."

That link isn't the first known instance of seeds-retrying-seeds every minute by libtorrent clients -- that has been known for years.

[arvidn]

the intention is to stop contacting seeds (if oneself is also a seed). I'll take a look, thanks for the report!

[arvidn]

In my test I have two peers, one seed and one downloader. After the transfer is complete neither peer is attempting to connect to the other.

Could you help me reproduce the conditions where this is happening?

[Pentaphon]

Could you help me reproduce the conditions where this is happening?

@reardonia also interested in seeing this.

[Seeker2]

In my test I have two peers, one seed and one downloader. After the transfer is complete neither peer is attempting to connect to the other.

In my case, I had both qBitTorrent and uTorrent set to uTP only connection type (no TCP peers/seeds).

[reardonia]

In my test I have two peers, one seed and one downloader. After the transfer is complete neither peer is attempting to connect to the other.

Could you help me reproduce the conditions where this is happening?

(sorry for delayed response, due to vacation)

Repro is simple. Start qBittorrent (or similar LT-RB client) with full seed torrent already in place. If the tracker sends back the full list of peers, inclusive of seeds, then the LT-RB client will connect every peer that the tracker returned. However, even though it learns from those peers that they are in fact seeds, it continues to contacts them every ~60sec. The problem seems to be that LT-RB is discarding the upload_only indications and relying on the tracker to send it peers that are downloading still. The problem is that many trackers do not distinguish between seeds and downloaders.

Note: to repro you MUST start with app with seed in place. Don't start a new download as you did. The concern we have is for long-terms seeds where the bt application is stopped and restarted many times.

[ValeZAA]

However, even though it learns from those peers that they are in fact seeds, it continues to contacts them every ~60sec.

What happens if it is 3 minutes? 6 minutes? Is there a standard how to do it?

even appearing to cause massive file losses and/or hard drive damage for other BitTorrent

Then this a bug either in hardware, or firmware of HDD or software of windows. Could not happen on NVMe, now can it

[Seeker2]

even appearing to cause massive file losses and/or hard drive damage for other BitTorrent

Then this a bug either in hardware, or firmware of HDD or software of windows. Could not happen on NVMe, now can it

Almost resembles a completely dying old computer...

...except no:
#3542 (comment)
"The weird error message (which seems to imply a drive failure with lots of missing files) is just a clever side-effect of that."

The problem could be even worse on NVMe, due to its speed.

[reardonia]

In my test I have two peers, one seed and one downloader. After the transfer is complete neither peer is attempting to connect to the other.

Could you help me reproduce the conditions where this is happening?

Hoping to pick this back up with you @arvidn

The problem is with long-lived seeds. Ignore situation of a fresh download where one peer downloads completely.

Instead, consider two libtorrent peers (seeds) that are connecting to the same tracker. That tracker returns, on announce, all known peers, rather than only returning leechers to seeders. I think you may be fooled because you are testing with a tracker that returns leechers to seeders, but does not return seeders to seeders. In this common situation where the tracker returns peers without filtering for seeds (especially common on private trackers), libtorrent clients (qBittorrent and Deluge) will keep attempting connections to known seeds. the libtorrent clients disconnect right after handshake, after learning that the other peer is a seed (during the LTEP exchange, which is when they see the "upload_only" flag). But that knowledge is never retained, and the libtorrent peers retry once per minute, ad infinitum.

Does this help clarify?

I should also add: libtorrent peers-that-are-seeds will drop a connection as soon as learning that the other peer is a seed, by "upload_only" flag amongst others. However, the libtorrent-peer that drops the connection never sends it's own "upload_only" flag during potential LTEP exchange. The connection seems to be dropped before LTEP completion. This in turn confuses non-libtorrent peers who aren't sure why the connection was dropped, something that would be otherwise deduced from seeing the "upload_only", aka seed, flag.

[Pentaphon]

the libtorrent clients disconnect right after handshake, after learning that the other peer is a seed (during the LTEP exchange, which is when they see the "upload_only" flag). But that knowledge is never retained, and the libtorrent peers retry once per minute, ad infinitum.

This seems like something that could be deduced from the code itself.
Is this what happens in the current implementation @arvidn ? If so, it should be using that knowledge to try other peers instead of wasting time and resources on known peers.