filter local addresses (for WAN) and localhost addresses (for LAN)

[marten-seemann]

Filters are only set for the Dual DHT. Is that correct? Do they also need to be set for other ways to initialize the DHT?

[Jorropo]

Seems duplicated with PrivateRoutingTableFilter and PublicRoutingTableFilter. it looks duplicated but this filtering records sent

[guillaumemichel]

What do you think of setting the Public IP Addr Filter as the default address filter?

go-libp2p-kad-dht/internal/config/config.go

Lines 107 to 132 in e94c190

	// Defaults are the default DHT options. This option will be automatically
	// prepended to any options you pass to the DHT constructor.
	var Defaults = func(o *Config) error {
	o.Validator = record.NamespacedValidator{}
	o.Datastore = dssync.MutexWrap(ds.NewMapDatastore())
	o.ProtocolPrefix = DefaultPrefix
	o.EnableProviders = true
	o.EnableValues = true
	o.QueryPeerFilter = EmptyQueryFilter
	o.RoutingTable.LatencyTolerance = time.Minute
	o.RoutingTable.RefreshQueryTimeout = 1 * time.Minute
	o.RoutingTable.RefreshInterval = 10 * time.Minute
	o.RoutingTable.AutoRefresh = true
	o.RoutingTable.PeerFilter = EmptyRTFilter
	o.MaxRecordAge = providers.ProvideValidity
	o.BucketSize = defaultBucketSize
	o.Concurrency = 10
	o.Resiliency = 3
	// MAGIC: It makes sense to set it to a multiple of OptProvReturnRatio * BucketSize. We chose a multiple of 4.
	o.OptimisticProvideJobsPoolSize = 60
	return nil
	}

The private DHT could then override this default filter with the IP Loopback filter.

[marten-seemann]

@guillaumemichel, yes, that makes sense.

I'm wondering if we should be more careful which addresses we send out as well. We probably shouldn't even tell other nodes about private addresses, if we're connected to them over the public internet. I suggest to do that in a follow-up PR though.

[guillaumemichel]

Many tests are failing when setting the Public IP address filter as a default option (probably because the tests create IpfsDHTs using private IP addresses).

It is fine for now to leave it on the dual DHT only (no default address filter). This way we won't break anyone that may use a (non-dual) DHT with private addresses.

I added basic tests for the addrFilter, please have a look and add more checks if some are missing. I am good to merge after that

[sukunrt]

@guillaumemichel the tests look good to me.

[guillaumemichel]

Thanks @sukunrt!

@Jorropo are you good to merge?

[Jorropo]

@guillaumemichel I'm good if you are.

[sukunrt]

@Jorropo @guillaumemichel
What's required to get this landed in kubo?

[hsanjuan]

Hey, so this is causing my tests in cluster to break because I create a swarm of dual-dht hosts on localhost. It was very hard to arrive here.

I don't fully understand the rationale on why Dual DHT will ignore localhost nodes now. Seems a bit arbitrary (localhost becomes second class versus everyone else in the LAN?). Probably not many people hit this, but when they hit it they will have a hard time figuring out why those dht lookups fail.

[Jorropo]

The LAN dht ignores localhost in order to not announce Localhost IPs to other nodes on your LAN (which cause libp2p to first attempt all localhost addresses before trying the LAN ones).
I think what happen in your case is that you have exclusively localhost addresses so all addresses are being filtered out which is a usecase we overlooked.

As a workaround you can add LAN ips to all the Kubo processes on the same machine, which has similar performance as running through localhost (MTU will be worst but QUIC does not use the higher MTU of localhost anyway)

I guess you want a triple DHT setup where you have a localhost only DHT (since we would like to avoid advertising localhost addresses on the DHT),
ideally Dual wouldn't exists anymore and you could do that with a custom Kubo config using the composable routing helpers but we never finished the composable refactor.

[hsanjuan]

Understood. The fix is very simple for me (just tests setting dual.LanDHTOption(dht.AddressFilter(nil)),).

Given the explanation, it's better the way it is now.

[hsanjuan]

It's also an issue coming from test-local-hosts being setup to explicitly need DHT lookups for dials, coming from a time when testing the rest of the ipfs stack was more important. I don't think anyone is going to do it this way nowadays.