Segregate routable and non-routable networks

[Stebalien]

Currently, we announce all IP addresses we're listening, even private ones, and will dial any address we find for a peer, even private ones. We do this intentionally so we can connect to nodes on our local network. Unfortunately, this causes issues like ipfs/kubo#5511 and really pisses off users (for obvious reasons).

One reasonable solution (IMO) is to segregate routable and non-routable networks. That is:

1. If we learn about a non-routable IP address from a routable network, discard it.
2. If we know a non-routable IP for a peer, only distribute it to peers with non-routable addresses.

We'll still be able to discover and dial nodes on non-routable networks through local discovery (mDNS). Additionally, if we're running IPFS on a VPN and all peers have non-routable addresses, everything should "just work".

This will mostly require changes to the DHT and the Identify protocol.

Future work:

• Local Peer Exchange Protocol notes#3
• Separate DHTs? We could have a private one and a public one.

[rob-deutsch]

Potentially a silly question, but what's the definition of "routable network" and "non-routable network" in this proposal?

[vyzo]

The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.

[raulk]

I've been thinking about this as well in the context of libp2p/libp2p#47.

One possible design is to add an API in go-multiaddr-net that allows to query Routability() of multiaddrs, returning an "enum" with the following values:

• PUBLIC.
• PRIVATE => as per Private Network.
• PROXIMITY => future: Bluetooth or proximity-based technologies.
• LOOPBACK.
• NA => not applicable.

[raulk]

^ Identify could use this API to filter addresses. And the peerstore could use this to prune existing entries.

[upperwal]

Maybe after segregation we can try connecting top to down like if WAN network is same reject it and move to LAN network if that is same dial using loopback on different port.

---

Peer A:
WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B:
WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)

If A wants to connect with B: Because they are on the same LAN and WAN network they can connect using loopback:port

---

Peer A:
WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B:
WAN (1.2.3.4/24) -> LAN (192.168.1.200) -> LOOPBACK (127.0.0.1:4000)

WAN is same, connect on LAN.

---

Peer A:
WAN (1.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:3000)

Peer B:
WAN (2.2.3.4/24) -> LAN (192.168.1.100) -> LOOPBACK (127.0.0.1:4000)

Totally different network. Dial on WAN.

Can this work?

[Stebalien]

@upperwal I'm having trouble parsing your first sentence.

[upperwal]

@Stebalien So instead of trying all the observed addresses we can segregate the addresses into different categories like PRIVATE, PUBLIC or LOOPBACK (as @raulk suggested) and then we can try connecting in the following order:

1. PUBLIC: Connect using PUBLIC endpoint only if peer A and B are on different PUBLIC IP's. If they are on the same PUBLIC IP's a shorter path exist. Case 3 above shows this.
2. PRIVATE: If they are on the same PUBLIC IP check if they are on the same PRIVATE IP. If they are not, connect using PRIVATE endpoint. Case 2.
3. LOOPBACK: If their PUBLIC and PRIVATE IP's or endpoints are same they are on the same machine. Use loopback:port to dial. Case 1.

[Stebalien]

I'm not sure that'll solve the issue. Many nodes simply aren't reachable so we'll end up spamming a bunch of non-routable addresses anyways. Additionally, IMO, we generally want to take the opposite approach when dialing. We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.

edit: I need to read entire comments before responding.

[rob-deutsch]

The usual interpretation is publicly routable address range -- check wikipedia for a list of private networks.

Okay. Maybe it wasn't such a silly question, because I assumed a different definition.

I think we should limit the distribution of non-routable addresses even more. I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:

2. If we know a non-routable IP for a peer, only distribute it to peers with non-routable addresses in same same subnet as the non-routable IP.

Also, could the proposal be amended to say that we send only our non-routable IP to non-routable nodes in the same subnet. I.e. we don't also send the routable IP. This will avoid NAT and IP-level routing problems, and I would definitely prefer it. However, I don't know if there are edge cases, or advanced network setups, that this might be non-ideal for. I'm especially unsure about IPv6.

In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.

[Stebalien]

I think that non-routable IPs should only be distributed to peers with non-routable addresses in the same subnet as the non-routable IP we want to distribute. I.e. the line in the OP could should be changed is:

(that was actually my first thought as well)

Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.

In general, how do other protocols/software handle this problem of public/private addresses? Is there some prior art we can refer to.

Manual configuration and service discovery through mDNS (local multicast). We have the second but the first probably isn't going to work well for us (although we could provide some kind of configuration language...).

---

However, I'd would like to amend my original proposal:

1. Localhost addresses should only be advertised to other localhost nodes.
2. IPv6 link-local addresses (when we eventually add support for them) should only be advertised to nodes on the same link. This isn't an issue currently (we don't support these) but we should keep it in mind.

[upperwal]

We should prefer private/loopback addresses as those addresses should fail faster (and the resulting connections may often be faster). However, we don't want to do this unless we have a reason to believe that these addresses will actually work.

@Stebalien The given approach does prioritise loopback/private addresses and it also gives you a way to check if these addresses work.

Let me explain it again.

Solution to this problem is, to somehow identify the location (network) of peer A and B.

if samePhysicalHost(A, B):
  dial(loopback)
else if sameLocalNetwork(A, B):
  dial(private_ip)
else
  dial(public_ip)

1. Now we start by comparing the public_ip (comparing is simple string matching which can happen on the node) of the destination node with the source node, If they match, we can safely assume they are on the same network hence destination should not be dialed using public_ip (we skip dial using this address so no overhead). If they don't match, they are on different network, safe to dial using this address and stop dialing using other addresses

2. We do the same with private _ip but only if public_ip matches because now the best candidate is private_ip. If private_ip for both matches we again skip using it (because they are on the same private network, loopback is the best option here). If they do not match we assume the private network for peer A and B are different hence safe to dail using private_ip. Do not dial with any other address.

3. If private_ip and public_ip is same go with the loopback because both the peers are on the same machine.

This way we are checking for the subnets in which peer A and B lies and dialing accordingly.

This can also work for complex networks like you described above given we can query the subnet router for it's interface address (and subnets will have different IP signatures so distinguishable). If not, a fallback strategy can be used where:

If peer A and B are on different subnet within an organisation connected to a VPN with observable addresses as follows:

Peer A: 1.2.3.4/24:1234 -> [some subnet X which isn't visible] -> 192.168.1.2 -> 127.0.0.1:3000
Peer B: 1.2.3.4/24:6789 -> [some subnet Y which isn't visible] -> 192.168.1.2 -> 127.0.0.1:4000

Now we will assume that they are on the same host but they are not. We can try connecting using loopback but if it fails we try private IP if that fails we goto the public one. This case is applicable only when we could not find the subnet IP.

Man, I need a diagram. 😛 I hope I am much more clear this time.

[rob-deutsch]

Unfortunately, that won't work on complex networks: it's entirely possible to have multiple private subnets joined into a single network. As far as I know, this is actually an extremely common setup on corporate networks. Different buildings/floors will be on different subnets and the subnets will be connected by, e.g., a VPN.

That's a very good point, but there's a few other sides to the coin:

1. Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.

2. Existent but unrouteable private addresses - Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN). In a way, this is an extension of the problem we're trying to solve here i.e. not publishing private IPs to public networks.

(To be clear, I am not married to any particular proposal solution here. I just think its an interesting topic, and I hope I'm contributing to the conversation.)

For what its worth, I think that your proposal is a strict improvement on the current setup, and I would be all-for it being implemented, but I'm not sure I'm convinced that its the end of the road in terms of solutions.

[Stebalien]

@rob-deutsch

Information leakage - It's 'bad form' to pass information about one subnet to another subnet. Multi-subnet corporate networks are common, but so are PCs that connect to a local network and a privacy-conscious VPN. Or home network and corporate network.

Yeah, this bugs me too (related ipfs/kubo#1771). And you're right, that's bad form. Really, we should just make this configurable (to some extent). Personally, I'd start with my proposal (as you said, it's a strict improvement but is less restrictive) and we can then add configuration options for more restrictive setups.

Existent but unrouteable private addresses. Another rare case, but I might have a node that's connected to two independant private networks that can't actually talk to each other (e.g. VPN)

This I'm less worried about. Yes, it can happen but oh well, a few dials fail in rare cases. IMO, the bigger issue is dialing private IP addresses on every dial.

---

@upperwal

Ah. I saw the in-order part, read the headers, and completely misread your proposal. That's actually a really cool solution.

There are a few drawbacks:

• We could both have different public IP addresses but be on the same VPN.
• One or both peers may not know their public IP addresses. They may even have multiple public IP addresses on some of the nastier NATs. In this case, we'd have to just dial every address.
• It still leaks private information (@rob-deutsch's issue).

However, it's a really cool heuristic and we can probably just plug it into the dialer with little work.

[Stebalien]

@upperwal want to try plugging this logic into go-libp2p-swarm/swarm_dial.go (if you have time)?

[upperwal]

@Stebalien Yeah, its a simple solution with little refactor. Sure, let me give it a try.

[upperwal]

@Stebalien

I am able to order the ip list as []public, []private, []loopback by implementing Routability()

unordered
[/ip4/192.168.2.229/tcp/28447 /ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/127.0.0.1/tcp/4001 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip6/::1/tcp/4001 /ip4/14.114.26.227/tcp/10782]

ordered
[/ip4/14.114.26.227/tcp/12540 /ip4/14.114.26.227/tcp/10024 /ip4/14.114.26.227/tcp/10782 /ip4/192.168.2.229/tcp/28447 /ip4/192.168.203.113/tcp/4001 /ip4/172.17.0.1/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip6/::1/tcp/4001]

Next step would be to find the correct subnet. The only problem is that the peer dialing does not know it's observed address. InterfaceListenAddresses() can only give [/ip4/127.0.0.1/tcp/52575 /ip4/192.168.1.101/tcp/52575 /ip6/::1/tcp/52576]. Hence segregating subnet would be difficult. Any suggestion?

[cannium]

A quick and (somehow) dirty solution would be to pass *BasicHost when creating swarm instance(https://github.com/libp2p/go-libp2p/blob/master/config/config.go#L94), and get observed addresses by *BasicHost.IDService().OwnObservedAddrs()
A more graceful solution might need some refactoring, store observed addresses together with peerstore. Just my personal opinion.

[vyzo]

Unfortunately that can't work -- it would create a circular dependency.

[cannium]

So some refactoring is unavoidable...

[upperwal]

I think identity protocol can help (with little refactoring) where destination peer can send us back our observed address during protocol negotiation. Observed address can be stored and made available by some API.

Edit: Keeping our observed address can help us take intelligent and optimised decisions when it comes to routing and connectivity.

[Stebalien]

Damn. That's annoying. I don't know of a simple fix.

Next step would be to find the correct subnet.

We don't really need to know anything about subnets to get this to work, we just need our observed external address.

[upperwal]

Damn. That's annoying.

I know.

By "Next step would be to find the correct subnet.", I meant to find the shortest path to dial on and yes that would only require our observed address.

@Stebalien Is #427 related to the other peer's observed addresses (the one I would be connected to)?

[bigs]

@vyzo that's a fair place to start, indeed. That will be a quick win before making deeper (and more dangerous) changes.

It's been bugging me that the dialing logic in go-libp2p-swarm is hardcoded and not abstractable, so I'm turning it into a "dialing strategy" we can have multiple implementations of, to select dynamically.

i think this is reasonable, just be careful when making choices about composability. i think this is long overdue, but i'd be sure to account for the multiple wrapping layers here and try your best to tease them apart! something like:

type Dialer interface {
  func Dial(...) ...
}

type DialerStrategy interface {
  Dialer

  func Wrap(Dialer) Dialer
}

then we could have

var MultiaddrDialer Dialer // manet stuff
var SyncDialer DialerWrapper // lets us parallelize and cancel upon success
var ResourceLimitedDialer DialerWrapper // lets us constrict file handle usage

[bigs]

one thing i'm immediately wondering about/concerned about is how "peer-to-peer friendly" an identity service is. feels like it has the potential to centralize things quite a bit if gone about wrong. perhaps a node doesn't announce until it's fully bootstrapped, which might include identifying itself by multiple peers in different subnets?

furthermore, i think raul's concerns about backwards compatibility point towards a dialer-focused solution as the best steps forward.

[rob-deutsch]

Also, could I suggest a modification to your first point?

... I should announce the proper addresses of mine myself and peers I know about.

Dunno, I think this is too much magic. Peer A should not make a decision about how Peer B addresses Peer C. Also, peers certifying their own routing records would prohibit third parties from modifying them, as the signature would no longer match.

Regardless of how much magic this is, isn't it required?

For example, lets say that Peer A and Peer B both have their own public addresses but are also on the same private network (because they're in a datacenter or VPN).

Peer A and Peer B might tell eachother both their public and private IP addresses. But what happens when Peer A wants to tell Peer C (on the other side of the internet) about Peer B? Will Peer A tell Peer C the private addresses too?

[upperwal]

@Stebalien

Damn. That's annoying. I don't know of a simple fix.

got a trivial solution. Calling dht.FindPeer(self) would return our observable Multiaddr from one of our immediate neighbour.

[Stebalien]

got a trivial solution. Calling dht.FindPeer(self) would return our observable Multiaddr from one of our immediate neighbour.

I'd rather not make any network requests to discover this... @raulk is now working on a dialing refactor to make things like this possible.

[upperwal]

cool

[upperwal]

@raulk Any update on this?

I noticed that identify already knows about observedAddrs which can be used for remote address selection in swarm before dialling. We can identify the topology based on our addresses (observed + local + private) and remote addresses. This way we dial to only one remote address.

This will require ObservedAddrSet to be available within swarm. As swarm is created before basicHost.

go-libp2p/config/config.go

Lines 105 to 116 in 3e2dc09

	swrm := swarm.NewSwarm(ctx, pid, cfg.Peerstore, cfg.Reporter)
	if cfg.Filters != nil {
	swrm.Filters = cfg.Filters
	}
	var h host.Host
	h, err = bhost.NewHost(ctx, swrm, &bhost.HostOpts{
	ConnManager: cfg.ConnManager,
	AddrsFactory: cfg.AddrsFactory,
	NATManager: cfg.NATManager,
	EnablePing: !cfg.DisablePing,
	})

Maybe we can have something like swrm.SetObservableAddrs()

This is a quick fix. Let me know if you are working on something bigger and I would be happy to contribute.

Edit: This will also help close libp2p/go-libp2p-pubsub#40 and resolve error ERROR pubsub: already have connection to peer: PubSub error (except in the case where A and B simultaneously connects to each other)

[raulk]

@upperwal Here's a WIP PR for a change that modularises the swarm dialer in a way that allows applications to determine how they want to prioritise multiaddrs, plan the dials over time, and throttle each attempt based on resources: libp2p/go-libp2p-swarm#88.

Maybe we can have something like swrm.SetObservableAddrs()

We'll definitely need to expose the observed addresses on the Network interface somehow. Haven't thought about the API, but Planners in dialerv2 will want this info.

[bonedaddy]

Any updates to this?

[Stebalien]

Progress:

1. Two DHTs: Private & Public DHTs #803.
2. Scoped addresses: Model "address scopes" in signed peer records #793