[META] Private Addresses In Public

[Stebalien]

You may have noticed that IPFS currently announces private addresses on the public DHT. While IPFS does support local discovery through MDNS, (a) MDNS isn't always perfectly reliable and (b) when trying to find a specific peer, we don't actually query MDNS.

However, announcing private addresses to the public network is far from an ideal solution. There have been many discussions about this issue (#1226, #1173, #4343, #5418, #5511) and some deeper discussions in libp2p/go-libp2p#436.

Problems include:

• Unnecessary network activity. Local dials are usually pretty cheap but they're still a waste.
• Triggers port scanning warnings (Running IPFS on VPS providers triggers netscan detection, risking account termination #4343). This is actually a rather complicated issue that includes other issues like dialing too many nodes (should be fixed by incoming DHT improvements), accumulating too many observed addresses (should be fixed by signed peer routing records), and just the general "IPFS is a p2p application that needs to dial a lot of nodes".

However, we have good news!

1. Libp2p is introducing signed peer routing records. This means we're getting atomic peer routing records that can't be spoofed, modified, extended, etc. This will land in 0.6.0.
2. Once that lands, libp2p will add a record "scope" (Model "address scopes" in signed peer records libp2p/go-libp2p#793).
3. Go-ipfs 0.5.0 should (unless something changes) ship ships two DHTs. A public one and a private one: Private & Public DHTs libp2p/go-libp2p#803.

Once all three of those land, we should be able to restrict private addresses to the private DHT.

[RubenKelevra]

I hate to bump this, but it's quite a low hanging fruit and causing many headaches - as it requires manual interventions on every installed node to not push private IPs to the DHT and also not try to connect to private IPs received from the public DHT.

Is there a chance to get this on the roadmap for 0.9? :)

[froid1911]

bump. lots of people and orgs are hosting ipfs on hetzner.

[simon-something]

Bump. Just got a warning from Hetzner...

[keesvanbochove]

Bump. Just got a warning from Hetzner...

This is happening when you are running with server profile? Is it a similar issue like #5418 (comment)?

[simon-something]

Bump. Just got a warning from Hetzner...

This is happening when you are running with server profile? Is it a similar issue like #5418 (comment)?

I thought I did, but they removed my warning after config patching it, so now I guess the issue was on my side - sorry for the unnecessary bump

[FarisZR]

Bump. Just got a warning from Hetzner...

This is happening when you are running with server profile? Is it a similar issue like #5418 (comment)?

I thought I did, but they removed my warning after config patching it, so now I guess the issue was on my side - sorry for the unnecessary bump

Can you give more details on your setup ? Are you using a dedicated server ? Or cloud ?

I'm trying to install ipfs on a hetzner cloud server, I got a warning on my first try.

[Mouradif]

I just got my server locked by hetzner due to this 😭

[ghost]

Hetzner's network team gets very unpleasant about these.

Maybe IPFS could check the first non-loopback interface to see if that address is RFC1918 and then make local peer discovery opt-in at that point?

[ghost]

Update: I got ejected from Hetzner again; server profile does not fix this issue.

(Why? Because it doesn't ignore the CGNAT range and apparently people run IPFS on nodes with tailscale installed)

I'm dropping IPFS from my stack for the time being.

[ShadowJonathan]

@acuteaura IPFS does not rate limit connections at all, and this repo has not acknowledged deep-running issues like this for multiple years, you're SOL, I heavily recommend permanently dropping IPFS from your stack.

[sharp2448]

@ShadowJonathan do you know if there is any good alternative?
Because I'm running graph node with subgraph, so I'm looking if I can use different approach which doesn't involve IPFS node ruining...

[Winterhuman]

@sharp2448 go-ipfs v0.13 will have a ResourceManager so rate limiting will be possible pretty soon.

[sharp2448]

@Winterhuman Ok, sounds good. Thanks for info

[Jorropo]

@varuzam this is a different issue, please comment on #8585

I am not able to reproduce the issue you are seeing, if you can help do so please comment on #8585.

[Jorropo]

@ROBERT-MCDOWELL this is a different issue, please comment on #8585

[ROBERT-MCDOWELL]

duplicate issue.... #8585

[TheRook]

@Stebalien Sending out RFC1918 addresses over the public network is always a bug - this should be forbidden and any client doing this needs to be patched. Do you think this is a symptom of the client not knowing its external IP?

What if there was an RPC that just echos back the IP address revived on the socket handle? With this RPC, the moment a peer has a connection they know how they routed out. This could be used to identify a double-NAT, which can still be traversed via UPnP as well as debug other issues like local peer discovery for peers who share a NAT but are on different network segments preventing mDNS, which is common in universities and corporate networks where multiple buildings or floors share the same up-link.

Now if the client knows its IP address, they could hash this address (no port number) and see if anyone looks it up on the global DHT network. If every client tries to lookup their own external IP - they'll find each-other and can exchange their RFC1918 local address so that they can connect directly over the local network which is much higher bandwidth. Clients knowing about each other on the local network is also good as it prevents them from competing for resources, or clobbering each other's NAT routes.

[mahdiyari]

I think you guys need 5 more years to get this sorted out.

[Mouradif]

I think you guys need 5 more years to get this sorted out.

Feel free to open a pull request

[TheRook]

Yeah, this feature already exists in I2p and cjdns. Is IPFS going to adopt open source solutions or are we going to wait 10 years for someone to reinvent the wheel on this one?

…

On Sun, Mar 5, 2023, 3:23 AM Mourad Kejji ***@***.***> wrote: I think you guys need 5 more years to get this sorted out. Feel free to open a pull request — Reply to this email directly, view it on GitHub <#6932 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD7MN6GPESBCGLMCCZF32LW2RZSLANCNFSM4K3WWXVQ> . You are receiving this because you commented.Message ID: ***@***.***>

[ShadowJonathan]

what "open source solutions"? Kubo is open source

[TheRook]

Freenet launched this feature 23 years ago... Kubo isn't using open source. With all due respect, Kubo is reinventing wheels and then releasing the source code.

…

On Sun, Mar 5, 2023, 12:31 PM Jonathan de Jong ***@***.***> wrote: what "open source solutions"? Kubo is open source — Reply to this email directly, view it on GitHub <#6932 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAD7MN75GE7SRJKDKVH2LG3W2TZ3XANCNFSM4K3WWXVQ> . You are receiving this because you commented.Message ID: ***@***.***>

[ShadowJonathan]

Freenet isn't Kubo, sorry to break the news

if you'd want to implement your double-nat discovery ideas, i'd suggest taking it up with https://github.com/ipfs/specs, some chats, or some other effort, and/or make a pull request adding an experimental module for this discovery method, and then get it properly integrated, you can use as much preexisting Open Source code and libraries as you want doing that

[Jorropo]

@TheRook this isn't a technological issue and have absolutely nothing to do with i2p or cjdns.

This issue is about the fact that I can do ipfs id somePeerId, and the Kubo node will probably just gives me all of it's private addresses, including LAN, Loopback, ...
Filtering for this is already implemented in Kubo: ipfs config profile apply server, then restart your node.

This is not default because this fully disable all fast local connections that are possible when you run more than one ipfs node on a single network.
I think the main interesting comparision is webrtc, for example the way firefox implements local connections is using mdns (it will announce randomly generated mdns names in the SDPs instead of private ips).
We have something similar in kubo (except we build a LAN DHT using mdns), the issue is that mdns rarely work with network topologies complex than 1 single FFA /24 LAN (like some cloud vpc network ...).

If you want to have a usefull discussion it would be about changing default (if you think that advertising private addresses in public is insane defaults) and good points would be:

• How many users run on complex LAN topologies, is that actually negligeable ?
• Showing mdns & lan dht or alternative working on cloud vpc network. (does it actually work good enough to be a complete replacement ?)
• More background on why you think that "Sending out RFC1918 addresses over the public network is always a bug", I am not convainced about this, what is an attacker gonna do anyway with thoses addresses it's not like learning them allows them to send packets to thoses addresses ? I get why some people would want this but this seems less important compared to the other bugs & features we have to fix becayse the impact seems very small and there is a workaround exists for people who really need it (ipfs config profile apply server).
  My quick read of RFC1918 yield that it was created to give private ranges for people that want to get around routing overhead and IP scarcity. Using a few public IPs for let's say a complete entreprise site with thousands machines (where the corresponding all public range would be costy). The security considerations is how this single NAT gateway topology allows users to add filtering on that gateway box more easily.
  Also many IPv6 setups provided by default by ISPs give an ipv6 range to each customer and the DHCP then gives public routable and add a firewall in front. Assuming you want to keep lan ip addresses private this is even worst, because then there is no real private ipv6 address anymore then, each machine have a unique ipv6 address which is broadcasted used with anyone you talk, the only difference is which side of the firewall is your interlocutor on.

If mdns solves almost all local connection needs and the one aren't solved are edge cases where people can be reasonably asked to change some configuration and this have any real impact at all.

We could change the defaults but stuff like this can't be done lightly:

• We lack data about how well the current alternative work (mdns) and my background networking knowledge tell me it wont work.
• You, We, ... never know who we are gonna break (some providers may relies on vpc to share data between their nodes without paying high transit costs, that seems like a usefull realistical usage).

[TheRook]

@Jorropo

There are a few open bugs that I believe are a symptom of the same problem; libp2p's nat traversal fails when it is behind multiple nats. Libp2p's hole punching is unstable when multiple clients attempt to use the same source port on the same subnet.

One problem here is that a new node cannot know what the external IP address is by looking at it's local interface. Putting this on the human, means only smart humans will be able to use it - and that isn't a general solution.

What if we create two new RPCs that facilitate network connectivity. Lets call them /net/ip and /net/ping. A new node can reach-out to a bootstrapped libp2p node and call /net/ip to get their external IP address so that external advertisements use an external ip address. The ping feature triggers a connect-back request to debug routing issues. Before the a new node even attempts to update the NAT it can check to see if the public_ip:default_port already has a node by trying to establish a connection with it.

The /net/ping RPC can be used to get a connectback from a public node on the network. This RPC call will trigger a unique connect back to the same IP address where the call originated. This RPC takes two arguments; it will take the port number and a random nonce to pass along. The ping RPC returns a success that the ping was sent or failure if the host didn't respond. If the new node gets a success from the ping RPC, but never got the nonce then there is another client on the network using that IP:Port combination and the client must not attempt to route using this port - it needs to select another port in order to route out. Ping is unlikely to be used for DDoS because it only sends a ping back to the IP address that sent the request - which should be enough to verify connectivity.

Now these two RPCs should resolve external routing conflicts and the ping RPC call can be used to verify a route upon request - one benefit here is that the node will know right away when NAT traversal starts working. This approach gives us another way for nodes to find eachother. If both nodes share the same IP address - they can still communicate as peers via the public interface even if they are on separate subnets.

But there is another issue that you brought up which is shortcomings of mdns. Two major reasons why mdns will fail is that the two clients are not on the same subnet - but still share the same NAT gateway, and the other being that mdns is filtered at the firewall level. If mdns is filtered a node is still free to send a UDP broadcast data-gram on another port - for example a HELLO broadcast via libp2p's UDP 162 is useful in helping identify any other nodes on the subnet. If they are on different subnets and share the same NAT gateway, they see the pattern above to establish connectivity via the public IP and a distinct port number.

[TheRook]

The real cool part of this solution is that every node - regardless of network topography will have the same unique connection string. Every node in the network will have a consistent and unique public_ip:port tuple to connect, and at no point would there ever be a need to advertise a local IP address to any node. This means that at most 65,535 libp2p nodes can all share the same public IP address - all of which can communicate with each-other and the global network.

[TheRook]

@Jorropo please confirm that your issue is solved with the two new RPC calls. Additionally, under no circumstance must libp2p node advertise a RFC1918 address to over to a non-RFC1918 address as this is guaranteed to result in a routing error in all cases.

[TheRook]

@Jorropo There there two additional methods on top of mDNS that can be used for any number of nodes to identify each-other on the same network. Most notability, if two nodes who are not on the same subnet - but share the same NAT can communicate so long nodes are made aware of their external IP address.

Let me be clear RFC1918 address cannot be routed to over the internet as per the RFC definition and must never be advertised under as this will always be an error condition. This is always a bug, and there is a good patch here.

[TheRook]

@Jorropo please confirm.