Libp2p does not recover from interface being temporally down

[hsanjuan]

Our storage cluster suffers from time to time from some issues where libp2p hosts do dial_backoff forever after a problem with the listen interface. It starts with:

Jul 16 04:02:21 cluster2.fsn kernel: igb 0000:03:00.0 eth0: igb: eth0 NIC Link is Down
Jul 16 04:20:29 cluster2.fsn kernel: igb 0000:03:00.0 eth0: igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

Libp2p does not recover from this event. It shows:

i/o deadline reached (a few times after iface is down)
connection reset (5 secs later)
dial attempt failed: context deadline exceeded (+1min later)
dial backoff (+1 min later)

From there, dial attempt failed and dial backoffs keep continuously happening until the peers are restarted, well after the network interface is back up. After re-starting the peers, everything works well again.

This are standard libp2p basic hosts, from 5.0.17. The services they run will continuously attempt to re-open streams in such failures.

Ideally, libp2p should be able to recover from such events automatically, but I'm not sure how difficult this is or why exactly it stays erroring.

[Stebalien]

I don't think this is due to the backoff logic, probably something else.

• Are the peer's changing their IP addresses?
• Can they dial peers outside the cluster?
• What listen addresses are you using in your config?

[arzahs]

Hello. Is there a solution for the issue? I got the same.

github.com/libp2p/go-libp2p v0.11.0
github.com/libp2p/go-libp2p-core v0.6.1

[jkassis]

peer A creates a stream that initializes a connection to peer B. peer A's network interface connects to a new wifi network (goes out of range of the one it was in and then hops to another). the connection / stream doesn't work. not completely unexpected. but when it hops back to the previous wifi network and gets the same assigned IP address, the connection / stream doesn't resume.

not entire sure about the design objectives here, but this is kindof a show-stopper for P2P iOT.

[jkassis]

how can i at least just check the status of the connection? Conn.Stat doesn't seem to do it.

[Stebalien]

but when it hops back to the previous wifi network and gets the same assigned IP address, the connection / stream doesn't resume.

That's not something we can fix (without stream migration, see libp2p/specs#328). When peer B switches networks, all of its connections get cut by the OS. The problem is that peer A won't see this till it hits a timeout.

[Stebalien]

But to answer your last question, you can use the ping protocol (search for "ping" in the main libp2p repo) to quickly check. But you'll still have to wait some timeout before you can really declare the connection "dead".

[MarcoPolo]

I know this is an old issue, but I just verified that this no longer happens.