IPFS starts too many new connections on add

[magik6k]

Version information:

current master at 181dd00

Type:

Bug

Severity:

High

Description:

When adding a large file the daemon starts creating many new connections eventually failing with too many open files error. This can be easily reproduced by running the daemon and then dd bs=1M if=/dev/urandom count=1000 | ipfs add --pin=false

Script/command for daemon monitoring:

export DPID=$(pidof ipfs); watch -n0 'printf "sockets: %s\nleveldb: %s\nflatfs: %s\n" $(ls /proc/${DPID}/fd/ -l | grep "socket:" | wc -l) $(ls /proc/${DPID}/fd/ -l | grep "\\/datastore\\/" | wc -l) $(ls /proc/${DPID}/fd/ -l | grep "\\/blocks\\/" | wc -l); netstat -anpt 2>/dev/null | grep "$DPID/ipfs" | sort -k6 | column -N "a,b,c,d,e,f,g" -J | jq ".table[].f" --raw-output | uniq -c'

When the add is running the daemon sometimes starts creating way more new connections(SYN_SENT) than it should, up to 1500 when it runs out of FD limit.

This is directly causing #3792 and is likely one of the reasons why many lower-end routers can't handle ipfs.

[Kubuxu]

This mean that connection listener is not working in go-libp2p-swarm.

I suspect something might be wrong with maftm.TCP or we are passing wrong multiaddr there, There is similar weird issue with fallback dialer being selected even though there is perfectly ok dialer to use.

[magik6k]

After digging a bit more, turns out that:

• dialLimiter.finishDial gets called with dialJobs that are on waitingOnFd list
• dialLimiter.finishDial can get called 10x for one job
• In both cases this lowers dl.fdConsuming, after a while(<1min since daemon start) daemon 'consumes' -10000 connections
• (I wonder if it may overflow after running daemon for a long time and cause it to not be able to connect to any new peers)
• Other less significant issue is that addrutil.IsFDCostlyTransport(which uses mafmt.TCP) doesn't match /ip4/1.2.3.4/tcp/123/ws(or /utp) addrs. This ignores the FD limit too but it adds way less to used FDs

[magik6k]

Fix is in libp2p/go-libp2p-swarm#26, bubbling it up is probably blocked by #4094.

[Kubuxu]

It should be bubble up in 0.4.10, it is a big bug that hits many users and makes go-ipfs 1. hard to use 2. seem more resource expensive than it can be.

[Kubuxu]

Oh, I forgot 4.10 is out.

[magik6k]

Reopening as libp2p/go-libp2p-swarm#26 isn't yet propagated and this got auto-closed.