Deadlocks and memory leaks

[inetic]

Version information:

We're using asio-ipfs a C++ binding to go-ipfs.
That library uses IPFS v0.4.19 from https://dist.ipfs.io/go-ipfs/v0.4.19/go-ipfs-source.tar.gz
Golang: go1.11.2.linux-amd64
Machine it runs on: Linux 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ free -h # Before the application starts
              total        used        free      shared  buff/cache   available
Mem:           1.9G        126M        1.0G        580K        850M        1.6G
Swap:            0B          0B          0B

Note however that we're seeing similar issues on machines with 6GB of RAM

CPU: Intel(R) Atom(TM) CPU C3955 @ 2.10GHz

The machine is a fresh scaleway VPS that doesn't have any other services running on it.

Type:

panic

Description:

When the application starts, we add around 500 entries (nothing big, mostly html, css, jpg) into IPFS using Unixfs().Add(...). Then we leave the application running. Looking at the logs, it seems that after ~11 hours, the OOM killer kills the app which then generates about 55MB long log file. The log file spits out traces to ~14000 goroutines most of which are hanging somewhere in semacquire, select, syscall, chan receive and syn.Cond.Wait. These goroutines have been stuck there anywhere between 1 and 673 minutes.

Here is the log (compressed to 3.3MB)

[Stebalien]

So, you definitely ran out of memory and it looks like QUIC is the culprit (looks like quic-go/quic-go#1811).

[Stebalien]

Depends on libp2p/go-libp2p-quic-transport#53

[Stebalien]

@inetic could you re-try with the latest master?

[inetic]

Sorry, I was off for a few days. Unfortunately we've removed our cmake scripts recently to test directly from git and we currently only download go-ipfs through https://dist.ipfs.io/go-ipfs

[Stebalien]

Got it. Keep an eye out for a release.

[inetic]

@Stebalien sorry to bring this back. We've switched to v0.4.20. I let our programs run on the same VPS and saw this log when I came back to work today. Now this machine doesn't have a lot of RAM and it's entirely possible that our own code allocates more than it should, but greping through the log it seems there are 309 goroutines that have been stuck for more than 99 minutes.

Hope the log will be useful to you.

Also, we've switched back to being able to build go-ipfs from github. Not sure we'll be able to checkout an arbitrary commit, but we were able to test v0.4.20-rc1 and v0.4.20-rc2 by modifying our go.mod.

[Stebalien]

I assume you have about 700 connections? If so, that trace looks completely normal.

Could you post a heap trace? That shouldn't be taking gigabytes.

[inetic]

I assume you bad about 700 connections? If so, that trace looks completely normal.

Ah, OK, thanks.

Could you post a heap trace? That shouldn't be taking gigabytes.

Could you give me some pointers on how to do that? All I've ever done in golang was this C++ bindings to go-ipfs.

[Stebalien]

https://github.com/ipfs/go-ipfs/blob/master/docs/debug-guide.md#beginning