ipfs add breaks somewhere in the middle of upload

[mlovci]

Commands to reproduce with docker, the equivalent commands running ipfs without docker also break:

export ipfs_staging=$HOME/ipfs
export ipfs_data=$HOME/ipfs_permanent
`boot2docker |grep export`
boot2docker ssh -L 5001:localhost:5001 -L 4001:localhost:4001 -L 8080:localhost:8080 -fN
docker run -d --name ipfs_host -v $ipfs_staging:/export -v $ipfs_data:/root/.go-ipfs -p 8080:8080 -p 4001:4001 -p 5001:5001 jbenet/go-ipfs:latest
sleep 30 && docker exec ipfs_host ipfs swarm peers
docker exec ipfs_host ipfs add -r /export/caprin_clip

Facts:

• $ipfs_staging has a 102MB directory of files called caprin_clip as seeded at QmZA1Dwb8ggkkm3GLKA2WeTDLLSAKWGqZ913EmsKLcYEnC
• docker exec ipfs_host ipfs add -r /export/caprin_clip produces the progress message, but stops at: 512.00 KB / 11.63 MB 4.30 %
• this also fails on other files (i.e. i tried to tar/gzip /root/.go-ipfs and upload it but it also broke)
• I'm running OSX 10.7.5 with 147GB free space on my disk and no apparent permissions problems.
• IRC user otimm suggested ulimit -n 2048 but that didn't fix the problem
• discussion on IRC #ipfs starting PM on April 13, 2015
• $ipfs_data is empty when i start the container

[notslang]

The same thing is happening for me with ipfs version 0.3.5 on Ubuntu. I'm trying to add 146.24GB of music (a mix of small MP3s, and some slightly larger FLAC files in a directory structure that's never more than 2 levels deep).

So far, it has broken at 1.29GB, 7.25GB, and 10.69GB. Each time it breaks, it doesn't output an error message... I just re-run the command and it quickly catches up to the point where it broke and continues from there. The ipfs datastore is on a disk with 1.3TB of free space and the RAM usage of the ipfs process is only at 113MB, so that's not too bad. CPU usage is oddly low - between 1% and 0% of my AMD FX-8150 (8-core). It's also probably not an issue with leaking file descriptors - it only ever has 11-12 files open at a time (some devices, some files from .ipfs/datastore, the music file being processed, the repo.lock, and a block).

Also, this only seems to be happening when I'm running the ipfs daemon at the same time. After killing the daemon & re-running the add command, I've gotten through 15GB (and counting) without it breaking.

[whyrusleeping]

@slang800 did you build from source? if so, could you provide us with the git commit sha? Are there any sorts of symlinks in the data youre adding?

[notslang]

did you build from source?

I just used go get -u github.com/ipfs/go-ipfs/cmd/ipfs. If there's a way to check the SHA that it got, then I'll do that, but I'm not familiar enough with go to know how.

any sorts of symlinks

Nope. You can actually see the finished data here

[ghost]

Except for memory leaks, ipfs add should be fixed -- please reopen if you still see the issue.