IPFS add hangs

[rklaehn]

Version information:

$ ipfs version --all
go-ipfs version: 0.4.16-
Repo version: 7
System version: amd64/linux
Golang version: go1.10.2

Type:

Bug

Description:

ipfs add hangs when adding a small directory. We use an ipfs node for our artefact repository, so every time a ci job runs a directory gets added to this node. The node has a moderately big store. Limit is 100GB, current size is ~30 GB, with 250k nodes.

$ ipfs repo stat
NumObjects: 257005
RepoSize:   31613163529
StorageMax: 100000000000
RepoPath:   /data/ipfs/repo
Version:    fs-repo@7

We are using the flat file store. We have already increased the ulimit so this is not an issue.

When running ipfs add to add another directory, the command adds all files and computes the hash of the root directory. But then it just sits there and never terminates, so our CI (travis ci) is stuck.

If I try to add the temporary directory that is added to ipfs manually, I get the same result. Everything works, but the command never terminates:

added QmWQjYQ4J2NEN4WmUtwTE93pexXwqDvLgtXzxaKjXgDoUT tmp.p7kWktMJyW/favicon.ico
added QmW4u8C8UYfbeXU9iHnuk16tPhCK2KGnFr1dMFs3vutrn1 tmp.p7kWktMJyW/fonts/MaterialIcons-Regular.012cf6a10129e2275d79d6adac7f3b02.woff
added QmeKV18YuLsfg4kMBqY2KftAxts7VY62ktbhp9rQgToUWa tmp.p7kWktMJyW/fonts/MaterialIcons-Regular.570eb83859dc23dd0eec423a49e147fe.woff2
added QmSeQFVEKaqh6NVKPDTZrFyhx2aStvv8S3JssziDTmwfg1 tmp.p7kWktMJyW/fonts/MaterialIcons-Regular.a37b0c01c0baf1888ca812cc0508f6e2.ttf
added QmZfhST2ntifLy7bnAySLWGhrffZdM3ensKLms1uyAhy1E tmp.p7kWktMJyW/fonts/MaterialIcons-Regular.e79bfd88537def476913f3ed52f4f4b3.eot
added Qmc9QyM6RcDJwYS2L1UexMukz2mc3Kr3ykPvinf5ig5hDz tmp.p7kWktMJyW/fonts/TitilliumWeb-Bold.a9f517c1ea874345a3951d5ec4ea545c.ttf
added QmWvaJuY5dpCVpSkL5zLppjppviXdmeu1seKiadMLjMMzi tmp.p7kWktMJyW/fonts/TitilliumWeb-Regular.4f5848e6a415f5a4964a272d8b633164.ttf
added QmfSK5WAbpuDvzjaVW2dntc1zWhoqWgaNRevcoMj1uWVZ1 tmp.p7kWktMJyW/fonts/TitilliumWeb-SemiBold.600f2d0861eeddb39f0d8b0ff13ef59c.ttf
added QmXeyZ7DoKw8ifiTecHS3nm2W8jxe7ZDqR9nfcrfTLoaji tmp.p7kWktMJyW/fonts/fontawesome-webfont.674f50d287a8c48dc19ba404d20fe713.eot
added QmPumHB4BS7sJVc7eY7K95txfoNmNTnivYPEnLwSThheT8 tmp.p7kWktMJyW/fonts/fontawesome-webfont.af7ae505a9eed503f8b8e6982036873e.woff2
added QmbXtm3u3o22tXJz8Fn26Bgx4mYKSRTHHbJet56rjZmps7 tmp.p7kWktMJyW/fonts/fontawesome-webfont.b06871f281fee6b241d60582ae9369b9.ttf
added Qmd14HpB82q9g9sDS6ZXszXM8hNVxUax3kHKH6FYdtDaAm tmp.p7kWktMJyW/fonts/fontawesome-webfont.fee66e712a8a08eef5805a46892932ad.woff
added QmaThZzZKmYUcTehXzHjnhTYnv3wgdTqbRGYwBjLNVviLB tmp.p7kWktMJyW/iframe.html
added QmUNds4shKi7PEnbqq95teaAURB68aYhDBsCt5caajcYEz tmp.p7kWktMJyW/index.html
added QmTfaP78QX9vaGgNd7FK1qQw4owuiDNzs4Yd3xLC2Gtbv6 tmp.p7kWktMJyW/static/0.3ec4b32805871bc5c24a.bundle.js
added QmRcYuRrgLiq1tY9ArCz7vbnnNztVceUDcx6jGefCTcNAE tmp.p7kWktMJyW/static/0.3ec4b32805871bc5c24a.bundle.js.map
added QmYL2xJ6aAKZavcTK9Zzwq3AiTKhtb6KzvD9QJMq265X7Q tmp.p7kWktMJyW/static/manager.389f0ba23ed163c4af5c.bundle.js
added QmchoQQpHiocmtiNsgyJeu8ghYUGypmxq5XwHQsHTEW6vd tmp.p7kWktMJyW/static/manager.389f0ba23ed163c4af5c.bundle.js.map
added QmW2y6vZmKhgN3LYmQjt1B4oVFigcAn7rRxTV77vPHfV93 tmp.p7kWktMJyW/static/preview.8aa0dbba104b4a095088.bundle.js
added QmdYWdDKKAH5AiZd2oNW7adC1Mxq3Tmz9HSxtp7e5d23Sc tmp.p7kWktMJyW/static/preview.8aa0dbba104b4a095088.bundle.js.map
added Qmdz73NXrzaPVZxYyhni92E9CWsVqmYna3eMydijxgMxot tmp.p7kWktMJyW/static/runtime~manager.8f8d2dd9fa483b151cc1.bundle.js
added QmbyRR4nixXxmqmVujkUEQdD5JPzYZSF7goAbAUe7dNzzQ tmp.p7kWktMJyW/static/runtime~manager.8f8d2dd9fa483b151cc1.bundle.js.map
added QmPjkvPJJn8iDU879vH8gduki5bbnZFP3P8vVthRCVmJJk tmp.p7kWktMJyW/static/runtime~preview.8aa3302459456749015d.bundle.js
added QmPCUG6SFW875ENYGt1vrwzeofS74SA1i2JYZ2wVoZwATY tmp.p7kWktMJyW/static/runtime~preview.8aa3302459456749015d.bundle.js.map
added QmUf2LQ2KR9NBaJ8eRz9uRLvfDh382wUmDd3JfhQJzSwcp tmp.p7kWktMJyW/static/vendors~preview.890b95e9baef25de24ce.bundle.js
added Qmbtf47BBzTkvMH8NnWCn3cFERXEHNhM5fD2NMxDXGk3eF tmp.p7kWktMJyW/static/vendors~preview.890b95e9baef25de24ce.bundle.js.map
added QmdLekuj4atZ1VD4vp9potF8XVzYTSi8p8poCNYcZMygSx tmp.p7kWktMJyW/undefined/node_modules/pdfjs-dist/build/pdf.worker.min.js
added QmVVyGuU37AQ1sCYtLgjRiw2u9b9fo6j6BfNWHHESoBrTo tmp.p7kWktMJyW/fonts
added QmQa9rZxMHonqJU8AYE66knheZo4dSheVwP14CUPBQ7p9G tmp.p7kWktMJyW/static
added QmXmL2X7Zri8M42m4qkzEQUX5yvZMdKBhLsVC1W93i1Py4 tmp.p7kWktMJyW
 12.28 MiB / 12.31 MiB [=============================================================================================================================]

We tried restarting the node, but it does not help.

One thing that might be relevant is that there is a lot of recursive pinning going on on that same node, which could put some pressure on the file store. Maybe it is some kind of deadlock?

I am willing to provide more info. For now we are going to move CI to a different node and use the badger file store. I really hope that helps, since our entire team is stuck when CI fails...

This could be related to #3885 , but in our case the folder that is added is not large, so I don't consider it a dupe.

[Stebalien]

This is looks like: #5221. Basically, the slow part is adding the pin.

[rklaehn]

So you're saying that if I was adding without pinning it would work?

Thanks for looking at this, by the way.

[Stebalien]

It should. If it doesn't, please reopen. Note: you can add a "best effort" pin by linking the file into mfs (ipfs files cp /ipfs/MyHash /some_file_name).