Silent crash ipfs-http-client@42.0.0

[aquiladev]

I use ipfs-http-client@42.0.0 in my project
When I try to add directory to IPFS I have silent crash quite often.
I try to upload an artifact of small client web app (~ 4.5Mb), in 1 min uploading stops (even when I set timeout much bigger). Code execution continues, but I don't receive rootHash
I use infura IPFS for it.

const IpfsHttpClient = require('ipfs-http-client');
const { globSource } = IpfsHttpClient;

const ipfs = IpfsHttpClient({ host, port, protocol, timeout });
const files = await all(globSource(path, { recursive: true }));
const source = await all(ipfs.add(files, { pin: true, timeout }));

for await (const file of source) {
  if (root === file.path)
    rootHash = file.cid.toString();
}

[hugomrdias]

reduced this to

const all = require('it-all')
const path = require('path')
const client = require('./packages/ipfs-http-client')

async function main () {
  console.log(await all(client.globSource(path.join(process.cwd(), 'examples'), { recursive: true })))
}

main()

but this works

const all = require('it-all')
const path = require('path')
const client = require('./packages/ipfs-http-client')

async function main () {
  console.log(await all(client.globSource(path.join(process.cwd(), 'docs'), { recursive: true })))
}

main()

run both inside the js-ipfs repo to repro

[achingbrain]

Both of these work for me, though the first is quite slow as there are 37304 files/folders under examples and only 27 under docs.

This might be related to #2838

@aquiladev could you put a small demo repo together with a runnable example of the crash please?

[aquiladev]

I upgraded to version 44.3.0, but still facing the issue.
I try to upload 4.5MB react app, there are 18 files only. I use Infura service fo that, but after 60 seconds, upload stops (some files uploaded), and I don't have root hash. I set timeout, but it does not help.

You can have a look on repo https://github.com/aquiladev/ipfs-action
There is a test [infura] should upload real dapp dir, you only need to change INPUT_PATH parameter to any directory with similar size.

[achingbrain]

There's a bug in your uploader code - you don't need to wrap the globSource & ipfs.add commands in await all(...) - that will just convert the iterator into an array, reading the whole thing into memory which is bad news.

Change:

const files = await all(globSource(path, { recursive: true })).catch((err) => { throw err; });
const source = await all(api.add(files, { pin: true, timeout })).catch((err) => { throw err; });

to:

const files = globSource(path, { recursive: true });
const source = api.add(files, { pin: true, timeout });

After making this change the tests start to pass for me, though the time they take varies quite a lot - between 5 seconds and well over minute, probably because Infura is overloaded.

Sometimes they fail, though the ipfs.add stream exits correctly. I think what's happening here is the request is actually timing out due to load at Infura's end but due to #2519 it looks like the request successfully ends.