Unable to upload to IPFS API: passing multiple files to ipfs.files.add sometimes fails

[lidel]

Background

I was playing with ipfs.files.api over js-ipfs-api and go-ipfs (PR #479) and noticed an odd edge case. Sometimes, for specific payloads, go-ipfs returns an error in the middle of payload:

{"Name":"T-RexFaceEmoji.jpg_original","Hash":"QmR1G8Jq4rNiYwCKyNWEZhNS7i8EWrCpz2xDjKsB9ofi82","Size":"23723"}
{"Name":"FlyingSaucerEmoji.jpg_original","Hash":"QmVWxHcLVHYpFJP6R1ZLBZcpAZsxyFMfU7BU84D2LAMSed","Size":"13071"}
{"Message":"http: invalid Read on closed Body","Code":0,"Type":"error"}

Then, upon receiving this response, js-ipfs-api does not throw an error, but silently ignores the last line and returns a partial result list with only two first items.

How to reproduce

I was able to reproduce it with brand new (dockerized) node and latest Companion from #479, so I assume the bug is real, but would be great if someone else was able to reproduce and comment if they got the same result:

1. Download three files from:
   https://ipfs.io/ipfs/QmUGmyGCJEntjyxgcBmptkZ34k7PsgnCzXPoqwiWBDTK6L and save them on your disk
2. Start local instance of go-ipfs v0.4.15.
   (I was able to reproduce the problem with ephemeral dockerized instance run via docker run --rm -it -p 8080:8080 -p 4001:4001 -p 127.0.0.1:5001:5001 ipfs/go-ipfs:v0.4.15)
3. Try to upload the three files from step 1 (all at once, select them with Ctrl) via IPFS Companion's Quick Upload screen, you should get the error below:
   

How to fix

Not sure yet, but there are two bugs at play:

1. go-ipfs returned http: invalid Read on closed Body for some reason
   • Upstream bug: Error while files.add over HTTP API (http: invalid Read on closed Body) kubo#5168
     • Temporary workaround (requires custom build of go-ipfs): ipfs/kubo@aad8f81
2. ipfs-api ignored the error and returned a partial response (2 instead of 4 hashes) as a valid result
   • Upstream bug: https://github.com/ipfs/js-ipfs-api/issues/797

For now, Companion checks if result list has a proper count and displays an error if things are fishy, but it should be addressed upstream.

I am parking it for now and will pick up in spare time.
Until then, if anyone has any quick ideas, feel free to update this issue with any new findings.

[lidel]

Upload via browser action fixed in #509 using workaround from ipfs/kubo#5168 (comment).
Landed in v2.4.0.10120 (Beta)

Upstream bugs can be tracked at: ipfs/kubo#5168 and https://github.com/ipfs/js-ipfs-api/issues/797

[lidel]

Reopening.

Chrome does not allow modification of Connection header in onBeforeSendHeaders and it remains at keep-alive, which means workaround from #509 won't work in Chrome:

Firefox

Chrome

[lidel]

Good news!

ipfs/kubo#5168 (comment) provides another workaround: send /api/v0/add with Expect: 100-continue

And guess what: Chrome does allow setting Expect via onBeforeSendHeaders WebExtension API:

This means we have a workaround that works for both Firefox AND Chrome.

Update: I checked Chrome docs and they do not include Expect on "restricted" list, but state this may change in future:

So it is unclear if this is a feature, but its safe to use it for now and fix uploads for Chrome users.

Will PR shortly.