add with 1MB file triggers Uncaught Error: stream.push() after EOF

[olizilla]

Uploading a file > 1MB, as a pull-stream fails, with the error:

Uncaught Error: stream.push() after EOF
    at new NodeError (errors-browser.js:30)
    at readableAddChunk (_stream_readable.js:269)
    at PullDuplexStream.push../node_modules/readable-stream/lib/_stream_readable.js.Readable.push (_stream_readable.js:240)
    at next (index.js:55)
    at FileReader.loaded (index.js:22)

The issue is occuring in master on webui.ipfs.io since upgrading v30.1.2 from v29.1.0

See: ipfs/ipfs-webui#1010

[olizilla]

Also of note, in ipfs-webui we use https://github.com/tableflip/pull-file-reader

which has a default chunk size of 1MB https://github.com/tableflip/pull-file-reader/blob/master/index.js#L8 which suspiciously conincides with the threshold for triggering this issue.

[olizilla]

@hugomrdias this is blocking a relase of webui and desktop. Are you digging in to it or should I jump on it?

[olizilla]

investigating this some more...

• we dont test ipfs.add in the browser due to ipfs shutdown errors out on Windows ipfs/kubo#4078
• issue can be recreated in with the upload-file-via-browser example in this repo

[hugomrdias]

im on it, will report back asap

[olizilla]

@hugomrdias I'm on it. The problem seems to be in pull-file-reader

[olizilla]

There is nothing in pull-stream-reader to stop it from emitting the cb(true) for when the offset exeeds the filesize, while it is still waiting for a data callback from the FileReader...

in the above, with a 2Mb file, we see it trigger the offset > filesize callback before the final data chunk.

[hugomrdias]

But this worked before right? what changed in ipfs to trigger this problem?

[olizilla]

It sure did! I think something else must have been dealing with it, when it should always have been an error. I just added a quick, "count the pending chunks" check to pull-stream-reader and it fixes it! Will PR a cleaner fix now.

[olizilla]

[olizilla]

@hugomrdias oh no wait, you are right, this is super weird, the pull-stream sink should not be asking for more chunks before it gets a callback from the previous one... that's the weirdness that is causing this. I could guard against it in pull-file-reader, but it seems like we have a rogue pull-stream sink in ipfs now that is just pulling as fast as it can, rather than waiting for each chunk

[olizilla]

Moved the guard to pull-to-stream, and we have a fix! hugomrdias/pull-to-stream#1