feat: move regular files api to top level, add addFromFs and addFromUrl

[daviddias]

Related to #118

[daviddias]

Moving all the non MFS commands to the top level. This preps the users for the future Files API as they will get used to call the add, cat, get and ls from there, but when the MFS parts get coalesced with the non MFS parts, they will just get the lovely surprise that there will be a "drive" looking thing that contains all the files they added.

[daviddias]

Added the addFromFs (no one should have to do this: https://github.com/ipfs/js-ipfs/blob/master/src/cli/commands/files/add.js#L18-L102) and addFromUrl.

Once ipfsx is ready, we can squash all these 5 adds into one function.

[daviddias]

Happy to work on these changes if everyone feels great (or ok) about them :)

[achingbrain]

Why not go all the way and make MFS top level? Considering most MFS commands also take /ipfs/QM.. as valid paths..

[daviddias]

@achingbrain I believe that is the plan for the future with the whole API redesign. Unfortunately there are some conflicts such as .ls and if we don't address them at the internals level, it will be hard to pick what is correct.

[alanshaw]

Yep, and IPFSX already has a unified ls:
ipfs-shipyard/ipfsx#1
https://github.com/alanshaw/ipfsx/blob/master/API.md#ls

[achingbrain]

I think we should jump straight to the ideal API, rather than make users update their code twice - once to this and then again to ipfsx.

[daviddias]

@achingbrain even if that means 4+ months?

In order for ipfsx to land, we will need:

• (A) Refactor the whole codebase plus test that a stream muxer powered by async generators will be equally performant as the one we have with Streams.
• (B) Agree on the de-facto files API

I believe that we should do these in parallel

[alanshaw]

I'm +1 on moving the files API stuff to the top level. IPFS's prime purpose is to deal with files, so these calls should be privileged. They are also the most widely used calls so saving on typing helps developers. WIN WIN.

I think that addFromURL and addFromFs are convenient right now, but these operations will become really easy if add starts to take an async iterator like IPFSX does and so I don't think they'll be needed.

Right now in IPFSX if I want to add these things I'd do something like:

// Add a file
const stream = fs.createReadStream('/path/to/file')
const { cid } = await node.add(stream).first()

// Add a URL
const response = await fetch('http://giffatron.com/cat.gif')
const { cid } = await node.add(response.body.getReader()).first()

Ok so, caveat, browser ReadableStream isn't a async iterator quite yet but we can shim temporarily.

[daviddias]

@alanshaw what about users that want to add entire directories from their local file system in Node.js (analogous to the CLI ipfs add)? Even with ipfsx, you would still need to do the globbing manually that is happening here https://github.com/ipfs/js-ipfs/blob/master/src/cli/commands/files/add.js#L18-L102 every single time a user is using a embedded node.

[hugomrdias]

I agree with @achingbrain we should not do a breaking change just to do another in a few months , I don't believe there sufficient value in it. We can improve docs and make small tweaks to improve experience though.
I think we should aim for ipfsx I looked into it just now for a second time didn't know it was already so complete. Looks freaking awesome @alanshaw please tell me how I can help !! We need this 🎉🎉🎉

[alanshaw]

@alanshaw what about users that want to add entire directories from their local file system in Node.js (analogous to the CLI ipfs add)?

Good point, I didn't consider that.

[daviddias]

@hugomrdias here is a good milestone towards that. Convert https://github.com/libp2p/js-libp2p-mplex to async-generators and run the stream muxer tests on it https://github.com/libp2p/interface-stream-muxer/tree/master/src and compare it with the current implementation in terms of speed and memory usage.

[daviddias]

Can I get a quick vote? I want to make sure to understand if people agree/disagree if people want to ship A and B -- #378 (comment) -- in one swipe.

• If you think that we should do A & B from feat: move regular files api to top level, add addFromFs and addFromUrl #378 (comment) refactor in parallel, 🎉 this comment
• If you think that we should do A & B from feat: move regular files api to top level, add addFromFs and addFromUrl #378 (comment) refactor simultaneously, 😄 this comment.
• If you don't feel confident of what this means exactly, 😕 this comment.

[alanshaw]

I think we should aim for ipfsx I looked into it just now for a second time didn't know it was already so complete. Looks freaking awesome @alanshaw please tell me how I can help !! We need this 🎉🎉🎉

❤️

There's so much to do. I'm trying to create RFC issues for everything now so that there can be some discussion before an implementation lands. The biggest contribution you can make is using it, suggesting changes, and proposing things you'd like to see.

Also, is there any opposition to moving the ipfsx repo to ipfs shipyard or is it too soon? I think by moving it to shipyard the community might feel it has some longevity and be more open to try it out and it'll also aid discovery.

[daviddias]

is there any opposition to moving the ipfsx repo to ipfs shipyard or is it too soon? I think by moving it to shipyard the community might feel it has some longevity and be more open to try it out and it'll also aid discovery.

Yeah, do it :)

[daviddias]

From IRC (important for context on the vote #378 (comment)):

[alanshaw]

#filescatreadablestream => #catreadablestream...and others ;)

[daviddias]

Have it all working on ipfs-inactive/js-ipfs-http-client#878. Now onto js-ipfs.

@alanshaw @hugomrdias @achingbrain Please review meanwhile the docs and tests in this PR. Thank you!

[alanshaw]

I've added some typo fixes. I'm assuming the tests for addFromFs, addFromUrl, addFromStream have been copied from js-ipfs-api so I haven't reviewed them in depth (since they should have already been reviewed!).