Add ability to specify CID in entries in addAll

[RangerMauve]

Hello, I've been messing around with the interface and I've found that I've been wanting to construct directories using existing directory and files.

My proposed workflow looks something like this:

• Have some existing files in a directory that I get via ipfs.ls
• Have some new files that I need to merge with the directory
• ipfs.addAll is invoked with the new files,
• As well, the old files are part of the list by specifying their cid instead of the content field
• A new directory gets constructed with the uploaded files and the new files

I think this would involve modifying the dagbuilder to detect entry.cid and to effectively pass it through as an importResult, though it might be necessary for users to specify an entire ImportResult rather than just the CID.

That would then be processed by the tree-builder as normal.

Would this be something that's in scope of the IPFS API? I would imagine it would make modifying existing data a lot easier. Would a PR be welcome that adds this functionality?

[BigLep]

HI @RangerMauve : what you're describing sounds a lot like how the MFS is implemented (see https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#the-mutable-files-api ). Is that sufficient for your needs?

[lidel]

My understanding is that you want more ergonomic replacement for legacy command ipfs object patch to add new data to an existing DAG without fetching and storing an entire DAG locally (lazy-loading).

If so, augmenting end-user API like ipfs add will be problematic because ipfs.add* operations pin by default, and pinning prefetches every block in the DAG.

This means that passing a CID that is not reachable could block or fail the entire operation, etc. Not saying improving the add API is a bad idea, just that there are wider UX considerations, and if we make changes only here, we risk diverging API and ergonomics between JS API and CLI and HTTP API in js-ipfs/go-ipfs.

I'm closing this, assuming we don't want to go there (due to complexity and the lazy-loading requirement).

Instead, like @BigLep suggested, use MFS as the lazy-loaded way of patching existing unixfs directories:

$ ipfs files mkdir /test-dir # create a new dir, or copy existig one to MFS
$ ipfs files cp /ipfs/yourCID/some-huge-dir /test-dir/new-big-subdir # add new file to the dir without fetching contents
$ ipfs files stat /test-dir | head -1 # read CID of updated dir

fysa we may get codec-agnostic ipfs dag path in the future (ipfs/kubo#4782), but for now MFS is the best way of doing lazy-loaded directory construction.