[BOUNTY] Implement UnixFSv1.5 in go-ipfs

[Stebalien]

This issue has a bounty!

Successfully closing this issue by producing a production-ready, mergeable PR can earn you not only the undying love of the IPFS community — it can net you a financial reward. See the current list of bounty issues and their values here.

The need in brief

Add UnisFSv1.5 metadata support per the spec https://github.com/ipfs/specs/blob/master/UNIXFS.md#data-format, which includes: specifying metadata, respecting existing metadata, and displaying metadata.

This aims to support large data import use cases (ex package managers), who need to retain file metadata, particularly last modified time (mtime), in order to selectively sync only data that has changed. Up until now if you wanted to host a large data set on IPFS, like a package manager's repository, it would be difficult to update - but with file mtime you can only reimport the changed files, making things much more efficient.

Deliverable

• Implement the new mode & mtime fields and obey the new rules around default modes & mtimes as per the metadata spec.
• Plumb this metadata all the way through from ipfs add on one IPFS node to ipfs get on another.
• An end-to-end sharness test that validates that this metadata is preserved.

Requirements

• This feature must be interoperable with js-ipfs's implementation of the UnixFSv1.5 spec.
• The behavior of ipfs add must not change except when the user has requested that metadata be preserved via the appropriate flags (see the implementation details below). Specifically, the final hash produced by ipfs add must not change.
• The final hash of files added with this feature enabled must match the hash produced by javascript.

Guidelines

• Please use the IPFS Go Contributing Guidelines as your north star — adherence to these guidelines are crucial when it comes to awarding a potential bounty!
• Please DO NOT bundle a general code refactor with the completion of this bounty!

[momack2]

For context, here is the js-ipfs unixfsv1.5 implementation done by @achingbrain: ipfs/js-ipfs#2621, and the blog post describing the shipped work: https://blog.ipfs.io/2020-02-13-js-ipfs-0-41/#unixfs-v1-5

If you want to start working on this, ping @Stebalien for a quick walkthrough of where to start and what already exists to build on top!

[Stebalien]

Implementation Notes

Implementing this feature will touch several sub-components of go-ipfs:

• The go-ipfs-files library will need to be updated to:
  • Support modification times and modes.
    • When reading from and writing to disk.
    • When reading from the web (WebFile).
    • When sending/receiving over HTTP using the "multipart" file types.
  • Expose modification times and modes via the Node interface.
• Implementations of the Node interface from go-ipfs-files will need to be updated:
  • github.com/ipfs/go-unixfs/file/unixfile.go
  • github.com/ipfs/go-ipfs-http-client/apifile.go
• The protobuf definitions in github.com/ipfs/go-unixfs/pb/unixfs.proto will need to be updated and regenerated according to the spec.
• The go-unixfs (de)serialization logic in go-unixfs will also need to be updated to (de)serialize modes/mtimes.
• The go-ipfs file adder (github.com/ipfs/go-ipfs/core/coreunix/add.go) will need to be extended to handle modification times.
• The go-ipfs "add" command (ipfs add) will need two new flags (defaulting to false):
  • --preserve-mtime -- record the file modification time when adding.
  • --preserve-mode -- record the file mode when adding.

The new go-ipfs-files Node interface should be:

// Node is a common interface for files, directories and other special files
type Node interface {
	io.Closer

	// Mode returns the file's Mode
	Mode() os.FileMode

	// ModTime returns the files last modification time.
	//
	// If the last modification time is unknown/unspecified, ok is false and
	// the mtime is the Unix epoch (default timestamp).
	ModTime() (mtime time.Time, ok bool)

	// Size returns size of this file (if this file is a directory, total size of
	// all files stored in the tree should be returned). Some implementations may
	// choose not to implement this
	Size() (int64, error)
}

[kalmi]

@Stebalien Hi, @momack2 said to ping you for a walkthrough if one would like to start working on this, but I think it is safe to assume that your last comment is that walkthrough, and I will just go ahead. Thanks!

[achingbrain]

ipfs add also needs the following new options:

• --mode - specify a file mode
• --mtime - specify the seconds component of the mtime
• --mtime-nsecs - specify the nanosecond component of the mtime

And there are two new mfs commands:

• ipfs files touch [path] (update mtime to the current time)
  • --mtime - optionally specify the seconds component of the mtime to update to
  • --mtime-nsecs - optionally specify the nanosecond component of the mtime to update to
• ipfs files chmod [mode] [path]

[momack2]

Hey @kalmi! How are things going? Did you get your questions answered / are you unblocked?

[kalmi]

Hi!

Thanks. I'm not blocked, I had some personal issues last week, sorry for the silence. I plan to spend most of the upcoming long weekend on this. So far I have got ipfs get to use perm modes passed from the tarwriter, and I might have overdone the handling of the edge cases that can happen during ipfs get, but probably not. I want it to update modes for files and directories that already exist without doubling the number of syscalls, and that proved to be a bit tricky.

Status (WIP, "done" is very loosely defined at this point, this list is bound to be incomplete):

• go-unixfs
  • Regenerate unixfs.pb.go
  • Update go-unixfs serialization logic 
• go-ipfs-files
  • Implement ModTime and Mode (default-returning) stub for all Node implementers
  • Change Node interface
  • Use ModTime and Mode in tarwriter (used by ipfs get)
  • serialfile
  • WebFile
    • Check if there is such a thing in js-ipfs
    • Use headers (What headers?)
  • multipart
    • Check used headers in js-ipfs
    • Use headers
  • filewriter
    • Find where this is actually used
  • others?
• go-ipfs-http-client (yet to investigate)
• ipfs get (this mostly touches whyrusleeping/tar-utils in a backward compatible way)
  • mode perms
    • umask is respected (maybe too much, even for preexisting file, that might be misguided)
    • mode perms are updated for preexisting directories and files
  • set non mode perm bits
  • set mtime
  • sharness
• ipfs add
  • go-ipfs/core/coreunix/add.go updated
  • --preserve-mtime implemented
  • --preserve-mode implemented
  • --mode
  • --mtime, --mtime-nsec
  • sharness
• mfs commands
  • touch
  • chmod
  • sharness
• Investigate OS time resolution differences
• Consider abusing NTFS alternate streams to store mode
• FUSE?

[Stebalien]

I'd drop the last two for now. Everything else looks good to me.

Note: If you're trying to find where different types/functions are being used, I recommend grep.app. E.g., .NewMultiFileReader.

[momack2]

Hey @kalmi! Any status update? Are you blocked?

[kalmi]

@momack2 I am making progress, albeit unfortunately it is slower than I originally expected due to external factors. Now I am at the point where I believe the get side of things to be mostly done, and I am moving on to extending add, and I hope to be able to write back the added files with metadata somewhen in the next few days. I wanted to test aginst jsipfs before this as a sanity check, but I had some bad experiences it with it, and eventually decided to push it to the end. In the meantime, I will use the hashes from its test cases for the go-ipfs tests.

I am making sure that all changes to the libraries are backward compatible. The Node interface change specified by Stebalien is not, but I guess that's reasonable and acceptable. When I have everything working, I will start making PRs in the proper merge order.

Do you have a reasonable timeframe in mind for this that I should be aware of?

[momack2]

Hey @kalmi! It'd be great to include this feature in the next release, go-ipfs 0.6. I think the feature cut for that release is ~2 weeks from Wednesday. Do you think that's a reasonable goal?

Note, the sooner to put up draft PRs for review (and the more modular/well tested they are) the easier it will be to land this quickly. =]

[achingbrain]

I wanted to test aginst jsipfs before this as a sanity check, but I had some bad experiences it with it

Could you open an issue on ipfs/js-ipfs please and we'll see if we can get to the bottom of the problem you had?

[Stebalien]

@kalmi Any status updates?

[kstuart]

@achingbrain looks like there are still some issues running the tests, specifically when running the browser tests process doesn't exist:

FAILED TESTS:
  interface-ipfs-core over ipfs-http-client tests against go-ipfs
    .add
      ✖ should add with mtime as hrtime
        Chrome Headless 91.0.4455.2 (Linux x86_64)
      TypeError: process.hrtime is not a function
          at Context.eval (interface-ipfs-core/src/add.js:269:29)

Could you please investigate, the package browser-hrtime might provide a solution.

[zacharywhitley]

Any idea what the current status of this is? I'd really like the feature and it seems soooooooo close. I'm not interested in the bounty but would really like to see it get over the finish line and if there's any way I could give it that final push I'd be happy to help out.

[kstuart]

@zacharywhitley so far this year it has been challenging finding the time to focus on this, I'll be looking to move the PRs out of draft within the next 6/8 weeks, if not sooner.

[markg85]

@kstuart I'm really curious about your progress here!

I would very much like to use this additional stat information in places where i'm (going to) list content from folders stored on IPFS.

Just for my information. This additional stat info you're providing is also there when using the HTTP API, right? Not just internal to the unixfs layer? I'm asking because i intent to add IPFS browse support in KODI as a VFS plugin where it in the backend would use the HTTP API (it's near certain that i will be doing this). Likewise i want to add a KIO SLAVE (it's for KDE's file browsing capabilities) to support IPFS natively in KDE applications too, it too would fetch file data via the HTTP API. It's unlikely i'll be doing the KDE side of things anytime soon though, that really is just a wishlist item for me personally that keeps getting pushed back.

[kstuart]

@markg85 yes, if stored with mode and/or time that information is returned from the HTTP API. As per screenshot it returns the same fields as js-ipfs, except for WithLocality field.

I need to review Mtime value, but aside from that you can rely on those fields being present if they were stored.

[zacharywhitley]

Really looking forward to this. Quick question that I could probably answer by looking at the branch but you can make things easy on my if you don't mind my masking. Is this going to include an update to fuse to pass the info on to mounts? That's what I really want it for. I'm looking to replace SDKMan with IPFS so installing a new JDK is as simple as pinning the directory for that version and updating a couple of env vars.

[markg85]

Nice!
Can't wait for this to land and an IPFS version be released with this!
I don't want to rush you... but which version to you "soft target" for including this? 0.11?

[kstuart]

@zacharywhitley it's not in scope for this ticket, don't know what the plan is there.

@markg85 all I can say is I'm currently working on it, barring any major issues it shouldn't be much longer.

[zacharywhitley]

Thanks. That's fine. I was planning on writing a separate fuse implementation that also supported MFS so as long as it's there.

[djdv]

@zacharywhitley
For what it's worth, a while ago I had code for this within the go-ipfs repo.
There's a rough demo of that version here: https://www.youtube.com/watch?v=FXi3yYO2H1w
(this would later see some changes)

More recently, I've been asked to make a standalone version of this, and have been porting those features over to it.
A rough demo of that is here: https://www.youtube.com/watch?v=q-hKANpM9Xg
While not done yet, the port of the MFS code is planned and should probably happen soon-ish.
Although I'll need to coordinate some means of actually syncing changes back to the go-ipfs node in cases where you want to target it's ipfs files root rather than just IPNS keys, or arbitrary CIDs. Basically something like ipns name publish for the files root, where I can just tell it to replace its root CID with another one.

If you need help with yours, feel free to ping me.

[zacharywhitley]

Oh wow. That's fantastic and thank you for the offer of help. I will definitely keep in touch. You're from Philly? I went to Drexel University and used to work up in King of Prussia.

I've always thought that IPFS mount was an unfortunate rough spot for IPFS that didn't get the love it deserved. It's called IPFS and you go to use it as a filesystem and it's a little disappointing. I guess this is more an MFS thing but I always thought it would be nice if it also supported snapshots. I'm thinking of it creating a hidden directory every time the root changes to move the old root to something like .snapshot-20210826-3489. It's immutable. I should be able to go to an old version.

[djdv]

You're from Philly?

I am, I'm at the borders of Tacony and Holmesburg. :^)

It's called IPFS and you go to use it as a filesystem and it's a little disappointing

Strongly agree. It was the only reason I downloaded it in the first place, only to be greeted with a "not supported yet" message.
And from what users tell me, the only platform left that it does work on isn't practically usable.
Someone posted a few days ago about a 75-80% CPU reduction using my initial unoptimized version, and a fraction of the time.
Compared to the current release. #5003 (comment)
Not too bad.

snapshots

Yeah this would be really cool. It's something I like a lot in systems like ZFS, and shouldn't be too difficult to come up with a scheme for in IPFS.

the port of the MFS code is planned and should probably happen soon-ish

For what it's worth I at least ported over the read-only portions of this for now. They're here.
It seemed to work, but I haven't wired up a way to pass a CID to it yet via the CLI. I need to fetch it from node using a different API. Unfortunately it looks like no progress has been made on this ipfs/interface-go-ipfs-core#19
So I'm going to have to come up with some roundabout solution for now. Likely I won't mount and sync the root, but instead some subdirectory like /mount, until we can do something in the IPFS APIs to be able to set the root, or manipulate it directly.

[ec1oud]

How's it going with this? Seems like little progress lately?

[kstuart]

Last update was late June which refactored/improved/fixed the initial implementation, added support for recursively adding directory trees with mode/modification time and restoring them to the file-system and archive files.

Left to do is supporting mfs chmod and touch commands on directories, I was not initially able to identify an implementation path in the time available for this so need to study the code-base some more, progress on this is dependent on my schedule which is quite tight.

I've avoided moving dependent PRs out of draft status until all are ready, however if the dependent PRs are merged in order they shouldn't break anything, so I'll probably look to enable that shortly to avoid potential future conflict.

if you're not aware progress and dependency tracking is in main PR #7754.

[kstuart]

This should now be feature complete and all related PRs have been promoted from Draft status.

[zacharywhitley]

oh hell ya!!! Thank you. I've really been looking forward to this. Being able to run executables out of a fuse mounted ipfs is going to be great.

[ec1oud]

Cool, sounds good.

Well I was hoping for arbitrary metadata too (extended attributes) but that's not in 1.5 apparently.