New Tutorial: Mutable File System

[terichadbourne]

Following up on a Slack convo with @mikeal (cc @olizilla & @marcooliveira based on recent related convos)

The existing ProtoSchool workshops (basics and blog) use the DAG API, but we never introduce it as one of many APIs for working with IPFS. I had incorrectly thought that the data being stored in IPFS in these examples just happened to be JavaScript objects because it was a simple example to demonstrate, not realizing that it's the only type of data that can be stored with that particular API and that you need to use a different API to use IPFS for file transfer.

Since file transfer is a more practical use case that a beginner could see a need for, it would be nice to also build a simple lesson using the file API. We could either build a net new beginner workshop using the file API or replace the existing basics workshop. My inclination is to do the former and make it clear that one workshop is about transferring data and one about transferring files, linking between them so users can see the alternatives. We might also want to link from this lesson about transferring files yourself to the GUI file sharing service at share.ipfs.io so users have an example of the types of applications they could build using this API.

The blog workshop would remain using the DAG API, since it's a more advanced lesson about structured data that makes sense in that format.

Note that this is not being proposed as part of the 1.0 release.

[mikeal]

The thing I was stuck on mentally for a while was "how do we get a file into the browser based code workflow." I thought that we'd have to figure out a way to incorporate UI programming into the lesson.

The other day I thought of a much better solution. We create a new Lesson sub-component called "FileLesson" and in the exercise box we have a drop area for files. Instead of a run function that takes no arguments the run function will take the arguments of the files dropped into the drop area.

The first lesson should pass automatically if they drop anything in, just to get them accustomed to the workflow. The next lesson or two should just be a quick primer on working with File objects in the browser, cause the API is pretty rough around the edges.

Then we can get into how to store files in IPFS :)

[terichadbourne]

Would the lesson not be possible with them typing filepaths (my/computer/awesome.doc) because there's some lack of ability to access their file system from the editor, or is it just friendlier the way you're describing @mikeal?

[mikeal]

@terichadbourne not unless we have them download an Electron desktop app or something. The browser security model won't allow us any direct file access without the user explicitly dropping the file into our web application.

[mikeal]

Alternatively, instead of a "drop area" we could have a button that asks them to select a file. I usually implement a drop area that also kicks off that workflow if you click it instead of dropping a file in.

[lidel]

Some relevant prior art:

• js-ipfs/examples#js-ipfs-examples-and-tutorials
  • browser-browserify (old&ugly boilerplate with basic file picker)
  • exchange-files-in-browser (advanced demo with nice UI for dropping a file in)
  • browser-mfs (various MFS demos)
• js-ipfs-http-client/examples/upload-file-via-browser (basic "Choose File" file picker)

Creating a workshop for Files API (MFS) may be a good opportunity to refine or deprecate examples above.

ps. Files API has been recently refactored – ipfs/js-ipfs#1720

[marcooliveira]

@lidel's comment has a lot of nice examples. In terms of choosing the right use case for an introduction, I don't think I'm the right person to choose. I will add that we do have an example on https://js.ipfs.io/ as well, worth considering.

The key point, in my view, is that we should eventually go through all existing artefacts that have introductions/examples and, instead of having them live there, point to proto.school. If anything, proto.school could be embeddable, so that the user is kept in place, but the learning material should live in one place, for everyone's sanity, making it easier to maintain documentation and examples, further freeing people to upgrade APIs, and not fear outdated examples/doc being forgotten somewhere.

[mikeal]

Just so that we're all on the same page, by "File" we mean MFS :)

[terichadbourne]

@mikeal has started working on the functionality to add files in WIP PR #111. I'll circle around to this after launch to build out the lesson content.

[terichadbourne]

I plan to start work on this tutorial next week, and I think I'll need to poke around and explore existing documentation / tutorials on the FIle API before proposing the specific concepts and exercises we might want to introduce and asking for feedback in this issue.

My first step will be poking around the https://js.ipfs.io/ site and seeing what resources I find my way to first, circling back to this trail for some specific suggestions. If anyone would like to watch me attempt this as some free usability testing for your own projects, I'm happy to coordinate time to get on Zoom together. (@alanshaw I believe you expressed an interest in this.) Otherwise I'm more than happy to explore myself and request help / share feedback when I get lost.

I'm planning to set aside 9-12 Eastern (2-5 UK/Portugal) on Tuesday 1/22 to start poking around. If anyone would like to spectate for part of that period, feel free to send a Calendly invite (https://calendly.com/teri-chadbourne) for the time that works for you for a Zoom call.

After I feel I have a grip on what the File API is capable of, I'll circle back to the work @mikeal has done to enable file uploads in WIP PR #111 and see what additional instructions we'll need to provide just to introduce users to how to get files into their browser to start working with.

I'll be sharing ideas for lesson design here as I go, so I look forward to feedback and suggestions as I go!

[alanshaw]

👍 I'll join you Tuesday

[terichadbourne]

@alanshaw and I just spent some time going through the current resources available to learn the file API.

It seems like there might actually be 2 different tutorials worth building here:

** Tutorial 1: Adding and retrieving files using the file API **
Teaches regular part of API for files on IPFS (not MFS)
This would be very similar in scope to: https://github.com/ipfs/js-ipfs/tree/master/examples/ipfs-101

• Introduce how to use @mikeal's new functionality to get files from your desktop to where they can be played with in ProtoSchool
• Introduce the file API - it uses the dag API (which we learned in previous lessons) under the hood but has lots of extra functionality built in to make it work with files. We could have done this on our own, but the file API saves us time.
• Add a file with node.add
• See how the CID is returned
• (Use the CID to view it on the gateway?)
• Using that same CID, use node.cat to retrieve the file
• Show how to add a file using wrapWithDirectory: true so that there will be a path (essentially a filename) and not just a hash returned
• Use node.get to retrieve the file and see both path and content (highlight differences between cat and get)

** Tutorial 2: Editing files and directories with MFS **
Teaches MFS Files API

• Make sure everyone has completed the tutorial described above before this one.
• Upload some files to start with (or we preload the files?)
• Introduce basic concept of MFS, a "virtual file system on top of IPFS that exposes a Unix like API over a virtual directory. It enables users to write and read from paths without having to worry about updating the graph." Note that many commands are similar to what you use in the terminal for files on your own machine.
• Make a directory with file.mkdir
• Copy a file into the new directory with file.cp
• Move a file into it with file.mv
• Use file.ls to list the files in the directory
• Remove a file with file.rm
• Run file.ls again to show it's gone
• Try files.stat on a file and the directory
• Write to a .txt file with file.write using buffer to add a string
• Highlight the difference between node.add from the previous lesson (importing a file into IPFS) and file.write in this lesson (changing content of the file)
• Use file.read to read the contents
• Do stat again to see that the CID of the file and its containing directory have changed because the content changed
• Explain that the "mutable" nature of this file system is just virtual. Files and folders are still getting new CIDs as their content changes, as with everywhere in IPFS

There's been a separate proposal to add "next steps" pages at the end of each lesson that would include resources that can be used to continue learning, apply knowledge through project contributions, etc. Something like the much more complex exchange-files-in-browser tutorial mentioned by @lidel could be linked to after this second tutorial as an example of what you could build using the Files API. @alanshaw noted that a separate ProtoSchool tutorial on peer discovery and transports would be needed before you can use multiaddr and understand all the aspects of that tutorial.

I'd love feedback on the structure of these potential tutorials before I start attempting to start building them!

[lidel]

Sample files

* Upload some files to start with (or we preload the files?

I may be handy to provide a downloadable asset (eg. IPFS logo image) for later use in excercise:

• it saves time on picking a cat pic from the internet
• enables us to validate initial add went as expected
• as a side effect, it demonstrates the CID remains the same

Adding as a balanced tree (add) vs trickle-dag (files.write)

* Highlight the difference between `node.add` from the previous lesson (importing a file into IPFS) and `file.write` in this lesson (changing content of the file)

Writing down some notes, for additional clarity:

Both ipfs-inactive/support#45 and ipfs/ipfs-webui#676 paint a good picture why this difference is important to explain.

A TL;DR is in ipfs-inactive/support#45 (comment):

The hashes are different because ipfs files write uses different linking structure than ipfs add. ipfs files write's linking structure [trickle-dag, list of lists] is optimized for random seeking and writing after initial creation and ipfs add structure [merkle-dag, balanced tree] is optimized for reduction of link count.

add builds a new DAG while file.write is append-at-the-end of existing one.
Underling data is still deduplicated as long both operations use the same chunking settings.

My mental shortcut is that trickle-dag is useful for things like streaming (eg. video can be played before entire DAG is fetched) or append-only data structures.

If you feel it is useful to visualize the difference, I provided some reference materials in ipfs/ipfs-gui#11 (comment) (with links to DAG inspector (in old and new webui)

Bit worth highlighting:

Underling data is still deduplicated as they use the same chunking

cat vs file.read

Use file.read to read the contents

This method is missing from go-ipfs, perhaps we should use cat instead?

We should show difference between them.

AFAIK the only difference between cat and files.read is that files.read accepts MFS paths. Is there anything else here?

[alanshaw]

This method is missing from go-ipfs, perhaps we should use cat instead?

Is missing? https://docs.ipfs.io/reference/api/http/#api-v0-files-read

[lidel]

@alanshaw Ah, my bad. Updated #91 (comment), thanks!

@terichadbourne I think before Tutorial 2 we should include a lesson (or maybe entire turorial) about object.patch.addLink in the context of unixfs:

Creating directory trees on the fly

The goal of this lesson/tutorial would be to show how files can be organized into arbitrary directory trees using programmatic approach and emphasize how powerful it is thanks to content-addressing and deduplication provided by IPFS.

• We should show how to obtain a CID of an empty directory via ipfs.object.new('unixfs-dir')
  • this is important, we don't want to hardcode CID as the default can change in the future (due to hash funciton, new cid version, unixfsv2 etc)
• Then, how to attach a newly uploaded file to an empty directory using ipfs.object.patch.addLink(<dir>, <file>)
• Then attach another file to directory produced in previous step, show we now have a directory with two files

  directory2
  ├── file1
  └── file2
  

• Create a new, separate directory and add a file there, then add this new directory to the old one (which already had 2 files) and show the user that now we have a directory tree:

  directory4
  ├── directory3
  │   └── file3
  ├── file1
  └── file2
  

• Highlight that one can create directories using CID of data that is not present on local node (without the need for fetching it) and the same file will get deduplicated across all directories linking to it.

[terichadbourne]

@fsdiogo Here's one bug I've noticed:

Sometimes when I go back to a file upload lesson I've worked on previously, it somehow pre-populates the string "passed" instead of either my own cached code or the default code for the lesson. It's also not showing the "reset code" option when I do this, so I have to type a character in order to make that option appear.

[terichadbourne]

The previous PR with file upload functionality has been merged. Work in progress for the MFS tutorial is in this new PR: #200

[terichadbourne]

I've opened a new issue for any discussion of the non-MFS files tutorial at #203.

[terichadbourne]

Reworked lesson layout with @olizilla as follows for the MFS tutorial:
(Teaches MFS Files API)

1. Working with files in an IPFS node (text only) - what a node is, how we create one in ProtoSchool
2. Stat empty root to see you already have a CID
3. Working with files in ProtoSchool - how the file uploader works and how they can work with uploaded files in the code editor
4. Add a file to MFS (write) - includes brief into to what MFS is and how it's similar to file system on your computer
5. View the contents of a directory (ls)
6. Stat again to see the CID of root has changed - Explain that the "mutable" nature of this file system is just virtual. Files and folders are still getting new CIDs as their content changes, as with everywhere in IPFS
7. Add a directory (mkdir)
8. Move a file into it (mv)
9. Copy in an IPFS path (CID) that's a .txt (success.txt from Qme1zmi8dxBiVM7K9y5J3oPxiWWBgzA7n9M6tkmkz8kSwV if I pinned it correctly)
10. Use file.read to read the contents
11. rm all the things!
12. one last stat to show we got back to the same empty dir CID