The dag and files distinction

[jbenet]

One complicating piece for the user is the distinction of ipfs and ipfs unix. right now ipfs {add, cat, ls} conflate the two. And there may be a future where different datastructures can be mounted to expose things differently.

Overall, we need to go through the codebase and:

• clearly distinguish merkledag and unixfs uses
• when we make tour clearly talk about this
• optimize flow with usability in mind, but not at the cost of expressivity (what you can do) or confusion (is it easy to understand what is going on)

NOTE: the following is speculative. these mount examples are a lot of work and not in our near future. They're just an example of how the different models work

note well what each of these commands does!!

> ipfs block mount blocks/
> cat blocks/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv
�+
"� Sfl ∂ÎJ�8J�_gÃH5è"p�·¥E}Ü�BR–˶™–��bar��+
"� Ä€�úU3:?����∂,∫€X3T∑ùrtÌ(Fl|≠˝üõ��baz�r�+
"� ï�ôbz¢≥UglàÃ∂�€`fl�˘d8-Úà�Ê°”o°É©��foo�
���

# like `ipfs object get`
> ipfs dag mount dag/
> ls dag/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv
bar
baz/
foo
> cat dag/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv/.ipfs-object
{
  "Links": [
    {
      "Name": "bar",
      "Hash": "QmTz3oc4gdpRMKP2sdGUPZTAGRngqjsi99BPoztyP53JMM",
      "Size": 12
    },
    {
      "Name": "baz",
      "Hash": "QmX1ebVUtfY11ZCpVmqyE5mDoN62SpLd8eLPpg5GGV1ABt",
      "Size": 114
    },
    {
      "Name": "foo",
      "Hash": "QmYNmQKp6SuaVrpgWRsPTgCQCnpxUYGq76YEKBXuj2N4H6",
      "Size": 12
    }
  ],
  "Data": "CAE="
}
> cat dag/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv/.ipfs-data
CAE=

# like `ipfs {cat, ls}` currently work
> ipfs unix mount unix/
> cat unix/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv
> ls unix/QmNtpA5TBNqHrKf3cLQ1AiUKXiE4JmUodbG5gXrajg8wdv
bar
baz/
foo

# 
> ipfs-bitcoin mount bitcoin/
> ls bitcoin/
HEAD
> ls bitcoin/HEAD
parent/
proof-of-work
transactions/
> ls bitcoin/HEAD/transaction/0/inputs
1LaxoTrQy51LnB289VmoSAgN6J6UrJbfL9
> ls bitcoin/HEAD/transaction/0/outputs
1QFbx3bKA8LABAGEaSe7EiP9JCxe2j4fN7
1taxinvBLwDAb1tjyTYzhcyb1fNKfivAB
1FTgzPJCbpCWYfF6VxPdmCMPUDBfygut2h
1DffZcvcQue3gn15G9tTshu2Y9YkzEZE2f
1HCFDzKx94uwtnvjJEXBZP12dseiNDK6if
> cat bitcoin/HEAD/transaction/0/full
input 1LaxoTrQy51LnB289VmoSAgN6J6UrJbfL9
output 1QFbx3bKA8LABAGEaSe7EiP9JCxe2j4fN7
output 1taxinvBLwDAb1tjyTYzhcyb1fNKfivAB
output 1FTgzPJCbpCWYfF6VxPdmCMPUDBfygut2h
output 1DffZcvcQue3gn15G9tTshu2Y9YkzEZE2f
output 1HCFDzKx94uwtnvjJEXBZP12dseiNDK6if

[btc]

Right now, coreio conflates the two concepts.

• decide whether it belongs in unixfs world.

https://github.com/jbenet/go-ipfs/tree/master/core/io

[whyrusleeping]

@jbenet for the alpha, would you say that this issue is mostly one of documentation?

[whyrusleeping]

@jbenet If youre gonna be done with your issues tonight, could you take this one on?

[jbenet]

• go merkledag api
• merkledag examples
• ipfs object to ipfs dag
• making an ipfs unixfs cmd

[daviddias]

👍 to ipfs object to ipfs dag

ipfs unixfs exists as files now, wooo :)

[timthelion]

I can verify that I'm confused. I find the following in particular to be especially startling:

$ ipfs dag get QmZsbEAP56wfrsWREvFneezd1RkUScjPJRUNYuqVSJUzU9 | ipfs dag put
zdpuAtM2TxQ3SQiMP99NoGusEQmy2jVjwwgzGew183s4DtCnP

What!? Different hash?

And the new has doesn't seem to work as a file:

http://ipfs.hobbs.cz/ipfs/zdpuAtM2TxQ3SQiMP99NoGusEQmy2jVjwwgzGew183s4DtCnP

shows:

ipfs cat /ipfs/zdpuAtM2TxQ3SQiMP99NoGusEQmy2jVjwwgzGew183s4DtCnP: unrecognized node type

[Stebalien]

So, the problem here is that IPFS nodes are stored in the DagProtobuf format (the legacy original^†, more limited IPLD format) but ipfs dag put encodes IPLD nodes in DagCBOR (the new, more flexible IPLD format). Basically, your data is getting re-encoded in a different format so the hash changes.

It would be great to fix this however, the only solution we (wel, @jbenet) can think of is to not re-encode. That is, make ipfs dag get HASH return encoded nodes by default (unless the user explicitly decodes them). Unfortunately, spitting out raw binary isn't very user-friendly.

_{^†@diasdavid pointed out to me that DagProtobuf isn't really "legacy", just the original format and the one used by IPFS.}

[Stebalien]

ipfs cat /ipfs/zdpuAtM2TxQ3SQiMP99NoGusEQmy2jVjwwgzGew183s4DtCnP: unrecognized node type

This is a known bug. Currently, IPFS only supports DagProtobuf nodes. Eventually, it should support all IPLD node types but this will take some work.

[jonnycrunch]

@Stebalien any update on ipfs supporting all IPLD node types? I'd like to publish IPLD objects to IPNS for my decentralized ID (DID) spec. Thanks!

[Stebalien]

@jonnycrunch that's actually a different issue. I was referring to the fact that unixfs should only care about the structure of the IPLD object, not the format. However, that won't let you stick arbitrary ipld objects into ipfs (as ipfs files, anyways).

You're issue is really that we haven't implemented the /ipld namespace yet. See: #4666