Implement stripping LD directives from IPLD Node, and transforming IPLD to JSON-LD

[mildred]

This is a work in progress, but I wanted to have a place to discuss implementation

@jbenet, can you review the test case ipld_test.go and tell me if you think this goes in the right direction. Every test case has a few keys I want ti implement:

• json: the JSON derived from the original document stripped from LD directives
• links: subset of json obtained using the same algorithm but focused on searching links
• jsonld: (not implemented yet) IPLD node transformed to JSON-LD

At first, I wrote flattening algorithm without using the Walk function you created, and I then noticed that I should converge with your implementation. I modified the Walk function to allow preprocessing of Nodes (transforming the JSON tree). This would allow walking either the IPLD node unmodified, the node without the LD directives, or the LD model. You can walk the model using three methods.

I still have to write a few more test cases, and I plan to remove most of the code in node_index.go and put it somewhere else.

[mildred]

I rebased my branch to get small and nice commits.

@jbenet, could you review this and tell me what you think ? What this PR does is escape keys in JSON for special characters like @, and add transformation to plain JSON without directives, and to JSON-LD.

The commit history tells you what you need to know:

• Change link definition {mlink:} instead of {@type:mlink,hash:}

• walk.go: functions now operate on []string instead of string path

  Use the more flexible slice of components instead of the /
  separated path. There is an easy conversion between the two.

• Walk: added ability to transform the node structure

• Implement extract directives from IPLD, and implement Links() on top of it

  This permit to specify JSON-LD directives without conflicting with links (have
  links possibly named "@context" is now possible because special characters in
  keys are escaped).

• Implement conversion to JSON-LD

[jbenet]

Change link definition {mlink:} instead of {@type:mlink,hash:}

ok, i think this matches https://github.com/diasdavid/node-ipld 👍

walk.go: functions now operate on []string instead of string path

i had it this way because it's a nicer UX to pass in a single string. I'm explicitly matching the UX expected by most golang users: http://golang.org/pkg/path/filepath/#WalkFunc 👎

Implement extract directives from IPLD
This permit to specify JSON-LD directives without conflicting with links (have
links possibly named "@context" is now possible because special characters in
keys are escaped).

Implement conversion to JSON-LD

I think this belongs in a different package, one specific to using JSON-LD with IPLD, not here.

I'm 👎 on including all this here.

[mildred]

i had it this way because it's a nicer UX to pass in a single string. I'm explicitly matching the UX expected by most golang users: http://golang.org/pkg/path/filepath/#WalkFunc 👎

It's nice to have a nice API, but we are not dealing with paths, really, but JSON keys that can sometimes contain '/' characters (as opposed to filesystem paths). If you want to handle JSON-LD or arbitrary JSON, you must be able to handle that nicely. And as a programmer, I prefer dealing with a []string than split a path and unescape / characters.

It's always possible to provide another function to have an API using plain string paths.

You might argue that you specifically disallow keys with / for links, and I would argue back that the Walk function is not restricted to finding links I used it to implement JSON and JSON-LD conversion (in further commits).

I think this belongs in a different package, one specific to using JSON-LD with IPLD, not here.

I would perhaps agree with you, but in the future. Having it here has the advantage that we don't forget JSON and JSON-LD compatibility. Moving it elsewhere means it's easy to make a change that will break this compatibility. Having tests to ensure that is something nice.

If ou don't think this belong in this repository, what's go-ipls for ? Is it just useful to serialize/unserialize data and find the list of links ? I would think this was also the place where you can use the IPLD format in various useful ways. If not here, where would this belong to ?

[mildred]

I have an additional question, how do you handle the case of the link named @context (example: I have a file with that name) if you don't implement plain JSON compatibility/conversion (that implements special characters escaping) in this package ?

And it won't be nice to do the work internally (escape the special characters) and not provide an external API to use that (forcing reimplementation elsewhere)

[jbenet]

I would perhaps agree with you, but in the future. Having it here has the advantage that we don't forget JSON and JSON-LD compatibility. Moving it elsewhere means it's easy to make a change that will break this compatibility. Having tests to ensure that is something nice.

Agreed about tests, but these can be in a different package.

If ou don't think this belong in this repository, what's go-ipls for ? Is it just useful to serialize/unserialize data and find the list of links

This package is just for the bare minimum needed to read/write ipfs data to disk/wire/etc. the bare minimum everyone is going to have to write in other languages, and understand, to grasp the format.

We can make it a subpackage of this repo. maybe jsonld/ or something?

I have an additional question, how do you handle the case of the link named @context (example: I have a file with that name) if you don't implement plain JSON compatibility/conversion (that implements special characters escaping) in this package

Yeah, as i was discussing with the json-ld folks, perhaps we should. @diasdavid?

And it won't be nice to do the work internally (escape the special characters) and not provide an external API to use that (forcing reimplementation elsewhere)

yeah if we'll do that, we should have functions here for sure.

[mildred]

This package is just for the bare minimum needed to read/write ipfs data to disk/wire/etc. the bare minimum everyone is going to have to write in other languages, and understand, to grasp the format.

Ok, should the Links() function be there as well, or should we move it in a subpackage ?

[jbenet]

Links() belongs here, it is part of the core spec. it's fine if it does not use your new code.

[mildred]

I updated this pull request, most of the code is still here but organized a bit differently:

• The Walk function takes back a path string argument. I think this is a wrong decision because if we need to use it, we'll have to convert it back to a string slice (I think) and this would cause unnecessary conversions.
• The Walk function is still modified to escape / characters in path elements. This makes it possible to use the Walk function for other things than just links (but arbitrary JSON structure)
• The Walk function is still modified to allow filtering while walking the data structure. This is necessary to allow handling of directives in JSON
• The algorithm that takes a node and strips its directives (@ prefixed keys) is still there, to prevent parsing directive as links. It is used in the Links() function and the algorithm is exposed a a public function of the module.
• The conversion to JSON-LD has been moved to the jsonld sub-package.

@jbenet If you're ok with the / character escaping in paths and the @ character escaping in JSON, I'd like for you to comment on the design and code of this PR. Else, we'd need to sit down and redefine what we need. My goal here is to get IPLD working so we can get work done with it. I just want it to be a solid foundation, with no escaping and conflict problems in the datastructure. And as you say, we can always iterate using multicodec.

Mildred

[jbenet]

let's not have this on the Node, make it a separate function, like:

func StripDirectives(n Node) Node { ... }

(want to keep the interface of Node very, very minimal. same reason things like walk are separate functions)

[jbenet]

@mildred thanks for the revision!

I updated this pull request, most of the code is still here but organized a bit differently:

• The Walk function takes back a path string argument. I think this is a wrong decision because if we need to use it, we'll have to convert it back to a string slice (I think) and this would cause unnecessary conversions.

ok thanks.

• The Walk function is still modified to escape / characters in path elements. This makes it possible to use the Walk function for other things than just links (but arbitrary JSON structure)

This escaping should just happen on the path string, not on the node itself.

• The Walk function is still modified to allow filtering while walking the data structure. This is necessary to allow handling of directives in JSON

Filtering may be ok.

• The algorithm that takes a node and strips its directives (@ prefixed keys) is still there, to prevent parsing directive as links. It is used in the Links() function and the algorithm is exposed a a public function of the module.

I'm still not sure about this.

• The conversion to JSON-LD has been moved to the jsonld sub-package.

thanks!

@jbenet If you're ok with the / character escaping in paths and the @ character escaping in JSON,

I think I am ok with this, and +1 on escaping / and leading @ (is it just leading @ or all @? what's less confusing to end users? probably all.)

However, before we pull the trigger, we should confirm what we win by doing this now instead of later? and whetehr doing it later is ok? Like, do we lose anything by providing a raw-json-without-and-no-@context-special-handling-at-all interface? (that just looks for specially formed link ojects within it)

If we can't do it in the future without breaking anything (i suspect we cant), then it makes sense to put it it now, but would be good to have this concretely shown why now. (I'm sorry if this is repeating oursleves-- this stuff is seriously confusing, and worry about getting it either wrong or incompatible with what @diasdavid is doing in node).

I'd like for you to comment on the design and code of this PR. Else, we'd need to sit down and redefine what we need. My goal here is to get IPLD working so we can get work done with it.

Excellent, me too. i've wanted this done for a long time. sorry to keep massaging this but just want to make sure because this is THE core api, it is the core of IPFS as a whole, so it best be very right, and very polished. Whatever we do here also means @diasdavid may have to change things on his end too, so i'd like to minimize that right now.

I just want it to be a solid foundation, with no escaping and conflict problems in the datastructure. And as you say, we can always iterate using multicodec.

Very much agreed!

Thanks for bearing with me, and pushing this stuff!

Juan

[mildred]

i do not understand why Walk needs to transform the node. the components in the path can be escaped just fine without touching the node itself.

The idea was riven by the compatibility we would have with JSON-LD. The problem is we want to nest index maps (the @container: @index), and JSON-LD doesn't permit it. So we must define a way to translate our model to JSON-LD.

By formulating that, I realize this algorithm shouldn't probably take place when looking for links, but rather when converting to JSON-LD. So the code can probably be moved in the jsonld package and the Links() function can be simplified.

Anyway, to understand how this algorithm works, we transform the following JSON (IPLD wire format):

{
  dir1: {
    @attrs: {
      mode: 0644,
      @index: files
    }
    README.md: { mlink: ... }
  }
}

To the following JSON-LD file:

{
  dir1: {
    mode: 0644,
    @context: { files: { @container: @index } } // not included currently in the JSON-LD, supposed to be part of the schema
    files: {
      README.md: { mlink: ... }
    }
  }
}

And, this was also converted (by mistake I now think) to the same JSON when stripping directives (minus the directives). I'll have to change that, thank you for finding this mistake. This would lead to the path dir1/files/README.md instead of dir1/README.md.

I'll answer the rest later (and ipdate the PR probably)

[mildred]

I think I am ok with this, and +1 on escaping / and leading @ (is it just leading @ or all @? what's less confusing to end users? probably all.)

What's implemented is escaping of all @ characters, not just the first. But I was wondering which one was the best when implementing it, and I'm still not sure.

However, before we pull the trigger, we should confirm what we win by doing this now instead of later? and whetehr doing it later is ok? Like, do we lose anything by providing a raw-json-without-and-no-@context-special-handling-at-all interface? (that just looks for specially formed link ojects within it)

We gain time to think about the format and its implications by not doing the escaping right now, and that would be welcomed. I find it difficult to have a clear idea on what it is that we want really.

• we can easily delay not implementing / escaping by requiring keys not to contain the / character at all. We could then improve the format without changing the multicodec name

• if we allow use of @ character without escaping right now, this means we will have to change the multicodec if we want to make use of it later. That will also mean to have two decoders in IPFS to allow for backwards compatibility.

  The @ character could be used to make IPLD wire format compatible with JSON-LD for documents containing a recursive tree structure (as opposed to IPLD documents describing a one level directory entry). This is because JSON-LD doesn't permit having multi-level indexed maps, they must be objects with a defined type between each index container.

  We coulld also use the / character which use has also been reserved for the same purpose (we could for instance say that instead of escaping normal JSON keys containing a @ with \@, we could escape directives with /, for example, we would put /@context instead of @context).

  Note that directives might be useful beyond just JSON-LD.

Having said that, I'm quite in favor of escaping @directive with /@directive, and key/with/slash with key//with//slash (I'd imagine).

[jbenet]

What's implemented is escaping of all @ characters, not just the first. But I was wondering which one was the best when implementing it, and I'm still not sure.

I think escaping all is "less magic-y" so we probably should do that

We gain time to think about the format and its implications by not doing the escaping right now, and that would be welcomed. I find it difficult to have a clear idea on what it is that we want really.

agreed. me too.

[rest]

I think we should probably escape @ now, if we're going to later (that way dont have to burn another multicodec that people have to support). We can then chose to escape or not escape / for now. (if we escape anything, we might as well escape / too...).

Having said that, I'm quite in favor of escaping @directive with /@directive, and key/with/slash with key//with//slash (I'd imagine).

what is your reason again for escaping with / ? was it a windows thing?

the problem is that escaping with \ is standard in lots and lots of systems, it's become the escape character.

• 
• and https://en.wikipedia.org/wiki/Escape_character

it would be most intuitive if we keep \. what do we lose on the windows side again?

[jbenet]

• let's also keep in mind IPFS permits undesirable paths ipfs/kubo#1710

[mildred]

Sorry, I wasn't available these past 4 days, but I'll work back on this issue starting soon.

what is your reason again for escaping with / ? was it a windows thing?

Windows, I wouldn't even think of it. I haven't been using it since like forever.

The thing is that / is already special as it is not allowed in path elements. Try to rename a file with a / in it, you won't be able to. The idea is that to parse links, it won't be necessary to process every JSON key and perform the unescape process. The algorithm is simple: if the key contains a /, it is not a valid pathname and isn't a link. Else, it's a pathname and no unescape step is necessary. It's just speeding up the process.

[jbenet]

That's bundling many assumptions. Also i do not think we should reuse / for escape, even if not chosing \ i do not think we should use / for precisely the reason that users tend to think of it as a path delimiter.

I think it is cleanest to stick to the very well-established expectation: special characters are escaped with \.

[mildred]

@jbenet, I'll probably close this PR and make another one with just the character escaping implemented. But before I do that, I'd like to get a few things right first:

On the characters we want to escape (the special characters mentioned before):

• We might want to escape the / character in pathnames, or simply disallow the / character (I'm in favor of the latter)
• Our JSON is not compatible right away with JSON-LD, we need a processing step during wich we need to parse IPLD-specific directives to shift the JSON objects around and make it look like JSON-LD. We need a special character that we will need to escape to introduce these directives:
  • Choose @ and make sure our directives don't clash with JSON-LD (what I implemented)
  • Choose /, we disallowed it from the path components, we don't have to escape it (easy, my prefered)
  • Choose \, TeX style
  • Choose #, C++ style
  • Choose %, #, $ or any other fancy character
• Me might also want to escape the JSON-LD directives (this the @ character) if we allow them in the JSON. We can also say we only allow IPLD specific directives and not escape the @ character (what I'd do).

Once we decided what we want to escape (/ or not, IPLD directive character, JSON-LD directive or not) we have to specify how to escape it:

• If / is the character we want to escape for directives, we don't need to escape it. Path components can't contain this character. (I'd like this solution)
• we can say we double the character to escape it (Ada style "I ""quote"" my words", I like it, it'simple).
  • file with @ character: file@1.txt encoded to file@@1.txt
  • directive: @prefix
• we can add a \ (or any escape character), C style: weird\file@name would be encoded to weird\\file\@name
• we can escape by adding the ASCII or unicode character in hexadecimal (URl encoding style): weird%file@name would be encoded to weird%25file%40name. We can pick something else than % to introduce the escape sequence (= remings me of quoted printable), or use unicode characters instead of ASCII byte. What's nice about it is that the escaped character (@ in this case) doesn't appear on the result string (if we want to escape / but still be able to split on /).

I don't like the \ escape character so much because it's used everywhere. If you have a string with escape sequences that must be embedded in a language that uses \ for escape sequences (like JSON), you end up with long strings of \\\\ that you must count to understand what it means.

For example the filename file@1.txt could be encoded for IPLS as file\@1.txt which would appear in JSON as "file\\\@1.txt". If you try to match that with a regexp, you'll band your head on the walls :)

Can we make a decision on those points? I think you already mentionned your preference for the \ escaping. But what special character do we want to escape? What special character should we choose for introducing our IPLD specific directives? I'm no longer convinced we should choose @ and risk conflicts with JSON-LD by inventing new directive names.

The next step would be to implement the character escaping, and at a later stage, implement IPLD directives (using the chosen special character) to transform our JSON to JSON-LD if we want to. But that's not as important as the rest.

[jbenet]

@jbenet, I'll probably close this PR and make another one with just the character escaping implemented. But before I do that, I'd like to get a few things right first:

thanks for all the help here!

On the characters we want to escape (the special characters mentioned before):

• We might want to escape the / character in pathnames, or simply disallow the / character (I'm in favor of the latter)

I think we should disallow it for now, but not close the door on allowing it and escaping it later.

• Our JSON is not compatible right away with JSON-LD

right. but to confirm, if users provide JSON-LD as input (their own) we can take it, right? (supposing we escape things as needed)

we need a processing step during wich we need to parse IPLD-specific directives to shift the JSON objects around and make it look like JSON-LD. We need a special character that we will need to escape to introduce these directives:

• Choose @ and make sure our directives don't clash with JSON-LD (what I implemented)

I'm for this one. I think @ is a good, safe choice. Don't think we'll clash much. and it's used as directives in lots of stuff already. (JSON-LD, CSS, LESS, python, java, ...)

(1) So: Let's say we use @ for our directives.

• Choose /, we disallowed it from the path components, we don't have to escape it (easy, my prefered)
• Choose \, TeX style
• Choose #, C++ style
• Choose %, #, $ or any other fancy character

$ is another option, and i've seen it used a lot. it will likely clash with straight dumps from databases, so will be annoying.

One worry of clashing with json-ld, is if we may want to have a directive like a type -- @type -- we clash, though we may be able to make it work with JSON-LD too, so might be fine. (or use @ipfs-type or something)

• Me might also want to escape the JSON-LD directives (this the @ character) if we allow them in the JSON. We can also say we only allow IPLD specific directives and not escape the @ character (what I'd do).

I think we should escape them. i want to allow people to give us any data (the paths being the only exception, and may not be an exception later). @s are very common too (email!).

(2) So: let's just escape @ with \.

Once we decided what we want to escape (/ or not, IPLD directive character, JSON-LD directive or not) we have to specify how to escape it:

• If / is the character we want to escape for directives, we don't need to escape it. Path components can't contain this character. (I'd like this solution)
• we can say we double the character to escape it (Ada style "I ""quote"" my words", I like it, it'simple).

This is common, but i've seen it confuse many people. So i've taught many programming classes in the past, so i've a sense of what trips people up, and having different escape characters across programs can be a very confusing thing. People can grasp double escapes much easier (though yes, it's a bit annoying too), than recalling which escape character is for which layered system.

• file with @ character: file@1.txt encoded to file@@1.txt
• directive: @prefix
  • we can add a \ (or any escape character), C style: weird\file@name would be encoded to weird\\file\@name
  • we can escape by adding the ASCII or unicode character in hexadecimal (URl encoding style): weird%file@name would be encoded to weird%25file%40name.

I really, really dislike URL encoding style. It's very frustrating for people.

We can pick something else than % to introduce the escape sequence (= remings me of quoted printable), or use unicode characters instead of ASCII byte. What's nice about it is that the escaped character (@ in this case) doesn't appear on the result string (if we want to escape / but still be able to split on /).

I don't like the \ escape character so much because it's used everywhere.

This is precisely why we need it. the cognitive overload on users, particularly people who are new to programming in general, is very very low.

If you have a string with escape sequences that must be embedded in a language that uses \ for escape sequences (like JSON), you end up with long strings of \\\\ that you must count to understand what it means.

Yes, but this is also the simplest, most common and familiar thing to do.

For example the filename file@1.txt could be encoded for IPLS as file\@1.txt which would appear in JSON as "file\\\@1.txt". If you try to match that with a regexp, you'll band your head on the walls :)

Regexping the raw data? you'd be regexping over cbor most of the time. if converting to json, can also pull out the strings. And yes, in the worst case, you count over \\\\ but not over something like \@\@$ and have to wonder what is an escape and what isn't. when these things layer deep (>3 systems), having the same escape character avoids much confusion, trials, and errors.

Can we make a decision on those points? I think you already mentionned your preference for the \ escaping. But what special character do we want to escape? What special character should we choose for introducing our IPLD specific directives? I'm no longer convinced we should choose @ and risk conflicts with JSON-LD by inventing new directive names.

Is the conflict risk very high? i really don't want to invent a new, complex language with all sorts of things. If at all possible it would be a watered down version of JSON-LD.

So far, I strongly think:

• Special character for directives: @
• Escape character: \

If clashes with JSON-LD is seen as a major problem, then:

• Special character for directives: $
• Escape character: \

But i think this ($) is going to complicate things more.

The next step would be to implement the character escaping, and at a later stage, implement IPLD directives (using the chosen special character) to transform our JSON to JSON-LD if we want to. But that's not as important as the rest.

Yep. Steps:

• character escaping
• IPLD directives (just context, right now?)
• IPLD directives to transform JSON to JSON-LD (can this be done inside the context? we could also use @ipld instead of @context and put everything there. )

Actually i really like the @ipld idea, either scoping everything there, or prefixing other directives with @ipld-. it may work with JSON-LD very well, if we can define mappings there to transform an "IPLD object --> JSON-LD document". and because it's all scoped under a namespaced thing, then we dont clash with other stuff. It may even be a suggestion worth making to JSON-LD, to scope "extensions" or "related formats".

[mildred]

right. but to confirm, if users provide JSON-LD as input (their own) we can take it, right? (supposing we escape things as needed)

Even without escaping, JSON-LD is JSON, so we can. With escaping we wouldn't create links from JSON-LD directives. The problem is that the links generated might not be the most interesting ones.

Is the conflict risk very high? i really don't want to invent a new, complex language with all sorts of things. If at all possible it would be a watered down version of JSON-LD.

The major need would be for wrapping index container in intermediate LD objects. I proposed @attr earlier and if we petition for it, we could imagine it being taken in the next JSON-LD standard. Although I'm not so sure ad they would also need to implement an escaping mechanism of their own.

So ok, \ because it is used everywhere, and @ not to have too much characters to escape. We cal always use @ipld-attrs if we would like to avoid clashes.

All JSON keys participate in IPLD links, except keys containing a / character or an unescaped @ character.

Yep. Steps:

• character escaping
  We just talked about it
• IPLD directives (just context, right now?)

What do you propose this should do? We allow its inclusion (we already do). I don't suppose you want to implement a full JSON-LD processor in IPLD, do you?

• IPLD directives to transform JSON to JSON-LD (can this be done inside the context? we could also use @ipld instead of @context and put everything there. )

The @context is not going to be ambiguous if we use it the same way JSON-LD does. However I don't see a way to wrap the index containers reliably from the context whn looking at the unixfs example. You can easily tell in the JSON itself when the file hierarchy stops, in the context, however, it's more complicated.

I'd say, we include it inline in the JSON (I'd love to have it on the context, I just don't see how).

[jbenet]

What do you propose this should do? We allow its inclusion (we already do). I don't suppose you want to implement a full JSON-LD processor in IPLD, do you?

haha no :) -- was thinking of how to handle the mlink: <path> thing, and a @context helps with that. but it's not priority right now. we can always clean that up in impls later without changing the format.

The @context is not going to be ambiguous if we use it the same way JSON-LD does. However I don't see a way to wrap the index containers reliably from the context whn looking at the unixfs example. You can easily tell in the JSON itself when the file hierarchy stops, in the context, however, it's more complicated.
I'd say, we include it inline in the JSON (I'd love to have it on the context, I just don't see how).

Ok. We can just punt on this until later. one step at a time :).

thanks for all the discussion here @mildred! 👏 👏

[mildred]

@jbenet What do you think of the code ?

[jbenet]

is it double because json? or what?

(outside of json, in our own datastruct, we may not need to escape twice?)

[jbenet]

(or maybe not, to preserve the data nicely. not sure)

[mildred]

It's double because it's the Go string escaping. Is \ still the Go escape character for double quote strings?

[jbenet]

ahh, right :) 👍 (i asked for it ;) )

[jbenet]

@mildred sorry for the delay here-- lost track. Code LGTM. I'm good to merge it. You?

[jbenet]

no need for ;. would've expected go fmt to remove it.

[jbenet]

I'll just merge it and we can go from there.

[mildred]

Thank you

[jbenet]

@mildred no, thank you, this is great! 👍 👏 👏