Introducing @dir

[gkellogg]

In some situations it is important/necessary to include the base direction of a text, alongside its language; see the “Requirements for Language and Direction Metadata in Data Formats” for further details. In practice, in a vanilla JSON, it would require something like:

"title": [ { "value": "Moby Dick", "lang": "en" },
           { "value": "موبي ديك", "lang": "ar"  "dir": "rtl"}
         ]  

(the example comes from that document).

At this moment, I believe the only way you can reasonably express that in JSON-LD is via cheating a bit:

"title": [ { "@value": "Moby Dick", "@language": "en" },
           { "@value": "موبي ديك",  "@language": "ar"  "dir": "rtl"}
         ]  

and making sure that the dir term is not defined in the relevant @context so that, when generating the RDF output, that term is simply ignored. But that also means that there is no round-tripping, that term will disappear after expansion.

The difficulty lies in the RDF layer, in fact; RDF does not have any means (alas!) to express text direction. On the other hand, this missing feature is a general I18N problem whenever JSON-LD is used (there were issues when developing the Web Annotation Model, these issues are popping up in the Web Publication work, etc.).

Here is what I would propose as a non-complete solution

1. Let us introduce a @dir term, alongside @language. This means this term can be used in place of dir above, ie, it is a bona-fide part of a string representation, and would therefore be kept in the compaction/expansion steps, can also be used for framing.
2. In JSON-LD 1.1, @dir is ignored when transforming into RDF. I.e., only the language tag would be used.
3. We may initiate some work in the RDF community to solve this issue. There may be several ways, each of them require the RDF community to chime in
   3.1. Define a mechanism of "parametrized" standard datatypes that represent a (language,direction) pair. One would then get something like[] ex:title "موبي ديك"^^rdf:internationalText(ar,rtl) ;
   3.2. Go for a "generalized" RDF where strings can also appear as subjects (that has been a matter of dispute for a long time...). That would give the possibility to add such attribute to texts like directions
   3.3. Some other mechanisms that I cannot think about
4. In a future JSON-LD 1.* the @dir value can be properly mapped onto an RDF representing the right choices (if such choices are worked out)

Original issue Introducing @dir ? #583

[iherman]

As the one who raised this: I am in favour of closing this.

[azaroth42]

Closing, wontfix, as new features should be compatible with the RDF data model.
WG Resolution: https://www.w3.org/2018/json-ld-wg/Meetings/Minutes/2018/2018-08-03-json-ld#section2-2

[azaroth42]

To be discussed at TPAC 2019, given lack of interest in a separate WG to fix text direction in RDF generally

[iherman]

This issue was discussed in a meeting.

• RESOLVED: (1/) We add <code>@direction</code> as a keyword to the JSON-LD Syntax to assert the base text direction of a literal string*
• RESOLVED: (2/) <code>@direction</code> is legal as part of a value object in the json-ld internal model but may not be used if <code>@type</code> is present*
• RESOLVED: (3/) <code>@direction</code> is legal in an expanded term definition if you could have used <code>@language</code> in that definition and applies to all values of that term in the same way as <code>@language*</code>
• RESOLVED: (4/) The value space of <code>@direction</code> is one of “ltr” and “rtl” (“auto” would have been the same as not asserting the keyword)*
• RESOLVED: (5/) We expand the possible values of language maps to be either a string (as per today) or a value object with direction and without language or type*
• RESOLVED: (6/) <code>@direction</code> can be present in a context node, and sets in the active context that the default direction for all string literals*
• RESOLVED: (7/) WG agrees that <code>@container:</code> <code>@direction</code> is not a useful feature for JSON-LD 1.1 (unlike <code>@container:</code> <code>@language)*</code>

View the transcript

Rob Sanderson:
{
"@context": {
"@vocab": "https://example.com/",
"schema": "https://schema.org/",
"name": "schema:name"
},
"name": {
"@value": "fish",
"@language": "en",
"@direction": "ltr"
}
}
Pierre-Antoine Champin: +1
Pierre-Antoine Champin: { “@context”: [“http://schema.org/”, { “@direction”: null }], “tag:p”: { “@value”: “foo”, “@language”: “en”, “@direction”: “ltr” } }
Manu Sporny: Discussion around use of @direction and how it affects JSON-LD 1.0 processors.
Manu Sporny: Seems that there is an issue wrt. the use of @vocab and @direction… processors are choking.
Manu Sporny: A bug exists in 1.0 and 1.1 that doesn’t flag the use of @direction and @vocab.
Rob Sanderson: Issue created: #259
Manu Sporny: Current 1.0 processors are going to ignore use of @direction in various places.
Pierre-Antoine Champin: {"@version": null} :-D
Rob Sanderson: Moving on to whether or terms can start with @ symbol.
… Currently, the spec is not clear wrt. SHOULD NOT vs. MUST NOT.
Benjamin Young: This is why HTML is not versioned.
… I’m less concerned about two documents processing two documents differently… we don’t want to touch old data, but we don’t want to prevent accomodation of the future at the same time.
Gregg Kellogg: If HTML pages had something that wasn’t interpreted, but then it would… like span, or blink, or marquee … then there could be issues.
Benjamin Young: When I use a newer processor, I’m aware that it’s going to look different… I don’t touch old data.
… I’d suggest you use a switching mechanism in processor… nuclear option is to use 1.1.
… If I point towards the past… I provide HTML5 document to HTML 3 processor, you don’t lose data
… If you give @version to 1.0 processor, it explodes.
Manu Sporny: Yes, but the only consumers are schema.org and they’ll just do 1.1.
Benjamin Young: yes, the evergreen browser argument
Pierre-Antoine Champin: Changing 1.0 parsers, if they don’t fail on 1.1 parsers, they’ll break on other things
Benjamin Young: See Feature Detection issue
Gregg Kellogg: We have to do something intelligent when we hit @container: subway – 1.0 will throw a syntax error, it should throw errors when it sees an issue. My feeling is that we should remove the explicit version announcement, and do feature detection when things are discovered and throw errors when they don’t understand.
Ivan Herman: Another reason why I’m in favor of incremental stuff, in a weaker way than what you say, is because we have to look ahead, we’ll have 1.1 now, but there’ll be a different discussion on how to proceed when this WG stops… listening to presentation from yesterday, next charter whatever 2.0 - essentially, admin and names put aside, we’d be in a position to put one feature that we want to add to the language like direction.
… If in a year from now, we add @direction, what do we do – version 1.1.1 or version 1.2?
… We get into versioning/naming issue… it’s difficult to maintain… sometimes they release 5.3, they do incremental changes, then implementations catch up… we might have the same sort of future ahead of us, we should be careful to not make things too difficult… 1.1 as a standard adds new features, not necessarily a thing called a 1.1 processor, we hope we have processors that implement everything, that’s the goal.
Manu Sporny: so. the original @version proposal was a list of features
… so if you were to process this document, you must support those features
… if a processor can’t support those, then it just blows up
… and 1.1 was just an alias for a bundle of features
… the issue we saw was that there might be a feature…well…
… one, announcing them doesn’t mean you’ll be using them
… or it might just get forgotten that you’ve left the “opt-in” around when you ended up not using it after all
… so, the feature detection works best when it’s also BC friendly
… and that turns out to be rather hard
… we’d have to go through each feature and determine how detectable they are
… but the reason against that is that it’s a huge amount of spec text
… and we weren’t sure that every new feature weren’t able to figure those out
… and they might determined it differently
… which is why we were ultimately in favor of 1.1
Ivan Herman: so does that mean if we go into continuous development
Manu Sporny: …yeah. I’m against that also
… I was opposed to this 1.1 work earlier…the 4 year gap was helpful for the stability of the data and community
Ivan Herman: k…but if we keep shipping, then we’d just keep shipping these version number things
Manu Sporny: yeah…but that gets complicated again
… and if you look at HTML5, feature detection is insane
… the earlier issues were about feature announcement not detection
Gregg Kellogg: the point of 1.1 was not so we’d have a 1.2
… it was so we’d be locking things down
… and that’s principally done in context processing
… and we did not do that properly in 1.0
… so, virtually, I’d say 90% of features are discovered in @context processing
… The original version was 1.1, was not to do 1.2, it was so we could detect things that should not be there in 1.0… if 1.2 added @direction, a 1.1 processor would fail if it saw it because it was not in the whitelist. 1.2 is not necessary to say @version 1.2 to do the right thing… virtually, 90% of feature issues are discovered in context processing, the exception is when processing expanded documents where there are some features used.
… the others show up in expansion
… We do look for those in 1.1 mode and 1.0 mode… frame processing mode…
… I think where we are detecting these things in expansion processing, maybe more needed for forward compatability, everything else is pretty much be in the @context processing phase.
Ivan Herman: If we had it today, @direction wouldn’t be detected in @context.
Gregg Kellogg: If we had it in the context it would be.
Ivan Herman: If I just use it as part of the value, it’s a 1.1 feature that I don’t see in the @context.
Gregg Kellogg: Anything that looks like a keyword and the processor doesn’t understand it, it should abort.
… I disagree w/ manu, I think we can do feature detection.
Manu Sporny: I don’t think I said we’d use 1.1 when doing JSON-LD 5.0
Ivan Herman: isn’t this an issue for stream processor?
Gregg Kellogg: Well, you could throw an error if you hit something…
… I think we’d be in a better position if we don’t use @version.
rob: We need to break for lunch soon.
Sanders Tolk: In some cases, it’s important to break down with newer features used… it’s a conscious choice to move from a 1.0 to a 1.1 interpreter…
… There is something to be said to an interpreter breaking on a feature it doesn’t understand, vs. a feature thing… parser, or implementation…
… Depending on level of confidence, you use one or the other… this is something that people should be aware of anyway.
Gregg Kellogg: In some cases, in order to get behavior, you should be in validation mode.
Benjamin Young: I want as much of data out of top of page, don’t care about reviews, give me what you can give me.
Sanders Tolk: To be able to post a snippet in a web page, does this have any unknown features for 1.1… so people can know, am I going outside of that…
Gregg Kellogg: It is using features that are defined in 1.1… if your parser dies, sometimes it helps if you have it all gathered.
6. Text directions, Part II
Rob Sanderson: The issue that we want to talk about is notion of internationalization and feature of text direction of strings, which is not possible in underlying RDF data model. Proposal that came out of discussion was that it would be better for the infrastructure overall if JSON-LD did text direction. Absence of it already existing at RDF concepts layer… it would be better back port it into RDF from JSON-LD.
Addison Phillips: We move the upper stories of the Eiffel tower first :)
… We are concerned that there is a window opened right now that so many specs are dependent on JSON-LD, we would like to make sure common serialization form for handling this, if we don’t have something, we might have non-interop solutions for this.
… We are interested in seeing if we can do something at this level, see if we can move this down to RDF level.
Rob Sanderson: One question that would be good to answer, why isn’t language sufficient to do this?
… Why can’t we just say English is ltr, arabic is rtl, and be done…
Addison Phillips: You can infer base direction, but not the same as base direction information, like when you don’t have accurate language metadata, or applying base direction isn’t applicable.
Richard Ishida: See Article on Base direction and language tag
Addison Phillips: Rather than doing coercion, we should have metadata slot rather than interfering… that’s the basis of it. Key aspect is that it has to do w/ mixed directional text… text that flows in one direction isn’t a problem, text that starts or ends with an opposite direction, bring multiple strings together, you have spillover effects that are desirable. That’s a challenge that we see as fairly fundamental.
Charles Neville: There is another problem in RDF in particular, which is that RDF doesn’t know anything about the language… lang=ar, you know it’s rtl… the only modern language that you can’t detect is azeri.
Richard Ishida: No, not true by a long shot.
Charles Neville: RDF doesn’t understand ar-kw ar-?? is all arabic
Richard Ishida: And new subtags are added all the time…
Charles Neville: You’d have to parse RDF language tags to get this partial solution
Richard Ishida: There was a point where we thought it would work… the URL that I just posed in IRC is the result of that.
… The conclusion we came to was that language wouldn’t do the job… wrote two articles on Kashmiri - the language tagging that I used in both documents was ks, didn’t put script info because I didn’t need to… there is a lot of human stuff going on there. You don’t know if it’s rtl or ltr. Even I didn’t feel like I needed to put subscript and relying on language.
Ivan Herman: Coming back to what Addison said, whatever we do, there are cases that we cannot handle directly, and we shouldn’t even go there. If we have text that internally mixes language, the only way we could handle that is to fall back on HTML encoding of content.
… metadata only on the one string.
… When you have mixed text, we can’t do that.
Addison Phillips: We just want base direction of whole string… there are bidi things that can happen on whole string. Supplied in form of HTML… set the base here… we’re not talking about internal changes
Dan Brickley: What RDF understands of the language structure… a lot of stuff you just pass through RDF… if you want to be able to write SPARQL queries, you’d have to have advanced knowledge of those language fields, you push the solution into the application.
Addison Phillips: Language tag mapping, don’t know what the subtags mean… SPARQL just does string matching, if you say ‘bs’ which stands for “bosnian”, you need a “bs range” to match.
… You still don’t need to know what the tags and subtags mean… the hyphens tell you what the boundaries mean… it’s deliberate, keep away from needing to read registry, use complex logic.
… If you are doing complex matching, you need application logic.
Dan Brickley: We don’t need to address fancy SPARQL query.
Sandera Stolk: I was going to talk about lang matches, doesn’t fix solution for direction… you’re probably aware of, if a solution is found for direction in JSON-LD, there may be multiple other types of language interpretation attributes to help with something that goes beyond direction… if there is a solution found in JSON-LD, it should also be a solution that should allow for other attributes as time goes by… it can be placed into the same construct.
Charles Neville: If we follow this plan, what breaks?
Ivan Herman: What’s the plan? Add syntax for @direction and add a new property
Charles Neville: See the proposed json-ld syntax
Manu Sporny: Group looks at directions in rdf-dir-literal document.
Ivan Herman: Note that the document we are referring to (link above) describes a range of possible solutions. We suggested implementing the json-ld syntax in json-ld as a starting point, because it doesn’t yet have a representation in RDF semantics
… We ended up saying this, if I put RDF hat on, there are downsides in practice, there are duplicated ways to represent literals in RDF… literal w/ language that RDF has today, in parallel we have another approach, might break some deployed OWL mechanisms, object properties, data properties, not a good solution, but possible solution…
Gregg Kellogg: We also discussed that this was a transitional solution.
Ivan Herman: Hope is in time, there is a more general mechanism in RDF, don’t know if this will happen. Current proposed approach, maps nicely to JSON-LD, covers what sander wants.
… What we discussed today, yes, this is doable, but what happens w/ existing JSON-LD data out there, and what are the effects… and answer is not that simple…
… Adding the feature is easy, what happens to the process is not a big deal….
Manu Sporny: I really pushed to fix this, for verifiable credentials to be able to move on…
… we rushed in to getting good ideas on the table. I am going to check whether this is really the right course.
… seems like language tag does 95% of what we want, a lot of times.
… so it is true that people extract some text direction out of the language tag
Addison Phillips: mostly right now it isn’t used…
… most people without a base direction appy the unicode algorithm.
Charles Neville: anecdata suggests that in practice, people are really implementing stuff withthe language tag and accept that as generally good enough for customers.
Manu Sporny: There is a reality on the ground that people are using the lang tag. All of these specs getting changed might not be as good as working out how to do this through teh lang tag.
… we could deal with it in the lang tag and not have to change the RDF layer
Benjamin Young: the mentioned list of specs https://w3c.github.io/rdf-dir-literal/draft-charter.html#normative
Addison Phillips: One thing is that by design implemnentations don’t need to know about the content of the language tag - and most of them don’t.
… there is a massive number of subtags.
Addison Phillips: One of the things I mentioned earlier, by design, implementations don’t need to know about subtags/language tags… we can maybe smuggle serialization schemes… the direction metadata doesn’t belong in the language tag, it’s semiorthogonal not totally unrelated, separate piece of data, I’ll admit that people have struggled… we failed to provide a mechanism to do this, we should have had this conversation before the paint was dry.
… we have looked, in https://w3c.github.io/rdf-dir-literal at ways to “smuggle” the information metadata in via the lanugage tag. It is semi-orthogonal, being a separate bit of data, that we have to be able to serialise and deserialise. There are a lot of places we suffer because we haven’t yet managed to solve the problem, since we didn’t do it from the start.
Richard Ishida: r12a’s slides start here: https://www.w3.org/International/talks/1810-paris/#metadata
Addison Phillips: People who have data catalogs who have base direction metadata, when people have this affordance, it helps to be able to do this… does it work a lot of the time, yes, but corner cases are really not that unusual…
… people with data catalogues who have the metadata have an easy time if they can set the direction directly, Carrying around lists of subtag matching works a lot of the time but is pretty ugly, and the corner cases are not that unusual.
… we launched a site in an RTL language, and hit a pile of bugs. I don’t always have convenient mechanisms to manage this, so I am interested in a way to provide direction metadata in JSON-LD
… Where I’m headed to here, I am not a purist, if we had an impure serialization scheme that had greater compat, that wouldn’t be bad…
… Talking w/ RDF community, we’re creating an interesting hack, doesn’t speak to extensibility… don’t know if it’s a hack… I’d be open to a serialization, looks hacky relative to other alternatives… maybe we can be compatible w/ existing implementation things.
Dan Brickley: I liked the idea of doing something in JSON-LD… if we could backport for 1.0, we could get something happening in 1.0, Google might get involved.
… We’ve encouraged people do to good solid JSON-LD 1.0 files, there are millions and millions of this….
… The smallest little thing we can do is important, if we can do some surface hackery, we can parse this stuff…
Gregg Kellogg: Yes, some transitional mechanism.
Ivan Herman: We are having a discussion about versioning discussion…
Gregg Kellogg: parameterize RDF algorithms stuff…
Dan Brickley: Parser being invoked w/ arguments, people run RDFa parser in certain mode.
Rob Sanderson: There are two separate questions… the first one, do we feel comfortable adding @direction to JSON-LD for @value objects regardless of how it transforms to RDF.
… There are things that don’t round trip in JSON-LD, such as indexes, there is precendent for non-roundtrippable stuff in JSON-LD… if cost to RDF layer is high, we can defer the serialization through into RDF… the proposal is 1) do we want to do this, to 2) how do we want to do this.
Charles Neville: I think we want to do this, happy to give up round trippability… what’s going to break?
… If that’s a fast path, this is a good solution.
… Forcing things that don’t parse language to parse language, that’s a big ecosystem thing.
Manu Sporny: I think the vast majority of appications don’t pay attention to the language tags, or direction.
Ivan Herman: In browsers, they don’t establish direction with language they do it with direction metadata.
Richard Ishida: https://github.com/w3c/string-meta/wiki/Base-direction-and-language-tags
Richard Ishida: The normal way of detecting language is to look at first strong character heuristics, for most things in strings, that’s the way to do it… far more efficient to look at first character… that works for 95% of time time, for the other 5%, that’s the solution we’re looking for, not having language, not having direction, and when direction on character is wrong…
Manu Sporny: I just wanted to make sure we are making an educated decision.
Addison Phillips: I think the right thing to have done is split the two – language and dir must be separated.
Richard Ishida: Yes, they’re not doing the same thing
sander: Pinpoint something wrt RDF - our company uses multiple language tags to discuss the same term, we find it’s difficult to find proper name for same term.
… We would be more reluctant if we adopt this that this is a different language encoding to …
Ivan Herman: If you look at this thing in RDF version, if there is no @direction, it maps against a literal and language tag… it’s some bnode with properties, it’s very different.
sander: I wanted to point out that if capturing such information would require another pattern, then such adoption for orgs to adopt direction, if that causes structural thing, that would hinder even looking for these things.
… smuggling direction in language might help adoption because it’s easier to do… let’s not make this have additional barriers.
Gregg Kellogg: To clarify, are we done at 3?
Rob Sanderson: This is something we want to do…
Gregg Kellogg: We get together we have an hour, we do the same discussion, we should put a stake in the ground. Something is better than nothing.
Manu Sporny: I’m committed and convinced to do @direction
Rob Sanderson: Is it possible for you to be back after break, danbri?
Dan Brickley: What about the future? When you have arabic, french, in Japanese?
… Are we defining something where future direction happens…?
Charles Neville: What about text orientation and stuff… LTR and RTL…
Addison Phillips: There is a high degree of confidence on number of text directions…
Rob Sanderson: on the Agenda is continue until exhaustion
… in favor of progress
… I think that is contentious to add the surface syntax
… so reopen that original issue
… and then look at RDF round tripping after that’s resolved
… the worst scenario is to go to CR with no solution
… it is better to go CR with @direction and without a mapping into RDF, than to not put in @direction
Gregg Kellogg: curious about danbri’s thoughts
Dan Brickley: that’s up to Google (Bing, etc), but we can encourage users of Schema.org to publish it
Rob Sanderson: (better to also have consensus about the RDF transitional reflection, but that seems more complicated)
Dan Brickley: but can’t promise SEO bots will consume it
Ivan Herman: if I put in a @language into my web site data, what happens
Dan Brickley: we certainly have the notion of languages of pages, etc.
… but the more narrow @language usage would be up to the various products/consumers of that data
… and I can’t speak for what they do
… I think it’s likely that we’re not doing a great job with @language–let alone @direction, etc.
… so I don’t think our tooling will catch fire, etc.
… it would be nice if there were an appendix or something that said even if you’re publishing in 1.0, that you can still put @direction in and hope for the best
Gregg Kellogg: there have been some calls in Schema.org to add some properties to say
… things like breadcrumbs might make use of the @list capability
… or strings that used a map structure
… where there would be a @context to understand those
… whereas at Google they’re just checking directly
… might you consider a similar pattern for @direction
Dan Brickley: I think you’re designing hacks for a system neither of us know much about :)
Rob Sanderson: let’s project some examples
Dan Brickley: I was thinking whatever Voodoo you work up–so in 1.1 it’s these little @ things…
Gregg Kellogg: so, yes, we have @language and soon, we think, @direction
… I can set an @language for the whole document or a certain object
… and when it gets expanded, you have that language data as well as the value
Dan Brickley: so each publishing web site would make use of its own context?
Benjamin Young: let’s make sure we’re all talking using the same terms
… I don’t we’re all sure what expanded form is, etc.
Rob Sanderson: [projects some examples]
Dan Brickley: for examples of google’s use of json-ld (and schema.org), see Search Gallery at https://developers.google.com/search/docs/guides/search-gallery
Manu Sporny: so I feel we’re overusing an unbuilt feature
… how often to we expect people to use the @direction feature actually?
… most authors won’t want to grow to great pains to express it
Addison Phillips: no, but most machines written to care will
… so people with direction metadata will want to consistently use it all over the place
Dan Brickley: Can’t we just use the dir attribute on a script tag?
… You can pick it up from the HTML dir element…
Benjamin Young: No, we’re not allowed to do that.
… You can send it into processing algorithms for JSON-LD…
Ivan Herman: We looked at this, HTML5 spec says to not use lang for script tags.
Dan Brickley: https://html.spec.whatwg.org/multipage/scripting.html#data-block
… is there a way to do this at the top of the file?
Ivan Herman: no
Gregg Kellogg: you can at the top of the file for @language
… and presumably we would for @direction
Ivan Herman: so, yes, it could be there, and on each value
… it is an open question whether the container mapping thing would also be used for language direction
… basically, yesterday’s proposal was to mimic @language, but for the @direction data
Benjamin Young: clearer? or worse?
Dan Brickley: but what if context’s aren’t used?
Dan Brickley: eg https://gist.github.com/danbri/9627452ec7dcbe63af9cb5a631acfdf8#file-gistfile1-txt
Manu Sporny: so you’d just use @language and @value directly as an object for the value of the property–i.e. a value object
Dan Brickley: so, here’s an example we’ve been playing with at Google
… we don’t want to do this on every field–as you can see it’d be painful and verbose
Gregg Kellogg: yeah, we would let you set for the whole document–as with @language
Manu Sporny: I feel like we’re handing people a foot gun
… everything would become ltr or rtl–even if it’s not supposed to
Gregg Kellogg: well, it depends on how they visualize the output to know if they’ve done it wrong
Manu Sporny: people don’t visualize it
Benjamin Young: they just put the JSON into text areas and call it done
Manu Sporny: yeah…most of them don’t even care about @language
Dan Brickley: so do it only down at the property level
Addison Phillips: I can see doing it there
Dan Brickley: would it make sense to have it in the schema languages?
… like certainly fields are “directable”?
… so gsin’s or isbn’s are not directable or language-able
Addison Phillips: isbn’s are a great example
… they have to be a string, includes letters, hyphens, etc.
… but it’s not in a language
… it’s a string literal
… and the world is few of these
… but it’s also full of language-bearing strings which have both language and direction
… in some cases that can be inferred, but in some cases we can’t
… so we do have direction fields in our databases to track that
Manu Sporny: so, one option is to allow @direction at the top and have it cascade to the entire document
… the other extreme/option is to put it on only the values
Ivan Herman: so it is up to Schema.org to decide if they want to promote @direction at the top
… we are here talking about the standard
… and I don’t see why we would disallow this
Addison Phillips: the reason you do this with @language is because you don’t want to repeat it
Ivan Herman: the same applies to @direction
Gregg Kellogg: we have various things to apply @language across the whole document
… I can say that ISBN is no language
… by setting @language: null on just that value
… for symmetries sake it would be odd for us to do this only for @language, but not @direction
… even though we might not have obvious use cases, there is an orthogonality argument to make to do it for the whole document
Ivan Herman: I agree
Addison Phillips: I agree
Gregg Kellogg: something I can see us doing is that if there’s a default @direction, then it could be limited to only things that bear a @language
Ivan Herman: no, ISBN is a good example
… they always go “rtl”
Addison Phillips: MAC addresses are a great example
… they draw backwards if they flow the wrong way
Gregg Kellogg: so, we should do this in all the same places we allow @language
group: [general positive responses]
Gregg Kellogg: I presume it doesn’t make since to have @direction maps?
Addison Phillips: @language maps are very useful
… the question then is how to do both if you use @direction
Gregg Kellogg: we could do nested objects inside nested objects
group: [growns]
tung: so we define only the language, and guess the direction?
Gregg Kellogg: you could leave the direction off
… you could provide both
… or the string could just use the default direction
tung: if the user provides a direction, but if the user does not state a direction it falls back to language?
Addison Phillips: the language map is the space in which you’d define a language
… but on each value
Duy-Tung Luu: what about vertical text?
Addison Phillips: vertical text is a presentational concern
Manu Sporny: I think we keep breaking our own principles here
… now we’re saying we’re using language for text direction when it’s not available?
Addison Phillips: they’re not completely independent
Manu Sporny: yeah…so the fact that this was thoroughly confusing to a bunch of developers…
… the stuff presented earlier about not guessing direction from language…but now we are
uests: David_Clarke
David Clarke: Yiddish can be in Romain characters or Hebrew characters
… so declaring the direction is still important
Manu Sporny: page-wide dir on HTML was maybe a mistake
Addison Phillips: no, there’s a reason to have it on the outside of the container for a bunch of reasons
… there is some linkage between language and direction
… and what we want is to give people the things they need to reach beyond what language provides
… if you ever watch folks attempting to develop or write this stuff, it’s really really hard
… we’re trying to provide enough affordance to take a string out of a page and have it present properly
… without having to introspect, etc.
… we’d like for folks to be able to express their data in all the various standards without having to rewrite all the things
… that’s why having direction is important here
… because it handles all the other bits beyond the text–bullet points, etc.
… works in html, etc.
*Proposed resolution: (1/) We add @direction as a keyword to the JSON-LD Syntax to assert the base text direction of a literal string** (Rob Sanderson)
Manu Sporny: +1
Gregg Kellogg: +1
Rob Sanderson: +1
Benjamin Young: it’s really similar to RDFa using a document level dir for all the things
… and being overridden elsewhere
Benjamin Young: +1
Simon Steyskal: +1
Ivan Herman: +1
Pierre-Antoine Champin: +1
Dan Brickley: +1
Resolution #3: (1/) We add @direction as a keyword to the JSON-LD Syntax to assert the base text direction of a literal string*
*Proposed resolution: (2/) @direction is legal as part of a value object in the json-ld internal model but may not be used if @type is present** (Rob Sanderson)
Manu Sporny: +1
Pierre-Antoine Champin: +1
Rob Sanderson: +1
Benjamin Young: +1
Ivan Herman: +1
Gregg Kellogg: +1
Simon Steyskal: +1
Resolution #4: (2/) @direction is legal as part of a value object in the json-ld internal model but may not be used if @type is present*
Duy-Tung Luu: why no @direction with @type?
Gregg Kellogg: because you can’t have @language with @type
tung: but what if the @type has a language?
Gregg Kellogg: that’s a different case
tung: can we use a class to define language?
Gregg Kellogg: that was considered
*Proposed resolution: (3/) @direction is legal in an expanded term definition if you could have used @language in that definition and applies to all values of that term in the same way as @language** (Rob Sanderson)
Rob Sanderson: I think we’re back in solution space, and should get back to resolutions
Gregg Kellogg: +1
Manu Sporny: +1
Benjamin Young: +1
Pierre-Antoine Champin: +1
Ivan Herman: +1
eg: {"label": {"@id": "rdfs:label", "@direction": "rtl"}}
Resolution #5: (3/) @direction is legal in an expanded term definition if you could have used @language in that definition and applies to all values of that term in the same way as @language*
*Proposed resolution: (4/) The value space of @direction is one of “ltr” and “rtl” (“auto” would have been the same as not asserting the keyword)** (Rob Sanderson)
Rob Sanderson: +1
Gregg Kellogg: +1
Ivan Herman: +1
Simon Steyskal: +1
Manu Sporny: +1
Benjamin Young: +1
Pierre-Antoine Champin: +1
Resolution #6: (4/) The value space of @direction is one of “ltr” and “rtl” (“auto” would have been the same as not asserting the keyword)*
*Proposed resolution: (5/) We expand the possible values of language maps to be either a string (as per today) or a value object with direction and without language or type** (Rob Sanderson)
Rob Sanderson: +1
Gregg Kellogg: +1
Pierre-Antoine Champin: +1
Ivan Herman: +1
Benjamin Young: +1
Manu Sporny: +1
eg: {“en”: {“@value”: “Rob”, “@direction”: “ltr”}}
Resolution #7: (5/) We expand the possible values of language maps to be either a string (as per today) or a value object with direction and without language or type*
*Proposed resolution: (6/) @direction can be present in a context node, and sets in the active context that the default direction for all string literals** (Rob Sanderson)
Rob Sanderson: +1
Gregg Kellogg: +1
Ivan Herman: +1
Manu Sporny: +1
Simon Steyskal: +1
Benjamin Young: +
Pierre-Antoine Champin: +1
Benjamin Young: +1
Resolution #8: (6/) @direction can be present in a context node, and sets in the active context that the default direction for all string literals*
Proposed resolution: (7 agrees that @container: @direction is not a useful feature for JSON-LD 1.1 (unlike @container: @language) (Rob Sanderson)
Rob Sanderson: +1
Manu Sporny: +1
Dan Brickley: +1
Pierre-Antoine Champin: +1
Benjamin Young: +1
Simon Steyskal: +1
Resolution #9: (7/) WG agrees that @container: @direction is not a useful feature for JSON-LD 1.1 (unlike @container: @language)*
Ivan Herman: +1

[iherman]

This issue was discussed in a meeting.

• RESOLVED: do not, by default, handle <code>@direction</code> expression in toRDF (or fromRDF)

View the transcript

Ivan Herman: what we agreed yesterday is the syntax. We will have direction in json-ld. Within json-ld world it just works
… however, the question is if we generate RDF, what do we do?
Ralph Swick: can we be more precise about “generate RDF”?
Ivan Herman: Produce an RDF graph with a literal with a direction
… then one approach is that the processor has two api run-time options. One option is to say forget it. It would just disappear and generate a traditional language literal
… the other is to define new RDF terms for language and direction, where instead of a literal it’s a blank node
… a kind of skos-xl like structure for literals
… the positive aspect of the structure is that it round trips through the graph
… the information isn’t lost
… the negative aspect is if RDF data goes this way, we create a parallel universe for language+direction literals
… json-ld users can pick which way they want
… but what’s the default
… don’t have an answer, perhaps via a coin flip
… that’s where we are
… there are other possibilities for how to reflect into RDF
… could also be a new datatype
… there’s always a parallel universe
Rob Sanderson: [looks at rdf literals and base direction doc]
Ivan Herman: question is whether there’s a problem to define two new terms in rdf: namespace
Ralph Swick: are you already comfortable with the processor option?
Gregg Kellogg: there are options for other things, though most people use default
… Not acceptable to not round trip. Should do something that preserves it.
… lots of discussion. Future group would be expected to replace or endorse the structure. A processor option would let the future processor adapt
… can’t keep it entirely in the JSON-LD domain
Rob Sanderson: I generally agree. two comments, though
… for CR, if we mark it at risk, and it turns out that….
Ivan Herman: mark what?
Rob Sanderson: well…whatever we define here
… and at implementation phase it turns out there’s push back against the structured approach
… because it results in an “always punning” scenario
… and if, like us, you don’t have punning as an option
… then the language tag approach would be preferred
… which would be my personal preference
… because it puts things in an expected place
… the SPARQL queries already have to search for “en-*”
… there are of course other cons
… but if it turns out that during implementation that the structured approach is not the best default
… do we have to go back through CR in order to change that default?
Gregg Kellogg: the i18n seemed to strongly object to the use (well…“hacking”) of the language tag
Ivan Herman: I’d even expect them to formally object to that solution
… at this point I don’t think it’s a choice
Rob Sanderson: Addison did say he’d rather have it in the language string than to not keep it at all
Manu Sporny: I don’t think the structure solves things. We haven’t had this for a long time.
Ralph Swick: And getting objections as a result
Manu Sporny: Yes. We should be careful to do this right. It’s a half way solution that introduces problems. We should do the thing that we think is the final solution
Rob Sanderson: .. we can be a forcing function. By default it’s dropped, it doesn’t impact them if you don’t want it
Manu Sporny: so opt in
… what it looks like in RDF - I agree that the i18n folks dont’ want it in the tag - but there could be a ^direction
… of the options I’ve seen that looks the cleanest
… don’t have to upgrade infrastructure now until it’s implemented
… BCP47 doesn’t have a caret in any language
… so a clean token to split the string on
… if that’s the final option, then we should force it in that direction
Ivan Herman: Am fine with opt in
… but opt in should mean generate something in RDF which is valid today
… generating invalid RDF is unusable anyway
… no idea what the community will come to a solution, if ever, but something that no processor will accept today is useless
… caret means we create another syntax
… could have created something with a string and lots of carets bla bla :)
… won’t have it today, might not ever
… so generating it is pointless. Only one that’s valid is the structure
Pierre-Antoine Champin: If we go the way of an option, we should consider the equivalent for rdf to json-ld
… that might be none trivial for an intermediate node solution
Gregg Kellogg: to manu’s point, the algorithms to make a dataset rather than nquads serialization, adding a direction is something a dataset would need to store
… we have uris bnodes and literals
… only if language tagged can they have a language
… literal would have a direction and a language … you’ve added a column in a store
… an rdf WG might decide to do that
… deafening silence was the reaction from the community
Ralph Swick: strong +1 to opt in
… default should be to break the fewest tools
… option to do something better is most useful if the better is something that tools can work with
… suggests some choices
… don’t know how it works now but could have non binary option, and in 1.2 could then have a 3 or 4 value option with different syntaxes
… urge to do something useful to existing tools
Manu Sporny: What goes along with the option could be a function to decide what to output
… e.g. RDF in some format
… if you want new triples then great. BUt it lets the community to decide how to output it so they can integrate with different toolsets
… no clear solution that meets all the requirements
… if the solution is new triples, we have no interest in implementing it
… don’t have a strong requirement to do that, so would just wait until the problem has a final solution
… also to note that the argument of existing ecosystem not doing this is invoked a lot. Invoked in 1.0 but we defined various things and they were pulled into 1.1
… arguing this is not the state of the world today can be valid, but can’t push the ecosystem forward
Ralph Swick: [so if Manu has no strong requirement at the moment, having infinite extensibility with a function call-out seems way overblown]
Manu Sporny: concerned by implementation complexity. People saying they can’t implement a 1.1 processor
… every time they look at the spec they see a massive set of features
… looks like it will cost a lot of money so they pick and choose a small set and pay attention to the things they think are important
Proposed resolution: do not, by default, handle @direction expression in toRDF (or fromRDF) (Benjamin Young)
Manu Sporny: every new feature makes it worse – too many options and we make it very difficult.
Manu Sporny: +1
Rob Sanderson: Clarification - the default option for toRDF is to drop it
Benjamin Young: +1
Ivan Herman: +1
Manu Sporny: a future proposal will add in more options
Pierre-Antoine Champin: +0.5
Rob Sanderson: +1
Gregg Kellogg: +0.1
Resolution #1: do not, by default, handle @direction expression in toRDF (or fromRDF)
Rob Sanderson: from before – data type is also a valid option that solves the objectproperty / dataproperty distinction in OWL

[iherman]

This issue was discussed in a meeting.

• RESOLVED: Add a parameter option to the API function toRDF, with three possible values: null or absent (do not process, the default), datatype (reflect to a datatype per 2.1.2.1.2) and bnode (reflect to a bnode structure per 2.1.4)
• RESOLVED: Add a value to the options parameter to the API function fromRDF, with the above three options plus “any” which will convert via any conversion function available as discovered in the data

View the transcript

Rob Sanderson: We agreed in the AM that we want a processor option, which defaults to drop the direction.
Ivan Herman: The datatype version familiy of language datatypes is also a viable option.
… 2.1.2.1.2 creates an ugly but valid datatype to describe language and direction #XX_YY.
… This wouldn’t be used in JSON-LD, but in the resulting quad.
… I now think this should be one option, and the other comound literal
Benjamin Young: This is adding switching in from/to RDF algorithms. 1.0 did not actually have API entrypoints for this.
… We could lower the bar for implementations by moving them into a separate interrace.
Pierre-Antoine Champin: I advocate the datatype solution over the compound node.
… The opt-in/out options would be different, but they would still be literals.
… In SPARQL, STR(literal) still gives you the expected results.
… Detecting these in the to/from algorithm would be easier and fewer corner-cases.
… On the semantics side, I think the DT solution makes it easy to define semantic interopration with existing language tags (intersecting value spaces).
… You could make interaction with existing lang-tagged strings smoother than as considering them as two different spieces of literals.
Ivan Herman: To be very precise, we’re talking about 2.1.2.1.2 in the document
Pierre-Antoine Champin: https://w3c.github.io/rdf-dir-literal/#datatype-family
Rob Sanderson: I wanted to disuss 2.1.1.2.
… I think we could subclass rdf:langString to create rtl/ltr versions of that sub-type.
… Hm, Guess not, given RDF spec.
Pierre-Antoine Champin: I agree with gkellogg that the option in fromRdf might sound strange, but the message might not be clear. If there’s no option, we bake in the processing, and close the door for other possible options.
Benjamin Young: section references come from https://w3c.github.io/rdf-dir-literal/#compound-literal
Proposed resolution: Add a parameter option to the API function toRDF, with three possible values: null or absent (do not process, the default), datatype (reflect to a datatype per 2.1.2.1.2) and bnode (reflect to a bnode structure per 2.1.4) (Rob Sanderson)
Rob Sanderson: +1
Benjamin Young: +1
Gregg Kellogg: +1
Ivan Herman: +1
Pierre-Antoine Champin: +1
Resolution #2: Add a parameter option to the API function toRDF, with three possible values: null or absent (do not process, the default), datatype (reflect to a datatype per 2.1.2.1.2) and bnode (reflect to a bnode structure per 2.1.4)
Benjamin Young: What if you have both bnodes and datatypes in an input? Could we convert them?
Proposed resolution: Add a value to the options parameter to the API function fromRDF, with the above three options plus “any” which will convert either datatype or bnode as discovered in the data (Rob Sanderson)
Rob Sanderson: +1
Ivan Herman: +1
Benjamin Young: +1
Gregg Kellogg: +1
Gregg Kellogg: For example `preserveDirection: enum(null, dt, combo, any)
Proposed resolution: Add a value to the options parameter to the API function fromRDF, with the above three options plus “any” which will convert via any conversion function available as discovered in the data (Rob Sanderson)
Pierre-Antoine Champin: I see this as an extension point, some may have more than two.
Ivan Herman: +1
Benjamin Young: +1
Gregg Kellogg: +1
Rob Sanderson: +1
Pierre-Antoine Champin: +1
Resolution #3: Add a value to the options parameter to the API function fromRDF, with the above three options plus “any” which will convert via any conversion function available as discovered in the data

[iherman]

This issue was discussed in a meeting.

• RESOLVED: the WG is happy to merge #276, but leaves it to the editors whether they do it now or after the API changes is also in a PR
• ACTION: create issue on values of language maps not including <code>@direction</code> (Gregg Kellogg)

View the transcript

Benjamin Young: See Issue Syntax#1§1
Benjamin Young: See Pending PR on @direction
Benjamin Young: no much to discuss; just to make sure everyone is aware of the PR
Gregg Kellogg: I will only merge it when the tests pass;
… currently working on my implementation;
… maybe only merge when we have an API PR.
… The syntax PR does not address the ability for a language map to contain value objects with @value and @direction.
… Currently they only support strings.
… So there is no way to override the direction in a language map.
Ivan Herman: I think we can defer this to a future version.
Benjamin Young: is there an issue for this already?
Ivan Herman: raise an issue, so we can make it clear if we decide to postpone it.
Action #3: create issue on values of language maps not including @direction (Gregg Kellogg)
Dave Longley: the syntax doc proposes 2 or 3 ways to represent this in RDF in order to roundtrip.
… We should only recommend one.
… We would have to maintain them for many years.
… My preferred option would be to drop the @direction by default,
… with an option to serialize it in the way we think is the best for future RDF.
… That would force people to update their RDF lib,
… but that what we were expecting if a dedicated WG had been created.
Ivan Herman: half of your wishes are already fulfilled:
… by default, the @direction is dropped.
… We picked two options for roundtripping:
… * the i18n family of datatypes,
… * the “compound literal” approach (not a very good name).
… At the meeting in Fukuoka, even around the table,
… there was no clear consensus about which one was the best option.
… The idea is to let the community decide which one they want to use.
Dave Longley: Why not a 3rd option where we add a property to a literal?
Ivan Herman: because this is not valid RDF.
… The failure to create a WG shows that the RDF community is not willing to change the core of RDF.
… The advantage of the two proposed approaches is that they work with the deployed RDF infrastructure,
… even though they require some application-level knowledge to be used.
Dave Longley: What I’m hearing is “more technical depts, and no interop”
… If this is something that people need, and we introduce a hack to get it work,
… people will adopt the hack.
Ivan Herman: you can call it a hack, but this is the only thing that people seem ready to accept.
… Trying to force RDF to change is a bigger hack in my opinion.
Dave Longley: JSON-LD 1.0 did influence the RDF WG at the time, to extend RDF.
Ivan Herman: this is the big difference; there was an RDF WG at the time.
… There is none at the moment, nor can I foresee one in the near future.
Dave Longley: sometimes, standard follow implementations, rather than the opposite.
Pierre-Antoine Champin: I’d argue that the proposed solutions are not so hacky from and RDF point of view
… I prefer the datatype solution
… but none of them seem a scandal to me
… if there was an RDF WG working on this, they would consider one of these as the future standard
Ivan Herman: +1 to pchampin
Gregg Kellogg: I agree, these solutions are reasonable from the point of view of RDF.
… Also, the default option is to drop direction.
… The majority of data will not use direction, and not produce RDF differently than today.
Dave Longley: dropping it is a good default;
… but I expect Digital Bazaar being force to keep it in some way.
… For example, in VC, when the graph is signed, we need to keep the whole information.
Gregg Kellogg: from a digital signature point of view,
… I would not consider the direction to be relevant.
… Would two values differing only by direction be really different?
Dave Longley: this is scary; that could be turned into a terrible vulnerability.
Ivan Herman: I would like a resolution to allow Greg to merge the direction PR;
… this has lingered for too long.
Proposed resolution: the WG is happy to merge #276, but leaves it to the editors whether they do it now or after the API changes is also in a PR (Ivan Herman)
Ivan Herman: +1
Gregg Kellogg: +1
Harold Solbrig: +1
Pierre-Antoine Champin: +1
Benjamin Young: +1
Tim Cole: +1
Dave Longley: +0
Ruben Taelman: +1
Resolution #2: the WG is happy to merge #276, but leaves it to the editors whether they do it now or after the API changes is also in a PR

[iherman]

With base direction added to the spec the issue is now part of the official I18N review; closing this one.