Adds BNF and text to extend LANGTAG to support base direction

[gkellogg]

… and term construtors for creating directional language-tagged strings.

Fixes #32. Depends on w3c/rdf-concepts#48.

---

Preview | Diff

[gkellogg]

Considering the fuzziness of the updated LANGTAG, we might want to consider splitting the terminals and use a grammar more like the following:

literal           ::= STRING_LITERAL_QUOTE ('^^' IRIREF | LANGTAG DIRECTION? )?
LANGTAG           ::= "@" [a-zA-Z]+ ( "-" [a-zA-Z0-9]+ )*
DIRECTION         ::= "--" [a-zA-Z0-9]+

The current text makes the term language tag fuzzy, as it is both associated with the language tag from RDF Concepts and the LANGTAG terminal production.

[Tpt]

Considering the fuzziness of the updated LANGTAG, we might want to consider splitting the terminals and use a grammar more like the following:

literal           ::= STRING_LITERAL_QUOTE ('^^' IRIREF | LANGTAG DIRECTION? )?
LANGTAG           ::= "@" [a-zA-Z]+ ( "-" [a-zA-Z0-9]+ )*
DIRECTION         ::= "--" [a-zA-Z0-9]+

I am not sure this rewriting works. It makes both DIRECTION and LANGTAG separated terminals so it allows whitespaces between them making "test"@en --ltr correct.

[afs]

I agree with @Tpt. We should not allow white space.

In Turtle, WS includes newlines

"test"
@en
--ltr

In addition to appearance, it has the effect that the token DIRECTION may be recognized on it's own. It makes the use of any the character sequence -- anywhere as a future feature problematic. This already applies to @, which is why we have the text about @prefix and @base in Turtle. It is unfortunate that white space was allowed between " and @ at all but that was many years ago.

[afs]

If the concern that LANGTAG is now fuzzy is widely shared, then rename the token as LANGDIR or LANGTAGDIR (pref: the former - don't explicitly mention TAG)

As direction only applies when a language tag is present, calling it LANGTAG does not concern me because the constructor text explains the situation. If the general feeling is that a new name is better, than that's fine by me as well.

[gkellogg]

IMO, the fuzziness is more about the use of language tag to refer to the value matched by LANGTAG as well as the concept of a language tag. It can be solved with some more creative editing.

[afs]

I think LANGDIR is used in the current latest but github isn't clear for me so I mentioned it.

I did find class="type langTag in the doc.

[TallTed]

Circumflex accent is not the same as caret, remove that before ^^.

It appears that there is not quite such a strict line between caret and circumflex.

https://en.wikipedia.org/wiki/Caret

Caret is the name used familiarly for the character ^, provided on most QWERTY keyboards by typing ⇧ Shift+6. The symbol has a variety of uses in programming and mathematics. The name "caret" arose from its visual similarity to the original proofreader's caret, a mark used in proofreading to indicate where a punctuation mark, word, or phrase should be inserted into a document. The formal ASCII standard (X3.64.1977) calls it a "circumflex".[1]

I think we should be more explicit in the descriptive text, as well as in the Unicode character notation. We should state that the ^^ are two Unicode CIRCUMFLEX ACCENT (not to be confused with the visually similar Unicode CARET, ‸; COMBINING CIRCUMFLEX ACCENT,  ̂; or FULLWIDTH CIRCUMFLEX ACCENT, ＾).

[afs]

There is a non-blocking issue with renaming LANGTAG as LANGDIR.

LANGDIR is a function name proposed for the accessor for the language direction. This fits into the general style of function naming in SPARQL (c.f. LANGMATCHES, isURI) and SPARQL already has LANG for the language tag accessor.

Possibilities:

• use LANG_DIR as the token name. This also reflects that fact that the token syntax has two parts. _ is already used in some token names.
• use LANGTAG_DIR or LANGTAGDIR for the token name.

[TallTed]

Blocked by rdf-concepts #48

[TallTed]

Similar notation of these Unicode code points should be added to the EBNF, where these characters are currently specified only visually, as '^^', which is not sufficiently specific.

Suggested change

	preceded by '<code>^^</code>'
	preceded by two circumflex accent characters, '<code>^^</code>'

[afs]

There's another "circumflex accent".

As a single name isn't clear, just say that "^ is codepoint U+005E".

[TallTed]

So far as I have found, while there are other characters where "circumflex accent" is part of their name, there is no other character whose full name is "circumflex accent".

If I'm just searching badly, please provide the codepoint(s) of such other character(s).

[gkellogg]

See suggested wording in w3c/rdf-turtle#34 (comment).

[TallTed]

There may be more of this. Trying to make all the specs spell out the special characters and sequences in the same way.