Add Uchar module to the standard library.

[dbuenzli]

As I already made clear in previous discussions on the caml-list, I find that OCaml's current support for Unicode is outstanding (au propre comme au figuré).

I don't think introducing a Unicode string data structure and a corresponding syntax for literals would be a good thing do to. Since, if one wanted to that in a correct and useful way, it would entail importing a good deal of the Unicode processing machinery (e.g. normalization) in the compiler and I really think it's better to leave that outside the compiler. Unicode processing can perfectly be left to a set of modularized, external libraries. I also think it's actually a good idea to proceed that way as libraries are in a better position to evolve with the standard (e.g. newly encoded characters on Unicode standard updates may imply changes to normalisation results and would entail updates to the compiler).

There is however one thing that I really find missing to get utterly excellent Unicode support in OCaml: an abstract datatype, in the standard library, to represent an Unicode scalar value (by abusing terminology: an Unicode character). An Unicode scalar value is simply an integer in the ranges 0x0000…0xD7FF or 0xE000…0x10FFFF.

Such a data type would allow independent libraries dealing with unicode characters (e.g. ulex, camomile, uutf, uunf, uucp, uucd) to interchange data without relying on ints and as such strengthen the abstractions and guarantees a bit; avoid documentation warnings blabla that the given ints need to be in the above range, avoid needless (re)checks if data flows among modules, well you get the idea, the basic advantages of data abstraction...

This proposal simply adds such a minimal data type along with a few functions which by themselves don't do much except integrating with the standard library; doing real Unicode processing is left to external libraries, as it should be.

One question is whether a Pervasives.uchar type equal to Uchar.t should be introduced (not part of this proposal). I don't think it's essential, it could be a nice touch though.

[Chris00]

Should it be abstract or a private int?

[dbuenzli]

Good question. I personally have no problem in doing

match Uchar.to_int u with 
| 0x000A -> ...
... 

I don't know what's the stance of the dev team about using so called "Language extensions" in the stdlib.

[ghost]

You would still need to write Uchar.to_int u or write a coercion if t was defined as private int. Having t = private int is more an optimization: if the compiler knows that Uchar.t is always represented by an immediate value, the code generator can skip calls to caml_modify and/or float array checks.

[c-cube]

I like this idea of only adding standard types in the compiler library. It makes interoperability much easier and still doesn't require Inria people to support and maintain such complicated things as comprehensive unicode support... I don't see any drawback to this PR.

[whitequark]

I think this is an excellent idea!

[bobot]

Could you add a hash function? Just an alias for to_int, but it is useful for application with Hashtbl.Make.

[dbuenzli]

Right, added a hash function.

[bobot]

I agree it is a nice idea to add the abstract datatype in the standard library, and only that. What is the opinion of other unicode ocaml library makers? @yoriyuki @alainfrisch

[yoriyuki]

I do not see the point to add Uchar module without standard Unicode string data type and literals. They are needed for the precisely same reason to Uchar, interoperability between Unicode processing libraries. We do not need normalization etc. inside the stdlib.

To that said, adding Uchar is a good step toward more satisfactory Unicode support in OCaml. I have only minor comments.

• Code points like 0xFFFF are also non-character. Should we raise the error or not?
• Should we mark the function which raises the exception by, say _exn? (I know it is a controversial point)

[dbuenzli]

Le lundi, 14 juillet 2014 à 12:57, Yoriyuki Yamagata a écrit :

I do not see the point to add Uchar module without standard Unicode string data type and literals. They are needed for the precisely same reason to Uchar, interoperability between Unicode processing libraries. We do not need normalization etc. inside the stdlib.

I disagree with that, if you introduce an Unicode string data type and literals, then you most likely also want pattern matching on them. And if you want pattern matching on them you need to take normalization into account, in particular you want to be able to specify in which normalisation form your literal is supposed to be, otherwise it is useless, deceiving and could even be the source of a new class of potential security bugs. Formal unicode string literals without normalisation would be irresponsible IMHO.

It is currently perfectly possible to write unnormalized UTF-8 literals in OCaml which is entirely sufficient for many programs out there and a function away to translate into the representation of your particular library at the cost of a negligible initial runtime cost. Introducing the Uchar module greatly enhance the possibility of modular implementations of Unicode and allow for exemple ulex to talk to uunf with strong invariants guaranteed by the abstraction.

Code points like 0xFFFF are also non-character. Should we raise the error or not?
I would say no. For the following reasons (reference are to the pdf of Unicode 6.2):

1. Applications are allowed to use non-characters internally (D12 p. 68 Coded character sequence, bullet 2+3). Also on page 24. we have:

"Noncharacter code points are reserved for internal use, such as for sentinel values. They should never be interchanged. They do, however, have well-formed representations in Unicode encoding forms and survive conversions between encoding forms. This allows sentinel values to be preserved internally across Unicode encoding forms, even though they are not designed to be used in open interchange."

1. Applications should not interchange (serialize to UTF-X) non-characters (D14 p. 68) but stricto sensu these code points may happen in interchange as they do not produce invalid sequences of bytes: UTF-X are explicitely defined as a map from scalar values to byte code units (see 3.9 p. 89., D79 p. 90), non-characters are part of scalar values. More specifically on p. 560 we have:

"Applications are free to use any of these noncharacter code points internally but should never attempt to exchange them. If a noncharacter is received in open interchange, an application is not required to interpret it in any way. It is good practice, however, to recognize it as a noncharacter and to take appropriate action, such as replacing it with U+FFFD replacement character, to indicate the problem in the text. It is not recommended to simply delete noncharacter code points from such text, because of the potential security issues caused by deleting uninterpreted characters."

As such it's better if we have a way to represent these characters since UTF-X decoders can then pass them to the application which is then free to take the appropriate context dependent action.

Should we mark the function which raises the exception by, say _exn? (I know it is a controversial point)
I would say no. a) that's for people who like Hungarian notation b) Remember that Invalid_argument means programming error, you are not supposed to behave in a way that raises this exception and should not try to catch it, except at the toplevel as part of a general recovering procedure, see [1].

Best,

Daniel

[1] https://sympa.inria.fr/sympa/arc/caml-list/2007-10/msg00475.html

[yoriyuki]

For the latter two points, I now concur. I am not against to merging your patch.

For the first point,

I disagree with that, if you introduce an Unicode string data type and literals, then you most likely also want
pattern matching on them. And if you want pattern matching on them you need to take normalization into
account, in particular you want to be able to specify in which normalisation form your literal is supposed to
be, otherwise it is useless, deceiving and could even be the source of a new class of potential security bugs.
Formal unicode string literals without normalisation would be irresponsible IMHO.

If you mean that comparison and pattern matching should be always respect to canonical equivalence, and all string literals should be in normal forms, then I disagree with you. Code-point comparison has a place, like comparison which is used in binary trees, say, OCaml's Set. String literals in non-normalized form have also in place, for example, passing strings to legacy encodings. Unicode security is complex issue. Leave it to the programmer and we should satisfy that the necessary tools are provided by the compiler and libraries.

It is currently perfectly possible to write unnormalized UTF-8 literals in OCaml which is entirely sufficient for
many programs out there and a function away to translate into the representation of your particular library
at the cost of a negligible initial runtime cost.

Using the raw byte string which is encoded by UTF-8, as an alternative to proper Unicode string, is a troubling tendency. UTF-8 encoding can be broken, and creates serious security issues. It is much worse than your normalization apocalypse.

But, this topic (whether we need a standard Unicode string or not) is not related to your patch. If you want to continue the discussion, let us move to caml-list,

[dbuenzli]

Le lundi, 14 juillet 2014 à 16:08, Yoriyuki Yamagata a écrit :

If you mean that comparison and pattern matching should be always respect to canonical equivalence,

That's exactly not what I said. First I never talked about comparison at all, pattern matching is about equality and what I was precisely suggesting is that the equality you'd like (i.e. the underlying unicode equivalence) depends on context, which is why literals should be able to indicate the normal form you want them to be in, in order to be useful in pattern matching. You could say we want the literal notation without the pattern matching but that would feel odd as this would mismatch all other literal notations we have in the language.

Code-point comparison has a place, like comparison which is used in binary trees, say, OCaml's Set.
Again, never talked about comparison here. Pay attention to the words I use.

Unicode security is complex issue. Leave it to the programmer and we should satisfy that the necessary tools are provided by the compiler and libraries.

That's precisely the aim of this proposal.

Using the raw byte string which is encoded by UTF-8, as an alternative to proper Unicode string, is a troubling tendency. UTF-8 encoding can be broken, and creates serious security issues.

I don't think so, you are not supposed and can't use them blindly: if you do any processing with them you must have them go through some validating function (which will detect malformed sequences) if only to be able to normalize them so that you can match them against normalized user provided input.

Best,

Daniel

[yoriyuki]

That's exactly not what I said. First I never talked about comparison at all, pattern matching is about
equality and what I was precisely suggesting is that the equality you'd like (i.e. the underlying unicode
equivalence) depends on context, which is why literals should be able to indicate the normal form you want
them to be in, in order to be useful in pattern matching. You could say we want the literal notation without the
pattern matching but that would feel odd as this would mismatch all other literal notations we have in the
language.

Comparison has a broader meaning, which includes equality test, I think. Although my example of Set is using comparison in narrow sense, there is a plenty of the case which code-point equality test are used. (say, hash table)

As for pattern matching, code-point comparison is enough. If you need canonical equivalence or others, you can preprocess the input and making a normal form for literals by hand or use when clauses.

I don't think so, you are not supposed and can't use them blindly: if you do any processing with them you must have them go through some validating function (which will detect malformed sequences) if only to be able to normalize them so that you can match them against normalized user provided input.

Of course we must have them validated, but there is no guarantee whether such validation is performed from the type system. Having abstract Unicode string enforces validation, and increases safety.

[dbuenzli]

Le lundi, 14 juillet 2014 à 17:48, Yoriyuki Yamagata a écrit :

Comparison has a broader meaning, which includes equality test, I think. Although my example of Set is using comparison in narrow sense, there is a plenty of the case which code-point equality test are used. (say, hash table)

I think you are making this discussion more confusing than it should be. Binary comparison which includes binary equality has its uses, especially when you have normalized your inputs including your string literals and you actually know in which normal form they are.

As for pattern matching, code-point comparison is enough. If you need canonical equivalence or others, you can preprocess the input and making a normal form for literals by hand or use when clauses.

Well it's enough if you want people to write broken Unicode programs. Making a normal form by hand is certainly painful and when clauses are impossible: you need to normalize the literal constant of the pattern, otherwise you are just acting on variables which you can already perfectly do right now:

let ustr nf s = (* function that validates the UTF-8 encoded s and normalizes to nf *)
let cst = ustr `NFD "Éole"

match ustr `NFD x with
| x when x = cst -> ...

Overall I think that unicode string literals without pattern matching and normalization is just a waste of time for everybody.

Daniel

[yoriyuki]

I think you miss my points.

I think you are making this discussion more confusing than it should be. Binary comparison which includes binary equality has its uses, especially when you have normalized your inputs including your string literals and you actually know in which normal form they are.

My point here is that, there are cases that binary comparison and equality is enough or even necessary without normalization.

First examples of such kinds are data-structures which only requires consistent equality or ordering over Unicode string. The second example is to interact the legacy encoding, which, say, distinguishes Ω (unit) and Greek Ω.

Well it's enough if you want people to write broken Unicode programs. Making a normal form by hand is certainly painful and when clauses are impossible: you need to normalize the literal constant of the pattern, otherwise you are just acting on variables which you can already perfectly do right now:

let ustr nf s = (* function that validates the UTF-8 encoded s and normalizes to nf *)
let cst = ustr `NFD "Éole"

match ustr `NFD x with
| x when x = cst -> ...

Overall I think that unicode string literals without pattern matching and normalization is just a waste of time for everybody.

Again, you miss my point. My point is that, by introducing abstract Unicode string type, we can enforce that the internal representation of Unicode string (say, UTF-8) is valid by type system. We need string literal for just a convenience to write down such abstract data type. We do not need pattern matching for this purpose.

Beside, if you use UTF-8 encoded byte string to represent Unicode string, a.[0], a.[1]... are bytes of UTF-8 encoded string, not first and second Unicode characters. I think it is conceptually ugly.

[dbuenzli]

My point here is that, there are cases that binary comparison and equality is enough or even necessary without normalization.

They are certainly not the average case, there may be a few specific cases or some data sets may give you the illusion that this is the case, until you fall on a damned decomposed é. Even if you want to deal with something "relatively simple" like latin1 characters it's not going to be enough, better not lure programmers in fallacies; it seems they have already enough hard time understanding all of this. I think you miss both the social and technical point here.

Again, you miss my point. My point is that, by introducing abstract Unicode string type, we can enforce that the internal representation of Unicode string (say, UTF-8) is valid by type system.

I perfectly get that point: it has the same basis as this very proposal on which we agree. Sure it would be useful. But then it's much more contentious, for example I expect there will already be disagreement over the actual internal representation (e.g. I would make them immutable arrays of ints, not UTF-8 encoded strings), over what the minimal support should be (as we have right at the moment). Then if you want to introduce literals you will need to hook an UTF-8 decoder in the compiler then you will need to find an actual syntax in the very crowded surface syntax of OCaml, and this for not much gain in my opinion, that is unless we get pattern matching and normalization, which, unlike what you suggest is a basic need in most cases to perform correct unicode processing. I prefer nothing than broken things that will confuse everyone. I prefer small things that improve my coding life than nothing because the change was too invasive.

We need string literal for just a convenience to write down such abstract data type. We do not need pattern matching for this purpose.

I don't like the idea of having literals on which you cannot pattern match. This is conceptually ugly.

Beside, if you use UTF-8 encoded byte string to represent Unicode string, a.[0], a.[1]... are bytes of UTF-8 encoded string, not first and second Unicode characters.

As I already said on the caml-list indexing Unicode characters is worthless in general. From an abstract character point of view, for layout purposes, etc. direct indexing doesn't bring you anything, so I don't really care about that and in real programs it has never been a problem for me not to have direct indexing. The UTF-8 encoded sources files/strings may not be a perfect solution but it works well enough in real programs. Having that as a basis we can move to consolidate it, step by step.

I think it is conceptually ugly.

It's not a concept ! I was not made for that… It's a way to move forward. Progress is made in small steps. I'm already glad we don't have the conceptual mess other languages have with their Unicode support. Again, rather have nothing than broken things. The actual literal notation you'd like is a function call away, from a pragmatic point of view I'd say it is not at the moment (if ever) worth pursuing the idea (that is unless the dev. team is willing to commit to some form of useful unicode string support in the compiler).

[chambart]

It was suggested that this function should be named is_valid because we don't want to encourage to open this module and Uchar.is_char is ugly

[dbuenzli]

I don't see the connection to opening the modules. Why not another name but Uchar.is_valid wouldn't make sense at all, we are talking about a function that checks whether [u] can be represented by char. Maybe is_latin1 ? That would makes it less consistent with Uchar.of_char and Uchar.to_char but why not. What do you think ?

[alainfrisch]

I think the question was rather on is_uchar.

[dbuenzli]

Ah ! Makes more sense. Ok'll rename it.

[chambart]

Oups sorry for the misleading typo...

[mshinwell]

Daniel, in your first comment, you put in emphasis "in the standard library". Can you provide some more justification for that? (In particular, with the advent of OPAM simplifying the writing of new libraries, could we put this in a "base Unicode" library that the other Unicode libraries all depend on?)

[dbuenzli]

Le mardi, 4 novembre 2014 à 11:50, Mark Shinwell a écrit :

Daniel, in your first comment, you put in emphasis "in the standard library". Can you provide some more justification for that? (In particular, with the advent of OPAM simplifying the writing of new libraries, could we put this in a "base Unicode" library that the other Unicode libraries all depend on?)

We could of course publish this module separately but it would be a real maintenance burden (not code-wise, infrastructure-wise) for such small functionality — 31 loc which are basically cast in stone. In the end every program using some form of unicode character (and which don't these days ?) would end up with this tiny package in their dependency list and the only benefit would be, in my opinion, to introduce noise in the whole infrastructure; e.g. if you take uutf, uucp, ulex or camomile they don't have any dependencies at the moment. Having it in the standard library is also a better way of enforcing use of that representation for such a fundamental type.

Best,

Daniel

[dbuenzli]

Renamed Uchar.is_uchar to Uchar.is_valid.

[dbuenzli]

Removed UTF-8 comment as per request.

[damiendoligez]

I'm in favor of adding this to the stdlib.

[avsm]

Is there anything blocking this from being merged into trunk now? It would be useful to be able to start depending on it, and putting in a transitionary package into OPAM for older compiler revisions (as we did for bytes)

[gasche]

I wouldn't mind merging it if there was a clear consensus in favor, but right now I'm not sure there is -- apparently it wasn't discussed at the last developer meeting? Maybe you could ask other developers for their opinion.

[dbuenzli]

It seems this PR goes against the very idea of the stdlib. So let's just close this.

[lpw25]

Reopening. Whilst I appreciate Daniel's frustration, this is a pull request with fairly broad support that I would very much like to see merged.

[murmour]

I knew something was wrong with this otherwise stellar pull request: "non negative" should be either "non-negative" or "nonnegative" (in case you find hyphens outrageous). Thank God we caught this early!

[dbuenzli]

Thanks. Dash added.

[damiendoligez]

Let's merge it now.

[gasche]

@damiendoligez any reason not to merge it yourself?

[alainfrisch]

Minor nitpicks: can you add an entry to Changes and update copyright headers to 2015 for new files?

[hcarty]

@alainfrisch @damiendoligez If the Changes and copyright changes are holding a merge, I can submit a separate PR with those changes after this gets in.

[alainfrisch]

I can submit a separate PR with those changes after this gets in.

That would be very nice to you!

[dbuenzli]

I don't see why copyright dates should be changed they all correspond to the year when the code was written.

[alainfrisch]

Yeah ok, what matters is really the Changes file.