Support unicode superscripts

[yongqli]

Currently Rust allows unicode subscripts to continue identifiers; I've added support for superscripts as well.

See below for discussions:
#4928
https://users.rust-lang.org/t/allow-unicode-superscripts-in-identifiers/1892

r? @nikomatsakis

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @nikomatsakis (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. The way Github handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[killercup]

Maybe also test let σ⁴² = 2;

[emberian]

I'm not in favor. Get the Unicode Consortium to extend XID_continue, let's not deviate from them ad-hoc.

[emberian]

(Note also that I believe extending the surface syntax in this way would require an RFC.)

[petrochenkov]

-1 to this random extension.
#4928 can be resolved by moving from Default Identifier Syntax to richer Alternative Identifier Syntax (with amendments), as defined by UAX #31 (this is what C++ did, so Clang currently accept σ⁴² as an identifier).
The paper linked from #4928 contains a good overview of the problem and relevant standards.

[steveklabnik]

(Note also that I believe extending the surface syntax in this way would require an RFC.)

Given that unicode identifiers are still behind a feature flag, I don't think so. There would be one to stabilize it, of course...

[nagisa]

Alternatively, we could accept all characters from Letter,Modifier and Number,Other categories as non-first identifier character. This is what Haskell does.

[nagisa]

Given the controversy of this and the fact that this is a somewhat major language change, this should probably be an rfc.

[nikomatsakis]

Hmm. This has historically been an area with a (somewhat) surprising amount of controversy. I am not persuaded that we have to block on the Unicode consortium, but it is nice to follow existing precedent. Have other languages deviated from XID_Continue?

[nikomatsakis]

Many thanks to the author of this PR, but I am inclined not to accept it. I think instead someone should write-up an RFC describing a coherent policy, which we should then implement and stabilize. Speaking personally, I think that an RFC that can argue from some precedent (C++, Haskell, etc) will be much stronger than what which carves out a new space of identifiers, regardless of how good that space is (I am unclear whether this PR is carving out a new space or not, so I'm not implying anything about this PR per se).

UPDATE: Edited to remove somewhat unfortunate phrasing.

[nikomatsakis]

FWIW -- I'd love to be able to use unicode superscripts :)