Normative: List new Unicode v14 Script/Script_Extensions values

[mathiasbynens]

The following new values for the already-supported properties Script and Script_Extensions are added:

• Cypro_Minoan (Cpmn)
• Old_Uyghur (Ougr)
• Tangsa (Tnsa)
• Toto (Toto)
• Vithkuqi (Vith)

Issue: #2514
Tests: tc39/test262#3199

[bakkot]

@mathiasbynens I believe last time we said we were going to ask for consensus for these. Do you want to present this or would you like the editors to take care of it? I imagine it will be an extremely short agenda item either way.

[mathiasbynens]

My memory is that we would indeed ask for consensus for any new properties being added, but not for any new values to already-supported properties. Otherwise we’d be in this weird state where we claim to support the latest Unicode, and we claim to support Script_Extensions in property escapes, yet we somehow don’t support one of the Script_Extensions values in the latest Unicode release.

IMHO, asking for consensus on new values for already-supported properties would be equivalent to asking for TC39 consensus on the set of new ID_Start characters in every new Unicode release. I can’t imagine anyone objecting to a specific ID_Start character but not others, and similarly I cannot imagine anyone objecting to a specific Script_Extensions value but not another. We either support Script_Extensions as a whole, or we don’t — and that decision was already made as part of the \p{…} proposal.

The meeting notes from when this was last discussed are here: https://github.com/tc39/notes/blob/master/meetings/2020-06/june-2.md#introducing-unicode-support But I agree it doesn’t clearly capture a decision on this specific case. My memory and intuition goes back to when I first proposed \p{…}.

[bakkot]

@mathiasbynens That sounds reasonable, but I don't think we explicitly got consensus for that at the last meeting. I want to bring this PR to the next meeting and more explicitly get consensus on

• new properties will require plenary approval
• new values for existing properties will be marked normative but merged by editors without plenary approval

On the other hand, it's not totally clear to the editors what the purpose of the list of values in General_Category and Script/Script_Extensions serves, if the editors are always going to fast-track any updates to them without plenary approval, vs just having a normative reference to Unicode in place of those two tables.

[mathiasbynens]

@mathiasbynens That sounds reasonable, but I don't think we explicitly got consensus for that at the last meeting. I want to bring this PR to the next meeting and more explicitly get consensus on

• new properties will require plenary approval
• new values for existing properties will be marked normative but merged by editors without plenary approval

This sounds like re-establishing consensus on something that was already part of the approved \p{…} proposal, but I understand that we might be disagreeing on that.

As part of the \p{…} proposal we decided to support all Script, Script_Extensions, and General_Category character property values, and (at the very least) the intention was for that list of values to be kept in sync with upstream Unicode over time. I thought this intention was part of what was approved at the time.

If relitigated, it’d be problematic if this would not re-gain consensus for the reasons I described.

[michaelficarra]

@mathiasbynens any reason not to remove the tables that enumerate those values then?

[mathiasbynens]

@mathiasbynens any reason not to remove the tables that enumerate those values then?

We discussed that, no? https://github.com/tc39/notes/blob/master/meetings/2020-06/june-2.md#introducing-unicode-support They were originally added to reduce the risk of interoperability issues, and IMHO we shouldn’t change that.

[michaelficarra]

@mathiasbynens It was clear that we didn't want to remove the table that lists properties. It was not clear that we didn't want to remove the table that lists value.

[mathiasbynens]

Besides interop, listing the values + aliases explicitly is useful since we decided not to support loose matching in ECMAScript’s \p{…}. If we defer to the Unicode Standard, it becomes ambiguous what the “canonical” way to write each value / value alias really is, since Unicode doesn’t define it. (We generally use the casing that’s found in Unicode’s data files, but as far as Unicode is concerned, these labels are case-insensitive.)

[michaelficarra]

Added a topic for next plenary: tc39/agendas#1084

[mathiasbynens]

Per the 2021-12-14 TC39 meeting, patches that add new upstream Unicode values & value aliases to already-supported-in-ECMAScript properties do not require explicit consensus. Removing the label. Thanks for presenting this, @michaelficarra!

[bakkot]

@mathiasbynens One of the things we said we'd do as part of this is to document exactly how the names of the entries in this table are chosen, either in the spec text itself or in some metadata somewhere. Can you help us figure out the right way of saying it? Would "these names correspond to the first spelling (including casing) used for each value in Scripts.txt" be accurate?

[michaelficarra]

It was also suggested that we include that editorial process in the spec itself, so I plan to discuss the merits of that in the next editor call.

[mathiasbynens]

@mathiasbynens One of the things we said we'd do as part of this is to document exactly how the names of the entries in this table are chosen, either in the spec text itself or in some metadata somewhere. Can you help us figure out the right way of saying it? Would "these names correspond to the first spelling (including casing) used for each value in Scripts.txt" be accurate?

What you were suggesting might lead to the same results for the Script/Script_Extensions names (I haven’t checked) but wouldn’t explain where we get the canonical value aliases from. The source I’ve been using is PropertyValueAliases.txt (since we need the values + any value aliases as well). This applies to both the Script/Script_Extensions table as well as the General_Category table.

Note that this logic unfortunately doesn’t cover ALL properties listed in the spec — in particular Any, ASCII, and Assigned don’t appear in PropertyValueAliases.txt nor in PropertyAliases.txt since they’re technically not properties (at the Unicode level). For those, we just settled on a casing that felt consistent with the other properties/values (as part of the \p{…} proposal). Then again, that is a one-off scenario that wouldn’t occur in these annual PRs — we’d absolutely still want to get consensus on adding brand new properties to ECMAScript (just not for any new values for existing properties).

[markusicu]

Right, these are the files to look at:

• https://www.unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt
• https://www.unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt

Formally speaking, the particular use of case/space/underscore is unlikely to change but not (I think) guaranteed to not change.

[bakkot]

I've pushed up a commit with the following paragraph just before the table-unicode-general-category-values.html and table-unicode-script-values.html tables:

The spellings of entries in these tables (including casing) were chosen to match the first occurrence of each property in the files PropertyAliases.txt and PropertyValueAliases.txt in the Unicode Character Database at the time each entry was added to this specification. However, because the precise spellings in those files are not guaranteed to be stable, implementations are required to follow this table rather than those files.

@mathiasbynens PTAL. If this does not seem correct to you, can you suggest an alternative?

[mathiasbynens]

LGTM 👍 Some nits/thoughts:

• Let’s add links to these files? https://unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt and https://unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt are stable URLs.
• Should the file names be wrapped in <code>?

[michaelficarra]

So like this?

Suggested change

<p>The spellings of entries in these tables (including casing) were chosen to match the first occurrence of each property in the files PropertyAliases.txt and PropertyValueAliases.txt in the Unicode Character Database at the time each entry was added to this specification. However, because the precise spellings in those files are not guaranteed to be stable, implementations are required to follow this table rather than those files.</p>

<p>The spellings of entries in these tables (including casing) were chosen to match the first occurrence of each property in the files <a href="https://unicode.org/Public/UCD/latest/ucd/PropertyAliases.txt"><code>PropertyAliases.txt</code></a> and <a href="https://unicode.org/Public/UCD/latest/ucd/PropertyValueAliases.txt"><code>PropertyValueAliases.txt</code></a> in the Unicode Character Database at the time each entry was added to this specification. However, because the precise spellings in those files are not guaranteed to be stable, implementations are required to follow this table rather than those files.</p>

[mathiasbynens]

[gibson042]

Would that also apply to the other UCD file references (cf. #2594)?

[michaelficarra]

@gibson042 Yes, please do.

[bakkot]

Done.