Measurement and Currency feedback from Siri's experience

[zbraniecki]

George Rhoten from Apple shared their experience of why they decided not to use ICU4C/ICU4J MeasurementFormat/CurrencyFormat for Siri:

    The major unit and minor units need to be independently pronounced. So we need to handle the following:

    $1.00 → one dollar
    $2 → two dollars
    $1.01 → one dollar and one cent
    $1.02 → one dollar and two cents
    $2.01 → two dollars and one cent
    $2.02 → two dollars and two cents

    The unit needs to sensitive to the region pronunciation.

    USD with US region: dollar/dollars
    CAD with CA region: dollar/dollars
    USD with CA region: US dollar/US dollars
    CAD with US region: Canadian dollar/Canadian dollars

    The number needs to convey the grammatical gender of the noun
    The entire phrase needs to convey grammatical case of the number and noun
    Measure words can be needed for East Asian languages
    The word order may change depending on the plural rule, like in Hebrew
    Fractions of currency subunits can be needed, like 2.4 cents
    Round to whole units can be needed, like $1.02 → $1 or $1,234,567 → $1.2 million
    Subunit translation is needed, like the notion of cent.
    Reasonable behavior is needed for rare or historical currencies.
    Reasonable non-ISO 4217 handling is needed, like when Bitcoins are handled.

The other measurement concepts were just a subset of those issues. Usually it was just around getting a number and noun into grammatical agreement.

 [Measure words](https://en.wikipedia.org/wiki/Measure_word) tend to provide a measure for a unit. For example, the measure word in the phrase “3 sheets of paper”, the measure word is “sheet”.

Since we want to power ECMA-402/MF2.0 integration with ICU4X, this kind of feedback may be useful for us as we design ICU4X Number/Measure/Currency formatters.

[sffc]

Going point by point through George's comment:

The major unit and minor units need to be independently pronounced...

By necessity, strings for textual display are not going to be the same as spoken strings. The question here is therefore whether we consider TTS to be in-scope of ICU4X.

Personally, I would like to see TTS support; it's something that my directors at Google frequently ask about ("when is ICU ever going to have good spoken language support?"). It could be in scope for implementation in 2022, but we should have some early discussions to lay down some plans on how to design ICU4X such that we don't box ourselves out of TTS down the road.

The number needs to convey the grammatical gender of the noun
The entire phrase needs to convey grammatical case of the number and noun

@hugovdm, @younies, @macchiati, and others did a great deal of work in 2020 to gather more data on the gender and case of measurement units. That data is now in CLDR. When we work on full unit support in ICU4X, I anticipate that we will build it with inflections as a first-class feature.

Measure words can be needed for East Asian languages

Not exactly sure what this one means. We do have CLDR data now that replaces the entire number with a standalone word in measurement unit and compact notation patterns.

The word order may change depending on the plural rule, like in Hebrew

CLDR measurement unit data has a lot of support for plural rules.

Fractions of currency subunits can be needed, like 2.4 cents
Subunit translation is needed, like the notion of cent.

Is this not already supported in ICU? That would be a limitation only on CLDR data having patterns for the "cent" currency unit.

Round to whole units can be needed, like $1.02 → $1 or $1,234,567 → $1.2 million

ICU supports this since ICU 60.

Reasonable behavior is needed for rare or historical currencies.

Does ICU/CLDR not already have reasonable behavior for such currencies?

Reasonable non-ISO 4217 handling is needed, like when Bitcoins are handled.

The ISO 4217 code for Bitcoin is XBT. Do we need something more?

[grhoten]

Going point by point through George's comment:

The major unit and minor units need to be independently pronounced...

By necessity, strings for textual display are not going to be the same as spoken strings. The question here is therefore whether we consider TTS to be in-scope of ICU4X.

Personally, I would like to see TTS support; it's something that my directors at Google frequently ask about ("when is ICU ever going to have good spoken language support?"). It could be in scope for implementation in 2022, but we should have some early discussions to lay down some plans on how to design ICU4X such that we don't box ourselves out of TTS down the road.

I'm open to collaboration on such a topic. Though our implementation may be hard to fit into ICU/CLDR due to different design philosophies.

The number needs to convey the grammatical gender of the noun
The entire phrase needs to convey grammatical case of the number and noun

@hugovdm, @younies, @macchiati, and others did a great deal of work in 2020 to gather more data on the gender and case of measurement units. That data is now in CLDR. When we work on full unit support in ICU4X, I anticipate that we will build it with inflections as a first-class feature.

I haven't seen the latest on this, but unless it properly conveys the RBNF rule to use with such scenarios, it still won't work for our use cases.

Measure words can be needed for East Asian languages

Not exactly sure what this one means. We do have CLDR data now that replaces the entire number with a standalone word in measurement unit and compact notation patterns.

Within the CLDR framework, this normally isn't an issue because measure words would typically be a part of the unit being formatted.

The word order may change depending on the plural rule, like in Hebrew

CLDR measurement unit data has a lot of support for plural rules.

Fractions of currency subunits can be needed, like 2.4 cents
Subunit translation is needed, like the notion of cent.

Is this not already supported in ICU? That would be a limitation only on CLDR data having patterns for the "cent" currency unit.

ICU and CLDR have no support for subunit names. CLDR also doesn't have always have data for colloquial names. It seems to prefer the excessively formal names and precise names.

Round to whole units can be needed, like $1.02 → $1 or $1,234,567 → $1.2 million

ICU supports this since ICU 60.

It's in separate API in ICU, and we use it all already. Our implementation combines it all together in a unified format as just properties of a number being formatted.

Reasonable behavior is needed for rare or historical currencies.

Does ICU/CLDR not already have reasonable behavior for such currencies?

Not in the scope of pronouncing a quantity. My examples above start to fail when the local name doesn't exist or the subunit name doesn't exist.

Reasonable non-ISO 4217 handling is needed, like when Bitcoins are handled.

The ISO 4217 code for Bitcoin is XBT. Do we need something more?

That's a nice dream. If only the data feeds out there would actually follow such guidance, I'd be a happier person. There are several Bitcoin names and codes. Some even insist on using BTC, which is totally wrong. So we had to turn several of the Bitcoin variants to a 4 letter uppercase codes (e.g. BTC --> XBTC or XBT, ETH --> XETH for Etherium and LTC --> XLTC for litecoin). Then we provide a non-abbreviated form of the name that can be pronounced. These currencies also go well beyond the typical 2 digit subunits. Bitcoins can have long fractions.

One of our data feeds insist that GBp stands for British pence instead of GBP standing for British pounds. It's so frustrating when standards aren't followed, and we can't all agree on currency consistency.