Locale of constructed object & language negotiation in data loading

[sffc]

ICU has three types of locales for constructed objects (return value of getLocale() of NumberFormat, for example):

• Requested local
• Valid locale
• Actual locale

It's a very confusing model, so much so that ICU isn't adding new getLocale() methods.

What model do we want to adopt in ICU4X?

[zbraniecki]

In Fluent and Mozilla we use the following lists:

• Requested Locales (e.g. user requested ["es-CL", "es", "fr"]
• Available Locales (e.g. we have ["es", "de", "it", "en-GB", "en-US"])
• Last Fallback Locale (e.g. "en-US")
• Resolved Locales (e.g. we negotiated it down to ["es", "en-US"])

For ICU4X the interesting question is about available locales in the scenario where we don't have the same number of locales available per component. For example we may have PluralRules for 110 locales, but NumberFormat data for 94 locales.

So, the negotiation needs to know what components are you asking for:

let resolved = DataProvider::negotiateLocales(requested, &[Component::PluralRules, Component::NumberFormat]);

and then we need to know whether the user wants to only get locales supported by all requested components (or data chunks), or the best available per component. Let's say we have NumberFormat for es-CL but PluralRules only for es - do we want the returned list to contain ["es-CL", "es"] and then NumberFormat will use es-CL and PluralRules will use es or do we want the resolved list to only contain es?

The answer in this example may feel natural to go for more accuracy, since plural rules are per-language and not per region, but what if we had:

• NumberFormat in ["es-CL", "es", "fr", "it", "de"]
• PluralRules in ["fr", "de"]

Do we then want the resolved to contain es at all? Or just ["fr"] ? I think this time we'd want ["fr", "en-US"].

Finally, once we get to the single instance, it may be tempting to say that an instance of a component is in a single locale, so we don't need a fallback chain, but that only works if the component doesn't depend on other components.

If NumberFormat depends on PluralRules, we want to make sure that they support the same locale, as in the example above - we want to use fr for both.
But MessageFormat may depend on many components. Having a single locale on it means that we either hard fail if one of the dependencies doesn't have that locale or we hard fallback on last fallback locale. It may be useful to store the fallback chain so that we have a more graceful error path if some component used by MessageFormat (say, RelativeTimeFormat, DisplayNames etc.) cannot return a value in es and we want to fallback on the next locale.

[sffc]

All great points. Answering the question "what locales do you support?" is really hard.

Maybe we should ask ourselves, what is the primary use case for asking that question? Based on my experience, I think the most common reason people ask that question is because they have custom fallback logic in the case when a locale is not supported. However, with pluggable data providers, we give users a more official way to add that fallback logic. What other use cases are we looking at?

If we don't have any clear use cases, then we could choose to keep locale negotiation exclusively in the data provider object, and not have any "GetLocale" methods on the constructed objects at all.

[sffc]

I want to extend the scope of this issue to include a very closely related topic, which we discussed in yesterday's ICU4X meeting. How do we handle language negotiation when performing data loading, particularly when different categories of data support different locales?

[sffc]

Changing this to S-epic because one we agree on the design, we will need to implement this in the data providers.

[mihnita]

Posted document in the icu4x-sc / Drop Box

[sffc]

Action items:

• Add API that gives a list of suggested locales
• Share the doc with Markus and Mark Davis
• Add the design doc to the repository