Locale vs Preferences

[zbraniecki]

In #1833 I started working on generalized icu_preferences utility and wanted to discuss how we want to approach this abstraction.

The outline is that there are three types of information that are passed to the constructor of a formatter with three sources of authority:

1. Developer provides information on general type of formatting
2. User provides their preferences
   2.a) In form of a Locale
   2.b) In form of preferences

An example (shaped after ECMA-402 Intl.DateTimeFormat) may look like this:

let dtf = DateTimeFormat::new(
    locid!("en-US"),                                        // Provided from the user
    options!(
        dateLength: DateLength::Long,           // Decided by the developer
        timeLength: TimeLength::Medium,      // Decided by the developer
        preferences: preferences!(
            hour_cycle: HourCycle:H24             // Provided from the user
        )
    )
);

The division between locale and preferences may be, depending on how we architect it, seen as arbitrary, ergonomic of functional.

The locale itself is a set of preferences - preferred language, region, script, variants, hour cycle, calendar, numbering system, etc. Preferences could be seen as a shadow of the same information.

In such approach, the duality of those two in the API serve as an ergonomic API for the developer who can freely pass the locale they retrieved from the user or embedded system and use the preferences to adjust any overrides they'd like to make.

If we approach it this way, then there's little difference between:

let loc = locid!("en-US");

let dtf = DateTimeFormat::new(
    loc,
    options!(
        dateLength: DateLength::Long,
        timeLength: TimeLength::Medium,
        preferences: preferences!(
            hour_cycle: HourCycle:H24
        )
    )
);

and:

let mut loc = locid!("en-US");
loc.extensions.unicode.set(unicode_ext_key("hc"), unicode_ext_value!("h24"));

let dtf = DateTimeFormat::new(
    loc,
    options!(
        dateLength: DateLength::Long,
        timeLength: TimeLength::Medium,
    )
);

except that the developer may have to clone the locale to apply the adjustment before passing to the constructor.

If we were to give up on preferences we'd need to think about the impact on ECMA-402 bindings which would then basically require us to always clone the Locale whenever preferences in ECMA-402 constructor are set, modify the cloned Locale and pass it to the constructor.
The value is that now we don't need a resolution between Locale and preferences and have a single location for all such data.

We could even then implement ergonomic API for ecma-402 preferences to be applied in-bulk onto Locale.

But this approach has a scaling tradeoff.
Historically, ecma-402 preferences and relevant unicode extensions have not been completely matching. The binding would have to guard against that.

This is slightly problematic when we consider preferences that have been intentionally not included because it is confusing. An example of such is cu in Locale is an extension stating what is the preferred currency. But in NumberFormat options it is an information about what is the currency of the value.
I hope that in ICU4X the information about the currency will be associated with the type, not formatter (CurrencyFormatter will receive Currency which will be a number+currency).
Similar problem is with tz.

Conceptually, my larger concern is that we'd link any preferences to unicode extension keys. We'd design the API to never allow for a preference to exist that has no equivalent as unicode extension key.
Are we comfortable with it?

I feel that there are user specified preferences such as preferred patterns for given lengths or skeletons, that will never make it to Unicode Extensions. I don't know if we'd ever want to support it, but I imagine a world in which we'd eventually want to support something like:

let loc = locid!("en-US");

let dtf = DateTimeFormat::new(
    loc,
    options!(
        dateLength: DateLength::Long,
        timeLength: TimeLength::Medium,
        preferences: preferences!(
            date_patterns: {
                long: "Y/m/d E",
                medium: "Y/m/d",
            }
        )
    )
);

If we lock ourselves in unicode extensions, supporting that would require us to add that to UE and I doubt it'll ever happen. This is not an extreme scenario. Operating Systems like Windows and MacOS allow users to override wide range of preferences - date/time patterns for lengths, decimal separators, sign symbols etc.

With the current split between preferences and Locale we can take information from Locale, but allow for preferences to store more, and potentially any of the ones from their OS.
Would we want to give up on this?

[sffc]

Discussion:

• @sffc - I'm not convinced nesting preferences inside options is the right way. We should consider either flattening it into options, or attaching it to the locale via a trait.
• @robertbastian - Is the locale mandatory? It seems like we should have one argument that is a combination of locale and preferences.
• @sffc - We should consider the locale being a trait, which is implemented by Locale, LanguageIdentifier, Preferences, etc. Previous discussion: Signature of ICU4X APIs that take a Locale #1662

[zbraniecki]

Resuming this thread in relation to the conversation I've been having with Shane in #1833.

I explored multiple approaches to API of the constructor between Locale/Options with an aim to remain open minded and avoid cargo culting.

For the following reasoning, there are two entities we'll be discussing - Locale and Preferences. Both are effectively structs of keys and values, with the difference being that Locale has a strong string representation and is limited by UTS 35 standard and Preferences are a strongly typed structure with potentially no string representation and may contain any keys and values we want to provide to any constructor.

Visualizing it, Locale is:

{
    language: language!("en"),
    region: region!("US"),
    extensions: {
      unicode: {
        hour_cycle: value!("h24"),
      },
}

or: en-US-u-hc-h24.

Preferences are very similar in "valid" case, but can be stronger typed:

{
    language: language!("en"),
    region: region!("US"), 
    hour_cycle: HourCycle::H24,
}

Notice, that for hour_cycle in Preferences is always valid making the code inside DTF be able to operate on it without having to validate it each time. In case of operating on Locale the value!() could not match to any of the HourCycle and the internal code has to reason about it and decide how to handle such cases.

If we want to encode something not present in UTS35, in Preferences we can just add it:

{
    language: language!("en"),
    region: region!("US"), 
    date_style: DateStyle::Long,
}

but in Locale we either cannot or we have to use private extension:

{
    language: language!("en"),
    region: region!("US"), 
    private: vec![
      key!("date_style"),
      key!("long"),   
    ]
}

// or

"en-US-x-date_style-long"

As you can see, the encoding of date style as private extension is at best awkward. It doesn't enforce key-value pair, the order could be flipped, or scattered, and in the end it's quite hard to assume that it will convey the right information and the right value.

Locale is effectively a subset of Preferences. All keys and values that Locale encodes are either encodable in Preferences or discarded.

The convention used in ECMA-402 is that we separate those two and constructor API accepts both separately and then merge internally.

I initially struggled with the "internal merging" as in:

impl DTF {
    pub fn new(locale: Locale, prefs: Preferences) -> Self {
        let prefs = prefs.merge(locale); // <-- what if we merge `en-US-u-hc-h87` ?
        Self { prefs }
    }
}

fn foo() {
    let loc = locale!("en-US-u-hc-h12");
    let prefs = Preferences {
      hour_cycle: HourCycle:H23,
    };
    let dtf = DTF::new(loc, prefs);
}

Shane convinced me that it is ok to just discard any unknown values here, so basically locale in this case is indistinguishable from en-US and the implicit merge is okay.

What's left is the question if the API is better if keps separate or merged. The alternative that I'm nurturing is:

impl DTF {
    pub fn new(prefs: Preferences) -> Self {
        Self { prefs }
    }
}


fn foo() {
    let loc = locale!("en-US-u-hc-h12");
    let prefs = Preferences {
      hour_cycle: HourCycle:H23,
    };
    prefs.merge(loc);
    let dtf = DTF::new(prefs);
}

There are couple reasons why I'm attracted to this diversion from ECMA-402.

Primarily it feels to me (intuitively) that it gives us a foundation that is closer to the ontological value of those two types - they are both preferences, and they can be merged. Once we acknowledge that the proposed API yields itself better to different models of merging - it's easy to have merge_lossy and try_merge and maybe prefs.merge(locale, Strategy::Foo) and so on giving developer control over the operation rather than locking us in implicit merging strategy.
It seems attractive to me, but I'm open to interpretation that no such value can ever exist and the default strategy ("discard unknown") is the only one we'll ever need and it's ok to "hardcode" it by making it implicit.

The second value I see in it is that it makes it clear that DTF only cares about some merged value of those two. API that takes locale and prefs makes it look like those are two different things that have a value for DTF remaining separate while in fact the first and only thing DTF constructor will do with it is merge.
It feels against Rust API design culture to hide it and lock ourselves in the design.

On the flipside Shane argues two things about the separation:

1. There is a value in forcing developer to pass locale. Otherwise they could accidentally construct without it:

let prefs = Preferences {
  hour_cycle: HourCycle::H24,
};
DTF::new(prefs);

2. There two types strictly differ in source - Locale is "user preferences bag" and Preferences is "developer preferences bag".

I'm sympathetic to (1) although I have reservations from shaping API with a goal of "educating" developers. I also can imagine scenarios where it may be acceptable:

let prefs = get_user_prefs(); // { hour_cycle: HourCycle::H12 };
let dtf = DTF::new(prefs); // user did not specify language or region, all we know that we'll use user specified prefs

I'm much less sympathetic to (2) - I think it is presumptuous to assume where which bag comes from and further it can serve as a self-fulfilling prophecy - what ICU/ECMA-402 provides sets boundaries on what developers will do with it.
Today it is very hard or impossible for ICU/ECMA-402 user to merge Locale and Prefs so nobody will try and likely everyone will just do what is easy.

But realization that Locale and Prefs are mostly the same thing and providing API to operate on it I think opens up ability for developers to create more creative strategies of handling more sophisticated scenarios of information/preference sources including chained merges, multi-stage sources etc.

Locale can be first provided by Android into an application, the application can merge it with it's "defaults" and then merge further with cached user preferences and finally merge with the particular callsite preferences in the following, admittingly elaborate, example:

fn main() {
  let loc = OSEnvironment::get_user_preferred_locale();
  
  let defaults = Preferences {
    date_style: DateStyle::Long,
  };

  let app_loc = App::get_user_selected_locale();
  
  let loc = app_loc.merge(os_loc);

  {
    // UI screen
    let screen1_prefs = Preferences {
      time_style: TimeStyle::Medium,
    };
    let prefs = screen1_prefs.merge(loc);
    let dtf = DTF::new(prefs);
    widget.label = dtf.format_to_string(datetime);
  }
}

I think we can achieve a good design around it with the following tenants:

1. It should be obvious/easy to pass locale to the constructor.
2. It should be possible to control the merging of locales and preferences by the developer
3. It must be possible to back ECMA-402 API easily with ICU4X API

The latter can be achieved by either matching 1-1 ECMA-402 or by offering the explicit merge that matches ECMA-402 strategy for invalid values.

@sffc - wdyt?

[sffc]

On the flipside Shane argues two things about the separation:

1. There is a value in forcing developer to pass locale. Otherwise they could accidentally construct without it:

Yes; this is what I call the "First Principle of i18n Library Design". The Locale is the most important argument, and it should be clear where and when to use it in the API.

2. There two types strictly differ in source - Locale is "user preferences bag" and Preferences is "developer preferences bag".

This mischaracterizes my position. The line between is not between Locale and Preferences; it is between Preferences and Options.

I'm okay with your reasoning about Preferences being a more expressive form of Locale, with type safety and a limited number of common extensions for preferences that can't be expressed with BCP-47. I have to say that I'm not completely sold on the use cases you laid out, but I'm willing to go along with it.

However, Options serves completely different purpose. It is how the developer specifies thing like, "I have only 100 pixels, so I want a narrow format", or "this price is in USD and I need to display it with that currency", or "this is a disjunctive list of items, so it should use localised 'or' rather than 'and'".

One of your examples has:

    let screen1_prefs = Preferences {
      time_style: TimeStyle::Medium,
    };

My position is that TimeStyle is not a Preference; it is an Option. (Note that TimeStyle is different than the user's preferred datetime formatting pattern, which may come from a GUI in the OS, which I agree is a Preference.)

Another way to see this is that Preferences are things we don't know until runtime, whereas Options are things we typically know at compile time. See my doc data_phases.md.

I think it's a strength that ECMA-402 has a fairly clear separation here. For Preferences, ECMA-402 only supports locales, and I think it's reasonable for ICU4X to iterate on that with a type-safe Preferences bag. However, it's a completely different thing to say that we want to merge the Options in with the Preferences, and this is a bridge I'm much less comfortable with.

Locale can be first provided by Android into an application, the application can merge it with it's "defaults" and then merge further with cached user preferences and finally merge with the particular callsite preferences in the following, admittingly elaborate, example:

I agree with merging a locale with the application defaults and cached user preferences to create a resolved preferences bag at application load time. I would argue this would be better done via APIs on Locale itself, without needing a separate preferences bag type, but I agree with the concept.

Where I don't agree is in the "callsite preferences". This is a much weaker use case. The call site provides Options, not Preferences, which are mostly-orthogonal sets of things that control the behavior of the formatter.

I think we can achieve a good design around it with the following tenants:

1. It should be obvious/easy to pass locale to the constructor.
2. It should be possible to control the merging of locales and preferences by the developer
3. It must be possible to back ECMA-402 API easily with ICU4X API

The latter can be achieved by either matching 1-1 ECMA-402 or by offering the explicit merge that matches ECMA-402 strategy for invalid values.

As explained above, I think (2) is a red herring. We're not aligned on the premise.

[zbraniecki]

As a follow up to Jun 30'th ICU4X meeting, here's a table of all constructors in ECMA-402 and all options they have, and how I think of them in the User vs Developer driven.

Here's the summary - https://docs.google.com/spreadsheets/d/17eXmqFicLUfDkxgzSnGvZ6g9fEO4XbQhN6IoRtCQnyI/edit?usp=sharing

In particular, I believe that there are no direct overlaps between User Preferences and Developer Settings.

I know of scenarios where a developer wants to force a particular preferences (say, numberingSystem to be latn), and I believe it should be done via:

let hardcodedPrefs = DateTimeFormatPreferences {
    numberingSystem: Some(NumberingSystem::Latn),
   ..Default::default()
};

loc.merge_prefs(hardcodedPrefs, true);

let dtf = DateTimeFormat::try_new(loc, Default::default();