[DO NOT MERGE] Internationalisation support

[querkmachine]

A tech spike into internationalisation (i18n) support in GOV.UK Frontend, particularly within our JavaScript files, which are currently hardcoded. This spike specifically explores a bespoke solution for Frontend, which was the conclusion I came to following the research and tech spikes into third-party solutions (internal document).

Specification

The spike was undertaken based on this rough, self-imposed specification:

• Must be able to pass translation strings from outside of Frontend (e.g. via JS inside of HTML).
• Must store localisation strings as key-value pairs.
• Must support interpolation of dynamic content within translation strings.
• Must support alternative strings for plural forms.
  • Must support different pluralisation rules of different languages.
• Must support fallback ('default') text.
• Must support JS modules being individually initialised (not via initAll).
• Optionally, allow localisation strings to be passed as nested JS objects.
• Optionally, allow implementors to customise which language's pluralisation rules are used.
• Optionally, allow implementors to customise how interpolated strings are detected (defaulting to %{key_name} otherwise, as GOV.UK uses).

How it works

This is a rough, step-by-step of how the example implemention in this spike works. The actual i18n.js file is chock full of documentation too. It's worth a read for how individual methods work.

When using initAll

initAll accepts a configuration object where child objects are separated by the component name. This object is split up and each component passed the relevant part of the configuration.

Currently only two component namespaces are implemented:

• accordion for the Accordion component
• character_count for the Character Count component

Each component's configuration object may contain a nested i18n object, which includes information on the locale to use (used for pluralisation rules and number formatting) and a key-value based list of translations (the 'translation map').

This configuration object may also include other configuration in future (though that is outside of the scope of this work).

window.GOVUKFrontend.initAll({
  accordion: {
    i18n: {
      locale: "cy",
      translations: {
        show_all_sections: "Dangoswch bob adran",
        hide_all_sections: "Cuddiwch bob adran"
      }
    }
  },
  character_count: {
    maxlength: 500
  }
})

When initialising components separately

Individually initialised components use the same configuration object format as initAll, but with the component namespace removed.

new window.GOVUKFrontend.Accordion(document.getElementById("accordion"), {
  i18n: {
    locale: "cy",
    translations: {
      show_all_sections: "Dangoswch bob adran",
      hide_all_sections: "Cuddiwch bob adran"
    }
  }
}).init()

Continue...

1. Each component takes the provided configuration object and performs a shallow merge with a default configuration included within the component.
2. If internationalisation information has been provided (via an i18n key in the config object), each component creates a new instance of the I18n class and passes in the i18n object.
3. i18n.mjs — Reads in the information given.
4. i18n.mjs — If no locale information is given, it attempts to read this from html[lang]. If that is null or empty, it falls back to "en". This is made available as this.locale.
5. i18n.mjs — The translation map is made available as this.translations.
6. Components may then call t() with a lookup key from the previously provided translation map and, optionally, an object containing additional placeholder data. The lookup key is used to find the associated translation string, which may be processed, before being returned to the calling component. See examples.

Examples

Assume we've instantiated a non-existent example component with the following configuration for the Pirate language:

new window.GOVUKFrontend.Example(
  $element,
  {
    animate_flags: false,
    i18n: {
      locale: "en",
      translations: {
        page_name: "Pirate's Cove Antique Treasures",
        welcome_user: "Welcome aboard, Cap'n %{name}",
        ship_count_one: "Ye got %{count} fine ship under ye command",
        ship_count_other: "Ye have %{count} ships under ye command",
      }
    }
  }
)

In the component's code, we pass the i18n information through to the I18n class, and assign the resulting instance to a variable we can use.

this.i18n = new I18n(options.i18n)

Using t() later in the component will result in these outputs. Note how in the last example that the number has been formatted appropriate to the given locale.

this.i18n.t("page_name")
// => Pirate's Cove Antique Treasures

this.i18n.t("welcome_user", { name: "Picard" })
// => Welcome aboard, Cap'n Picard

this.i18n.t("welcome_user", { count: 1 })
// => Ye got 1 fine ship under ye command

this.i18n.t("welcome_user", { count: 2000 })
// => Ye have 2,000 ships under ye command

Currently...

Currently the spike supports:

• Localisations being passed into Nunjucks macros.
• Localisations being passed globally to all components, via initAll.
• Localisations being passed to individual components, via each component's constructor.
• Key-value based localisation strings.
• A specific method (t) for selecting a lookup key and interpolating dynamic content.
• Pluralisation support.
  • Uses unique, explicitly named keys for different plural forms (based on Unicode CLDR's specification).
  • Has an internal reference of the pluralisation rules used by different languages.
  • Also supports multiple plural forms in fallbacks.
• Fallback text defined within the component.
• Overriding the locale used to determine which pluralisation rules to use.

It doesn't support:

• Alternative grammar forms, gendered forms, or switching strings based on an arbitrary context.
• Bi-directional (Bidi) text strings (mixing left-to-right and right-to-left strings) without munging up formatting.
  • This can be handled manually by implementors by including the appropriate control characters (LTR, RTL).
• Pluralisation for numbers that are negative, decimals or part of a range ("3–5 dogs").

Improvements that could still be made:

• Fallbacks that are 'mismatched' to what is expected aren't checked for (e.g. a pluralised fallback object for a non-pluralised string).
• The list of languages currently supported is pretty much arbitrary, and ones we don't need to support can be removed.

[lfdebrux]

I had a good conversation with @querkmachine about this the other day. Noting the gist of it here in case it's useful/interesting for others.

I asked about whether she'd considered using gettext style calling, where the key is the string in English. She said she had, but decided against it, as she pointed out that if we changed the text for a component, that would force a team with a translation to update that translation, which seems like an unnecessary pain for users, which I agree with.

That does though tangentially raise interesting questions about how we encourage conformity; are there ever cases where we will want to make a 'breaking change' to our content, to force teams with translations to update theirs? Of course, there is the wider point that if every team is doing their own translation, then we can expect their to be much consistency anyway.

Anyway, I think after discussing it, how @querkmachine has done it using a lookup table with simple keys is in fact the right way to go.

I've also been thinking of ways to reduce the need to pass around configuration, but I need to do some experiments with that first...

[querkmachine]

It's been suggested that this version of the i18n code (or something similar to it) could be done in an IE-compatible v4 release. On this basis, I've made some minor changes and added polyfills to enable functionality back to IE8.

After we stop shipping JavaScript to IE, we can then do a revision to remove polyfills and most of the pluralisation code, as we could then start to leverage the Internationalization API's PluralRules instead.

[querkmachine]

Based on a comment @lfdebrux made during a demo, I've tried modifying the function to use a singleton pattern.

In this case, the 'default' I18n instance (which is usually the one created within initAll, otherwise the first one created) is made available to other components via a static getInstance method. This simplifies things a little, as we no longer need to pass a specific I18n instance around between components—the components can access the instance themselves.

I've kept the ability to manually pass I18n instances into components on the assumption that a service might want to customise specific uses of a component differently to other uses of the same component.

[querkmachine]

I've spent some time making the code a little more robust and working on some of the "Improvements that could still be made" bits.

---

A component neglecting to give a translation string a key, or giving it a key that is falsy in JavaScript lingo, will now output an error in the console.

• This won't stop code execution however, as it'll continue to try and use the fallback text. If none is defined, it'll just output "UNDEFINED" (this could probably be better).
• This is a situation that is internal to the Design System JS and is very unlikely to occur in production unless there's some massive failure in our code review process—it's more just about making sure things doesn't fail quietly should the eventuality somehow arise.

The count option is now checked to make sure that it's actually a number when evaluated for plural rules.

• As our current code can only handle integers and positive numbers at the moment, it also forces count through abs and floor to make sure it's a positive integer.
• We can stop forcing it when we start using Intl.PluralRules, as that supports decimals and negative numbers.
• As far as I'm aware, no languages have plural rules specific to negative numbers anyway.
• If the count turns out not to be a number, the 'other' plural rule is immediately returned.
• count is still available, unmodified, for placeholder text.

Translation strings are now checked if they have placeholders before being passed through replacePlaceholders.

• Stand back! I know regular expressions!

I've removed the ability to customise placeholder separators.

• During testing I realised that, if a user customises the placeholder separators, that means that all of the placeholders in the fallback text would cease to work, which isn't ideal.
• Having a different set of separators defined for passed-through translation strings versus fallback translation strings seemed like it would add a lot of complexity (in code and also in mentally parsing what's happening) for little benefit.
• Having customisable placeholder format was a 'nice to have' in my self-imposed spec anyway, so removing it doesn't seem terribly detrimental.

[NickColley]

Non-blocking feedback:

1. Should we expand the public API? Recommendation: keep it private until a) there is evidenced user need b) it's been in production for a while
2. Is a consistent API for individually initialized components worthwhile?

Feel free to disregard any and all feedback as I am a community member.

[querkmachine]

@vanitabarrett and I had a discussion about how this work will be covered by automated tests. Currently tests are ran against the review app, however the app currently uses the same global initAll call for all of the examples, and it's not simple to have some examples use this and others not within the current structure.

Instead, we will unit test the internationalisation JavaScript independently from the review app (as per Vanita's spike, #2611), and provide an example page in the app for visual checks and manual tests.

[querkmachine]

The PR has been rebased to include the Nunjucks localisation work from #2648.

An example of localised components in action (including localisation performed both via Nunjucks and via JavaScript) can be found here: https://govuk-frontend-pr-2614.herokuapp.com/full-page-examples/localisation

[querkmachine]

Something to consider, potentially before we do a pre-release, is the actual naming of the JavaScript localisation keys.

I've tried to be descriptive about the purpose of each key when naming them, however this has resulted in some key names being fairly verbose.

The list of JS keys currently:

• accordion.hide_all_sections
• accordion.show_all_sections
• accordion.hide_this_section
• accordion.show_this_section
• character_count.characters_under_limit_other
• character_count.characters_under_limit_zero
• character_count.characters_under_limit_one
• character_count.characters_under_limit_two
• character_count.characters_under_limit_few
• character_count.characters_under_limit_many
• character_count.characters_over_limit_other
• character_count.characters_over_limit_zero
• character_count.characters_over_limit_one
• character_count.characters_over_limit_two
• character_count.characters_over_limit_few
• character_count.characters_over_limit_many
• character_count.words_under_limit_other
• character_count.words_under_limit_zero
• character_count.words_under_limit_one
• character_count.words_under_limit_two
• character_count.words_under_limit_few
• character_count.words_under_limit_many
• character_count.words_over_limit_other
• character_count.words_over_limit_zero
• character_count.words_over_limit_one
• character_count.words_over_limit_two
• character_count.words_over_limit_few
• character_count.words_over_limit_many

It feels like these could be shortened, for example accordion.hide_this_section could be accordion.hide_section.

Likewise, the word 'limit' in the character_count set is potentially superfluous: characters_over, characters_under, etc. might be obvious enough in this context.

Thoughts?

[domoscargin]

Think these gem-c classes have snuck in.

[querkmachine]

I copied this page from one of the other full-page examples, though I'm happy to remove given it's pretty much decorative here.

[domoscargin]

Is it potentially a slight issue that we split HTML and text into their own properties on other fields, but not here?

[querkmachine]

I figured splitting it up might not be necessary as it's fairly likely to be HTML. There are other parameters that aren't split (like visuallyHiddenText), though that does include 'text' in the name.

Would it be better named as contentLicenceHtml or do you think it's better to split it up?

[querkmachine]

copyright is a more awkward example of this, I think.

The coat of arms is set as the background image of the crown copyright link. As such, a user cannot use HTML to set a custom link unless hideCrownCopyrightArms: true (which removes the link and thus the arms), in which case they can. The expected value of copyright basically switches between being HTML or plain text depending on that context.

[domoscargin]

Would be good to clarify what will be required and suggested in this case. Does the Design System/GDS care if users don't add a new copyright/license to replace these?

[querkmachine]

I'm not sure how necessary that is, might be a question for policy?

For public-facing GOV.UK services, the default (or a translation of the default) is expected, I imagine.

For internal/caseworking systems, they're more likely to remove or change the content licence and potentially remove the coat of arms for space purposes.

For third-parties that use Frontend as a boilerplate, they'll probably remove or replace all of it.

[domoscargin]

Useful from the perspective of users in terms of us not introducing breaking changes in the future, though p'raps a bit more mental load for users to take on right now. I wouldn't be against this, but I don't feel strongly that it MUST happen.

[querkmachine]

I've made a fork of this work based on a suggestion made by @36degrees, for consideration as an alternative approach.

You can compare it against this PR here: spike-i18n-support...spike-i18n-support-alt

Changes

The end result is pretty much identical, however the way that localisation strings are configured and scoped is different.

Instead of strings being defined within a i18n object which is separated by component, i18n is instead now a possible parameter within each component's individual options object, essentially reversing the prior relationship.

- { i18n: { accordion: { translations: { show_all_sections: "Show all sections" } } } }
+ { accordion: { i18n: { translations: { show_all_sections: "Show all sections" } } } }

Localisations are now part of a component's own configuration, along with any other configuration options. This allows for future configuration options to be nested within the same component object, or for the wrapping object to be discarded entirely when it's unnecessary (for example, when passing a configuration directly into a component).

new Accordion($element, {
  remember_state: false,
  i18n: {
-   accordion: {
-     translations: {
-       show_all_sections: "Show all sections"
-     }
-   }
+   translations: {
+     show_all_sections: "Show all sections"
+   }
  }
})

Other changes

• The singleton pattern has been removed, as each component now instantiates its own I18n instance. Similarly, an I18n instance is no longer created by initAll.
• The fallback parameter has been removed, as these are now part of a component's default configuration options. They're now defined as part of defaultOptions.
• The 'flattening' functionality for nested translation objects has been removed as it provides less benefit in this pattern.
• Previously, the locale parameter was only provided once, which propagated to all components. This now has to be specified for each component (unless the document language is being used).

What didn't change

Ollie's original proposal was to (roughly) reduce the I18n class to essentially be helper functions that don't need an instantiated instance at all.

In this situation, I18n.prototype.t would only be responsible for transforming strings (pluralisation selection and placeholders), with components passing translation strings directly, rather than having a lookup key system.

However, the locale information used for pluralisation is currently created on instantiation. Without this, locale information would need to be passed as an option each time t was used.

I initially made a 'halfway' where I18n was instantiated solely for locale information to be created and string selection was otherwise handled within the component, however this seemed more convoluted than necessary—if we're already creating an I18n instance and passing configuration into it, you might as send the translations object at the same time and retain the lookup key system.

The process of selecting strings within components became much more verbose when not using named keys, too.

// Previously, this would suffice
- this.i18n.t('show_all_sections')

// You'd now need to pass in the key relative to the current component and always pass in locale info
+ this.i18n.t(this.options.i18n.translations.show_all_sections, { locale: this.options.i18n.locale })

Ultimately I didn't continue with this approach, as it made implementation within components more complicated for seemingly little additional benefit.

Comparisons

Using `initAll` to pass translation strings to multiple components at once

Old:

<script>
  window.GOVUKFrontend.initAll({
    i18n: {
      locale: "cy",
      translations: {
        accordion: {
          show_all_sections: "Dangos pob adran",
          hide_all_sections: "Cuddiwch bob adran",
          show_this_section: "Dangoswch yr adran hon",
          hide_this_section: "Cuddio'r adran hon"
        },
        character_count: {
          characters_over_limit_zero: "Rydych chi %{count} llythyrau drosodd",
          characters_over_limit_one: "Rydych chi %{count} llythyr drosodd",
          characters_over_limit_two: "Rydych chi %{count} lythyren drosodd",
          characters_over_limit_few: "Rydych chi %{count} llythyren drosodd",
          characters_over_limit_many: "Rydych chi %{count} llythyren drosodd",
          characters_over_limit_other: "Rydych chi %{count} o lythyrau drosodd",
          characters_under_limit_zero: "Mae gennych %{count} llythyren ar ôl",
          characters_under_limit_one: "Mae gennych %{count} llythyr ar ôl",
          characters_under_limit_two: "Mae gennych %{count} lythyren ar ôl",
          characters_under_limit_few: "Mae gennych %{count} llythyren ar ôl",
          characters_under_limit_many: "Mae gennych %{count} llythyr ar ôl",
          characters_under_limit_other: "Mae gennych %{count} llythyren ar ôl"
        }
      }
    }
  })
</script>

New:

<script>
  window.GOVUKFrontend.initAll({
    accordion: {
      i18n: {
        locale: "cy",
        translations: {
          show_all_sections: "Dangos pob adran",
          hide_all_sections: "Cuddiwch bob adran",
          show_this_section: "Dangoswch yr adran hon",
          hide_this_section: "Cuddio'r adran hon"
        }
      }
    },
    character_count: {
      i18n: {
        locale: "cy",
        translations: {
          characters_over_limit_zero: "Rydych chi %{count} llythyrau drosodd",
          characters_over_limit_one: "Rydych chi %{count} llythyr drosodd",
          characters_over_limit_two: "Rydych chi %{count} lythyren drosodd",
          characters_over_limit_few: "Rydych chi %{count} llythyren drosodd",
          characters_over_limit_many: "Rydych chi %{count} llythyren drosodd",
          characters_over_limit_other: "Rydych chi %{count} o lythyrau drosodd",
          characters_under_limit_zero: "Mae gennych %{count} llythyren ar ôl",
          characters_under_limit_one: "Mae gennych %{count} llythyr ar ôl",
          characters_under_limit_two: "Mae gennych %{count} lythyren ar ôl",
          characters_under_limit_few: "Mae gennych %{count} llythyren ar ôl",
          characters_under_limit_many: "Mae gennych %{count} llythyr ar ôl",
          characters_under_limit_other: "Mae gennych %{count} llythyren ar ôl"
        }
      }
    }
  })
</script>

Passing translation strings when individually instantiating a component

Old:

<script>
  new window.GOVUKFrontend.Accordion($element, {
    i18n: {
      locale: "cy",
      translations: {
        accordion: {
          show_all_sections: "Dangos pob adran",
          hide_all_sections: "Cuddiwch bob adran",
          show_this_section: "Dangoswch yr adran hon",
          hide_this_section: "Cuddio'r adran hon"
        }
      }
    }
  }).init()
</script>

New:

<script>
  new window.GOVUKFrontend.Accordion($element, {
    i18n: {
      locale: "cy",
      translations: {
        show_all_sections: "Dangos pob adran",
        hide_all_sections: "Cuddiwch bob adran",
        show_this_section: "Dangoswch yr adran hon",
        hide_this_section: "Cuddio'r adran hon"
      }
    }
  }).init()
</script>

[querkmachine]

The alternative approach described in my last comment has now been merged into this branch, as we felt it was a change that was ultimately necessary for our future API plans.

I've also rebased from main and modified this branch to use .mjs files as the main branch does.

[vanitabarrett]

Closing this PR as it contains code developed for the internationalisation spike, which we've completed. The branch will remain so we can refer back to it and/or generate pre-releases as needed. I'm going to draft some new issues that break down the work into sizeable chunks we can start working on implementing this with a release in mind.