Extract map for i18n strings to use for directed load of i18n strings per view.

[nerrad]

Description

Ongoing work. See #6015 for details.

Types of changes

When complete this will:

• introduce a new webpack plugin for building the map of i18n strings to chunk.
• introduce a mechanism server side for registering only the necessary locale strings for the scripts being loaded on page.

Checklist:

• My code is manually (user) tested.
• My code has unit tests.
• My code follows the WordPress code style.
• My code has proper inline documentation.

[nerrad]

Contained is the proof of concept webpack plugin. I realized after building that the plugin I did is not following the WebPack 4 plugin api (still works, just throws deprecated warnings). Should be fairly trivial to convert (I hope to convert to webpack 4 if needed, however it probably could be tweaked to support both).

The pull isn't done, there still needs to be the php side of things done. But I thought I'd get this so it can be iterated on.

[nerrad]

Alright, now I have added the php component for this and as far as I can tell it should work swimmingly. I still have to test with actual translations but I think I have enough for the POC now so we can review the current work and decide if this is a process we want to iterate on. Once that decision is made (along with any requested tweaks), I'll go ahead and clean up code style and add some phpunit tests for the php side of things, but I could use some help with the javascript tests because I haven't done many of those.

For the php side, although it still in some ways is gutenberg specific, I still think it gives a bit of an idea of how things maybe could happen in WP core. Whether that gets generalized more before merging to gutenberg or gutenberg ships with a variation of this and the generalization takes place in the related trac ticket in tandem... that's up to you guys :)

[nerrad]

So in testing I'm seeing the locale data get built okay, but I think there may be an issue with where wp.i18n.setLocale is getting called in this pull because not all strings that are provided are getting used in translations for the view.

So only some things are getting translated (even though it looks like the strings ARE getting registered). Since things do work on master for the translation I'm testing with I'm comparing the two branches to see what's happening.

[nerrad]

K figured it out. Apparently it was due to a couple things:

1. I was passing the domain through as the second argument on wp.i18n.setLocale, whereas master was not setting that argument. I don't know if that's expected behaviour or not, but it appears adding gutenberg as the second argument for domain breaks translations (maybe because there's translations not setting the domain?). I tried passing an empty string and translations still didn't get pulled in for some things.
2. I wasn't setting up the locale "header" info correctly (the object with domain/plurals in it)

After that I noticed that translations would get used only when a component was updated in the view. Traced that to where wpi81n.setLocale was called in page load. So when I switched that to be attached to wp-i18n and invoked immediately after that script was loaded all was well. I think that makes sense anyways, it should always get invoked immediately after loading wp.i18n right?

So, the first point I made above does sound like a possible bug with usage of $domain as the second argument for wp.i18n.setLocale.

[aduth]

whereas master was not setting that argument. I don't know if that's expected behaviour or not, but it appears adding gutenberg as the second argument for domain breaks translations (maybe because there's translations not setting the domain?).

Master doesn't pass an argument through for domain because those strings are assumed to be in the default domain (see default argument).

I think that makes sense anyways, it should always get invoked immediately after loading wp.i18n right?

Either that, or immediately before the script using the strings (passing 'before' as wp_add_inline_script third argument). Which may be more accurate to say they're dependents of those original scripts, rather than of wp-i18n the utility.

[nerrad]

Either that, or immediately before the script using the strings (passing 'before' as wp_add_inline_script third argument). Which may be more accurate to say they're dependents of those original scripts, rather than of wp-i18n the utility.

Hmm, however with wp-i18n being the dependency for any scripts actually declaring translation strings, I think it may be better if the inline script is actually registered to that handle as it's pretty much guaranteed its going to be loaded right?

[nerrad]

So after the meeting today, I'm wondering if I should do the webpack plugin as a separate pull request on wordpress/pacakges (converting to webpack4 package? or should it be wp3&4 compatible?).

[aduth]

I think eventually we'd want it to live in the packages repository. It becomes a bit more difficult to test that way, in the context of Gutenberg, particularly if there are Gutenberg-specific pieces that need to remain here. In the past I've introduced them here and progressively moved them to packages (e.g. @wordpress/custom-templated-path-webpack-plugin).

While core still uses Webpack 3, Gutenberg uses Webpack 4 and I think it would be fine to target only Webpack 4.

[aduth]

I think it may be better if the inline script is actually registered to that handle as it's pretty much guaranteed its going to be loaded right?

As far as guarantees, sure. Conceptually it seems to invert awareness of scripts to be less of a handle associating their own strings, and more of the i18n script knowing which other scripts exist on the current screen. Not saying one or the other is better, but as traditionally with scripts defining their own dependencies, it seemed more intuitive to me that the relationship be with locale data set on the script using them.

[nerrad]

As far as guarantees, sure. Conceptually it seems to invert awareness of scripts to be less of a handle associating their own strings, and more of the i18n script knowing which other scripts exist on the current screen. Not saying one or the other is better, but as traditionally with scripts defining their own dependencies, it seemed more intuitive to me that the relationship be with locale data set on the script using them.

Right and this gets to the comment @youknowriad made in chat about ensuring wp.i18n.setLocale can "accumatively" load string objects. Should I pursue that path here?

[aduth]

I'm okay with that, sure. It gets a bit weird so far as what's expected to be passed to setLocaleData, since there's more than just strings, but also "header" details. It may just be a matter of using something like Lodash's _.merge. It could be a bit more difficult now that the i18n module has been extracted out (#6007). Would we need anything to "unset" locale data?

[nerrad]

So for accumulative wp.i18n.setLocale it looks like we'll need to work out an approach for how Jed is instantiated.

I see that the locale data is assigned to the options object per:

i18n.options.locale_data[ domain ] = localeData;

So would we just need to merge additional incoming localeData for the domain on subsequent calls to setLocale?

[nerrad]

It could be a bit more difficult now that the i18n module has been extracted out (#6007).

Right, I think this is a change we'd make at the package level right?

Would we need anything to "unset" locale data?

I can't think of a usecase offhand.

[nerrad]

Here's the pull on packages/i18n: WordPress/packages#105

[nerrad]

I'm also not sure whether translation-map.json should be commited with the repo or not? Currently it builds on both dev and production builds (would be needed for both I think) so having it committed might be unnecessary? If agreed, I can untrack and force push.

[aduth]

Yeah, we probably don't want to commit it, as it's a build artifact.

[nerrad]

Okay, apologies but I used git filter-branch to rewrite the history in this branch removing translation-map.json from tracking. Then I force pushed. So context was lost for some of the comments made in this pull request. However I think that's okay in favor of having a cleaner history?

translation-map.json is no longer tracked.

[aduth]

However I think that's okay in favor of having a cleaner history?

I use git push -f as part of my regular workflow, so it's not an issue as part of the branch / pull request process, and I agree cleaner history should be prioritized. All the same, in the end we'll do a "Squash and Merge" so there's only one commit to the master branch.

[nerrad]

I had noticed that the msgID format for _x and _nx strings is {context}\u0004{singular_string} so I corrected that. This took care of a few strings that weren't translating in my tests.

[nerrad]

The latest round of changes:

• fixes a bug in the webpack plugin where it wasn't writing strings for _nx and _x correctly to the map.
• reworks enqueuing of translations so that we enqueue translations per script. This utilizes the new functionality in wp.i18n.setLocale brought in from the package update for merging in additional string translations on consecutive calls.
• I tweaked how gutenberg registers its domain php side. I realize one of the reasons (maybe) why gutenberg doesn't use a defined $domain for wp.i18n is because of its eventual merge to wp core where a domain is not used/needed. Yet with it being in the WordPress.org plugin repository it has the domain of gutenberg for translations there. I want the way strings are registered via GB_Scripts to be representative of how it could work with multiple plugins registering their scripts this way so that on (or even before) merge it likely can be brought in with only a few changes to the WP_Scripts api. So in certain places I modifed things to ensure that gutenberg is registered via wp.i18n.setLocale with an empty (which ends up being default via wp.i18n) domain.

[nerrad]

So one evolution of this I'm thinking about is removing the need to require a $chunk_name when registering scripts and instead just attach scripts to handles (which should be unique anyways). The webpack plugin could be made configurable by receiving (optionally) a alias map. Something as simple as this maybe (obviously since this follows a pattern the aliases object can be scripted a bit, but for clarity I did full form):

new wpi18nExtractor( {
    aliases: {
        data: 'wp-data',
        coreData: 'wp-core-data',
        utils: 'wp-utils',
        hooks: 'wp-hooks',
        date: 'wp-date',
        element: 'wp-element',
        components: 'wp-components',
        blocks: 'wp-blocks',
        viewport: 'wp-viewport',
        editor: 'wp-editor',
        editPost: 'wp-edit-post',
        plugins: 'wp-plugins'
    }
} );

Then the translation-map.json would be generated attaching strings to the chunk alias instead of the chunk name. That would allow us to drop the need for a $chunk_name argument php side for parsing the map.

Thoughts?

[nerrad]

I decided to go ahead and implement the changes mentioned in my previous comment. I think php side, this aligns better with the work being done in https://core.trac.wordpress.org/ticket/20491.

[nerrad]

I'm not sure that this is going to land in GB. We're using a variation of this for Event Espresso. I'm wondering if anyone would find benefit if I publish the webpack plugin as a package in our repo?

[aduth]

I'm not sure that this is going to land in GB.

Is there a reason for this feeling aside from needing support in pushing it forward? Is it still on a trajectory to be an official core module?

[nerrad]

It's hard to communicate sentiment via text so let me clarify that my earlier statement wasn't intended to communicate I'm bummed or upset about it. I'm just communicating my perception of the state of things from conversations that have happened so far in the i18n chats in slack for wp core. I think this has served as a good example of what can be done but I'm not sure it will land as is (especially the gb php pieces). I'm fine with leaving this pull open for reference for now (or it might get iterated on once there's a clearer direction for what needs to happen).

Currently Event Espresso is using the webpack plugin as is right now in our own gutenberg integration work (with some improvements we've discovered it needed). I'm fine with dogfooding it for our needs for the time being while waiting for WP core to come up with a solution. However, as stated in my comment, I'm just wondering if there'd be value in us releasing it as an npm package via our (ee's) repo in case theirs others that could benefit from it - I'd also be fine with submitting it as a pull for the WordPress/packages but understand any reluctance to doing so being it's not known clearly what the direction will be for wpi18n yet.

[nerrad]

Is it still on a trajectory to be an official core module?

If I go by lack of feedback on it outside of the GB team I would say its not on trajectory to be an official core module.

[aduth]

Is it safe to say that this could be considered a post-5.0 enhancement? Or is it blocking or backward-incompatible in any way?

[nerrad]

tldr; this doesn't block merge.

The intent behind this was to complement what GB introduced with the @wordpress/i18n package in that it provides a mechanism for plugins to implement wp.i18n within their javascript. Since the merge of Gutenberg brings this to WP core, the goal was to come up with a suitable api for plugins to use as well once something landed in wp core (in the merge).

There was supposed to be further discussion on this (and other proposed solutions) in a trac ticket (and wpi18n slack but those meetings petered out) and I still think that should happen at some point. I still don't know what the plans are for translating GB strings once GB lands in core? Are GB strings just going to be extracted to php a-la pot-to-php? Will it be expected that plugins with javascript i18n will do something similar?

I don't think this necessarily holds up the merge for 5.0 but I do think that there's still a glaring hole in what plugins should be doing for i18n in javascript.

In the js projects I'm involved in right now, I use a variation of what I proposed in here to manage our i18n strings, but it'd definitely be better if there was a canonical api that WP offered.

I'm fine with this pull being bumped or even closed. If so, I'll probably just release the plugin I did on npm separately (will make it easier for us and others to pull in to projects if/as needed :) )

[aduth]

Yeah, there's a few localization things which I think would fall under the same realm of "needs to be done, but not blocking for a 5.0", including another I filed at #9846 and have since temporarily discontinued putting effort toward.

For 5.0, I think the pot-to-php workflow is a reasonable stop-gap solution.

At this point my main concern would be any foreseen incompatibilities that might occur in the eventual implementation (i.e. necessary for an API freeze).

[nerrad]

Closing this in favor of the work being done here: https://core.trac.wordpress.org/ticket/45103