Message files splitting

[tricoder42]

It seems that one unsolved problem in JavaScript internationalization is automatic splitting of message files, similar to code splitting.

I would like to tackle this problem in v3 and figure out, what API changes we need to make to get it done.

Related issues: #474 #362

---

Code splitting creates main bundle with loader and then smaller chunks, which can be loaded separately. We know, what modules are included in each chunk (webpack stats data) and we know, what messages are included in each module (origin of message in lingui catalogs). Therefore we can determine what messages are included in each chunk and generate message files for each chunk.

As with code splitting, even locales need to have main bundle, which will include locale specific data like plural rules, date/number formats, etc.

The only question is, how to load chunks of message files.

Webpack handles loading of code chunks automagically (== I have no idea how it works). When we type import "module" and the module isn't loaded yet, webpack automatically loads the correct chunk.

We can't do that with locales. Usually, we would write import "locale/en.json", but that would load whole catalog. In fact, we need just import "locale/en.acf422.json", but that filename is generated automatically. We probably need to hook into webpack require and when chunk xyz is loaded, then locale locale/en.xyz.json must be loaded as well.

Finally, the last piece in puzzle - once we have locale chunks loaded automatically, we need to load them into Lingui object.

---

Notes

While trying to figure out how to load message catalog automatically with code chunk, I discovered next-client-pages-loader from Next.js, which injects some extra code to generated chunk.

We need to either override default webpack require or call another require before/after chunk is loaded, so adding extra code to generted bundle might be related.

---

Checklist:

• Generate locale files for each chunk
• Figure out, how to hook into webpack's require and load locale chunk along with code chunks
• Figure out how to feed loaded locale chunks into Lingui

[theKashey]

1. ✅Extract messages from the source code.
2. ✅Build the app
3. Add plugin, tap on the compilation emit, to analyze how modules are distributed in chunks ( stat.json)
4. Form the same "chunks" from translated messages, these translations-per-chunks could be added into webpack emit as well.
5. Add another plugin, to hook into __webpack_require__.e(requireEnsure) (like mini-css-extract-plugin does) to load translation the translation chunk next to the code chunk.

[theKashey]

• selected for development :)

8 months later... 🐌 🐢

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[renchap]

Still valid I think :)

[semoal]

Probably this feature could be a nice one on a 3.1.0 of lingui, @tricoder42 if you have some time to open other issue to write a little roadmap of 3.1.0, I could help as much as I can 👍🏻

[tricoder42]

@semoal I've just created a milestone and added two new possible features.

This issue isn't trivial because it depends on the framework. I would start with Next.js because it's closer to me. Just clarify — we're talking about splitting the compiled catalog, not the source one. We can already split the source catalog using catalogs attribute and, to be hones, even one large source catalog isn't an issue. The problem is when you send such large catalog to the client, which potentially might explode the bundle size.

It would be great, if we could split the catalog based on generated chunks, but I have no idea how to hook into Webpack. @ScriptedAlchemy mentioned module federation yesterday.

[hectorcoronado]

• Open another issue

[semoal]

Hi @theKashey , since we have right now version 3.X.X. really clean, we released some interesting new features and some hot-fixes so probably the next milestones are reasonably clean.

If you can provide some details of what you got, I'm open to help you or start working on it. This feature will be amazing.

[semoal]

There's something I'm not getting here, clearly, and I'm wondering if you could please provide some insight?

Feel free to open another issue if you found something not working correctly 👍🏻

[theKashey]

@semoal - all I've got is in the comment above. The general idea is to extract "translations" in the same way some CSS-in-JS do a static extraction, producing CSS files from js ones.
The only difference that such files have to pull translation from the other source and (optional) support runtime language selection.

[micahbule]

Any particular update on this if we can already do this or if there's any existing workaround?

[JSteunou]

until having it all automatic hooked on build split feature from webpack or other builders, could it be possible to add this feature through a naive implementation with namespace concept. On every call to Trans / t adding a namespace parameter which will be used to split the catalogue files.

To avoid boilerplate maybe some configuration could be added to automatically set this namespace for all files of a folder / folders / glob pattern.

Loading the appropriate catalogue would still be manual though but nothing too difficult nor painful...

What do you think?

[semoal]

It's a good idea if you want to collaborate @JSteunou, we're open to collaborations.

[JSteunou]

My hands are full but if at some point we are in the need of this feature at work I might sell it to be able to work on it on work time. But that could take years because the todo list is already huge so if someone want to tackle it before, go.

[SanichKotikov]

Hi,

Is there any way to extract messages separately for different pages(folders)? For example, I have next project structure:

/src
  /_common
  /page1
  /page2
  /...

and I want to get next jsons (with messages):

/locales
  /_common
    /en.json
    /fr.json
  /page1
    /en.json
    /fr.json
  /page2
    /en.json
    /fr.json

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[JSteunou]

nooo stale bot please keep it up keep it up :)