[WIP] Use Crowdin translations for reactjs.org

[bvaughn]

Relates to PR #82

TODO

• Update branch and resolve any merge conflicts
• Update plug-in for Gatsby 2.0

Pre-release checklist

• Make sure unlocalized docs URLs redirect to the correct default locale (English?)
• Test permalink redirects
• Test the error decoder (e.g. /docs/error-decoder.html?invariant=62&args[]=)
• Create locale selection page (in progress as /translations.html)
• Remember localization preference when navigating between translated and non-translated URLs.
• Audit new code for "TODO" comments
• Figure out why yarn install is failing on Circle CI
• Make sure Netlify deployment works

Maybe I should fork the (root project) Gatsby config files and use a feature flag approach to conditionally require the translation capable ones. Maybe that way we could land this as a work-in-progress without being fully committed.

Post-release checklist

• Verify older version redirects
• Verify Algolia integration

How does it work?

Only content from the content/docs directory is localized. All other sections/pages remain English only.

Downloading content from Crowdin

This PR adds a new root directory, crowdin. This directory contains some JavaScript files as well as a symlink for the default language (English) that points to the content/docs directory:

.
└── crowdin
   ├── __translated__ #----------------------- Initially empty, except for English
   │   └── en-US
   │       └── docs -> ../../../content/docs
   ├── __untranslated__ #--------------------- Contains symlinks to untranslated content
   │   ├── blog -> ../../content/blog
   │   └── # ...
   ├── config.js #---------------------------- Crowdin configuration settings
   └── download.js #-------------------------- Node Download script

A new Yarn task has been added (yarn crowdin:download) that uses the Crowdin API to download translations into an __exported__ subdirectory:

.
└── crowdin
   ├── __exported__
   │   └── # Crowdin expoert goes here ...
   ├── __translated__
   │   └── # ...
   ├── __untranslated__
   │   └── # ...
   └── # ...

Next the task identifies which languages have been translated past a certain threshold (specified by crowdin/config.js). For these languages, the script creates symlinks in the __translated__ subdirectory:

.
└── crowdin
   ├── __exported__
   │   └── # ...
   ├── __translated__
   │   ├── en-US
   │   │   └── docs -> ../../../content/docs
   │   ├── es-ES
   │   │   └── docs -> ../../__exported__/path/to/docs/es-ES/docs
   │   └── # Other languages that pass the threshold ...
   ├── __untranslated__
   │   └── # ...
   └── # ...

Gatsby integration

A new (local) gatsby-plugin-crowdin plugin has been created that knows how to create localized links to certain sections of the website (e.g. things within the translated "/docs" directory).

The gatsby-source-filesystem plugin has also been configured to read all content from the crowdin/__translated__/ and crowdin/__untranslated__/ (symlinked) directories rather than content. This way it consumes translated content when available. (Crowdin provides default language fallbacks for pages/sections that have not yet been translated for any given locale.)

This configuration is done via gatsby-config.js:

{
  resolve: 'gatsby-source-filesystem',
  options: {
    name: 'untranslated',
    path: `${__dirname}/crowdin/__untranslated__/`,
  },
},
{
  resolve: 'gatsby-source-filesystem',
  options: {
    name: 'translated',
    path: `${__dirname}/crowdin/__translated__/`,
  },
},

Because of the default initial symlink (crowdin/__translated__/en-US/docs -> content/docs) Gatsby will still serve English content when run locally, even if the Crowdin script has not been run. This should enable fast iteration and creation of new content.

Translations can be updated by running yarn crowdin:download (or automatically as part of CI deployment).

Language selector

A new Yarn task (yarn crowdin:update-languages) has been added to determine which translated languages have been downloaded. (This task is automatically run before yarn dev or yarn build in order to just-in-time update the list.) The task writes a list of locales to a local JSON file, languages.json:

.
└── crowdin
   ├── __exported__
   │   └── # ...
   ├── __translated__
   │   └── # ...
   ├── __untranslated__
   │   └── # ...
   ├── translated-languages.json # This is the list of local translations
   └── # ...

This languages.json list is imported into a translations page (pages/translations.js) and used to create a list of links to translated docs.

Locale persistence

By default, legacy links to docs pages (e.g. /docs/hello-world.html) are re-routed to a new page (docs-language-redirect.js) that determines which locale to redirect to (e.g. /en-US/docs/hello-world.html). This is done as follows:

• First it checks localStorage for the user's selected language. If one is found, it is used.
• Next it checks the user's preferred languages (using navigator.languages). If any have been translated, it is used.
• Lastly it falls back to English.

Each time a user visits a localized docs path, the website updates their currently selected language (in localStorage) so that subsequent visits (within this session or a new session) will restore their selected language.

Netlify also supports conditional redirects based on the Accept-Language header and GeoIP. It might be nice to use these instead (when running on the server) to pre-empt the client side redirect. This would need to be tested though, to make sure the two work well together.

[reactjs-bot]

Deploy preview for reactjs failed.

Built with commit a8ad83e

https://app.netlify.com/sites/reactjs/deploys/5b02eb3db3127403e81a46bb

[bvaughn]

The approach I'm currently taking is not very satisfying. It relies a lot on symlinks (that may cause pain for Windows contributors) and it only translates one section of the site (not even including the home page).

I spoke with Eric today about how it would be nice to translate more of the site (e.g. the home page, the side navs, even blog posts) and we came up with a rough plan that I will discuss with Dan on Monday as sort of a sanity check:

1. Move all of the files/folders inside of /content to /content/en-US (and update Crowdin to read from this new location). This will cause merge conflicts for all open PRs, but rebasing shouldn't be too bad and I think it's worth ripping the bandaid off.
2. Configure our Crowdin download script to download new languages to the root /content directory also (e.g. /content/zh-CN). Add a .gitignore file so that everything other than the default en-US directory is ignored.
3. Update Gatsby config to read all content from /content. This will automatically add locale ids into the URLs, since it's how Gatsby generates paths. (This will also enable the site to be run locally in an English-only mode without downloading Crowdin data.)
4. Add a prestart/prebuild Node script that concats all of the sitenav YML files (e.g. /content/docs/nav.yaml) into a single YML file in the root (e.g. /content/docs-nav.yml). This file will contain an entry for every translated language. This way we can embed it into our page templates at build time and change between locales at runtime. (This is kind of nasty but I don't know of a better way to handle this within our current constraints- if we wan to be able to translate the side nav.)
5. Make sure page links and nav links all stay within the current localization.

Eric is going to make the necessary changes to our Crowdin config file and then get back to me. I'll start making the necessary changes to our download script and Gatbsy configs.

We will need to decide on a redirect strategy for old, non-localized URLs. We could update Gatsby to generate server-side redirects for these so that they automatically point to English. Netlify also supports conditional redirects based on the Accept-Language header and GeoIP. It would also be great if we could rely on these (on the server) and maybe use an in-browser redirect for the client. Either way, we'll want to make sure navigation within a locale keeps you within that locale and doesn't require client side redirects for every click. (This would probably break Gatsby's preloading anyway.)

[facebook-github-bot]

Thank you for your pull request. We require contributors to sign our Contributor License Agreement, and yours has expired.

Before we can review or merge your code, we need you to email cla@fb.com with your details so we can update your status.

[koba04]

@bvaughn
What's the current status of this issue?
I've been hearing that "I want to contribute to React by translating the documentation. What should I do?".
Currently, they can translate the documentation on Crowdin, but the effort doesn't produce anything. 😢

Could we restart this work with integrate-crowdin branch?

[bvaughn]

Hey @koba04!

I kind of dropped the ball here because other things came up. Thanks for the ping. I'm going to see if we can find a way to finish it out. I agree that it's important.

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[threepointone]

This was 'fixed' in #1605. Thanks Brian!