feat: atlas push pull scripts: FC-55 #317

Amr-Nash · 2024-05-09T23:01:05Z

Description

This pull request provides the CI scripts and developer tooling for the Atlas Translations Management Design
proposal related to OEP-58.

Testing instructions

Testing the `pull_translations`:

make ATLAS_OPTIONS='--repository=Zeit-Labs/openedx-translations --revision=fc_55_sample' pull_translations

When the above command is executed:

The None-English translations will be pulled from openedx-translations repo into I18N/ directory.
Then each language file will be split and distributed to the modules one by one
In the end, the pulled translations will be removed and only the split translations can be found in the modules' directories

If the script is working as intended:

The above command should not produce errors when executed.
The pulled translations should be correctly distributed among the modules in the expected paths.
The I18N directory should not exist after the script has been executed.
The app should have all its translations in place and can now be deployed.

Testing the `extract_translations`:

make extract_translations

When the above command is executed:

The English sources from all values/strings.xml files in all modules will be extracted
The keys of each translation entry will be updated to contain the name of the module followed by a dot e.g. "Module_name.OLD_KEY_of_the_entry".
Then all of the updated entries will be put in one file: I18N/src/main/res/values/strings.xml

If this script is working as intended:

The above command should not produce errors when executed.
Every translatable entry from the English sources of each module should exist in I18N/src/main/res/values/strings.xml with the corresponding comment, if present, placed before it.
The keys in I18N should contain the name of the module from which the entry originated, as specified.
The values and comments should be identical to the original ones.

Q and A

1. When are source strings extracted? what do the files look like?

An automated action will be developed to extract the English sources if they are changed in the Android repo, then the action will push the sources to the translations repo.
The sources will be in one file: I18N/src/main/res/values/strings.xml and it will look like this:

<resources>
	<string name="Module_one.FIRST_ENTRY_KEY_IN_MODULE_ONE">English Translation for the first entry in module one.</string>
	<string name="Module_one.SECOND_ENTRY_KEY_IN_MODULE_ONE">English Translation for the second entry in module one.</string>
        .
        .
	<string name="Module_two.FIRST_ENTRY_KEY_IN_MODULE_TWO">Translation for the first entry in module two</string>
        .
        .
</resources>

2. Does anything happen to those files before they are added to openedx-translations

The only extracted file is the English sources from the modules in the Android repo as it is the only translation that lives there.

3. When translation files are pulled via atlas what do the files look like?

When the translation files are pulled via atlas the files' structure is going to look like the structure below:

I18N/src/main/res/
----values-ar/
--------strings.xml
----values-uk/
--------strings.xml
----values-es/
--------strings.xml

As an example the values-es/strings.xml would look like:
I18N/src/main/res/values-es/strings.xml

<resources>
	<string name="Module_one.FIRST_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la primera entrada del módulo uno.</string>
	<string name="Module_one.SECOND_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la segunda entrada del módulo uno.</string>
        .
        .
	<string name="Module_two.FIRST_ENTRY_KEY_IN_MODULE_TWO">Traducción de la primera entrada del módulo dos.</string>
        .
        .
</resources>

4) What happens to those files after `atlas pull` to make them work in the app build process?

After atlas pull is completed, all language translation files are retrieved, similar to the process described in the previous question. Subsequently, a Python script is executed to process each file. This script iterates through each translation file, splits it, removes the module name from the keys, and organizes each entry into its corresponding module.

For instance, the I18N/src/main/res/values-es/strings.xml file is split into two separate files:

Module_one/src/main/res/values-es/strings.xml
Module_two/src/main/res/values-es/strings.xml

Inside Module_one/src/main/res/values-es/strings.xml, the content would resemble:

<resources>
	<string name="FIRST_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la primera entrada del módulo uno.</string>
	<string name="SECOND_ENTRY_KEY_IN_MODULE_ONE">Traducción al inglés de la segunda entrada del módulo uno.</string>
        .
        .
</resources>

Similarly, inside Module_two/src/main/res/values-es/strings.xml, it would appear as:

<resources>
	<string name="FIRST_ENTRY_KEY_IN_MODULE_TWO">Traducción de la primera entrada del módulo dos.</string>
        .
        .
</resources>

Once the Python script completes its task and the files are split, the make script removes the entire I18N directory and its contents.

5) Why is this needed?

This new process is being introduced to have the best combination of Developer Experience and Translator Experience.

The best experience for Translators requires combining source strings into as few transifex resources as possible. The best experience for Engineers requires splitting translation source files to fit within the modular architecture.

6) What should live in openedx translations?

Translations that were done by the open-edx translators are going to be stored in openedx translations.

7) What should live in atlas?

Atlas is the tool that we are using to pull the translations from openedx translations; therefor, no translations are stored in atlas.

8) What should live in the openedx-app-Android repo?

In the Android app repo only the English sources are going to be stored.

TODO

Refactor the code into a single Python script to improve code reuse
Add CI scripts for make extract_translations
Test with the Android team
Review by Brian Smith

openedx-webhooks · 2024-05-09T23:01:09Z

Thanks for the pull request, @Amr-Nash! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

Feel free to add as much of the following information to the ticket as you can:

supporting documentation
Open edX discussion forum threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will be done via the GitHub pull request interface. As a reminder, our process documentation is here.

Please let us know once your PR is ready for our review and all tests are green.

Amr-Nash · 2024-05-23T13:52:32Z

@OmarIthawi could you please review this PR if we are done with the ios PR?

OmarIthawi · 2024-05-26T09:43:53Z

Tested on both extract and pull (below):

Fresh (before pull)	After pulling Ukrainian translations

OmarIthawi

Thanks @Amr-Nash! I've tested the pull request and it worked on both combine and pull.

The README looks good and I have no comments on it.

I built the application with Android studio and it worked okay.

Please address the minor comment and it should be good to go.

i18n_scripts/translation.py

OmarIthawi · 2024-05-26T10:16:03Z

@brian-smith-tcril @volodymyr-chekyrta please take a look and approve the tests to run.

OmarIthawi · 2024-05-31T20:24:11Z

Thanks @Amr-Nash, please squash the commits to fix the Lint Commit Messages failure https://github.com/openedx/openedx-app-android/actions/runs/9293840766/job/25604477406?pr=317

OmarIthawi

Thanks for addressing the error handling.

Amr-Nash · 2024-05-31T21:41:55Z

Thanks @Amr-Nash, please squash the commits to fix the Lint Commit Messages failure https://github.com/openedx/openedx-app-android/actions/runs/9293840766/job/25604477406?pr=317

Done.

@OmarIthawi

Thanks for addressing the error handling.

My pleasure.

OmarIthawi

One more comment please Amr.

i18n_scripts/translation.py

OmarIthawi

Please also add the language renaming after split so it works with fr_CA language for example.

Makefile

i18n_scripts/requirements.txt

README.md

volodymyr-chekyrta

Tested ✅
Works fine for me 👍

README.md

Amr-Nash · 2024-06-09T15:29:28Z

@OmarIthawi I squashed the commits. Now, it is ready to be merged.

Extract will extract the English language resources from all modules to the I18N folder. Pull will pull all other languages translations from the openedx-traslations repository to the i18n folder then split them to thier modules.

openedx-webhooks · 2024-06-12T19:24:39Z

@Amr-Nash 🎉 Your pull request was merged! Please take a moment to answer a two question survey so we can improve your experience in the future.

openedx-webhooks added the open-source-contribution PR author is not from Axim or 2U label May 9, 2024

volodymyr-chekyrta added product review PR requires product review before merging and removed product review PR requires product review before merging labels May 10, 2024

Amr-Nash mentioned this pull request May 13, 2024

make the combine script Zeit-Labs/openedx-app-android-contrib#1

Closed

Amr-Nash force-pushed the combine_strings branch 2 times, most recently from 9a947c7 to d51c1a8 Compare May 13, 2024 14:42

OmarIthawi mentioned this pull request May 18, 2024

feat: add openedx-app-android | FC-55 openedx/openedx-translations#6072

Closed

2 tasks

Amr-Nash force-pushed the combine_strings branch from 020253f to ac1eade Compare May 21, 2024 15:51

Amr-Nash marked this pull request as ready for review May 25, 2024 15:52

OmarIthawi suggested changes May 26, 2024

View reviewed changes

i18n_scripts/translation.py Outdated Show resolved Hide resolved

OmarIthawi mentioned this pull request May 30, 2024

[Android] Internationalization Infrastructure #124

Open

OmarIthawi approved these changes May 31, 2024

View reviewed changes

Amr-Nash force-pushed the combine_strings branch from 686841f to 01d61c4 Compare May 31, 2024 21:35

OmarIthawi reviewed Jun 3, 2024

View reviewed changes

i18n_scripts/translation.py Outdated Show resolved Hide resolved

OmarIthawi reviewed Jun 5, 2024

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

i18n_scripts/requirements.txt Outdated Show resolved Hide resolved

Amr-Nash commented Jun 6, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

volodymyr-chekyrta previously approved these changes Jun 7, 2024

View reviewed changes

volodymyr-chekyrta reviewed Jun 7, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

Amr-Nash dismissed volodymyr-chekyrta’s stale review via b0fedf0 June 7, 2024 12:03

Amr-Nash force-pushed the combine_strings branch from 13c9e86 to 22d5f7a Compare June 9, 2024 15:27

feat: atlas push pull scripts: FC-55

456cd50

Extract will extract the English language resources from all modules to the I18N folder. Pull will pull all other languages translations from the openedx-traslations repository to the i18n folder then split them to thier modules.

OmarIthawi force-pushed the combine_strings branch from 22d5f7a to 456cd50 Compare June 12, 2024 16:17

brian-smith-tcril approved these changes Jun 12, 2024

View reviewed changes

volodymyr-chekyrta merged commit 065d583 into openedx:develop Jun 12, 2024
3 checks passed

This was referenced Jun 13, 2024

feat: add mobile apps and with support for pre-extract step | FC-55 openedx/openedx-translations#6394

Merged

chore: Remove in-code ukranian translations | FC-55 #342

Merged

PavloNetrebchuk mentioned this pull request Nov 6, 2024

Develop to main. Release v2 #399

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: atlas push pull scripts: FC-55 #317

feat: atlas push pull scripts: FC-55 #317

Amr-Nash commented May 9, 2024 •

edited

Loading

openedx-webhooks commented May 9, 2024 •

edited

Loading

Amr-Nash commented May 23, 2024

OmarIthawi commented May 26, 2024

OmarIthawi left a comment

OmarIthawi commented May 26, 2024

OmarIthawi commented May 31, 2024

OmarIthawi left a comment

Amr-Nash commented May 31, 2024 •

edited

Loading

OmarIthawi left a comment

OmarIthawi left a comment

volodymyr-chekyrta left a comment

Amr-Nash commented Jun 9, 2024

openedx-webhooks commented Jun 12, 2024

feat: atlas push pull scripts: FC-55 #317

feat: atlas push pull scripts: FC-55 #317

Conversation

Amr-Nash commented May 9, 2024 • edited Loading

Description

Testing instructions

Testing the pull_translations:

When the above command is executed:

If the script is working as intended:

Testing the extract_translations:

When the above command is executed:

If this script is working as intended:

Q and A

1. When are source strings extracted? what do the files look like?

2. Does anything happen to those files before they are added to openedx-translations

3. When translation files are pulled via atlas what do the files look like?

4) What happens to those files after atlas pull to make them work in the app build process?

5) Why is this needed?

6) What should live in openedx translations?

7) What should live in atlas?

8) What should live in the openedx-app-Android repo?

TODO

openedx-webhooks commented May 9, 2024 • edited Loading

Amr-Nash commented May 23, 2024

OmarIthawi commented May 26, 2024

OmarIthawi left a comment

Choose a reason for hiding this comment

OmarIthawi commented May 26, 2024

OmarIthawi commented May 31, 2024

OmarIthawi left a comment

Choose a reason for hiding this comment

Amr-Nash commented May 31, 2024 • edited Loading

OmarIthawi left a comment

Choose a reason for hiding this comment

OmarIthawi left a comment

Choose a reason for hiding this comment

volodymyr-chekyrta left a comment

Choose a reason for hiding this comment

Amr-Nash commented Jun 9, 2024

openedx-webhooks commented Jun 12, 2024

Amr-Nash commented May 9, 2024 •

edited

Loading

Testing the `pull_translations`:

Testing the `extract_translations`:

4) What happens to those files after `atlas pull` to make them work in the app build process?

openedx-webhooks commented May 9, 2024 •

edited

Loading

Amr-Nash commented May 31, 2024 •

edited

Loading