Transifex integration (translation management)

[tofi86]

Hey guys,

as I've been translating the rest of the messages to german a couple of weeks ago, I've been wondering whether we could improve the translation workflow somehow.
I'm fearing the situation when translations are drifting apart because their maintainers doesn't watch changes to the english master message files...

I remembered that I once signed up for Transifex (though never used it until now), which offers free translation management for Open-Source projects, and created an IDPF organization and an EpubCheck project:

https://www.transifex.com/projects/p/epubcheck/

I added all current translations to the transifex project and set up the transifex commandline integration (http://docs.transifex.com/developer/client/) with this PullRequest.

This is how translating messages in Transifex looks like (if you signed up):
https://www.transifex.com/projects/p/epubcheck/translate/#fr/ErrorsWarnings/35761294

I think, Transifex makes translation management a bit easier, as it automatically keeps the english message files up to date with the github repo and translators can easily see when new messages need to be translated.

However, it doesn't (automatically) syncs itself with GitHub – this has to be done by hand at the moment (but: there exists tools for syncing – see below). A manual workflow for this could look like the following:

• a contributor translates/adds/changes messages at the transifex project
  • someone of the epubcheck core team
    • pulls changes from transifex to his local repository with tx pull -a
    • then pushes changes to the github repo manually
• a contributor translates/adds/changes messages at the github project (PulLRequest, push)
  • someone of the epubcheck core team
    • updates his local github repository
    • then pushes changes to the transifex repo manually with tx push -s -t

If you want to test the transifex integration, sign up at transifex for free and @rdeltour or me will add you as a contributor to the project.

Next steps include:

• setting up the transifex commandline client (http://docs.transifex.com/developer/client/setup)
  • install python and pip (http://pip.readthedocs.org/en/latest/installing.html)
  • install the transifex client: sudo pip install --upgrade transifex-client
  • create a ~/.transifexrc file with your transifex login credentials (http://docs.transifex.com/developer/client/setup#transifexrc)
• pull translations from transifex repository to your local git repo: tx pull -a

Wanna give it a try and tell me what you think?

TODOs

The transifex integration is just a first draft for this task. There are still a few issues to solve:

• maven integration?
  • it would be nice to pull transifex changes to the github repository automatically before building and testing epubcheck
• syncing the transifex and github repositories
  • http://blog.transifex.com/post/67760039192/github-transifex-integration
  • https://github.com/jsilland/txgh
• special char escaping
  • when pulling message files from transifex which include "special characters" (like `!:/, transifex automatically escapes these characters:
    • example 1
      • github repo: file_not_found=Fichier introuvable : '%1$s'
      • transifex checkout: file_not_found=Fichier introuvable \: '%1$s'
    • example 2
      • github repo: argument_needed=Es wird mindestens ein Parameter erwartet!
      • transifex checkout: argument_needed=Es wird mindestens ein Parameter erwartet\!
  • I don't know if this behaviour is okay for the java integration?!
• multi-line messages
  • pulling message files from transifex which include multi-line messages (e.g. the EpubCheck help message) results in a one-liner:

help_text = EpubCheck v%1$s\n\nPour lancer cet outil, le premier argument doit être le nom (avec le chemin)\ndu fichier à vérifier.\n\nSi un fichier non-epub est vérifié, la version epub du fichier\ndoit être fournie en utilisant -v et le type du fichier en utilisant -mode.\nLa version par défaut est 3.0.\n\nModes et versions supportés \: \n--mode opf -v 2.0\n--mode opf -v 3.0\n--mode xhtml -v 2.0\n--mode xhtml -v 3.0\n--mode svg -v 2.0\n--mode svg -v 3.0\n--mode nav -v 3.0\n--mode mo  -v 3.0 // Pour la vérification des Media Overlays\n--mode exp  // Pour des archives EPUB éclatés\n\nL'outil accepte aussi les options suivantes \:\n--save \t       \= sauve l'epub créé depuis l'epub éclaté\n--out <file>     \= émet un fichier XML de validation (utilisé - pour émettre dans stdout).\n--xmp <file>     \= émet un fichier XMP de validation (utilisé - pour émettre dans stdout).\n--json <file>    \= émet un fichier JSON de validation (utilisé - pour émettre dans stdout)\n-m <file>        \= identique à --mode\n-o <file>         \= identique à --out\n-x <file>        \= identique à --xmp\n-j <file>        \= identique à --json\n--failonwarnings[+|-] \= Par défautn l'outil renvoie 1 si des erreurs sont trouvées et 0 s'il n'y a pas d'erreur.\n                        En utilisant --failonwarnings le processus sortira avec un status 1\n                        si des avertissements ou des erreurs sont trouvés et 0 seulement s'il n'y a ni erreur ni avertissement.\n-q, --quiet      \= aucun message émis sur la console, seulement les erreurs, seulement sur la sortie\n-f, --fatal      \= n'émet que les erreurs fatales sur le sortie\n-e, --error      \= n'émet que les erreurs et les erreurs fatales sur le sortie\n-w, --warn       \= émet les erreurs fatales, les erreurs et les avertissements sur la sortie\n-u, --usage      \= inclis aussi les message d'utilisation des epub sur la sortie\n                    (par défaut à OFF); si activé, les messages d'utilisation seront toujours inclus dans la sortie\n\n-l, --listChecks [<file>] \= listes les identifiants de message et leur degré de sévérité dans un fichier de personnalitation des messages dans le fichier  <file>\n                          ou la console\n-c, --customMessages [<file>] \= remplace les niveaux de sévérité des messages par ceux du fichier de personnalitation des messages défini dans <file>\n\n-h, -? ou --help \= affiche ce message d'aide\n

I'm open for suggestions and your opinions regarding this topic.

• Is it too much overhead?
• Shall we just don't care about outdated translations?
• Are there better tools for translation management?
• How to solve some of the TODOs?

Cheers,
Tobias

[tofi86]

rebased against current master with german translation

[murata2makoto]

I joined the Transifex. Yes, this is good to keep message in different languages to be synchronized.

I'm actually surprised by the fact that quite a few new messages have not been translated yet. On Transifex, I translated some of them. Will do more.

[pettarin]

My two cents : until there is full integration between GitHub/mvn and Transifex, using the latter seems to bring more problems than benefits.

A few comments on my experience writing the Italian translation:

1. I had to figure out on my own, mostly extrapolating from the preexisting translations like FR and DE, what characters should be escaped (and how to do that) in the various message files
2. placing somewhere the Jing EN templates might be helpful (I worked from the DE ones)
3. the Italian help message (the one printed when epubcheck is executed with -h) has lines with at most 80 characters (78 actually) for old fashioned die-hard consolers like me

[tofi86]

Rebased this branch on current master (epubcheck 4.0.0)

[tofi86]

Even though there's still no fully automated GitHub <-> Transifex workflow, @rdeltour and I decided to merge this first step of the Transifex integration to master in 4.0.1

I wrote a bash script wich solves issues with special char escaping and multi-line messages (for details see first post on the top) which works on top of the Transifex Commandline Client.
I'll doing some rework today or tomorrow and will push it to the PR. This script could eventually be used in a Travis or Maven integration in the future...

Also I set up a wiki page about contributing translations: https://github.com/IDPF/epubcheck/wiki/Translating I'm going to add some information soon on how to pull translations from Transifex and push to GitHub.

[tofi86]

With f3c67b4 I added a bash script to pull and normalize Transifex translations as described in my last comment.

To run this script you need to have a Transifex account and install the Transifex Commandline Client! Instructions: http://docs.transifex.com/client/

You need to run this script from the repository's root directory!
e.g. ./src/build/transifex-pull.sh --all

usage: transifex-pull.sh [--all | <2-digit-country-code>]
examples:
  transifex-pull.sh --all
  transifex-pull.sh de

Note: tested on Mac OS X only!

[tofi86]

@rdeltour ready to merge! Once this is done, I'm going to perform a transifex-pull on the master branch to initialy normalize the current translations.
This will delete comments in message properties files like

# This is the German translation of MessageBundle.properties file
# Translation by Tobias Fischer (https://github.com/tofi86)

and will replace them with the default comments from the original Message properties file

# This is the default MessageBundle.properties file

This is how Transifex handles comments in source files: it just copies them over to the translated file. no chance to get custom comments through the process.

[tofi86]

This is weird. @rdeltour's merge of this branch didn't include all commits. master is missing 4c25396 and f3c67b4. Going to merge these manually.

[rdeltour]

@rdeltour's merge of this branch didn't include all commits.

argh. maybe I forgot to pull before rebasing? Sorry for the hassle, let me know if I can help.

[tofi86]

@rdeltour no worries. Seems to be merged fine now. Double commit's though, but it works...