Improve translation system workflow (#17214)

There are several limitations to translating with the Crowdin workflow such as:
• No way to locally verify html output for documentation
• Crowdin is confused by empty target tags
• Incorrect double escaping of xml.

Description of development approach
markdownTranslate:
• No longer incorrectly xml escape content before inserting it into the lxml tree. lxml already does this.
• No longer xml unescaped content coming from the lxml tree - lxml already does this.translateXliff: add a note to certain exceptions showing the line number of the markdown file to find markdown syntax errors easier.
• translateXliff: for lines that are purely structural (no translation still ensure that the line matches the skeleton, to catch markdown syntax errors much earlier.
md2html.py
• Restructured no longer to be an SCons tool, rather it is now a standalone script in user_docs, so that other code such as the nvdaL10nUtil script can use it.
nvdaL10util:
Added a new nvdal10nUtil program, which provides the following commands:
• xliff2md: converts a translated xliff file back into its original markdown file
nvdaL10util xliff2md <xliff file> <output xliff file>
• md2html: converts a markdown file into html, including support for the keyCommands document.
nvdaL10util md2html -t {userGuide|changes|keyCommands} -l <lang> <markdown file> <output html file>
• xliff2html: converts a xliff file all the way to html, including support for the keyCommands document.
nvdaL10util xliff2html -t {userGuide|changes|keyCommands} -l <lang> <xliff file> <output html file>
• stripXliff: takes a translated xliff file and generates a new xliff file that has all unnecessary content removed, so that when uploaded, only valid, added/changed translation trings are submitted to Crowdin. It removes: 
◦ empty target tags
◦ initial English source target tags
• Optionally removes any translation that already appears in the old xliff, so that the new file only contains added / changed translations since last download.
nvdaL10nUtil stripXliff [-o <old xliff>] <translated xliff file> <output stripped xliff file>
Copy
Documentation updates
• No longer talk about Crowdin approvals - this has been disabled due to inaccessibility
• No longer recommend Poedit's Crowdin cloud translation feature as it is buggy. Instead just recommend ownloading, translating with Poedit, and uploding again.

.gitignore

@@ -27,6 +27,7 @@ source/louis
 dlldata.c
 *.pdb
 .sconsign.dblite
+user_docs/*/*.md.sub
 user_docs/*/*.html
 user_docs/*/*.css
 extras/controllerClient/x86/nvdaController.h

.pre-commit-config.yaml

@@ -1,4 +1,4 @@
-exclude: ^user_docs/(?!en/).*$
+exclude: ^user_docs/(?!en/).+/.+\.md$
 
 # https://pre-commit.ci/
 # Configuration for Continuous Integration service

appveyor/scripts/setSconsArgs.ps1

@@ -1,5 +1,5 @@
 $ErrorActionPreference = "Stop";
-$sconsOutTargets = "launcher developerGuide changes userGuide keyCommands client moduleList"
+$sconsOutTargets = "launcher developerGuide changes userGuide keyCommands client moduleList nvdaL10nUtil"
 if(!$env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:feature_buildAppx) {
 	$sconsOutTargets += " appx"
 }

projectDocs/dev/developerGuide/sconscript

@@ -12,7 +12,20 @@ env = env.Clone()
 devDocsOutputDir = outputDir.Dir("devDocs")
 
 # Build the developer guide and move it to the output directory
-htmlFile = env.md2html("developerGuide.md")
+mdFile = env.File("developerGuide.md")
+# first substitute some variables such as NvDA version and URL into the markdown file
+mdFileSub = env.Substfile(
+	target=mdFile.abspath.replace(".md", ".md.sub"),
+	source=mdFile,
+	SUBST_DICT={
+		"NVDA_VERSION": env["version"],
+	},
+)
+htmlFile = env.Command(
+	target=mdFile.abspath.replace(".md", ".html"),
+	source=mdFileSub,
+	action=[f'@{sys.executable} user_docs/md2html.py -t developerGuide "$SOURCE" "$TARGET"'],
+)
 devGuide = env.Command(
 	target=devDocsOutputDir.File("developerGuide.html"), source=htmlFile, action=Move("$TARGET", "$SOURCE")
 )

projectDocs/translating/crowdin.md

@@ -1,9 +1,9 @@
 # Translating using Crowdin
 
-Crowdin is used to translate the main NVDA interface.
+Crowdin is used to translate the main NVDA interface and user documentation.
 NVDA's Crowdin project: <https://crowdin.com/project/nvda>.
 
-This document covers setting up a Crowdin account, connecting it with PoEdit, and translating the main interface using Crowdin and PoEdit.
+This document covers setting up a Crowdin account, connecting it with PoEdit, and translating the main interface and user documentation using Crowdin and PoEdit.
 
 ## Setup
 
@@ -17,47 +17,29 @@ Please include your Crowdin username and the languages you wish to translate.
 
 It is recommended that you use the latest version of PoEdit and NVDA for translating.
 Alternatively, you can use the [Crowdin web interface](https://support.crowdin.com/online-editor/) directly.
-As PoEdit only supports viewing approved strings, large translators team need to co-ordinate submitting unapproved strings to prevent conflicts.
+As PoEdit only supports viewing approved strings, large translator teams need to co-ordinate submitting unapproved strings to prevent conflicts.
 Using Crowdin's interface avoids this problem.
 
-PoEdit supports connecting with Crowdin directly.
 PoEdit's homepage is: <http://www.poedit.net/>
 
 1. Download the latest Windows PoEdit version at <https://poedit.net/download>
 1. Install it by following the on-screen instructions, the default options should be sufficient.
-1. When launching PoEdit:
-    1. Choose "Translate cloud project"
-    1. Connect your Crowdin account
-    1. Select NVDA and the language you wish to translate
 
 ### Translation reviews
-
-Translated strings will need to be reviewed and approved by a proofreader before being included in NVDA.
-A proofreader is required for each language.
-Proofreader status is granted on a case-by-case basis by messaging the [translators mailing list](https://groups.io/g/nvda-translations) or <info@nvaccess.org>
-
-Proofreaders approve strings using the [Crowdin web interface](https://support.crowdin.com/online-editor/).
-PoEdit does not support viewing unapproved strings from other translators.
-When manually uploading to Crowdin from PoEdit, proofreaders are able to auto-approve all submitted strings.
+Due to accessibility issues, for now translation approvals have been disabled on Crowdin.
+Any translation uploaded to Crowdin is automatically available in the project.
+However, joining the project as a translator is by invitation only.
 
 ## Translation workflows
 
-There are 3 common workflows for translating with Crowdin:
+There are 2 common workflows for translating with Crowdin:
 
-1. Only on Crowdin's web interface, either with:
-    - only one proofreader approving their own translations,
-    - or with many translators making suggestions and a proofreader approving them.
-1. Multiple translators translating on PoEdit.
-    - Using Crowdin cloud synchronization.
-    - Proofreaders approve the translations on Crowdin's web interface.
-1. Translating on PoEdit without cloud synchronization and performing manual uploads to Crowdin.
-    - Translators with proofreader status can upload strings manually with automatic approval.
-    As such, this may be a preference for single or small-team translators using PoEdit.
-    - Manual uploads without cloud synchronization means conflicts can occur, translator teams must be co-ordinated if following this approach.
+1. Translating strings directly via Crowdin's interface. Or
+1. Downloading from Crowdin, translating with Poedit and uploading again.
 
 ## Translating using PoEdit
 
-After opening a .po file you will be placed on a list with all of the strings to translate.
+After opening a .po or .xliff file you will be placed on a list with all of the strings to translate.
 
 You can read the status bar to see how many strings have already been translated, the number of untranslated messages, and how many are fuzzy.
 A fuzzy string is a message which has been automatically translated, thus it may be wrong.
@@ -77,12 +59,15 @@ Each time you press this key, PoEdit saves the po file, and if you check compile
 
 NVDA provides additional shortcuts for PoEdit which are described in [the User Guide](https://www.nvaccess.org/files/nvda/documentation/userGuide.html#Poedit).
 
-If you are unsure of meaning of the original interface message, consult automatic comments (also called translator comments), by pressing `control+shift+a`.
+If you are unsure of the meaning of the original interface message, consult automatic comments (also called translator comments), by pressing `control+shift+a`.
 Some comments provide an example output message to help you understand what NVDA will say when speaking or brailling such messages.
 
-## Translating the interface
+## Translating NVDA's interface
+
+* Download nvda.po from the Files section of your language on Crowdin.
+* Open the po file in Poedit, translate, and save the file.
+* Upload the po file back to Crowdin.
 
-Open "nvda.po" for the language you want to translate in PoEdit.
 Alternatively, you can use the [Crowdin interface directly](https://support.crowdin.com/online-editor/).
 
 ### Messages with formatting strings
@@ -160,11 +145,56 @@ In Crowdin, this information appears at the end of the context section.
 
 ### Testing the interface translation
 
-1. To test the current interface messages, save the current nvda.po file, and copy the nvda.mo file to the following location: `nvdadir/locale/langcode/LC_MESSAGES`
+1. To test the current interface messages, save the current nvda.po file in Poedit, and copy the nvda.mo file to the following location: `nvdadir/locale/langcode/LC_MESSAGES`
     - `nvdadir`: the directory where NVDA has been installed
     - `langcode`: the ISO 639-1 language code for your language (e.g. en for English, es for Spanish, etc.)
 1. Restart NVDA, then go to the NVDA menu, go to Preferences and choose General Settings, or press `NVDA+control+g` to open General Settings.
 1. From the language list, select your language (if it is listed), press `enter` and say yes when you're asked to restart NVDA.
 1. The messages you have translated should now be heard or brailled in your native language provided that the synthesizer you are using supports your language or a braille code for your language exists.
 
 Whenever you add or update your interface translations, repeat the steps above (copying the updated .mo file and restarting NVDA) to test your updated translation messages.
+
+## Translating User Documentation
+
+Documentation available for translation includes:
+
+* The NVDA user guide (userGuide.xliff)
+* The NVDA What's New document (changes.xliff)
+
+To translate any of these files:
+
+* Download the xliff file from the Files section of your language on Crowdin.
+    * Make sure to choose "Download" not "Export xliff".
+* Make a copy of this file.
+* Open the xliff file in Poedit, translate, and save the file.
+* Use the nvdaL10nUtil program to strip the xliff so that it only contains translations that were added / changed. E.g.
+```
+nvdaL10nUtil stripXliff -o <old xliff file> <translated xliff file> <output stripped xliff file>
+```
+* Upload the xliff file back to Crowdin. If it is a stripped xliff file, it is safe to check the `allow target to match source` checkbox.
+
+Alternatively, you can use the [Crowdin interface directly](https://support.crowdin.com/online-editor/).
+
+### Translating markdown
+
+The English NVDA user documentation is written in markdown syntax.
+The xliff file you are directly translating has been generated from that markdown file.
+It contains the content of any line that requires translation, shown in the order it appears in the original markdown file.
+
+Structural lines that do not contain any translatable content (such as blank lines, hidden table header rows, table header body separator lines etc) are not included here.
+
+Structural syntax from the beginning and end of lines (such as heading prefix like `###`, heading anchors like `{#Introduction}`, and initial and final vertical bars on table rows) has been removed from the content to translate, but is available to view in the translator notes for that line.
+Content may still however contain inline markdown syntax such as links, inline code fences (``` `` ```), and table column separators (`|`).
+This syntax must be kept intact when translating.
+
+All strings for translation contain translator notes which include:
+* Line: the original line number in the markdown file.
+* prefix: any structural markdown on the line before this content.
+* Suffix: any structural markdown on the line after this content.
+
+### Verifying your translation
+
+When ever you have saved the xliff file with Poedit, you can use the nvdaL10nUtil program to generate the html version of the documentation file. E.g.
+```
+nvdaL10nUtil xliff2html -t [userGuide|changes|keyCommands] <xliff file> <output html file>
+```

projectDocs/translating/readme.md

@@ -23,19 +23,14 @@ For further information please see the [Release Process page](https://github.com
 You can view [Crowdin](https://crowdin.com/project/nvda) for an up to date report on the status of translating the NVDA interface.
 If you would like to improve or would like to work on a new language, please write to the [NVDA translations mailing list](https://groups.io/g/nvda-translations).
 
-The translation status of user documentation (User Guide and Changes) can only be checked by translators.
-
 ## New Localization
 
 Start by subscribing to the translation list above so that you can get help and advice.
 
 The current process for translation is split between multiple processes:
 
-- Crowdin for the NVDA interface
-- The legacy SVN translation system for the User Guide and Changes files.
-This is planned to move to Crowdin.
-- The legacy SVN translation system for Character Descriptions, Symbols and Gestures.
-This is planned to move to GitHub.
+- Crowdin for the NVDA interface and user documentation
+- Github for Character Descriptions, Symbols and Gestures.
 
 Read [Files to be Localized](#files-to-be-localized) to learn the translation for process for these.
 
@@ -57,6 +52,6 @@ Note that linked guides may be out of date, as the translation system is undergo
 - characterDescriptions.dic: names of characters in your language, see [Translating Character Descriptions](https://www.nvaccess.org/files/nvda/documentation/developerGuide.html#characterDescriptions) for more info.
 - symbols.dic: names of symbols and punctuation in your language, see [Translating Symbols](https://www.nvaccess.org/files/nvda/documentation/developerGuide.html#symbolPronunciation) for more information.
 - gestures.ini: remapping of gestures for your language, see [Translating Gestures](https://www.nvaccess.org/files/nvda/documentation/developerGuide.html#TranslatingGestures) for more information.
-- userGuide.md: the User Guide, see [Translating the User Guide](https://github.com/nvaccess/nvda/wiki/TranslatingUserGuide) for more information.
-- changes.md (optional): a list of changes between releases, see [Translating Changes](https://github.com/nvaccess/nvda/wiki/TranslatingChanges) for more information.
+- userGuide.md: the User Guide, see [Translating using Crowdin](./crowdin.md) for more information.
+- changes.md (optional): a list of changes between releases, see [Translating using Crowdin](./crowdin.md) for more information.
 - Add-ons (optional): a set of optional features that users can install, see [Translating Addons](https://github.com/nvaccess/nvda/wiki/TranslatingAddons) for more information.

pyproject.toml

@@ -73,6 +73,10 @@ ignore_packages = [
 	"importlib-metadata",
 	"opencv-python",
 
+	# Incompatible Apache licenses.
+	# For building NVDA, but not included.
+	"nuitka",
+
 	# Incompatible Apache licenses.
 	# System testing frameworks, not bundled with NVDA.
 	"robotframework",

requirements.txt

@@ -21,6 +21,8 @@ pycaw==20240210
 
 # Packaging NVDA
 py2exe==0.13.0.2
+# xliff2html is packaged with nuitka 
+nuitka==2.4.8
 
 # Creating XML unit test reports
 unittest-xml-reporting==3.2.0

sconstruct

@@ -149,7 +149,6 @@ env = Environment(
 	tools=[
 		"textfile",
 		"gettextTool",
-		"md2html",
 		"doxygen",
 		"recursiveInstall",
 		"m4",
@@ -346,7 +345,23 @@ for xliffFile in env.Glob(os.path.join(userDocsDir.path, "*", "*.xliff")):
 	)
 # Allow all markdown files to be converted to html in user_docs
 for mdFile in env.Glob(os.path.join(userDocsDir.path, "*", "*.md")):
-	htmlFile = env.md2html(mdFile)
+	# first substitute some variables such as NVDA version and URL into the markdown file
+	mdFileSub = env.Substfile(
+		target=mdFile.abspath.replace(".md", ".md.sub"),
+		source=mdFile,
+		SUBST_DICT={
+			"NVDA_VERSION": env["version"],
+			"NVDA_URL": versionInfo.url,
+			"NVDA_COPYRIGHT_YEARS": versionInfo.copyrightYears,
+		},
+	)
+	lang = os.path.split(os.path.dirname(mdFile.path))[-1]
+	docType = os.path.basename(mdFile.path).split(".")[0]
+	htmlFile = env.Command(
+		target=mdFile.abspath.replace(".md", ".html"),
+		source=mdFileSub,
+		action=[f'@{sys.executable} user_docs/md2html.py -l {lang} -t {docType} "$SOURCE" "$TARGET"'],
+	)
 	styleInstallPath = os.path.dirname(mdFile.abspath)
 	installedStyle = env.Install(styleInstallPath, styles)
 	installedHeadingsStyle = env.Install(styleInstallPath, numberedHeadingsStyle)
@@ -362,11 +377,14 @@ for mdFile in env.Glob(os.path.join(userDocsDir.path, "*", "*.md")):
 	env.Depends(htmlFile, mdFile)
 
 # Create key commands files
-for userGuideFile in env.Glob(os.path.join(userDocsDir.path, "*", "userGuide.md")):
-	keyCommandsHtmlFile = env.md2html(
-		userGuideFile.abspath.replace("userGuide.md", "keyCommands.html"), userGuideFile
+for userGuideFileSub in env.Glob(os.path.join(userDocsDir.path, "*", "userGuide.md.sub")):
+	lang = os.path.split(os.path.dirname(userGuideFileSub.path))[-1]
+	keyCommandsHtmlFile = env.Command(
+		target=userGuideFileSub.abspath.replace("userGuide.md.sub", "keyCommands.html"),
+		source=userGuideFileSub,
+		action=[f'@{sys.executable} user_docs/md2html.py -l {lang} -t keyCommands "$SOURCE" "$TARGET"'],
 	)
-	env.Depends(keyCommandsHtmlFile, userGuideFile)
+	env.Depends(keyCommandsHtmlFile, userGuideFileSub)
 
 # Build unicode CLDR dictionaries
 env.SConscript("cldrDict_sconscript", exports=["env", "sourceDir"])
@@ -713,3 +731,11 @@ source = env.Dir(os.path.join(os.getcwd(), "dist"))
 # Putting the target in the output dir automatically causes AppVeyor to package it as an artefact
 target = env.File(os.path.join(outputDir.abspath, "library_modules.txt"))
 env.Alias("moduleList", env.GenerateModuleList(target, source))
+
+nvdaL10nUtil = env.Command(
+	target=outputDir.File("nvdaL10nUtil.exe"),
+	source="user_docs/nvdaL10nUtil.py",
+	ENV=os.environ,
+	action=f"nuitka --assume-yes-for-downloads --remove-output --standalone --onefile --output-dir={outputDir.abspath} --include-module=mdx_truly_sane_lists --include-module=markdown_link_attr_modifier --include-module=mdx_gh_links user_docs/nvdaL10nUtil.py",
+)
+env.Alias("nvdaL10nUtil", nvdaL10nUtil)