Add support for custom symbol dictionaries (#16823)

Closes #16739

Summary of the issue:
There is a desire to provide symbol pronunciation rules in optional symbol dictionaries, e.g. for ancient languages. IN #16739, we concluded that having the ability to provide them in add-ons would be most helpful.

Description of user facing changes
Removed the option Include Unicode Consortium data (including emoji) when processing characters and symbols from the speech settings category and replaced it by a checkable list box to enable optional dictionaries. This only contains the CLDR dictionary by default, but can be expanded by add-ons.
Added the ability for add-ons to provide extra dictionaries. They could be mandatory, meaning that enabling the add-on means enabling the dictionary. If optional, they are listed in the listbox mentioned above.
Description of development approach
Added a SymbolDictionaryDefinition class to characterProcessing that contains all data about a dictionary.
Changed the logic in characterProcessing to create speech symbol processors based on enabled definitions.
For dictionary registration in add-ons, I chose a similar approach as for custom braille tables, e.g. you define the dictionaries in a subsection of the add-on manifest.

projectDocs/dev/developerGuide/developerGuide.md

@@ -949,9 +949,11 @@ Then the handler needs to be registered - preferably in the constructor of your
     addonHandler.isCLIParamKnown.register(processArgs)
 ## Packaging Code as NVDA Add-ons {#Addons}
 
-To make it easy for users to share and install plugins, drivers and braille translation tables, they can be packaged in to a single NVDA add-on package which the user can then install into a copy of NVDA via the Add-on Store found under Tools in the NVDA menu.
+Add-ons make it easy for users to share and install plugins, drivers, speech symbol dictionaries and braille translation tables.
+They can be packaged in to a single NVDA add-on package, which the user can then install into a copy of NVDA via the Add-on Store found under Tools in the NVDA menu.
 Add-on packages are only supported in NVDA 2012.2 and later.
-An add-on package is simply a standard zip archive with the file extension of "`nvda-addon`" which contains a manifest file, optional install/uninstall code and one or more directories containing plugins, drivers and/or and braille translation tables.
+An add-on package is simply a standard zip archive with the file extension of "`nvda-addon`".
+It can contain a manifest file, install/uninstall code and directories containing plugins, drivers, speech symbol dictionaries and braille translation tables.
 
 ### Non-ASCII File Names in Zip Archives {#nonASCIIFileNamesInZip}
 
@@ -1013,21 +1015,23 @@ The lastTestedNVDAVersion field in particular is used to ensure that users can b
 It allows the add-on author to make an assurance that the add-on will not cause instability, or break the users system.
 When this is not provided, or is less than the current version of NVDA (ignoring minor point updates e.g. 2018.3.1) then the user will be warned not to install the add-on.
 
-The manifest can also specify information regarding the additional braille translation tables provided by the add-on.
-Please refer to the [braille translation tables section](#BrailleTables) later on in this document.
+The manifest can also specify information regarding the additional speech symbol dictionaries and braille translation tables provided by the add-on.
+Please refer to the [speech symbol dictionaries](#AddonSymbolDictionaries) and [braille translation tables](#BrailleTables) sections.
 
 #### An Example Manifest File {#manifestExample}
-    --- start ---
-    name = "myTestAddon"
-    summary = "Cool Test Add-on"
-    version = "1.0.0"
-    description = "An example add-on showing how to create add-ons!"
-    author = "Michael Curran <mick@example.com>"
-    url = "https://github.com/nvaccess/nvda/blob/master/projectDocs/dev/addons.md"
-    docFileName = "readme.html"
-    minimumNVDAVersion = "2021.1"
-    lastTestedNVDAVersion = "2022.3.3"
-    --- end ---
+
+```ini
+name = "myTestAddon"
+summary = "Cool Test Add-on"
+version = "1.0.0"
+description = "An example add-on showing how to create add-ons!"
+author = "Michael Curran <mick@example.com>"
+url = "https://github.com/nvaccess/nvda/blob/master/projectDocs/dev/addons.md"
+docFileName = "readme.html"
+minimumNVDAVersion = "2021.1"
+lastTestedNVDAVersion = "2022.3.3"
+```
+
 ### Plugins and Drivers {#pluginsAndDrivers}
 
 The following plugins and drivers can be included in an add-on:
@@ -1036,6 +1040,7 @@ The following plugins and drivers can be included in an add-on:
 * Braille display drivers: Place them in a brailleDisplayDrivers directory in the archive.
 * Global plugins: Place them in a globalPlugins directory in the archive.
 * Synthesizer drivers: Place them in a synthDrivers directory in the archive.
+* [Speech symbol dictionaries](#AddonSymbolDictionaries): Place them in the directory for one or more [locales](#localizingAddons) with a file name of `symbols-<name>.dic`, e.g. `locale\en\symbols-greek.dic`.
 * [Braille translation tables](#BrailleTables): Place them in a brailleTables directory in the archive.
 
 ### Optional install / Uninstall code {#installUninstallCode}
@@ -1067,6 +1072,7 @@ This directory should contain directories for each language it supports, using t
 
 Each of these language directories can contain a locale-specific manifest file called manifest.ini, which can contain a small subset of the manifest fields for translation.
 These fields are summary and description.
+You can also override the `displayName` field for speech symbol dictionaries and braille translation tables.
 All other fields will be ignored.
 
 #### Locale-specific Messages {#localeMessages}
@@ -1077,6 +1083,47 @@ To allow plugins in your add-on to access gettext message information via calls
 This function cannot be called in modules that do not belong to an add-on, e.g. in a scratchpad subdirectory.
 For more information about gettext and NVDA translation in general, please read the [Translating NVDA page](https://github.com/nvaccess/nvda/blob/master/projectDocs/translating/readme.md)
 
+#### Speech symbol dictionaries {#AddonSymbolDictionaries}
+
+You can provide custom speech symbol dictionaries in add-ons to improve symbol pronunciation.
+The process to create custom speech symbol dictionaries is very similar to that of the [translation process of existing symbols](#symbolPronunciation).
+Note that [complex symbols](#complexSymbols) are not supported.
+
+Custom dictionaries must be placed in a language directory and have a filename in the form `symbols-<name>.dic`, where `<name>` is the name that has to be provided in the add-ons manifest.
+All locales implicitly inherit the symbol information for English, though any of this information can be overridden for specific locales.
+
+When adding a dictionary not marked as mandatory, some information must be provided such as its display name, since it should be shown in the speech category of the settings dialog.
+A dictionary can also be marked mandatory, in which case it is always enabled with the add-on.
+When an add-on ships with dictionaries, this information is included in its manifest in the optional `symbolDictionaries` section.
+For example:
+
+```ini
+[symbolDictionaries]
+[[greek]]
+displayName = Greek
+mandatory = false
+
+[[hebrew]]
+displayName = Biblical Hebrew
+mandatory = true
+```
+
+In the above example, `greek` is a dictionary that is optional and will be listed in the speech category of NVDA's settings dialog under the "Extra dictionaries for character and symbol processing" setting.
+Its file will be stored as `locale\en\symbols-greek.dic`, whereas French translations of the symbols are stored in `locale\fr\symbols-greek.dic`.
+When using NVDA in French, symbols that aren't defined in the French dictionary inherit the symbol information for English.
+
+Also in the example, the `hebrew` dictionary is marked mandatory and will therefore always be enabled as long as the add-on is active.
+Its file will be stored as `locale\en\symbols-hebrew.dic`, whereas French translations of the symbols are stored in `locale\fr\symbols-hebrew.dic`.
+
+Note that for the display name of the dictionary to be translated, an entry should be added to a [locale manifest](#localeManifest).
+For example, add the following to `locale\fr\manifest.ini`:
+
+```ini
+[symbolDictionaries]
+[[hebrew]]
+displayName = Hébreu Biblique
+```
+
 ### Add-on Documentation {#AddonDoc}
 
 Documentation for an add-on should be placed in a doc directory in the archive.
@@ -1121,6 +1168,15 @@ Providing a custom table, whether it has the same file name as a standard table
 The only exception to this rule applies to tables that are included within other tables.
 While they don't have to be included in the manifest of the add-on, they can only be included from other tables that are part of the same add-on.
 
+Note that for the display name of the table to be translated, an entry should be added to a [locale manifest](#localeManifest).
+For example, add the following to `locale\fr\manifest.ini`:
+
+```ini
+[brailleTables]
+[[no-no-8dot.utb]]
+displayName = Norvégien Braille informatique 8 points - Remplacement
+```
+
 Custom tables can also be placed in the brailleTables subdirectory of the scratchpad directory.
 In this case, the table metadata can be placed in a `manifest.ini` file in the root of the scratchpad in the exact same format as the example above.
 Basically, this means that, whether using an add-on or the scratchpad, the requirements and implementation steps are equal.

source/addonHandler/__init__.py

@@ -1031,6 +1031,13 @@ class AddonManifest(ConfigObj):
 		input = boolean(default=true)
 		output = boolean(default=true)
 
+# Symbol Pronunciation
+[symbolDictionaries]
+	# The key is the symbol dictionary file name (not the full path)
+	[[__many__]]
+		displayName = string()
+		mandatory = boolean(default=false)
+
 # NOTE: apiVersion:
 # EG: 2019.1.0 or 0.0.0
 # Must have 3 integers separated by dots.
@@ -1071,6 +1078,10 @@ def __init__(self, input, translatedInput=None):
 				value = tableConfig.get("displayName")
 				if value:
 					self["brailleTables"][fileName]["displayName"] = value
+			for fileName, dictConfig in self._translatedConfig.get("symbolDictionaries", {}).items():
+				value = dictConfig.get("displayName")
+				if value:
+					self["symbolDictionaries"][fileName]["displayName"] = value
 
 	@property
 	def errors(self):