Other Language Support

[The-Randalorian]

Other languages should be supported. These should have custom TTS, STT and Actions plugins that can be swapped out for the proper components. That being said, these other plugins should still be based off the original English plugins to maintain compatibility with other plugins like home automation. The English Actions plugin in particular must maintain a very similar structure, as this is the main way other plugins interface with open assistant.

The good news is many of the core libraries have or are working on multiple language support. Stanza, the sentence unpacking library, supports a lot of languages, as seen here. The STT library DeepSpeech has experimental Mandarin Chinese models, with other languages probably being worked on. The TTS library pyttsx3 also supports multiple languages.

Most of the work needed for this will probably be in the speech and general knowledge components of the Actions plugin. The speech plugin will need to be tested to verify test sentences are handled properly (many band-aid functions will probably have to be made). The general knowledge plugin will probably need to be modified to account for different words used in the language (for example, the list of colors will need to be changed to an appropriate list for the language.)

This issue will eventually be closed and replaced with separate issues for different languages. That will happen when more languages are supported than not.

Edit: clarity/accuracy

[arguiot]

Hello, I wanted to know concretely how to add support for a language. Do you expect each language to have its own logic? Or is there a system that manages languages (like what Olivia, a similar project, have)?

I have to admit that the code doesn't really speak to me. I'm not sure what to do, and I'm not sure how much work to do for this project.

[The-Randalorian]

Tl;dr Start by taking a look at "englishActions/speech.py", change the strings on lines 38-41 from "en" to your language ("fr"). Head to "englishActions/general_knowledge.py" and change any English words to an appropriate translation. Modify the test cases at the bottom of the file "englishActions/speech.py" and run it to test the basic functionality quickly.

I expect each language to have some of it's own logic, but not much. Most of the "logic" will be boilerplate/patches/quick fixes for some errors and changing some of the built in knowledge base (IE there is a list of color names that needs to be changed, or the list of verbs that can be used to turn on the lights).

Basically, anything labelled "english" needs to be modified for other languages. I would start by downloading the English code and heading to the "englishActions/speech.py" file. On lines 38, 40 and 41, change the string "en" to your language of choice ("fr" for example). That tell change stanza, the library that works to comprehend the sentence, to switch to french. That will handle most of the conversion for you. It might be fine, but I would disable the bandaid functions at the bottom of this file. These functions apply very simple and quick fixes to stanza's output to fix language errors. Disable them by changing the list post_bandaids from it's current value to an empty list post_bandaids = []

Then go to "englishActions/general_knowledge.py". This file contains the classes that Open Assistant uses to understand how to interpret objects. Start by going through all the built in strings and see if they need to be changed, and figure out what to change them to. For example, lines 231-238 have a list of verbs used to figure out how well liked something is. Lines 400 and 403 have nouns that describe a person of a specific gender ("guy", "man", "girl", "woman").

To test everything, run "englishActions/speech.py". That will run the test cases at the bottom of the file and also let you enter text to have the program try to interpret.