Text classification in different languages

[the42]

I followed the helpful tutorial on Text classification. It seems this type of classification assumes English input? What would the yaml configuration look like if my target language is german?

What languages does ludwig support in general?

[arnavgarg1]

Hi @the42, the tutorial does assume the text input is in English, however, this is not a requirement!

Ludwig does support at least one text encoder out of the box that is multilingual: XLMRoberta. Here's a masking example I used that's in Hindi. I also tried an equivalent in Spanish.

Additionally, if you would like, you can use a pretrained multilingual text encoder from Huggingface by using the AutoTransformer text encoder and specifying the pretrained model/path from Huggingface.

Let me know if this helps!