some thai word2vec problem

[billy800413]

hi, i'm trying to train the word2vec. i download thwiki-20170701-pages-articles.xml.bz2 and convert it to text. but i see all word can't be read in text, because it misses the vowels.

Code to Reproduce

from gensim.corpora import WikiCorpus
wiki_corpus = WikiCorpus('thwiki-20170701-pages-articles.xml.bz2')
with open("wiki_texts.txt",'w',encoding='utf-8') as output:
for text in wiki_corpus.get_texts():
output.write(' '.join(text) + '\n')

Expected Results
words with vowels

Actual Results
words without vowels

Versions
Python 3.4.3
gensim (2.2.0)

[piskvorky]

@billy800413 can you give some examples of the Expected/Actual results?

I'm thinking it may be somehow related to text preprocessing or tokenization for Thai.

[gojomo]

I may be wrong, but isn't Thai also a language without spaces-between-words, making tokenization potentially much more difficult & dependent on contextual-analysis?

[piskvorky]

It is, that's why I ask for the examples :)
Vowels are also realized very differently in Thai (below, above, ...).

[billy800413]

in thai, the word will be with some vowels(example วัตถุท้องฟ้า), but i see all words without vowels after converting thwiki-20170701-pages-articles.xml.bz2 to text (example วตถทองฟา). i find a data from
https://sites.google.com/site/rmyeid/projects/polyglot -> th_wiki_text.tar.lzma, and it also from wiki but the words with vowel. So now i use th_wiki_text.tar.lzma and ployglot to tokensize all sentence to train my word2vec.

[piskvorky]

Eh, I see they ripped off gensim, just copy&pasting gensim code and changing the author & license, without any attribution 😳
https://bitbucket.org/aboSamoor/polyglot2/src/2aba7d03dd2bfc97aa8c620161c2489df827b538/polyglot2/polyglot2.py?at=master&fileviewer=file-view-default

@billy800413 regarding Thai tokenization: do you have a tokenizer that can split Thai text into words? If so, we can help you plug it in.

[billy800413]

the code which i use to do tokennization:
import polyglot
from polyglot.text import Text, Word
zen = Text(sentence)
words=zen.words #the type is a list
for word in words:
output.write(word +' ')

[piskvorky]

OK, so before running your code in gensim, you want to plug in this Thai tokenizer. You can do that simply by replacing the tokenize function here with your own tokenizer:

import polyglot
from polyglot.text import Text, Word

def tokenize(content):
    zen = Text(content)
    return [word for word in zen.words]  # or any other text processing you like

I don't know how fast Polyglot is (on Thai texts), but gensim has no Thai tokenizer, so it's probably your best bet anyway.

[menshikh-iv]

Resolved in #1537, now WikiCorpus is more flexible @billy800413.