Biterm Topic Model (BTM) is a word co-occurrence based topic model that learns topics by modeling word-word co-occurrences patterns (e.g., biterms). (In constrast, LDA and PLSA are word-document co-occurrence topic models, since they model word-document co-occurrences.)
A biterm consists of two words co-occurring in the same context, for example, in the same short text window. Unlike LDA models the word occurrences, BTM models the biterm occurrences in a corpus. In generation procedure, a biterm is generated by drawn two words independently from a same topic. In other words, the distribution of a biterm b=(wi,wj) is defined as:
P(b) = \sum_k{P(wi|z)*P(wj|z)*P(z)}.
With Gibbs sampling algorithm, we can learn topics by estimate P(w|k) and P(z).
More detail can be referred to the following paper:
Xiaohui Yan, Jiafeng Guo, Yanyan Lan, Xueqi Cheng. A Biterm Topic Model For Short Text. WWW2013.
The code has been test on linux. If you on windows, please install cygwin (with bc, wc, make).
The code includes a runnable example, you can run it on Linux by:
$ cd script
$ sh runExample.sh
It trains BTM over the documents in sample-data/doc_info.txt and output the topics. The doc_info.txt contains all the training documents, where each line represents one document with words separated by space as:
word1 word2 word3 ....
(Note: the sample data is only used for illustration of the usage of the code. It is not the data set used in the paper.)
You can change the paths of data files and parameters in script/runExample.sh to run over your own data.
Indeed, the runExample.sh processes the input documents in 4 steps.
1. Index the words in the documents
To simplify the main code, we provide a python script to map each word to a unique ID (starts from 0) in the documents.
$ python script/indexDocs.py <doc_pt> <dwid_pt> <voca_pt>
doc_pt input docs to be indexed, each line is a doc with the format "word word ..."
dwid_pt output docs after indexing, each line is a doc with the format "wordId wordId ..."
voca_pt output vocabulary file, each line is a word with the format "wordId word"
2. Topic learning
The next step is to train the model using the documents represented by word ids.
$ ./src/btm est <K> <W> <alpha> <beta> <n_iter> <save_step> <docs_pt> <model_dir>
K int, number of topics
W int, size of vocabulary
alpha double, Symmetric Dirichlet prior of P(z), like 1
beta double, Symmetric Dirichlet prior of P(w|z), like 0.01
n_iter int, number of iterations of Gibbs sampling
save_step int, steps to save the results
docs_pt string, path of training docs
model_dir string, output directory
The results will be written into the directory "model_dir":
- k20.pw_z: a K*M matrix for P(w|z), suppose K=20
- k20.pz: a K*1 matrix for P(z), suppose K=20
3. Inference topic proportions for documents, i.e., P(z|d)
If you need to analysis the topic proportions of each documents, just run the following common to infer that using the model estimated.
$ ./src/btm inf <type> <K> <docs_pt> <model_dir>
K int, number of topics, like 20
type string, 4 choices:sum_w, sum_b, lda, mix. sum_b is used in our paper.
docs_pt string, path of docs to be inferred
model_dir string, output directory
The result will be output to "model_dir":
- k20.pz_d: a N*K matrix for P(z|d), suppose K=20
4. Results display
Finally, we also provide a python script to illustrate the top words of the topics and their proportions in the collection.
$ python script/topicDisplay.py <model_dir> <K> <voca_pt>
model_dirthe output dir of BTM
Kthe number of topics
voca_ptthe vocabulary file
- 2015-01-12, v0.5, improve the usability of the code
- 2012-09-25, v0.1
If there is any question, feel free to contact: Xiaohui Yan(xhcloud@gmail.com).