[MRG] Lda training visualization in visdom

docs/notebooks/Training_visualizations.ipynb

@@ -0,0 +1,189 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Setup Visdom\n",
+    "\n",
+    "Install it with:\n",
+    "\n",
+    "`pip install visdom`\n",
+    "\n",
+    "Start the server:\n",
+    "\n",
+    "`python -m visdom.server`\n",
+    "\n",
+    "Visdom now can be accessed at http://localhost:8097 in the browser.\n",
+    "\n",
+    "\n",
+    "# LDA Training Visualization\n",
+    "\n",
+    "To monitor the LDA training, a list of Metrics can be passed to the LDA function call for plotting their values live as the training progresses.  \n",
+    "\n",
+    "Let's plot the training stats for an LDA model being trained on Lee corpus. We will use the four evaluation metrics available for topic models in gensim: Coherence, Perplexity, Topic diff and Convergence. (using separate hold_out and test corpus for evaluating the perplexity)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Using TensorFlow backend.\n"
+     ]
+    }
+   ],
+   "source": [
+    "import os\n",
+    "import re\n",
+    "import gensim\n",
+    "from gensim.models import ldamodel\n",
+    "from gensim.corpora.dictionary import Dictionary\n",
+    "\n",
+    "# Set file names for train and test data\n",
+    "test_data_dir = '{}'.format(os.sep).join([gensim.__path__[0], 'test', 'test_data'])\n",
+    "lee_train_file = test_data_dir + os.sep + 'lee_background.cor'\n",
+    "lee_test_file = test_data_dir + os.sep + 'lee.cor'\n",
+    "\n",
+    "def read_corpus(fname):\n",
+    "    texts = []\n",
+    "    with open(fname, encoding=\"ISO-8859-1\") as f:\n",
+    "        for line in f:\n",
+    "            # lower case all words\n",
+    "            lowered = line.lower()\n",
+    "            # remove punctuation and split into seperate words\n",
+    "            words = re.compile('\\w+').findall(lowered)\n",
+    "            texts.append(words)\n",
+    "    return texts\n",
+    "\n",
+    "training_texts = read_corpus(lee_train_file)\n",
+    "test_texts = read_corpus(lee_test_file)\n",
+    "\n",
+    "# Split test data into hold_out and test corpus\n",
+    "holdout_texts = test_texts[:25]\n",
+    "test_texts = test_texts[25:]\n",
+    "\n",
+    "dictionary = Dictionary(training_texts)\n",
+    "\n",
+    "training_corpus = [dictionary.doc2bow(text) for text in training_texts]\n",
+    "holdout_corpus = [dictionary.doc2bow(text) for text in holdout_texts]\n",
+    "test_corpus = [dictionary.doc2bow(text) for text in test_texts]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from gensim.models.callbacks import CoherenceMetric, DiffMetric, PerplexityMetric, ConvergenceMetric\n",
+    "\n",
+    "# define perplexity callback for hold_out and test corpus\n",
+    "pl_holdout = PerplexityMetric(corpus=holdout_corpus, logger=\"visdom\", viz_env=\"LdaModel\", title=\"Perplexity (hold_out)\")\n",
+    "pl_test = PerplexityMetric(corpus=test_corpus, logger=\"visdom\", viz_env=\"LdaModel\", title=\"Perplexity (test)\")\n",
+    "\n",
+    "# define other remaining metrics available\n",
+    "ch_umass = CoherenceMetric(corpus=training_corpus, coherence=\"u_mass\", logger=\"visdom\", viz_env=\"LdaModel\", title=\"Coherence (u_mass)\")\n",
+    "diff_kl = DiffMetric(distance=\"kullback_leibler\", logger=\"visdom\", viz_env=\"LdaModel\", title=\"Diff (kullback_leibler)\")\n",
+    "convergence_jc = ConvergenceMetric(distance=\"jaccard\", logger=\"visdom\", viz_env=\"LdaModel\", title=\"Convergence (jaccard)\")\n",
+    "\n",
+    "callbacks = [pl_holdout, pl_test, ch_umass, diff_kl, convergence_jc]\n",
+    "\n",
+    "# training LDA model\n",
+    "model = ldamodel.LdaModel(corpus=training_corpus, id2word=dictionary, passes=5, num_topics=5, callbacks=callbacks)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "When the model is set for training, you can open http://localhost:8097 to see the training progress."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "-21.8411539992\n"
+     ]
+    }
+   ],
+   "source": [
+    "# to get a metric value on a trained model\n",
+    "print(CoherenceMetric(corpus=training_corpus, coherence=\"u_mass\").get_value(model=model))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The four types of graphs which are plotted for LDA:\n",
+    "\n",
+    "**Coherence**\n",
+    "\n",
+    "Coherence is a measure used to evaluate topic models. A good model will generate coherent topics, i.e., topics         with high topic coherence scores. Good topics are topics that can be described by a short label based on the topic     terms they spit out. \n",
+    "\n",
+    "<img src=\"Coherence.gif\">\n",
+    "\n",
+    "Now, this graph along with the others explained below, can be used to decide if it's time to stop the training. We     can see if the value stops changing after some epochs and that we are able to get the highest possible coherence       of our model.  \n",
+    "\n",
+    "\n",
+    "**Perplexity**\n",
+    "\n",
+    "Perplexity is a measurement of how well a probability distribution or probability model predicts a sample. In LDA, topics are described by a probability distribution over vocabulary words. So, perplexity can be used to compare probabilistic models like LDA.\n",
+    "\n",
+    "<img src=\"Perplexity.gif\">\n",
+    "\n",
+    "For a good model, perplexity should be as low as possible.\n",
+    "\n",
+    "\n",
+    "**Topic Difference**\n",
+    "\n",
+    "Topic Diff calculates the distance between two LDA models. This distance is calculated based on the topics, by either using their probability distribution over vocabulary words (kullback_leibler, hellinger) or by simply using the common vocabulary words between the topics from both model.\n",
+    "\n",
+    "<img src=\"Diff.gif\">\n",
+    "\n",
+    "In the heatmap, X-axis define the Epoch no. and Y-axis define the distance between the identical topic from consecutive epochs. For ex. a particular cell in the heatmap with values (x=3, y=5, z=0.4) represent the distance(=0.4) between the topic 5 from 3rd epoch and topic 5 from 2nd epoch. With increasing epochs, the distance between the identical topics should decrease.\n",
+    "  \n",
+    "  \n",
+    "**Convergence**\n",
+    "\n",
+    "Convergence is the sum of the difference between all the identical topics from two consecutive epochs. It is basically the sum of column values in the heatmap above.\n",
+    "\n",
+    "<img src=\"Convergence.gif\">\n",
+    "\n",
+    "The model is said to be converged when the convergence value stops descending with increasing epochs."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.4.3"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

gensim/models/callbacks.py

@@ -0,0 +1,149 @@
+import gensim
+import logging
+import copy
+import numpy as np
+from queue import Queue
+
+# Visdom is used for training stats visualization
+try:
+    from visdom import Visdom
+    VISDOM_INSTALLED = True
+except ImportError:
+    VISDOM_INSTALLED = False
+
+
+class Metric(object):
+    def __init__(self):
+        pass
+
+    def get_value(self, **parameters):
+        for parameter, value in parameters.items():
+            setattr(self, parameter, value)
+
+
+class CoherenceMetric(Metric):
+    def __init__(self, corpus=None, texts=None, dictionary=None, coherence=None, window_size=None, topn=None, logger=None, viz_env=None, title=None):
+        self.corpus = corpus
+        self.dictionary = dictionary
+        self.coherence = coherence
+        self.texts = texts
+        self.window_size = window_size
+        self.topn = topn
+        self.logger = logger
+        self.viz_env = viz_env
+        self.title = title
+
+    def get_value(self, **kwargs):
+        # only one of the model or topic would be defined
+        self.model = None
+        self.topics = None
+        super(CoherenceMetric, self).get_value(**kwargs)
+        cm = gensim.models.CoherenceModel(self.model, self.topics, self.texts, self.corpus, self.dictionary, self.window_size, self.coherence, self.topn)
+        return cm.get_coherence()
+
+
+class PerplexityMetric(Metric):
+    def __init__(self, corpus=None, logger=None, viz_env=None, title=None):
+        self.corpus = corpus
+        self.logger = logger
+        self.viz_env = viz_env
+        self.title = title
+
+    def get_value(self, **kwargs):
+        super(PerplexityMetric, self).get_value(**kwargs)
+        corpus_words = sum(cnt for document in self.corpus for _, cnt in document)
+        perwordbound = self.model.bound(self.corpus) / corpus_words
+        return np.exp2(-perwordbound)
+
+
+class DiffMetric(Metric):
+    def __init__(self, distance="jaccard", num_words=100, n_ann_terms=10, normed=True, logger=None, viz_env=None, title=None):
+        self.distance = distance
+        self.num_words = num_words
+        self.n_ann_terms = n_ann_terms
+        self.normed = normed
+        self.logger = logger
+        self.viz_env = viz_env
+        self.title = title
+
+    def get_value(self, **kwargs):
+        super(DiffMetric, self).get_value(**kwargs)
+        diff_matrix, _ = self.model.diff(self.other_model, self.distance, self.num_words, self.n_ann_terms, self.normed)
+        return np.diagonal(diff_matrix)
+
+
+class ConvergenceMetric(Metric):
+    def __init__(self, distance="jaccard", num_words=100, n_ann_terms=10, normed=True, logger=None, viz_env=None, title=None):
+        self.distance = distance
+        self.num_words = num_words
+        self.n_ann_terms = n_ann_terms
+        self.normed = normed
+        self.logger = logger
+        self.viz_env = viz_env
+        self.title = title
+
+    def get_value(self, **kwargs):
+        super(ConvergenceMetric, self).get_value(**kwargs)
+        diff_matrix, _ = self.model.diff(self.other_model, self.distance, self.num_words, self.n_ann_terms, self.normed)
+        return np.sum(np.diagonal(diff_matrix))
+
+
+class Callback(object):
+    def __init__(self, metrics):
+        # list of metrics to be plot
+        self.metrics = metrics
+
+    def set_model(self, model):
+        self.model = model
+        self.previous = None
+        # check for any metric which need model state from previous epoch
+        if any(isinstance(metric, (DiffMetric, ConvergenceMetric)) for metric in self.metrics):
+            self.previous = copy.deepcopy(model)
+            # store diff diagnols of previous epochs
+            self.diff_mat = Queue()
+        if any(metric.logger=="visdom" for metric in self.metrics):
+            if not VISDOM_INSTALLED:
+                raise ImportError("Please install Visdom for visualization")
+            self.viz = Visdom()
+            # store initial plot windows of every metric (same window will be updated with increasing epochs) 
+            self.windows = []
+        if any(metric.logger=="shell" for metric in self.metrics):
+            # set logger for current topic model
+            model_type = type(self.model).__name__
+            self.log_type = logging.getLogger(model_type)
+
+    def on_epoch_end(self, epoch, topics=None):
+        # plot all metrics in current epoch
+        for i, metric in enumerate(self.metrics):
+            value = metric.get_value(topics=topics, model=self.model, other_model=self.previous)
+            metric_label = type(metric).__name__[:-6]
+            # check for any metric which need model state from previous epoch
+            if isinstance(metric, (DiffMetric, ConvergenceMetric)):
+                self.previous = copy.deepcopy(self.model)
+
+            if metric.logger=="visdom":
+                if epoch==0:
+                    if value.ndim>0:
+                        diff_mat = np.array([value])
+                        viz_metric = self.viz.heatmap(X=diff_mat.T, env=metric.viz_env, opts=dict(xlabel='Epochs', ylabel=metric_label, title=metric.title))
+                        # store current epoch's diff diagonal
+                        self.diff_mat.put(diff_mat)
+                        # saving initial plot window
+                        self.windows.append(copy.deepcopy(viz_metric)) 
+                    else:
+                        viz_metric = self.viz.line(Y=np.array([value]), X=np.array([epoch]), env=metric.viz_env, opts=dict(xlabel='Epochs', ylabel=metric_label, title=metric.title))
+                        # saving initial plot window
+                        self.windows.append(copy.deepcopy(viz_metric))
+                else:
+                    if value.ndim>0:
+                        # concatenate with previous epoch's diff diagonals
+                        diff_mat = np.concatenate((self.diff_mat.get(), np.array([value])))
+                        self.viz.heatmap(X=diff_mat.T, env=metric.viz_env, win=self.windows[i], opts=dict(xlabel='Epochs', ylabel=metric_label, title=metric.title))
+                        self.diff_mat.put(diff_mat)
+                    else:
+                        self.viz.updateTrace(Y=np.array([value]), X=np.array([epoch]), env=metric.viz_env, win=self.windows[i])
+
+            if metric.logger=="shell":
+                statement = " ".join(("Epoch:", epoch, metric_label, "estimate:", str(value)))
+                self.log_type.info(statement)
+