Added EnsembleLda for stable LDA topics

.github/workflows/tests.yml

@@ -39,7 +39,8 @@ jobs:
       #
       - name: Update sbt
         run: |
-          echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
+          echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | sudo tee /etc/apt/sources.list.d/sbt.list
+          echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | sudo tee /etc/apt/sources.list.d/sbt_old.list
           curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo apt-key add
           sudo apt-get update -y
           sudo apt-get install -y sbt

docs/notebooks/ensemble_lda_with_opinosis.ipynb

@@ -0,0 +1,178 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "scrolled": false
+   },
+   "outputs": [],
+   "source": [
+    "import logging\n",
+    "from gensim.models import EnsembleLda, LdaMulticore\n",
+    "from gensim.models.ensemblelda import rank_masking\n",
+    "from gensim.corpora import OpinosisCorpus\n",
+    "import os"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "enable the ensemble logger to show what it is doing currently"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "elda_logger = logging.getLogger(EnsembleLda.__module__)\n",
+    "elda_logger.setLevel(logging.INFO)\n",
+    "elda_logger.addHandler(logging.StreamHandler())"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def pretty_print_topics():\n",
+    "    # note that the words are stemmed so they appear chopped off\n",
+    "    for t in elda.print_topics(num_words=7):\n",
+    "        print('-', t[1].replace('*',' ').replace('\"','').replace(' +',','), '\\n')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Experiments on the Opinosis Dataset\n",
+    "\n",
+    "Opinosis [1] is a small (but redundant) corpus that contains 289 product reviews for 51 products. Since it's so small, the results are rather unstable.\n",
+    "\n",
+    "[1] Kavita Ganesan, ChengXiang Zhai, and Jiawei Han, _Opinosis: a graph-based approach to abstractive summarization of highly redundant opinions [online],_ Proceedings of the 23rd International Conference on Computational Linguistics, Association for Computational Linguistics, 2010, pp. 340–348. Available from: https://kavita-ganesan.com/opinosis/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Preparing the corpus\n",
+    "\n",
+    "First, download the opinosis dataset. On linux it can be done like this for example:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!mkdir ~/opinosis\n",
+    "!wget -P ~/opinosis https://github.com/kavgan/opinosis/raw/master/OpinosisDataset1.0_0.zip\n",
+    "!unzip ~/opinosis/OpinosisDataset1.0_0.zip -d ~/opinosis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "path = os.path.expanduser('~/opinosis/')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Corpus and id2word mapping can be created using the load_opinosis_data function provided in the package.\n",
+    "It preprocesses the data using the PorterStemmer and stopwords from the nltk package.\n",
+    "\n",
+    "The parameter of the function is the relative path to the folder, into which the zip file was extracted before. That folder contains a 'summaries-gold' subfolder."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "opinosis = OpinosisCorpus(path)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Training"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**parameters**\n",
+    "\n",
+    "**topic_model_kind** ldamulticore is highly recommended for EnsembleLda. ensemble_workers and **distance_workers** are used to improve the time needed to train the models, as well as the **masking_method** 'rank'. ldamulticore is not able to fully utilize all cores on this small corpus, so **ensemble_workers** can be set to 3 to get 95 - 100% cpu usage on my i5 3470.\n",
+    "\n",
+    "Since the corpus is so small, a high number of **num_models** is needed to extract stable topics. The Opinosis corpus contains 51 categories, however, some of them are quite similar. For example there are 3 categories about the batteries of portable products. There are also multiple categories about cars. So I chose 20 for num_topics, which is smaller than the number of categories."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "elda = EnsembleLda(\n",
+    "    corpus=opinosis.corpus, id2word=opinosis.id2word, num_models=128, num_topics=20,\n",
+    "    passes=20, iterations=100, ensemble_workers=3, distance_workers=4,\n",
+    "    topic_model_class='ldamulticore', masking_method=rank_masking,\n",
+    ")\n",
+    "pretty_print_topics()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The default for **min_samples** would be 64, half of the number of models and **eps** would be 0.1. You basically play around with them until you find a sweetspot that fits for your needs."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "elda.recluster(min_samples=55, eps=0.14)\n",
+    "pretty_print_topics()"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

docs/src/apiref.rst

@@ -20,13 +20,15 @@ Modules:
     corpora/lowcorpus
     corpora/malletcorpus
     corpora/mmcorpus
+    corpora/opinosiscorpus
     corpora/sharded_corpus
     corpora/svmlightcorpus
     corpora/textcorpus
     corpora/ucicorpus
     corpora/wikicorpus
     models/ldamodel
     models/ldamulticore
+    models/ensemblelda
     models/nmf
     models/lsimodel
     models/ldaseqmodel

docs/src/auto_examples/howtos/run_doc.ipynb

@@ -15,7 +15,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "\nHow to Author Gensim Documentation\n==================================\n\nHow to author documentation for Gensim.\n"
+        "\nHow to Author Gensim Documentation\n==================================\n\nHow to author documentation for Gensim.\n\n"
       ]
     },
     {
@@ -61,7 +61,7 @@
       "cell_type": "markdown",
       "metadata": {},
       "source": [
-        "Correctness\n-----------\n\nIncorrect documentation can be worse than no documentation at all.\nTake the following steps to ensure correctness:\n\n- Run Python's doctest module on your docstrings\n- Run your documentation scripts from scratch, removing any temporary files/results\n\nUsing data in your documentation\n--------------------------------\n\nSome parts of the documentation require real-world data to be useful.\nFor example, you may need more than just a toy example to demonstrate the benefits of one model over another.\nThis subsection provides some tips for including data in your documentation.\n\nIf possible, use data available via Gensim's\n`downloader API <https://radimrehurek.com/gensim/gensim_numfocus/auto_examples/010_tutorials/run_downloader_api.html>`__.\nThis will reduce the risk of your documentation becoming obsolete because required data is no longer available.\n\nUse the smallest possible dataset: avoid making people unnecessarily load large datasets and models.\nThis will make your documentation faster to run and easier for people to use (they can modify your examples and re-run them quickly).\n\nFinalizing your contribution\n----------------------------\n\nFirst, get Sphinx Gallery to build your documentation::\n\n    make -C docs/src html\n\nThis can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.\nOnce this completes successfully, open ``docs/auto_examples/index.html`` in your browser.\nYou should see your new tutorial or guide in the gallery.\n\nOnce your documentation script is working correctly, it's time to add it to the git repository::\n\n    git add docs/src/gallery/tutorials/run_example.py\n    git add docs/src/auto_examples/tutorials/run_example.{py,py.md5,rst,ipynb}\n    git add docs/src/auto_examples/howtos/sg_execution_times.rst\n    git commit -m \"enter a helpful commit message here\"\n    git push origin branchname\n\n.. Note::\n  You may be wondering what all those other files are.\n  Sphinx Gallery puts a copy of your Python script in ``auto_examples/tutorials``.\n  The .md5 contains MD5 hash of the script to enable easy detection of modifications.\n  Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.\n  Finally, ``sg_execution_times.rst`` contains the time taken to run each example.\n\nFinally, make a PR on `github <https://github.com/RaRe-Technologies/gensim>`__.\nOne of our friendly maintainers will review it, make suggestions, and eventually merge it.\nYour documentation will then appear in the gallery alongside the rest of the example.\nAt that stage, give yourself a pat on the back: you're done!\n\n"
+        "Correctness\n-----------\n\nIncorrect documentation can be worse than no documentation at all.\nTake the following steps to ensure correctness:\n\n- Run Python's doctest module on your docstrings\n- Run your documentation scripts from scratch, removing any temporary files/results\n\nUsing data in your documentation\n--------------------------------\n\nSome parts of the documentation require real-world data to be useful.\nFor example, you may need more than just a toy example to demonstrate the benefits of one model over another.\nThis subsection provides some tips for including data in your documentation.\n\nIf possible, use data available via Gensim's\n`downloader API <https://radimrehurek.com/gensim/gensim_numfocus/auto_examples/010_tutorials/run_downloader_api.html>`__.\nThis will reduce the risk of your documentation becoming obsolete because required data is no longer available.\n\nUse the smallest possible dataset: avoid making people unnecessarily load large datasets and models.\nThis will make your documentation faster to run and easier for people to use (they can modify your examples and re-run them quickly).\n\nFinalizing your contribution\n----------------------------\n\nFirst, get Sphinx Gallery to build your documentation::\n\n    make --directory docs/src html\n\nThis can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.\nOnce this completes successfully, open ``docs/auto_examples/index.html`` in your browser.\nYou should see your new tutorial or guide in the gallery.\n\nOnce your documentation script is working correctly, it's time to add it to the git repository::\n\n    git add docs/src/gallery/tutorials/run_example.py\n    git add docs/src/auto_examples/tutorials/run_example.{py,py.md5,rst,ipynb}\n    git add docs/src/auto_examples/howtos/sg_execution_times.rst\n    git commit -m \"enter a helpful commit message here\"\n    git push origin branchname\n\n.. Note::\n  You may be wondering what all those other files are.\n  Sphinx Gallery puts a copy of your Python script in ``auto_examples/tutorials``.\n  The .md5 contains MD5 hash of the script to enable easy detection of modifications.\n  Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.\n  Finally, ``sg_execution_times.rst`` contains the time taken to run each example.\n\nFinally, open a PR at `github <https://github.com/RaRe-Technologies/gensim>`__.\nOne of our friendly maintainers will review it, make suggestions, and eventually merge it.\nYour documentation will then appear in the `gallery <https://radimrehurek.com/gensim/auto_examples/index.html>`__,\nalongside the rest of the examples. Thanks a lot!\n\n"
       ]
     }
   ],
@@ -81,7 +81,7 @@
       "name": "python",
       "nbconvert_exporter": "python",
       "pygments_lexer": "ipython3",
-      "version": "3.7.3"
+      "version": "3.6.5"
     }
   },
   "nbformat": 4,

docs/src/auto_examples/howtos/run_doc.py

@@ -155,7 +155,7 @@
 #
 # First, get Sphinx Gallery to build your documentation::
 #
-#     make -C docs/src html
+#     make --directory docs/src html
 #
 # This can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.
 # Once this completes successfully, open ``docs/auto_examples/index.html`` in your browser.
@@ -176,7 +176,7 @@
 #   Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.
 #   Finally, ``sg_execution_times.rst`` contains the time taken to run each example.
 #
-# Finally, make a PR on `github <https://github.com/RaRe-Technologies/gensim>`__.
+# Finally, open a PR at `github <https://github.com/RaRe-Technologies/gensim>`__.
 # One of our friendly maintainers will review it, make suggestions, and eventually merge it.
-# Your documentation will then appear in the gallery alongside the rest of the example.
-# At that stage, give yourself a pat on the back: you're done!
+# Your documentation will then appear in the `gallery <https://radimrehurek.com/gensim/auto_examples/index.html>`__,
+# alongside the rest of the examples. Thanks a lot!

docs/src/auto_examples/howtos/run_doc.py.md5

@@ -1 +1 @@
-512a76ce743dd12482d21784a76b60fe
+96cefb1417d54ac8010e38cc739d5ff1

docs/src/auto_examples/howtos/run_doc.rst

@@ -1,10 +1,12 @@
-.. note::
-    :class: sphx-glr-download-link-note
+.. only:: html
+
+    .. note::
+        :class: sphx-glr-download-link-note
 
-    Click :ref:`here <sphx_glr_download_auto_examples_howtos_run_doc.py>` to download the full example code
-.. rst-class:: sphx-glr-example-title
+        Click :ref:`here <sphx_glr_download_auto_examples_howtos_run_doc.py>`     to download the full example code
+    .. rst-class:: sphx-glr-example-title
 
-.. _sphx_glr_auto_examples_howtos_run_doc.py:
+    .. _sphx_glr_auto_examples_howtos_run_doc.py:
 
 
 How to Author Gensim Documentation
@@ -78,6 +80,15 @@ At the very top, you need a docstring describing what your script does.
 
 
 
+.. rst-class:: sphx-glr-script-out
+
+ Out:
+
+ .. code-block:: none
+
+
+    '\nTitle\n=====\n\nBrief description.\n'
+
 
 
 The title is what will show up in the gallery.
@@ -167,7 +178,7 @@ Finalizing your contribution
 
 First, get Sphinx Gallery to build your documentation::
 
-    make -C docs/src html
+    make --directory docs/src html
 
 This can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.
 Once this completes successfully, open ``docs/auto_examples/index.html`` in your browser.
@@ -188,17 +199,17 @@ Once your documentation script is working correctly, it's time to add it to the
   Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.
   Finally, ``sg_execution_times.rst`` contains the time taken to run each example.
 
-Finally, make a PR on `github <https://github.com/RaRe-Technologies/gensim>`__.
+Finally, open a PR at `github <https://github.com/RaRe-Technologies/gensim>`__.
 One of our friendly maintainers will review it, make suggestions, and eventually merge it.
-Your documentation will then appear in the gallery alongside the rest of the example.
-At that stage, give yourself a pat on the back: you're done!
+Your documentation will then appear in the `gallery <https://radimrehurek.com/gensim/auto_examples/index.html>`__,
+alongside the rest of the examples. Thanks a lot!
 
 
 .. rst-class:: sphx-glr-timing
 
-   **Total running time of the script:** ( 0 minutes  1.226 seconds)
+   **Total running time of the script:** ( 0 minutes  0.171 seconds)
 
-**Estimated memory usage:**  9 MB
+**Estimated memory usage:**  6 MB
 
 
 .. _sphx_glr_download_auto_examples_howtos_run_doc.py:
@@ -211,13 +222,13 @@ At that stage, give yourself a pat on the back: you're done!
 
 
 
-  .. container:: sphx-glr-download
+  .. container:: sphx-glr-download sphx-glr-download-python
 
      :download:`Download Python source code: run_doc.py <run_doc.py>`
 
 
 
-  .. container:: sphx-glr-download
+  .. container:: sphx-glr-download sphx-glr-download-jupyter
 
      :download:`Download Jupyter notebook: run_doc.ipynb <run_doc.ipynb>`
 

docs/src/auto_examples/howtos/sg_execution_times.rst

@@ -5,9 +5,14 @@
 
 Computation times
 =================
-**00:01.226** total execution time for **auto_examples_howtos** files:
+**00:00.171** total execution time for **auto_examples_howtos** files:
 
-- **00:01.226**: :ref:`sphx_glr_auto_examples_howtos_run_doc.py` (``run_doc.py``)
-- **00:00.000**: :ref:`sphx_glr_auto_examples_howtos_run_compare_lda.py` (``run_compare_lda.py``)
-- **00:00.000**: :ref:`sphx_glr_auto_examples_howtos_run_doc2vec_imdb.py` (``run_doc2vec_imdb.py``)
-- **00:00.000**: :ref:`sphx_glr_auto_examples_howtos_run_downloader_api.py` (``run_downloader_api.py``)
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_doc.py` (``run_doc.py``)                       | 00:00.171 | 6.1 MB |
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_compare_lda.py` (``run_compare_lda.py``)       | 00:00.000 | 0.0 MB |
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_doc2vec_imdb.py` (``run_doc2vec_imdb.py``)     | 00:00.000 | 0.0 MB |
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_downloader_api.py` (``run_downloader_api.py``) | 00:00.000 | 0.0 MB |
++----------------------------------------------------------------------------------------+-----------+--------+

docs/src/auto_examples/index.rst

@@ -188,6 +188,27 @@ Learning-oriented lessons that introduce a particular gensim feature, e.g. a mod
 
    /auto_examples/tutorials/run_fasttext
 
+.. raw:: html
+
+    <div class="sphx-glr-thumbcontainer" tooltip="Introduces Gensim&#x27;s EnsembleLda model">
+
+.. only:: html
+
+ .. figure:: /auto_examples/tutorials/images/thumb/sphx_glr_run_ensemblelda_thumb.png
+     :alt: Ensemble LDA
+
+     :ref:`sphx_glr_auto_examples_tutorials_run_ensemblelda.py`
+
+.. raw:: html
+
+    </div>
+
+
+.. toctree::
+   :hidden:
+
+   /auto_examples/tutorials/run_ensemblelda
+
 .. raw:: html
 
     <div class="sphx-glr-thumbcontainer" tooltip="Introduces the Annoy library for similarity queries on top of vectors learned by Word2Vec.">