Merge remote-tracking branch 'upstream/develop' into levenshtein-soft…

…cossim

.circleci/config.yml

@@ -30,7 +30,7 @@ jobs:
           name: Build documentation
           command: |
             source venv/bin/activate
-            tox -e docs -vv
+            tox -e compile,docs -vv
 
       - store_artifacts:
           path:  docs/src/_build

.gitignore

@@ -7,6 +7,8 @@
 *.o
 *.so
 *.pyc
+*.pyo
+*.pyd
 
 # Packages #
 ############

.travis.yml

@@ -13,7 +13,10 @@ language: python
 matrix:
   include:
     - python: '2.7'
-      env: TOXENV="flake8"
+      env: TOXENV="flake8,flake8-docs"
+
+    - python: '3.6'
+      env: TOXENV="flake8,flake8-docs"
 
     - python: '2.7'
       env: TOXENV="py27-linux"
@@ -24,5 +27,13 @@ matrix:
     - python: '3.6'
       env: TOXENV="py36-linux"
 
+    - python: '3.7'
+      env:
+        - TOXENV="py37-linux"
+        - BOTO_CONFIG="/dev/null"
+      dist: xenial
+      sudo: true
+
+
 install: pip install tox
 script: tox -vv

CHANGELOG.md

@@ -1,5 +1,107 @@
 Changes
 ===========
+## 3.6.0, 2018-09-20
+
+### :star2: New features
+* File-based training for `*2Vec` models (__[@persiyanov](https://github.com/persiyanov)__, [#2127](https://github.com/RaRe-Technologies/gensim/pull/2127) & [#2078](https://github.com/RaRe-Technologies/gensim/pull/2078) & [#2048](https://github.com/RaRe-Technologies/gensim/pull/2048))
+
+  New training mode for `*2Vec` models (word2vec, doc2vec, fasttext) that allows model training to scale linearly with the number of cores (full GIL elimination). The result of our Google Summer of Code 2018 project by Dmitry Persiyanov.
+
+  **Benchmark**
+  - Dataset: `full English Wikipedia`
+  - Cloud: `GCE`
+  - CPU: `Intel(R) Xeon(R) CPU @ 2.30GHz 32 cores`
+  - BLAS: `MKL`
+
+
+  | Model | Queue-based version [sec] | File-based version [sec] | speed up | Accuracy (queue-based) | Accuracy (file-based) |
+  |-------|------------|--------------------|----------|----------------|-----------------------|
+  | Word2Vec | 9230 | **2437** | **3.79x** | 0.754 (± 0.003) | 0.750 (± 0.001) |
+  | Doc2Vec | 18264 | **2889** | **6.32x** | 0.721 (± 0.002) | 0.683 (± 0.003) |
+  | FastText | 16361 | **10625** | **1.54x** | 0.642 (± 0.002) | 0.660 (± 0.001) |
+
+  Usage:
+
+  ```python
+  import gensim.downloader as api
+  from multiprocessing import cpu_count
+  from gensim.utils import save_as_line_sentence
+  from gensim.test.utils import get_tmpfile
+  from gensim.models import Word2Vec, Doc2Vec, FastText
+
+
+  # Convert any corpus to the needed format: 1 document per line, words delimited by " "
+  corpus = api.load("text8")
+  corpus_fname = get_tmpfile("text8-file-sentence.txt")
+  save_as_line_sentence(corpus, corpus_fname)
+
+  # Choose num of cores that you want to use (let's use all, models scale linearly now!)
+  num_cores = cpu_count()
+
+  # Train models using all cores
+  w2v_model = Word2Vec(corpus_file=corpus_fname, workers=num_cores)
+  d2v_model = Doc2Vec(corpus_file=corpus_fname, workers=num_cores)
+  ft_model = FastText(corpus_file=corpus_fname, workers=num_cores)
+
+  ```
+  [Read notebook tutorial with full description.](https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/Any2Vec_Filebased.ipynb)
+
+
+### :+1: Improvements
+
+* Add scikit-learn wrapper for `FastText` (__[@mcemilg](https://github.com/mcemilg)__, [#2178](https://github.com/RaRe-Technologies/gensim/pull/2178))
+* Add multiprocessing support for `BM25` (__[@Shiki-H](https://github.com/Shiki-H)__, [#2146](https://github.com/RaRe-Technologies/gensim/pull/2146))
+* Add `name_only` option for downloader api (__[@aneesh-joshi](https://github.com/aneesh-joshi)__, [#2143](https://github.com/RaRe-Technologies/gensim/pull/2143))
+* Make `word2vec2tensor` script compatible with `python3` (__[@vsocrates](https://github.com/vsocrates)__, [#2147](https://github.com/RaRe-Technologies/gensim/pull/2147))
+* Add custom filter for `Wikicorpus` (__[@mattilyra](https://github.com/mattilyra)__, [#2089](https://github.com/RaRe-Technologies/gensim/pull/2089))
+* Make `similarity_matrix` support non-contiguous dictionaries (__[@Witiko](https://github.com/Witiko)__, [#2047](https://github.com/RaRe-Technologies/gensim/pull/2047))
+
+
+### :red_circle: Bug fixes
+
+* Fix memory consumption in `AuthorTopicModel` (__[@philipphager](https://github.com/philipphager)__, [#2122](https://github.com/RaRe-Technologies/gensim/pull/2122))
+* Correctly process empty documents in `AuthorTopicModel` (__[@probinso](https://github.com/probinso)__, [#2133](https://github.com/RaRe-Technologies/gensim/pull/2133))
+* Fix ZeroDivisionError `keywords` issue with short input (__[@LShostenko](https://github.com/LShostenko)__, [#2154](https://github.com/RaRe-Technologies/gensim/pull/2154))
+* Fix `min_count` handling in phrases detection using `npmi_scorer` (__[@lopusz](https://github.com/lopusz)__, [#2072](https://github.com/RaRe-Technologies/gensim/pull/2072))
+* Remove duplicate count from `Phraser` log message (__[@robguinness](https://github.com/robguinness)__, [#2151](https://github.com/RaRe-Technologies/gensim/pull/2151))
+* Replace `np.integer` -> `np.int` in `AuthorTopicModel` (__[@menshikh-iv](https://github.com/menshikh-iv)__, [#2145](https://github.com/RaRe-Technologies/gensim/pull/2145))
+
+
+### :books: Tutorial and doc improvements
+
+* Update docstring with new analogy evaluation method (__[@akutuzov](https://github.com/akutuzov)__, [#2130](https://github.com/RaRe-Technologies/gensim/pull/2130))
+* Improve `prune_at` parameter description for `gensim.corpora.Dictionary` (__[@yxonic](https://github.com/yxonic)__, [#2128](https://github.com/RaRe-Technologies/gensim/pull/2128))
+* Fix `default` -> `auto` prior parameter in documentation for lda-related models (__[@Laubeee](https://github.com/Laubeee)__, [#2156](https://github.com/RaRe-Technologies/gensim/pull/2156))
+* Use heading instead of bold style in `gensim.models.translation_matrix` (__[@nzw0301](https://github.com/nzw0301)__, [#2164](https://github.com/RaRe-Technologies/gensim/pull/2164))
+* Fix quote of vocabulary from `gensim.models.Word2Vec` (__[@nzw0301](https://github.com/nzw0301)__, [#2161](https://github.com/RaRe-Technologies/gensim/pull/2161))
+* Replace deprecated parameters with new in docstring of `gensim.models.Doc2Vec` (__[@xuhdev](https://github.com/xuhdev)__, [#2165](https://github.com/RaRe-Technologies/gensim/pull/2165))
+* Fix formula in Mallet documentation (__[@Laubeee](https://github.com/Laubeee)__, [#2186](https://github.com/RaRe-Technologies/gensim/pull/2186))
+* Fix minor semantic issue in docs for `Phrases` (__[@RunHorst](https://github.com/RunHorst)__, [#2148](https://github.com/RaRe-Technologies/gensim/pull/2148))
+* Fix typo in documentation (__[@KenjiOhtsuka](https://github.com/KenjiOhtsuka)__, [#2157](https://github.com/RaRe-Technologies/gensim/pull/2157))
+* Additional documentation fixes (__[@piskvorky](https://github.com/piskvorky)__, [#2121](https://github.com/RaRe-Technologies/gensim/pull/2121))
+
+### :warning: Deprecations (will be removed in the next major release)
+
+* Remove
+    - `gensim.models.wrappers.fasttext` (obsoleted by the new native `gensim.models.fasttext` implementation)
+    - `gensim.examples`
+    - `gensim.nosy`
+    - `gensim.scripts.word2vec_standalone`
+    - `gensim.scripts.make_wiki_lemma`
+    - `gensim.scripts.make_wiki_online`
+    - `gensim.scripts.make_wiki_online_lemma`
+    - `gensim.scripts.make_wiki_online_nodebug`
+    - `gensim.scripts.make_wiki` (all of these obsoleted by the new native  `gensim.scripts.segment_wiki` implementation)
+    - "deprecated" functions and attributes
+
+* Move
+    - `gensim.scripts.make_wikicorpus` ➡ `gensim.scripts.make_wiki.py`
+    - `gensim.summarization` ➡ `gensim.models.summarization`
+    - `gensim.topic_coherence` ➡ `gensim.models._coherence`
+    - `gensim.utils` ➡ `gensim.utils.utils` (old imports will continue to work)
+    - `gensim.parsing.*` ➡ `gensim.utils.text_utils`
+
+
 ## 3.5.0, 2018-07-06
 
 This release comprises a glorious 38 pull requests from 28 contributors. Most of the effort went into improving the documentation—hence the release code name "Docs 💬"!
@@ -202,7 +304,7 @@ Apart from the **massive overhaul of all Gensim documentation** (including docst
     - `gensim.parsing.*` ➡ `gensim.utils.text_utils`
 
 
-## 3.3.0, 2018-01-02
+## 3.3.0, 2018-02-02
 
 :star2: New features:
 * Re-designed all "*2vec" implementations (__[@manneshiva](https://github.com/manneshiva)__, [#1777](https://github.com/RaRe-Technologies/gensim/pull/1777))

MANIFEST.in

@@ -4,14 +4,29 @@ include CHANGELOG.md
 include COPYING
 include COPYING.LESSER
 include ez_setup.py
+
 include gensim/models/voidptr.h
+include gensim/models/fast_line_sentence.h
+
 include gensim/models/word2vec_inner.c
 include gensim/models/word2vec_inner.pyx
 include gensim/models/word2vec_inner.pxd
+include gensim/models/word2vec_corpusfile.cpp
+include gensim/models/word2vec_corpusfile.pyx
+include gensim/models/word2vec_corpusfile.pxd
+
 include gensim/models/doc2vec_inner.c
 include gensim/models/doc2vec_inner.pyx
+include gensim/models/doc2vec_inner.pxd
+include gensim/models/doc2vec_corpusfile.cpp
+include gensim/models/doc2vec_corpusfile.pyx
+
 include gensim/models/fasttext_inner.c
 include gensim/models/fasttext_inner.pyx
+include gensim/models/fasttext_inner.pxd
+include gensim/models/fasttext_corpusfile.cpp
+include gensim/models/fasttext_corpusfile.pyx
+
 include gensim/models/_utils_any2vec.c
 include gensim/models/_utils_any2vec.pyx
 include gensim/corpora/_mmreader.c

README.md

@@ -119,29 +119,23 @@ Documentation
 Adopters
 --------
 
-
-
-| Name                                   | Logo                                                                                                                           | URL                                                                                              | Description                                                                                                                                                                                                           |
-|----------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|                                                                                                                         
-| RaRe Technologies                            | ![rare](docs/src/readme_images/rare.png) | [rare-technologies.com](http://rare-technologies.com)                                                           | Machine learning & NLP consulting and training. Creators and maintainers of Gensim. |
-| Mindseye                            | ![mindseye](docs/src/readme_images/mindseye.png) | [mindseye.com](http://www.mindseyesolutions.com/)                                                           | Similarities in legal documents                                                    | 
-| Talentpair                            | ![talent-pair](docs/src/readme_images/talent-pair.png) | [talentpair.com](http://talentpair.com)                                                           | Data science driving high-touch recruiting                                                    | 
-| Tailwind          | ![tailwind](docs/src/readme_images/tailwind.png)| [Tailwindapp.com](https://www.tailwindapp.com/)| Post interesting and relevant content to Pinterest              |
-| Issuu          | ![issuu](docs/src/readme_images/issuu.png) | [Issuu.com](https://issuu.com/)| Gensim’s LDA module lies at the very core of the analysis we perform on each uploaded publication to figure out what it’s all about.
-| Sports Authority        |               ![sports-authority](docs/src/readme_images/sports-authority.png)               | [sportsauthority.com](https://en.wikipedia.org/wiki/Sports_Authority)| Text mining of customer surveys and social media sources |
-| Search Metrics        | ![search-metrics](docs/src/readme_images/search-metrics.png) | [searchmetrics.com](http://www.searchmetrics.com/)| Gensim word2vec used for entity disambiguation in Search Engine Optimisation
-| Cisco Security        | ![cisco](docs/src/readme_images/cisco.png) | [cisco.com](http://www.cisco.com/c/en/us/products/security/index.html)|  Large-scale fraud detection
-|  12K Research         | ![12k](docs/src/readme_images/12k.png)| [12k.co](https://12k.co/)|   Document similarity analysis on media articles
-|  National Institutes of Health         | ![nih](docs/src/readme_images/nih.png) | [github/NIHOPA](https://github.com/NIHOPA/pipeline_word2vec)|   Processing grants and publications with word2vec
-|  Codeq LLC         | ![codeq](docs/src/readme_images/codeq.png) | [codeq.com](https://codeq.com)|   Document classification with word2vec
-| Mass Cognition     | ![mass-cognition](docs/src/readme_images/mass-cognition.png) | [masscognition.com](http://www.masscognition.com/)                                  | Topic analysis service for consumer text data and general text data |
-| Stillwater Supercomputing     | ![stillwater](docs/src/readme_images/stillwater.png) | [stillwater-sc.com](http://www.stillwater-sc.com/)                                  | Document comprehension and association with word2vec |
-| Channel 4     | ![channel4](docs/src/readme_images/channel4.png) | [channel4.com](http://www.channel4.com/)                                  | Recommendation engine |
-| Amazon     |  ![amazon](docs/src/readme_images/amazon.png) | [amazon.com](http://www.amazon.com/)                                  |  Document similarity|
-| SiteGround Hosting     |  ![siteground](docs/src/readme_images/siteground.png) | [siteground.com](https://www.siteground.com/)                                  | An ensemble search engine which uses different embeddings models and similarities, including word2vec, WMD, and LDA. |
-| Juju  | ![juju](docs/src/readme_images/juju.png) | [www.juju.com](http://www.juju.com/) | Provide non-obvious related job suggestions. |
-| NLPub | ![nlpub](docs/src/readme_images/nlpub.png) | [nlpub.org](https://nlpub.org/) | Distributional semantic models including word2vec. |
-|Capital One | ![capitalone](docs/src/readme_images/capitalone.png) | [www.capitalone.com](https://www.capitalone.com/) | Topic modeling for customer complaints exploration. |
+| Company | Logo | Industry | Use of Gensim |
+|---------|------|----------|---------------|                          
+| [RARE Technologies](http://rare-technologies.com) | ![rare](docs/src/readme_images/rare.png) | ML & NLP consulting | Creators of Gensim – this is us! |
+| [Amazon](http://www.amazon.com/) |  ![amazon](docs/src/readme_images/amazon.png) | Retail |  Document similarity. |
+| [National Institutes of Health](https://github.com/NIHOPA/pipeline_word2vec) | ![nih](docs/src/readme_images/nih.png) | Health | Processing grants and publications with word2vec. |
+| [Cisco Security](http://www.cisco.com/c/en/us/products/security/index.html) | ![cisco](docs/src/readme_images/cisco.png) | Security |  Large-scale fraud detection. |
+| [Mindseye](http://www.mindseyesolutions.com/) | ![mindseye](docs/src/readme_images/mindseye.png) | Legal | Similarities in legal documents. |
+| [Channel 4](http://www.channel4.com/) | ![channel4](docs/src/readme_images/channel4.png) | Media | Recommendation engine. |
+| [Talentpair](http://talentpair.com) | ![talent-pair](docs/src/readme_images/talent-pair.png) | HR | Candidate matching in high-touch recruiting. |
+| [Juju](http://www.juju.com/)  | ![juju](docs/src/readme_images/juju.png) | HR | Provide non-obvious related job suggestions. |
+| [Tailwind](https://www.tailwindapp.com/) | ![tailwind](docs/src/readme_images/tailwind.png) | Media | Post interesting and relevant content to Pinterest. |
+| [Issuu](https://issuu.com/) | ![issuu](docs/src/readme_images/issuu.png) | Media | Gensim's LDA module lies at the very core of the analysis we perform on each uploaded publication to figure out what it's all about. |
+| [Search Metrics](http://www.searchmetrics.com/) | ![search-metrics](docs/src/readme_images/search-metrics.png) | Content Marketing | Gensim word2vec used for entity disambiguation in Search Engine Optimisation. |
+| [12K Research](https://12k.co/) | ![12k](docs/src/readme_images/12k.png)| Media |   Document similarity analysis on media articles. |
+| [Stillwater Supercomputing](http://www.stillwater-sc.com/) | ![stillwater](docs/src/readme_images/stillwater.png) | Hardware | Document comprehension and association with word2vec. |
+| [SiteGround](https://www.siteground.com/) |  ![siteground](docs/src/readme_images/siteground.png) | Web hosting | An ensemble search engine which uses different embeddings models and similarities, including word2vec, WMD, and LDA. |
+| [Capital One](https://www.capitalone.com/) | ![capitalone](docs/src/readme_images/capitalone.png) | Finance | Topic modeling for customer complaints exploration. |
 
 -------
 

appveyor.yml

@@ -28,6 +28,11 @@ environment:
       PYTHON_ARCH: "64"
       TOXENV: "py36-win"
 
+    - PYTHON: "C:\\Python37-x64"
+      PYTHON_VERSION: "3.7.0"
+      PYTHON_ARCH: "64"
+      TOXENV: "py37-win"
+
 init:
   - "ECHO %PYTHON% %PYTHON_VERSION% %PYTHON_ARCH%"
   - "ECHO \"%APPVEYOR_SCHEDULED_BUILD%\""