examples/phrase-extraction-with-manatee.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial: Phrase Extraction with Manatee\n",
    "\n",
    "This tutorial shows how to extract phrases from corpora by explicit rules (grammar patterns) and learn their distributed representations with POcs2Vec."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequistities\n",
    "\n",
    "You will need <a href=\"https://nlp.fi.muni.cz/trac/noske\">Manatee</a> corpus manager with compiled SUSANNE corpus.  As for knowledge, you should be happy with basics of <a href=\"https://www.sketchengine.co.uk/documentation/corpus-querying/\">CQL</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from itertools import islice\n",
    "\n",
    "from diphra.lex import Lexicon\n",
    "from diphra.pocs import POcs, VerticalPOcs, HDF5POcs\n",
    "from diphra.models import pocs2vec\n",
    "from diphra.extractors import ManateeExtractor\n",
    "from diphra.extractors.manatee_extractor import match_cql"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Pattern Matching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "matches = match_cql(\n",
    "    corpus='susanne',\n",
    "    struct='p',\n",
    "    lex_attr='lemma',\n",
    "    cql_attr='tag',\n",
    "    query='1:\"VV.*\" 2:\"RP\" \"AT.*\"? 3:\"N.*\"',\n",
    "    template='to {1.lemma} {2.word} {3.word} | {3.tag}'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to take over bank | NNJ1c\n",
      "to cover up places | NN2\n",
      "to carry out obligations | NN2\n",
      "to take up matter | NN1n\n",
      "to take over Johnston | NP1s\n",
      "to point out state | NNL1n\n",
      "to bring on state | NNL1n\n",
      "to report out Tuesday | NPD1\n",
      "to toss in towel | NN1c\n",
      "to carry on battle | NN1n\n"
     ]
    }
   ],
   "source": [
    "for struct_id, label2pos, label2lex_id, text in islice(matches, 10):\n",
    "    print text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, `match_cql` cares only about tokens that are labeled &mdash; there is no way how to get an unlabeled token into the template (though such feature is planned as it would have nice use cases).\n",
    "\n",
    "Interpretation of the quadruples yielded by `match_cql` is:\n",
    "\n",
    " - a sentence ID (numbered from 0)\n",
    " - a dict mapping labels to in-sentence positions (numbered from 0)\n",
    " - a dict mapping labels to lemma IDs\n",
    " - a filled in template"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ManateeExtractor *class*\n",
    "\n",
    "ManateeExtractor manages the process of extracting many different kinds of phrases from huge corpora.  To achieve scalability (currently mainly memory-friendliness), it involves things like *external sort* or *bounded counting*.  In the end, you end up with a large pocs dataset that you can put into POcs2Vec for machine interpretation.  \n",
    "\n",
    "The motivation behind this extractor was: *knowledge of CQL should be enough to experiment with distributional semantics of phrases*.\n",
    "\n",
    "### Phrase Types\n",
    "\n",
    "<img src=\"dep-tree.png\" style=\"float: right;margin-left: 20px;\" />\n",
    "\n",
    "A phrase type is **a tree of word categories**.  It is crudely based on the concept of *dependency syntactic trees*.  Its main purpose is to augment phrases with some structure, so that we can decompose phrases to heads and collocates.  Root is the head, subtrees are collocates.\n",
    "\n",
    "The phrase type depicted on the right covers phrases like TO&nbsp;TAKE&nbsp;THE&nbsp;BULL&nbsp;BY&nbsp;THE&nbsp;HORNS or TO&nbsp;TAKE&nbsp;THE&nbsp;WORLD&nbsp;BY&nbsp;STORM (determiners can be ignored in phrase types).\n",
    "\n",
    "Each phrase in ManateeExtractor is eventually identified by:\n",
    "\n",
    " - a phrase type tree\n",
    " - an assignment of lexical IDs (usually lemmas) to nodes of the tree\n",
    "\n",
    "An important implication is that the textual representation can be ambiguous, i.e. DUCK with phrase type `(noun, )` is something different than DUCK with phrase type `(verb, )`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workspace\n",
    "\n",
    "ManateeExtractor uses disk space for many intermediate results.  Firstly, you need to make a workspace &mdash; a directory with a configuration file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ManateeExtractor.make(\n",
    "    directory=\"./extraction-directory\", corpus=\"susanne\",\n",
    "    struct=\"p\", lex_attr=\"lemma\", cql_attr=\"tag\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize an extractor just by providing a workspace directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "extractor = ManateeExtractor('./extraction-directory')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrase Grammar\n",
    "\n",
    "You have to decide which phrase types you want and write CQL queries for each of them.  ManateeExtractor will handle everything else.  You only give it a dictionary object `grammar`, whose keys are phrase type trees in *Polish Notation* (technically nested tuples of strings).\n",
    "\n",
    "Each item in the `grammar` must contain a list of `patterns`.  Each pattern consist of a `query` and a `template`, which are processed by `match_cql` function which you are already familiar with.  However, we need to interconnect labeled tokens with nodes of the phrase type tree.  The convention is very simple &mdash; a&nbsp;token labeled as `i` corresponds to `i`-th node of the phrase type tree in pre-order traversal.\n",
    "\n",
    "Also, you can have additional labeled tokens and use them in templates only (like determiners) &mdash; ManateeExtractor will automatically select the most common textual form for each phrase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grammar = {\n",
    "    (('verb', 'adv'), 'noun'): dict(\n",
    "        # The best way to start is to list several\n",
    "        # concrete examples of what we eventually want\n",
    "        # (diversity encouraged).\n",
    "        examples=[\n",
    "            'to pick up speed',\n",
    "            'to get off the ground', \n",
    "            'to take time out',\n",
    "            'to keep an eye out'\n",
    "        ],\n",
    "        # Now craft some patterns that will cover\n",
    "        # the defined examples.\n",
    "        patterns=[\n",
    "            dict(  # verb adv noun\n",
    "                query='1:\"VV.*\" 2:\"RP\" 3:\"N.*\"',\n",
    "                template='to {1.lemma} {2.word} {3.word}'\n",
    "            ),\n",
    "            dict(  # verb adv det noun\n",
    "                query='1:\"VV.*\" 2:\"RP\" 4:\"AT.*\" 3:\"N.*\"',\n",
    "                template='to {1.lemma} {2.word} {4.word} {3.word}'\n",
    "            ),\n",
    "            dict(  # verb noun adv\n",
    "                query='1:\"VV.*\" 3:\"N.*\" 2:\"RP\"',\n",
    "                template='to {1.lemma} {3.word} {2.word}'\n",
    "            ),\n",
    "            dict(  # verb det noun adv\n",
    "                query='1:\"VV.*\" 4:\"AT.*\" 3:\"N.*\" 2:\"RP\"',\n",
    "                template='to {1.lemma} {4.word} {3.word} {2.word}'\n",
    "            ),\n",
    "        ]\n",
    "    )\n",
    "    # ...\n",
    "    # ... analogically add more phrase types\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words are incorporated into the grammar in the exactly same way as phrases.  After all, a word is just a phrase of length 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "grammar.update({\n",
    "    ('det', ): dict(patterns=[dict(query='1:\"AT.*\"', template='{1.lemma}')]),\n",
    "    ('noun', ): dict(patterns=[dict(query='1:\"N.*\"', template='{1.lemma}')]),\n",
    "    ('adj', ): dict(patterns=[dict(query='1:\"J.*\"', template='{1.lemma}')]),\n",
    "    ('prep', ): dict(patterns=[dict(query='1:\"I.*\"', template='{1.lemma}')]),\n",
    "    ('verb', ): dict(patterns=[dict(query='1:\"V.*\"', template='to {1.lemma}')]),\n",
    "    # ...\n",
    "    # ... add all other word categories you wish to incorporate (preferably all)\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grammar Check\n",
    "\n",
    "Phrase grammars can be quite exhaustive and the extraction may take tens of hours for >1B corpora.  Before we run anything, we want to be absolutely sure that we've done a good job.  If there's a problem, we want to know where exactly it occurs.  So firstly, we put the grammar into `ManateeExtractor.check_grammar` function.  It will try to execute all the included CQL queries and print specified number of outputs for each of them.\n",
    "\n",
    "What one fool can mess up, one call to `check_grammar` should detect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('adj',) / #1\n",
      "\n",
      "\tgrand\n",
      "\trecent\n",
      "\tprimary\n",
      "\texecutive\n",
      "\toverall\n",
      "\tsuperior\n",
      "\n",
      "('prep',) / #1\n",
      "\n",
      "\tof\n",
      "\tin\n",
      "\tof\n",
      "\tof\n",
      "\tof\n",
      "\tfor\n",
      "\n",
      "('noun',) / #1\n",
      "\n",
      "\tFulton\n",
      "\tcounty\n",
      "\tjury\n",
      "\tFriday\n",
      "\tinvestigation\n",
      "\tAtlanta\n",
      "\n",
      "('det',) / #1\n",
      "\n",
      "\tthe\n",
      "\tan\n",
      "\tno\n",
      "\tthe\n",
      "\tthe\n",
      "\tthe\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #1\n",
      "\n",
      "\tto take over bank\n",
      "\tto cover up places\n",
      "\tto carry out obligations\n",
      "\tto bring on state\n",
      "\tto report out Tuesday\n",
      "\tto cut down expenses\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #2\n",
      "\n",
      "\tto take up the matter\n",
      "\tto take over the Johnston\n",
      "\tto point out the state\n",
      "\tto toss in the towel\n",
      "\tto carry on the battle\n",
      "\tto draw up a plan\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #3\n",
      "\n",
      "\tto take petitions out\n",
      "\tto move Cooke across\n",
      "\tto give time off\n",
      "\tto take time out\n",
      "\tto get Miller out\n",
      "\tto put coffee on\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #4\n",
      "\n",
      "\tto pass the bill on\n",
      "\tto keep the people in\n",
      "\tto keep the pressure on\n",
      "\tto get a day off\n",
      "\tto get the secrets off\n",
      "\tto shoot the bastards down\n",
      "\n",
      "('verb',) / #1\n",
      "\n",
      "\tto say\n",
      "\tto produce\n",
      "\tto take\n",
      "\tto say\n",
      "\tto have\n",
      "\tto deserve\n",
      "\n"
     ]
    }
   ],
   "source": [
    "extractor.check_grammar(grammar, nb_outputs=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phrase Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of phrase types: 6\n",
      "There are colliding phrase types (3):\n",
      "\t- ('noun',)\n",
      "\t- ('verb',)\n",
      "\t- (('verb', 'adv'), 'noun')\n",
      "Colliding phrase types will be overwritten.\n",
      "\n",
      "('adj',) / #1\n",
      "[[ Progress: 100%, Occurrences: 9,311 ]]\n",
      "\n",
      "('prep',) / #1\n",
      "[[ Progress: 100%, Occurrences: 16,488 ]]]\n",
      "\n",
      "('noun',) / #1\n",
      "[[ Progress: 100%, Occurrences: 35,415 ]]]\n",
      "\n",
      "('det',) / #1\n",
      "[[ Progress: 100%, Occurrences: 13,319 ]]]\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #1\n",
      "[[ Progress: 100%, Occurrences: 27 ]]\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #2\n",
      "[[ Progress: 100%, Occurrences: 52 ]]\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #3\n",
      "[[ Progress: 100%, Occurrences: 9 ]]\n",
      "\n",
      "(('verb', 'adv'), 'noun') / #4\n",
      "[[ Progress: 100%, Occurrences: 18 ]]\n",
      "\n",
      "('verb',) / #1\n",
      "[[ Progress: 100%, Occurrences: 23,523 ]]]\n",
      "\n",
      "Phrase extraction succesfully finished!\n"
     ]
    }
   ],
   "source": [
    "extractor.extract_phrases(grammar)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each phrase type is processed independently.  The statement really is equivallent to the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# for k, v in grammar.items():\n",
    "#     extractor.extract_phrases({k: v})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build Lexicon\n",
    "\n",
    "At this  point, we have a separate (and potentially huge) vocabulary for each phrase type.  What needs to be done is to **unify** it into a single lexicon and **restrict** its size &mdash; e.g. throw away phrases with frequency&nbsp;&lt;&nbsp;60 and/or keep only 1&nbsp;million most frequent items.\n",
    "\n",
    "Now... SUSANNE is a tiny, tiny corpus.  The configuration here is therefore a bit funny."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting phrases from raw vocabularies ...\n",
      "\tPhrase Type: ('prep',) \n",
      "\tPhrase Type: ('noun',) \n",
      "\tPhrase Type: (('verb', 'adv'), 'noun') \n",
      "\tPhrase Type: ('det',) \n",
      "\tPhrase Type: ('verb',) \n",
      "\tPhrase Type: ('adj',) \n",
      "\n",
      "Detecting vassals (coeff=0.75) ...\n",
      "[[ Progress: 100%, Phrases: 10,644, Vassals: 7 ]]]\n",
      "\n",
      "Persisting lexicon to disk ...\n"
     ]
    }
   ],
   "source": [
    "extractor.build_lexicon(min_count=1, max_items=int(3e5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\t3\t0\tthe\t9616\n",
      "1\t4\t41\tto be\t4875\n",
      "2\t0\t9\tof\t4668\n",
      "3\t3\t63\ta\t2951\n",
      "4\t0\t24\tin\t2723\n",
      "5\t4\t32\tto have\t1429\n",
      "6\t0\t51\tto\t1358\n",
      "7\t0\t39\tfor\t1067\n",
      "8\t1\t11\t-\t996\n",
      "9\t0\t179\twith\t895\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "head extraction-directory/lexicon.vert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8812\t2\t1582,702,3291\tto move the flights over\t1\n",
      "8813\t2\t1713,702,1805\tto look the setup over\t1\n",
      "8814\t2\t1812,702,182\tto hand the money over\t1\n",
      "8815\t2\t21,669,9047\tto take a carbine down\t1\n",
      "8816\t2\t2438,669,8948\tto ease the Winchester down\t1\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "grep -P \"\\t[0-9]+,[0-9]+,[0-9]+\\t\" extraction-directory/lexicon.vert |head -5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Columns in `lexicon.vert` have the following interpretation:\n",
    "\n",
    " - phrase ID\n",
    " - phrase type ID\n",
    " - comma-separated lex. attribute IDs (for phrase type tree nodes in pre-order)\n",
    " - textual representation\n",
    " - frequency\n",
    "\n",
    "The list of phrase types is pickled in `phrase_types.pickle` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iterate over Phrase Occurrences\n",
    "\n",
    "Now that we have a lexicon of bounded size, we can iterate over **unsorted** phrase occurrences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "of 0 (9,)\n",
      "in 1 (4,)\n",
      "of 1 (19,)\n",
      "of 1 (29,)\n",
      "of 1 (32,)\n",
      "for 1 (35,)\n"
     ]
    }
   ],
   "source": [
    "for phrase, sentence_i, positions in islice(extractor.iter_pocs(), 6):\n",
    "    print phrase, sentence_i, positions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... or better replacing textual reprs. with phrase IDs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2 0 (9,)\n",
      "4 1 (4,)\n",
      "2 1 (19,)\n",
      "2 1 (29,)\n",
      "2 1 (32,)\n",
      "7 1 (35,)\n"
     ]
    }
   ],
   "source": [
    "for phrase, sentence_i, positions in islice(extractor.iter_pocs(use_ids=True), 6):\n",
    "    print phrase, sentence_i, positions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sort Phrase Occurrences\n",
    "\n",
    "We yet need to sort the pocs.  To do that, we first persist them to disk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "POcs(func=lambda: extractor.iter_pocs(use_ids=True)).save_to_vertical(\"my_pocs.unsorted.vert.gz\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\t0\t9\n",
      "4\t1\t4\n",
      "2\t1\t19\n",
      "2\t1\t29\n",
      "2\t1\t32\n",
      "7\t1\t35\n",
      "4\t1\t38\n",
      "12\t2\t9\n",
      "2\t2\t19\n",
      "4\t2\t24\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "gzip: stdout: Broken pipe\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "zcat my_pocs.unsorted.vert.gz |head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and apply an external sort that is happpy with compressed files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sorting lines 0 - 20,000\n",
      "Sorting lines 20,000 - 40,000\n",
      "Sorting lines 40,000 - 60,000\n",
      "Sorting lines 60,000 - 80,000\n",
      "Sorting lines 80,000 - 100,000\n",
      "Number of chunks: 5\n",
      "Merging chunks 0 - 5\n"
     ]
    }
   ],
   "source": [
    "VerticalPOcs.pocs_sort(\n",
    "    inp_fn=\"my_pocs.unsorted.vert.gz\",\n",
    "    out_fn=\"my_pocs.sorted.vert.gz\",\n",
    "    line_chunk_size=int(2e4)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set `line_chunk_size` according to your RAM.  Here it is only 20k solely for the purpose of demonstrating the chunking mechanism on small data.  10&ndash;100m is more realistic for >1B corpora."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\t0\t0\n",
      "929\t0\t1\n",
      "137\t0\t2\n",
      "1298\t0\t3\n",
      "365\t0\t4\n",
      "18\t0\t5\n",
      "970\t0\t6\n",
      "15\t0\t7\n",
      "900\t0\t8\n",
      "2\t0\t9\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "gzip: stdout: Broken pipe\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "zcat my_pocs.sorted.vert.gz |head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile POcs\n",
    "\n",
    "This step is optional, though highly recommended -- it significantly speeds up POcs2Vec."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "HDF5POcs.build(\n",
    "    pocs=VerticalPOcs(\"my_pocs.sorted.vert.gz\"),\n",
    "    output_name=\"my_pocs.sorted.hdf5\",\n",
    "    max_len=3,  # max. number of nodes in a phrase type tree in your grammar\n",
    "    use_ids=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## POcs2Vec &mdash; Distributional Semantics of Phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ PROGRESS: 93.05% | 1,057,834 pocs/s | 2,353,660 tc/s | 155,883 downsampled/s ]]\n",
      "worker thread finished; awaiting finish of 3 more threads\n",
      "worker thread finished; awaiting finish of 2 more threads\n",
      "worker thread finished; awaiting finish of 1 more threads\n",
      "worker thread finished; awaiting finish of 0 more threads\n"
     ]
    }
   ],
   "source": [
    "my_pocs = HDF5POcs(\"my_pocs.sorted.hdf5\")\n",
    "\n",
    "pocs2vec(pocs=my_pocs, output_name=\"phrase_model\",\n",
    "         dim=32, nb_epochs=128, window=6, sample=int(1e3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lexicon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lexicon = Lexicon.load(\"extraction-directory/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lexicon.load_vecs(\"phrase_model.target.npy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "to awaken\n",
      "repetition\n",
      "to mess\n",
      "gregarious\n",
      "secret\n",
      "doubt\n",
      "encouragement\n",
      "visit\n",
      "uncertain\n",
      "stink\n",
      "to cheat\n"
     ]
    }
   ],
   "source": [
    "for p in lexicon.most_similar(lexicon[\"to have\"][0]):\n",
    "    print p.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}