NLTK.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to NLP with NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Natural Language Processing (NLP) is often taught at the academic level from the perspective of computational linguists. However, as data scientists, we have a richer view of the natural language world - unstructured data that by its very nature has latent information that is important to humans. NLP practioners have benefited from machine learning techniques to unlock meaning from large corpora, and in this class we’ll explore how to do that particularly with Python and with the Natural Language Toolkit (NLTK). \n",
    "\n",
    "NLTK is an excellent library for machine-learning based NLP, written in Python by experts from both academia and industry. Python allows you to create rich data applications rapidly, iterating on hypotheses. The combination of Python + NLTK means that you can easily add language-aware data products to your larger analytical workflows and applications. \n",
    "\n",
    "## Quick Overview of NLTK\n",
    "NLTK stands for the Natural Language Toolkit and is written by two eminent computational linguists, Steven Bird (Senior Research Associate of the LDC and professor at the University of Melbourne) and Ewan Klein (Professor of Linguistics at Edinburgh University). NTLK provides a combination of natural language corpora, lexical resources, and example grammars with language processing algorithms, methodologies and demonstrations for a very pythonic \"batteries included\" view of Natural Language Processing.   \n",
    "\n",
    "As such, NLTK is perfect for researh driven (hypothesis driven) workflows for agile data science. Its suite of libraries includes:\n",
    "\n",
    "- tokenization, stemming, and tagging\n",
    "- chunking and parsing\n",
    "- language modeling\n",
    "- classification and clustering\n",
    "- logical semantics\n",
    "\n",
    "NLTK is a useful pedagogical resource for learning NLP with Python and serves as a starting place for producing production grade code that requires natural language analysis. It is also important to understand what NLTK is _not_:\n",
    "\n",
    "- Production ready out of the box\n",
    "- Lightweight\n",
    "- Generally applicable\n",
    "- Magic\n",
    "\n",
    "NLTK provides a variety of tools that can be used to explore the linguistic domain but is not a lightweight dependency that can be easily included in other workflows, especially those that require unit and integration testing or other build processes. This stems from the fact that NLTK includes a lot of added code but also a rich and complete library of corpora that power the built-in algorithms. \n",
    "\n",
    "### The Good parts of NLTK\n",
    "\n",
    "- Preprocessing\n",
    "    - segmentation\n",
    "    - tokenization\n",
    "    - PoS tagging\n",
    "- Word level processing\n",
    "    - WordNet\n",
    "    - Lemmatization\n",
    "    - Stemming\n",
    "    - NGrams\n",
    "- Utilities\n",
    "    - Tree\n",
    "    - FreqDist\n",
    "    - ConditionalFreqDist\n",
    "    - Streaming CorpusReaders\n",
    "- Classification\n",
    "    - Maximum Entropy\n",
    "    - Naive Bayes\n",
    "    - Decision Tree\n",
    "- Chunking\n",
    "- Named Entity Recognition\n",
    "- Parsers Galore!\n",
    "\n",
    "### The Bad parts of NLTK\n",
    "\n",
    "- Syntactic Parsing\n",
    "\n",
    "    - No included grammar (not a black box)\n",
    "    - No Feature/Dependency Parsing\n",
    "    - No included feature grammar\n",
    "\n",
    "- The sem package\n",
    "    \n",
    "    - Toy only (lambda-calculus & first order logic)\n",
    "\n",
    "- Lots of extra stuff (heavyweight dependency)\n",
    "\n",
    "    - papers, chat programs, alignments, etc.\n",
    "\n",
    "Knowing the good and the bad parts will help you explore NLTK further - looking into the source code to extract the material you need, then moving that code to production. We will explore NLTK in more detail in the rest of this notebook. \n",
    "\n",
    "## Installing NLTK\n",
    "\n",
    "This notebook has a few dependencies, most of which can be installed via the python package manger - `pip`. \n",
    "\n",
    "1. Python 2.7 or later (anaconda is ok)\n",
    "2. NLTK\n",
    "3. The NLTK corpora \n",
    "4. The BeautifulSoup library\n",
    "5. The gensim libary\n",
    "\n",
    "Once you have Python and pip installed you can install NLTK as follows:\n",
    "\n",
    "    ~$ pip install nltk\n",
    "    \n",
    "    ~$ pip install matplotlib\n",
    "    \n",
    "    ~$ pip install beautifulsoup4\n",
    "    \n",
    "    ~$ pip install gensim\n",
    "\n",
    "Note that these will also install Numpy and Scipy if they aren't already installed. \n",
    "\n",
    "To download the corpora, open a python interperter:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will open up a window with which you can download the various corpora and models to a specified location. For now, go ahead and download it all as we will be exploring as much of NLTK as we can. Also take note of the `download_directory` - you're going to want to know where that is so you can get a detailed look at the corpora that's included. I usually export an enviornment variable to track this:\n",
    "\n",
    "    ~$ export NLTK_DATA=/path/to/nltk_data\n",
    "    \n",
    "_Take a moment to explore what is in this directory_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with Example Corpora\n",
    "\n",
    "NLTK ships with a variety of corpora, let's use a few of them to do some work. Get access to the text from _Moby Dick_ as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "moby = nltk.text.Text(nltk.corpus.gutenberg.words('melville-moby_dick.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nltk.text.Text` class is a wrapper around a sequence of simple (string) tokens - intended only for _the initial exploration of text_ usually via the Python REPL. It has the following methods:\n",
    "\n",
    "- common_contexts\n",
    "- concordance\n",
    "- collocations\n",
    "- count\n",
    "- plot\n",
    "- findall\n",
    "- index\n",
    "\n",
    "You shouldn't use this class in production level systems, but it is useful to explore (small) snippets of text in a meaningful fashion.\n",
    "\n",
    "The corcordance function performs a search for the given token and then also provides the surrounding context:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "moby.concordance(\"monstrous\", 55, lines=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given some context surrounding a word, we can discover similar words, e.g. words that that occur frequently in the same context and with a similar distribution: Distributional similarity: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print moby.similar(\"ahab\")\n",
    "austen = nltk.text.Text(nltk.corpus.gutenberg.words('austen-sense.txt'))\n",
    "print\n",
    "print austen.similar(\"monstrous\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, this takes a bit of time to build the index in memory, one of the reasons it's not suggested to use this class in production code. Now that we can do searching and similarity, find the common contexts of a set of words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "moby.common_contexts([\"ahab\", \"starbuck\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_your turn, go ahead and explore similar words and contexts - what does the common context mean?_\n",
    "\n",
    "NLTK also uses matplotlib and pylab to display graphs and charts that can show dispersions and frequency. This is especially interesting for the corpus of innagural addresses given by U.S. presidents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "inaugural = nltk.text.Text(nltk.corpus.inaugural.words())\n",
    "inaugural.dispersion_plot([\"citizens\", \"democracy\", \"freedom\", \"duties\", \"America\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To explore much of the built in corpus, use the following methods:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Lists the various corpora and CorpusReader classes in the nltk.corpus module\n",
    "for name in dir(nltk.corpus):\n",
    "    if name.islower() and not name.startswith('_'): print name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# For a specific corpus, list the fileids that are available:\n",
    "print nltk.corpus.shakespeare.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print nltk.corpus.gutenberg.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print nltk.corpus.stopwords.fileids()\n",
    "nltk.corpus.stopwords.words('english')\n",
    "import string\n",
    "print string.punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These corpora export several vital methods:\n",
    "\n",
    "- paras (iterate through each paragraph)\n",
    "- sents (iterate through each sentence)\n",
    "- words (iterate through each word)\n",
    "- raw   (get access to the raw text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "corpus = nltk.corpus.brown\n",
    "print corpus.paras()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print corpus.sents()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print corpus.words()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print corpus.raw()[:200] # Be careful!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Your turn! Explore some of the text in the available corpora_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequency Analyses\n",
    "\n",
    "In statistical machine learning approaches to NLP, the very first thing we need to do is count things - especially the unigrams that appear in the text and their relationships to each other. NLTK provides two very excellent classes to enable these frequency analyses:\n",
    "\n",
    "- `FreqDist`\n",
    "- `ConditionalFreqDist` \n",
    "\n",
    "And these two classes serve as the foundation for most of the probability and statistical analyses that we will conduct.\n",
    "\n",
    "First we will compute the following:\n",
    "\n",
    "- The count of words\n",
    "- The vocabulary (unique words)\n",
    "- The lexical diversity (the ratio of word count to vocabulary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reuters = nltk.corpus.reuters # Corpus of news articles\n",
    "counts  = nltk.FreqDist(reuters.words())\n",
    "vocab   = len(counts.keys())\n",
    "words   = sum(counts.values())\n",
    "lexdiv  = float(words) / float(vocab)\n",
    "\n",
    "print \"Corpus has %i types and %i tokens for a lexical diversity of %0.3f\" % (vocab, words, lexdiv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "counts.B()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print counts.most_common(40)  # The n most common tokens in the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print counts.max() # The most frequent token in the corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print counts.hapaxes()[0:10]  # A list of all hapax legomena "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "counts.freq('stipulate') * 100 # percentage of the corpus for this token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "counts.plot(200, cumulative=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from itertools import chain \n",
    "\n",
    "brown = nltk.corpus.brown\n",
    "categories = brown.categories()\n",
    "\n",
    "counts = nltk.ConditionalFreqDist(chain(*[[(cat, word) for word in brown.words(categories=cat)] for cat in categories]))\n",
    "\n",
    "for category, dist in counts.items():\n",
    "    vocab  = len(dist.keys())\n",
    "    tokens = sum(dist.values())\n",
    "    lexdiv = float(tokens) / float(vocab)\n",
    "    print \"%s: %i types with %i tokens and lexical diveristy of %0.3f\" % (category, vocab, tokens, lexdiv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Your turn: compute the conditional frequency distribution of bigrams in a corpus_\n",
    "\n",
    "Hint:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for ngram in nltk.ngrams([\"The\", \"bear\", \"walked\", \"in\", \"the\", \"woods\", \"at\", \"midnight\"], 5):\n",
    "    print ngram\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing Text\n",
    "\n",
    "NLTK is great at the preprocessing of Raw text - it provides the following tools for dividing text into it's constituent parts:\n",
    "\n",
    "- `sent_tokenize`: a Punkt sentence tokenizer:\n",
    "\n",
    "    This tokenizer divides a text into a list of sentences, by using an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences.  It must be trained on a large collection of plaintext in the target language before it can be used.\n",
    "    \n",
    "    However, Punkt is designed to learn parameters (a list of abbreviations, etc.) unsupervised from a corpus similar to the target domain. The pre-packaged models may therefore be unsuitable: use PunktSentenceTokenizer(text) to learn parameters from the given text.\n",
    "    \n",
    "    \n",
    "- `word_tokenize`: a Treebank tokenizer \n",
    "\n",
    "    The Treebank tokenizer uses regular expressions to tokenize text as in Penn Treebank. This is the method that is invoked by ``word_tokenize()``.  It assumes that the text has already been segmented into sentences, e.g. using ``sent_tokenize()``.\n",
    "    \n",
    "\n",
    "- `pos_tag`: a maximum entropy tagger trained on the Penn Treebank\n",
    "\n",
    "    There are several other taggers including (notably) the BrillTagger as well as the BrillTrainer to train your own tagger or tagset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text = u\"Medical personnel returning to New York and New Jersey from the Ebola-riddled countries in West Africa will be automatically quarantined if they had direct contact with an infected person, officials announced Friday. New York Gov. Andrew Cuomo (D) and New Jersey Gov. Chris Christie (R) announced the decision at a joint news conference Friday at 7 World Trade Center. “We have to do more,” Cuomo said. “It’s too serious of a situation to leave it to the honor system of compliance.” They said that public-health officials at John F. Kennedy and Newark Liberty international airports, where enhanced screening for Ebola is taking place, would make the determination on who would be quarantined. Anyone who had direct contact with an Ebola patient in Liberia, Sierra Leone or Guinea will be quarantined. In addition, anyone who traveled there but had no such contact would be actively monitored and possibly quarantined, authorities said. This news came a day after a doctor who had treated Ebola patients in Guinea was diagnosed in Manhattan, becoming the fourth person diagnosed with the virus in the United States and the first outside of Dallas. And the decision came not long after a health-care worker who had treated Ebola patients arrived at Newark, one of five airports where people traveling from West Africa to the United States are encountering the stricter screening rules.\"\n",
    "\n",
    "for sent in nltk.sent_tokenize(text): \n",
    "    print sent\n",
    "    print\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for sent in nltk.sent_tokenize(text):\n",
    "    print list(nltk.wordpunct_tokenize(sent))\n",
    "    print"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for sent in nltk.sent_tokenize(text):\n",
    "    print list(nltk.pos_tag(nltk.word_tokenize(sent)))\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these taggers work _pretty_ well - but you can (and should train them on your own corpora). \n",
    "\n",
    "## Stemming and Lemmatization\n",
    "\n",
    "We have an immense number of word forms as you can see from our various counts in the `FreqDist` above - it is helpful for many applications to normalize these word forms (especially applications like search) into some canonical word for further exploration. In English (and many other languages) - mophological context indicate gender, tense, quantity, etc. but these sublties might not be necessary:\n",
    "\n",
    "Stemming = chop off affixes to get the root stem of the word:\n",
    "\n",
    "    running --> run\n",
    "    flowers --> flower\n",
    "    geese   --> geese \n",
    "    \n",
    "Lemmatization = look up word form in a lexicon to get canonical lemma\n",
    "\n",
    "    women   --> woman\n",
    "    foxes   --> fox\n",
    "    sheep   --> sheep\n",
    "    \n",
    "There are several stemmers available:\n",
    "\n",
    "    - Lancaster (English, newer and aggressive)\n",
    "    - Porter (English, original stemmer)\n",
    "    - Snowball (Many langauges, newest)\n",
    "    \n",
    "The Lemmatizer uses the WordNet lexicon\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.stem.lancaster import LancasterStemmer\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "text = list(nltk.word_tokenize(\"The women running in the fog passed bunnies working as computer scientists.\"))\n",
    "\n",
    "snowball = SnowballStemmer('english')\n",
    "lancaster = LancasterStemmer()\n",
    "porter = PorterStemmer()\n",
    "\n",
    "for stemmer in (snowball, lancaster, porter):\n",
    "    stemmed_text = [stemmer.stem(t) for t in text]\n",
    "    print \" \".join(stemmed_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "lemmas = [lemmatizer.lemmatize(t) for t in text]\n",
    "print \" \".join(lemmas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the lemmatizer has to load the WordNet corpus which takes a bit.\n",
    "\n",
    "Typical normalization of text for use as features in machine learning models looks something like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "## Module constants\n",
    "lemmatizer  = WordNetLemmatizer()\n",
    "stopwords   = set(nltk.corpus.stopwords.words('english'))\n",
    "punctuation = string.punctuation\n",
    "\n",
    "def normalize(text):\n",
    "    for token in nltk.word_tokenize(text):\n",
    "        token = token.lower()\n",
    "        token = lemmatizer.lemmatize(token)\n",
    "        if token not in stopwords and token not in punctuation:\n",
    "            yield token\n",
    "\n",
    "print list(normalize(\"The eagle flies at midnight.\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Recognition\n",
    "\n",
    "NLTK has an excellent MaxEnt backed Named Entity Recognizer that is trained on the Penn Treebank. You can also retrain the chunker if you'd like - the code is very readable to extend it with a Gazette or otherwise. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print nltk.ne_chunk(nltk.pos_tag(nltk.word_tokenize(\"John Smith is from the United States of America and works at Microsoft Research Labs\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also wrap the Stanford NER system, which many of you are also probably used to using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from nltk.tag import StanfordNERTagger\n",
    "\n",
    "# change the paths below to point to wherever you unzipped the Stanford NER download file\n",
    "stanford_root = '/Users/benjamin/Development/stanford-ner-2014-01-04'\n",
    "stanford_data = os.path.join(stanford_root, 'classifiers/english.all.3class.distsim.crf.ser.gz')\n",
    "stanford_jar  = os.path.join(stanford_root, 'stanford-ner-2014-01-04.jar')\n",
    "\n",
    "st = StanfordNERTagger(stanford_data, stanford_jar, 'utf-8')\n",
    "for i in st.tag(\"John Bengfort is from the United States of America and works at Microsoft Research Labs\".split()):\n",
    "    print '[' + i[1] + '] ' + i[0]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "### Preprocessing\n",
    "\n",
    "The primary responsibility you will have before any task involving NLP is to ingest and transform raw text into a corpus that can then be used for performing further evaluations. NLTK provides many corpora for you to work with for exploration, but you must become able to design and construct your own corpora, and to implement `nltk.CorpusReader` objects - classes that in a memory safe and efficient way are able to read entire corpora and analyze them. \n",
    "\n",
    "Many people get away with the `nltk.PlainTextCorpusReader` - which uses built-in taggers and tokenizers to deal with raw text. However, this methodology leaves you at the mercy of the tagging model that you have provided, and does not allow you to make corrections that are saved in between runs. Instead you should preprocess your text to allow it to be read by the `nltk.corpus.TaggedCorpusReader` or the penultimate corpus, the `nltk.corpus.BracketParseCorpusReader`. \n",
    "\n",
    "In this task, you will transform raw text into a format that can then be read by the `nltk.corpus.TaggedCorpusReader`. See the documentation at [http://www.nltk.org/api/nltk.corpus.reader.html](http://www.nltk.org/api/nltk.corpus.reader.html) for more information on this reader. \n",
    "\n",
    "You will find 20-40 documents of recent tech articles from Engadget and Tech Crunch at the following link: [http://bit.ly/nlpnltkcorpus](http://bit.ly/nlpnltkcorpus) - please download them to your local file system. Write a Python program that uses NLTK to preprocess these documents into a format that can be easily read by the `nltk.corpus.TaggedCorpusReader`. \n",
    "\n",
    "Note that you will have to process these files and remove HTML tags and you might have to do other tasks related to the clean up; to do this I suggest you use the third party library BeautifulSoup which can be found at [http://www.crummy.com/software/BeautifulSoup/](http://www.crummy.com/software/BeautifulSoup/). See also Chapter 3 in the NLTK book for more information.\n",
    "\n",
    "#### Evaluation\n",
    "\n",
    "1. What is the word count and vocabulary of this corpus?\n",
    "2. What is the lexical diversity of the corpus?\n",
    "3. What are the 5 most common lexical categories (parts of speech)?\n",
    "4. What are the 10 most common unigrams, the 10 most common bigrams? (please exclude stopwords, using the `nltk.corpus.stopwords('english')` list)\n",
    "5. How many nouns are in the corpus?\n",
    "\n",
    "### Parsing\n",
    "\n",
    "Given a seed inventory of pre-terminal and non-terminal symbols (grammatical categories) and a sample lexicon, write a grammar for English noun phrases. Your grammar should cover all legal structures of noun phrases used by the grammatical categories provided. You must include the following:\n",
    "\n",
    "- noun-noun compounds (\"brick wall\", \"lawn chair\")\n",
    "- relative clauses of the form Rel-Cl ⟶ Rel-Pro V NP (\"[the ball] that hit her\") \n",
    "\n",
    "**Note:** You do not need to cover more than one PP in a row, more than one adjective in a row, noun-noun compounds of length > 2, quantifiers followed by determiners (\"all of these\") or mass nouns (\"beer\", \"sincerity\")\n",
    "\n",
    "You should then write a program that uses an NLTK parser and the grammar you constructed that will return a syntactic tree if the input is a noun phrase or None if the input is ungrammatical. Your program will have to take the input sentence, tokenize it and then tag it according to the lexicon (you can assume that words in this lexicon do not have multiple senses) - you'll then have to pass the grammar phrase (the tags) to the parser.\n",
    "\n",
    "#### Tagset\n",
    "\n",
    "    N = noun\n",
    "    NP = noun phrase\n",
    "    Adj = adjective\n",
    "    AdjP = adjective phrase\n",
    "    Adv = adverb\n",
    "    Prep = preposition\n",
    "    PP = prepositional phrase\n",
    "    Quant = quantifier\n",
    "    Ord = ordinal numeral\n",
    "    Card = cardinal numeral\tRel-Cl = relative clause\n",
    "    Rel-Pro = relative pronoun\n",
    "    V = verb\n",
    "    S = sentence\n",
    "    Det = determiner\n",
    "    Dem-Det = demonstrative determiner\n",
    "    Wh-Det = wh-determiner\n",
    "    PPron = personal pronoun\n",
    "    PoPron = possessive pronoun\n",
    "\n",
    "#### Sample Lexicon\n",
    "\n",
    "    a            Det\n",
    "    an           Det\n",
    "    at           Prep\n",
    "    airplane     NSg\n",
    "    airplanes    NPl\n",
    "    airport      NSg\n",
    "    airports     NPl\n",
    "    any          Quant\n",
    "    beautiful    Adj\n",
    "    big          Adj\n",
    "    eat          V\n",
    "    eats         V3Sg\n",
    "    finished     VPastPP\n",
    "    four         Card\n",
    "    fourth       Ord\n",
    "    he           PPron\n",
    "    his          PoPron\n",
    "    in           Prep\n",
    "    many         Quant\n",
    "    my           PoPron\n",
    "    new          Adj\n",
    "    of           Prep\n",
    "    offered      VPastPP\n",
    "    on           Prep\n",
    "    restaurant   NSg\n",
    "    restaurants  NPl\n",
    "    runway       NSg\n",
    "    runways      NPl\n",
    "    second       Ord\n",
    "    some         Quant\n",
    "    that         Dem-DetSg\n",
    "    that         Rel-Pro\n",
    "    the          Det\n",
    "    this         Dem-DetSg\n",
    "    these        Dem-DetPl\n",
    "    third        Ord\n",
    "    those        Dem-DetPl\n",
    "    three        Card\n",
    "    two          Card\n",
    "    very         Adv\n",
    "    which        Wh-Det\n",
    "    who          Wh-Det\n",
    "    you          PPron\n",
    "\n",
    "#### Evaluation Phrases\n",
    "\n",
    "- \"Four new airports\"\n",
    "- \"Very new airport runways\"\n",
    "- \"His second house\"\n",
    "- \"Some beautiful dishes which a restaurant offered\"\n",
    "- \"The runway that the airport built\"\n",
    "\n",
    "### Document Classification\n",
    "\n",
    "In the first week you created an ingestion mechanism and an NLTK corpus reader for a set of RSS feeds. These feeds potentially have topics associated with them (broad tags like tech, news, sports, etc). In this question you'll build a classifier on a data set of RSS feeds that is provided in the course materials to decide whether or not you can categorize the various topics using one of the classifiers you learned in this week.\n",
    "\n",
    "The corpus is constructed as follows. Each individual blog post is in its own HTML file stored in a directory labled with the topic. Use the `nltk.CategorizedCorpusReader` or the `nltk.CategorizedPlaintextCorpusReader` to construct your corpora (you may review how the movie reviews data set is structured). To do this you need to pass to the corpus the path to the root of your corpus, and a regular expression to match file names. You also need to use a regular expression passed as the `cat_pattern` keyword argument, which is used to match the category labels. Here is an example for the spam corpus:\n",
    "\n",
    "    from nltk.corpus import CategorizedPlaintextCorpusReader as EmailCorpus\n",
    "\n",
    "    corpus   = EmailCorpus(\"./data/nbspam\", r'(?!\\.).*\\.[a-f0-9]+',\n",
    "                   cat_pattern=r'(spam|ham)/.*', encoding='iso-8859-1')\n",
    "\n",
    "    print corpus.categories()\n",
    "    print corpus.fileids()\n",
    "\n",
    "Create a test set, a dev test set, and a training set from randomly shuffled documents that are in the corpus to use in your development. Save these sets to disk with pickles to ensure that you can develop easily with them.\n",
    "\n",
    "Create a function that extracts features per document. Choose any features you would like. One idea is to use the most common unigrams; you might be able to use common bigrams as well. If you can think of any other features, feel free to include them as well (maybe an includes_recipe feature, etc.)) You may want to consider a TF-IDF feature to improve your results.\n",
    "\n",
    "Train the classifier of your choice on the training data, and then improve it with your dev set. Report your final accuracy and the most informative features by running the accuracy checker on the final test set.\n",
    "\n",
    "#### Evaluation\n",
    "\n",
    "1. Report accuracy and most informative features of classifier (4 points)\n",
    "2. Show complete work with submitted Python code (6 points)\n",
    "3. Create a corpus reader that extends a built-in NLTK corpus reader (4 points)\n",
    "4. Create an efficient feature extractor (4 points)\n",
    "5. Achieve an accuracy with your classifier of greater than 85% (2 points)\n",
    "\n",
    "### Product Classification\n",
    "\n",
    "The second question involves comparing and contrast the Naive Bayes Classifier with the Maximum Entropy classifier. You will be given an abbreviated data set of product names and their descriptions as well as their label (tops, bottoms, shoes, etc.) - similarly to question one, create a corpus that can read the CSV file - you may want to look at the `nltk.corpus.WordListCorpusReader` for inspiration about how to create such a corpus (each product is on a single line).\n",
    "\n",
    "Create test and training sets from the data then build both a NaiveBayes and Maxent classifier - make sure that you save these classifiers to disk using the `pickle` module! The Maxent classifier in particular will take a long time to train. Once they're trained; report the accuracy of each as well as the most informative features. Are there any surprises? Which classifier performs better?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}