scraping/scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following provides an example of\n",
    "<ul>\n",
    "<li>IPython Notebook</li>\n",
    "<li>Parsing HTML using lxml</li>\n",
    "<li>Data manipulation w/ pandas</li>\n",
    "<li>Classification w/ Multinomial Naive Bayes using scikit-learn</li>\n",
    "</ul>\n",
    "\n",
    "Now lets see if we can teach our computer how to tell the difference between a Wall Street Journal headline and a Gawker headline. \n",
    "\n",
    "Inspiration for this notebook/presentation was provided by [this](http://nbviewer.ipython.org/github/nealcaren/workshop_2014/blob/master/notebooks/6_Classification.ipynb) and [this](http://nbviewer.ipython.org/github/cs109/content/blob/master/HW3_solutions.ipynb). If you want to run this notebook, or one like it, it'll be helpful for you to check out [Anaconda](https://store.continuum.io/cshop/anaconda/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get the Data!\n",
    "If we want to build a model with a discerning taste for quality news, we're going to have to find him some examples. Lucky for us, Gawker and the Wall Street Journal both have websites. \n",
    "### To the internet (with dev tools)!\n",
    "#### [Gawker](http://gawker.com)\n",
    "After poking around the gawker homepage with the help of command + option + i (mac) and a Chrome plugin, [XPath Helper](https://chrome.google.com/webstore/detail/xpath-helper/hgimnogjllphhhkhlmebbmlgjoejdpjl?hl=en), we can see that all the article titles can be grabbed with the xpath, `\"//header/h1/a/text()\"`. lxml lets us retrieve all the text that matches our xpath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We got 17 titles. Here are the first 5:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['College Kids Not So Into the Free Press. Whatever',\n",
       " \"Today's Best Deals: Running Shoes, Outdoor Clothes, Anker Jump Starter, and More\",\n",
       " \"Who's Named in the Panama Papers?\",\n",
       " \"NYPD's Blowhard Union Boss Is Feeling a Little Scared and a Little Confused\",\n",
       " 'This Is Bad, Even For Sarah Palin']"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lxml import html \n",
    "x = html.parse('http://gawker.com')\n",
    "titles = x.xpath('//header/h1/a/text()')\n",
    "print \"We got {} titles. Here are the first 5:\".format(len(titles))\n",
    "titles[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This page only has 20 titles on it. If we want to properly train our model, we're going to need more examples. There's a \"More Stories\" button at the bottom of the page, which brings us to another, similarly structured page with new titles and another \"More Stories\" button. To get more examples, we'll repeat the process above in a loop with each successive iteration hitting the page pointed to by the \"More Stories\" button. In order to figure out what the link is, go do some more investigating! You can also just look at the code below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "headlines = x.xpath('//header/h1/a/text()')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved 17 titles from url: http://gawker.com/\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1459814100852\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1459777920161\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1459616727760\n",
      "Retrieved 16 titles from url: http://gawker.com/?startTime=1459518120384\n",
      "Retrieved 17 titles from url: http://gawker.com/?startTime=1459440720292\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1459378619355\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1459344905458\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1459282860784\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1459224087623\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1459180800411\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1459096920497\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1458942097476\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1458873001132\n",
      "Retrieved 23 titles from url: http://gawker.com/?startTime=1458838501396\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1458762000470\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1458691200399\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1458648838619\n",
      "Retrieved 17 titles from url: http://gawker.com/?startTime=1458568349668\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1458411300607\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1458317400101\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1458236400438\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1458168300774\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1458139800781\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1458082978370\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1458052800282\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1457977740670\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457906400579\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457799193651\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457712900315\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1457649720778\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457582709616\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1457542700682\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457492894819\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457456760380\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1457390396828\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457316282784\n",
      "Retrieved 17 titles from url: http://gawker.com/?startTime=1457221920353\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1457130679407\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1457063685144\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1457037300965\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456967315072\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1456942260395\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456885757675\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1456850488611\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1456790401314\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456763744146\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456691039055\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456527300918\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456502460180\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1456436100526\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456414440291\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1456336700198\n",
      "Retrieved 17 titles from url: http://gawker.com/?startTime=1456273083458\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1456201802624\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1456168500179\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1456088148835\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1455993374937\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1455903180013\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1455843616785\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1455804120343\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1455730221178\n",
      "Retrieved 17 titles from url: http://gawker.com/?startTime=1455659880488\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1455591918939\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1455553200047\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1455469971709\n",
      "Retrieved 17 titles from url: http://gawker.com/?startTime=1455383318369\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1455289920340\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1455223200774\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1455199761203\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1455134100662\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1455073539497\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1455052500595\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1454976900789\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454951100525\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454868325898\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1454784480648\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1454693820811\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454626184034\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454595538930\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1454525160578\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1454461200308\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1454439412740\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454380200472\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1454348400739\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1454277180763\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454172298806\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454086800199\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1454019780001\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1453993980250\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1453930800340\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1453866002019\n",
      "Retrieved 17 titles from url: http://gawker.com/?startTime=1453830666472\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1453765105238\n",
      "Retrieved 21 titles from url: http://gawker.com/?startTime=1453699281227\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1453646100280\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1453500960040\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1453436298952\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1453399920758\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1453332702220\n",
      "Retrieved 19 titles from url: http://gawker.com/?startTime=1453263812584\n",
      "Retrieved 18 titles from url: http://gawker.com/?startTime=1453236600032\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1453165200269\n",
      "Retrieved 20 titles from url: http://gawker.com/?startTime=1453091040767\n"
     ]
    }
   ],
   "source": [
    "# These are the xpaths we determined from snooping \n",
    "next_button_xpath = \"//div[@class='row load-more']//a[contains(@href, 'startTime')]/@href\"\n",
    "headline_xpath = '//header/h1/a/text()'\n",
    "\n",
    "# We'll use sleep to add some time in between requests\n",
    "# so that we're not bombarding Gawker's server too hard. \n",
    "from time import sleep\n",
    "\n",
    "# Now we'll fill this list of gawker titles by starting\n",
    "# at the landing page and following \"More Stories\" links\n",
    "gawker_titles = []\n",
    "base_url = 'http://gawker.com/{}'\n",
    "next_page = \"http://gawker.com/\"\n",
    "while len(gawker_titles) < 2000 and next_page:\n",
    "    dom = html.parse(next_page)\n",
    "    headlines = dom.xpath(headline_xpath)\n",
    "    print \"Retrieved {} titles from url: {}\".format(len(headlines), next_page)\n",
    "    gawker_titles += headlines\n",
    "    next_pages = dom.xpath(next_button_xpath)\n",
    "    if next_pages: \n",
    "        next_page = base_url.format(next_pages[0]) \n",
    "    else:\n",
    "        print \"No next button found\"\n",
    "        next_page = None\n",
    "    sleep(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Holy smokes, we got 2001 Gawker headlines!\n"
     ]
    }
   ],
   "source": [
    "with open('gawker_titles.txt', 'wb') as out:\n",
    "    out.writelines(gawker_titles)\n",
    "# with open('gawker_titles.txt') as f:\n",
    "#     gawker_titles = f.readlines()\n",
    "    \n",
    "print \"Holy smokes, we got {} Gawker headlines!\".format(len(gawker_titles))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Wall Street Journal](http://online.wsj.com/public/page/archive-2014-1-1.html)\n",
    "Now we'll do a similar thing with WSJ now. Here we notice that they have a section of the site where they have lists of articles for each day in the past year. There are links to the different archive dates all over the page, and we can see that the links all have the same structure, with different dates in the URL. Lets iterate over a bunch of dates. I grabbed the articles from the first day of each month this year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieved 106 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-1-1.html\n",
      "Retrieved 21 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-2-1.html\n",
      "Retrieved 31 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-3-1.html\n",
      "Retrieved 284 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-4-1.html\n",
      "Retrieved 386 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-5-1.html\n",
      "Retrieved 120 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-6-1.html\n",
      "Retrieved 310 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-7-1.html\n",
      "Retrieved 300 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-8-1.html\n",
      "Retrieved 162 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-9-1.html\n",
      "Retrieved 388 WSJ headlines from url: http://online.wsj.com/public/page/archive-2014-10-1.html\n"
     ]
    }
   ],
   "source": [
    "wsj_url = \"http://online.wsj.com/public/page/archive-2014-{}-1.html\"\n",
    "wsj_headline_xpath = \"//h2/a/text()\"\n",
    "wsj_headlines = []\n",
    "for i in range(1, 11): \n",
    "    dom = html.parse(wsj_url.format(i))\n",
    "    titles = dom.xpath(wsj_headline_xpath)\n",
    "    wsj_headlines += titles\n",
    "    print \"Retrieved {} WSJ headlines from url: {}\".format(len(titles), wsj_url.format(i))   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Jeez, Louise! We got 2108 WSJ headlines!\n"
     ]
    }
   ],
   "source": [
    "with open('wsj_titles.txt', 'wb') as out:\n",
    "    out.writelines(wsj_headlines)\n",
    "# with open('wsj_titles.txt') as f:\n",
    "#     wsj_headlines = f.readlines()\n",
    "    \n",
    "print \"Jeez, Louise! We got {} WSJ headlines!\".format(len(wsj_headlines))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll use pandas to build a data frame with two columns: \"gawker\", which contains a boolean value indicating whether the value in the \"title\" column came from Gawker's website. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>gawker</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4104</th>\n",
       "      <td>False</td>\n",
       "      <td>Australia to Fly Support Missions in Iraq</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4105</th>\n",
       "      <td>False</td>\n",
       "      <td>Zalando Stock Debut Disappoints</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4106</th>\n",
       "      <td>False</td>\n",
       "      <td>China September Official Manufacturing PMI Hol...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4107</th>\n",
       "      <td>False</td>\n",
       "      <td>Asian Shares Mixed in Quiet Trading</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4108</th>\n",
       "      <td>False</td>\n",
       "      <td>BOJ Tankan: Japan Corporate Sentiment Improves</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     gawker                                              title\n",
       "4104  False          Australia to Fly Support Missions in Iraq\n",
       "4105  False                    Zalando Stock Debut Disappoints\n",
       "4106  False  China September Official Manufacturing PMI Hol...\n",
       "4107  False                Asian Shares Mixed in Quiet Trading\n",
       "4108  False     BOJ Tankan: Japan Corporate Sentiment Improves"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "gawk_records = [{'gawker': True, 'title': title} for title in gawker_titles]\n",
    "wsj_records = [{'gawker': False, 'title': title} for title in wsj_headlines]\n",
    "df = pd.DataFrame.from_records(gawk_records + wsj_records)\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Teach the Machine!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basic goal of machine learning is to learn a model or function that maps our inputs/observations to outputs/predictions. The model that we'll be building to make our predictions is known as Naive Bayes. We can compute by counting the probability that any given word shows up in a Gawker title, or P(Word | Gawker). We want the probability of a that a given body of text is a Gawker title, or P(Gawker | Words).\n",
    "\n",
    "$$P(Gawker | Words) = P(Word_1|Gawker)*P(Word_2|Gawker)*..*P(Word_n | Gawker)*P(Gawker)$$\n",
    "\n",
    "All of the probabilities on the right side of the equation are empirically determined from the text data. Luckily, instead of having to write all of the code to determine those probabilities ourselves, sklearn's CountVectorizer does all of the dirty work for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(min_df=5, max_df=.3, ngram_range=(1,2))\n",
    "# X, Y = make_xy(df)\n",
    "X = vectorizer.fit_transform(df.title)\n",
    "X = X.tocsc()  # some versions of sklearn return COO format\n",
    "Y = df.gawker.values.astype(np.int) # Need numbers instead of bools for "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a data array, X, whose rows correspond to titles and whose columns correspond to words. So the value X[i, j] is the number of times word j shows up in article title i. Each row has a corresponding member in vector, Y. If the headline associated with X[i] came from Gawker, Y[i] == 1. Otherwise, Y[i] == 0. Now our data are in a format we want it to be in. Time to seperate our data into training and testing sets before building and evaluating our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "X_train, X_test, Y_train, Y_test = cross_validation.train_test_split(X,Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultinomialNB(alpha=0.5, class_prior=None, fit_prior=False)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clf = naive_bayes.MultinomialNB(fit_prior=False, alpha=0.5)\n",
    "clf.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the machine!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll see how good our model is by testing the accuracy of its predictions on articles it hasn't seen before. The accuracy metric reported below is simply the percentage of titles it classifies correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 84.91%\n"
     ]
    }
   ],
   "source": [
    "print \"Accuracy: %0.2f%%\" % (100 * clf.score(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad for a stupid machine! We can take a closer look at how the model is making predictions. Let's look at which words are the Gawkeriest and which ones are the Wall Street Journaliest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gawker words\t P(gawker| word)\n",
      "            one week 0.99\n",
      "              titles 0.99\n",
      "            get free 0.99\n",
      "              choose 0.99\n",
      "   ingredient recipe 0.99\n",
      "             week of 0.99\n",
      "         apron fresh 0.99\n",
      "            month of 0.99\n",
      "         choose from 0.99\n",
      "           of oyster 0.99\n",
      "WSJ words\t P(gawker | word)\n",
      "              market 0.02\n",
      "           what news 0.02\n",
      "       manufacturing 0.01\n",
      "           investors 0.01\n",
      "              growth 0.01\n",
      "                 ipo 0.01\n",
      "                 ceo 0.01\n",
      "               sales 0.01\n",
      "               china 0.01\n",
      "              profit 0.01\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "words = np.array(vectorizer.get_feature_names())\n",
    "\n",
    "x = np.eye(X_test.shape[1])\n",
    "probs = clf.predict_log_proba(x)[:, 0]\n",
    "ind = np.argsort(probs)\n",
    "\n",
    "good_words = words[ind[:10]]\n",
    "bad_words = words[ind[-10:]]\n",
    "\n",
    "good_prob = probs[ind[:10]]\n",
    "bad_prob = probs[ind[-10:]]\n",
    "\n",
    "print \"Gawker words\\t P(gawker| word)\"\n",
    "for w, p in zip(good_words, good_prob):\n",
    "    print \"%20s\" % w, \"%0.2f\" % (1 - np.exp(p))\n",
    "    \n",
    "print \"WSJ words\\t P(gawker | word)\"\n",
    "for w, p in zip(bad_words, bad_prob):\n",
    "    print \"%20s\" % w, \"%0.2f\" % (1 - np.exp(p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might be complaining that our model should do better. But some titles can be tough. Would you have classified these mis-predicted article titles correctly? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mis-predicted WSJ quotes\n",
      "---------------------------\n",
      "Study Finds Over One Million Caring for Iraq, Afghan War Veterans\n",
      "\n",
      "Photo of the Week\n",
      "\n",
      "Body of Ferry Victim Found by Fishermen\n",
      "\n",
      "From Florida Boy to Alleged Suicide Bomber in Syria\n",
      "\n",
      "How the 'Jesus' Wife' Hoax Fell Apart\n",
      "\n",
      "\n",
      "Mis-predicted Gawker quotes\n",
      "--------------------------\n",
      "Benetton Will Contribute to Fund For Rana Plaza Victims in Bangladesh\n",
      "\n",
      "NYU Urges Staffers to Help Pay Students' Outrageously Expensive Tuition\n",
      "\n",
      "Nobody In China Wants Tibetan Mastiffs Anymore\n",
      "\n",
      "Ukraine Cease-Fire Under Threat in Key Town, Holding Elsewhere\n",
      "\n",
      "Leaders Reach Shaky Cease-fire Deal to End War in Ukraine \n",
      "\n"
     ]
    }
   ],
   "source": [
    "prob = clf.predict_proba(X)[:, 0]\n",
    "predict = clf.predict(X)\n",
    "\n",
    "bad_wsj = np.argsort(prob[Y == 0])[:5]\n",
    "bad_gawker = np.argsort(prob[Y == 1])[-5:]\n",
    "\n",
    "print \"Mis-predicted WSJ quotes\"\n",
    "print '---------------------------'\n",
    "for row in bad_wsj:\n",
    "    print df[Y == 0].title.irow(row)\n",
    "    print\n",
    "\n",
    "print\n",
    "print \"Mis-predicted Gawker quotes\"\n",
    "print '--------------------------'\n",
    "for row in bad_gawker:\n",
    "    print df[Y == 1].title.irow(row)\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make your own headline and see if it belongs in Gawker or WSJ!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P(gawker) = 0.971367221347\n"
     ]
    }
   ],
   "source": [
    "probs = clf.predict_proba(vectorizer.transform([\"Your title here\"]))\n",
    "print \"P(gawker) = {}\".format(probs[0][1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}