practical9.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical 9\n",
    "\n",
    "## Text Mining\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read in some packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\AAAAAAAAAAAAAA\\anaconda\\lib\\site-packages\\sklearn\\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "# Import pandas to read in data\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Import models and evaluation functions\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "from sklearn import metrics\n",
    "from sklearn import cross_validation\n",
    "\n",
    "# Import vectorizers to turn text into numeric\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# Import plotting\n",
    "import matplotlib.pylab as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text classification\n",
    "We are going to look at some Amazon reviews and classify them into positive or negative."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "The file `data/books.csv` contains 2,000 Amazon book reviews. The data set contains two features: the first column (contained in quotes) is the review text. The second column is a binary label indicating if the review is positive or negative.\n",
    "\n",
    "Let's take a quick look at the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'head' 不是内部或外部命令，也不是可运行的程序\n",
      "或批处理文件。\n"
     ]
    }
   ],
   "source": [
    "!head -3 data/books.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the data into a pandas data frame. You'll notice two new attributed in `pd.read_csv()` that we've never seen before. The first, `quotechar` is tell us what is being used to \"encapsulate\" the text fields. Since our review text is surrounding by double quotes, we let pandas know. We use a `\\` since the quote is also used to surround the quote. This backslash is known as an escape character. We also let pandas now this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"data/books.csv\", quotechar=\"\\\"\", escapechar=\"\\\\\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_text</th>\n",
       "      <th>positive</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>THis book was horrible.  If it was possible to...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I like to use the Amazon reviews when purchasi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>THis book was horrible.  If it was possible to...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>I'm not sure who's writing these reviews, but ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>I picked up the first book in this series (The...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         review_text  positive\n",
       "0  THis book was horrible.  If it was possible to...         0\n",
       "1  I like to use the Amazon reviews when purchasi...         0\n",
       "2  THis book was horrible.  If it was possible to...         0\n",
       "3  I'm not sure who's writing these reviews, but ...         0\n",
       "4  I picked up the first book in this series (The...         0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text as a set of features\n",
    "Going from text to numeric data is very easy. Let's take a look at how we can do this. We'll start by separating out our X and Y data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_text = data['review_text']\n",
    "Y = data['positive']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will turn `X_text` into just `X` -- a numeric representation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a vectorizer that will track text as binary features\n",
    "binary_vectorizer = CountVectorizer(binary=True)\n",
    "\n",
    "# Let the vectorizer learn what tokens exist in the text data\n",
    "binary_vectorizer.fit(X_text)\n",
    "\n",
    "# Turn these tokens into a numeric matrix\n",
    "X = binary_vectorizer.transform(X_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modeling\n",
    "We have a ton of features, let's use them in some different models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Area under the ROC curve for our classifier is 0.848\n"
     ]
    }
   ],
   "source": [
    "# Create a model\n",
    "logistic_regression = LogisticRegression()\n",
    "\n",
    "# Use this model and our data to get 5-fold cross validation AUCs\n",
    "aucs = cross_validation.cross_val_score(logistic_regression, X, Y, scoring=\"roc_auc\", cv=5)\n",
    "\n",
    "# Print out the average AUC rounded to three decimal points\n",
    "print (\"Area under the ROC curve for our classifier is \" + str(round(np.mean(aucs), 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try using full counts instead of a binary representation. I've just copy and pasted what is above and removed the `binary=True` from the vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Area under the ROC curve for our classifier is 0.849\n"
     ]
    }
   ],
   "source": [
    "# Create a vectorizer that will track text as binary features\n",
    "count_vectorizer = CountVectorizer()\n",
    "\n",
    "# Let the vectorizer learn what tokens exist in the text data\n",
    "count_vectorizer.fit(X_text)\n",
    "\n",
    "# Turn these tokens into a numeric matrix\n",
    "X = count_vectorizer.transform(X_text)\n",
    "\n",
    "# Create a model\n",
    "logistic_regression = LogisticRegression()\n",
    "\n",
    "# Use this model and our data to get 5-fold cross validation AUCs\n",
    "aucs = cross_validation.cross_val_score(logistic_regression, X, Y, scoring=\"roc_auc\", cv=5)\n",
    "\n",
    "# Print out the average AUC rounded to three decimal points\n",
    "print (\"Area under the ROC curve for our classifier is \" + str(round(np.mean(aucs), 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try using TF-IDF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Area under the ROC curve for our classifier is 0.87\n"
     ]
    }
   ],
   "source": [
    "# Create a vectorizer that will track text as binary features\n",
    "tfidf_vectorizer = TfidfVectorizer()\n",
    "\n",
    "# Let the vectorizer learn what tokens exist in the text data\n",
    "tfidf_vectorizer.fit(X_text)\n",
    "\n",
    "# Turn these tokens into a numeric matrix\n",
    "X = tfidf_vectorizer.transform(X_text)\n",
    "\n",
    "# Create a model\n",
    "logistic_regression = LogisticRegression()\n",
    "\n",
    "# Use this model and our data to get 5-fold cross validation AUCs\n",
    "aucs = cross_validation.cross_val_score(logistic_regression, X, Y, scoring=\"roc_auc\", cv=5)\n",
    "\n",
    "# Print out the average AUC rounded to three decimal points\n",
    "print (\"Area under the ROC curve for our classifier is \" + str(round(np.mean(aucs), 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Group work\n",
    "#### Features\n",
    "Tfidf is looking pretty good! How about adding n-grams? Stop words? Lowercase transforming?\n",
    "\n",
    "`CountVectorizer()` and `TfidfVectorizer()` can be modified to handle all of these things. Work in groups and try a few different combinations of these settings for anything you want: binary counts, numeric counts, tf-idf counts. Here is how you would use these settings:\n",
    "\n",
    "- \"`ngram_range=(1,2)`\": would include unigrams and bigrams\n",
    "- \"`stop_words=\"english\"`\": would use a standard set of English stop words\n",
    "- \"`lowercase=False`\": would turn off lowercase transformation (it is actually on by default)!\n",
    "\n",
    "You can use some of these like this:\n",
    "\n",
    "`tfidf_vectorizer = TfidfVectorizer(ngram_range=(1,2), lowercase=False)`\n",
    "\n",
    "#### Models\n",
    "You can also swap out the line creating a logistic regression with one making a naive Bayes. This is also one line:\n",
    "\n",
    "`naive_bayes = BernoulliNB()`\n",
    "\n",
    "You can then go ahead and use `naive_bayes` inplace of `logistic_regression`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Area under the ROC curve for our classifier is 0.879\n"
     ]
    }
   ],
   "source": [
    "# Work with your teams here!\n",
    "# Try different features, models, or both!\n",
    "# What is the highest AUC you can get?\n",
    "tfidf_vectorizer_1 = TfidfVectorizer(ngram_range=(1,2))\n",
    "\n",
    "tfidf_vectorizer_1.fit(X_text)\n",
    "\n",
    "X = tfidf_vectorizer_1.transform(X_text)\n",
    "\n",
    "logistic_regression_1 = LogisticRegression()\n",
    "\n",
    "aucs = cross_validation.cross_val_score(logistic_regression_1, X, Y, scoring = \"roc_auc\", cv = 5)\n",
    "\n",
    "print (\"Area under the ROC curve for our classifier is \" + str(round(np.mean(aucs), 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering\n",
    "We have examined two ways of dealing with categorical data: binarizing/dummy variables and numerical scaling. We will practice these here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"data/categorical.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Minutes</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Marital</th>\n",
       "      <th>Satisfaction</th>\n",
       "      <th>Churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>Male</td>\n",
       "      <td>Single</td>\n",
       "      <td>Low</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>220</td>\n",
       "      <td>Female</td>\n",
       "      <td>Married</td>\n",
       "      <td>Very Low</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>500</td>\n",
       "      <td>Female</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>High</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>335</td>\n",
       "      <td>Male</td>\n",
       "      <td>Single</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>450</td>\n",
       "      <td>Male</td>\n",
       "      <td>Married</td>\n",
       "      <td>Very High</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Minutes  Gender   Marital Satisfaction  Churn\n",
       "0      100    Male    Single          Low      0\n",
       "1      220  Female   Married     Very Low      0\n",
       "2      500  Female  Divorced         High      1\n",
       "3      335    Male    Single      Neutral      0\n",
       "4      450    Male   Married    Very High      1"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binarizing\n",
    "Get a list of features you want to binarize, go through each feature and create new features for each level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "features_to_binarize = [\"Gender\", \"Marital\"]\n",
    "\n",
    "# Go through each feature\n",
    "for feature in features_to_binarize:\n",
    "    # Go through each level in this feature (except the last one!)\n",
    "    for level in data[feature].unique()[0:-1]:\n",
    "        # Create new feature for this level\n",
    "        data[feature + \"_\" + level] = pd.Series(data[feature] == level, dtype=int)\n",
    "    # Drop original feature\n",
    "    data = data.drop([feature], 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Minutes</th>\n",
       "      <th>Satisfaction</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Gender_Male</th>\n",
       "      <th>Marital_Single</th>\n",
       "      <th>Marital_Married</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>Low</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>220</td>\n",
       "      <td>Very Low</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>500</td>\n",
       "      <td>High</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>335</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>450</td>\n",
       "      <td>Very High</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Minutes Satisfaction  Churn  Gender_Male  Marital_Single  Marital_Married\n",
       "0      100          Low      0            1               1                0\n",
       "1      220     Very Low      0            0               0                1\n",
       "2      500         High      1            0               0                0\n",
       "3      335      Neutral      0            1               1                0\n",
       "4      450    Very High      1            1               0                1"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numeric scaling\n",
    "We can also replace text levels with some numeric mapping we create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data['Satisfaction'] = data['Satisfaction'].replace(['Very Low', 'Low', 'Neutral', 'High', 'Very High'], \n",
    "                                                    [-2, -1, 0, 1, 2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Minutes</th>\n",
       "      <th>Satisfaction</th>\n",
       "      <th>Churn</th>\n",
       "      <th>Gender_Male</th>\n",
       "      <th>Marital_Single</th>\n",
       "      <th>Marital_Married</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>220</td>\n",
       "      <td>-2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>500</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>335</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>450</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Minutes  Satisfaction  Churn  Gender_Male  Marital_Single  Marital_Married\n",
       "0      100            -1      0            1               1                0\n",
       "1      220            -2      0            0               0                1\n",
       "2      500             1      1            0               0                0\n",
       "3      335             0      0            1               1                0\n",
       "4      450             2      1            1               0                1"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}