4 - Week/bayesian_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Bayesian Learning\n",
    "\n",
    "In this notebook you will learn how to implement the Naive Bayes classifier in Python and how to use the version\n",
    "implemented in scikit-learn."
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## The Dataset\n",
    "\n",
    "In this notebook you will be working with the\n",
    "[Twenty Newsgroups dataset](https://archive.ics.uci.edu/ml/datasets/Twenty+Newsgroups).\n",
    "This dataset consists of 20,000 messages taken from 20 newsgroups. The aim of this classification task is to predict\n",
    "from which group each message came from.\n",
    "\n",
    "The newsgroups are:\n",
    "<PRE>\n",
    "    alt.atheism\n",
    "    comp.graphics\n",
    "    comp.os.ms-windows.misc\n",
    "    comp.sys.ibm.pc.hardware\n",
    "    comp.sys.mac.hardware\n",
    "    comp.windows.x\n",
    "    misc.forsale\n",
    "    rec.autos\n",
    "    rec.motorcycles\n",
    "    rec.sport.baseball\n",
    "    rec.sport.hockey\n",
    "    sci.crypt\n",
    "    sci.electronics\n",
    "    sci.med\n",
    "    sci.space\n",
    "    soc.religion.christian\n",
    "    talk.politics.guns\n",
    "    talk.politics.mideast\n",
    "    talk.politics.misc\n",
    "    talk.religion.misc\n",
    "</PRE>\n",
    "\n",
    "The messages are typical postings and thus have headers including subject lines,\n",
    "signature files, and quoted portions of other messages.\n",
    "\n",
    "We will download this dataset directly from the UCI repository. Note that this dataset is made of\n",
    "20 folders, one per newsgroup, containing\n",
    "1,000 files each. Each file is a message. You can open these files with an editor."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from urllib.request import urlretrieve\n",
    "\n",
    "dataset_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/20newsgroups-mld/20_newsgroups.tar.gz'\n",
    "urlretrieve(dataset_url, '20_newsgroups.tar.gz')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The file is compressed, therefore we will now uncompress it."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import tarfile\n",
    "\n",
    "with tarfile.open('20_newsgroups.tar.gz') as f:\n",
    "    f.extractall('.')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will now loop across the files in the folder and create a dataset list\n",
    "containing tuples made of message texts (the content of the file) and labels (the folder name).\n",
    "For each message we will also remove the header."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import glob\n",
    "import os\n",
    "\n",
    "dataset = []\n",
    "# loop across the newsgroups files\n",
    "for path in glob.glob('20_newsgroups/**/*'):\n",
    "    folder, y, document_id = path.split(os.sep)\n",
    "    with open(path, 'r', encoding='latin-1') as file:\n",
    "        # by splitting an rejoining using a double new line we remove the header from each message.\n",
    "        x = '\\n\\n'.join(file.read().split('\\n\\n')[1:])\n",
    "        dataset.append((x, y))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's now print the values of the target variable, and the number of messages per class."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "classes = {}\n",
    "for x, y in dataset:\n",
    "    if y not in classes:\n",
    "        classes[y] = 0\n",
    "    classes[y] += 1\n",
    "\n",
    "print('#classes: ', len(classes))\n",
    "\n",
    "for clazz in classes:\n",
    "    print(clazz, classes[clazz])"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Let's now have a look at the content of one example."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "x, y = dataset[101]\n",
    "\n",
    "print('Message:')\n",
    "print(x)\n",
    "\n",
    "print('Label:')\n",
    "print(y)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will now randomize the dataset and perform the train-test split."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "\n",
    "# this sets the numpy to print numbers with float precision (this setting affects only the prints not the actual values)\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "# shuffles the list in place\n",
    "random.shuffle(dataset)\n",
    "\n",
    "xs, ys = np.split(dataset, [1], axis=1)\n",
    "xs = xs.reshape(-1)\n",
    "ys = ys.reshape(-1)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will now select 80% of the dataset for training and 20% for testing."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "n_train = len(xs) * 80 // 100\n",
    "xs_train, xs_test = np.split(xs, [n_train], axis=0)\n",
    "ys_train, ys_test = np.split(ys, [n_train], axis=0)\n",
    "\n",
    "print('training set shape:\\t', xs_train.shape)\n",
    "print('test set shape:\\t\\t', xs_test.shape)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### Preprocessing\n",
    "\n",
    "Since the kind of data we are working with is text and the learners that we will be using later require examples made of\n",
    "categorical values, we will now convert these texts into bag-of-words."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def preprocess(text):\n",
    "    tokens = re.split(r'[^\\w]', text) # split on everything unless it is a word\n",
    "    tokens = [t.lower() for t in tokens if t]\n",
    "    res = set(tokens)\n",
    "    return res\n",
    "\n",
    "xs_train_prep = [preprocess(x) for x in xs_train]"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## The Naive Bayes Classifier\n",
    "\n",
    "We will now implement the Naive Bayes Classifier in Python from scratch."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "\n",
    "class NaiveBayes:\n",
    "\n",
    "    def __init__(self):\n",
    "        # keep the count of the number of times a token appears for a given class\n",
    "        self.count_token_given_class = defaultdict(lambda: defaultdict(int))\n",
    "        # keep the count of the number of times an example belongs to a class\n",
    "        self.count_class = defaultdict(int)\n",
    "        # keep the count of the number of times a token is in an example\n",
    "        self.count_token = defaultdict(int)\n",
    "\n",
    "    def add_example(self, x, y):\n",
    "        \"\"\"\n",
    "        Add one example to the list of training examples.\n",
    "        :param x: The set of tokens\n",
    "        :param y: The label associated to this example\n",
    "        \"\"\"\n",
    "        self.count_class[y] += 1\n",
    "        count = self.count_token_given_class[y]\n",
    "        for token in x:\n",
    "            if token not in count:\n",
    "                count[token] = 0\n",
    "            count[token] += 1\n",
    "            if token not in self.count_token:\n",
    "                self.count_token[token] = 0\n",
    "            self.count_token[token] += 1\n",
    "\n",
    "    def add_examples(self, xs, ys):\n",
    "        \"\"\"\n",
    "        Add a list of examples to the list of training examples.\n",
    "        :param xs: A list of token sets\n",
    "        :param ys: A list of labels associated to the examples\n",
    "        \"\"\"\n",
    "        for x, y in zip(xs, ys):\n",
    "            self.add_example(x, y)\n",
    "\n",
    "    def classify(self, x_q):\n",
    "        scores = {}\n",
    "        for clazz in self.count_class:\n",
    "            p_class = self.count_class[clazz]\n",
    "            count = self.count_token_given_class[clazz]\n",
    "            p_tokens = 1.0\n",
    "            for token in x_q:\n",
    "                p_token_given_class = 0\n",
    "                if token in self.count_token:\n",
    "                    p_token_given_class = count[token]/self.count_token[token]\n",
    "                p_tokens *= p_token_given_class\n",
    "            scores[clazz] = p_class * p_tokens\n",
    "\n",
    "        max_score = -float('inf')\n",
    "        max_class = None\n",
    "        for clazz, score in scores.items():\n",
    "            if score > max_score:\n",
    "                max_score = score\n",
    "                max_class = clazz\n",
    "\n",
    "        return max_class"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now train this classifier."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "nb_clf = NaiveBayes()\n",
    "nb_clf.add_examples(xs_train_prep, ys_train)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can now classify any set of tokens."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "nb_clf.classify({'linux', 'is', 'beautiful'})"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We will now evaluate the performance of this classifier computing its accuracy on the train and test set."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "def accuracy(ys, ys_hat):\n",
    "    res = 0\n",
    "    for y, y_hat in zip(ys, ys_hat):\n",
    "        if y == y_hat:\n",
    "            res += 1\n",
    "    res /= len(ys)\n",
    "    return res\n",
    "\n",
    "ys_train_pred = []\n",
    "for x in xs_train_prep:\n",
    "    y_hat = nb_clf.classify(x)\n",
    "    ys_train_pred.append(y_hat)\n",
    "\n",
    "# preprocess test set\n",
    "xs_test_prep = [preprocess(x) for x in xs_test]\n",
    "\n",
    "ys_test_pred = []\n",
    "for x in xs_test_prep:\n",
    "    y_hat = nb_clf.classify(x)\n",
    "    ys_test_pred.append(y_hat)\n",
    "\n",
    "print('Train accuracy of NB', accuracy(ys_train, ys_train_pred))\n",
    "print('Test accuracy of NB', accuracy(ys_test, ys_test_pred))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "The performance of this Naive Bayes classifier is quite poor because if a token exists\n",
    "in the tested example that was not part of the training set,\n",
    "the score computed by the classifier for this example will always be 0, and\n",
    "the classifier will return one class at random."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "nb_clf.classify({'linux', 'is', 'beautiful', 'djsklajdklsajdkl'})"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In order to avoid this issue we need to smooth the probabilities using the m-estimate."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "class MEstimateNaiveBayes(NaiveBayes):\n",
    "\n",
    "    def __init__(self, m = 1):\n",
    "        super().__init__()\n",
    "        self.m = m\n",
    "\n",
    "    def classify(self, x_q):\n",
    "        scores = {}\n",
    "        p = 1.0/len(self.count_class)\n",
    "        for clazz in self.count_class:\n",
    "            logp_class = np.log(self.count_class[clazz])\n",
    "            count = self.count_token_given_class[clazz]\n",
    "            logp_tokens = 0.0\n",
    "            for token in x_q:\n",
    "                num_token_given_class = p * self.m\n",
    "                if token in count:\n",
    "                    num_token_given_class = np.log(count[token] + p * self.m)\n",
    "                den_token_given_class = - np.log(p)\n",
    "                if token in self.count_token:\n",
    "                    den_token_given_class = - np.log(self.count_token[token] + p)\n",
    "                logp_tokens += num_token_given_class + den_token_given_class\n",
    "            scores[clazz] = logp_class + logp_tokens\n",
    "\n",
    "        max_score = -float('inf')\n",
    "        max_class = None\n",
    "        for clazz, score in scores.items():\n",
    "            if score > max_score:\n",
    "                max_score = score\n",
    "                max_class = clazz\n",
    "\n",
    "        return max_class"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now train this classifier."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "mnb_clf = MEstimateNaiveBayes()\n",
    "mnb_clf.add_examples(xs_train_prep, ys_train)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now perform the same test as before."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "mnb_clf.classify({'linux', 'is', 'beautiful'})"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "mnb_clf.classify({'linux', 'is', 'beautiful', 'xzydsads'})"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "And evaluate the model."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "ys_train_pred = []\n",
    "for x in xs_train_prep:\n",
    "    y_hat = mnb_clf.classify(x)\n",
    "    ys_train_pred.append(y_hat)\n",
    "\n",
    "ys_test_pred = []\n",
    "for x in xs_test_prep:\n",
    "    y_hat = mnb_clf.classify(x)\n",
    "    ys_test_pred.append(y_hat)\n",
    "\n",
    "print('Train accuracy of MNB', accuracy(ys_train, ys_train_pred))\n",
    "print('Test accuracy of MNB', accuracy(ys_test, ys_test_pred))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Try to change the `m` parameter value to get a better result."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Naive Bayes in Scikit-Learn\n",
    "\n",
    "Now we will implement the same classifier but using the scikit-learn implementation,\n",
    "the `MultinomialNB` model."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "nb_clf = MultinomialNB()\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "In order to use this classifier we need to convert each message into a vector using the bag-of-words approach.\n",
    "Scikit-learn provides the `CountVecorizer` class, which converts a collection of texts into a\n",
    "matrix of token counts."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vectorizer = CountVectorizer()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We now vectorize the training set by using the method `fit_transform`."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "xs_train_prep = vectorizer.fit_transform(xs_train)\n"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can now train the Naive Bayes classifier."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "nb_clf.fit(xs_train_prep, ys_train)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now we evaluate this classifier plotting its confusion matrix.\n",
    "A confusion matrix is often used to describe the performance of a\n",
    "classification model on a test set.\n",
    "Each row of the matrix represents the instances in a predicted class, while each column represents\n",
    "the instances in an actual class."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "from matplotlib import pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "xs_test_prep = vectorizer.transform(xs_test)\n",
    "ys_test_pred = nb_clf.predict(xs_test_prep)\n",
    "\n",
    "mat = confusion_matrix(ys_test, ys_test_pred)\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 10))\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False,\n",
    "            xticklabels=classes, yticklabels=classes, ax=ax)\n",
    "plt.xlabel('true label')\n",
    "plt.ylabel('predicted label')"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "From this matrix we can assess which pair of classes get the most confused.\n",
    "\n",
    "We now evaluate its accuracy."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(ys_test, ys_test_pred))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Interpret the results. What is the difference between macro and micro avg?"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}