lesson3-cnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Subjectivity classification with CNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we implement the approched described in this [paper](https://arxiv.org/pdf/1408.5882.pdf) for classifiying sentences using Convolutional Neural Networks. In particular, we will classify sentences into \"subjective\" or \"objective\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subjectivity Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The subjectivity dataset has 5000 subjective and 5000 objective processed sentences. To get the data:\n",
    "```\n",
    "wget http://www.cs.cornell.edu/people/pabo/movie-review-data/rotten_imdb.tar.gz\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unpack_dataset():\n",
    "    ! wget http://www.cs.cornell.edu/people/pabo/movie-review-data/rotten_imdb.tar.gz\n",
    "    ! mkdir data\n",
    "    ! tar -xvf rotten_imdb.tar.gz -C data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('data/plot.tok.gt9.5000'),\n",
       " PosixPath('data/subjdata.README.1.0'),\n",
       " PosixPath('data/quote.tok.gt9.5000')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "PATH = Path(\"data\")\n",
    "list(PATH.iterdir())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the readme file:\n",
    "- quote.tok.gt9.5000 contains 5000 subjective sentences (or snippets)\n",
    "- plot.tok.gt9.5000 contains 5000 objective sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the movie begins in the past where a young boy named sam attempts to save celebi from a hunter . \r\n",
      "emerging from the human psyche and showing characteristics of abstract expressionism , minimalism and russian constructivism , graffiti removal has secured its place in the history of modern art while being created by artists who are unconscious of their artistic achievements . \r\n",
      "spurning her mother's insistence that she get on with her life , mary is thrown out of the house , rejected by joe , and expelled from school as she grows larger with child . \r\n",
      "amitabh can't believe the board of directors and his mind is filled with revenge and what better revenge than robbing the bank himself , ironic as it may sound . \r\n",
      "she , among others excentricities , talks to a small rock , gertrude , like if she was alive . \r\n",
      "this gives the girls a fair chance of pulling the wool over their eyes using their sexiness to poach any last vestige of common sense the dons might have had . \r\n",
      "styled after vh1's \" behind the music , \" this mockumentary profiles the rise and fall of an internet startup , called icevan . com . \r\n",
      "being blue is not his only predicament ; he also lacks the ability to outwardly express his emotions . \r\n",
      "the killer's clues are a perversion of biblical punishments for sins : stoning , burning , decapitation . \r\n",
      "david is a painter with painter's block who takes a job as a waiter to get some inspiration . \r\n"
     ]
    }
   ],
   "source": [
    "! head data/plot.tok.gt9.5000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## String cleaning functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from collections import defaultdict\n",
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_file(path):\n",
    "    \"\"\" Read file returns a shuttled list.\n",
    "    \"\"\"\n",
    "    with open(path, encoding = \"ISO-8859-1\") as f:\n",
    "        content = np.array(f.readlines())\n",
    "    return content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_vocab(content):\n",
    "    \"\"\"Computes Dict of counts of words.\n",
    "    \n",
    "    Computes the number of times a word is on a document.\n",
    "    \"\"\"\n",
    "    vocab = defaultdict(float)\n",
    "    for line in content:\n",
    "        words = set(line.split())\n",
    "        for word in words:\n",
    "            vocab[word] += 1\n",
    "    return vocab          "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split train and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_content = read_file(PATH/\"quote.tok.gt9.5000\")\n",
    "obj_content = read_file(PATH/\"plot.tok.gt9.5000\")\n",
    "sub_content = np.array([line.strip().lower() for line in sub_content])\n",
    "obj_content = np.array([line.strip().lower() for line in obj_content])\n",
    "sub_y = np.zeros(len(sub_content))\n",
    "obj_y = np.ones(len(obj_content))\n",
    "X = np.append(sub_content, obj_content)\n",
    "y = np.append(sub_y, obj_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array(['will god let her fall or give her a new path ?',\n",
       "        \"the director's twitchy sketchbook style and adroit perspective shifts grow wearisome amid leaden pacing and indifferent craftsmanship ( most notably wretched sound design ) .\",\n",
       "        \"welles groupie/scholar peter bogdanovich took a long time to do it , but he's finally provided his own broadside at publishing giant william randolph hearst .\",\n",
       "        'based on the 1997 john king novel of the same name with a rather odd synopsis : \" a first novel about a seasoned chelsea football club hooligan who represents a disaffected society operating by brutal rules .',\n",
       "        'yet , beneath an upbeat appearance , she is struggling desperately with the emotional and physical scars left by the attack .'],\n",
       "       dtype='<U691'), array([1., 0., 0., 1., 1.]))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[:5], y_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting vocab from training sets\n",
    "data_vocab = get_vocab(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#data_vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0.7924, -1.8440, -1.3614],\n",
       "         [-0.8127, -0.1256, -0.5875],\n",
       "         [ 0.9269, -1.3714, -1.0822],\n",
       "         [-1.5044,  0.2768, -0.3427],\n",
       "         [ 0.7924, -1.8440, -1.3614]]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# an Embedding module containing 10 (words) tensors of size 3\n",
    "embed = nn.Embedding(10, 3)\n",
    "a = torch.LongTensor([[1,2,4,5,1]])\n",
    "embed(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0487, -0.4587,  0.3779],\n",
       "        [ 0.7924, -1.8440, -1.3614],\n",
       "        [-0.8127, -0.1256, -0.5875],\n",
       "        [-0.5279,  0.0987, -0.6294],\n",
       "        [ 0.9269, -1.3714, -1.0822],\n",
       "        [-1.5044,  0.2768, -0.3427],\n",
       "        [-0.5540, -1.5207, -0.0946],\n",
       "        [-0.5207,  0.4329,  1.4940],\n",
       "        [-0.5993,  0.4766, -1.5010],\n",
       "        [ 1.4477,  1.3778, -1.3264]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## here is the randomly initialized embeddings\n",
    "embed.weight.data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing embedding layer with Glove embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get glove pre-trained embeddings:\n",
    "    `wget http://nlp.stanford.edu/data/glove.6B.zip`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section I am keeping the whole Glove embeddings. You can decide to keep just the words on your training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the 0.418 0.24968 -0.41242 0.1217 0.34527 -0.044457 -0.49688 -0.17862 -0.00066023 -0.6566 0.27843 -0.14767 -0.55677 0.14658 -0.0095095 0.011658 0.10204 -0.12792 -0.8443 -0.12181 -0.016801 -0.33279 -0.1552 -0.23131 -0.19181 -1.8823 -0.76746 0.099051 -0.42125 -0.19526 4.0071 -0.18594 -0.52287 -0.31681 0.00059213 0.0074449 0.17778 -0.15897 0.012041 -0.054223 -0.29871 -0.15749 -0.34758 -0.045637 -0.44251 0.18785 0.0027849 -0.18411 -0.11514 -0.78581\r\n",
      ", 0.013441 0.23682 -0.16899 0.40951 0.63812 0.47709 -0.42852 -0.55641 -0.364 -0.23938 0.13001 -0.063734 -0.39575 -0.48162 0.23291 0.090201 -0.13324 0.078639 -0.41634 -0.15428 0.10068 0.48891 0.31226 -0.1252 -0.037512 -1.5179 0.12612 -0.02442 -0.042961 -0.28351 3.5416 -0.11956 -0.014533 -0.1499 0.21864 -0.33412 -0.13872 0.31806 0.70358 0.44858 -0.080262 0.63003 0.32111 -0.46765 0.22786 0.36034 -0.37818 -0.56657 0.044691 0.30392\r\n"
     ]
    }
   ],
   "source": [
    "! head -2 data/glove.6B.50d.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would like to initialize the embeddings from our model with the pre-trained Glove embeddings. After initializing we should \"freeze\" the embeddings at least initially. The rationale is that we first want the network to learn weights for the other parameters that were randomly initialize. After that phase we could finetune the embeddings to our task. \n",
    "\n",
    "`embed.weight.requires_grad = False` freezes the embedding parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following code initializes the embedding. Here `V` is the vocabulary size and `D` is the embedding size. `pretrained_weight` is a numpy matrix of shape `(V, D)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loadGloveModel(gloveFile=PATH/\"glove.6B.300d.txt\"):\n",
    "    \"\"\" Loads word vectors into a dictionary.\"\"\"\n",
    "    f = open(gloveFile,'r')\n",
    "    word_vecs = {}\n",
    "    for line in f:\n",
    "        splitLine = line.split()\n",
    "        word = splitLine[0]\n",
    "        word_vecs[word] = np.array([float(val) for val in splitLine[1:]])\n",
    "    return word_vecs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_vecs = loadGloveModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "400000 21415\n"
     ]
    }
   ],
   "source": [
    "print(len(word_vecs.keys()), len(data_vocab.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def delete_rare_words(word_vecs, data_vocab, min_df=2):\n",
    "    \"\"\" Deletes rare words from data_vocab\n",
    "    \n",
    "    Deletes words from data_vocab if they are not in word_vecs\n",
    "    and don't have at least min_df occurrencies in data_vocab.\n",
    "    \"\"\"\n",
    "    words_delete = []\n",
    "    for word in data_vocab:\n",
    "        if data_vocab[word] < min_df and word not in word_vecs:\n",
    "            words_delete.append(word)\n",
    "    for word in words_delete: data_vocab.pop(word)\n",
    "    return data_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21415"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_vocab.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clean up issues here\n",
    "data_vocab = delete_rare_words(word_vecs, data_vocab, min_df=2) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18756"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_vocab.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_embedding_matrix(word_vecs, data_vocab, min_df=2, D=300):\n",
    "    \"\"\"Creates embedding matrix from word vectors. \"\"\"\n",
    "    data_vocab = delete_rare_words(word_vecs, data_vocab, min_df)\n",
    "    V = len(data_vocab.keys()) + 2\n",
    "    vocab2index = {}\n",
    "    W = np.zeros((V, D), dtype=\"float32\")\n",
    "    vocab = [\"\", \"UNK\"]\n",
    "    # adding a vector for padding\n",
    "    W[0] = np.zeros(D, dtype='float32')\n",
    "    # adding a vector for rare words \n",
    "    W[1] = np.random.uniform(-0.25,0.25,D)\n",
    "    vocab2index[\"UNK\"] = 1\n",
    "    i = 2\n",
    "    for word in data_vocab:\n",
    "        if word in word_vecs:\n",
    "            W[i] = word_vecs[word]\n",
    "            vocab2index[word] = i\n",
    "            vocab.append(word)\n",
    "            i += 1\n",
    "        else:\n",
    "            W[i] = np.random.uniform(-0.25,0.25,D)\n",
    "            vocab2index[word] = i\n",
    "            vocab.append(word)\n",
    "            i += 1   \n",
    "    return W, np.array(vocab), vocab2index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "pretrained_weight, vocab, vocab2index = create_embedding_matrix(word_vecs, data_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18758"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pretrained_weight) # note that index 0 is for padding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "        [ 0.0150, -0.0471,  0.1818,  ...,  0.2418,  0.1772,  0.0814],\n",
       "        [-0.0977, -0.5308, -0.2896,  ...,  0.4010, -0.1188,  0.1739],\n",
       "        ...,\n",
       "        [-0.1369, -0.0570, -0.1921,  ..., -0.2036, -0.4955, -0.2766],\n",
       "        [-0.3170, -0.4958,  0.3020,  ..., -0.1990,  0.0607, -0.1257],\n",
       "        [ 0.2028, -0.3397, -0.1055,  ...,  0.5841, -0.4893,  0.0245]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "D = 300\n",
    "V = len(pretrained_weight)\n",
    "emb = nn.Embedding(V, D)\n",
    "emb.weight.data.copy_(torch.from_numpy(pretrained_weight))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question: How many parameters do we have in this embedding matrix?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding training and validation sets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using 1D Convolutional neural networks as our model. CNNs assume a fixed input size so we need to assume a fixed size and truncate or pad the sentences as needed. Let's find a good value to set our sequence length to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_len = np.array([len(x.split()) for x in X_train])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile(x_len, 95) # let set the max sequence len to N=40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'will god let her fall or give her a new path ?'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# returns the index of the word or the index of \"UNK\" otherwise\n",
    "vocab2index.get(\"will\", vocab2index[\"UNK\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 8,  2,  3,  9,  4, 11, 12,  9, 10,  5,  6,  7])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([vocab2index.get(w, vocab2index[\"UNK\"]) for w in X_train[0].split()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_sentence(s, N=40):\n",
    "    enc = np.zeros(N, dtype=np.int32)\n",
    "    enc1 = np.array([vocab2index.get(w, vocab2index[\"UNK\"]) for w in s.split()])\n",
    "    l = min(N, len(enc1))\n",
    "    enc[:l] = enc1[:l]\n",
    "    return enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 8,  2,  3,  9,  4, 11, 12,  9, 10,  5,  6,  7,  0,  0,  0,  0,  0,\n",
       "        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "        0,  0,  0,  0,  0,  0], dtype=int32)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encode_sentence(X_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8000, 40)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train = np.vstack([encode_sentence(x) for x in X_train])\n",
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2000, 40)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test = np.vstack([encode_sentence(x) for x in X_test])\n",
    "x_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Playing and debugging CNN layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "V = len(pretrained_weight)\n",
    "D = 300\n",
    "N = 40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "        [ 0.0150, -0.0471,  0.1818,  ...,  0.2418,  0.1772,  0.0814],\n",
       "        [-0.0977, -0.5308, -0.2896,  ...,  0.4010, -0.1188,  0.1739],\n",
       "        ...,\n",
       "        [-0.1369, -0.0570, -0.1921,  ..., -0.2036, -0.4955, -0.2766],\n",
       "        [-0.3170, -0.4958,  0.3020,  ..., -0.1990,  0.0607, -0.1257],\n",
       "        [ 0.2028, -0.3397, -0.1055,  ...,  0.5841, -0.4893,  0.0245]])"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "emb = nn.Embedding(V, D)\n",
    "emb.weight.data.copy_(torch.from_numpy(pretrained_weight))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 40)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = x_train[:2]\n",
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 8,  2,  3,  9,  4, 11, 12,  9, 10,  5,  6,  7,  0,  0,  0,  0,  0,  0,\n",
       "          0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "          0,  0,  0,  0],\n",
       "        [14, 36, 32, 17, 30, 13, 35, 22, 21, 23, 27, 28, 24, 20, 13, 29, 15, 34,\n",
       "         25, 19, 33, 26, 16, 18, 31,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "          0,  0,  0,  0]])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = torch.LongTensor(x)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 40, 300])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1 = emb(x)\n",
    "x1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 300, 40])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1 = x1.transpose(1,2)  # needs to convert x to (batch, embedding_dim, sentence_len)\n",
    "x1.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_3 = nn.Conv1d(in_channels=D, out_channels=100, kernel_size=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "x3 = conv_3(x1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 100, 38])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x3.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_4 = nn.Conv1d(in_channels=D, out_channels=100, kernel_size=4)\n",
    "conv_5 = nn.Conv1d(in_channels=D, out_channels=100, kernel_size=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 100, 37]) torch.Size([2, 100, 36])\n"
     ]
    }
   ],
   "source": [
    "x4 = conv_4(x1)\n",
    "x5 = conv_5(x1)\n",
    "print(x4.size(), x5.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the convolution all apply to the same `x1`. How do we combine now the results of the convolutions? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 100, 1])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 100 3-gram detectors\n",
    "x3 = nn.ReLU()(x3)\n",
    "x3 = nn.MaxPool1d(kernel_size = 38)(x3)\n",
    "x3.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 100, 1])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 100 4-gram detectors\n",
    "x4 = nn.ReLU()(x4)\n",
    "x4 = nn.MaxPool1d(kernel_size = 37)(x4)\n",
    "x4.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 100, 1])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 100 5-gram detectors\n",
    "x5 = nn.ReLU()(x5)\n",
    "x5 = nn.MaxPool1d(kernel_size = 36)(x5)\n",
    "x5.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 100, 3])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# concatenate x3, x4, x5\n",
    "out = torch.cat([x3, x4, x5], 2)\n",
    "out.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 300])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = out.view(out.size(0), -1)\n",
    "out.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After this we have a fully connected network. Let's write a network that implements this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1D CNN model for sentence classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notation:\n",
    "* V -- vocabulary size\n",
    "* D -- embedding size\n",
    "* N -- MAX Sentence length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SentenceCNN(nn.Module):\n",
    "    \n",
    "    def __init__(self, V, D, glove_weights):\n",
    "        super(SentenceCNN, self).__init__()\n",
    "        self.glove_weights = glove_weights\n",
    "        self.embedding = nn.Embedding(V, D, padding_idx=0)\n",
    "        self.embedding.weight.data.copy_(torch.from_numpy(self.glove_weights))\n",
    "        self.embedding.weight.requires_grad = False ## freeze embeddings\n",
    "\n",
    "        self.conv_3 = nn.Conv1d(in_channels=D, out_channels=100, kernel_size=3)\n",
    "        self.conv_4 = nn.Conv1d(in_channels=D, out_channels=100, kernel_size=4)\n",
    "        self.conv_5 = nn.Conv1d(in_channels=D, out_channels=100, kernel_size=5)\n",
    "        \n",
    "        self.dropout = nn.Dropout(p=0.5)\n",
    "        self.fc = nn.Linear(300, 1)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.embedding(x)\n",
    "        x = x.transpose(1,2)\n",
    "        x3 = F.relu(self.conv_3(x))\n",
    "        x4 = F.relu(self.conv_4(x))\n",
    "        x5 = F.relu(self.conv_5(x))\n",
    "        x3 = nn.MaxPool1d(kernel_size = 38)(x3)\n",
    "        x4 = nn.MaxPool1d(kernel_size = 37)(x4)\n",
    "        x5 = nn.MaxPool1d(kernel_size = 36)(x5)\n",
    "        out = torch.cat([x3, x4, x5], 2)\n",
    "        out = out.view(out.size(0), -1)\n",
    "        out = self.dropout(out)\n",
    "        return self.fc(out)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "V = len(pretrained_weight)\n",
    "D = 300\n",
    "N = 40\n",
    "model = SentenceCNN(V, D, glove_weights=pretrained_weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10, 40)\n"
     ]
    }
   ],
   "source": [
    "# testing the model\n",
    "x = x_train[:10]\n",
    "print(x.shape)\n",
    "x = torch.LongTensor(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 1])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat = model(x)\n",
    "y_hat.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that I am not bodering with mini-batches since our dataset is small."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "#def train(model, x_train, y_train):\n",
    "model = SentenceCNN(V, D, glove_weights=pretrained_weight) #.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_metrics(m):\n",
    "    model.eval()\n",
    "    x = torch.LongTensor(x_test) #.cuda()\n",
    "    y = torch.Tensor(y_test).unsqueeze(1) #).cuda()\n",
    "    y_hat = m(x)\n",
    "    loss = F.binary_cross_entropy_with_logits(y_hat, y)\n",
    "    y_pred = y_hat > 0\n",
    "    correct = (y_pred.float() == y).float().sum()\n",
    "    accuracy = correct/y_pred.shape[0]\n",
    "    print(\"test loss %.3f and accuracy %.3f\" % (loss.item(), accuracy.item()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test loss 0.695 and accuracy 0.484\n"
     ]
    }
   ],
   "source": [
    "# accuracy of a random model should be around 0.5\n",
    "test_metrics(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this filters parameters with p.requires_grad=True\n",
    "parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "optimizer = torch.optim.Adam(parameters, lr=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epocs(model, epochs=10, lr=0.01):\n",
    "    parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    optimizer = torch.optim.Adam(parameters, lr=lr)\n",
    "    model.train()\n",
    "    for i in range(epochs):\n",
    "        x = torch.LongTensor(x_train)  #.cuda()\n",
    "        y = torch.Tensor(y_train).unsqueeze(1)\n",
    "        y_hat = model(x)\n",
    "        loss = F.binary_cross_entropy_with_logits(y_hat, y)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        print(loss.item())\n",
    "    test_metrics(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7023484110832214\n",
      "0.46630650758743286\n",
      "0.5858073830604553\n",
      "0.35865071415901184\n",
      "0.45489776134490967\n",
      "0.33390170335769653\n",
      "0.28767043352127075\n",
      "0.33160310983657837\n",
      "0.3383573889732361\n",
      "0.28842759132385254\n",
      "test loss 0.299 and accuracy 0.874\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10, lr=0.005)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.25689801573753357\n",
      "0.2492968738079071\n",
      "0.24048243463039398\n",
      "0.23228466510772705\n",
      "0.22797048091888428\n",
      "0.22122417390346527\n",
      "0.21561495959758759\n",
      "0.21407529711723328\n",
      "0.2064850628376007\n",
      "0.2005021870136261\n",
      "test loss 0.253 and accuracy 0.891\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10, lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.19603541493415833\n",
      "0.19891881942749023\n",
      "0.19023893773555756\n",
      "0.19017376005649567\n",
      "0.1822727471590042\n",
      "0.1785450130701065\n",
      "0.17655792832374573\n",
      "0.17357498407363892\n",
      "0.16655944287776947\n",
      "0.16596028208732605\n",
      "test loss 0.240 and accuracy 0.903\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10, lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[torch.Size([100, 300, 3]), torch.Size([100]), torch.Size([100, 300, 4]), torch.Size([100]), torch.Size([100, 300, 5]), torch.Size([100]), torch.Size([1, 300]), torch.Size([1])]\n"
     ]
    }
   ],
   "source": [
    "# how to figure out the parameters\n",
    "parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "print([p.size() for p in parameters])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unfreezing the embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unfreezing the embeddings\n",
    "model.embedding.weight.requires_grad = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[torch.Size([18758, 300]), torch.Size([100, 300, 3]), torch.Size([100]), torch.Size([100, 300, 4]), torch.Size([100]), torch.Size([100, 300, 5]), torch.Size([100]), torch.Size([1, 300]), torch.Size([1])]\n"
     ]
    }
   ],
   "source": [
    "parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "print([p.size() for p in parameters])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.16300448775291443\n",
      "0.18081034719944\n",
      "0.1502874493598938\n",
      "0.14908277988433838\n",
      "0.15106000006198883\n",
      "0.13690626621246338\n",
      "0.1242230236530304\n",
      "0.12123315036296844\n",
      "0.11981801688671112\n",
      "0.11310970038175583\n",
      "test loss 0.230 and accuracy 0.906\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10, lr=0.001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.1044512689113617\n",
      "0.09997980296611786\n",
      "0.10306422412395477\n",
      "0.09937475621700287\n",
      "0.09839469194412231\n",
      "0.0987946018576622\n",
      "0.09739076346158981\n",
      "0.0969555452466011\n",
      "0.09545455873012543\n",
      "0.09494327753782272\n",
      "test loss 0.222 and accuracy 0.908\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10, lr=0.0001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Improve tokenization\n",
    "* Apply this method to a different dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CNN is adapted from here https://github.com/junwang4/CNN-sentence-classification-pytorch-2017/blob/master/cnn_pytorch.py.\n",
    "Code for the original paper can be found here https://github.com/yoonkim/CNN_sentence/blob/master/process_data.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}