cbow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\" style=\"margin-top: 1em;\"><ul class=\"toc-item\"><li><span><a href=\"#Text-Classification\" data-toc-modified-id=\"Text-Classification-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Text Classification</a></span><ul class=\"toc-item\"><li><span><a href=\"#Subjectivity-Dataset\" data-toc-modified-id=\"Subjectivity-Dataset-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Subjectivity Dataset</a></span></li><li><span><a href=\"#Tokenization\" data-toc-modified-id=\"Tokenization-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Tokenization</a></span><ul class=\"toc-item\"><li><span><a href=\"#Simple-Tokenization\" data-toc-modified-id=\"Simple-Tokenization-1.2.1\"><span class=\"toc-item-num\">1.2.1&nbsp;&nbsp;</span>Simple Tokenization</a></span></li><li><span><a href=\"#Much-better-tokenization-with-Spacy\" data-toc-modified-id=\"Much-better-tokenization-with-Spacy-1.2.2\"><span class=\"toc-item-num\">1.2.2&nbsp;&nbsp;</span>Much better tokenization with Spacy</a></span></li></ul></li><li><span><a href=\"#Split-dataset-in-train-and-validation\" data-toc-modified-id=\"Split-dataset-in-train-and-validation-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Split dataset in train and validation</a></span></li><li><span><a href=\"#Word-to-index-mapping\" data-toc-modified-id=\"Word-to-index-mapping-1.4\"><span class=\"toc-item-num\">1.4&nbsp;&nbsp;</span>Word to index mapping</a></span></li><li><span><a href=\"#Sentence-encoding\" data-toc-modified-id=\"Sentence-encoding-1.5\"><span class=\"toc-item-num\">1.5&nbsp;&nbsp;</span>Sentence encoding</a></span></li><li><span><a href=\"#Embedding-layer\" data-toc-modified-id=\"Embedding-layer-1.6\"><span class=\"toc-item-num\">1.6&nbsp;&nbsp;</span>Embedding layer</a></span></li><li><span><a href=\"#Continuous-Bag-of-Words-Model\" data-toc-modified-id=\"Continuous-Bag-of-Words-Model-1.7\"><span class=\"toc-item-num\">1.7&nbsp;&nbsp;</span>Continuous Bag of Words Model</a></span></li></ul></li><li><span><a href=\"#Training-the-CBOW-model\" data-toc-modified-id=\"Training-the-CBOW-model-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Training the CBOW model</a></span></li><li><span><a href=\"#Data-loaders-for-SGD\" data-toc-modified-id=\"Data-loaders-for-SGD-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Data loaders for SGD</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pytorch libraries\n",
    "%matplotlib inline\n",
    "import torch \n",
    "import torch.autograd as autograd \n",
    "import torch.nn as nn \n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification\n",
    "In this part of the tutorial we develop a continuous bag of words (CBOW) model for a text classification task described [here]( https://people.cs.umass.edu/~miyyer/pubs/2015_acl_dan.pdf). The CBOW model was first described [here](https://arxiv.org/pdf/1301.3781.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subjectivity Dataset\n",
    "The subjectivity dataset has 5000 subjective and 5000 objective processed sentences. To get the data:\n",
    "```\n",
    "wget http://www.cs.cornell.edu/people/pabo/movie-review-data/rotten_imdb.tar.gz\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def unpack_dataset():\n",
    "    ! wget http://www.cs.cornell.edu/people/pabo/movie-review-data/rotten_imdb.tar.gz\n",
    "    ! mkdir data\n",
    "    ! tar -xvf rotten_imdb.tar.gz -C data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-02-19 11:49:47--  http://www.cs.cornell.edu/people/pabo/movie-review-data/rotten_imdb.tar.gz\n",
      "Resolving www.cs.cornell.edu (www.cs.cornell.edu)... 132.236.207.20\n",
      "Connecting to www.cs.cornell.edu (www.cs.cornell.edu)|132.236.207.20|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 519599 (507K) [application/x-gzip]\n",
      "Saving to: ‘rotten_imdb.tar.gz’\n",
      "\n",
      "rotten_imdb.tar.gz  100%[===================>] 507.42K   558KB/s    in 0.9s    \n",
      "\n",
      "2020-02-19 11:49:49 (558 KB/s) - ‘rotten_imdb.tar.gz’ saved [519599/519599]\n",
      "\n",
      "x quote.tok.gt9.5000\n",
      "x plot.tok.gt9.5000\n",
      "x subjdata.README.1.0\n"
     ]
    }
   ],
   "source": [
    "unpack_dataset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "plot.tok.gt9.5000   quote.tok.gt9.5000  subjdata.README.1.0\r\n"
     ]
    }
   ],
   "source": [
    "!ls data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the movie begins in the past where a young boy named sam attempts to save celebi from a hunter . \r\n",
      "emerging from the human psyche and showing characteristics of abstract expressionism , minimalism and russian constructivism , graffiti removal has secured its place in the history of modern art while being created by artists who are unconscious of their artistic achievements . \r\n"
     ]
    }
   ],
   "source": [
    "! head -2 data/plot.tok.gt9.5000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('data/plot.tok.gt9.5000'),\n",
       " PosixPath('data/subjdata.README.1.0'),\n",
       " PosixPath('data/quote.tok.gt9.5000')]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "PATH = Path(\"data\")\n",
    "list(PATH.iterdir())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "Tokenization is the task of chopping up text into pieces, called tokens.\n",
    "\n",
    "spaCy is an open-source software library for advanced Natural Language Processing. Here we will use it for tokenization.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need each line in the file \n",
    "def read_file(path):\n",
    "    \"\"\" Read file returns a list of lines.\n",
    "    \"\"\"\n",
    "    with open(path, encoding = \"ISO-8859-1\") as f:\n",
    "        content = f.readlines()\n",
    "    return content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_lines = read_file(PATH/\"plot.tok.gt9.5000\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the movie begins in the past where a young boy named sam attempts to save celebi from a hunter . \\n'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obj_lines[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['the', 'movie', 'begins', 'in', 'the', 'past', 'where', 'a',\n",
       "       'young', 'boy', 'named', 'sam', 'attempts', 'to', 'save', 'celebi',\n",
       "       'from', 'a', 'hunter', '.'], dtype='<U8')"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(obj_lines[0].strip().lower().split(\" \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Much better tokenization with Spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install -U spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first time run this\n",
    "#!python3 -m spacy download en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "tok = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "obj_lines = read_file(PATH/\"plot.tok.gt9.5000\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5000"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(obj_lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the movie begins in the past where a young boy named sam attempts to save celebi from a hunter . \\n'"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obj_lines[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = tok(obj_lines[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([the, movie, begins, in, the, past, where, a, young, boy, named,\n",
       "       sam, attempts, to, save, celebi, from, a, hunter, ., \n",
       "], dtype=object)"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([x for x in test])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split dataset in train and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_content = read_file(PATH/\"quote.tok.gt9.5000\")\n",
    "obj_content = read_file(PATH/\"plot.tok.gt9.5000\")\n",
    "sub_content = np.array([line.strip().lower() for line in sub_content])\n",
    "obj_content = np.array([line.strip().lower() for line in obj_content])\n",
    "sub_y = np.zeros(len(sub_content))\n",
    "obj_y = np.ones(len(obj_content))\n",
    "X = np.append(sub_content, obj_content)\n",
    "y = np.append(sub_y, obj_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('smart and alert , thirteen conversations about one thing is a small gem .',\n",
       " 0.0)"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[0], y[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array(['will god let her fall or give her a new path ?',\n",
       "        \"the director's twitchy sketchbook style and adroit perspective shifts grow wearisome amid leaden pacing and indifferent craftsmanship ( most notably wretched sound design ) .\",\n",
       "        \"welles groupie/scholar peter bogdanovich took a long time to do it , but he's finally provided his own broadside at publishing giant william randolph hearst .\",\n",
       "        'based on the 1997 john king novel of the same name with a rather odd synopsis : \" a first novel about a seasoned chelsea football club hooligan who represents a disaffected society operating by brutal rules .',\n",
       "        'yet , beneath an upbeat appearance , she is struggling desperately with the emotional and physical scars left by the attack .'],\n",
       "       dtype='<U691'), array([1., 0., 0., 1., 1.]))"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[:5], y_train[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word to index mapping\n",
    "In interest of time we will tokenize without spaCy. Here we will compute a vocabulary of words based on the training set and a mapping from word to an index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_vocab(content):\n",
    "    \"\"\"Computes Dict of counts of words.\n",
    "    \n",
    "    Computes the number of times a word is on a document.\n",
    "    \"\"\"\n",
    "    vocab = defaultdict(float)\n",
    "    for line in content:\n",
    "        words = set(line.split())\n",
    "        for word in words:\n",
    "            vocab[word] += 1\n",
    "    return vocab      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Getting the vocabulary from the training set\n",
    "word_count = get_vocab(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "#word_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21415"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(word_count.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's delete words that are very infrequent\n",
    "for word in list(word_count):\n",
    "    if word_count[word] < 5:\n",
    "        del word_count[word]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4065"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(word_count.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Finally we need an index for each word in the vocab\n",
    "vocab2index = {\"<PAD>\":0, \"UNK\":1} # init with padding and unknown\n",
    "words = [\"<PAD>\", \"UNK\"]\n",
    "for word in word_count:\n",
    "    vocab2index[word] = len(words)\n",
    "    words.append(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [],
   "source": [
    "#vocab2index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence encoding\n",
    "Here we encode each sentence as a sequence of indices corresponding to each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_len = np.array([len(x.split()) for x in X_train])\n",
    "x_val_len = np.array([len(x.split()) for x in X_val])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43.0"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.percentile(x_train_len, 95) # let set the max sequence len to N=40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'will god let her fall or give her a new path ?'"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# returns the index of the word or the index of \"UNK\" otherwise\n",
    "vocab2index.get(\"?\", vocab2index[\"UNK\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4,  3,  5, 11, 12,  8,  9, 11, 10,  2,  6,  7])"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([vocab2index.get(w, vocab2index[\"UNK\"]) for w in X_train[0].split()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_sentence(s, N=40):\n",
    "    enc = np.zeros(N, dtype=np.int32)\n",
    "    enc1 = np.array([vocab2index.get(w, vocab2index[\"UNK\"]) for w in s.split()])\n",
    "    l = min(N, len(enc1))\n",
    "    enc[:l] = enc1[:l]\n",
    "    return enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4,  3,  5, 11, 12,  8,  9, 11, 10,  2,  6,  7,  0,  0,  0,  0,  0,\n",
       "        0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "        0,  0,  0,  0,  0,  0], dtype=int32)"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encode_sentence(X_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_len = np.minimum(x_train_len, 40)\n",
    "x_val_len = np.minimum(x_val_len, 40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(8000, 40)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train = np.vstack([encode_sentence(x) for x in X_train])\n",
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2000, 40)"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_val = np.vstack([encode_sentence(x) for x in X_val])\n",
    "x_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embedding layer\n",
    "Most deep learning models use a dense vectors of real numbers as representation of words (word embeddings), as opposed to a one-hot encoding representations. The module torch.nn.Embedding is used to represent word embeddings. It takes two arguments: the vocabulary size, and the dimensionality of the embeddings. The embeddings are initialized with random vectors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.0000,  0.0000,  0.0000,  0.0000],\n",
       "        [-0.6142,  0.2136, -0.6799,  0.6064],\n",
       "        [ 1.3913,  0.3764, -0.9674, -0.9030],\n",
       "        [-0.1504, -0.0164,  2.1774,  0.8860],\n",
       "        [ 1.4216, -0.4580,  1.3691,  0.8674],\n",
       "        [-0.7108,  1.2483,  0.5496, -1.5263],\n",
       "        [-1.2288, -0.6853,  0.2598, -0.9845],\n",
       "        [ 0.4001,  0.3452,  0.5711,  0.5329],\n",
       "        [ 0.5904, -0.7116, -0.1716, -0.2356],\n",
       "        [ 1.4376,  1.5275, -0.4301,  0.3887]], requires_grad=True)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# an Embedding module containing 10 words with embedding size 4\n",
    "# embedding will be initialized at random\n",
    "embed = nn.Embedding(10, 4, padding_idx=0)\n",
    "embed.weight"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `padding_idx` has embedding vector 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.6142,  0.2136, -0.6799,  0.6064],\n",
       "         [ 1.4216, -0.4580,  1.3691,  0.8674],\n",
       "         [-0.6142,  0.2136, -0.6799,  0.6064],\n",
       "         [-0.7108,  1.2483,  0.5496, -1.5263],\n",
       "         [-0.6142,  0.2136, -0.6799,  0.6064],\n",
       "         [ 0.0000,  0.0000,  0.0000,  0.0000]]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# given a list of ids we can \"look up\" the embedding corresponing to each id\n",
    "# can you see that some vectors are the same?\n",
    "a = torch.LongTensor([[1,4,1,5,1,0]])\n",
    "embed(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This would be the representation of a sentence with words with indices [1,4,1,5,1] and a padding at the end. Bellow we have an example in which we have two sentences. the first sentence has length 3 and the last sentence has length 2. In order to use a tensor we use padding at the end of the second sentence. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = torch.LongTensor([[1,4,1], [1,3,0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model takes an average of the word embedding of each word. Here is how we do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = torch.FloatTensor([3, 2]) # here is the size of the vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[-0.6142,  0.2136, -0.6799,  0.6064],\n",
       "         [ 1.4216, -0.4580,  1.3691,  0.8674],\n",
       "         [-0.6142,  0.2136, -0.6799,  0.6064]],\n",
       "\n",
       "        [[-0.6142,  0.2136, -0.6799,  0.6064],\n",
       "         [-0.1504, -0.0164,  2.1774,  0.8860],\n",
       "         [ 0.0000,  0.0000,  0.0000,  0.0000]]], grad_fn=<EmbeddingBackward>)"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embed(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1932, -0.0307,  0.0092,  2.0803],\n",
       "        [-0.7646,  0.1972,  1.4974,  1.4924]], grad_fn=<SumBackward1>)"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embed(a).sum(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0644, -0.0102,  0.0031,  0.6934],\n",
       "        [-0.3823,  0.0986,  0.7487,  0.7462]], grad_fn=<DivBackward0>)"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum_embs = embed(a).sum(dim=1) \n",
    "sum_embs/ s.view(s.shape[0], 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Continuous Bag of Words Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CBOW(nn.Module):\n",
    "    def __init__(self, vocab_size, emb_size=100):\n",
    "        super(CBOW, self).__init__()\n",
    "        self.word_emb = nn.Embedding(vocab_size, emb_size, padding_idx=0)\n",
    "        self.linear = nn.Linear(emb_size, 1)\n",
    "        \n",
    "    def forward(self, x, s):\n",
    "        x = self.word_emb(x)\n",
    "        x = x.sum(dim=1)/ s\n",
    "        x = self.linear(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CBOW(vocab_size=5, emb_size=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter containing:\n",
       "tensor([[ 0.0000,  0.0000,  0.0000],\n",
       "        [ 0.4837,  0.0118, -1.5768],\n",
       "        [ 0.5992,  0.5553,  0.8514],\n",
       "        [ 0.8974, -0.0957,  0.5334],\n",
       "        [-1.9593, -0.5764, -0.4522]], requires_grad=True)"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.word_emb.weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.9028],\n",
       "        [-0.7842]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s = s.view(s.shape[0], 1)\n",
    "model(a, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training the CBOW model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4067\n"
     ]
    }
   ],
   "source": [
    "V = len(words)\n",
    "model = CBOW(vocab_size=V, emb_size=50)\n",
    "print(V)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "def val_metrics(model):\n",
    "    model.eval()\n",
    "    x = torch.LongTensor(x_val) #.cuda()\n",
    "    y = torch.Tensor(y_val).unsqueeze(1) #).cuda()\n",
    "    s = torch.Tensor(x_val_len).view(x_val_len.shape[0], 1)\n",
    "    y_hat = model(x, s)\n",
    "    loss = F.binary_cross_entropy_with_logits(y_hat, y)\n",
    "    y_pred = y_hat > 0\n",
    "    correct = (y_pred.float() == y).float().sum()\n",
    "    accuracy = correct/y_pred.shape[0]\n",
    "    return loss.item(), accuracy.item()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.7011229991912842, 0.5065000057220459)"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# accuracy of a random model should be around 0.5\n",
    "val_metrics(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epocs(model, epochs=10, lr=0.01):\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "    for i in range(epochs):\n",
    "        model.train()\n",
    "        x = torch.LongTensor(x_train)  #.cuda()\n",
    "        y = torch.Tensor(y_train).unsqueeze(1)\n",
    "        s = torch.Tensor(x_train_len).view(x_train_len.shape[0], 1)\n",
    "        y_hat = model(x, s)\n",
    "        loss = F.binary_cross_entropy_with_logits(y_hat, y)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        val_loss, val_accuracy = val_metrics(model)\n",
    "        print(\"train_loss %.3f val_loss %.3f val_accuracy %.3f\" % (loss.item(), val_loss, val_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_loss 0.705 val_loss 0.712 val_accuracy 0.500\n",
      "train_loss 0.701 val_loss 0.619 val_accuracy 0.728\n",
      "train_loss 0.613 val_loss 0.552 val_accuracy 0.795\n",
      "train_loss 0.542 val_loss 0.467 val_accuracy 0.854\n",
      "train_loss 0.448 val_loss 0.381 val_accuracy 0.882\n",
      "train_loss 0.351 val_loss 0.326 val_accuracy 0.872\n",
      "train_loss 0.283 val_loss 0.277 val_accuracy 0.892\n",
      "train_loss 0.225 val_loss 0.251 val_accuracy 0.904\n",
      "train_loss 0.189 val_loss 0.247 val_accuracy 0.903\n",
      "train_loss 0.167 val_loss 0.243 val_accuracy 0.910\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10, lr=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_loss 0.144 val_loss 0.241 val_accuracy 0.913\n",
      "train_loss 0.139 val_loss 0.240 val_accuracy 0.913\n",
      "train_loss 0.136 val_loss 0.240 val_accuracy 0.910\n",
      "train_loss 0.133 val_loss 0.240 val_accuracy 0.909\n",
      "train_loss 0.130 val_loss 0.239 val_accuracy 0.910\n",
      "train_loss 0.126 val_loss 0.238 val_accuracy 0.910\n",
      "train_loss 0.123 val_loss 0.238 val_accuracy 0.910\n",
      "train_loss 0.119 val_loss 0.238 val_accuracy 0.911\n",
      "train_loss 0.116 val_loss 0.238 val_accuracy 0.910\n",
      "train_loss 0.113 val_loss 0.238 val_accuracy 0.909\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10, lr=0.01)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data loaders for SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nearly all of deep learning is powered by one very important algorithm: **stochastic gradient descent (SGD)**. SGD can be seeing as an approximation of **gradient descent** (GD). In GD you have to run through *all* the samples in your training set to do a single itaration. In SGD you use *only one* or *a subset*  of training samples to do the update for a parameter in a particular iteration. The subset use in every iteration is called a **batch** or **minibatch**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we are going to create a data loader. The data loader provides the following features:\n",
    "* Batching the data\n",
    "* Shuffling the data\n",
    "* Load the data in parallel using multiprocessing workers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_sentence2(s, N=40):\n",
    "    enc = np.zeros(N, dtype=np.int32)\n",
    "    enc1 = np.array([vocab2index.get(w, vocab2index[\"UNK\"]) for w in s.split()])\n",
    "    l = min(N, len(enc1))\n",
    "    enc[:l] = enc1[:l]\n",
    "    return enc, l"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 4,  3,  5, 11, 12,  8,  9, 11, 10,  2,  6,  7,  0,  0,  0,  0,  0,\n",
       "         0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,  0,\n",
       "         0,  0,  0,  0,  0,  0], dtype=int32), 12)"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encode_sentence2(X_train[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SubjectivityDataset(Dataset):\n",
    "    def __init__(self, X, y):\n",
    "        self.x = X\n",
    "        self.y = y\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.y)\n",
    "    \n",
    "    def __getitem__(self, idx):\n",
    "        x = self.x[idx]\n",
    "        x, s = encode_sentence2(x)\n",
    "        return x, self.y[idx], s\n",
    "    \n",
    "sub_dataset_train = SubjectivityDataset(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = DataLoader(sub_dataset_train, batch_size=5, shuffle=True)\n",
    "x, y, s = next(iter(train_loader))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[1098,  171,   10,  594,  118,   28,   13,   39, 1386,  417,   51,  130,\n",
       "          1891,    1,   51,  424, 2638,  959,   51,    1, 1895,   28,   29,  929,\n",
       "            90, 3126,  696,   10,    1,   51,    1,    1,  171,   14,  113,   20,\n",
       "             0,    0,    0,    0],\n",
       "         [  14,  441,  196,   42,   90,    1,   74,  188,  187, 1817,   14,  657,\n",
       "            20,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "             0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "             0,    0,    0,    0],\n",
       "         [  14,  505,   51,  617, 3557,   96,  433, 1077,  796, 1740,  264,   13,\n",
       "           264,   28,  212, 1432,   14, 2930,   20,    0,    0,    0,    0,    0,\n",
       "             0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "             0,    0,    0,    0],\n",
       "         [  14, 1198,   51,  438,    1,    1, 2929, 2101,   81,  240,    1, 2930,\n",
       "            74,  181, 1939,  414,   42,   90, 2262,   20,    0,    0,    0,    0,\n",
       "             0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "             0,    0,    0,    0],\n",
       "         [   1, 2434,   10,    1,   28, 3068, 2994,   28,   57,    1,  267,   65,\n",
       "             1,    1,    1,   13,   14, 1092,  134,  706,    1,    1,   20,    0,\n",
       "             0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "             0,    0,    0,    0]], dtype=torch.int32),\n",
       " tensor([0., 1., 0., 0., 0.]),\n",
       " tensor([36, 13, 19, 20, 23]))"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, y, s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CBOW(vocab_size=V, emb_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = DataLoader(sub_dataset_train, batch_size=500, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epocs(model, epochs=10, lr=0.01):\n",
    "    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n",
    "    for i in range(epochs):\n",
    "        total_loss = 0\n",
    "        total = 0\n",
    "        model.train()\n",
    "        for x, y, s in train_loader:\n",
    "            x = x.type(torch.LongTensor)  #.cuda()\n",
    "            y = y.type(torch.FloatTensor).unsqueeze(1)\n",
    "            s = s.type(torch.Tensor).view(s.shape[0], 1)\n",
    "            y_hat = model(x, s)\n",
    "            loss = F.binary_cross_entropy_with_logits(y_hat, y)\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            total_loss += x.size(0)*loss.item()\n",
    "            total += x.size(0)\n",
    "        train_loss = total_loss/total\n",
    "        val_loss, val_accuracy = val_metrics(model)\n",
    "        \n",
    "        print(\"train_loss %.3f val_loss %.3f val_accuracy %.3f\" % (train_loss, val_loss, val_accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train_loss 0.644 val_loss 0.579 val_accuracy 0.775\n",
      "train_loss 0.493 val_loss 0.416 val_accuracy 0.850\n",
      "train_loss 0.332 val_loss 0.307 val_accuracy 0.887\n",
      "train_loss 0.235 val_loss 0.258 val_accuracy 0.898\n",
      "train_loss 0.182 val_loss 0.238 val_accuracy 0.906\n",
      "train_loss 0.148 val_loss 0.231 val_accuracy 0.910\n",
      "train_loss 0.125 val_loss 0.230 val_accuracy 0.910\n",
      "train_loss 0.107 val_loss 0.233 val_accuracy 0.910\n",
      "train_loss 0.092 val_loss 0.240 val_accuracy 0.908\n",
      "train_loss 0.080 val_loss 0.246 val_accuracy 0.909\n"
     ]
    }
   ],
   "source": [
    "train_epocs(model, epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "nav_menu": {},
  "toc": {
   "nav_menu": {
    "height": "116px",
    "width": "251px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}