Drop_Dataset_with_Seq2Seq_with_attention.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7-BF6VfBruup"
   },
   "source": [
    "Drop dataset if a Question Answering datset focusing on reading comprehension of models. Each datapoint has:\n",
    "\n",
    "1.   passage: which we refer as context in this notebook. A passage is derived from wikipedia and its corresponding wikipedia url is mentioned in the key 'wiki_url.\n",
    "2.   Qa_pairs: For a single passage, the dataset has mutiple questions. Each question has the answer of the following type:\n",
    "\n",
    "  *   Number: When the answer is a specific number\n",
    "  *   Date: When the answer is a date\n",
    "  *   Span: When the answer is a text like names or places\n",
    "\n",
    "What makes this datset interesting is the presence of multiple questions for the same context. So lets say a batch of 32 contexts might have total 900 questions on them total or 560. This varying number of number of questions for each passage brings up design issues.\n",
    "\n",
    "A goto choice could be that for each question you create an example like\n",
    "<context,question,answer>\n",
    "\n",
    "But this would mean passing the same context again and again through the encoder for questions belonging to the same context which would take time for longer contexts since we are modelling the problem as a seq2seq problem. To avoid this, we create an example like the dataset itself.\n",
    "<context,(question1..question_n),(answer1..answer_n)>\n",
    "\n",
    "This required making customField and a customExample class. To have a batch of contexts and yet allow different number of questions in each batch.\n",
    "\n",
    "\n",
    "The architecture contains three RNN cells. The procedure looks like this:\n",
    "1. We pass context to an encoder.\n",
    "2. We then pass the question through a different encoder cell which takes care of different number of questions for each batch of contexts. We use the last hidden vector of a context passed through context encoder as an input to cell for its question.\n",
    "3. We then use the hidden vectors of the question encoder as an input to the 'answer decoder'.\n",
    "\n",
    "We are initially only focussing on loss and perplexity and not on exact matches of the answer.\n",
    "\n",
    "In this notebook, we add attention components to the 'answer decoder'. We do not use attention from the output of the question encoder RNN because questions in the dataset don't have much information. We use attention on the hidden states of 'context encoder' because the answer is part or derived from the context. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "9iEKPzfocn5K"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import json\n",
    "from torchtext import data\n",
    "from itertools import chain\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "import time\n",
    "from torch.nn import Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "R__vGEjydJeD",
    "outputId": "af36bc2d-2603-43f9-a197-2c8ef1059978"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  drop_dataset.zip\n",
      "  inflating: drop_dataset/drop_dataset_dev.json  \n",
      "  inflating: drop_dataset/drop_dataset_train.json  \n",
      "  inflating: drop_dataset/license.txt  \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "--2020-12-29 12:14:28--  https://s3-us-west-2.amazonaws.com/allennlp/datasets/drop/drop_dataset.zip\n",
      "Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.180.216\n",
      "Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.180.216|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 8308692 (7.9M) [application/zip]\n",
      "Saving to: ‘drop_dataset.zip’\n",
      "\n",
      "     0K .......... .......... .......... .......... ..........  0% 3.18M 2s\n",
      "    50K .......... .......... .......... .......... ..........  1% 3.25M 2s\n",
      "   100K .......... .......... .......... .......... ..........  1% 3.28M 2s\n",
      "   150K .......... .......... .......... .......... ..........  2%  224M 2s\n",
      "   200K .......... .......... .......... .......... ..........  3% 3.35M 2s\n",
      "   250K .......... .......... .......... .......... ..........  3%  167M 2s\n",
      "   300K .......... .......... .......... .......... ..........  4%  190M 1s\n",
      "   350K .......... .......... .......... .......... ..........  4%  191M 1s\n",
      "   400K .......... .......... .......... .......... ..........  5%  209M 1s\n",
      "   450K .......... .......... .......... .......... ..........  6% 3.53M 1s\n",
      "   500K .......... .......... .......... .......... ..........  6%  160M 1s\n",
      "   550K .......... .......... .......... .......... ..........  7% 96.4M 1s\n",
      "   600K .......... .......... .......... .......... ..........  8%  212M 1s\n",
      "   650K .......... .......... .......... .......... ..........  8%  194M 1s\n",
      "   700K .......... .......... .......... .......... ..........  9%  208M 1s\n",
      "   750K .......... .......... .......... .......... ..........  9%  210M 1s\n",
      "   800K .......... .......... .......... .......... .......... 10%  198M 1s\n",
      "   850K .......... .......... .......... .......... .......... 11%  217M 1s\n",
      "   900K .......... .......... .......... .......... .......... 11%  201M 1s\n",
      "   950K .......... .......... .......... .......... .......... 12% 3.84M 1s\n",
      "  1000K .......... .......... .......... .......... .......... 12%  196M 1s\n",
      "  1050K .......... .......... .......... .......... .......... 13%  227M 1s\n",
      "  1100K .......... .......... .......... .......... .......... 14%  204M 1s\n",
      "  1150K .......... .......... .......... .......... .......... 14%  148M 1s\n",
      "  1200K .......... .......... .......... .......... .......... 15%  168M 1s\n",
      "  1250K .......... .......... .......... .......... .......... 16%  205M 0s\n",
      "  1300K .......... .......... .......... .......... .......... 16% 98.7M 0s\n",
      "  1350K .......... .......... .......... .......... .......... 17%  105M 0s\n",
      "  1400K .......... .......... .......... .......... .......... 17%  112M 0s\n",
      "  1450K .......... .......... .......... .......... .......... 18%  224M 0s\n",
      "  1500K .......... .......... .......... .......... .......... 19%  157M 0s\n",
      "  1550K .......... .......... .......... .......... .......... 19%  215M 0s\n",
      "  1600K .......... .......... .......... .......... .......... 20%  215M 0s\n",
      "  1650K .......... .......... .......... .......... .......... 20%  221M 0s\n",
      "  1700K .......... .......... .......... .......... .......... 21%  203M 0s\n",
      "  1750K .......... .......... .......... .......... .......... 22%  226M 0s\n",
      "  1800K .......... .......... .......... .......... .......... 22%  223M 0s\n",
      "  1850K .......... .......... .......... .......... .......... 23%  207M 0s\n",
      "  1900K .......... .......... .......... .......... .......... 24%  218M 0s\n",
      "  1950K .......... .......... .......... .......... .......... 24%  256M 0s\n",
      "  2000K .......... .......... .......... .......... .......... 25% 5.12M 0s\n",
      "  2050K .......... .......... .......... .......... .......... 25%  236M 0s\n",
      "  2100K .......... .......... .......... .......... .......... 26%  269M 0s\n",
      "  2150K .......... .......... .......... .......... .......... 27%  274M 0s\n",
      "  2200K .......... .......... .......... .......... .......... 27%  104M 0s\n",
      "  2250K .......... .......... .......... .......... .......... 28%  305M 0s\n",
      "  2300K .......... .......... .......... .......... .......... 28%  247M 0s\n",
      "  2350K .......... .......... .......... .......... .......... 29%  249M 0s\n",
      "  2400K .......... .......... .......... .......... .......... 30%  295M 0s\n",
      "  2450K .......... .......... .......... .......... .......... 30%  268M 0s\n",
      "  2500K .......... .......... .......... .......... .......... 31%  252M 0s\n",
      "  2550K .......... .......... .......... .......... .......... 32%  280M 0s\n",
      "  2600K .......... .......... .......... .......... .......... 32%  254M 0s\n",
      "  2650K .......... .......... .......... .......... .......... 33%  257M 0s\n",
      "  2700K .......... .......... .......... .......... .......... 33%  202M 0s\n",
      "  2750K .......... .......... .......... .......... .......... 34%  274M 0s\n",
      "  2800K .......... .......... .......... .......... .......... 35% 76.9M 0s\n",
      "  2850K .......... .......... .......... .......... .......... 35% 60.4M 0s\n",
      "  2900K .......... .......... .......... .......... .......... 36% 49.0M 0s\n",
      "  2950K .......... .......... .......... .......... .......... 36% 29.8M 0s\n",
      "  3000K .......... .......... .......... .......... .......... 37%  117M 0s\n",
      "  3050K .......... .......... .......... .......... .......... 38%  216M 0s\n",
      "  3100K .......... .......... .......... .......... .......... 38%  142M 0s\n",
      "  3150K .......... .......... .......... .......... .......... 39%  223M 0s\n",
      "  3200K .......... .......... .......... .......... .......... 40%  233M 0s\n",
      "  3250K .......... .......... .......... .......... .......... 40%  290M 0s\n",
      "  3300K .......... .......... .......... .......... .......... 41%  236M 0s\n",
      "  3350K .......... .......... .......... .......... .......... 41%  290M 0s\n",
      "  3400K .......... .......... .......... .......... .......... 42%  290M 0s\n",
      "  3450K .......... .......... .......... .......... .......... 43%  272M 0s\n",
      "  3500K .......... .......... .......... .......... .......... 43%  215M 0s\n",
      "  3550K .......... .......... .......... .......... .......... 44%  241M 0s\n",
      "  3600K .......... .......... .......... .......... .......... 44%  266M 0s\n",
      "  3650K .......... .......... .......... .......... .......... 45%  240M 0s\n",
      "  3700K .......... .......... .......... .......... .......... 46%  230M 0s\n",
      "  3750K .......... .......... .......... .......... .......... 46%  270M 0s\n",
      "  3800K .......... .......... .......... .......... .......... 47%  273M 0s\n",
      "  3850K .......... .......... .......... .......... .......... 48%  286M 0s\n",
      "  3900K .......... .......... .......... .......... .......... 48%  216M 0s\n",
      "  3950K .......... .......... .......... .......... .......... 49%  258M 0s\n",
      "  4000K .......... .......... .......... .......... .......... 49%  178M 0s\n",
      "  4050K .......... .......... .......... .......... .......... 50%  217M 0s\n",
      "  4100K .......... .......... .......... .......... .......... 51% 15.2M 0s\n",
      "  4150K .......... .......... .......... .......... .......... 51%  197M 0s\n",
      "  4200K .......... .......... .......... .......... .......... 52%  195M 0s\n",
      "  4250K .......... .......... .......... .......... .......... 52%  217M 0s\n",
      "  4300K .......... .......... .......... .......... .......... 53%  190M 0s\n",
      "  4350K .......... .......... .......... .......... .......... 54%  187M 0s\n",
      "  4400K .......... .......... .......... .......... .......... 54%  212M 0s\n",
      "  4450K .......... .......... .......... .......... .......... 55%  206M 0s\n",
      "  4500K .......... .......... .......... .......... .......... 56%  162M 0s\n",
      "  4550K .......... .......... .......... .......... .......... 56%  209M 0s\n",
      "  4600K .......... .......... .......... .......... .......... 57%  205M 0s\n",
      "  4650K .......... .......... .......... .......... .......... 57%  175M 0s\n",
      "  4700K .......... .......... .......... .......... .......... 58%  179M 0s\n",
      "  4750K .......... .......... .......... .......... .......... 59%  228M 0s\n",
      "  4800K .......... .......... .......... .......... .......... 59%  207M 0s\n",
      "  4850K .......... .......... .......... .......... .......... 60%  198M 0s\n",
      "  4900K .......... .......... .......... .......... .......... 61%  178M 0s\n",
      "  4950K .......... .......... .......... .......... .......... 61%  254M 0s\n",
      "  5000K .......... .......... .......... .......... .......... 62%  252M 0s\n",
      "  5050K .......... .......... .......... .......... .......... 62%  213M 0s\n",
      "  5100K .......... .......... .......... .......... .......... 63%  173M 0s\n",
      "  5150K .......... .......... .......... .......... .......... 64%  218M 0s\n",
      "  5200K .......... .......... .......... .......... .......... 64%  221M 0s\n",
      "  5250K .......... .......... .......... .......... .......... 65%  246M 0s\n",
      "  5300K .......... .......... .......... .......... .......... 65%  151M 0s\n",
      "  5350K .......... .......... .......... .......... .......... 66%  206M 0s\n",
      "  5400K .......... .......... .......... .......... .......... 67%  199M 0s\n",
      "  5450K .......... .......... .......... .......... .......... 67%  230M 0s\n",
      "  5500K .......... .......... .......... .......... .......... 68%  149M 0s\n",
      "  5550K .......... .......... .......... .......... .......... 69%  207M 0s\n",
      "  5600K .......... .......... .......... .......... .......... 69%  207M 0s\n",
      "  5650K .......... .......... .......... .......... .......... 70%  189M 0s\n",
      "  5700K .......... .......... .......... .......... .......... 70%  195M 0s\n",
      "  5750K .......... .......... .......... .......... .......... 71%  195M 0s\n",
      "  5800K .......... .......... .......... .......... .......... 72%  200M 0s\n",
      "  5850K .......... .......... .......... .......... .......... 72%  203M 0s\n",
      "  5900K .......... .......... .......... .......... .......... 73%  177M 0s\n",
      "  5950K .......... .......... .......... .......... .......... 73%  204M 0s\n",
      "  6000K .......... .......... .......... .......... .......... 74%  209M 0s\n",
      "  6050K .......... .......... .......... .......... .......... 75%  203M 0s\n",
      "  6100K .......... .......... .......... .......... .......... 75%  189M 0s\n",
      "  6150K .......... .......... .......... .......... .......... 76%  220M 0s\n",
      "  6200K .......... .......... .......... .......... .......... 77%  220M 0s\n",
      "  6250K .......... .......... .......... .......... .......... 77%  215M 0s\n",
      "  6300K .......... .......... .......... .......... .......... 78%  208M 0s\n",
      "  6350K .......... .......... .......... .......... .......... 78%  242M 0s\n",
      "  6400K .......... .......... .......... .......... .......... 79%  244M 0s\n",
      "  6450K .......... .......... .......... .......... .......... 80%  231M 0s\n",
      "  6500K .......... .......... .......... .......... .......... 80%  210M 0s\n",
      "  6550K .......... .......... .......... .......... .......... 81%  197M 0s\n",
      "  6600K .......... .......... .......... .......... .......... 81% 17.0M 0s\n",
      "  6650K .......... .......... .......... .......... .......... 82%  189M 0s\n",
      "  6700K .......... .......... .......... .......... .......... 83%  110M 0s\n",
      "  6750K .......... .......... .......... .......... .......... 83%  205M 0s\n",
      "  6800K .......... .......... .......... .......... .......... 84%  211M 0s\n",
      "  6850K .......... .......... .......... .......... .......... 85%  204M 0s\n",
      "  6900K .......... .......... .......... .......... .......... 85%  173M 0s\n",
      "  6950K .......... .......... .......... .......... .......... 86%  262M 0s\n",
      "  7000K .......... .......... .......... .......... .......... 86%  204M 0s\n",
      "  7050K .......... .......... .......... .......... .......... 87%  225M 0s\n",
      "  7100K .......... .......... .......... .......... .......... 88%  180M 0s\n",
      "  7150K .......... .......... .......... .......... .......... 88%  222M 0s\n",
      "  7200K .......... .......... .......... .......... .......... 89%  188M 0s\n",
      "  7250K .......... .......... .......... .......... .......... 89%  141M 0s\n",
      "  7300K .......... .......... .......... .......... .......... 90%  192M 0s\n",
      "  7350K .......... .......... .......... .......... .......... 91%  235M 0s\n",
      "  7400K .......... .......... .......... .......... .......... 91%  165M 0s\n",
      "  7450K .......... .......... .......... .......... .......... 92%  190M 0s\n",
      "  7500K .......... .......... .......... .......... .......... 93%  163M 0s\n",
      "  7550K .......... .......... .......... .......... .......... 93%  191M 0s\n",
      "  7600K .......... .......... .......... .......... .......... 94%  196M 0s\n",
      "  7650K .......... .......... .......... .......... .......... 94%  219M 0s\n",
      "  7700K .......... .......... .......... .......... .......... 95%  206M 0s\n",
      "  7750K .......... .......... .......... .......... .......... 96%  238M 0s\n",
      "  7800K .......... .......... .......... .......... .......... 96%  224M 0s\n",
      "  7850K .......... .......... .......... .......... .......... 97%  247M 0s\n",
      "  7900K .......... .......... .......... .......... .......... 97%  212M 0s\n",
      "  7950K .......... .......... .......... .......... .......... 98%  171M 0s\n",
      "  8000K .......... .......... .......... .......... .......... 99%  219M 0s\n",
      "  8050K .......... .......... .......... .......... .......... 99%  182M 0s\n",
      "  8100K .......... ...                                        100%  212M=0.1s\n",
      "\n",
      "2020-12-29 12:14:29 (55.6 MB/s) - ‘drop_dataset.zip’ saved [8308692/8308692]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "directory=/content/drop_dataset/\n",
    "if [ ! -d \"$directory\" ]; then\n",
    "  # Download drop dataset\n",
    "  wget -c \"https://s3-us-west-2.amazonaws.com/allennlp/datasets/drop/drop_dataset.zip\"\n",
    "  # unzip the file\n",
    "  unzip drop_dataset.zip\n",
    "fi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "1-z9FK_AeZiQ"
   },
   "outputs": [],
   "source": [
    "train_file = \"/content/drop_dataset/drop_dataset_train.json\"\n",
    "test_file = \"/content/drop_dataset/drop_dataset_dev.json\"\n",
    "\n",
    "train = json.load(open(train_file))\n",
    "test = json.load(open(test_file))\n",
    "\n",
    "train_data = []\n",
    "for i, j in enumerate(train):\n",
    "    ques_ = []\n",
    "    ans_ = []\n",
    "    for l in train[j]['qa_pairs']:\n",
    "        ques_.append(l['question'])\n",
    "        m = l['answer']\n",
    "        if len(m['spans']) > 0:\n",
    "            ans_.append(\" \".join(m['spans']))\n",
    "        elif len(m['number']) > 0:\n",
    "            ans_.append(m['number'])\n",
    "        else:\n",
    "            ans_.append(\" \".join([m['date'][o] for o in m['date']]))\n",
    "    train_data.append([train[j]['passage'], ques_, ans_])\n",
    "\n",
    "test_data = []\n",
    "for i, j in enumerate(test):\n",
    "    ques_ = []\n",
    "    ans_ = []\n",
    "    for l in test[j]['qa_pairs']:\n",
    "        ques_.append(l['question'])\n",
    "        m = l['answer']\n",
    "        if len(m['spans']) > 0:\n",
    "            ans_.append(\" \".join(m['spans']))\n",
    "        elif len(m['number']) > 0:\n",
    "            ans_.append(m['number'])\n",
    "        else:\n",
    "            ans_.append(\" \".join([m['date'][o] for o in m['date']]))\n",
    "    test_data.append([test[j]['passage'], ques_, ans_])\n",
    "\n",
    "class CustomExample(data.Example):\n",
    "    # a problem might happen while trying to convert to tensors or while making vocab\n",
    "    @classmethod\n",
    "    def fromCustomList(cls, data, fields):\n",
    "        ex = cls()\n",
    "        for index, ((name, field), val) in enumerate(zip(fields, data)):\n",
    "            if index == 0:\n",
    "                if isinstance(val, str):\n",
    "                    val = val.rstrip('\\n')\n",
    "                setattr(ex, name, field.preprocess(val))\n",
    "            else:\n",
    "                field_processed_val = []\n",
    "                if not isinstance(val, list):\n",
    "                    val = [val]\n",
    "                for i in val:\n",
    "                    if isinstance(i, str):\n",
    "                        i = i.rstrip('\\n')\n",
    "                    field_processed_val.append(field.preprocess(i))\n",
    "                setattr(ex, name, field_processed_val)\n",
    "        return ex\n",
    "\n",
    "\n",
    "class CustomField(data.Field):\n",
    "    # Creating this class for custom padding and numercalizing of batches of questions\n",
    "    # problem was batch_size number of passages will have way more number of questions on them\n",
    "    # so append all question to make 1 batch for the second RNN cell.\n",
    "\n",
    "    def pad(self, minibatch):\n",
    "        # If I get OOM, restrict number of questions per passage here\n",
    "        self.lengths = [len(i) for i in minibatch]\n",
    "        m = chain.from_iterable(minibatch)\n",
    "        return super(CustomField, self).pad(m)\n",
    "\n",
    "\n",
    "context = data.Field(sequential=True, tokenize='spacy', init_token='<sos>', eos_token='<eos>')\n",
    "question = CustomField(sequential=True, tokenize='spacy', init_token='<sos>', eos_token='<eos>')\n",
    "answer = CustomField(sequential=True, tokenize='spacy', init_token='<sos>', eos_token='<eos>')\n",
    "\n",
    "fields = [('context', context), ('question', question), ('answer', answer)]\n",
    "train_Examples = [\n",
    "    CustomExample.fromCustomList([i[0], i[1], i[2]], fields) for i\n",
    "    in train_data]\n",
    "test_Examples = [\n",
    "    CustomExample.fromCustomList([i[0], i[1], i[2]], fields) for i\n",
    "    in test_data]\n",
    "\n",
    "train_dataset = data.Dataset(train_Examples, fields)\n",
    "test_dataset = data.Dataset(test_Examples,fields)\n",
    "context.build_vocab(train_dataset,min_freq=2,max_size = 30000)\n",
    "question.build_vocab(train_dataset,min_freq=2,max_size = 10000)\n",
    "answer.build_vocab(train_dataset,min_freq=2,max_size = 5000)\n",
    "\n",
    "BATCH_SIZE = 32\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "train_iterator, test_iterator = data.BucketIterator.splits((train_dataset, test_dataset), batch_size=BATCH_SIZE,\n",
    "                                                            sort_key=lambda x: len(x.context), sort_within_batch=True,device=device)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "K1Q129_le3KS",
    "outputId": "5ed6968e-e8fe-46cf-fe49-a870a21a2fe1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context:\n",
      "['To', 'start', 'the', 'season', ',', 'the', 'Lions', 'traveled', 'south', 'to', 'Tampa', ',', 'Florida', 'to', 'take', 'on', 'the', 'Tampa', 'Bay', 'Buccaneers', '.', 'The', 'Lions', 'scored', 'first', 'in', 'the', 'first', 'quarter', 'with', 'a', '23-yard', 'field', 'goal', 'by', 'Jason', 'Hanson', '.', 'The', 'Buccaneers', 'tied', 'it', 'up', 'with', 'a', '38-yard', 'field', 'goal', 'by', 'Connor', 'Barth', ',', 'then', 'took', 'the', 'lead', 'when', 'Aqib', 'Talib', 'intercepted', 'a', 'pass', 'from', 'Matthew', 'Stafford', 'and', 'ran', 'it', 'in', '28', 'yards', '.', 'The', 'Lions', 'responded', 'with', 'a', '28-yard', 'field', 'goal', '.', 'In', 'the', 'second', 'quarter', ',', 'Detroit', 'took', 'the', 'lead', 'with', 'a', '36-yard', 'touchdown', 'catch', 'by', 'Calvin', 'Johnson', ',', 'and', 'later', 'added', 'more', 'points', 'when', 'Tony', 'Scheffler', 'caught', 'an', '11-yard', 'TD', 'pass', '.', 'Tampa', 'Bay', 'responded', 'with', 'a', '31-yard', 'field', 'goal', 'just', 'before', 'halftime', '.', 'The', 'second', 'half', 'was', 'relatively', 'quiet', ',', 'with', 'each', 'team', 'only', 'scoring', 'one', 'touchdown', '.', 'First', ',', 'Detroit', \"'s\", 'Calvin', 'Johnson', 'caught', 'a', '1-yard', 'pass', 'in', 'the', 'third', 'quarter', '.', 'The', 'game', \"'s\", 'final', 'points', 'came', 'when', 'Mike', 'Williams', 'of', 'Tampa', 'Bay', 'caught', 'a', '5-yard', 'pass', '.', ' ', 'The', 'Lions', 'won', 'their', 'regular', 'season', 'opener', 'for', 'the', 'first', 'time', 'since', '2007']\n",
      "Questions :\n",
      "[['How', 'many', 'points', 'did', 'the', 'buccaneers', 'need', 'to', 'tie', 'in', 'the', 'first', '?'], ['How', 'many', 'field', 'goals', 'did', 'the', 'Lions', 'score', '?'], ['How', 'long', 'was', 'the', 'Lion', \"'s\", 'longest', 'field', 'goal', '?'], ['Who', 'caught', 'the', 'touchdown', 'for', 'the', 'fewest', 'yard', '?'], ['Who', 'caught', 'the', 'shortest', 'touchdown', 'pass', '?'], ['How', 'many', 'field', 'goals', 'were', 'scored', 'in', 'the', 'first', 'quarter', '?'], ['How', 'many', 'yards', 'was', 'the', 'shortest', 'touchdown', 'scoring', 'play', '?'], ['How', 'many', 'touchdowns', 'were', 'scored', 'in', 'the', 'second', 'quarter', '?'], ['Which', 'player', 'scored', 'the', 'first', 'points', 'of', 'the', 'game', 'for', 'Tampa', 'Bay', '?'], ['How', 'many', 'yards', 'Hanson', 'score', 'with', 'in', 'the', 'first', '?'], ['How', 'many', 'field', 'goals', 'were', 'made', 'in', 'the', 'first', 'quarter', '?'], ['Who', 'threw', 'the', 'first', 'touchdown', 'pass', 'of', 'the', 'game', '?'], ['How', 'many', 'touchdowns', 'were', 'scored', 'in', 'the', '2nd', 'half', '?'], ['Who', 'caught', 'the', 'longest', 'touchdown', 'reception', 'of', 'the', 'game', '?'], ['How', 'many', 'points', 'were', 'scored', 'first', '?']]\n",
      "Answers:\n",
      "[['3'], ['2'], ['28-yard'], ['Mike', 'Williams'], ['Calvin', 'Johnson'], ['3'], ['1'], ['3'], ['Connor', 'Barth'], ['23'], ['3'], ['Matthew', 'Stafford'], ['2'], ['Calvin', 'Johnson'], ['3']]\n"
     ]
    }
   ],
   "source": [
    "print(\"Context:\")\n",
    "print(train_Examples[0].context)\n",
    "print(\"Questions :\")\n",
    "print(train_Examples[0].question)\n",
    "print(\"Answers:\")\n",
    "print(train_Examples[0].answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cD8KKuwPe5Sk",
    "outputId": "8832c308-ad2d-4334-e04e-81249794af65"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5565\n"
     ]
    }
   ],
   "source": [
    "print(len(train_Examples))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "tuVd5BFdmvWF"
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "  def __init__(self,context_dim,emb_dim,hid_dim,n_layers,dropout,bidirectional):\n",
    "    super().__init__()\n",
    "    self.hid_dim = hid_dim\n",
    "    \n",
    "    self.embedding = nn.Embedding(context_dim,emb_dim)\n",
    "\n",
    "    self.rnn = nn.LSTM(input_size=emb_dim,hidden_size = hid_dim,num_layers= n_layers,dropout= dropout,bidirectional = bidirectional)\n",
    "\n",
    "    self.dropout = nn.Dropout(dropout)\n",
    "  \n",
    "  def forward(self,context):\n",
    "    \n",
    "    embedded = self.dropout(self.embedding(context))\n",
    "\n",
    "    outputs, (hidden,cell_state) = self.rnn(embedded)\n",
    "\n",
    "    return outputs,hidden,cell_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "B-1WjaBQm0TH"
   },
   "outputs": [],
   "source": [
    "class Encoder1(nn.Module):\n",
    "  def __init__(self,question_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout):\n",
    "    super().__init__()\n",
    "    self.hid_dim = hid_dim\n",
    "\n",
    "    self.embedded = nn.Embedding(question_dim,emb_dim)\n",
    "\n",
    "    self.rnn = nn.LSTM(input_size=emb_dim,hidden_size = hid_dim,num_layers= n_layers,dropout= dropout,bidirectional = bidirectional)\n",
    "\n",
    "    self.dropout = nn.Dropout(dropout)    \n",
    "    \n",
    "\n",
    "  def fill_tensor(self,hidden, cell_state, question):\n",
    "    # The idea is to make a new tensor that fills the hidden and cell state according to the lengths of the question\n",
    "    \n",
    "    hidden_shape = hidden.shape\n",
    "    question_shape = question.shape\n",
    "    t_hidden = torch.zeros(hidden_shape[0],question_shape[-1],hidden_shape[-1]).to(device)\n",
    "    t_cell_state = torch.zeros(hidden_shape[0],question_shape[-1],hidden_shape[-1]).to(device)\n",
    "    pos = 0\n",
    "    for i in range(hidden_shape[1]):\n",
    "      t_hidden[:,pos:pos+self.lengths[i],:] = hidden[:,i,:].unsqueeze(1)\n",
    "      t_cell_state[:,pos:self.lengths[i],:] = cell_state[:,i,:].unsqueeze(1)\n",
    "      pos += self.lengths[i]\n",
    "    return t_hidden,t_cell_state\n",
    "\n",
    "  def forward(self, question, hidden, cell_state,lengths):\n",
    "    self.lengths = lengths\n",
    "\n",
    "    assert sum(lengths) == question.shape[-1]\n",
    "    input = self.dropout(self.embedded(question))\n",
    "\n",
    "    # filled tensor\n",
    "    filled_hidden, filled_cell_state = self.fill_tensor(hidden,cell_state,question)\n",
    "\n",
    "    outputs , (hidden, cell_state) = self.rnn(input,(filled_hidden,filled_cell_state))\n",
    "\n",
    "    return outputs ,hidden, cell_state\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "id": "KTM-Nlo3aSGp"
   },
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "class Attention(nn.Module):\n",
    "  def __init__(self, attn_vector_size,hid_dim,bidirectional,n_layers, type_='dot'):\n",
    "    super().__init__()\n",
    "    self.attn_vector_size = attn_vector_size\n",
    "    self.bidirectional = bidirectional\n",
    "    self.n_layers = n_layers\n",
    "    self.type_ = type_\n",
    "    self.hid_dim = hid_dim\n",
    "    self.directions = 2 if bidirectional else 1\n",
    "    # The final vector is concatenation of (n_layers*directions) different vectors and the final size is 'attn_vector_size'\n",
    "    # so it should be divisible with (n_layers*directions) \n",
    "    assert attn_vector_size%(n_layers*self.directions) == 0 \n",
    "\n",
    "    self.transform_ = nn.Linear(self.directions*self.hid_dim,int(attn_vector_size/(n_layers*self.directions)))\n",
    "  \n",
    "  def forward(self,encoder_outputs,decoder_hidden_state):\n",
    "    # Interesting: what happens when there are multiple layers and bidirectionality.. Do I find attention vector for each layer and append it to \n",
    "    # input of each layer.. but we give LSTMs input only once..so I think I need a attention vector including information from all layers\n",
    "    # given as a single input once.\n",
    "\n",
    "    # Since I am already working with very long sequences..to avoid too many params, I am using dot product as a score function\n",
    "    if not self.type_ == 'dot': raise NotImplementedError\n",
    "    \n",
    "    # The idea is to find attention for each layer and direction of decoder hidden state and later combine all these and give a final vector\n",
    "    # so the attention input has attention information from all decoder hidden states\n",
    "\n",
    "    # encoder_outputs : [src_len,batch_size,num_of_directions*hid_dim]\n",
    "    # decoder_hidden_state : [n_layers*no_of_directions,batch_size,hid_dim]\n",
    "\n",
    "    decoder_hidden_state_len = len(decoder_hidden_state)\n",
    "    src_len = encoder_outputs.shape[0]\n",
    "\n",
    "    encoder_view = encoder_outputs.view(-1,self.directions,self.hid_dim) # encoder_view : [src_len*batch_size,num_of_directions,hid_dim]\n",
    "\n",
    "    encoder_outputs = encoder_outputs.permute(1,0,2) # encoder_outputs : [batch_size,src_len,num_of_directions*hid_dim]\n",
    "    all_attention_vectors = []\n",
    "\n",
    "    for i in range(decoder_hidden_state_len):\n",
    "      \n",
    "      # Calculating alpha\n",
    "      hidden_state = decoder_hidden_state[i] # hidden state : [batch_size,hid_dim]\n",
    "\n",
    "      hidden_state = hidden_state.unsqueeze(0) # hidden_state : [1,batch_size,hid_dim]\n",
    "      hidden_state = hidden_state.permute(1,2,0) # hidden_state :[batch_size,hid_dim, 1]\n",
    "      hidden_state = hidden_state.repeat(src_len,1,1) # hidden_state :[src_len*batch_size,hid_dim, 1]\n",
    "\n",
    "      temp = torch.bmm(encoder_view,hidden_state) # temp : [src_len*batch_size,num_of_directions,1]\n",
    "      temp = temp.sum(1) # temp : [src_len*batch_size,1]\n",
    "      temp = temp.reshape(-1,1,src_len) # temp :[batch_size,1,src_len]\n",
    "\n",
    "      alpha = F.softmax(temp,dim=-1)  # alpha :[batch_size,1,src_len]\n",
    "\n",
    "      # Calculating attention vector\n",
    "      c = torch.bmm(alpha,encoder_outputs) # s: [batch_size,1,num_of_directions*hid_dim]\n",
    "\n",
    "      transform_c = self.transform_(c) # transform_s : [batch_size,1,attn_vector_size/(n_layers*self.directions)]\n",
    "      transform_c = transform_c.permute(1,0,2) # transform_s : [1,batch_size,attn_vector_size/(n_layers*self.directions)]\n",
    "      all_attention_vectors.append(transform_c)\n",
    "\n",
    "\n",
    "    # return vector of shape [1,batch_size,self.attn_vector_size(emb_dim)]\n",
    "    return torch.cat(all_attention_vectors,dim = -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "k3ay_sB4y0FK"
   },
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "  def __init__(self,answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout):\n",
    "    super().__init__()\n",
    "\n",
    "    ## I wont be passing the hidden state from last LSTM to each input and fc_out of eaxh time step and emb dimension to fc_out\n",
    "    self.hid_dim = hid_dim\n",
    "\n",
    "    self.embedded = nn.Embedding(answer_dim,emb_dim)\n",
    "\n",
    "    self.rnn = nn.LSTM(input_size=2*emb_dim,hidden_size = hid_dim,num_layers= n_layers,dropout= dropout,bidirectional = bidirectional)\n",
    "\n",
    "    self.attention = Attention(emb_dim,hid_dim,bidirectional,n_layers)\n",
    "\n",
    "    self.dropout = nn.Dropout(dropout)\n",
    "    \n",
    "    no_of_directions = 2 if bidirectional else 1\n",
    "\n",
    "    self.fc_out = nn.Linear(no_of_directions*hid_dim,answer_dim)\n",
    "\n",
    "  def forward(self, answer, encoder_outputs, hidden, cell_state):\n",
    "    # this forward will be 1 step at a time\n",
    "\n",
    "    # emdeb answer with dropout\n",
    "    answer = answer.unsqueeze(0)\n",
    "    embedded = self.dropout(self.embedded(answer))\n",
    "\n",
    "    attention_vector = self.attention(encoder_outputs,hidden)\n",
    "\n",
    "    # concatenate embedded with attention vector\n",
    "    embedded = torch.cat([embedded,attention_vector],dim=-1)\n",
    "\n",
    "    # pass it through cell\n",
    "    output , (hidden,cell_state) = self.rnn(embedded,(hidden,cell_state))\n",
    "\n",
    "    # change dimension of output for fc_out\n",
    "    # output should be [1,batch_size,no_of_directions*hid_dim]\n",
    "    output = output.squeeze(0)\n",
    "    \n",
    "    # return hidden and cell state\n",
    "    prediction = self.fc_out(output)\n",
    "\n",
    "    return prediction, hidden, cell_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "id": "o0In0cetqfVR"
   },
   "outputs": [],
   "source": [
    "\n",
    "import random\n",
    "class Seq2Seq(nn.Module):\n",
    "  def __init__(self,context_dim,question_dim,answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout):\n",
    "    super().__init__()\n",
    "    self.context_encoder = Encoder(context_dim,emb_dim,hid_dim,n_layers,dropout,bidirectional)\n",
    "    self.question_encoder = Encoder1(question_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout)\n",
    "    self.answer_decoder = Decoder(answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout)\n",
    "\n",
    "    self.answer_dim = answer_dim\n",
    "  def custom_strech(self,context,lengths):\n",
    "    total_len = sum(lengths)\n",
    "\n",
    "    #checking batch size in  context is same as lenght of the list 'lenghts'\n",
    "    assert context.shape[1] == len(lengths)\n",
    "    t_context = torch.zeros(context.shape[0],total_len,context.shape[-1]).to(device)\n",
    "    pos = 0\n",
    "    for j,i in enumerate(lengths):\n",
    "      t_context[:,pos:pos+i,:] = context[:,j,:].unsqueeze(1)\n",
    "      pos = pos+i\n",
    "\n",
    "    return t_context\n",
    "  \n",
    "  def forward(self,context,question,answer,lengths,teacher_forcing_ratio = 0.5):\n",
    "\n",
    "    # encode context\n",
    "    context_encoder_outputs,hidden,cell_state = self.context_encoder(context)\n",
    "\n",
    "    # strech encoder context output so that the batch size is same while calculating attention\n",
    "    context_encoder_outputs = self.custom_strech(context_encoder_outputs,lengths)\n",
    "\n",
    "    # encode question\n",
    "    _,hidden,cell_state = self.question_encoder(question,hidden,cell_state,lengths)\n",
    "\n",
    "    # pass the answer one by one\n",
    "    answer_len = len(answer)\n",
    "    batch_size = answer.shape[1]\n",
    "\n",
    "    outputs = torch.zeros(answer_len,batch_size,self.answer_dim).to(device)\n",
    "\n",
    "    for i,j in enumerate(range(answer_len)):\n",
    "      k = answer[j]\n",
    "      if not i == 0:\n",
    "        k = prediction.argmax(1) if random.random() < teacher_forcing_ratio else k\n",
    "      # using attention over context vectors rather than questions\n",
    "      prediction,hidden,cell_state = self.answer_decoder(k,context_encoder_outputs,hidden,cell_state)\n",
    "      outputs[j] = prediction\n",
    "\n",
    "    return outputs\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-nczYUvTzMnk",
    "outputId": "63f84347-1b84-46d0-c417-8fee95c65354"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py:61: UserWarning: dropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout=0.5 and num_layers=1\n",
      "  \"num_layers={}\".format(dropout, num_layers))\n"
     ]
    }
   ],
   "source": [
    "context_dim = len(context.vocab)\n",
    "question_dim = len(question.vocab)\n",
    "answer_dim = len(answer.vocab)\n",
    "emb_dim = 50\n",
    "hid_dim = 50\n",
    "n_layers = 1\n",
    "bidirectional = True\n",
    "dropout = 0.5\n",
    "\n",
    "model = Seq2Seq(context_dim,question_dim,answer_dim,emb_dim,hid_dim,n_layers,bidirectional,dropout).to(device)\n",
    "\n",
    "def init_weights(m):\n",
    "    for name, param in m.named_parameters():\n",
    "      if not isinstance(m, Embedding):\n",
    "        nn.init.normal_(param.data, mean=0, std=0.01)\n",
    "        \n",
    "model.apply(init_weights)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "\n",
    "\n",
    "TRG_PAD_IDX = answer.vocab.stoi[answer.pad_token]\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index = TRG_PAD_IDX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "SZHL7hR14A5S"
   },
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion, clip):\n",
    "    \n",
    "    model.train()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    for i, batch in enumerate(iterator):\n",
    "        \n",
    "        context_ = batch.context\n",
    "        question_ = batch.question\n",
    "        answer_ = batch.answer\n",
    "        lengths = question.lengths # I know this is not the right way to do..maybe create a new filed for lenghts but they depend of the chosen batch??\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        output = model(context_, question_,answer_,lengths)\n",
    "        \n",
    "        trg = answer_\n",
    "        #trg = [trg len, batch size]\n",
    "        #output = [trg len, batch size, output dim]\n",
    "        \n",
    "        output_dim = output.shape[-1]\n",
    "        \n",
    "        output = output[1:].view(-1, output_dim)\n",
    "        trg = trg[1:].view(-1)\n",
    "        \n",
    "        #trg = [(trg len - 1) * batch size]\n",
    "        #output = [(trg len - 1) * batch size, output dim]\n",
    "        \n",
    "        loss = criterion(output, trg)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "nCvWtM2J5BeO"
   },
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    \n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for i, batch in enumerate(iterator):\n",
    "\n",
    "            context_ = batch.context\n",
    "            question_ = batch.question\n",
    "            answer_ = batch.answer\n",
    "            lengths = question.lengths\n",
    "        \n",
    "\n",
    "            output = model(context_, question_,answer_,lengths,0) #turn off teacher forcing\n",
    "\n",
    "            trg = answer_\n",
    "            #trg = [trg len, batch size]\n",
    "            #output = [trg len, batch size, output dim]\n",
    "\n",
    "            output_dim = output.shape[-1]\n",
    "            \n",
    "            output = output[1:].view(-1, output_dim)\n",
    "            trg = trg[1:].view(-1)\n",
    "\n",
    "            #trg = [(trg len - 1) * batch size]\n",
    "            #output = [(trg len - 1) * batch size, output dim]\n",
    "\n",
    "            loss = criterion(output, trg)\n",
    "\n",
    "            epoch_loss += loss.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "PGBoT91y5Ln6"
   },
   "outputs": [],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HoDqVqra5NpP",
    "outputId": "0676c138-7d18-41af-a507-760f4540c172"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Time: 2m 4s\n",
      "\tTrain Loss: 5.240 | Train PPL: 188.745\n",
      "\t Val. Loss: 4.057 |  Val. PPL:  57.780\n",
      "Epoch: 02 | Time: 2m 3s\n",
      "\tTrain Loss: 4.369 | Train PPL:  78.936\n",
      "\t Val. Loss: 3.732 |  Val. PPL:  41.743\n",
      "Epoch: 03 | Time: 2m 3s\n",
      "\tTrain Loss: 4.083 | Train PPL:  59.320\n",
      "\t Val. Loss: 3.389 |  Val. PPL:  29.641\n",
      "Epoch: 04 | Time: 2m 4s\n",
      "\tTrain Loss: 3.982 | Train PPL:  53.623\n",
      "\t Val. Loss: 3.336 |  Val. PPL:  28.112\n",
      "Epoch: 05 | Time: 2m 1s\n",
      "\tTrain Loss: 3.876 | Train PPL:  48.231\n",
      "\t Val. Loss: 3.198 |  Val. PPL:  24.472\n",
      "Epoch: 06 | Time: 2m 1s\n",
      "\tTrain Loss: 3.675 | Train PPL:  39.447\n",
      "\t Val. Loss: 3.025 |  Val. PPL:  20.597\n",
      "Epoch: 07 | Time: 2m 4s\n",
      "\tTrain Loss: 3.487 | Train PPL:  32.685\n",
      "\t Val. Loss: 2.899 |  Val. PPL:  18.153\n",
      "Epoch: 08 | Time: 2m 4s\n",
      "\tTrain Loss: 3.364 | Train PPL:  28.902\n",
      "\t Val. Loss: 2.745 |  Val. PPL:  15.560\n",
      "Epoch: 09 | Time: 2m 3s\n",
      "\tTrain Loss: 3.272 | Train PPL:  26.355\n",
      "\t Val. Loss: 2.657 |  Val. PPL:  14.248\n",
      "Epoch: 10 | Time: 2m 0s\n",
      "\tTrain Loss: 3.174 | Train PPL:  23.898\n",
      "\t Val. Loss: 2.541 |  Val. PPL:  12.693\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "N_EPOCHS = 10\n",
    "CLIP = 1\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "    \n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss = train(model, train_iterator, optimizer, criterion, CLIP)\n",
    "    valid_loss = evaluate(model, test_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "    \n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut2-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train PPL: {math.exp(train_loss):7.3f}')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. PPL: {math.exp(valid_loss):7.3f}')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "Drop Dataset with Seq2Seq with attention.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}