machine_learning_diary/deep_learning/14_Advanced_RNNs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bOChJSNXtC9g"
   },
   "source": [
    "# Advanced RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OLIxEDq6VhvZ"
   },
   "source": [
    "In this notebook we're going to cover some advanced topics related to RNNs.\n",
    "\n",
    "1. Conditioned hidden state\n",
    "2. Char-level embeddings\n",
    "3. Encoder and decoder\n",
    "4. Attentional mechanisms\n",
    "5. Implementation\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "41r7MWJnY0m8"
   },
   "source": [
    "# Set up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EJDhjHCHY0_a"
   },
   "outputs": [],
   "source": [
    "# Load PyTorch library\n",
    "!pip3 install torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p0FbOd6IZmzX"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "from argparse import Namespace\n",
    "import collections\n",
    "import copy\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bOsqAo4XZpXQ"
   },
   "outputs": [],
   "source": [
    "# Set Numpy and PyTorch seeds\n",
    "def set_seeds(seed, cuda):\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    if cuda:\n",
    "        torch.cuda.manual_seed_all(seed)\n",
    "        \n",
    "# Creating directories\n",
    "def create_dirs(dirpath):\n",
    "    if not os.path.exists(dirpath):\n",
    "        os.makedirs(dirpath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "QHfvEzQ9ZweF",
    "outputId": "ca3e04cf-b266-4049-9443-b5581839e6f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using CUDA: True\n"
     ]
    }
   ],
   "source": [
    "# Arguments\n",
    "args = Namespace(\n",
    "    seed=1234,\n",
    "    cuda=True,\n",
    "    batch_size=4,\n",
    "    condition_vocab_size=3, # vocabulary for condition possibilities\n",
    "    embedding_dim=100,\n",
    "    rnn_hidden_dim=100,\n",
    "    hidden_dim=100,\n",
    "    num_layers=1,\n",
    "    bidirectional=False,\n",
    ")\n",
    "\n",
    "# Set seeds\n",
    "set_seeds(seed=args.seed, cuda=args.cuda)\n",
    "\n",
    "# Check CUDA\n",
    "if not torch.cuda.is_available():\n",
    "    args.cuda = False\n",
    "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
    "print(\"Using CUDA: {}\".format(args.cuda))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VoMq0eFRvugb"
   },
   "source": [
    "# Conditioned RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZUsj7HjBp69f"
   },
   "source": [
    "Conditioning an RNN is to add extra information that will be helpful towards a prediction. We can encode (embed it) this information and feed it along with the sequential input into our model. For example, suppose in our document classificaiton example in the previous notebook, we knew the publisher of each news article (NYTimes, ESPN, etc.). We could have encoded that information to help with the prediction. There are several different ways of creating a conditioned RNN.\n",
    "\n",
    "**Note**: If the conditioning information is novel for each input in the sequence, just concatenate it along with each time step's input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Kc8H9JySmtLa"
   },
   "source": [
    "1. Make the initial hidden state the encoded information instead of using the initial zerod hidden state. Make sure that the size of the encoded information is the same as the hidden state for the RNN.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pKlb9SjfpbED"
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/conditioned_rnn1.png\" width=400>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jbrlQHx2x8Aa"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "cFoiV-fqmvRo",
    "outputId": "96da576e-6edf-4837-a64d-7955084706a9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 4, 100])\n"
     ]
    }
   ],
   "source": [
    "# Condition\n",
    "condition = torch.LongTensor([0, 2, 1, 2]) # batch size of 4 with a vocab size of 3\n",
    "condition_embeddings = nn.Embedding(\n",
    "    embedding_dim=args.embedding_dim, # should be same as RNN hidden dim\n",
    "    num_embeddings=args.condition_vocab_size) # of unique conditions\n",
    "\n",
    "# Initialize hidden state\n",
    "num_directions = 1\n",
    "if args.bidirectional:\n",
    "    num_directions = 2\n",
    "    \n",
    "# If using multiple layers and directions, the hidden state needs to match that size\n",
    "hidden_t = condition_embeddings(condition).unsqueeze(0).repeat(\n",
    "    args.num_layers * num_directions, 1, 1).to(args.device) # initial state to RNN\n",
    "print (hidden_t.size())\n",
    "\n",
    "# Feed into RNN\n",
    "# y_out, _ = self.rnn(x_embedded, hidden_t)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "REgyaMDgmtHw"
   },
   "source": [
    "2. Concatenate the encoded information with the hidden state at each time step. Do not replace the hidden state because the RNN needs that to learn. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "yUIg5o-dpiZF"
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/conditioned_rnn2.png\" width=400>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "eQ-h28o-pi4X",
    "outputId": "f92d4aac-3430-4e77-cb69-57e0d84b1eff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 4, 100])\n"
     ]
    }
   ],
   "source": [
    "# Initialize hidden state\n",
    "hidden_t = torch.zeros((args.num_layers * num_directions, args.batch_size, args.rnn_hidden_dim))\n",
    "print (hidden_t.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2Z6hYSIdqBQ4"
   },
   "outputs": [],
   "source": [
    "def concat_condition(condition_embeddings, condition, hidden_t, num_layers, num_directions):\n",
    "    condition_t = condition_embeddings(condition).unsqueeze(0).repeat(\n",
    "        num_layers * num_directions, 1, 1)\n",
    "    hidden_t = torch.cat([hidden_t, condition_t], 2)\n",
    "    return hidden_t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Tjyzq_s5pixL",
    "outputId": "189c4e95-a624-4921-9a57-79469b17a32d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 4, 200])\n"
     ]
    }
   ],
   "source": [
    "# Loop through the inputs time steps\n",
    "hiddens = []\n",
    "seq_size = 1\n",
    "for t in range(seq_size):\n",
    "    hidden_t = concat_condition(condition_embeddings, condition, hidden_t, \n",
    "                                args.num_layers, num_directions).to(args.device)\n",
    "    print (hidden_t.size())\n",
    "    \n",
    "    # Feed into RNN\n",
    "    # hidden_t = rnn_cell(x_in[t], hidden_t)\n",
    "    ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A-0_81jMXg_J"
   },
   "source": [
    "# Char-level embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "w0yUKKpq3pu_"
   },
   "source": [
    "Our conv operations will have inputs that are words in a sentence represented at the character level|  $\\in \\mathbb{R}^{NXSXWXE}$  and outputs are embeddings for each word (based on convlutions applied at the character level.) \n",
    "\n",
    "**Word embeddings**: capture the temporal correlations among\n",
    "adjacent tokens so that similar words have similar representations. Ex. \"New Jersey\" is close to \"NJ\" is close to \"Garden State\", etc.\n",
    "\n",
    "**Char embeddings**: create representations that map words at a character level. Ex. \"toy\" and \"toys\" will be close to each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-SZgVuwebm_4"
   },
   "source": [
    "![](https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/char_embeddings.png?raw=true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QOdIvz0G3O8C"
   },
   "outputs": [],
   "source": [
    "# Arguments\n",
    "args = Namespace(\n",
    "    seed=1234,\n",
    "    cuda=False,\n",
    "    shuffle=True,\n",
    "    batch_size=64,\n",
    "    vocab_size=20, # vocabulary\n",
    "    seq_size=10, # max length of each sentence\n",
    "    word_size=15, # max length of each word\n",
    "    embedding_dim=100,\n",
    "    num_filters=100, # filters per size\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "raztXIeYXYJT"
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, embedding_dim, num_embeddings, num_input_channels, \n",
    "                 num_output_channels, padding_idx):\n",
    "        super(Model, self).__init__()\n",
    "        \n",
    "        # Char-level embedding\n",
    "        self.embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
    "                                       num_embeddings=num_embeddings,\n",
    "                                       padding_idx=padding_idx)\n",
    "        \n",
    "        # Conv weights\n",
    "        self.conv = nn.ModuleList([nn.Conv1d(num_input_channels, num_output_channels, \n",
    "                                             kernel_size=f) for f in [2,3,4]])\n",
    "\n",
    "    def forward(self, x, channel_first=False, apply_softmax=False):\n",
    "        \n",
    "        # x: (N, seq_len, word_len)\n",
    "        input_shape = x.size()\n",
    "        batch_size, seq_len, word_len = input_shape\n",
    "        x = x.view(-1, word_len) # (N*seq_len, word_len)\n",
    "        \n",
    "        # Embedding\n",
    "        x = self.embeddings(x) # (N*seq_len, word_len, embedding_dim)\n",
    "        \n",
    "        # Rearrange input so num_input_channels is in dim 1 (N, embedding_dim, word_len)\n",
    "        if not channel_first:\n",
    "            x = x.transpose(1, 2)\n",
    "        \n",
    "        # Convolution\n",
    "        z = [F.relu(conv(x)) for conv in self.conv]\n",
    "        \n",
    "        # Pooling\n",
    "        z = [F.max_pool1d(zz, zz.size(2)).squeeze(2) for zz in z] \n",
    "        z = [zz.view(batch_size, seq_len, -1) for zz in z] # (N, seq_len, embedding_dim)\n",
    "        \n",
    "        # Concat to get char-level embeddings\n",
    "        z = torch.cat(z, 2) # join conv outputs\n",
    "        \n",
    "        return z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "MzHVs8Xe0Zph",
    "outputId": "70f5c4bf-7fc0-418d-d277-9ee2ea80b57a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 10, 15])\n"
     ]
    }
   ],
   "source": [
    "# Input\n",
    "input_size = (args.batch_size, args.seq_size, args.word_size)\n",
    "x_in = torch.randint(low=0, high=args.vocab_size, size=input_size).long()\n",
    "print (x_in.size())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "0B_Xscby2PMQ",
    "outputId": "caf6288d-b44f-47c0-bc9d-66a74b9d42f1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method Module.named_modules of Model(\n",
      "  (embeddings): Embedding(20, 100, padding_idx=0)\n",
      "  (conv): ModuleList(\n",
      "    (0): Conv1d(100, 100, kernel_size=(2,), stride=(1,))\n",
      "    (1): Conv1d(100, 100, kernel_size=(3,), stride=(1,))\n",
      "    (2): Conv1d(100, 100, kernel_size=(4,), stride=(1,))\n",
      "  )\n",
      ")>\n"
     ]
    }
   ],
   "source": [
    "# Initial char-level embedding model\n",
    "model = Model(embedding_dim=args.embedding_dim, \n",
    "              num_embeddings=args.vocab_size, \n",
    "              num_input_channels=args.embedding_dim, \n",
    "              num_output_channels=args.num_filters,\n",
    "              padding_idx=0)\n",
    "print (model.named_modules)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "8DIgeEZFXYR2",
    "outputId": "a0e376c7-fa16-4faa-8a85-b8ad7d65dd39"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 10, 300])\n"
     ]
    }
   ],
   "source": [
    "# Forward pass to get char-level embeddings\n",
    "z = model(x_in)\n",
    "print (z.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nzTscaE10HFA"
   },
   "source": [
    "There are several different ways you can use these char-level embeddings:\n",
    "\n",
    "1. Concat char-level embeddings with word-level embeddings, since we have an embedding for each word (at a char-level) and then feed it into an RNN. \n",
    "2. You can feed the char-level embeddings into an RNN to processes them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nyCQ13_ckV_c"
   },
   "source": [
    "# Encoder and decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_sixbu74kbJk"
   },
   "source": [
    "So far we've used RNNs to `encode` a sequential input and generate hidden states. We use these hidden states to `decode` the predictions. So far, the encoder was an RNN and the decoder was just a few fully connected layers followed by a softmax layer (for classification). But the encoder and decoder can assume other architectures as well. For example, the decoder could be an RNN that processes the hidden state outputs from the encoder RNN. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kfK1mAp1dlpT"
   },
   "outputs": [],
   "source": [
    "# Arguments\n",
    "args = Namespace(\n",
    "    batch_size=64,\n",
    "    embedding_dim=100,\n",
    "    rnn_hidden_dim=100,\n",
    "    hidden_dim=100,\n",
    "    num_layers=1,\n",
    "    bidirectional=False,\n",
    "    dropout=0.1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p_OJFyY97bF_"
   },
   "outputs": [],
   "source": [
    "class Encoder(nn.Module):\n",
    "    def __init__(self, embedding_dim, num_embeddings, rnn_hidden_dim, \n",
    "                 num_layers, bidirectional, padding_idx=0):\n",
    "        super(Encoder, self).__init__()\n",
    "        \n",
    "        # Embeddings\n",
    "        self.word_embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
    "                                            num_embeddings=num_embeddings,\n",
    "                                            padding_idx=padding_idx)\n",
    "        \n",
    "        # GRU weights\n",
    "        self.gru = nn.GRU(input_size=embedding_dim, hidden_size=rnn_hidden_dim, \n",
    "                          num_layers=num_layers, batch_first=True, \n",
    "                          bidirectional=bidirectional)\n",
    "\n",
    "    def forward(self, x_in, x_lengths):\n",
    "        \n",
    "        # Word level embeddings\n",
    "        z_word = self.word_embeddings(x_in)\n",
    "   \n",
    "        # Feed into RNN\n",
    "        out, h_n = self.gru(z)\n",
    "        \n",
    "        # Gather the last relevant hidden state\n",
    "        out = gather_last_relevant_hidden(out, x_lengths)\n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HRXtaGPlpyH7"
   },
   "outputs": [],
   "source": [
    "class Decoder(nn.Module):\n",
    "    def __init__(self, rnn_hidden_dim, hidden_dim, output_dim, dropout_p):\n",
    "        super(Decoder, self).__init__()\n",
    "        \n",
    "        # FC weights\n",
    "        self.dropout = nn.Dropout(dropout_p)\n",
    "        self.fc1 = nn.Linear(rnn_hidden_dim, hidden_dim)\n",
    "        self.fc2 = nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "    def forward(self, encoder_output, apply_softmax=False):\n",
    "        \n",
    "        # FC layers\n",
    "        z = self.dropout(encoder_output)\n",
    "        z = self.fc1(z)\n",
    "        z = self.dropout(z)\n",
    "        y_pred = self.fc2(z)\n",
    "\n",
    "        if apply_softmax:\n",
    "            y_pred = F.softmax(y_pred, dim=1)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SnKyCPVj-OVi"
   },
   "outputs": [],
   "source": [
    "class Model(nn.Module):\n",
    "    def __init__(self, embedding_dim, num_embeddings, rnn_hidden_dim, \n",
    "                 hidden_dim, num_layers, bidirectional, output_dim, dropout_p, \n",
    "                 padding_idx=0):\n",
    "        super(Model, self).__init__()\n",
    "        self.encoder = Encoder(embedding_dim, num_embeddings, rnn_hidden_dim, \n",
    "                               num_layers, bidirectional, padding_idx=0)\n",
    "        self.decoder = Decoder(rnn_hidden_dim, hidden_dim, output_dim, dropout_p)\n",
    "        \n",
    "    def forward(self, x_in, x_lengths, apply_softmax=False):\n",
    "        encoder_outputs = self.encoder(x_in, x_lengths)\n",
    "        y_pred = self.decoder(encoder_outputs, apply_softmax)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "hfeoErsc-Tum",
    "outputId": "8d95f223-076a-4ae7-b3cd-118040ea633e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method Module.named_parameters of Model(\n",
      "  (encoder): Encoder(\n",
      "    (word_embeddings): Embedding(1000, 100, padding_idx=0)\n",
      "    (gru): GRU(100, 100, batch_first=True)\n",
      "  )\n",
      "  (decoder): Decoder(\n",
      "    (dropout): Dropout(p=0.1)\n",
      "    (fc1): Linear(in_features=100, out_features=100, bias=True)\n",
      "    (fc2): Linear(in_features=100, out_features=4, bias=True)\n",
      "  )\n",
      ")>\n"
     ]
    }
   ],
   "source": [
    "model = Model(embedding_dim=args.embedding_dim, num_embeddings=1000, \n",
    "              rnn_hidden_dim=args.rnn_hidden_dim, hidden_dim=args.hidden_dim, \n",
    "              num_layers=args.num_layers, bidirectional=args.bidirectional, \n",
    "              output_dim=4, dropout_p=args.dropout, padding_idx=0)\n",
    "print (model.named_parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LAsOI6jEmTd0"
   },
   "source": [
    "# Attentional mechanisms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vJN5ft5Sc_kb"
   },
   "source": [
    "When processing an input sequence with an RNN, recall that at each time step we process the input and the hidden state at that time step. For many use cases, it's advantageous to have access to the inputs at all time steps and pay selective attention to the them at each time step. For example, in machine translation, it's advantageous to have access to all the words when translating to another language because translations aren't necessarily word for word. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jb6A6WfbXje6"
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/attention1.jpg\" width=650>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mNkayU0rf-ua"
   },
   "source": [
    "Attention can sound a bit confusing so let's see what happens at each time step. At time step j, the model has processed inputs $x_0, x_1, x_2, ..., x_j$ and has generted hidden states $h_0, h_1, h_2, ..., h_j$. The idea is to use all the processed hidden states to make the prediction and not just the most recent one. There are several approaches to how we can do this.\n",
    "\n",
    "With **soft attention**, we learn a vector of floating points (probabilities) to multiply with the hidden states to create the context vector.\n",
    "\n",
    "Ex. [0.1, 0.3, 0.1, 0.4, 0.1]\n",
    "\n",
    "With **hard attention**, we can learn a binary vector to multiply with the hidden states to create the context vector. \n",
    "\n",
    "Ex. [0, 0, 0, 1, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gYSIAVQqu3Ab"
   },
   "source": [
    "We're going to focus on soft attention because it's more widley used and we can visualize how much of each hidden state helps with the prediction, which is great for interpretability. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9Ch21nZNvDHO"
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/attention2.jpg\" width=650>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "o_jPXuT8xlqd"
   },
   "source": [
    "We're going to implement attention in the document classification task below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0iNnQzdxnGvn"
   },
   "source": [
    "# Document classification with RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n38ZJoVZnGaE"
   },
   "source": [
    "We're going to implement the same document classification task as in the previous notebook but we're going to use an attentional interface for interpretability.\n",
    "\n",
    "**Why not machine translation?** Normally, machine translation is the go-to example for demonstrating attention but it's not really practical. How many situations can you think of that require a seq to generate another sequence? Instead we're going to apply attention with our document classification example to see which input tokens are more influential towards predicting the genre."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Fu7HgEqbnGFY"
   },
   "source": [
    "## Set up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "elL6BxtCmNGf"
   },
   "outputs": [],
   "source": [
    "from argparse import Namespace\n",
    "import collections\n",
    "import copy\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import re\n",
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DCf2fLmPbKKI"
   },
   "outputs": [],
   "source": [
    "def set_seeds(seed, cuda):\n",
    "    np.random.seed(seed)\n",
    "    torch.manual_seed(seed)\n",
    "    if cuda:\n",
    "        torch.cuda.manual_seed_all(seed)\n",
    "        \n",
    "# Creating directories\n",
    "def create_dirs(dirpath):\n",
    "    if not os.path.exists(dirpath):\n",
    "        os.makedirs(dirpath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "TTwkuoZdmMlF",
    "outputId": "f0441822-890f-450c-807c-ba70ec58a573"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using CUDA: True\n"
     ]
    }
   ],
   "source": [
    "args = Namespace(\n",
    "    seed=1234,\n",
    "    cuda=True,\n",
    "    shuffle=True,\n",
    "    data_file=\"news.csv\",\n",
    "    vectorizer_file=\"vectorizer.json\",\n",
    "    model_state_file=\"model.pth\",\n",
    "    save_dir=\"news\",\n",
    "    train_size=0.7,\n",
    "    val_size=0.15,\n",
    "    test_size=0.15,\n",
    "    pretrained_embeddings=None,\n",
    "    cutoff=25,\n",
    "    num_epochs=5,\n",
    "    early_stopping_criteria=5,\n",
    "    learning_rate=1e-3,\n",
    "    batch_size=128,\n",
    "    embedding_dim=100,\n",
    "    kernels=[3,5],\n",
    "    num_filters=100,\n",
    "    rnn_hidden_dim=128,\n",
    "    hidden_dim=200,\n",
    "    num_layers=1,\n",
    "    bidirectional=False,\n",
    "    dropout_p=0.25,\n",
    ")\n",
    "\n",
    "# Set seeds\n",
    "set_seeds(seed=args.seed, cuda=args.cuda)\n",
    "\n",
    "# Create save dir\n",
    "create_dirs(args.save_dir)\n",
    "\n",
    "# Expand filepaths\n",
    "args.vectorizer_file = os.path.join(args.save_dir, args.vectorizer_file)\n",
    "args.model_state_file = os.path.join(args.save_dir, args.model_state_file)\n",
    "\n",
    "# Check CUDA\n",
    "if not torch.cuda.is_available():\n",
    "    args.cuda = False\n",
    "args.device = torch.device(\"cuda\" if args.cuda else \"cpu\")\n",
    "print(\"Using CUDA: {}\".format(args.cuda))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xfiWhgX5mMQ5"
   },
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "baAsxXNFmMCF"
   },
   "outputs": [],
   "source": [
    "import urllib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3tJi_HyOmLw-"
   },
   "outputs": [],
   "source": [
    "url = \"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/data/news.csv\"\n",
    "response = urllib.request.urlopen(url)\n",
    "html = response.read()\n",
    "with open(args.data_file, 'wb') as fp:\n",
    "    fp.write(html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "wrI_df4bmLjB",
    "outputId": "a6436ebb-e1b5-4661-aee0-6ba1c6c94c0b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Business</td>\n",
       "      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Business</td>\n",
       "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Business</td>\n",
       "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Business</td>\n",
       "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Business</td>\n",
       "      <td>Oil prices soar to all-time record, posing new...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   category                                              title\n",
       "0  Business  Wall St. Bears Claw Back Into the Black (Reuters)\n",
       "1  Business  Carlyle Looks Toward Commercial Aerospace (Reu...\n",
       "2  Business    Oil and Economy Cloud Stocks' Outlook (Reuters)\n",
       "3  Business  Iraq Halts Oil Exports from Main Southern Pipe...\n",
       "4  Business  Oil prices soar to all-time record, posing new..."
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(args.data_file, header=0)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "TreK7nqEmLTN",
    "outputId": "a75e4563-b073-4006-e157-ce8c78f2423f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Business: 30000\n",
      "Sci/Tech: 30000\n",
      "Sports: 30000\n",
      "World: 30000\n"
     ]
    }
   ],
   "source": [
    "by_category = collections.defaultdict(list)\n",
    "for _, row in df.iterrows():\n",
    "    by_category[row.category].append(row.to_dict())\n",
    "for category in by_category:\n",
    "    print (\"{0}: {1}\".format(category, len(by_category[category])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "35nb3LxLmLCA"
   },
   "outputs": [],
   "source": [
    "final_list = []\n",
    "for _, item_list in sorted(by_category.items()):\n",
    "    if args.shuffle:\n",
    "        np.random.shuffle(item_list)\n",
    "    n = len(item_list)\n",
    "    n_train = int(args.train_size*n)\n",
    "    n_val = int(args.val_size*n)\n",
    "    n_test = int(args.test_size*n)\n",
    "\n",
    "  # Give data point a split attribute\n",
    "    for item in item_list[:n_train]:\n",
    "        item['split'] = 'train'\n",
    "    for item in item_list[n_train:n_train+n_val]:\n",
    "        item['split'] = 'val'\n",
    "    for item in item_list[n_train+n_val:]:\n",
    "        item['split'] = 'test'  \n",
    "\n",
    "    # Add to final list\n",
    "    final_list.extend(item_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "Y48IvuSfmK07",
    "outputId": "b26ae123-5a2d-4ef6-a27e-7d32c8aa6f6d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "train    84000\n",
       "test     18000\n",
       "val      18000\n",
       "Name: split, dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "split_df = pd.DataFrame(final_list)\n",
    "split_df[\"split\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "RWuNBxAXmKk2",
    "outputId": "9accd13e-d4f1-416f-84e6-acd756dcc238"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>category</th>\n",
       "      <th>split</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Business</td>\n",
       "      <td>train</td>\n",
       "      <td>general electric posts higher rd quarter profit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Business</td>\n",
       "      <td>train</td>\n",
       "      <td>lilly to eliminate up to us jobs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Business</td>\n",
       "      <td>train</td>\n",
       "      <td>s amp p lowers america west outlook to negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Business</td>\n",
       "      <td>train</td>\n",
       "      <td>does rand walk the talk on labor policy ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Business</td>\n",
       "      <td>train</td>\n",
       "      <td>housekeeper advocates for changes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   category  split                                            title\n",
       "0  Business  train  general electric posts higher rd quarter profit\n",
       "1  Business  train                 lilly to eliminate up to us jobs\n",
       "2  Business  train  s amp p lowers america west outlook to negative\n",
       "3  Business  train        does rand walk the talk on labor policy ?\n",
       "4  Business  train                housekeeper advocates for changes"
      ]
     },
     "execution_count": 30,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def preprocess_text(text):\n",
    "    text = ' '.join(word.lower() for word in text.split(\" \"))\n",
    "    text = re.sub(r\"([.,!?])\", r\" \\1 \", text)\n",
    "    text = re.sub(r\"[^a-zA-Z.,!?]+\", r\" \", text)\n",
    "    text = text.strip()\n",
    "    return text\n",
    "    \n",
    "split_df.title = split_df.title.apply(preprocess_text)\n",
    "split_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m-a0OpqhmKJc"
   },
   "source": [
    "## Vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RUMQ_MwumJ8F"
   },
   "outputs": [],
   "source": [
    "class Vocabulary(object):\n",
    "    def __init__(self, token_to_idx=None):\n",
    "\n",
    "        # Token to index\n",
    "        if token_to_idx is None:\n",
    "            token_to_idx = {}\n",
    "        self.token_to_idx = token_to_idx\n",
    "\n",
    "        # Index to token\n",
    "        self.idx_to_token = {idx: token \\\n",
    "                             for token, idx in self.token_to_idx.items()}\n",
    "\n",
    "    def to_serializable(self):\n",
    "        return {'token_to_idx': self.token_to_idx}\n",
    "\n",
    "    @classmethod\n",
    "    def from_serializable(cls, contents):\n",
    "        return cls(**contents)\n",
    "\n",
    "    def add_token(self, token):\n",
    "        if token in self.token_to_idx:\n",
    "            index = self.token_to_idx[token]\n",
    "        else:\n",
    "            index = len(self.token_to_idx)\n",
    "            self.token_to_idx[token] = index\n",
    "            self.idx_to_token[index] = token\n",
    "        return index\n",
    "\n",
    "    def add_tokens(self, tokens):\n",
    "        return [self.add_token[token] for token in tokens]\n",
    "\n",
    "    def lookup_token(self, token):\n",
    "        return self.token_to_idx[token]\n",
    "\n",
    "    def lookup_index(self, index):\n",
    "        if index not in self.idx_to_token:\n",
    "            raise KeyError(\"the index (%d) is not in the Vocabulary\" % index)\n",
    "        return self.idx_to_token[index]\n",
    "\n",
    "    def __str__(self):\n",
    "        return \"<Vocabulary(size=%d)>\" % len(self)\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.token_to_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "1LtYf3lpExBb",
    "outputId": "48282730-351e-423d-86f0-c93492ecb6f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Vocabulary(size=4)>\n",
      "4\n",
      "0\n",
      "Business\n"
     ]
    }
   ],
   "source": [
    "# Vocabulary instance\n",
    "category_vocab = Vocabulary()\n",
    "for index, row in df.iterrows():\n",
    "    category_vocab.add_token(row.category)\n",
    "print (category_vocab) # __str__\n",
    "print (len(category_vocab)) # __len__\n",
    "index = category_vocab.lookup_token(\"Business\")\n",
    "print (index)\n",
    "print (category_vocab.lookup_index(index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z0zkF6CsE_yH"
   },
   "source": [
    "## Sequence vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QtntaISyE_1c"
   },
   "source": [
    "Next, we're going to create our Vocabulary classes for the article's title, which is a sequence of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ovI8QRefEw_p"
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4W3ZouuTEw1_"
   },
   "outputs": [],
   "source": [
    "class SequenceVocabulary(Vocabulary):\n",
    "    def __init__(self, token_to_idx=None, unk_token=\"<UNK>\",\n",
    "                 mask_token=\"<MASK>\", begin_seq_token=\"<BEGIN>\",\n",
    "                 end_seq_token=\"<END>\"):\n",
    "\n",
    "        super(SequenceVocabulary, self).__init__(token_to_idx)\n",
    "\n",
    "        self.mask_token = mask_token\n",
    "        self.unk_token = unk_token\n",
    "        self.begin_seq_token = begin_seq_token\n",
    "        self.end_seq_token = end_seq_token\n",
    "\n",
    "        self.mask_index = self.add_token(self.mask_token)\n",
    "        self.unk_index = self.add_token(self.unk_token)\n",
    "        self.begin_seq_index = self.add_token(self.begin_seq_token)\n",
    "        self.end_seq_index = self.add_token(self.end_seq_token)\n",
    "        \n",
    "        # Index to token\n",
    "        self.idx_to_token = {idx: token \\\n",
    "                             for token, idx in self.token_to_idx.items()}\n",
    "\n",
    "    def to_serializable(self):\n",
    "        contents = super(SequenceVocabulary, self).to_serializable()\n",
    "        contents.update({'unk_token': self.unk_token,\n",
    "                         'mask_token': self.mask_token,\n",
    "                         'begin_seq_token': self.begin_seq_token,\n",
    "                         'end_seq_token': self.end_seq_token})\n",
    "        return contents\n",
    "\n",
    "    def lookup_token(self, token):\n",
    "        return self.token_to_idx.get(token, self.unk_index)\n",
    "    \n",
    "    def lookup_index(self, index):\n",
    "        if index not in self.idx_to_token:\n",
    "            raise KeyError(\"the index (%d) is not in the SequenceVocabulary\" % index)\n",
    "        return self.idx_to_token[index]\n",
    "    \n",
    "    def __str__(self):\n",
    "        return \"<SequenceVocabulary(size=%d)>\" % len(self.token_to_idx)\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.token_to_idx)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "g5UHjpi3El37",
    "outputId": "563d51cd-0de7-49e8-cda8-7d94075d45fb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<SequenceVocabulary(size=4400)>\n",
      "4400\n",
      "4\n",
      "general\n"
     ]
    }
   ],
   "source": [
    "# Get word counts\n",
    "word_counts = Counter()\n",
    "for title in split_df.title:\n",
    "    for token in title.split(\" \"):\n",
    "        if token not in string.punctuation:\n",
    "            word_counts[token] += 1\n",
    "\n",
    "# Create SequenceVocabulary instance\n",
    "title_word_vocab = SequenceVocabulary()\n",
    "for word, word_count in word_counts.items():\n",
    "    if word_count >= args.cutoff:\n",
    "        title_word_vocab.add_token(word)\n",
    "print (title_word_vocab) # __str__\n",
    "print (len(title_word_vocab)) # __len__\n",
    "index = title_word_vocab.lookup_token(\"general\")\n",
    "print (index)\n",
    "print (title_word_vocab.lookup_index(index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1_wja0EfQNpA"
   },
   "source": [
    "We're also going to create an instance fo SequenceVocabulary that processes the input on a character level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "5SpfS0BXP9pz",
    "outputId": "4668016c-b513-470a-db4a-8830f1531ec4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<SequenceVocabulary(size=35)>\n",
      "35\n",
      "4\n",
      "g\n"
     ]
    }
   ],
   "source": [
    "# Create SequenceVocabulary instance\n",
    "title_char_vocab = SequenceVocabulary()\n",
    "for title in split_df.title:\n",
    "    for token in title:\n",
    "        title_char_vocab.add_token(token)\n",
    "print (title_char_vocab) # __str__\n",
    "print (len(title_char_vocab)) # __len__\n",
    "index = title_char_vocab.lookup_token(\"g\")\n",
    "print (index)\n",
    "print (title_char_vocab.lookup_index(index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4Dag6H0SFHAG"
   },
   "source": [
    "## Vectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VQIfxcUuKwzz"
   },
   "source": [
    "Something new that we introduce in this Vectorizer is calculating the length of our input sequence. We will use this later on to extract the last relevant hidden state for each input sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tsNtEnhBEl6s"
   },
   "outputs": [],
   "source": [
    "class NewsVectorizer(object):\n",
    "    def __init__(self, title_word_vocab, title_char_vocab, category_vocab):\n",
    "        self.title_word_vocab = title_word_vocab\n",
    "        self.title_char_vocab = title_char_vocab\n",
    "        self.category_vocab = category_vocab\n",
    "\n",
    "    def vectorize(self, title):\n",
    "       \n",
    "        # Word-level vectorization\n",
    "        word_indices = [self.title_word_vocab.lookup_token(token) for token in title.split(\" \")]\n",
    "        word_indices = [self.title_word_vocab.begin_seq_index] + word_indices + \\\n",
    "            [self.title_word_vocab.end_seq_index]\n",
    "        title_length = len(word_indices)\n",
    "        word_vector = np.zeros(title_length, dtype=np.int64)\n",
    "        word_vector[:len(word_indices)] = word_indices\n",
    "        \n",
    "        # Char-level vectorization\n",
    "        word_length = max([len(word) for word in title.split(\" \")])\n",
    "        char_vector = np.zeros((len(word_vector), word_length), dtype=np.int64)\n",
    "        char_vector[0, :] = self.title_word_vocab.mask_index # <BEGIN>\n",
    "        char_vector[-1, :] = self.title_word_vocab.mask_index # <END>\n",
    "        for i, word in enumerate(title.split(\" \")):\n",
    "            char_vector[i+1,:len(word)] = [title_char_vocab.lookup_token(char) \\\n",
    "                                           for char in word] # i+1 b/c of <BEGIN> token\n",
    "                \n",
    "        return word_vector, char_vector, len(word_indices)\n",
    "    \n",
    "    def unvectorize_word_vector(self, word_vector):\n",
    "        tokens = [self.title_word_vocab.lookup_index(index) for index in word_vector]\n",
    "        title = \" \".join(token for token in tokens)\n",
    "        return title\n",
    "    \n",
    "    def unvectorize_char_vector(self, char_vector):\n",
    "        title = \"\"\n",
    "        for word_vector in char_vector:\n",
    "            for index in word_vector:\n",
    "                if index == self.title_char_vocab.mask_index:\n",
    "                    break\n",
    "                title += self.title_char_vocab.lookup_index(index)\n",
    "            title += \" \"\n",
    "        return title\n",
    "    \n",
    "    @classmethod\n",
    "    def from_dataframe(cls, df, cutoff):\n",
    "        \n",
    "        # Create class vocab\n",
    "        category_vocab = Vocabulary()        \n",
    "        for category in sorted(set(df.category)):\n",
    "            category_vocab.add_token(category)\n",
    "\n",
    "        # Get word counts\n",
    "        word_counts = Counter()\n",
    "        for title in df.title:\n",
    "            for token in title.split(\" \"):\n",
    "                word_counts[token] += 1\n",
    "        \n",
    "        # Create title vocab (word level)\n",
    "        title_word_vocab = SequenceVocabulary()\n",
    "        for word, word_count in word_counts.items():\n",
    "            if word_count >= cutoff:\n",
    "                title_word_vocab.add_token(word)\n",
    "                \n",
    "        # Create title vocab (char level)\n",
    "        title_char_vocab = SequenceVocabulary()\n",
    "        for title in df.title:\n",
    "            for token in title:\n",
    "                title_char_vocab.add_token(token)\n",
    "        \n",
    "        return cls(title_word_vocab, title_char_vocab, category_vocab)\n",
    "\n",
    "    @classmethod\n",
    "    def from_serializable(cls, contents):\n",
    "        title_word_vocab = SequenceVocabulary.from_serializable(contents['title_word_vocab'])\n",
    "        title_char_vocab = SequenceVocabulary.from_serializable(contents['title_char_vocab'])\n",
    "        category_vocab = Vocabulary.from_serializable(contents['category_vocab'])\n",
    "        return cls(title_word_vocab=title_word_vocab, \n",
    "                   title_char_vocab=title_char_vocab, \n",
    "                   category_vocab=category_vocab)\n",
    "    \n",
    "    def to_serializable(self):\n",
    "        return {'title_word_vocab': self.title_word_vocab.to_serializable(),\n",
    "                'title_char_vocab': self.title_char_vocab.to_serializable(),\n",
    "                'category_vocab': self.category_vocab.to_serializable()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 340
    },
    "colab_type": "code",
    "id": "JtRRXU53El9Y",
    "outputId": "9a30879c-af8a-4630-fc4d-c55251238175"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<SequenceVocabulary(size=4404)>\n",
      "<SequenceVocabulary(size=35)>\n",
      "<Vocabulary(size=4)>\n",
      "word_vector: (10,)\n",
      "char_vector: (10, 10)\n",
      "title_length: 10\n",
      "[   2    1 4151 1231   25    1 2392 4076   38    3]\n",
      "[[ 0  0  0  0  0  0  0  0  0  0]\n",
      " [ 7 15  4  5  7  0  0  0  0  0]\n",
      " [21  5 18  5  7  5  7  0  0  0]\n",
      " [26 13  6 16  0  0  0  0  0  0]\n",
      " [12 17  5  0  0  0  0  0  0  0]\n",
      " [26 13 23 25  9  5 18 15  6  0]\n",
      " [12  5  6  6 13 16  0  0  0  0]\n",
      " [12 15 20  7  6  8 23  5  6 12]\n",
      " [30  0  0  0  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0  0  0  0]]\n",
      "<BEGIN> <UNK> federer wins the <UNK> tennis tournament . <END>\n",
      " roger federer wins the wimbledon tennis tournament .  \n"
     ]
    }
   ],
   "source": [
    "# Vectorizer instance\n",
    "vectorizer = NewsVectorizer.from_dataframe(split_df, cutoff=args.cutoff)\n",
    "print (vectorizer.title_word_vocab)\n",
    "print (vectorizer.title_char_vocab)\n",
    "print (vectorizer.category_vocab)\n",
    "word_vector, char_vector, title_length = vectorizer.vectorize(preprocess_text(\n",
    "    \"Roger Federer wins the Wimbledon tennis tournament.\"))\n",
    "print (\"word_vector:\", np.shape(word_vector))\n",
    "print (\"char_vector:\", np.shape(char_vector))\n",
    "print (\"title_length:\", title_length)\n",
    "print (word_vector)\n",
    "print (char_vector)\n",
    "print (vectorizer.unvectorize_word_vector(word_vector))\n",
    "print (vectorizer.unvectorize_char_vector(char_vector))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uk_QvpVfFM0S"
   },
   "source": [
    "## Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oU7oDdelFMR9"
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pB7FHmiSFMXA"
   },
   "outputs": [],
   "source": [
    "class NewsDataset(Dataset):\n",
    "    def __init__(self, df, vectorizer):\n",
    "        self.df = df\n",
    "        self.vectorizer = vectorizer\n",
    "\n",
    "        # Data splits\n",
    "        self.train_df = self.df[self.df.split=='train']\n",
    "        self.train_size = len(self.train_df)\n",
    "        self.val_df = self.df[self.df.split=='val']\n",
    "        self.val_size = len(self.val_df)\n",
    "        self.test_df = self.df[self.df.split=='test']\n",
    "        self.test_size = len(self.test_df)\n",
    "        self.lookup_dict = {'train': (self.train_df, self.train_size), \n",
    "                            'val': (self.val_df, self.val_size),\n",
    "                            'test': (self.test_df, self.test_size)}\n",
    "        self.set_split('train')\n",
    "\n",
    "        # Class weights (for imbalances)\n",
    "        class_counts = df.category.value_counts().to_dict()\n",
    "        def sort_key(item):\n",
    "            return self.vectorizer.category_vocab.lookup_token(item[0])\n",
    "        sorted_counts = sorted(class_counts.items(), key=sort_key)\n",
    "        frequencies = [count for _, count in sorted_counts]\n",
    "        self.class_weights = 1.0 / torch.tensor(frequencies, dtype=torch.float32)\n",
    "\n",
    "    @classmethod\n",
    "    def load_dataset_and_make_vectorizer(cls, df, cutoff):\n",
    "        train_df = df[df.split=='train']\n",
    "        return cls(df, NewsVectorizer.from_dataframe(train_df, cutoff))\n",
    "\n",
    "    @classmethod\n",
    "    def load_dataset_and_load_vectorizer(cls, df, vectorizer_filepath):\n",
    "        vectorizer = cls.load_vectorizer_only(vectorizer_filepath)\n",
    "        return cls(df, vectorizer)\n",
    "\n",
    "    def load_vectorizer_only(vectorizer_filepath):\n",
    "        with open(vectorizer_filepath) as fp:\n",
    "            return NewsVectorizer.from_serializable(json.load(fp))\n",
    "\n",
    "    def save_vectorizer(self, vectorizer_filepath):\n",
    "        with open(vectorizer_filepath, \"w\") as fp:\n",
    "            json.dump(self.vectorizer.to_serializable(), fp)\n",
    "\n",
    "    def set_split(self, split=\"train\"):\n",
    "        self.target_split = split\n",
    "        self.target_df, self.target_size = self.lookup_dict[split]\n",
    "\n",
    "    def __str__(self):\n",
    "        return \"<Dataset(split={0}, size={1})\".format(\n",
    "            self.target_split, self.target_size)\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.target_size\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        row = self.target_df.iloc[index]\n",
    "        title_word_vector, title_char_vector, title_length = \\\n",
    "            self.vectorizer.vectorize(row.title)\n",
    "        category_index = self.vectorizer.category_vocab.lookup_token(row.category)\n",
    "        return {'title_word_vector': title_word_vector, \n",
    "                'title_char_vector': title_char_vector, \n",
    "                'title_length': title_length, \n",
    "                'category': category_index}\n",
    "\n",
    "    def get_num_batches(self, batch_size):\n",
    "        return len(self) // batch_size\n",
    "\n",
    "    def generate_batches(self, batch_size, collate_fn, shuffle=True, \n",
    "                         drop_last=False, device=\"cpu\"):\n",
    "        dataloader = DataLoader(dataset=self, batch_size=batch_size,\n",
    "                                collate_fn=collate_fn, shuffle=shuffle, \n",
    "                                drop_last=drop_last)\n",
    "        for data_dict in dataloader:\n",
    "            out_data_dict = {}\n",
    "            for name, tensor in data_dict.items():\n",
    "                out_data_dict[name] = data_dict[name].to(device)\n",
    "            yield out_data_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "id": "_Dpb6ZHJFMeb",
    "outputId": "6d2dbb9b-2921-4267-9a3e-dbd9033f6bd4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Dataset(split=train, size=84000)\n",
      "[ 2 51  1 52 53 26 54  3]\n",
      "[[ 0  0  0  0  0  0  0  0  0  0]\n",
      " [18  5  9 12  8  0  0  0  0  0]\n",
      " [18 15 18  4  5 16  0  0  0  0]\n",
      " [25  8  6 27  7 20 14 12 11 22]\n",
      " [26 13 12 17  0  0  0  0  0  0]\n",
      " [ 9  8 25 15  7  0  0  0  0  0]\n",
      " [18  5  8  9  0  0  0  0  0  0]\n",
      " [ 0  0  0  0  0  0  0  0  0  0]]\n",
      "8\n",
      "0\n",
      "<BEGIN> delta <UNK> bankruptcy with labor deal <END>\n",
      " delta dodges bankruptcy with labor deal  \n",
      "tensor([3.3333e-05, 3.3333e-05, 3.3333e-05, 3.3333e-05])\n"
     ]
    }
   ],
   "source": [
    "# Dataset instance\n",
    "dataset = NewsDataset.load_dataset_and_make_vectorizer(df=split_df,\n",
    "                                                       cutoff=args.cutoff)\n",
    "print (dataset) # __str__\n",
    "input_ = dataset[10] # __getitem__\n",
    "print (input_['title_word_vector'])\n",
    "print (input_['title_char_vector'])\n",
    "print (input_['title_length'])\n",
    "print (input_['category'])\n",
    "print (dataset.vectorizer.unvectorize_word_vector(input_['title_word_vector']))\n",
    "print (dataset.vectorizer.unvectorize_char_vector(input_['title_char_vector']))\n",
    "print (dataset.class_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_IUIqtbvFUAG"
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xJV5WlDiFVVz"
   },
   "source": [
    "embed → encoder → attend → predict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rZCzdZZ9FMhm"
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "c9wipRZt7feC"
   },
   "outputs": [],
   "source": [
    "class NewsEncoder(nn.Module):\n",
    "    def __init__(self, embedding_dim, num_word_embeddings, num_char_embeddings,\n",
    "                 kernels, num_input_channels, num_output_channels, \n",
    "                 rnn_hidden_dim, num_layers, bidirectional, \n",
    "                 word_padding_idx=0, char_padding_idx=0):\n",
    "        super(NewsEncoder, self).__init__()\n",
    "        \n",
    "        self.num_layers = num_layers\n",
    "        self.bidirectional = bidirectional\n",
    "        \n",
    "        # Embeddings\n",
    "        self.word_embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
    "                                            num_embeddings=num_word_embeddings,\n",
    "                                            padding_idx=word_padding_idx)\n",
    "        self.char_embeddings = nn.Embedding(embedding_dim=embedding_dim,\n",
    "                                            num_embeddings=num_char_embeddings,\n",
    "                                            padding_idx=char_padding_idx)\n",
    "        \n",
    "        # Conv weights\n",
    "        self.conv = nn.ModuleList([nn.Conv1d(num_input_channels, \n",
    "                                             num_output_channels, \n",
    "                                             kernel_size=f) for f in kernels])\n",
    "        \n",
    "        \n",
    "        # GRU weights\n",
    "        self.gru = nn.GRU(input_size=embedding_dim*(len(kernels)+1), \n",
    "                          hidden_size=rnn_hidden_dim, num_layers=num_layers, \n",
    "                          batch_first=True, bidirectional=bidirectional)\n",
    "        \n",
    "    def initialize_hidden_state(self, batch_size, rnn_hidden_dim, device):\n",
    "        \"\"\"Modify this to condition the RNN.\"\"\"\n",
    "        num_directions = 1\n",
    "        if self.bidirectional:\n",
    "            num_directions = 2\n",
    "        hidden_t = torch.zeros(self.num_layers * num_directions, \n",
    "                               batch_size, rnn_hidden_dim).to(device)\n",
    "        \n",
    "    def get_char_level_embeddings(self, x):\n",
    "        # x: (N, seq_len, word_len)\n",
    "        input_shape = x.size()\n",
    "        batch_size, seq_len, word_len = input_shape\n",
    "        x = x.view(-1, word_len) # (N*seq_len, word_len)\n",
    "        \n",
    "        # Embedding\n",
    "        x = self.char_embeddings(x) # (N*seq_len, word_len, embedding_dim)\n",
    "        \n",
    "        # Rearrange input so num_input_channels is in dim 1 (N, embedding_dim, word_len)\n",
    "        x = x.transpose(1, 2)\n",
    "        \n",
    "        # Convolution\n",
    "        z = [F.relu(conv(x)) for conv in self.conv]\n",
    "        \n",
    "        # Pooling\n",
    "        z = [F.max_pool1d(zz, zz.size(2)).squeeze(2) for zz in z] \n",
    "        z = [zz.view(batch_size, seq_len, -1) for zz in z] # (N, seq_len, embedding_dim)\n",
    "        \n",
    "        # Concat to get char-level embeddings\n",
    "        z = torch.cat(z, 2) # join conv outputs\n",
    "        \n",
    "        return z\n",
    "        \n",
    "    def forward(self, x_word, x_char, x_lengths, device):\n",
    "        \"\"\"\n",
    "        x_word: word level representation (N, seq_size)\n",
    "        x_char: char level representation (N, seq_size, word_len)\n",
    "        \"\"\"\n",
    "        \n",
    "        # Word level embeddings\n",
    "        z_word = self.word_embeddings(x_word)\n",
    "        \n",
    "        # Char level embeddings\n",
    "        z_char = self.get_char_level_embeddings(x=x_char)\n",
    "        \n",
    "        # Concatenate\n",
    "        z = torch.cat([z_word, z_char], 2)\n",
    "        \n",
    "        # Feed into RNN\n",
    "        initial_h = self.initialize_hidden_state(\n",
    "            batch_size=z.size(0), rnn_hidden_dim=self.gru.hidden_size,\n",
    "            device=device)\n",
    "        out, h_n = self.gru(z, initial_h)\n",
    "        \n",
    "        return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zeEcdA287gz4"
   },
   "outputs": [],
   "source": [
    "class NewsDecoder(nn.Module):\n",
    "    def __init__(self, rnn_hidden_dim, hidden_dim, output_dim, dropout_p):\n",
    "        super(NewsDecoder, self).__init__()\n",
    "        \n",
    "        # Attention FC layer\n",
    "        self.fc_attn = nn.Linear(rnn_hidden_dim, rnn_hidden_dim)\n",
    "        self.v = nn.Parameter(torch.rand(rnn_hidden_dim))\n",
    "        \n",
    "        # FC weights\n",
    "        self.dropout = nn.Dropout(dropout_p)\n",
    "        self.fc1 = nn.Linear(rnn_hidden_dim, hidden_dim)\n",
    "        self.fc2 = nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "    def forward(self, encoder_outputs, apply_softmax=False):\n",
    "        \n",
    "        # Attention\n",
    "        z = torch.tanh(self.fc_attn(encoder_outputs))\n",
    "        z = z.transpose(2,1) # [B*H*T]\n",
    "        v = self.v.repeat(encoder_outputs.size(0),1).unsqueeze(1) #[B*1*H]\n",
    "        z = torch.bmm(v,z).squeeze(1) # [B*T]\n",
    "        attn_scores = F.softmax(z, dim=1)\n",
    "        context = torch.matmul(encoder_outputs.transpose(-2, -1), \n",
    "                               attn_scores.unsqueeze(dim=2)).squeeze()\n",
    "        if len(context.size()) == 1:\n",
    "            context = context.unsqueeze(0)\n",
    "        \n",
    "        # FC layers\n",
    "        z = self.dropout(context)\n",
    "        z = self.fc1(z)\n",
    "        z = self.dropout(z)\n",
    "        y_pred = self.fc2(z)\n",
    "\n",
    "        if apply_softmax:\n",
    "            y_pred = F.softmax(y_pred, dim=1)\n",
    "        return attn_scores, y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "yVDftS-G7gwy"
   },
   "outputs": [],
   "source": [
    "class NewsModel(nn.Module):\n",
    "    def __init__(self, embedding_dim, num_word_embeddings, num_char_embeddings,\n",
    "                 kernels, num_input_channels, num_output_channels, \n",
    "                 rnn_hidden_dim, hidden_dim, output_dim, num_layers, \n",
    "                 bidirectional, dropout_p, word_padding_idx, char_padding_idx):\n",
    "        super(NewsModel, self).__init__()\n",
    "        self.encoder = NewsEncoder(embedding_dim, num_word_embeddings,\n",
    "                                   num_char_embeddings, kernels, \n",
    "                                   num_input_channels, num_output_channels, \n",
    "                                   rnn_hidden_dim, num_layers, bidirectional, \n",
    "                                   word_padding_idx, char_padding_idx)\n",
    "        self.decoder = NewsDecoder(rnn_hidden_dim, hidden_dim, output_dim, \n",
    "                                   dropout_p)\n",
    "        \n",
    "    def forward(self, x_word, x_char, x_lengths, device, apply_softmax=False):\n",
    "        encoder_outputs = self.encoder(x_word, x_char, x_lengths, device)\n",
    "        y_pred = self.decoder(encoder_outputs, apply_softmax)\n",
    "        return y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jHPYCPd7Fl3M"
   },
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "D3seBMA7FlcC"
   },
   "outputs": [],
   "source": [
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "HnRKWLekFlnM"
   },
   "outputs": [],
   "source": [
    "class Trainer(object):\n",
    "    def __init__(self, dataset, model, model_state_file, save_dir, device, \n",
    "                 shuffle, num_epochs, batch_size, learning_rate, \n",
    "                 early_stopping_criteria):\n",
    "        self.dataset = dataset\n",
    "        self.class_weights = dataset.class_weights.to(device)\n",
    "        self.device = device\n",
    "        self.model = model.to(device)\n",
    "        self.save_dir = save_dir\n",
    "        self.device = device\n",
    "        self.shuffle = shuffle\n",
    "        self.num_epochs = num_epochs\n",
    "        self.batch_size = batch_size\n",
    "        self.loss_func = nn.CrossEntropyLoss(self.class_weights)\n",
    "        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)\n",
    "        self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(\n",
    "            optimizer=self.optimizer, mode='min', factor=0.5, patience=1)\n",
    "        self.train_state = {\n",
    "            'done_training': False,\n",
    "            'stop_early': False, \n",
    "            'early_stopping_step': 0,\n",
    "            'early_stopping_best_val': 1e8,\n",
    "            'early_stopping_criteria': early_stopping_criteria,\n",
    "            'learning_rate': learning_rate,\n",
    "            'epoch_index': 0,\n",
    "            'train_loss': [],\n",
    "            'train_acc': [],\n",
    "            'val_loss': [],\n",
    "            'val_acc': [],\n",
    "            'test_loss': -1,\n",
    "            'test_acc': -1,\n",
    "            'model_filename': model_state_file}\n",
    "    \n",
    "    def update_train_state(self):\n",
    "\n",
    "        # Verbose\n",
    "        print (\"[EPOCH]: {0:02d} | [LR]: {1} | [TRAIN LOSS]: {2:.2f} | [TRAIN ACC]: {3:.1f}% | [VAL LOSS]: {4:.2f} | [VAL ACC]: {5:.1f}%\".format(\n",
    "          self.train_state['epoch_index'], self.train_state['learning_rate'], \n",
    "            self.train_state['train_loss'][-1], self.train_state['train_acc'][-1], \n",
    "            self.train_state['val_loss'][-1], self.train_state['val_acc'][-1]))\n",
    "\n",
    "        # Save one model at least\n",
    "        if self.train_state['epoch_index'] == 0:\n",
    "            torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
    "            self.train_state['stop_early'] = False\n",
    "\n",
    "        # Save model if performance improved\n",
    "        elif self.train_state['epoch_index'] >= 1:\n",
    "            loss_tm1, loss_t = self.train_state['val_loss'][-2:]\n",
    "\n",
    "            # If loss worsened\n",
    "            if loss_t >= self.train_state['early_stopping_best_val']:\n",
    "                # Update step\n",
    "                self.train_state['early_stopping_step'] += 1\n",
    "\n",
    "            # Loss decreased\n",
    "            else:\n",
    "                # Save the best model\n",
    "                if loss_t < self.train_state['early_stopping_best_val']:\n",
    "                    torch.save(self.model.state_dict(), self.train_state['model_filename'])\n",
    "\n",
    "                # Reset early stopping step\n",
    "                self.train_state['early_stopping_step'] = 0\n",
    "\n",
    "            # Stop early ?\n",
    "            self.train_state['stop_early'] = self.train_state['early_stopping_step'] \\\n",
    "              >= self.train_state['early_stopping_criteria']\n",
    "        return self.train_state\n",
    "  \n",
    "    def compute_accuracy(self, y_pred, y_target):\n",
    "        _, y_pred_indices = y_pred.max(dim=1)\n",
    "        n_correct = torch.eq(y_pred_indices, y_target).sum().item()\n",
    "        return n_correct / len(y_pred_indices) * 100\n",
    "    \n",
    "    def pad_word_seq(self, seq, length):\n",
    "        vector = np.zeros(length, dtype=np.int64)\n",
    "        vector[:len(seq)] = seq\n",
    "        vector[len(seq):] = self.dataset.vectorizer.title_word_vocab.mask_index\n",
    "        return vector\n",
    "    \n",
    "    def pad_char_seq(self, seq, seq_length, word_length):\n",
    "        vector = np.zeros((seq_length, word_length), dtype=np.int64)\n",
    "        vector.fill(self.dataset.vectorizer.title_char_vocab.mask_index)\n",
    "        for i in range(len(seq)):\n",
    "            char_padding = np.zeros(word_length-len(seq[i]), dtype=np.int64)\n",
    "            vector[i] = np.concatenate((seq[i], char_padding), axis=None)\n",
    "        return vector\n",
    "        \n",
    "    def collate_fn(self, batch):\n",
    "        \n",
    "        # Make a deep copy\n",
    "        batch_copy = copy.deepcopy(batch)\n",
    "        processed_batch = {\"title_word_vector\": [], \"title_char_vector\": [], \n",
    "                           \"title_length\": [], \"category\": []}\n",
    "             \n",
    "        # Max lengths\n",
    "        get_seq_length = lambda sample: len(sample[\"title_word_vector\"])\n",
    "        get_word_length = lambda sample: len(sample[\"title_char_vector\"][0])\n",
    "        max_seq_length = max(map(get_seq_length, batch))\n",
    "        max_word_length = max(map(get_word_length, batch))\n",
    "\n",
    "\n",
    "        # Pad\n",
    "        for i, sample in enumerate(batch_copy):\n",
    "            padded_word_seq = self.pad_word_seq(\n",
    "                sample[\"title_word_vector\"], max_seq_length)\n",
    "            padded_char_seq = self.pad_char_seq(\n",
    "                sample[\"title_char_vector\"], max_seq_length, max_word_length)\n",
    "            processed_batch[\"title_word_vector\"].append(padded_word_seq)\n",
    "            processed_batch[\"title_char_vector\"].append(padded_char_seq)\n",
    "            processed_batch[\"title_length\"].append(sample[\"title_length\"])\n",
    "            processed_batch[\"category\"].append(sample[\"category\"])\n",
    "            \n",
    "        # Convert to appropriate tensor types\n",
    "        processed_batch[\"title_word_vector\"] = torch.LongTensor(\n",
    "            processed_batch[\"title_word_vector\"])\n",
    "        processed_batch[\"title_char_vector\"] = torch.LongTensor(\n",
    "            processed_batch[\"title_char_vector\"])\n",
    "        processed_batch[\"title_length\"] = torch.LongTensor(\n",
    "            processed_batch[\"title_length\"])\n",
    "        processed_batch[\"category\"] = torch.LongTensor(\n",
    "            processed_batch[\"category\"])\n",
    "        \n",
    "        return processed_batch  \n",
    "  \n",
    "    def run_train_loop(self):\n",
    "        for epoch_index in range(self.num_epochs):\n",
    "            self.train_state['epoch_index'] = epoch_index\n",
    "      \n",
    "            # Iterate over train dataset\n",
    "\n",
    "            # initialize batch generator, set loss and acc to 0, set train mode on\n",
    "            self.dataset.set_split('train')\n",
    "            batch_generator = self.dataset.generate_batches(\n",
    "                batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
    "                shuffle=self.shuffle, device=self.device)\n",
    "            running_loss = 0.0\n",
    "            running_acc = 0.0\n",
    "            self.model.train()\n",
    "\n",
    "            for batch_index, batch_dict in enumerate(batch_generator):\n",
    "                # zero the gradients\n",
    "                self.optimizer.zero_grad()\n",
    "                \n",
    "                # compute the output\n",
    "                _, y_pred = self.model(x_word=batch_dict['title_word_vector'],\n",
    "                                       x_char=batch_dict['title_char_vector'],\n",
    "                                       x_lengths=batch_dict['title_length'],\n",
    "                                       device=self.device)\n",
    "                \n",
    "                # compute the loss\n",
    "                loss = self.loss_func(y_pred, batch_dict['category'])\n",
    "                loss_t = loss.item()\n",
    "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
    "\n",
    "                # compute gradients using loss\n",
    "                loss.backward()\n",
    "\n",
    "                # use optimizer to take a gradient step\n",
    "                self.optimizer.step()\n",
    "                \n",
    "                # compute the accuracy\n",
    "                acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
    "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
    "\n",
    "            self.train_state['train_loss'].append(running_loss)\n",
    "            self.train_state['train_acc'].append(running_acc)\n",
    "\n",
    "            # Iterate over val dataset\n",
    "\n",
    "            # initialize batch generator, set loss and acc to 0, set eval mode on\n",
    "            self.dataset.set_split('val')\n",
    "            batch_generator = self.dataset.generate_batches(\n",
    "                batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
    "                shuffle=self.shuffle, device=self.device)\n",
    "            running_loss = 0.\n",
    "            running_acc = 0.\n",
    "            self.model.eval()\n",
    "\n",
    "            for batch_index, batch_dict in enumerate(batch_generator):\n",
    "\n",
    "                # compute the output\n",
    "                _, y_pred = self.model(x_word=batch_dict['title_word_vector'],\n",
    "                                       x_char=batch_dict['title_char_vector'],\n",
    "                                       x_lengths=batch_dict['title_length'],\n",
    "                                       device=self.device)\n",
    "\n",
    "                # compute the loss\n",
    "                loss = self.loss_func(y_pred, batch_dict['category'])\n",
    "                loss_t = loss.to(\"cpu\").item()\n",
    "                running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
    "\n",
    "                # compute the accuracy\n",
    "                acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
    "                running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
    "\n",
    "            self.train_state['val_loss'].append(running_loss)\n",
    "            self.train_state['val_acc'].append(running_acc)\n",
    "\n",
    "            self.train_state = self.update_train_state()\n",
    "            self.scheduler.step(self.train_state['val_loss'][-1])\n",
    "            if self.train_state['stop_early']:\n",
    "                break\n",
    "          \n",
    "    def run_test_loop(self):\n",
    "        # initialize batch generator, set loss and acc to 0, set eval mode on\n",
    "        self.dataset.set_split('test')\n",
    "        batch_generator = self.dataset.generate_batches(\n",
    "            batch_size=self.batch_size, collate_fn=self.collate_fn, \n",
    "            shuffle=self.shuffle, device=self.device)\n",
    "        running_loss = 0.0\n",
    "        running_acc = 0.0\n",
    "        self.model.eval()\n",
    "\n",
    "        for batch_index, batch_dict in enumerate(batch_generator):\n",
    "            # compute the output\n",
    "            _, y_pred = self.model(x_word=batch_dict['title_word_vector'],\n",
    "                                   x_char=batch_dict['title_char_vector'],\n",
    "                                   x_lengths=batch_dict['title_length'],\n",
    "                                   device=self.device)\n",
    "\n",
    "            # compute the loss\n",
    "            loss = self.loss_func(y_pred, batch_dict['category'])\n",
    "            loss_t = loss.item()\n",
    "            running_loss += (loss_t - running_loss) / (batch_index + 1)\n",
    "\n",
    "            # compute the accuracy\n",
    "            acc_t = self.compute_accuracy(y_pred, batch_dict['category'])\n",
    "            running_acc += (acc_t - running_acc) / (batch_index + 1)\n",
    "\n",
    "        self.train_state['test_loss'] = running_loss\n",
    "        self.train_state['test_acc'] = running_acc\n",
    "    \n",
    "    def plot_performance(self):\n",
    "        # Figure size\n",
    "        plt.figure(figsize=(15,5))\n",
    "\n",
    "        # Plot Loss\n",
    "        plt.subplot(1, 2, 1)\n",
    "        plt.title(\"Loss\")\n",
    "        plt.plot(trainer.train_state[\"train_loss\"], label=\"train\")\n",
    "        plt.plot(trainer.train_state[\"val_loss\"], label=\"val\")\n",
    "        plt.legend(loc='upper right')\n",
    "\n",
    "        # Plot Accuracy\n",
    "        plt.subplot(1, 2, 2)\n",
    "        plt.title(\"Accuracy\")\n",
    "        plt.plot(trainer.train_state[\"train_acc\"], label=\"train\")\n",
    "        plt.plot(trainer.train_state[\"val_acc\"], label=\"val\")\n",
    "        plt.legend(loc='lower right')\n",
    "\n",
    "        # Save figure\n",
    "        plt.savefig(os.path.join(self.save_dir, \"performance.png\"))\n",
    "\n",
    "        # Show plots\n",
    "        plt.show()\n",
    "    \n",
    "    def save_train_state(self):\n",
    "        self.train_state[\"done_training\"] = True\n",
    "        with open(os.path.join(self.save_dir, \"train_state.json\"), \"w\") as fp:\n",
    "            json.dump(self.train_state, fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "id": "ICkiOaGtFlk-",
    "outputId": "bf789f19-1786-4beb-ca83-88084438c8c4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method Module.named_modules of NewsModel(\n",
      "  (encoder): NewsEncoder(\n",
      "    (word_embeddings): Embedding(3406, 100, padding_idx=0)\n",
      "    (char_embeddings): Embedding(35, 100, padding_idx=0)\n",
      "    (conv): ModuleList(\n",
      "      (0): Conv1d(100, 100, kernel_size=(3,), stride=(1,))\n",
      "      (1): Conv1d(100, 100, kernel_size=(5,), stride=(1,))\n",
      "    )\n",
      "    (gru): GRU(300, 128, batch_first=True)\n",
      "  )\n",
      "  (decoder): NewsDecoder(\n",
      "    (fc_attn): Linear(in_features=128, out_features=128, bias=True)\n",
      "    (dropout): Dropout(p=0.25)\n",
      "    (fc1): Linear(in_features=128, out_features=200, bias=True)\n",
      "    (fc2): Linear(in_features=200, out_features=4, bias=True)\n",
      "  )\n",
      ")>\n"
     ]
    }
   ],
   "source": [
    "# Initialization\n",
    "dataset = NewsDataset.load_dataset_and_make_vectorizer(df=split_df,\n",
    "                                                       cutoff=args.cutoff)\n",
    "dataset.save_vectorizer(args.vectorizer_file)\n",
    "vectorizer = dataset.vectorizer\n",
    "model = NewsModel(embedding_dim=args.embedding_dim, \n",
    "                  num_word_embeddings=len(vectorizer.title_word_vocab), \n",
    "                  num_char_embeddings=len(vectorizer.title_char_vocab),\n",
    "                  kernels=args.kernels,\n",
    "                  num_input_channels=args.embedding_dim,\n",
    "                  num_output_channels=args.num_filters,\n",
    "                  rnn_hidden_dim=args.rnn_hidden_dim,\n",
    "                  hidden_dim=args.hidden_dim,\n",
    "                  output_dim=len(vectorizer.category_vocab),\n",
    "                  num_layers=args.num_layers,\n",
    "                  bidirectional=args.bidirectional,\n",
    "                  dropout_p=args.dropout_p, \n",
    "                  word_padding_idx=vectorizer.title_word_vocab.mask_index,\n",
    "                  char_padding_idx=vectorizer.title_char_vocab.mask_index)\n",
    "print (model.named_modules)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "tuaRZ4DiFlh1",
    "outputId": "4ac9c984-583f-4aa2-b47d-d79d1ff220e0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[EPOCH]: 00 | [LR]: 0.001 | [TRAIN LOSS]: 0.77 | [TRAIN ACC]: 69.6% | [VAL LOSS]: 0.55 | [VAL ACC]: 80.5%\n",
      "[EPOCH]: 01 | [LR]: 0.001 | [TRAIN LOSS]: 0.51 | [TRAIN ACC]: 81.9% | [VAL LOSS]: 0.49 | [VAL ACC]: 82.3%\n",
      "[EPOCH]: 02 | [LR]: 0.001 | [TRAIN LOSS]: 0.43 | [TRAIN ACC]: 84.5% | [VAL LOSS]: 0.45 | [VAL ACC]: 83.6%\n",
      "[EPOCH]: 03 | [LR]: 0.001 | [TRAIN LOSS]: 0.39 | [TRAIN ACC]: 86.0% | [VAL LOSS]: 0.47 | [VAL ACC]: 83.4%\n",
      "[EPOCH]: 04 | [LR]: 0.001 | [TRAIN LOSS]: 0.36 | [TRAIN ACC]: 87.3% | [VAL LOSS]: 0.44 | [VAL ACC]: 84.6%\n"
     ]
    }
   ],
   "source": [
    "# Train\n",
    "trainer = Trainer(dataset=dataset, model=model, \n",
    "                  model_state_file=args.model_state_file, \n",
    "                  save_dir=args.save_dir, device=args.device,\n",
    "                  shuffle=args.shuffle, num_epochs=args.num_epochs, \n",
    "                  batch_size=args.batch_size, learning_rate=args.learning_rate, \n",
    "                  early_stopping_criteria=args.early_stopping_criteria)\n",
    "trainer.run_train_loop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 335
    },
    "colab_type": "code",
    "id": "mzRJIz88Flfe",
    "outputId": "fcc04182-d954-43da-cb04-98fb7fd69917"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAAE+CAYAAAD4XjP+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xl8lfWd9//XWbInJ/u+J0ACYd/3\nJYggoNQFl9ZlanvPaG37u2esgzftFOtgtb+7zFRtq12ctnY6tRVBRBQUJMgmu4BAwp6d7HtCcnJy\n7j8SDkRWIcmVk7yfj0cf5VzXdc555+CD5JPP9/p8TU6n04mIiIiIiIgYzmx0ABEREREREWmnAk1E\nRERERKSXUIEmIiIiIiLSS6hAExERERER6SVUoImIiIiIiPQSKtBERERERER6CRVoIjcpLS2Nc+fO\nGR1DRESkRzz44IPcddddRscQ6fNUoImIiIjINR0/fpyAgABiYmI4cOCA0XFE+jQVaCJdrLm5mR//\n+MfMnTuXO+64g5deegmHwwHAf//3f3PHHXcwb9487rvvPk6cOHHN4yIiIr3B6tWrmTdvHgsXLuTd\nd991HX/33XeZO3cuc+fO5ZlnnqGlpeWqx3ft2sWcOXNcz7308auvvsqPfvQj7rvvPv74xz/S1tbG\nT37yE+bOnUtmZibPPPMMdrsdgMrKSp544glmz57NnXfeybZt28jKymLhwoWdMt9zzz1s3Lixuz8a\nkS5nNTqASF/zpz/9iXPnzrFu3TpaW1t5+OGHef/995k9ezYvv/wymzdvxt/fnw8//JCsrCyio6Ov\neHzgwIFGfykiIiI4HA4+/vhjnnrqKSwWCytWrKClpYXS0lJ+9rOf8e677xIREcH3vvc93nzzTebN\nm3fF48OGDbvm+2zZsoU1a9YQEhLChg0b2Lt3L++//z5tbW3cfffdfPDBByxatIgVK1aQmprK66+/\nztGjR/nmN7/J1q1bKSsrIzs7m/T0dIqKisjLy2P69Ok99CmJdB0VaCJdLCsri8cffxyr1YrVauXO\nO+9k+/btzJ8/H5PJxMqVK1m4cCF33HEHAHa7/YrHRUREeoNt27YxbNgw/P39ARg/fjybN2+murqa\nUaNGERkZCcCKFSuwWCy88847Vzy+b9++a77PiBEjCAkJAWDu3LnMmjULDw8PAIYNG0Z+fj7QXsj9\n7ne/A2DIkCFs2rQJT09P5s6dy7p160hPT2fjxo3Mnj0bT0/Prv9ARLqZljiKdLHKykoCAwNdjwMD\nA6moqMDDw4M//vGP7N+/n7lz5/L1r3+dnJycqx4XERHpDVatWkVWVhZjx45l7NixfPTRR6xevZqq\nqipsNpvrOi8vL6xW61WPX8+l3zsrKytZsmQJc+fOZd68eWzatAmn0wlAdXU1AQEBrmsvFI4LFixg\n3bp1AGzcuJH58+ff2hcuYhAVaCJdLCwsjOrqatfj6upqwsLCgPbf9L3yyivs3LmTqVOnsmzZsmse\nFxERMVJNTQ27d+9m165d7N27l71797Jnzx4OHz6M2WymqqrKdW19fT3l5eUEBwdf8bjFYnHdkw1Q\nW1t71ff9z//8T6xWK2vXrmX9+vXMmDHDdS4oKKjT6xcUFGC32xk3bhytra1s3ryZEydOMHny5K76\nGER6lAo0kS42c+ZMVq5cicPhoLGxkTVr1jBjxgxycnL4/ve/T0tLC56engwdOhSTyXTV4yIiIkZb\nt24dEydO7LRU0Gq1MnXqVFpaWti/fz8FBQU4nU6WLVvGypUrmTFjxhWPh4eHU1ZWRkVFBQ6Hg7Vr\n1171fSsqKhg0aBCenp5kZ2dz4MABGhsbAcjMzGT16tUAnDx5knvuuQeHw4HZbGb+/Pn8+7//O5mZ\nma7lkSLuRvegidyCRx55BIvF4nq8fPlyHnnkEfLz81mwYAEmk4l58+a57iuLi4tj4cKFeHh44Ofn\nx49//GMGDRp0xeMiIiJGe/fdd3nssccuOz5nzhx+/etf8/zzz/PYY49hsVgYNmwY3/zmN/Hy8rrq\n8XvvvZevfe1rxMTEsGjRIo4dO3bF93388cdZsmQJq1atYuzYsSxZsoQf/vCHDB8+nGeeeYYlS5aQ\nmZmJn58fP//5z/H29gbalzn+4Q9/0PJGcWsm54UFvSIiIiIibqy8vJy7776brKysTr9AFXEnWuIo\nIiIiIn3CK6+8wkMPPaTiTNyaCjQRERERcWvl5eXMnj2b8vJyHn/8caPjiNwSLXEUERERERHpJdRB\nExERERER6SVUoImIiIiIiPQSPT5mv6ys7pZfIzjYl6qqxi5I0zPcKa+ydg93ygrulVdZu0dXZQ0P\nD+iCNP1Hf/se6U5Zwb3yKmv3UNbu4055uyLrtb4/umUHzWp1r8k87pRXWbuHO2UF98qrrN3DnbJK\nZ+70d+dOWcG98ipr91DW7uNOebs7q1sWaCIiIiIiIn2RCjQREREREZFeQgWaiIiIiIhIL9HjQ0JE\nRETcTUNDA0uWLKGmpga73c5TTz3Fb3/7W9f50tJS7r77bp544gnXsVdffZW1a9cSGRkJwF133cXi\nxYt7PLuIiLgXFWgiIiLXsXr1apKTk3n66acpKSnhscceY/369a7z3/72t1m0aNFlz3v00Ud5+OGH\nezKqiIi4OS1xFBERuY7g4GCqq6sBqK2tJTg42HVux44dJCUlER0dbVQ8ERHpQ9RBExERuY4FCxaw\natUq5syZQ21tLb/5zW9c5958802WLl16xeetX7+eTZs24enpyY9+9CPi4+N7KrKIiLgpFWgiIv1I\nVtYmZs6cfd3rXnjhBRYuvJeYmNgeSNX7rVmzhpiYGN544w2ys7NZunQpq1atoqSkhMbGRhISEi57\nzowZM5g4cSLjxo1j3bp1LF++vFNhdyXBwb5dsr+OO20Q7k5Zwb3yKmv3UNbu4055uzOrCjQRkX6i\nuLiIjRs33FCB9sMf/pCysroeSOUe9u/fz9SpUwFIT0+ntLQUh8PBli1bmDhx4hWfM3z4cNefMzMz\n+fnPf37d96mqarzlrOHhAW7zd+dOWcG98ipr91DW7uNOebsi67UKPLcr0JpbHKzdepqRKcF4e7pd\nfBERw/zHf/yMY8eOMG3aOG6//Q6Ki4v4xS9+zYsvPk9ZWSlNTU08/vg/MmXKNB555BG++91/YfPm\nTTQ01JOXl0thYQHf//7TTJo0xegvpcclJiZy8OBB5s6dS2FhIX5+flgsFg4fPsysWbOu+Jzly5cz\nb948xo4dy+7duxk4cGAPpxYRka5yvqWV3HN15J6rY9ywGIJ9uq8OcbsKJye/it++e5jZY+L4xpxB\nRscREXEbDz30CKtW/Z3k5FTy8s7y61//nqqqSsaPn8gddyyksLCAf/u3Z5kyZVqn55WWlvDzn7/C\nZ5/tYM2ad/plgfbAAw+wdOlSHn74YVpbW3nuuecAKCsrIzQ01HVdWVkZr776Ks8//zyLFy9m2bJl\nWK1WTCYTy5cvNyi9iIh8FY62NgrLGjhTXMuZ4lpOF9VSWN6A09l+Pq+8gW/PH9xt7+92BdqQpBCi\nQ/3IOlDIbWPjiAz2NTqSiMhX9vdPTrInu7RLX3NcegT3Zw64oWsHD84AICDAxrFjR3jvvVWYTGZq\na2suu3b48JEAREREUF9f33WB3Yifnx8vv/zyZcdff/31To/Dw8N5/vnnAUhLS+Ott97qkXwiInJz\nnE4nFTXnOd1RjJ0pquVsSR0t9jbXNZ4eZgbGBpIcYyM52sas8Yk01J3vtkxuV6BZLWYeXTCYn725\nl3e2nOY7XxtqdCQREbfj4eEBwMcfr6e2tpZf/er31NbW8u1vP3LZtRbLxaEVzgu/PhQREXFDDeft\nrq7YmaL2oqy20e46bzJBbJg/KTEBJEe3F2Sx4X5YzBd3J/P19lCB9mVThseQHG1jb3Ypp4pqSI0J\nNDqSiMhXcn/mgBvudnUVs9mMw+HodKy6upro6BjMZjNbtnyC3W6/yrNFRETci73VQV5pPWeKats7\nZEW1lFQ1dbom1ObF2PQIUqJtJEcHkBgVYPicC7cs0EwmE/fPSuVn/3OAtzefYsnXR2EymYyOJSLS\nqyUmJpOTk010dAxBQUEAzJyZybPP/gtHj37BggV3ERERwR/+8DuDk4qIiHw1bU4nJZWN7Z2xjg5Z\nfmk9jraLKz98vKxkJAWTHGMjJTqQ5OgAAv29DEx9ZW5ZoAGkJQQzckAYn58s5+DJCkYODDM6kohI\nrxYcHMyqVes6HYuOjuFPf7p4n9Ttt98BXBwhnJJyscuXkjKAX/7ytz0TVkRE5Bpq6ps5faEzVlzL\nmeI6mppbXeetFhMJkQHtnbGYAFJiAokI9sHsBk0dty3QAO6dmcrBU+W8nXWSYakhndaGioiIiIiI\n+7sw4v7SgqyytrnTNZEhvowcEEZKxyCP+Ah/PKzuWRu4dYEWG+bHtOExfHqwiG2HipkxMtboSCIi\nIiIicpMujLg/fWGQR3EtRZeMuAew+XowckBYx1JFG0nRAfh5exgXuou5dYEGsGhqMp8dPce7W88w\ncUgUXp6W6z9JREREREQM5XQ6Ka85z5niWs7tzOWLU+XknaujpfVLI+7jgkiJtrm6YyE2rz49f8Lt\nC7TgAC/mjktg7Y6zbNiTx11Tko2OJCIiIiIiX1LfZHftNXZhqWLdl0bcx4X7k9xRjKVE24gO8+13\ntzG5fYEGMG9CAlmfF/LhrjxmjozF5udpdCQRERERkX7L3uogr6T+4lTF4lpKLxtx78249GCSo22M\nHhJFoJdFq+HoIwWaj5eVRVOT+e+PjvPe9jM8fHua0ZFERERERPqFNqeTcxWNrvH2p4trKfjSiHtf\nLysZySGu7lhytI3AS5oqF6YHSx8p0ACmj4jh4z35bPm8iNvGxhMV4mt0JBERt3TffXfywQfrrn+h\niIj0S1V1zR2j7dsLsrPnamlqdrjOWy0mEqMC2ouxjoIsItinT9831pX6TIFmtZi5b2Yqv1r9Be9s\nOcVTdw8zOpKIiIiIiFtram7l7Lm6TveOVdV1HnEfHerLqIE2V3csPsIfq6V/3TfWlfpMgQYwelA4\nqbE29uWUcbKwhgGxgUZHEhHpNR5//Bv89KcriIqK4ty5Yv7P/3ma8PAImpqaOH/+PP/8z88wZMhQ\no2OKiIhBWh0XR9yfuXTE/SXX2Pw8L+43FmMjOSoA3z404r436FMFmslk4v5ZA3jxv/fz980n+T/f\nGK1WqohIh+nTZ7F9+6fce+/9bN26henTZ5GaOpDp02eyb98e/vKXP/HCC//X6JgiItIDnE4nZTXn\nXYXY6aJackvqsF8y4t7Lw8Kg+CDXfmMpMTaCA/r2iPveoE8VaAAD44IYNTCMAyfKOXCinNGDwo2O\nJCJymVUn3+dA6eEufc1REcO4Z8DCq56fPn0Wv/zlL7j33vvZtm0L3/3uP/PWW3/mr3/9M3a7HW9v\n7y7NIyIivUddYwtnius63TtW33RxxL3ZZCIu3K+9K9ZRjMWE+mE2qxjraX2uQAO4b2YqB09W8HbW\nKYanhmoNrIgIkJKSSkVFGSUl56irq2Pr1izCwiL4t3/7d7Kzj/LLX/7C6IgiItIF7K1tZJ+tZN/R\nc657x0qrO4+4Dwv0ZkhS+4j75GgbiZEBGnHfS/TJAi061I/pI2PIOlDI1kPFzBoVa3QkEZFO7hmw\n8Jrdru4yadJUfvvbXzNt2gyqq6tITR0IwJYtm2ltbe3xPCIicuuamls5WVjD8fxqTuRXc7q4jlbH\nxaWKft5WhiaHuMbbJ0fbtG9wL9YnCzSARVOS2PnFOdZsO8OkjEi8PfvslyoicsNmzJjFE088zh//\n+FfOn29i+fJlbN68kXvvvZ+NGz9i3br3jI4oIiLXUdPQwon8ao7nV3O8oJr80nqcHZM8TEB8hD/D\nB4YTHeLTPuI+SCPu3UmfrVoC/b2YNyGBNdvOsGF3PoumJhsdSUTEcIMHZ7Blyy7X47/8ZaXrz1On\nzgBgwYK78PPzo7FRG4aKiBjN6XRSVt3E8fwajhe0d8hKqi4uV7RaTAyIDWRQfBAD44IYEBuIr7dV\nGz+7sT5boAHMHR/P5gOFrN+Vx8yRMQT6exkdSURERETkqtranBSU1Xd0x2o4UVBNTX2L67yPl4Vh\nKaEMig9kYFwQydEBeFh171hf0qcLNG9PK1+bmsybG3JYs/0sj85NMzqSiIiIiIiLvbWNM8W1nCio\n5nh+DScLa2hqvnhPcKCfJ2PTIxgU194liwv312TFPq5PF2gA00ZE89GefD79vIg5Y+OIDvUzOpKI\niIiI9FPXG+gREezDmEHhDIxvL8h0/1j/0+cLNIvZzH0zU/nlqsOszDrF9+4dbnQkEREREeknauqb\nOV7QUZB9eaCHqX2gx6C4oI57yAJ1S470/QINYNTAMAbEBXLgRDnH86sZFB9kdCQRERER6WOcTiel\n1U0d3bH2oR6lnQZ6mBkYG8jA+PaCLDWmfaCHyKX6xX8RJpOJ+2cN4Kd/3sfbm0+y9JExahWLiIiI\nyC25bKBHfjU1Ddca6GHDw2o2MLG4g35RoAEMiA1kTFo4+3LK2JdTxtj0CKMjiYiIiIgbabE7XEsV\n2wd6VNPU7HCdD/T3ZFx6hGu5ogZ6yM3oNwUawL0zUvn8RDnvbDnFyIFhWC36DYaIiFxfQ0MDS5Ys\noaamBrvdzlNPPcVvf/tbGhsb8fX1BWDJkiUMHTrU9Ry73c6zzz5LUVERFouFF198kfj4eKO+BBG5\nCY3n2wd6tBdk1Zw9V4e99eJAj8hgH8akBXXcQxZIuAZ6SBfoVwVaVIgvM0bG8Mn+Qj49WETm6Dij\nI4mIiBtYvXo1ycnJPP3005SUlPDYY48RHh7Oiy++yKBBg674nPfffx+bzcaKFSvYtm0bK1as4Be/\n+EUPJxeRr6LTQI/8avLLOg/0SI4JJCUqQAM9pFv1qwIN4K4pyWz/4hxrtp1hUkYUPl797iMQEZGv\nKDg4mJycHABqa2sJDg6+7nN27tzJ1772NQAmT57M0qVLuzWjiHw1NzTQo6MzNjAuiAGxgSTEBVNW\nVmdgaukP+l11YvPz5I4JCby79Qzrd+Vx9/QUoyOJiEgvt2DBAlatWsWcOXOora3lN7/5DStWrOCV\nV16hqqqK1NRUli5dire3t+s55eXlhISEAGA2mzGZTLS0tODp6WnUlyHSr7W1Ockvred4QXt37ERB\nzZcGelgZnhrKwI4NoZOiNNBDjNHvCjSAueMS2Ly/kA178pg5KpbgALWnRUTk6tasWUNMTAxvvPEG\n2dnZLF26lCeffJK0tDQSEhJYtmwZf/nLX/jWt7511ddwXlgndQ3Bwb5YrZZbzhseHnDLr9FT3Ckr\nuFfe/p61xe7gRH41R05XcORMBdlnK2k83+o6H2LzYuqIGDJSQslICSUhyoblBgZ69PfPtTu5U97u\nzNovCzQvTwtfm5bMn9bnsGbbGf7hjnSjI4mISC+2f/9+pk6dCkB6ejqlpaVkZmZisbQXU5mZmXzw\nwQednhMREUFZWRnp6enY7XacTud1u2dVVY23nDU8PMBtlmC5U1Zwr7z9MeuXB3qcKa6l1XHxFyOR\nIb6MGRTuun/sywM9KivqeyxrT3CnrOBeebsi67UKvH5ZoAFMHR7NR3vy2XqoiDnj4okN8zM6koiI\n9FKJiYkcPHiQuXPnUlhYiK+vL9/61rd45ZVXsNls7Nq1i4EDB3Z6zpQpU1i/fj3Tpk1j8+bNTJgw\nwaD0In1TdX1zx8j79v3H8kvruVCOmUyQEBHAwPhABsUFMTA+iEA/LS8W99BvCzSL2czimQN45Z1D\nvJN1iu/fN9zoSCIi0ks98MADLF26lIcffpjW1lZ+8pOfUFVVxT/8wz/g4+NDZGQk3/ve9wB48skn\nee2115g/fz47duzgoYcewtPTk5deesngr0LEfTmdTkqrmjo2hG4f6lFa/aWBHvHtAz0GxQWRGhuo\nQXDitvr1f7kjBoQyKD6Iz0+Wk5NXRVrC9adyiYhI/+Pn58fLL7982fH58+dfduy1114DcO19JiJf\nnQZ6SG/hdDqpaaklt7aAvLoCCuqKmJw8mhG2Ed32nv26QDOZTNw/awDL39zL3zef4kePjtHmgiIi\nIiI9zN7q4ExxnatDdqqwhqZmh+t8kL8n4wdHdIy9DyI23A+zfmaTblDTXEd+XQG5dQXkdRRltS2d\n7zdLDotlhK37MvTrAg0gJcbGuPQI9mSXsjenjHHpEUZHEhEREenzauqb+fRQMdl51ZzIr7psoMfY\ntPbu2MD4IMIDvfVLdOlydS315NUVugqxvLoCqptrOl0T5BXIiLAMEmxxJAS0/y85NqpbB5rcUIH2\n05/+lIMHD2IymVi6dCnDh7ffr1VSUsIPfvAD13X5+fk8/fTT3Hnnnd2TtpvcOyOF/cfLeCfrFKMG\nhmG1qEUuIiIi0tWcTienCmvZtL+AvdmlONqcmE0Qr4Ee0s0a7I3tRVhtQXtRVldA5fmqTtfYPAMY\nGjqYBFsciQFxxAfEEejV86P/r1ug7d69m9zcXP72t79x6tQpli5dyt/+9jcAIiMj+fOf/wxAa2sr\njzzyCJmZmd2buBtEBPsyc1Qsm/YVkHWgkNvGxhsdSURERKTPaLE72HW0hE37C8graR9nHxPmx+zR\nsSyYPoDG+vMGJ5S+pKm1ify6Qtd9Y3m1BZSfr+x0jb+HH0NC00js6Iol2OII8go0KHFn1y3Qdu7c\nyW233QZAamoqNTU11NfX4+/v3+m61atXM3fuXPz83HNc/Z1Tkth+uJj3tp9l8tBofL37/epPERER\nkVtSWt1E1v5Cth4qouF8K2aTiTFp4cweHUdaQhAmkwk/Hw8VaHLTzreeJ7+uyLVEMa+2gNKm8k7X\n+Fl9GRwyyFWIJQTEEuwV1GuXzV63CikvLycjI8P1OCQkhLKysssKtLfffpv/+q//6vqEPcTm68n8\niYms+vQ0H+7K5d4ZqUZHEhEREXE7bU4nR89UsmlfAYdOVeAEAnw9WDg5kZkjYwmxeRsdUdxUs6OF\ngi8VYyWNZTi5eP+ij9WbtOABlxRjcYR6B/faYuxKvnKbyOl0XnbswIEDpKSkXFa0XUlwsC9Wq+Wr\nvu1lrrX79s166I7BZH1exMd7C1g8J43QQJ8ue+3uyNtdlLV7uFNWcK+8yto93CmriBiv8bydbYfP\n8cn+Akqr2vcoS42xkTkmjrFpERqDL19Ji8NOYX3xJfeNFVDcUNKpGPO2eDEgKLlTMRbmE4LZ5N7/\nrV23QIuIiKC8/GKbsLS0lPDw8E7XZGVlMWnSpBt6w6qqxq8Y8XLh4QHdNjll0ZQk/vBhNm+8e5hv\nzh/cJa/ZnXm7mrJ2D3fKCu6VV1m7R1dlVZEn0vcVlNbzyf4Cdhw5R4u9DavFzJRhUWSOjiM5uhtn\nkUufYW9r5VRlLgcLc8irbR9xX9xQQpuzzXWNp9mDlMBEVyGWGBBHuG+Y2xdjV3LdAm3KlCm8+uqr\nPPjggxw5coSIiIjLOmWHDx++4mad7mjKsGg+2pPPtsPFzBkXT1z49buCIiIiIv1Jq6ONAyfK2bSv\ngOP51QCE2rzJnBLL1OHRBPhqCqNcmaPNQVHDuU6dscL6czicF/e98zBbSQyId90vlhAQR5RfRJ8s\nxq7kugXa6NGjycjI4MEHH8RkMrFs2TJWrVpFQEAAc+bMAaCsrIzQ0NBuD9sTzGYTi2el8ou3D7Ey\n6xT/e3H37RIuIiIi4k5q6pvZcrCIrAOFVNe3AJCRFEzmmDhGpIZhNrvPfT7S/RxtDs41lroKsdy6\nAgrri2lta3VdYzVZiPOPIS0imXCPSBJtcUT5RmAx3/otUe7qhu5Bu3SvM4D09PROj9euXdt1iXqB\nYSmhpCcEcehUBcdyqxicGGx0JBERERFDOJ1OThXV8sm+AvZ07F3m7WnhtjFxzBodS3Soe07wlq7V\n5myjpLHMtUQxv66A/Loi7G121zUWk4UY/yjXEsUEWxzRfpFYzVa3ug2gu2mW/BWYTCYWzxrAv/9p\nL29vPsmPHhuL2Y0mv4iIiIjcqmvtXTYxIwofL/0Y2V+1Odsoa6q42BmrLSC/vpAWR4vrGrPJTLRf\npKsQSwiII8Y/Gg+z/ru5Hn1CV5EcbWP84Ah2Hytlz7FSJgyJNDqSiIiISLcrq25i84FCth68+t5l\n0n84nU7KmyrJq8snt+O+sfy6Is47Lu5dZ8JEtF8kCQFxxNtiSQyII9Y/Bk+Lh4HJ3ZcKtGu4Z0Yq\n+3LKeGfLKUYPCtd4WBEREemTenrvsvqWBnLr8jlbmw8FDhzN4GnxwNPsgYfFs+P/2x97Wjw7znni\nYfZo/3PH4/58n1J3cDqdVJ6vuqQQKyS3roCm1ibXNSZMRPiGMyxgCIkdnbG4gBi8LBoM01VUoF1D\nRJAPmaPj+HhvPpsPFHL7uHijI4mIiIh0mcbzdrZ37F1W0k17l9kddvLri8itzedsbR5na/Mpb6q4\n5deF9mV0nmZPPCxWPM0XCzlPy8UCz+PC8UvPmS8+vrQQvPR4Xy8EnU4n1c01rmmKuR2bPzfYO2+J\nFeETRkZoWvteYwFxxAfE4G3VZuPdSQXaddw5JYlth4tZu/0MU4dF4eutVq2IiIi4t+7au6zN2UZp\nY3mnYqywvrjTCHVfqw+DQwaRZEsgyRZPYmQUpRU1tLS10OKwY2+z0+Kw09LWgt1hp8XRQkvbJccv\nPL70mo5ztS11rtfoSu2FoAdeHl5YsV5W5LU/9rykC3h5kXexELz02s7FpcVk6bYlpDXNta77xfI6\nirG6lvpO14R5h5AWPOCSYiwWXw+fbskjV6cC7Tr8fTxYMCmRlVmn+OCzPO6bmWp0JBEREZGvrNXR\nxucde5fldNHeZbUtde3FWE17MZZbl09T68V7k6wmC3EBMa5iLMkWT7hPWKciJDwkgABH107vczqd\n2Ntar1vktXypyLtQ9F24xt7WeklB2ILD5KCppZm6lnrX8a50oRD0+NJyz2t3/750vOM5AS1eHC08\n7VquWNNS2+m9gr2CGBk+tL0Y61iq6Ofh26Vfj9wcFWg34LYxcWzaV8DHe/PJHN3167BFREREusuF\nvcu2fF5EVV0zcHN7l7U4Wsj9qHt5AAAgAElEQVSrK3R1xnJr86k8X9XpmgifMIaGDiEpsL0Yi/WP\nMWRqn8lkcnWv6MLFT18eBX9ZIdhR9NkvKfIuFn7X7v5dWgi2dBSG9V1UCAZ5BTI8LOOSYiyWAE//\nW/04pJuoQLsBnh4W7pmewhvrjrF662m+tWCI0ZFERERErsrpdHKysOayvctmj4kj8wb2LmtztnGu\noZSzHUsVc2vzKWo4R5uzzXWNv4cfQ0PTSbIlkGiLJ9EW3+86MN1VCH7ZhULQ3ta5+2fvtCy0oyjs\n+LOvnyc2gkgIiCPQ6+aXrUrPU4F2gyZlRLFhdz47Dp/j9nEJxEfotw4iIiLSu7TYHew6VsKnh4o5\nVVAD3NjeZdXNNe3FWE17MZZbl0/zJXtaeZitHUsU24uxJFsCod7BGrnfQy4tBG+0CNbGz+5LBdoN\nMptNLJ6Vyn/+/SBvZ53kX+4faXQkEREREeAKe5eZTYwZFE7mmDjSv7R32fnW85ctVaxurnGdN2Ei\n0je8/b6xwPbOWKxfdJ+cZCjSG6lA+wqGJocwODGYL05XcuRsJRlJIUZHEhERkX6qzenk6NlKPtlX\nyMGT5Z32LrsnMw1aW3G0OSisL+5UjBU3lODE6Xodm2cAw8My2icq2uJJtMXhY9XkPhGjqED7Ckwm\nE/fPGsBP/riHtzefZPA/jMOs1r6IiIj0oMbzrWw/XNxp77KUGBuZo2JJTfaksLGQD3M/4Ni5U+TX\nFXQaMOFp9iA1KMm1VDHZlkCQV6CWKor0IirQvqLEqAAmZkTy2ZESdh0tYVJGlNGRREREpB8oKKvn\nk/2F7PziHM12B1ZPB8NGmIiKa6HGmcd7NR9Qt/vivlYmTET7RV4ccR+YQJRvhJYqivRyKtBuwj3T\nUtibXcqqLacZmxaBh9VsdCQRERHpgy7sXbZxXy4nKgow+9fgk1JHQFA9Dc4qTgIny9qvDfIKZGT4\nMJJs8YxMSCPAEYK31cvQ/CLy1alAuwlhQT7MHhPHht35fLK/gLnjE4yOJCIiIn2E0+nkbGUJHx05\nxJHS07R4VGKOrMU7un3EvQNwmL0YZBvg2vw50RZPkFeg6zU0wU/EfalAu0kLJiWx9WAx7+84y9Th\n0fh5d+PmFyIiYqiGhgaWLFlCTU0Ndrudp556ivDwcJ5//nnMZjM2m40VK1bg43NxsMKqVat4+eWX\nSUho/yXe5MmTefLJJ436EqQXq7c3kFtbwNmaXI6VnSGvvgCHqX1DaULA4jQR5RvJgJAkkgLalypG\n+oZjNmkFj0hfpALtJvn7eLBwchJ/33ySdTtzuX/WAKMjiYhIN1m9ejXJyck8/fTTlJSU8NhjjxEW\nFsazzz7L8OHD+dnPfsaqVav4xje+0el58+fPZ8mSJQallt7I3tZKQV2Ra/Pns7V5lDVVdLqmrcUH\nL3scg8OSmDYwgwEh8XhaPA1KLCI9TQXaLZg9JpZN+/LZuLeAzNGxhAVqJK2ISF8UHBxMTk4OALW1\ntQQHB/P666/j7+8PQEhICNXV1UZGlF6ozdlGWVNF++bPdfmcrcmnoL4Ih9PhusaKJ9SFY6+1QUMQ\nGZHJ3D564GV7l4lI/6EC7RZ4WC3cPT2F379/jNWfnuF/3TnE6EgiItINFixYwKpVq5gzZw61tbX8\n5je/cRVnjY2NrFmzhpdffvmy5+3evZtvfetbtLa2smTJEoYM0feJvqyupb7TfmNna/Npam1ynbeY\nLMT6R2MjgvJib3JPW2g670uAryezR8Ywc2QsITZvA78CEekNVKDdookZUWzYnc9nR85x+7h4EqMC\njI4kIiJdbM2aNcTExPDGG2+QnZ3N0qVLWbVqFY2NjTz55JM8/vjjpKamdnrOiBEjCAkJYebMmRw4\ncIAlS5awdu3aa75PcLAvVuutj0APD3ef70XulBUu5m1pbeF0VT4nK89wsuIsJyrPUtbQealilH84\nY0KGMiA0iTj/eE6dbGPDjnxyyhoASEsIZsHUZKaOiMGjC/7er5bVHShr93CnrOBeebszqwq0W2Tu\n2Lx6xd8+Z2XWSZ5+cJTRkUREpIvt37+fqVOnApCenk5paSktLS185zvfYeHChdxzzz2XPSc1NdVV\ntI0aNYrKykocDgcWy9V/EK+qarzlrO40vc9dsjqdTkoaSylvK+Vw0Qlya/IobDhHm7PNdY2fhy8Z\noekk2uI7NoGOw9/Dr33vst2F/NcXOe17l1nMTBkWReboOJKjbQBUd8Hf+5e5y2cLytpd3CkruFfe\nrsh6rQJPBVoXyEgOISM5hCNnKvniTAVDk0ONjiQiIl0oMTGRgwcPMnfuXAoLC/Hz8+ONN95g/Pjx\nLF68+IrP+d3vfkd0dDQLFy7k+PHjhISEXLM4k96lqbWJY5UnOFaRw9HK41Q317jOWc1WEgPiLxlx\nn0CYT4jrnrELe5d9sj+H7Lz2exNDbV7cOSWJacOjCfDVwA8RuToVaF1k8cxUjp6p5O3NpxiSFIJZ\nN/aKiPQZDzzwAEuXLuXhhx+mtbWV5557jmeeeYa4uDh27twJwIQJE/jud7/Lk08+yWuvvcadd97J\nM888w1tvvUVraysvvPCCwV+FXIvT6aSgvpijFdkcqcjhTG2uq0Pm5+HL2MiRjIhNJ8wcQYx/FFbz\n5T9C1TS08OnnhWR9XkRVXfuY/IykYDLHxDEiNQyzWT8biMj1qUDrIgmRAUwaGsWOL87x2ZFzTB4a\nbXQkERHpIn5+fpcNAdm2bdsVr33ttdcAiIqK4s9//nO3Z5Ob12hvIrvqBEcqsjlWkUNNS/uSJRMm\nEm3xDAkZxJDQdBJtcZhN5isua3I6nZwuqmXT/gL2HCvF0ebE29PC7DFxZI6OJTrUz4gvTUTcmAq0\nLnT3tBR2Hytl1aenGZce0S03/IqIiMjNaXO2UVBfxNGKHI5W5HCmNs/VJfP38GNc5GgyQtMYHDII\nf89rF1Ytdge7jpXwyf5Ccs+1F20xYX7MHh3LxIwofLz0I5aI3Bz969GFQgO9uW1sHOt35bFxXwF3\nTEg0OpKIiEi/1mhv5FjlcY5U5HC0Moe6lnqgvUuWZItnSGgaGaHpxAfEYjaZr/t65dVNbD5QyKcH\ni2g434rJBGMGhZM5Jk57l4lIl1CB1sUWTEpk68Ei3t+Ry7ThMfj7eBgdSUREpN9oc7ZRUFfUUZBl\nc6YmDydOAAI8/JkQNYYhoWmkhwzE3+PGlx8eOVvJtrVH2X3kHE4gwNeDBZMSmTkyltBA7V0mIl1H\nBVoX8/P2YOHkJP72yUne33GWB2cPNDqSiIhIn1ZvbyC74jhHKnM4VnGcOvvFLllyYAJDQtLJCE0j\nLiDmhrpkX7b9cDFvrDsGQEqMjdmj4xibHoGH9au/lojI9ahA6waZo+PYtK+AT/YXMHtMnFttuici\nItLbtTnbyK8r5EhFNkcrcjhbm+/qktk8A5gYNZYhoYNIDxmEn4fvLb1Xq6ONd7eewWox8+J3phDq\np5UxItK9VKB1Aw+rmXump/DbtUdZ/elphgyMMDqSiIiIW6tvaXDdS3asMod6ewMAZpOZlMBEhoS2\nd8li/aNvqkt2NdsOF1NRe57bxsaRnhTiNhvpioj7UoHWTcYPiWTD7nw+O1rCyYJqAr000VFERORG\ntTnbyK0taN+XrDKHvNoCV5cs0DOAidFjyQhNJz14IL4ePt2SodXRxrodZ/Gwmpk/UYO/RKRnqEDr\nJmaTiftnpfJ/3/qcP6w9wv937zBNdhIREbmGupb6ji5ZNscqj9NgbwTau2SpQUlkhKQzODSNOP/o\nHvmeuvVQMRW1zcwZG0+Qv1e3v5+ICKhA61aDk0IYlhLKoZPlfHGmkmEpoUZHEhER6TXa2to4U5Pb\nPnGxIoe8uku7ZDYmR49jSGg66SED8LF2T5fsauytbazbeRZPq5n5ExN69L1FpH9TgdbNFs9M5Ysz\nFby9+SQZSSGYzeqiiYhI/1XbUsexivYuWU71SepbLt5LNiAo2bUvWYxflKErT7YdKqKytpnbx8UT\nqO6ZiPQgFWjdLC7Cn8yx8Wzak8+OL84xdXi00ZFERER6TJuzjbO1eR1dsmzy6gpd50J9ghkRk8GQ\n0HTSggfgY+0d+4nZW9t4f2cunlYzd+jeMxHpYSrQesA35g7m0wOFrN56mvGDI/D00MAQERHpu2qa\n6zhW2b5s8VjlcRpbmwCwmCwMCkp1dcmGJw2gvLze4LSX+/RgEVV1zcwbn0Cgn6fRcUSkn1GB1gPC\ng32YMzaeDz7L5eO9+SyYlGR0JBERkS7jaHNwpjaPox1dsvz6Ite5YK8gRkUMJyM0jbTgAXhf0iXr\njcOz7K2O9nvPPMzMm6B7z0Sk56lA6yHzJyby6cEiPvgsl+kjYgjw1W/kRETEfdU013K0IocjlTlk\nV56g6ZIuWVrwAFeXLMo3olcWYlez5fMiqutbmDchAZu6ZyJiABVoPcTX28qdk5P466YTvL8jl4du\nG2h0JBERkRvmaHNwuiaXox1LFwsu6ZKFeAczJnIEGSFpDAoegLfVPYdqtNgdrPssFy8Pi7pnImIY\nFWg9aNboWDbuy+eT/QXMHhtHRFDPjgwWERH5Kqqba9q7ZBU55FSdoKn1PABWk4X04IEdXbI0It2s\nS3Y1Ww4WUVPfwh0TE7BppYuIGEQFWg+yWszcOyOV19ccYdWWUzyxaKjRkURERFzau2Rn2ycuVuZQ\nWF/sOhfqHcK4yFEMCW3vknlZ+lYB02J38MHOXLw8Lcwbr+6ZiBhHBVoPG5seQfLuPHYfK2Xu+FqS\no21GRxIRkX6s6ny1616ynMoTnHc0A2A1WxkcMqi9SxaSRoRveJ/okl1N1udF1DS0sGBSou4TFxFD\nqUDrYWaTicUzB/D///UAf//kJP/69VF9+hueiIj0Lq1trRe7ZBU5FDWcc50L8wllQugYhoSkMSg4\nFc8+1iW7mma7gw8+y8Xb08Jcdc9ExGAq0AyQnhjM8NRQDp2q4NCpCkYMCDM6koiI9GFV56s5UpHN\n0YocsqtO0OxoAcDDbGVISJrrXrII33CDkxoj60AhtQ0tLJyciL+Ph9FxRKSfU4FmkPtmpnL4dAVv\nZ51iaEoIFrPZ6EgiItJH2NtaOVV9xrV08VxDietchE8YQ0Lbi7KBQal4Wvp3QdLc4uDDju7Z7ePU\nPRMR46lAM0hcuD9Th0Wz9VAx2w+fY/qIGKMjiYiImztedZI/ZH/GoZJsWlxdMg8yQtM77iVLJ9w3\n1OCUvcvmA4XUNtpZODlJ3TMR6RVUoBnoa9NS2HW0hHe3nmbCkEi8PCxGRxIRETf2UW4WxyqPE+Eb\nRkZIe1E2ICil33fJrqa5xcGHu3Lx8bIwd3y80XFERAAVaIYKDvDi9vHxvL8jl4/35LNwcpLRkURE\nxI09nvEN/AKtOBtVkN2ITw4UUNdo564pSfh56zMTkd5BNz4Z7I4J7Tckf/BZLrWNLUbHERERN+br\n4UOYX4jRMdzC+ZZWPvwsDx8vK7ePU/dMRHoPFWgG8/GysmhqMudbHKzdftboOCIiIv3CJ/sLqW+y\nc/u4eHzVPRORXuSGljj+9Kc/5eDBg5hMJpYuXcrw4cNd54qLi/mXf/kX7HY7Q4YM4fnnn++2sH3V\njJExfLwnn6wDhdw2No7IYF+jI4mIyCUaGhpYsmQJNTU12O12nnrqKcLDw3nuuecASEtL4yc/+Umn\n59jtdp599lmKioqwWCy8+OKLxMerU9MbNDW3sn5XHr5eVuaM1d+JiPQu1+2g7d69m9zcXP72t7/x\nwgsv8MILL3Q6/9JLL/H444+zcuVKLBYLRUVF3Ra2r7JazNw7MxVHm5N3tpw2Oo6IiHzJ6tWrSU5O\n5s9//jMvv/yy6/vh0qVLeeutt6ivr2fLli2dnvP+++9js9n461//yhNPPMGKFSsMSi9f9sn+gvbu\n2fh4fL11O76I9C7XLdB27tzJbbfdBkBqaio1NTXU19cD0NbWxr59+8jMzARg2bJlxMRoXPzNGJsW\nTnK0jb3ZpZwqqjE6joiIXCI4OJjq6moAamtrCQoKorCw0LWiZNasWezcubPTc3bu3MmcOXMAmDx5\nMvv37+/Z0HJFl3bPbhuj7pmI9D7XLdDKy8sJDg52PQ4JCaGsrAyAyspK/Pz8ePHFF3nooYf028Fb\nYDKZuH9WKgBvf3ISp9NpcCIREblgwYIFFBUVMWfOHB5++GH+9V//FZvN5jofGhrq+t54QXl5OSEh\n7QM7zGYzJpOJlhYNgzLaxn0FNJxvZa66ZyLSS33lf5kuLRycTiclJSU8+uijxMbG8o//+I9kZWUx\nc+bMqz4/ONgXq/XW9/sKDw+45dfoSTeSNzw8gKyDxew6co4zpQ1MGBrdA8munMNdKGv3cae8yto9\n3Clrd1uzZg0xMTG88cYbZGdn89RTTxEQcPHzuZFfqt3INf3xe2RPZm08b+fjPfkE+Hrw4LzBNzUc\nRJ9t91DW7uFOWcG98nZn1usWaBEREZSXl7sel5aWEh4eDrQv+YiJiSEhIQGASZMmceLEiWsWaFVV\njbcYuf0DKSuru+XX6SlfJe+dkxLZffQcb7z3BYnhvljMPTto050+W2XtPu6UV1m7R1dldadvttey\nf/9+pk6dCkB6ejrNzc20tra6zpeUlBAREdHpOREREZSVlZGeno7dbsfpdOLp6XnN9+lv3yN7Ouva\n7Weob7Jzz/QUGurO01B3/is9X59t91DW7uFOWcG98nZF1mt9f7zuT/9Tpkxhw4YNABw5coSIiAj8\n/f0BsFqtxMfHc/bsWdf55OTkWwrb38WE+TF9RAzFFY1sO1RsdBwREQESExM5ePAgAIWFhfj5+ZGa\nmsrevXsB+Oijj5g2bVqn50yZMoX169cDsHnzZiZMmNCzoaWTxvOtbNidj7+PB7PHxBkdR0Tkqq7b\nQRs9ejQZGRk8+OCDmEwmli1bxqpVqwgICGDOnDksXbqUZ599FqfTyaBBg1wDQ+TmLZqazM4j53h3\n6xkmDonCy/PWl7uIiMjNe+CBB1i6dCkPP/wwra2tPPfcc4SHh/PjH/+YtrY2RowYweTJkwF48skn\nee2115g/fz47duzgoYcewtPTk5deesngr6J/27g3n8bmVu6dkYKPl+49E5He64b+hfrBD37Q6XF6\nerrrz4mJifz1r3/t2lT9XJC/F/PGJ/De9rNs2JPHXVPUlRQRMZKfnx8vv/zyZcf/53/+57Jjr732\nGoBr7zMxXuN5Oxv2qHsmIu6hZ29wkhs2d3wCNl8PPtyVR02Dpn6JiIjcrI/25NPU3Mq8CQl4e6p7\nJiK9mwq0XsrHy8pdU5NpbnHw3vYzRscRERFxSw3n7Xy8t717ljk61ug4IiLXpQKtF5s+IobIYB+2\nHCiiuKLB6DgiIiJu56Pd+TQ1O7hjorpnIuIeVKD1YlaLmftmptLmdLJqy2mj44iIiLiV+iY7G/fl\nY/P1IHOU7j0TEfegAq2XGz0onNRYG/uOl3GysMboOCIiIm6j/d4zB/MmJGoisoi4DRVovZzJZOL+\nWQMA+PvmkzidToMTiYiI9H71TXY27s3H5ufJLN17JiJuRAWaGxgYF8ToQeGcLKjhwIlyo+OIiIj0\neht253G+xcH8CQl4eah7JiLuQwWam7h3Rgpmk4m3s07R6mgzOo6IiEivVdfYwsZ9Bdj8PJkxSt0z\nEXEvKtDcRHSoH9NHxlBS2cjWQ8VGxxEREem1NuzOp7nFwfyJieqeiYjbUYHmRhZNScLLw8Karadp\nam41Oo6IiEivU9vYwqZ9BQT6ezJzZIzRcUREvjIVaG4k0N+LeRMSqG20s2F3ntFxREREep0Nu/No\ntrd3zzzVPRMRN6QCzc3MHR+Pzc+TDbvzqalvNjqOiIhIr1Hb2MIn+woJUvdMRNyYCjQ34+1p5WtT\nk2m2O1iz/azRcURERHqN9bvau2cLJiXhYVX3TETckwo0NzRtRDRRIb58+nkRxRUNRscRERExXG1D\nC5/sLyA4wIvpI6KNjiMictNUoLkhi9nM4pmptDmdrMw6ZXQcERERw324K5cWexsLJiWqeyYibk0F\nmpsaOTCMgXGBHDhRzvH8aqPjiIiIGKamvpnN+wsJDvBi2nDdeyYi7k0FmpsymUwsnjUAgLc3n8Tp\ndBqcSERExBgf7sqjpbWNhZMS8bDqRxsRcW/6V8yNDYgNZExaOKeKatmXU2Z0HBERkR5XU9/M5gOF\nhNi8mKrumYj0ASrQ3Ny9M1KxmE2s3HKKVkeb0XFERER61Aef5WFvbWPhpCR1z0SkT9C/ZG4uKsSX\nGSNjKK1qYsvnRUbHERER6THV9c1kfV5IqM2bqcM1uVFE+gYVaH3AXVOS8fK08N72MzQ1txodR0RE\npEd8sDO3vXs2ORGrRT/SiEjfoH/N+gCbnyfzJyRQ12hn/a48o+OIiIh0u6q6ZrI+LyIs0Jspw9Q9\nE5G+QwVaH3H7uAQC/T3ZsCePqrpmo+OIiIh0qw925tLqaGPh5CR1z0SkT9G/aH2El6eFu6el0GJv\nY822M0bHERER6TaVtefZcrCQsEBvJg+NMjqOiEiXUoHWh0wZFkVMmB9bDxVRWN5gdBwREZFuse6z\nXFodTu5U90xE+iD9q9aHWMxm7puRitMJ72SdMjqOiIhIl6usPc/Wg0WEB3kzSd0zEemDrEYH+KpK\nGsv45Se/JcU/mSkxEwn0CjA6Uq8yYkAog+KD+PxkOTl5VaQlBBsdSUTE7b399tu89957rscHDx5k\nxIgRrselpaXcfffdPPHEE65jr776KmvXriUyMhKAu+66i8WLF/dc6D5q3c4L3bNkdc9EpE9yuwLN\n0eYgr7qQY2UnWX/2E0ZFDGN67GRSAhMxmUxGxzOcyWTi/lkDWP7mXv6++SQ/enSsPhcRkVu0ePFi\nV3G1e/duPvzwQ5YtW+Y6/+1vf5tFixZd9rxHH32Uhx9+uMdy9nUVNef59GAREcE+TBoaaXQcEZFu\n4Xa/eorxj+K1O3/Kg2n3EOkbzt6Sz/mP/b/mpT0vs6NoNy2OFqMjGi4lxsa49AjOFNexJ7vU6Dgi\nIn3Kr371K77zne+4Hu/YsYOkpCSiozXqvbut23kWR1v7vWcWs9v9CCMickPc8l83bw9vpsVOZOn4\nf+Z/j/onRoUPo6jhHH/JXskPt7/AqpPvU95UYXRMQ907IwWL2cSqLadpdbQZHUdEpE84dOgQ0dHR\nhIeHu469+eabPProo1e8fv369Xzzm9/kn/7pn8jPz++pmH1SeU0TWw8VExnsw8QMdc9EpO9yuyWO\nlzKZTAwMTmVgcCpV56vZVrSL7YW72JT3KZ/kbSUjNI3pcVMYHDIQs8kta9GbFhHsy6xRsWzcV0DW\ngUJuGxtvdCQREbe3cuVK7r77btfjkpISGhsbSUhIuOzaGTNmMHHiRMaNG8e6detYvnw5v/nNb675\n+sHBvlitllvOGR7uPvdn32jWv2WdwtHm5OvzBhMVGdjNqa6uL362vYGydg93ygrulbc7s7p1gXap\nYO8g7kyZy7yk2XxeepgtBTv4oiKbLyqyCfcJZXrcZCZGjcXXw8foqD1m4ZQktn9RzHvbzzJ5aDS+\n3n3mr1tExBC7du3iRz/6kevxli1bmDhx4hWvHT58uOvPmZmZ/PznP7/u61dVNd5yxvDwAMrK6m75\ndXrCjWYtq25i4+48IkN8GRJvM+zr64ufbW+grN3DnbKCe+XtiqzXKvD6XFvJw2xlXNQofjD2KZaM\n/T4To8dS1VzDOyfW8sPty/lr9jsU1hcbHbNH2Hw9mT8xkfomOx/uyjU6joiIWyspKcHPzw9PT0/X\nscOHD5Oenn7F65cvX87evXuB9sEiAwcO7JGcfdGFe8/umqJ7z0Sk7+vTLZUEWxyP2O7n7gEL2Fm0\nh62FO9lWtIttRbsYEJTMjLgpjAjLwGK+9eUkvdVtY+P5ZH8hH+3JZ9aoWEJs3kZHEhFxS2VlZYSE\nhFx2LDQ0tNPjV199leeff57FixezbNkyrFYrJpOJ5cuX93TkPqGsuonth88RHerLhMG690xE+r4+\nXaBd4O/hx5zEmcxOmM6Rimy2FOzgWOVxTlafIdDTxrTYiUyOmdAn91Tz8rDwtanJ/OHDbN7ddobH\n5w82OpKIiFsaOnQov//97zsde/311zs9Dg8P5/nnnwcgLS2Nt956q8fy9VVrd3RMbpyShNmsbWNE\npO/rFwXaBWaTmWFhQxgWNoSShlI+LdzJZ8X7eP/MR3x4dhOjIoYxI24KybaEPrV32JRh0Xy0J5/t\nh4u5fWw8cRH+RkcSERG5rtKqRnZ0dM/Gp6t7JiL9Q79dyB3pF8HiQYt4YcpSHhh0NxG+Yewt+ZwV\n+37Fz/a+wo6iPbQ47EbH7BJms4nFs1JxOmHlllNGxxEREbkha3ecpc3pZNHUZHXPRKTf6FcdtCvx\ntnozPW4S02IncqL6NFsKdnCo/Ah/yX6bd0+uY1LMOKbFTiLMJ+T6L9aLDUsJJT0hiEOnKjiWW8Xg\nxGCjI4mIiFxVSVUjO78oISbMj7FpEUbHERHpMf22g/ZlJpOJQcGp/K9hj/D8pGeZl5iJyWRiY94W\nntv5M14/9AeOVRynzememz6bTCYWzxoAwNubT9LmdBqcSERE5OrWbm/vnt2le89EpJ/p9x20Kwn2\nDuLO1HnMS76NA6WH2FKwg8PlxzhcfowIn7D2PdWix+Bjda891ZKjbUwYEsmuoyXsOVbKhCFazy8i\nIr3PucpGdh45R2y4H2PT1T0Tkf5FBdo1eJitjI8azfio0eTW5vNpwU72ln7OyhPv8d7p9YyPGs2M\n2MnE+EcZHfWG3TM9hb3Zpbyz5RSjB4XjYVUTVUREepe128/idMKiKcmY+9DQLhGRG6EC7QYl2uJ5\nZEg8dw9YwI7i3XxasJNthZ+xrfAzBgalMCNuCsPDhvT6PdXCg3yYPSaOj/bks/lAIbePizc6koiI\niEtxRQOfHT1HXLg/o8XurG0AACAASURBVNPCjY4jItLjVKB9Rf6eftyeOIvbEmZwuPwYnxbsILvq\nBCeqTxPkFcjUmIlMiR2PzbP37qm2cHISWw8Vs3b7GaYOi8LX28PoSCIiIkD75EanExZNTVL3TET6\nJRVoN8lsMjMiPIMR4Rmc69hTbVfxXt4/s4EPz25kdMRwZsRNJsmWYHTUy/j7eLBgUiIrs06x7rNc\nFs8cYHQkERERiisa2HW0hPgIf0YNUvdMRPonFWhdIMovgvsHLeKulLnsPrefLQU72FNygD0lB0gI\niGXh4NkM9EnD09J7OlW3jYlj074CPt5TQOaoOEIDvY2OJCIi/dx7Hfee3aV7z0SkH1OB1oXa91Sb\nzLTYSRyvOsWWwh0cKjvCr3e/iZ+HL5OjxzMtdiKhvWBPNU8PC/dMT+H/tXfn4VHX9/73nzOTPZmZ\nZJKZ7HsgCYGwypKwWrGIaNUeXLof7en5uZxed2t7aak9nqVq2x/20lZrT/U+PT3eHrVajoIbVgvI\nEgg7hC1k35PJnhAgIZn7jwkDEVmUJDMhr8d1ccGsec8H5ZtX3p/l/333CG9tLuO+FZO8XZKIiIxj\ntc0nKDzcSJIjjBkTo7xdjoiI1yigjQCDwUCmLYNMWwatp9rY3baHj0q28NeqjXxUtYnJUdksSsgj\nK2ICBi/+hHBeTgzrC6vZVtTAjbOTSHSEea0WEREZ39ZtLccFfGV+qlevjSIi3qY91keYLSiCr+Xe\nxs/zVvGt7LtIsiRwsPkwz+17iX/fsZqN1Vs5eeaUV2ozGg3cuSQdF/DGxhKv1CAiIlLr7GbnkSaS\no81Mm6DumYiMb+qgjRJ/kz9zYmcyJ3YmFZ1VfFJTwO7Gfbxx/G3Wlr3P7JiZLIyfN+pnquWk2piU\nEkFRWSuHKlpZbPfd3SdFROTatHZrhbpnIiKD1EHzghRLEt+adBc/z/8pX0m7iRC/EDbXFvBE4a95\nds9/sK/pIP0D/aNSi8Fg8Ozi+MaGEgYGXKPydUVERABqnN3sOtpESoyZqRmR3i5HRMTr1EHzInNA\nGDemLOFLSQspajnCppptHGsrobi9lPBAKwvi55EfNxtzwMiuDUuOMTM3J5rthxrZtLeGyUnhI/r1\nREREzlq7RWvPRETOd0UB7cknn2T//v0YDAZWrVpFbm6u57Hrr7+emJgYTCYTAKtXryY6Onpkqr1G\nmYwmptonM9U+mfoTjXxSU8COhl2sK/uA98v/yozoqSN+ptodC9LYdbSJZ17by7xJ0Syfl0xsZOiI\nfT0REZHyug52HXOSGmsmN13dMxERuIKAVlhYSGVlJa+//jqlpaWsWrWK119/fchzXnzxRUJD9c38\ncIgNjeauzNu4NX2Z50y1woY9FDbsIcmcwKKEPGY6puI/zGeqRYUH89AduazZXMbWoga2FTVwXbaD\nFfNSSNDujiIiMgJe/fAYoO6ZiMj5LhvQCgoKuOGGGwBIT0+no6OD7u5uwsL0TftICvYLYlFCHgvj\n53GsrYRParZxoPkwLx/5M2tK3iE/bg7z4+YSGRwxbF8zNz2SJbOT+XBbGeu2VVB4pInCI01MnxDF\nLfkppMRYhu1riYjI+FbV2EXBwXrS4ixMSVP3TETkrMsGtObmZnJycjy3bTYbTqdzSEB7/PHHqa2t\nZebMmTz88MP6KdgwMhgMZNkmkGWbQMvJNrbUbWdr3Q4+rNzAXys3MiVqEosS8siMyBiWcTcaDczM\ndDBjop2DZS2s21rB3uPN7D3eTG56JCvyUsiItw7DJxMRkfFs7dYKQN0zEZFP+9ybhLhcQ3f5+/73\nv8+CBQuwWq08+OCDrF+/nmXLll309RERIfj5mT5/pZ9iH2PbwQ9HvXbMZCUl8a3+2ymo2s0Hxzdy\noPkQB5oPEW+O4csTFrEwZQ4h/sHDUuuXHBaun5PCgePNvPbRMQ6UtnCgtIWpE6K464ZMJqdHev2i\nOpb+OxhLtcLYqle1joyxVKuMLVWNXewpdpKZHMHkVJu3yxER8SmXDWgOh4Pm5mbP7aamJux2u+f2\nbbfd5vnzwoULKS4uvmRAa2vr+aK1etjtZpzOrqt+n9EyEvVOCsth0vQcKjqr2FSzjT2N+/nPPa/z\nyv7/ZU7MLBYlzCMm9PNv1vJZtcZFBPHDlVMprm5n3bYK9h9vZv/xZiYkWLklP4WcFJtXgtpY+u9g\nLNUKY6te1ToyhqtWhTz5LG9vKQfga1/O8voP+kREfM1lz0HLz89n/fr1ABw6dAiHw+GZ3tjV1cV9\n991Hb28vADt37mTChAkjWK58WooliW9Pupuf5/+UW9KWEewXzCe12/j3HU/zm71/YJ+zaNjOVJuY\nGM7Dd03jp9+aybSMKI7XdPDr1/fz8//ezb7jzRd0V0VERD6tsqGLvcebyYi3Mn2i/fIvEBEZZy7b\nQZsxYwY5OTncfffdGAwGHn/8cdasWYPZbGbp0qUsXLiQu+66i8DAQCZNmnTJ7pmMHHNAGMtSrmdp\n0iIONh/2nKl2rK2EiMBwFsTPJW+YzlRLj7Py/b/Lpaqxi3XbKth9zMlv/nKAREcYK/JSmJlpx6if\niIqIyGc42z37ygKtPRMR+SxXtAbtRz/60ZDbWVlZnj9/+9vf5tvf/vbwViVfmMloYppjCtMcU6jr\nbmBzbQHbG3aztuwD3iv/KzOjp7EoIY9kS+JVf62kaDMP3j6FWmc37xZUsuNIIy+8VURsZAgr8lKY\nne3AZLxsk1ZERMaJ8vpO9pU0k5FgZVLy8O1CLCJyLfncm4TI2BEXFsNdmbdza/oydtTvYVPtVnY0\n7GZHw26SLYksis9jhiP3qs9Ui7eH8b1bc/jK/FTeLaik4FADL647zNtbyrl5bjLzJsfgZ1JQExEZ\n7852z27Tzo0iIhelgDYOBPsFszgxn4UJZ89UK+Bg82H+u/N11pS8Q17cbBbEz8UWdHU/zYy2hXDv\nzdncmp/Cezuq2HKgjj++f5S1W8tZPjeZ+bmx+A/DDp4iIqPtjTfeYO3atZ7bRUVFTJ48mZ6eHkJC\nQgB45JFHmDx5suc5fX19PProo9TV1WEymXjqqadITLz62QtjVVldJwdKW5iYYCVb3TMRkYtSQBtH\njAYj2baJZNsm0nKylc2129lWV+g5Uy3XnsPSiflEGaKvaq1aVHgw3/pyJrfkpfD+jko27avj5Q+L\nWbetgmVzklk0LY5AfwU1ERk7Vq5cycqVKwEoLCzk/fffp6SkhKeeeoqJEyd+5mveeecdLBYLTz/9\nNFu2bOHpp5/mmWeeGc2yfcrarWfXnqWpeyYicgkKaONUZLCN2zKWszx1Kbub9rOpZiv7nUXsdxYB\nYA+OJM2aQro1hbTwFKJD7BgNn2+aYoQ5kK/dMJGb56XwYWEVf9tTy2sfH+fdggq+PDuJJdPjCQ7U\nf4IiMrY8//zzrF69mh/+8IeXfF5BQYHnKJq8vDxWrVo1GuX5pNK6Dg6UtpCZGK7umYjIZei743Eu\nwOTPvNhZzI2ZSWVXNVWnKjlYV0x5Z6VnvRpAiF8wqdZk0qwppFmTSbEkEmAKuKKvYQ0NYOWSDG6a\nm8xfd1bz0e4a3txYyvvbK1k6K5EbZiUQEnR16+BEREbDgQMHiI2N9ZwH+pvf/Ia2tjbS09NZtWoV\nQUFBnuc2Nzdjs7kPYTYajRgMBnp7ewkIuLJ/O68lnrVnC1K9XImIiO9TQBMADAYDKZYkrkvPYaFj\nAQOuARpONFHaUUFZRwVlHZUcajnKoZajgHu6ZGJYPGnWZNLC3aEtPNB6ya8RFuzP7QvT+PLsRD7e\nU8uHhVW8taWc9TuruH5GAjdel4g5ZPx94yIiY8ebb77J7bffDsC3vvUtMjMzSUpK4vHHH+eVV17h\nvvvuu+hrr+SsyIiIEPyGYa2uLx0QfrSilaKyVnIzopg/M+mCx32p1isxlupVrSNDtY6csVTvSNaq\ngCafyWgwEhcWQ1xYDAvi5wLQcbqL8sGwVtZRQVVXLZVd1Wyo2QJAZFAEqdZk97RIawpxYTGfOS0y\nJMifW/JSWDorgQ17a1m/o4p3Cyr5665qlkyPZ9nsJKxhgaP6eUVErsSOHTt47LHHAFi6dKnn/uuv\nv5733ntvyHMdDgdOp5OsrCz6+vpwuVyX7Z61tfVcdY12uxmns+uq32e4/Nc7hwC4aXbiBXX5Wq2X\nM5bqVa0jQ7WOnLFU73DUeqmAp4AmV8waaPacsQbQ299HVVfNYIfNHdx2Ne5jV+M+AIJMgaRYkjwd\ntlRLEkF+56b/BAX4cdOcZL40I4FN++v4YEcV6wur+dueWhZOjeOmOUnYLEGfWYuIyGhrbGwkNDSU\ngIAAXC4Xf//3f89vfvMbLBYLO3bsYMKECUOen5+fzwcffMCCBQvYsGEDc+bM8VLl3lNS08Gh8lay\nkyPITNLaMxGRK6GAJl9YgMmfjPBUMsLdawpcLhdNPU5KBztsZR2VHG07ztG24wAYMBAfFutZx5Zm\nTcEWFE6Av4mlsxJZPC2erQfreW97JR/vrmHj3lrm58Zy09xkHOHB3vyoIiI4nU7PmjKDwcCdd97J\nd77zHYKDg4mOjuaf/umfALj//vt54YUXWL58Odu2beOee+4hICCAX/ziF94s3yve2lIGwFfma+2Z\niMiVUkCTYWMwGIgOdRAd6iAv7joAuntPUN5ZSWm7O7BVdVVT013HJ7XbAAgPtJ43LTKZBVPjmJ8b\ny/ZDjbxbUMGmfXVs3l/PvJxols9LJjYy1IufUETGs8mTJ/PSSy95bi9fvpzly5df8LwXXngBwHP2\n2XhVXN3O4Yo2JqVEMDEx3NvliIiMGQpoMqLCAkKZEjWJKVGTADgzcIbqrlrPOrbSjgr2Nh1gb9MB\nAAKM/iRbEkm3pvC1O5JpbYzlo+2NbC1qYFtRA9dlO1iRl0KC/Yuf0yYiIiPPs3Pj/DQvVyIiMrYo\noMmo8jP6kWpNJtWazJdYiMvlovlk65B1bCXt5RxvL/O8JnZKNNNc0dRVBbCz7ASFRxqZMdHBN2+e\nhDVQB16LiPia4up2jlS2kZNqIyPh0jv8iojIUApo4lUGgwF7SCT2kEjmxM4EoKfvJOWdVe7A1l5B\nRWcV9QONEAVBUWDsD6Sow8qPXy0k1ZrMHbOmk5kY6eVPIiIiZ721WWvPRES+KAU08Tkh/sHkRGaS\nE5kJQP9AP7Xd9ZR2VFDeUUlpRwXtpiZMtiZqOMazxz4i6JCNyY4MZiVmkmpNxhygKZDXmgHXAF29\nJ+jo7aDjdCftpzvpON1JdFsEkUYHCWFxBJh04LmItx2rauNoVTuTU21kxKt7JiLyeSmgic8zGU0k\nWRJIsiSwJHE+AK2n2mh2NbLh8H6ONpdxyr+Z3e3N7G7fDoAjOIq08BTP5iPRIQ4MBoM3P4ZchMvl\n4lT/qSGhq/10Bx29Q2939nYx4Bq48A0q3L8ZDUYSwmJJtiSRYkkkxZKII8T+mWfxicjIeWuze+3Z\nVxaoeyYi8kUooMmYZAuKINOexMTgLACOVDfx1q69lHdVYjS34Rxop+nkLrbX7wIg1C+EVGuyZ3v/\nZEuiui2joG/gDJ2nOy8IW58OYb39vRd9D5PBhDXQQoolEWuAhfBAK9ZAi/tXgAVXYB8Haoup7Kym\npquWqq5aNtcWABBkCiLZkkCyJZGUweBmDbSM1scXGXeOVLZxrLqdKWmRpMepeyYi8kUooMk1ITvR\nQXbil6ls6OKdggp272rCENxNZFwPMQmnaXc1UNRyhKKWI4D7m/5Ec7wnsKVZU7AGXvxEdxlqwDVA\nd9+JTwWuwQA2OAWx43Qn3X0nLvk+Zv8wHMFRhJ8NXIFWwgPO+3OghVD/kEt2wex2M1mh2YA7ENZ1\n11PeWUVlZzWVndUcayvhWFuJ5/kRgeGDgc39K9GcQJBf4PAMjMg45nK5PDs3au2ZiMgXp4Am15Tk\nGDMP3j6FWmc37xZUsuNII82lEBuZxR1zorA6TlDZVUVpRwVVXTVUdFbxt+rNAEQG2UizppAe7g5t\nsaHR43J63Mkzp6jtPEFZa50naLX3dtJxfhDr7fzs6YaDgkyBWAMtxIXFDna93KHrbPcrPNCCJcCM\nn3F4/wnyN/qRbEkk2ZLoua+nr4fKrhoqOqqp7KqioqOafc6D7HMeBNwHqMeGRg8GtiSSLYnEhkZj\nMmqHUJHP42hlG8XV7eSmR5IWp061iMgXpYAm16R4exjfuzWHW+en8l5BJQWHGnjlvSocEcHcPHcm\nX5l+MwOcobKzZsgW/zsb97CzcQ/gnh6Xak3ydNlSLEljutNyZuAMHae7BqcVnutyuTtfHZ77T19i\nuqHRYMQaYCHZnOgJWp7gdV4QC/ILGsVPdmkh/iFk2yaSbZsIuH/K33qqncquaio6qqjorKaqq4a6\nEw1sq98JuM/jSzQnkGI9NzUyIjBc6xhFLsLlcvGWumciIsNCAU2uaTG2EO69OZtb81N4b3slWw7W\n88f3j7J2awXL5yUzf0oKEyLch6gOuAZo6nFS2lFBWXslZZ0VHGkt5khrMeAOJ/FhsZ7Alm5NISIo\n3JsfD3DXfaKv51zQusgmG5ebbhjmH4o9OAproIUYaxSBA0Hnrfe6sumGY4HBYCAyOILI4AhmOHIB\n906h9ScaqRicGlnRWT14kHq553XmgLAhXbZkcyIh/sHe+hgiPuVwZRvHazqYlhFFaqy6ZyIiV0MB\nTcaFqPBgvrUsixV5KXywo4pN++t4ef0x1m0t56Y5ySycFkegv4mY0GhiQqPJj5sDQFdvN2UdlZ7t\n/as6q6nuqmVTzTYAwgOtgztFuneLjA+LHdapcafOnBoyrfCC9V6Duxv2u/ov+h4BpgDCAy3EhcYM\nmWbo6YAFWLEEmvE/b7qh3W7G6ewats/h60xGEwnmOBLMccyPnwu4x76qq9YT2Co6qzjYfISDzUc8\nr4sOsXsCW4olkfiw2GGftini61wuF28P7tx46/wU7xYjInIN0HcSMq7YLEF8belEbs5LYX1hFRv2\n1PLqx8d5p6CCL89OYsn0eIIDz/1vYQ4IY6o9h6n2HMC9CUV1Vw1lHZWUtVdQ2lHB7qb97G7aD7jD\nUIoliXRrMqnWFNKsSQT7Xdhl6R/op6O38xJby7uD2Kn+0xf9LGenGyaZ4z9jg41zQSzYh6YbjiVB\nfkFMjEhnYkS657720x3nBbZqqjqr2dGwmx0NuwHwM/qRGBbnWQeXYknCHhypqZFyTTtU0UpJrbt7\nlhKj7pmIyNVSQJNxyRoawJ1LMlg+N5kPd1bz8e5q3txYyvvbK1l6XSI3zEwgJOjCbfj9jX6eXR9J\nWoTL5cJ5stkd2DoqKO2opLithOLBXQPPbkCRHpVEW/e5QNbddwIXrovWF+YfSmSwzR20AqwXrvcK\ntBDmHzrmpxuONeGBVsLtVqbaJwPnpsWWD+4YWdFZRWVXDeWdVZ7XhPqFeALb1L5Mwl2ROkhdrhnn\nd8+09kxEZHgooMm4Fhbszx0L01g2O5GPd9fw4c5q3tpczvrCKr40M4GlsxIxhwRc9PUGgwFHiB1H\niJ25sbMA966B7sDmDm0VndXUVTYA7s0nwgOtxIQ6PGHr/J0NrQEWLIGWIdMNxXcZDUbPtNh5g3//\nvf191HTXurtsHe41bYdbj3G49RjvV3wEQFSQ7dxW/9YkEsLidS6fjElF5a2U1nUyfUIUyTE6qkRE\nZDjou0ARICTIn1vyU1l6XSIb9tayfkcV72yr5K87a1gyPZ4vz07EGnZlOziG+IcwOSqbyVHus7n6\nB/ohtI/eLhdBpiBNd7vGBZj8z3VZB3f77+49QUVnFc7+Jg7Xl1DZWT1kauzZDWjOP1A7OsSuDqn4\nNJ17JiIyMhTQRM4TFODHTXOSuX5GAp/sr+ODHVV8UFjFx3tqWDg1jpvmJGGzfL41XSajCXtYOM6T\n42fTDRkqLCCUyVHZ2O2zcUZ3DU6NbTk3LbKzmuruOqq7atlSux1wnyWXdN6B2smWRMIDrV7+JCLn\nHCxrpayuk5kT7SRFq3smIjJcFNBEPkOgv4mlsxJZPC2erQfrebegko9317Bxby3zc2NZPjcZe7i2\nWJcvxj01NgpHSBTXxUwH3OfU1XbXD9mE5Pz1jOBeA3f+Vv9J5oQxfTafjF3u7lkZALeqeyYiMqwU\n0EQuwd/PyOLp8czPjaXgUAPvFlSyaV8dm/fXMy8nmpvzUoixhXi7TLkG+Bn9PJuJLBy8r6fvJFVd\nNZ5t/is6q9jnLGKfswg4twlNynm7RsaGRg/rUQ8in+VAaQvl9V3MyrST6NCmNyIiw0kBTeQK+JmM\nLMiNI29yDDuPNPFOQSVbixrYdqiB67IcrMhLIcGub1JkeIX4B5Nlm0CWbQLg7lq0nW73BLbKzmqq\nOmuoO9HAtvqdAPgb/Ukyx593PlsStqBwrX2UYXP+2rNb89U9ExEZbgpoIp+DyWhkbk4MsydFs+eY\nk3e2VVB4pInCI03MmGjnlrwU7WQmI8ZgMGALisAWFMEMRy7g3oSmoaeJio4qKjqrqeyqpmzwYPWz\nzP5hpFgTSTYnDf6eSIi/pujKF7O/pIWKhi5mZTlIUPdMRGTYKaCJfAFGg4FZWQ5mZtrZX9rCuq0V\n7Cl2sqfYSW56JLfkpZAerw0dZOSZjCbiw2KJD4slP34OAKfOnKa6q9bTZavorOZg8xEONh/xvM4R\nEuXpsqVakogLi9XxDnJZZ7tnBuAr+SneLkdE5Jqkq7HIVTAYDEzLiGJqeiSHK9pYt7WcA6UtHCht\nYVJKBLfkpZCZFOHtMmWcCfILZEJEGhMi0jz3dZzu9IS1s78XNuyhsGEPAH4GEwnmeM/5bHNCpgA6\nm02G2lfSTGVjF7OzHcRrWreIyIhQQBMZBgaDgZxUGzmpNo5VtbFuWwWHK9o4XNHGxAQrX7spm4SI\nYIxGrQMS77AGWsi155BrzwFgwDVAU0/zkC6be0OSKjYB/9/RN3h8zo+JDLZ5t3DxGed3z27R2jMR\nkRGjgCYyzDKTIshMiqC0toN12yo4UNrCv7y4ndAgP6akRTIlPZLJqTbMIQHeLlXGMaPBSEyog5hQ\nB3NjZwHQ199HTXcdFZ3VGAL6de6aDLH3eDNVjd3MmRRNfFSot8sREblmKaCJjJD0eCv/z8qpVDZ0\nsf1oE4WHGth+uJHthxsxAGnxFnLTIslNjyIpOky77InX+Zv8SbUmk2pNxm4343TqcHVxGzive3ar\n1p6JiIwoBTSREZYcY2bWlDiaFnVS4zzBgdJmDpa2UFLbSWltJ/+7uRxrWABT0iLJTYtkUoqNkCD9\nrykivmNvsZPqpm7mToomNlLdMxGRkaTvAkVGicFgINERRqIjjJvnpXDiVB+Hyls5UNrCwbIWthyo\nZ8uBekxGAxMSrOSmRzElPZK4yBB110S87I033mDt2rWe20VFRbz66qv827/9G0ajEYvFwtNPP01w\n8LnjC9asWcOzzz5LUlISAHl5edx///2jXvvV8nTPDHCLumciIiNOAU3ES0KD/JmdHc3s7GgGXC4q\nG7rYX9LMwbIWjla1c7SqnT9vKCHSEkRueiS56ZFkJUcQ6G/yduki487KlStZuXIlAIWFhbz//vv8\n/Oc/59FHHyU3N5df/vKXrFmzhq9//etDXrd8+XIeeeQRb5Q8bPYcc1LjPMG8HHXPRERGgwKaiA8w\nGgykxlpIjbVw24I0Ok70UlTm3q6/qLyVDXtr2bC3Fj+TkazkcPfatYwoHOE6bFhktD3//POsXr2a\n4OBgwsLcW83bbDba29u9XNnwG3C5eHvr2e6Zdm4UERkNCmgiPsgaGkD+lFjyp8TSPzBAaW3n4Plq\nzRSVtVJU1sr/fHScGFsIuenunSEnJoTj72f0duki17QDBw4QGxuL3W733NfT08Pbb7/Ns88+e8Hz\nCwsLue+++zhz5gyPPPIIkyZNGs1yr9ruY05qnSfImxxDjC3E2+WIiIwLCmgiPs5kNDIxMZyJieH8\n3eJ0WjtPcaCshYOlLRyuaOPDndV8uLOawAATk5Ij3IEtLRKbJcjbpYtcc958801uv/12z+2enh7u\nv/9+7r33XtLT04c8d+rUqdhsNhYvXszevXt55JFHWLdu3SXfPyIiBD+/q5/GbLebr/o9BgZcvLt9\nJ0ajgW/fkoM9amQOph6OWkfTWKpXtY4M1TpyxlK9I1mrAprIGGOzBLF4WjyLp8XTd2aA4up2d3et\nrIW9x5vZe7wZgERHmCespcdbMBnVXRO5Wjt27OCxxx4D4MyZMzzwwAOsWLGCO+6444Lnpqene0Lb\n9OnTaW1tpb+/H5Pp4gGsra3nqmscriMSCo80UtXQRf6UGPxdrhE5dmGsHecwlupVrSNDtY6csVTv\ncNR6qYCngCYyhvn7GclJtZGTauMeJtDY1uPeFbLUvdFIdVM37xZUEhLox+Q0G7npkUxOi8SiQ7JF\nPrfGxkZCQ0MJCHD///Piiy8ye/Zsz+Yhn/biiy8SGxvLihUrKC4uxmazXTKc+ZKBAffOjUaDgVvy\nUrxdjojIuKKAJnINiY4IYemsEJbOSuR0bz9HqtoGA1szhUeaKDzShAFIibUwdXDtWnKMGaO28Re5\nLKfTic1m89x+5ZVXSEhIoKCgAIA5c+bw0EMPcf/99/PCCy9wyy238OMf/5jXXnuNM2fO8MQTT3ir\n9M+t8Ggj9S09zJ8SiyNCa89EREaTAprINSowwMS0jCimZUThck2krvkEB8paOFDSwvGaDsrrO3lr\nSzmWEH+mpLnD2uRUGyFB/t4uXcQnTZ48mZdeeslze8uWLZ/5vBdeeAGAmJgYXn755VGpbTgNDLhY\nt7UCk9HACp17JiIy6hTQRMYBg8FAvD2MeHsYN81JpufUGQ5XtHrWrm0tamBrUQNGg4GMBCvzcuNI\njwkjPipUh2SLjDM7jri7ZwtyY3WUh4iIFyigiYxDIUF+zMpyMCvLwYDLRVVjl2ft2vHqdoqr3ec5\n2SyB5A5217KTarJ3/wAAFclJREFUIwgK0D8ZIteygQEXawe7Z1p7JiLiHfpuS2ScMxoMpMRYSImx\ncGt+Kl09vVQ297B1Xy1FZS1s3FfHxn11+JkMZCZFuA/JTo8kWmciiVxzdhxupLG1h4VT44hS90xE\nxCsU0ERkCHNIAEtmRjI5KZz+gQHK67rYX9rMwdIWDpW3cqi8lVc/Po4jIpjcdHdYy0wMx38Yzm4S\nEe/pHxhg7dZy99qzvGRvlyMiMm4poInIRZmMRjISrGQkWPnqonTauk5zsKyFA6UtHKpo5aNdNXy0\nq4YAfyOTkm2ec9cirTokW2Ss2X6okca2kyyeFkeUVd0zERFvuaKA9uSTT7J//34MBgOrVq0iNzf3\nguc8/fTT7Nu3b0zuWCUiVybCHMjCqXEsnBrHmf4Bjle3u3eGLG1hX0kz+0rch2TH20M9UyHT4634\nmXRItogv6x8YYN0299qzm+eleLscEZFx7bIBrbCwkMrKSl5//XVKS0tZtWoVr7/++pDnlJSUsHPn\nTvz9tT23yHjhZzKSnWIjO8XGXddPoKn9JAdLWzhY1sKRyjbe31HF+zuqCA70IyfV5tlsxBqqQ7JF\nfE1BUSNNbSdZPD1eHXARES+7bEArKCjghhtuACA9PZ2Ojg66u7sJCwvzPOcXv/gFP/jBD3juuedG\nrlIR8WmO8GC+NDOBL81MoLevn6ODh2QfKG1h19Emdh1tAiAlxuyeCpkeSWqMBaNR2/iLeNOZ/gHW\nbSvHz2RgxTytPRMR8bbLBrTm5mZycnI8t202G06n0xPQ1qxZw+zZs4mPjx+5KkVkTAnwN5GbHkVu\nehQul4uG1h72l7i7a8XV7VQ0dLF2awVhwe5DsnPTI8lJtREWrC68yGgrONSAs/0US2bEY7OoeyYi\n4m2fe5MQl8vl+XN7eztr1qzhj3/8I42NjVf0+oiIEPyGYbc3u9181e8xmsZSvap1ZIylWmF463U4\nLORmxQDQc6qP/ced7DzcyO6jjRQcaqDgUANGA2Sl2JiVHc2s7GhSYi1XfEj2WBpb1Sq+5Ez/AOu2\nVuBnMnDzXHXPRER8wWUDmsPhoLm52XO7qakJu90OwPbt22ltbeXrX/86vb29VFVV8eSTT7Jq1aqL\nvl9bW89VF223m3E6u676fUbLWKpXtY6MsVQrjHy9GTFmMmLM3L0kneqmbvdUyLIWjlS0cri8lf9+\n7wgR5kCmpNnITY8iOzmC4MDP/udqLI3teKxVIc+3bStqoLnjFF+akaDumYiIj7hsQMvPz+e3v/0t\nd999N4cOHcLhcHimNy5btoxly5YBUFNTw09+8pNLhjMRkfMZDAaSos0kRZtZkZdC98k+ispbBjcb\naeWT/fV8sr8ek9HAxMRwz7lrMbaQK+6uichnO9M/wDvbKvAzGVmutWciIj7jsgFtxowZ5OTkcPfd\nd2MwGHj88cdZs2YNZrOZpUuXjkaNIjJOhAX7M3dSDHMnxTAw4KK8vvNcd62yjSOVbbz+txLs4UHk\npkUxJT2S+eEh3i5bZEzaerDe3T2bmUCEOdDb5YiIyKArWoP2ox/9aMjtrKysC56TkJCgM9BEZNgY\njQbS462kx1u5fWEa7d3uQ7IPDh6S/fGeGj7eU8Nzaw6SFmsmMymCrOQI0uMsBPhf/TpXkWvZ2e6Z\nv5+R5Vp7JiLiUz73JiEiIt4QHhbIgtw4FuS6D8kure3gQGkLx+s6OV7TTnFNB+u2uTc7SIuzkpUU\nTlZSBOnxFvyHYWMikWvJlgP1tHSe5oZZ6p6JiJy1cePHLF78pcs+74knnmDFiq8SFzcyu9groInI\nmONnMpKZFEFmUgR2u5nK6laKazo4VtXG0ap2jle3U1zdztqt7vU16XEWMpPCyU6OIC1OgU3Gt74z\nA7xToO6ZiMj56uvr+Oij9VcU0H7605+O6KZfCmgiMuaFBPkzLSOKaRlRgHsr/+LqDo5WtXGsyh3W\njp0X2DLiLe4pkUnhpMVZ8fczevkTiIyeLQfrae08zY3XJRIepu6ZiAjAr3/9S44cOcSCBddx4403\nUV9fxzPP/I6nnvo3nM4mTp48yb33fo/8/AV885vf5KGHfsiGDR9z4kQ3VVWV1NbW8P3vP8y8eflX\nXYsCmohcc0KC/Jk2IYppE9yB7cSpPndIq2r3hLajVe28Dfj7uTtsWYNr2FJjLQpscs3qO+Neexbg\nZ+Qmdc9ExEf9+W8l7DzaNKzveV2Wgzuvz7jo4/fc803WrPkzqanpVFVV8LvfvURbWyuzZ8/lpptW\nUFtbw89+9ij5+QuGvK6pqZHVq3/D9u3bePvtvyigiYhcidAgf6ZPsDN9gvsMx+6TfRyvdoe0Y+cF\nNraU4+9nJCPeSubgGjYFNrmWbD5QR1vXab48OxFraIC3yxER8UnZ2TkAmM0Wjhw5xNq1azAYjHR2\ndlzw3NzcaYD77Oju7u5h+foKaCIy7oQF+zN9op3pE88FtuLqc921s1v6QzkBfkbS4wc3HRnssPmZ\nFNhk7Ok708+7BZUE+BlZNkfdMxHxXXden3HJbtdI8/f3B+Cvf/2Azs5Onn/+JTo7O/nud795wXNN\npnPr2l0u17B8fQU0ERn3woL9mTHRzozzAtuxwe7a0aq2c4FtszuwZSRYyUyKIDspgpRYswKbjAmf\n7K+nres0y2YnqXsmIvIpRqOR/v7+Ife1t7cTGxuH0Whk06a/0dfXNyq1KKCJiHxKWLA/MzPtzMx0\nB7aunt7BDpu7y3a4wv3rf4EAfyMT4q2ec9hSYhTYxPe4u2cVBPgbWTYnydvliIj4nOTkVI4dO0ps\nbBzh4eEALF58PY8++kMOHy7i5ptvxeFw8Mc/vjjitSigiYhchjkkgJmZDmZmOgDo7OmluOrcpiOH\nKty/AAL9TWQknDuHLVmBTXzAxn11tHf3ctOcJCzqnomIXCAiIoI1a94dcl9sbBx/+tNrnts33ngT\nAHa7Gaezi7S0c9Mw09IyeO65PwxLLQpoIiKfkyUkgFlZDmZlDQa2E72eNWxHq9o5VN7KofJWwB3Y\nJiRYyUqOYG5uHNYgEyajApuMnt6+ft7bXkmgv0ndMxGRMUABTUTkKllChwa2jvMDW2UbReWtFJW3\n8ubGUgID3IEte/Cg7eSYMAU2GVGb9tXR0d3L8rnJmEPUPRMR8XUKaCIiw8waGsB1WQ6uOy+wHatq\no9J5gn3Hmigqa6WozN1hCwowMSEhnKxk95TIpGgFNhk+nu5ZgLpnIiJjhQKaiMgIs4YGMDs7mpsX\nuuesd3Sf5tjZTUcq2zhY1sLBshbAHdgmJrrDWmZSuAKbj3jjjTdYu3at53ZRURGvvvoq//Iv/wJA\nZmYm//qv/zrkNX19fTz66KPU1dVhMpl46qmnSExMHM2y2bi3lo4Tvdw8L5mwYP9R/doiIvLFKKCJ\niIwya1ggs7OjmZ0dDUB792nPtv5Hqto5UNrCgVJ3YAsOHOywJUWQlRxOksOM0WjwZvnj0sqVK1m5\nciUAhYWFvP/++zzxxBOsWrWK3NxcHn74YTZt2sSiRYs8r3nnnXewWCw8/fTTbNmyhaeffppnnnlm\n1Go+3dfPezuqCAow8eXZ6p6JiIwVCmgiIl4WHhbInEnRzJnkDmxtXac5Vu0+NPtoZdunApsfEwc3\nHclKiiDREabANsqef/55nnrqKb7xjW+Qm5sLwJIlSygoKBgS0AoKCrjtttsAyMvLY9WqVaNa54Y9\ntXSe6GVFnrpnIiJjiQKaiIiPiTAHMndSDHMnxQCDgW1wh8ijVW3sL21h/3mBLTMxnKykcDIV2Ebc\ngQMHiI2NxWQyYbFYPPdHRkbidDqHPLe5uRmbzQa4D0A1GAz09vYSEDDyG3Wc7u3n/R2VBAeauPE6\ndc9ERIbL3/3dLbz33ruXf+JVUEATEfFxEeZA5ubEMDfHHdhaO09xrNo9JfJoZTv7SprZV9IMQEig\nn3sNW3IEWUnhJDjCMBoU2IbLm2++ye23337B/S6X67KvvZLnRESE4Odn+kK1na+wuJmunj7uWjqR\n1CTbVb/fSLLbzd4u4XMZS/Wq1pGhWkfOWKjXNHi26UjWqoAmIjLG2CxBzMuJYd75ga3q7DlsbUMC\nW2iQ35BNRxTYrs6OHTt47LHHMBgMtLe3e+5vbGzE4XAMea7D4cDpdJKVlUVfXx8ul+uy3bO2tp6r\nrtFsCebNvxUTHOjH/JxonM6uq37PkXL2sNexYizVq1pHhmodOd6u9957v86TTz5NTEwMDQ31/OQn\nD2O3Ozh58iSnTp3iBz/4MZMmTaa/fwDgqmu9VMBTQBMRGeNsliDmTY5h3mR3YGvpOMWx6jbPLpF7\njzez9/inAtvgGrZ4e6gC2xVqbGwkNDTUE7LS0tLYtWsXs2bN4sMPP+Sb3/zmkOfn5+fzwQcfsGDB\nAjZs2MCcOXNGpc53t5bT1dPHrfkphAZp7ZmIjD1rSt5hb9PBYX3P6Y4p3JGx4qKPL1y4hK1bP+Gr\nX72TzZs3sXDhEtLTJ7Bw4WJ2797JK6/8iSee+L/DWtPFKKCJiFxjIq1B5FljyZscC0Bzx8nBXSLd\nXbZPB7bMJPd0yKykCOIU2C7K6XR61pQBrFq1in/+539mYGCAqVOnkpeXB8D999/PCy+8wPLly9m2\nbRv33HMPAQEB/OIXvxjxGk/1nmHNxhKCA/248brR3dJfRGQsW7hwCc899wxf/eqdbNmyiYce+gGv\nvfYyr776Mn19fQQFBY1aLQpoIiLXuChrMFFTgsmfMhjY2k+6z2GrdE+J3FPsZE+xe4OLsGB/MhPD\nmT89nikpEQpr55k8eTIvvfSS53ZGRgb/8z//c8HzXnjhBQDP2Wej6ePdNXSe6OUr81MJUfdMRMao\nOzJWXLLbNRLS0tJpaXHS2NhAV1cXmzdvJCrKwc9+9u8cPXqY554bvWNSFNBERMaZqPBgosKHBraj\n561h213sZHexk1/841wcESFerlY+j0PlrZhD/Fk6S90zEZHPa968+fzhD79jwYJFtLe3kZ4+AYBN\nmzZw5syZUatDAU1EZJyLCg9mfngw83NjcblcNHecwhjgR2SIOjBjzXdXTMIaHoJpYMDbpYiIjDmL\nFi3h//yfe/mv/3qVU6dO8vOfP86GDR/x1a/eyUcffci7764dlToU0ERExMNgMGAPD/b6blryxdgs\nQdgjQ/V3JyLyBWRn57Bp0w7P7VdeedPz5/nzFwFw8823EhoaSk/PyP07axyxdxYREREREZHPRQFN\nRERERETERyigiYiIiIiI+AgFNBERERERER+hgCYiIiIiIuIjFNBERERERER8hAKaiIiIiIiIj1BA\nExERERER8REKaCIiIiIiIj5CAU1ERERERMRHGFwul8vbRYiIiIiIiIg6aCIiIiIiIj5DAU1ERERE\nRMRHKKCJiIiIiIj4CAU0ERERERERH6GAJiIiIiIi4iMU0ERERERERHyEn7cLuJwnn3yS/fv3YzAY\nWLVqFbm5uZ7Htm3bxq9//WtMJhMLFy7kwQcf9GKll671+uuvJyYmBpPJBMDq1auJjo72VqkAFBcX\n88ADD/Cd73yHb3zjG0Me87WxvVStvja2v/rVr9i9ezdnzpzhH//xH7nxxhs9j/nauF6qVl8a15Mn\nT/Loo4/S0tLC6dOneeCBB1iyZInncV8a18vV6kvjer5Tp06xYsUKHnjgAe644w7P/b40tjLUWLo+\nwti6Rur6OHJ0jRx+ukaOLK9cH10+bMeOHa7vfe97LpfL5SopKXHdeeedQx6/6aabXHV1da7+/n7X\nPffc4zp+/Lg3ynS5XJevdcmSJa7u7m5vlPaZTpw44frGN77heuyxx1wvv/zyBY/70therlZfGtuC\nggLXd7/7XZfL5XK1tra6Fi1aNORxXxrXy9XqS+P67rvvuv7whz+4XC6Xq6amxnXjjTcOedyXxvVy\ntfrSuJ7v17/+teuOO+5w/eUvfxlyvy+NrZwzlq6PLtfYukbq+jhydI0cGbpGjixvXB99eopjQUEB\nN9xwAwDp6el0dHTQ3d0NQHV1NVarldjYWIxGI4sWLaKgoMAna/VFAQEBvPjiizgcjgse87WxvVSt\nvua6667j2WefBcBisXDy5En6+/sB3xvXS9Xqa5YvX84//MM/AFBfXz/kp2m+Nq6XqtVXlZaWUlJS\nwuLFi4fc72tjK+eMpesjjK1rpK6PI0fXyJGha+TI8db10aenODY3N5OTk+O5bbPZcDqdhIWF4XQ6\nsdlsQx6rrq72RpnApWs96/HHH6e2tpaZM2fy8MMPYzAYvFEqAH5+fvj5ffZfv6+N7aVqPctXxtZk\nMhESEgLAm2++ycKFCz1tel8b10vVepavjOtZd999Nw0NDfz+97/33Odr43rWZ9V6lq+N6y9/+Ut+\n9rOf8dZbbw2531fHVsbW9RHG1jVS18eRo2vkyNI1cvh56/ro0wHt01wul7dLuGKfrvX73/8+CxYs\nwGq18uCDD7J+/XqWLVvmpequLb44th999BFvvvkm//mf/+nVOq7ExWr1xXF97bXXOHLkCD/+8Y9Z\nu3at1y+Gl3KxWn1tXN966y2mTZtGYmKi12qQqzeWro+ga+Ro8dVx1TVyZOgaOby8eX306SmODoeD\n5uZmz+2mpibsdvtnPtbY2OjVFv+lagW47bbbiIyMxM/Pj4ULF1JcXOyNMq+Ir43t5fja2G7evJnf\n//73vPjii5jNZs/9vjiuF6sVfGtci4qKqK+vByA7O5v+/n5aW1sB3xvXS9UKvjWuABs3buTjjz/m\nzjvv5I033uB3v/sd27ZtA3xvbOWcsXR9hGvnGumLY3spvjiuukYOP10jR4Y3r48+HdDy8/NZv349\nAIcOHcLhcHimQyQkJNDd3U1NTQ1nzpxhw4YN5Ofn+2StXV1d3HffffT29gKwc+dOJkyY4LVaL8fX\nxvZSfG1su7q6+NWvfsV//Md/EB4ePuQxXxvXS9Xqa+O6a9cuz08vm5ub6enpISIiAvC9cb1Urb42\nrgDPPPMMf/nLX/jzn//MypUreeCBB8jLywN8b2zlnLF0fYRr5xrpi2N7Mb44rrpGjgxdI0eGN6+P\nBpePz4tYvXo1u3btwmAw8Pjjj3P48GHMZjNLly5l586drF69GoAbb7yR++67z2dr/dOf/sRbb71F\nYGAgkyZN4mc/+5lXW89FRUX88pe/pLa2Fj8/P6Kjo7n++utJSEjwubG9XK2+NLavv/46v/3tb0lN\nTfXcN2fOHDIzM31uXC9Xqy+N66lTp/jpT39KfX09p06d4qGHHqK9vd0n/y24XK2+NK6f9tvf/pb4\n+HgAnxxbGWosXR9h7FwjdX0cObpGjgxdI0feaF8ffT6giYiIiIiIjBc+PcVRRERERERkPFFAExER\nERER8REKaCIiIiIiIj5CAU1ERERERMRHKKCJiIiIiIj4CAU0ERERERERH6GAJiIiIiIi4iMU0ERE\nRERERHzE/w9Jo1uKveZHUwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4f13c7e748>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot performance\n",
    "trainer.plot_performance()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "4EmFhiX-FMaV",
    "outputId": "0a662c1d-2b4f-4043-92a9-ff0ce59de944"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test loss: 0.43\n",
      "Test Accuracy: 84.8%\n"
     ]
    }
   ],
   "source": [
    "# Test performance\n",
    "trainer.run_test_loop()\n",
    "print(\"Test loss: {0:.2f}\".format(trainer.train_state['test_loss']))\n",
    "print(\"Test Accuracy: {0:.1f}%\".format(trainer.train_state['test_acc']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zVU1zakYFMVF"
   },
   "outputs": [],
   "source": [
    "# Save all results\n",
    "trainer.save_train_state()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qLoKfjSpFw7t"
   },
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "51_PWM3xgXHJ"
   },
   "outputs": [],
   "source": [
    "class Inference(object):\n",
    "    def __init__(self, model, vectorizer, device=\"cpu\"):\n",
    "        self.model = model.to(device)\n",
    "        self.vectorizer = vectorizer\n",
    "        self.device = device\n",
    "  \n",
    "    def predict_category(self, dataset):\n",
    "        # Batch generator\n",
    "        batch_generator = dataset.generate_batches(\n",
    "            batch_size=len(dataset), shuffle=False, device=self.device)\n",
    "        self.model.eval()\n",
    "        \n",
    "        # Predict\n",
    "        for batch_index, batch_dict in enumerate(batch_generator):\n",
    "            # compute the output\n",
    "            attn_scores, y_pred =  self.model(\n",
    "                x_word=batch_dict['title_word_vector'],\n",
    "                x_char=batch_dict['title_char_vector'],\n",
    "                x_lengths=batch_dict['title_length'],\n",
    "                device=self.device,\n",
    "                apply_softmax=True)\n",
    "\n",
    "            # Top k nationalities\n",
    "            y_prob, indices = torch.topk(y_pred, k=len(self.vectorizer.category_vocab))\n",
    "            probabilities = y_prob.detach().to('cpu').numpy()[0]\n",
    "            indices = indices.detach().to('cpu').numpy()[0]\n",
    "\n",
    "            results = []\n",
    "            for probability, index in zip(probabilities, indices):\n",
    "                category = self.vectorizer.category_vocab.lookup_index(index)\n",
    "                results.append({'category': category, \n",
    "                                'probability': probability})\n",
    "\n",
    "        return attn_scores, results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "h8UoSJPggXC9"
   },
   "outputs": [],
   "source": [
    "# Load vectorizer\n",
    "with open(args.vectorizer_file) as fp:\n",
    "    vectorizer = NewsVectorizer.from_serializable(json.load(fp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "id": "woxuiUA2gXAX",
    "outputId": "d9adcdb7-b41a-43e5-8b86-1b385409762b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<bound method Module.named_modules of NewsModel(\n",
      "  (encoder): NewsEncoder(\n",
      "    (word_embeddings): Embedding(3406, 100, padding_idx=0)\n",
      "    (char_embeddings): Embedding(35, 100, padding_idx=0)\n",
      "    (conv): ModuleList(\n",
      "      (0): Conv1d(100, 100, kernel_size=(3,), stride=(1,))\n",
      "      (1): Conv1d(100, 100, kernel_size=(5,), stride=(1,))\n",
      "    )\n",
      "    (gru): GRU(300, 128, batch_first=True)\n",
      "  )\n",
      "  (decoder): NewsDecoder(\n",
      "    (fc_attn): Linear(in_features=128, out_features=128, bias=True)\n",
      "    (dropout): Dropout(p=0.25)\n",
      "    (fc1): Linear(in_features=128, out_features=200, bias=True)\n",
      "    (fc2): Linear(in_features=200, out_features=4, bias=True)\n",
      "  )\n",
      ")>\n"
     ]
    }
   ],
   "source": [
    "# Load the model\n",
    "model = NewsModel(embedding_dim=args.embedding_dim, \n",
    "                  num_word_embeddings=len(vectorizer.title_word_vocab), \n",
    "                  num_char_embeddings=len(vectorizer.title_char_vocab),\n",
    "                  kernels=args.kernels,\n",
    "                  num_input_channels=args.embedding_dim,\n",
    "                  num_output_channels=args.num_filters,\n",
    "                  rnn_hidden_dim=args.rnn_hidden_dim,\n",
    "                  hidden_dim=args.hidden_dim,\n",
    "                  output_dim=len(vectorizer.category_vocab),\n",
    "                  num_layers=args.num_layers,\n",
    "                  bidirectional=args.bidirectional,\n",
    "                  dropout_p=args.dropout_p, \n",
    "                  word_padding_idx=vectorizer.title_word_vocab.mask_index,\n",
    "                  char_padding_idx=vectorizer.title_char_vocab.mask_index)\n",
    "model.load_state_dict(torch.load(args.model_state_file))\n",
    "print (model.named_modules)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Yfc666nngW9p"
   },
   "outputs": [],
   "source": [
    "# Initialize\n",
    "inference = Inference(model=model, vectorizer=vectorizer, device=\"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1soPBCApgW7C"
   },
   "outputs": [],
   "source": [
    "class InferenceDataset(Dataset):\n",
    "    def __init__(self, df, vectorizer):\n",
    "        self.df = df\n",
    "        self.vectorizer = vectorizer\n",
    "        self.target_size = len(self.df)\n",
    "\n",
    "    def __str__(self):\n",
    "        return \"<Dataset(size={1})>\".format(self.target_size)\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.target_size\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        row = self.df.iloc[index]\n",
    "        title_word_vector, title_char_vector, title_length = \\\n",
    "            self.vectorizer.vectorize(row.title)\n",
    "        return {'title_word_vector': title_word_vector, \n",
    "                'title_char_vector': title_char_vector, \n",
    "                'title_length': title_length}\n",
    "\n",
    "    def get_num_batches(self, batch_size):\n",
    "        return len(self) // batch_size\n",
    "\n",
    "    def generate_batches(self, batch_size, shuffle=True, drop_last=False, device=\"cpu\"):\n",
    "        dataloader = DataLoader(dataset=self, batch_size=batch_size, \n",
    "                                shuffle=shuffle, drop_last=drop_last)\n",
    "        for data_dict in dataloader:\n",
    "            out_data_dict = {}\n",
    "            for name, tensor in data_dict.items():\n",
    "                out_data_dict[name] = data_dict[name].to(device)\n",
    "            yield out_data_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "tU70BuYvgW4Q",
    "outputId": "ddcbb69a-f502-4873-b627-4665ecd68d9b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter a title to classify: Sale of Apple's new iphone are skyrocketing.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'category': 'Sci/Tech', 'probability': 0.6141594},\n",
       " {'category': 'Business', 'probability': 0.3776285},\n",
       " {'category': 'World', 'probability': 0.006894056},\n",
       " {'category': 'Sports', 'probability': 0.001317991}]"
      ]
     },
     "execution_count": 90,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Inference\n",
    "title = input(\"Enter a title to classify: \")\n",
    "infer_df = pd.DataFrame([title], columns=['title'])\n",
    "infer_df.title = infer_df.title.apply(preprocess_text)\n",
    "infer_dataset = InferenceDataset(infer_df, vectorizer)\n",
    "attn_scores, results = inference.predict_category(dataset=infer_dataset)\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R3jrZ6ZkxN4r"
   },
   "source": [
    "# Interpretability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qrAieHoHxOt2"
   },
   "source": [
    "We can inspect the probability vector that is generated at each time step to visualize the importance of each of the previous hidden states towards a particular time step's prediction. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "k6uZY4J8vYgw"
   },
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 336
    },
    "colab_type": "code",
    "id": "UQwfUfwzjE_F",
    "outputId": "7fd81c48-c632-43af-cd54-c95c74ce50d0"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd4AAAE/CAYAAADohqLkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3X98zfX///Hb2WxjNmxsbObnZBiT\nZX6NLTHxplRS3hH1llLxSb+oFVJGiH5QUbwlFYtEpOS3xbD82IZRRovNfjObmf063z+67HzTu5i1\nc7Zz3K9dzqWdH689n2fnOI/zeD4fr+fTYDQajYiIiIhF2FV1B0RERG4mCrwiIiIWpMArIiJiQQq8\nIiIiFqTAKyIiYkEKvCIiIhZUo6o7ICIiUl4BzUIrfGxc0s5K7EnFKfCKiIjVMBgMVd2Ff0xDzSIi\nIhakjFdERKyGwWD9+aL1PwMRERErooxXRESshh3WP8erwCsiIlbDFoqrFHhFRMRq2NnAHK8Cr4iI\nWA1byHit/6uDiIiIFVHgFRERsSANNYuIiNUwqKpZRETEclRcJSIiYkG2UFylwCsiIlbDzgYCr/Xn\n7CIiIlZEgVdERMSCNNQsIiJWw2AD+aICr4iIWA0VV4mIiFiQLRRXKfCKiIjVsIUFNKx/sFxERMSK\nKPCKiIhYkIaaRUTEamjJSBEREQtSVbOIiIgFqapZRETEglTVLCIiIjdEGa+IiFgNWyiusv5nICIi\nYkWU8YqIiNVQVbOIiIgFqapZRETEglTVLCIiIjdEgVdERKyGwWCo8KU8ZsyYwYMPPsiwYcOIi4v7\ny8fMnTuXhx9++IaO+SMNNYuIiNUw5xzv/v37SUpKIjIyksTERMLDw4mMjLzqMSdPniQmJgYHB4dy\nH/M/z8Fsz0BERMSKREdH07dvXwB8fX3JyckhLy/vqse8+eabPPvsszd0zJ8p8IqIiNUw/IP/ricz\nMxM3NzfTdXd3dzIyMkzX16xZQ5cuXWjcuHG5j/krGmoWERGrYcmVq4xGo+nnCxcusGbNGpYuXUpa\nWlq5jvk7CrwiIiKAp6cnmZmZpuvp6el4eHgAsHfvXrKzsxk+fDiFhYX89ttvzJgx45rH/B0NNYuI\niNUwZ1VzcHAwmzZtAuDo0aN4enri4uICQP/+/dm4cSNffvklCxYswN/fn/Dw8Gse83eU8YqIiNUw\nZ1VzYGAg/v7+DBs2DIPBwNSpU1mzZg2urq6EhYWV+5jrMRjLMyAtIiJSDTzYeXSFj438aUkl9qTi\nNNQsIiJiQRpqFhERq2ELmyQo4xUREbEgZbwiImI1tB+viIiIBdnCULMCr4iIWA1b2I9XgVdERKyG\nLWS8Kq4SERGxIAVeERERC9JQs4iIWA1VNYuIiFiQLczxKvCKiIjVUFWziIiIBdlCxqviKhEREQtS\n4BUREbEgDTWLiIjVUFWziIiIBdnCHK8Cr4iIWA1lvCIiIhZkC6cTqbhKRETEgpTxioiI1bCz/oRX\nGa+IiIglKeMVERGroeIqERERC9LpRCIiIhZkCxmv5nhFREQsSBmviIhYDTsbOI9XgVdERKyGhppF\nRETkhijjFRERq6GqZhEREQuygbiroWYRERFLUsYrIiJWQ0PNIiIiFmQL2wIq8IqIiNXQ6UQiIiJy\nQ5TxioiI1dAcr4iIiAXZQNzVULOIiIglKeMVERGroaFmERERC9LpRCIiIhZkCxmv5nhFREQsSBmv\niIhYDXMnvDNmzCA2NhaDwUB4eDgBAQGm+7788ktWr16NnZ0dbdq0YerUqRgMBmbPns2BAwcoLi7m\niSeeoF+/ftdsQ4FXREQE2L9/P0lJSURGRpKYmEh4eDiRkZEAXL58mW+//ZbPP/8cBwcHRo4cyaFD\nhygsLOSXX34hMjKS8+fPc++99yrwioiI7TDnkpHR0dH07dsXAF9fX3JycsjLy8PFxYVatWqxbNky\n4PcgnJeXh4eHB97e3qasuE6dOly+fJmSkhLs7e3/th3N8YqIiNWwMxgqfLmezMxM3NzcTNfd3d3J\nyMi46jEfffQRYWFh9O/fnyZNmmBvb4+zszMAq1evJiQk5JpBFxR4RUTEihgMFb/cKKPR+D+3Pf74\n42zZsoWoqCgOHDhgun3Lli2sXr2aKVOmXPf3KvCKiIjVMGfG6+npSWZmpul6eno6Hh4eAFy4cIGY\nmBgAatasSUhICAcPHgQgKiqKhQsX8vHHH+Pq6nr951CRJy4iImJrgoOD2bRpEwBHjx7F09MTFxcX\nAIqLi3nppZe4dOkSAPHx8bRo0YLc3Fxmz57NokWLqFevXrnaUXGViIgIEBgYiL+/P8OGDcNgMDB1\n6lTWrFmDq6srYWFhPP3004wcOZIaNWrg5+dHnz59+PLLLzl//jwTJkww/Z5Zs2bh7e39t+0YjH81\niC0iIlINzb5nWoWPnbh2aiX2pOKU8YqIiNUw5+lElqLAKyIiVsPO+uOuAq+IiFgPW8h4VdUsIiJi\nQQq8IiIiFqShZhERsRq2MNSswCsiIlZDxVUiIiIWpIxXRETEgmwg7qq4SkRExJKU8YqIiNUozy5D\n1Z0yXhEREQtSxisiIlbDgPVnvAq8IiJiNWxgpFmBV0RErIfmeEVEROSGKOMVERGroQU0RERELMgG\n4q6GmkVERCxJGa+IiFgNDTWLiIhYkC3sTqShZhEREQtSxisiIlZDQ80iIiIWZANxV4FXRESsh1au\nEhERkRuijFdERKyGLczxKuMVERGxIGW8IiJiNWwg4S1fxpuTk8OsWbN44YUXANi2bRvZ2dlm7ZiI\niMifGQyGCl+qi3IF3ldffRUvLy/Onj0LQGFhIZMmTTJrx0RERP7MYKj4pbooV+DNzs5m5MiRODg4\nANC/f38KCgrM2jEREZE/szMYKnypLspdXFVUVGRK1TMzM8nPzzdbp0RERGxVuYqrRowYwf33309G\nRgZjx44lPj6eV155xdx9ExERsTnlCrwDBgygU6dOHDp0CEdHR15//XU8PT3N3TcREZGrVKMR4wor\n11DzyZMn+fzzzxkwYAB9+vTh7bff5ueffzZ330RERK5y01Q1T5s2jdDQUNP1IUOG8MYbb5itUyIi\nIn/FFqqayzXUXFJSQufOnU3XO3fujNFoNFunRERE/kp1ylwrqlyB19XVlS+++IKuXbtSWlpKVFQU\ntWvXNnffREREbE65Au/MmTOZO3cuK1asAKBTp07MnDnTrB0TERGxReUKvO7u7kRERJi7L/9Y4cWs\nKmnXsU79Ku1DVbf/5z4ENAu9xiPNIy5pp+nngswUi7cPULOBt+nn5f+Za/H2H/7v86aff/50tcXb\nB2g98n7Tz8mbNlm8/cZ33mn6+cr5NIu37+TW0PRzdfi3WNV/A3OwgZHm8gXeDRs2sHjxYnJycq6a\n292xY4e5+iUiIvI/qtMKVBVVrsA7f/58pk+fjre39/UfLCIiYiY2EHfLF3ibNWtGUFCQufsiIiJy\nTeauap4xYwaxsbEYDAbCw8MJCAgw3bd3717mzZuHnZ0dLVq0ICIiAju738/KLSgoYNCgQTz11FPc\nd99912yjXIG3U6dOzJs3jy5dumBvb2+6vXv37hV5XiIiItXO/v37SUpKIjIyksTERMLDw4mMjDTd\nP2XKFD799FMaNWrE//3f/xEVFWVa4+LDDz+kbt265WqnXIF3z549ABw6dMh0m8FgUOAVERGLMmfC\nGx0dTd++fQHw9fUlJyeHvLw8XFxcAFizZo3pZ3d3d86fPw9AYmIiJ0+e5Pbbby9XO+UKvMuXL/+f\n2zZVQcWiiIiIuWRmZuLv72+67u7uTkZGhinYlv0/PT2d3bt388wzzwAwa9YsJk+ezNq1a8vVTrkC\nb0pKCp999pkpuhcWFrJv3z7u/EPpvoiIiLlZcuWqv1qhMSsri7FjxzJ16lTc3NxYu3Ytt956K02a\nNCn37y1X4J04cSIhISFs376dESNGsHXrVmbPnl3+3ouIiFQCc8ZdT09PMjMzTdfT09Px8PAwXc/L\ny2PMmDFMmDCBnj17Ar+fVnvmzBl27NhBamoqjo6ONGrUiB49evxtO+UKvPb29jz++ONERUUxfPhw\n7r//fp577rlr/mIREZHKZs6MNzg4mPnz5zNs2DCOHj2Kp6enaXgZ4M0332TUqFGEhISYbnvnnXdM\nP8+fP5/GjRtfNzaWK/BeuXKF1NRUDAYDZ86cwdvbm+Tk5Bt9TiIiItVWYGAg/v7+DBs2DIPBwNSp\nU1mzZg2urq707NmTtWvXkpSUxOrVv68MN2jQIB588MEbbqdcgfexxx4jOjqa0aNHM3jwYOzt7Rk0\naNANN3Ytly5dMqX4Hh4eODs7V+rvFxER62fuKd4XXnjhqutt2rQx/XzkyJFrHjt+/PhytVGuwNui\nRQt8fX2B389zunTpEqdPny5XA9cTHx9PREQEFy9exM3NDaPRSHp6Og0bNmTKlCn4+flVSjsiImL9\nbH5bwIsXL3LhwgXCw8N56623TLcXFRUxadKkSjmlaMaMGURERJgCe5mjR4/y+uuv8/nnn//jNkRE\nRKqLawbeQ4cOsWzZMhISEhg1apTpdjs7O1NF1z9lNBr/J+gC+Pv7U1JSUiltiIiIbbCBhPfagTc0\nNJTQ0FA+//xzhg8fbpYOdOzYkbFjx9K3b1/c3d2B309i3rRpE126dDFLmyIiYp1umt2Jvv/+e7MF\n3pdffpmYmBiio6OJi4sDfj+Xaty4cXTq1MksbYqIiHWygbhbvsDbtm1b3n33XTp16oSDg4Pp9spa\nqzkoKEi7H4mIyE2hXIE3ISEBgJ9++sl0mzZJEBERS7P5quYyf7VJgoiIiKXZQNzFrjwPSkxMZOTI\nkQQGBnLbbbcxevRofvvtN3P3TURExOaUK+N94403+M9//kOXLl0wGo3s2bOHqVOnsnTpUnP3T0RE\nxMRgZ/0pb7kyXqPRyO23346zszO1a9cmLCxM59iKiIjFGQwVv1QX5Qq8RUVFHD161HQ9Li5OgVdE\nRKQCyjXUPGnSJJ5//nmysrKA38+znTVrllk7JiIi8mc3TVVzx44d+f7778nNzcVgMFy1P6GIiIil\n2EDcLV/gPXnyJO+99x4nT57EYDDg5+fH+PHjadGihbn7JyIiYmILGW+55nhfeuklQkJCWLBgAe+9\n9x7dunVj0qRJ5u6biIiIzSlXxlurVi3uv/9+03VfX99K2RJQRETkRthAwovBaDQar/eg999/Hz8/\nP4KDgyktLWXv3r0kJCTw9NNPYzQasbMrV+IsIiLyj0S99nGFj+312phK7EnFlSvj/eCDD/7y9KEF\nCxZgMBhMazmLiIiYlQ2kvOUKvH88h1dERKSq2EJxVbkCb1paGps2bSI3N5c/jkyPGzfObB0TERH5\nMxuIu+ULvGPGjMHf35+GDRuauz8iIiJ/yxbWai5X4K1Xrx4zZ840d19ERERsXrkCb1hYGN988w2d\nOnXC3t7edLu3t7fZOiYiImKLyhV4T5w4wfr166lXr57pNoPBwI4dO8zVLxERkf9x08zxxsbGEhMT\ng6Ojo7n7IyIi8rdumqrm9u3bc+XKFQVeERGpUjYQd8t/OtEdd9yBr6/vVXO8n3/+udk6JpaRkJCA\nu7u7KtZFKllpaalW9TODmybjHTt2rLn7YTWMRuNVL/yfr1fW77WElJQUnnvuOd5++23q1auHk5OT\nRdu/lr/6e5jzg+ybb76hTp06+Pn54eXlZZY2KiIrKwt3d/dq8WFTFe/Rqmy3ouLj4/Hz88PR0dFm\ng6+5PgdvFtcMvKWlpQB07tzZIp2prv74pjIYDBQWFlJYWIizs3Ol/KP64++/cOECtWrVskgQrFGj\nBsHBwaxevZoGDRpUmy9YZX+PPXv2cPjwYerUqUO/fv3w9PQ0S3urVq1i48aNjBo1CmdnZ7O0UREJ\nCQmsXbuWl19+ucr6UPZaHDhwgKioKIKCgvD19aVRo0ZmbTcxMRE3NzdcXV1xcHCo9h/sZf377bff\nmDVrFqWlpXzyySc2FXz//DlYXFxMSUkJTk5O1fq1qY6uGXjbtWv3l3/QshfgZlmjuexv8PPPP3Pp\n0iU++eQTDAYDgwcPpnfv3hX+vWV/x7Lf/+WXX7Jz506aNGlCkyZNGD58eKX0/8/KPgg8PT1p3749\nERERTJgwgfz8/GoReMqC7scff8x//vMfvvjiCzIyMnj22Wcrva2cnBw2btzIiy++iKurK9u2bSMl\nJQVfX1/69+9f6e2VV2FhIbfccgvnzp1jwYIFVbZKXNlr8fbbbzNy5Ejmzp3L0KFDeeCBB66adqpM\nq1atYsOGDXTq1InU1FSmT59OjRrlGpyrMmV/p6VLl3Lvvffy3Xff8cgjj7B06VKcnJxsIviWfU4l\nJSWRl5fHsmXLqFGjBoMHD6Zr164W7IfFmjKba74Tjh8/TkJCwv9cym63ddnZ2Vy4cIGsrCyWLl3K\nrFmziImJoW/fvjRq1IgWLVr8o99/6dIl08/ff/89W7Zs4fXXX+fSpUucPn36n3b/b5V9AKxdu5ai\noiJ8fHz47bff2LJlC1lZWWZr91pSUlJ46623TNcPHTrE2LFjKS0tpaioiEceeYSkpCQKCgoqtV0X\nFxduu+023nzzTWbOnMmZM2fw9vbm1KlTlGPjLrP49ttvmThxIrGxscyZM4eMjAx27txZJX0xGo0c\nO3aMN954Az8/P2rXrs1dd91FTk6O6f7KdPDgQTZs2MAHH3xAaWkpNWrUoEaNGlX2WpRH2cjgjh07\naN++PUOGDGHx4sW0bNmSxx9/nMLCQuzs7EyPs1apqaksXryYmTNnsmXLFjp27IiTkxNNmza1aD/K\nkpWKXKoL+9dee+21qu5EdZSbm8snn3xCcXExLi4u1KpVi6FDhxISEoKzszMrV65kwIABuLq63vDv\nNhqNpKWl8cADD3DrrbfSsGFDkpOT8fPzIzY2ltOnTzN9+nTi4+MpKCi46vzpyrJ+/Xq+/PJL/vWv\nf3HixAliYmIwGo04ODhQt25dateuXeltXouDgwNvvvkmp0+fJiQkhF9++YVvv/2W+Ph4pk6diqen\nJ19++SW33HILtWrVqrR27ezsaNu2LR06dGDYsGH07NmTtLQ0tm3bRlhYWJVkWoWFhWzcuJG4uDhS\nUlLo0qULGRkZ+Pv7U1paavYPkLKRmMLCQmrUqMGpU6d47bXXiI+P55133qFu3brMmDGD9u3b4+Li\nUmntJiYm4uzsjIODA7GxsZw4cYKZM2dy4sQJkpKSqt2CPWV/p+zsbJydnbl8+TLnzp3D09MTd3d3\nWrduzbp169i6dSsDBgy44fdSZmYmBQUFlfp+v1Hp6enk5eUBcPnyZezs7Lj33nsJCwvDzc2NdevW\n0a9fP4t+XqTsjQUDFbp497jVYv28FgXePyn7x+Tk5ITRaCQ+Pp4aNWrQvn17GjRoAPyenXbo0IGg\noKAKtWEwGHBxccHOzo63336bjh07mgJPfn4+7733HnZ2dnz66ac0bNiwUj9wjEYjJSUlLFu2jCFD\nhtCzZ0/uvPNOYmNj2b9/P8XFxTg6OuLr62uxb4hlbd53333897//5eTJkwwbNoyvvvqKjh070q9f\nPw4dOsTixYu5/fbbcXNzq9T2nZyc8PT0JD4+npUrV7Ju3TomT56Mh4dHpbZzPRs3biQjI4N27drR\nunVr2rRpQ3JyMgcPHmT16tXcdtttFin8MhgM7Ny5k/nz55OQkECfPn24fPkyRqORwYMHk5iYyPr1\n6+nRo0elvRY//vgju3btokWLFnz00UckJSWxePFi7Ozs+OKLLygsLKR9+/aV0lZlKBs6/vHHH3nu\nuefIycmhZs2aJCUlcenSJZydncnPz8fJyYmCggIyMjLo2LHjDbVx8uRJHBwcqFOnjpmexbVFR0cT\nERFBTEwMa9asISsri3bt2tGqVSuMRiNfffUVnTt3plOnThbt17m9cRXOeL2739hrYC4KvH+Snp5u\n+hbfpEkTHB0d2bt3L6Wlpbi7u1OzZk0+//xzOnToQPPmzW/495cNmRkMBgICAnBycmLmzJk88MAD\neHh4cODAARo3bsyOHTvYtWsX9957L3Xr1q2052cwGLCzsyMtLY309HR8fHyoU6cOffv25ezZs/Ts\n2ZPAwMAKZfIVYTQasbe35+TJkxQWFjJq1CgWL15MVlYWzz33HF9//TW7d+9m3bp1PP/88wQEBJit\nLzVr1sTOzo6hQ4f+42mEG1VaWsqpU6fYtWsXZ8+e5fjx4xgMBoYPH05AQAAFBQV06tQJd3d3s/fl\n5MmTLFy4kP79+5OTk8OaNWu4//77ycjI4L333mPz5s089thjBAYGVkp7iYmJPPHEE4SGhnLHHXfQ\nvHlztmzZwpUrV9ixYwcHDhxgxIgRlf6FqyLy8/NxcHDAYDBw6tQptm3bxr333ktycjJGo5GGDRuS\nkpJCTEwMH330Ec899xylpaVcuXLlhgNUw4YNqyzo7t69m08++YRx48YxevRofH19uXTpEt999x3N\nmjXDw8ODlStX0rVrV4uPRJyLjq3wsQq81YzRaCQlJYUBAwYQExPD8ePHqV+/Pl5eXjRu3JhDhw5R\nWlqK0Wikbt263HHHHRVqo+yb15YtW0hJSaFr1640adKEyZMn8+STT9KoUSPi4+NJSEggPDy8QsG9\nPNzd3dm+fTslJSW4urpy+PBhdu3axfjx4ys10F+PwWAgOjqaV199ld27d7Nnzx5ee+01Fi9eTG5u\nLpMnTyY4OJiQkBCzZzy1atWiefPmZhnav5Y1a9awZMkSfHx8qFevHm3btiUmJobvv/+exMREBg4c\nyO23326RoJucnMynn36Ku7s7jz32GB07duTcuXNs3bqViRMncs899xAWFkaHDh0qpb2vv/6a/Px8\njh07xqlTpwgKCqJdu3b4+/uTk5NDbm4uTz75JC1btqyU9v6Jy5cvM3v2bAICAigqKuKhhx6icePG\nPPLII7Rq1YqDBw9iZ2dH69atGTJkCL6+vvz222989dVXPProoxZ5/SrDmTNnGD16NA8//DB9+/YF\nfv8S0LBhQzIzM0lOTqZp06YUFhYSFhZm8f4p8NqQ/Px86tevj7OzMwUFBZw7dw6DwcC7775L7dq1\niYmJITU1FXt7e/r27Yu9vf0Nn+JQ9thVq1axYsUK6tSpw/vvv8+jjz5K8+bNmTJlCg888ACDBg2i\nT58+pqFtc6hTpw7NmzcnOjqajRs3Ehsby6RJk8x2ys7fSUxMZOHChcyYMYNHH32UzZs3ExcXx6xZ\ns3jvvfc4evQo/fr1s+iXAUtav34969evZ9y4cSxatIimTZsyYMAAunXrxrlz50hNTSU4ONis1eZl\n7+PLly/j4uLCyZMnSU1NxcHBgVatWtGxY0eOHz/OihUruPvuuyvttVi1ahXr16+nY8eOnDlzhqSk\nJPbu3UuXLl1o3bo1/v7+dOvWrVpkuqWlpTg6OhIQEMDFixdJTk6mf//+fPzxx7Ro0YK2bdvSqlUr\noqOjSU1NpXPnzri7uxMVFcVTTz2Fr69vVT+FcikuLsbNzY3i4mKOHz+Ot7e3acrFxcWF/Px81qxZ\nw/Dhw01nvVj6VK/UvbEYDFTo4qXAWz2UFToNGjSIrl27mr7B2dvbc+edd/Kvf/2LunXrkpyczC+/\n/MLmzZt54IEHbujctT++Mc+ePcuiRYt4//33OXLkCKdPn2bz5s2MGTOGkpISlixZwuDBg03DWebk\n5ubGbbfdRvfu3enTpw8+Pj5mba9M2d/DaDSyceNGdu/eTZMmTfDz86Nfv3588cUXpKSkMH36dBo0\naFCtFrSoTIWFhSQmJnLXXXdx6tQpzp49y0svvcSJEyfw9vamS5cu9OnTx+yBp2zU4Z133uHMmTM0\na9aM0tJS0tLSuHz5Mr6+vgQFBdGpUyfq169fKW1evHiR5cuX89JLLxEXF8fZs2dp3bo1CQkJ/PDD\nD/Tu3dti0x3XU1JSwk8//WQa9j19+jSTJ0+mf//+9O7dm1deeYWWLVvSpk0b/Pz8uOWWW2jUqBGu\nrq50797dajLdc+fOMWnSJHr27EmPHj1ITk5mw4YNNG/e3BR8mzVrxs6dO+nZs6dprQFLVwun7qv4\nHK9XNwXeaqGs0MnBwYGIiAi6detGly5dSE9PZ/fu3TRt2hR/f39CQ0MZMmQIgwcPvqFv/H8Mut98\n8w329va0atWKH3/8kYMHD7J48WJiY2OZPXs27du359lnn6Vu3boWezPb29vj7Oxs0fN3DQYDsbGx\npKam4urqSvPmzTlx4gSFhYW0bNmSBg0aEBcXR9++fW026K5cuZINGzawatUqvvnmG3Jzc5k/fz4G\ng4G5c+fi4+NDo0aNLLKQytGjR5k2bRpPPfUUKSkp5Obm4uTkhJ2dHQkJCRiNRnx9fSt1CN7JyQlv\nb2/27dvHjz/+yEcffURaWpop833ooYeqbH7zz8pqIsaOHcuKFSuYOHEiPj4+zJkzhz59+tCnTx+e\neeYZWrduTdu2bS0+VfFPlX1Gubq6kp6ezooVK+jZsydBQUGkp6ezfv16U/D95ptv2Lt3L3fddVeV\nrXSXti+uwhlvIwXeqnfmzBnOnTuHnZ0d3bp1o1atWrz66qv06NGDHj16kJWVxYEDB3BycjKt1FOz\nZs0KDy+vW7cOf39/unTpwvnz53F2diYoKIjc3FwCAwMJCQmxWNZZFcr+gR85coTw8HDOnj1Leno6\nRUVFtGjRgq+++orExEQ2b97M4MGDzTa/XdW2b9/O119/zd133016ejqxsbF4eXkREhLCDz/8QExM\nDAMHDrRYxhcXF0ft2rW5//77CQgIIC0tjaSkJAYOHEhWVhYBAQFmydoaNmxIUVERWVlZhISEkJyc\nTEhICOPHj682pw6VvWe9vb05cuQIv/76K/379zdl/2+//TahoaH0798fOzs7q/z3m5eXZwqinTp1\nIj09naVLlxISEkLnzp3JyMhg27Zt/Pzzz+zYsYMpU6ZU6Rfi1P3xFYu6BgONupqvOPNG3LSBt6yg\nJzY2lt9++43g4GDat29PnTp1CA8PNwXflJQUTpw4QUBAAPb29hXKRHNycpg/fz4TJ06kUaNG/Pjj\njxw/fpzc3Fx27NjBtm3bePZeVeS5AAAYvUlEQVTZZ21+owKDwcD+/fv54YcfGDt2LA8//DDFxcWc\nPn2a2rVrmzLf0NBQBg4cWNXdNYsTJ07w2Wef0a9fP/r370/37t2Jj48nOjqa3Nxcjh49yosvvkiz\nZs3M0n5xcbFpAZXMzEyMRiO1a9fmvffeo0mTJvj6+uLn58fKlSvp0KEDAwYMMOtQqcFgYOvWrezY\nsYOvv/6a0aNHm305yvIqC7onTpygoKCAHj16EBgYyHPPPUeHDh0IDg6mTp06zJ07lzFjxuDr61vt\nl7b8s+PHjzN27Fhyc3M5ffo07dq149Zbb8Xe3p6FCxfSu3dvbr31Vn755RfWrFnD7NmzadWqVZX2\nOXVfXIWPVeCtQvv27WPevHm8+uqrPP744wQHBwO/7zt8xx130KBBA1555RWCgoIICQmhTZs2/+gk\ndgcHB5KTk4mMjGTfvn04OTnRsmVLHB0dadCgASNGjKg23/DNoezDKC8vj40bN/L111/TtWtXWrRo\nQb169UxZ7913383ly5eJjY3F1dXVKrOH6zEYDCQlJXHgwAF8fHxo1qwZd9xxBz/99BONGzc2zWub\nQ3Z2Nhs3bqRly5YcPHiQF198kdjYWBwcHOjfvz8rV67EwcEBJycnvv32W9O/BXNydXXl1ltvpU6d\nOjz88MPV6jUvWwZyxowZGAwGPvroI8aMGUPdunWZNWsW8Hu2+OKLL5qGl60p6BYWFnL58mWOHTtG\nUVERR44c4fDhw6xcuZJevXpx+PBhfvzxR7p160ZoaCh33XUXjRs3rupukxYTX+E53kZdqkfgrd4L\noFaysiXbdu3axYgRI646r27WrFnEx8cTFhbGqFGjyM3N5YUXXmDdunX/eHUee3t7Ro0aRe/evWnZ\nsiW1atVi+/bt/PDDD7z99tvValcgczAYDOzatYuFCxcSHBxMjRo1WLhwIc2bN6dFixY0aNCAzz77\njIcffpiePXtSo0aNanH6iDnUr1+fRx99lK+++ooNGzZgMBjo0KEDixYt4uLFi2Zb/xh+z27i4+PJ\nzc3l2LFjTJkyBaPRaHpdRo8ezbvvvmvqo5+fn9n68kdlp6pUN1lZWXzwwQfMmzePw4cPm/6dDh06\nlLp16/L1118zdOhQqyme+qOoqCjWrl3L3LlzGTlyJIcPH+buu+/Gy8uLxMRE4uLicHJyYsOGDRw9\nepQNGzZY5fOsrm6qwFu2WEOdOnVMH3ClpaVs27aNixcvMmbMGFauXMktt9zC8OHDGTRoEI6OjpXS\ntqurK/7+/hw8eJCdO3eyb98+IiIibD7owu9ziIsWLWL69On88MMPtGjRgiNHjvD888/Tr18/UlJS\nGD16NPb29jRp0gQvL69qvyj+P1GvXj3uvfde1q5dS2RkJHZ2dvj7+5v9lKkePXoAsHXrVi5fvkyL\nFi1wd3fniSeeYNGiRXh4eLBs2TIKCgqoWbOmWftS3WVnZ1OrVi26devGrl272LZtG9OnT6eoqIh1\n69YxePBgevbsibOzs9UNL0dHR7NkyRKefvppALp06UJ+fj7R0dF06tSJkJAQQkNDARg3bhw1a9Y0\n6xfCG2VFf+q/Zd3bZdyAn376ieXLlwPg6OjIl19+CfxesdiqVSsiIiIIDQ2lQYMGFBUVAZiluKVF\nixYEBgYya9Ysqzm375+qVasWgwYN4vjx4+zevZtHHnmEnj17kpKSwnfffcedd97J7bffTnFxMYBN\nB90y7u7uDB48GD8/P7Nne2UjPenp6QQFBdG/f38aNGjAd999Z7pt9OjRREZGcu7cuZviy+CfZWdn\nk5CQQFFRESkpKbz66qsUFhaSkZHBkiVLiIiIwMvLi/3797N3716Ki4tN00/WFHR37drFf//7XyZM\nmHDVkre33347PXr04ODBg0RHR3P+/Hng98+r6nZmgbk3SZgxYwYPPvggw4YNIy7u6vnkK1euMGnS\nJO67776rbv/mm2+4++67ue+++9ixY8d127gpAm/ZOYplS9w9+uijNGzYkKeeegrAVD27ceNGTpw4\nQevWrQHMso2Xm5sboaGhZiueqY6aN29O9+7diYqKYsyYMfTq1YuGDRvi7+9P+/btmTRpEr/++utN\nEXD/qH79+jz00ENmm0e9cuUK8Pv7eM+ePYwbN47HHnuMnJwc2rZtS3p6Ops3byYtLY3u3buzYMEC\nvLy8rCqQVJYvvviClStXkpiYiLe3Nw0aNKBevXq8+uqr3HLLLSxYsIB33nmHTz/91LThgbX9nbKy\nsnjttde47bbbuPXW/79ZwEcffWRadzkwMJCoqCh+/vnnarsjVEVPJSrPy7V//36SkpKIjIwkIiKC\niIiIq+6fPXs2bdu2veq28+fP8/777/PFF1+wcOFCtm7det12bL64as+ePSxZsoQJEyZc9Wbr3bs3\nu3fvZtmyZZw7d47Dhw+zevVqZs2aZfFtrmydvb099erV4+jRoxQVFZGdnc25c+d45ZVX+Ne//oW9\nvT2+vr42uzrVtZhrj9acnBzWrl1LQEAAcXFxrFy5ksmTJ+Pj40NMTAxeXl54eXkRFxdHdnY2bdu2\ntfiOVNVB2TBx+/btOXbsGIcPH8bZ2ZmYmBjq1q1Ls2bNuOuuuygsLMTNzY3evXubijGtUUlJCb/8\n8gs1a9akefPmLFiwgISEBMaPH4+9vT0NGjSgVq1atG7dulrszf1X0n+Kr/CxnkHXXup09erVdOzY\nEX9/f9zd3VmyZAn33HOPacrxtttuw9vbm++++44HH3wQ+P30QDs7O9MuTeVZTtimU4wTJ04wbdo0\nJkyYcFUh1apVqwgICODNN9/kq6++Mg1xzp0796bKRC1t0KBBrFy5ktWrVzNu3DjTAgmjRo2q4p7Z\nHjs7O/r27cuZM2eYNm0aJSUlNG3alKZNm+Lg4MDnn3/OxIkTcXd3x93dvdJqGazZU089xdy5c9my\nZQs5OTm88sor9OjRg/z8fDp37sy///1vq8ty/8jZ2RkXFxcCAwP5+uuvWbVqFa6ursybN48aNWqw\nZs0aduzYwZw5c6r3dIMZX4PMzEz8/f1N193d3cnIyDAV2Lq4uHDhwoWrjjl79iwFBQWMHTuWixcv\nMn78eLp3737Ndmw68F6+fJmgoCDy8vJIS0ujYcOGpq3OBg8eDMA999xTrQoHbJmvry8TJ04kLy8P\nNzc3qytKsSaurq44OzvzySef4OXlxb59+3jjjTeYPHkyoaGh7N69m61bt/LEE09UdVerVNlymR98\n8AFdu3Zl6NChfPfddzg5OTFq1CgGDBhAVFQU/v7+Vv9e3bp1K9u2bSMiIgI3Nzfmz5/PhAkTqFGj\nBhs3buTbb7/l5Zdfrt5BFzDYWe51KO9w+4ULF1iwYAEpKSmMHDmS7du3X/P9YpOB9/Tp0zg7O9Ou\nXTuGDx/OunXrKC4u5uTJk+Tl5fHuu+/i4ODAunXrOHLkCC+//PINTb5LxTk4OJjWHtbf27zs7e1N\nw2Q1a9Zk586dpKWl8cgjj3DixAnGjh1b1V2scmfPnmX27Nm89tprODo60qpVK8aNG8ecOXNIS0vj\n/PnzDBkypKq7+Y+U7R18/vx5Ro4cSaNGjUy7DkVFRbF7927S0tJMa07fzDw9PcnMzDRdT09Pv+6+\n3PXr16dTp07UqFGDpk2bUrt2bbKzs6+5rrnNFVdFR0fzzDPPsGjRImbMmIGfnx8hISEkJCRw6NAh\nnnrqKRwcHPj2229Zt24dw4YNw87OTkFAbFL9+vUZOHAg7du3JygoiN27d/Puu+/y4osv0r17d9M0\ny82qbKOQjh070qZNGwAOHDjAHXfcYZGNSizBzs6O3NxcNm7cSNOmTUlJSSEpKYmMjAyCgoI4fPgw\nL774otUEXXMWVwUHB7Np0ybg9zXMPT09r7uOQ8+ePU17tp8/f578/PzrbmxiU8VVu3btYvny5Tz/\n/PP06tWLuLg4unfvTpMmTWjdujUZGRkYDAZ2797NDz/8wCuvvHLTnNIjN69atWrRtGlTUlNTqV27\nNk5OTsTGxnLnnXearbiruiqb3jh16hQnT56kXr16REZGkpGRwW233QZAZGQkDg4OPPLII5W2G1NV\nO3HiBNu2bcPT05N58+bh4ODA5cuXeeCBB0wLZ1iLjENHKxx5PQKvvae3l5cXJ0+e5L333iMqKoqp\nU6eya9cuzp49i6+vL//3f/9nWlN+9+7dODg4EBgYSEFBAdOmTWP9+vU8//zz140rBmN1rRm/QVlZ\nWQwdOpT77ruPcePGkZqaytChQ+nfvz/79u3j/fffp7CwkEWLFvHzzz8zd+5cBV25qWRnZ1NQUICH\nhwfTpk1j/Pjx1XLFKHPbuXMnH374IbVr16Z169a0atWKTz/9lNDQUHx8fFi/fj3PPPMMnTt3ruqu\nVprs7GxWrFhBq1at8PLyIiCgeiydWBHHFkdW+Nh2jz1YiT2pOJsJvPn5+SxfvpzTp08TGBjIrl27\n6N69O8OHD+fTTz9l6dKlrF+/nqSkJBo0aHBTfuCIlCkpKbkpiwpzcnJ49dVXmThxIk2aNGHp0qUU\nFRXRuXNnfvnlF86fP2/aAEGqp4QlFQ+8bUdXj8BrM8VVZaXyHTp0YPHixXTt2pXhw4cDMHLkSE6d\nOsWFCxeuKhUXuVndjEE3MTERNzc3cnNzSUxMpEmTJowcOZLJkydz5coVxo8fX9VdlHKwZFWzudjM\nBE9ZqXxYWBgTJkzg0qVLphVENm3aRHx8vM5VFLlJHT9+nAkTJpj2Hd61axf79+/H3t6eIUOGcPHi\nRdNKX1K9mXvJSEuw+oz3z6Xynp6e3H777QBs3ryZqKgoUlNTmTNnDp6enlXbWRGxuMTEROLj43Fw\ncKCoqIhevXqRk5PDO++8Q7du3di+fTvPPfdctT9/VWyH1We8f1Uqn5ycTGZmJt27d+fgwYO88MIL\nVlMqLyKV5+LFi0yaNImCggK6du3KRx99RHFxMcOHD+ell17Cx8eHadOm0atXr6ruqpSX4R9cqgmr\nz3jh9wUz8vPzTWvS9uvXj4KCAoYNG0bv3r1NSxOKyM2lTp063HPPPTRr1oygoCA2btzI8uXLGTp0\nKAEBAVZd3SvWy+ozXgAfHx969epFzZo1efnll3n00Ud58skncXR0VNAVuQklJiaSlZVFUVERt9xy\nC6tWraJly5bcddddXLlyhRUrVnDlypVquwOP/D1bmOO1mdOJROTm9cd1v0+ePMlrr71mml6aMGEC\nq1evpmbNmowcOZKDBw/i7u5u2g5UrMsvy7+q8LG3PFw9lv+0iYxXRG5e2dnZLFu2jJycHEpKSli1\nahURERG88MIL+Pn5ER4ezsWLFzl48CAAgYGBCrrWzO4fXKqJatQVEZEbl5iYyG+//UZkZCSlpaU4\nODhw/vx56tSpw/Dhw3n66acJCgoiPj6ezz77rKq7K/+QLQw1K/CKiFULCgqic+fOnD9/nqVLl3Lh\nwgVOnTrF6dOnAfDz8yM0NJQPPviA/Pz8Ku6tiI1UNYvIzWv//v2sWLGCXr16kZubS1xcHLGxsRw8\neJDExETq16/PvHnzOHr0KAcOHKC4uBh7e/tqlQHJzUWBV0SsVmxsLHPmzGH69Ol4eHhw+PBh8vPz\nKS4uZtq0aaSmplJUVISjoyOurq6Eh4dTo4Y+9qyZLXxh0jtQRKxWYWEhgYGBJCQksGvXLqKjoykq\nKiI9PZ0PPviAxx9/3BRow8LCqri3UimsP+4q8IqI9WrZsiV169Zl1apVPPnkkwwcOJBDhw6RnJzM\n3XffrezWBtnCJgk6j1dEbEZ0dDSLFi3iqaeeokuXLlXdHTGDU6vWVfjYlkMHV2JPKk5fB0XE6uXm\n5rJhwwY2bNjA448/rqAr1ZoyXhGxCUVFReTm5uLu7l7VXREzUsYrIlJNODg4KOjeBGygqFmBV0RE\nrIdOJxIREbEkG6hqVuAVERGrYQsZr9ZqFhERsSBlvCKVbPbs2cTHx3PlyhWOHTtGp06dABgyZAj3\n3HPP/zx+1apVHDhwgDfffNPSXRWxPtaf8CrwilS2iRMnAnD27Fkeeughli9fXsU9EpHqRIFXxELy\n8vKYMmUKaWlpFBcXc9999/Hggw9e9Zhdu3axYMEC/vvf/3LmzBlmzZpFSUkJxcXFTJ06lTZt2vDv\nf/+bkJAQDh48yK+//sqECRMYOHBgFT0rEcuyhTleBV4RC1m2bBnu7u7MmzePy5cvM2DAAHr27Gm6\n/9ixY7zzzjssXrwYFxcXXnjhBRYtWoSPjw9Hjhxh8uTJrFq1CoCCggI+/vhjoqOjmTNnjgKv3DRs\nYa1mBV4RC4mLi2PYsGEA1KpVi3bt2pGQkADAuXPneOKJJ1iyZAnu7u6kpaWRlJTEyy+/bDr+4sWL\npp+7du0KgLe3NxcuXLDgsxCpYsp4RaS8/jxE9sfVWn/99VdCQkJYunQpM2fOxNHRkZo1a/7t/LC9\nvb1Z+ypSXdnCULNOJxKxkI4dO/Ljjz8Cv8/3JiQk4O/vD0D37t15/fXXOX36NN9++y1ubm54eHiY\nHp+YmMiHH35YZX0XkcqjjFfEQkaOHMmUKVMYPnw4hYWFPPPMM3h5eZnut7e356233uLhhx8mICCA\nOXPmEBERwYcffkhJSclVw84iNy3rT3i1O5GIiFiPs999X+FjfQb0r8SeVJwyXhERsRqqahYREbEk\nGyiuUuAVERGroapmERERuSHKeEVExHpojldERMRyNNQsIiIiN0QZr4iIWA/rT3gVeEVExHpoqFlE\nRERuiDJeERGxHqpqFhERsRxzDzXPmDGD2NhYDAYD4eHhBAQEmO7bs2cP8+bNw97enpCQEJ5++mku\nXbrEpEmTyMnJoaioiKeffppevXpdsw0FXhERsR5mDLz79+8nKSmJyMhIEhMTCQ8PJzIy0nT/9OnT\nWbJkCQ0bNmTEiBHceeed7N27lxYtWvD888+TlpbGqFGj+P77a2/koDleERERIDo6mr59+wLg6+tL\nTk4OeXl5AJw5c4a6devi5eWFnZ0doaGhREdH4+bmxoULFwC4ePEibm5u121HgVdERKyGwWCo8OV6\nMjMzrwqc7u7uZGRkAJCRkYG7u/v/3Ddw4EBSUlIICwtjxIgRTJo06brtKPCKiIj8hfJsV79u3Tq8\nvb3ZvHkzy5Yt4/XXX7/uMZrjFRER62HGqmZPT08yMzNN19PT0/Hw8PjL+9LS0vD09OTgwYP07NkT\ngDZt2pCenk5JSQn29vZ/244yXhERsRrmHGoODg5m06ZNABw9ehRPT09cXFwA8PHxIS8vj7Nnz1Jc\nXMz27dsJDg6mWbNmxMbGApCcnEzt2rWvGXQBDMby5NIiIiLVQMa+3RU+1qNr8HUf89Zbb/HTTz9h\nMBiYOnUqx44dw9XVlbCwMGJiYnjrrbcA6NevH6NHj+bSpUuEh4eTlZVFcXExzzzzDN27d79mGwq8\nIiJiNTJj9lT42AZBPSqxJxWnoWYRERELUuAVERGxIFU1i4iI9bCB3YkUeEVExGrYwraACrwiImI9\nFHhFREQsx2AD2wKquEpERMSCFHhFREQsSEPNIiJiPTTHKyIiYkEKvCIiIpaj04lEREQsSVXNIiIi\nciOU8YqIiNUwGKw/X7T+ZyAiImJFlPGKiIj1UHGViIiI5aiqWURExJJU1SwiIiI3QhmviIhYDQ01\ni4iIWJINBF4NNYuIiFiQMl4REbEeNrCAhgKviIhYDYOqmkVERORGKOMVERHrYQPFVQq8IiJiNXQ6\nkYiIiCXZQHGV9T8DERERK6KMV0RErIaqmkVEROSGKOMVERHroeIqERERy1FVs4iIiCXZQFWzAq+I\niFgPFVeJiIjIjVDgFRERsSANNYuIiNVQcZWIiIglqbhKRETEcpTxioiIWJINZLzW/wxERESsiAKv\niIiIBWmoWURErIa5dyeaMWMGsbGxGAwGwsPDCQgIMN23Z88e5s2bh729PSEhITz99NPXPeavKPCK\niIj1MGNx1f79+0lKSiIyMpLExETCw8OJjIw03T99+nSWLFlCw4YNGTFiBHfeeSfZ2dnXPOavKPCK\niIjVMJixuCo6Opq+ffsC4OvrS05ODnl5ebi4uHDmzBnq1q2Ll5cXAKGhoURHR5Odnf23x/wdzfGK\niIj1MBgqfrmOzMxM3NzcTNfd3d3JyMgAICMjA3d39/+571rH/B1lvCIiYjUc69S3WFtGo9Esxyjw\nioiIAJ6enmRmZpqup6en4+Hh8Zf3paWl4enpiYODw98e83c01CwiIgIEBwezadMmAI4ePYqnp6dp\nrtbHx4e8vDzOnj1LcXEx27dvJzg4+JrH/B2DsSK5tIiIiA166623+OmnnzAYDEydOpVjx47h6upK\nWFgYMTExvPXWWwD069eP0aNH/+Uxbdq0uWYbCrwiIiIWpKFmERERC1LgFRERsSAFXhEREQtS4BUR\nEbEgBV4RERELUuAVERGxIAVeERERC1LgFRERsaD/B/GC4UPVjd8WAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4f13ddef28>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "attn_matrix = attn_scores.detach().numpy()\n",
    "ax = sns.heatmap(attn_matrix, linewidths=2, square=True)\n",
    "tokens = [\"<BEGIN>\"]+preprocess_text(title).split(\" \")+[\"<END>\"]\n",
    "ax.set_xticklabels(tokens, rotation=45)\n",
    "ax.set_xlabel(\"Token\")\n",
    "ax.set_ylabel(\"Importance\\n\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6CWwxX2aTBla"
   },
   "source": [
    "# Layer normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xPzmk6T3TD_9"
   },
   "source": [
    "Recall from our [CNN notebook](https://colab.research.google.com/github/GokuMohandas/practicalAI/blob/master/notebooks/11_Convolutional_Neural_Networks.ipynb) that we used batch normalization to deal with internal covariant shift. Our activations will experience the same issues with RNNs but we will use a technique known as [layer normalization](https://arxiv.org/abs/1607.06450) (layernorm) to maintain zero mean unit variance on the activations. \n",
    "\n",
    "With layernorm it's a bit different from batchnorm. We compute the mean and var for every single sample (instead of each hidden dim) for each layer independently and then do the operations on the activations before they go through the nonlinearities. PyTorch's [LayerNorm](https://pytorch.org/docs/stable/nn.html#torch.nn.LayerNorm) class abstracts all of this for us when we feed in inputs to the layer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tZTdk6OyTGq5"
   },
   "source": [
    "$ LN = \\frac{a - \\mu_{L}}{\\sqrt{\\sigma^2_{L} + \\epsilon}}  * \\gamma + \\beta $\n",
    "\n",
    "where:\n",
    "* $a$ = activation | $\\in \\mathbb{R}^{NXH}$ ($N$ is the number of samples, $H$ is the hidden dim)\n",
    "* $ \\mu_{L}$ = mean of input| $\\in \\mathbb{R}^{NX1}$\n",
    "* $\\sigma^2_{L}$ = variance of input | $\\in \\mathbb{R}^{NX1}$\n",
    "* $epsilon$ = noise\n",
    "* $\\gamma$ = scale parameter (learned parameter)\n",
    "* $\\beta$ = shift parameter (learned parameter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RAZGISTXTGnV"
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/GokuMohandas/practicalAI/master/images/layernorm.png\" width=400>\n",
    "\n",
    "The most useful location to apply layernorm will be inside the RNN on the activations before the non-linearities. However, this is a bit involved and though PyTorch has a [LayerNorm](https://pytorch.org/docs/stable/nn.html#torch.nn.LayerNorm) class, they do not have an RNN that has built in layernorm yet. You could implement the RNN yourself and manually add layernorm by following a similar setup like below.\n",
    "\n",
    "```python\n",
    "# Layernorm\n",
    "for t in range(seq_size):\n",
    "    # Normalize over hidden dim\n",
    "    layernorm = nn.LayerNorm(args.hidden_dim)\n",
    "    # Activating the module\n",
    "    a = layernorm(x)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1YHneO3SStOp"
   },
   "source": [
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gGHaKTe1SuEk"
   },
   "source": [
    "- attn visualization isn't always great\n",
    "- bleu score\n",
    "- ngram-overlap\n",
    "- perplexity\n",
    "- beamsearch\n",
    "- hierarchical softmax\n",
    "- hierarchical attention\n",
    "- Transformer networks\n",
    "- attention interpretability is hit/miss\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "14_Advanced_RNNs",
   "provenance": [],
   "toc_visible": true,
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}