AlexNet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AlexNet CIFAR-10 Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook implements the classic AlexNet convolutional network [1] and applies it to the CIFAR10 object classification dataset. The basic architecture is shown in the figure below:\n",
    "\n",
    "![](./images/alextnet/alextnet-1.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It contains 5 convolutional layers and 3 fully connected layers. Relu is applied after very convolutional and fully connected layer. Dropout is applied before the first and the second fully connected year. The image size in the following architecutre chart should be 227 * 227 instead of 224 * 224, as it is pointed out by Andrei Karpathy in his famous CS231n Course. More insterestingly, the input size is 224 * 224 with 2 padding in the pytorch torch vision. The output width and height should be (224–11+4)/4 + 1=55.25! The explanation here is pytorch Conv2d apply floor operator to the above result, and therefore the last one padding is ignored.\n",
    "\n",
    "Some of the highlights in AlexNet Architecture:\n",
    "\n",
    "- It uses `ReLU` activation function instead Sigmoid or Tanh functions. It speed by more than 5 times faster with same accuracy\n",
    "- It uses `Dropout` instead of regularisation to deal with overfitting. But the training time is doubled with dropout ratio of 0.5\n",
    "- More data and bigger model with 7 hidden layers, 650K units and 60M parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Size / Operation</th>\n",
       "      <th>Filter</th>\n",
       "      <th>Depth</th>\n",
       "      <th>Stride</th>\n",
       "      <th>Padding</th>\n",
       "      <th>Number of Parameters</th>\n",
       "      <th>Forward Computation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3* 227 * 227</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Conv1 + Relu</td>\n",
       "      <td>11 * 11</td>\n",
       "      <td>96</td>\n",
       "      <td>4</td>\n",
       "      <td></td>\n",
       "      <td>(11*11*3 + 1) * 96=34944</td>\n",
       "      <td>(11*11*3 + 1) * 96 * 55 * 55=105705600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>96 * 55 * 55</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Max Pooling</td>\n",
       "      <td>3 * 3</td>\n",
       "      <td></td>\n",
       "      <td>2</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>96 * 27 * 27</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Norm</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Conv2 + Relu</td>\n",
       "      <td>5 * 5</td>\n",
       "      <td>256</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>(5 * 5 * 96 + 1) * 256=614656</td>\n",
       "      <td>(5 * 5 * 96 + 1) * 256 * 27 * 27=448084224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>256 * 27 * 27</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Max Pooling</td>\n",
       "      <td>3 * 3</td>\n",
       "      <td></td>\n",
       "      <td>2</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>256 * 13 * 13</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Norm</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Conv3 + Relu</td>\n",
       "      <td>3 * 3</td>\n",
       "      <td>384</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>(3 * 3 * 256 + 1) * 384=885120</td>\n",
       "      <td>(3 * 3 * 256 + 1) * 384 * 13 * 13=149585280</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>384 * 13 * 13</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Conv4 + Relu</td>\n",
       "      <td>3 * 3</td>\n",
       "      <td>384</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>(3 * 3 * 384 + 1) * 384=1327488</td>\n",
       "      <td>(3 * 3 * 384 + 1) * 384 * 13 * 13=224345472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>384 * 13 * 13</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Conv5 + Relu</td>\n",
       "      <td>3 * 3</td>\n",
       "      <td>256</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>(3 * 3 * 384 + 1) * 256=884992</td>\n",
       "      <td>(3 * 3 * 384 + 1) * 256 * 13 * 13=149563648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>256 * 13 * 13</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Max Pooling</td>\n",
       "      <td>3 * 3</td>\n",
       "      <td></td>\n",
       "      <td>2</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>256 * 6 * 6</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Dropout (rate 0.5)</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>FC6 + Relu</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>256 * 6 * 6 * 4096=37748736</td>\n",
       "      <td>256 * 6 * 6 * 4096=37748736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>4096</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Dropout (rate 0.5)</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>FC7 + Relu</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>4096 * 4096=16777216</td>\n",
       "      <td>4096 * 4096=16777216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>4096</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>FC8 + Relu</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>4096 * 1000=4096000</td>\n",
       "      <td>4096 * 1000=4096000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>1000 classes</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Overall</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>62369152=62.3 million</td>\n",
       "      <td>1135906176=1.1 billion</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Conv VS FC</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>Conv:3.7million (6%) , FC: 58.6 million  (94% )</td>\n",
       "      <td>Conv: 1.08 billion (95%) , FC: 58.6 million (5%)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Size / Operation   Filter Depth Stride Padding  \\\n",
       "0         3* 227 * 227                                 \n",
       "1         Conv1 + Relu  11 * 11    96      4           \n",
       "2         96 * 55 * 55                                 \n",
       "3          Max Pooling    3 * 3            2           \n",
       "4         96 * 27 * 27                                 \n",
       "5                 Norm                                 \n",
       "6         Conv2 + Relu    5 * 5   256      1       2   \n",
       "7        256 * 27 * 27                                 \n",
       "8          Max Pooling    3 * 3            2           \n",
       "9        256 * 13 * 13                                 \n",
       "10                Norm                                 \n",
       "11        Conv3 + Relu    3 * 3   384      1       1   \n",
       "12       384 * 13 * 13                                 \n",
       "13        Conv4 + Relu    3 * 3   384      1       1   \n",
       "14       384 * 13 * 13                                 \n",
       "15        Conv5 + Relu    3 * 3   256      1       1   \n",
       "16       256 * 13 * 13                                 \n",
       "17         Max Pooling    3 * 3            2           \n",
       "18         256 * 6 * 6                                 \n",
       "19  Dropout (rate 0.5)                                 \n",
       "20          FC6 + Relu                                 \n",
       "21                4096                                 \n",
       "22  Dropout (rate 0.5)                                 \n",
       "23          FC7 + Relu                                 \n",
       "24                4096                                 \n",
       "25          FC8 + Relu                                 \n",
       "26        1000 classes                                 \n",
       "27             Overall                                 \n",
       "28          Conv VS FC                                 \n",
       "\n",
       "                               Number of Parameters  \\\n",
       "0                                                     \n",
       "1                          (11*11*3 + 1) * 96=34944   \n",
       "2                                                     \n",
       "3                                                     \n",
       "4                                                     \n",
       "5                                                     \n",
       "6                     (5 * 5 * 96 + 1) * 256=614656   \n",
       "7                                                     \n",
       "8                                                     \n",
       "9                                                     \n",
       "10                                                    \n",
       "11                   (3 * 3 * 256 + 1) * 384=885120   \n",
       "12                                                    \n",
       "13                  (3 * 3 * 384 + 1) * 384=1327488   \n",
       "14                                                    \n",
       "15                   (3 * 3 * 384 + 1) * 256=884992   \n",
       "16                                                    \n",
       "17                                                    \n",
       "18                                                    \n",
       "19                                                    \n",
       "20                      256 * 6 * 6 * 4096=37748736   \n",
       "21                                                    \n",
       "22                                                    \n",
       "23                             4096 * 4096=16777216   \n",
       "24                                                    \n",
       "25                              4096 * 1000=4096000   \n",
       "26                                                    \n",
       "27                           62369152=62.3 million    \n",
       "28  Conv:3.7million (6%) , FC: 58.6 million  (94% )   \n",
       "\n",
       "                                 Forward Computation  \n",
       "0                                                     \n",
       "1             (11*11*3 + 1) * 96 * 55 * 55=105705600  \n",
       "2                                                     \n",
       "3                                                     \n",
       "4                                                     \n",
       "5                                                     \n",
       "6         (5 * 5 * 96 + 1) * 256 * 27 * 27=448084224  \n",
       "7                                                     \n",
       "8                                                     \n",
       "9                                                     \n",
       "10                                                    \n",
       "11       (3 * 3 * 256 + 1) * 384 * 13 * 13=149585280  \n",
       "12                                                    \n",
       "13       (3 * 3 * 384 + 1) * 384 * 13 * 13=224345472  \n",
       "14                                                    \n",
       "15       (3 * 3 * 384 + 1) * 256 * 13 * 13=149563648  \n",
       "16                                                    \n",
       "17                                                    \n",
       "18                                                    \n",
       "19                                                    \n",
       "20                       256 * 6 * 6 * 4096=37748736  \n",
       "21                                                    \n",
       "22                                                    \n",
       "23                              4096 * 4096=16777216  \n",
       "24                                                    \n",
       "25                               4096 * 1000=4096000  \n",
       "26                                                    \n",
       "27                            1135906176=1.1 billion  \n",
       "28  Conv: 1.08 billion (95%) , FC: 58.6 million (5%)  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.read_csv(\"alextnet_params.csv\",na_filter =\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "References\n",
    "    \n",
    "- [1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. \"[Imagenet classification with deep convolutional neural networks.](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf)\" In Advances in Neural Information Processing Systems, pp. 1097-1105. 2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['',\n",
       " '/anaconda/envs/py36/lib/python36.zip',\n",
       " '/anaconda/envs/py36/lib/python3.6',\n",
       " '/anaconda/envs/py36/lib/python3.6/lib-dynload',\n",
       " '/anaconda/envs/py36/lib/python3.6/site-packages',\n",
       " '/anaconda/envs/py36/lib/python3.6/site-packages/IPython/extensions',\n",
       " '/data/home/zhangruochi/.ipython',\n",
       " '/Users/ZRC']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.append(\"/Users/ZRC\")\n",
    "sys.path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data import RandomSampler\n",
    "\n",
    "\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "\n",
    "from torchsummary import summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.utils import mem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPUMemory(total=11441, free=9705, used=1735)\n",
      "GPUMemory(total=11441, free=9705, used=1735)\n"
     ]
    }
   ],
   "source": [
    "print(mem.gpu_mem_get())\n",
    "torch.cuda.empty_cache()\n",
    "print(mem.gpu_mem_get())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from coke.visualization.image import show_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperparameters\n",
    "\n",
    "BATCH_SIZE = 64\n",
    "NUM_EPOCHS = 10\n",
    "LEARNING_RATE = 0.0001\n",
    "RANDOM_SEED = 7\n",
    "\n",
    "# Architecture\n",
    "NUM_CLASSES = 10\n",
    "GRAYSCALE = False\n",
    "\n",
    "# other\n",
    "torch.cuda.empty_cache()\n",
    "DEVICE = torch.device(\"cuda: 0\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_transforms = {\"train\": transforms.Compose([\n",
    "                            transforms.Resize((224,224)),\n",
    "                            transforms.ToTensor()]),\n",
    "                    \"val\": transforms.Compose([\n",
    "                      transforms.Resize((224,224)),\n",
    "                      transforms.ToTensor()])\n",
    "                   }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "train_dataset = datasets.CIFAR10(root = \"data\",\n",
    "                                train = True,\n",
    "                                transform = data_transforms[\"train\"],\n",
    "                                download=True)\n",
    "\n",
    "test_dataset = datasets.CIFAR10(root = \"data\",\n",
    "                                train = False,\n",
    "                                transform = data_transforms[\"val\"],\n",
    "                                download=False)\n",
    "\n",
    "train_dataloader = DataLoader(dataset = train_dataset,\n",
    "                             batch_size=BATCH_SIZE,\n",
    "                             shuffle=True,\n",
    "                             num_workers=4)\n",
    "\n",
    "test_dataloader = DataLoader(dataset = test_dataset,\n",
    "                             batch_size=BATCH_SIZE,\n",
    "                             shuffle=False,\n",
    "                             num_workers=4)\n",
    "\n",
    "data_loader = {\"train\": train_dataloader, \"val\": test_dataloader}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 3, 224, 224]), torch.Size([64]))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_samples,labels = next(iter(train_dataloader))\n",
    "batch_samples.shape,labels.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "show_batch(batch_samples.permute(0,2,3,1), labels.numpy(), (4,4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AlextNetZrc(torch.nn.Module):\n",
    "\n",
    "    def __init__(self, num_classes, grascale = False):\n",
    "        \n",
    "        super(AlextNetZrc, self).__init__()\n",
    "        \n",
    "        if grascale:\n",
    "            in_channels = 1\n",
    "        else:\n",
    "            in_channels = 3\n",
    "            \n",
    "        self.block_1 = torch.nn.Sequential(\n",
    "    \n",
    "            torch.nn.Conv2d(in_channels = in_channels,\n",
    "                            out_channels = 96,\n",
    "                            kernel_size = 11,\n",
    "                            stride = 4,\n",
    "                            padding = 4),\n",
    "            torch.nn.ReLU(inplace = True),\n",
    "            torch.nn.MaxPool2d(kernel_size = 3, stride = 2),\n",
    "            torch.nn.BatchNorm2d(96),\n",
    "            \n",
    "            torch.nn.Conv2d(in_channels = 96,\n",
    "                            out_channels = 256,\n",
    "                            kernel_size = 5,\n",
    "                            padding = 2),\n",
    "            torch.nn.ReLU(inplace = True),\n",
    "            torch.nn.MaxPool2d(kernel_size = 3, stride = 2),\n",
    "            torch.nn.BatchNorm2d(256),\n",
    "        \n",
    "        )\n",
    "        \n",
    "        \n",
    "        self.block_2 = torch.nn.Sequential(*[self.__block_2_helper(in_channels,out_channels) for in_channels,out_channels in [(256,384),(384, 384),(384,256)]])\n",
    "        \n",
    "        self.max_pool = torch.nn.MaxPool2d(kernel_size = 3, stride = 2)\n",
    "        \n",
    "        self.classifier = torch.nn.Sequential(\n",
    "            torch.nn.Dropout(0.5),\n",
    "            torch.nn.Linear(256*6*6,4096),\n",
    "            torch.nn.ReLU(inplace = True),\n",
    "            \n",
    "            torch.nn.Dropout(0.5),\n",
    "            torch.nn.Linear(4096,4096),\n",
    "            torch.nn.ReLU(inplace = True),\n",
    "            \n",
    "            torch.nn.Linear(4096,num_classes)\n",
    "        )\n",
    "        \n",
    "    def forward(self,x):\n",
    "        x = self.block_1(x)\n",
    "#         print(x.size())\n",
    "        x = self.block_2(x)\n",
    "#         print(x.size())\n",
    "        x = self.max_pool(x)\n",
    "        \n",
    "        x = torch.flatten(x,start_dim=1, end_dim=-1)\n",
    "        logits = self.classifier(x)\n",
    "        probas = F.softmax(logits, dim=1)\n",
    "        return logits, probas\n",
    "        \n",
    "        \n",
    "        \n",
    "    def __block_2_helper(self, in_channels, out_channels):\n",
    "        return torch.nn.Sequential(\n",
    "            torch.nn.Conv2d(in_channels,out_channels,kernel_size = 3,padding = 1),\n",
    "            torch.nn.ReLU(inplace = True)\n",
    "        )\n",
    "\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_weights(layer):\n",
    "    if isinstance(layer, torch.nn.Linear):\n",
    "        torch.nn.init.xavier_uniform_(layer.weight)\n",
    "#         torch.nn.init.normal_(layer.weight, 0.0, 0.1)\n",
    "        if layer.bias is not None:\n",
    "            torch.nn.init.constant_(layer.bias.data, 0)\n",
    "    elif isinstance(layer, torch.nn.Conv2d):\n",
    "        torch.nn.init.kaiming_normal_(layer.weight)\n",
    "        if layer.bias is not None:\n",
    "            torch.nn.init.constant_(layer.bias.data, 0) \n",
    "            \n",
    "model = AlextNetZrc(num_classes = NUM_CLASSES)\n",
    "model.apply(init_weights)\n",
    "model = model.to(DEVICE)\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr = LEARNING_RATE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "            Conv2d-1           [-1, 96, 56, 56]          34,944\n",
      "              ReLU-2           [-1, 96, 56, 56]               0\n",
      "         MaxPool2d-3           [-1, 96, 27, 27]               0\n",
      "       BatchNorm2d-4           [-1, 96, 27, 27]             192\n",
      "            Conv2d-5          [-1, 256, 27, 27]         614,656\n",
      "              ReLU-6          [-1, 256, 27, 27]               0\n",
      "         MaxPool2d-7          [-1, 256, 13, 13]               0\n",
      "       BatchNorm2d-8          [-1, 256, 13, 13]             512\n",
      "            Conv2d-9          [-1, 384, 13, 13]         885,120\n",
      "             ReLU-10          [-1, 384, 13, 13]               0\n",
      "           Conv2d-11          [-1, 384, 13, 13]       1,327,488\n",
      "             ReLU-12          [-1, 384, 13, 13]               0\n",
      "           Conv2d-13          [-1, 256, 13, 13]         884,992\n",
      "             ReLU-14          [-1, 256, 13, 13]               0\n",
      "        MaxPool2d-15            [-1, 256, 6, 6]               0\n",
      "          Dropout-16                 [-1, 9216]               0\n",
      "           Linear-17                 [-1, 4096]      37,752,832\n",
      "             ReLU-18                 [-1, 4096]               0\n",
      "          Dropout-19                 [-1, 4096]               0\n",
      "           Linear-20                 [-1, 4096]      16,781,312\n",
      "             ReLU-21                 [-1, 4096]               0\n",
      "           Linear-22                   [-1, 10]          40,970\n",
      "================================================================\n",
      "Total params: 58,323,018\n",
      "Trainable params: 58,323,018\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.57\n",
      "Forward/backward pass size (MB): 12.11\n",
      "Params size (MB): 222.48\n",
      "Estimated Total Size (MB): 235.17\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "summary(model, (3,224,224))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_accuracy(model, data_loader, device):\n",
    "    model.eval()\n",
    "    correct_pred, num_examples = 0, 0\n",
    "    for i, (features, targets) in enumerate(data_loader):\n",
    "            \n",
    "        features = features.to(device)\n",
    "        targets = targets.to(device)\n",
    "\n",
    "        logits, probas = model(features)\n",
    "        _, predicted_labels = torch.max(probas, 1)\n",
    "        num_examples += targets.size(0)\n",
    "        correct_pred += (predicted_labels == targets).sum()\n",
    "    return correct_pred.float()/num_examples * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, data_loader, optimizer, num_epochs,batch_size, device,metric_func, random_seed = 7):\n",
    "    # Manual seed for deterministic data loader\n",
    "    torch.manual_seed(random_seed)\n",
    "    \n",
    "    loss_list = []\n",
    "    train_acc_list, valid_acc_list = [], []\n",
    "    \n",
    "    for epoch in range(num_epochs):\n",
    "        # set training mode\n",
    "        model.train() \n",
    "        for batch_idx, (features, targets) in enumerate(data_loader[\"train\"]):\n",
    "            features = features.to(device)\n",
    "            targets = targets.to(device)\n",
    "\n",
    "\n",
    "            ## forward pass\n",
    "            logits, probas = model(features)\n",
    "            loss = F.cross_entropy(logits,targets)\n",
    "\n",
    "            # backward pass\n",
    "            # clear the gradients of all tensors being optimized\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "            ### Login\n",
    "            loss_list.append(loss.item())\n",
    "            if not batch_idx % 50:\n",
    "                print ('Epoch: {0:03d}/{1:03d} | Batch {2:03d}/{3:03d} | Loss: {4:.2f}'.format(\n",
    "                    epoch+1, num_epochs, batch_idx, \n",
    "                         len(train_dataset)//batch_size, loss))\n",
    "\n",
    "        with torch.set_grad_enabled(False):\n",
    "            train_acc = metric_func(model, data_loader[\"train\"], device)\n",
    "            valid_acc = metric_func(model, data_loader[\"val\"], device)\n",
    "            \n",
    "            print('Epoch: {0:03d}/{1:03d} training accuracy: {2:.2f}'.format(\n",
    "                  epoch+1, num_epochs, train_acc))\n",
    "            \n",
    "            print('Epoch: {0:03d}/{1:03d} validation accuracy: {2:.2f}'.format(\n",
    "                  epoch+1, num_epochs, valid_acc))\n",
    "            \n",
    "            train_acc_list.append(train_acc)\n",
    "            valid_acc_list.append(valid_acc)\n",
    "            \n",
    "    return loss_list, train_acc_list, valid_acc_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 001/010 | Batch 000/781 | Loss: 0.88\n",
      "Epoch: 001/010 | Batch 050/781 | Loss: 1.09\n",
      "Epoch: 001/010 | Batch 100/781 | Loss: 1.20\n",
      "Epoch: 001/010 | Batch 150/781 | Loss: 0.98\n",
      "Epoch: 001/010 | Batch 200/781 | Loss: 0.80\n",
      "Epoch: 001/010 | Batch 250/781 | Loss: 1.08\n",
      "Epoch: 001/010 | Batch 300/781 | Loss: 0.64\n",
      "Epoch: 001/010 | Batch 350/781 | Loss: 0.85\n",
      "Epoch: 001/010 | Batch 400/781 | Loss: 0.73\n",
      "Epoch: 001/010 | Batch 450/781 | Loss: 0.89\n",
      "Epoch: 001/010 | Batch 500/781 | Loss: 0.65\n",
      "Epoch: 001/010 | Batch 550/781 | Loss: 0.46\n",
      "Epoch: 001/010 | Batch 600/781 | Loss: 0.67\n",
      "Epoch: 001/010 | Batch 650/781 | Loss: 0.46\n",
      "Epoch: 001/010 | Batch 700/781 | Loss: 0.49\n",
      "Epoch: 001/010 | Batch 750/781 | Loss: 0.56\n",
      "Epoch: 001/010 training accuracy: 59.63\n",
      "Epoch: 001/010 validation accuracy: 57.38\n",
      "Epoch: 002/010 | Batch 000/781 | Loss: 1.16\n",
      "Epoch: 002/010 | Batch 050/781 | Loss: 0.95\n",
      "Epoch: 002/010 | Batch 100/781 | Loss: 0.97\n",
      "Epoch: 002/010 | Batch 150/781 | Loss: 0.88\n",
      "Epoch: 002/010 | Batch 200/781 | Loss: 1.15\n",
      "Epoch: 002/010 | Batch 250/781 | Loss: 1.10\n",
      "Epoch: 002/010 | Batch 300/781 | Loss: 0.95\n",
      "Epoch: 002/010 | Batch 350/781 | Loss: 0.80\n",
      "Epoch: 002/010 | Batch 400/781 | Loss: 0.98\n",
      "Epoch: 002/010 | Batch 450/781 | Loss: 0.84\n",
      "Epoch: 002/010 | Batch 500/781 | Loss: 0.96\n",
      "Epoch: 002/010 | Batch 550/781 | Loss: 0.99\n",
      "Epoch: 002/010 | Batch 600/781 | Loss: 0.80\n",
      "Epoch: 002/010 | Batch 650/781 | Loss: 0.96\n",
      "Epoch: 002/010 | Batch 700/781 | Loss: 0.89\n",
      "Epoch: 002/010 | Batch 750/781 | Loss: 0.83\n",
      "Epoch: 002/010 training accuracy: 72.42\n",
      "Epoch: 002/010 validation accuracy: 68.39\n",
      "Epoch: 003/010 | Batch 000/781 | Loss: 0.76\n",
      "Epoch: 003/010 | Batch 050/781 | Loss: 0.95\n",
      "Epoch: 003/010 | Batch 100/781 | Loss: 0.77\n",
      "Epoch: 003/010 | Batch 150/781 | Loss: 0.95\n",
      "Epoch: 003/010 | Batch 200/781 | Loss: 0.81\n",
      "Epoch: 003/010 | Batch 250/781 | Loss: 1.01\n",
      "Epoch: 003/010 | Batch 300/781 | Loss: 0.54\n",
      "Epoch: 003/010 | Batch 350/781 | Loss: 0.75\n",
      "Epoch: 003/010 | Batch 400/781 | Loss: 0.71\n",
      "Epoch: 003/010 | Batch 450/781 | Loss: 0.65\n",
      "Epoch: 003/010 | Batch 500/781 | Loss: 0.65\n",
      "Epoch: 003/010 | Batch 550/781 | Loss: 0.51\n",
      "Epoch: 003/010 | Batch 600/781 | Loss: 0.87\n",
      "Epoch: 003/010 | Batch 650/781 | Loss: 0.64\n",
      "Epoch: 003/010 | Batch 700/781 | Loss: 0.69\n",
      "Epoch: 003/010 | Batch 750/781 | Loss: 0.75\n",
      "Epoch: 003/010 training accuracy: 77.58\n",
      "Epoch: 003/010 validation accuracy: 72.70\n",
      "Epoch: 004/010 | Batch 000/781 | Loss: 0.71\n",
      "Epoch: 004/010 | Batch 050/781 | Loss: 0.54\n",
      "Epoch: 004/010 | Batch 100/781 | Loss: 0.67\n",
      "Epoch: 004/010 | Batch 150/781 | Loss: 0.74\n",
      "Epoch: 004/010 | Batch 200/781 | Loss: 0.61\n",
      "Epoch: 004/010 | Batch 250/781 | Loss: 0.61\n",
      "Epoch: 004/010 | Batch 300/781 | Loss: 0.53\n",
      "Epoch: 004/010 | Batch 350/781 | Loss: 0.78\n",
      "Epoch: 004/010 | Batch 400/781 | Loss: 0.47\n",
      "Epoch: 004/010 | Batch 450/781 | Loss: 0.50\n",
      "Epoch: 004/010 | Batch 500/781 | Loss: 0.43\n",
      "Epoch: 004/010 | Batch 550/781 | Loss: 0.61\n",
      "Epoch: 004/010 | Batch 600/781 | Loss: 0.53\n",
      "Epoch: 004/010 | Batch 650/781 | Loss: 0.60\n",
      "Epoch: 004/010 | Batch 700/781 | Loss: 0.70\n",
      "Epoch: 004/010 | Batch 750/781 | Loss: 0.68\n",
      "Epoch: 004/010 training accuracy: 80.75\n",
      "Epoch: 004/010 validation accuracy: 73.94\n",
      "Epoch: 005/010 | Batch 000/781 | Loss: 0.69\n",
      "Epoch: 005/010 | Batch 050/781 | Loss: 0.62\n",
      "Epoch: 005/010 | Batch 100/781 | Loss: 0.68\n",
      "Epoch: 005/010 | Batch 150/781 | Loss: 0.56\n",
      "Epoch: 005/010 | Batch 200/781 | Loss: 0.67\n",
      "Epoch: 005/010 | Batch 250/781 | Loss: 0.64\n",
      "Epoch: 005/010 | Batch 300/781 | Loss: 0.55\n",
      "Epoch: 005/010 | Batch 350/781 | Loss: 0.51\n",
      "Epoch: 005/010 | Batch 400/781 | Loss: 0.67\n",
      "Epoch: 005/010 | Batch 450/781 | Loss: 0.41\n",
      "Epoch: 005/010 | Batch 500/781 | Loss: 0.69\n",
      "Epoch: 005/010 | Batch 550/781 | Loss: 0.60\n",
      "Epoch: 005/010 | Batch 600/781 | Loss: 0.52\n",
      "Epoch: 005/010 | Batch 650/781 | Loss: 0.61\n",
      "Epoch: 005/010 | Batch 700/781 | Loss: 0.62\n",
      "Epoch: 005/010 | Batch 750/781 | Loss: 0.57\n",
      "Epoch: 005/010 training accuracy: 83.53\n",
      "Epoch: 005/010 validation accuracy: 75.54\n",
      "Epoch: 006/010 | Batch 000/781 | Loss: 0.43\n",
      "Epoch: 006/010 | Batch 050/781 | Loss: 0.56\n",
      "Epoch: 006/010 | Batch 100/781 | Loss: 0.53\n",
      "Epoch: 006/010 | Batch 150/781 | Loss: 0.55\n",
      "Epoch: 006/010 | Batch 200/781 | Loss: 0.32\n",
      "Epoch: 006/010 | Batch 250/781 | Loss: 0.42\n",
      "Epoch: 006/010 | Batch 300/781 | Loss: 0.79\n",
      "Epoch: 006/010 | Batch 350/781 | Loss: 0.34\n",
      "Epoch: 006/010 | Batch 400/781 | Loss: 0.56\n",
      "Epoch: 006/010 | Batch 450/781 | Loss: 0.25\n",
      "Epoch: 006/010 | Batch 500/781 | Loss: 0.50\n",
      "Epoch: 006/010 | Batch 550/781 | Loss: 0.47\n",
      "Epoch: 006/010 | Batch 600/781 | Loss: 0.62\n",
      "Epoch: 006/010 | Batch 650/781 | Loss: 0.54\n",
      "Epoch: 006/010 | Batch 700/781 | Loss: 0.45\n",
      "Epoch: 006/010 | Batch 750/781 | Loss: 0.55\n",
      "Epoch: 006/010 training accuracy: 86.19\n",
      "Epoch: 006/010 validation accuracy: 76.50\n",
      "Epoch: 007/010 | Batch 000/781 | Loss: 0.43\n",
      "Epoch: 007/010 | Batch 050/781 | Loss: 0.31\n",
      "Epoch: 007/010 | Batch 100/781 | Loss: 0.46\n",
      "Epoch: 007/010 | Batch 150/781 | Loss: 0.37\n",
      "Epoch: 007/010 | Batch 200/781 | Loss: 0.29\n",
      "Epoch: 007/010 | Batch 250/781 | Loss: 0.39\n",
      "Epoch: 007/010 | Batch 300/781 | Loss: 0.32\n",
      "Epoch: 007/010 | Batch 350/781 | Loss: 0.27\n",
      "Epoch: 007/010 | Batch 400/781 | Loss: 0.32\n",
      "Epoch: 007/010 | Batch 450/781 | Loss: 0.35\n",
      "Epoch: 007/010 | Batch 500/781 | Loss: 0.39\n",
      "Epoch: 007/010 | Batch 550/781 | Loss: 0.27\n",
      "Epoch: 007/010 | Batch 600/781 | Loss: 0.26\n",
      "Epoch: 007/010 | Batch 650/781 | Loss: 0.43\n",
      "Epoch: 007/010 | Batch 700/781 | Loss: 0.41\n",
      "Epoch: 007/010 | Batch 750/781 | Loss: 0.46\n",
      "Epoch: 007/010 training accuracy: 88.80\n",
      "Epoch: 007/010 validation accuracy: 78.31\n",
      "Epoch: 008/010 | Batch 000/781 | Loss: 0.37\n",
      "Epoch: 008/010 | Batch 050/781 | Loss: 0.38\n",
      "Epoch: 008/010 | Batch 100/781 | Loss: 0.35\n",
      "Epoch: 008/010 | Batch 150/781 | Loss: 0.31\n",
      "Epoch: 008/010 | Batch 200/781 | Loss: 0.35\n",
      "Epoch: 008/010 | Batch 250/781 | Loss: 0.22\n",
      "Epoch: 008/010 | Batch 300/781 | Loss: 0.30\n",
      "Epoch: 008/010 | Batch 350/781 | Loss: 0.36\n",
      "Epoch: 008/010 | Batch 400/781 | Loss: 0.23\n",
      "Epoch: 008/010 | Batch 450/781 | Loss: 0.27\n",
      "Epoch: 008/010 | Batch 500/781 | Loss: 0.31\n",
      "Epoch: 008/010 | Batch 550/781 | Loss: 0.32\n",
      "Epoch: 008/010 | Batch 600/781 | Loss: 0.31\n",
      "Epoch: 008/010 | Batch 650/781 | Loss: 0.57\n",
      "Epoch: 008/010 | Batch 700/781 | Loss: 0.35\n",
      "Epoch: 008/010 | Batch 750/781 | Loss: 0.78\n",
      "Epoch: 008/010 training accuracy: 89.54\n",
      "Epoch: 008/010 validation accuracy: 77.83\n",
      "Epoch: 009/010 | Batch 000/781 | Loss: 0.32\n",
      "Epoch: 009/010 | Batch 050/781 | Loss: 0.28\n",
      "Epoch: 009/010 | Batch 100/781 | Loss: 0.39\n",
      "Epoch: 009/010 | Batch 150/781 | Loss: 0.36\n",
      "Epoch: 009/010 | Batch 200/781 | Loss: 0.26\n",
      "Epoch: 009/010 | Batch 250/781 | Loss: 0.29\n",
      "Epoch: 009/010 | Batch 300/781 | Loss: 0.28\n",
      "Epoch: 009/010 | Batch 350/781 | Loss: 0.37\n",
      "Epoch: 009/010 | Batch 400/781 | Loss: 0.41\n",
      "Epoch: 009/010 | Batch 450/781 | Loss: 0.44\n",
      "Epoch: 009/010 | Batch 500/781 | Loss: 0.25\n",
      "Epoch: 009/010 | Batch 550/781 | Loss: 0.34\n",
      "Epoch: 009/010 | Batch 600/781 | Loss: 0.32\n",
      "Epoch: 009/010 | Batch 650/781 | Loss: 0.17\n",
      "Epoch: 009/010 | Batch 700/781 | Loss: 0.31\n",
      "Epoch: 009/010 | Batch 750/781 | Loss: 0.30\n",
      "Epoch: 009/010 training accuracy: 92.01\n",
      "Epoch: 009/010 validation accuracy: 78.56\n",
      "Epoch: 010/010 | Batch 000/781 | Loss: 0.29\n",
      "Epoch: 010/010 | Batch 050/781 | Loss: 0.11\n",
      "Epoch: 010/010 | Batch 100/781 | Loss: 0.30\n",
      "Epoch: 010/010 | Batch 150/781 | Loss: 0.28\n",
      "Epoch: 010/010 | Batch 200/781 | Loss: 0.30\n",
      "Epoch: 010/010 | Batch 250/781 | Loss: 0.21\n",
      "Epoch: 010/010 | Batch 300/781 | Loss: 0.25\n",
      "Epoch: 010/010 | Batch 350/781 | Loss: 0.23\n",
      "Epoch: 010/010 | Batch 400/781 | Loss: 0.33\n",
      "Epoch: 010/010 | Batch 450/781 | Loss: 0.30\n",
      "Epoch: 010/010 | Batch 500/781 | Loss: 0.20\n",
      "Epoch: 010/010 | Batch 550/781 | Loss: 0.22\n",
      "Epoch: 010/010 | Batch 600/781 | Loss: 0.26\n",
      "Epoch: 010/010 | Batch 650/781 | Loss: 0.48\n",
      "Epoch: 010/010 | Batch 700/781 | Loss: 0.22\n",
      "Epoch: 010/010 | Batch 750/781 | Loss: 0.18\n",
      "Epoch: 010/010 training accuracy: 93.01\n",
      "Epoch: 010/010 validation accuracy: 78.42\n"
     ]
    }
   ],
   "source": [
    "loss_list, train_acc_list, valid_acc_list = train_model(model, \n",
    "            data_loader, \n",
    "            optimizer, \n",
    "            NUM_EPOCHS, \n",
    "            device = DEVICE, \n",
    "            batch_size = BATCH_SIZE,\n",
    "            metric_func = compute_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOydd3xT1fvH3ydp0rCVLbMgS/YoGxGUpSiCoD/AhaLg4ItfNy5QHF/cflEB2coS9etAWQLKnkWGgGwKlFlWmUmb5Pz+yGjSjN60SZO25/165UVy7rnnPg3tfe4553k+j5BSolAoFApFduiibYBCoVAo8gfKYSgUCoVCE8phKBQKhUITymEoFAqFQhPKYSgUCoVCE8phKBQKhUITcZEaWAgxFbgTOC2lbOjn+IvA/R523ASUk1KeE0IkA5cAG2CVUiZGyk6FQqFQaENEKg9DCNERuAx8489hZOl7F/CslPJW5+dkIFFKeSYixikUCoUiZCK2JCWlXAmc09h9ADAnUrYoFAqFIvdEbElKK0KIokAPYJhHswR+F0JI4Csp5UQtY5UtW1YmJCSE30iFQqEooGzevPmMlLKclr5RdxjAXcAaKaXnbKS9lPK4EKI8sEQIsds5Y/FBCDEEGAJQrVo1kpKSIm+xQqFQFBCEEIe19o2FKKn+ZFmOklIed/57GvgJaBXoZCnlRCllopQysVw5TU5SoVAoFDkgqg5DCFEKuAX4xaOtmBCihOs90A3YER0LFQqFQuEikmG1c4BOQFkhRAowCjAASCknOLv1AX6XUl7xOLUC8JMQwmXfbCnlokjZqVAoFAptRMxhSCkHaOgzHZiepe0g0CQyVikUikBkZGSQkpKC2WyOtimKCGAymahSpQoGgyHHY8TCprdCoYgBUlJSKFGiBAkJCThn+IoCgpSSs2fPkpKSQo0aNXI8TixseisUihjAbDZTpkwZ5SwKIEIIypQpk+vZo3IYCoXCjXIWBZdw/N8qhxFFUi9ZWLzzZLTNUCgUCk0ohxFFHpi8gaEzNmPOsEXbFIUiJhBC8OCDD7o/W61WypUrx5133gnAvHnzGDNmTNAxjh8/Tr9+/QCYPn06w4YNC9o/K++99162fQYNGsQPP/wQ0rg5YevWrSxYsCDi19GKchhR5Mi5qwDYIyQAqVDkN4oVK8aOHTu4du0aAEuWLKFy5cru47169WLEiBFBx6hUqVKubuZaHEZeoRyGwgflLxSKTG6//Xbmz58PwJw5cxgwIDNC33PGMGjQIIYPH067du2oWbOm20kkJyfTsGGmQPbRo0fp0aMHdevW5a233nK39+7dmxYtWtCgQQMmTnTI1Y0YMYJr167RtGlT7r/fUX3hm2++oXHjxjRp0sRr9rNy5Uqfa2fF37mHDx/mtttuo3Hjxtx2220cOXIEgO+//56GDRvSpEkTOnbsSHp6OiNHjmTu3Lk0bdqUuXPn5u6LDQMqrDaKuPaglL9QxBpv/bqTXccvhnXM+pVKMuquBtn269+/P6NHj+bOO+9k+/btPProo6xatcpv3xMnTrB69Wp2795Nr1693EtRnmzcuJEdO3ZQtGhRWrZsSc+ePUlMTGTq1KmULl2aa9eu0bJlS/r27cuYMWP44osv2Lp1KwA7d+7k3XffZc2aNZQtW5Zz585pvnagc4cNG8ZDDz3Eww8/zNSpUxk+fDg///wzo0ePZvHixVSuXJkLFy5gNBoZPXo0SUlJfPHFF5q/50iiZhhRRMWjKBS+NG7cmOTkZObMmcMdd9wRtG/v3r3R6XTUr1+fU6dO+e3TtWtXypQpQ5EiRbjnnntYvXo1AGPHjqVJkya0adOGo0ePsm/fPp9z//jjD/r160fZsmUBKF26tOZrBzp33bp1DBw4EIAHH3zQbU/79u0ZNGgQkyZNwmaLzX1NNcOIASJVxEqhyClaZgKRpFevXrzwwgssX76cs2fPBuwXHx/vfh/o7yhrOKkQguXLl7N06VLWrVtH0aJF6dSpk98cBSllwHDU7K4d7Fx/9k2YMIENGzYwf/58mjZt6p7lxBJqhqFQKGKORx99lJEjR9KoUaNcj7VkyRLOnTvHtWvX+Pnnn2nfvj1paWlcf/31FC1alN27d7N+/Xp3f4PBQEZGBgC33XYb3333ndtpeS5JZUegc9u1a8e3334LwKxZs+jQoQMABw4coHXr1owePZqyZcty9OhRSpQowaVLl3L9HYQL5TBigGlrkkm32qNthkIRM1SpUoVnnnkmLGN16NCBBx98kKZNm9K3b18SExPp0aMHVquVxo0b88Ybb9CmTRt3/yFDhtC4cWPuv/9+GjRowGuvvcYtt9xCkyZNeO655zRfN9C5Y8eOZdq0aTRu3JgZM2bw3//+F4AXX3yRRo0a0bBhQzp27EiTJk3o3Lkzu3btiplN74jV9I4GiYmJMj8VUGo4ajGXLVYAXuhWh2G31o6yRYrCzD///MNNN90UbTMUEcTf/7EQYrOUMlHL+WqGESNccjoOhUKhiFWUw4giXtthBWeip1AoCijKYSgUCoVCE8phxAhqgqFQKGId5TAUCoVCoQnlMKKJSvVWKBT5COUwooinvziRZibtWkbUbFEoYgG9Xk/Tpk1p2LAhd911FxcuXIjIddq1axeRcQs6ymHECL9uO07nj5ZH2wyFIqoUKVKErVu3smPHDkqXLs2XX34ZkeusXbs2IuOGg1jVkYIIOgwhxFQhxGkhxI4AxzsJIdKEEFudr5Eex3oIIfYIIfYLIYKL3+djsurMnLuSHiVLFIrYo23bthw7dgyA5cuXu4sogUPxdfr06QAkJCQwatQomjdvTqNGjdi9ezcAb775Jo8++iidOnWiZs2ajB071n1+8eLF3eN26tSJfv36Ua9ePe6//363LtSCBQuoV68eHTp0YPjw4V7Xd5GcnMzNN99M8+bNad68udsR/d///Z9XHYtBgwbxv//9D5vNxosvvkjLli1p3LgxX331lduOzp07M3DgQLccij/5dYApU6ZQp04dOnXqxOOPP+6We09NTaVv3760bNmSli1bsmbNmlx8+/6JpPjgdOAL4JsgfVZJKb3+F4QQeuBLoCuQAmwSQsyTUu6KlKEKhSILC0fAyb/DO2bFRnB78Gp5Lmw2G8uWLWPw4MGa+pctW5a//vqLcePG8dFHHzF58mQAdu/ezZ9//smlS5eoW7cuTz75JAaDwevcLVu2sHPnTipVqkT79u1Zs2YNiYmJDB06lJUrV1KjRg2vmhyelC9fniVLlmAymdi3bx8DBgwgKSmJ/v37M3fuXO644w7S09NZtmwZ48ePZ8qUKZQqVYpNmzZhsVho37493bp1AzJl2GvUqAHgV37dYrHw9ttv89dff1GiRAluvfVWmjRpAsAzzzzDs88+S4cOHThy5Ajdu3fnn3/+0fT9aSViDkNKuVIIkZCDU1sB+6WUBwGEEN8CdwPKYSgUBRxX8aLk5GRatGhB165dNZ13zz33ANCiRQt+/PFHd3vPnj2Jj48nPj6e8uXLc+rUKapUqeJ1bqtWrdxtrmsXL16cmjVrum/eAwYM8HrKd5GRkcGwYcPYunUrer2evXv3Ao4iUMOHD8disbBo0SI6duxIkSJF+P3339m+fbu74FJaWhr79u3DaDTSqlUr9/XAoTn1008/Abjl10+ePMktt9zilkq/99573ddcunQpu3Zl3iYvXrzIpUuXKFGihKbvUAvRljdvK4TYBhwHXpBS7gQqA0c9+qQAraNhnEJRaNE4Ewg3rj2MtLQ07rzzTr788kuGDx9OXFwcdnumQGdWKXKX1Lher8dqtfq0+zsWrI9Wjb1PP/2UChUqsG3bNux2OyaTCQCTyUSnTp1YvHgxc+fOdc9QpJR8/vnndO/e3Wuc5cuXU6xYMa/P/uTXg9llt9tZt24dRYoU0WR7TojmpvdfQHUpZRPgc+BnZ7u/YNOA35IQYogQIkkIkZSamhoBMyOHBql8haJQUqpUKcaOHctHH31ERkYG1atXZ9euXVgsFtLS0li2bFlEr1+vXj0OHjxIcnIyQECl2LS0NG644QZ0Oh0zZszw2rDu378/06ZNY9WqVW4H0b17d8aPH++WT9+7dy9XrlzxO64/+fVWrVqxYsUKzp8/j9Vq5X//+5/7nG7dunlV5otEPY2oOQwp5UUp5WXn+wWAQQhRFseMoqpH1yo4ZiCBxpkopUyUUiaWK1cuojYrFIq8o1mzZjRp0oRvv/2WqlWrct9997llx5s1axbRaxcpUoRx48bRo0cPOnToQIUKFShVqpRPv6eeeoqvv/6aNm3asHfvXq9ZQrdu3Vi5ciVdunTBaDQC8Nhjj1G/fn2aN29Ow4YNGTp0qN9ZTyD59cqVK/Pqq6/SunVrunTpQv369d12jR07lqSkJBo3bkz9+vWZMGFC2L+XiMqbO/cwfpNSNvRzrCJwSkophRCtgB+A6oAe2AvcBhwDNgEDnctVQclv8ua1X1tAhs37+08e0zNK1igKO0re3JvLly9TvHhxpJQ8/fTT1K5dm2effTbaZrntslqt9OnTh0cffZQ+ffpoOje38uYR28MQQswBOgFlhRApwCjAACClnAD0A54UQliBa0B/6fBeViHEMGAxDucxVYuzyI9kdRYKhSJ2mDRpEl9//TXp6ek0a9aMoUOHRtskwBEuvHTpUsxmM926daN37955dm1VQCmKJIyY79OmZhiKaKFmGAUfVUBJoVCEjYL0AKnwJhz/t8phKBQKwBEKevbsWeU0CiBSSs6ePesO+80p0c7DiBmmrj5E+1plqVsxfEkuCkV+okqVKqSkpJDfwtMV2jCZTD5Ji6GiHAYO7zv6t13E6QT737sjz64rBKiHOUWsYDAYvDKNFYqsqCUpYO4mR2K51a7u3gqFQhEI5TCAbSlp0TZBoVAoYh7lMKKIUgZRKBT5CeUwFAqFQqEJ5TD8sGjHCd6d75AJvnA1nVd+/BtzRvirYGUtoATw05YUEkbMJ/WSJezXUygUitygHIYfnpj5F5NWHQLg0yV7mbPxCN8nHc3mrNDxtyQ1e8MRAA6d8VWwVCgUimiiHEY22Jxxr3kVP+UKs31k2ka/0iEKhUIRLZTDADYfPud+v+XI+Ty7rj8n5Gq7kh67heAVCkXhRDkMYO+py+73fcatjaIlCoVCEbsoh6GRSITA+i0tqFK/FQpFjKIcRoyh3IVCoYhVlMOIMaI9wbDa7GxKPpd9R4VCUehQDkPhxadL93LvhHV5uvmvUCjyB8phZEMkn/j95O1FnT0nHQEAKnFQoVBkRTkMrUTg7u7PGWVtuppuDft1FQqFIicohxGEWIhYqj9yMQdTL2ffUaFQKCKMchhBOJFmjuj4fictfpzU/tPKYSgUiugTMYchhJgqhDgthNgR4Pj9QojtztdaIUQTj2PJQoi/hRBbhRBJkbIx2vgr2ORvTuNPpFChUCjymkjOMKYDPYIcPwTcIqVsDLwNTMxyvLOUsqmUMjFC9kUdrStesbA0plAoFBFzGFLKlUDAgH4p5VoppSt2cz2Qu+rkESI/3aqtNrtyLgqFImLEyh7GYGChx2cJ/C6E2CyEGBIlmxyGOO+/h/NIbtzf/V7rklSt1xby6k9/h9kihUKhcBB1hyGE6IzDYbzs0dxeStkcuB14WgjRMcj5Q4QQSUKIpNTU1LDbd+qiY+N78upDXu0/bznG2gNnwn693DJno2/dDiklS3edwu5nz0ShUCi0ElWHIYRoDEwG7pZSnnW1SymPO/89DfwEtAo0hpRyopQyUUqZWK5cubDbGGiJ599ztzJw0obwX8/PItjB1MtuO9KtduZsPBLSzX/ssv089k0SczYdCZudCoWi8BE1hyGEqAb8CDwopdzr0V5MCFHC9R7oBviNtCqI+PNP/1m4m2lrkgGYsOIAr/z4Nz9uOaZ5zE+XOr7ekxEOE1YoFAWbuEgNLISYA3QCygohUoBRgAFASjkBGAmUAcY51+itzoioCsBPzrY4YLaUclGk7AyGEHm/6X3uSrrf9q1HL3gdv3gtI0/sOXruKusOnOW+llXz5HoKhSJ2iZjDkFIOyOb4Y8BjftoPAk18z8h7At28I0mgZMHsHNdHi/eE3xig34S1nLpooU/zyhj0Ud/yUigUUSRiDqMg0HPsaooY9NE2w4uU89f8tn/x5/5sz71ktnLqopkKJU2ar3f2ct47TYVCEZuoR8ZsuJYRO7W1T6RdY+k/p4CcaSFOX5tM6/eWhXSOiqtSKBQulMPIJ0gpQ5IcP3clnUvmnO9zZHUUSpxEoVAohxEiHy7eTcKI+Xl+3d+2nyDDpv15v/nbS2j7nz/Cdn0101AoFMphhMiXfx6I2rUfnb7J/T7tWgYdP/iTfuPXBux/2ZLzWhpqRqFQKLKiNr3zEWkeobSfLd0HwJFzV/Pk2sqBKBQKNcMogOw8nha2sZSYoUKhcKEcRgGk59jVYR8zHG4jKfkcfcatId1qD8NoCoUir1EOoxDyx+5TPm07j6fx4eLdZHUN4Sze9MqPf7PlyAWSz+aN8q9CoQgv2e5hCCH0UsrYSUZQ+HAizX8yH8C783f5tK3df5ZSRYy0qH69u63v+LWYM+zcXLusV1+1JKVQKFxomWHsF0J8KISoH3FrChnlSsSHZZynZv0V8NikVYd82iavPkTfLNFVducqkWtGocVNrNibSnI2dULW7D/DGz8XGu1IhaJAo8VhNAb2ApOFEOud9SdKRtiuQoE+TMs9W45cyPUYLln1QBb5m2g8PHUjnT5aHnTc+ydvYMb6w7kzTqFQxATZOgwp5SUp5SQpZTvgJRyqsyeEEF8LIWpF3MICjL/aF3nN7pMX3Uq4nkQijDb6P61CocgNmvYwgJ7AI0AC8DEwC7gZWADUiaB9igjy67bj/GvOFgAMev9LUVrrNC3ZdYpGlUtRsVRgYcP9py/nxEyFQhEjaEnc2wf8CXwopfRc+P4hWOlURfZEez/Z5SzCwePfJFH5uiKsGXFr2MZUKBSxhaY9DCnl4CzOAgAp5fAI2BSzhFtDKhaXaALuYXhYezXdv+TIsQuBo7Wyw2qzkzBiPp8u2Zt9Z4VCERW0OIzyQohfhRBnhBCnhRC/CCFqRtyyQkB+DFn9aUsK9UcuZs/JS36PT1p5MEfjptscYVoTc3i+QqGIPFocxmzgO6AiUAn4HpgTSaMUscsfu1MBx2a5P95d8E+2YyhdKoUif6LFYQgp5QwppdX5mklsrqbkO2J5giGBDFvuJDwCnR/sx46FyDGFQuEfLQ7jTyHECCFEghCiuhDiJWC+EKK0EKJ0pA3Mj+w/fZn1B89m288eQx7DZYpnasj2lDTf4yGM+cUf2ZeNdSHUvEOhiHm0REn9n/PfoVnaH8XxsKj2M7LQ5ZMVACSP6Rm0X2n7eXrq1zHT1jUvzNKEpw/zdB7pNjsmj/rmnv1OXTT7HctzE3zdgeAOVM0sFIrYR0viXo0gr6DOQggx1blR7lcbQjgYK4TYL4TYLoRo7nHsYSHEPufr4dB/tNhnGm/wjmEaTYX2J/FIYXUmXLhu2wLv2cSTMzd79V+174z7fSA5dc/zB0xa736f26WuvEJKiTWf2KpQ5AXZOgwhhEEIMVwI8YPzNUwIYdA4/nSgR5DjtwO1na8hwHjnNUvjyChvDbQCRgkhrg80SH4lwznB66NfFWVLfBkyw9tBrNl/lsU7T7o//++vlGzH0AWQPhnyzWaftlhcknp/0R5qvbYQi1VpbyoUoG0PYzzQAhjnfLVwtmWLlHIlcC5Il7uBb6SD9cB1QogbgO7AEinlOSnleWAJwR1P1Fh34CynL5oDRg15svvkRa+bjwUjAA/HLUEQG0+yp9Iyl5d+2nLM69jQGZs5meZ/+cmTtGsZHEy9TCCprNzka+Qls5waWBZVv0OhALTtYbSUUjbx+PyHEGJbmK5fGTjq8TnF2RaoPfyYL/KxYTzLbM1YYG8T8umPTt+ETUrSrfagexaplyz0+GwV9zSvzCf3NQUpqUZmXYqFxlfokf5+jn6EcLLnVGZ+xaIdJ32Ob0wO5v8d9P5yDYfOXGFAq6phtU2hUEQXLTMMmxDiRtcHZ9JeuObo/p5BZZB23wEc6rlJQoik1NTU0C0wFqOJOMDDcb+Hfq4TVwW5YOvdly2O7Oi/Dp93NFw9R3GuMtbaG4B6uqMkmwZSmRz8DFEm63LSoWwkz/MLahteofBGi8N4EUdo7XIhxArgD+D5MF0/BfB8DK0CHA/S7oOUcqKUMlFKmViuXLnQLdDpWWpvTjOxj3jSQz79Wkam76z12kI2aAinBeCcI6N5q70W91necDevMT1DWcJXkzsv+GypfzmPORuP+m3PLXd/sZrEd5b4PZZhszNj/WFsWlUTNSCAbzceYe3+M9n2VSgKMkEdhhBCB1zDsSk93PmqK6X8M0zXnwc85IyWagOkSSlPAIuBbkKI652b3d2cbRFhg/0mjMJGV53vZmyojJq30+vzRXOG183L/e68o7DRYVmBjfImapm/cfdJMj3Jnbp1ubYlt2gt17EtJXwOTktqyraUNM5c9u/cJ648yBs/7+DVH/8Om00AI378m4GTN4R1TIUivxHUYUgp7cDHUkqLlHK7lHKblNKidXAhxBxgHVBXCJEihBgshHhCCPGEs8sC4CCwH5gEPOW87jngbWCT8zXa2RYRNthvAqCtzrecaaicv5p5IzNn2Gj85u+8OW+n7xrbOYfDSJGOWZGVOBLMs92HvzB+ThVxOtf25BeCOaf9py+Rdi1D0ziufnOTcj+7yY9aXwpFJNGyJPW7EKKvEKGXh5NSDpBS3iClNEgpq0gpp0gpJ0gpJziPSynl01LKG6WUjaSUSR7nTpVS1nK+poV67VC4iokkex3q63JfGe7itUwlV0uGY0/jl63HOHvF4WcvXHXe+C4c4ZKhnDtSykWCeTap0lHQ8AfjW7m2JzfkZahrsHtzl09Wcs+4NXlmS1a0/OpfsVhpNGoxy/cUHievKHxocRjP4RActAghLgohLgkhso8hzWessTeksThAmQjtH0xa6ZhRuJ+ULxwmLf4Gv31bWiYAUFGcp00YZj35iUD35gOp2W+k7zt1yR2A4GLPyUtsdgUaRJD9py9zyWLl49+VPLui4KIl07uElFInpTRKKUs6Pxe4mt6/2tqgF5KB+mURGV+vy3InPLOPC6YqAfvfZXkHgAmGTyNiT6yS01Wg81fS6frpSqavTfZq7/7ZSvqO9ynlki0XzRlcSdceDOhydC6Jk7+OnGftAbVJrihYaMn09rmD+mvL7+yXVdhnr8zzhh8ikkSn83QYV8/B5ZOkFr0xYP+/ZU0O2G/gOnGFiYaPw26PFrL6uEiidcHTX24IwJUARZ1yyh3/DS373rV853J494xby8BJapNcUbAI6DCEECanREdZZ7RSaecrAUddjALHantDAGqLY9n0DEwgET2vm+8pRyRVMIcB8GrGYwB002+mly7v1/BzsG2lmfHLDwSU3LDa7H43nC9cTdekAhwOUs6Hlo3unmGofXJFASbYDGMosBmo5/zX9foF+DLypuU9U20O9ZHbdRtzPEbWNXQXXrpKZ/Y4/ikSXOh3g7yJLpYPABhr/JK8TiWLpITH+4t2e1Xn87zR1nptoU94MsAnuSjfuv+0/wqB4Ub5C0VBJqDDkFL+V0pZA3hBSlnTQ6G2iZTyizy0Mc84KssD8KzhfzkeI1C+mJfDOHsQ4opwyZh9ouF+WYVfbO0ASDbdT3kiv4GbV/jbI3DpNn2zzjdiTcrAgob+nuyPnL3qft/lk5Ws2pezLHot4bWZM4zIuYxtRy+QMGI+k1cdZPicLRG7jkIRCC2b3p8LIdoJIQYKIR5yvfLCuLxH8IOtIwDtdH4V2XOM3vObPncQStcAoSVIDV7KGOJ+v9H0NMmmgWyPf4w/jM/xqH4hpbgcVlujgS3AjfbZuVvd72esP6x5rwOg44fe+aV/H4tcBn1ehCDP2uBwou/M/4d52/wKHygUEUXLpvcM4COgA9DS+UqMsF1R43OnttNLcXPDOm6SZ2jnuYNQWnvdKQtGL/kQgJLiKjV1JxlpmME20xA+Noyjk24LT+l/oRj5Qw32+IVrJIyYz5r9Z5iz4YjfPlkVcz1vy33Hr2Xb0QuOdg336w8W7cmRnWqZSaFwoEWtNhGoLwtJ2uthWQGAproDYR33oDOPQIfdIQtSpzuhBGNtlDeRYJ5NfZHMgvhX2W2vihmj286++tX01a8G4CXDXFqZv+Q0sV1C5JetjqfkJ2du5rqixmx6O1i2OzMxbvPh84yct5OnOt3Iaz+FRwpk69EL9P4y9ACDSG16n7uSTvO3l/B+30bhHVihyAFa1kR2ABUjbUjsIPjd1gKAOHIXqjluuaOSnuc95AbOgi0dygSPkArELplAgnk2PdLfp3f62ySYZzPeepdPv2nGD8gvz8YXzVaOnLvq015/5CKfNn9KuENnbA6oLRUqM9fnLNs/ax5GuDh81vHzzo6QkKNCEQpaHEZZYJcQYrEQYp7rFWnD8pJRd9X3+vybrS0AvXShJ3x58pVHFJCLejrn0ksIS1LZ8b51AAnm2e7XQXtFGugOM8nwSdiuEQ2uakiccy1J5RSrzc7LP2yn6ejA8vZaZg1Z8zByy9hl+0jKUnukcMzxFbGMFofxJtAbeA/42ONVYHikfQ2vz2vtDQD4xDiBG3ORk+HikjlzptJStxd0BqjSKtfjBuLe9FEAdNVvpmQB2BCPFOsPnqXWawuZm3TUrfGV061r1wzDbLWFRVr9kyV76Tch+orFCoUnwRL36gFIKVcA66WUK1wvQLNibX7kDKWYbb0VgJ668Gbr1hVHoFxdMJjCOq4nZynFHZb3AFgf/6+IXSe/syZLfYs3/eR+ACz4+4RP22WLlU+W7HUXzXI5mqPnrtH10xVhtVOhiBWCzTBme7zP+qgzLgK2xBSvWgcDMDhuQVjHras7CuXrZ98xl+ySCZyS11FUWAqVTHpumL422W+01St+amt8uGg3Y5ft8xveelCDUKJWPOcqEUy8Vyg0EcxhiADv/X0ugAh+sbWjlLhKScJzAyjJZSqJc1ChQVjGy+biatMAACAASURBVI5H0l8CYHX8v2kofPdTFDnHVWkxw2ZnzMLdUSuu9NzcrSSMmB+VaysKH8Echgzw3t/nAslk6x0A3KEPz82grkhxvMkjh7FLVne//y3+dSoQsRpU+ZJwbSJPWHGA1Eu5W6W12yXvLfiHlPPe0WKeT2b+7P1xS+732BQKrQRzGFWEEGOFEJ97vHd9rpxH9kWVv2UN9tkr01e/MizjuQo0tZlyAovVlgdRL4IE82x+tbUB4JE43zBVhTfhzth+69ed2DVsgu84nsbElQf5VxbJj3NXwhMunBfklV6XInoEcxgv4hAbTPJ47/r8UuRNiwUEP9pupqVuL9XEqVyPlqjbw3FZmpOU5vTF4E+kNcsVy/X1XPwrYzjzbG15Iu43SlPgal/lmLzYE5i2JpltKYFDf+12yTfrkt0aWlkjrAZ/neTnrNhj3rbjdPlkJUt25f7vRBG7BBMf/DrYKy+NjCY/2dpjl4K++tDqI/ijqTjAX/Y6gMCazVPn0I7hy9OAzOW1dSpqKs85ddHChBUH/AoTztt2nJG/7GTssn1RsCx8/HPC8SCy95SaZRRktKnfFWJOUoa9sgrPxP2YK5G/ipylqi6Vw05F3HDE6ofCdunILI8XGQyIUFXB/MaU1Yd82kKddVgCyNl78q85fzFm4W72nvL9/blkceTouPJALl7LUJvYOSTDZg9YXkARHpTD0ICrTsarcbOz6RmYZjqHTMgOuyNJ0B7CBkZi9fBoQnWw/BeA/ximcLNue1jGzM/4yyTPzmEcv3CNtfvPuKVIRv7iP3fDkwyb4//aardz5rKF6Ws8HFWW34PTQTbPVVhtcLp9upI6ry+MthkFmog6DCFEDyHEHiHEfiHECD/HPxVCbHW+9gohLngcs3kci6oUyXe2TgD8X9zyHI+R4NwDWWlvDMCUVYew2QM/DXneR8I1F0mR5Rhn7QXADOOYbHpLqohUqoZh7ybW+Pj3PW6p8FC57eMVDJy8gcvm0HXGpIRhs//izV93sS/L0o0WZ6CkQYLjT2dMEV6yVasVQnwAvANcAxYBTYB/SylnZnOeHkdlvq5ACrBJCDFPSrnL1UdK+axH/38BzTyGuCalbBrCz5Irnutah+83H+XoOX/S4ILvrR25N24lTcR+tslaIY9fVZzmnCzOFYoAMDfpKCVMgb/+djeWdb9/uvONPDo9PJufH1j7c4M4Sx/9GpJNA72OzbF25hXr4zQUB/kt/nV3+2Tr7bxjfTAs148FPv9jf47PdeVf5BTX0pNr1uEinM7gz92nKWGKIzGhdPgGVSjQNsPoJqW8CNyJ48ZfB0fUVHa0AvZLKQ9KKdOBb4G7g/QfAMzRMG5EGH5bbeYPvzng8f9YHTfXX+JHoif0m0YVkequ6OfiiiXwU2qxeD0ApYsZqVa6aMjXC8arGYP9tg+I+5Nk00AvZwHwWNxCbhTHqCeOkBT/hE/EWCvxDwuNL2PIpbpvXnAtG0HDlXvPBD0eLsKtautZ6/yR6ZuUDpUiImhxGAbnv3cAc6SUWrO/KgOemswpBMjfEEJUB2oAf3g0m4QQSUKI9UKI3hqvmSuCPeWdoyTznCq2XXSbQx7b4TC8S7KKAOsQvZpUcpd6FQQu+5pTrmEiwTybluZx/GRrzwxrF4am/9urzzpbfWqaZ9I/3eE8lsW/yKL4EZQVF1kZ/yyLjC/zgH4Jm+Kf4Lv4t7lJd5Td8Q/zatws5hpHhy07Ptw8823w0qZa65hvTM5ZEmSg//OcVAPc4OEk+k9cz+UsDyAJI+bzeR5EX52/kh704UdRcNBSQOlXIcRuHEtSTwkhygFmDef5+8sIdOvrD/wgpfR8/KsmpTwuhKgJ/CGE+FtK6VPVSAgxBBgCUK1aNQ1mBSGbG/OzGU/RS7+Or4yfMTx9GPPs7TQNq8NOFZHK7/aWXu2ekVL3NK9MEYOeOxtXou2NZdxPwvcmViU+LjJbTalcx7MZT7s/1zTPpKo4zXlZnIsUB2C93b/uVT3dUd7RTfNq0wvJkDhHhM8nhnE8nvE8MsbiKn6PYp6A175UgN+1QJLu245eoHb54l5t/zdxvddnm8130I+X7OVft9XO1rar6VYMeh0GfWj/XxeuptPs7SUhnRMJTl3UcktS5BYtNb1HAG2BRCllBnCF4EtLLlKAqh6fqwCBChH3J8tylJTyuPPfg8ByvPc3PPtNlFImSikTy5Ur569LjulyUwWvzzb0/GV37F+MNX5BW132ETIAlcQZjMLGIRm4DtUn9zXl3T6NaHtjGQCKGPXsfrsHL3WvS/Uyxbi5dtmA54YLOzoOy4puZ+GihXk8X1u7Usv8DQnm2XSyZKrb77dX4gdbR26xfMIlWcTd3kW/hT+Mz0fc5vxEbpehtNwUXeq5oVJ/5GLuzcEylmfN9WgydY1viLQi/Gip6X0vYJVS2oQQrwMzgUoaxt4E1BZC1BBCGHE4BZ9oJyFEXeB6PBRxhRDXCyHine/LAu2BXVnPDTe+f9C+f+APp2cGe92i26ZpXFeElKv8q1ZMBj06nWOidldjLV95ZDhLKUZZH8HqnJAmyxtIMM8iwTyLLukf8ULGExyWFWlkmUKCeTZtzZ8DUEN3io4avyNF7lm+9zS1Xst5WOnWHBSjOpWNYoGiYKFl/vmGlPKSEKID0B34Ghif3UlSSiswDFgM/AN8J6XcKYQYLYTo5dF1APBtlprhNwFJQohtwJ/AGM/oqmhyiaIkmGez1X4j7TTOMBLESQAO2XNe6Tbcm6S5RxBItPgEZWhpdijgf2N8Pw9tim2kzPzGLl7LYOb6w2H9X33mW/9P++ZcRnYFIt1qZ9eJnEvNNBy1mEembfRpN2fYeOzrJJJVmGzMoWUPw/Xb1hMYL6X8RQjxppbBpZQLgAVZ2kZm+ewzlpRyLZDnVe+zrisH2wS3oaOp7gANxCF2yhqBOwLVxSmuSSOnuc7v8ff65PmPGnFSuY5T8joqiAvsin+E+pZp2Z9UiHjxh+2aN9hzy5iFu3mzl7dC8oHUyxh0OqqVyXkEXm73DS5brPy5J9WnffW+Myz95xRSSqYMaunnTD/E2vNUAUXLDOOYEOIr4D5ggXOpKLZ2MiNEsN/Bj633AnCPfnW24ySIUxyR5QNuAA9snf1mfbhVVPOCbpYPACgqLFyvRA+RZCbo5cRZ5DRX40Sa77Vu+3gFHT/8M0fjzdpwmA8W7c6ZMRpQ9/7YRcuN/z4cy0o9pJQXgNJoy8PId2T9RQ0m37HW3hCAwXEL0RF8o7GqOM2RLDkYhYE0itPN4liSGhT3e5StiT7+xAdDYfX+nOWIZL3sVD8aWqHw2k87GLfcJ2AxuuS/56l8iZYoqavAAaC7EGIYUF5KWSD/+j3/oPs0q5xt/sMGez0AOmajy1RJnOWYjHyUUyyyV1blvCzOM3E/8nrcjGibowBG/xYT24GKfIiWKKlngFlAeedrplPGo8CT3RPhQ86IqenGDwL2KYqZkuIqJ2UuZRry8RPUCxlDAUfG+ErjM1G2JnoUtKWW1MuRjZAKSWyxoH25MYqWJanBQGsp5UjnhnUb4PHImhUdShUxeH3ObgXBgpFT0rGRbcL/H09F4cgIzrXDyCGlixmjcl1Pltlb8IQzk7yaLjXgd1UYiIbibLBf477j19Lq3aWax7pkznC/v2fc2lxYlclDUzMjpVq8vYRpzpyKpf+cptcX2e8RKvIOLQ5DgJd4ko18/bwbmDi9jk/ua+L+rEWC/OUMh+98x+A/CqihSAbghCyTK9tciXuhRlTFyn/UInsrhqc7sspv1QWX53AguUW3jZ3xj1BbpNBZt4Vk08BsXzdwNvuho0S01GaDXXfz4fNBJdWz8uTMv8JgkTcr92ZGSp29ks7aA5n/h9tT/Eum3Pn5Kh6csiGzIVZ+0Qs4WsJqpwEbhBA/OT/3BqZEzqTookW+wZPl9qbYpaCffiXvZ/QnNUvobBnh+IU/GCTLWws3lCpC8pieALz609+5Gita/GZvy1i+ZJxxLDCWBuYpXKEITcR+fokfyR+2ppyW11FDd5LWuswonCXx2isCrzP9izrmr0nHkH3nKBDtaLcLV3NXIzzpcHANrbRrGWTY7CFLjITKjmP5P+ru5y3H+PfcrWx+vQtlisdH2xxNZOswpJSfCCGWAx1w+PFHpJRaHhHzPdqS5QQPZLzCbON7bDI9RYLZu8hSRXEOszT4OJK8IpaK7tizTGh3mrxVc2/V+yaeXZHxFBOOJ+AXM4awytYIM0auEY+JdFrpdrPc3pTruMSG+GHohGSv6WESzLOIvcfO6C20W212ks9epcsnKzSfM2bhbkqY4ni6s3Y5/4krD3Iw9Qo7j6fxxcDmtAhT8a9syYd7GDPWO2qyHDpzpWA4DCGEDtgupWwIhH8uGoN43mC1qsS6QmwBSnCVS2QmQ90gznFCliavbl7VyxSlRbXr+XHLMQBeuf0mujesiM0uafJW9IPbulo+YEn8S1yWJoqLzMSv320t6Kb3VgHO6nyzYsHIEnsiAKlcT33LVHabHgGgrW4X6+wNgp2e50SvAJLkPwt3+y1JG4wJKxyhs54OQ8sMaek/DimcvuPXcuC9O9DrYs1xB+fI2at8tmwvP/51jLlD2tC6Zu6WkwsSQR2GlNIuhNgmhKgmpTySV0bFCpVKmTT3vc/yBt/Fv83fpsd4IWMoP9g6AoKK4hwnc7l/kR01yhZzVxtb8WJnALfD6NuiSkSvHSr7ZBW3IyiCmUf0i5lvb81hWREyXHfUnN1gzMRT1zydPaZBzDG+m63DKSycSDOzbPfpsIwVqkRN6iULFUP4OwoHFquNId9sZsTt9ahXsQQQWFbeH4O/3sS+047663M2Hom4w8hPkyMtC403ADuFEMuEEPNcr0gbFgu826cRnw/wK5Lrw18yU0L6I8NXvBU3HXDkYJwgshFSf77QKaLjR4prmBhnu9vhLIBg+lRasWB0Kwr31sVWhE20bgw7j18MeXbjKfshpcQehqIsM9YfzrE+1MHUy3y36ahP+wvfb/OpAzJj3WFW7E3l1Z/+psYrC3xk4E9fNHPk7FWSz1zxKjHgIsND8TeS/2f5a97lQMum91sRtyJGKRYfx11NKpF2LYPle06z9J/AT2lW4kgwz6auOMLi+BE8HLeEd60PUImzUcny3vJGV7fSbWHjkfSX2GYawmfGccw3tyFD06955DFn2HJUKCkatH5vmfv9ewv+YdKqQxx4746QN+1dD/Y2u+SNn3dwfVEDSa935cZXFwQ/MQt3fr6aq+k27mtZ1av9h80p1ChbzKvtnfn/eH3eeMixUb/16AXqVSxBK4+f7V+31uL5bnUDXjcvlhHzU632gDMMIUQtIUR7KeUKzxcOp5uSdybmLa5a2g+0qe5ue6BNdSY/rE0EbY+sxp82R2jut8a30QnJKZlHG38eXF/M6JNXUlhIozhjrY4ijR8bshVWzjMenOKrzJofmLYmGdAWZu6PhBHz6TfBkbNx/moGVrt/KZ1gibKBCktpJeX8VXp/uYbXf97h1b7hkG/UVz66f+c5wZakPgMu+Wm/6jxWIKlYykTymJ65iu54OsORzdxctx+Ac7JkWGxTaOcLax8AeunXZav1pQiO1blsI2XOo+62HMmstRFolqLFH73y4985Usm9eM2xbLUjxBmeBAZMXM87Sk4FCO4wEqSUPiJJUsokICFiFhUArmLiQY9CS7FCXCFaokrHwND0ZwHYGf9olK3JP6zeF1jgUCLDsnwSyOloGXrOxiO89ENw7bZg1wzVfikl6w6eZXIuBRuD2ZSfCOYwgoU2FAlyTAGssjcm0Tye2dbOLLG3iLY5hZLFzpDbIiKdRBFuOW5JA3GIYlxjouFjbhTHwjx+dHhgygZ2HfefFLf7xCWuhViMKd3qO7sLdJ8MtCQ1f/sJr88r9vrW0LiSbvVp87qmy2GEuOAUqLfdLjl8VtsG/ms//e0lf5KfCeYwNgkhfDSjhBCDgc1++iuycIZSvGp93CdhLVy8eVd9vhvaVnN/k0EfETtiF0E/i6Ne1116bfWqq4pT3KzbzttxU+mh8/wjl+iwU0ukOGVI7md+/GvsNA2mm34zy+JfpBI5kx+PNe4Yu8pv+91frgl5rCW7Tvm0hTpJeXp29ilgp7MpFRtwsz6HM6bxKw5wy4fL2XvK36q9N7M2HPGSP/ExwcNRrtqXyrvzY3f5K1j4yL+Bn4QQ95PpIBIBI9An0obFIq1rlPa7SZbX1KtYgnSrnUHtg1f6y8qPT7Wj26crAXirVwNGzfMuMdu5bjm/FdDyM0myHr/Y2tFPv5IvrH28Mu712HhcP59e+nXU1x32OfdBtIny7bZXpZ7uKGtNw+lo+ZQjIdZuV8DVbGYI2XH47FVN/YItSf2y9Rg31y7n1ZZy3n+hK1fk1bEL16hToQSHzlzhwtV0mlXTvvfpz4m5AiNe61nf3WbOsCEExMdF/4Ev4KOvlPKUlLIdjrDaZOfrLSllWynlybwxL7aoXaG4T1u5Enmf0r/o3x35Iwe5F3UqlKCtMwmpQSXfjfi3ezd0ixwWJD6x9qOYsLDJ9BSdPIQPN8Q/zQjDt36dRTDOyhJMsN5JE/NEEsyz6ZH+Po+lPw/AZMNHYbU9v6O16NPmw+epP3Jxjq6x9egF9mTzpO9aksqas+Hi2IVrPPPtVp6cudnL+Ww7esFv/6x+p/NHy+kTJvXerDQYtZi6ry/idC5L4oYDLVpSfwI5q+VYwPD3dFKhZDypIah9RpuvHmrBgdOXidP5PitUub4oMwa3pufYVewMsI4NjhyPZm8viaSZYeWwrMg1aaSISGe68UO/fT639maerR37ZGZmfCku85D+d3bJ6lyUxXjHMJWB6a9xllI+5y+1t2CPvQp1dSmUJY0zfvoURv7wk2Hub5kqN2gZz/UsfyLN/03XtdeSXQTW6Ytm1h44y5ocVj/MCa7kwlbvLXMLkEaL2MhoUuQZJU0GmlW7nu0p/p+cgGy1f66PgRoboXKTZRorjM9SXed9A+tg+YyUAImVaRTnc9s97s/d0wMXygJ4LuMp5se/yh369Xxj6557owsomw+f92mLdI5Khs3/WpREYrFq38j3TPorjERUg1gI0UMIsUcIsV8I4RNnKoQYJIRIFUJsdb4e8zj2sBBin/P1cCTt1MpdTSr5tA1oVS0KlvjSsU65kJbHgq3llvHjEAa1S8iBVbGE4Jb0z0gwzyLBPJt65mnUM08L6Cxywk6ZwB57FUYbvqaqCO9TdEEiGpnNXy7f77d9U/J56r6+iCvOpapQaoOEi/yUKBgxhyGE0ANfArcD9YEBQoj6frrOlVI2db4mO88tDYwCWgOtgFFCiLxPl85Cm5plqHxdZkTxlje6cn/r6txaL++lP7LyzaOt2PRaF839K18fODK6QknfiOpm1aIjzx5+HLMnM/GYCf/+0+fOhMEvDJ+HdF45zlOSnOks5TdCDW0NB2lXM4Iev2R2OIxQM8qX/XOKv474zpj8kXrJgtVDp8q1TvZGluxzT8whhjFHmkjOMFoB+6WUB6WU6cC3wN0az+0OLJFSnpNSngeWAD0iZGeOcUlvtKkZnfKruaFs8XjuaOS/qNNrPW/yaevlZ3aVEx4NMbIrv/GbvS3b7DVpojtId90mTec8F/cdm0xPs930OHfq1vGkfp67gmAb3S6EylTPNdltvufUic1cf0RzqdqW7y5ltJ+McZcyblbSrXaenOmdwbBox0lvp5PHRNJhVAY85SVTnG1Z6SuE2C6E+EEI4VIW03puVHGJ+0W7ilpO0fvZ+AYoYTJQxTkD6dOsMs/cVluTPPTqlzvTvlYZjFmqrcXHZX6O0+fP7yoUBqU7KgR+ZfyUOAKHiwrsPKX/heFxP7vbvjB+zsuGb92fvzW+wyHTA5TjAs/HfcccwzvcKI7RUByM3A8QYbTkLuQ1v247nm2fGq/Mz/V1ft/pWKrccuS8OzQ3EHVeX+gT5v7EzM3uOiXRIJIOw9+dIasb/xWHBEljYCnwdQjnOjoKMUQIkSSESEpNjXwOgb9s1PyY4q+V57rW4dmudXzaXbMrkyHzV6jK9UWZ9Vgb9r57u1dfz28smMBcQeE8JVlmc8ji7zc95LdPY3GARcYRvGSYC8ACWys+yrjXffw3Wxtez3jE/XmT6Sn+FfczbfW7WBb/Ir/Fv+6ehbwXNzmCP034WbM/9uquz9noK52eFS2/ui9+v43Tl4JHWn2waHeuQnCPXcgcf/W+MySMmM+JNP/5IuEmkg4jBfDUIq4CeLlxKeVZKaVrl2kS0ELruR5jTJRSJkopE8uVK+evS1gp+Lc7bSx/oRN/vtDJ735HVkweM4xC4C8AGJzxgvt9E5G54VrbmSk+L/4N6uocos+PpT/PUxn/5gtbHxLMs0kwz2ZYxnBm2ro6S81mclr67iUNjPuDuqLQ1TfLFk/Bw7zi+80pvPPbP0H7jFueuxnCxkNn3bOT2RsdOUR/Hc6bnzWSDmMTUFsIUUMIYQT6A16Fl4QQN3h87AW4vunFQDchxPXOze5uzraoU1hueNlxfTGjTx0CT/56oyvv9nGUru1SPzPzufB8fYLm5gkA/BI/EoGdfvoVLIl/yd3jjCxJB8tnLA2qNSbcTiTBPJtWlnFenx9Pfw6A9wxTKEzfbiwTbMUhWAhv4ze13eIOpF7hvq+0Sd2Em4jlYUgprUKIYThu9HpgqpRypxBiNJAkpZwHDBdC9AKswDlgkPPcc0KIt3E4HYDRUsroa3IA1xU1cPKi2WspJpTyj7HEi93qcuaShXUHc75EEMiBli5mxODcI9EJwT+jeyAEfLBoT46vld84R0k22OvRWrebQ6YH3O077dXpmf6fsFxjiT2RtzMe4A3DTAbrFzLFdkdYxlWEh/cXeYteng8SrXXRHLo8Sl4/wEY0D0NKuUBKWUdKeaOU8l1n20ins0BK+YqUsoGUsomUsrOUcrfHuVOllLWcr2mRtDMUbrrBIakx6q4G7raBrarRv2VVht9WO9Bpbkbd5S+yODpUK1OUOUPaaO5f1Bialo0r8kQARYx6TAZ9VEIqo4lrA9zFixlDwuYsXHxt64ZFGnjDMJO2up3Zn6DIM8bncPnph82h1ajLq2dWlemdQzwjf4oY9Yzp2xiL1YaUkuuKGnk7QMGVh9om8NavsaVGOfvx1vwRpPysi7Ujbg1J3rpMMUeeQyWP3JVQn4jGDmjG8Dlbsu8Yo1zDRIJ5NuCIipIReEazEsfr1kf40DCROcZ3qWmeGTGFZEX2uCKhQmH3SW8pnhe+3xYuc8KK+q0KI/Fxep7vVpfBHfJXrkG7G8vy+p3eM58yxR03e0+ZkOuKGrmhlHfCX7AZw203lWfCA80ZdmutHNvWovr1lI+CwGMkiISzcPG9rRP/dSYNHjQ9oHI3okioNUMAtuZgg/7w2St5ntinHIbCL5MeasGYexp5zQ78EWzGIISgR8MbMOg9o6RCm2KUKmJgo58M9va1yoQ0TmHgU2s/9/v79CvCPn4H3d/01//Bv/Q/ukN6e+rWh/06hZERP/4d8jm3fLjcnafhKqMbaZTDCBGXXlMJkyFH5+eX7fHyJUz0j4BO1oNtE0LqH+j7mvVYG5pUdYSYTnk4MXdGFRgEjcyOnIxh+p8pRvax+RU4R1vdToz434wV2Omi20yyaSAzjf9hjGEyzxt+cB//0jgWFZ0VffIqv0ntYYTI893qUKdCCbrcFH39qPxIrfLFmfBAC56Yqa1oY7Aqgd8NbcPFa9aAootb3ujKuoNneWpW9hXbCgqXKMp9ljf4Lv5tdpoG80LGUH6w3QJAc7GXH+PfZKy1N3HYqSzOcLc+9ASy/fZKfG7tTUVxnlcMc5hg+IwnMp4N94+iiEGUwwiR+Dg9/VpUyb6jImRqly/upauz6qXOQaXW4+P0lCvh7VBurl2WVfscukHF4uMoEmJkV0Fgo7yJM7IkZcVFPjJ8xTpbfdaYnnEf95Qi0crD6S+zwt7Eq02PjVcMc+ih34Qhw0qGup0UeNSSVB6TT1M2AhKumXCTKqWYN6wD1xXNXOqrWrpoyONMfDBzecrzuy6IlQSDkWiZwOfW3gBezuKTjH4ctpcnXToc6ZD0Z91JgDdbPuVOyzs0Mk8mwTyLxuZJ3GsZSYJ5to+zALChd1canGscnQc/lSLaKIeRB9TxU9q1oPBMF0fuSSg/o79JQ8VSprDMBooY9dzfuprzOoIOtcrSv2VV3u/bONdj5zc8N8GPy9LUMM9krO0ebkn/jDqWGSSYZ/O7vaW7z1FZgR2yJpcoCgguUoxNsl7Qayy1NweguW4/tUVouQMFmZ3H00gYkXuxwlhDOYwIseqlzsQ574zta3k/3SaP6cn84R2iYVbYuS+xKm/3bsisx7QnAN5arzyD2iW464t7Mudx/+MsebYjfzx/C8a47H9l3767If+M7oFeJzDodYzp2zjbaK+CiB2de/bQzvJFhMJ6BZ0tHwOwJP4ldCqcF4Azl9Pz9HpXLHkTXqscRoSoWrqoW0vpuiK+FewSygTWYcpvPNimekjV/uL0Ot7s1YDuDTI1plwS8TfdUJID793BrtHeJU5rVyhBzXLFWfTMzXzQL/hsQacThXLvIlockjew2e6Yaf5ifF3TOTXFcaYYPuQzwxfcQGBpGoc8vGPdM550lhmfJ9k0kIpBzimMvPpT6GG5OUHtUkWQfi2qkm6T/F9iVT5duhfI1J0qFq+++kDodYKiRv/fT81yxalZLn8s8R187w7u+2odSX5qWBc0Bqa/xh7TIBrpkplueJ9BGS/79Lmei3xiGE9nvXcWc2+PSK3fbK1JEKdYYmvBs4b/udsP2StQQ5eZQb0i/jnqWr4mVjmZR3LjeY2aYUQQvU7wYJvqmpZRCiMd62TK0fdpHnP1sUKic11faX2dTjDr8da0qpH/KjKGigUj4Qy69gAAF0tJREFUHSyfAdBJv41bdZ6hzJKn9T+zxfSEl7O4KH2DGu7Ub6ChLtnLWQBezgIgXmSwzPh8+H6AMPPy//LmiT+vUY+5eUSDSiXZefxi9h0LETXLFSd5TM9om6GJ4vFxXLYEVhMNVL0wPk7PO70b0u3TlZEyLWZIkeWpa57Oz8aRvG+YxG2WD7lIMZJN97v7HLJX4ImMZ9kjM5NCm4l9tNTt5mtbd/aYBrnb19nq80DGK9gRdNVtpo9+NSMzBnGRYuwxDeJG3Qmai738JX0LfCkig3IYecSsx1pzINV/7V5FbHNviyrc07wKAyYFlsGIDzKLrFOhRCTMikksGHkt41F+jH+T7aYhXseamydwjpI+52yRtdlic+yBuIQas/K7vaVXRFdj8yS2mx7nybhfeTwjdmcaBQ21VpJHXFfUSIvqBX9poiAiBBSLD7yJPuqu+oVOtj0Yf0lfmf9a5m/8OouccpFiTLd2o6t+Mz8aR4ZtXEVwlMNQFDrigmSPB6JxFd/SqC4eaZ+/1Ikjj6C+eSrTrN3paXmXBPNsrBFYzPjIeh/gyAEZqv81V2O11/3NvfrlxJNOETJrZt+jW+kWWkw2DaSrLilX18nvqCUpRYHgm0dbBZUR8WT/e46qdIESq4Z1rsUXf+73e8xF8pieXudnl/EepxMhK4p+0K8xU1cfYvfJSyGdFwtcxcRb1ocjeo3LFCXRPJ6F8SN4xTCHsiKNd60PZH8ikCh280O8b3b6h4aJAPxtT6CRLtnn+CTjJ/xha8pjGS8Uypojhe8nVhRIOtYp55MgmZV6FUu4c2OCkdXxiCyauePubx6yffUrhb4cc19iVb5+tBXNqwWe3RR2zlCKB9JfAeDxuAWUxHefsBwX6Klbz0zDu+6Zgj9n4Ymns+hpeZeh6c/Sy/I2ALfqt/Jy3Jzw/RD5CDXDUBQaFv27Y9Djr/e8Cci+tvIdjW4IeOyR9gnc2fgG9p/WFuBQvkQ8py9ZAh6vUNLEj0+195rNNKpcir+PpWkavzCwR1bjHsub7o32OuavScfAk/p5vGz41u85F2Qx3rXez0Z7PbroNrPEnsgR6UoklTys/52jshwr7E2woWendCw7NjJPZobxPYbGzWeRrRVb/OzXhIuG4iC7ZEJMzWRixxKFIoo82elGHru5Jo/dXNOntoAuhD2PlgmlaVG9NP/XUlstkd/+5V8iJpB0zOAONfxusLucnScbX7tNkw0FAc+N9k8M40g2DfRyFtekke32GgxPf5oE82yaWibxva0Th2VFpth6ejgLAMHXtu78YW+ODe9gh0sU5Yl0h5T7T/GjaCgOhv1n6evcN/kt/nUOmh4g2TSQWKk5omYYCgUOaXUXnv7iobbVea6r/zj/FS924vgFs885oVDc5P0nuPjfHSlfIp7ri/nKyQAUMei5sVxxdhxTOT3eCBLMs/nN+Cp36je4W2+2fMpRL2eQe05ShqHpz/KV8VN+i3+d2uZvci3t3kAk86D+d0qLS3TT+9aKWW58jk7pnxDtEmwRnWEIIXoIIfYIIfYLIUb4Of6cEGKXEGK7EGKZEKK6xzGbEGKr8zUvknYqCieBJg6uJ/juDSow+u6GXFfU/827eplitL3RIaBYtoSjj1bJl73v3E7ymJ4UNcaxyaMEbd2KJQI6Cxf/uacRkx/yrjKo1AQcvJThyP0Ya+1Ngnl22J2Fi8X2lrybMRCA/xlH5aqG+nNx3zE//lX6xy13O4t+Tln5euZpACToTtFMBA/EyAsi9lsmhNADXwK3A/WBAUKI+lm6bQESpZSNgR+ADzyOXZNSNnW+ekXKTkXh5eB/etKnma8kSRunim4o4bKv3VGfMfc0omOAuhtGvfefmucNvlyJeDrXLcd/+zfVdK2ixji61K/gzpKvWbYY/9eyKoPaJXj1y7pZXxjYJRNIMM/mE2fIbSSZZOvJeVmcxrpDvB43K2A/HXbisFKWNGqIE1TkLCW4SjOxj2TTQHdBq4W2lsy2dqaBeQpJTll5M/EkmGfR0/JeRPdLtBLJJalWwH4p5UEAIcS3wN3ALlcHKeWfHv3XA9pi4hSKCHJz7XLsGt09oACiP4oY9UFroI8d0IwZ6w8zfvkBv8enPdIqZDsB1r1yKyVMBuLj9LzZqwHT1ya7j5kMatYRWQRtLF+wxzSIwXELKYKZV62P00m3lenGD7I/3YO+llFslnUDXmenTMi1teEgkr9RlYGjHp9TnG2BGAws9PhsEkIkCSHWCyF6R8JAhSIQ/pxF5euK8MxtOXvKq3RdEV7uEbwYUTCKOeXas27A31CqCMU9lsF2vJUpC1/CZOD3Z4NHhilyhwUj7c3/BWBg3J8kmwZqdhapsiSTrbdzn+WNIM4itojkDMPffNjv1qAQ4gEgEbjFo7malPK4EKIm8IcQ4m8ppc/jmRBiCDAEoFo1bZEpCkVOWDPi1qhdO+n1rry/aDdDO9YM2q94lj2UaOhYvXpHPd5bsDvPrxstjlGOR9JfZJrxQ3fbBxn3scDemmRZEcetUBLtDetwEEmHkQJU9fhcBTietZMQogvwGnCLlNIdkC6lPO7896AQYjnQDPBxGFLKicBEgMTExNiIPVMoArBmxK05um0UMTqWnPIDgzvULFQOA+BPezNqmmcGyZnI/84CIrsktQmoLYSoIYQwAv0Br2gnIUQz4Cugl5TytEf79UKIeOf7skB7PPY+FIpw8bBzozi7LPFwUfm6IvmmXOyed3rk6Dy9TtDuRt/yuwWdWEqwixQR+wmllFZgGLAY+Af4Tkq5UwgxWgjhinr6ECgOfJ8lfPYmIEkIsQ34ExgjpVQOQxF2mla9juQxPalQ0hRtU8JGgxzIkPgTZIyPy0xa61a/ApU9HF3X+pnhqqGU51XkbyLqEqWUC6SUdaSUN0op33W2jZRSznO+7yKlrJA1fFZKuVZK2UhK2cT575RI2qlQFCS+f6ItG14NnuV9YznvmvKB9mcGd6jB/7d379FVVXcCx78/khASkEBCGMJj8gDkqQSMkGDCQxQELGjHWQSnxRF5VMHa2tGCVqfV6Yx9zIy6xjWt02ltXYpatZSyxtJa63RpRyRSoKBSaKHKVIX6amtnqehv/tj7Jic3NzcnyX0ccn+fte665+677zm/5Jzkd88+Z+99/9pZ3L2qrl2dtU1t11J+7svHlBa13uq78dxxoePdfdP5oW8pDlrTaKMEZ5r19Damjynun9/uLq/+ef1474O2jmU7b1jAwMJ8pv79jtay+DMsP/U8N10Y33XKic0Psrx2JAV5/XjiM3MpG9h2pjF7bLgmvktn/SWlA/szf+LwUPWDJlWkbn4NE44lDGP6uP/ZfC5/evckq+/Zxa9PvBOq+e0Tc8cmfX/KyBLuunQG8/xc5jXlg5LW78w/XnwGAIMHFHDr8im0/PZNvr+nw70xJiL6/lUaY3Jc2aBCKssG8uD6Bu69ovMOgsumjQTgjubaUH1Glp5ZEXoolJj4Hu9BH2+o4vYVtRz5pyXdWueIPnT9KeosYRiTI4YNKqRpfHmH8q/+9TTA9UY/ettSltcm61/bO80zxyR9X0QQEa5orE7YU/2ey8/uUJaLd2RliyUMY3LcJWeNzti2CvL6tV4YT+amCyfz4q2LO5SPKS3mynljuX/NrIgM+J15XXXeTCdLGMaY0NbPreHOldND1f3c0klUlRWz9+aFrWX5eb3rwDa2fBCfvWAiszPUbyaKrluUvWFELGEYY0LbvHhS67WOrqxpquHJ6+ZTUlzAg+vqAVgwMT3DjQeTUhiTKwazeOqITt/v6ixobVM16+fUcOiLHc+C0i0/rx/fWd3+WlSmruNYwjAmR40b3rM7m3piVk0ZR29byszqUgC2bjiHR6+a3at1Fvoh4osL2zoYhp0XZPjgjp0N62tKE9b9h4va5oEfVJjPiMED2DB/HJuXTKIgyUX8dJpzejmPXzuHS2e58fPWz81MM5XdVmtMjvrBxkb+7/0PsrLt2jFDer2OJWdUcPT373B5YzUf+ikPRwwewEtv/Dlh/WXTRrJtb9stu7HPnFU5lPqaUq6aN44pgb4pF08fxevvvMfH6iv53Nb9gJtPfYs/W8qGhpq2C/zjhp/W2kM/UyNV2RmGMTmqqH8epV3M7pdtj13TxOPXzqGoII+aYe17p+f1E65eMJ5BhfmuH8dFU7lvzaxO17VwSvvmsNi0umubqrlu0cQOtwj/64raDk0/iTy0voEnPjO3y3phPPXZ+XzlkjNbX6+bU8OLt7aN6RWfrGI/Q3fmne8NO8MwxkRWrDf387cs6qImfLy+Mun78TMQfn7ZFAYU5DFvQvhe5l+8eGqHslgzW+2YIex5+a3Q60pk9NBi4HUA/mrGaG5YMglwv4cXXuk4j3vsLMnOMIwxxov1z+iOG/0/25j4L+EjhxRx58rpDCjIoyuxTSfr0b51wzndiq8ziW4XfmBdPf/1yaYO5asbqxk1pIhFSS7gp5IlDGNMn/LYNU3cetFU1jRVs/um8wE3XW0w33T3G/lj1zRxcyfjanUm2Dx2/QWJb4W9ZXnbHCetc7L7jBGMt6SogMkJRiEeWz6Ipzedy/DTMnOXlDVJGWP6lEkVg1ubskoH9ueZzQsYUNCvV3c0TRwxmIkjujfYYXCOlavmjePLPzzY7v11c2qoLGu7LvOxenfHk5LZZqbusIRhjOnTRpS0ffv+wcZGPvJvTzGhm//8w7pu0QRGDhlAXaW7rvEfq+oSDnEC7c8gwM3BDm3zyZcUFaQlxt6whGGMyRlnjC7hkSsbmDa697f1JrJhfvt5QIITTcWLH4gxNorw0jMqOPHHd1v7WESJXcMwxuSUsypLye+keeqO5lpWdjFAYm/9sx/ssXrYQGZWdews2K+fsLqxOtTF+EyzMwxjjPGW145K62i9AB+dMYrRQ4uYWV3aeudXZVlxWreZKpYwjDEmg0SEWYEe21vW1md0mJbesIRhjDEZ8I1VdQnPJBpOofk8LGEYY0wGnJfkAvipIq0XvUXkAhE5KCKHRWRTgvcLReRB//5OEakKvLfZlx8Uka7HBTDGGJNWaTvDEJE84C7gfOAYsEtEtqnq84FqVwBvquo4EWkGvgSsEJHJQDMwBRgJPC4ip6tqdobWTJPtVzey+6U3sx2GMcaEks4zjJnAYVX9jaq+BzwALI+rsxz4tl9+GFgg7raB5cADqvquqh4BDvv19SlTR5WwqqEq22EYY0wo6UwYo4CXA6+P+bKEdVT1JPA2UBbys8YYYzIonQkj0VAo8QMxdlYnzGfdCkTWiUiLiLScOHGimyEaY4wJK50J4xgQ7DI5GvhdZ3VEJB8oAd4I+VkAVPVuVa1T1bry8vIUhW6MMSZeOhPGLmC8iFSLSH/cRextcXW2AZf55UuAJ1RVfXmzv4uqGhgPPJvGWI0xxnQhbXdJqepJEdkI7ADygG+q6gERuQVoUdVtwH8C94rIYdyZRbP/7AEReQh4HjgJbOhrd0gZY8ypRlQTXho4JdXV1WlLS0u2wzDGmFOGiDynqnVh6tpotcYYY0KxhGGMMSaUPtUkJSIngN/28OPDgN+nMJxUsth6JqqxRTUusNh66lSOrVJVQ91i2qcSRm+ISEvYdrxMs9h6JqqxRTUusNh6KldisyYpY4wxoVjCMMYYE4oljDZ3ZzuAJCy2nolqbFGNCyy2nsqJ2OwahjHGmFDsDMMYY0woOZ8wupoVME3b/KaIHBeR/YGyUhH5sYgc8s9DfbmIyJ0+vn0iMiPwmct8/UMiclmibfUgtjEi8lMReUFEDojINVGJT0QGiMizIrLXx/YFX17tZ2w85Gdw7O/LMzqjo4jkicgvRGR7lOLy6z0qIr8UkT0i0uLLsr5P/TqHiMjDIvKiP+4aohCbiEzwv6/Y4w8i8qmIxPZp/zewX0S2+L+N9B9vqpqzD9wYV78GaoD+wF5gcga2OweYAewPlH0Z2OSXNwFf8stLgMdwQ77XAzt9eSnwG/881C8PTUFsFcAMv3wa8CtgchTi89sY5JcLgJ1+mw8Bzb78a8CVfvkq4Gt+uRl40C9P9vu6EKj2x0BeCn531wL3A9v960jE5dd9FBgWV5b1ferX+21gjV/uDwyJSmyBGPOAV4HKbMeGmxvoCFAUOM7+NhPHW0p+mafqA2gAdgRebwY2Z2jbVbRPGAeBCr9cARz0y18HVsbXA1YCXw+Ut6uXwji/j5tmN1LxAcXAbmAWrlNSfvw+xQ182eCX8309id/PwXq9iGc08BPgXGC7307W4wqs6ygdE0bW9ykwGPfPT6IWW1w8C4GnoxAbbRPMlfrjZzuwKBPHW643SUVpZr+/UNVXAPzzcF/eWYxpj92fuk7HfZOPRHy+2WcPcBz4Me5b0VvqZmyM304mZ3S8Hbge+NC/LotIXDEK/EhEnhORdb4sCvu0BjgBfMs3531DRAZGJLagZmCLX85qbKr6v8BXgZeAV3DHz3Nk4HjL9YQRema/LOr1rIQ92qjIIOAR4FOq+odkVTuJIy3xqeoHqlqL+0Y/E5iUZDsZiU1ELgSOq+pzweJsxxXnHFWdASwGNojInCR1MxlfPq559t9VdTrwDq6ZJwqxuQ26awHLgO92VbWTGFJ9vA0FluOakUYCA3H7tbNtpCyuXE8YoWf2y4DXRKQCwD8f9+WdxZi22EWkAJcs7lPVR6MWH4CqvgU8iWsrHiJuxsb47fR6RseQzgGWichR4AFcs9TtEYirlar+zj8fB76HS7ZR2KfHgGOqutO/fhiXQKIQW8xiYLeqvuZfZzu284AjqnpCVd8HHgVmk4HjLdcTRphZATMlOPvgZbhrB7HyVf4OjHrgbX8avANYKCJD/TeOhb6sV0REcBNbvaCq/xKl+ESkXESG+OUi3B/OC8BPcTM2JootFnPaZnRU1c2qOlpVq3DH0BOq+jfZjitGRAaKyGmxZdy+2E8E9qmqvgq8LCITfNEC3MRpWY8tYCVtzVGxGLIZ20tAvYgU+7/X2O8s/cdbqi4KnaoP3J0Nv8K1hd+YoW1uwbU9vo/L8lfg2hR/Ahzyz6W+rgB3+fh+CdQF1rMaOOwfl6cotkbcaek+YI9/LIlCfMCZwC98bPuBm315jT/QD+OaDQp9+QD/+rB/vyawrht9zAeBxSnct/Nou0sqEnH5OPb6x4HYcR6FferXWQu0+P26FXcnUVRiKwZeB0oCZVmPDfgC8KL/O7gXd6dT2o836+ltjDEmlFxvkjLGGBOSJQxjjDGhWMIwxhgTiiUMY4wxoVjCMMYYE4olDGMSEJE/+ecqEbk0xeu+Ie71z1O5fmPSxRKGMclVAd1KGCKS10WVdglDVWd3MyZjssIShjHJ3QY0iZsP4dN+8MOviMguP+fBegARmSduHpH7cZ22EJGtfrC/A7EB/0TkNqDIr+8+XxY7mxG/7v3i5q5YEVj3k9I2Z8R9voevMRmV33UVY3LaJuDvVPVCAP+P/21VPVtECoGnReRHvu5MYKqqHvGvV6vqG34Yk10i8oiqbhKRjeoGUIz3UVyv52nAMP+Zn/n3pgNTcGP9PI0bv+qp1P+4xnTOzjCM6Z6FuPGC9uCGfS/DjcED8GwgWQB8UkT2As/gBnkbT3KNwBZ1I/K+Bvw3cHZg3cdU9UPccC1VKflpjOkGO8MwpnsEuFpV2w0eJyLzcENzB1+fh5uQ5s8i8iRuTJ+u1t2ZdwPLH2B/uyYL7AzDmOT+iJuqNmYHcKUfAh4ROd2PABuvBHjTJ4uJuGHYY96PfT7Oz4AV/jpJOW4q316PVmtMqti3FGOS2wec9E1L9wB34JqDdvsLzyeAixJ87ofAJ0RkH24k0GcC790N7BOR3eqGQY/5Hm5qzb24EYOvV9VXfcIxJutstFpjjDGhWJOUMcaYUCxhGGOMCcUShjHGmFAsYRhjjAnFEoYxxphQLGEYY4wJxRKGMcaYUCxhGGOMCeX/AdHVbAjKeolJAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(loss_list, label='Minibatch cost')\n",
    "plt.plot(np.convolve(loss_list, \n",
    "                     np.ones(200,)/200, mode='valid'), \n",
    "         label='Running average')\n",
    "\n",
    "plt.ylabel('Cross Entropy')\n",
    "plt.xlabel('Iteration')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXiU5dXH8e8hCSQhhEBC2EIM+w4hRHZZxA03FFFEqIILbtW21lbe1mprN9tat7qigmhRiqDVWjeKIJsFAREtCGEJISQkIRAI2Zfz/vFMWEMSIM9Mkjmf65orM09mOYzym5t77uc+oqoYY4zxH418XYAxxhjvsuA3xhg/Y8FvjDF+xoLfGGP8jAW/Mcb4mUBfF1ATUVFRGhcX5+syjDGmXlm/fv1+VW118vF6EfxxcXGsW7fO12UYY0y9IiK7KztuUz3GGONnLPiNMcbPWPAbY4yfqRdz/JUpKSkhNTWVwsJCX5fSYAQHBxMTE0NQUJCvSzHGuKjeBn9qairNmjUjLi4OEfF1OfWeqpKdnU1qaiodO3b0dTnGGBfV26mewsJCIiMjLfRriYgQGRlp/4Iyxg/U2+AHLPRrmb2fxviHejvVY4wxDVFhSRk7s/JIysxle+YRbkjsQIeWobX6Ghb8Zyk7O5uxY8cCsG/fPgICAmjVyjlBbu3atTRu3Lja55g+fTozZ86ke/fup73P888/T0REBFOmTKmdwo0xdUJ+cSk7Mp2AT8o8QlLGEbZn5pJyIJ9yT5uUgEbCgNgIC/66IjIyko0bNwLw61//mrCwMB588MET7qOqqCqNGlU+ozZnzpxqX+fee+8992KNMT6TW1jC9swjJGUecX5mOEGferDg6H2CAoSOUU3p3a454+Pb07V1GF2jmxEXFUqTwIBar8mCv5Zt376da665hhEjRrBmzRo+/PBDfvOb37BhwwYKCgqYNGkSjzzyCAAjRozgueeeo0+fPkRFRXHXXXfx8ccfExoayvvvv090dDQPP/wwUVFR/PjHP2bEiBGMGDGCzz//nEOHDjFnzhyGDRtGXl4eN998M9u3b6dXr14kJSXx6quvEh8f7+N3wxj/kZNffDTgkzKOHJ2qST90bMFE48BGdG4VRkJsCyYldqBr6zC6RDfjvMhQggK895Vrgwj+3/zrf2xOO1yrz9mrXTiPXtX7rB67efNm5syZw0svvQTA448/TsuWLSktLWXMmDFMnDiRXr16nfCYQ4cOMWrUKB5//HEeeOABZs+ezcyZM095blVl7dq1fPDBBzz22GN88skn/O1vf6NNmzYsWrSIb775hoSEhLOq2xhTvewjRU64Zx5hu2f0npR5hKzcoqP3CQkKoEt0GEM7RdLFM3rvGh1Gh5ahBDTy/SKKBhH8dU3nzp05//zzj95+++23ee211ygtLSUtLY3NmzefEvwhISGMGzcOgIEDB7JixYpKn3vChAlH75OcnAzAypUreeihhwDo378/vXuf3QeWMcahqmTmFh0duTsh71w/mF9y9H5hTQLpEh3G6G6tjk7PdIkOo31ECI3qQMCfToMI/rMdmbuladOmR68nJSXxzDPPsHbtWiIiIpg6dWqla+WP/zI4ICCA0tLSSp+7SZMmp9xHVWuzfGP81p4D+by1NoWF61NPGMGHBwfSrXUzLuvThi6e0XvX1mG0CQ+ul8ugG0Tw12WHDx+mWbNmhIeHk56ezqeffspll11Wq68xYsQIFixYwAUXXMC3337L5s2ba/X5jWnISsvKWfJ9Jm+tSWF5UhYCXNijNRd0jaJrdBhdWofRKqxJvQz407Hgd1lCQgK9evWiT58+dOrUieHDh9f6a9x3333cfPPN9OvXj4SEBPr06UPz5s1r/XWMaUjSDxUwf+0e/vHVHvYdLqRNeDA/GtuVSed3oG3zEF+X5yqpD9MEiYmJenIjli1bttCzZ08fVVS3lJaWUlpaSnBwMElJSVxyySUkJSURGHjmn+v2vpqGrLxcWZ6Uxbw1KSzZkoECI7u2YsrgWC7sEU2gF1fWeIOIrFfVxJOP24i/AThy5Ahjx46ltLQUVeXll18+q9A3pqHKyi1iwbo9vL02hdSDBUSFNeauUZ2ZPCi21k+Oqg8sHRqAiIgI1q9f7+syjKlTVJUvd2Yzb00Kn363j9JyZWinSGaO68ElvdrQOLBhje7PhAW/MaZBOZhXzKINqby1JoWd+/NoHhLEtGFxTB4cS+dWYb4ur06w4DfG1HuqyvrdB3lrTQoffptOcWk5A89rwZMXduHyvm0JDqr9bQ/qMwt+Y0y9dbiwhH9+vZd5/01ha0YuYU0CufH8Dtw0OJYebcJ9XV6d5Wrwi8iPgDsAAV5R1adFpCXwDyAOSAZuUNWDbtZhjGlYvk09xLw1u3l/YxoFJWX0bd+cxyf05ar+7WjaxMaz1XHt2w0R6YMT+oOA/sCVItIVmAksUdWuwBLP7Xpn9OjRfPrppycce/rpp7nnnntO+5iwMGd+MS0tjYkTJ572eU9eunqyp59+mvz8/KO3L7/8cnJycmpaujH1Un5xKfPXpnDV31Zy1XMreX9jGuPj2/HBD4fzr/tGcOOgWAv9GnLzXeoJ/FdV8wFE5AvgWmA8MNpzn7nAMuAhF+twxeTJk5k/fz6XXnrp0WPz58/nL3/5S7WPbdeuHQsXLjzr13766aeZOnUqoaHOMrSPPvrorJ/LmLru+32HeWtNCu9t2EtuUSndWzfjsfG9uWZAe8KDg3xdXr3k5nqm74CRIhIpIqHA5UAHoLWqpgN4fkZX9mARmSEi60RkXVZWlotlnp2JEyfy4YcfUlTk7OeRnJxMWloa8fHxjB07loSEBPr27cv7779/ymOTk5Pp06cPAAUFBdx4443069ePSZMmUVBwbI/uu+++m8TERHr37s2jjz4KwLPPPktaWhpjxoxhzJgxAMTFxbF//34AnnzySfr06UOfPn14+umnj75ez549ueOOO+jduzeXXHLJCa9jTF1TWFLGuxtSue7F1Vz29Armf7WHi3u1ZtHdQ/nkxxdw89A4C/1z4NqIX1W3iMifgMXAEeAboPKdxyp//CxgFjhn7lZ5549nwr5vz77YyrTpC+MeP+2vIyMjGTRoEJ988gnjx49n/vz5TJo0iZCQEN577z3Cw8PZv38/Q4YM4eqrrz7tPh8vvvgioaGhbNq0iU2bNp2wpfLvf/97WrZsSVlZGWPHjmXTpk3cf//9PPnkkyxdupSoqKgTnmv9+vXMmTOHNWvWoKoMHjyYUaNG0aJFC5KSknj77bd55ZVXuOGGG1i0aBFTp06tnffKmHOkqmTnFZNyIJ+PNqWzcEMqOfkldIpqysNX9OS6hBhaNK2+q52pGVcnxFT1NeA1ABH5A5AKZIhIW1VNF5G2QKabNbipYrqnIvhnz56NqvKLX/yC5cuX06hRI/bu3UtGRgZt2rSp9DmWL1/O/fffD0C/fv3o16/f0d8tWLCAWbNmUVpaSnp6Ops3bz7h9ydbuXIl11577dHdQSdMmMCKFSu4+uqr6dix49HGLMdv6WyMN5SWlbPvcCF7DxawN6eAvQcLSDtUQKrndlpOAYUl5QAENhIu7dOGKYNjGdopskFtjlZXuL2qJ1pVM0UkFpgADAU6ArcAj3t+njoXcqaqGJm76ZprruGBBx442l0rISGB119/naysLNavX09QUBBxcXGVbsN8vMr+x961axdPPPEEX331FS1atGDatGnVPk9V+y5VbOcMzpbONtVjalNBcRl7c/JJPVhAWk4he3PyTwj5fYcLj/aRrRAV1pj2ESH0aNOMsT2iaRcRQvuIEAbEtqBVsyaVv5CpFW5/Bb5IRCKBEuBeVT0oIo8DC0TkNiAFuN7lGlwTFhbG6NGjufXWW5k8eTLgdNKKjo4mKCiIpUuXsnv37iqfY+TIkcybN48xY8bw3XffsWnTJsDZzrlp06Y0b96cjIwMPv74Y0aPHg1As2bNyM3NPWWqZ+TIkUybNo2ZM2eiqrz33nu8+eabtf8HN35FVcnJL2FvzrER+t6Dzih9r+dyIK/4hMcENBLaNg+mXUQIQzpF0r6FE+rtW4QcDXg7qcp33J7quaCSY9nAWDdf15smT57MhAkTmD9/PgBTpkzhqquuIjExkfj4eHr06FHl4++++26mT59Ov379iI+PZ9CgQYDTSWvAgAH07t37lO2cZ8yYwbhx42jbti1Lly49ejwhIYFp06YdfY7bb7+dAQMG2LSOqVZeUSmb0w8fG6V7wr1iGia/uOyE+4cEBRwN874xzZ1Q9wR7+4gQWocH14kWg6Zyti2zOYG9r/4leX8ec79M5p11qRwpOrb2omXTxrSLCPYEeujRQI/xjNhbhAbZ3Hs9YNsyG2MAZ0/6Fdv38/qqXSzblkVgI+GKvm25Or4dsS1DaRcRQmhji4aGzP7rGuMnjhSV8u6GVF5fnczOrDyiwprwo7FduWlwLNHNgn1dnvGieh38qmr/3KxF9WHaz5y5iumchetSyS0qJb5DBE9Piufyvm39ek96f1Zvgz84OJjs7GwiI22db21QVbKzswkOtpFfQ1AxnTN3dTJLt2Yenc65ZVgcA2Jb+Lo842P1NvhjYmJITU2lLm7nUF8FBwcTExPj6zLMObDpHFMT9Tb4g4KC6Nixo6/LMKZOOHk6p39Mc5vOMadVb4PfGH9X2XTO5X3bMs2mc0w1LPiNqWcqm865/8KuTBkcS3S4TeeY6lnwG1NPJO/P440vd/POuj1Hp3OemtSfy/u2pUmgbX9gas6C35g6TFVZkbSf1z3TOQEiXNHPpnPMubHgN6YOqpjOmbs6mR1ZeUSFNeY+z3ROa5vOMefIgt+YOuTk6Zx+Np1jXGDBb4yPVUznzF2dzOee6ZzL+7Zl2vA4BnSIsBMUTa2z4DfGR/KLS1m0YS+vr9p1bDpnTBemDDnPpnOMqyz4jfGyvTkFvLE6mbfXpnC4sJS+7Zvz5A39uaKfTecY77DgN8YLVJX1uw8ye9UuPv1fBqrKuD5tmT48joHntbDpHONVFvzGuKi4tJyPvk1n9qpdbEo9RHhwILeP6MgPhp5HTItQX5dn/JQFvzEuyD5SxFtrUnjzv7vJzC2iU6um/PaaPlyX0N6anBifc/X/QBH5CXA7oMC3wHTgJWAUcMhzt2mqutHNOozxlu/3HWbOymTe27iX4tJyRnZrxZ8mxjGqaysaWQ9aU0e4Fvwi0h64H+ilqgUisgC40fPrn6nqQrde2xhvKi9XPv8+k9mrdrF6RzbBQY2YODCG6cPi6Nq6ma/LM+YUbv+bMxAIEZESIBRIc/n1jPGaI0WlvLNuD3NXJ5OcnU/b5sE8dFkPJg/qQERoY1+XZ8xpuRb8qrpXRJ4AUoAC4DNV/UxEbgJ+LyKPAEuAmapadPLjRWQGMAMgNjbWrTKNOWMp2fnM/TKZBV85Z9cmxEbw4KXdubR3G4ICbO97U/eJW31WRaQFsAiYBOQA7wALccJ+H9AYmAXsUNXHqnquxMREXbdunSt1GlMTqsqaXQeYvXIXi7dkHD27dvpw2yzN1F0isl5VE08+7uZUz0XALlXN8hTwLjBMVf/u+X2RiMwBHnSxBmPOSWFJGf/6Jo05q5LZnH6YFqFB3DO6Mz8YEkeb5nZ2ramf3Az+FGCIiITiTPWMBdaJSFtVTRfnjJVrgO9crMGYs5KZW8i8/6Ywb81u9h8pplvrMB6f0JdrBrQnOMjOrjX1m5tz/GtEZCGwASgFvsaZ2vlYRFoBAmwE7nKrBmPO1Hd7DzF71S4+/Cad4rJyLuwRza3DOzK8S6SdXWsaDFdX9ajqo8CjJx2+0M3XNOZMlZUrizfvY/aqZNbuOkBo4wAmD+rALcPi6NQqzNflGVPr7BRC47cOF5aw4Ks9vL46mdSDBbSPCOHhK3pyfWIHmocE+bo8Y1xjwW/8TurBfF5buYsFX+0hr7iMQXEtefiKnlzUszWBthzT+AELfuM3vtt7iJeX7+Sjb9MR4Kr+7bhtREf6tG/u69KM8SoLftOgqSrLtmXxyvKdrN6RTViTQG4dHsf04R1pFxHi6/KM8QkLftMgFZeW8/7Gvby6YhdbM3JpEx7M/43rweTBsYQH2/y98W8W/KZBOVRQwttrU5izahcZh4vo0aYZf72+P1f1b0fjQJu/NwYs+E0DkZZTwOyVu5j/1R6OFJUyoksUf57Yn5Fdo2z9vTEnseA39dr/0g7xyvKdfLgpHQWu7NeWOy7oZF/YGlMFC35T76gqK5L2M2v5TlZu30/TxgHcMiyOW0d0pL19YWtMtSz4Tb1RXFrOh5vSmLV8J9/vyyW6WRMeuqwHNw2OtROujDkDFvymzsstdL6wnb0ymX2HC+nWOoy/TOzH1fHtaBJoG6YZc6Ys+E2dlX6ogDmrknl7TQq5RaUM7RTJHyf0ZXT3VvaFrTHnwILf1Dlb0g/zyvKdfPBNGuWqXNGvHXdc0JF+MRG+Ls2YBsGC39QJqsqq7dnMWrGT5duyCG0cwNQh53HbiI50aBnq6/KMaVAs+I1PlZSV8+9N6cxavpPN6YeJCmvCzy7tzpTBsdaw3BiXWPAbnzhSVMr8tSnMWZXM3pwCOrdqyp+uczpc2Re2xrjLgt941YG8Yl5ZsZO//3c3uYWlDO7YksfG92ZM92gaNbIvbI3xBgt+4xWHC0t4dcUuXluxk4KSMsb1acsdIzsR38G+sDXG21wNfhH5CXA7oMC3wHSgLTAfaInTj/cHqlrsZh3GdwqKy5j7ZTIvfbGDnPwSLu/bhgcu7kaX6Ga+Ls0Yv+Va8ItIe+B+oJeqFojIAuBG4HLgKVWdLyIvAbcBL7pVh/GNotIy5q/dw3NLt5OVW8To7q148JLutoeOMXWA21M9gUCIiJQAoUA6TrP1mzy/nwv8Ggv+BqO0rJx3v97LM/9JYm9OAYM6tuSFKQmcH9fS16UZYzxcC35V3SsiTwApQAHwGbAeyFHVUs/dUoH2lT1eRGYAMwBiY2PdKtPUkvJy5aPv0nly8TZ2ZuXRt31z/jihLxfYtsjG1DluTvW0AMYDHYEc4B1gXCV31coer6qzgFkAiYmJld7H+J6qsnRrJk98uo3N6Yfp1jqMl6YO5NLerS3wjamj3JzquQjYpapZACLyLjAMiBCRQM+oPwZIc7EG46Ivd2Tzl0+/Z0NKDrEtQ3lqUn+u7t+eAFuWaUyd5mbwpwBDRCQUZ6pnLLAOWApMxFnZcwvwvos1GBds3JPDE59uZeX2/bQJD+YP1/bl+sQYggKstaEx9YGbc/xrRGQhzpLNUuBrnKmbfwPzReR3nmOvuVWDqV3f7zvMXz/bxuLNGbRs2piHr+jJ1CHnERxkZ9oaU5+4uqpHVR8FHj3p8E5gkJuva2rXrv15PLV4G//alEZYk0B+enE3po/oSFgTO//PmPrI/uaa00rLKeDZJUm8sz6VxgGNuGtUZ+4c2ck2TzOmnqs2+EXkh8A8VT3ohXpMHZCVW8QLy7Yz778pAPxgyHncM6Yz0c2CfVyZMS5ShfxsOLQHDu0FLYOAxhAQ5PnZGBoFHXf7uOMnX6/jK9pqMuJvA3wlIhuA2cCnqmrLKxugQ/klzFqxg9krkykuK2diQgz3X9TVGpibhqGsBA6nwaFUJ9xz9nhCvuJ6KpQW1M5rHf2AqOLD4eiHSWDVvx84DaJ71E5dHtUGv6o+LCK/Ai7B2WvnOc/2C6+p6o5arcb4RF5RKa+vTublL3ZwuLCUq/q34ycXdaVTqzBfl2bcVF4Ge9fDtk9g26dOKIZGHndpCU2jTjrmOR4aCU3C69bItujIcaGeclLAp0JuGmj5iY9p2gqax0B0T+h2qXO9eQdo3t4J77Ji5wOjrBjKS45dP/542emOF0N56Wnuf9zjivOg7ODpH9f9Mu8HP4CqqojsA/bhrNBpASwUkcWq+vNarch4TWFJGfPWpPDisu3sP1LMRT2jeeDi7vRqF+7r0oxbCg/Bjs+doE/6zJnakACIHQodBkH+AefYwWTnQyE/2wm8yjQKOvXDIDTypA+Llid+aASd5b8eVSEv66SReqrnuifkC06ajW4UCOHtoHksdLzgWKhHdPCEe8zZ11PP1WSO/36c9fb7gVeBn6lqiYg0ApIAC/56pqSsnIXrU3l2SRLphwoZ3iWSWZd0JyG2ha9LM27I3uEZ1X8Cu1c7o8ngCOh6iTPK7TIWQk7z314VinKdD4DTXjwfFhn/c34WHOQ0J+RDUNPKPxCaen6GtISS/BOnYQ6leqZhCk98rsZhx4I8ZpAT5BGxx0K9WRtoZEuNK1OTEX8UMEFVdx9/UFXLReRKd8oybigvV/61KY2nFm8jOTufAbER/PX6/gzrEuXr0kxtKiuBlC+dUf22TyB7u3O8VU8Y+kPodhnEnA8BNfjrLwLB4c6lZceavX55GRTknPThsP/ED4mKy4EdzrGiw6c+T9NoJ9Rb93FqPj7UIzo4H151aaqpHqlJ8H8EHKi4ISLNcLZaXqOqW1yrzNSq1IP53P33DXy79xA92jTjtVsSubBHtO2n01DkZcP2xU7Qb1/iBGlAY4i7AAbdCd0ugRZx3qmlUYAzgm8aWfPHlBYd+1AICoHw9hBkq8jcUpPgfxFIOO52XiXHTB32dcpB7nhjPUWlZTxzYzxX9WtnbQ7rO1XI3Hzsi9k9awF1Rsm9xjsj5E6joUk9+YI+sAmEt3UuxnU1CX45fvmmZ4rHTvyqJ/71TRo/fecbWoc3Yf6Mwdb5qj4rKYTkFcfC/tAe53jbeBj1kDNf3zYeGtmeSaZqNQnwnZ4veCuapdyDs+2CqcNUlb99vp0nF2/j/LgWvDR1IJFhTXxdljlTh9Mh6VMn6Hcuc774DAqFTmNg5M+cL2htlGzOUE2C/y7gWeBhnK/ql+BpkGLqpsKSMmYu2sQ/N6YxYUB7/nhdX5oE2uqGeqG8HNI3HluFk/6Nc7x5B4if4kzhxI2w+W9zTmpyAlcmTq9cUw/sP1LEnW+uZ/3ug/zs0u7cM7qzfYFb1xUdgZ1LPWH/GeRlgjRyliiOfdQJ++ietoLF1JqarOMPxmmI3hs4OsxQ1VtdrMuchaSMXKa//hVZuUU8f1MCV/SzKYA6paQQDuyE/VthfxLs3wZZWyHre+cMzSbNnTX13S6DLhed2aoYY85ATaZ63gS+By4FHgOmALaMs475YlsWP5y3gSZBAfzjzqHEd4jwdUn+K/+AJ9i3OuG+P8kJ+JzdJ24Z0DwWoro6q2+6XgKxQ5w9WoxxWU2Cv4uqXi8i41V1roi8BXzqdmGm5t78Mplf/2szXaPDeG3a+bapmjeUlzuravZvO+7iCfj8/cfuF9AEIrtA2/7Q7waI6uaEfWQXaNzUd/Ubv1aT4K/YqCNHRPrg7NcT51pFpsZKy8r53b+38PrqZMb2iOaZyQOsOUptKylwtjw4YXpmm3M27PE7OYa0gKju0H0ctOp+LOAjzrNtA0ydU5OUmCUiLXBW9XwAhAG/crUqU63cwhLue/trlm3N4vYRHfm/y3tak/NzkZftGbWfNP+ek8KxfWfE2TYgqht0HAmtunkCvpuzMZkx9USVwe/ZiO2wpwnLcqCTV6oyVdpzIJ/b5n7Fjqw8/nBtX24aHOvrkuqXw2mw8wtnP5uKgC84cOz3gcEQ2RXaD4T4m5yRe1Q3aNkZGof6rm5jakmVwe85S/eHwIIzfWIR6Q7847hDnYBHgAjgDiDLc/wXqvrRmT6/v1q/+yB3vrmOotJy5k4fxIiuNtKsVsFBSF7phP2uL5ywB2eTr+he0POqE6dnmsfa2a+mQavJVM9iEXkQJ8TzKg6q6oHTPwRUdSsQDyAiAcBe4D2cZi5PqeoTZ1u0v3p/415+tnATbZsHM3/G+XSJrif7sHhbSQGk/NcJ+Z3LnJOgtNw54/W8YZBwM3Qc5ez6aAFv/FBNgr9ivf69xx1TzmzaZyywQ1V328lEZ05VeWZJEk//J4lBcS156QcDadnUGp4fVVbqnO26c5lz2bMWyoqcRhwx58PIn0OnUdA+EQLtfTOmJmfu1nAT7irdCLx93O0fisjNwDrgp5U1cheRGXi2hoiN9d857MKSMn6+cBMffJPGdQkx/GFCH9t+QdU56ali6iZ55bH93Fv3hUF3OCP684bVn90pjfEiqa5vuiegT6Gqb9ToBUQaA2lAb1XNEJHWON28FPgt0La6s4ATExN13bp1NXm5BiUrt4g731zHhpQcfn5Zd+4e5cfbL+Ts8UzdeML+SIZzvEWcE/KdRjk/bXWNMUeJyHpVTTz5eE2mes4/7nowzrTNBqBGwQ+MAzaoagZAxU9PUa8AH9bwefzK1n253Pr6V2TnFfHilATG9fWz7RfyD8Cu5cfm6Q94NoRt2urEoG9xnk/LNKY+qslUz33H3xaR5jjbONTUZI6b5hGRtqqa7rl5LfDdGTyXX1i6NZP73vqa0MYBLLhzKP1i/GD7heI8Z3nlTk/Q7/sWUGjcDOKGw/l3OGEf3cs2KzPmHJ3NaZ75QNea3FFEQoGLgTuPO/xnEYnHmepJPul3fu/1Vbt47MPN9GgTzmvTEmnbvIFuv1BWAnvXH5u62bMWykucdoExg2DML5wRffsE27/GmFpWk905/8WxUxcbAb2o4bp+Vc0HIk869oMzrNEvlJaV89iHm3njy91c1LM1z9wYT9OGtP3CkUxI+9q57F0Pu1dD8RFAoG0/GHqPE/SxQ+0kKWNcVpNkOX69fSmwW1VTXarHLx0uLOGHb33N8m1ZzBjZiYcu61G/t1/IPwBpGzxBv9H5eXiv55finCzVb5IzdRN3AYS29Gm5xvibmgR/CpCuqoUAIhIiInGqmuxqZX5iz4F8bn39K3btz+PxCX25cVA9W7pakOOcIFUxmk/72tl+uEJkF2dZZbsBzqVNP1tiaYyP1ST43wGGHXe7zHPs/Mrvbmpq/e4DzHhjPSVl5bxx6yCGdanjSxGLciF904khf2DHsd+3iHPm5M+/zQn5tv0huLnPyjXGVK4mwR+oqsUVN1S12LM235yDiu0X2jUPZva08+nUqo6NgovzIeM72LvhWMjv38bRr3vCY6BdvLOJWcVo3qZsjKkXagugWFYAABOUSURBVBL8WSJytap+ACAi43FOwDJnQVV56j9JPLskicEdW/LS1IG08PX2C6VFTsgfHclvhMwtoGXO78NaQ7sE6HOdJ+TjISzatzUbY85aTYL/LmCeiDznuZ0KVHo2r6laYUkZD77zDR9uSuf6gTH8/tq+NA708iZhZSWQufnE6ZqMzc5SSoDQSCfku19+bCQf7mcnjxnTwNXkBK4dwBARCcPZ4iHX/bIanqzcIu54Yx3fpOYwc1wP7hzZybvbLxzYBR89CLtWOBuYgTP/3m4ADPvhsZBv3sFOkDKmgavJOv4/AH9W1RzP7RY4G6s97HZxDcWJ2y8M5LI+bbz34qqw8S34+OcgAc4GZu0TnJBv0dFC3hg/VJOpnnGq+ouKG6p6UEQux2nFaKpRWlbOnW+uo6SsnHfuHEbfGC+ucsk/AB/+GDa/D+cNh2tfcloHGmP8Wk2CP0BEmqhqETjr+IEm7pbVcPz723SSs/N5aepA74b+zi/gvbsgLxPGPgrDf2RNv40xQM2C/+/AEhGZ47k9HZjrXkkNR3m58sLSHXSJDuOSXq2986KlRbDkMfjyOadv7OT/ONM6xhjjUZMvd/8sIpuAiwABPgFsL9wa+Pz7TLZm5PLkDf1p5I0tGDK3wKI7IONbSLwVLvkdNG7q/usaY+qVmu4Ctg8oB24AdgGLXKuogVBVnlu6nZgWIVzVv53bLwZrX4HFv4LGYTB5PnQf5+5rGmPqrdMGv4h0w2mZOBnIxmm2Lqo6xku11Wtf7sxm454cfntNH4ICXFyrn5sB798L2xdD10tg/PN2cpUxpkpVjfi/B1YAV6nqdgAR+YlXqmoAXly2g6iwJlw/MMa9F9n6sRP6xXlw+RNw/u22PNMYU62qhqLX4UzxLBWRV0RkLM4cv6nGN3tyWJG0nzsu6EhwkAsraYrz4F8/hrdvhPB2MOMLZ32+hb4xpgZOO+JX1feA90SkKXAN8BOgtYi8CLynqp95qcZ654Vl2wkPDmTKEBe+A0/7GhbdDtk7YNj9cOHDEGira40xNVft5LOq5qnqPFW9EogBNgIzXa+snkrKyOXT/2UwbXhHwmqzg1Z5Gaz4K7x6EZQUwC0fwCW/tdA3xpyxM0omVT0AvOy5mEq8uGwHIUEBTB8WV3tPmpPinIy1exX0vhaufApCWtTe8xtj/IprTV1FpDvOSqAKnYBHgDc8x+Nwmq3foKoH3arDm/YcyOf9b9KYNiyu9rZa3vQO/PunoOVwzUvQ/0abyzfGnBPX1hmq6lZVjVfVeGAgkA+8hzNNtERVuwJLaEDTRi8v30EjgTsu6HTuT1aQ48zlv3s7RPeAu1dC/GQLfWPMOXNtxH+SscAOVd3taeQy2nN8LrAMeMhLdbgm83AhC9alMnFgDG2aB5/bkyWvgvfuhMNpMOaXMOIBCPDWfypjTEPnrTS5EXjbc721qqYDqGq6iFR6tpGIzABmAMTG1v0dJV9bucvZiXNk57N/ktJiWPZHWPmU07/2ts8gJrHWajTGGHBxqqeCpz/v1TgN2mtMVWepaqKqJrZq1cqd4mpJTn4xf//vbq7s1464qLPcG2d/Erx2Max8EgZMhbtWWugbY1zhjRH/OGCDqmZ4bmeISFvPaL8tkOmFGlw1d/Vu8orLuHv0WYz2VWH9HPj0l87SzBvehF5X136Rxhjj4Y2Gr5M5Ns0D8AFwi+f6LcD7XqjBNXlFpcxZvYuLekbTs234GT54P8y/CT78CXQYDHd/aaFvjHGdqyN+EQkFLgbuPO7w48ACEbkNSAGud7MGt729NoWc/BLuGdPlzB6YtBj+eQ8U5sClf4TBd0EjLzdeN8b4JVeDX1XzgciTjmXjrPKp94pKy5i1fCdDO0WSEFvDE6pKCmDxI7B2FkT3gh+8B236uFuoMcYcx9YInoNF6/eSmVvEkzfE1+wB6Zvg3Tsg63sYco/TEjHoHJd+GmPMGbLgP0ulZeW89MUO+sc0Z3iXyKrvXF7utEL8/LcQ0hKmvgtdGsQ/eowx9ZAF/1n697fppBzI55dXDESqO5v2n3fBpn9AjyvhqmehaTUfFMYY4yIL/rNQ0US9a3QYF/espol68kon9Ef8xJnasS0XjDE+ZstIzkJFE/V7xnSuuol6eTl89jCEt4dRD1noG2PqBBvxn6ETmqj3q6aJ+v/edRqnXPMSBIV4p0BjjKmGjfjPUEUT9btGdSawqibqJYXwn99Am77Qb5L3CjTGmGrYiP8MvbB0B62aNWFidU3U174Mh1Jg/Ad2YpYxpk6xRDoDG/fksHJ7DZqo5x+A5X+FrpdCp1HeK9AYY2rAgv8MvLB0O81DgrhpcDVN1L/4MxTnwsWPeacwY4w5Axb8NbQtI5fPNmdwy7C4qpuoZ++Ar16BhJudzlnGGFPHWPDX0IvLdhDauAZN1P/zawhoAqN/4Y2yjDHmjFnw10BKdj4ffJPGTYNiq26inrIGtnwAI34Mzao5scsYY3zEgr8GXl6+gwARbq+qiboqfPZLCGsDQ+/1XnHGGHOGbDlnNTIPF/LOulSuq66J+uZ/QupXcPVz0Pgs2y8aY4wX2Ii/Gq+u3EVpeTl3japitF9a5MztR/eG+Ju8VpsxxpwNG/FXoaKJ+lX923FeZBWj+K9eg4PJMHURNKpifb8xxtQBNuKvwuurk8mvrol6wUH44k/Q+ULocpH3ijPGmLPkavCLSISILBSR70Vki4gMFZFfi8heEdnouVzuZg1n60hRKXNWJXNRz9b0aFNFE/XlT0DhIbj4t94rzhhjzoHbUz3PAJ+o6kQRaQyEApcCT6nqEy6/9jl5e00KhwpKuGdMFaP9g8lO79wBU6xvrjGm3nAt+EUkHBgJTANQ1WKguNpuVXVAYUkZr6zYybDO1TRRX/IYSACM+aX3ijPGmHPk5lRPJyALmCMiX4vIqyJS8Q3pD0Vkk4jMFpEqktU3Fm1IJTO3iHvHdDn9nVLXwXeLYNh9EF7NvvzGGFOHuBn8gUAC8KKqDgDygJnAi0BnIB5IB/5a2YNFZIaIrBORdVlZWS6WeaLSsnJe/mIn/TtEMKzzaXrjqjqdtZpGw/D7vVabMcbUBjeDPxVIVdU1ntsLgQRVzVDVMlUtB14BBlX2YFWdpaqJqprYqlUrF8s8UUUT9XtHdz59E/Xv/w0pX8KYX0CTZl6rzRhjaoNrwa+q+4A9ItLdc2gssFlE2h53t2uB79yq4UxVNFHv1jqMi07XRL2sBBY/AlHdYcAPvFugMcbUArdX9dwHzPOs6NkJTAeeFZF4QIFk4E6Xa6ixJZ4m6k9Pij99E/V1c+DADrhpAQTY+W/GmPrH1eRS1Y1A4kmH6+QwuaKJeoeWIVzZr23ldyo8BMv+CB1HQtdLvFugMcbUEjtz1+PLHdl8syeHO0dW0UR95VNQcMA5WaseLEs1xpjKWPB7PL9se9VN1HP2wJcvQL8boV28d4szxphaZMEPfJ1ykFXbs6tuov65Z5R/4cPeLc4YY2qZBT/wwrIdVTdRT/saNv0DhtwDER28W5wxxtQyvw/+rftyWbw5g2mna6KuCp/9CkIjnZaKxhhTz/l98L+4bDuhjQOYdrom6ts+heQVMPr/ILi5V2szxhg3+HXwVzRRnzL4NE3Uy0ph8a8gsgsMnOb1+owxxg1+fQbSS8t3ENio0embqG+YC/u3wY1vQUCQd4szxhiX+O2IP+NwIQvXpTIxMYbW4ZU0US/KdU7Wih0G3etkrxhjjDkrfjvif3XFTqeJ+sjTNFpZ9QzkZcHkf9jJWsaYBsUvR/wH84qZtyaFq/u3IzYy9NQ7HNoLq5+DPhMhZqD3CzTGGBf5ZfAfa6J+mkYrS38PWgZjH/FuYcYY4wV+F/xHikp5fXUyF/dqTfc2leylv+9b2PgWDL4TWpzmhC5jjKnH/C74jzZRH13J3H5FZ62QCLjgp94vzhhjvMCvgr+iifrwLpEMqKyJ+vYlsHMZjHoIQupcK2BjjKkVfhX8R5uoVza3X17mnKzVoiMk3ub94owxxkv8ZjlnaVk5L32xg/gOEQytrIn6xnmQuRmunwuBlZzFa4wxDYTfjPg/3JTOngMF3Dumy6lN1IuOwOe/h5hB0Gu8bwo0xhgv8YsRf3m58sKy7XRv3YyxPaJPvcOXz8GRfTDpTTtZyxjT4Lk64heRCBFZKCLfi8gWERkqIi1FZLGIJHl+uv4t6n+2ZLAt4wh3j+58ahP13H2w6llnpN9hkNulGGOMz7k91fMM8Imq9gD6A1uAmcASVe0KLPHcdo2q8vyyHadvor70D1BWDGMfdbMMY4ypM1wLfhEJB0YCrwGoarGq5gDjgbmeu80FrnGrBoDVnibqd42qpIl6xmb4+k0YdAdEnmbPHmOMaWDcHPF3ArKAOSLytYi8KiJNgdaqmg7g+VnJpDuIyAwRWSci67Kyss66iOeXbie6WROuS6ikifriR6BxMxj5s7N+fmOMqW/cDP5AIAF4UVUHAHmcwbSOqs5S1URVTWzVqtVZFbAh5SCrd2RzxwWdTm2ivmMpbF8MIx+E0JZn9fzGGFMfuRn8qUCqqq7x3F6I80GQISJtATw/M90q4I3VyZ4m6rEn/qK8zOmjGxELg2a49fLGGFMnubacU1X3icgeEemuqluBscBmz+UW4HHPz/fdquEPE/qydV8uTU9uor7pH5DxLVz3GgRV0oTFGGMaMLfX8d8HzBORxsBOYDrOvzIWiMhtQApwvVsvHto48NQ9eYrzYclvoV0C9LnOrZc2xpg6y9XgV9WNQGIlvxrr5utW6b8vQG4aXPeqnaxljPFLfrNlAwBHMmHlU9DjSogb7utqjDHGJ/wr+Jc9DqWFcNFvfF2JMcb4jP8Ef9Y2WP86DJwOUadpuWiMMX7Af4L/P49CUCiMdnWHCGOMqfP8I/h3rYCtH8EFD0DTKF9XY4wxPtXwg7+83OmjGx4DQ+72dTXGGONzDX8//u8WQfpGuPZlCArxdTXGGONzDXvEX1IIS34DbfpB3xt8XY0xxtQJDXvEv+YlOLQHxj8PjRr2Z5wxxtRUw07DsGiInwqdRvm6EmOMqTMa9og//ibnYowx5qiGPeI3xhhzCgt+Y4zxMxb8xhjjZyz4jTHGz1jwG2OMn7HgN8YYP2PBb4wxfsaC3xhj/Iyoqq9rqJaIZAG7fV3HOYoC9vu6iDrE3o9j7L04kb0fJzqX9+M8VW118sF6EfwNgYisU9XKGs/7JXs/jrH34kT2fpzIjffDpnqMMcbPWPAbY4yfseD3nlm+LqCOsffjGHsvTmTvx4lq/f2wOX5jjPEzNuI3xhg/Y8FvjDF+xoLfZSLSQUSWisgWEfmfiPzI1zX5mogEiMjXIvKhr2vxNRGJEJGFIvK95/+Rob6uyVdE5CeevyPficjbIhLs65q8SURmi0imiHx33LGWIrJYRJI8P1vUxmtZ8LuvFPipqvYEhgD3ikgvH9fkaz8Ctvi6iDriGeATVe0B9MdP3xcRaQ/cDySqah8gALjRt1V53evAZScdmwksUdWuwBLP7XNmwe8yVU1X1Q2e67k4f7Hb+7Yq3xGRGOAK4FVf1+JrIhIOjAReA1DVYlXN8W1VPhUIhIhIIBAKpPm4Hq9S1eXAgZMOjwfmeq7PBa6pjdey4PciEYkDBgBrfFuJTz0N/Bwo93UhdUAnIAuY45n6elVEmvq6KF9Q1b3AE0AKkA4cUtXPfFtVndBaVdPBGUQC0bXxpBb8XiIiYcAi4MeqetjX9fiCiFwJZKrqel/XUkcEAgnAi6o6AMijlv4pX9945q7HAx2BdkBTEZnq26oaLgt+LxCRIJzQn6eq7/q6Hh8aDlwtIsnAfOBCEfm7b0vyqVQgVVUr/gW4EOeDwB9dBOxS1SxVLQHeBYb5uKa6IENE2gJ4fmbWxpNa8LtMRARnDneLqj7p63p8SVX/T1VjVDUO54u7z1XVb0d1qroP2CMi3T2HxgKbfViSL6UAQ0Qk1PN3Zix++kX3ST4AbvFcvwV4vzaeNLA2nsRUaTjwA+BbEdnoOfYLVf3IhzWZuuM+YJ6INAZ2AtN9XI9PqOoaEVkIbMBZCfc1frZ1g4i8DYwGokQkFXgUeBxYICK34Xw4Xl8rr2VbNhhjjH+xqR5jjPEzFvzGGONnLPiNMcbPWPAbY4yfseA3xhg/Y8FvDCAiZSKy8bhLrZ1BKyJxx++4aIyv2Tp+YxwFqhrv6yKM8QYb8RtTBRFJFpE/ichaz6WL5/h5IrJERDZ5fsZ6jrcWkfdE5BvPpWLbgQARecWz3/xnIhLisz+U8XsW/MY4Qk6a6pl03O8Oq+og4Dmc3UXxXH9DVfsB84BnPcefBb5Q1f44++78z3O8K/C8qvYGcoDrXP7zGHNaduauMYCIHFHVsEqOJwMXqupOz2Z7+1Q1UkT2A21VtcRzPF1Vo0QkC4hR1aLjniMOWOxppoGIPAQEqerv3P+TGXMqG/EbUz09zfXT3acyRcddL8O+XzM+ZMFvTPUmHffzS8/11RxrDTgFWOm5vgS4G472Fg73VpHG1JSNOoxxhBy3eyo4fXArlnQ2EZE1OAOlyZ5j9wOzReRnOF20KnbV/BEwy7ObYhnOh0C669UbcwZsjt+YKnjm+BNVdb+vazGmtthUjzHG+Bkb8RtjjJ+xEb8xxvgZC35jjPEzFvzGGONnLPiNMcbPWPAbY4yf+X8WvxYImcVIVAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(1, NUM_EPOCHS+1), train_acc_list, label='Training')\n",
    "plt.plot(np.arange(1, NUM_EPOCHS+1), valid_acc_list, label='Validation')\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}