ee046746_tut_01-2_deep_learning_pytorch_basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/bubbles/100/000000/3d-glasses.png\" style=\"height:50px;display:inline\"> EE 046746 - Technion - Computer Vision\n",
    "\n",
    "#### Tal Daniel\n",
    "\n",
    "## Tutorial 1.5 - Deep Learning - PyTorch Basics\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut_xiv_pytorch_logo.png\" style=\"height:100px\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "\n",
    "* [Introduction & Motivation](#-Introduction-&-Motivation)\n",
    "* [Up & Running with PyTorch](#-PyTorch-Basics)\n",
    "    * [PyTorch Basics](#-PyTorch-Basics)\n",
    "    * [Datasets: MNIST, Fashion-MNIST](#-Datasets-in-PyTorch---MNIST-&-Fashion-MNIST)\n",
    "    * [Multi-Layer Perceptrons (MLPs)](#-Multi-Layer-Perceptron-(MLP)-in-PyTorch)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "\n",
    "# pytorch imports\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.utils.data import DataLoader, Dataset, ConcatDataset\n",
    "import torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <img src=\"https://img.icons8.com/color/48/000000/fire-element.png\" style=\"height:50px;display:inline\" /> Introduction & Motivation\n",
    "\n",
    "As you saw in the ML course, even though the idea of a neural network is a simple one, the implementation and calculations are not very sympathetic. Neural networks are a strong tool that might be useful to many sorts of tasks and thus simple frameworks were established to help engineers focus on the task rather than the actual implemntation of the neural networks and making the backpropagation easy as one line of code. There are several deep learning frameworks, where the two most popular ones are <a href=\"https://tensorflow.org\">Tensorflow</a> (maintained by Google) and <a href=\"https://pytorch.org\">PyTorch</a> (maintained by Facebook). TF and PyTorch have different workflows, though in recent versions they share some of the workflows.\n",
    "\n",
    "<img src=\"./assets/tut_xiv_pytorch_logo.png\" style=\"height:100px\" />\n",
    "PyTorch is \"an open source deep learning platform that provides a seamless path from research prototyping to production deployment\". In simple words, PyTorch is meant to make deep learning programming more easier for Python programmers by adapting the Python workflow and \"easy-to-code\" language.  Also, the PyTorch official website provides great tutorials to get you up and running in most deep learning fields (Computer Vision, Reinforcement Learning, Natural Language Processing).\n",
    "\n",
    "At its core, PyTorch provides two main features:\n",
    "\n",
    "* An n-dimensional Tensor, similar to numpy but can run on GPUs\n",
    "* Automatic differentiation for building and training neural networks\n",
    "\n",
    "The word \"Tensor\" might seem initmidating, but it is just an n-dimensional array or matrix.  A PyTorch Tensor is conceptually identical to a numpy array: a Tensor is an n-dimensional array, and PyTorch provides many functions for operating on these Tensors. Behind the scenes, Tensors can keep track of a computational graph and gradients, but they’re also useful as a generic tool for scientific computing. Most of the function and methods available in `numpy` are also available in PyTorch for tensors. \n",
    "\n",
    "#### Why not just use NumPy?\n",
    "Numpy is a great framework, but it cannot utilize GPUs to accelerate its numerical computations. For modern deep neural networks, GPUs often provide speedups of 50x or greater, so unfortunately numpy won’t be enough for modern deep learning.\n",
    "\n",
    "### <img src=\"https://img.icons8.com/dusk/64/000000/diversity.png\" style=\"height:50px;display:inline\"> PyTorch Basics\n",
    "We will now introduce some the basics. The main thing to keep in mind is that PyTorch was designed to have similar API as NumPy's. That means that almost every NumPy's function is in PyTorch, and most probably has the same name and same parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x: \n",
      " tensor([[-0.9316, -0.2761,  0.7423],\n",
      "        [-0.7188, -0.6464, -0.8161],\n",
      "        [ 0.2640,  0.6787, -0.3176],\n",
      "        [ 0.3718, -0.8876, -0.8888],\n",
      "        [-0.1430, -0.6374, -0.8229]])\n",
      "x size: torch.Size([5, 3])\n",
      "same as x shape: torch.Size([5, 3])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# create a tensor initialized from a random uniform distribution\n",
    "x = torch.Tensor(5, 3).uniform_(-1, 1)\n",
    "print(\"x: \\n\", x)\n",
    "print(\"x size:\", x.size())\n",
    "print(\"same as x shape:\", x.shape)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y: \n",
      " tensor([[0.5636, 0.0164, 0.2488],\n",
      "        [0.4739, 0.0954, 0.7210],\n",
      "        [0.1011, 0.9942, 0.8188],\n",
      "        [0.7817, 0.6225, 0.3460],\n",
      "        [0.9175, 0.1450, 0.6944]])\n",
      "x + y = \n",
      " tensor([[-0.3680, -0.2596,  0.9911],\n",
      "        [-0.2449, -0.5510, -0.0952],\n",
      "        [ 0.3651,  1.6729,  0.5012],\n",
      "        [ 1.1534, -0.2652, -0.5428],\n",
      "        [ 0.7745, -0.4924, -0.1285]])\n",
      "x - y = \n",
      " tensor([[-1.4953, -0.2925,  0.4935],\n",
      "        [-1.1927, -0.7417, -1.5371],\n",
      "        [ 0.1630, -0.3155, -1.1363],\n",
      "        [-0.4099, -1.5101, -1.2348],\n",
      "        [-1.0605, -0.7825, -1.5172]])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# basic math oprations\n",
    "y = torch.rand(5, 3)\n",
    "print(\"y: \\n\", y)\n",
    "print(\"x + y = \\n\", x + y)\n",
    "print(\"x - y = \\n\", x - y)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x[3, :] =  tensor([ 0.3718, -0.8876, -0.8888])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# indexing - same as NumPy\n",
    "print(\"x[3, :] = \", x[3, :])\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.ones([2, 2]): \n",
      " tensor([[1., 1.],\n",
      "        [1., 1.]])\n",
      "torch.zeros([1, 2]): \n",
      " tensor([[0., 0.]])\n",
      "y.reshape(-1, 1): \n",
      " tensor([[0.0746],\n",
      "        [0.9194],\n",
      "        [0.7642],\n",
      "        [0.7937],\n",
      "        [0.5881],\n",
      "        [0.2956],\n",
      "        [0.6239],\n",
      "        [0.1899],\n",
      "        [0.8838],\n",
      "        [0.4624],\n",
      "        [0.2099],\n",
      "        [0.3366],\n",
      "        [0.2956],\n",
      "        [0.7547],\n",
      "        [0.9789]])\n",
      "y.view(-1, 1): \n",
      " tensor([[0.0746],\n",
      "        [0.9194],\n",
      "        [0.7642],\n",
      "        [0.7937],\n",
      "        [0.5881],\n",
      "        [0.2956],\n",
      "        [0.6239],\n",
      "        [0.1899],\n",
      "        [0.8838],\n",
      "        [0.4624],\n",
      "        [0.2099],\n",
      "        [0.3366],\n",
      "        [0.2956],\n",
      "        [0.7547],\n",
      "        [0.9789]])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# ones, zeros, reshaping - same as NumPy\n",
    "print(\"torch.ones([2, 2]): \\n\", torch.ones([2,2]))\n",
    "print(\"torch.zeros([1, 2]): \\n\", torch.zeros([1,2]))\n",
    "print(\"y.reshape(-1, 1): \\n\", y.reshape(-1, 1)) \n",
    "# -1 just means 'don't care about the dimension this axis, make it happen'\n",
    "# more efficiently, use `view` as it doesn't copy the variable (saves memory)\n",
    "print(\"y.view(-1, 1): \\n\", y.view(-1, 1))\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z: \n",
      " tensor([[1, 2],\n",
      "        [3, 4]], dtype=torch.int32)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# torch <-> numpy\n",
    "z = torch.tensor(np.array([[1, 2], [3, 4]]))\n",
    "print(\"z: \\n\", z)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z: \n",
      " tensor([[1, 2],\n",
      "        [3, 4]], dtype=torch.int32)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# torch <-> numpy, alternative version\n",
    "a = np.array([[1, 2], [3, 4]])\n",
    "z = torch.from_numpy(a)\n",
    "print(\"z: \\n\", z)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "p = x.numpy(): \n",
      " [[-0.93162763 -0.27608442  0.7423003 ]\n",
      " [-0.71882963 -0.6463584  -0.8161427 ]\n",
      " [ 0.26402986  0.678723   -0.31756258]\n",
      " [ 0.3717742  -0.8876412  -0.88880277]\n",
      " [-0.14297056 -0.6374365  -0.8228916 ]]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "p = x.numpy()\n",
    "print(\"p = x.numpy(): \\n\", p)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "xy = x @ y.t(): \n",
      " tensor([[-0.3449,  0.0674,  0.2391, -0.6432, -0.3794],\n",
      "        [-0.6189, -0.9907, -1.3835, -1.2466, -1.3199],\n",
      "        [ 0.0810, -0.0391,  0.4415,  0.5190,  0.1202],\n",
      "        [-0.0262, -0.5493, -1.5727, -0.5695, -0.4048],\n",
      "        [-0.2958, -0.7218, -1.3220, -0.7933, -0.7950]])\n",
      "xy = torch.matmul(x, y.t()): \n",
      " tensor([[-0.3449,  0.0674,  0.2391, -0.6432, -0.3794],\n",
      "        [-0.6189, -0.9907, -1.3835, -1.2466, -1.3199],\n",
      "        [ 0.0810, -0.0391,  0.4415,  0.5190,  0.1202],\n",
      "        [-0.0262, -0.5493, -1.5727, -0.5695, -0.4048],\n",
      "        [-0.2958, -0.7218, -1.3220, -0.7933, -0.7950]])\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# tensor multiplication\n",
    "xy = x @ y.t()\n",
    "print(\"xy = x @ y.t(): \\n\", xy)\n",
    "# or\n",
    "xy = torch.matmul(x, y.t())\n",
    "print(\"xy = torch.matmul(x, y.t()): \\n\", xy)\n",
    "print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "xy = x * y: \n",
      " tensor([[-0.5251, -0.0045,  0.1847],\n",
      "        [-0.3406, -0.0616, -0.5884],\n",
      "        [ 0.0267,  0.6748, -0.2600],\n",
      "        [ 0.2906, -0.5525, -0.3075],\n",
      "        [-0.1312, -0.0924, -0.5714]])\n",
      "xy = torch.mul(x, y): \n",
      " tensor([[-0.5251, -0.0045,  0.1847],\n",
      "        [-0.3406, -0.0616, -0.5884],\n",
      "        [ 0.0267,  0.6748, -0.2600],\n",
      "        [ 0.2906, -0.5525, -0.3075],\n",
      "        [-0.1312, -0.0924, -0.5714]])\n"
     ]
    }
   ],
   "source": [
    "# element-wise\n",
    "xy = x * y\n",
    "print(\"xy = x * y: \\n\", xy)\n",
    "# or\n",
    "xy = torch.mul(x, y)\n",
    "print(\"xy = torch.mul(x, y): \\n\", xy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/processor.png\" style=\"height:50px;display:inline\"> Device - CPU or GPU?\n",
    "* The greatest advantage of using the deep learning frameworks in the ability to utilize the GPUs. In PyTorch, we can code it to automatically choose which device to use.\n",
    "* A good practice is to always define the device at the beginning of your code, and then just send the models and tensors to that devie using `.to(device)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n",
      "current device:  0\n",
      "device:  cuda:0\n",
      "tensor([[-1.2017, -0.6636],\n",
      "        [ 1.2575, -0.0579]], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "# check if there is a GPU available\n",
    "print(torch.cuda.is_available())\n",
    "# check what is the current available device\n",
    "if torch.cuda.is_available():\n",
    "    print(\"current device: \", torch.cuda.current_device())\n",
    "# automatically choose device\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")  # use gpu 0 if it is available, o.w. use the cpu\n",
    "print(\"device: \", device)\n",
    "\n",
    "# create a random tensor and send it to the device\n",
    "a = torch.randn([2, 2]).to(device)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src=\"https://img.icons8.com/cute-clipart/64/000000/documents-folder.png\" style=\"height:50px;display:inline\"> Datasets in PyTorch - MNIST & Fashion-MNIST\n",
    "* The first thing we need to do before we build our networks is load the data, separate it to train and test sets (and sometimes also validation) and create batches of it to train using one of the Gradient Descent optimizers.\n",
    "* Fortunately, PyTorch provide a simple data structure to load the data (usually from files) called `Dataset` and another data structure, called `DataLoader` that creates batches out of the Dataset (and it even takes care of shuffling it if you wish).\n",
    "* We now introduce two very popular datasets:\n",
    "\n",
    "1. **MNIST**- The MNIST database (1998) of handwritten digits (0-9, a total of 10 digits) has a training set of 60,000 examples, and a test set of 10,000 examples. Images are of size 28 x 28 pixels with one color channel. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image. It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting. MNIST is broadly used as a baseline to many ML papers, even today.\n",
    "2. **Fashion-MNIST** - Fashion-MNIST is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. Zalando intends Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits.\n",
    "\n",
    "Let's load the dataset (PyTorch provides an auto-download feature)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: \n",
      " torch.Size([128, 1, 28, 28])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEYCAYAAACQgLsAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcLElEQVR4nO3debBU1bn38d/DIAjkokxOCAS4MUFkEowiDkm4IiiKAoGC1xitGzWKsQpFo2iBA/EWVsw1RtDULeOA1xhBCCISfFMMMQ4llOKE+IIFyg3ITDgMYVrvH7u56bUOp/v06dXT4fupoopfn917P33O4jy9e7H3MuecAACIqUGpCwAA1D80FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0x0RzMbO1ZjawhMdfb2YXl+r4yB1jBrlgvFQXpbmY2Wgze9fMdpvZptTfbzYzi7H/QjGz182sKvXngJntT8tP1nGfM8xscsQa70urqcrM9prZITM7MdYxSoEx4+0z9pi5wszeMrMdZrbBzJ4ysxax9l8KjBdvn7HHy2lm9mpqrDgzax9jv3k3FzO7XdJjkh6RdLKkkyTdJOl8ScfV8JyG+R43BufcYOdcC+dcC0kvSJp6JDvnbgq3N7NGJajxwbSaWkj6paQ/O+e2F7uWWBgzBfcNSfdLOkXSmZK+Kek/SlBHFIyXgjssab6kEVH36pyr8x9JLSXtljQ8y3bPSJqeegG7JQ1MPfc5SZslrZN0r6QGqe0nS5qR9vxOkpykRqm8WNKDkv4qaZekhZLapG1/TWqfWyVNlLRW0sBa1PhQ8NjA1HPvkbRR0u8k/bukxWnbNErV1knSzZIOSNovqUrS7NQ26yWNl/SRpJ2SXpTUpA7fb0u9rrH5/NxK+YcxU9wxk9rXDyW9X+qfPeOlvMeLpKap47SP8bPL98zlPElNJP2xFtuOkTRFybuqNyU9ruSH31nSRZJ+JOm6HI49JrV9OyXvXu6QJDPrpmSQXSPpVEmtJeVzmtdeUgtJHZT8YGvknJsm6SVJv3DJO5Or0r78Q0n/puT1np2qT2bWMPXxxbm1qOV7kk6UNDvnV1E+GDNpijBmJOlCSZ/k9hLKBuMlTZHGSxT5Npc2krY45w4eeSDts969ZnZh2rZ/dM791Tl3WEnnHSXpbufcLufcWiUf91yTw7F/55z73Dm3V9IfJPVKPT5C0jzn3FLn3D8k3afktK+uDkqa7JzbnzpWXf2nc26jc26rpHlH6nXOHXLOneCce6cW+7hW0h+cc3vyqKPUGDO1l/eYMbPBSn5JTsqjjlJivNRejN8x0eTbXLZKapP+OaFzrr9z7oTU19L3/1Xa39soeSewLu2xdZJOy+HYG9P+vkdJ55eSdxL/eyzn3O5ULXX1tXNufx7PP6KmemvFzJpLGi7p2Qi1lBJjpvbyHTP9JT0v6Wrn3JoI9ZQC46X28hovseXbXN6W9A9JV9Zi2/TbL29R8s6iY9pjHST9T+rvuyU1S/vayTnUtEHS6UeCmTVTctpaV+Fto7PVVqjbTI+Q9LWS0/1Kxpgpwpgxs76S5kj6kXNucez9FxHjpXi/Y6LKq7k453Yo+V8p08xshJm1MLMGZtZLUvMMzzuk5DRzipl9w8w6KpmMmpHa5ANJF5pZBzNrKenuHMqaKelyMxtgZsdJekBxr+dZIamHmZ1lZser+scNXyv5zDO2ayU961Izb5WKMVP4MWNmPZVMbN/snJsfa7+lwHgpzu8YM2uqZG5LkpqYWZNM29dG3t8Q59xUJT+0OyVtUvLCn5J0l6S3Mjz1ViUd+gsl78b/W9LTqX2+oWTS6kNJy5V8fljbej6RdEtqfxskbVfyPymicM59KukXSv43ySpJS4NN/ktSTzPbbmYzs+0vNdlWZWbnZdimg5JJ2efrXHgZYcwUfMzcoeSd9DNp11SsqPsrKC3GS2HHS+ojx72SdqQeWq3k+5YXq/A3wgCAMnRM3P4FAFBcNBcAQHQ0FwBAdDQXAEB0GW+SZmbM9lc451xR7xrLmKl8xRwzjJfKV9N44cwFABAdzQUAEB3NBQAQHc0FABAdzQUAEB3NBQAQHc0FABAdzQUAEB3NBQAQHc0FABBdxtu/APXNHXfc4eXjjz/eyz169PDyiBEjsu5z+vTpXn777be9/Pzz9WKNNyAnnLkAAKKjuQAAoqO5AACiM+dqvuM1t8OufMf6LfdfeuklL9dmDiVfa9as8fLAgQO9/OWXXxa8hnxwy/3i+ta3vuXlzz77zMu33Xablx9//PGC15QLbrkPACgamgsAIDqaCwAgOq5zQb2S7xxL+Hn3n/70Jy937ty52nOGDh3q5S5dunh57NixXn744Ydzqgn1W+/evb18+PBhL69fv76Y5UTDmQsAIDqaCwAgOpoLACA65lxQ0fr27evlq666KuP2n3zyiZevuOIKL2/ZssXLVVVVXj7uuOOq7fOdd97xcs+ePb3cunXrjDXh2NarVy8v796928uzZ88uZjnRcOYCAIiO5gIAiI7mAgCIrqRzLuE1CD/5yU+8/Le//c3L+/bt8/ILL7zg5Y0bN3p59erV+ZaIMnfKKad42cy/zVE4xzJo0CAvb9iwIafj3X777dUe69atW8bnvPbaazkdA/Vb9+7dvTxu3Dgv15f1fzhzAQBER3MBAERHcwEARFfSOZepU6d6uVOnTjk9/8Ybb/Tyrl27vBx+3l4K4X2Bwte8bNmyYpZT77z66qte7tq1q5fDMbFt27a8jjd69OhqjzVu3DivfeLY8u1vf9vLzZs393J4f7xKxZkLACA6mgsAIDqaCwAgupLOuYTXtfTo0cPLK1eu9PJ3vvMdL/fp08fLF198sZfPPfdcL3/11VdePv3002td6xEHDx708ubNm70cXncRCtdPZ84lrnXr1kXd34QJE7wcrnd+NO+++27GjGPbnXfe6eVwzNaX3wmcuQAAoqO5AACio7kAAKIz51zNXzSr+Ytl6MQTT/RyuE7C8uXLvdyvX7+cjxHe3+zzzz/3cjhP1KpVKy/fcsstXp4+fXrONeTCOWfZt4qn0sZM6PLLL/fyyy+/7OWjreeyadMmL4fXwixZsiRSdcVRzDFT6eMlm6Ndu/fFF194OfwdEl4HU+5qGi+cuQAAoqO5AACio7kAAKIr6XUusW3fvt3LixYtyrj9n//857yPOXz4cC+H8z4fffSRl+vLfYPqq759+3r5aHMsofBnWmlzLCiciy66KOs24bVy9QVnLgCA6GguAIDoaC4AgOjq1ZxLMbRr187L06ZN83KDBn6/fuCBB7yc73oiiGvOnDlevuSSSzJu/9xzz1V77N57741aE+qPs846K+s24RpP9QVnLgCA6GguAIDoaC4AgOiYc8lReG+wtm3bejm81mbVqlUFrwm1F663079/fy83adLEy1u2bPHyQw89VG2fVVVVkapDpQvXkLruuuuqbfP+++97+Y033ihoTaXCmQsAIDqaCwAgOpoLACA65lyyOP/8873885//POP2w4YN8/LHH38cvSbU3axZs7zcunXrjNvPmDHDy2vWrIleE+qPgQMHejlcz0mSFixY4OVwjaj6gjMXAEB0NBcAQHQ0FwBAdDQXAEB0TOhnMWTIEC83btzYy+GCY2+//XbBa0LtXXHFFV7u06dPxu0XL17s5UmTJsUuCfVYz549veycq7bNzJkzi1VOSXHmAgCIjuYCAIiO5gIAiI45l8Dxxx/v5UsvvdTL+/fv93L4mfyBAwcKUxhqJbwo8p577vFyOGcW+uCDD7zMTSmRycknn+zlCy64wMtHu3Ht7NmzC1pTueDMBQAQHc0FABAdzQUAEB1zLoEJEyZ4uXfv3l4Obzr31ltvFbwm1N7tt9/u5X79+mXcfs6cOV7muhbk4sc//rGX27Vr5+XXX3+9iNWUF85cAADR0VwAANHRXAAA0R3Tcy6XXXZZtcfuu+8+L//973/38gMPPFDQmpCf8ePH57T9uHHjvMx1LchFx44dM359+/btRaqk/HDmAgCIjuYCAIiO5gIAiO6YmnMJ7zv161//uto2DRs29PL8+fO9/M4778QvDCXTqlUrL8e4N9zOnTsz7jO8v1nLli0z7u+EE07wcq7zSocOHfLyXXfd5eU9e/bktD/80+WXX57x66+++mqRKik/nLkAAKKjuQAAoqO5AACiq9dzLuH8SXhfsG9+85vVnrNmzRovh9e9oH758MMPo+/z5Zdf9vKGDRu8fNJJJ3l51KhR0WvIZOPGjV6eMmVKUY9fyQYMGODlcD0X/BNnLgCA6GguAIDoaC4AgOjq9ZxLly5dvHz22WdnfU54DUE4B4PyFl6XdOWVVxa9hpEjR+b1/IMHD3r58OHDGbefO3eul5ctW5Zx+7/85S91Kwy66qqrvBzO677//vteXrp0acFrKlecuQAAoqO5AACio7kAAKKrV3Mu4doKCxcuzLj9hAkTqj02b968qDWhuK6++mov33nnnV4O7+uVzZlnnunlulyT8vTTT3t57dq1GbefNWuWlz/77LOcj4k4mjVr5uUhQ4Zk3H7mzJleDu/rdizhzAUAEB3NBQAQHc0FABCdOedq/qJZzV8sQ+E9ku6+++6M259zzjnVHst2jUClcc5ZMY9XaWMG1RVzzJT7eAnn6JYsWeLlTZs2eXnMmDFePhbWyqlpvHDmAgCIjuYCAIiO5gIAiK6ir3MJ11a49dZbS1QJgProwIEDXu7fv3+JKqk8nLkAAKKjuQAAoqO5AACiq+g5lwsuuMDLLVq0yLh9uDZLVVVV9JoAAJy5AAAKgOYCAIiO5gIAiK6i51yyWbFihZd/8IMfeHnbtm3FLAcAjhmcuQAAoqO5AACio7kAAKKrV+u5oDrWc0GuWM8FuWA9FwBA0dBcAADR0VwAANFlnHMBAKAuOHMBAERHcwEAREdzAQBER3MBAERHcwEAREdzAQBER3MBAERHcwEAREdzAQBER3MBAERHcwEARHdMNBczW2tmA0t4/PVmdnGpjo/cMWaQC8ZLdVGai5mNNrN3zWy3mW1K/f1mMyvqQlW5MrPXzawq9eeAme1Py0/WcZ8zzGxy5Dr/j5mtS9X1ipmdEHP/pcCY8fYZfcyk7ft5M3Nm1qkQ+y8Wxou3z6jjxcxOM7NXzWxDaqy0j7HfvJuLmd0u6TFJj0g6WdJJkm6SdL6k42p4TsN8jxuDc26wc66Fc66FpBckTT2SnXM3hdubWaNi12hmPSRNkzRWyff3gKTfFLuOmBgzxZF6J9uxVMePhfFScIclzZc0IupenXN1/iOppaTdkoZn2e4ZSdNTL2C3pIGp5z4nabOkdZLuldQgtf1kSTPSnt9JkpPUKJUXS3pQ0l8l7ZK0UFKbtO2vSe1zq6SJktZKGliLGh8KHhuYeu49kjZK+p2kf5e0OG2bRqnaOkm6Wckv//2SqiTNTm2zXtJ4SR9J2inpRUlNavk9nirpubR8hqR/SGqWz8+uVH8YM4UfM6nnN5a0QlLPI8cq9c+e8VK+4yW1j6ap47SP8bPL98zlPElNJP2xFtuOkTRF0jckvSnpcSU//M6SLpL0I0nX5XDsMant2yl593KHJJlZNyWD7BpJp0pqLSmf07z2klpI6qDkB1sj59w0SS9J+oVL3plclfblH0r6NyWv9+xUfTKzhma2w8zOrWG3Zyr5JXHkGKuUvNP417q9nJJjzKQp0JiRktf2fyV9UudXUR4YL2kKOF6iy7e5tJG0xTl38MgDZvZW6oXsNbML07b9o3Pur865w0o67yhJdzvndjnn1kr6pVLfjFr6nXPuc+fcXkl/kNQr9fgISfOcc0udc/+QdJ+SX8Z1dVDSZOfc/tSx6uo/nXMbnXNbJc07Uq9z7pBz7gTn3Ds1PK+Fknci6f6u5B9QJWLM1F6dxoyZdZR0vZJ355WO8VJ7df0dUxD5Npetktqkf07onOvvnDsh9bX0/X+V9vc2St4JrEt7bJ2k03I49sa0v+9R8ktYSt5J/O+xnHO7U7XU1dfOuf15PP+ImurNpkrSvwSP/YuSU/VKxJipvbqOmV9LmuScq9Qxko7xUnt1HS8FkW9zeVvJ5/9X1mLb9PWUtyh5Z5E+2dhB0v+k/r5bUrO0r52cQ00bJJ1+JJhZMyWnrXUVrgOdrbbY60Z/ouRzc0mSmX1Lyc/t/0U+TrEwZgo/Zn4g6VEz26jks3hJes/MRkU+TjEwXgo/Xgoir+binNsh6X5J08xshJm1MLMGZtZLUvMMzzuk5DRzipl9I3UaP17SjNQmH0i60Mw6mFlLSXfnUNZMSZeb2QAzO07SA4p7Pc8KST3M7CwzO17SpODrXyv5zDOWGZKGmVl/M2uu5PW87JzbE/EYRcOYKcqY6azkI5FeSj57l6QhkuZGPEZRMF6KMl5kZk2VzG1JUhMza5Jp+9rI+xvinJuq5Id2p6RNSl74U5LukvRWhqfeqqRDf6Fk8u2/JT2d2ucbSiatPpS0XMnnh7Wt5xNJt6T2t0HSdv3z3VvenHOfSvqFkv9NskrS0mCT/5LU08y2m9nMbPtLTbZVmdl5NRzvQ0njJP1eyfe3iZLvXcVizBR8zGxKffa+Ucn3VpI25/l5fskwXgo7XlIfOe6VtCP10Gol37e8WOq/oAEAEM0xcfsXAEBx0VwAANHRXAAA0dFcAADRZbxJmpkx21/hnHNFvWssY6byFXPMMF4qX03jhTMXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQZb7lf6Zo3b+7lRx55xMs33nhjtecsX77cyyNHjvTyunXrIlUHAPUXZy4AgOhoLgCA6GguAIDozLmaVxmt9CVIu3bt6uWVK1dmfU6DBn6//dnPfublJ554Iv/Ciohljn19+vTx8iuvvOLlTp06FbGaxCWXXOLlcJx+9dVXxSyHZY6LbOjQoV6eO3eul8eNG+flJ5980suHDh0qTGG1xDLHAICiobkAAKKjuQAAoqtX17m0bdvWy88++2yJKkG5GjRokJebNGlSokr+KfzM/frrr/fy6NGji1kOCqx169ZenjZtWsbtf/Ob33j56aef9vLevXvjFBYZZy4AgOhoLgCA6GguAIDoKnrOJbwGZdiwYV4+55xz8j7GhRde6OXwOpgVK1Z4eenSpXkfE/E0auQP8SFDhpSokpqF97MbP368l8N75O3evbvgNaFwwt8p7du3z7j9iy++6OV9+/ZFr6kQOHMBAERHcwEAREdzAQBEV9FzLr/61a+8fPjw4ejHuPrqqzPmcH2XUaNGeTn8PB3F9b3vfc/L5513npenTp1azHKO6sQTT/Ryt27dvNysWTMvM+dSOY52HdXEiRNz2sfzzz/v5Uz3gywnnLkAAKKjuQAAoqO5AACiq6j1XObPn+/lwYMHeznGnMvWrVu9XFVV5eWOHTvmtL+GDRvmXVM+jrX1XLp37+7lxYsXezn8+Z599tleDn/exRDWOGDAAC+fcsopXt68eXNB62E9l3j69u1b7bH33nsv43MOHjzo5caNG0etKTbWcwEAFA3NBQAQHc0FABBdWV/nctFFF3n5jDPO8HI4x5LrnEu4FrUkLVy40Ms7d+708ve//30vZ/s/6z/96U+9PH369FxKRI7uvfdeL4f35br00ku9XIo5llatWnk5HOeFuF4LpTF8+PCcnxP+DqpUnLkAAKKjuQAAoqO5AACiK6s5l06dOnn597//vZfbtGmT0/7C+37NmjXLy/fff3+15+zZsyenfd5www1ebtu2rZfDe1c1bdrUy+H62AcOHMh4fPhGjBjh5XC9ltWrV3t52bJlBa8pm3CeLpxjCa972bFjR6FLQoGEa7cczf79+72c673HyhVnLgCA6GguAIDoaC4AgOjKas4lXO881zmWJUuWeHn06NFe3rJlS90KSxPOuTz88MNefvTRR70crsURzsHMnTvXy2vWrMm3xGPKyJEjvRx+v6dNm1bMco4qnEscO3aslw8dOuTlhx56yMvMw1WO/v37Z8xHE67P88EHH0StqVQ4cwEAREdzAQBER3MBAERXVnMuuQqvWbj++uu9HGOOJZtwziT8PL1fv34Fr+FY0rJlSy+fe+65Gbcvh3u5hddChXOJK1eu9PKiRYsKXhMKoy7/3sthjBYCZy4AgOhoLgCA6GguAIDoynrOpUGDzL3vu9/9bpEqqZmZv3x0WHO21zB58mQvX3PNNVHqqq+aNGni5dNOO83LL774YjHLqZUuXbpk/PrHH39cpEpQaH379s26TXivOOZcAACoJZoLACA6mgsAIDqaCwAgurKa0L/pppu8HC6iVI6GDh3q5d69e3s5fA1hDif0kdmuXbu8HN7kr0ePHl5u1aqVl7dt21aYwtK0a9fOy+GCZqE333yzkOWggAYMGODlMWPGZH3Ozp07vbx+/fqoNZULzlwAANHRXAAA0dFcAADRldWcSzh/UQ7atm3r5W7dunn5nnvuyWl/mzdv9jILQeVm7969Xg4XVxs+fLiXX3vtNS+Hi7nlqnv37tUe69y5s5fDxcGccxn3WQlzizi61q1beznbRdOS9MYbbxSqnLLCmQsAIDqaCwAgOpoLACC6sppzKUcTJ0708i233JLT89euXevla6+91stffvllnepCYtKkSV4ObyR62WWXeTnfG1sebQG6cE4lXAwsm2eeeSafklBC2a5hCm9SKUlPPfVUocopK5y5AACio7kAAKKjuQAAorNM/wffzDL/B/3IVq1a5eXw+oFQ48aNo9cwf/58L59xxhle7tChQ077W7BggZeLfS2Pc86ybxVPscdMNr169fJy165d89rfzJkzs27z7LPPenns2LEZt2/UqLymPos5ZsptvGTTvn17L69bt87L4XUuR1sI7qyzzopfWAnVNF44cwEAREdzAQBER3MBAERXVh/2htcoZLtPz+DBgzN+/be//a2XTz311Kw1hMfM975P5Xi/tGNJuN5LmAvhiy++yGn78H5lR/ucHuWhf//+Xs72O2rOnDmFLKesceYCAIiO5gIAiI7mAgCIrqzmXKZPn+7lqVOnZtx+3rx5Xs42P1KX+ZNcn/Pkk0/mfAzUL+HcYZhDzLFUjnD9llB477nHHnuskOWUNc5cAADR0VwAANHRXAAA0ZXVnMsrr7zi5QkTJng5XM++GMI171euXOnlG264wcsbNmwoeE0ob+H9+jLdvw+VZdCgQRm/Hq7PtHPnzkKWU9Y4cwEAREdzAQBER3MBAERXVnMu4doIo0eP9vKwYcO8fNtttxW8pilTpnj5iSeeKPgxUdmaNm2a8et79+4tUiXIV7hmVJcuXTJuv2/fPi8fOHAgek2VgjMXAEB0NBcAQHQ0FwBAdGU15xJaunRpxrxw4UIvh9echGupzJ0718vhei9S9ftAffrpp7UrFki57rrrvLxjxw4vP/jgg8UsB3kI7y24bNkyL4dr8axevbrgNVUKzlwAANHRXAAA0dFcAADRlfWcSzYLFizImIFSeO+997z86KOPennRokXFLAd5OHTokJcnTpzo5fC+ccuXLy94TZWCMxcAQHQ0FwBAdDQXAEB0lmmtCTNjIYoK55zLvIB7ZIyZylfMMcN4qXw1jRfOXAAA0dFcAADR0VwAANHRXAAA0dFcAADR0VwAANHRXAAA0dFcAADR0VwAANHRXAAA0dFcAADRZby3GAAAdcGZCwAgOpoLACA6mgsAIDqaCwAgOpoLACA6mgsAILr/D0ganRzqqR6MAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "# MNIST\n",
    "mnist_train_dataset = torchvision.datasets.MNIST(root='./datasets/',\n",
    "                                           train=True, \n",
    "                                           transform=torchvision.transforms.ToTensor(),\n",
    "                                           download=True)\n",
    "\n",
    "mnist_test_dataset = torchvision.datasets.MNIST(root='./datasets',\n",
    "                                          train=False, \n",
    "                                          transform=torchvision.transforms.ToTensor())\n",
    "\n",
    "# Data loader\n",
    "mnist_train_loader = torch.utils.data.DataLoader(dataset=mnist_train_dataset,\n",
    "                                           batch_size=batch_size, \n",
    "                                           shuffle=True, drop_last=True)\n",
    "\n",
    "# We use drop_last=True to avoid the case where the data / batch_size != int\n",
    "\n",
    "mnist_test_loader = torch.utils.data.DataLoader(dataset=mnist_test_dataset,\n",
    "                                          batch_size=batch_size, \n",
    "                                          shuffle=False)\n",
    "\n",
    "# let's plot some of the samples from the test set\n",
    "examples = enumerate(mnist_test_loader)\n",
    "batch_idx, (example_data, example_targets) = next(examples)\n",
    "print(\"shape: \\n\", example_data.shape)\n",
    "fig = plt.figure()\n",
    "for i in range(6):\n",
    "    ax = fig.add_subplot(2,3,i+1)\n",
    "    ax.imshow(example_data[i][0], cmap='gray', interpolation='none')\n",
    "    ax.set_title(\"Ground Truth: {}\".format(example_targets[i]))\n",
    "    ax.set_axis_off()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shape: \n",
      " torch.Size([128, 1, 28, 28])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEYCAYAAACQgLsAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5CU1Zk/8O9REeQ2XIaLgDPIVUUBkSAg4kZQC7NKGTCJbmGVumUZN1rZNbqRzf50iVoxGxM3pnR1jRo1USx/8QeiGJQK4uIFA0FQ5A7DxZmB4X6Tm+f3R7exn2/39OmePjPTDd9PFVU8093v+3a/Z/rMe573Ocd57yEiIhLTSc19ACIicvxR5yIiItGpcxERkejUuYiISHTqXEREJDp1LiIiEt0J0bk45zY458Y34/43O+f+rrn2L/lTm5F8qL2ki9K5OOe+55z70Dm33zm3Nfn/25xzLsb2G4tzbrZzbl/y3xHn3OGU+L8buM0XnHP3RTxG55z7P865jc65Pc65Pzjn2sbafnNRmzHbjN1mrnbOveec2+Wcq3bOPVHqbUbtxWwzdnvp6Zx7LdlWvHOuV4ztFty5OOfuBPBfAP4TQHcA3QDcCuAiAKfW85qTC91vDN77Cd77tt77tgB+D+DnX8Xe+1v5+c65U5r+KHETgO8BGAWgJ4D2SHzeJUttptG1A/AfAE4HMAjAmQB+1gzHEYXaS6P7EsAbACZH3ar3vsH/AJQB2A9gUuB5zwJ4PPkG9gMYn3ztcwC2AagC8BMAJyWffx+AF1Je3xuAB3BKMp4H4KcAFgDYC2AOgPKU509JbnM7gH8DsAHA+ByO8X762fjka6cCqAHwDIB/BDAv5TmnJI+tN4DbABwBcBjAPgCvJp+zGcC/AFgGYDeAFwG0zPEz/n8A/jklHgvgAIBWhZy75vqnNtP4bSbDcX4HwF+b+9yrvRR3ewHQKrmfXjHOXaFXLqMAtAQwI4fnXg/gAST+qvpfAI8icfL7ALgEwA0Absxj39cnn98Vib9efgQAzrlzkGhkUwD0ANAZQCGXeb0AtAVQgcSJrZf3/jEA0wE86BN/mVyT8vB3AFyGxPu9IHl8cM6dnBy+GFnPZl3yX2p8GoC+DXgvxUBtJkUjtRk2FsCn+b2FoqH2kqKJ2ksUhXYu5QDqvPdHv/pByljvQefc2JTnzvDeL/Def4lEz/tdAPd47/d67zcAeBjJDyNHz3jvV3nvDwJ4GcDQ5M8nA5jlvZ/vvT8E4N+RuOxrqKMA7vPeH07uq6Ee8d7XeO+3A5j11fF674957zt47z+o53WzAdzinKt0znUAcHfy560LOJbmpDaTu4a2mb9xzk1A4kvy3gKOozmpveSu4PYSU6Gdy3YA5anjhN770d77DsnHUre/KeX/5Uj8JVCV8rMqJHIKuapJ+f8BJHp+IPGXxN/25b3fnzyWhqr13h8u4PVfqe94Q/4HwCsA5iNxyTs3+fPNEY6pOajN5K6hbQYA4JwbDeB5AN/23q+NcDzNQe0ldwW1l9gK7VzeB3AIwMQcnps6/XIdEn9ZVKb8rALAluT/98P+Zd49j2OqBnDGV4FzrjUSl60NxdNGh44t6jTTyb86fuK9r/TenwFgBRINuybw0mKlNtPIbQYAnHPDkcjX3eC9nxd7+01I7aUJ2ktjKKhz8d7vQuKulMecc5Odc22dcyc554YCaJPldceQuMx8wDnXzjlXiUQy6oXkU5YAGOucq3DOlQG4J4/DegXA3zvnxjjnTgUwDXHreT4GMNg5d55z7jSkDzfUIjHmGYVzrtw51yd5S/K5AH6BxCV0STQwpjbTJG1mCBKJ7du892/E2m5zUHtp/PYCAM65VkjktgCgpXOuZbbn56LgD8R7/3MkTtrdALYi8cafAPCvAN7L8tLbkeih1yGRfPsDgKeT23wLiaTVUgCLkBg/zPV4PgXwT8ntVQPYiYhDSN775QAeROJukpVIDFelegrAEOfcTufcK6HtJZNt+5xzo+p5ShcAbyLxWc0C8IT3/umGHn8xUJtp9DbzIyT+kn42pabi44a/g+al9tK47SU55HgQwK7kj9Yg8bkVxJXoH8AiIlLETojpX0REpGmpcxERkejUuYiISHTqXEREJLqsk6Q555TtL3He+yadNbbY20y7du1MPGLECBPPnTsXhRo2bJiJ9+3bZ+JVq1YVvI/G1JRtprnbC0+qzDc4jRs3zsR33HGHiZcsWWLi7t1tScqaNWvS9tm2ra1t7Nixo4mPHDli4j597F3H11xzDYpJfe1FVy4iIhKdOhcREYkua51Lc1+ySuGO92GxVq1amfiHP/yhia+77joT8xBEly5dTHzgwAETd+rUKe9j+uKLL0x88KCdi/DYsWMmfuedd0z81FNPmfjNN9/M+xgKcSINi510kv37+ssv7fyT7777ronHjBmT1/b37NmT9rPWre2cs6ecYrMT3Ab5+VdddZWJZ83Kuf6zUWhYTEREmow6FxERiU6di4iIRKecy3HueMu5PPTQQya+5ZZbTMy3GnO+g2O+7fO0004zcYsWLUx88snpS7MfPmyX4uAxcx7Xb9nSTjjL++R9vP/++yYeO3YsGtOJlHMJ2bt3r4n5XNfV1Zk4lE8B0tvg0aNHTcy3R/fr18/Ed911l4l/8YtfpO2jKSnnIiIiTUadi4iIRKfORUREoss6/YtIc+Ocyt13323imhq72jNPtRJy6qmnmphrVDjOlKPk2gjO0zDeJh8z18GMHj3axK+99pqJue5B4uGpWjjH0r59exNzfu3QoUNp2+ScGufgMr0m1RlnnJH18WKhKxcREYlOnYuIiESnzkVERKJTzkWK2k9/+lMT81xNnO/gugKeAp3t3Lkz6/a4BqFNmzZp2+D5zbZv325iHmPnnAqPuXOdQ21trYm5zqW8vNzEnBeQ3HXr1i3r41wXxTk4zrlkqoviNsVtjrfJbb5r165Zj7FY6MpFRESiU+ciIiLRqXMREZHolHORolZWVmZirgHgMW7OsTz22GMmfvLJJ028aNEiE1dXV5u4V69eJua5pgBg48aNJuYxcZ6P6vTTTzfx5s2bTczvkWspeC4yXgZXOZeGO/fcc7M+HpqLjvNpHAPpbZZxnobbA+fYipWuXEREJDp1LiIiEp06FxERiU45FylqXAPC83JxTQibOnWqiXfv3m1iHt/m9TjmzZtn4m9+85tZ9wcAy5cvN/HZZ59tYs6h3HHHHSa+//77Tbxt2zYT85j9RRddZOKFCxcGj1EyGzx4sIk5X8btj9sLt1c+1wCwY8eOrMfAbZq3uX///qyvLxa6chERkejUuYiISHTqXEREJDrlXCLjMfzQvEEstLYDr6e9Zs2afA+xqPH6Kow/T/682HPPPWfiiRMnZn1+p06dTMw5lmnTpqW9hud+uu6667Jus6KiwsTTp083MedcOMfCtRPnn39+2jFJw4wYMcLE3N44x8LzhHFd1uLFi9P2MXToUBPz/Hb8O8/73LRpU9o2i5GuXEREJDp1LiIiEp06FxERie6Eyrnw/eOZaiR4jLVnz54mHjVqlIlnz55t4kLvQQ+tnz1p0iQTP/TQQwXtr9j06NEj6+N8fnhuJ8bnL+Taa6/N+jjncID02gfOu3388ccm5rnF9u3bl88hpunfv39Br5evcU0SzyXG7a9t27Ym5rnpRo4cmbaP0BowHPMaRaE6mWKhKxcREYlOnYuIiESnzkVERKI7oXIujMdPM7n44otNfOGFF5qYcwS//vWvCzomXgvkiiuuMDHXVBxv8l2rokWLFibmMXLOuYTW0njnnXeyPv6nP/0p7We8nsr27dtNfOWVV5r4z3/+s4k5J8M5GD5mrq3gNWyk4bhOJbTePedc/vjHP+a9T87RZVoDJlWoFqxY6MpFRESiU+ciIiLRqXMREZHoTqicC49t8ngqAAwfPtzEfN97bW2tibnG4NVXXzUx35POdRlVVVUm7ty5s4l5PQheb/14w2vWs9D6LQcOHDAx5yN4zJy3N3DgQBP/7Gc/M3Hfvn2z7h8APvvsMxOfddZZJq6srDTxbbfdZmKupeI2xGuM5FvLI/XjnCe3p9DcgC+++GJwH1zLxnPPcc6O8VxjxUpXLiIiEp06FxERiU6di4iIRHdc51xC9QFt2rRJew3PLcXjo61atTJxu3btTMxj+HwM/PigQYNMzGs18FoPPM/Q8aZLly5ZH+ecCefROOaakQceeMDEXCdz+eWXm3jIkCEmPvfcc9OOidsA51g4b8Prt/D6Hiy0RhC/B2k4zmdw+wn9/nENUybvv/++iTnHxuebhXIyxUJXLiIiEp06FxERiU6di4iIRNesA/icfwitc8CPc5zvHD233npr2s9qampMzGt19O7d28Scg+E6mNB4Oa//wjUMXOfCa8Zz3qjQ9WSaG691wvjz4zbC+Yfdu3ebeOrUqVm3z8/n83nOOedkfT2Q3oY4j8RtioXadWhOvHx/DyR33L44jxtajwkANmzYYOIxY8aYOFTLxW20WOnKRUREolPnIiIi0alzERGR6Bo15xLKqYTm6Yk9tnzdddeZONM6GIsXLzYxj7F26NDBxHzPOc8DxeuTcE1E6J52zinwffg8t9mSJUuybq/YhepcGOeo5s6da+KxY8eamOdm4zbDa2VwXcPevXuDx8RthnMwnKfjbfKYOtfBhOocOC+4du3arM+X+vF3FJ/bhny23AZDueVSpSsXERGJTp2LiIhEp85FRESia9ScS2jskMcaOebxcN5eKMdy4403mpjX6uB5vID0HAnnjXg9li1btpiYcyqcN+L1IXj8PZSnYldccYWJSz3nwjktxmuW8/j17373OxPz+vX8+bPQXHC5zO0WGqfnWiWulXjmmWdMHJp7jHEbVs6l4Y4cOWJiriv75JNP8t7m66+/buK7777bxNwGS9Xx8S5ERKSoqHMREZHo1LmIiEh06lxERCS6ghL6ocQTJzY5OcrJ7lDRJOvRo4eJv/3tb5uYk++rV682MSeHgfRka+fOnU3MRXv8HrnIkfFNCDzRHT/OE1HyZ3TRRRdl3V+p6dSpk4lDn++2bdtMzIurMT5/nGyPUcAWmniSH+fCzQ8//DCv7R88eNDEoYkPJXehIuf169fnvc2lS5eamM9/aPG3UpmcVlcuIiISnToXERGJTp2LiIhElzXnEpoYMt8cSWg8myctrKysNPFZZ51lYl5YisfT9+zZY2Iu0OOFuIBwwRu/Zz5Gfv2uXbtMzEVZocWveDydzwlPejho0CCUMj5HnJPiotN9+/aZ+Oyzz866/dBElawhOZh8J2zl9xzaJ2+f20y+k3/K17gol3N8fG4+//zzvPfBRbMslOdRzkVERE5Y6lxERCQ6dS4iIhJd1pxLaGLIbt26mZjzDzzJG8dch3LmmWeamMc7OV/B4+089lxWVpZ1f5nGPnmfPNEh5wB4zL66ujrrMfD2uS6Da286duxoYh5v5QXPuC6n1IRqQtjKlStN3Ldv36zP5+2FFmpqSM1IqM6F2xC3ka1bt2bdPm+Pj5EnrpTc1dbWmpjbE3/2AwYMyHsfnBtmoe/dUC1dsdCVi4iIRKfORUREolPnIiIi0eU1t9j48eNNzHN7cU6ka9euJubxba7x4NdzDQfnIzjfwGPPXKPC+Y1Mc6PxPniMlXMefIy7d+82MX8GIXyM/Blx3ohzPqF76IsdL8YVGn9etWqViceOHZvX9hm3oXwXb8v0Gm5noXPEtRYch/JqvGCd5O6jjz4yMddNcb5syJAh0Y+Bv7cYH0Ox0pWLiIhEp85FRESiU+ciIiLRZR2Avvzyy0188803m3jFihUm5hoPntuL8xd8v3doTh3Ob3C+gcfnee4wHgvn/AWQnuPgucI4z8O1Pjy3F78+33mD+J72L774IuvzQzUSxY7nUgvlXPh88fxznMcLrUHUEKFaGT7G0Hvq16+fiWtqakzMbZB/j0qlDqIYzZ8/38Q33nijibk9DRs2rOB9cnsIfUeE2k+x0JWLiIhEp85FRESiU+ciIiLRZc25LFy40MQjR4408XnnnWfi0HrufH8/51B27NiRNeYaEs65cE6F6wEGDhxo4kxj05yn4fFzvq+d18PesGGDibk2iO9hD9VN8Ge2ZcsWE3Nei+t0Sk2+489ct8LnnOeGC20vpCHru3DOJXQMEydONDG3qfPPPz/r9nk+Osnde++9Z2LOcfLvY4wcJ38PhuazK7QNNxVduYiISHTqXEREJDp1LiIiEl3WnAuv/z5t2rSsG+Px/gsvvNDEvPbB6NGjTdy7d28TDx482MS8Hkxo3icei+YczrJly8DeeustE8+ePdvEPAYbMnPmTBNXVFSYuK6uzsQ8/soxj/nyPEOrV6/O6/iKDedcWrVqlfX5PPcT5+H48+EcDbeR0Hh3psfznX8sNGbOvwec15s8eXLW13NtleSuqqrKxJzT5Jwpt88+ffqYeN26dcF9cu1MaP475VxEROSEpc5FRESiU+ciIiLR5bWeSwivaT937tys8eOPPx5z90Xp6quvbu5DKCk8T1YoB8I1HTxfHG+Pcyws9HimfAr/jOPQGjFcvzVq1CgT85o1of1nmjNPGoZzLJzv4BxfQ3IuPCcj59w4V9wY8+M1htI4ShERKSnqXEREJDp1LiIiEl3UnItIofief17fhWupHn74YROPGzfOxJx/yHctjFA+Bch/LqjQukPz5s0z8axZs0x87733Zt0e5wGkfqEapVdffdXE119/vYk5/zFmzBgTv/3228Fj4DWZQsfI9YfFSlcuIiISnToXERGJTp2LiIhEp5yLFBVeY4fzCZyT4fwCz9XWv39/E69du9bE+dYMhPIrmZ7DtTM8P1ynTp1MzGuE8Hti/BlVVlYGj1ESQjmXGTNmmPiGG24wMbfHSZMmmfi+++4LHgPPJRbK8+U7v2Fz0ZWLiIhEp85FRESiU+ciIiLRKeciRYXXMOd5tni8mefd4jWDjgc8XxWv8cPzX3300UeNfkzHC865cX6M13PauXOnifmzD81Nl8knn3xi4vPOO8/EXOvVo0ePvPfRHHTlIiIi0alzERGR6NS5iIhIdMq5SFFZuHChibnuJd/1WY4HLVq0MDGP83OtD6+rJPXLd665jRs3mnjkyJEmbtOmjYlHjx6dtg3OK/Lcc61atTIxn//y8vLcDraZ6cpFRESiU+ciIiLRqXMREZHolHORorJ582YTL1682MRc5xJaC4PnbeIx9lzmCmtsfAx8jGvWrDHx66+/buKysjITf/DBBxGP7viWaX2ebJ588kkTr1ixwsQvvfSSiTm/ksnzzz9vYj6fXNf07rvvBrdZDHTlIiIi0alzERGR6NS5iIhIdC7fMUcREZEQXbmIiEh06lxERCQ6dS4iIhKdOhcREYlOnYuIiESnzkVERKJT5yIiItGpcxERkejUuYiISHTqXEREJDp1LiIiEt0J0bk45zY458Y34/43O+f+rrn2L/lTm5F8qL2ki9K5OOe+55z70Dm33zm3Nfn/21wxrMSUhXNutnNuX/LfEefc4ZT4vxu4zRecc/dFPMaezrnXnHPVzjnvnOsVa9vNSW3GbDNqm0lus6tz7kXn3G7n3E7n3HMxt9/U1F7MNkviO6bgzsU5dyeA/wLwnwC6A+gG4FYAFwE4tZ7XnFzofmPw3k/w3rf13rcF8HsAP/8q9t7fys93zjXHyp1fAngDwORm2HejUJtpEjMAbAJwBoCuAH7VTMdRMLWXRtc43zHe+wb/A1AGYD+ASYHnPQvg8eQb2A9gfPK1zwHYBqAKwE8AnJR8/n0AXkh5fW8AHsApyXgegJ8CWABgL4A5AMpTnj8luc3tAP4NwAYA43M4xvvpZ+OTr50KoAbAMwD+EcC8lOeckjy23gBuA3AEwGEA+wC8mnzOZgD/AmAZgN0AXgTQMs/PulVyP70KOWfN/U9tpvHbDIArAaz96rMp5X9qL6X7HVPolcsoAC2R+Csp5HoADwBoB+B/ATyKxMnvA+ASADcAuDGPfV+ffH5XJP56+REAOOfOQaKRTQHQA0BnAIVc5vUC0BZABRIntl7e+8cATAfwoE/8ZXJNysPfAXAZEu/3guTxwTl3snNul3NuZAHHWErUZlI0UpsZCWAlgBecc9udcwudc2MKeD/NSe0lRSl9xxTauZQDqPPeH/3qB86595Jv5KBzbmzKc2d47xd4779Eouf9LoB7vPd7vfcbADyM5IeRo2e896u89wcBvAxgaPLnkwHM8t7P994fAvDvSFz2NdRRAPd57w8n99VQj3jva7z32wHM+up4vffHvPcdvPcfFLDtUqI2k7uGtpleACYg8dd2dySGlGY65zoVcCzNRe0ld0X1HVNo57IdQHnqOKH3frT3vkPysdTtb0r5fzkSfwlUpfysCkDPPPZdk/L/A0j0/EDiL4m/7ct7vz95LA1V670/XMDrv1Lf8Z5o1GZy19A2cxDAGu/9s977I9773wOoReIqoNSoveSuqL5jCu1c3gdwCMDEHJ6bup5yHRJ/WVSm/KwCwJbk//cDaJ3yWPc8jqkaiSQmAMA51xqJy9aG4nWgQ8emdaOzU5tp/DaztBG22VzUXkr0O6agzsV7vwvAfwB4zDk32TnX1jl3knNuKIA2WV53DInLzAecc+2cc5VIJKNeSD5lCYCxzrkK51wZgHvyOKxXAPy9c26Mc+5UANMQt57nYwCDnXPnOedOA3AvPV6LxJhnNM65VkiMOwNAS+dcy2zPL2ZqM03SZv4vgG7OuX9Ijrd/F0AXJL6oS4raS+l+xxT8gXjvf47ESbsbwFYk3vgTAP4VwHtZXno7Ej30OiSSb38A8HRym28hkbRaCmAREuOHuR7PpwD+Kbm9agA7kbiTIgrv/XIADyJxN8lKAPPpKU8BGJKsLXgltL3kL/8+51zGIYvkcMBBALuSP1qDxOdWstRmGrfNeO/rkPhL/x4k7hz6EYCrvfc7Gv4umo/aS2l+x7jkLWgiIiLRnBDTv4iISNNS5yIiItGpcxERkejUuYiISHRZJ0lzzhVVtr9Lly4mvuWWW0y8e/duEx88mL3YlZ8PAHyDw8kn2/nvTj3VzpO3detWE8+bN8/Ehw/HqI1qOO99k84am2+bOekk+/fNl1/aQufQpLeF3pAycqSdEaNNG3t3K59vbg+ZtGxp7+Lctm2biefP55t/iktTtpli+45h/Pt89OhREx86dMjErVq1MvGGDRvStsnP6datm4n37dtnYm5z/DvzrW99K20fTam+9qIrFxERiU6di4iIRJe1zqXYLlm///3vm/hXv7JLVOzYYWvEqqurTdynjy1q3bw5ve5p9erVJj777LNN/MUXX5j47bffNvHSpUtN/Pzzz6ftoykV+7BYvms9hYbB2rVrZ+JLL73UxMOGDTPxhAkTTLxy5cqs+2vbNn26ps6d7cwfdXV1Jj7ttNNMzMMcr732molnzpxp4o0bN6btszGdyMNi7du3N/HatWtNzMPgrHXr1ibmISwg/Tvk2LFjJj5w4ICJeZiVj2HcuHFZj6mxaVhMRESajDoXERGJTp2LiIhE11zrezdI165dTcy3+fHYJeMcTKbbSnn8nMdg9+zZY+IePXqYeMWKFVmPQSzOaXAOJpRj4dvRBwwYYGI+x3x+pk+fbuKhQ4eamG81PeWU9F8ZztNwG+ExdL6lvrKy0sS//OUvs77+xz/+sYk///zztGOShuHbhLn98fnnUgOOd+7cmbYPbpP8HcP75O+5UIlFsdCVi4iIRKfORUREolPnIiIi0ZVUzoXzITytBtexcN0L10DwNAsA0KFDBxNzDoC3wdOVLFu2LG2bUr98cyxc68Rtgsenjxw5YmKuO+CagXfeecfE11xzjYlramrAOC/D74HbBNfWrFq1ysQ8LRHnZO6//34T33TTTWnHJA0zadIkE3fq1MnEmzZtMjHnYLh9cdvI9BzO8/A2y8rKTHz66aeb+IILLjDxokWL0vbZHHTlIiIi0alzERGR6NS5iIhIdCWVc6mqqjLxkCFDTMz5D465XiDTdPg8Hspj7DwGy89XnUt+QjmXM844w8QVFRUmXrdunYkzzf2Vav/+/Sbm6c55Linefv/+/dO2uX37dhMvXLjQxGPHjjXxli1bTMxj7jwXGdc1dO/e3cRTpkwxMc9nl29e60R28803m5hr4zjPy7V3PCV/r1690vbB30P8PcVzj/E2uc2OGDHCxMq5iIjIcUudi4iIRKfORUREoiupnAuPTfLaKTyezmPNffv2NXHHjh3T9sGv4fVdGI/J8/ioZMfnlPXr18/E/PlyTQDXLvFaGDyvEz+f65zeeOMNEz/44INpx8g5ET4mjmtra03MSyvzXFO81DLXTpx//vkm5pyLciy5GzhwoIk5f8H5sBYtWpiYc7D8nQSkn0/GdU4c8+8Mz29YLHTlIiIi0alzERGR6NS5iIhIdCWVc+Gx482bN5t4+fLlWV8/efJkE/O8VAAwaNAgE8+fP9/EPAbLNQs8nsr3tEt++HxwDQDnVBiPeXPOhdcA4nwH1znMmTMnbR+cB+JtrlmzxsSc1+O6Fc7RcB0M+8Y3vpH1cakfz9PFnz3PPcd1LfydxLVzXKcFpLdhzvtxHoePiV+faf6yYqArFxERiU6di4iIRKfORUREoiupnMtnn31m4nHjxmV9nMciOSfDc0ABwBNPPGFiXr+B8zy8RnaprG9dKnhuJr7nP5Rz4THz1q1bmzi0JjrnfLi2Ckifb47XtOc6BK6l4bmiOM/Dx7B+/XoT87pFnPfLNIeeJPBnn6kuJRXny/j3nfO4f/nLX9K2ce6555qY65z27t1rYq6d4Rwf52CKha5cREQkOnUuIiISnToXERGJrqRyLjxezuOjXC/A+RDG4+1A+hg+j3eG1lrgmoRivQe9WPEYOOP1Wnh+OM6JHDlyxMRc58J43iY+f5nmo+McB4/Lczvj2greB2+PczSM2+jgwYNNnGncXxJ4LjH+fQ7lYLjOhc8tz40HAH/9619NPGDAABNv3LjRxNyGuY6qWL9jdOUiIiLRqXMREZHo1LmIiEh0JZVz4fFPzsGE1jngsW8e+5Z6yVQAAAqlSURBVATSx1BD6zfwGD6Pj0p+zjzzTBOH1mfhGgE+f1yDwucvNG8X5zN4vBtIb3ddunTJuk1+D9wuuV1z3QO/nvME/Bkq51K/s846y8T8HcPtK7SefV1dXXCfH3zwgYmHDBliYm5PfL5D85kVC125iIhIdOpcREQkOnUuIiISXUnlXHhtFB6b5PF5xo8vWbIkuE/OuYTWUlDOpTAVFRUm5s+bcyCh11dVVZmYx6c5Z8Yxtxkec8+0T94Gv4bbTKgOhts9tzGOuW5C6sd1KDx3Hdcc8bnkvO6zzz4b3Odvf/tbE996660mDtVihdYPKha6chERkejUuYiISHTqXEREJLqSyrlwjoXHmvn+b45DORkgfX0GHnPl++BLZfyzVPAYNn+ee/bsMTHXALRv397E3GY4v8Hb5/FubkOZ1o/h13BdCs9HxnkkzuvxeywvLzfxrl27TMx5qKFDh6Ydo2TG7YV///n8c/vhuqlHHnkkuE+uO+I2Glq/hfOGxfqdoysXERGJTp2LiIhEp85FRESiK6mcC8/bw+OhPFbJ+ZJc1prmvAyvzcHb2LJli4l5/FTyw+u18Pgyr9HDNSYzZszIuj1uM5y345wKxzzGnmkbPC7P85dxG+E2tWLFChNfffXVJg7NLRWaL02+xueOc6r8WfO8bzU1NSZet25d3sewfft2E/N3Drd5zsEV6/nWlYuIiESnzkVERKJT5yIiItGVVM6lurraxJxTYTw+mmm8nPF4OY/Bcg1CaB4gyQ/nOLjugO/55/Hp5cuXm/jiiy82cajWiWsGeP16Hv8G0sfl+Rh5XJ+Pma1atcrE3I759TxXGR+z1G/Hjh0mDn1HcA7vzTffLPgYOG/DOblt27aZmOumivU7SFcuIiISnToXERGJTp2LiIhEV1I5F17XgmPOj3DdC6+nnglvg3MAXJPA96hLfjjHxXm00Hgy5zM+//xzE4fyGzyvF+dceA31TOc7NKddKOfC73H16tUm5pwLt2v+DPmYOU+Qyxx7J4rQPHD82fbt29fEd955Z9btZ1p/iHMq69evN3HPnj1NzPV9fEy9evXKegzNRVcuIiISnToXERGJTp2LiIhEp85FRESiK6mEPidbOTEZSnRyMVImnEzlhC8nnIt10rhSwZPwcbI7tFgTT9rIj3PMBY58wwYX1fFNI5mK7LiNbN261cTcbvk98uNcLBxaDCq0wFX37t1NvGbNmqzbO5GEJv3kmyH4O4aLdlmmG1I4of/pp5+a+MwzzzQxF2536dLFxJkKe4uBrlxERCQ6dS4iIhKdOhcREYmupHIujMe/QwVQuYxN8hgqFyi1b9/exDwmL/nhSRb5nHHRKj9/06ZNJuaiOC4o5EkCeX88ps75jkw5tlAhJud5eJ88rs8x53B4zD70Hrp27Wpi5Vy+tnTpUhOPGDHCxJyT45wstyeWy+KBr7/+uolvv/12E3Mb7tatm4mLtZBbVy4iIhKdOhcREYlOnYuIiERX0jmXzp07m5jHQ6+88koTP/HEE8FtLl682MQ8Brt582YTF+tCPaWCazK4dokXwhowYICJV6xYkfX1nO9gfP44j8fHxzkgIL3OhPMymSYvTMUTqvLkqcuWLTNxu3btTMy5RB7n5xyOfO3ll1828U033WRizp9xzvXSSy818Zw5c0wcmjgVAFauXGli/o7h88ntiY+pWOjKRUREolPnIiIi0alzERGR6Eo653LJJZeYmBfymTBhgomnTJkS3OYnn3xiYh4P/8EPfmBivk9+0aJFwX3I1zhvxjkSriHhOhf+/HneJc5PMK4R4boGzn9kmueLx8T5mDmvw4uH8esrKipMvHbtWhOPHj066/44D1WsY/LFgM8nnxvOV3H75O8UzrmEcn5A+mJgXMdSWVmZ9Zgy5QGLga5cREQkOnUuIiISnToXERGJrqRyLnzPOI9l9+/f38Q8h1IuY5M8RlpWVmbiCy+80MSZ1veQ3A0bNszEnD/gmMejucZj+PDhJua53zi/wTG3KV7vI1NdE/+M6xC4VodjbnNDhgwx8e7du00cqqvhuaj4M3nllVcgmXE+g9sff4dwHVwMfD75d4TXlOJjLBa6chERkejUuYiISHTqXEREJLqSyrnwPE+hsUce284F51C4DoJzMPy45IfrSHi8uWfPnibmupUlS5aYeOjQoSbetWuXiVu3bp31eDivx3UvmXIuXCvB74nzNpxj4bxP7969TTxz5kwTP/300ybm+bF4/9XV1WnHLJktWLDAxNdff72Jee0UnssuhqqqKhNzrR1/74XmrmsuxXlUIiJS0tS5iIhIdOpcREQkupJOGPBYNs+hxGPPueDxcB5P55xMaA1tye6ZZ57J+jjXHfTp08fE69atM/GkSZNMzHUwvD0er+YcTXl5uYkz1TWF8jKcC+Tc4bZt20w8cuRIE/M6RDx/Go/7F+tcU6XgN7/5jYknT55sYs6P8Vx3ofaZi71795qY84zcvriNFwtduYiISHTqXEREJDp1LiIiEl1J51xCcyw1ZOyZ8zhc98Bj9Lz+g8TF+QRev4XHo3l9mB07dpiY65Jqa2tNzPkR3l6mNdG5zXBOhXMyoforrsXhucZmz56d9fXScFu2bDEx5+B43jauOeG5xhqSc+H20bFjx6z75PZVLHTlIiIi0alzERGR6NS5iIhIdCWdc+nevbuJQ+tq5ILH+EPrfXDeRwoTynFx3dGYMWNMHMqB8fni7ffr18/E69evz7o9IH2NGX4PnAvkNWb4mHjc/5JLLjEx51x4f5zzkfqFPrs5c+aYmOteON82ceJEE7/00kt5HxPX53Eb5ThTHrAY6MpFRESiU+ciIiLRqXMREZHoSjrnwjUKXbt2NTHPE5YLnqeHx/j5nvKtW7fmvQ+pH4958+fPBg4caGJeb55rAnh7AwYMMPGGDRtMzOPfPXr0SDsGzqnwmDjXzvAYOY/bc8y5RcafmXIwuQvl9N544w0TX3vttSbmfFmvXr0KPqZQG+baLa7FKha6chERkejUuYiISHTqXEREJLqSzrnweOjw4cNNzDUqueC1FPbs2WNiHl/nMXqJi+uKeEy8srLSxDw+vXr1ahNzm1i5cqWJeTz7nHPOyfp6IH2NFz5GblOhMXXO6/FcY6G5ypRzyV3oO2LBggUm5hqksrIyE3N+jOeFA4CPP/446z75O4fPP+eStZ6LiIicMNS5iIhIdOpcREQkupLOufB6LZwPCdVI5IJrFHg9Bx6DlbhC+YKpU6ea+K677jLxhAkTTMxrnvPcYTw3GZ9/Xu8eSF9vg9eY6dSpk4l5LjLOwdTV1Zn40UcfNXFoPZiG5BpPVPnmozZu3Gjiq666ysScD7nsssvSthHKuXD74TbIuD0VC125iIhIdOpcREQkOnUuIiISXUnnXJ5//nkTX3zxxSaOsdb4zJkzsz6+bNmygvch9QvlD3hup2nTpmV9fkVFhYm5joXHr9u3b2/iXNYI4rnBeByex+25loLXFJLi8cADD5i4pqbGxHzu582bl/c+pk+fbmKeQ3HXrl0mnjt3bt77aAq6chERkejUuYiISHTqXEREJDqneYdERCQ2XbmIiEh06lxERCQ6dS4iIhKdOhcREYlOnYuIiESnzkVERKL7/w23te2HX7uCAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Fashion-MNIST\n",
    "fmnist_train_dataset = torchvision.datasets.FashionMNIST(root='./datasets/',\n",
    "                                           train=True, \n",
    "                                           transform=torchvision.transforms.ToTensor(),\n",
    "                                           download=True)\n",
    "\n",
    "fmnist_test_dataset = torchvision.datasets.FashionMNIST(root='./datasets',\n",
    "                                          train=False, \n",
    "                                          transform=torchvision.transforms.ToTensor())\n",
    "\n",
    "# Data loader\n",
    "fmnist_train_loader = torch.utils.data.DataLoader(dataset=fmnist_train_dataset,\n",
    "                                           batch_size=batch_size, \n",
    "                                           shuffle=True, drop_last=True)\n",
    "\n",
    "fmnist_test_loader = torch.utils.data.DataLoader(dataset=fmnist_test_dataset,\n",
    "                                          batch_size=batch_size, \n",
    "                                          shuffle=False)\n",
    "\n",
    "# let's plot some of the samples from the test set\n",
    "examples = enumerate(fmnist_test_loader)\n",
    "batch_idx, (example_data, example_targets) = next(examples)\n",
    "print(\"shape: \\n\", example_data.shape)\n",
    "fig = plt.figure()\n",
    "for i in range(6):\n",
    "    ax = fig.add_subplot(2,3,i+1)\n",
    "    ax.imshow(example_data[i][0], cmap='gray', interpolation='none')\n",
    "    ax.set_title(\"Ground Truth: {}\".format(example_targets[i]))\n",
    "    ax.set_axis_off()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/broadcasting.png\" style=\"height:50px;display:inline\"> Multi-Layer Perceptron (MLP) in PyTorch\n",
    "* Other names: Fully-Connected (FC) Network (FCN), Dense Network\n",
    "\n",
    "An MLP is composed of one input layer, one or more hidden layers and a final output layer. Every layer, except the output layer includes a bias neuron which is fully connected to the next layer. When the number of hidden layers is larger than 2, the network is usually called a deep neural network (DNN).\n",
    "\n",
    "The algorithm is composed of two main parts: forward pass and backward pass. In the forward pass, for each training instance, the algorithm feeds it to the network and computes the output of every neuron in each consecutive layer (using the network for prediction is just doing a forward pass). Then, the output error (the difference between the desired output and the actual output from the network) is computed.\n",
    "\n",
    "After the output error calculation, the network calculates how much each neuron in the last hidden layer contributed to the output error (using the chain rule). It then proceeds to measure how much of these error contributions came from each neuron in the previous layers until reaching the input layer. This is the backward pass: measuring the error gradient across all the connection weights in the network by propagating the error gradient backward in the network (this is the backpropagation process).\n",
    "\n",
    "In short: for each training instance the backpropagation algorithm first makes a prediction (forward pass), measures the error, then goes in reverse to measure the error contribution from each connection (backward pass) and finally, using Gradient Descent, updates the weights in the direction that reduces the error.\n",
    "\n",
    "<img src=\"./assets/tut_xiv_mlp.jpg\" style=\"height:200px\">\n",
    "\n",
    "#### Implementing an MLP\n",
    "\n",
    "We will now see how easy it is to implement a neural network using PyTorch. The first thing to know is that every neural network (NN) is inherited from parent class `torch.nn.Module`, which establishes a shared workflow, that is, every (!) NN in PyTorch has the same properties in terms of functunality. This global design helps with building a common language. We will now meet PyTorch's basic blocks and see why it is so popular. \n",
    "\n",
    "We will implement the following network:\n",
    "* Input dimension: 28 * 28\n",
    "* Output dimension: 10 (as the number of classes)\n",
    "* Batch size: 128\n",
    "* Hidden Layers: 1\n",
    "* Hidden Units: 256\n",
    "* Optimizer: Adam (Learning Rate: 0.001)\n",
    "* Activation: ReLU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a two-layer MLP\n",
    "# method 1\n",
    "class TwoLayerNet(torch.nn.Module):\n",
    "    def __init__(self, D_in, H, D_out):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate two nn.Linear modules and assign them as\n",
    "        member variables.\n",
    "        Parameters:\n",
    "            D_in - dimensions of inputs\n",
    "            H - number of hidden units per layer\n",
    "            D_out - dimensions of outputs\n",
    "        \"\"\"\n",
    "        # initialzing the parent object (important!)\n",
    "        super(TwoLayerNet, self).__init__()\n",
    "        # define the first layer (hidden)\n",
    "        self.linear1 = torch.nn.Linear(D_in, H)\n",
    "        # define the second layer (output)\n",
    "        self.linear2 = torch.nn.Linear(H, D_out)\n",
    "        # define the activation function\n",
    "        self.relu = torch.nn.ReLU()\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Tensor of input data and we must return\n",
    "        a Tensor of output data. We can use Modules defined in the constructor as\n",
    "        well as arbitrary operators on Tensors.\n",
    "        Parameters:\n",
    "            x - tensor of inputs (shape: [BATCH_SIZE, D_in])\n",
    "        \"\"\"\n",
    "        h_relu = self.relu(self.linear1(x))\n",
    "        y_pred = self.linear2(h_relu)\n",
    "        return y_pred\n",
    "    \n",
    "# method 2\n",
    "class TwoLayerNetPiped(torch.nn.Module):\n",
    "    def __init__(self, D_in, H, D_out):\n",
    "        \"\"\"\n",
    "        In the constructor we instantiate two nn.Linear modules and assign them as\n",
    "        member variables.\n",
    "        Parameters:\n",
    "            D_in - dimensions of inputs\n",
    "            H - number of hidden units per layer\n",
    "            D_out - dimensions of outputs\n",
    "        \"\"\"\n",
    "        # initialzing the parent object (important!)\n",
    "        super(TwoLayerNetPiped, self).__init__()\n",
    "        # Create a pipeline - a sequence of layers\n",
    "        self.pipe = torch.nn.Sequential(\n",
    "            torch.nn.Linear(D_in, H), \n",
    "            torch.nn.ReLU(),\n",
    "            torch.nn.Linear(H, D_out))\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        In the forward function we accept a Tensor of input data and we must return\n",
    "        a Tensor of output data. We can use Modules defined in the constructor as\n",
    "        well as arbitrary operators on Tensors.\n",
    "        Parameters:\n",
    "            x - tensor of inputs (shape: [BATCH_SIZE, D_in])\n",
    "        \"\"\"\n",
    "        return self.pipe(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### When to use method 1 and when to use method 2?\n",
    "If you want easy access to the weights of the layers (to do some manipulating on them or create your algorithm) then it is better to use method 1, otherwise, use method 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyper-parameters:\n",
    "num_epochs = 10\n",
    "learning_rate = 0.001\n",
    "\n",
    "# Device configuration, as before\n",
    "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "# create model, send it to device\n",
    "model = TwoLayerNetPiped(D_in=28*28, H=256, D_out=10).to(device)\n",
    "\n",
    "# Loss and optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch [1/10], Step [100/468], Loss: 0.4783, Time: 1.8672 secs\n",
      "Epoch [1/10], Step [200/468], Loss: 0.1721, Time: 3.2621 secs\n",
      "Epoch [1/10], Step [300/468], Loss: 0.1589, Time: 4.5964 secs\n",
      "Epoch [1/10], Step [400/468], Loss: 0.1732, Time: 5.9777 secs\n",
      "Epoch [2/10], Step [100/468], Loss: 0.2179, Time: 8.2756 secs\n",
      "Epoch [2/10], Step [200/468], Loss: 0.2117, Time: 9.6090 secs\n",
      "Epoch [2/10], Step [300/468], Loss: 0.1143, Time: 10.9415 secs\n",
      "Epoch [2/10], Step [400/468], Loss: 0.1430, Time: 12.2827 secs\n",
      "Epoch [3/10], Step [100/468], Loss: 0.1151, Time: 14.6643 secs\n",
      "Epoch [3/10], Step [200/468], Loss: 0.0915, Time: 16.0267 secs\n",
      "Epoch [3/10], Step [300/468], Loss: 0.2148, Time: 17.3831 secs\n",
      "Epoch [3/10], Step [400/468], Loss: 0.1106, Time: 18.6888 secs\n",
      "Epoch [4/10], Step [100/468], Loss: 0.0481, Time: 20.8634 secs\n",
      "Epoch [4/10], Step [200/468], Loss: 0.0289, Time: 22.1266 secs\n",
      "Epoch [4/10], Step [300/468], Loss: 0.0971, Time: 23.4104 secs\n",
      "Epoch [4/10], Step [400/468], Loss: 0.0954, Time: 24.7019 secs\n",
      "Epoch [5/10], Step [100/468], Loss: 0.0921, Time: 26.8355 secs\n",
      "Epoch [5/10], Step [200/468], Loss: 0.0399, Time: 28.0898 secs\n",
      "Epoch [5/10], Step [300/468], Loss: 0.0403, Time: 29.3446 secs\n",
      "Epoch [5/10], Step [400/468], Loss: 0.0204, Time: 30.6226 secs\n",
      "Epoch [6/10], Step [100/468], Loss: 0.0443, Time: 32.8190 secs\n",
      "Epoch [6/10], Step [200/468], Loss: 0.0593, Time: 34.2322 secs\n",
      "Epoch [6/10], Step [300/468], Loss: 0.1105, Time: 35.5956 secs\n",
      "Epoch [6/10], Step [400/468], Loss: 0.0168, Time: 36.9479 secs\n",
      "Epoch [7/10], Step [100/468], Loss: 0.0191, Time: 39.2662 secs\n",
      "Epoch [7/10], Step [200/468], Loss: 0.0214, Time: 40.6056 secs\n",
      "Epoch [7/10], Step [300/468], Loss: 0.0379, Time: 41.9390 secs\n",
      "Epoch [7/10], Step [400/468], Loss: 0.0780, Time: 43.2827 secs\n",
      "Epoch [8/10], Step [100/468], Loss: 0.0424, Time: 45.5493 secs\n",
      "Epoch [8/10], Step [200/468], Loss: 0.0149, Time: 46.8704 secs\n",
      "Epoch [8/10], Step [300/468], Loss: 0.0186, Time: 48.2039 secs\n",
      "Epoch [8/10], Step [400/468], Loss: 0.0212, Time: 49.5511 secs\n",
      "Epoch [9/10], Step [100/468], Loss: 0.0161, Time: 51.8537 secs\n",
      "Epoch [9/10], Step [200/468], Loss: 0.0053, Time: 53.0754 secs\n",
      "Epoch [9/10], Step [300/468], Loss: 0.0075, Time: 54.2974 secs\n",
      "Epoch [9/10], Step [400/468], Loss: 0.0357, Time: 55.5677 secs\n",
      "Epoch [10/10], Step [100/468], Loss: 0.0183, Time: 57.7369 secs\n",
      "Epoch [10/10], Step [200/468], Loss: 0.0526, Time: 58.9671 secs\n",
      "Epoch [10/10], Step [300/468], Loss: 0.0375, Time: 60.2029 secs\n",
      "Epoch [10/10], Step [400/468], Loss: 0.0186, Time: 61.4466 secs\n"
     ]
    }
   ],
   "source": [
    "# Train the model\n",
    "model.train()  # training mode\n",
    "total_step = len(mnist_train_loader)\n",
    "start_time = time.time()\n",
    "for epoch in range(num_epochs):\n",
    "    for i, (images, labels) in enumerate(mnist_train_loader):\n",
    "        # each i is a batch of 128 samples\n",
    "        images = images.to(device).view(batch_size, -1)  # represent images as column vectors\n",
    "        labels = labels.to(device)\n",
    "        \n",
    "        # Forward pass\n",
    "        outputs = model(images)\n",
    "        loss = criterion(outputs, labels)\n",
    "        \n",
    "        # Backward and optimize - ALWAYS IN THIS ORDER!\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        if (i + 1) % 100 == 0:\n",
    "            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}, Time: {:.4f} secs' \n",
    "                   .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(), time.time() - start_time))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy of the model on the 10000 test images: 97.78 %\n"
     ]
    }
   ],
   "source": [
    "# Test the model\n",
    "model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance), or use:\n",
    "with torch.no_grad(): # \"don't keep track of the gradients\" ,can also use .detach()\n",
    "    correct = 0\n",
    "    total = 0\n",
    "    for images, labels in mnist_test_loader:\n",
    "        images = images.to(device).view(images.size(0), -1)\n",
    "        labels = labels.to(device)\n",
    "        outputs = model(images)\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted == labels).sum().item()\n",
    "\n",
    "    print('Test Accuracy of the model on the 10000 test images: {} %'.format(100 * correct / total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "* PyTorch - <a href=\"https://www.youtube.com/watch?v=_H3aw6wkCv0&t=2s\">Stefan Otte: Deep Neural Networks with PyTorch | PyData Berlin 2018</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}