ee046746_tut_06_07_generative_adversarial_networks_gan.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"https://img.icons8.com/bubbles/100/000000/3d-glasses.png\" style=\"height:50px;display:inline\"> EE 046746 - Technion - Computer Vision\n",
    "\n",
    "#### Tal Daniel\n",
    "\n",
    "## Tutorial 06-07 - Generative Adversarial Networks (GANs)\n",
    "---\n",
    "\n",
    "<img src=\"./assets/tut_gan_morphing.gif\" style=\"height:200px\">\n",
    "\n",
    "* <a href=\"https://becominghuman.ai/with-gans-world-s-first-ai-generated-painting-to-recent-advancement-of-nvidia-b08ddfda45b1\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "\n",
    "* [What are Generative Adversarial Networks?](#-What-Are-Generative-Adversarial-Networks-(GANs)?)\n",
    "    * [Discriminative Vs. Generative](#-Discriminative-vs.-Generative)\n",
    "    * [Adversarial Training](#-Adversarial-Training)\n",
    "    * [A Game Theory Perspective - Nash Equilibrium](#-GANs---A-Game-Theory-Persepective---Nash-Equilibrium)\n",
    "* [GANs Training Steps](#-GANs-Training-Steps)\n",
    "    * Formulation\n",
    "    * Algorithm\n",
    "* [2D Demo](#-2D-Demo)\n",
    "* [GANs (Serious) Problems](#-GANs-(Serious)-Problems)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* [Vanilla-GAN on MNIST with PyTorch](#-Vanilla-GAN-on-MNIST-with-PyTorch)\n",
    "* [Deep Convolutional GANs (DCGANs)](#-Deep-Convolutional-GANs-(DCGANs))\n",
    "* [The Latent Space](#-The-Latent-Space)\n",
    "* [Conditional GANs](#-Conditional-GANs)\n",
    "* [GANs Today](#-GANs-Today)\n",
    "* [Tips for Training GANs](#-Tips-for-Training-GANs)\n",
    "* [Applications](#-Applications)\n",
    "    * [Image-to-Image Translation (Pix2Pix)](#-Image-to-Image-Translation-(Pix2Pix))\n",
    "    * [CycleGAN](#-CycleGAN)\n",
    "    * [Realistic Neural Talking Head Models](#-Realistic-Neural-Talking-Head-Models)\n",
    "    * [Face Aging with Conditional GANs](#-Face-Aging-with-Conditional-GANs)\n",
    "* [Cool GAN Projects (with Code)](#-Cool-GAN-Projects-(with-Code))\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# pytorch\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "##  <img src=\"./assets/mnist_anim_icon.gif\" style=\"height:40px;display:inline\"> What Are Generative Adversarial Networks (GANs)?\n",
    "---\n",
    "* **Generative** - learn a generative model that can generate new data.\n",
    "* **Adversarial** - trained in an *adversarial* setting (there is some competition during the model's training).\n",
    "* **Networks** - the model is implemented using deep neural networks.\n",
    "\n",
    "GANs were first introduced in <a href=\"http://papers.nips.cc/paper/5423-generative-adversarial-nets\">Generative Adversarial Networks</a>, NIPS 2014, by Goodfellow et al."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/resize-horizontal.png\" style=\"height:50px;display:inline\"> Discriminative vs. Generative\n",
    "---\n",
    "* So far, we have only seen *discriminative* models\n",
    "    * Given an image $X$, predict a label $Y$\n",
    "    * That is, we learn $P(Y\\mid X)$\n",
    "* The problem with discriminative models:\n",
    "    * During training, labels are required, as it is a *supervised* setting.\n",
    "    * Can't model $P(X)$, i.e., the probability of seeing a certain image.\n",
    "    * As a result, can't *sample* from $P(X)$, i.e., **can't generate new images**.\n",
    "* **Generative** models can overcome these limitations!\n",
    "    * They can model $P(X)$, implicitly (e.g. GANs) or explicitly (e.g. Variational Autoencoders - VAEs).\n",
    "    * Given a trained model, can generate new images (or data in general)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_taxonomy.jpg\" style=\"height:350px\">\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/1701.00160\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Explicit** density estimation: explicitly define and solve for $p_{model}(x)$.\n",
    "* **Implicit** density estimation: learn a model that can sample from $p_{model}(x)$ without explicitly defining it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/ios/50/000000/battle.png\" style=\"height:50px;display:inline\"> Adversarial Training\n",
    "---\n",
    "* **Goal**: given training data, generate new samples from the same distribution.\n",
    "<img src=\"./assets/tut_gan_train_to_sample.PNG\" style=\"height:250px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* In **general adversarial setting** (can also be discriminative):\n",
    "    * We can generate adversarial samples to *fool* a dsicriminative model.\n",
    "    * Using adversarial samples, we can make models more **robust**.\n",
    "    * Doing this will require the adversarial samples to be of better quality over time.\n",
    "        * This will require more effort in generating such quality samples!\n",
    "    * Repeating this process will result in a better *discriminative* model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_adv_setting.png\" style=\"height:300px\">\n",
    "\n",
    "* <a href=\"https://www.researchgate.net/publication/325370539_Protecting_Voice_Controlled_Systems_Using_Sound_Source_Identification_Based_on_Acoustic_Cues\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **GANs** extend this idea to *generative* models:\n",
    "    * **Generator**: generate fake samples, tries to fool the *Discriminator*.\n",
    "    * **Discriminator**: tries to dsitinguish between real and fake samples.\n",
    "    * Train them **against** each other!\n",
    "    * Repeat this and get a better *Generator* over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_architecture.png\" style=\"height:300px\">\n",
    "\n",
    "* <a href=\"https://www.researchgate.net/publication/334100947_Partial_Discharge_Classification_Using_Deep_Learning_Methods-Survey_of_Recent_Progress\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_gen_disc.png\" style=\"height:300px\">\n",
    "\n",
    "* <a href=\"https://towardsdatascience.com/comprehensive-introduction-to-turing-learning-and-gans-part-2-fd8e4a70775\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/plasticine/100/000000/controller.png\" style=\"height:50px;display:inline\"> GANs - A Game Theory Persepective - Nash Equilibrium\n",
    "---\n",
    "* GAN is based on a **zero-sum** cooperative game (minimax).\n",
    "    * In short, if one wins the other loses.\n",
    "* In game theory, the GAN model **converges** when the *discriminator* and the *generator* reach a **Nash equilibrium**.\n",
    "* **Nash equilibrium** - as both sides want to beat the other, a Nash equilibrium happens when *one player will not change its action regardless of what the opponent may do*.\n",
    "* **Cost functions may not converge using gradient descent in a minimax game**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/plasticine/100/000000/workflow.png\" style=\"height:50px;display:inline\"> GANs Training Steps\n",
    "---\n",
    "* Training the **Discriminator**\n",
    "    * **Freeze** the *Generator* and generate fake samples (that is, when backpropagating, don't update the generator weights)\n",
    "    \n",
    "<img src=\"./assets/tut_gan_train_disc.PNG\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Training the **Generator**\n",
    "    * **Freeze** the *Discriminator*, and update the Generator to get a higher score (like the real data) from the Discriminator\n",
    "    \n",
    "<img src=\"./assets/tut_gan_train_gen.PNG\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/color/96/000000/rubiks-cube.png\" style=\"height:50px;display:inline\"> Formulation & Algorithm\n",
    "---\n",
    "* For a Discriminator (binary classifier) $D$, a Generator $G$ and a reward function $V$, the GAN's objective function: $$ \\min_G \\max_D V(D,G)$$\n",
    "* It is formulated as a **minimax game**, where:\n",
    "    * The **Discriminator** $D$ is trying to *maximize* its reward $V(D, G)$\n",
    "    * The **Generator** $G$ is trying to *minimize* the Discriminator's reward (or maximize its loss)\n",
    "        * Why? Because minimizing the Discriminator's reward means that the Discriminator can not tell the difference between real and fake samples, thus, the Generator is \"winning\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* In our case, the reward function $V$: $$ V(D,G) = \\mathbb{E}_{x \\sim p(x)} \\left[ \\log D(x) \\right] + \\mathbb{E}_{z\\sim q(z)} \\left[\\log(1 - D\\left(G(z)\\right)) \\right] $$\n",
    "    * Recall from ML course that for binary classification (real or fake) we use the <a href=\"https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#cross-entropy\">Binary Cross Entropy (BCE)</a> loss function.\n",
    "* The **Nash equilibrium** is reached when:\n",
    "    * $P_{data}(x) = P_{gen}(x), \\forall x$\n",
    "    * $D(x) = \\frac{1}{2}$ (completely random classifier).\n",
    "    \n",
    "* <a href=\"https://srome.github.io/An-Annotated-Proof-of-Generative-Adversarial-Networks-with-Implementation-Notes/\">Proof of Nash Equilibrium in GANs</a> - However out of the scope of this course, the mathematical proof is very nice and important. It is recommended to go over it if you are interested in working with GANs in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_gen_disc_loss.png\" style=\"height300px\">\n",
    "\n",
    "* <a href=\"https://towardsdatascience.com/comprehensive-introduction-to-turing-learning-and-gans-part-2-fd8e4a70775\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_algo.PNG\" style=\"height:500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/cute-clipart/64/000000/test-tube.png\" style=\"height:50px;display:inline\"> 2D Demo\n",
    "\n",
    "<a href=\"https://poloclub.github.io/ganlab/\">GAN Lab <br> <img src=\"./assets/tut_gan_ganlab.PNG\" style=\"height:400px\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/box-important.png\" style=\"height:50px;display:inline\"> GANs (Serious) Problems\n",
    "---\n",
    "* **Non-convergence**: the model parameters oscilate, destabilize and (almost) never converge.\n",
    "* **Mode Collapse**: the *Generator* collapses, which produces limited varieties of samples.\n",
    "    * For example, on a 2D eight-Gaussians dataset: <img src=\"./assets/tut_gan_mode_collapse.png\" style=\"height:250px\">\n",
    "    * <a href=\"https://mc.ai/gan-unrolled-gan-how-to-reduce-mode-collapse/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **Vanisihng/Diminishing Gradient**: the discriminator gets *too good* such that the generator gradient vanishes and learns nothing.\n",
    "    * Proof: consider the second term in the objective function which is relavent only for the generator.\n",
    "        * Recall that the output of binary classification is the output of the *sigmoid* function, $\\sigma$.\n",
    "        * $\\nabla_{\\theta_G}V(D,G) = \\nabla_{\\theta_G}\\mathbb{E}_{z\\sim q(z)} \\left[\\log(1 - D\\left(G(z)\\right)) \\right]$\n",
    "        * $\\nabla_a \\log\\left(1 - \\sigma(a)\\right) = \\frac{-\\nabla_a \\sigma(a)}{1 - \\sigma(a)} = \\frac{-\\sigma(a)\\left(1 - \\sigma(a)\\right)}{1 - \\sigma(a)} = -\\sigma(a) = -D\\left(G(z)\\right)$\n",
    "        * So if $D$ is confident (that the sample is fake), the gradient goes to 0, i.e. $D\\left(G(z)\\right) \\to 0$\n",
    "    * Possible remedy: replace the problematic term with $$ \\mathbb{E}_{z\\sim q(z)} \\left[\\log(1 - D\\left(G(z)\\right)) \\right] \\rightarrow -\\mathbb{E}_{z\\sim q(z)} \\left[\\log D\\left(G(z)\\right) \\right] $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* **GANs are highly sensitive to hyper-parameters!**\n",
    "    * Even the slightest change in hyper-parameters may lead to any of the above, e.g. even changing the learning rate from 0.0002 to 0.0001 may lead to instability. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/officel/80/000000/fire-element.png\" style=\"height:50px;display:inline\"> Vanilla-GAN on MNIST with PyTorch\n",
    "---\n",
    "* Based on example by <a href=\"https://github.com/rasbt/deeplearning-models\">Sebastian Raschka</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# ------------------------- <img src=\"https://img.icons8.com/color/96/000000/code.png\" style=\"height:50px;display:inline\"> CODE TIME -------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### SETTINGS\n",
    "##########################\n",
    "\n",
    "# Device\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# Hyperparameters\n",
    "# Remember that GANs are highly sensitive to hyper-parameters\n",
    "random_seed = 123\n",
    "generator_learning_rate = 0.001\n",
    "discriminator_learning_rate = 0.001\n",
    "NUM_EPOCHS = 100\n",
    "BATCH_SIZE = 128\n",
    "LATENT_DIM = 100  # latent vectors dimension [z]\n",
    "IMG_SHAPE = (1, 28, 28)  # MNIST has 1 color channel, each image 28x8 pixels\n",
    "IMG_SIZE = 1\n",
    "for x in IMG_SHAPE:\n",
    "    IMG_SIZE *= x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image batch dimensions: torch.Size([128, 1, 28, 28])\n",
      "Image label dimensions: torch.Size([128])\n",
      "shape: \n",
      " torch.Size([128, 1, 28, 28])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEYCAYAAACQgLsAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAcLElEQVR4nO3debBU1bn38d/DIAjkokxOCAS4MUFkEowiDkm4IiiKAoGC1xitGzWKsQpFo2iBA/EWVsw1RtDULeOA1xhBCCISfFMMMQ4llOKE+IIFyg3ITDgMYVrvH7u56bUOp/v06dXT4fupoopfn917P33O4jy9e7H3MuecAACIqUGpCwAA1D80FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0x0RzMbO1ZjawhMdfb2YXl+r4yB1jBrlgvFQXpbmY2Wgze9fMdpvZptTfbzYzi7H/QjGz182sKvXngJntT8tP1nGfM8xscsQa70urqcrM9prZITM7MdYxSoEx4+0z9pi5wszeMrMdZrbBzJ4ysxax9l8KjBdvn7HHy2lm9mpqrDgzax9jv3k3FzO7XdJjkh6RdLKkkyTdJOl8ScfV8JyG+R43BufcYOdcC+dcC0kvSJp6JDvnbgq3N7NGJajxwbSaWkj6paQ/O+e2F7uWWBgzBfcNSfdLOkXSmZK+Kek/SlBHFIyXgjssab6kEVH36pyr8x9JLSXtljQ8y3bPSJqeegG7JQ1MPfc5SZslrZN0r6QGqe0nS5qR9vxOkpykRqm8WNKDkv4qaZekhZLapG1/TWqfWyVNlLRW0sBa1PhQ8NjA1HPvkbRR0u8k/bukxWnbNErV1knSzZIOSNovqUrS7NQ26yWNl/SRpJ2SXpTUpA7fb0u9rrH5/NxK+YcxU9wxk9rXDyW9X+qfPeOlvMeLpKap47SP8bPL98zlPElNJP2xFtuOkTRFybuqNyU9ruSH31nSRZJ+JOm6HI49JrV9OyXvXu6QJDPrpmSQXSPpVEmtJeVzmtdeUgtJHZT8YGvknJsm6SVJv3DJO5Or0r78Q0n/puT1np2qT2bWMPXxxbm1qOV7kk6UNDvnV1E+GDNpijBmJOlCSZ/k9hLKBuMlTZHGSxT5Npc2krY45w4eeSDts969ZnZh2rZ/dM791Tl3WEnnHSXpbufcLufcWiUf91yTw7F/55z73Dm3V9IfJPVKPT5C0jzn3FLn3D8k3afktK+uDkqa7JzbnzpWXf2nc26jc26rpHlH6nXOHXLOneCce6cW+7hW0h+cc3vyqKPUGDO1l/eYMbPBSn5JTsqjjlJivNRejN8x0eTbXLZKapP+OaFzrr9z7oTU19L3/1Xa39soeSewLu2xdZJOy+HYG9P+vkdJ55eSdxL/eyzn3O5ULXX1tXNufx7PP6KmemvFzJpLGi7p2Qi1lBJjpvbyHTP9JT0v6Wrn3JoI9ZQC46X28hovseXbXN6W9A9JV9Zi2/TbL29R8s6iY9pjHST9T+rvuyU1S/vayTnUtEHS6UeCmTVTctpaV+Fto7PVVqjbTI+Q9LWS0/1Kxpgpwpgxs76S5kj6kXNucez9FxHjpXi/Y6LKq7k453Yo+V8p08xshJm1MLMGZtZLUvMMzzuk5DRzipl9w8w6KpmMmpHa5ANJF5pZBzNrKenuHMqaKelyMxtgZsdJekBxr+dZIamHmZ1lZser+scNXyv5zDO2ayU961Izb5WKMVP4MWNmPZVMbN/snJsfa7+lwHgpzu8YM2uqZG5LkpqYWZNM29dG3t8Q59xUJT+0OyVtUvLCn5J0l6S3Mjz1ViUd+gsl78b/W9LTqX2+oWTS6kNJy5V8fljbej6RdEtqfxskbVfyPymicM59KukXSv43ySpJS4NN/ktSTzPbbmYzs+0vNdlWZWbnZdimg5JJ2efrXHgZYcwUfMzcoeSd9DNp11SsqPsrKC3GS2HHS+ojx72SdqQeWq3k+5YXq/A3wgCAMnRM3P4FAFBcNBcAQHQ0FwBAdDQXAEB0GW+SZmbM9lc451xR7xrLmKl8xRwzjJfKV9N44cwFABAdzQUAEB3NBQAQHc0FABAdzQUAEB3NBQAQHc0FABAdzQUAEB3NBQAQHc0FABBdxtu/APXNHXfc4eXjjz/eyz169PDyiBEjsu5z+vTpXn777be9/Pzz9WKNNyAnnLkAAKKjuQAAoqO5AACiM+dqvuM1t8OufMf6LfdfeuklL9dmDiVfa9as8fLAgQO9/OWXXxa8hnxwy/3i+ta3vuXlzz77zMu33Xablx9//PGC15QLbrkPACgamgsAIDqaCwAgOq5zQb2S7xxL+Hn3n/70Jy937ty52nOGDh3q5S5dunh57NixXn744Ydzqgn1W+/evb18+PBhL69fv76Y5UTDmQsAIDqaCwAgOpoLACA65lxQ0fr27evlq666KuP2n3zyiZevuOIKL2/ZssXLVVVVXj7uuOOq7fOdd97xcs+ePb3cunXrjDXh2NarVy8v796928uzZ88uZjnRcOYCAIiO5gIAiI7mAgCIrqRzLuE1CD/5yU+8/Le//c3L+/bt8/ILL7zg5Y0bN3p59erV+ZaIMnfKKad42cy/zVE4xzJo0CAvb9iwIafj3X777dUe69atW8bnvPbaazkdA/Vb9+7dvTxu3Dgv15f1fzhzAQBER3MBAERHcwEARFfSOZepU6d6uVOnTjk9/8Ybb/Tyrl27vBx+3l4K4X2Bwte8bNmyYpZT77z66qte7tq1q5fDMbFt27a8jjd69OhqjzVu3DivfeLY8u1vf9vLzZs393J4f7xKxZkLACA6mgsAIDqaCwAgupLOuYTXtfTo0cPLK1eu9PJ3vvMdL/fp08fLF198sZfPPfdcL3/11VdePv3002td6xEHDx708ubNm70cXncRCtdPZ84lrnXr1kXd34QJE7wcrnd+NO+++27GjGPbnXfe6eVwzNaX3wmcuQAAoqO5AACio7kAAKIz51zNXzSr+Ytl6MQTT/RyuE7C8uXLvdyvX7+cjxHe3+zzzz/3cjhP1KpVKy/fcsstXp4+fXrONeTCOWfZt4qn0sZM6PLLL/fyyy+/7OWjreeyadMmL4fXwixZsiRSdcVRzDFT6eMlm6Ndu/fFF194OfwdEl4HU+5qGi+cuQAAoqO5AACio7kAAKIr6XUusW3fvt3LixYtyrj9n//857yPOXz4cC+H8z4fffSRl+vLfYPqq759+3r5aHMsofBnWmlzLCiciy66KOs24bVy9QVnLgCA6GguAIDoaC4AgOjq1ZxLMbRr187L06ZN83KDBn6/fuCBB7yc73oiiGvOnDlevuSSSzJu/9xzz1V77N57741aE+qPs846K+s24RpP9QVnLgCA6GguAIDoaC4AgOiYc8lReG+wtm3bejm81mbVqlUFrwm1F663079/fy83adLEy1u2bPHyQw89VG2fVVVVkapDpQvXkLruuuuqbfP+++97+Y033ihoTaXCmQsAIDqaCwAgOpoLACA65lyyOP/8873885//POP2w4YN8/LHH38cvSbU3axZs7zcunXrjNvPmDHDy2vWrIleE+qPgQMHejlcz0mSFixY4OVwjaj6gjMXAEB0NBcAQHQ0FwBAdDQXAEB0TOhnMWTIEC83btzYy+GCY2+//XbBa0LtXXHFFV7u06dPxu0XL17s5UmTJsUuCfVYz549veycq7bNzJkzi1VOSXHmAgCIjuYCAIiO5gIAiI45l8Dxxx/v5UsvvdTL+/fv93L4mfyBAwcKUxhqJbwo8p577vFyOGcW+uCDD7zMTSmRycknn+zlCy64wMtHu3Ht7NmzC1pTueDMBQAQHc0FABAdzQUAEB1zLoEJEyZ4uXfv3l4Obzr31ltvFbwm1N7tt9/u5X79+mXcfs6cOV7muhbk4sc//rGX27Vr5+XXX3+9iNWUF85cAADR0VwAANHRXAAA0R3Tcy6XXXZZtcfuu+8+L//973/38gMPPFDQmpCf8ePH57T9uHHjvMx1LchFx44dM359+/btRaqk/HDmAgCIjuYCAIiO5gIAiO6YmnMJ7zv161//uto2DRs29PL8+fO9/M4778QvDCXTqlUrL8e4N9zOnTsz7jO8v1nLli0z7u+EE07wcq7zSocOHfLyXXfd5eU9e/bktD/80+WXX57x66+++mqRKik/nLkAAKKjuQAAoqO5AACiq9dzLuH8SXhfsG9+85vVnrNmzRovh9e9oH758MMPo+/z5Zdf9vKGDRu8fNJJJ3l51KhR0WvIZOPGjV6eMmVKUY9fyQYMGODlcD0X/BNnLgCA6GguAIDoaC4AgOjq9ZxLly5dvHz22WdnfU54DUE4B4PyFl6XdOWVVxa9hpEjR+b1/IMHD3r58OHDGbefO3eul5ctW5Zx+7/85S91Kwy66qqrvBzO677//vteXrp0acFrKlecuQAAoqO5AACio7kAAKKrV3Mu4doKCxcuzLj9hAkTqj02b968qDWhuK6++mov33nnnV4O7+uVzZlnnunlulyT8vTTT3t57dq1GbefNWuWlz/77LOcj4k4mjVr5uUhQ4Zk3H7mzJleDu/rdizhzAUAEB3NBQAQHc0FABCdOedq/qJZzV8sQ+E9ku6+++6M259zzjnVHst2jUClcc5ZMY9XaWMG1RVzzJT7eAnn6JYsWeLlTZs2eXnMmDFePhbWyqlpvHDmAgCIjuYCAIiO5gIAiK6ir3MJ11a49dZbS1QJgProwIEDXu7fv3+JKqk8nLkAAKKjuQAAoqO5AACiq+g5lwsuuMDLLVq0yLh9uDZLVVVV9JoAAJy5AAAKgOYCAIiO5gIAiK6i51yyWbFihZd/8IMfeHnbtm3FLAcAjhmcuQAAoqO5AACio7kAAKKrV+u5oDrWc0GuWM8FuWA9FwBA0dBcAADR0VwAANFlnHMBAKAuOHMBAERHcwEAREdzAQBER3MBAERHcwEAREdzAQBER3MBAERHcwEAREdzAQBER3MBAERHcwEARHdMNBczW2tmA0t4/PVmdnGpjo/cMWaQC8ZLdVGai5mNNrN3zWy3mW1K/f1mMyvqQlW5MrPXzawq9eeAme1Py0/WcZ8zzGxy5Dr/j5mtS9X1ipmdEHP/pcCY8fYZfcyk7ft5M3Nm1qkQ+y8Wxou3z6jjxcxOM7NXzWxDaqy0j7HfvJuLmd0u6TFJj0g6WdJJkm6SdL6k42p4TsN8jxuDc26wc66Fc66FpBckTT2SnXM3hdubWaNi12hmPSRNkzRWyff3gKTfFLuOmBgzxZF6J9uxVMePhfFScIclzZc0IupenXN1/iOppaTdkoZn2e4ZSdNTL2C3pIGp5z4nabOkdZLuldQgtf1kSTPSnt9JkpPUKJUXS3pQ0l8l7ZK0UFKbtO2vSe1zq6SJktZKGliLGh8KHhuYeu49kjZK+p2kf5e0OG2bRqnaOkm6Wckv//2SqiTNTm2zXtJ4SR9J2inpRUlNavk9nirpubR8hqR/SGqWz8+uVH8YM4UfM6nnN5a0QlLPI8cq9c+e8VK+4yW1j6ap47SP8bPL98zlPElNJP2xFtuOkTRF0jckvSnpcSU//M6SLpL0I0nX5XDsMant2yl593KHJJlZNyWD7BpJp0pqLSmf07z2klpI6qDkB1sj59w0SS9J+oVL3plclfblH0r6NyWv9+xUfTKzhma2w8zOrWG3Zyr5JXHkGKuUvNP417q9nJJjzKQp0JiRktf2fyV9UudXUR4YL2kKOF6iy7e5tJG0xTl38MgDZvZW6oXsNbML07b9o3Pur865w0o67yhJdzvndjnn1kr6pVLfjFr6nXPuc+fcXkl/kNQr9fgISfOcc0udc/+QdJ+SX8Z1dVDSZOfc/tSx6uo/nXMbnXNbJc07Uq9z7pBz7gTn3Ds1PK+Fknci6f6u5B9QJWLM1F6dxoyZdZR0vZJ355WO8VJ7df0dUxD5Npetktqkf07onOvvnDsh9bX0/X+V9vc2St4JrEt7bJ2k03I49sa0v+9R8ktYSt5J/O+xnHO7U7XU1dfOuf15PP+ImurNpkrSvwSP/YuSU/VKxJipvbqOmV9LmuScq9Qxko7xUnt1HS8FkW9zeVvJ5/9X1mLb9PWUtyh5Z5E+2dhB0v+k/r5bUrO0r52cQ00bJJ1+JJhZMyWnrXUVrgOdrbbY60Z/ouRzc0mSmX1Lyc/t/0U+TrEwZgo/Zn4g6VEz26jks3hJes/MRkU+TjEwXgo/Xgoir+binNsh6X5J08xshJm1MLMGZtZLUvMMzzuk5DRzipl9I3UaP17SjNQmH0i60Mw6mFlLSXfnUNZMSZeb2QAzO07SA4p7Pc8KST3M7CwzO17SpODrXyv5zDOWGZKGmVl/M2uu5PW87JzbE/EYRcOYKcqY6azkI5FeSj57l6QhkuZGPEZRMF6KMl5kZk2VzG1JUhMza5Jp+9rI+xvinJuq5Id2p6RNSl74U5LukvRWhqfeqqRDf6Fk8u2/JT2d2ucbSiatPpS0XMnnh7Wt5xNJt6T2t0HSdv3z3VvenHOfSvqFkv9NskrS0mCT/5LU08y2m9nMbPtLTbZVmdl5NRzvQ0njJP1eyfe3iZLvXcVizBR8zGxKffa+Ucn3VpI25/l5fskwXgo7XlIfOe6VtCP10Gol37e8WOq/oAEAEM0xcfsXAEBx0VwAANHRXAAA0dFcAADRZbxJmpkx21/hnHNFvWssY6byFXPMMF4qX03jhTMXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQ0FwBAdDQXAEB0NBcAQHQZb7lf6Zo3b+7lRx55xMs33nhjtecsX77cyyNHjvTyunXrIlUHAPUXZy4AgOhoLgCA6GguAIDozLmaVxmt9CVIu3bt6uWVK1dmfU6DBn6//dnPfublJ554Iv/Ciohljn19+vTx8iuvvOLlTp06FbGaxCWXXOLlcJx+9dVXxSyHZY6LbOjQoV6eO3eul8eNG+flJ5980suHDh0qTGG1xDLHAICiobkAAKKjuQAAoqtX17m0bdvWy88++2yJKkG5GjRokJebNGlSokr+KfzM/frrr/fy6NGji1kOCqx169ZenjZtWsbtf/Ob33j56aef9vLevXvjFBYZZy4AgOhoLgCA6GguAIDoKnrOJbwGZdiwYV4+55xz8j7GhRde6OXwOpgVK1Z4eenSpXkfE/E0auQP8SFDhpSokpqF97MbP368l8N75O3evbvgNaFwwt8p7du3z7j9iy++6OV9+/ZFr6kQOHMBAERHcwEAREdzAQBEV9FzLr/61a+8fPjw4ejHuPrqqzPmcH2XUaNGeTn8PB3F9b3vfc/L5513npenTp1azHKO6sQTT/Ryt27dvNysWTMvM+dSOY52HdXEiRNz2sfzzz/v5Uz3gywnnLkAAKKjuQAAoqO5AACiq6j1XObPn+/lwYMHeznGnMvWrVu9XFVV5eWOHTvmtL+GDRvmXVM+jrX1XLp37+7lxYsXezn8+Z599tleDn/exRDWOGDAAC+fcsopXt68eXNB62E9l3j69u1b7bH33nsv43MOHjzo5caNG0etKTbWcwEAFA3NBQAQHc0FABBdWV/nctFFF3n5jDPO8HI4x5LrnEu4FrUkLVy40Ms7d+708ve//30vZ/s/6z/96U+9PH369FxKRI7uvfdeL4f35br00ku9XIo5llatWnk5HOeFuF4LpTF8+PCcnxP+DqpUnLkAAKKjuQAAoqO5AACiK6s5l06dOnn597//vZfbtGmT0/7C+37NmjXLy/fff3+15+zZsyenfd5www1ebtu2rZfDe1c1bdrUy+H62AcOHMh4fPhGjBjh5XC9ltWrV3t52bJlBa8pm3CeLpxjCa972bFjR6FLQoGEa7cczf79+72c673HyhVnLgCA6GguAIDoaC4AgOjKas4lXO881zmWJUuWeHn06NFe3rJlS90KSxPOuTz88MNefvTRR70crsURzsHMnTvXy2vWrMm3xGPKyJEjvRx+v6dNm1bMco4qnEscO3aslw8dOuTlhx56yMvMw1WO/v37Z8xHE67P88EHH0StqVQ4cwEAREdzAQBER3MBAERXVnMuuQqvWbj++uu9HGOOJZtwziT8PL1fv34Fr+FY0rJlSy+fe+65Gbcvh3u5hddChXOJK1eu9PKiRYsKXhMKoy7/3sthjBYCZy4AgOhoLgCA6GguAIDoynrOpUGDzL3vu9/9bpEqqZmZv3x0WHO21zB58mQvX3PNNVHqqq+aNGni5dNOO83LL774YjHLqZUuXbpk/PrHH39cpEpQaH379s26TXivOOZcAACoJZoLACA6mgsAIDqaCwAgurKa0L/pppu8HC6iVI6GDh3q5d69e3s5fA1hDif0kdmuXbu8HN7kr0ePHl5u1aqVl7dt21aYwtK0a9fOy+GCZqE333yzkOWggAYMGODlMWPGZH3Ozp07vbx+/fqoNZULzlwAANHRXAAA0dFcAADRldWcSzh/UQ7atm3r5W7dunn5nnvuyWl/mzdv9jILQeVm7969Xg4XVxs+fLiXX3vtNS+Hi7nlqnv37tUe69y5s5fDxcGccxn3WQlzizi61q1beznbRdOS9MYbbxSqnLLCmQsAIDqaCwAgOpoLACC6sppzKUcTJ0708i233JLT89euXevla6+91stffvllnepCYtKkSV4ObyR62WWXeTnfG1sebQG6cE4lXAwsm2eeeSafklBC2a5hCm9SKUlPPfVUocopK5y5AACio7kAAKKjuQAAorNM/wffzDL/B/3IVq1a5eXw+oFQ48aNo9cwf/58L59xxhle7tChQ077W7BggZeLfS2Pc86ybxVPscdMNr169fJy165d89rfzJkzs27z7LPPenns2LEZt2/UqLymPos5ZsptvGTTvn17L69bt87L4XUuR1sI7qyzzopfWAnVNF44cwEAREdzAQBER3MBAERXVh/2htcoZLtPz+DBgzN+/be//a2XTz311Kw1hMfM975P5Xi/tGNJuN5LmAvhiy++yGn78H5lR/ucHuWhf//+Xs72O2rOnDmFLKesceYCAIiO5gIAiI7mAgCIrqzmXKZPn+7lqVOnZtx+3rx5Xs42P1KX+ZNcn/Pkk0/mfAzUL+HcYZhDzLFUjnD9llB477nHHnuskOWUNc5cAADR0VwAANHRXAAA0ZXVnMsrr7zi5QkTJng5XM++GMI171euXOnlG264wcsbNmwoeE0ob+H9+jLdvw+VZdCgQRm/Hq7PtHPnzkKWU9Y4cwEAREdzAQBER3MBAERXVnMu4doIo0eP9vKwYcO8fNtttxW8pilTpnj5iSeeKPgxUdmaNm2a8et79+4tUiXIV7hmVJcuXTJuv2/fPi8fOHAgek2VgjMXAEB0NBcAQHQ0FwBAdGU15xJaunRpxrxw4UIvh9echGupzJ0718vhei9S9ftAffrpp7UrFki57rrrvLxjxw4vP/jgg8UsB3kI7y24bNkyL4dr8axevbrgNVUKzlwAANHRXAAA0dFcAADRlfWcSzYLFizImIFSeO+997z86KOPennRokXFLAd5OHTokJcnTpzo5fC+ccuXLy94TZWCMxcAQHQ0FwBAdDQXAEB0lmmtCTNjIYoK55zLvIB7ZIyZylfMMcN4qXw1jRfOXAAA0dFcAADR0VwAANHRXAAA0dFcAADR0VwAANHRXAAA0dFcAADR0VwAANHRXAAA0dFcAADRZby3GAAAdcGZCwAgOpoLACA6mgsAIDqaCwAgOpoLACA6mgsAILr/D0ganRzqqR6MAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "##########################\n",
    "### MNIST DATASET\n",
    "##########################\n",
    "\n",
    "# Note transforms.ToTensor() scales input images\n",
    "# to 0-1 range\n",
    "train_dataset = datasets.MNIST(root='./datasets', \n",
    "                               train=True, \n",
    "                               transform=transforms.ToTensor(),\n",
    "                               download=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root='./datasets', \n",
    "                              train=False, \n",
    "                              transform=transforms.ToTensor())\n",
    "\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE, \n",
    "                          shuffle=True)\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE, \n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "for images, labels in train_loader:  \n",
    "    print('Image batch dimensions:', images.shape)\n",
    "    print('Image label dimensions:', labels.shape)\n",
    "    break\n",
    "    \n",
    "# let's see some digits\n",
    "examples = enumerate(test_loader)\n",
    "batch_idx, (example_data, example_targets) = next(examples)\n",
    "print(\"shape: \\n\", example_data.shape)\n",
    "fig = plt.figure()\n",
    "for i in range(6):\n",
    "    ax = fig.add_subplot(2,3,i+1)\n",
    "    ax.imshow(example_data[i][0], cmap='gray', interpolation='none')\n",
    "    ax.set_title(\"Ground Truth: {}\".format(example_targets[i]))\n",
    "    ax.set_axis_off()\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### MODEL\n",
    "##########################\n",
    "\n",
    "class GAN(torch.nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(GAN, self).__init__()\n",
    "        \n",
    "        # generator: z [vector] -> image [matrix]\n",
    "        self.generator = nn.Sequential(\n",
    "            nn.Linear(LATENT_DIM, 128),\n",
    "            nn.LeakyReLU(inplace=True),\n",
    "            nn.Dropout(p=0.5),\n",
    "            nn.Linear(128, IMG_SIZE),\n",
    "            nn.Tanh()\n",
    "        )\n",
    "        \n",
    "        # discriminator: image [matrix] -> label (0-fake, 1-real)\n",
    "        self.discriminator = nn.Sequential(\n",
    "            nn.Linear(IMG_SIZE, 128),\n",
    "            nn.LeakyReLU(inplace=True),\n",
    "            nn.Dropout(p=0.5),\n",
    "            nn.Linear(128, 1),\n",
    "            nn.Sigmoid()\n",
    "        )\n",
    "\n",
    "            \n",
    "    def generator_forward(self, z):\n",
    "        img = self.generator(z)\n",
    "        return img\n",
    "    \n",
    "    def discriminator_forward(self, img):\n",
    "        pred = model.discriminator(img)\n",
    "        return pred.view(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# constant the seed\n",
    "torch.manual_seed(random_seed)\n",
    "\n",
    "# build the model, send it ti the device\n",
    "model = GAN().to(device)\n",
    "\n",
    "# optimizers: we have one for the generator and one for the discriminator\n",
    "# that way, we can update only one of the modules, while the other one is \"frozen\"\n",
    "optim_gener = torch.optim.Adam(model.generator.parameters(), lr=generator_learning_rate)\n",
    "optim_discr = torch.optim.Adam(model.discriminator.parameters(), lr=discriminator_learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "##########################\n",
    "### Training\n",
    "##########################\n",
    "\n",
    "start_time = time.time()    \n",
    "\n",
    "discr_costs = []\n",
    "gener_costs = []\n",
    "for epoch in range(NUM_EPOCHS):\n",
    "    model = model.train()\n",
    "    for batch_idx, (features, targets) in enumerate(train_loader):\n",
    "\n",
    "        features = (features - 0.5) * 2.0  # normalize between [-1, 1]\n",
    "        features = features.view(-1, IMG_SIZE).to(device) \n",
    "        targets = targets.to(device)\n",
    "\n",
    "        # generate fake and real labels\n",
    "        valid = torch.ones(targets.size(0)).float().to(device)\n",
    "        fake = torch.zeros(targets.size(0)).float().to(device)\n",
    "        \n",
    "        ### FORWARD PASS AND BACKPROPAGATION\n",
    "        \n",
    "        # --------------------------\n",
    "        # Train Generator\n",
    "        # --------------------------\n",
    "        \n",
    "        # Make new images\n",
    "        z = torch.zeros((targets.size(0), LATENT_DIM)).uniform_(-1.0, 1.0).to(device)\n",
    "        generated_features = model.generator_forward(z)\n",
    "        \n",
    "        # Loss for fooling the discriminator\n",
    "        discr_pred = model.discriminator_forward(generated_features)\n",
    "        \n",
    "        # here we use the `valid` labels because we want the discriminator to \"think\"\n",
    "        # the generated samples are real\n",
    "        gener_loss = F.binary_cross_entropy(discr_pred, valid)\n",
    "        \n",
    "        optim_gener.zero_grad()\n",
    "        gener_loss.backward()\n",
    "        optim_gener.step()\n",
    "        \n",
    "        # --------------------------\n",
    "        # Train Discriminator\n",
    "        # --------------------------        \n",
    "        \n",
    "        discr_pred_real = model.discriminator_forward(features.view(-1, IMG_SIZE))\n",
    "        real_loss = F.binary_cross_entropy(discr_pred_real, valid)\n",
    "        \n",
    "        # here we use the `fake` labels when training the discriminator\n",
    "        discr_pred_fake = model.discriminator_forward(generated_features.detach())\n",
    "        fake_loss = F.binary_cross_entropy(discr_pred_fake, fake)\n",
    "        \n",
    "        discr_loss = 0.5 * (real_loss + fake_loss)\n",
    "\n",
    "        optim_discr.zero_grad()\n",
    "        discr_loss.backward()\n",
    "        optim_discr.step()        \n",
    "        \n",
    "        discr_costs.append(discr_loss)\n",
    "        gener_costs.append(gener_loss)\n",
    "        \n",
    "        \n",
    "        ### LOGGING\n",
    "        if not batch_idx % 100:\n",
    "            print ('Epoch: %03d/%03d | Batch %03d/%03d | Gen/Dis Loss: %.4f/%.4f' \n",
    "                   %(epoch+1, NUM_EPOCHS, batch_idx, \n",
    "                     len(train_loader), gener_loss, discr_loss))\n",
    "\n",
    "    print('Time elapsed: %.2f min' % ((time.time() - start_time)/60))\n",
    "    \n",
    "print('Total Training Time: %.2f min' % ((time.time() - start_time)/60))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-2344.9500000000003, 49243.95)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEzCAYAAAA7AhgJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3gU1frA8e+bSq+C0hRQEOlIEUEBsYDixV5RxHrxqoheC1jRa0HsKBZ+KnZRUUFFAelFuvTeAoQWWkIoCSnn98fMJptkN9lNtiSz7+d58mQzOzvnzCR59+w5Z94jxhiUUko5T1S4K6CUUio4NMArpZRDaYBXSimH0gCvlFIOpQFeKaUcKibcFXAXFRVlypcvH+5qKKVUmXH8+HFjjPHYWC9VAb58+fIcO3Ys3NVQSqkyQ0ROeHtOu2iUUsqhNMArpZRDaYBXSimHKlV98Eop32VkZJCYmEhaWlq4q6JCoFy5ctSvX5/Y2FifX6MBXqkyKjExkcqVK9OwYUNEJNzVUUFkjOHgwYMkJibSqFEjn1+nXTRKlVFpaWnUrFlTg3sEEBFq1qzp96c1DfBKlWEa3CNHcX7XjgjwaRlZjFuaiKY+VkqpXI4I8K9NWs9jP65g5sb94a6KUhFj37593HrrrTRu3Jj27dtz/vnn88svv4StPjNnzuTvv/8u8TGuvPLKANUo/BwR4JNS0wFITcsMc02UigzGGK6++mq6devG1q1bWbp0KWPHjiUxMTGo5WZmev8fL06AL+x4TuCIAO/qmdIuGqVCY/r06cTFxTFw4MCcbWeccQYPPfQQAFlZWTz++ON07NiR1q1b8/HHHwNWEO7RowfXX389zZo1o1+/fjn/t0uXLqV79+60b9+eXr16sWfPHgB69OjBU089Rffu3Xn33Xf57bffOO+882jXrh2XXHIJ+/btIyEhgY8++oi3336btm3bMmfOHLZv387FF19M69atufjii9mxYwcAAwYM4NFHH+Wiiy7iySef9HqOhw4d4uqrr6Z169Z07tyZlStXAjBr1izatm1L27ZtadeuHampqezZs4du3brRtm1bWrZsyZw5cwJ/0YvBEdMkdaBJRboXflvD2t1HAnrM5nWr8Py/Wnh8bs2aNZx77rleX/vpp59StWpVFi9eTHp6Ol27duWyyy4DYNmyZaxZs4a6devStWtX5s2bx3nnncdDDz3EhAkTqFWrFt9//z1PP/00n332GQDJycnMmjULgMOHD7NgwQJEhE8++YQRI0bw5ptvMnDgQCpVqsRjjz0GwL/+9S/69+/PHXfcwWeffcagQYMYP348ABs3bmTq1KlER0d7PYfnn3+edu3aMX78eKZPn07//v1Zvnw5b7zxBqNGjaJr164cPXqUcuXKMXr0aHr16sXTTz9NVlYWx48f9/+CB4EjArxSKrweeOAB5s6dS1xcHIsXL2bKlCmsXLmScePGAZCSksKmTZuIi4ujU6dO1K9fH4C2bduSkJBAtWrVWL16NZdeeilgfQKoU6dOzvFvuummnMeJiYncdNNN7Nmzh5MnT3qdFz5//nx+/vlnAG6//XaeeOKJnOduuOGGQoM7wNy5c/npp58A6NmzJwcPHiQlJYWuXbvy6KOP0q9fP6699lrq169Px44dueuuu8jIyODqq6+mbdu2/l7CoNAAr5QDeGtpB0uLFi1ygh/AqFGjOHDgAB06dACs7tL33nuPXr165XndzJkziY+Pz/k5OjqazMxMjDG0aNGC+fPneyyvYsWKOY8feughHn30Ufr27cvMmTMZNmyYT3V2/6TvfjxvPHX5ighDhgyhT58+/PHHH3Tu3JmpU6fSrVs3Zs+ezcSJE7n99tt5/PHH6d+/v0/1CiZH9cErpUKjZ8+epKWl8eGHH+Zsc++W6NWrFx9++CEZGRmA1SVSWCrws88+m/379+cE+IyMDNasWeNx35SUFOrVqwfAF198kbO9cuXKpKam5vzcpUsXxo4dC8A333zDBRdc4Nc5duvWjW+++Qaw3phOOeUUqlSpwpYtW2jVqhVPPvkkHTp0YP369Wzfvp3atWtz7733cvfdd/PPP//4VVawOKoFr2OsSoWGiDB+/HgeeeQRRowYQa1atahYsSKvvfYaAPfccw8JCQmce+65GGOoVatWTv+3J3FxcYwbN45BgwaRkpJCZmYmgwcPpkWLgp9Mhg0bxg033EC9evXo3Lkz27ZtA6w+9+uvv54JEybw3nvvMXLkSO666y5ef/11atWqxZgxY/w6x2HDhnHnnXfSunVrKlSokPNm8s477zBjxgyio6Np3rw5l19+OWPHjuX1118nNjaWSpUq8eWXX/pVVrBIaZp5UrFiRVOcBT8eHruMCct3885Nbbm6Xb0g1Eyp0mfdunWcc8454a6GCiFPv3MROW6M8djn5IguGhdD6XmzUkqpcHNEgNc+eKWUKsgRAd6lFPU2KaVU2DkiwP+95SAAU9bsC3NNlFKq9HBEgHflolm643CYa6KUUqWHIwK8UkqpgjTAK6WKJTo6mrZt29KiRQvatGnDW2+9RXZ2NgBLlixh0KBBJS7jo48+8ntOeZcuXYpd3ueff87u3buL/Xqw5s+/8cYbJTpGoOiNTkqpYilfvjzLly8HICkpiVtvvZWUlBReeOEFOnTokJO2oLgyMzPzZKv0VUlywn/++ee0bNmSunXr+vyarKysIvPahIujWvCaVFKp8KhduzajR4/m/fffxxiTZ+EMT+l1AUaMGEGrVq1o06YNQ4YMAQqmBnZvDffo0YNHHnmEbt26cc4557B48WKuvfZamjRpwjPPPJNTl0qVKgGFpyZ+8cUX6dixIy1btuS+++7DGMO4ceNYsmQJ/fr1o23btpw4cYJp06bRrl07WrVqxV133UV6ujXe17BhQ1588UUuuOACfvzxR6/XZfny5XTu3JnWrVtzzTXXcPiwNU44cuRImjdvTuvWrbn55psLvU4l4agWvFIR688hsHdVYI95Wiu4fLjPuzdu3Jjs7GySkpLybPeUXvfPP/9k/PjxLFy4kAoVKnDo0KGc/d1TA+dPJBYXF8fs2bN59913ueqqq1i6dCk1atTgzDPP5JFHHqFmzZp59veUmviCCy7gwQcf5LnnngOsTJO///47119/Pe+//z5vvPEGHTp0IC0tjQEDBjBt2jSaNm1K//79+fDDDxk8eDAA5cqVY+7cuYVek/79+/Pee+/RvXt3nnvuOV544QXeeecdhg8fzrZt24iPjyc5OdnrdSopR7XgtYtGqfDylPrElV535MiRJCcnExMTw9SpU7nzzjupUKECADVq1MjZ3z01cH59+/YFoFWrVrRo0YI6deoQHx9P48aN2blzZ4H9XamJo6KiclITA8yYMYPzzjuPVq1aMX36dI+JzTZs2ECjRo1o2rQpAHfccQezZ8/2qZ5gJUVLTk6me/fuBV7funVr+vXrx9dff01MTIzX61RSDmvBa4RXEcqPlnawbN26lejoaGrXrs26detytntKr2uM8bpQT2GpfF2phqOiovKkHY6KivK4/J6n1MRpaWn85z//YcmSJTRo0IBhw4aRlpZW4LVF5enyJeWwNxMnTmT27Nn8+uuv/O9//2PNmjUer1OzZs2KXQY4rAWvlAqP/fv3M3DgQB588MECgdtTet3LLruMzz77LCfFsHsXTbC5gvkpp5zC0aNHcxYlgbwph5s1a0ZCQgKbN28G4KuvvsppjfuiatWqVK9ePWf5Ptfrs7Oz2blzJxdddBEjRowgOTmZo0ePerxOJeWwFrxSKlROnDhB27ZtycjIICYmhttvv51HH320wH6e0uvGx8ezfPlyOnToQFxcHFdccQWvvPJKSOpdrVo17r33Xlq1akXDhg3p2LFjznMDBgxg4MCBlC9fnvnz5zNmzBhuuOEGMjMz6dixo9+zer744gsGDhzI8ePHady4MWPGjCErK4vbbruNlJQUjDE88sgjVKtWjWeffbbAdSopR6QLbjhkIgCnVIpjyTOXBrpaSpVKmi448kR0umBdfFsppXI5IsDf0N5awHdAl4bhrYhSSpUijgjwFeOtoYTysaXzbjKlgqU0dbGq4CrO79oRAV6pSFSuXDkOHjyoQT4CGGM4ePCg3zc/OWoWjXbBq0hSv359EhMT2b9/f7irokKgXLly1K9f36/XBD3Ai0g0sATYZYy5MphlaUNGRZLY2FgaNWoU7mqoUiwUXTQPA+uK3EsppVRABTXAi0h9oA/wSTDLUUopVVCwW/DvAE8A2d52EJH7RGSJiCzxlEtCKaVU8QQtwIvIlUCSMWZpYfsZY0YbYzoYYzoEInuaUkopSzBb8F2BviKSAIwFeorI10EsTymllJugBXhjzFBjTH1jTEPgZmC6Mea2YJWnlFIqL73RSSmlHCoknd7GmJnAzKCXE+wClFKqDHFEC17vYFVKqYIcEeD1DlallCrIEQFeKaVUQY4I8NpFo5RSBTkiwCullCpIA7xSSjmUBnillHIoDfBKKeVQjgrwunSZUkrlclSAV0oplctRAV5EWLEzmdcmrQ93VZRSKuwcFeCNMVw1ah4fztwS7qoopVTYOSLAC3qnk1JK5eeIAK+UUqogxwZ4nVGjlIp0jg3wSikV6Rwb4LUBr5SKdM4N8OGugFJKhZljA7xSSkU6xwZ4HWRVSkU6RwR4XfBDKaUKckSA99RY1/a7UirSOSLAe6I9NEqpSOeIAK9dNEopVZAjArwnRjtplFIRzrEBXimlIp1jA7z2wSulIp2jArwGdaWUyuWIAK9jrEopVZAjArynhru25pVSkc4RAV4ppVRBjgjwnrpodJqkUirSBS3Ai0g5EVkkIitEZI2IvBCssgozevYWPpmzNRxFK6VUWMUE8djpQE9jzFERiQXmisifxpgFQSwzh6sP/pU/1gNwz4WNQ1GsUkqVGkEL8MbK13vU/jHW/gpZv4l20CilIl1Q++BFJFpElgNJwF/GmIXBLE8ppVSuoAZ4Y0yWMaYtUB/oJCIt8+8jIveJyBIRWZKZmVmy8tza7brgh1Iq0oVkFo0xJhmYCfT28NxoY0wHY0yHmJjA9Rg98O2ygB1LKaXKomDOoqklItXsx+WBS4D1wSoPQNwmTM7euD+YRSmlVKkXzFk0dYAvRCQa643kB2PM70Esj0PHTwbz8EopVaYEcxbNSqBdsI7vzrXgx4czt4SiOKWUKhMccSerUkqpgjTAK6WUQ2mAV0oph9IAr5RSDqUBXimlHEoDvFJKOZQGeC/mbNrPD4t3hrsaSilVbMG80SlkRDyvypqVXfx8NLd/ugiAGzs2KPYxlFIqnBzRgveWWGzt7iMhrolSSpUejgjwSimlCnJEgPfWReNls1JKRQRHBHillFIF+RTgReRMEYm3H/cQkUGuVMBKKaVKJ19b8D8BWSJyFvAp0Aj4Nmi18tPW/UeL3kkppSKMrwE+2xiTCVwDvGOMeQQr33upcOREyZb6U0opJ/I1wGeIyC3AHYBr0Y7Y4FSpGLwMpu5OPhHaeiilVCnia4C/EzgfeNkYs01EGgFfB69afvJyP9O70zaFth5KKVWK+HQnqzFmLTAIQESqA5WNMcODWTF/lIuLLnIfY4zX6ZRKKeVEvs6imSkiVUSkBrACGCMibwW3ar5rUbdKkfuUIGtBjh8W76ThkImkZWSV/GBKKRVkvnbRVDXGHAGuBcYYY9oDlwSvWv4JVbv87akbATh0TBf3VkqVfr4G+BgRqQPcSO4ga6nnJUWNUkpFBF8D/IvAZGCLMWaxiDQGSv0IZrC63PV9QylVFvg6yPoj8KPbz1uB64JVqWCwMk4WHfEXbD1IfztVcH46RKuUKkt8HWStLyK/iEiSiOwTkZ9EpH6wKxcOo2Zs5mRWdqH7eEtPrJRSpYmvXTRjgF+BukA94Dd7W5nxx+q9JT6GTrNUSpUlvgb4WsaYMcaYTPvrc6BWEOsVcIO+W1as173421qOn8ybCsG9AW+M4fXJ69m4L7Uk1VNKqYDzNcAfEJHbRCTa/roNOBjMivkjkA3r/K30z+ZtY/TsrV73/2vtPkbN2MItoxcErhJKKRUAvgb4u7CmSO4F9gDXY6UvKPOMMfy4ZCepaRle98ku5C6p+75aCsDxk/7d/PT5vG0s2nbIr9copZQ/fArwxpgdxpi+xphaxpjaxpirsW56KvOW7Uzm8XEreWb8asDzTJmZG/dzw0d/k5ltDb6mnCj4ZhAl0HDIRAaM8TwDx93elDSG/baWGz+eX6K6K6VUYUqyotOjAatFGD318yoA9qeme91nZWIKixMOs++Itc/zv64psI+ra2fmhv1FlvnRrC3FqapSSvmlJAG+1E8p8WU24/q9qT7v6+IpF42v4wD7U9OLnIbpiaZHKFx2tuGeLxYzb/OBcFdFqVKjJAG+1EwGlwC+1/gSqF1vBvO35I4z+1qDji9P5duFO/yq0+pdKZz7v78YtzTRr9eV1ImTWYyasZnNSaV/htDxjCymrkvivi+XhLsqSpUahQZ4EUkVkSMevlKx5sSXCiYA7zX+HMO158Fjud06xZ0jvzIxuch9NtifMv4Ocev0zSkbeH3yBi55azabdBqoUmVOoQHeGFPZGFPFw1dlY0yhaQ5EpIGIzBCRdSKyRkQeDmzVS8bTQKm/3Lt1ijtVs+/784q8M/b1yRuKd/ASOpqeO/9/75G0sNTBX6XmY6VSpUBJumiKkgn81xhzDtAZeEBEmgejIG9dNN7+2f9au482L0xhcULuNEV/+uA9BeSoEkzGL6r17wqu+UsdNWOzTrW0lfoBIaXCIGgB3hizxxjzj/04FViHleYgZPIH4qXbrWD4yPfLAZi6bl+xj71hbyrJx3MHPoMxCJqVbch0G5DNfz6vT95Q7KmWy3YcZvWulEL3cX/fCWX6nc1JqbR5YQp7UnRNXaVKwqdskiUlIg2BdsBCD8/dB9wHEBcXF9R67Dh0nPZn1Mjpevh4VsE7VH1tCfZ6Z7bf5Rd2w5QnPd+cyc5Dx3N+DmSMveaDvwFIGN4ngEcNjK/mbyflRAaTV+9lQNdGfr1W88AplSuYXTQAiEgl4CdgsL0qVB7GmNHGmA7GmA4xMYF9v3FNgfTHoeMl75v35r3pm/3af/vB4wFZajAQSkk1vNI8cEoVFNQALyKxWMH9G2PMz8EsyxfGeJ7DDnAkzWrV7zh4LGjlz95U9E1QLmt3F3gvDEPrNLxRs7S/qShV2gUtwIs1cvgpsM4YU2oW6HZ1TeS3bo8VUH2Z7rjDrdvEH8fSM4veyXbFyDkFtv26Yjcn/Mx5UxLpmbllhTLUlyQtcyCmzCrlFMFswXcFbgd6ishy++uKIJZXJGNyA3lJ+JtYLCk1jdW7UnzqMnrql1W8+uc6r8+f89wkr59CAi0zKzdYathUquwJ2iCrMWYuIWr4+Zwm4Kj3fDMAjYdODEqfd883ZuWZU57fsfRMKsbHkJaR5dNdrukZ2ZSLjfb43M5Dx4mPiaJ2lXLFrq+LBnWlyragD7KWJm9N2Vjo88Ea0CwsuAO89ddGu3zfKlBYN8SFI2bQ6ZVpvlfOdUxjePG3tWzdfzRnm3t9wrFMoT9F5r8XYvWuFKauLf40WKWcIKICfGntn3X1dQcyp46/th44xmfztnGPey6X0nm5fHLle3PznotSESiyAnwYAtaM9UlF7uNvYF8bgHEEb/alpPHw2GWkZWTlacGXlfVodR68UrkiKsCHw1wfEoS5YqevMfTW/1vI1wu2l6BW3h07mcWE5bsZu2hHgbVnS7Nwvf9kZmXz8NhluiavKpUcEeB9/d/ODNFdQ9nZptAFRPIrTmx6Z2rh4wnusrINSan+JQsb9ttan8cEwu3NKRt4bsLqsJS9fm8qE5bvZvDY5WEpX6nCOCLAlzbvTN1Ix5ensu9Iml/BO1it0DenbKDTy9NI8pAR0pU+wVMahXCHd/fys7MNMzckefwk8d70zfywJLHAa5SKdBrgg2DqOqvf/bxXpvHJ3G1F7j9z437+XLXHrzIOHC08udkHMzezcV8qKScy+GCmtUTgZrcZMi6Nn/qDlBMZ3Dx6QYHn3IOpp8CZnW144Jt/8mTlDARPb3SfzdvGgDGLmbR6b6GvPZmZzZQ1he+jVKRwRoAvIwOA3mw/eJz7v/knoMccMWkDl709mzYvTMnZ9u+vlnrc96Xf13LQQzbMonq0Uk5kMHHVHu75IvizVVxJ1/b5kJf+Pi/nqVSkCUk2yUgTrm6Ce75YXOjzqWmZzFifxD87DufZfsLLnbGFDaxu2X+UauVjASvQb9qXSpNTK/tZ45LJzMrm9k8XhbRMb7RrSJVGjmjBl7b2eyDSIRSHq2uoMHd+vrhAVktvg6neWvBLtx/m4jdnMWZeQs62S9/2P32yPzxVZV9qOvO3HvTwTOiU8Q+PIZOZlc22A8FL5Kc8c0SAj3Sfzt1WosUx/ljluc961kbP2S8T7H/UZTsPe3w+UDx9gnCfj+/L1M3e78xm+J/rA1ovJzp4NJ0DRaTyKIlX/ljPRW/MZHeyLuISShrgHeB/v6/l/FenB7cQU/DhvM3BaT27bvwqqr/dl1mc6/em8tGsLT6V+/3iHXQd7t91DPZM0l9X7GbiSv8G4Iuj/UtT6fDS1DzbLhwxnUvemhWQ4y+wP2kVZ+Wz7GwTsgR7TuOIAK8fk0PLW8v5+MmCOXf+b/ZWWj4/2edjT1i+i8/mWTOP/m+O9T0723i8hyHQwfXJn1axq5S1MAd9t4wHvg3sALyvdh46weakgjOvQu35X9fQ7NlJZJWW1W8CaP3eIxwOwnKfLo4I8E5RFu4rmrZuH1/O93wXrWutW3cv/7GuyGRr7h72cMPQg9/94zHLZmnILRSIxsWhYyfDNm4TKiW5Tt8stP7eSvvd1MXR+5059B01N2jH1wCvfHIkLYOt+49y9xdLWOVlse41Hlahcn99cT9m5x8jeG3Seq7/8G+/3hBnbih6ADpc+oycw+XvFlzgpbTwZ6GaQNqVfIKmz/yZM9gf5dCP6jsPBe9ToyMCfDizMEaKh8cup+ebhffHugJutofUCK2HTaHZs5MKrEg1bd0+v/tlP5y5hSXbD/vVfh8wZjErdiYX2H40PbPAXbyBbil+tWA7a3Z7flME2JNiXauMrOyAlhsoMzf4vtRkIE1evZeTmbnXxKHxPagcEeCdojgDUKWJKzC+N30znV6e5rE/2727JjUtg7u/WMKdnxc+f9+bT+du9Wv/5BN5F1Q/lp5Jy+cn89ok77NsjqVn+pRXaN2eIzQcMtHjc8+OX02fkdbH8M1JqV5nPD3186oiy1Fl19xNB/hkjn9/syWlAb4UWe6hhVmWGKxUAW/bidCKmpEye6OVaXN7IQudpxzPG5R/XbE75/HXC4pe/aowKXbAn7B8d57t7g343u/OpuPLeWeXuCvq06P7AFrSkTQueWu21xlP031ILV3W+bWIi+T/uWw34W/7dCEvTfS+HGcwaIAvRf4T4HQFobYnJY12L04pekdg2Y7DPs0OaZPveEu3B27uvSvWRBUSN4rqH80/0Lsr+UROWgWA/01cm/P4whEzCj2WPwFsf2o6k1YHf/oklI7B7NJk8Nhl9H6n+Df2bTtwjC6vek7+F2iOCPBl/I3dUY75uCD5QbdkaaH69Z3IN41z2jprSb/8gbUk4azr8OlcOGJGztxt9z7k9MzC+9j9+Tu+47NFDPz6H79mKBXXsh3JZAZofOD3lbuZs8l7n/60dfsYOW1Tkcf5e8sBGg6ZmOdeiV9X7OaSt2YVGEM5kpbB31uKXpfBV+OX72b93uLn//983jZ2p6Txh58JBovDGQE+3BVQJWKAX5Yleu3DDpSBX//DX27rtD43YQ0QnAbCf39c4ffcbfdqNBwy0eO0U5edh61PCVlZ/r8dpRzP4N4vlxSZmdPl07nbePOvvOsPpGdmMW5poscB6cysbI6kZeRpobqu8cezt3L7p4tY7WUm1t1fLMlZo7gwX/5tTZ38dfnunDfqR79fzuakowXumbj/66Xc+n8LC3T3ldT6vUf4cKZvN9ElHj5e9E5B4IgAr8qWtIysnLnNLo98vyIkZc/zsMJW/gC/xUNaZW+89cH/smwXAIeP+z5wnr8ermNAwXqnplkt94PH0kn2owywur3+WruPgV/7nnVzfb55+q2GTeGxH1fkecMEa4nKs57+k9bDphS6+PvsQlrxLr687778xzruLiKb6Qa7tX0ywLOU+r4/z+sAfXa2YeyiHTnrLT8Qpu5XDfAqpO76fDHNnp3EDLepd6H8BBbtocM9//zqy+zEaWMXeR7EfW7Cap4ct9Kn8hZsLTxXvnv+l8IGbPt9stDj9p5vzqLti39x40fzc4JJIKSm5W3tureJ96ScyOl6+mPVHg4fO8mLv60lIyu7wECxa7ps/nMryZz2GeuTmFRIzn9fBnK7Dp9eYFW0Sav35px3aloGOw4W3up2XQNPn2J+X7WHIT+v4r1pVmK/orrngsUR6YK1D77s8HaTVKh4GlDdfvB4gX/StbuPMMTLtEXXnbyvXd+6xPUZ8lPuG0Vhg73uPAWURQmH2JJ0jOZ1q+TZPm/zAfp9spArWp3GB/3ac+PH830qo9Uw74Pl7t1O45fvJjoqip/+SaRNg6oF/hc7vTyNn+7vUqDbpCT/soO+W+Zxe1FxwH2weFfyCd6ZuoneLU+jVqV42tt5eNo2qEa52Cg2Jx3jwNF0Eob3KbI+xhQs2zVDy/UJLlwzgBwR4JXyVZSXKJo/X05hc+MDZdG2Q3kGSYsKAo2HTmTmYxcxZ7NvNx5lZGXntPxddwMv2hbY1bcAsrKz7e/G45jDdR/+XWBbUS14b/mHwL9B8O0HjxW6+lnCgeOsSsxtdBRnqrKn+rjehF3nGahBan9pgFdhl1GMgcLicv3DvfT72jzb88/+8ZYq2Z2ndWx9tXT74QKt6V3JJ1ixM5k2Dap5Ls/AxFV7vM4IGT17C5nZhvdvPReg0NkqhfE0GOn+oSH/G1FaRnbOPt94yBnkSf74vnxnMr+63Y/wwLf/8KeXQWBvM4dcf0cJB4/R1F58pvvrM4usi6+/xY9nbaFVvaoFX28M+bBf/H8AACAASURBVD+TuP42XOe5yS1p2xf2J8Cxi3f6WHLxaYBXYReKqX4u0fZ/nC9r5RZl1IzN9DyndrFeuz/V8xzoq0bNY8HQi72+rrBPFuPtAPnWjdnExUSRme+N01sfvfsA7r+/WlLker/5ufrDl+7w/R4F9zeJLfuPcs0H8/K8iXgL7r5YvSuFlYkpNDut6BXGRPA5wr/qZV2BZ8avZvh1ebvrcu+x8P5JpSRTLX3liEHWsn6Hmwqd92dsDliumUUJh4KSAbTzq95nn/hi3uYDHEnL4P0ZeVfu8tav7j6jafKafUXeTObtv81Txk9vflyyk/lbDrJw60EufnNWia+j+7TL9MxsHvtxBVe+lzdLY3pGcLpJxi7eScKBYzQcMpGv5idgjMlJkBbu0KQteBVxpvmwtKGv8s/EKA027kvl2QmrSTyc9y7ckx5mcgz5aWWBlr4nszbuZ/bG/XRrWisgdVy/N5Vb/m+BX6/JyMomNtpzm9Q9mI9bmuhxnwtHzPA4aJoagE+QrrGOZyesISPLFOiDDxdHtODdW2QvXd0yjDVRZcE9XxY+b9qTdXuO0OaFKXmyZM7ZdMCndXA9CeYygq/+ub5AcPdm7OKdTMk3l92bL+cnACXrPikJX9MDFPYJZNSMzfzPbfxFIM/P+f2wZCcNh0xkZWLhg6/uCele/H0tc+1uL19nRgWL41rwZ9aqFO4qKAf6dO42Uk5kBCx1bkIRc6xLI1c7qrCAGExb9pd80e7XJ2/wa/8n7Psd+r4/r9D9oqIAtyEO199JuLuPHdGCd+fpRhalSsqVqz2S/7pceevDyT2RWyAEagglrZD+ffdpmKHmiBa8+zQ7L110SpWIK6VwtgOXjfPV2j1HilwIPdj8bYEX5d9f+Z6uoThGz97K6NmhzQHvLmjhUEQ+E5EkEVkdrDJc3s2TfS6S21gq2J78KbIX5TivkPwyoeC+HoAqWjDbu58DvYN4fI/CPS1JKaX8daufM4p8FbQAb4yZDQT+vugihHtaklJK+evvLQeDctyw98GLyH3AfQBxcXElP16Jj6CUUs4Q9iFJY8xoY0wHY0yHmJiSv99oA14ppSxhD/CBVtQiyEopFSmcF+A1viulFBDcaZLfAfOBs0UkUUTuDlZZwXZNu3rhroJSSvktmLNobjHG1DHGxBpj6htjPg1WWe4Ka8EnDO9DlXL+9/PXqVou53F8jHXJRlzXmsd7nU1bL7m7lVIq3BzXRVO9QuEzcfq0ruv3MXs2s3J+/3T/+TSrYy2JdtaplXjgorPy3Or86R0d/DpujKZVUEoFkaMC/Me3t6dutfK8dl2rnG1DLm+WZ58mtYtORjZu4Pl5fq5ZKZ6E4X1of0YNXr2mFd2a1qKFvfbly1e35Jw6VVjx3GVcfM6p/Lt7YwAqxEXz0W3tCy3nts5nADB/aM+iTy6Elj93KRXjosNdDaVUCTkiwP8negIDoifRq8VpAHRsWMPrvgO6NOT0GhUKPV6HhjVIGN6Hl6+xUg/Xrhyf81zzulX48q5OxMdYAbBlvar8+fCFVK0QC8DQy88hYXgf1r7Ym94tT+On+7sUOP6bN7Thu3s783Sfc5j+3+7UqVqe3x+6IOf5yYO75Tzu1eLUnMdVy8cWWu9AeP/WdlSrEMfsJy7Ks71crCP+VJSKKI74r30i9nuGxX6Z83Njt5TB+XNDRUUJ/yskZ/zNHRvkPO533hkkDO9Dxfjiz89vf0Z1nulzDnExUUwe3I2f7u/Cde3rc/6ZNYmNjsqpa0u3tR7PPq0ykwZfyJd3daJetdw3o5mP9fBYxke3tWfjS5fn2TbKXpfTk7NqV+I/Pc7Ms801ttCyrlWPmpXi8zz/UM8mnH1q7hJoN3ao7/FclVKlR9jvZA02T93c3ZvWYsT1rXNyPQNMfbQbGVmGc+w+9kC658LG3HNhY79e0+y0KjQ7LXchgTEDOlK9Yhwjb2nH/C0H2bL/KIu2WZkgmp5aibiYKBY9fTGdXraSQbWoW4V5Q3oSGy3M33KQh8cuB8izos193RpzND2TuOgoalaK59Cxk9Ry+7RSpVwMR9Ks1W7OrFWRUf3acclb1qILz/+rBe3PqM72g8eZMWs6qVTgp/v70HDIRAAanVKRbQdKnr9bqZI6VzayzZzGYXz93y64iHawbXnliqAc1xEt+MK48sNXytcK79smd7D149vbc1btykEJ7v5YMPTiAgsuX3tuPSYP7sZF9kBv3zZ1efXaVvzw7/PZ8FJvpv+3e86ngNqVy9GgRnnAmk1Ur1p5alcux1VtPU/zrFYhjvrVK1C7SjmioyRPcAdY8fxlbHzpcn75Txd6t6zDWbUrkzC8T86nmps6ns4TvZvxZ/xQ5sY/nOe1P+YbxwBoWNNz19iqYZfRsaHV+o8hkzoULy9HFNn0jZqHkE0U2VThKOVJo4XkXWBbyEaw8ndX50jO9i5Rq6mN7wtHA2yN78e3sS8V2D4n7mFui/4LgEocp51sKrBPcUSRzdS4x6hJ3hzj7l18RTPE4H2ZuvKkcVHUsjzbKnGcaLKIIpsGso9Logqm2S3892aYHvcoV0TlTap1uuzL+V0U9trKWHngy5FeaN09+Tl+GD/E/Y8bomfyRMzYQvf9V9TfJJTrRyvxLcVvY9mNL1nlz5RdNJWd1BfPC8YEax2LCGjBC2MGdKTJqXkHV10zWG7p1CCn7z7cTnObjukiIpztWh3++CFY/i1MeRq6DyH+oqG53VHZ2ZCdmdMllXNHb0YaLPiAGM4k089ft+xdRVxKIu2a+d+6qGAP0t7Z5Qx++Hs9VTjOt/07UuvDczgeXZmjmVF0TX8PgMrlYrmhfQP2bN+Y80bxeMZ9dJL1XPf010S9djoAhx7fT3pmFt8s2MH7MzZze/QU0ojj76wW7KIWI2JHc330bM7M3M3DMb8AsCm7Hk2idvFI3W+osH0a2a1v4dV1lwKwNfs0GkftZUTGjVw76A3O+vDWPOfQOO1rvot7iSaxB/ggrRcpVKSFJPBFVi9GxY6kgSQRJYYu0WtJiLZe+0FmX/4T8ysAL0WN4aXYMXmO2SHtQw5QhU9i36B51HZ6pw/n3KjNxJLJTTcPIPPHu3k980b2meqcJbtpFbWVM2U3H2X+i5tjZnBx3SzOStrN0nL3A/Bh5r+o330ALbM3klDuVtJMLN9WvpO7jo7OKfPvrOZ0ibZWYXoz43r+GzsOgEdPDuStuI+K/F1+lHklA2N+t34HUTWokW19cpyb1YLZ2a05O2on6SaWW2NmANA67f/4PO411mQ35Jbe3Tg6632qZVhLG34QN5LN2eM4Kyo37e/a7DNIpTwnTQwXRnvPLr4xux5No3YBkGrKk2wqMTzzFkbFjQTgvycHssXU5cXYMbSO2sby7MY8nPEgAE2idvF6lHVN9pnqXB69iDayhfJyEoBbTz7FqNiRVJejAPwW/wyZJooYyebdzGtIN3E8Eft9nvqMyezFnTGTWZXdkJpyhLpyiPtOPsJ10XOYk92Kl2LH0Cf9FZ4q/zNdswsuE/lDZndujJll/ZCyFqoG/n4bCdQK84FQsWJFc+xYMT7WD7P7r4fltmpcXQUf397eawA/kpZBxbiYot89Tx6DxCXQuHvRdTmaBJVqw7KvoWYTOP08n04BgEPbQKJg/wZocikc2w9bpkObm63nx/aD9b/nfc2Vb0OHu2DcXbD6Jx6KfYF+6WOp99BkGtSqCrNehxkvMSyjP9OrXltg8LSAxKXwSU+4Z7r1HXKva1YGrBpn1efoPoiJh/LVc6//I2vh7eZQrhq0vpHUbs9T+Y0A/9Gefj7smB/YYypVGgwr3spPInLcGFPR43NODfDfP3MVWUTR/bFvqVetfMkq9sMdsHY8DF4F1U73vt/aX+GH2/NuK18dHtsE0bFWC3zjJGh5Hcx5Cy4YDLHlYfcySN4BP/TPfV2La2HNz7k/12tvvXmk7Mx7/Aad4e7JudegENkVaxN1zG2R6LjKcDLVquMNX0DDC+BFewZSqxtg1Y9FHlMpFSAa4L3wEOBd2xIf2Er9WjV9P1bSOqh5lhWQXd7vCAc2wgOLoNbZ1rZjB+D72+H6z6BKHTi0FUa2837cbk/A7BEFt593Pyz80Pf6KaWcKQgB3vl98McOgLcAf2gr/HI/NO1ltZC/7Jv73FO7Ic6+Zkfs/sJRnXKfr90CktbAwo/g0hcKD+7gObiDBnelVNA4PsAXYIzVGj+laW5Q3ulhuaxXikhpkLTG+j7vHetLKaVKGcdPkyxg4cdWS/wFTRKmfFTtjNzHDywK7DHLFT12ElRdHoLnXCtrivXJ9fZf4O6pnvevdgbc72GQ+8avIMrDndZ3TYEugwpu73iPNZD/5Hbo8RQMTYSB8+Dc/nD+g9Z17vVK7v4Va0MHOyHtXZNh0HJ4bLM1sP98slVO/wnW833ehL7v57421p6eW/V0a9zJpcU1MGQnPLwSatj3qUTHW10lza+2fj67D9zxGzyxDZ49CAMmwuDVVpmnd4EbPof/boDuT+Ye9/bx1vP1O0LXwdZ5PZtvCunV+T65P1hw2mkgOLMPPv0ovGrN3th9xwLqNjqn4L4q/MpVs/75LhgM5/SFzBPWLKJ32/h/rPMfhPnvF75P79eg80Dr8c7F8Okl1uN7Z0DWSZj7DpzaHGqcCZOGQnoK1G0H983Me5zsbDhxGJZ8Bi2vhffsu4bvmmINxre/E5Z9aR2n1Q2w4U9Y8wtsmAgd74WzLoEze0KMnRjvtUZw4hA8ut4az3E5eSz3k2S/n6DyaXD8gFWnxCXW4P9/10FcJcjOgjlvwsxX4OZvYaw95XPoLuvc4itb40oHNlsTBWJ8WB7T9b/S5y04nGDNnjq1hbUtJRF2L4c/HodeL1vXweWTS63g2eKa3PN5ty0cdrsf4fnkohdvWPkj/HwP9BgKPYYUXV9fZWVCVHTB8tNTrb+/uIrW73jdBOvvMsrHvExZmdbvsVJtz8//0B/WToBnkqwZaAESeYOsI9tZ/etA+sBFxMfGQIWaUL6aBvhwqVwXHl4OL9W2gl73J+GUJp73dc1aAug3zhofmfaC9Y9e2Z7yevKY9Ub+ZlPr5+cO5c4AuuoDK4Au/Rw2/AHXfASje8BD/0C13FQUZKRZwa+clxvcjPFtBZlD26BqA4guosfzwGao3rDo/dwlrbe+125W+H5gBfnt86BRN+vaHNxkvRkU1/xRkHYELhpa/GPkt2OB9aZep3XR+2Znw/JvoPVNvr0hRajIC/DuQfw/C+GD87BuPS4951om1WwCA36HrTPhl3973qfTv2H9RGjcA5Z/nbvdnxkCJ5Kt4/cYUnSAGtMHUnfDoGWwY6F1s0jVgnlylHKqyA7wbW6FFd+WvHKl0fVjIGGO1VXgrv2d1hz6rg/DtBetbR3utubfSxSM6e3b8VteD6utux657lNodX3e55N3wDt2auaH/oEKNSC+KkTZQzs7FkD5GrB/PTTvi1Iq8CJrmuTJfG8QZSW4t7wOts227l4FuGea1cWwb7V1IxRYAz4bJlpBtPvjVr9ny2utPt3kndDpXjDZVl/rv+yZPZXrwMbJcOVbuWW53gizs607UiufBpnp8PKpViBv2gtiyoFE5wb4Om0L1rna6XDR09aAVM0zCz5/emfre62mJb48Sin/Oa8FfyIZXjuj8P2D7Zn9Vp/h8UOQcdy6eeqb663R+nc99D1WrgP/XW/dPPW6HSjduzSMsVIWnNkz9KuKZ2XC/nVwWqui91VKhVxkteAlTDM/b/4Oxt5iPXYNCFWoAdSw+oS99UE/us4K8AAVT7Gmh51IzruPCJx1ccHXhkJ0jAZ3pcooZwX4kn4aGboL3mhitbrdxVeBdDutbOcHoOllsG8tTHabXdDUx37tCx6F+h2g8UUQHVdwRoUrFYJSSpWQwwJ8NsWeKdP6ZoivBD2fgclP5W6vXBceXAw/3mHNIult33xRr4MV4FvfZPVh+9p1csnzxaufUkr5yVl98GDdLPJWEXOGB0yEz/vAqS2tQUzI7UIxBo4fzO0L73iPdWecL9aMt+bbN7rQv/orpVQxRVYf/Ny3i96n4QW5AT1xCexyu01YxOoL7zoY1v0KvV71vewWV/tXV6WUCiLnteCL0uEua5EMpZRygMJa8I5INpYY5+OC1l0e0uCulIoYjgjw9S+617cdL3khuBVRSqlSxBEB3qvTz8993PY237PCKaWUAzgjwHuboihuAb33K573UUoph3JGgPemesPcx+FeWEEppULM2QG+tx9THJVSymGcNw/e5cye1kIOxVypXCmlyjpntOCzswpuu/aT0NdDKaVKkaAGeBHpLSIbRGSziARwUcV8DmwouC2+UtCKU0qpsiBoAV5EooFRwOVAc+AWEWkelMIOb8/783WfBnRRW6WUKouC2QffCdhsjNkKICJjgauAtQEv6aavrYU0Bi2D8tUDfnillCqLgtlFUw/Y6fZzor0tDxG5T0SWiMiSzMzM4pVUrgo8maDBXSml3AQzwHu6+6hAZjNjzGhjTAdjTIeYGOdO6lFKqVALZoBPBBq4/Vwf2B3E8pRSSrkJZoBfDDQRkUYiEgfcDPwaxPKUUkq5CVqfiDEmU0QeBCYD0cBnxpg1wSpPKaVUXs5Y8EMppSKU4xf8UEopVZAGeKWUcigN8Eop5VClqg9eRE4CK4v58lOAAwGsTlmk10CvQaSfP0TeNWhtjInz9ESpCvAlISJLjDEdwl2PcNJroNcg0s8f9Bq40y4apZRyKA3wSinlUE4K8KPDXYFSQK+BXoNIP3/Qa5DDMX3wSiml8nJSC14ppZQbDfBKKeVQZT7Ah2zd1xARkc9EJElEVrttqyEif4nIJvt7dbfnhtrnvkFEerltby8iq+znRoqI2NvjReR7e/tCEWkYyvMriog0EJEZIrJORNaIyMP29ki6BuVEZJGIrLCvwQv29oi5Bi4iEi0iy0Tkd/vniLsGJWKMKbNfWFkqtwCNgThgBdA83PUq4Tl1A84FVrttGwEMsR8PAV6zHze3zzkeaGRfi2j7uUXA+VgLr/wJXG5v/w/wkf34ZuD7cJ9zvvOvA5xrP64MbLTPM5KugQCV7MexwEKgcyRdA7dr8SjwLfB7pP0vBOT6hbsCJfzlnw9Mdvt5KDA03PUKwHk1zBfgNwB17Md1gA2ezhcrNfP59j7r3bbfAnzsvo/9OAbrjj8J9zkXci0mAJdG6jUAKgD/AOdF2jXAWiRoGtDTLcBH1DUo6VdZ76Lxad1XBzjVGLMHwP5e297u7fzr2Y/zb8/zGmNMJpAC1AxazUvA/sjcDqsFG1HXwO6aWA4kAX8ZYyLuGgDvAE8A2W7bIu0alEhZD/A+rfvqYN7Ov7DrUiaumYhUAn4CBhtjjhS2q4dtZf4aGGOyjDFtsVqxnUSkZSG7O+4aiMiVQJIxZqmvL/GwrUxfg0Ao6wE+UtZ93ScidQDs70n2dm/nn2g/zr89z2tEJAaoChwKWs2LQURisYL7N8aYn+3NEXUNXIwxycBMoDeRdQ26An1FJAEYC/QUka+JrGtQYmU9wEfKuq+/AnfYj+/A6pd2bb/Zng3QCGgCLLI/uqaKSGd7xkD/fK9xHet6YLqxOyFLA7u+nwLrjDFvuT0VSdeglohUsx+XBy4B1hNB18AYM9QYU98Y0xDr/3q6MeY2IugaBES4BwFK+gVcgTXTYgvwdLjrE4Dz+Q7YA2RgtTDuxuoXnAZssr/XcNv/afvcN2DPDrC3dwBW28+9T+5dy+WAH4HNWLMLGof7nPOd/wVYH5NXAsvtrysi7Bq0BpbZ12A18Jy9PWKuQb7r0YPcQdaIvAbF/dJUBUop5VBlvYtGKaWUFxrglVLKoTTAK6WUQ2mAV0oph9IAr5RSDqUBXjmGiBy1vzcUkVsDfOyn8v38dyCPr1QwaIBXTtQQ8CvAi0h0EbvkCfDGmC5+1kmpkNMAr5xoOHChiCwXkUfsxF2vi8hiEVkpIv8GEJEeYuWe/xZYZW8bLyJL7Tzs99nbhgPl7eN9Y29zfVoQ+9ir7ZzjN7kde6aIjBOR9SLyjVse8uEistauyxshvzoqYsSEuwJKBcEQ4DFjzJUAdqBOMcZ0FJF4YJ6ITLH37QS0NMZss3++yxhzyE4RsFhEfjLGDBGRB42V/Cu/a4G2QBvgFPs1s+3n2gEtsHKfzAO6isha4BqgmTHGuFISKBUM2oJXkeAyoL+dfnch1u3uTeznFrkFd4BBIrICWICViKoJhbsA+M5Y2R/3AbOAjm7HTjTGZGOlXGgIHAHSgE9E5FrgeInPTikvNMCrSCDAQ8aYtvZXI2OMqwV/LGcnkR5Yib3ON8a0wcoHU86HY3uT7vY4C4gxVt7xTljZMq8GJvl1Jkr5QQO8cqJUrOX+XCYD99tpiBGRpiJS0cPrqgKHjTHHRaQZ1jJ5Lhmu1+czG7jJ7uevhbXk4iJvFbPz3Fc1xvwBDMbq3lEqKLQPXjnRSiDT7mr5HHgXq3vkH3ugcz9W6zm/ScBAEVmJlZFwgdtzo4GVIvKPMaaf2/ZfsJaGW4GVBfMJY8xe+w3Ck8rABBEph9X6f6R4p6hU0TSbpFJKOZR20SillENpgFdKKYfSAK+UUg6lAV4ppRxKA7xSSjmUBnillHIoDfBKKeVQGuCVUsqhNMArpZRDaYBXSimH0gCvlFIOpQFeKaUcSgO8Uko5lAZ4pZRyqFKVD15EJmGta1kcpwAHAlid0l5uOMvWc3Z+ueEsW8/ZPweMMb09PeGYfPAissQY0yFSyg1n2XrOzi83nGXrOQeOdtEopZRDaYBXSimHclKAHx1h5YazbD1n55cbzrL1nAPEMX3wSiml8nJSC14ppZQbDfBKKeVQZT7Ai0hvEdkgIptFZEiQy/pMRJJEZLXbthoi8peIbLK/Vw9CuQ1EZIaIrBORNSLycCjKFpFyIrJIRFbY5b4QinLz1SFaRJaJyO+hKltEEkRklYgsF5EloSrXLqeaiIwTkfX27/v8EPyez7bP1fV1REQGh/CcH7H/vlaLyHf2310ofs8P22WuEZHB9raglOtv7BCRoXZM2yAivYpbbpkO8CISDYwCLgeaA7eISPMgFvk5kP+GgiHANGNME2Ca/XOgZQL/NcacA3QGHrDPM9hlpwM9jTFtgLZAbxHpHIJy3T0MrHP7OVRlX2SMaes2NzlU5b4LTDLGNAPaYJ17UMs2xmywz7Ut0B44DvwS7HIBRKQeMAjoYIxpCUQDNwe7bBFpCdwLdMK6zleKSJMglvs5PsYO+3/7ZqCF/ZoP7FjnP2NMmf0Czgcmu/08FBga5DIbAqvdft4A1LEf1wE2hOC8JwCXhrJsoALwD3BeqMoF6tt/+D2B30N1vYEE4JR820JRbhVgG/bkh3D8jQGXAfNCeM71gJ1ADaw763+36xDUsoEbgE/cfn4WeCKY5foaO/LHMWAycH5xyizTLXhy/zhcEu1toXSqMWYPgP29djALE5GGQDtgYSjKtrtIlgNJwF/GmJCUa3sH658u221bKMo2wBQRWSoi94Ww3MbAfmCM3S31iYhUDFHZLjcD39mPg16uMWYX8AawA9gDpBhjpoSg7NVANxGpKSIVgCuABiEo1523sgIW18p6gBcP2xw771NEKgE/AYONMUdCUaYxJstYH93rA53sj7ZBJyJXAknGmKWhKC+frsaYc7G6/h4QkW4hKjcGOBf40BjTDjhGcLu/8hCROKAv8GMIy6wOXAU0AuoCFUXktmCXa4xZB7wG/AVMAlZgdYWWBgGLa2U9wCdiveu61Ad2h7gO+0SkDoD9PSkYhYhILFZw/8YY83MoywYwxiQDM7H6BENRblegr4gkAGOBniLydSjKNsbstr8nYfVFdwpFuVh/z4n2pySAcVgBP1S/58uBf4wx++yfQ1HuJcA2Y8x+Y0wG8DPQJRRlG2M+Ncaca4zpBhwCNoWiXDfeygpYXCvrAX4x0EREGtmtj5uBX0Nch1+BO+zHd2D1jweUiAjwKbDOGPNWqMoWkVoiUs1+XB7rn3F9sMsFMMYMNcbUN8Y0xPq9TjfG3BbsskWkoohUdj3G6g9eHexyAYwxe4GdInK2veliYG0oyrbdQm73DCEqdwfQWUQq2H/nF2MNLIfi/6q2/f104Fqscw/VtaaQsn4FbhaReBFpBDQBFhWrhEAPmoT6C6vvbCOwBXg6yGV9h9VPmIH1Lns3UBNrIHCT/b1GEMq9AOsj2kpguf11RbDLBloDy+xyVwPP2duDfs756tGD3EHWYJ9zY6yP6yuANa6/qVCdM9ZspSX2NR8PVA/R31gF4CBQ1W1bqM75BayGw2rgKyA+ROc8B+sNdAVwcTDP2d/YATxtx7QNwOXFLVdTFSillEOV9S4apZRSXmiAV0oph9IAr5RSDqUBXimlHEoDvFJKOZQGeOV4IpKVL1tiwO4OFZGG7hkClSpNYsJdAaVC4ISx0i0oFVG0Ba8illh5318TK+f9IhE5y95+hohME5GV9vfT7e2nisgvYuXHXyEiXexDRYvI/9l5xafYd/0iIoNEZK19nLFhOk0VwTTAq0hQPl8XzU1uzx0xxnQC3sfKXon9+EtjTGvgG2CkvX0kMMtY+fHPxbrTFaxbyUcZY1oAycB19vYhQDv7OAODdXJKeaN3sirHE5GjxphKHrYnYC1ostVO5rbXGFNTRA5g5enOsLfvMcacIiL7gfrGmHS3YzTESqPcxP75SSDWGPOSiEwCjmKlHBhvjDka5FNVKg9twatIZ7w89raPJ+luj7PIHdvqg7XiWHtgqYjomJcKKQ3wKtLd5PZ9vv34b6wMlgD9gLn242nA/ZCzEEoVbwcVkSiggTFmBtaiJdWAAp8ilAombVGoSFDeXpXKZZIxxjVVMl5EFmI1dm6xtw0CPhORx7FWWLrT3v4wMFpE7sZqqd+PlSHQk2jgaxGpbh9U/gAAAE5JREFUirWAw9vGyqmvVMhoH7yKWHYffAdjzIFw10WpYNAuGqWUcihtwSullENpC14ppRxKA7xSSjmUBnillHIoDfBKKeVQGuCVUsqh/h+6wAyUX7HARwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "##########################\n",
    "### Evaluation\n",
    "##########################\n",
    "\n",
    "ax1 = plt.subplot(1, 1, 1)\n",
    "ax1.plot(range(len(gener_costs)), gener_costs, label='Generator loss')\n",
    "ax1.plot(range(len(discr_costs)), discr_costs, label='Discriminator loss')\n",
    "ax1.set_xlabel('Iterations')\n",
    "ax1.set_ylabel('Loss')\n",
    "ax1.legend()\n",
    "\n",
    "# Set scond x-axis\n",
    "ax2 = ax1.twiny()\n",
    "newlabel = list(range(NUM_EPOCHS+1))\n",
    "iter_per_epoch = len(train_loader)\n",
    "newpos = [e*iter_per_epoch for e in newlabel]\n",
    "\n",
    "ax2.set_xticklabels(newlabel[::10])\n",
    "ax2.set_xticks(newpos[::10])\n",
    "\n",
    "ax2.xaxis.set_ticks_position('bottom')\n",
    "ax2.xaxis.set_label_position('bottom')\n",
    "ax2.spines['bottom'].set_position(('outward', 45))\n",
    "ax2.set_xlabel('Epochs')\n",
    "ax2.set_xlim(ax1.get_xlim())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABEUAAACoCAYAAAAGjlD3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZBU5bnH8edV2XdwWAOCigsoio4reEuiKC7RGHPjUgpiXEoxakwZFTXGqFHDdYlbSi4SNdGoJSYS4hKMWBJLQURNEEWUGnAQWWQbkG3wvX8wucU5zzPMS2/ndJ/vpyol58fp7rdnztOnOZn+jfPeCwAAAAAAQNbskvQCAAAAAAAAksBFEQAAAAAAkElcFAEAAAAAAJnERREAAAAAAJBJXBQBAAAAAACZxEURAAAAAACQSXldFHHOjXDOzXPOfeacu75QiwKQH2YTSCdmE0gnZhNIJ2YTpeC897nd0LldReRTERkuIrUi8q6InOO9n9vYbXbffXfft2/fnB4Ple+9995b4b2vSnod5Y7ZRKExm4WRhdm03lM45xJYSTYwm4WRhdlEaTGbhcFsopBqampkxYoV5puS3fK438NF5DPv/QIREefcMyJyuog0epD27dtXZs2alcdDlhZv7qK+/fZble2yS+E+geWcW1iwO8u2ip/NJOR6/FfC6wizWTBlO5uhx/HmzZtV1rx586Ks6T8qYcZyxWwWTNnOJtKJ2SwYZhMFU11d3ejf5fMv2l4i8sV227UNWYRz7hLn3Czn3Kzly5fn8XAAAjGbQDoxm0A6MZtAOjGbKIl8LopY/zeM+r9rvPfjvffV3vvqqip+igwoAWYTSCdmE0gnZhNIJ2YTJZHPRZFaEem93fZ3ROTL/JYDoACYTSCdmE0gnZhNIJ2YTZREPp0i74pIf+dcPxFZLCJni8i5hVhUrp9J3rp1q8p23XVXlS1btkxlXbt2VVm5fya50B0g1m3jj1HIjhHkrGizWexemWJ3A+Rz/7k+z3J7Hcm1fBtBijabxWadl1u0aKGyYveHWJKYsWK/Flqs9zgomLKdTSRv6dKlSS+hkjGbyJl1rm5MzhdFvPf1zrkrRORVEdlVRCZ67z/K9f4AFAazCaQTswmkE7MJpBOziVLJ5ydFxHv/koi8VKC1ACgQZhNIJ2YTSCdmE0gnZhOlwGcdAAAAAABAJnFRBAAAAAAAZFJeH58pllyL2qxSVYtVqlqJ6urqVNahQ4eCPkauBXeUOZanYhcJhh5PoaXKud5/lvE1Sr9CFhKHFoZapaq5ynV+kzJ37lyVDRgwIOf727JlS2S7WbNmQbdL89cIyLJu3bolvQQAhp35dws/KQIAAAAAADKJiyIAAAAAACCTuCgCAAAAAAAyiYsiAAAAAAAgkxIvWl29erXKOnbsmNN9FbJ8rhSs9a5du1Zlbdu2VVm8OMYqyyt0qaolzV9fpEehixUpHAQKo9jlyZZ85rfY5/l4CapIeKlq6NpCilVDC3ABNG3dunUqs95bA8guzrAAAAAAACCTuCgCAAAAAAAyiYsiAAAAAAAgkxLvFGnXrl3B7qvc+i3q6+tVtmHDBpXttpv+NrVp0yayvWjRIrVPv379VGZ9Xjr062atI94VYX1WvNy6XlB4fBY+KrRjxZodK7O+vtZ+cdYchtwOycry66fVtWHNzoQJE1T2gx/8ILL94osvqn2GDh2qsv79+wetLfT7EnLe5DUTlWrTpk0qa9GiRVEfk/4QAE3hrAsAAAAAADKJiyIAAAAAACCTuCgCAAAAAAAyKa9OEedcjYjUichWEan33lcXYlEA8sNsAunEbALpxGwC6cRsohQKUbQ6zHu/ItcbWwVjlnj5X66FZjvzmLmyigrHjBmjsvvvv19l3bt3z+kxrVJVi/XcrcLX5s2bB+0XIsulgAnLazYLyToGrOPJKvMtNus1wpphaz+rGLlly5aR7ZkzZ6p9evbsqbI1a9ao7KCDDlKZVZZszbU1wyE2b96c0+2wUxKZzUoovbaOdat89fPPP1dZr169ItvW+XbSpEkqK3QxZPw5VML3pYKk5rwZcn4RsY+VVatWqeyTTz5R2VFHHRXZXrx4sdrntddeU9nTTz+tsqlTp6rs8ssvV9n555+vso4dO0a2u3Tpovbp1KmTyor9fh6pkprZLLQFCxao7K233ops/+QnP1H7WO8bLc2aNVOZ9XpgnTePP/74yHYln5v4+AwAAAAAAMikfC+KeBH5u3PuPefcJYVYEICCYDaBdGI2gXRiNoF0YjZRdPn+rPoQ7/2XzrmuIjLVOfeJ9/7N7XdoOHgvERHp06dPng8HIBCzCaQTswmkE7MJpBOziaLL6ydFvPdfNvx3mYj8WUQON/YZ772v9t5XV1VV5fNwAAIxm0A6MZtAOjGbQDoxmyiFnH9SxDnXRkR28d7XNfz5BBH51c7ej1Uw1sjj5XS7JEqYrALJBx98UGVpKYiyyi2XL1+uslxfZCiRK61CzWax5VOqahUrxotQQwtJrUK6Cy+8UGXTpk1TmVXAGH8M61jv27evylq1aqWyG264QWULFy5U2XXXXaey+Ndol13CroHnUyCJHUt6Niv1ddc6tuOlqiIi++yzT2T7iiuuUPvsueeeKstnJkLK3q3vSxIl8VmW9GxarHOCxTofWsfssmXLVNa/f//I9ldffaX2sc7Vq1evDlrbww8/HJT16NEjsl1drX+5SHytIiLjxo1TWei5DuUhjbNpWb9+vcqmTJmiMusXYxx33HEqW7duXWEWJvb74RNPPFFl1uzMmzevYOtIu3w+PtNNRP7ccDLfTUSe9t6/UpBVAcgHswmkE7MJpBOzCaQTs4mSyPmiiPd+gYjo3xcJIFHMJpBOzCaQTswmkE7MJkqFnzEDAAAAAACZxEURAAAAAACQSfn+St685Vr8lpbCuA0bNqjMKrgqdPFTvLzUKnd96qmnVLZo0SKV3XTTTSpr27ZtHquLSsv3CpUj5Jiy5nDmzJkqu/POO1VmlapaM2YV0LVr167Jferq6lR21llnBe3XtWtXlVllxpTNISus8/ADDzygsvnz50e2a2tr1T5WwanFKre0ivHat28fdH9xlKpWtpAibOucYx0Xn332mcrmzp2rMqvgtGXLlpHtAQMGqH3icyNir9eaiVBLliyJbP/1r39tcp/G1gEUm/Wey8r2339/ld1+++0qK2SpqsUqd7XeX1prmz59emT7mGOOKdzCUoZXEwAAAAAAkElcFAEAAAAAAJnERREAAAAAAJBJXBQBAAAAAACZlHjRarFZxTf5FH/G788qUQwtfgpd25YtW1Q2ceLEyHa3bt3UPtddd53KHnvsMZVZBT/NmzdXmeWMM86IbP/ud79T+3Tv3j3ovoBQVqFbvIDub3/7m9rnvPPOU1nr1q1V1rdvX5Xdd999Kjv++ONVFi+veuaZZ9Q+nTt3VtngwYNV1qdPH5U1a9ZMZZs3b1ZZvGh248aNap94yR6Qdlb55FVXXaWyzz//vMn7evPNN1V28cUXB63DOs9bBeXWeT7+2nTqqacGPSYqR8j7ROu9n1WW//7776ts4MCBKhs3bpzKbr755sj2hx9+qPax3g9ahYzWsb7XXnup7I477lDZnDlzItvWe+uOHTuqDEhC6L/VrDm0Zj9e0C8SVr560EEHqWzChAlB67BeN/7whz+oLNey8HLET4oAAAAAAIBM4qIIAAAAAADIJC6KAAAAAACATOKiCAAAAAAAyKSKL1rNp1TVKs2JlznGt0VEvv76a5W98MILKhs5cqTK1q5dq7La2lqVzZ49O7I9efJktU+HDh1UdvbZZ6vswAMPVNmjjz6qsgMOOEBlf/7zn1WGymAd/1bJZ7FZ5W3x419E5M4774xsW8emVXB18MEHq8wqaW3Tps0O1/kf8bk77LDD1D4rVqxQWVVVVdA6jj76aJVZBY/xolVKVVFo1mxu3bpVZVZpYq6s1yWrHM4qY46/H7BmomfPnjmvzXq/Ya2DYlWEeOCBB1R2ySWXqMwq/Lberx1zzDEqixcSX3DBBWqfc889V2VWgao151bB9xNPPKGyLl26RLbPPPNMtQ/nMKSFdR7q1KmTyqxzpDUnQ4YMUVn83GH9O9IqQQ59n26VtFZXV6uskOfvtOMnRQAAAAAAQCZxUQQAAAAAAGQSF0UAAAAAAEAmNXlRxDk30Tm3zDk3Z7uss3NuqnNufsN/9QepABQVswmkE7MJpBOzCaQTs4mkhbSnPC4iD4nIk9tl14vIP7z3dznnrm/Yvi7kAePFMVbxYYi6ujqVtWvXLqf7aoxVLhMvSHzxxRfVPlbp6RdffKGyuXPnqmzq1Kkqs0ol6+vrI9vLli1T+yxdulRl++67b9Bj/uIXv1DZPffco7IQ1tfo9NNPz+m+EPG4FHA249JcrmTNulVKGmcVklrlVa1bt1ZZaGnz4sWLI9tWSV2/fv1U9txzz6mspqZGZVbBl1WgHMIqAcunnBr/73Ep4mxa0vK9LOTrhvWcrIJyq8zRem9xzTXXRLatEuTQ9yRWoaxVvJ7re5yVK1eqrHPnzjndFyIelxLPZgjrWP/5z3+uspkzZ6rMKlX98ssvVTZhwgSVxYtWH3zwQbXPjTfeqLI1a9aobPPmzSrbsGGDyqzz2iOPPBLZvuOOO9Q+F154ocqsczXK1uOSwtm0WGWm1nszaw7/+Mc/qmzOnDkq+/TTTyPbkyZNUvtY/5ayfsmGVfhtzU5Iefrw4cPVPtOmTVNZOWrybO29f1NE4mfn00XkP/XRT4jI9wu8LgBNYDaBdGI2gXRiNoF0YjaRtFw7Rbp575eIiDT8t2tjOzrnLnHOzXLOzVq+fHmODwcgELMJpBOzCaQTswmkE7OJkil60ar3frz3vtp7X11VVVXshwMQiNkE0onZBNKJ2QTSidlEvnL98O9S51wP7/0S51wPEdGFFo2If7429HPQ8f3Wr1+v9rH6Aqz7sj73aH0W2LrS+Oyzz0a2f/vb36p9lixZorILLrhAZeedd57Kxo4dq7L33ntPZfHPQv/+979X+xx00EEqsz4HZ32u7LbbblNZyOfTre/nKaec0uTtUDA5z2ZcaB9B/HueT4+BdfxYn4V87bXXVBbv2bFcddVVKuvUSfd2WT0A1v1bfTyvv/56ZNt6HbG6fe6++26VjR49WmWF7E6iP6SkCjabFuvzzM2bNy/Y/VtzaJ1LrXOMdX4NYfUnTJkyJei21nM/6aSTItvDhg3LaV0i9nOyvkYhM2bdjv6QkirqbIYIfS0+4ogjgvZ76KGHVDZo0CCV9e7dO7JtdSBY/SFWj9VXX32lsnnz5qmsTZs2Krv00ksj27fccovah/6QTEp8NkNZ577vfOc7Knv11VdVtv/++6vs1FNPjWxb/xYcM2aMyqw+yRYtWqjsxz/+scqsTrAuXbpEtgcPHqz2qRS5/qTIZBEZ1fDnUSKimzQBJIHZBNKJ2QTSidkE0onZRMmE/EreP4nI2yKyr3Ou1jn3YxG5S0SGO+fmi8jwhm0AJcRsAunEbALpxGwC6cRsImlNfnzGe39OI391XIHXAmAnMJtAOjGbQDoxm0A6MZtIWtGLVgEAAAAAANIo16LVggktl4rvZ5WQhd6XVcBmFZ3Nnz9fZW+88UZke/HixWofq0D1pptuUlmvXr1UZhXIWqU5Dz/8cGTbKtGpqalR2fTp01Vmfd2sIixLvGjP+tpaxT1WoSbKUyHLOkPva9KkSSoLKXz94Q9/qLKtW7cGPebcuXNVds0116gsPq9WC/pZZ52lsqOOOkple+yxh8o2bNigspCCR0pVK4f1+lnIUtXQAvSVK1eqrEePHjk95sKFC1U2atQolVlFw1bh6wknnKCyeLFqoWfCKmi2vpbxGW7VqlVB14HkWK/F1nFRbFYBo/WLAWbMmBHZ3rhxo9pnr732Upn1XtViHf/t27dXWV1dXWT7nHP0Dw2Evi7lI/5+INeSaEDEPmZPPPFElZ1xxhkqi5cU19bWqn3ic9MY6xeRvPTSSyqz/i0Zn4HQxyxH/KQIAAAAAADIJC6KAAAAAACATOKiCAAAAAAAyCQuigAAAAAAgExKvGjVElKmtGrVKrVPy5YtVWYVhlqlV19//bXK3nnnHZX95S9/UVlcvLhKROSiiy5SmVXAOHv2bJUtWLBAZVYZVpxV+NWsWTOVbdmyRWWhpX2UNyJXoaVpVhGqVVIcn/XQ+frkk09U1rp1a5Xde++9QbeNP4dBgwapfQ455BCVWWV2mzZtUplVNGmhzLhyhb7uhpQGWucJi3WOtGYzZG319fUqe/3111Vmla+GnPtE7DkJKWMOZc2XlVnvN1asWBHZ7t27d87rQLokUapqsY7tyy67TGVPPPFEZHvEiBFqnylTpqhszpw5Oa/N+gUFe++9d2R78uTJap/TTjst58cMRbEqCin0F01Yv4yjuro6sp3Pa8u6deuC9gs5v1pzeOyxx6rspz/9qcrS8vrYmHSvDgAAAAAAoEi4KAIAAAAAADKJiyIAAAAAACCTuCgCAAAAAAAyKfGi1dCyxbjOnTurzCqSsUrkrMccPny4yj788MMm12H56KOPgrJQ1tcjnlnP0/oajRo1SmXWcz/rrLNUdsABB6jMKm4NQUErrGPAms1ly5aprKamRmXffPNNZLu2tlbtYxU/7bnnnip7//33VdaiRQuVWSVa3bt3j2wff/zxap/9999fZRar9G3z5s0qs4qRmTGElAZaxWfWHO6+++4qe/7551VmlYbGC1kXLVqk9rn77rubvF1jDjzwQJXdeuutQbfNlTWHb7zxhsoOPvhglcW/RmvWrFH7dOjQIffFAQbrHGad6+LeeuutnB/TOg9Zx/aECRMi22+//bbaZ99991VZ//79VWa9flnnaiAJ1kw888wzTd4utBQ99L11rqzi5VdeeUVlRx99dNA6rP2Swk+KAAAAAACATOKiCAAAAAAAyCQuigAAAAAAgExq8qKIc26ic26Zc27OdtkvnXOLnXMfNPzv5OIuE0AcswmkE7MJpBOzCaQTs4mkhTQPPS4iD4nIk7H8Pu/9/+zMg3nvZePGjZHMKke1Ct3irIJPq8z0iCOOUNnq1atVVldX1+RjloJVBlVfX6+ykNKcpUuXqsx67tbXqEePHk3ePxL3uBRoNkV0qeGWLVvUPi1bttzZu90pVpHUypUrVXbooYeqbP78+ZHtFStWqH2sQsPZs2cHrS3+2iUi0rVrV5XFi1UvuugitU9owbRVlGm9RliFlCElmyiax6WAsxlnHT9WZpWohrCOResYGzFiRNA61q5dG9l+6KGH1D69evVS2YIFC1RmlUVa2R577KGyXL8eFusxjzzySJVZ5+84q3jSei0s5Poz7HEp4mxWojFjxqjsnXfeUZn1nsGak+eee05l/fr1i2xb70tfe+01lY0fP15lVmkzysLjktHZtN5bX3zxxZHtp59+Wu1j/VKMG2+8UWWnnHKKyj755JOdWeIOWee5YcOGqSxeqCwisnDhQpWdc845hVnYTmryDOu9f1NE9L9KACSK2QTSidkE0onZBNKJ2UTS8vm/Ha5wzv2r4cedOjW2k3PuEufcLOfcrOXLl+fxcAACMZtAOjGbQDoxm0A6MZsoiVwvivxORPYSkYNFZImI3NPYjt778d77au99dVVVVY4PByAQswmkE7MJpBOzCaQTs4mSyemiiPd+qfd+q/f+WxH5XxE5vLDLApALZhNIJ2YTSCdmE0gnZhOlFFK0qjjnenjvlzRsniEic3a0/3a3k+bNm0eyLl265LIE00knnaSyqVOnquzww/VMnXbaaSqzSm0GDx4c2f7ggw/UPrfeeqvKrLLI559/XmVWsZxVKtu7d+/I9l577aX2sYoma2trVRYvwROxvy8h5ZCUwyUr19kU0d8nqyAtVyHFwNYaRETatm2rspdfflllo0ePjmw/+uijah+r2Nm6f6tUddOmTSobPny4yn75y19Gtq1ZsoosQ8tXLZSqpl8+s2ncV1AWIp/ZbNOmTdA6OnbsGNn++9//rvaxzk1Wcak116tWrVKZVUhcSNbXrXXr1iqzzt8hOG+WTiFns9wtXrxYZWeffXbQbR955BGVjRw5UmXWe85u3bpFtq3XEesXIljn6tAi6lxfM1E6lTibVmn5CSecoLJjjjkmsn3DDTeofeL/Fmzs/ufM0V82ayY2bNigsvi5eeDAgWof6z2o9Z75/PPPV9mzzz6rsqQ0+a7BOfcnETlWRHZ3ztWKyC0icqxz7mAR8SJSIyKXFnGNAAzMJpBOzCaQTswmkE7MJpLW5EUR7731e3EeK8JaAOwEZhNIJ2YTSCdmE0gnZhNJ4+czAQAAAABAJnFRBAAAAAAAZFJxm8gMxSwPO/roo1U2duxYld1///0qGzdunMpuv/12lS1btiyy3adPn6C1zZs3T2XnnKN/UswqTI2X1ImIjB8/PrIdL4AVsYtWrdLH0JLGkFKq0O9vfX190H4onWKWjoXet1XU+/7776ssXkAlInLyySdHtocOHar2sY7Pzp07q+xXv/qVyl555RWVHXfccSrr2bNnZNuar0J/rTdv3qyyeKk1YAk9Fq1SNmueQsrbnnrqKbWPVVpsFahaJY2DBg1SmeWbb76JbFvFqKGsr5tVcGfNPyWqSMKWLVtU9uqrr0a2b7755qD7ateuncouu+yyoNtaBc3x14hWrVqpfQ455BCVLV++XGVWyTKlqkhC/JwjYp93hgwZorLLL788sv3www+rfdavX68y65cHhLLmer/99otsn3nmmWof6xeHhLLmOimcmQEAAAAAQCZxUQQAAAAAAGQSF0UAAAAAAEAmlbxTpFBWrlypssmTJ6vM+uxu6OcNrf369u0b2bY+P231ZVhdCVbngbWflR144IGRbet5du3aVWXFFvqZ6tAeE2SLNYcnnniiyg444ACVdevWLbLdoUOHoPu3Xkus/hBr1keMGKGyZs2aRbatmbDua+HChSqLv940JqQ/xFqH9VrVokWLoMdEtoR+Jt/aL358Wp8hts5h8Z4BEZFOnTqpzOooqKmpUdkee+wR2bbmMJ/ugfjs53t/QCGtXr1aZfFuu3Xr1gXd1xVXXJHzOjZu3KiyeIeINZv77ruvyqZMmaKyJUuWqKxXr147s0SgIEJ7q6zjPf5vUKt35NJLL1XZqFGjAlcXJn4Oy6c/xDJt2jSV7b333gV9jFD8pAgAAAAAAMgkLooAAAAAAIBM4qIIAAAAAADIJC6KAAAAAACATCrbolWrRNEqarOKBENZBafxwplvvvlG7TN69GiVTZ06VWVr1qwJWodVQrN8+fLIdlpKpEILVCmfqxxWQVRc6PfbmjmrqMoqFo2/JoQ+5rXXXqsy6zlZBaRW6WP8OVgzYZXZ9enTZ4frzBeFxygF65y7aNGiyHZoQfnuu++usnihcmP317t37yb3s94z5KOQ81ToElhgv/32U9mWLVuavJ113F199dU5r6Nly5Y5PeZXX32lskGDBqnsvPPOU5n1PjrXGWM2UWhff/21yu65557IdkihvoguT96Z21rq6uoi2+3bt1f7rF27Nuf7v/DCC3O+baHxkyIAAAAAACCTuCgCAAAAAAAyiYsiAAAAAAAgk5q8KOKc6+2cm+ac+9g595Fz7qqGvLNzbqpzbn7Df/WH6wEUDbMJpBOzCaQTswmkE7OJpIUUrdaLyM+897Odc+1E5D3n3FQRuUBE/uG9v8s5d72IXC8i1xVvqVFWoZlVfmQVM3Xv3j3o/mpqalR21113Rbat0rdZs2apLLRUtW/fvirr2rWryqzngMxJxWzGC8Y2btyo9rGK1ax5tYoP46XCInYpaa4Faf/85z+bvJ2IXS5lFcPGM+s5tW3bNugxkxBSnIsmpWI2i806VlatWqWyMWPGRLZHjBih9rHm0Coyt15LrBmz1hafzTQXJqZlHRUoE7O5evVqlW3dulVlmzZtavK+2rRpo7Lp06er7MwzzwxcnRafxXnz5ql9Ro0apbIhQ4ao7JVXXlHZ3LlzVbbnnnuqLP41sp47s1k0mZhN67xjFR6PHTs2sv3AAw+ofYYNG6ayfAq/rcLXnj17RrY3b94cdF/NmjVT2ZFHHqmyNM1Tkz8p4r1f4r2f3fDnOhH5WER6icjpIvJEw25PiMj3i7VIABqzCaQTswmkE7MJpBOziaTtVKeIc66viAwWkRki0s17v0Rk24EsIvrHGbbd5hLn3Czn3Czr//EFkD9mE0gnZhNIJ2YTSCdmE0kIvijinGsrIpNE5GrvffAvJPbej/feV3vvq6uqqnJZI4AdYDaBdGI2gXRiNoF0YjaRlKCLIs65ZrLtAH3Ke/9CQ7zUOdej4e97iMiy4iwRQGOYTSCdmE0gnZhNIJ2YTSSpyaJVt60B5TER+dh7f+92fzVZREaJyF0N/30xlwVYRYVWaVoIq6zFulr49ttvq8wqa7JKYuKFqbW1tWqf9evXq6xHjx4qe+GFF1S29957q6xLly4qS1MxDZJR7NnMlVWEaLHKpl566SWVDRw4UGVW6Wmc9Toyf/58lS1atEhl1nwNGDBAZS1atFBZPiVXcUkUQfLakr+0zmY+rOLG3XbTbyEWLFigsvh50yoxv+CCC1R25ZVXqqx58+YqswokrTks5GyiPFXibFq/UOCDDz5QmVVcHGLdunUqGz16tMqOO+44lbVu3VpldXV1Kjv00EMj2ytXrgy6nfX+uLq6WmUzZsxQWeh7FZRGJc6mxXqPZWV33HFHZNs69y1cuFBloedD61z9m9/8RmWhxapxDz74oMpuueUWleX6b/5iCPntM0NE5HwR+bdz7j+vsmNl28H5nHPuxyKySET+uzhLBNAIZhNIJ2YTSCdmE0gnZhOJavKiiPf+nyLS2P91qC8LAygJZhNIJ2YTSCdmE0gnZhNJS8/PrAAAAAAAAJQQF0UAAAAAAEAmhXSKFFVowUq8cDC0DNAqQvyv//qvoNv27t1bZePHj2/ydlZxo1WQs2TJEpVZBZIUH6KcbNq0SWXWHFoly0OHDlWZVXg8ZcoUlR1++OGR7XPPPVftM3PmTJVt2bJFZZbTTrZ15BAAAAoYSURBVDtNZcUubiz27H/xxRcq69WrV1EfE+lnzXCHDh2CbmudN6+99trItjW/1pxb92WVyFmFr+UkiUJlVI7u3bur7MQTT1RZvMxUROS9996LbFvnQ+t9ulW+etRRR6nsjDPOUNmdd96pshDWOvbZZx+VTZ48WWXWa4RVIGm9VweKrWfPnk3uY50n+vTpo7Lvfe97KrvmmmtUdvLJJ6vMOvfH7bfffir717/+pTLrnH7ppZc2ef9J4idFAAAAAABAJnFRBAAAAAAAZBIXRQAAAAAAQCZxUQQAAAAAAGRS2bSThZSOlaKsLF42Zz3mgAEDVLZ+/XqVWQVRlViu9uGHH6rsoIMOSmAlKAWrVNWaE6uktG3btipbtWqVyhYsWKCye++9N7L9/vvvq33q6+tVZhk4cKDKqqqqVGaVxcZn2Jpp63ahpdO5sr4HVpElYM2wxTqOP/74Y5U9/fTTke0f/ehHap9hw4apzDpvWkWIrVu3Vlmx56mQrNcIq1C22MXOqBzWMXXEEUeoLH5O/PTTT9U+1rnDKlqtqalR2bhx43a0zEZ17txZZffdd5/KRo4cGXR/1msVpaooJ9ZMW2XB06dPD8pCf8lA/JcYzJgxI+h25ah83jUAAAAAAAAUEBdFAAAAAABAJnFRBAAAAAAAZBIXRQAAAAAAQCaVTdFqiEKXlFoFNvFiJusxrYK39u3bF25hZSbLzx3bhM6mNTudOnVS2csvv6yyVq1aRbY7duyo9lm5cqXKvvvd76rstttuU1l1dbXKcpVEobL1mKUop0ZlCC3+tOZp6NChke3QMuYsH4ubNm1SmVUoC4S6//77VRYvQT7ttNPUPldeeWXQfVlF6TfddJPK4qXoIiInnHBCZHvSpElqn2bNmqksVDkVLwOhvvzyS5VZ74/33HNPlb311lsq69+/v8oGDRqU4+rKD68SAAAAAAAgk7goAgAAAAAAMomLIgAAAAAAIJOa7BRxzvUWkSdFpLuIfCsi4733v3XO/VJELhaR5Q27jvXev1SshcatX79eZW3atCnoY8T7Qyz19fUq2223iqpq2aF474r1NevXr1+plpMpaZ3NfFif+7WyadOmqezbb7+NbK9bt07tY30m3+oOiveTFFpauhLSso5Kk9bZfPLJJ1U2cuTIoNta/SEW65iyOkRCblduCtmLQn9IcaR1NpNy7rnnNrnPxIkTc77/X//610EZwGzuvL59+wZlljPPPLOwi6kAIf96rxeRn3nvZzvn2onIe865qQ1/d5/3/n+KtzwAO8BsAunEbALpxGwC6cRsIlFNXhTx3i8RkSUNf65zzn0sIr2KvTAAO8ZsAunEbALpxGwC6cRsImk71SninOsrIoNFZEZDdIVz7l/OuYnOOf17M7fd5hLn3Czn3Kzly5dbuwDIE7MJpBOzCaQTswmkE7OJJARfFHHOtRWRSSJytfd+rYj8TkT2EpGDZduVvXus23nvx3vvq7331VVVVQVYMoDtMZtAOjGbQDoxm0A6MZtISlAjqHOumWw7QJ/y3r8gIuK9X7rd3/+viEwp1KJWrlypss6dO0e2Q0tVC1l8ZrFKVeOFjyJ2WWS5Oeyww1T27rvvJrAS/EepZzPN4jPWvn17tY/1elAJhYaV+ppTzko9myHnutBSVeTGem9hFTmHlLhbrO8xdh7nTSCdmE0kqcl3zW7bWf4xEfnYe3/vdnmP7XY7Q0TmFH55ABrDbALpxGwC6cRsAunEbCJpIT8pMkREzheRfzvnPmjIxorIOc65g0XEi0iNiFxalBUCaAyzCaQTswmkE7MJpBOziUSF/PaZf4qI9XkTfkc0kCBmE0gnZhNIJ2YTSCdmE0njQ+cAAAAAACCTgopWSy1eqioisn79+sh2aNFqIUtVRURqamoi23369FH7VGrBYa6lqsUuu0XplPv3spzWujNyfc0pZAkkkhVS8pn172389asUrwchX/PQ19VKff1C+dm4caPKWrZsmcBKAKAwKvNf7wAAAAAAAE3goggAAAAAAMgkLooAAAAAAIBM4qIIAAAAAADIJGcVfBXtwZxbLiILRWR3EVlRsgcujnJ/Dmlc/x7e+6qkF5FFzGaqpHH9zGZCmM1USeP6mc2EMJupksb1M5sJqaDZLPf1i6TvOTQ6lyW9KPL/D+rcLO99dckfuIDK/TmU+/pRHJVwXJT7cyj39aM4KuG4KPfnUO7rR3FUwnFR7s+h3NeP4ij346Lc1y9SXs+Bj88AAAAAAIBM4qIIAAAAAADIpKQuioxP6HELqdyfQ7mvH8VRCcdFuT+Hcl8/iqMSjotyfw7lvn4URyUcF+X+HMp9/SiOcj8uyn39ImX0HBLpFAEAAAAAAEgaH58BAAAAAACZxEURAAAAAACQSSW/KOKcG+Gcm+ec+8w5d32pHz8XzrmJzrllzrk522WdnXNTnXPzG/7bKck17ohzrrdzbppz7mPn3EfOuasa8rJ5Dig+ZrP0mE2EKLfZLPe5FGE2EYbZLD1mE00pt7kUKf/ZrIS5LOlFEefcriLysIicJCIDROQc59yAUq4hR4+LyIhYdr2I/MN7319E/tGwnVb1IvIz7/3+InKkiIxp+LqX03NAETGbiWE2sUNlOpuPS3nPpQiziSYwm4lhNtGoMp1LkfKfzbKfy1L/pMjhIvKZ936B936ziDwjIqeXeA07zXv/poisjMWni8gTDX9+QkS+X9JF7QTv/RLv/eyGP9eJyMci0kvK6Dmg6JjNBDCbCFB2s1nucynCbCIIs5kAZhNNKLu5FCn/2ayEuSz1RZFeIvLFdtu1DVk56ua9XyKy7UAQka4JryeIc66viAwWkRlSps8BRcFsJozZRCMqZTbL9phmNtEIZjNhzCYMlTKXImV6TJfrXJb6oogzMn4ncIk459qKyCQRudp7vzbp9SBVmM0EMZvYAWYzQcwmdoDZTBCziUYwlwkq57ks9UWRWhHpvd32d0TkyxKvoVCWOud6iIg0/HdZwuvZIedcM9l2kD7lvX+hIS6r54CiYjYTwmyiCZUym2V3TDObaAKzmRBmEztQKXMpUmbHdLnPZakvirwrIv2dc/2cc81F5GwRmVziNRTKZBEZ1fDnUSLyYoJr2SHnnBORx0TkY+/9vdv9Vdk8BxQds5kAZhMBKmU2y+qYZjYRgNlMALOJJlTKXIqU0TFdCXPpvC/tTxQ5504WkftFZFcRmei9v6OkC8iBc+5PInKsiOwuIktF5BYR+YuIPCcifURkkYj8t/c+XpCTCs65oSIyXUT+LSLfNsRjZdtnvcriOaD4mM3SYzYRotxms9znUoTZRBhms/SYTTSl3OZSpPxnsxLmsuQXRQAAAAAAANKg1B+fAQAAAAAASAUuigAAAAAAgEzioggAAAAAAMgkLooAAAAAAIBM4qIIAAAAAADIJC6KAAAAAACATOKiCAAAAAAAyKT/A56eJ7myrFaEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1440x180 with 5 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "##########################\n",
    "### VISUALIZATION\n",
    "##########################\n",
    "\n",
    "model.eval()\n",
    "# Make new images\n",
    "z = torch.zeros((5, LATENT_DIM)).uniform_(-1.0, 1.0).to(device)\n",
    "generated_features = model.generator_forward(z)\n",
    "imgs = generated_features.view(-1, 28, 28)\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=5, figsize=(20, 2.5))\n",
    "\n",
    "\n",
    "for i, ax in enumerate(axes):\n",
    "    axes[i].imshow(imgs[i].to(torch.device('cpu')).detach(), cmap='binary')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/layers.png\" style=\"height:50px;display:inline\"> Deep Convolutional GANs (DCGANs)\n",
    "---\n",
    "* Key ideas:\n",
    "    * Replace fully-connected (FC) hidden layers with convolutions.\n",
    "        * Use fractional/dialated convolutions (to perform the up-convolution from vectors to images).\n",
    "    * Use *Batch Normalization* after each layer.\n",
    "    * Activations:\n",
    "        * Hidden layers are activated with ReLUs.\n",
    "        * Output layer is activated with Tanh (i.e, the pixel values are normalized between $[-1, 1]$)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_dcgan.PNG\" style=\"height:350px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/ultraviolet/80/000000/mark-view-as-hidden.png\" style=\"height:50px;display:inline\"> The Latent Space\n",
    "---\n",
    "* As we learn how to transform a latent vector, $z$, to images, we actullay learn a latent continuous space.\n",
    "* This continuous spcae allows us to perform interpolation and arithmetics.\n",
    "* As this space is continuous, unlike the original data (images), it was found that some operations (like summing) perform really well when done on the latent space.\n",
    "* As you can see below, those operations were demonstrated in the paper <a href=\"https://arxiv.org/abs/1511.06434\">Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, Alec Radford, Luke Metz, Soumith Chintala, ICLR 2016</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_latent_space.PNG\" style=\"height:450px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/80/000000/maracas.png\" style=\"height:50px;display:inline\"> Conditional GANs\n",
    "---\n",
    "* As you probably have noticed, we don't too much control over the latent space, e.g., with vanilla-GAN trained on MNIST we can't control what digit we are generating.\n",
    "* **Conditional-GANs** - a simple modification to the original GAN framework that *conditions* the model on additional information for better multi-modal learning.\n",
    "* In practice, we usually use the labels of the datasets to perform the conditioning.\n",
    "    * For example, on MNIST we will use the one-hot vector representation of the digit ($1 \\rightarrow [0,1,0,0,0,0,0,0,0,0]$) along with the images from that class.\n",
    "* Leads to many practical applications of GANs when we have *explicit supervision available*.\n",
    "* There is more than one way to perform conditioning, some approaches are presented below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_cgans.png\" style=\"height:300px\">\n",
    "\n",
    "* <a href=\"https://arxiv.org/abs/1411.1784\">Conditional Generative Adversarial Nets, Mehdi Mirza, Simon Osindero</a>\n",
    "* <a href=\"https://assemblingintelligence.wordpress.com/2017/05/10/conditional-gans/\">Conditional GANs</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/today.png\" style=\"height:50px;display:inline\"> GANs Today\n",
    "---\n",
    "* GANs are **HARD to train** and many researche studies try to improve training stability.\n",
    "* **WGAN** - Wasserstein GANs use the Wasserstein (Earth Movers) distance as the loss function. Training is more stabilized than vanilla-GAN.\n",
    "    * **WGAN-GP** - improves upon the original WGAN by using *Gradient Penalty* in the loss function (instead of *value clipping*)\n",
    "    * <a href=\"https://arxiv.org/abs/1701.07875\">WGAN Paper</a>, <a href=\"https://github.com/Zeleni9/pytorch-wgan\">PyTorch Code</a>\n",
    "    * <a href=\"https://arxiv.org/abs/1704.00028\">WGAN-GP Paper</a>, <a href=\"https://github.com/Zeleni9/pytorch-wgan\">PyTorch Code</a>\n",
    "* **EBGAN** - Energy-Based GANs use *autoencoders* in their architecture (with the autoencoder loss).\n",
    "    * <a href=\"https://arxiv.org/abs/1609.03126\">EBGAN Paper</a>, <a href=\"https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/ebgan/ebgan.py\">PyTorch Code</a>\n",
    "* **BEGAN** - Boundary Equilibrium GANs combines *autoencoders* and Wassertein distance to balance the generator and discriminator during training.\n",
    "    * <a href=\"https://arxiv.org/abs/1703.10717\">BEGAN Paper</a>, <a href=\"https://github.com/anantzoid/BEGAN-pytorch\">PyTorch Code</a>\n",
    "* **Mimicry** - a lightweight PyTorch library aimed towards the reproducibility of GAN research - <a href=\"https://github.com/kwotsin/mimicry\">GitHub</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/to-do.png\" style=\"height:40px;display:inline\"> Tips for Training GANs\n",
    "---\n",
    "All tips are here: <a href=\"https://github.com/soumith/ganhacks\">Tips for Training GANs</a>\n",
    "* Normalize the inputs - usually between $[-1, 1]$. Use TanH for the Generator output.\n",
    "* Use the modified loss function to avoid the vanishing gradients.\n",
    "* Use a spherical Z - sample from a Gaussian distribution instead of uniform distribution.\n",
    "* BatchNorm (when batchnorm is not an option use instance normalization).\n",
    "* Avoid Sparse Gradients: ReLU, MaxPool - the stability of the GAN game suffers if you have sparse gradients.\n",
    "    * LeakyReLU is good (in both G and D)\n",
    "    * For Downsampling, use: Average Pooling, Conv2d + stride\n",
    "    * For Upsampling, use: PixelShuffle, ConvTranspose2d + stride"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Use Soft and Noisy Labels\n",
    "    * Label Smoothing, i.e. if you have two target labels: Real=1 and Fake=0, then for each incoming sample, if it is real, then replace the label with a random number between 0.7 and 1.2, and if it is a fake sample, replace it with 0.0 and 0.3.\n",
    "    * Make the labels the noisy for the discriminator: occasionally flip the labels when training the discriminator\n",
    "* Track failures early:\n",
    "    * D loss goes to 0 -- failure mode.\n",
    "    * Check norms of gradients: if they are over 100 things are not good...\n",
    "    * When things are working, D loss has low variance and goes down over time vs. having huge variance and spiking.\n",
    "* **Don't** balance loss via statistics (unless you have a good reason to)\n",
    "    * For example, don't do that: `while lossD > A: train D` or `while lossG > B: train G`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"./assets/gen_anim_icon.gif\" style=\"height:40px;display:inline\"> Applications\n",
    "---\n",
    "* There has been great progress in GANs, and everyday there is \"a new GAN\".\n",
    "* The current quality of the generations is almost indistinguishable from real images.\n",
    "    * <a href=\"https://thispersondoesnotexist.com/\">StyleGAN V2 - ThisPersonDoesNotExist.com</a>\n",
    "    * <a href=\"https://github.com/lucidrains/stylegan2-pytorch\">SyleGAN V2 - PyTorch Code</a>\n",
    "* There are many appliactions which we do not cover, but we provide links to projects at the end of this tutorial. We encourage you to explore areas that you find interesting and integrate them in your homework and projects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/doodle/96/000000/transition-both-directions--v1.png\" style=\"height:50px;display:inline\"> Image-to-Image Translation (Pix2Pix)\n",
    "---\n",
    "* Training is conditioned on the images from the source domain.\n",
    "* Conditional GANs provide an effective way to handle many complex domains without worrying about designing structured loss functions explicitly.\n",
    "    * These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping.\n",
    "* <a href=\"https://phillipi.github.io/pix2pix/\">Project Page</a>\n",
    "    * <a href=\"https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix\">PyTorch Code on GitHub</a>\n",
    "* <a href=\"https://affinelayer.com/pixsrv/\">Edges-to-Cats Demo</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_pix2pix.png\" style=\"height:350px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_pix2pixarch.jpg\" style=\"height:300px\">\n",
    "\n",
    "* <a href=\"https://neurohive.io/en/popular-networks/pix2pix-image-to-image-translation/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Pix2pix uses a conditional generative adversarial network (cGAN) to learn a function to map from an input image to an output image. \n",
    "* The **Generator** transforms the input image to get the output image. \n",
    "* The **Discriminator** measures the similarity of the input image to an unknown image (either a target image from the dataset or an output image from the generator) and tries to guess if it real or fake."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/process.png\" style=\"height:50px;display:inline\"> CycleGAN\n",
    "---\n",
    "* For many tasks, paired training data will not be available (like in Pix2Pix).\n",
    "* **CycleGAN** -  an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples.\n",
    "* The goal is to learn a mapping $G: X \\to Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss.\n",
    "* Because this mapping is highly under-constrained, it is coupled with an inverse mapping $F: Y \\to X$ and introduce a cycle consistency loss to push $F(G(X)) \\approx X$ (and vice versa). $$\\text{Loss}_{cyc}(G, F, X, Y) = \\frac{1}{m}\\sum_{i=1}^m \\left[\\mid\\mid F\\left(G(x_i)\\right) - x_i\\mid\\mid_1\\right] + \\left[\\mid \\mid G\\left(F(y_i)\\right) - y_i\\mid \\mid_1\\right]$$\n",
    "* The complete loss: $$ \\text{Loss}_{full} = \\text{Loss}_{adv} + \\lambda  \\text{Loss}_{cyc}$$\n",
    "    * $\\lambda =10$ in the original implementation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* <a href=\"https://junyanz.github.io/CycleGAN/\">Project Page</a>\n",
    "    * <a href=\"https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix\">PyTorch Code on GitHub</a>\n",
    "    \n",
    "    <img src=\"./assets/tut_gan_horse_zebra.gif\" style=\"height:200px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_cycle_gan_arch.PNG\" style=\"height:350px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_cyclegan_arch.png\" style=\"height:300px\">\n",
    "\n",
    "* <a href=\"https://towardsdatascience.com/image-to-image-translation-using-cyclegan-model-d58cfff04755\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_cycle_gan.jpg\" style=\"height:350px\">\n",
    "\n",
    "* Video: <a href=\"https://www.youtube.com/watch?v=lCR9sT9mbis\">[CycleGAN] Rendering Cityscapes in GTA Style</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/portrait.png\" style=\"height:50px;display:inline\"> Realistic Neural Talking Head Models\n",
    "---\n",
    "* In order to create a personalized talking head model, it usually requires training on a large dataset of images of a single person. \n",
    "* However, in many practical scenarios, such personalized talking head models need to be learned from a few image views of a person, potentially even a single image. \n",
    "* In the paper **\"Few-Shot Adversarial Learning of Realistic Neural Talking Head Models\"**, a system with such few-shot capability is presented. \n",
    "* The model performs lengthy meta-learning on a large dataset of videos, and after that it is able to frame few- and one-shot learning of neural talking head models of previously unseen people as adversarial training problems with high capacity generators and discriminators.\n",
    "* <a href=\"https://arxiv.org/abs/1905.08233v1\">Paper Link</a>\n",
    "    * <a href=\"https://github.com/vincent-thevenin/Realistic-Neural-Talking-Head-Models\">PyTorch Code on GitHub</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_talking_head.PNG\" style=\"height:450px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_living_portrait.gif\" style=\"height:250px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* <a href=\"https://papers.nips.cc/paper/8935-first-order-motion-model-for-image-animation\">First Order Motion Model for Image Animation</a> - Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, Nicu Sebe - NeuroIPS 2019\n",
    "    * <a href=\"https://github.com/AliaksandrSiarohin/first-order-model\">Code and Colab Demo</a>\n",
    "    \n",
    "<img src=\"./assets/tut_gan_first_order_motion.PNG\" style=\"height:300px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/cute-clipart/64/000000/age.png\" style=\"height:50px;display:inline\"> Face Aging with Conditional GANs\n",
    "---\n",
    "* A GAN-based method for automatic face aging.\n",
    "* This model puts emphasize on preserving the original person’s identity in the aged version of his/her face.\n",
    "* Introduces a novel approach for \"Identity-Preserving\" optimization of GAN's latent vectors. \n",
    "* The objective evaluation of the resulting aged and rejuvenated face images is done by the state-of-the-art face recognition and age estimation solutions.\n",
    "* <a href=\"https://arxiv.org/abs/1702.01983\">Paper Link</a>\n",
    "    * <a href=\"https://github.com/guyuchao/IPCGANs-Pytorch\">PyTorch Code on GitHub</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_face_age_arch.PNG\" style=\"height:350px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"./assets/tut_gan_face_age.PNG\" style=\"height:500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/cute-clipart/64/000000/cool.png\" style=\"height:50px;display:inline\"> Cool GAN Projects (with Code)\n",
    "---\n",
    "* <a href=\"https://github.com/nashory/gans-awesome-applications\">gans-awesome-applications</a>\n",
    "* <a href=\"https://github.com/znxlwm/pytorch-generative-model-collections\">pytorch-generative-model-collections</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "* Introduction to GANs - <a href=\"https://www.youtube.com/watch?v=9JpdAg6uMXs\"> Introduction to GANs, NIPS 2016 | Ian Goodfellow, OpenAI</a>\n",
    "* Generative Models -  <a href=\"https://www.youtube.com/watch?v=5WoItGTWV54\"> Stanford CS231n - Lecture 13 | Generative Models</a>\n",
    "* Deep Generative Modeling - <a href=\"https://www.youtube.com/watch?v=yFBFl1cLYx8\"> MIT 6.S191 (2019): Deep Generative Modeling </a>\n",
    "* Face Editing - <a href=\"https://www.youtube.com/watch?v=dCKbRCUyop8\"> Face editing with Generative Adversarial Networks </a>\n",
    "* Wasserstein GANs - <a href=\"https://www.youtube.com/watch?v=31mqB4yGgQY\"> Nuts and Bolts of WGANs, Kantorovich-Rubistein Duality, Earth Movers Distance </a>\n",
    "* Energy-Based GANs - <a href=\"https://www.youtube.com/watch?v=x4sI5qO6O2Y\"> Energy-Based Adversarial Training and Video Prediction, NIPS 2016 | Yann LeCun, Facebook AI Research </a>\n",
    "* CycleGAN - <a href=\"https://www.youtube.com/watch?v=T-lBMrjZ3_0\"> Zebras, Horses & CycleGAN - Computerphile </a>\n",
    "* Pix2Pix - <a href=\"https://www.youtube.com/watch?v=vrvwfFej_r4\"> Neural Networks: pix2pix (Conditional GANs) </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "\n",
    "* Slides from <a href=\"http://slazebni.cs.illinois.edu/spring17/\">CS 598 LAZ</a>\n",
    "* Slides by Lihi Zelnik-Mannor\n",
    "* Slides from <a href=\"http://ci2cv.net/16720b/\">CMU - 16720B – Computer Vision</a>\n",
    "* Some material from Alexander Amini and Ava Soleimany, MIT 6.S191: Introduction to Deep Learning, <a href=\"http://introtodeeplearning.com/\">IntroToDeepLearning.com</a>\n",
    "* Icons from <a href=\"https://icons8.com/\">Icon8.com</a> - https://icons8.com"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}