Report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q Network Banana Hunter\n",
    "### Training a Deep Q Network to Collect Yellow Bananas\n",
    "\n",
    "**Author: [jstol](https://github.com/jstol)**\n",
    "\n",
    "This project was completed as part of the Udacity [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893).\n",
    "\n",
    "Code within this project is written from scratch (with some inspiration and tips taken from previous [DQN homework](https://github.com/udacity/deep-reinforcement-learning/tree/dc65050c8f47b365560a30a112fb84f762005c6b/dqn) in the Udacity program).\n",
    "\n",
    "## Environment\n",
    "The goal is to train an agent through (Deep) Q-Learning that is capable of navigating a world containing yellow and blue bananas. Yellow bananas result in a reward of +1, and blue bananas -1. The task is episodic, with the episode ending after a fixed amount of time. An agent that \"solves\" the environment is defined as achieving a reward of +13 or greater on average over 100 episodes.\n",
    "\n",
    "The environment is provided by Udacity, and based on Unity's [ML-Agents](https://github.com/Unity-Technologies/ml-agents) framework.\n",
    "\n",
    "Observations are delivered in the form of 37-dimensional floating point vectors, and include measurements such as the velocity of the agent. The actions available to the agent are `0` (walk forwards), `1` (walk backwards), `2` (turn left), and `3` (turn right).\n",
    "\n",
    "## Model\n",
    "The agent in this project learns through [Q-learning](https://en.wikipedia.org/wiki/Q-learning), specifically through a Deep Q Network (DQN) \\[1\\]. A simple feed-forward network (with 3 hidden layers) is used. The network architecture is configurable, but by default the agent makes use of the architecture as seen below:\n",
    "\n",
    "![network architecture](images/network_architecture.png \"The Project Network Architecture\")\n",
    "\n",
    "As seen in the code below, the agent is able to solve the environment in 1,170 episodes.\n",
    "\n",
    "### Hyperparameters\n",
    "Adam was used as the optimizer, and ReLU activation functions were used within the hidden layers. The agent can be configured with hyperparameters as desired, but the following settings are used by default:\n",
    "- `update_freq` = 4: Frequency to update the DQNs.\n",
    "- `buffer_size` = 20,000: Size of the replay buffer to maintain.\n",
    "- `batch_size` = 64: Size of the batch to use for SGD.\n",
    "- `gamma` = 0.99: Reward discount factor.\n",
    "- `lr` = 0.001: Learning rate to use (Adam).\n",
    "- `tau` = 0.001: The mixing factor to use when updating the target DQN. If None, use hard updates.\n",
    "- `eps_start` = 1.0: Starting epsilon value.\n",
    "- `eps_min` = 0.05: Minimum allowable epsilon value.\n",
    "- `eps_decay` = 0.998: Epsilon decay factor.\n",
    "- `max_num_episodes` = 2,000: Maximum number of episodes to let the agent run for.\n",
    "\n",
    "### Future Work\n",
    "The version of DQN used in this project stays fairly true to the original paper; however, it could be improved to aid in the agent's learning. For example, through the use of [Double Q-Learning](https://www.semanticscholar.org/paper/Deep-Reinforcement-Learning-with-Double-Q-Learning-Hasselt-Guez/3b9732bb07dc99bde5e1f9f75251c6ea5039373e), [Dueling Q Networks](https://www.semanticscholar.org/paper/Dueling-Network-Architectures-for-Deep-Learning-Wang-Schaul/4c05d7caa357148f0bbd61720bdd35f0bc05eb81), or [Prioritized Experience Replay](https://www.semanticscholar.org/paper/Prioritized-Experience-Replay-Schaul-Quan/c6170fa90d3b2efede5a2e1660cb23e1c824f2ca).\n",
    "\n",
    "### References\n",
    "  1. Mnih, Volodymyr, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, \n",
    "Martin A. Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis \n",
    "Antonoglou, Helen. King, Dharshan Kumaran, Daan Wierstra, Shane Legg and Demis Hassabis. “[Human-level control through \n",
    "deep reinforcement learning](https://www.nature.com/articles/nature14236).” Nature 518 (2015): 529-533."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "\n",
    "import numpy as np\n",
    "import torch\n",
    "from tqdm.notebook import tqdm\n",
    "from unityagents import UnityEnvironment\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This module should be on PYTHONPATH - it contains the training logic\n",
    "from dqn import train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 37\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "model_output_path = '../models/checkpoint.pth'\n",
    "env_path = '../data/Banana.app'\n",
    "env = UnityEnvironment(file_name=env_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35ba58f9a7ca40b18e89a23f8ff9c923",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Episode 0: N/A (Current best avg: N/A)', max=2000.0, styl…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Solved in 1170 episodes (13.06 > 13)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "avg_window_size = 100\n",
    "avg_window_target = 13\n",
    "scores = []\n",
    "scores_window = deque(maxlen=avg_window_size)\n",
    "max_avg_score = float('-inf')\n",
    "max_num_episodes = 2000\n",
    "\n",
    "# Explore/train\n",
    "agent, train_step_generator = train(env, max_num_episodes=max_num_episodes)\n",
    "tqdm_iterator = tqdm(enumerate(train_step_generator), total=max_num_episodes,\n",
    "                     desc='Episode 0: N/A (Current best avg: N/A)')\n",
    "for episode, score in tqdm_iterator:\n",
    "    scores.append(score)\n",
    "    scores_window.append(score)\n",
    "    max_avg_score = max(max_avg_score, np.mean(scores_window)) \\\n",
    "        if len(scores_window) == avg_window_size else max_avg_score\n",
    "\n",
    "    tqdm_iterator.set_description(f\"Episode {episode}: {score} (Current best avg: {max_avg_score})\")\n",
    "\n",
    "    if max_avg_score >= avg_window_target:\n",
    "        print(f\"Solved in {episode + 1} episodes ({max_avg_score} > {avg_window_target})\")\n",
    "        break\n",
    "        \n",
    "# Save the model after we're done\n",
    "torch.save(agent.dqn.state_dict(), model_output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Reward as Training Progresses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO2dd5wV1fn/P882eu/VtVFVioACFrAFRGOiSdQYS9SYGP2ZqN8Y1NiifkMS+zfRaNTErrETUZAqKAIuvcMCS2dZysICW+89vz+m3Llzp5yZO3PbPu/Xa19778yZc87cufd5zvOc5zyHhBBgGIZhmLx0d4BhGIbJDFghMAzDMABYITAMwzAqrBAYhmEYAKwQGIZhGJWCdHdAho4dO4ri4uJ0d4NhGCarWLx48T4hRCfZ8lmhEIqLi1FSUpLubjAMw2QVRLTVS3l2GTEMwzAAWCEwDMMwKqwQGIZhGACsEBiGYRgVVggMwzAMgBAVAhH1IqLZRLSGiFYT0W/U4w8T0U4iWqb+XRxWHxiGYRh5wgw7bQBwtxBiCRG1ArCYiKar554WQjwRYtsMwzCMR0JTCEKI3QB2q6+riGgtgB5htccwDGPH0m0HUVSQh4Hd26S7KxlNSuYQiKgYwBAAC9VDtxPRCiJ6lYja2VxzCxGVEFFJRUVFKrrJMEyO8sPn52PCc1+nuxsZT+gKgYhaAvgQwG+FEIcBvADgRACDoVgQT1pdJ4R4SQgxTAgxrFMn6ZXXDMMwjE9CVQhEVAhFGbwlhPgIAIQQ5UKIiBAiCuCfAEaE2QeGYRhGjjCjjAjAKwDWCiGeMhzvZij2QwCrwuoDwzAMI0+YUUajAVwLYCURLVOP3QfgaiIaDEAAKAPwyxD7wDAMw0gSZpTR1wDI4tTnYbXJMAzD+IdXKjMMwzAAWCEwDMMwKqwQGIZhGACsEBiGYRgVVggMwzAMAFYIDMMwjAorBIZhGAYAKwSGYRhGhRUCwzAMA4AVAsMwDKPCCoFhGIYBwAqBYRiGUWGFwDAMwwBghcAwDMOosEJgGIYJiGN1DSjdW5VUHYdr6rFl39GAeuQNVggMwzABceubS3DBU3NRH4n6ruPy5+dj7BNzguuUB1ghMAzDBMT8TfsAAFEhfNdRuvdIUN3xDCsEhmGYgCHLzSIzH1YIDMMwASPg30JIJ6wQGIZhAkLzFCXhMUorrBAYhmEYAKwQGIZhpNi6/ygiUbmhP1sIDMMwGc7MteW+rtu6/yjO/escPDNjg2M5of/PTo3ACoFhmEbD+nJ/i8bKD9cCAL7dtF+qfBAWgkiDmcEKgWGYRoPfcFCSvCxIIS7pnQoUVggMwzQa8nwuD5CV88L0PxmSWdzml9AUAhH1IqLZRLSGiFYT0W/U4+2JaDoRbVT/twurDwzDMEZkR/rJEoSlkFMKAUADgLuFEAMAnAngNiIaAGAigJlCiJMBzFTfMwzDhE74LiP1v69WrOtKJaEpBCHEbiHEEvV1FYC1AHoAuAzAa2qx1wD8IKw+MAzDGDEKdiEEKqpqHctXHqvzlagutkBNYG9VDeojURw8WgcAOFRdj5r6iGsduWYh6BBRMYAhABYC6CKE2K2e2gOgSyr6wDAMQwaN8MHiHRj++Ays2FFpW37wH6fjt+8u099Li2i14HvfbceIx2fikue+xpBHp0MIgUGPfImfvPitaxU5OalMRC0BfAjgt0KIw8ZzQnG0Wd42Ed1CRCVEVFJRURF2NxmGaQQYPT/fblZCSDeUW2cX1eYBpqzc7dnRpK1DmLNekV1auKs26F+x45BrHTlnIRBRIRRl8JYQ4iP1cDkRdVPPdwOw1+paIcRLQohhQohhnTp1CrObDMM0EuLmAlR5ayfsG5IYomuyvM7kbvJSo/C/pYJvwowyIgCvAFgrhHjKcGoygOvV19cD+DSsPjAMwxjJM2gETTjbTRg3RJJQCOr/2ob4uQIvo/5IGiyEghDrHg3gWgAriUhzwt0HYBKA/xDRTQC2AvhJiH1gGIbRMU8qO1EfjSZcI4tWd12DyULwIOPT4TIKTSEIIb6GvTV2fljtMgzD2EEW0t1O4EdUCyHOqpAU0lops0LwIuRzbg6BYZjsorou4vjeK7UNETRIhG1GowI19ZGk23ODIH9PmoWQTxSnNOojUddQVCGAmvoIKqvr4457EfI5tQ6BYZjsYtXOQ+j/4FRMXbUHAPDl6j3o/+BUx7BMN/r+YSqufWWRa7nHpqxFvwemov+DU/HJ0p2+23NjY3kV+j84FZOX74rNIdg4MrQ5BLMFMeDBqRj5p5mO7bz41Sb0e2Aqtu4/Fne83sO8BFsIDMOkjeWq4P9qgxL4N2dDhXrcPUTSCS2804n3vtumv565zjLwMBDW7lbCP2etLddH4G6TyuaJ6PqIwL4jdY7tfLJsl+VxL4vccnIdAsMw2YF5pCxcwjLDIsy0z14miBtUl5GSEM/7SgQrPCmENGgEVggMw9iQOELOdrRbsV0Ra0Bbh5CXRxKl5ahv4DkEhmGyCE0QqQPklGUI1dsPsW4vEUPaaN6PQrSr2rxQzQmeQ2AYJm2Y5Z6WfiF37ANr4W4VigoY5xAAr5+CnSj34jJKx8I0VggMw8RhlkN+LQQvcwGpEn26y0jIu4zy84xWhVw7dvfuRSGkYwvNMFcqM0xOUTxxCq4e0Rt/uvzUdHclFDSx917Jdhyta8BnK3Y7lncjmVxAybB292GMf3YeXr9xBM7pE58HTbMQJi+PRQERlHDUC5+ei0tO64bPVuzGgnvPN6yfIM9K0YuFUDxxCm4663is31OFr0v36cc5yohhMpx3Fm1zL5QDGJWB301lvOQCCtIttWjLAQDA9DXlCefsttBcvPUggNh9l+49orts/KWusD6u6QOtH1ok0Stfb4lTBgDPITAMk0ZsBZ9Paa2FbXomRDkom7pCQCQV5WPn7omYhv1OVpTfjy8ZWCEwDOOI37DTZLKFJoNTd60sBDsLKJkRut2VUd3qUNp0UppsITAMkzbsBKNfd069hyFukKLPSY7aRRRZXWIezXvrhPVhrU6tF06pLHgdAsMwGYffKCO/AlUEpB6s+m1pIVi5jIS1QJbtmV05cyipVeI/rT9sITAMkz5sBL9fhZAul5ETshPkAvEKzXPiChthLkwT1VYL1bS2WCEwDAMA+J/3l+PUh6YFVt/Dk1ejeOIUX9f6jTKyi7nfVVmN4olTMGtdYhQQ4Owqqa6LoHjiFNz38UoUT5yCEY/PwFl/ngVAiS4qnjgFmyqs90gGgKmr9yQc+/VbS7B+T1XCcd3fb1PXPR8sR/HEKfhqQ+Ke77YWgrb6G4RIVGDkn2ZZtBvffiphhcAwGcgHi3egqrYhsPr+Pb/MtUzQK5LtPEbLtytZVf/z3Q7PdR48pmQZfXuhEv67t6oWOw5WAwAmL1fSZs/f5J5d1cyc9fEZVoUQev9F/An95X9KlP5/sDjxPuzDTmMn3Bap8ToEhmHShp388Z/LKHiJVmC3kAAxAeqnu+aeCsiP0K3as5sH0V1J5F4/ZztlGCZn8OLxkBXi+Q4KwW1/AycS+io8KASbSWkrIgY3lJu8ZwuBYZi0YSdH/a5DsJNnVsdlZZ9d2ChgmLD1YSNYjeiNAtmpXev6rNHDTiUshHTkMmKFwDCMI35dRn4nRZ0uc6pTO5fnYEV4aVPWZeOlNWM7wmWZBlsIDMOkDbtRsN8oozAGuE51JrPDm7leAWGpfKyat/zcJCaVXecQ2EJgmMzi/2ZuxIjHZ9ia7ze/VoKbX/vOd/1/+mItzntiju/rvfKhRUSMG04WwpA/fokX5myyPGf8yJ6fU4rTH52O0x6ehle/3gJACQHVQkfjrjNI080VR1A8cQrW7j6ccM7M++q9af19/dutKJ44RSrcdmdlddz7hojAXf9Zrr/XPoIVDvtLX/vKQst7MBIxuLXcxP11ry6yDWsNC1YIDOPAk9M3YG9Vre35GWvLMWOt/03hX/xqMzbvO+r7eq/8eeo623N+RtYHj9Xb1mkUin+Zuh77j9bhcE0DStTMohpOn+8Xq5R1A3q6aolBcxA7vNU0xPw5shtoztsYy1ZqN7g3LkyTtQAK81O3RRErBIaRIB15ZcLAz234FUd+PzPjdXWqYC7MV0SVTJV+XVy2/XFtT/4a49IDWYXQtDBfqlwQsEJgGAnSpQ9SGWliN7L2v2Oa/75oaIu3itRRsowQ9TGnnID5c3d8DpZhp24uI/nPpxkrBIbJLNIRAggkmXEzILyGXGr4TVJnbE5TCLqFIGUiJK8RzIrH653Ypr/Ww06pcVkIRPQqEe0lolWGYw8T0U4iWqb+XRxW+wwTJOkSy0FvtO6cGtrmeAhtyV6npYcuKvDiMkoeL5vTWLmoZFJXyOr6XLEQ/g1gnMXxp4UQg9W/z0Nsn2GynuB3zfIupf1bCMlTa55DkNAyQUwqmxWxH0VqhTFhnuw6h6aFqXPkFIRVsRBiLhEVh1U/w6SS7z09N+GYU1ZNI8/M2IBl2yvRoqgAPdo1Q3VdBG8s2IoJp3VzvdbKQli54xB+9spCzLr7XHRo2USqDxr7jtTpYZgbHhuvj7ydIAAHjtZh7BNz8MZNI7B0WyU+WroTn942Wi9z82sleP6aoRj7xBw8dOkAXDSwq283W10kinHPzMUPhvQwzCHk4fk5pfigxDps9sZ/x0J/l26r9NWuEWPfK4/V4YoX5iddJwC89912Qxty1zQpSJ2FEJpCcOB2IroOQAmAu4UQB60KEdEtAG4BgN69e6ewewyTiFVo6BvfbpW69pkZGy2PTzFsZG+H1RzCP77ahEPV9Zi/aT8uHdRdqg9WHDxWhy6tm+rv7aJzBIBvSvfhUHU9XvxqM6asTOz3jLXl2H+0Fjsrq/HAp6sUheC3X0frsG5PFSZ9sQ5XDO2pdQ5/mbre9ppZ6/yH/lphjAZy2tUM8Oai2rj3iH6R7BxCs6LccBlZ8QKAEwEMBrAbwJN2BYUQLwkhhgkhhnXq1ClV/WMYaWobwt8F3cqtoE3W+s0xpCF7uRBCb8tJiOXrZWLX+cFoFameopRn/gx7lTBJttG5lTcLMFlSqhCEEOVCiIgQIgrgnwBGpLJ9hgmS2vpI6G1YuYySyeppxGwR2NUnEAvldBJi2lyDJryNRb301bjTmpbdNNXBVl4Ugu+wXIkyDSm+8ZQqBCIyOk1/CGCVXVmGyXTSZiEEJCO8CDKtbFxyNpuOaMLUeNZpHwMzcVtXSlgmYeAl3NfPQjgikrKgrPZcDpPQ5hCI6B0AYwB0JKIdAB4CMIaIBkP5rpQB+GVY7TNM2NQ2hG8hWI0QNZdRssE0si4nIWKC2dgbs29dE9oxl5G5LTkha7SKZCyTMAh7YK6krnAvl2oLIcwoo6stDr8SVnsMk2pSYSFYjVSDcxnJIvRRsHFUa94CUlcIusvIKNidWzPKe6NVpF2X6gV6XuYswkwP3uAyoR00vFKZyXle+XoL7v94JQDgsxW7UDxxCmav9x+V8j/vL8cbC7aixjCH8JMXv0VDJIrSvUdw8bPzcKi6HvM2VuDiZ+d5qvv2t5fgr9NiyeIiUYFrX1kYt+evJiJ+9eYSfLJ0J6588Vvc8c5S/McQ0ggAL8zZhMuf/8a2D0MenY7XDHst2603eGbGxpiFYJBP5qR2mgytqm3AU9M3xI2AnXY6U66NFY5YKJJH/rvG8fqgefzztdJl3/1uO95YIBdxplF5rB5HJfbMTmWEEcAKgWkEPPrZGrylbsp++9tLAQC3vrnYd30fLN6BBz5ZFTdqXbTlAPYfrcPTMzZgze7DmLuhAne8sxRr1LTNsny2Yjf+PjuWTrqyuh7zNu7DrW8u0Y8ZhfJv31uGhVsOYPLyXbjnwxVxdf156jos2Vbp2IeHJq821Gs9Gl23p0p33RhLvG4KuzWOqp+buTEudcXA7q1t+6C0HXsdichbFn4Y1Ktt4HU+8In36dDFWy0j7uP48NaRfrrjG1YITKMkPwRBA8SEmduIWJZD1fUAgIK4FMipT6QhE3aa4NYxvG3bvNCxfqPysAo7DRLjgjo3WjUNb6mW2/qGovw8nNS5VWjtW8EKgWmUhDHyFCImzJKtX1MolcfqAMRH6YQ1vyqTnsHJtW6eUzAWdetznIVgMYeQLrxER3nF/HmZScets0JgGJ+Yfe4CQnebFOS574jlRFM1pcTBo6pCMAyVw7IPnLKTxuYQ7MuYJ9m9KC6j5dFgEXaaLvLzwhORbhPGQVmZXmCFwDABIURMmOXnUVIj+SZqhstK1WVUGGchhKMSZFL+C2G/30BdgoUgDK9d2ja8jrcQXC4MmSDat9Np9S6ZC8Nyazrh6CAjoqFO54UQS5zOM0ymEoRITdhEBbGRbl6SkkSzECqPKQoh3zCHEJqF4FBxnr4OQSA/jxC1GN3WNZjDUOXqNp83Tk6nfrYkniBkcj4RGiw+ADcLIRNdRk+qf38HsBDAS1BSTixUjzFM4CzfXok731umC4a5Gyrw0Kfeojj+OXcz3lYjizTKDAnqjtQ24ObXSrDnUI3vfi4xZdX826xSfXRbkEf6hLAfNAvhoDqHsP1ANe5SPxMn4Trwwal6iK0XKo/V4e73l9ue/+NnSjRSNGrv13/qyw1x740Kc8bacum+HK2LhfO+MGeTQ8nwOVab/OJDu8/rk6U7Ha9rksJ9EDQcFYIQYqwQYiyURHRD1WRzpwMYAsD5bhjGJze/XoKPl+5ExRFl8/XrXl2E1yQzi2o8/vla3GcSjA8aQiwBRUg9Nd0+g6ZX3lm0TXcZJTsZqk1mVhuE40dLd2LfkVrHUfPRuogeYuuFN13i6DeUK1k6BYTtROuisgNx79M9ug+CKom1Am7YTUPsV+eH7HjnF2ck3bZXZOcQ+goh9F+XEGIVgP7hdIlp7Bj91WETdKqYqK4QAqrP9BlEhJCaQ/A6zyA7eRsVHtxhuaARAsDPXEBhPqU85BSQT12xkoheBvCm+v4aACscyjOMb2yzbgqRVNSJlZAMOkdO0Ftemvsnm8IhrLQaQgjp6Be/eyrnGn7mk9IRYQTIK4QbANwK4Dfq+7lQ9jZgmNAwC5T6iEBRQbA/lKBz5GgWgp96rRSWuZ6IyxyChtHVFCRCyI94g9/+Mzvx4z5MR4QRIKEQiCgfwBfqXMLT4XeJaezYpRNOVnhbCdKgR/RafX6yVFp1xcpCkBl514SUiTUqhPSIl+0DBT+D/WSj1PziOocghIgAiBJRmxT0h2F0zALSLW7btT4LERV0TL8WSujLQrA4ZlYIUZFmCwHyo9ew1ktkG34shHSt0JZ1GR2BMo8wHYAeuyeEuCOUXjGNGqtEagCwaschjDqpo+U1myqO4O2F23D/xf3jRlele6sc29IE9z++2oQXv9qE+RPPR7OifMxevxfbDxzz3HdNgLulJTDzfsn2uDBDTR58U7o/rtwfPlmFBZvjo3msqKmXb//xKWvwz3lbpMp62cCe1YF/Mn0O4SP1j2FCxy5Nwk9fXoiySRMsr/nlG4tRuvcIrjmjN07o1FI/fv2r3+mvLV1Gqtyc9IWSyvmDxdtx7chi/Pxf3yUWlkBLWObVZfS7D+RiNGSUgdIPeYUgqwy8kor9IsLmgv6dMWOt/1TpgNyzuGJoT2w/cAyXDOqGBz9dndkWghDitbA7wjBm/HgczK4ao8tFxkef7Byzts9yqne6MpMJo/OakNxWqWRg9zaeFULfLq2wvjxmmdZIKMbfj+uLzq2bqlbp6lCyvMogpRCI6GQAfwIwAEBT7bgQ4oSQ+sUwntAWS5lTCscnjXaP4kkWbVSc7F64fvbpNZLqLSetOFaX/KKudONnpG6+xJzSw/oairs2XRaCrB76F5Qw0wYAYwG8jtiaBIZJO9p+AQ2miWfjugUZCyFZdIWQbgsh/foA1R7mMTKVVLnyta+p9twyXSE0E0LMBEBCiK1CiIcBWDtzGSZJzD8OGQrU/ABe/dZBKwRtW81U7wFsJhMifKrrs99l5Cf8M5moooghW246kJ1UriWiPAAbieh2KHmMWrpcwzC+8DM4KlQthBqTEDLWZSUizYI7WUGqWQZJu4ySlAfpVwdAdQ64jPw8Bz9bKGjNaOtY0qUQZLv+GwDNAdwB4HQAPwNwfVidYrKPl+dtxlqHvXujUYG/TluHvYcTs4sKIfDU9A3YVVkNIOY/jwiRkOX0ulcX4aZ/f4fFWw/GZTP9rkzZn7a6LhKXZXTHwWpDQ4n9WrD5QJwScNvWUJZkXUbr9jiHy7pxj2TUUpiEFb2USlK1hkDfkS7gXFhekVUIB4QQR4QQO4QQPxdCXCGEWBBqz5is4rEpazH+2Xm25xdvO4i/z95kmWJ53Z4qPDdzI379lrK9hvbj2HOoJiHL6dwNFZi5bi+ueGF+QjZTQBHoT0yzzmAqIDBY3WC9Wxs9NgKbKo7or82bvPgl3S6jLYZU34x/fjikh+dr/OTb0q4p7tgCZ5/cEU/8eJDnOoJAViG8SkSbiOhdIrqNiE4NtVdMzqENwq1W0GrnzO4eP/79qBAJE8vGdrq2VhRB2+ZFced6tG2mlglGkAdlaWQbbZoVhlJv2aQJ+O/tZ4VSt5P87tCiyP6kXX02x9s71KX1oTA/D2/cdAaG9G7nud0gkF2HcC4RFQEYDmAMgClE1FII0T7MzjHZgYwQ1eKqrYS85i/Vzun+VB+j7KhDRlTjjmbGPhvLBzWwT3YOIVspzA/P1xFW4A3Bfs7Fjy/fj7snzTuF6siuQzgLwNnqX1sAnwGw9w8wjQoZIaoJXauBs+4/tcj9770vwjHXjtaGsep8l9BUP9Sn2WWULpJJT+5GWKGYRGT74P3cTzblLjIjG2U0B8BiKIvTPhdCOG/1A4CIXgVwCYC9QohT1GPtAbwHoBhAGYCfCCEOeu41k1HIuHY0oRu1EJR67iLNQtCUhw+3i7LFo/U5pX6lTmOf8+IshGAEeaSR5n4OczLUT/SODEF3OYjFbOlC9iPuCOCPAEYCmEpEM4joUZdr/g1gnOnYRAAzhRAnA5ipvmeyHCmFkBcfZx2PlrvI+E7OQhCmHcQiri6jxD67hab6Id0L09JFsius01F30MLYT31hfm5ekFIIQohKAJsBbIGyv/KJAM5xuWYuAHMmrssAaHmRXgPwAy+dZdLL2wu36RvVf7ZiF1bsUDJfygyGtVGTURBHogJ/m7URC7coGT0FgHkbK7BZbcPKmjCzs7I6ztUkhLAdoS3dVqkrj00VsSicqav26MffXrhNql03Xvxqc9J1ZCPpCpdMhqDdXL4UQoZ8blIKgYg2A3gSQHsoKSz6CiHO9dFeFyHEbvX1HgBdHNq8hYhKiKikoqLCR1NMkESjAvd9vBKXvzAfAHD720vx/b99o5yTGMlrX3ijhfDFqt144ssNuP/jVfq5a19ZpJ+XsRB++cbiuPajAo6Jwaxk/eOfr9Utg31HajFjbblru4w1yQjXji2bOJ4Pa0vOoJWYebR/1fBeuHXMic7XZIhCkJ1DOEkIEahTVAghiMj2CQshXgLwEgAMGzascdrfGYT2AA4cTZw+klEIWhmjkK+qabAsoyETZXSouj6uXCRqbyE49dV4/HBN9q+wTRfJCLavfz8W/R6Yans+rGwcQbtrzHMdk644DQDw3nfbU9YHv8jOIZxERDOJaBUAENFpRPQHH+2VE1E3tY5uAJJLNM6kDKfQUhmXkXa50R1jzgJpbkJGIZgDRIQQvmYJjXWY10Mw8iQTLeN2aWgKIWBZ7C/KKNg++EVWIfwTwL0A6gFACLECwFU+2puMWMqL6wF86qMOJg04/Ra9WAhGGZ+oELxbCHlECS4jmX4kHo+9ZoXgn2SEq9vWnOG5jIKeQ/C/UjndyCqE5kKIRaZjjnY1Eb0D4FsAfYloBxHdBGASgAuJaCOAC9T3TBbgJPPlFILy3yjkzWkizMJcam4C8W4oNyViX2XsRC7s9JUukhFrboI5eyyE1FwTBrJzCPuI6ESovxoi+hGUaCNbhBBX25w6X757TKbgNDqTmfyNWQgGwWsaiZvbkFnsm0cE4+yWmxKxO288HNYG9Y2BMF1GYW36E7Qs9lNftlkItwF4EUA/ItoJ4LcAfhVar5iM4UhtA95csNVxdGZ17v2S7dh3pNZQRim0+1As2+kS04bt5Ydr496X7j0CNzbvO4rJK3bp76NCOPq37PYkNgobjjJKgiTkmptQDM9CCHhSOUOEux9k1yFsFkJcAKATgH4AzgUQTqYpJqN4ZPJq/OGTVZi/aZ9tGfPIbcfBY/jdBytw65uLDWUSrzMqDCte/UYuffIDn8RSZEdFYuI6GYz9Szb1dGNGyybrl6tH9LI9d2LnxC1Yuhuy1trRs10z9OkSf+2vx5yITq2UMNfAXUY2/p8HLxlgebyoIE0bKFvg2BMiak1E9xLR34joQgDHoEwGlwL4SSo6yKQXLcz0mIMbxey3b1BTTuytMloIidd5GUmd0LGFnpHUiagQ+v7KXqh38E+FmbDNK/26tvJ8zWWDu4fQk0RGndgB7W2U8RnHy+XBfOjSgbbnWjZJ9HC/+vPhjvWd1rMNvv79eXj3lpFxx+8Z1w+Tbx8NIIRJZcPrU3q01l//YEgPlE1K3Ghy2YMXBtp+MrjNIbwB4CCUyeFfALgfyv3+UAixLOS+MRmEF5eR1RaYSft/JX+z0ahAxH55iy1OCkERGNm7FCYDdtOUFrpes4vK1utUKmh1bxwf5UskYMokF5ObQjhBCHEqABDRy1AmknsLIRK3vWJyEhlZYjtRa7jaWKY+EkVhfp4nU122aFT4Uz5O+xekazvDXEL2WbuFnpoJIjLJrUmHZKiW1DbErGkZ4zKD9IHrHIK+F6EQIgJgByuDxonT78HsMiJTsjrzaz9x/rITf1EhAh8RexVSYeIvxj2EjnhEdhTsdVN72eLOzTtX4nUEbwxbLsgxC2EQEWkb5RKAZup7gpJ9orX9pUwu4bhS2cZlFF8mVqi6PoJWTb3trCVtIfh0GTnhVUhlGqlyGTnJtbBknpv1JrOYza1veQR4GcIYF1zKWJeZpBAc1ZcQIl8I0Vr9ayWEKDC8ZmWQwdQ2RPDx0h0Jgryk7MjoRZMAACAASURBVIBtOGddQxQfLVGumbexAl9v3IfFW923q1i5s9LyeCQq8OHiHYhERVybtfXKD8aTy4jkdmZ7c+E2rC8PNkqIXUbJE1acvfwcQmI57evk9ni9CmyjQiiQ8Bll0tcrc+KdmEB5dsZG3PneckxfEx9T/6N/fIsLnvrK8prn55Tirv8sx2crduPaVxbhZ68sxKFqxWvoJIvvfG953Hvt97P7UA3ufn85Xv+2DI/8d41+XvOxeknoRSCp+YwDR+swZYXjmknPZNIIzk9PrhxuH8qZSoYdF9w+wT87szeAeOvN8TFZnNP2OL7j/JMd2/L6/E833Of1I4tdy2fKojSAFULOoi3yqqyudykZo0INE608lpjR1EseGbPy2HM4ftrJz3bD6fzNpHoEd9eFfSzDE2Xo2jo+Lv+GUcUYfVJHy7Jtm7u77Wbc5SfLPXD/xf3j3gsh8MGto3DNGb191Wfm1jEnoWzShLhns/yhizzV0bQwH2WTJuCaM45zLOf1+fdo1wxlkyagbNIEXDDANsN/RsIKIUfRBagH/7E2ErK6JBk/tDmJXUPUu8so2T4kQ6qVUTL36UV4yRT1/Iz0/+E+LO0+jaN3p88tmWfodQSfCWG+fmGFkKPE9IH8t1P7kVntGOblS24O+zQni5PJYmomnWZ1RrmMXEMk5fsqc19e7j2VOf21fsV1z+JrFYRwzqDHHzqsEHIUq8Vh7tcoF1mF5Hv5XZnlfaKF4EMhIPxRp1PbqSSZ+/SyEb2MoPN676T/j79S3y87oA9Tq8dNCZn36faD1wFBNisQVgg5SmwPYx/XWFoIXuYQ4suaFYIfCyGdZNKknxueRvQBWwipROtXKuZ3vLbBLiMmdNbuPqxvcC+D9juevmaPozBfvPUg9lYpk77aXsRvL9qWUM6YAXT7gWP666mrEiN6zPJ+8vJdce8/KNmBaNTbArJjdQ1Z/UPzQjK+cC+yS0bQBaUPgrbuYi4jybDTJG4kU5ViGLBCyBLGPzsPY56YI11e+wHMXl+RIJCNXPHCfFz87Ndx12yxUDzTVscUwtl/ma2//tWbSyxqdf7xv1eyHev2VHlKMVG2/1jasgnJyIMOLbxnWLVD5j7thLlZeH1vYFfbOmR8/snIwlEndtBfx1w3chVe5BKdE5tUjh1rVpTvqX9mbhx9vOVxr8rkLJuoLo0L+ncGEAudzSRYIeQoxq9w+WHnbCNaGuqgBkIyHqG6SNSTO8uNc/t0csyGen6/zr7rlhkhXnxqN/xUDam8Z1xf23KjT4oJye8P8paF9BdnxwTWmzefgbJJE/DP64bFldG62qppAcomTcBIg1A2903medvd+/PXDE04Zi769i/OxBs3jQCQuAjs3vGxbKNmyiZNwEum+0poS/2GGxVMUUGebbiu262WTZqABy+1Tk+db5CSX/1ujGNIcNmkCTi1ZxvHtl6+fjjKJk3AYz841Xd4cViwQshRjD9O2YF4UKaxTHtKzqHgNEJ+HoWWplo6X476v5VFmuZYXbHKbFdA23wuRJQgCM2foe5KkeivzPO2KyL7mWh5oDSXkTbaLszPSyoqifLi/7uWT+KrYfycUhlJlQ5YIeQoxi+urNgNaoJOxhUkfCShcyqfR+Ro2iejemRcBkSx8NomBfauC6NwCWK7Sa/35VWg2fVR2o1iKhYLdhCBCGnXbKf+m0hoC8juCCIZWCHkKOm0EGQUQiQa7B65eRReeKhsvVo0ldMOWEala3RFGK0Fu0+FkCiQ7KwsO4FtPCwTomp3716/K2aXkRDJfd+0eqxqyIv77quWSTLWiOFSPyHT2QQrhBwlfr2O9ZfYHF6aapeRd4XgvGeBU/eTck9JxuvLKASj4DduntJUchtFMv23uy27z8J4WM5lZF1G3o2muYzU6/LCtxBkUk77aQuI3+sgF2GFkCFUVNWidG8sS+fR2gYs356YRfRobQMAYO/hGvynZDsWbTmAmvoIhBBYsHk/hBAo3VuFEkOW0lU7D1m2GTFJkyD0wd6qGpSUWW9kb2Ta6j3YZghfTZb8PHIUcEm5jCTLacKiiaOFYFQIseNNC2NuJrlNXeIFrX6tdt69Csl5BrvjsuGe1u8jQgTiMrOqwirDaHLKJ/a6pt5HIq4sghVChjD6z7NwwVNz9fd3vLMUl/39G1TVxCenu/UtJcxzxP/OxD0frMBPXvwW9360EtNWl+OqlxbgzQVbccFTc7F612H9ms9X7rFcDGY+FoSFMOLxmXjYkNnUjn99U+a4S5kVmqC0iiZysxCS4Uenu2cLJSKMP6UbAKBPF/t9j40WwnmGyKcB3b1lk9fu1WxlXaVmNr16RDAhjbYWgoWmuOS0bvYVqd3UJ5mFu5B2ChKwTF2hcu1I52R1spzQsQUA4MfDYs+/W5umdsVzAlYIGYJ5Ne+SbcoI3yw0l25L3J9g+Y5K7DiojLa37LMedVu5Z8zHMikvuxVab9+95cyEc3nkYiGoF79yvXM4o5mfntEbvzr3BGz634v1Y5/9v7P013+YEMvq+eNhPbH+sXHo1b65bX2aIG3fogjn9YvF2hd3aBHrq3qnpY+PR+nj43H3hX2UE4ZJBCuX0YbHxuOms47HxsfH43ffiw99vXxID6V9Fym88L7zUfr4+Fh/JaKM/nzFqVj/2DhcOTxRCen9VO/JuBre7eu27tHxtuecXEYTx/Vzqdmd0sfHY7qa6fXXY07EhsfGY8Nj49Gltb1CyIUJZ7cd05g0YWf6W7kTjMfsvpRWYZ7mCbJsSdFg5aN3izLSy3nUes0K80FEcXvjGl1CxjaJyDHCCIiNkAsc+qE9pgLVp1SQn3i/VlFG2udiNbLW7tvtI8rPo7j27CZjzVlG7e6bDBaBcp3yPyphIThtTOQ0qWz1PfD61Y77DIhQVOBegdMzzRbYQshQ4vcjjr2xG+m7JfESIlGZhDWpHDZWvczPS10SumQEjh+ZYR0UoAnacKNe7OL8jffrtEe2WXFpn12yEWbkYCGki1zYWS8tFgIRlQGogrJVaYMQwpsd3wjQfugC8SN5q9+RTLI4q6ge83UWA9GMQg8htBACrlFGPtu0HoEm9kkWuz0n4uq07UtsvK4LWsnmpUfSEtcB8YK4psF+otV8vX7/ASkyN33gNWVGMuRnkHLySzpdRmOFEPvS2H5WEIkKNBjmEaxGi0K4Jw+LRAUieSaFkKXZ4qx+d+Qyh6CXC6Itj3UY0Vw3ZoEYFyZseixOj0l2pG1vY5iOmW7Y7jM1joZrJSJvtPs1uoyCIJPcnF7dkZkIzyFkGGt2HUaTwjw9vO1wTT2aFsaG7kIA35TG69GdldX6ZLLdlpn7j9TpO5UBQOneI3E+zy37jmLexuzQz1Y/u4IQo4zMJOOmMEbZ2GGn3K0WG3oVrHECVELZ2d1rvIUg7zIyrkNIJan4bvAcgn8EgC+JaDER3WJVgIhuIaISIiqpqKhIcffSx8XPzcP5T36FOnXj4cufnx9nIdQ2RHHNywsTrntHTVn9weIdlvWOeWJOXFjrBU99hTveXaq/v/CprzJeIVw2WImUaV6UOI7JI3LM7Hle304A4qN5ZLASuJpw6dU+Fv4q65LQLQQPfRjSuy0AYHhxO+mVyna49dJu3YAZo+wb2ju2qfyI49sDAE5TE7x1V0OEL1Szl1LAFkLQ9OtqHzLsBs8h+OcsIcROIuoMYDoRrRNCzDUWEEK8BOAlABg2bFiGfn3C51B1fWijqRU7YgvWsmFJ/gOXDMCdF/axTHOcn0f49ZgTMWvdXizemhiae/2oYlw2uAfaOaSpXv7gRXhz4Vb8ddp6/Vh9JNEdQiCsfuR7yM8jvLlgq6d70OZpHJ+p6dSoEzti6QMXol2LIvzfrFLLohNOdVgD4EJxh+Yo269YmGbFZq8QYicuNKSqvmhgV72vANCtTTMse/BCtGlWGHdd2JPhGvoCNsnyn94+Om4A5oVMmuD2S1osBCHETvX/XgAfAxiRjn5kC9nq6w+a/DzSBYuZPFLcIW1tzhORozIAgNbNCvQVw5qbzso/TgS0aKKU9bo1pJzLKJHEvqtBB2pFzSX3ArDqZ5vmhrrNFoJE2KkZc1/bNi8yRAUpx1LlMvI6x9CkIB8tHLLVOsEuIx8QUQsiaqW9BnARgFWp7kc2wfrAHfOEnp9U2Ep6aYUidShfZ2UhJPG7t51Ulk0FYXrvVSGRzWs7bDN0+4zb8rO1azJoFlkqJp95UtkfXQB8rD6gAgBvCyGmpqEfWUOqJ+CyEXPIX2F+Huoj/hORNSnMB2oaLJOZxS3K0vL8S9ZrF3ZqxNP+1RZ9kkUXkob2zNXY1WvlSvPSZqr21fayP0Sy5IKFkHKFIITYDGBQqtvNZrJtU/p0YJ7QS3aCT7cQLGLsrcNe5erVu5XkI41FGWlrM+SucxspJ6xDsCle67D2wIl8XQdlpssoGXLBQsjwpUjhs21/cBk3zQgh9A3pD9fU4+DROv3crspq1EeiqI9Esauy2qWe0LqYM5h/+H4VglaNlp7CSvDFbT7k8dnIRBk51Wm+z1hZSZeTSzFz/XYC1a9CiIWd+rrce3vapHIKZHUuLExr1Aph+ppynPPX2fhy9Z5Q6n9/8Q6c/ZfZWLTlAIY9OgNDHp0OQElhPWrSLNz/8Uo8+OlqjJo0y7Eedhm5c1qP+H1sk/1xagnqRhS3Tzhn1DX9uylZSgf1aitVb2xS2f6ZytYFxDKrnn5cO8dyw9TzJ3ZqqR87vXfiNbKfWvc2SjjpOIdQXyv6qv0d3Kst2jZ3nuS34pQeHrPCmt577a8dxu9Ax5ZNAABj1NDmbKZRL0xbvUsJu1y16zAuCuiLYmSJGv5YuvdI3OSkNrqatroczQrdo0My3WV0z7i+uGFUMVbuOITN+47i3o9WBlb38OJ2+K4sMYzUzAWG0EfAv/muXVXcoTke//1YXfBZFgJwTp9OmHfPWMcMp98f1B2Tl+8CEEtA55S64tJB3V37qV0/4vj2mHfPWPRsZ9FPA1cO74XRJ3VEp1ZN9GNXjeiF90q22/ZDY+F956NV0wIMeHCafqxv11ZYeN/5aNvcOqrLjjNO6KD3l4gwf+J5rgMiI+//chSOqHuCOPH8NUPx67eWGNJkk6/+2kFEunk2756xOHiszjETarbQqBWC3UblQWEXAaK9rY9EpRRChusDdGvTFM2LCnDGCR3Q2ibs0y8yC76sPsMg3Lk921kLeXOfnJSBuS+aQvBr9elTEIbL3doHFAHWq33zuER01rmMEo/ZCTq/AtDY3+4We1s40awo33Idipn2auircWAQpMDOIyURW6xP3u4jU2nULiOvCcK8EssBH39cW1cgG6mRTS6jwFdrSlRn1WSY/lyvt2gUvNpktd9HGsZtZc+3Sx7tNxPWPG8qkuWlg0atEPQNv0P6SWgje/OXR3MB1Ufk9pXNJoWQjtWaViPdMCM+kolcsXUZeRQwfq1a90llX9VmHLH9F0LTCDlJo1YIsdzs4dSv/2bNFoLHBqNZtI1rOmKxrVpMVhA4PSGvNRvLF2orpfxaCCFIolyUbTELIZy7y4EIU0sauUJQ/luNwCNRYRuSagwfPVrbYLtBiN2iJaNCcNpcRCObLISgXUYytVn95v0qJqld17wKGUNxPcrIpBFSNTK3UijGnuSKhRC1mb8LCnYZ5SD6D9tC3v5tVinO+etsLN56IO74/E37MOTR6ZixphwAMPChaRj/7DzrBvQvZfyXxyjgDx6zTldt5K2FW13LZApeFUJLl7wxMj/os09ODPfr73HTejNWOnigWmehxHaKZ5/c0fJ4UPNWvjf8MXS9gzrxagxZ1QSdU9SSXT6pdNKtTfyEcWzBXjiC2+75ZjuNOsrIKdHWsu1KqGOlSWAv3VYJAFi87aAe6rhl31HL+m08Rp5dRnM3eE9L3a9rK6zbU+X5OkAJo7v5tRKsL/d+fRPTfseF+YR3bzkTV7zwLQBgwb3n47a3l2Dx1oO4cEAXPH/NUJx8/xeOdS6493zb3EQz7z4XPSwiVX4wuAduG3MSurRukpDJddH956OuIYrahig27KnCrW8t0X37TvLjXz8fjt2VNZbpt828dO0w/H12Kf42Oz47qUzqCkcClG+92jfH9DvPQXHHFvjXN2VK9Wr9n//mbByyGazM/d3Y4Drhwnf3X+A6x/f178eiVdN4JaXNsfhIaSXFc1cPwepdh2wj0bKVRq0QtNGQlXy2k9naPsQyA2F9lyiTHebVBVTg41s9uFdb3wqhU6sm6Ny6iS+F0NQUAtq0IB+nHxdb3NW1TVOcdVJHLN56EP27tY751G0gELq2sQ8XNC60ir8OGGBjJXRuFatPE9BmRWYlhDq3ahp3rRPNivJxXIfmal9izy9fX6mbHjeg+Zt0cpdWludbNy1E66bWlkCbgGL5ZTCum7DDSihr825hzSE0LYz/XucKjdpl5GS+az9Y82heeysT1qiVNdfvde8BP375ZOQNkX9T26wQkiVsn7b20TYpUPodtmc4MJeR77BV5zvMpC0pkyFsl1Gu0sgVgvUEnxGz7I54+KLZKRWvLiM/E6TJjEApiSkzL8ork36qCRZCAAN4qyrsvjeyn0UmfWaZjPYTy9VooLBo1Aohz2G0JvTRvTAdV32TEt807coEK8NjGKkfszeZUFoibz+kMCMuwh7gaYsDmxSG91Mw3oPd55q6KKPkzmcLQh+4pbkjWUbOKwQtn70QIi63fW1DRP/yW42m9dG9+r/yWB0iUaEfFyJxpbHWhh6KqlZbXR/fbrVEqKmRw9XukUhmkllsR/CmhLy25aV02OF9NeqOaEWmuYxMD1f0+3wby8K0mIWQIzeUInJ6UvmzFbtw+9tLMf3Oc/DR0p14Yc4mrHt0HGrroxj0xy/1CSsrF04s3zxQVVOPwX+cjssGd0c3NdnZ0zM24OkZG+Ku+cu09Xhhzib9/VB1c/RH/rtGP9b3D973Atp1qMbzNUL4jzQiosAEw+DescydHUxbK4bxW+3btRVmrtsrNRkJxMJeh6iZP3uo4ZbHd7SerPaEhczu2qaJ2l7sc/HiEhzYvQ0Wbjmg5+oJitOPa4fFWw/mjM+9c2vlc+7btZVLScZITiuE6epagdW7DuPNb5VY/tr6KCqO1AIAKqqU/1YKIWYJCD309NNlu3DLOSfYtvfuom1x78tC3GvBDSEE3vvlSAx65EvL89eNPA6vf2u9viGPvM4FxJedefe5aNWkAOWHa3FCpxYAgOl3nmMrxGbcdS7aNCvErspqnNylJaavKUdVTQP+8In3nVXvurAPzuvXWTqFdHHHFvjw1lE4VU2ffV6/LnjvljMx3CLttV+Mn87px7XHlDvO0hO8aff+0lxlIHHdyOMc67r34n6YcFo39Ovqb52FncD/98+HY9uB9Hxf5/5uLPIDjg8dXtwe7/9qJIZapPhm7MlphWAkfm9c4XAuvkQkKuJcPFEH57zZPG3wuc1gEESF8wKin57R21YhEBEKXMJBjZjdF1ooaGdDdklzeKORkzor5bVR/WWDe2Duhgq1L9LdAAAU5OdhmEdhbt5L4IwTOnhr1CMDu8f2btDuXcNqTYWRwvw8170P/NCqaWFcv1JJ7w7hxPIHqdQbCzk/h6ChCf2IxXxBfcRqVln5FxXx6SWcJmvNwiudaavdmm5a4BweWihhIZgjc6SRCOHJnmQd9njx88ci3hgmfTQahaDJoEhUJMijOouN1DWXUVQIVNdFEo5bYTbH07mxjVvYqVtOeRkLQV/dG8LEb6r23E0FMlZObnjumWyn0SgEDStBaWUhaEeiUYEadYezwnxyVgim9w1pTFPqJlDdLASZic4mLnUEQa5McjJMNpDTCsFKJkaiiYZ8ncWG4ZpAra6P6BZCQV6e7SpjIRL3NkinheA2wG5a5PzoZdJl+HYZqThZFrlgH+SQkcM0EnJaIWgYBfWoSbMSEtZtKK9C8cQp+HL1Hpz68DQUT5yCJWoSu0f+uwYfLFb2nW1SmIdPl+60bOOCp75C+eHauGPpnENwcxmZ4+7N7DxY7dqGFjrZoWWwIZAAYplig6855XhxqbESYdJJTisEu7wx5Yfj4/r3quGnny7bhaqaxA28a1ULolubZmhvI/w2VVhnPPXKhFO74bUbR+Cf1w3DV78bE3fu5euGJZT/x8+G6q9fu3EEJpzWDYCzYPn0ttEgIrz/q5GY/T9jLMus3nXYsZ9v3DQCz1w5GK/eMMwy/bQTLPMsyALN9/Xvx2LKHWeluxtMiOR02GlQo60jtYqSiKRgTuDSQd1wbh9rAaul2zZiXIB1bp9OqKmPYMqK3Y7WiRajP7y4ve1cg1sWUk0JnNcvsU9BoG8ulAWC0g49/bmHewhrO9cg6NmuOXpyWH9Ok9MWgob5R+bVt39EtRoaIokRSsHjVQLGl9fz7Ut21G7S1k/KbSYe7RHIRRnx582kn5xWCNoP0RxFVGsRZqpcYH1YsxDqU2AhJDsi1hP2JdmPVOyNnM2jf4bJRdKiEIhoHBGtJ6JSIpoYVjvGtQdGtIRmsmjzCpEUWAjJykinfaK9kO6kYCIHJpUz2f3DMFakXCEQUT6AvwMYD2AAgKuJaECYbZrDSr1mG9UsBKsUF0HjFnfvNnLXV7wmKYusVnSng9xYh8BRRkx2kA4LYQSAUiHEZiFEHYB3AVwWRkNagraHJq+OOz7pi3WW5aes2O1Y374jddhZ6R6OmQxuosOcvbG5acWxtu2hedNxGYwb3vcKca9YGaHXKon7yBTaNVci0mQyr2plgs5iakfHMEKFmawnHVFGPQBsN7zfAeAMcyEiugXALQDQu3dvXw1dNrg7Vu86hA3lR3xdb+TJHw/C3e8vT7qeTq2aoF/XVhjSqy2emxXbgP2skzri69J9CX71d35xJq7+5wL9/es3jsAzMzbijQVKYrr+3Vpj8u2j9aig049rh/+7eggu6K9E/3z2/87ChvIqnNipJS77+zeOfZt25znYoobPPnvVYDwzYyMuHNAF09eUY8Hm/Vi3pwo/Or0nfnamc0bOIBhxfHs8d/UQXGQRWZUtjD+lK56+chAuOa07zjm5o6Owv2FUMdq3KMRlg3qE3q/XbxyBPg7JBpnGS8aGnQohXgLwEgAMGzbMlyE9pm9nnNunE46/93PbMnkkt4Bswmnd8PHSnfi6dJ+fruj864bhOKVHG3y2YlfccS2qx6wQRp4Yn3mzQ8sm+N7ArnhjwVYML1ZiAE/rGZ/q+dJB3fXXp/Rog1N6yGWx7NG2mZ5ts23zIjz8/YEAgNEndcRjn63Buj1V6NOlJQZLppZOlu8b7iMbISL8cEhPAHDNwJqfFysbNufYhDUzTDpcRjsB9DK876keCwVXn7xkmucmBXloGsA2i5obK9/Ur9gkqru/WZtGKMhL3eMLanN4wLDGIPmqGIYJkHQohO8AnExExxNREYCrAExOQz8AyIdXEhGaFCafzE1XCKZ2dTkr0Z161aThtQIMwwRJyl1GQogGIrodwDQA+QBeFUKsdrksNLzE2zcLQCFo4ZwJCkHIj5q1FdNuq4mDhPP1M0zuk5Y5BCHE5wDsHfspxMtWkUG6jPJs2pWJ/29QF9p56XuyhNFSTkSUMkwOkdMrlWXo0lo+rDEIC0FLGW22TLQN6N02rgGgu646S24kHwRtmithoMbQVN91qVt7aqGlDMNkBhkbZZQKfj+uHy4f2gPfbtqPPl1a4eLn5gEA+nVthXV7qgAo4aDXjyoGADR1UAiPXjYQZfuP4aIBXXDlS0qY6A2jivHv+WV6mSd/PAjd1SieYce1x10X9sGJnVqiQ8siDOzeGiOO74BhEvvlnnNyRzz6g1NwxVBvIYr/umE4erZL3LP3i9+crWd8tePms05Ai6ICXD3CXwiwkZ+PPh5NCvJxzRn+63r9xhFS8f0Mw8jTqBXCj4f1RMeWTfCDIYpgvXpEb7yzaBsuHdQd6/asBwC8eXNsiYSmEM7t0wlfqZvAa/Rs3xzXjiyOO/bw9wfGKYQrTo+FFTYryscd558cV/6nkgKSiHCtj7UAY/t1tjzev1tr9O/mfG1RQZ6uGJOlMD/5ujh0kmGCp1G7jMyhnw1qagq7ncA0hWA1EZ3u3D8MwzDJ0rgVgilsU0uCZ68QlONWE8Jm5cIwDJNtNGqFYB7pN+gKwXquoJmDhcD6gGGYbKdRKwRz2GaDGt/fxCa8VHMZWaWW5iyVDMNkO41bIZiG9c2LlDl2LbTSnIxMcxlZhZ+67T/Qummjnr9nGCYLaFRS6qmfDMLmiqP422wly6jZQnjgkgE4oVMLjO3bGX+6/FSMPCE+sdzw4vb4xdnH46oRvXFu307o3601xj2jhKoa9w/45LbR2K2myf73z4fj46U7cecFfXz3+x8/G+ppvQTDMIwfGpVCuHyoEvapKQRz4rs2zQrx6zEnAYBlvH2rpoW4f4Kyl8+JnVoCAMb07YQ56yvi9jAe3KutnhF0TN/OGNPXOtxTlnGnuMSEMgzDBECjdhkFgRZumoLtlhmGYUKFFUKS6AqBZ5UZhslyWCEkiTYNwQqBYZhshxVCksQshDR3hGEYJklYISSJtsl9CjNRMwzDhEKjiDJ68drT49YcfHjrKGworwqk7gcvHYDOrZvqm9ozDMNkKySywPc9bNgwUVJSku5uMAzDZBVEtFgIMUy2PLuMGIZhGACsEBiGYRgVVggMwzAMAFYIDMMwjAorBIZhGAYAKwSGYRhGhRUCwzAMA4AVAsMwDKOSFQvTiKgCwFafl3cEsC/A7qQbvp/Mhu8n88m1e3K6n+OEEJ1kK8oKhZAMRFTiZaVepsP3k9nw/WQ+uXZPQd4Pu4wYhmEYAKwQGIZhGJXGoBBeSncHAobvJ7Ph+8l8cu2eArufnJ9DYBiGYeRoDBYCwzAMIwErBIZhGAZAjisEIhpHROuJqJSIJqa7PzIQ1nN1PgAABo1JREFUUS8imk1Ea4hoNRH9Rj3enoimE9FG9X879TgR0XPqPa4goqHpvYNEiCifiJYS0Wfq++OJaKHa5/eIqEg93kR9X6qeL05nv+0gorZE9AERrSOitUQ0Msufz53qd20VEb1DRE2z6RkR0atEtJeIVhmOeX4eRHS9Wn4jEV2fjntR+2F1P39Vv28riOhjImprOHevej/rieh7huPe5Z8QIif/AOQD2ATgBABFAJYDGJDufkn0uxuAoerrVgA2ABgA4C8AJqrHJwL4s/r6YgBfACAAZwJYmO57sLinuwC8DeAz9f1/AFylvv4HgFvV178G8A/19VUA3kt3323u5zUAN6uviwC0zdbnA6AHgC0AmhmezQ3Z9IwAnANgKIBVhmOengeA9gA2q//bqa/bZdD9XASgQH39Z8P9DFBlWxMAx6syL9+v/Ev7FzLED3UkgGmG9/cCuDfd/fJxH58CuBDAegDd1GPdAKxXX78I4GpDeb1cJvwB6AlgJoDzAHym/hD3Gb7c+nMCMA3ASPV1gVqO0n0PpvtpowpQMh3P1ufTA8B2VRAWqM/oe9n2jAAUmwSop+cB4GoALxqOx5VL9/2Yzv0QwFvq6zi5pj0fv/Ivl11G2hddY4d6LGtQzfEhABYC6CKE2K2e2gOgi/o60+/zGQD3AIiq7zsAqBRCNKjvjf3V70U9f0gtn0kcD6ACwL9UN9jLRNQCWfp8hBA7ATwBYBuA3VA+88XI7mcEeH8eGf2cTNwIxcoBAr6fXFYIWQ0RtQTwIYDfCiEOG88JReVnfLwwEV0CYK8QYnG6+xIgBVDM+ReEEEMAHIXiktDJlucDAKpv/TIoiq47gBYAxqW1UwGTTc/DDSK6H0ADgLfCqD+XFcJOAL0M73uqxzIeIiqEogzeEkJ8pB4uJ6Ju6vluAPaqxzP5PkcD+D4RlQF4F4rb6FkAbYmoQC1j7K9+L+r5NgD2p7LDEuwAsEMIsVB9/wEUBZGNzwcALgCwRQhRIYSoB/ARlOeWzc8I8P48Mv05gYhuAHAJgGtUJQcEfD+5rBC+A3CyGi1RBGUCbHKa++QKERGAVwCsFUI8ZTg1GYAW+XA9lLkF7fh1avTEmQAOGUzltCKEuFcI0VMIUQzl858lhLgGwGwAP1KLme9Fu8cfqeUzamQnhNgDYDsR9VUPnQ9gDbLw+ahsA3AmETVXv3va/WTtM1Lx+jymAbiIiNqpVtNF6rGMgIjGQXG9fl8IccxwajKAq9Tor+MBnAxgEfzKv3RPBoU8MXMxlCidTQDuT3d/JPt8FhTzdgWAZerfxVD8tDMBbAQwA0B7tTwB+Lt6jysBDEv3Pdjc1xjEooxOUL+0pQDeB9BEPd5UfV+qnj8h3f22uZfBAErUZ/QJlKiUrH0+AB4BsA7AKgBvQIlYyZpnBOAdKPMf9VAsuJv8PA8ovvlS9e/nGXY/pVDmBDSZ8A9D+fvV+1kPYLzhuGf5x6krGIZhGAC57TJiGIZhPMAKgWEYhgHACoFhGIZRYYXAMAzDAGCFwDAMw6iwQmAaJUQUIaJlhj/HbJBE9Csiui6AdsuIqGOy9TBMGHDYKdMoIaIjQoiWaWi3DErs+75Ut80wbrCFwDAG1BH8X4hoJREtIqKT1OMPE9H/qK/vIGW/ihVE9K56rD0RfaIeW0BEp6nHOxDRl6TsN/AylIVRWls/U9tYRkQvElF+Gm6ZYXRYITCNlWYml9GVhnOHhBCnAvgblGytZiYCGCKEOA3Ar9RjjwBYqh67D8Dr6vGHAHwthBgI4GMAvQGAiPoDuBLAaCHEYAARANcEe4sM440C9yIMk5NUq4LYincM/5+2OL8CwFtE9AmU1BWAknLkCgAQQsxSLYPWUDY7uVw9PoWIDqrlzwdwOoDvlBRCaIZYAjaGSQusEBgmEWHzWmMCFEF/KYD7iehUH20QgNeEEPf6uJZhQoFdRgyTyJWG/98aTxBRHoBeQojZAH4PJf1zSwDzoLp8iGgMgH1C2cdiLoCfqsfHQ0mEByiJ135ERJ3Vc+2J6LgQ74lhXGELgWmsNCOiZYb3U4UQWuhpOyJaAaAWytaKRvIBvElEbaCM8p8TQlQS0cMAXlWvO4ZY6uVHALxDRKsBzIeSbhpCiDVE9AcAX6pKph7AbQC2Bn2jDCMLh50yjAEOC2UaM+wyYhiGYQCwhcAwDMOosIXAMAzDAGCFwDAMw6iwQmAYhmEAsEJgGIZhVFghMAzDMACA/w+6J2tqRJ36PQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the scores per episode\n",
    "plt.figure()\n",
    "plt.plot(scores)\n",
    "plt.xlabel('Episode')\n",
    "plt.ylabel('Reward')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize a Smart Agent\n",
    "> *Note*: Sometimes this cell requires a fresh kernel restart.\n",
    "\n",
    "The code below will load the model parameters for a pre-trained agent and run the agent in the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dqn import env_step, create_default_agent\n",
    "\n",
    "# Load model\n",
    "smart_agent = create_default_agent(env)\n",
    "smart_agent.dqn.load_state_dict(torch.load(model_output_path))\n",
    "\n",
    "# Visualize 4 episodes\n",
    "brain_name = env.brain_names[0]\n",
    "smart_agent_scores = []\n",
    "for _ in range(4):\n",
    "    smart_agent_score = 0.0\n",
    "    env_info = env.reset(train_mode=False)[brain_name]\n",
    "    state = env_info.vector_observations[0]\n",
    "    while True:\n",
    "        # By default, the agent uses an epsilon of 0.0, which is equivalent to using a greedy policy\n",
    "        action = smart_agent.act(state)\n",
    "        state, reward, done = env_step(env, brain_name, action)\n",
    "        smart_agent_score += reward\n",
    "        if done:\n",
    "            break\n",
    "    smart_agent_scores.append(smart_agent_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[14.0, 15.0, 10.0, 17.0]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print out the scores collected by the smart agent in each episode\n",
    "smart_agent_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleanup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Close the environment\n",
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}