DDPG_Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DDPG Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='../../Reacher_Linux_NoVis/Reacher.x86_64')\n",
    "# env = UnityEnvironment(file_name='../../Reacher_Linux/Reacher.x86_64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the agent with ddpg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "from ddpg_agent import Agent\n",
    "import torch\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initialising ReplayBuffer\n",
      "Episode 1\tScore: 0.62\tAverage Score: 0.62\n",
      "Episode 2\tScore: 1.15\tAverage Score: 0.89\n",
      "Episode 3\tScore: 1.65\tAverage Score: 1.14\n",
      "Episode 4\tScore: 2.67\tAverage Score: 1.52\n",
      "Episode 5\tScore: 3.15\tAverage Score: 1.85\n",
      "Episode 6\tScore: 4.79\tAverage Score: 2.34\n",
      "Episode 7\tScore: 4.67\tAverage Score: 2.67\n",
      "Episode 8\tScore: 5.94\tAverage Score: 3.08\n",
      "Episode 9\tScore: 7.32\tAverage Score: 3.55\n",
      "Episode 10\tScore: 7.51\tAverage Score: 3.95\n",
      "Episode 11\tScore: 7.69\tAverage Score: 4.29\n",
      "Episode 12\tScore: 7.59\tAverage Score: 4.56\n",
      "Episode 13\tScore: 7.59\tAverage Score: 4.80\n",
      "Episode 14\tScore: 10.21\tAverage Score: 5.18\n",
      "Episode 15\tScore: 10.33\tAverage Score: 5.53\n",
      "Episode 16\tScore: 9.10\tAverage Score: 5.75\n",
      "Episode 17\tScore: 10.80\tAverage Score: 6.05\n",
      "Episode 18\tScore: 11.19\tAverage Score: 6.33\n",
      "Episode 19\tScore: 11.31\tAverage Score: 6.59\n",
      "Episode 20\tScore: 12.84\tAverage Score: 6.91\n",
      "Episode 21\tScore: 15.36\tAverage Score: 7.31\n",
      "Episode 22\tScore: 13.97\tAverage Score: 7.61\n",
      "Episode 23\tScore: 13.02\tAverage Score: 7.85\n",
      "Episode 24\tScore: 13.72\tAverage Score: 8.09\n",
      "Episode 25\tScore: 13.73\tAverage Score: 8.32\n",
      "Episode 26\tScore: 12.38\tAverage Score: 8.47\n",
      "Episode 27\tScore: 13.32\tAverage Score: 8.65\n",
      "Episode 28\tScore: 13.52\tAverage Score: 8.83\n",
      "Episode 29\tScore: 12.89\tAverage Score: 8.97\n",
      "Episode 30\tScore: 14.34\tAverage Score: 9.15\n",
      "Episode 31\tScore: 13.63\tAverage Score: 9.29\n",
      "Episode 32\tScore: 13.18\tAverage Score: 9.41\n",
      "Episode 33\tScore: 15.57\tAverage Score: 9.60\n",
      "Episode 34\tScore: 14.21\tAverage Score: 9.73\n",
      "Episode 35\tScore: 12.89\tAverage Score: 9.82\n",
      "Episode 36\tScore: 14.72\tAverage Score: 9.96\n",
      "Episode 37\tScore: 14.35\tAverage Score: 10.08\n",
      "Episode 38\tScore: 12.84\tAverage Score: 10.15\n",
      "Episode 39\tScore: 13.34\tAverage Score: 10.23\n",
      "Episode 40\tScore: 14.50\tAverage Score: 10.34\n",
      "Episode 41\tScore: 15.63\tAverage Score: 10.47\n",
      "Episode 42\tScore: 13.29\tAverage Score: 10.54\n",
      "Episode 43\tScore: 15.71\tAverage Score: 10.66\n",
      "Episode 44\tScore: 13.86\tAverage Score: 10.73\n",
      "Episode 45\tScore: 16.11\tAverage Score: 10.85\n",
      "Episode 46\tScore: 15.19\tAverage Score: 10.94\n",
      "Episode 47\tScore: 16.71\tAverage Score: 11.07\n",
      "Episode 48\tScore: 16.71\tAverage Score: 11.18\n",
      "Episode 49\tScore: 14.89\tAverage Score: 11.26\n",
      "Episode 50\tScore: 16.78\tAverage Score: 11.37\n",
      "Episode 51\tScore: 16.62\tAverage Score: 11.47\n",
      "Episode 52\tScore: 17.72\tAverage Score: 11.59\n",
      "Episode 53\tScore: 15.16\tAverage Score: 11.66\n",
      "Episode 54\tScore: 16.35\tAverage Score: 11.75\n",
      "Episode 55\tScore: 16.25\tAverage Score: 11.83\n",
      "Episode 56\tScore: 16.45\tAverage Score: 11.91\n",
      "Episode 57\tScore: 16.66\tAverage Score: 11.99\n",
      "Episode 58\tScore: 16.78\tAverage Score: 12.08\n",
      "Episode 59\tScore: 16.95\tAverage Score: 12.16\n",
      "Episode 60\tScore: 18.38\tAverage Score: 12.26\n",
      "Episode 61\tScore: 18.86\tAverage Score: 12.37\n",
      "Episode 62\tScore: 17.61\tAverage Score: 12.46\n",
      "Episode 63\tScore: 17.75\tAverage Score: 12.54\n",
      "Episode 64\tScore: 17.58\tAverage Score: 12.62\n",
      "Episode 65\tScore: 18.33\tAverage Score: 12.71\n",
      "Episode 66\tScore: 17.81\tAverage Score: 12.78\n",
      "Episode 67\tScore: 17.98\tAverage Score: 12.86\n",
      "Episode 68\tScore: 18.35\tAverage Score: 12.94\n",
      "Episode 69\tScore: 19.33\tAverage Score: 13.03\n",
      "Episode 70\tScore: 19.07\tAverage Score: 13.12\n",
      "Episode 71\tScore: 19.01\tAverage Score: 13.20\n",
      "Episode 72\tScore: 20.27\tAverage Score: 13.30\n",
      "Episode 73\tScore: 20.13\tAverage Score: 13.40\n",
      "Episode 74\tScore: 19.93\tAverage Score: 13.48\n",
      "Episode 75\tScore: 20.35\tAverage Score: 13.58\n",
      "Episode 76\tScore: 21.23\tAverage Score: 13.68\n",
      "Episode 77\tScore: 23.03\tAverage Score: 13.80\n",
      "Episode 78\tScore: 22.69\tAverage Score: 13.91\n",
      "Episode 79\tScore: 21.71\tAverage Score: 14.01\n",
      "Episode 80\tScore: 21.72\tAverage Score: 14.11\n",
      "Episode 81\tScore: 21.44\tAverage Score: 14.20\n",
      "Episode 82\tScore: 23.24\tAverage Score: 14.31\n",
      "Episode 83\tScore: 22.06\tAverage Score: 14.40\n",
      "Episode 84\tScore: 23.41\tAverage Score: 14.51\n",
      "Episode 85\tScore: 24.56\tAverage Score: 14.63\n",
      "Episode 86\tScore: 26.02\tAverage Score: 14.76\n",
      "Episode 87\tScore: 26.57\tAverage Score: 14.89\n",
      "Episode 88\tScore: 28.28\tAverage Score: 15.05\n",
      "Episode 89\tScore: 28.58\tAverage Score: 15.20\n",
      "Episode 90\tScore: 27.42\tAverage Score: 15.33\n",
      "Episode 91\tScore: 28.78\tAverage Score: 15.48\n",
      "Episode 92\tScore: 29.69\tAverage Score: 15.64\n",
      "Episode 93\tScore: 28.14\tAverage Score: 15.77\n",
      "Episode 94\tScore: 28.79\tAverage Score: 15.91\n",
      "Episode 95\tScore: 29.16\tAverage Score: 16.05\n",
      "Episode 96\tScore: 27.99\tAverage Score: 16.17\n",
      "Episode 97\tScore: 29.06\tAverage Score: 16.31\n",
      "Episode 98\tScore: 30.04\tAverage Score: 16.45\n",
      "Episode 99\tScore: 27.94\tAverage Score: 16.56\n",
      "Episode 100\tScore: 29.21\tAverage Score: 16.69\n",
      "Episode 100\tAverage Score: 16.69\n",
      "Episode 101\tScore: 29.42\tAverage Score: 16.98\n",
      "Episode 102\tScore: 31.15\tAverage Score: 17.28\n",
      "Episode 103\tScore: 30.16\tAverage Score: 17.56\n",
      "Episode 104\tScore: 28.55\tAverage Score: 17.82\n",
      "Episode 105\tScore: 30.66\tAverage Score: 18.10\n",
      "Episode 106\tScore: 30.55\tAverage Score: 18.35\n",
      "Episode 107\tScore: 31.00\tAverage Score: 18.62\n",
      "Episode 108\tScore: 29.18\tAverage Score: 18.85\n",
      "Episode 109\tScore: 25.96\tAverage Score: 19.04\n",
      "Episode 110\tScore: 30.22\tAverage Score: 19.26\n",
      "Episode 111\tScore: 27.49\tAverage Score: 19.46\n",
      "Episode 112\tScore: 28.50\tAverage Score: 19.67\n",
      "Episode 113\tScore: 30.61\tAverage Score: 19.90\n",
      "Episode 114\tScore: 29.58\tAverage Score: 20.09\n",
      "Episode 115\tScore: 29.16\tAverage Score: 20.28\n",
      "Episode 116\tScore: 31.46\tAverage Score: 20.51\n",
      "Episode 117\tScore: 30.87\tAverage Score: 20.71\n",
      "Episode 118\tScore: 31.02\tAverage Score: 20.90\n",
      "Episode 119\tScore: 31.01\tAverage Score: 21.10\n",
      "Episode 120\tScore: 30.40\tAverage Score: 21.28\n",
      "Episode 121\tScore: 29.58\tAverage Score: 21.42\n",
      "Episode 122\tScore: 30.15\tAverage Score: 21.58\n",
      "Episode 123\tScore: 31.05\tAverage Score: 21.76\n",
      "Episode 124\tScore: 29.22\tAverage Score: 21.92\n",
      "Episode 125\tScore: 29.72\tAverage Score: 22.08\n",
      "Episode 126\tScore: 30.09\tAverage Score: 22.25\n",
      "Episode 127\tScore: 30.57\tAverage Score: 22.43\n",
      "Episode 128\tScore: 29.70\tAverage Score: 22.59\n",
      "Episode 129\tScore: 30.23\tAverage Score: 22.76\n",
      "Episode 130\tScore: 29.85\tAverage Score: 22.92\n",
      "Episode 131\tScore: 31.76\tAverage Score: 23.10\n",
      "Episode 132\tScore: 31.95\tAverage Score: 23.29\n",
      "Episode 133\tScore: 30.62\tAverage Score: 23.44\n",
      "Episode 134\tScore: 32.45\tAverage Score: 23.62\n",
      "Episode 135\tScore: 31.48\tAverage Score: 23.80\n",
      "Episode 136\tScore: 31.11\tAverage Score: 23.97\n",
      "Episode 137\tScore: 30.25\tAverage Score: 24.13\n",
      "Episode 138\tScore: 30.10\tAverage Score: 24.30\n",
      "Episode 139\tScore: 29.61\tAverage Score: 24.46\n",
      "Episode 140\tScore: 31.91\tAverage Score: 24.64\n",
      "Episode 141\tScore: 30.18\tAverage Score: 24.78\n",
      "Episode 142\tScore: 29.81\tAverage Score: 24.95\n",
      "Episode 143\tScore: 31.83\tAverage Score: 25.11\n",
      "Episode 144\tScore: 32.32\tAverage Score: 25.29\n",
      "Episode 145\tScore: 31.65\tAverage Score: 25.45\n",
      "Episode 146\tScore: 33.01\tAverage Score: 25.63\n",
      "Episode 147\tScore: 31.36\tAverage Score: 25.77\n",
      "Episode 148\tScore: 31.92\tAverage Score: 25.93\n",
      "Episode 149\tScore: 32.31\tAverage Score: 26.10\n",
      "Episode 150\tScore: 33.66\tAverage Score: 26.27\n",
      "Episode 151\tScore: 33.99\tAverage Score: 26.44\n",
      "Episode 152\tScore: 32.22\tAverage Score: 26.59\n",
      "Episode 153\tScore: 32.43\tAverage Score: 26.76\n",
      "Episode 154\tScore: 33.18\tAverage Score: 26.93\n",
      "Episode 155\tScore: 32.26\tAverage Score: 27.09\n",
      "Episode 156\tScore: 31.38\tAverage Score: 27.24\n",
      "Episode 157\tScore: 34.03\tAverage Score: 27.41\n",
      "Episode 158\tScore: 33.15\tAverage Score: 27.57\n",
      "Episode 159\tScore: 31.83\tAverage Score: 27.72\n",
      "Episode 160\tScore: 33.02\tAverage Score: 27.87\n",
      "Episode 161\tScore: 32.85\tAverage Score: 28.01\n",
      "Episode 162\tScore: 31.70\tAverage Score: 28.15\n",
      "Episode 163\tScore: 31.64\tAverage Score: 28.29\n",
      "Episode 164\tScore: 32.76\tAverage Score: 28.44\n",
      "Episode 165\tScore: 33.48\tAverage Score: 28.59\n",
      "Episode 166\tScore: 31.66\tAverage Score: 28.73\n",
      "Episode 167\tScore: 30.86\tAverage Score: 28.86\n",
      "Episode 168\tScore: 32.48\tAverage Score: 29.00\n",
      "Episode 169\tScore: 31.84\tAverage Score: 29.13\n",
      "Episode 170\tScore: 31.71\tAverage Score: 29.25\n",
      "Episode 171\tScore: 31.34\tAverage Score: 29.38\n",
      "Episode 172\tScore: 30.41\tAverage Score: 29.48\n",
      "Episode 173\tScore: 30.23\tAverage Score: 29.58\n",
      "Episode 174\tScore: 29.67\tAverage Score: 29.68\n",
      "Episode 175\tScore: 31.19\tAverage Score: 29.78\n",
      "Episode 176\tScore: 30.51\tAverage Score: 29.88\n",
      "Episode 177\tScore: 30.17\tAverage Score: 29.95\n",
      "Episode 178\tScore: 31.49\tAverage Score: 30.04\n",
      "\n",
      "Environment solved in 78 episodes!\tAverage Score: 30.04\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4XGeV+PHvmVHvvTfbsi33GidOc3qvhCWEbAhtA2xYYAMsZSkJsJQsS/2FEggktAQCaUC6Y+IU27EcN7lKliWrd416m3l/f9w7I8mWLNnxaCTN+TyPHo3uzNU9Gklz5m3nFWMMSimlgpcj0AEopZQKLE0ESikV5DQRKKVUkNNEoJRSQU4TgVJKBTlNBEopFeQ0ESilVJDTRKCUUkFOE4FSSgW5kEAHMBkpKSmmoKAg0GEopdSMsmPHjmZjTOpEj/NbIhCRCGAzEG5f5y/GmK+JyMPABsBlP/QDxphdJ/teBQUFFBcX+ytUpZSalUSkcjKP82eLoB+4xBjTJSKhwOsi8px93+eMMX/x47WVUkpNkt8SgbGq2XXZX4baH1rhTimlphm/DhaLiFNEdgGNwEvGmG32Xf8jIntE5AciEj7OuXeJSLGIFDc1NfkzTKWUCmp+TQTGGLcxZiWQA6wTkaXAF4Ei4CwgCfj8OOc+aIxZa4xZm5o64ViHUkqp0zQl00eNMe3AJuAqY0ydsfQDvwHWTUUMSimlxua3RCAiqSKSYN+OBC4HDopIpn1MgJuAEn/FoJRSamL+nDWUCTwiIk6shPNnY8zfReQVEUkFBNgFfMyPMSillJqAP2cN7QFWjXH8En9dUymljtfRN0hPv5uM+IiTPq5nYIi/7a7l3WtycTpkiqKbHrTEhFJq1mrvGeDmB97gXx/aNuFjX9hXz+f/upd/HmqcgsimF00ESqlZqW/Qzb/9tpgjTd0caeqid8B90sfXtvcB8I+9dVMR3rSiiUApNSs9vauG7RVtXLs8E2PgSFPXSR9f77ISwUv7G+gfOnnSGKl3wE1te+87ijXQNBEopWalHZVtJEWH8elL5wNQ2th50sfXufpwCHT2DfF6afOkr/Plp0q48YE3sIopzEyaCJRSs9LOY+2syImnICWaEIdQ2jDcImjo6OO547qA6jt6OXdeCnERIZPuHmrrHuBve2pp6uynqbP/jMY/lTQRKKVmLLfH8MTb1Qy5PaOOd/YNUtbUxaq8REKdDuakRFPaOJwIvvPcQT7+h7c51tLjO1bv6iM3KYrLFqez6WAjxhjcHsNv3jg6brfSX9+uZmDIunZZ48m7nqYzTQRKqRlr44EG7vnzbjYdGl2PbE+1C2NgZW4CAPPTY3wv1K7eQZ613/G/sK8egP4hN81dA2TGR3D2nCTaegapaOnhzSPN3Pe3/Vz1w8388OXDo65hjOGP244xJyUagLIJxiCmM00ESqkZa1dVOwAH6zpGHd95rA2AFXYiKEyLpbKlm75BN8/sqqF/yENydBjPlVgJobHD6tbJiI9gVV4iAG9XtrGtvBWnQ7hoYRo/fLmUo83dvmtsr2ijvLmbuy8uJCY8RFsESik1GZ19g6c0I2civkTQ0HnC8Xmp0cRHhgIwPy0Gj4Hypm7+VFzFosw4PnBuAW8fa6fe1eeb9ZMZH0Fhagyx4SHsrGpj29EWlmbH8/mrigDYWt7iu8bmw004HcKVS9KZlxYzKhH0Dbp9yWgm0ESglJoyt/zsTe59Zv87+h6VLd0cqOvA7THsqbY2OhzZIjDGsPNYOytzE33H5qfHAPCtZw9QUtPBrWtzuHpZBgAv7q+nvsOaOpoZH4nDIazMS2DLkRZ2V7k4e04S81KjSY0NZ8uR4UTwxpFmlufEExsRSmFqzKhxhEffOsbNP32TPdXt7+hnnSqaCJRSU6K7f4jDDV08X1KH23N6Uy37Bt3c8dBb/OuvtnGgroOu/iGyEyKpaOmhb9BqaTyzu5aW7gHOnpPkO29OSjROh/B6WTM3rszivevyKEyLpTAthuf21lNnryHwlqFYlZvAkaZuBtwezp6ThIiwfm4yW8pbMMbQ2TfInmoX581LAaAwLYaGjn46+gYBKKmxEtMvXzt6ek/WFNNEoJTyG4/H0Gi/2y5vsvrX23oG2VV1et0mv3i1nGOtPbR0D/CNv1sti/eszcXtMZQ1dnG0uZsvPbGXtfmJvGt1tu+88BAnX71uMT+5bRU/eu8qIkKdAFy1JINtR1vYV9tBbEQIMeFW+TXvOIEIrC2wEsr6eck0dfZzpKmbt4624vYYzp2XDFiJAOCI3T10sN5KBM/uraOqdXhm0nSliUAp5ReVLd3c/qttnPPtjZQ1do7qOnn5wMnr+fx+ayXv+fmWUceqWnv46T/LuHZZJgvTY9l2tJXYiBBfF8/B+k7u+fMuQkMc/Pi2VYQ4R7+83XluAdevyBp17KqlGXgMPF9SR+aIonTe2UaLMuJ84wzr51ov+lvLW3jzSAthIQ5W51sJw5sIyhq7GHJ7KG3o4oYVWQjw/ZcO0zMwNKnnLFA0ESilzri+QTc3//RNdle34zGwtbyVssYunA5hbX4ir0yQCLYcaeGtilZauwd8x54vqad/yMMXryniA+cVALAiJ4G5KdGEhTh4cPMRdh5r50tXLyIrIXJScS7JiiMnMZJBtyEjfvicxOgwLluUxs2rhlsV+clRZMZH8MibFTy3t461+Ym+lkVuYiRhTgdlTVarZMDt4eKiVO5Yn8+TO2tY/+1XTljANp1oIlAqyFS2dPt9FezR5m5auwf41s3LSIkJY+exdo40dZGfFMVVSzM41NB50i6T6jbrvtIRs4HKm7tJjAolJzGKm1Zmk50QyYYFqYQ4HcxPi+FwQxdzU6NHdQlNRES4eqnVosiMG12m+ld3nsW/XTh31GM/euFcegbc1Lr6uGxRuu++EKeDosxYtpa3cqDeirkoI46vXb+Ev358PQXJUfznn3dRUuOadGxt3QN857mDlE/B+gRNBEoFmY/+bgcf+W2xX2vjeMcDCtNiWJmbyM6qNsoau5iXFsPli9NxiDWDZ7wYqtqs6ZyHR0zJrGzppsBevBUZ5mTzf13se6FemBELwD2XLzihS2giVy3NBJhwvwKAD5w3hze+cAkl913JB+1Widf1y7PYXdXOs3vqCHEI81Kt7qI1+Un86s6zSIgM46O/24Grd3BSce2sauPnrx6ZktIVmgiUmmbeLGseVfrgTPJ4DOXN3eyuamfzKRRWO1Xed7FzU6NZlZdAeVM3R5u7KUyLIT85mi9cXcRzJfU8uLn8hHO7+4d8XUIjWwQVzd3MSY72fT1y85h/WZPL+9fnc439on4qVuUm8MlL53PDyqyJH2yLCQ/B2m132I0rs3AIPL+vnsK0GMJChl9eU2PD+dF7V1LT3jtmF1FVaw9fe7rEN/MJ4O3KdpwOYVlO/Cn/TKdKE4FS04jHY7jrdzv4+t/Hn2vf1NnP/7146LQWZjV19ftq4/xkYynP7K7l47/fwSf++DZ/3HZszHPufWYfD2wqO6XrlDd3kxkfQVRYCKvyrIHXIY/xvUv+twvmcs2yDO5/4RAN9qwir5oRJZ0P24mgb9DqjskfkQhGWj8vma/fuBTHaews5nAI91y+wBfb6UqLi+C8Qms6aZHdQhlp3ZwkchIjeXF/wwn3Pb2rhke2VPLMrlrfsZ1VbSzKjCUqzJ87Cls0ESg1jdS099LVP8QbZc2j3h16GWP40pN7+ckrZRRXnPoUTG/f+8ULUymubOOTj+5kd1U7bx1t5atPl/ju93L1DPL7rZX8bXftWN9uXOVNVn89wPKcBLyvz97ZNSLCPZcvxO0xvro/x8e4ID3GVzG00m4hFaREnVIcU+2W1TkAFGXGnXCfiHDF4gxeL2umu3/0LKJ9tdZ009+8WeErdre7ysWqEYvi/EkTgVLTiLdmfu+ge1Q5gy1HWvjd1koe217FS/Y7yok2WvE+ZuTOXNV23/tnrljIHefk86P3ruT1z1/Ck3efB3BCV83LBxoYsruTxloENlYfvzGG8qZu5qZYL/ox4SEsSLfeIXuTA1hJoSgjlr/vGZ0IqlqtGC8uSqOle4CWrn5fjR9vgbfp6qqlGdx+dh7XLhu7i+qKJekMDHnYfHh0kbx9tR1Ehzk5UNfBtqPWDKuu/iFfa8rf/JYIRCRCRN4Skd0isk9E7rOPzxGRbSJSJiJ/EpEwf8Wg1Exz2H4HHBbiYNPB4SmW33r2AF95qoQvPrGXpdlxxE6iyNnAkIfrfvw6P9w4XDXTmwjmpcbwjZuWcuPKbBwOITshkltW5/DY9ioaO4e7ap63q3MODHmoaRu9C1fvgJsrfrCZn2wsHXW8qaufzv6hUS/6F8xPoTAthriI0FGPvX5FFjsq20bt8FXd1kN4iMM3b7+0sYvKFisRjNc1NF1EhDr5n5uXkZs0dstlbX4iiVGho7qHOvoGOdbaw4fPn0NCVCgPbi7nbbtO0eq8md8i6AcuMcasAFYCV4nIOcB3gR8YYwqBNuDDfoxBqRnlcH0n6XHhXFCYwka7Jr4xhoqWbi5fnM5nLl/AT25bzdzjipyNpaa9l95B96g5+1WtPaTEhBMZ5jzh8R+/aB5Dbg+PvFkBWIO2mw83sSzbGqwsaxpd2O2BTWWUNnbx6FvH8NitBW9rABjV5/5fVxXxzCfOO+Ga1y233jn/Y0SroLqtl5zESN9MoNKGTipaukmKDvMt7pqpQpwOLl2UzrN76/jNG0cZcnvYb3cLrc5P5GMb5vHKwUb+78XDJEWHkZ88NV1hfksExuL9Sw21PwxwCfAX+/gjwE3+ikGpmeZwYycL0mO5uCiN6rZeyhq7aO8ZpLNviLPnJPEfl85nTko0hamjE4GrZ5AL79/E7qrhImcV9rvo0sYu3wCs90V2LAUp0VwwP5Wndtbi8Rj+eaiJ/iEPd19cCMCRxuESzOVNXTy4uZys+AhqXX3srGrnd1squOD+Tb5uj5EtglCnY8xBz/zkaFbkxPOjjaW+FbhWjFFkxEUQGx5CSU0HR5u7KZiiF0V/+9yVC1k3J4n7/rafz/1lj298YElWPB+9cC43rMiiuaufVbkJJ8xM8he/jhGIiFNEdgGNwEvAEaDdGOMdKakGJr/6Q6lZ4itPlfD9Fw+NOuax6+UsSI/lfHv2SXFlm+8FfWS3SGFaDI2dw0XOKlq6Odbaw47K4QHkkVNQvS/O1W094yYCsKZA1rT3suNYGw+/eZSMuAguX5xOSkyYL/EMDHn47OO7CQ9x8PuPnE1YiIPfbang/ucPUd3Wy0//eYSIUAdZ8ZNb3fvj21ZxfmEKP95YypefLKGqrYfcpEhrcHVJBo/vqGJ3lcu3hmCmS4+L4LcfWsddF87lyZ01PLmzmrTYcFJjwxER7n/3cm5elc17zsqdspj8mgiMMW5jzEogB1gHFE32XBG5S0SKRaS4qalp4hOUmiG6+od4bPuxE6YRVrX10DfoYUF6DHlJUUSHOTlY18ExewXuyHfExxc58y5Sqh8xFbOypYfIUCdZ8RG8eqgJj8dQ0947bv81wBVLMogIdfC1p/exvaKNuy8pxOkQ5qbG+Hbg+sbf9/P2sXa+fcsy5qbGsGFBKk/tqqVn0M2XrimyY42e9FTO/ORofn7HGu6+eB5P7KyhvWeQnEQrxvtuXMLc1Bh6B90UTPPxgVMhInzikkLiI0MpqelgSdbwLKOIUCc/uHUlVy7JmLJ4pmTWkDGmHdgErAcSRMTbRswBasY550FjzFpjzNrU1NSpCFOpd6RnYIgPP7x9wjICr5c2M+g2HGvtGTXrxjtQPD89FodDKMqM40B9JxXNViIY+QI+z+528b5Lb/cmAtdwIjjW2k1+chQbFqbyRlkzNe29DLrNSVsEMeEhXLYonf11HWQnRPKetdZ0yEJ7TOK5vXX8bmsld104l+uWWwuwvP387z0rl7sunMe3bl7Gxy+aN4lnbLS7Ly4kwy7z4I0xJjyEX9yxhnmp0ay3K33OFnERoXxsg/U8Lcny/6Kxk/HnrKFUEUmwb0cClwMHsBLCu+2H3Qk87a8YlJpK/zzUxMaDjb7tD8fzykGrJdAz4Kapa7h8gHfx1Hz73X5RRiwH6zqobOkmIy7CV+AMIC8pijCngyP2wOx4LYK8pCg2LEijs3+In716BMD3bns83kJr/3FJIeEh1jXnpcbg6h3kK0+XsDgzjv+6cqHv8VctzeBzVy7kc/ax952dx40rT73HNyoshK9ctxiHjF6QNS81ho2fuYizCpJOcvbMdOe5+dy8KpvrVpz6iugzyZ9L1jKBR0TEiZVw/myM+buI7AceE5FvAjuBh/wYg1JT5vkSa6qldxbIWDwewysHm0iODqOle4DKlh7SYiPYcqSFh9+sYE5KNLH2FMuizDj+sO0YW8tbTpg9EuJ0UJAS5WsRdNiJwLtK1+OxWhwXLUzlskVprMpL8K0czj1JiwDgkqI0nvz3c32lmGG4K6q5a4AH3rd6VD2f8BCnb0D5nbp2eSYXLLjihGmms1VUWAg/uHVloMPw66yhPcaYVcaY5caYpcaYr9vHy40x64wxhcaYfzHG+L+iklJ+1j/k5hV73v/+EdsmejyG771wiG324rCSWhfNXf3cfnYeYL1rL65o5fZfbSU2IoSf3r7ad+7iTOtdsVVa4cR38fNGbI/Y3mPV5ql39WGMoaGzj/4hD3nJ0YQ4HXz/PSuJCLX+3Scq0SwirMpLHDVjxdtKuXZ5JmfP9W8XTbAkgelEVxYrdQa8WdZCV/8Q5xUm09DRT7Pd5fPDjaX8v01l/Op1a8vCVw42IgK3n5OPQ+BYSzcvH2jE6RCeuvs8Fo0oTeBdjQtjL6TKTYqitr0XY4yva6h/yIOrd9BXkiHfHleYkxLNd29Zzq1rc0d1MU1WVkIkP//XNXzrpmWnfK6a/vxfzUipIPB8ST0x4SF85IK5vFHWwoG6DvoHPfx4YykRoQ7eOtqKx2N4rbSZ5dnxpMdFkGXvtVvn6mVpdvwJ74RjI0LJTYqkqrV3zBZBSkwY/UMeuvqHaO8ZLm1c5+rzTR0ded6NK7NPq+/e66qlUzeLRU0tbREodRqMMdS5hssibC5tYsOCVFbmWP3qJTUdfPMf+ynKiOVr1y/B1TvI9opWdlW1c8F8axZcfrLVx7+72sWacUoJLMqwWgj5SSe2CFJiwgGr397VO+gre1zf0UdlazdOh0x6py4V3DQRKHWKjDF88x8HWP/tVyiuaKWho486Vx+r8xNJjA4jKz6Ch988SkVLD3dfXMiFC6wX/h+8fBi3x3D+fGuxWF5SNPvrOhgY8rC2YOxEsDQ7HqdDyB+j6qY3ETR19uPqHfT14ze4+qho6SE7IZLQU9ykRQUn/StR6hR9/6XDPGT3+b96uIlddlmHlbnWXPDFWXE0dPSTnRDJ1UszyE6IJC8piq3lrUSFOX2FxEZ223g3QT/eh86fw58/un7MAdThFsFwIhCxuoa2H2311QhSaiKaCJQ6BT0DQzywqYwbVmSxIieereUt7K5qJ8QhvkVBi+0B3w+cW+CbZnnO3CT7c7KvC8e7UjgvKYq02LG3SYwJD2HNOEkiJdYq3Nvc1U97zyApMeEkR4fzz0ONNHb2c9FCXYipJkcTgVKnoKyxC4+Bq5dmcM68ZHZVtbOlvIWizFjfbJwrl2Zw0cJUbl03XCvmHHvK5QV2txBYXUNglSY+HcnR4TgEatv76B10Ex8ZSkZ8OLurrZXNGzQRqEnSRKDUKfCWgViQEcs5c5MZdBt2HmsftfhqSVY8D39w3ajunMsWp/OetTlcv2J4X9w5KdFkJ0RyxWnWlHE6hKToMN9agoSoUF+JhqXZceO2MpQ6nk4fVeoUHG7oJCzEQX5SFGmx4TgdgttjWJFz8p2k4iJCuf/dK0Ydiwxz8sYXLnlH8aTEhPsKz8VFhpJuJ4KLF6a9o++rgou2CJQ6BYcbOpmXGkOI00FsRChL7QHZkS2CqZQSE06lXZ00ISqMzHgrEVykiUCdAm0RKHUKDtd3sm7OcPGzKxan09TRN2o3rqmUEhPm20s4PjKU65Zn0TfoCVhiUjOTJgKlJqmjb5BaVx8LRlTG/PiGeXzkgjmTrr1/pnmnkAIkRIZSkBLNZ0dUBlVqMjQRKDVJpd6B4rThROBwCOGOU6/dc6akxA4ngpm+n68KHB0jUGqSvPsFLBzRIgi0kS2COE0E6jRpIlBqkg43dBIV5iR7GtXvSYmxFpXFRoTgDFD3lJr5NBEoNQmDbg+bDjayJCsuYOMBY0m1u4YSorQ1oE6fJgKlJuHx4moqWnq468JT34vXn1LtriEdH1DvhCYCpSbQN+jmRxsPszovgcsWTa/5+UnRYYhAQmRYoENRM5gmAqUm8PSuGho6+vnslQtHbd84HYQ4HSRGhWmLQL0jOn1UqQm8XdlOYlQo6/28V+/p+vRl8ykYYytLpSZLE4FSE9hb42Jpdvy0aw14vX99QaBDUDOc37qGRCRXRDaJyH4R2Scin7KP3ysiNSKyy/64xl8xKPVO9Q26OdzQqZu8qFnNny2CIeAzxpi3RSQW2CEiL9n3/cAY8z0/XlupM+JgfSdDHqOJQM1qfksExpg6oM6+3SkiB4Bsf11PKX/YW2Nt8rJUE4GaxaZk1pCIFACrgG32oU+IyB4R+bWInN72TEpNgZJqFwlRoeQkTp/VxEqdaX5PBCISA/wV+LQxpgP4GTAPWInVYvi/cc67S0SKRaS4qanJ32EqNaa9NS6WTeOBYqXOBL8mAhEJxUoCfzDGPAFgjGkwxriNMR7gl8C6sc41xjxojFlrjFmbmqp7r6qppwPFKlj4c9aQAA8BB4wx3x9xPHPEw24GSvwVg1LvREmNiyGPYfkE21AqNdP5c9bQecAdwF4R2WUf+xJwm4isBAxQAXzUjzEoddq2lrcAcPaIHcmUmo38OWvodWCsjtVn/XVNpc6kLeUtFGXEkhitdXzU7Ka1hpQaQ/+Qm+KKNtbPm55lJZQ6kzQRKDWG3VUu+oc8nDNN6wspdSZpIlBqDFuOtCCi4wMqOGgiUGoMW8tbWJQRR0KUjg+o2U8TgVJAW/cA3/j7fnoH3Ay5PeyqamedtgZUkNBEoILCjso2qtt6xr3/pQMNPPT6UTYdauRQQye9g25W5en6ARUcdD8CNesZY/jII9uJDHXy5N3nkR4XccJjjjR2AbCtvIW2ngEAVuZqIlDBQVsEatZr6uynrWeQWlcfH3p4Oz0DQyc8ptROBFvLW9l1rJ2k6DDykqKmOlSlAkITgZr1yuwX+Q+cW8C+2g6eeLtm3Mccaujk9bJmVuYmaKE5FTQ0EahZz/tu/2Mb5pGXFMUrBxtH3d836KaqrYfzC1MAqHP1abeQCiqaCNSsV9bYRWx4COlx4VxSlMYbZc30Drh995c3dWMM3LImm/AQ619CE4EKJpoI1KxX1thFYXoMIsJli9LpH/LwRlmz7/7Sxk4AFmXGsSbf2idphSYCFUQ0EahZr7Sxi8LUGADWzUkiOszJxhHdQ0cau3AIzEmJ5s5zC/jgeQXER4YGKlylppxOH1WzWnvPAM1d/cxPtxJBWIiDCxek8o89tbh6BzhnbjJlTV3kJ0cTHuLkyiUZXLkkI8BRKzW1NBGoWc07G2h+Wqzv2PvOzqO0sYu9NS6e3VtPqFPYsCAtUCEqFXDaNaRmNW8iKEyL8R27YH4qL9+zgVc/ezG3n53HoNv4WgxKBSNtEahZ7XBDFxGhDrITIk+4z+EQvnnTUs4qSNJy0yqoaSJQs9rm0iZW5SbicIy9OExEuGlV9hRHpdT0ol1DatY63NBJWWMX1yzTwV+lTkYTgZq1nt1bhwhcuVQTgVIno4lAzVrP7q3jrIIk0mJPrDaqlBrmt0QgIrkisklE9ovIPhH5lH08SUReEpFS+3Oiv2JQwau0oZPDDV1cuywz0KEoNe35s0UwBHzGGLMYOAe4W0QWA18ANhpj5gMb7a+VOm1/3HaMoq88x9f/tp/Gjj6MMXz97/uJDHVytY4PKDUhv80aMsbUAXX27U4ROQBkAzcCF9kPewT4J/B5f8WhZr+X9tfjEOGRLRU8vqOKS4vSeK20mW/etFS7hZSahCkZIxCRAmAVsA1It5MEQD2QPs45d4lIsYgUNzU1TUWYagbyeAw7Ktu4YUUWL9+zgUUZcTy1q5YLF6Ry+9l5gQ5PqRnB7+sIRCQG+CvwaWNMx8jNPowxRkTMWOcZYx4EHgRYu3btmI9Rqqypi46+IdbkJzInJZrH7jqHVw42clZBkm4so9Qk+TURiEgoVhL4gzHmCftwg4hkGmPqRCQTaBz/Oyh1ctsrWgE4qyAJsFYLX7Z4zEamUmoc/pw1JMBDwAFjzPdH3PUMcKd9+07gaX/FoGaffbUuni+p8329o6KNlJgw8pN1f2GlTtekE4GInC8iH7Rvp4rInAlOOQ+4A7hERHbZH9cA3wEuF5FS4DL7a6Um5StPlfCx37/Nt587gDGG4so21uQnajeQUu/ApLqGRORrwFpgIfAbIBT4PdaL/ZiMMa8D4/13XnpqYSoFLV397KxqJycxkl+8Ws4/9tRR3dbLHefkBzo0pWa0ybYIbgZuALoBjDG1QOxJz1DqJEpqXHz28d0Muj2TPuefh5owBn52+xq+e8sylufEsyInniuW6JiAUu/EZAeLB0bO8BGRaD/GpGa5IbeHzz6+m4P1ndy2Lpc1+UmTOu+Vg42kx4WzNDuOZTnx3HqWTg9V6kyYbIvgzyLyCyBBRP4NeBn4pf/CUrPZb7dUcrDe2jB+e0XbpM4ZGPKw+XATlxSl6XiAUmfYpBKBMeZ7wF+wpoIuBL5qjPmJPwNTs1NH3yA/eOkwGxakMiclmuIxEkFJjYtfbi7HmOHlI2+UNdPZP8QlRdoNpNSZNmHXkIg4gZeNMRcDL/k/JDWb7TzWTmf/EHddOJendtbw8oEGjDGj3uV/+7kDvFHWwvz0GC5amEZLVz9fenIvuUmRnF+YEsDolZqdJmwRGGPcgEdE4qcgHjXL7a1uB2BZTjxrCxJp6xnkSFO37/6q1h7eKGsB4FvPHqCte4BPPraTlu4Bfnb7GiLDnAFMu7DUAAAa50lEQVSJW6nZbLKDxV3AXhF5CXvmEIAx5pN+iUrNWnuqXcxNiSYuIpS19mrg4opWEqNCiYkI4fHiKkTgy9cu5ht/38/672ykf8jD/bcsZ2m2vhdRyh8mmwiesD+U8rnjoW2cV5jCxzbMm/Q5e6pdnD3XSgBzU6JJig7jey8e5otP7iUrPpL+ITcXzk/lQ+cVsK28ha7+If772kUsydIkoJS/TCoRGGMeEZEwYIF96JAxZtB/Yanprra9l9dKmwlxyKQTQWNHH/UdfSzPSQCsjeM3LEjlxX31fPDcObxR1kxNey+3rctDRHjw/Wv9+SMopWyTXVl8EdbeARVYq4VzReROY8xm/4WmprM3ypoBqHP1TfqcvTUuAJbnDL+7/84ty/jWzcuIDHMy5PZwoK6TpdlxZzZYpdRJTbZr6P+AK4wxhwBEZAHwKLDGX4Gp6e3NI9aAbm1776TP2VPtwiGwJGv4hT48ZHjwN8TpYFmOdgEpNdUmu6As1JsEAIwxh7HqDakgZIzhdbtF0NE3RHf/0KTO2VHZxvy0WKLC/L4NhlLqFEw2ERSLyK9E5CL745dAsT8DU4HXP+Rm0O3xLewaGPLwt9217K520dTZz/q5yQDUucZvFXgTwK2/2MrrZc1ctDB1SmJXSk3eZN+afRy4G/BOF30N+KlfIlLTwp+Lq/ivv+wB4LZ1eXz7Xct4rqSOTz22ixCHtfjrX9bmsKW8hdr2PgrTTqxBWFzRyr1/20dJTQeJUaF8513LeM/a3Cn9OZRSE5tsIggBfuTdYMZebRzut6hUwL1e2kxydBiZCRG8XmbtGV3a0IXTIazMTUBkeFew41sEg24PP95YygObyshKiOSbNy3lplXZxIRrl5BS09Fk/zM3Ym0i02V/HQm8CJzrj6BU4O2rdbEqL5Fl2fH8cONh+gbdHGnqIj8pir983Pq1D7o9iEBtex9vljXz3ecPcl5hCm+UNbO72sW71+TwtesXExuhw0lKTWeTTQQRxhhvEsAY0yUiujfgLPPKwQbmpMSQHhdOeXM31y3PYl5aNMbAkaYuypu6mZs6XIE81OkgNSacOlcvf327hv11HZTUdhATHsJPb1/NNcsyA/jTKKUma7KJoFtEVhtj3gYQkbXA5OcNqmnPGMMn/riT9XOT+feLCzHGmuaZZ+8FXNrQxdGWbjYcN9ibmRBJnauPo83dXFKUxndvWU6o00G0dgMpNWNM9r/108DjIlJrf50J3OqfkNRU8ngMDofQ1NVPz4Cb18qaWTfH6vtfkh1PSkwYDoFXDzcxMORhbsroPYmy4iN480gLrt5BPnTeHBKiwgLxYyil3oGTTh8VkbNEJMMYsx0oAv4EDALPA0enID41jq7+ISpbuid+4HH+truWVw9bg79vljWz9N4XaOrsp6rVauANDHn4zRsVxEeGkhUfQXiIk7ykKF452AjAvLSYUd8vMz4SV69VbcRbQ0gpNbNMtI7gF8CAfXs98CXgAaANePBkJ4rIr0WkUURKRhy7V0RqRGSX/XHNO4g9qP3klVKu/8nrDJ3Cnr8ej+GrT5fwwCtlAOyqbqdnwM2+WhfVbT0AOATqO/pYkhXn2yOgMC3G92J/QosgIQKAuIgQijK0NIRSM9FEicBpjGm1b98KPGiM+asx5itA4QTnPgxcNcbxHxhjVtofz55auDNTnauXFfe9yK6q9jP2PY80dtPRN8Shhs5Jn3OooZO2nkFq7LIQNW3W5yNN3VS1WongqqUZwOgyEN5WQHxkKEnRo7t+MuMjAWsqqdOhW0gqNRNNmAhExDuOcCnwyoj7Tjq+YBekaz3ZY4LFwbpOXL2DvHqo6Yx9T++L+akkly12faA6Vy+Dbo/vexxp6qKqtZeUmDBuXJkNMKr2f2GqlQjmpkafsF9wpt0i0G4hpWauiRLBo8CrIvI01iyh1wBEpBBwneY1PyEie+yuo8TT/B4zSq294GpP9ZlrEXi7cnYeO4VEUG4lAo+Belefr2DckcYuqtt7yEmM4rJF6XzvX1b4WgZgdQ0BzEuNOeF7LsuO5+6L53HL6pzT/lmUUoF10kRgjPkf4DNY3Tznm+HdxB3Af5zG9X4GzANWAnVYVU3HJCJ3iUixiBQ3NZ25d9KB4H3B3V3dPmpD9tPV0TdIZ59V6G1ki2DI7aGieewBZLfHsK28hdwkqyunqq3nuK6hXnKTonA6hHevyRlVFbQwLYbwEAeLM08cAwh1OvjclUUkx+hCc6VmqsnsWbzVGPOkMWbkFpWHvWsKToUxpsEY4zbGeIBfAutO8tgHjTFrjTFrU1NndqGy2narZn9z1wC1p1C/fzzeF/CijFjKGrt8A7lP7Kzhou/9kxf21Z9wzoG6Djr6hnzv3PfXdtA94CY9Lpzmrn6q23rITYwc83qxEaG89J8b+Ndz8t9x7Eqp6Wey1UfPCBEZudT0ZqBkvMfOJrXtvcTaC6x2n4EBY28iuG659XR6u5z213YA8Nk/7z6hZeAdH7hldQ4isLXcGr65cL6VZD0GcpPGXyyelxxFWMiU/rkopaaI3/6zReRRYAuwUESqReTDwP0isldE9gAXA//pr+tPJ7WuXs6fn0KY08Hu0xgnuO9v+7j7j8MNMO8g79XLMhEZHic40tRFblIkTqfwxSf2jvoeW8pbmJsSTW5SFOmxEWyvsBLBBQuGW1u5iVo1RKlg5Lc6AMaY28Y4/JC/rjddeTyGelcf1y7LYlFm7Gm1CF4oqafW1cd/XtZFYVoMNe29hIU4mJMcTWFqjG+coKyxi3PmJpMaG87Db1Qw6PYQ6nQw5Pbw1tFWbliZBUBOYiTFlW0AnDM3iTCngwG3xzd+oJQKLtrW97Pmrn4G3YbshAhW5Cawt9qF2zP5AeOWrn7fuMIftlUCVtdQdkIkDrsk9M5jbXT1D1Hn6qMwLYYlWXEMuD2UNVp1AktqO+jqH/JtJJNjjwVEhFpF4wpSonAIZCVoIlAqGGki8DPvi3hmfCRnz0mme8DNDvvd+I7KNp7eVcNL+xvGnU3k3fA9LymKv+6opnfATXW7lQgAVuUl0tYzyCZvCYjUaJZkWWsA9tljBt7xgXN8icDqAspOiEREKMqIIy8pilCn/jkoFYy0RKSfeaeOZiVEkpsUSahTePlAA2mx4bz752/iff3/68fPZU3+8LIKV+8g8ZGh7K22EsG9NyzmQw8X88TOamraerm0KA2AlbkJAPxlRzVgTfWckxJNZKiTkhprT4At5S3MT4shNdaa4plttwi8LYAvX7fINx1VKRV89C2gnw0ngghiI0I5Z24yL+9v4PdbK3GK8MiHrBm0O49ZrYSegSG+9nQJK+57kad21rC3xsXclGguXpjGmvxE/veFQzR39ftezBekxxAV5uS10iZCHEJ+cjROh7AoM5b9tR0Muj0UV7Syfl6yLyZv15D3c1psxJiLxZRSwUETgZ/VtvcRFeYkPtLapeuKxemUN3fz+22VXLkkgw0LUslOiPTN/Lnz12/xyJZKEqJC+ckrpeypdrEsJx4R4TvvWkZ3v/XO3fsiHuJ0sCw7Ho+xpnh6u3eWZMWzv66DzYeb6Blw+8YHrHOHu4aUUkoTgZ/VtveSGR/hq9Fz6aJ0APoGPdyx3lqgtTIvgV1V7VQ0d7O9oo3PX1XEfTcs4UhTN/UdfSyz6/7MT4/l7outWn/5ycNVQFflWV1KhSPe1S/JiqOrf4gvPrGX/OQoLlmU5rsvLymKD503h6t1BzGlFDpG4Hd1rt5Rs3GyEiJZkZtA/6Cbs+0NYFblJvCPPXU8+tYxwFoolhkfwf3PH6KmvXdUAbhPXFzIWQVJrM5L8B3zjhMUjtgrwHtOY2c/v7hjzaiSEU6H8NXrF/vhp1VKzUSaCPysuq2Xy4+r0fPL969BEF8rYZX9ov7wmxUUZcT6Vvh+6tL5fPu5A6MSQYjTwXmFKaO+31kFicSGh/h2FgOYnx5DWIiDtfmJXLE43S8/m1JqdtBE4EeNnX20dA+wID121PG02IhRXy/JiifEIfQPebh8xIv2e87K5d1rcnBMUOc/OSacPfdeMapEdHiIkz9+5GzmpJxYOloppUbSMQI/8s7jX5x18p27IkKdvsdcfty794mSgNdYL/ZrC5K0KqhSakLaIvCj/ZNMBAAbFqTSM+D2DQwrpdRU0UTgR/trO8hLiiIuInTCx95z+QI+del87cZRSk05TQR+tL+uY8zNXMYiIoQ4NQkopaaejhH4SVf/EEebu0dtAq+UUtORJgI/OVg3+fEBpZQKJE0EfuKdMeStBKqUUtOVjhGcYX2Dbh56/Si/3VJBSkw46XE6fVMpNb1pIjjDvvTkXp54u4Zz5yXz6csW6CwgpdS0p4ngDPrLjmqeeLuGT106n/+8fEGgw1FKqUnRMYIzpLt/iK8+XcLZc5L45KXzAx2OUkpNmiaCM+RQQyc9A24+csFcnJMsC6GUUtOB3xKBiPxaRBpFpGTEsSQReUlESu3PiSf7HtNdW/cAFc3dAByu7wRg4XEF5pRSarrzZ4vgYeCq4459AdhojJkPbLS/nnEG3R4+9/huzv7WRq7+0Wv0DAxxqKGTyFCnb+cwpZSaKfyWCIwxm4HW4w7fCDxi334EuMlf1/enreUtPL6jmhW58fQOutlV1c7hhk4WpMdMulqoUkpNF1M9RpBujKmzb9cD4+6YIiJ3iUixiBQ3NTVNTXSTVNfeB8DXrl+CCGw/2sbhhq4T9h1QSqmZIGCDxcYYA5iT3P+gMWatMWZtamrqFEY2sVpXLwAL0mNZmB7Li/vraersZ2GGJgKl1Mwz1YmgQUQyAezPjVN8/TOi3tVHSkw4YSEOzipI8pWT0BaBUmommupE8Axwp337TuDpKb7+GVHn6iMrwdpu8qwR+wRrIlBKzUT+nD76KLAFWCgi1SLyYeA7wOUiUgpcZn8949S5esmIsxNBgTUDNi4iROsKKaVmJL+VmDDG3DbOXZf665pTpc7Vx/q5yQBkxkeSkxhJZnyE1hVSSs1IWmvoFHX1D9HZN0RmwvB6gR+9dxXhIbpIWyk1M2kiOEX19oyhzPgI37E1+TN6gbRSKsjp29hTVOey1hB4xwiUUmqm00RwiryLybIStJSEUmp20ERwirwtgjSdIaSUmiU0EZyi+o5eUmLCCQ9xBjoUpZQ6IzQRnKLa9r5RA8VKKTXT6ayhCWyvaOWFknrOK0xhdX4ite29FKREBzospZQ6YzQRjMPVM8i3nzvAY9urAPjV60d9951XmBKosJRS6ozTRDCGTQcb+dxfdtPWM8hdF87l7osKKa5spbypG6dDuHpZRqBDVEqpM0YTwRi+9OReEqLCeORD61iSFQ/ApYvSuXRRgANTSik/0MHi49S7+qhz9fG+dXm+JKCUUrOZJoLj7K5uB2BFbkKAI1FKqamhieA4u6vaCXEIS7LiAh2KUkpNCU0Ex9ld3U5RZiwRobpgTCkVHDQRjODxGPZUuViRo91CSqngoYlghKMt3XT2D+n4gFIqqGgiGGF3lTVQvFITgVIqiGgiGGHnsXaiw5zMS40JdChKKTVlNBGMsKOyjVV5iTgduvewUip4BCQRiEiFiOwVkV0iUhyIGI7X1T/EwfoOVuu2k0qpIBPIEhMXG2OaA3j9UXYda8djYK0mAqVUkNGuIduOyjZEYGWeDhQrpYJLoBKBAV4UkR0icleAYhiluLKVhemxxEWEBjoUpZSaUoHqGjrfGFMjImnASyJy0BizeeQD7ARxF0BeXp7fAnlhXz0x4SHsOtbODSuz/HYdpZSargKSCIwxNfbnRhF5ElgHbD7uMQ8CDwKsXbvW+COO7v4hPvb7HRj7u68t0PEBpVTwmfJEICLRgMMY02nfvgL4+lTHAXCooRNj4H1n59E36OaSovRAhKGUUgEViBZBOvCkiHiv/0djzPMBiIODdZ0AfHzDPHKTogIRglJKBdyUJwJjTDmwYqqvO5ZD9R3EhIeQnRAZ6FCUUipggnr66IH6Thakx+DQlcRKqSAWtInAGMOh+k6KMnUDGqVUcAvaRFDf0Yerd5CijNhAh6KUUgEVtIngYL01ULwwXROBUiq4BW0iOGQngqIM7RpSSgW3oE0Ee2tcZMZHEB+lJSWUUsEtKBPB/toOnttbx+WLdQGZUkoFXSIwxvDVp0tIiArjnssXBDocpZQKuKBLBM+V1FNc2cYXrioiISos0OEopVTABV0ieHZvHamx4bx7TU6gQ1FKqWkhqBLBkNvDa6XNbFiQqquJlVLKFlSJYHd1O67eQS5amBroUJRSatoIqkTwz0NNOAQuKNREoJRSXkGXCFbnJeraAaWUGiFoEkFNey97a1zaLaSUUscJmkTwvRcOERbi4MaV2YEORSmlppWgSAS7q9p5cmcNHz5/ju5EppRSx5n1icAYwzf+vp+UmDD+/aJ5gQ5HKaWmnVmfCJ7da60kvufyhcRG6CCxUkodb1Yngr5BN99+7gBFGbHcelZuoMNRSqlpaVYngt+8UUF1Wy9fvnYxTl1JrJRSYwpIIhCRq0TkkIiUicgX/HWdtNhw/mVNDufPT/HXJZRSasYLmeoLiogTeAC4HKgGtovIM8aY/Wf6WresyeEWLS6nlFInFYgWwTqgzBhTbowZAB4DbgxAHEoppQhMIsgGqkZ8XW0fU0opFQDTdrBYRO4SkWIRKW5qagp0OEopNWsFIhHUACPncubYx0YxxjxojFlrjFmbmqr1gZRSyl8CkQi2A/NFZI6IhAHvBZ4JQBxKKaUIwKwhY8yQiHwCeAFwAr82xuyb6jiUUkpZpjwRABhjngWeDcS1lVJKjTZtB4uVUkpNDTHGBDqGCYlIE1B5mqenAM1nMBx/0lj9Q2P1D43VP85krPnGmAln28yIRPBOiEixMWZtoOOYDI3VPzRW/9BY/SMQsWrXkFJKBTlNBEopFeSCIRE8GOgAToHG6h8aq39orP4x5bHO+jECpZRSJxcMLQKllFInMasTwVRtgHM6RCRXRDaJyH4R2Scin7KP3ysiNSKyy/64JtCxAohIhYjstWMqto8lichLIlJqf06cBnEuHPHc7RKRDhH59HR5XkXk1yLSKCIlI46N+TyK5cf23+8eEVk9DWL9XxE5aMfzpIgk2McLRKR3xPP782kQ67i/cxH5ov28HhKRK6dBrH8aEWeFiOyyj0/N82qMmZUfWOUrjgBzgTBgN7A40HGNiC8TWG3fjgUOA4uBe4HPBjq+MeKtAFKOO3Y/8AX79heA7wY6zjH+BuqB/OnyvAIXAquBkomeR+Aa4DlAgHOAbdMg1iuAEPv2d0fEWjDycdPkeR3zd27/n+0GwoE59uuEM5CxHnf//wFfncrndTa3CKb1BjjGmDpjzNv27U7gADNvX4YbgUfs248ANwUwlrFcChwxxpzuYsQzzhizGWg97vB4z+ONwG+NZSuQICKZUxPp2LEaY140xgzZX27Fqh4ccOM8r+O5EXjMGNNvjDkKlGG9XkyJk8UqIgK8B3h0quKB2d01NGM2wBGRAmAVsM0+9Am76f3r6dDdYjPAiyKyQ0Tuso+lG2Pq7Nv1QHpgQhvXexn9DzUdn1cY/3mc7n/DH8JqsXjNEZGdIvKqiFwQqKCOM9bvfDo/rxcADcaY0hHH/P68zuZEMCOISAzwV+DTxpgO4GfAPGAlUIfVTJwOzjfGrAauBu4WkQtH3mmsduy0mYJmlzi/AXjcPjRdn9dRptvzOB4R+W9gCPiDfagOyDPGrALuAf4oInGBis82I37nx7mN0W9epuR5nc2JYFIb4ASSiIRiJYE/GGOeADDGNBhj3MYYD/BLprDJejLGmBr7cyPwJFZcDd6uCvtzY+AiPMHVwNvGmAaYvs+rbbzncVr+DYvIB4DrgNvtxIXdzdJi396B1e++IGBBctLf+XR9XkOAdwF/8h6bqud1NieCab0Bjt0X+BBwwBjz/RHHR/YB3wyUHH/uVBORaBGJ9d7GGjAswXo+77QfdifwdGAiHNOod1bT8XkdYbzn8Rng/fbsoXMA14gupIAQkauA/wJuMMb0jDieKiJO+/ZcYD5QHpgofTGN9zt/BniviISLyBysWN+a6vjGcBlw0BhT7T0wZc/rVI2UB+IDa9bFYaws+t+Bjue42M7H6gLYA+yyP64BfgfstY8/A2ROg1jnYs2y2A3s8z6XQDKwESgFXgaSAh2rHVc00ALEjzg2LZ5XrORUBwxi9U1/eLznEWu20AP23+9eYO00iLUMq3/d+zf7c/uxt9h/G7uAt4Hrp0Gs4/7Ogf+2n9dDwNWBjtU+/jDwseMeOyXPq64sVkqpIDebu4aUUkpNgiYCpZQKcpoIlFIqyGkiUEqpIKeJQCmlgpwmAjWriYhbRlcjPWkVWhH5mIi8/wxct0JEUk7jvCtF5D67IulzE5+h1DsXEugAlPKzXmPMysk+2BgzpeWTx3ABsMn+/HqAY1FBQlsEKijZ79jvF2uPhbdEpNA+fq+IfNa+/Umx9ovYIyKP2ceSROQp+9hWEVluH08WkRfF2lviV1iLwbzX+lf7GrtE5BfelaLHxXOrXYP+k8APsUoifFBEps1qeDV7aSJQs13kcV1Dt464z2WMWQb8P6wX3+N9AVhljFkOfMw+dh+w0z72JeC39vGvAa8bY5Zg1WLKAxCRRcCtwHl2y8QN3H78hYwxf8KqQFtix7TXvvYN7+SHV2oytGtIzXYn6xp6dMTnH4xx/x7gDyLyFPCUfex8rGX/GGNesVsCcVibjbzLPv4PEWmzH38psAbYbpWXIpLxi/MtYLiOTLSx9qlQyu80EahgZsa57XUt1gv89cB/i8iy07iGAI8YY7540gdZ23+mACEish/ItLuK/sMY89ppXFepSdOuIRXMbh3xecvIO0TEAeQaYzYBnwfigRjgNeyuHRG5CGg21j4Sm4H32cevBryboGwE3i0iafZ9SSKSf3wgxpi1wD+wds+6H6uw30pNAmoqaItAzXaR9jtrr+eNMd4ppIkisgfoxypbPZIT+L2IxGO9q/+xMaZdRO4Ffm2f18Nw+ej7gEdFZB/wJnAMwBizX0S+jLW7mwOr4uTdwFjbZ67GGiz+d+D7Y9yvlF9o9VEVlESkAqusc3OgY1Eq0LRrSCmlgpy2CJRSKshpi0AppYKcJgKllApymgiUUirIaSJQSqkgp4lAKaWCnCYCpZQKcv8f11xF4836T38AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def ddpg(n_episodes=300, max_t=1000):\n",
    "    \"\"\" Deep Deterministic Policy Gradients\n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "    \"\"\"\n",
    "    scores_window = deque(maxlen=100)\n",
    "    scores = np.zeros(num_agents)\n",
    "    scores_episode = []\n",
    "    \n",
    "    agents =[] \n",
    "    \n",
    "    for i in range(num_agents):\n",
    "        agents.append(Agent(state_size, action_size, random_seed=0))\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations\n",
    "        \n",
    "        for agent in agents:\n",
    "            agent.reset()\n",
    "            \n",
    "        scores = np.zeros(num_agents)\n",
    "            \n",
    "        for t in range(max_t):\n",
    "            #actions = [agents[i].act(states[i]) for i in range(num_agents)]\n",
    "            actions = np.array([agents[i].act(states[i]) for i in range(num_agents)])\n",
    "#             if t == 0:\n",
    "#                 print(\"actions\", actions)\n",
    "            env_info = env.step(actions)[brain_name]        # send the action to the environment\n",
    "            next_states = env_info.vector_observations     # get the next state\n",
    "            rewards = env_info.rewards                     # get the reward\n",
    "            dones = env_info.local_done        \n",
    "            \n",
    "            for i in range(num_agents):\n",
    "                agents[i].step(t,states[i], actions[i], rewards[i], next_states[i], dones[i]) \n",
    " \n",
    "            states = next_states\n",
    "            scores += rewards\n",
    "            if t % 20:\n",
    "                print('\\rTimestep {}\\tScore: {:.2f}\\tmin: {:.2f}\\tmax: {:.2f}'\n",
    "                      .format(t, np.mean(scores), np.min(scores), np.max(scores)), end=\"\") \n",
    "            if np.any(dones):\n",
    "                break \n",
    "        score = np.mean(scores)\n",
    "        scores_window.append(score)       # save most recent score\n",
    "        scores_episode.append(score)\n",
    "\n",
    "        print('\\rEpisode {}\\tScore: {:.2f}\\tAverage Score: {:.2f}'.format(i_episode, score, np.mean(scores_window)), end=\"\\n\")\n",
    "        if i_episode % 100 == 0:\n",
    "            print('\\rEpisode {}\\tAverage Score: {:.2f}'.format(i_episode, np.mean(scores_window)))\n",
    "        if np.mean(scores_window)>=30.0:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(Agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(Agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            break\n",
    "            \n",
    "    return scores_episode\n",
    "\n",
    "scores = ddpg()\n",
    "\n",
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}