Continuous_Control_v8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mYou are using pip version 10.0.1, however version 18.1 is available.\r\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\r\n"
     ]
    }
   ],
   "source": [
    "# when running in Udacity workspace\n",
    "!pip -q install ./python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from ddpg_agent import Agent\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import random\n",
    "import time\n",
    "import torch\n",
    "from unityagents import UnityEnvironment\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "## when running locally\n",
    "env = UnityEnvironment(file_name='Reacher20.app')\n",
    "\n",
    "## when running via AWS EC2\n",
    "# env = UnityEnvironment(file_name='Reacher_Linux_NoVis/Reacher.x86_64')\n",
    "\n",
    "## when running in Udacity workspace\n",
    "# env = UnityEnvironment(file_name='/data/Reacher_Linux_NoVis/Reacher.x86_64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 0.06299999859184027\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name]     # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents)                          # initialize the score (for each agent)\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# DDPG function\n",
    "\n",
    "def ddpg(n_episodes=500, max_t=1000, solved_score=30.0, consec_episodes=100, print_every=1, train_mode=True,\n",
    "         actor_path='actor_ckpt.pth', critic_path='critic_ckpt.pth'):\n",
    "    \"\"\"Deep Deterministic Policy Gradient (DDPG)\n",
    "    \n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int)      : maximum number of training episodes\n",
    "        max_t (int)           : maximum number of timesteps per episode\n",
    "        train_mode (bool)     : if 'True' set environment to training mode\n",
    "        solved_score (float)  : min avg score over consecutive episodes\n",
    "        consec_episodes (int) : number of consecutive episodes used to calculate score\n",
    "        print_every (int)     : interval to display results\n",
    "        actor_path (str)      : directory to store actor network weights\n",
    "        critic_path (str)     : directory to store critic network weights\n",
    "\n",
    "    \"\"\"\n",
    "    mean_scores = []                               # list of mean scores from each episode\n",
    "    min_scores = []                                # list of lowest scores from each episode\n",
    "    max_scores = []                                # list of highest scores from each episode\n",
    "    best_score = -np.inf\n",
    "    scores_window = deque(maxlen=consec_episodes)  # mean scores from most recent episodes\n",
    "    moving_avgs = []                               # list of moving averages\n",
    "    \n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=train_mode)[brain_name] # reset environment\n",
    "        states = env_info.vector_observations                   # get current state for each agent      \n",
    "        scores = np.zeros(num_agents)                           # initialize score for each agent\n",
    "        agent.reset()\n",
    "        start_time = time.time()\n",
    "        for t in range(max_t):\n",
    "            actions = agent.act(states, add_noise=True)         # select an action\n",
    "            env_info = env.step(actions)[brain_name]            # send actions to environment\n",
    "            next_states = env_info.vector_observations          # get next state\n",
    "            rewards = env_info.rewards                          # get reward\n",
    "            dones = env_info.local_done                         # see if episode has finished\n",
    "            # save experience to replay buffer, perform learning step at defined interval\n",
    "            for state, action, reward, next_state, done in zip(states, actions, rewards, next_states, dones):\n",
    "                agent.step(state, action, reward, next_state, done, t)             \n",
    "            states = next_states\n",
    "            scores += rewards        \n",
    "            if np.any(dones):                                   # exit loop when episode ends\n",
    "                break\n",
    "\n",
    "        duration = time.time() - start_time\n",
    "        min_scores.append(np.min(scores))             # save lowest score for a single agent\n",
    "        max_scores.append(np.max(scores))             # save highest score for a single agent        \n",
    "        mean_scores.append(np.mean(scores))           # save mean score for the episode\n",
    "        scores_window.append(mean_scores[-1])         # save mean score to window\n",
    "        moving_avgs.append(np.mean(scores_window))    # save moving average\n",
    "                \n",
    "        if i_episode % print_every == 0:\n",
    "            print('\\rEpisode {} ({} sec)  -- \\tMin: {:.1f}\\tMax: {:.1f}\\tMean: {:.1f}\\tMov. Avg: {:.1f}'.format(\\\n",
    "                  i_episode, round(duration), min_scores[-1], max_scores[-1], mean_scores[-1], moving_avgs[-1]))\n",
    "        \n",
    "        if train_mode and mean_scores[-1] > best_score:\n",
    "            torch.save(agent.actor_local.state_dict(), actor_path)\n",
    "            torch.save(agent.critic_local.state_dict(), critic_path)\n",
    "                  \n",
    "        if moving_avgs[-1] >= solved_score and i_episode >= consec_episodes:\n",
    "            print('\\nEnvironment SOLVED in {} episodes!\\tMoving Average ={:.1f} over last {} episodes'.format(\\\n",
    "                                    i_episode-consec_episodes, moving_avgs[-1], consec_episodes))            \n",
    "            if train_mode:\n",
    "                torch.save(agent.actor_local.state_dict(), actor_path)\n",
    "                torch.save(agent.critic_local.state_dict(), critic_path)  \n",
    "            break\n",
    "            \n",
    "    return mean_scores, moving_avgs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1 (89 sec)  -- \tMin: 0.0\tMax: 2.1\tMean: 0.6\tMov. Avg: 0.6\n",
      "Episode 2 (92 sec)  -- \tMin: 0.0\tMax: 2.3\tMean: 0.9\tMov. Avg: 0.8\n",
      "Episode 3 (93 sec)  -- \tMin: 0.7\tMax: 3.6\tMean: 2.0\tMov. Avg: 1.2\n",
      "Episode 4 (94 sec)  -- \tMin: 0.7\tMax: 4.8\tMean: 2.5\tMov. Avg: 1.5\n",
      "Episode 5 (94 sec)  -- \tMin: 1.1\tMax: 6.3\tMean: 2.9\tMov. Avg: 1.8\n",
      "Episode 6 (96 sec)  -- \tMin: 2.5\tMax: 7.3\tMean: 4.4\tMov. Avg: 2.2\n",
      "Episode 7 (96 sec)  -- \tMin: 1.3\tMax: 6.5\tMean: 3.5\tMov. Avg: 2.4\n",
      "Episode 8 (97 sec)  -- \tMin: 2.6\tMax: 11.5\tMean: 4.9\tMov. Avg: 2.7\n",
      "Episode 9 (97 sec)  -- \tMin: 3.0\tMax: 9.0\tMean: 5.8\tMov. Avg: 3.0\n",
      "Episode 10 (98 sec)  -- \tMin: 2.9\tMax: 9.8\tMean: 6.9\tMov. Avg: 3.4\n",
      "Episode 11 (100 sec)  -- \tMin: 4.2\tMax: 18.1\tMean: 8.1\tMov. Avg: 3.9\n",
      "Episode 12 (101 sec)  -- \tMin: 4.3\tMax: 12.9\tMean: 8.0\tMov. Avg: 4.2\n",
      "Episode 13 (101 sec)  -- \tMin: 3.2\tMax: 11.1\tMean: 7.2\tMov. Avg: 4.4\n",
      "Episode 14 (103 sec)  -- \tMin: 2.7\tMax: 18.8\tMean: 9.0\tMov. Avg: 4.8\n",
      "Episode 15 (103 sec)  -- \tMin: 3.1\tMax: 13.4\tMean: 8.3\tMov. Avg: 5.0\n",
      "Episode 16 (105 sec)  -- \tMin: 2.5\tMax: 16.2\tMean: 8.6\tMov. Avg: 5.2\n",
      "Episode 17 (107 sec)  -- \tMin: 2.2\tMax: 18.2\tMean: 8.5\tMov. Avg: 5.4\n",
      "Episode 18 (109 sec)  -- \tMin: 2.9\tMax: 12.8\tMean: 7.8\tMov. Avg: 5.5\n",
      "Episode 19 (112 sec)  -- \tMin: 4.6\tMax: 13.2\tMean: 9.9\tMov. Avg: 5.8\n",
      "Episode 20 (114 sec)  -- \tMin: 3.7\tMax: 16.0\tMean: 9.8\tMov. Avg: 6.0\n",
      "Episode 21 (117 sec)  -- \tMin: 7.4\tMax: 26.0\tMean: 12.8\tMov. Avg: 6.3\n",
      "Episode 22 (117 sec)  -- \tMin: 6.1\tMax: 19.7\tMean: 12.0\tMov. Avg: 6.6\n",
      "Episode 23 (119 sec)  -- \tMin: 6.0\tMax: 25.7\tMean: 12.4\tMov. Avg: 6.8\n",
      "Episode 24 (122 sec)  -- \tMin: 8.6\tMax: 19.2\tMean: 13.2\tMov. Avg: 7.1\n",
      "Episode 25 (125 sec)  -- \tMin: 4.9\tMax: 19.6\tMean: 14.3\tMov. Avg: 7.4\n",
      "Episode 26 (124 sec)  -- \tMin: 5.5\tMax: 18.6\tMean: 13.0\tMov. Avg: 7.6\n",
      "Episode 27 (129 sec)  -- \tMin: 8.7\tMax: 24.5\tMean: 14.6\tMov. Avg: 7.8\n",
      "Episode 28 (129 sec)  -- \tMin: 10.5\tMax: 20.4\tMean: 15.9\tMov. Avg: 8.1\n",
      "Episode 29 (132 sec)  -- \tMin: 11.5\tMax: 18.9\tMean: 15.7\tMov. Avg: 8.4\n",
      "Episode 30 (134 sec)  -- \tMin: 9.2\tMax: 26.8\tMean: 15.2\tMov. Avg: 8.6\n",
      "Episode 31 (137 sec)  -- \tMin: 8.4\tMax: 23.1\tMean: 14.8\tMov. Avg: 8.8\n",
      "Episode 32 (139 sec)  -- \tMin: 9.7\tMax: 27.8\tMean: 15.5\tMov. Avg: 9.0\n",
      "Episode 33 (142 sec)  -- \tMin: 10.5\tMax: 24.1\tMean: 18.5\tMov. Avg: 9.3\n",
      "Episode 34 (145 sec)  -- \tMin: 13.2\tMax: 25.6\tMean: 18.4\tMov. Avg: 9.6\n",
      "Episode 35 (145 sec)  -- \tMin: 13.5\tMax: 22.8\tMean: 18.0\tMov. Avg: 9.8\n",
      "Episode 36 (149 sec)  -- \tMin: 9.2\tMax: 22.9\tMean: 18.2\tMov. Avg: 10.1\n",
      "Episode 37 (155 sec)  -- \tMin: 8.9\tMax: 27.3\tMean: 19.5\tMov. Avg: 10.3\n",
      "Episode 38 (151 sec)  -- \tMin: 15.2\tMax: 25.2\tMean: 20.2\tMov. Avg: 10.6\n",
      "Episode 39 (154 sec)  -- \tMin: 15.3\tMax: 25.1\tMean: 21.0\tMov. Avg: 10.8\n",
      "Episode 40 (158 sec)  -- \tMin: 16.4\tMax: 26.7\tMean: 21.2\tMov. Avg: 11.1\n",
      "Episode 41 (160 sec)  -- \tMin: 16.6\tMax: 36.4\tMean: 22.1\tMov. Avg: 11.4\n",
      "Episode 42 (161 sec)  -- \tMin: 15.3\tMax: 30.9\tMean: 24.1\tMov. Avg: 11.7\n",
      "Episode 43 (163 sec)  -- \tMin: 17.9\tMax: 33.1\tMean: 26.4\tMov. Avg: 12.0\n",
      "Episode 44 (166 sec)  -- \tMin: 17.2\tMax: 32.5\tMean: 24.9\tMov. Avg: 12.3\n",
      "Episode 45 (172 sec)  -- \tMin: 21.8\tMax: 34.9\tMean: 27.2\tMov. Avg: 12.6\n",
      "Episode 46 (168 sec)  -- \tMin: 16.0\tMax: 35.7\tMean: 28.5\tMov. Avg: 13.0\n",
      "Episode 47 (178 sec)  -- \tMin: 22.9\tMax: 37.3\tMean: 28.9\tMov. Avg: 13.3\n",
      "Episode 48 (171 sec)  -- \tMin: 20.6\tMax: 35.8\tMean: 30.0\tMov. Avg: 13.7\n",
      "Episode 49 (173 sec)  -- \tMin: 25.2\tMax: 35.4\tMean: 31.0\tMov. Avg: 14.0\n",
      "Episode 50 (178 sec)  -- \tMin: 25.2\tMax: 37.2\tMean: 31.6\tMov. Avg: 14.4\n",
      "Episode 51 (179 sec)  -- \tMin: 27.3\tMax: 38.4\tMean: 32.6\tMov. Avg: 14.7\n",
      "Episode 52 (175 sec)  -- \tMin: 29.0\tMax: 36.3\tMean: 32.9\tMov. Avg: 15.1\n",
      "Episode 53 (177 sec)  -- \tMin: 21.6\tMax: 39.1\tMean: 33.1\tMov. Avg: 15.4\n",
      "Episode 54 (184 sec)  -- \tMin: 28.1\tMax: 38.8\tMean: 34.4\tMov. Avg: 15.8\n",
      "Episode 55 (181 sec)  -- \tMin: 29.4\tMax: 38.6\tMean: 34.1\tMov. Avg: 16.1\n",
      "Episode 56 (180 sec)  -- \tMin: 28.7\tMax: 39.6\tMean: 33.8\tMov. Avg: 16.4\n",
      "Episode 57 (179 sec)  -- \tMin: 28.6\tMax: 38.6\tMean: 34.2\tMov. Avg: 16.7\n",
      "Episode 58 (184 sec)  -- \tMin: 29.2\tMax: 39.3\tMean: 34.9\tMov. Avg: 17.0\n",
      "Episode 59 (184 sec)  -- \tMin: 32.9\tMax: 39.4\tMean: 36.8\tMov. Avg: 17.4\n",
      "Episode 60 (182 sec)  -- \tMin: 31.9\tMax: 39.2\tMean: 36.4\tMov. Avg: 17.7\n",
      "Episode 61 (184 sec)  -- \tMin: 29.3\tMax: 39.6\tMean: 36.7\tMov. Avg: 18.0\n",
      "Episode 62 (178 sec)  -- \tMin: 33.1\tMax: 39.0\tMean: 36.3\tMov. Avg: 18.3\n",
      "Episode 63 (175 sec)  -- \tMin: 25.7\tMax: 39.6\tMean: 36.7\tMov. Avg: 18.6\n",
      "Episode 64 (182 sec)  -- \tMin: 34.3\tMax: 39.1\tMean: 37.3\tMov. Avg: 18.9\n",
      "Episode 65 (177 sec)  -- \tMin: 35.0\tMax: 39.5\tMean: 38.3\tMov. Avg: 19.2\n",
      "Episode 66 (179 sec)  -- \tMin: 36.4\tMax: 39.5\tMean: 38.3\tMov. Avg: 19.5\n",
      "Episode 67 (178 sec)  -- \tMin: 33.5\tMax: 39.6\tMean: 39.0\tMov. Avg: 19.8\n",
      "Episode 68 (181 sec)  -- \tMin: 34.8\tMax: 39.6\tMean: 38.7\tMov. Avg: 20.0\n",
      "Episode 69 (179 sec)  -- \tMin: 37.4\tMax: 39.5\tMean: 38.8\tMov. Avg: 20.3\n",
      "Episode 70 (181 sec)  -- \tMin: 32.9\tMax: 39.6\tMean: 38.2\tMov. Avg: 20.6\n",
      "Episode 71 (182 sec)  -- \tMin: 36.3\tMax: 39.6\tMean: 38.6\tMov. Avg: 20.8\n",
      "Episode 72 (176 sec)  -- \tMin: 37.5\tMax: 39.6\tMean: 39.1\tMov. Avg: 21.1\n",
      "Episode 73 (173 sec)  -- \tMin: 38.0\tMax: 39.5\tMean: 38.9\tMov. Avg: 21.3\n",
      "Episode 74 (171 sec)  -- \tMin: 35.9\tMax: 39.4\tMean: 38.4\tMov. Avg: 21.6\n",
      "Episode 75 (172 sec)  -- \tMin: 36.8\tMax: 39.6\tMean: 38.5\tMov. Avg: 21.8\n",
      "Episode 76 (176 sec)  -- \tMin: 37.1\tMax: 39.6\tMean: 38.4\tMov. Avg: 22.0\n",
      "Episode 77 (174 sec)  -- \tMin: 37.8\tMax: 39.6\tMean: 39.0\tMov. Avg: 22.2\n",
      "Episode 78 (182 sec)  -- \tMin: 37.9\tMax: 39.6\tMean: 39.1\tMov. Avg: 22.4\n",
      "Episode 79 (177 sec)  -- \tMin: 38.5\tMax: 39.6\tMean: 39.3\tMov. Avg: 22.7\n",
      "Episode 80 (172 sec)  -- \tMin: 37.4\tMax: 39.6\tMean: 39.3\tMov. Avg: 22.9\n",
      "Episode 81 (176 sec)  -- \tMin: 38.1\tMax: 39.7\tMean: 39.1\tMov. Avg: 23.1\n",
      "Episode 82 (172 sec)  -- \tMin: 37.6\tMax: 39.6\tMean: 38.9\tMov. Avg: 23.3\n",
      "Episode 83 (176 sec)  -- \tMin: 38.2\tMax: 39.6\tMean: 39.1\tMov. Avg: 23.4\n",
      "Episode 84 (174 sec)  -- \tMin: 35.4\tMax: 39.6\tMean: 38.8\tMov. Avg: 23.6\n",
      "Episode 85 (176 sec)  -- \tMin: 35.8\tMax: 39.6\tMean: 39.0\tMov. Avg: 23.8\n",
      "Episode 86 (173 sec)  -- \tMin: 37.4\tMax: 39.7\tMean: 39.1\tMov. Avg: 24.0\n",
      "Episode 87 (174 sec)  -- \tMin: 37.3\tMax: 39.6\tMean: 39.1\tMov. Avg: 24.2\n",
      "Episode 88 (178 sec)  -- \tMin: 37.0\tMax: 39.6\tMean: 38.9\tMov. Avg: 24.3\n",
      "Episode 89 (176 sec)  -- \tMin: 37.3\tMax: 39.5\tMean: 38.9\tMov. Avg: 24.5\n",
      "Episode 90 (171 sec)  -- \tMin: 37.4\tMax: 39.5\tMean: 38.7\tMov. Avg: 24.6\n",
      "Episode 91 (175 sec)  -- \tMin: 36.6\tMax: 39.5\tMean: 38.5\tMov. Avg: 24.8\n",
      "Episode 92 (172 sec)  -- \tMin: 36.8\tMax: 39.6\tMean: 38.7\tMov. Avg: 25.0\n",
      "Episode 93 (173 sec)  -- \tMin: 36.0\tMax: 39.6\tMean: 38.2\tMov. Avg: 25.1\n",
      "Episode 94 (172 sec)  -- \tMin: 36.0\tMax: 39.6\tMean: 38.6\tMov. Avg: 25.2\n",
      "Episode 95 (177 sec)  -- \tMin: 36.6\tMax: 39.6\tMean: 38.8\tMov. Avg: 25.4\n",
      "Episode 96 (172 sec)  -- \tMin: 30.7\tMax: 39.6\tMean: 37.9\tMov. Avg: 25.5\n",
      "Episode 97 (176 sec)  -- \tMin: 31.7\tMax: 39.6\tMean: 36.5\tMov. Avg: 25.6\n",
      "Episode 98 (175 sec)  -- \tMin: 30.4\tMax: 39.6\tMean: 36.4\tMov. Avg: 25.7\n",
      "Episode 99 (172 sec)  -- \tMin: 31.0\tMax: 39.4\tMean: 38.0\tMov. Avg: 25.9\n",
      "Episode 100 (180 sec)  -- \tMin: 31.7\tMax: 39.7\tMean: 38.2\tMov. Avg: 26.0\n",
      "Episode 101 (179 sec)  -- \tMin: 35.3\tMax: 39.7\tMean: 38.9\tMov. Avg: 26.4\n",
      "Episode 102 (184 sec)  -- \tMin: 33.5\tMax: 39.6\tMean: 38.4\tMov. Avg: 26.7\n",
      "Episode 103 (174 sec)  -- \tMin: 30.7\tMax: 39.6\tMean: 38.0\tMov. Avg: 27.1\n",
      "Episode 104 (177 sec)  -- \tMin: 35.3\tMax: 39.1\tMean: 37.7\tMov. Avg: 27.5\n",
      "Episode 105 (173 sec)  -- \tMin: 32.8\tMax: 38.8\tMean: 37.7\tMov. Avg: 27.8\n",
      "Episode 106 (175 sec)  -- \tMin: 34.5\tMax: 39.4\tMean: 37.7\tMov. Avg: 28.1\n",
      "Episode 107 (174 sec)  -- \tMin: 32.9\tMax: 39.6\tMean: 38.1\tMov. Avg: 28.5\n",
      "Episode 108 (180 sec)  -- \tMin: 31.7\tMax: 39.6\tMean: 38.2\tMov. Avg: 28.8\n",
      "Episode 109 (187 sec)  -- \tMin: 37.3\tMax: 39.6\tMean: 38.9\tMov. Avg: 29.1\n",
      "Episode 110 (182 sec)  -- \tMin: 36.2\tMax: 39.6\tMean: 38.6\tMov. Avg: 29.5\n",
      "Episode 111 (189 sec)  -- \tMin: 35.8\tMax: 39.2\tMean: 37.8\tMov. Avg: 29.8\n",
      "Episode 112 (177 sec)  -- \tMin: 31.7\tMax: 39.3\tMean: 37.8\tMov. Avg: 30.1\n",
      "\n",
      "Environment SOLVED in 12 episodes!\tMoving Average =30.1 over last 100 episodes\n"
     ]
    }
   ],
   "source": [
    "# run the training loop\n",
    "agent = Agent(state_size=state_size, action_size=action_size, random_seed=1)\n",
    "scores, avgs = ddpg()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8zPcfwPHXJ3uJkCCISBA7ZmIrqoOWGlWj/aGT7lar1V3dS3erpdWWVtFqaauqdo22dgSxgiBEJiIi+/P743PUCkEul7t7Px+PPJL73vfu3pdL7n2f9f4orTVCCCGcl4utAxBCCGFbkgiEEMLJSSIQQggnJ4lACCGcnCQCIYRwcpIIhBDCyUkiEEIIJyeJQAghnJwkAiGEcHJutg6gJIKCgnRYWJitwxBCCLuybt26NK11lYudZxeJICwsjLVr19o6DCGEsCtKqb0lOU+6hoQQwslZPREopVyVUhuUUnMsl8OVUquUUvFKqRlKKQ9rxyCEEKJ4ZdEieATYetrlt4D3tdb1gMPAXWUQgxBCiGJYdYxAKRUC3Ai8BjymlFLA1cCtllMmA2OBzy71vvPz80lMTCQnJ6eUohUl4eXlRUhICO7u7rYORQhRSqw9WPwB8CRQwXI5EDiitS6wXE4Eal7OHScmJlKhQgXCwsIw+UVYm9aa9PR0EhMTCQ8Pt3U4QohSYrWuIaVULyBFa73uMm8/Qim1Vim1NjU19Zzrc3JyCAwMlCRQhpRSBAYGSitMCAdjzTGCjsBNSqkEYDqmS+hDIEApdbIlEgIcON+NtdYTtdZRWuuoKlXOPw1WkkDZk9+5EI7Hal1DWuungacBlFJdgdFa69uUUj8CAzDJYTjwi7ViEEKUrpz8Qv7ccog9acfxdHPFy92FBsEVaF27Ep5urrYOT1wmWywoGwNMV0q9CmwAJtkghlLh6upKZGQk+fn5uLm5MWzYMEaNGoWLiwtLly6lT58+1KlTh+zsbKpVq8aTTz5Jr169ABg7dixffPEFVapUoaCggNdff52bbroJgO+++463336bwsJC3NzciI6OZty4cQQEBNjy6Qo7dSwnnydnxuLt4cqw9mG0qHVpf0cpmTlsOnCUv3akMnvDATJzCs45x9vdlfZ1A+nZNJjrmwbj7+VOWlYuK+PTCPDx4KqIoDNak2lZuaxNOMy6vRkkpGfjqhRurorQyj50bVCVVqEBuLnKMqeyUiaJQGu9FFhq+Xk30KYsHtfavL29iYmJASAlJYVbb72VzMxMXnrpJQA6d+7MnDlzAIiJiaFv3754e3vTvXt3AEaNGsXo0aPZunUrnTt3JiUlhfnz5/P+++/zxx9/ULNmTQoLC5k8eTLJycmSCJyc1pqnf97E8p1p3NM5nMFtQvFyd6WoSHPkRD6Vfc9dkpNxPI/hX61ma1Imnm4u/Lz+AM1DKvLOLc2pX63CeR7lP5sSj/LA9+vZl5ENgIebCz2aBDMouhZtwyuTX6g5nldAzL4jLN+ZyuLtKSyemcKzszdTu7IPO1OyTt1X69qVGH1dA9KycpmxZj8r4tNO3WedIF+0hvzCIuZtPsT4pbuo4OVGaGUfAnzcCfLzpGfT6nRvVBV3GySHDfsO8+XyPTzXqxHVK3qX2v3uz8hm3uZD9G1ZkyoVPEvtfi+H0lrbNICSiIqK0meXmNi6dSuNGjWyUUSGn58fWVn//bHv3r2b6Oho0tLS+Ouvvxg3btypRADw1Vdf8dtvvzFr1izGjh2Ln58fo0ePBiA4OJjY2FhuvvlmXn75Zbp161bmz6ekysPv3hl9sngn4+bvIDzIlz1px6lawZMaAd5sP3SME/mFPNitHqOvb3Dq/OTMHP735Sr2ZWTz2f9aER1WmVkbDvDx4ni0hh9GtqNOFb/zPtbWpEyGfPEvvh5u3NkpnMiaFWlSwx9fz+I/O2qtidl/hF83HmRX6nHahlemU70gNh88ygcLd5J6LBeAmgHeDGgdwlX1g2has+IZXUqZOfms3JnGsp1pJGfmcPREPnvTs0nLyiXIz5ObW9ekT/OaNKpuklhcUib/7Eqne6NqhAf5nhHPoaM5zFiznx/X7adL/Sq81i/ykn/nR7LzuOHD5Rw8mkN4kC/T7mlHcEWvS76fs2XlFtDv05XsTMnC082FwdG1uK9rvVK579MppdZpraMudp5d1Bq6mJd+20LcwcxSvc/GNfx5sXeTS7pNnTp1KCwsJCUl5bzXt2rVinfeeeec46tWrcLFxYUqVaqwZcsWWrVqdVkxC8c1J/Yg4+bvoF/Lmrw3sDn/7E5nwl+7yckvZFB0LVKO5fDJknhCA30YGFWLuIOZ3D15DUdP5PPNHW1oXzcQgGHtw+hQN4jBE//h1i9W8cPI9gCs25fBsZwC6gT54enuwr3frsPLzZXpI9pRq7JPiWJUStEytBItQyudcbx5rQD6tazJLzEHqRHgTad6Qbi6nH/Sgb+XOz0jq9MzsvqpYwWFRfy1I5Vpq/fz5fI9TPhrNxFV/cgvLCIh3bRWPly0k09vbcVV9auQlpXL2/O28dP6AxQWaWoH+jB11T76twqhde1K533c89Fa89RPm0g5lssrfZrw5h/buPWLf5k+oh1V/S//DVtrzZMzN7IrNYt3BjRjTUIG36/ex4K4ZOaNugp/r7Jfo+MQicBenN36ev/99/nuu++oUKECM2bMOGdGzqZNmxg6dCjHjh3j9ddfZ9CgQWUZrrAhrTWzNhxg04GjHDqaw+JtKUTVrsSbN0eilKJD3SA61A06dX5+YRHHctbwzM+bSD6aw+d/7aKClzszRranac2KZ9x3vap+fHd3WwZP/Jdu7y6lsOjcXoEgP0++v6dtiZPAxfh4uDGkTehl3dbN1YXujarRvVE10rNymbv5EHNjk3BzVYzsUpfImhUZ/eNG7vhmDQOjQvg9NonsvEKGta/N7R3CCPLzpNu4pbz82xZm3d8Rl2KS0Nm+X72PeVsO8cwNDRnaPoxG1f0Z9tVqbpnwD5//rzWNqvtf1vOZuGw3czeZ+70lqha3RNVicJtQBnz2N6/8Fsc7tzS/rPu9Eg6RCC71k7u17N69G1dXV6pWrcrWrVvPuX7Dhg1ndKmcHCM4XZMmTVi/fj3dunUjMjKSmJgYHnzwQU6cOGH1+EX58eeWQzz2w0Z8PVypVtGLrg2q8Hq/yGJn5ri7uvDJra24+bO/eXfBDprW9OfLYdHFdjU0DPbn+7vb8cPa/dSt6kdU7UpU8vFgd1oW+zOy6VA3qNSSQGkK9PNkaLvaDG1X+4zjP93XgUemxzBt9X461gvkpZuaUK/qf2MgY3o05PEfNzJrwwFubh1S7P3nFxaxbEcqszYc4M8th+gcEcTdneoAEBVWme/ubst9362j3/iVvNE/kn4ti7+vkwqLNFNX7WX5zjTiDmZy4MgJboyszj2d65w6p1VoJe7vWo9PlsRzfZNgrmlc7VJ/NVfEIRJBeZCamsq9997Lgw8+eN659rGxsbzyyit8+eWXF7yfp59+mtGjR/PLL78QEmL+yCQJOJf8wiLemrediKp+/PFI5xLPnqno7c7kO9vwa8xBhneojY/Hhf+9G9fwZ+xNZ36ICq7oBXUvO3Sb8fV0Y+LQ1sSnZhFR1e+c/8F+LWsy5d+9vDVvGz2aBhc71nHbl6tYvSeDSj7uDI4OZdS19c9oQbQKrcSchzrz4PfrGTVjI8t2pPFUz4ZUK6arKD0rl0dnxLB8Zxp1gnxpVbsSd3QM47a2tc+J8eHuESzcmsxTP29iVnAFQip5l9m6HUkEV+DEiRO0aNHi1PTRoUOH8thjj526fvny5bRs2ZLs7GyqVq3KRx99dGrGUHFuuOEGUlNT6dmzJ4WFhQQEBNC0aVOuv/56az8dUU58v2ofe9KO89XtUZc8hbJmgDf3dbXDd/JS4OKiip0J5eKieLF3Y/qP/5vv/t3LyC7n/o52Jh9j9Z4MHuxWj4e7R+Dhdv7ffZUKnky9uy0fLdrJ53/t5s8thxh5VV0iqplxi7yCInILisjJL2TSij2kH8/jjf6RDI6udcE3dg83F94b2II+n66g89tL8Pdyo25VP569oRFRYZUv75dSQjJrSFwy+d1bT2ZOPl3fWUrD4ApMvbutrOQuZQM++5uM43kserzLOb/bDxbu4MNFO1n1THeqVijZYPC+9GxemxvHn1uSz3t9WKAPn9za6pxxmgvZdiiTf3elE5+axa6U4zzXqxFNapT89qdzqllDQjiKz5buIuN4Hs/c0EiSgBUMjK7FkzNjWbv3MNFnfcqeuymJ6LDKJU4CAKGBPkwYGkVC2nFyC4pwc1V4uLrg6eaCp5srfl5uxc6QKk7DYH8aBl/eQPTlkqV7QpQTi7clM+GvXQxoHXJJnyBFyd0YWR1fD1dmrNl/xvGdycfYkZxFr2bVi7nlhYUF+dIguAJ1q/hRq7IPVf29qOjjfslJwFYkEQhRDsQdzOSh7zfQuIY/L/cpH7PgHJGvpxu9m9fg99gkjuXknzr++6YklIIeTYNtGJ3tSCIQwgom/53AVW8vYf2+w6eOaa3ZsO8wx3PPrNWTnJnDXZPXUMHLnUnDoy8620dcmYHRtTiRX8ic2KRTxy6nW8iRSCIQwgp+Xp/IvoxsBk/4lx/X7mfDvsMMmvAv/cb/TY8Pl7F6TwYA87cc4saPVnD0RD6Tbo8qdhqiKD0tawUQUdWP6Wv2k1dQRHyK6Ra6MfLyuoUcgXz0EKKUHT6eR+yBo9zeIYydKcd4YmYsAEF+Hjx+bX1+XJfIoIn/0Dq0Emv3HqZxdX/G3dKGxjXKdoDQWSmlGNwmlFfmxNHw+T/w83RDKejppN1CIImg3Pn888/x8fFh2LBhtg5FXKYV8WloDTe1qEGzmhX5eHE8LkpxV+dw/DxNEbc3/tjKD2sTeaR7BA90q1fsnHVhHXd0CKNKBU/ik4+xLyObsCDfK6ofZO8kEZQz9957r61DEFdo+c5U/L3caFazIm6uLoy6tv4Z1/t6uvFq30jG9m4iNfdtxMVFcVPzGrYOo9yQv8LLlJCQQMOGDbn99tupX78+t912GwsXLqRjx45ERESwevVqADIyMujbty/NmjWjXbt2xMbGUlRURFhYGEeOHDl1fxERESQnJzN27FjGjRsHQNeuXRkzZgxt2rShfv36LF++HIDs7GwGDhxI48aN6devH23btuXsBXcAL7/8MtHR0TRt2pQRI0agtWbbtm20afPfdhAJCQlERpryvHPnzqVhw4a0bt2ahx9++NQmOqLktNYs35lGx3pBF32TlyQgygurtQiUUl7AMsDT8jgztdYvKqW+AboARy2n3q61jrmiB3v0UYi5srs4R4sW8MEHFzwlPj6eH3/8ka+++oro6Gi+//57VqxYwa+//srrr7/O7NmzefHFF2nZsiWzZ89m8eLFDBs2jJiYGPr06cOsWbO44447WLVqFbVr16ZatXMLTRUUFLB69Wrmzp3LSy+9xMKFCxk/fjyVKlUiLi6OzZs306JFi/PG9+CDD/LCCy8AMHToUObMmUPv3r3Jy8tjz549hIeHM2PGDAYNGkROTg4jR45k2bJlhIeHM2TIkCv/HTqhXalZJB3N4eHu599nW4jyyJofSXKBq7XWzYEWQA+lVDvLdU9orVtYvkr5HbzshIeHExkZiYuLC02aNKF79+4opYiMjCQhIQGAFStWMHToUACuvvpq0tPTyczMZNCgQcyYMQOA6dOnF1tiun///gC0bt36jPscPHgwAE2bNqVZs2bnve2SJUto27YtkZGRLF68mC1btgAwcODAU499MhFs27aNOnXqEB4eDiCJ4DL9tcPsvNWpXtBFzhSi/LDm5vUaOLl9l7vlyzqFjS7yyd1aPD3/217OxcXl1GUXFxcKCs7d1/V07du3Jz4+ntTUVGbPns1zzz13wcdwdXW96H2eLicnh/vvv5+1a9dSq1Ytxo4dS05ODgCDBg3illtuoX///iiliIiIOLXlprgyy3emUifIt1yWcBaiOFbtpFRKuSqlYoAUYIHWepXlqteUUrFKqfeVUrbdrNPKOnfuzNSpUwFYunQpQUFB+Pv7o5SiX79+PPbYYzRq1IjAwMAS32fHjh354YcfAIiLi2PTpk3nnHPyTT8oKIisrCxmzpx56rq6devi6urKK6+8cqol0qBBA3bv3n2q1XGyxSBKLregkH93p9M5QloDwr5YddaQ1roQaKGUCgBmKaWaAk8DhwAPYCIwBnj57NsqpUYAIwBCQy9vZ6PyYOzYsdx55500a9YMHx8fJk+efOq6QYMGER0dzTfffHNJ93n//fczfPhwGjduTMOGDWnSpAkVK55ZmyYgIIB77rmHpk2bEhwcTHR09BnXDxo0iCeeeII9e/YA4O3tzfjx4+nRowe+vr7nnC8u7vfYJHLyi7iqvowPCPtSZmWolVIvANla63GnHesKjNZaX3B6ipShPlNhYSH5+fl4eXmxa9currnmGrZv346Hh8cV3W9WVhZ+fn5orXnggQeIiIhg1KhR55znzL/74uxIPkbfT1fSMLgCM0a2x11mBIlyoKRlqK3216qUqmJpCaCU8gauBbYppapbjimgL7DZWjE4quzsbDp16kTz5s3p168f48ePv+IkAPDFF1/QokULmjRpwtGjRxk5cmQpRGvfEg9nM3TSKpZsSyn2nKMn8hn57Tp8Pd347H+tJQkIu2PNrqHqwGSllCsm4fygtZ6jlFqslKoCKCAGkBVUl6hChQrnXTdwpUaNGnXeFoAz+/Yfs9fs8p1p3NUpnCd7NDhj3+ATeYWMmhHD/oxspo1oJ7WChF2y5qyhWKDleY5fXYqPIZt3lDF72NGutOQXFvHT+kS6NahCrco+TFqxh5XxadzfrR49mwazMzmLh6dvID4li1f7Nj1noxMh7IXdlpjw8vIiPT2dwMBASQZlRGtNeno6Xl7O8al30dYU0rLyGNq+Nlc3rEanekG88cc2Hp62geoVvUjPyiPAx51v72pD5wgZIBb2y24TQUhICImJiaSmpto6FKfi5eVFSEiIrcMoEzPW7KOavydXWd7kr2sSzDWNqrF4WwpT/t1LuzoePN+rMZV9r3x8RghbsttE4O7ufmoVrBClLenoCf7akcp9XeueURPIxUVxTeNqXNP43HIgQtgrmd4gxHnMXJtIkYaBUbVsHYoQVme3LQIhrOF4bgHzNh/i23/30r5OILUDfW0dkhBWJ4lACGB/Rjbjl+7il5gDZOcVUquyN6Ovr3/xGwrhACQRCKeWnpXLG39sY9aGA7gqRd+WNbglqhZRtSvJbDThNCQRCKf2wi9bWBCXzLD2tRl5VV2CKzrH1FghTieJQDituIOZ/L4piQe71WP09Q1sHY4QNiOzhoTTen/hDip4uXFP5zq2DkUIm5JEIJzSxv1HWBCXzD2d61DRx93W4QhhU5IIhFN6b8EOKvm4c0fHMFuHIoTNSSIQTmfmukT+2pHKyC51qeAlrQEhJBEIp/Lz+kSemLmRTvWCuL1DmK3DEaJckEQgnMbsDQd4/MeNdKgbyBfDovByd734jYRwApIIhFPYn5HNEzM30ja8Ml8Oi8bbQ5KAECdJIhBO4d3523FRig8GtZQkIMRZrLlnsZdSarVSaqNSaotS6iXL8XCl1CqlVLxSaoZSSoq5C6vacvAov2w8yJ2dwmXlsBDnYc0WQS5wtda6OdAC6KGUage8Bbyvta4HHAbusmIMQvD2vO34e7lzb5e6tg5FiHLJaolAG1mWi+6WLw1cDcy0HJ8M9LVWDEL8vSuNv3akcn/XulT0lqmiwo6cOAEffAA5OVZ/KKuOESilXJVSMUAKsADYBRzRWhdYTkkEalozBuHc3pu/g2B/L4bLVFFhL4qK4NtvoX59GDUK5syx+kNaNRForQu11i2AEKAN0LCkt1VKjVBKrVVKrZV9icXlWJuQwdq9hxnZpY5MFRX2YckSiIqCYcMgOBiWLoUBA6z+sGUya0hrfQRYArQHApRSJ6uehgAHirnNRK11lNY6qkqVKmURpnAwE5btJsDHnUHRst2kKOe2bYPeveHqqyE9Hb77Dlatgi5dyuThrTlrqIpSKsDyszdwLbAVkxBOprjhwC/WikE4r/iULMs+A2H4eEi1dVFOZWbC6NEQGQnLlsFbb8H27XDbbeBSdrP7rfkfUh2YrJRyxSScH7TWc5RSccB0pdSrwAZgkhVjEE7qi2W78XRzYXj72rYORYhzaQ3TpsHjj0NyMtx1F7z+Otio98NqiUBrHQu0PM/x3ZjxAiGsIjkzh1kbDjAouhaBfp62DkeIM+3YAfffD4sWQXQ0/Pqr+W5DsrJYOJSFccnc/NnfFGktG86I8iU3F156yXQDrV0L48fDP//YPAmAbFUpHEROfiEPTdvAgrhkIqr6MfXutoQG+tg6LCGMJUvg3ntNa2DwYHj/fTMrqJyQRCAcwuwNB1gQl8zo6+ozsktd3F2lsSvKgSNHzGDwpElQpw78+Sdcd52tozqH/LcIhzBt9T7qV/PjgW71JAkI29MafvoJGjeGb76BMWNg8+ZymQRAEoFwAJsPHGVj4lFubROKUsrW4Qhnl5Bg1gQMGADVqsHq1fDmm+DtbevIiiWJQNi9aav34enmQr+WIbYORTizwkJ47z1o0sSsCH73XVizBlq1snVkFyVjBMKuHc8t4JeYg9zYrDoVfaSonLCRTZvMWoA1a0xr4JNPIDTU1lGVmLQIhF37beNBsnILuK2t/fzTCQeSl2emhLZubbqEpk+HX36xqyQA0iIQdm7amv3Ur+ZHq9BKtg5FOJt16+COO0xr4LbbTMnooCBbR3VZpEUg7FbG8Tw27j/CTc1ryCCxKDu5ufDcc9C2rSkQ9+uvpkicnSYBkBaBsGPr9h4GoE14oI0jEU4jJsaUiN60CYYPNwvDKtl/a1RaBMJurU3IwMPVhWYhFW0dinB0hYVmCmibNpCWBr/9ZtYHOEASAEkEwk4cOprDl8t3U1SkTx1bk5BBZEhF2XRGWNfevdCtGzz9NPTtaxaG9epl66hKlSQCYRe+/TeBV3/fyr970gFTW2jTgaNEhTnGJzJRTk2bBs2awcaNMGUKzJgBlSvbOqpSJ4lA2IU1CWY8YPYGs6Hdxv1HyC/URNd2vH9KYWMpKTB7tikOd+utplroxo0wdCg46KQEGSwW5V5eQREb9x/BRcEfmw7xcp+mrLUMFLeuLS0CcQUKC2HrVvj3X1i50nzt3Gmu8/SEsWPh2WfBzbHfKh372QmHsPngUXILihjWvjZT/tnLgrhk1iRkEFHVj0q+HrYOT9gLrc2ir1WrzArg1ath/XrIzjbXBwZCx45w993me+vW4OVl05DLitUSgVKqFjAFqAZoYKLW+kOl1FjgHiDVcuozWuu51opD2L+1CRkAPNitHgvikvlpfSLr9h6mV7MaNo5MlHu7d8PChab2z7JlcMB0LeLlBS1bmjf9qCgzG6h+fYft+rkYa7YICoDHtdbrlVIVgHVKqQWW697XWo+z4mMLB7Im4TDhQb5U9feiT4uafP7XLgCiZaBYnC0tDRYvNm/+CxfCnj3mePXqcNVV0LkzdOgATZuCu9SmOsmaexYnAUmWn48ppbYCNa31eMIxaa1Zm5DBNY2qAdCv5emJQAaKnV5uLixfDgsWmDf+DRtMF5C/v5ny+fjjcM01Tv1pvyTKZIxAKRWG2ch+FdAReFApNQxYi2k1HD7PbUYAIwBC7ayAkyg9u1KPczg7/9SbfoPgCjSu7k/68VxCKpXf+u7Cinbtgj/+gHnzzBaQ2dnm032HDqYA3LXXmu4eBx/gLU1W/00ppfyAn4BHtdaZSqnPgFcw4wavAO8Cd559O631RGAiQFRUlD77euEcTo4PtD6tG+j9QS04lpMv9YWcRW6u6d+fMwfmzoX4eHO8Xj248064/nro2hX8/Gwapj2zaiJQSrljksBUrfXPAFrr5NOu/wKYY80YhH1bk3CYyr4e1AnyPXWsQXAFG0YkykR6Ovz+uyno9uefkJVlBnivvhoeeQR69oS6dW0dpcOw5qwhBUwCtmqt3zvteHXL+AFAP2CztWIQ9m/t3gyialeST//OYO9es5Br9mzTAigqgho1TInn3r1Nn7+Pj62jdEjWbBF0BIYCm5RSMZZjzwBDlFItMF1DCcBIK8Yg7FhKZg5707P5X9vatg5FWMvOnTBzptnofd06c6xpU3jmGVPXp1UrGeQtA9acNbQCON8rKGsGRIms3JUGQLs6UmbaoezYAT/+CD/8ALGx5libNvDWW9C/v+n7F2VKhtVFubVsRxqVfT1oUsPf1qGIK7V7t3njnzHD1PQHaN/ebPZ+8812t7Wjo5FEIMqloiLN8p1pdKoXhIuLdA3YpYMHzR6+06ebkg4A7dqZN/8BA6BWLdvGJ06RRCDKpa2HMknLyuWq+lVsHYq4FJmZ8PPPZuvGxYvN4q5WreDtt2HgQKgt4z3lkSQCUS4t32nGBzpH2O8+sE6joMCs7J08GX75BXJyzNTO5583ZZwbNLB1hOIiJBGIcmn5zlQaBlegmr9zVH+0S9u3w1dfwbffQlKS2bDlzjtN3f62bWW2jx2RRCDKney8AtbsOczwDtKNUO5kZ5sZP19+CStWgKurWdx1xx1w442mhr+wO5IIRLmzak8GeYVFMj5QnsTGwsSJpu//6FGIiDCbuQ8bZip7CrsmiUCUO8t2pOLp5iLVRW3txAkz3XPCBLODl6enme1zzz2mpLN0/TiMEicCpVQnIEJr/bVSqgrgp7XeY73QhDMqLNIs2ZZC2zqBeLm72joc57RzJ3z+OXz9NRw+bAZ733vPfPoPlMV9jqhEiUAp9SIQBTQAvgbcge8wZSSEKDWzNxwgIT2bx66TmSZlKj/fzPj5/HNYtMiUcO7fH+67D7p0kU//Dq6kLYJ+mP0E1gNorQ9adh0TotTk5Bfy7vztNAupSK9I6XcuE0lJpu9/wgTzc2govPaamf0THGzr6EQZKWkiyNNaa6WUBlBK+V7sBkJcqq9XJnDwaA7vDmwhq4mtSWuzgftHH5kZQAUF0KOHSQg9e5qZQMKplDQR/KCUmgAEKKXuwWzIIqvNAAAgAElEQVQk84X1whLOJuN4HuOXxNO9YVXa15V+aKsoKDBVPt9915R88PeHBx4wXxERto5O2FCJEoHWepxS6logEzNO8ILWesFFbibEBeUXFjF3UxLr9h5mZXwax/MKGNOzoa3DcjzZ2TBpkhnwTUgwb/qffmoWflWQHl5RgkSglHIFFmqtuwHy5i9KzStz4pjyz158PVxpFhLAfV3rUb+avDGVmiNHzBv+Bx9AWhp07Gh+7t0bXFxsHZ0oRy6aCLTWhUqpIqVURa310bIISji+f3alM+WfvdzeIYznezXGVcYESs+hQ+YNf/x4OHYMbrgBnn4aOnWydWSinCrpGEEWZqexBcDxkwe11g8XdwOlVC1gClANsxvZRK31h0qpysAMIAyzQ9lArfXhy4pe2KXsvALG/BRL7UAfxvRoKEmgtCQkwDvvmG6g/Hy45RZ46ilo0cLWkYlyrqSJ4GfL16UoAB7XWq+3TDVdZ0kktwOLtNZvKqWeAp4CxlzifQs79va87ezLyGbGiHZ4e8gMlSu2cye8+ipMnWq6fIYPhyeflAFgUWIlHSyerJTyAOpbDm3XWudf5DZJQJLl52NKqa1ATaAP0NVy2mRgKZIInMbf8WlM/ieB4e1r01a2oLwyu3fDK6+Y6p8eHvDww/DYYxASYuvIhJ0p6crirpg37QTMPsS1lFLDtdbLSnj7MMyCtFVANUuSADiE6ToSTuDw8TxG/RBDeJCvzA66EomJJgF89ZVZAfzwwzBmDFSTfyVxeUraNfQucJ3WejuAUqo+MA1ofbEbKqX8gJ+AR7XWmeq0peqnL1I7z+1GACMAQmU/U7untWbMT7FkHM9j0vBofDyk3uElS0+H1183M4GKimDkSHjmGahRw9aRCTtX0jlk7ieTAIDWegem3tAFKaXcMUlgqtb65BhDslKquuX66kDK+W6rtZ6otY7SWkdVqSLliO3dtNX7mR+XzJgeDWlas6Ktw7EvJ07AW2+ZXb8++ACGDIEdO+CTTyQJiFJR0kSwVin1pVKqq+XrC2DthW6gzEf/ScBWrfV7p131KzDc8vNw4JdLDVrYl+y8At74Yysd6wVyZ8dwW4djPwoLTQXQ+vXN7J/OnWHjRnMsLMzW0QkHUtL2+X3AA8DJ6aLLgfEXuU1HYChm2mmM5dgzwJuYkhV3AXuBgZcUsbA7szcc5FhOAY9dW19qCJWE1jBvHjzxBGzZAm3amA1hunSxdWTCQZU0EbgBH578ZG9ZbXzBPem01iswA8vn073EEQq7prVmyj8JNKnhT6vQSrYOp/zbuBFGj4aFC6FePVMU7uabpQy0sKqSdg0tArxPu+wNLCz9cISjWb0ng22HjjG8fRhK3syKt2+fmf/fsiWsX2/GArZsMTuCye9NWFlJWwReWuuskxe01llKKR8rxSTshNaaIs0FVwZP+WcvAT7u3NRCBjXP6/BheOMNUxIaTHfQU09BJWk9ibJT0kRwXCnVSmu9HkApFQWcsF5Ywh4MnbSa1XsyqFXZm/AgP4a2r02X0zacP3Q0h3lbDnFXp3DZdvJsOTlm1s/rr5vicMOGwcsvm41hhChjJU0EjwI/KqUOWi5XBwZZJyRhD/IKili1J52mNStSrYIXsYlHGP7Vam5sVp1R10QQm3iUaav3UaQ1/2tb29bhlh9FRTBtGjz7LOzdazaEeestaNbM1pEJJ3bBRKCUigb2a63XKKUaAiOB/sA8QDaud2K707LIL9Tc3iGMPi1qkltQyIS/dvPJknh+jzULx4P9vXj2hkaEBkovIgArV8Kjj8LatWYs4Msv4ZprbB2VEBdtEUwATv6ltsdM/3wIaAFMBAZYLzRRnm1NygSgUXV/ADzdXHm4ewS9m9dgYVwyUWGVaB4SINNFwZSEePJJ0xIICYEpU+C222RPAFFuXCwRuGqtMyw/D8KUkv4J+Om0tQHCCW1LOoaHqwvhQWduXx0e5Ms9V9WxUVTlTG6u2RbytddMl9Dzz5uaQL6y5bcoXy6aCJRSblrrAszc/xGXcFvhwLYeOkZENT/cXeVT7XnNmwcPPQTx8dC/v0kIshpYlFMX+y+eBvyllPoFM0toOYBSqh4gu5U5sW1JmTQM9rd1GOVPUhIMHgw9e4KrK8yfbzaMlyQgyrELfqrXWr+mlFqEmSU0X2t9slKoC2asQDih9KxcUo7l0qi67C98Sn4+fPYZvPCCmRr68stmXMDzggvwhSgXSrJn8b/nObbDOuEIe7Dt0DEAaRGctGiR2RMgLs7MAvr0U1MoTgg7IR284pL9N2PIyVsEu3eb/v9rrjGtgNmzTVeQJAFhZyQRiEu27dAxqlTwJNDPSbs9jh+H556Dxo3NG/9rr5m6QH36SF0gYZdk5o+4ZFuTMmkY7IStAa3hhx9MddDERLMW4K23oGZNW0cmxBWRFoEgJ7+Q/RnZJTq3oLCInclZNK7uZOMDO3bAtdeaGUFBQbBihdkjQJKAcADSIhC8MieOqav20So0gCFtQundvEaxReL2pB0nr7CIhs4yPpCba6qDvvEGeHubgeCRI83UUCEchLQInFxmTj4/rz9Ai1oBHDmRzxMzY+n76UrSs3LPe36cZaDYKWYMrVoFrVrBSy+ZzWG2bYP775ckIByO1RKBUuorpVSKUmrzacfGKqUOKKViLF83WOvxRcn8vC6RE/mFvNKnKYse68LEoa1JSD/O4In/knIs55zzV+3JwM1FUbeKnw2iLSPZ2WZfgA4dIDMT5s6F77+H4GBbRyaEVVizRfAN0OM8x9/XWrewfM214uOLi9Ba892qfTQPqUhkSEWUUlzXJJivb29D4uETDJ74LweO/LftxB+bkvh+1T5uiaqFh5uDNibnz4emTWHcOLjrLti82awSFsKBWe2/WWu9DMi46InCZlbtySA+JYvb2p25X0D7uoFMvrMNKZm59PxgGb/EHCDuYCaP/bCRVqEBjL2psY0itqK0NBg6FK6/Hjw8YOlSmDgRKla0dWRCWJ0tPtY9qJSKtXQdFbsfn1JqhFJqrVJqbWpqalnG5zSmrtqHv5cbvZudu41km/DK/P5wJ+pV9eOR6TEM+PxvKnq78/nQ1ni6OVAfudYwY4ZZEzB9uqkQGhMDXbrYOjIhykxZJ4LPgLqY/QySgHeLO1FrPVFrHaW1jqpSpUpxp4nLlJaVy7zNSQxoXQtvj/O/sdcO9OWHke15/Nr6VPb1YOKw1lSt4FXGkVrRoUNmZfDgwVC7ttk0/uWXwcuBnqMQJVCm00e11sknf1ZKfQHMKcvHF//5PTaJ/ELNkDa1Lniem6sLD3WP4KHuEWUUWRnQ2mwS89BDZpXw22/DqFHgJrOphXMq0xaBUqr6aRf7AZuLO1eUnqmr9vLz+sQzjv2xOYmIqn5EVHOS9QAnHTwIffuaVcH165tuoCeekCQgnJrV/vqVUtOArkCQUioReBHoqpRqAWggAbMHsrCi/RnZjP11C97urvRsWh1vD1fSs3JZvSeDB7rVs3V4ZUdr+PpreOwxs0jsnXdMK0DWBAhhvUSgtR5ynsOTrPV44vw+WRxPQZEmM6eA32IPMjCqFgu3JlOk4fomTjIvft8+uOceMzX0qqvMpvERDtTVJcQVctDJ4AIgIe04M9cnMrx9GBFV/Zj6714A5m0+REglb5rUcPDVwUVF8Pnn0KQJrFxpykMsWSJJQIizSCJwYB8t2ombi+L+rnW5rW0oGxOP8veuNFbGp9OjSTDKkUsm79wJV18N990HbdvCpk2mPISL/MkLcTb5r3BQ8SlZzI45wLD2tanq70W/ViF4u7sy+oeN5BUW0aOpg3YL5eeb0tDNmpmB4EmTYMECCA+3dWRClFuSCByQ1pqXfjMDxPd2qQtARW93bmpeg4NHc6hSwZNWocWu5bNfq1ZBVBQ89ZQpCxEXB3feKZvFCHERkgjs1O7ULOJTjp33ummr97N8ZxpP39DojF3EbmsXCsC1javh4uJAb45ZWfDII9C+PWRkmC0jf/4Zapy7YloIcS6ZPG2nnpgZS0LacRY+1oVKvh6njicezua13+PoVC+I29qGnnGbZiEBfDCoBe3rBpZ1uNYzbx7ce6+ZGfTAA/D661DBydZGCHGFpEVgh4qKNNuSMkk/nserv2894/iYn2JRSvHmzZHnHQzu27Im1fwdoITCwYMwaJDpAvL2NjuGffyxJAEhLoMkAjt04MgJjucVEh7ky0/rE1mxM41jOfncP3U9K+PTefbGRoRU8rF1mNaRmwvvvw+NGsEvv5jaQDExZu8AIcRlka4hO7TtkBkbeL1fJM/O2sRTP8fi6eZCQno2z97QiMHRF64fZJeKikyV0GefhT17TLnoTz6Bek60OloIK5EWgR3akWwSQWRIRV7vH0ni4RMcyc7nu7vacs9VdRxvfcD27dC1K9x6K/j7w59/mrEBSQJClAppEdihbYeOEVLJGz9PN9rVCWT6iHbUCfKlqiP0/Z8uJ8fUBHr1VfD1NaUh7rhDFoUJUcokEdihHYeO0eC0qqHt6jjQLCAwBeJ+/BHGjIGEBDMo/OGHUK2arSMTwiHJRys7k1dQxK7ULBoEO+DsGK3NKuAOHcybv78/LFxodg6TJCCE1UgisDN70o5TUKQdKxGcTACdO8N110FiInzxhdkxrHt3W0cnhMOTRGBnth3KBHCMRFBUZKaAtm1rEkBCgqkQGh8Pd98tewUIUUYkEdiZ7YeO4eaiqBPkZ+tQLl9REcyaBa1amd3C0tNhwgTYtctUCPX0vPh9CCFKjdUSgVLqK6VUilJq82nHKiulFiildlq+O2DlM+vakXyMOlV88XCzwxxeVAQ//WQSQP/+kJ0Nkyeb6aEjRkgCEMJGrPlu8g3Q46xjTwGLtNYRwCLLZXEJth06RoNgO9tQpqAAvv8eIiNhwAA4cQKmTDHVQYcNk/2ChbAxqyUCrfUyIOOsw32AyZafJwN9rfX4jigrt4DEwydoUM1OuoVycmDiRGjQwGwWrxRMm2YSwNChkgCEKCfK+j+xmtY6yfLzIUDmBF6E1ppFW1OoU8WXIyfyAcp/iyApCT77zGwTmZoKbdrAe+9B796yGEyIcshmH8m01loppYu7Xik1AhgBEBoaWtxpDu+vHancPWUtAH6e5uVqWF5nDMXGwrvvmk/9BQXQqxc8+ih06yabwwhRjpV1IkhWSlXXWicppaoDKcWdqLWeCEwEiIqKKjZhOLqV8Wl4uLrwfK9GrIxPp1BragZ42zqsMy1fbspAzJ9vSkHcey88/LDUAhLCTpR1IvgVGA68afn+Sxk/vt1ZGZ9Oq9oBDG0fxtD2YbYO50wrVsCLL8LixWbl7xtvwMiRUEkmgwlhT6w5fXQa8A/QQCmVqJS6C5MArlVK7QSusVwWxTh8PI+4pEw61A2ydShn2rgRbrzRrASOizP7A+zebfYKliQghN2xWotAaz2kmKuctmbAuD+306JWANc0LtkY+b+70wHoUB62lszPh7lzYdIk+O0384b/9ttme0gfB90ERwgnIfP3ykjS0RN8siQeTzcXfrqvA01rVrzobVbuSsPHw5XmtQLKIMJibN5sFn199x0cOgTBwfD88/DYYxBgw7iEEKVGEkEZWbItFQAfD1dGTFnLrw91wt3VhY8X7WRFfBpNa1akbXhlrm5YlUA/s8L2713ptAmvjLtrGU+5PHHC7AY2fjysWWPm+99wA9x1l/ku8/+FcCjyH11GFm9LoWaANxOGtmbA53/zvy9XkZyZw5ET+USHVWbh1mRmrkukekUvfn2wE4VFmt2pxxkSXYZTZ1NT4YMPTN2f9HRo3NhcHjIEqlYtuziEEGVKEkEp2p2aRY0Ab7zcz6yamZNfyMr4NAa0DqFpzYq8dXMzHpkeQ7s6lXm+V2Oa1KhIUZFm1Z4M7vhmNQ9MXc+A1iEAtC+L8YHkZLMT2GefmdZA377w0ENme0iZ/y+Ew5NEUEo27DtMv/F/4+XuQse6QdzUogY3Na+BUopVezI4kV/I1Q3Np+o+LWrSsV4Qgb4ep/YXdnFRtK8beCpJbDl4lAAfdxpXt+Iq4mPHzAKwceNMArj1VnjmGWjUyHqPKYQod2S9fyn55u8EKni6MTg6lO3Jx3hkegy/xZpqGku2peDp5nLGlpJBfp7n3WS+T4uajLiqDsfzCmlfJxAXFyt8Is/KMm/+9erBSy9Bz55mGui330oSEMIJSYugFKQcy2HupiSGtgvjhd6Neb5XY27+7G9e+GUz7epUZsn2FDrUDcTbo2QbrTx5fQMUlHiaaYkdOmR2/vrwQzMGcM018NprphaQEMJpSYugFExbtZ/8Qs3Q9rUBcHVRjLulOdl5hdwzZR1707NPdQuVhJurC0/f0IjosMqlE+CyZab8c61a8MILZkewf/4x20NKEhDC6UkiuEL5hUVMXbWXLvWrEB7ke+p4vap+PHFdAzbuPwJA1wY2mHWzcaOZ7tmlCyxdCo88Atu2we+/Q7t2ZR+PEKJckq6hK/TnlkOkHMvlrZvDzrnuzk7hLIhLJju/gFqVy3D1bVwcvPKKWQsQEGBWAD/4IHiXs2J1QohyQRLBFfpmZQKhlX3oUr/KOde5uiim3NWGwqIyKp4aG2v6/H/80VQBfeopePJJWQEshLggSQRXYNXudNbuPcyLvRsXO7vn7DUFVvH336by55w54OcHTz9tSkAEloMaRUKIck8SwRX4eHE8QX6eDGljg41ztIYlS8w+AEuWmDf9V14xReCkAqgQ4hJIIrhM6/ZmsCI+jWdvaFQ2n/pPt2WL6fNfuhSqVzeLwkaMMK0BIYS4RJIILtNHi+Kp7OvBbe3KsDVw5IjpAnrvPahQAT7+GO6+G7y8yi4GIYTDkURwGWL2H+GvHamM6dEQHw8r/woLC818/2++gdmzITcX7rwT3nwTqpw7QC2EEJdKEsEl0lrz9rxtBPi4n1pAZhVHjsBXX8Gnn5rdvypXhnvuMaWgW7Sw3uMKIZyOTRKBUioBOAYUAgVa6yhbxHE5ft14kL93pfNKnyb4eZbir09rM/tn7lwz+LtmDRQUQKdO8PrrpiKop2fpPZ4QQljYskXQTWudZsPHv2SZOfm8+vtWImtW5Na2pdQa2LPHfPKfOtX87OoK0dEwejQMHAgtW5bO4wghRDGka+gSvDd/B2lZuUwaHoXrlVQF1RoWL4aPPjL7/yoF3bubSqB9+oC/FUtPCyHEWWyVCDQwXymlgQla64lnn6CUGgGMAAgNtcE8/bPEHcxkyj8J/K9tbZqFXOZK3dxc+P57eP992LQJgoLM4q977zUF4YQQwgZslQg6aa0PKKWqAguUUtu01stOP8GSHCYCREVFlVGNhuJ9vXIP3u6ujL6uwaXfWGv4+Wd49FFITITISNMdNGSITP0UQticTaqPaq0PWL6nALOAclULWWuN1v/lnqzcAn7flESvZjWo6ON+aXcWF2c2fhkwwLQA5s83VUHvuEOSgBCiXCjzRKCU8lVKVTj5M3AdsLms4yiO1pqbPlnJ0z9vOnVszsaDZOcVMjC6hN03RUXw669w3XXQpImp/f/hh2Ym0LXXyj7AQohyxRZdQ9WAWZZtGt2A77XW82wQx3lt2H+ETQeOsunAUW5qXoMO9YKYsXY/9ar60Sq0BGMDcXFmvv/ff0PNmqYW0IgRsvhLCFFulXki0FrvBpqX9eOW1Kz1B/B0c6GqvyfPzd7MR0NasmHfEZ67sdF59xg+paDAlIB+7TVT/mHSJBg6FNwvsStJCCHKmEwfPU1eQRG/xR7k2sbVuCWqFsO/Ws1dk9fg7qro17Jm8TdMTYVBg8xCsCFD4IMPoKoNdiQTQojLIFtVnmbp9hSOZOfTv1VNutSvQq9m1UnOzOWaRtUI9CtmVe+6ddC6tRkHmDzZTA+VJCCEsCOSCE4za8MBAn096Bxh+vNf6NWYlqEB3HNVnXNPLioyVUA7dDCDvytWwLBhZRyxEEJcOekasjianc+irSnc2jYUd1eTH6v6ezHr/o7nnnzwINx+u6kK2qcPfPmlmRoqhBB2SFoEFrNjDpBXWMTNrUKKP6moCCZMgMaNYeVKmDgRZs2SJCCEsGtO3yLQWvPF8t28NW87zWsF0LRmMXV+Nm2C++4zCeDqq+HzzyEiomyDFUIIK3DqRHA8t4DRP27kj82H6NEkmHduaXbuFNH0dHjhBfPGX6mS2SBm2DBZFCaEcBhOmwgyjudxx9er2Xwwk2dvaMTdncP/SwIFBWY/4OnTYeZMyMqC++831UErV7Zp3EIIUdqcMhEkHT3B0Emr2ZeRzYT/teaaxtX+u3LLFujfH3bsMAvD+vSBMWOgaVPbBSyEEFbkdIkgMyefWz7/hyPZ+Uy5sw3t6gT+d+Xs2WY1sK8vzJgBvXuDt7ftghVCiDLgdLOGvl6RQOLhE3x9R/R/SSA5GR56CPr1g0aNzCKxgQMlCQghnIJTJYKjJ/KZtGI31zauRnRYZcjMhGefhTp14LPPzDjAsmWmWJwQQjgJp+oa+nrlHjJzCnjk6nqm62fUKEhKgsGDzUBw/fq2DlEIIcqc0yQC0xrYwy3VXWh650CzKrhVKzMu0KZc7YsjhBBlymkSwcRlu2i2bS1vLPwATmTDxx+bBWKurrYOTQghbMrhE0H8gQxmfvwDwQt+59sNv+PSqJFZG9Coka1DE0KIcsEmiUAp1QP4EHAFvtRav2mNx/nnjkdp+v1Enso7QZGrG3r47fDJx2Z6qBBCCMAGiUAp5Qp8ClwLJAJrlFK/aq3jSvuxPGuHsq3LDTS4fSD+vXqAfzF1hIQQwonZokXQBoi3bFmJUmo60Aco9UTQauxjwGOlfbdCCOFQbLGOoCaw/7TLiZZjQgghbKDcLihTSo1QSq1VSq1NTU21dThCCOGwbJEIDgC1TrscYjl2Bq31RK11lNY6qkqVKmUWnBBCOBtbJII1QIRSKlwp5QEMBn61QRxCCCGwwWCx1rpAKfUg8Cdm+uhXWustZR2HEEIIwybrCLTWc4G5tnhsIYQQZyq3g8VCCCHKhiQCIYRwckprbesYLkoplQrsvcybBwFppRhOeSLPzT7Jc7M/9vq8amutLzrt0i4SwZVQSq3VWkfZOg5rkOdmn+S52R9HfV4nSdeQEEI4OUkEQgjh5JwhEUy0dQBWJM/NPslzsz+O+rwAJxgjEEIIcWHO0CIQQghxAQ6dCJRSPZRS25VS8Uqpp2wdz+VSStVSSi1RSsUppbYopR6xHK+slFqglNpp+V7J1rFeLqWUq1Jqg1JqjuVyuFJqleW1m2GpS2V3lFIBSqmZSqltSqmtSqn2jvK6KaVGWf4eNyulpimlvOz1dVNKfaWUSlFKbT7t2HlfJ2V8ZHmOsUqpVraLvHQ4bCI4bSe0nkBjYIhSqrFto7psBcDjWuvGQDvgActzeQpYpLWOABZZLturR4Ctp11+C3hfa10POAzcZZOortyHwDytdUOgOeY52v3rppSqCTwMRGmtm2Lqhg3Gfl+3b4AeZx0r7nXqCURYvkYAn5VRjFbjsImA03ZC01rnASd3QrM7WuskrfV6y8/HMG8mNTHPZ7LltMlAX9tEeGWUUiHAjcCXlssKuBqYaTnFLp+bUqoicBUwCUBrnae1PoKDvG6YWmXeSik3wAdIwk5fN631MiDjrMPFvU59gCna+BcIUEpVL5tIrcORE4FD7oSmlAoDWgKrgGpa6yTLVYeAajYK60p9ADwJFFkuBwJHtNYFlsv2+tqFA6nA15Zury+VUr44wOumtT4AjAP2YRLAUWAdjvG6nVTc6+Rw7y2OnAgcjlLKD/gJeFRrnXn6ddpM/7K7KWBKqV5AitZ6na1jsQI3oBXwmda6JXCcs7qB7Ph1q4T5ZBwO1AB8ObdrxWHY6+tUUo6cCEq0E5q9UEq5Y5LAVK31z5bDySebpJbvKbaK7wp0BG5SSiVguu+uxvSrB1i6HMB+X7tEIFFrvcpyeSYmMTjC63YNsEdrnaq1zgd+xryWjvC6nVTc6+RQ7y3g2InAYXZCs/SZTwK2aq3fO+2qX4Hhlp+HA7+UdWxXSmv9tNY6RGsdhnmNFmutbwOWAAMsp9nrczsE7FdKNbAc6g7E4QCvG6ZLqJ1Sysfy93nyudn963aa4l6nX4FhltlD7YCjp3Uh2SettcN+ATcAO4BdwLO2jucKnkcnTLM0FoixfN2A6UtfBOwEFgKVbR3rFT7PrsAcy891gNVAPPAj4Gnr+C7zObUA1lpeu9lAJUd53YCXgG3AZuBbwNNeXzdgGmasIx/TkruruNcJUJgZibuATZiZUzZ/DlfyJSuLhRDCyTly15AQQogSkEQghBBOThKBEEI4OUkEQgjh5CQRCCGEk5NEIByaUqpQKRVz2tcFC7wppe5VSg0rhcdNUEoFXcbtrldKvWSpfPnHlcYhREm4XfwUIezaCa11i5KerLX+3JrBlEBnzKKszsAKG8cinIS0CIRTsnxif1sptUkptVopVc9yfKxSarTl54cte0DEKqWmW45VVkrNthz7VynVzHI8UCk131Kf/0vMoqOTj/U/y2PEKKUmWEqknx3PIKVUDKa08wfAF8AdSim7XA0v7IskAuHovM/qGhp02nVHtdaRwCeYN9+zPQW01Fo3A+61HHsJ2GA59gwwxXL8RWCF1roJMAsIBVBKNQIGAR0tLZNC4LazH0hrPQNTVXazJaZNlse+6UqevBAlIV1DwtFdqGto2mnf3z/P9bHAVKXUbEx5CDDlPm4G0FovtrQE/DH7DvS3HP9dKXXYcn53oDWwxpTkwZvii8zVB3ZbfvbVZu8JIaxOEoFwZrqYn0+6EfMG3xt4VikVeRmPoYDJWuunL3iSUmuBIMBNKRUHVLd0FT2ktV5+GY8rRIlJ15BwZoNO+/7P6VcopVyAWlrrJcAYoCLgByzH0rWjlOoKpGmzN8Qy4FbL8Z6Y4nJgioYJpuIAAADGSURBVJYNUEpVtVxXWSlV++xAtNZRwO+YGv9vY4oktpAkIMqCtAiEo/O2fLI+aZ7W+uQU0kpKqVggFxhy1u1cge8s200q4COt9RGl1FjgK8vtsvmvTPFLwDSl1Bbgb0yZZrTWcUqp54D5luSSDzwA7D1PrK0wg8X3A++d53ohrEKqjwqnZNkIJ0prnWbrWISwNekaEkIIJyctAiGEcHLSIhBCCCcniUAIIZycJAIhhHBykgiEEMLJSSIQQggnJ4lACCGc3P8B9ss2plNLlQIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the scores\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores, label='DDPG')\n",
    "plt.plot(np.arange(len(scores)), avgs, c='r', label='moving avg')\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.legend(loc='upper left');\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}