Continuous_Control.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Github on Jupyter Notebooks\n",
    "\n",
    "---\n",
    "\n",
    "Only needed for udacity workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The following paths are ignored by one of your .gitignore files:\n",
      "unity-environment.log\n",
      "Use -f if you really want to add them.\n",
      "On branch master\n",
      "Your branch is ahead of 'origin/master' by 1 commit.\n",
      "  (use \"git push\" to publish your local commits)\n",
      "\n",
      "nothing to commit, working tree clean\n",
      "To https://github.com/tomalbrecht/udacity_drlnd_p1.git\n",
      " ! [rejected]        master -> master (fetch first)\n",
      "error: failed to push some refs to 'https://tomalbrecht:D7zO0zgyWJDDYwYDeK@github.com/tomalbrecht/udacity_drlnd_p1.git'\n",
      "hint: Updates were rejected because the remote contains work that you do\n",
      "hint: not have locally. This is usually caused by another repository pushing\n",
      "hint: to the same ref. You may want to first integrate the remote changes\n",
      "hint: (e.g., 'git pull ...') before pushing again.\n",
      "hint: See the 'Note about fast-forwards' in 'git push --help' for details.\n"
     ]
    }
   ],
   "source": [
    "import getpass # only used for git password\n",
    "\n",
    "#!git initial setup\n",
    "!git config --global user.email \"thomas.albrecht@gmx.net\"\n",
    "!git config --global user.name \"Thomas Albrecht\"\n",
    "!git config --global push.default matching\n",
    "\n",
    "# add all changed files to this commit along with a message and credentials and push it\n",
    "!git add *\n",
    "!git add -u\n",
    "message =  input('git message: ')\n",
    "!git commit -m '{message}'\n",
    "username = input('git username: ')\n",
    "password = getpass.getpass('git password: ')\n",
    "#!git fetch 'https://{username}:{password}@github.com/tomalbrecht/udacity_drlnd_p2.git'\n",
    "#!git reset --hard origin/master\n",
    "!git push 'https://{username}:{password}@github.com/tomalbrecht/udacity_drlnd_p1.git'\n",
    "password = ''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous Control\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the second project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893) program.\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing the necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from unityagents import UnityEnvironment\n",
    "import numpy as np\n",
    "#import pandas as pd\n",
    "#import random\n",
    "import torch\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "from ddpg_agent import Agent\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Reacher.app\"`\n",
    "- **Windows** (x86): `\"path/to/Reacher_Windows_x86/Reacher.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Reacher_Windows_x86_64/Reacher.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Reacher_Linux/Reacher.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Reacher_Linux/Reacher.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Reacher_Linux_NoVis/Reacher.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name='Reacher_Linux_NoVis_20/Reacher.x86')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "In this environment, a double-jointed arm can move to target locations. A reward of `+0.1` is provided for each step that the agent's hand is in the goal location. Thus, the goal of your agent is to maintain its position at the target location for as many time steps as possible.\n",
    "\n",
    "The observation space consists of `33` variables corresponding to position, rotation, velocity, and angular velocities of the arm.  Each action is a vector with four numbers, corresponding to torque applicable to two joints.  Every entry in the action vector must be a number between `-1` and `1`.\n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726624e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Train the Agent with DDPG\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode 1\tTimestep 999\tScore: 0.96\tmin: 0.00\tmax: 2.37\n",
      "Episode 2\tTimestep 999\tScore: 2.61\tmin: 0.62\tmax: 4.89\n",
      "Episode 3\tTimestep 999\tScore: 4.95\tmin: 2.29\tmax: 8.70\n",
      "Episode 4\tTimestep 999\tScore: 7.22\tmin: 2.86\tmax: 14.39\n",
      "Episode 5\tTimestep 999\tScore: 8.91\tmin: 4.34\tmax: 17.46\n",
      "Episode 6\tTimestep 999\tScore: 10.49\tmin: 4.14\tmax: 19.53\n",
      "Episode 7\tTimestep 999\tScore: 19.34\tmin: 8.83\tmax: 32.67\n",
      "Episode 8\tTimestep 999\tScore: 23.34\tmin: 8.23\tmax: 36.20\n",
      "Episode 9\tTimestep 999\tScore: 29.26\tmin: 12.92\tmax: 38.89\n",
      "Episode 10\tTimestep 999\tScore: 34.92\tmin: 10.48\tmax: 39.17\n",
      "Episode 11\tTimestep 999\tScore: 36.82\tmin: 34.15\tmax: 39.49\n",
      "Episode 12\tTimestep 999\tScore: 36.90\tmin: 29.84\tmax: 39.40\n",
      "Episode 13\tTimestep 999\tScore: 35.91\tmin: 28.97\tmax: 39.44\n",
      "Episode 14\tTimestep 999\tScore: 34.80\tmin: 18.24\tmax: 39.06\n",
      "Episode 15\tTimestep 153\tScore: 4.60\tmin: 0.80\tmax: 5.84"
     ]
    }
   ],
   "source": [
    "def ddpg(n_episodes=2000, max_t=700):\n",
    "    \"\"\" Deep Deterministic Policy Gradients Methods\n",
    "    source: partially from https://github.com/udacity/deep-reinforcement-learning/tree/master/ddpg-bipedal\n",
    "    source: partially from my 1st project in this drlnd\n",
    "\n",
    "    Params\n",
    "    ======\n",
    "        n_episodes (int): maximum number of training episodes\n",
    "        max_t (int): maximum number of timesteps per episode\n",
    "\n",
    "        reminder:\n",
    "        epsilon = greedy factor for exploration vs. exploitation\n",
    "        alpha = something like learnrate. defines the step size when an update occurs\n",
    "        gamma = discount factor for rewards\n",
    "\n",
    "    \"\"\"    \n",
    "    \n",
    "    #scores = np.zeros(num_agents)      # list containing scores from each episode, per agent\n",
    "    scores_window = deque(maxlen=100)  # last 100 scores\n",
    "    agents = []                        # list of agents, depending on agents needed in the current environment\n",
    "    scores_episodes = []\n",
    "    for _ in range(num_agents):\n",
    "        agents.append(Agent(state_size, \n",
    "                            action_size, \n",
    "                            random_seed=0,\n",
    "                            buffer_size=int(1e6),  # replay buffer size (default: int(1e6))\n",
    "                            batch_size=1024,       # minibatch size (default: 128)\n",
    "                            gamma=0.98,            # discount factor (default: 0.99)\n",
    "                            tau=1e-3,              # for soft update of target parameters (default: 1e-3)\n",
    "                            lr_actor=1e-3,         # learning rate of the actor (default: 1e-3)\n",
    "                            lr_critic=1e-4,        # learning rate of the critic (default: 1e-4)\n",
    "                            #weight_decay=0.       # L2 weight decay (default: 3e-4)\n",
    "                            weight_decay=1e-9      # L2 weight decay (default: 3e-4)\n",
    "                            )\n",
    "                       )\n",
    "\n",
    "    for i_episode in range(1, n_episodes+1):\n",
    "        env_info = env.reset(train_mode=True)[brain_name]\n",
    "        states = env_info.vector_observations        \n",
    "        for agent in agents:\n",
    "            agent.reset()\n",
    "        \n",
    "        scores = np.zeros(num_agents) \n",
    "        \n",
    "        for t in range(max_t):\n",
    "            # interact with the enviornment\n",
    "            actions = np.array([agents[a].act(states[a]) for a in range(num_agents)])\n",
    "            env_info = env.step(actions)[brain_name]\n",
    "            next_states = env_info.vector_observations\n",
    "            rewards = env_info.rewards\n",
    "            dones = env_info.local_done\n",
    "            \n",
    "            for a in range(num_agents):\n",
    "                agents[a].step(t, states[a], actions[a], rewards[a], next_states[a], dones[a])\n",
    "                \n",
    "            states = next_states\n",
    "            scores += rewards\n",
    "            \n",
    "            if t % 10:\n",
    "                print('\\rEpisode {}\\tTimestep {}\\tScore: {:.2f}\\tmin: {:.2f}\\tmax: {:.2f}'\n",
    "                      .format(i_episode, t, np.mean(scores), np.min(scores), np.max(scores)), end=\"\")\n",
    "            \n",
    "            if np.any(dones):\n",
    "                break     \n",
    "        score = np.mean(scores)\n",
    "        scores_window.append(score)\n",
    "        scores_episode.append(score)\n",
    "        #scores.append(score)\n",
    "        \n",
    "        print('\\nEpisode {}\\tAverage Score: {:.2f}\\tScore: {:.2f}'.format(i_episode, np.mean(scores_window), score), end=\"\")\n",
    "        \n",
    "        if np.mean(scores_window) >= 30.0 and i_episode > 99:\n",
    "            print('\\nEnvironment solved in {:d} episodes!\\tAverage Score: {:.2f}'.format(i_episode-100, np.mean(scores_window)))\n",
    "            torch.save(Agent.actor_local.state_dict(), 'checkpoint_actor.pth')\n",
    "            torch.save(Agent.critic_local.state_dict(), 'checkpoint_critic.pth')\n",
    "            break\n",
    "            \n",
    "    return scores\n",
    "\n",
    "scores = ddpg(n_episodes=2000, max_t=1000)\n",
    "\n",
    "fig = plt.figure()\n",
    "#ax = fig.add_subplot(111)\n",
    "plt.plot(np.arange(len(scores)), scores)\n",
    "plt.ylabel('Score')\n",
    "plt.xlabel('Episode #')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Watch a Smart Agent!\n",
    "\n",
    "In the next code cell, you will load the trained weights from file to watch a smart agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'agent' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-33a837050f96>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mactor_local\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'checkpoint_actor.pth'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcritic_local\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'checkpoint_critic.pth'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mstate\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0magent\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'agent' is not defined"
     ]
    }
   ],
   "source": [
    "agent.actor_local.load_state_dict(torch.load('checkpoint_actor.pth'))\n",
    "agent.critic_local.load_state_dict(torch.load('checkpoint_critic.pth'))\n",
    "\n",
    "state = env.reset()\n",
    "agent.reset()   \n",
    "        \n",
    "while True:\n",
    "    actions = np.array([agents[i].act(states[i]) for i in range(num_agents)])\n",
    "\n",
    "    env_info = env.step(actions)[brain_name]\n",
    "    next_states = env_info.vector_observations\n",
    "    rewards = env_info.rewards\n",
    "    dones = env_info.local_done\n",
    "\n",
    "    states = next_states\n",
    "    scores += rewards\n",
    "\n",
    "    print('\\rScore: {:.2f}\\tmin: {:.2f}\\tmax: {:.2f}'\n",
    "              .format(np.mean(scores), np.min(scores), np.max(scores)), end=\"\") \n",
    "    \n",
    "    if np.any(dones):\n",
    "        break\n",
    "        \n",
    "print(\"\\nScores: {}\".format(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "drlnd",
   "language": "python",
   "name": "drlnd"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}