getting-started.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Navigation\n",
    "\n",
    "---\n",
    "\n",
    "In this notebook, you will learn how to use the Unity ML-Agents environment for the first project of the [Deep Reinforcement Learning Nanodegree](https://www.udacity.com/course/deep-reinforcement-learning-nanodegree--nd893).\n",
    "\n",
    "### 1. Start the Environment\n",
    "\n",
    "We begin by importing some necessary packages.  If the code cell below returns an error, please revisit the project instructions to double-check that you have installed [Unity ML-Agents](https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Installation.md) and [NumPy](http://www.numpy.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: unityagents in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (0.4.0)\n",
      "Requirement already satisfied: numpy in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (1.15.4)\n",
      "Requirement already satisfied: jupyter in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (1.0.0)\n",
      "Requirement already satisfied: grpcio==1.11.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (1.11.0)\n",
      "Requirement already satisfied: pytest>=3.2.2 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (4.0.2)\n",
      "Requirement already satisfied: pyyaml in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (3.13)\n",
      "Requirement already satisfied: tensorflow==1.7.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (1.7.1)\n",
      "Requirement already satisfied: Pillow>=4.2.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (5.3.0)\n",
      "Requirement already satisfied: matplotlib in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (3.0.2)\n",
      "Requirement already satisfied: protobuf==3.5.2 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (3.5.2)\n",
      "Requirement already satisfied: docopt in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from unityagents) (0.6.2)\n",
      "Requirement already satisfied: notebook in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from jupyter->unityagents) (5.7.3)\n",
      "Requirement already satisfied: nbconvert in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from jupyter->unityagents) (5.4.0)\n",
      "Requirement already satisfied: ipywidgets in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from jupyter->unityagents) (7.4.2)\n",
      "Requirement already satisfied: qtconsole in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from jupyter->unityagents) (4.4.3)\n",
      "Requirement already satisfied: jupyter-console in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from jupyter->unityagents) (6.0.0)\n",
      "Requirement already satisfied: ipykernel in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from jupyter->unityagents) (5.1.0)\n",
      "Requirement already satisfied: six>=1.5.2 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from grpcio==1.11.0->unityagents) (1.12.0)\n",
      "Requirement already satisfied: more-itertools>=4.0.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from pytest>=3.2.2->unityagents) (4.3.0)\n",
      "Requirement already satisfied: atomicwrites>=1.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from pytest>=3.2.2->unityagents) (1.2.1)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from pytest>=3.2.2->unityagents) (18.2.0)\n",
      "Requirement already satisfied: setuptools in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages/setuptools-28.8.0-py3.6.egg (from pytest>=3.2.2->unityagents) (28.8.0)\n",
      "Requirement already satisfied: py>=1.5.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from pytest>=3.2.2->unityagents) (1.7.0)\n",
      "Requirement already satisfied: pluggy>=0.7 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from pytest>=3.2.2->unityagents) (0.8.0)\n",
      "Requirement already satisfied: tensorboard<1.8.0,>=1.7.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorflow==1.7.1->unityagents) (1.7.0)\n",
      "Requirement already satisfied: absl-py>=0.1.6 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorflow==1.7.1->unityagents) (0.6.1)\n",
      "Requirement already satisfied: gast>=0.2.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorflow==1.7.1->unityagents) (0.2.0)\n",
      "Requirement already satisfied: wheel>=0.26 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorflow==1.7.1->unityagents) (0.32.3)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorflow==1.7.1->unityagents) (1.1.0)\n",
      "Requirement already satisfied: astor>=0.6.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorflow==1.7.1->unityagents) (0.7.1)\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from matplotlib->unityagents) (1.0.1)\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from matplotlib->unityagents) (2.3.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from matplotlib->unityagents) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from matplotlib->unityagents) (2.7.5)\n",
      "Requirement already satisfied: nbformat in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (4.4.0)\n",
      "Requirement already satisfied: prometheus-client in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (0.5.0)\n",
      "Requirement already satisfied: Send2Trash in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (1.5.0)\n",
      "Requirement already satisfied: pyzmq>=17 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (17.1.2)\n",
      "Requirement already satisfied: terminado>=0.8.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (0.8.1)\n",
      "Requirement already satisfied: jupyter-core>=4.4.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (4.4.0)\n",
      "Requirement already satisfied: ipython-genutils in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (0.2.0)\n",
      "Requirement already satisfied: traitlets>=4.2.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (4.3.2)\n",
      "Requirement already satisfied: tornado>=4 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (5.1.1)\n",
      "Requirement already satisfied: jupyter-client>=5.2.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (5.2.4)\n",
      "Requirement already satisfied: jinja2 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from notebook->jupyter->unityagents) (2.10)\n",
      "Requirement already satisfied: pandocfilters>=1.4.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbconvert->jupyter->unityagents) (1.4.2)\n",
      "Requirement already satisfied: testpath in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbconvert->jupyter->unityagents) (0.4.2)\n",
      "Requirement already satisfied: mistune>=0.8.1 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbconvert->jupyter->unityagents) (0.8.4)\n",
      "Requirement already satisfied: bleach in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbconvert->jupyter->unityagents) (1.5.0)\n",
      "Requirement already satisfied: pygments in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbconvert->jupyter->unityagents) (2.3.0)\n",
      "Requirement already satisfied: entrypoints>=0.2.2 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbconvert->jupyter->unityagents) (0.2.3)\n",
      "Requirement already satisfied: defusedxml in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbconvert->jupyter->unityagents) (0.5.0)\n",
      "Requirement already satisfied: ipython>=4.0.0; python_version >= \"3.3\" in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from ipywidgets->jupyter->unityagents) (5.0.0)\n",
      "Requirement already satisfied: widgetsnbextension~=3.4.0 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from ipywidgets->jupyter->unityagents) (3.4.2)\n",
      "Collecting prompt-toolkit<2.1.0,>=2.0.0 (from jupyter-console->jupyter->unityagents)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Using cached https://files.pythonhosted.org/packages/d1/e6/adb3be5576f5d27c6faa33f1e9fea8fe5dbd9351db12148de948507e352c/prompt_toolkit-2.0.7-py3-none-any.whl\n",
      "Requirement already satisfied: markdown>=2.6.8 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorboard<1.8.0,>=1.7.0->tensorflow==1.7.1->unityagents) (3.0.1)\n",
      "Requirement already satisfied: html5lib==0.9999999 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorboard<1.8.0,>=1.7.0->tensorflow==1.7.1->unityagents) (0.9999999)\n",
      "Requirement already satisfied: werkzeug>=0.11.10 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from tensorboard<1.8.0,>=1.7.0->tensorflow==1.7.1->unityagents) (0.14.1)\n",
      "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from nbformat->notebook->jupyter->unityagents) (2.6.0)\n",
      "Requirement already satisfied: ptyprocess; os_name != \"nt\" in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from terminado>=0.8.1->notebook->jupyter->unityagents) (0.6.0)\n",
      "Requirement already satisfied: decorator in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from traitlets>=4.2.1->notebook->jupyter->unityagents) (4.3.0)\n",
      "Requirement already satisfied: MarkupSafe>=0.23 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from jinja2->notebook->jupyter->unityagents) (1.1.0)\n",
      "Requirement already satisfied: simplegeneric>0.8 in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets->jupyter->unityagents) (0.8.1)\n",
      "Requirement already satisfied: pickleshare in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets->jupyter->unityagents) (0.7.5)\n",
      "Requirement already satisfied: pexpect; sys_platform != \"win32\" in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from ipython>=4.0.0; python_version >= \"3.3\"->ipywidgets->jupyter->unityagents) (4.6.0)\n",
      "Requirement already satisfied: wcwidth in /home/ammi/Development/deep-monkey/lib/python3.6/site-packages (from prompt-toolkit<2.1.0,>=2.0.0->jupyter-console->jupyter->unityagents) (0.1.7)\n",
      "\u001b[31mipython 5.0.0 has requirement prompt-toolkit<2.0.0,>=1.0.3, but you'll have prompt-toolkit 2.0.7 which is incompatible.\u001b[0m\n",
      "Installing collected packages: prompt-toolkit\n",
      "  Found existing installation: prompt-toolkit 1.0.15\n",
      "    Uninstalling prompt-toolkit-1.0.15:\n",
      "      Successfully uninstalled prompt-toolkit-1.0.15\n",
      "Successfully installed prompt-toolkit-2.0.7\n"
     ]
    }
   ],
   "source": [
    "!python -m pip install unityagents numpy\n",
    "\n",
    "from unityagents import UnityEnvironment\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will start the environment!  **_Before running the code cell below_**, change the `file_name` parameter to match the location of the Unity environment that you downloaded.\n",
    "\n",
    "- **Mac**: `\"path/to/Banana.app\"`\n",
    "- **Windows** (x86): `\"path/to/Banana_Windows_x86/Banana.exe\"`\n",
    "- **Windows** (x86_64): `\"path/to/Banana_Windows_x86_64/Banana.exe\"`\n",
    "- **Linux** (x86): `\"path/to/Banana_Linux/Banana.x86\"`\n",
    "- **Linux** (x86_64): `\"path/to/Banana_Linux/Banana.x86_64\"`\n",
    "- **Linux** (x86, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86\"`\n",
    "- **Linux** (x86_64, headless): `\"path/to/Banana_Linux_NoVis/Banana.x86_64\"`\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Banana.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Banana.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:unityagents:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of External Brains : 1\n",
      "        Lesson number : 0\n",
      "        Reset Parameters :\n",
      "\t\t\n",
      "Unity brain name: BananaBrain\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space type: continuous\n",
      "        Vector Observation space size (per agent): 37\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: discrete\n",
      "        Vector Action space size (per agent): 4\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "env = UnityEnvironment(file_name=\"../bin/Banana_Linux/Banana.x86_64\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "The simulation contains a single agent that navigates a large environment.  At each time step, it has four actions at its disposal:\n",
    "- `0` - walk forward \n",
    "- `1` - walk backward\n",
    "- `2` - turn left\n",
    "- `3` - turn right\n",
    "\n",
    "The state space has `37` dimensions and contains the agent's velocity, along with ray-based perception of objects around agent's forward direction.  A reward of `+1` is provided for collecting a yellow banana, and a reward of `-1` is provided for collecting a blue banana. \n",
    "\n",
    "Run the code cell below to print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 1\n",
      "Number of actions: 4\n",
      "States look like: [1.         0.         0.         0.         0.84408134 0.\n",
      " 0.         1.         0.         0.0748472  0.         1.\n",
      " 0.         0.         0.25755    1.         0.         0.\n",
      " 0.         0.74177343 0.         1.         0.         0.\n",
      " 0.25854847 0.         0.         1.         0.         0.09355672\n",
      " 0.         1.         0.         0.         0.31969345 0.\n",
      " 0.        ]\n",
      "States have length: 37\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents in the environment\n",
    "print('Number of agents:', len(env_info.agents))\n",
    "\n",
    "# number of actions\n",
    "action_size = brain.vector_action_space_size\n",
    "print('Number of actions:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "state = env_info.vector_observations[0]\n",
    "print('States look like:', state)\n",
    "state_size = len(state)\n",
    "print('States have length:', state_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment\n",
    "\n",
    "In the next code cell, you will learn how to use the Python API to control the agent and receive feedback from the environment.\n",
    "\n",
    "Once this cell is executed, you will watch the agent's performance, if it selects an action (uniformly) at random with each time step.  A window should pop up that allows you to observe the agent, as it moves through the environment.  \n",
    "\n",
    "Of course, as part of the project, you'll have to change the code so that the agent is able to use its experience to gradually choose better actions when interacting with the environment!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.0\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=False)[brain_name] # reset the environment\n",
    "state = env_info.vector_observations[0]            # get the current state\n",
    "score = 0                                          # initialize the score\n",
    "while True:\n",
    "    action = np.random.randint(action_size)        # select an action\n",
    "    env_info = env.step(action)[brain_name]        # send the action to the environment\n",
    "    next_state = env_info.vector_observations[0]   # get the next state\n",
    "    reward = env_info.rewards[0]                   # get the reward\n",
    "    done = env_info.local_done[0]                  # see if episode has finished\n",
    "    score += reward                                # update the score\n",
    "    state = next_state                             # roll over the state to next time step\n",
    "    if done:                                       # exit loop if episode finished\n",
    "        break\n",
    "    \n",
    "print(\"Score: {}\".format(score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When finished, you can close the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. It's Your Turn!\n",
    "\n",
    "Now it's your turn to train your own agent to solve the environment!  When training the environment, set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}