exp/6_atari.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Atari\n",
    "\n",
    "`通过这个实验，了解Deep-Q Learning。`\n",
    "\n",
    "所有的实验源代码都在`lib`目录下，来自[dennybritz](https://github.com/dennybritz/reinforcement-learning)，这里只做解读和归总。\n",
    "\n",
    "## 实验目录\n",
    "\n",
    "- [Gridworld](https://applenob.github.io/gridworld.html)：对应MDP的Dynamic Programming\n",
    "- [Blackjack](https://applenob.github.io/black_jack.html)：对应Model Free的Monte Carlo的Planning和Controlling\n",
    "- [Windy Gridworld](https://applenob.github.io/windy_gridworld.html)：对应Model Free的Temporal Difference的On-Policy Controlling，SARSA。\n",
    "- [Cliff Walking](https://applenob.github.io/cliff_walking.html)：对应Model Free的Temporal Difference的Off-Policy Controlling，Q-learning。\n",
    "- [Mountain Car](https://applenob.github.io/mountain_car.html)：对应Q-Learning with Linear Function Approximation。\n",
    "- [Atari](https://applenob.github.io/atari.html)：对应Deep-Q Learning。\n",
    "\n",
    "\n",
    "## 本文目录\n",
    "\n",
    "- [问题介绍](#问题介绍)\n",
    "- [Deep Q-Networks](#Deep-Q-Networks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 问题介绍\n",
    "\n",
    "这是一个常见的手机游戏，在游戏开始的时候fire出一个球，去把上方的砖块一块块击碎，每击碎一块加一分。球会一直按照直线运动，碰到边界或者砖块会反弹。如果球到了下方的边界，则损失一条命，一共五条命。agent需要在球落到下方的边界之前，赶到那个地方，将其反弹。\n",
    "\n",
    "- 动作集：noop（空操作），fire，left，right。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import gym\n",
    "from gym.wrappers import Monitor\n",
    "import itertools\n",
    "import numpy as np\n",
    "import os\n",
    "import random\n",
    "import sys\n",
    "import psutil\n",
    "import tensorflow as tf\n",
    "\n",
    "from lib import plotting\n",
    "from collections import deque, namedtuple\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用之前，要先单独安装atari：`pip install gym[atari]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-10-04 17:08:31,130] Making new env: Breakout-v0\n"
     ]
    }
   ],
   "source": [
    "env = gym.envs.make(\"Breakout-v0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Action space size: 4\n"
     ]
    }
   ],
   "source": [
    "# Atari Actions: 0 (noop), 1 (fire), 2 (left) and 3 (right) are valid actions\n",
    "VALID_ACTIONS = [0, 1, 2, 3]\n",
    "print(\"Action space size: {}\".format(env.action_space.n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation space shape: (210, 160, 3)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADntJREFUeJzt3X/sVfV9x/Hna1hNRruI9UcM4ABH2+myUUscmdN0c7VI\nmqJL2kGWyjYzNJGkjS4Z1mQjS5psXcGk2UaDkRQXC7pRK1mshbCmZtmwgkWEIgqU1q8QmLiIw6YO\neO+P8/mm1y/fy/dy3+f2nnt9PZKbe+/nnnPP+wRefM49nPu+igjMrHu/1O8CzAadQ2SW5BCZJTlE\nZkkOkVmSQ2SW1LMQSZovaZ+k/ZKW92o7Zv2mXvw/kaRJwMvAJ4AR4DlgcUT8sPaNmfVZr2ai64H9\nEXEwIt4BNgALe7Qts766oEfvOxV4teX5CPDb7RaW5MsmrIlej4jLJlqoVyHSOGPvCoqkpcDSHm3f\nrA4/7mShXoVoBJje8nwacLh1gYhYA6wBz0Q22Hr1meg5YLakmZIuBBYBm3q0LbO+6slMFBGnJC0D\nvgNMAtZGxJ5ebMus33pyivu8i2jg4dyqVavOe51777039R5j16/rPbKaUMNYY2vq0TZ3RMTciRby\nFQtmSb06sTB0ejFL9GO2q8MvYqYZJJ6JzJI8E9l5m2j2e6/NVJ6JzJI8E9mEJppZ+vG5rEk8E5kl\neSbqUB3/2jblPQZhm4PEM5FZkkNkluTLfsza82U/Zr8IjTixMG3atPfcf9BZ83X6d9IzkVmSQ2SW\n5BCZJTlEZkldh0jSdEnflbRX0h5Jny/jKyS9JmlnuS2or1yz5smcnTsF3BcRz0v6ALBD0pby2oMR\n8ZV8eWbN13WIIuIIcKQ8fkvSXqqmjWbvKbV8JpI0A/go8GwZWiZpl6S1kqbUsQ2zpkqHSNL7gY3A\nFyLiBLAauBqYQzVTrWyz3lJJ2yVtP3nyZLYMs75JhUjS+6gC9GhEfBMgIo5GxOmIOAM8RNXc/iwR\nsSYi5kbE3MmTJ2fKMOurzNk5AQ8DeyNiVcv4lS2L3Q7s7r48s+bLnJ27Afgc8KKknWXsi8BiSXOo\nGtgfAu5KVWjWcJmzc//B+L/+8FT35ZgNHl+xYJbUiK9CTMRfk7BeqKt3hGcisySHyCzJITJLcojM\nkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLSn+fSNIh4C3gNHAqIuZK\nugR4DJhB9RXxz0bE/2S3ZdZEdc1EvxcRc1p+VWw5sDUiZgNby3OzodSrw7mFwLryeB1wW4+2Y9Z3\ndYQogM2SdkhaWsauKG2GR9sNX17DdswaqY4eCzdExGFJlwNbJL3UyUolcEsBpkxxp2EbXOmZKCIO\nl/tjwBNUHU+PjjZxLPfHxlnPHVBtKGTbCE8uP6uCpMnALVQdTzcBS8piS4AnM9sxa7Ls4dwVwBNV\nR2EuAL4REU9Leg54XNKdwE+AzyS3Y9ZYqRBFxEHgt8YZPw7cnHlvs0HhKxbMkgaiA+q2+fP7XYIN\nof+s6X08E5klOURmSQ6RWZJDZJbkEJklDcTZuTO/dqLfJZi15ZnILMkhMktyiMySHCKzJIfILMkh\nMksaiFPcb/zK2/0uwawtz0RmSQ6RWVLXh3OSPkzV5XTULOCvgIuBPwf+u4x/MSKe6rpCs4brOkQR\nsQ+YAyBpEvAaVbefPwUejIiv1FKhWcPVdTh3M3AgIn5c0/uZDYy6zs4tAta3PF8m6Q5gO3Bftpn9\nGx95J7O62fher+dt0jORpAuBTwP/UoZWA1dTHeodAVa2WW+ppO2Stp88eTJbhlnf1HE4dyvwfEQc\nBYiIoxFxOiLOAA9RdUQ9izug2rCoI0SLaTmUG20fXNxO1RHVbGilPhNJ+mXgE8BdLcNfljSH6tci\nDo15zWzoZDugvg18cMzY51IVmQ2Ygbh27htnrup3CTaEbqnpfXzZj1mSQ2SW5BCZJTlEZkkOkVnS\nQJyde2fDin6XYMPolnp+XMUzkVmSQ2SW5BCZJTlEZkkOkVmSQ2SWNBCnuP/96Xn9LsGG0KduWVXL\n+3gmMktyiMySHCKzpI5CJGmtpGOSdreMXSJpi6RXyv2UMi5JX5W0X9IuSdf1qnizJuh0Jvo6MH/M\n2HJga0TMBraW51B1/5ldbkupWmiZDa2OQhQRzwBvjBleCKwrj9cBt7WMPxKVbcDFYzoAmQ2VzGei\nKyLiCEC5v7yMTwVebVlupIy9i5s32rDoxYkFjTMWZw24eaMNiUyIjo4eppX7Y2V8BJjestw04HBi\nO2aNlgnRJmBJebwEeLJl/I5ylm4e8OboYZ/ZMOrosh9J64GPA5dKGgH+Gvhb4HFJdwI/AT5TFn8K\nWADsB96m+r0is6HVUYgiYnGbl24eZ9kA7skUZTZIfMWCWZJDZJbkEJklOURmSQ6RWZJDZJbkEJkl\nOURmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWdKEIWrT/fTvJb1UOpw+IeniMj5D\n0k8l7Sy3r/WyeLMm6GQm+jpndz/dAvxGRPwm8DJwf8trByJiTrndXU+ZZs01YYjG634aEZsj4lR5\nuo2qLZbZe1Idn4n+DPh2y/OZkn4g6XuSbmy3kjug2rBI/VKepAeAU8CjZegIcFVEHJf0MeBbkq6N\niBNj142INcAagOnTp5/VIdVsUHQ9E0laAnwK+OPSJouI+FlEHC+PdwAHgA/VUahZU3UVIknzgb8E\nPh0Rb7eMXyZpUnk8i+rnVQ7WUahZU014ONem++n9wEXAFkkA28qZuJuAv5F0CjgN3B0RY3+SxWyo\nTBiiNt1PH26z7EZgY7Yos0HiKxbMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySH\nyCzJITJLcojMkhwisySHyCzJITJLcojMkrrtgLpC0mstnU4XtLx2v6T9kvZJ+mSvCjdrim47oAI8\n2NLp9CkASdcAi4Bryzr/NNq4xGxYddUB9RwWAhtK66wfAfuB6xP1mTVe5jPRstLQfq2kKWVsKvBq\nyzIjZews7oBqw6LbEK0GrgbmUHU9XVnGNc6y43Y3jYg1ETE3IuZOnjy5yzLM+q+rEEXE0Yg4HRFn\ngIf4+SHbCDC9ZdFpwOFciWbN1m0H1Ctbnt4OjJ652wQsknSRpJlUHVC/nyvRrNm67YD6cUlzqA7V\nDgF3AUTEHkmPAz+kanR/T0Sc7k3pZs1QawfUsvyXgC9lijIbJL5iwSzJITJLcojMkhwisySHyCzJ\nITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwis6Rumzc+1tK48ZCknWV8hqSf\ntrz2tV4Wb9YEE36zlap54z8Aj4wORMQfjT6WtBJ4s2X5AxExp64CzZquk6+HPyNpxnivSRLwWeD3\n6y3LbHBkPxPdCByNiFdaxmZK+oGk70m6Mfn+Zo3XyeHcuSwG1rc8PwJcFRHHJX0M+JakayPixNgV\nJS0FlgJMmTJl7MtmA6PrmUjSBcAfAo+NjpUe3MfL4x3AAeBD463vDqg2LDKHc38AvBQRI6MDki4b\n/RUISbOomjcezJVo1mydnOJeD/wX8GFJI5LuLC8t4t2HcgA3AbskvQD8K3B3RHT6ixJmA6nb5o1E\nxJ+MM7YR2Jgvy2xw+IoFsySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwis6TsVdy1eHPSGf7t\n4v/tdxk2jm3z56fWn/f00zVVUr/f2by5lvfxTGSW5BCZJTlEZkmN+ExkzdXkzzRN4ZnILMkzkb1n\n1TXLKiJqeaNUEVL/izA7246ImDvRQp18PXy6pO9K2itpj6TPl/FLJG2R9Eq5n1LGJemrkvZL2iXp\nuvy+mDVXJ5+JTgH3RcSvA/OAeyRdAywHtkbEbGBreQ5wK1WDktlULbFW1161WYNMGKKIOBIRz5fH\nbwF7ganAQmBdWWwdcFt5vBB4JCrbgIslXVl75WYNcV5n50o74Y8CzwJXRMQRqIIGXF4Wmwq82rLa\nSBkzG0odn52T9H6qTj5fiIgTVRvu8RcdZ+ysEwetHVDNBllHM5Gk91EF6NGI+GYZPjp6mFbuj5Xx\nEWB6y+rTgMNj37O1A2q3xZs1QSdn5wQ8DOyNiFUtL20ClpTHS4AnW8bvKGfp5gFvjh72mQ2liDjn\nDfhdqsOxXcDOclsAfJDqrNwr5f6SsryAf6Tqw/0iMLeDbYRvvjXwtn2iv7sR4f9sNTuHev6z1czO\nzSEyS3KIzJIcIrMkh8gsqSnfJ3odOFnuh8WlDM/+DNO+QOf786udvFkjTnEDSNo+TFcvDNP+DNO+\nQP3748M5sySHyCypSSFa0+8CajZM+zNM+wI1709jPhOZDaomzURmA6nvIZI0X9K+0thk+cRrNI+k\nQ5JelLRT0vYyNm4jlyaStFbSMUm7W8YGthFNm/1ZIem18me0U9KCltfuL/uzT9Inz3uDnVzq3asb\nMInqKxOzgAuBF4Br+llTl/txCLh0zNiXgeXl8XLg7/pd5znqvwm4Dtg9Uf1UX4P5NtVXXuYBz/a7\n/g73ZwXwF+Mse035e3cRMLP8fZx0Ptvr90x0PbA/Ig5GxDvABqpGJ8OgXSOXxomIZ4A3xgwPbCOa\nNvvTzkJgQ0T8LCJ+BOyn+nvZsX6HaFiamgSwWdKO0jsC2jdyGRTD2IhmWTkEXdtyeJ3en36HqKOm\nJgPghoi4jqrn3j2Sbup3QT00qH9mq4GrgTnAEWBlGU/vT79D1FFTk6aLiMPl/hjwBNXhQLtGLoMi\n1YimaSLiaEScjogzwEP8/JAtvT/9DtFzwGxJMyVdCCyianQyMCRNlvSB0cfALcBu2jdyGRRD1Yhm\nzOe226n+jKDan0WSLpI0k6pz7/fP680bcCZlAfAy1VmRB/pdTxf1z6I6u/MCsGd0H2jTyKWJN2A9\n1SHO/1H9y3xnu/rpohFNQ/bnn0u9u0pwrmxZ/oGyP/uAW893e75iwSyp34dzZgPPITJLcojMkhwi\nsySHyCzJITJLcojMkhwis6T/BzF6WOXJ/icoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f529d1e9470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADntJREFUeJzt3X/sVfV9x/Hna1hNRruI9UcM4ABH2+myUUscmdN0c7VI\nmqJL2kGWyjYzNJGkjS4Z1mQjS5psXcGk2UaDkRQXC7pRK1mshbCmZtmwgkWEIgqU1q8QmLiIw6YO\neO+P8/mm1y/fy/dy3+f2nnt9PZKbe+/nnnPP+wRefM49nPu+igjMrHu/1O8CzAadQ2SW5BCZJTlE\nZkkOkVmSQ2SW1LMQSZovaZ+k/ZKW92o7Zv2mXvw/kaRJwMvAJ4AR4DlgcUT8sPaNmfVZr2ai64H9\nEXEwIt4BNgALe7Qts766oEfvOxV4teX5CPDb7RaW5MsmrIlej4jLJlqoVyHSOGPvCoqkpcDSHm3f\nrA4/7mShXoVoBJje8nwacLh1gYhYA6wBz0Q22Hr1meg5YLakmZIuBBYBm3q0LbO+6slMFBGnJC0D\nvgNMAtZGxJ5ebMus33pyivu8i2jg4dyqVavOe51777039R5j16/rPbKaUMNYY2vq0TZ3RMTciRby\nFQtmSb06sTB0ejFL9GO2q8MvYqYZJJ6JzJI8E9l5m2j2e6/NVJ6JzJI8E9mEJppZ+vG5rEk8E5kl\neSbqUB3/2jblPQZhm4PEM5FZkkNkluTLfsza82U/Zr8IjTixMG3atPfcf9BZ83X6d9IzkVmSQ2SW\n5BCZJTlEZkldh0jSdEnflbRX0h5Jny/jKyS9JmlnuS2or1yz5smcnTsF3BcRz0v6ALBD0pby2oMR\n8ZV8eWbN13WIIuIIcKQ8fkvSXqqmjWbvKbV8JpI0A/go8GwZWiZpl6S1kqbUsQ2zpkqHSNL7gY3A\nFyLiBLAauBqYQzVTrWyz3lJJ2yVtP3nyZLYMs75JhUjS+6gC9GhEfBMgIo5GxOmIOAM8RNXc/iwR\nsSYi5kbE3MmTJ2fKMOurzNk5AQ8DeyNiVcv4lS2L3Q7s7r48s+bLnJ27Afgc8KKknWXsi8BiSXOo\nGtgfAu5KVWjWcJmzc//B+L/+8FT35ZgNHl+xYJbUiK9CTMRfk7BeqKt3hGcisySHyCzJITJLcojM\nkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLSn+fSNIh4C3gNHAqIuZK\nugR4DJhB9RXxz0bE/2S3ZdZEdc1EvxcRc1p+VWw5sDUiZgNby3OzodSrw7mFwLryeB1wW4+2Y9Z3\ndYQogM2SdkhaWsauKG2GR9sNX17DdswaqY4eCzdExGFJlwNbJL3UyUolcEsBpkxxp2EbXOmZKCIO\nl/tjwBNUHU+PjjZxLPfHxlnPHVBtKGTbCE8uP6uCpMnALVQdTzcBS8piS4AnM9sxa7Ls4dwVwBNV\nR2EuAL4REU9Leg54XNKdwE+AzyS3Y9ZYqRBFxEHgt8YZPw7cnHlvs0HhKxbMkgaiA+q2+fP7XYIN\nof+s6X08E5klOURmSQ6RWZJDZJbkEJklDcTZuTO/dqLfJZi15ZnILMkhMktyiMySHCKzJIfILMkh\nMksaiFPcb/zK2/0uwawtz0RmSQ6RWVLXh3OSPkzV5XTULOCvgIuBPwf+u4x/MSKe6rpCs4brOkQR\nsQ+YAyBpEvAaVbefPwUejIiv1FKhWcPVdTh3M3AgIn5c0/uZDYy6zs4tAta3PF8m6Q5gO3Bftpn9\nGx95J7O62fher+dt0jORpAuBTwP/UoZWA1dTHeodAVa2WW+ppO2Stp88eTJbhlnf1HE4dyvwfEQc\nBYiIoxFxOiLOAA9RdUQ9izug2rCoI0SLaTmUG20fXNxO1RHVbGilPhNJ+mXgE8BdLcNfljSH6tci\nDo15zWzoZDugvg18cMzY51IVmQ2Ygbh27htnrup3CTaEbqnpfXzZj1mSQ2SW5BCZJTlEZkkOkVnS\nQJyde2fDin6XYMPolnp+XMUzkVmSQ2SW5BCZJTlEZkkOkVmSQ2SWNBCnuP/96Xn9LsGG0KduWVXL\n+3gmMktyiMySHCKzpI5CJGmtpGOSdreMXSJpi6RXyv2UMi5JX5W0X9IuSdf1qnizJuh0Jvo6MH/M\n2HJga0TMBraW51B1/5ldbkupWmiZDa2OQhQRzwBvjBleCKwrj9cBt7WMPxKVbcDFYzoAmQ2VzGei\nKyLiCEC5v7yMTwVebVlupIy9i5s32rDoxYkFjTMWZw24eaMNiUyIjo4eppX7Y2V8BJjestw04HBi\nO2aNlgnRJmBJebwEeLJl/I5ylm4e8OboYZ/ZMOrosh9J64GPA5dKGgH+Gvhb4HFJdwI/AT5TFn8K\nWADsB96m+r0is6HVUYgiYnGbl24eZ9kA7skUZTZIfMWCWZJDZJbkEJklOURmSQ6RWZJDZJbkEJkl\nOURmSQ6RWZJDZJbkEJklOURmSQ6RWZJDZJbkEJklOURmSQ6RWdKEIWrT/fTvJb1UOpw+IeniMj5D\n0k8l7Sy3r/WyeLMm6GQm+jpndz/dAvxGRPwm8DJwf8trByJiTrndXU+ZZs01YYjG634aEZsj4lR5\nuo2qLZbZe1Idn4n+DPh2y/OZkn4g6XuSbmy3kjug2rBI/VKepAeAU8CjZegIcFVEHJf0MeBbkq6N\niBNj142INcAagOnTp5/VIdVsUHQ9E0laAnwK+OPSJouI+FlEHC+PdwAHgA/VUahZU3UVIknzgb8E\nPh0Rb7eMXyZpUnk8i+rnVQ7WUahZU014ONem++n9wEXAFkkA28qZuJuAv5F0CjgN3B0RY3+SxWyo\nTBiiNt1PH26z7EZgY7Yos0HiKxbMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySH\nyCzJITJLcojMkhwisySHyCzJITJLcojMkrrtgLpC0mstnU4XtLx2v6T9kvZJ+mSvCjdrim47oAI8\n2NLp9CkASdcAi4Bryzr/NNq4xGxYddUB9RwWAhtK66wfAfuB6xP1mTVe5jPRstLQfq2kKWVsKvBq\nyzIjZews7oBqw6LbEK0GrgbmUHU9XVnGNc6y43Y3jYg1ETE3IuZOnjy5yzLM+q+rEEXE0Yg4HRFn\ngIf4+SHbCDC9ZdFpwOFciWbN1m0H1Ctbnt4OjJ652wQsknSRpJlUHVC/nyvRrNm67YD6cUlzqA7V\nDgF3AUTEHkmPAz+kanR/T0Sc7k3pZs1QawfUsvyXgC9lijIbJL5iwSzJITJLcojMkhwisySHyCzJ\nITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwis6Rumzc+1tK48ZCknWV8hqSf\ntrz2tV4Wb9YEE36zlap54z8Aj4wORMQfjT6WtBJ4s2X5AxExp64CzZquk6+HPyNpxnivSRLwWeD3\n6y3LbHBkPxPdCByNiFdaxmZK+oGk70m6Mfn+Zo3XyeHcuSwG1rc8PwJcFRHHJX0M+JakayPixNgV\nJS0FlgJMmTJl7MtmA6PrmUjSBcAfAo+NjpUe3MfL4x3AAeBD463vDqg2LDKHc38AvBQRI6MDki4b\n/RUISbOomjcezJVo1mydnOJeD/wX8GFJI5LuLC8t4t2HcgA3AbskvQD8K3B3RHT6ixJmA6nb5o1E\nxJ+MM7YR2Jgvy2xw+IoFsySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwis6TsVdy1eHPSGf7t\n4v/tdxk2gW3z56fWn/f00zVVUo/f2by5lvfxTGSW5BCZJTlEZkmN+Exkg6Fpn2mawjORWZJnInvP\nqmtmVUTU8kapIqT+F2F2th0RMXeihTr5evh0Sd+VtFfSHkmfL+OXSNoi6ZVyP6WMS9JXJe2XtEvS\ndfl9MWuuTj4TnQLui4hfB+YB90i6BlgObI2I2cDW8hzgVqoGJbOpWmKtrr1qswaZMEQRcSQini+P\n3wL2AlOBhcC6stg64LbyeCHwSFS2ARdLurL2ys0a4rzOzpV2wh8FngWuiIgjUAUNuLwsNhV4tWW1\nkTJmNpQ6Pjsn6f1UnXy+EBEnqjbc4y86zthZJw5aO6CaDbKOZiJJ76MK0KMR8c0yfHT0MK3cHyvj\nI8D0ltWnAYfHvmdrB9Ruizdrgk7Ozgl4GNgbEataXtoELCmPlwBPtozfUc7SzQPeHD3sMxtKEXHO\nG/C7VIdju4Cd5bYA+CDVWblXyv0lZXkB/0jVh/tFYG4H2wjffGvgbftEf3cjwv/ZanYO9fxnq5md\nm0NkluQQmSU5RGZJDpFZUlO+T/Q6cLLcD4tLGZ79GaZ9gc7351c7ebNGnOIGkLR9mK5eGKb9GaZ9\ngfr3x4dzZkkOkVlSk0K0pt8F1GyY9meY9gVq3p/GfCYyG1RNmonMBlLfQyRpvqR9pbHJ8onXaB5J\nhyS9KGmnpO1lbNxGLk0kaa2kY5J2t4wNbCOaNvuzQtJr5c9op6QFLa/dX/Znn6RPnvcGO7nUu1c3\nYBLVVyZmARcCLwDX9LOmLvfjEHDpmLEvA8vL4+XA3/W7znPUfxNwHbB7ovqpvgbzbaqvvMwDnu13\n/R3uzwrgL8ZZ9pry9+4iYGb5+zjpfLbX75noemB/RByMiHeADVSNToZBu0YujRMRzwBvjBke2EY0\nbfannYXAhoj4WUT8CNhP9feyY/0O0bA0NQlgs6QdpXcEtG/kMiiGsRHNsnIIurbl8Dq9P/0OUUdN\nTQbADRFxHVXPvXsk3dTvgnpoUP/MVgNXA3OAI8DKMp7en36HqKOmJk0XEYfL/THgCarDgXaNXAZF\nqhFN00TE0Yg4HRFngIf4+SFben/6HaLngNmSZkq6EFhE1ehkYEiaLOkDo4+BW4DdtG/kMiiGqhHN\nmM9tt1P9GUG1P4skXSRpJlXn3u+f15s34EzKAuBlqrMiD/S7ni7qn0V1ducFYM/oPtCmkUsTb8B6\nqkOc/6P6l/nOdvXTRSOahuzPP5d6d5XgXNmy/ANlf/YBt57v9nzFgllSvw/nzAaeQ2SW5BCZJTlE\nZkkOkVmSQ2SW5BCZJTlEZkn/D/mLWOXbgfgoAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f529d1cccc0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAANEAAAD8CAYAAADpCEEHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADo9JREFUeJzt3X/sVfV9x/Hna1hNRruA9UcM4ABHu+myfWuJI3O6bq72\nK2mGLmmHWSrbzNBEkja6ZFiTjSxpsnUFk2YbDUZSXCzqZq1ksQzCmpplwwoWUYsoWlq/QmDCIg6b\nOuC9P87nm16/fC/fy32f6z33+nokN/fezz33nPcJvPicezj3fRURmFn3fq7fBZgNOofILMkhMkty\niMySHCKzJIfILKlnIZI0KmmvpH2SVvZqO2b9pl78P5GkacBLwCeBMeBp4OaI+EHtGzPrs17NRFcB\n+yLi1Yh4B3gIWNKjbZn11Tk9Wu8s4LWW52PAb7RbWJIvm7AmeiMiLpxqoV6FSJOMvSsokpYDy3u0\nfbM6/KiThXoVojFgTsvz2cCB1gUiYh2wDjwT2WDr1Weip4EFkuZJOhdYCmzq0bbM+qonM1FEnJC0\nAvg3YBqwPiJe6MW2zPqtJ6e4z7qIBh7OrVmz5qzfc+edd6bWMfH9da0jqwk1TDSxph5tc2dELJxq\nIV+xYJbUqxMLQ6cXs0Q/Zrs6vBczzSDxTGSW5JnIztpUs9/7babyTGSW5JnIpjTVzNKPz2VN4pnI\nLMkzUYfq+Ne2KesYhG0OEs9EZkkOkVmSL/sxa8+X/Zi9FxpxYmH27Nnvu/+gs+br9O+kZyKzJIfI\nLMkhMktyiMySug6RpDmSviNpj6QXJH2+jK+S9LqkXeW2uL5yzZonc3buBHBXRDwj6UPATklby2v3\nRsRX8uWZNV/XIYqIg8DB8vgtSXuomjaava/U8plI0lzgY8BTZWiFpN2S1kuaWcc2zJoqHSJJHwQe\nBb4QEceAtcBlwAjVTLW6zfuWS9ohacfx48ezZZj1TSpEkj5AFaAHI+KbABFxKCJORsQp4D6q5van\niYh1EbEwIhZOnz49U4ZZX2XOzgm4H9gTEWtaxi9pWewm4PnuyzNrvszZuauBzwHPSdpVxr4I3Cxp\nhKqB/X7gtlSFZg2XOTv3H0z+6w9PdF+O2eDxFQtmSY34KsRU/DUJ64W6ekd4JjJLcojMkhwisySH\nyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwis6T094kk7QfeAk4CJyJi\noaTzgYeBuVRfEf9sRPxPdltmTVTXTPQ7ETHS8qtiK4FtEbEA2Faemw2lXh3OLQE2lMcbgBt7tB2z\nvqsjRAFskbRT0vIydnFpMzzebviiGrZj1kh19Fi4OiIOSLoI2CrpxU7eVAK3HGDmTHcatsGVnoki\n4kC5Pww8RtXx9NB4E8dyf3iS97kDqg2FbBvh6eVnVZA0HbiequPpJmBZWWwZ8HhmO2ZNlj2cuxh4\nrOoozDnANyJis6SngUck3Qr8GPhMcjtmjZUKUUS8Cvz6JONHgOsy6zYbFL5iwSxpIDqgbh8d7XcJ\nNoT+s6b1eCYyS3KIzJIcIrMkh8gsySEySxqIs3OnfulYv0swa8szkVmSQ2SW5BCZJTlEZkkOkVmS\nQ2SWNBCnuI/+wtv9LsGsLc9EZkkOkVlS14dzkj5K1eV03HzgL4EZwJ8B/13GvxgRT3RdoVnDdR2i\niNgLjABImga8TtXt50+AeyPiK7VUaNZwdR3OXQe8EhE/qml9ZgOjrrNzS4GNLc9XSLoF2AHclW1m\nf/SX38m83Wxyb9SzmvRMJOlc4PeBfy5Da4HLqA71DgKr27xvuaQdknYcP348W4ZZ39RxOHcD8ExE\nHAKIiEMRcTIiTgH3UXVEPY07oNqwqCNEN9NyKDfePri4iaojqtnQSn0mkvTzwCeB21qGvyxphOrX\nIvZPeM1s6GQ7oL4NfHjC2OdSFZkNmIG4du4bpy7tdwk2hK6vaT2+7McsySEyS3KIzJIcIrMkh8gs\naSDOzr3z0Kp+l2DD6Pp6flzFM5FZkkNkluQQmSU5RGZJDpFZkkNkljQQp7j/ffOifpdgQ+jT16+p\nZT2eicySHCKzJIfILKmjEElaL+mwpOdbxs6XtFXSy+V+ZhmXpK9K2idpt6Qre1W8WRN0OhN9HRid\nMLYS2BYRC4Bt5TlU3X8WlNtyqhZaZkOroxBFxJPA0QnDS4AN5fEG4MaW8Qeish2YMaEDkNlQyXwm\nujgiDgKU+4vK+CzgtZblxsrYu7h5ow2LXpxY0CRjcdqAmzfakMiE6ND4YVq5P1zGx4A5LcvNBg4k\ntmPWaJkQbQKWlcfLgMdbxm8pZ+kWAW+OH/aZDaOOLvuRtBH4BHCBpDHgr4C/AR6RdCvwY+AzZfEn\ngMXAPuBtqt8rMhtaHYUoIm5u89J1kywbwB2ZoswGia9YMEtyiMySHCKzJIfILMkhMktyiMySHCKz\nJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMkuaMkRtup/+naQXS4fTxyTNKONz\nJf1E0q5y+1ovizdrgk5moq9zevfTrcCvRsSvAS8Bd7e89kpEjJTb7fWUadZcU4Zosu6nEbElIk6U\np9up2mI1wvbRUbaPTsy8We/U8ZnoT4FvtzyfJ+n7kr4r6Zp2b3IHVBsWqV/Kk3QPcAJ4sAwdBC6N\niCOSPg58S9IVEXFs4nsjYh2wDmDOnDmndUg1GxRdz0SSlgGfBv6otMkiIn4aEUfK453AK8BH6ijU\nrKm6mokkjQJ/Afx2RLzdMn4hcDQiTkqaT/XzKq/WUmmHFm3e/F5uzmzqELXpfno3cB6wVRLA9nIm\n7lrgryWdAE4Ct0fExJ9kMRsqU4aoTffT+9ss+yjwaLYos0HiKxbMkhwisySHyCzJITJLcojMkhwi\nsySHyCzJITJLcojMkhwisySHyCzJITJLcojMkhwisySHyCzJITJLcojMkrrtgLpK0ustnU4Xt7x2\nt6R9kvZK+lSvCjdrim47oALc29Lp9AkASZcDS4Erynv+UdK0uoo1a6KuOqCewRLgodI664fAPuCq\nRH1mjZf5TLSiNLRfL2lmGZsFvNayzFgZO407oNqw6DZEa4HLgBGqrqery7gmWXbS7qYRsS4iFkbE\nwunTp3dZhln/dRWiiDgUEScj4hRwHz87ZBsD5rQsOhs4kCvRrNm6CpGkS1qe3gSMn7nbBCyVdJ6k\neVQdUL+XK9Gs2brtgPoJSSNUh2r7gdsAIuIFSY8AP6BqdH9HRJzsTelmzVBrB9Sy/JeAL2WKMhsk\nvmLBLMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMySHCKzJIfILMkhMktyiMyS\nHCKzpG6bNz7c0rhxv6RdZXyupJ+0vPa1XhZv1gRTfrOVqnnj3wMPjA9ExB+OP5a0GnizZflXImKk\nrgLNmq6Tr4c/KWnuZK9JEvBZ4HfrLctscGQ/E10DHIqIl1vG5kn6vqTvSromuX6zxuvkcO5MbgY2\ntjw/CFwaEUckfRz4lqQrIuLYxDdKWg4sB5g5c+bEl80GRtczkaRzgD8AHh4fKz24j5THO4FXgI9M\n9n53QLVhkTmc+z3gxYgYGx+QdOH4r0BImk/VvPHVXIlmzdbJKe6NwH8BH5U0JunW8tJS3n0oB3At\nsFvSs8C/ALdHRKe/KGE2kLpt3khE/PEkY48Cj+bLMhscvmLBLMkhMktyiMySHCKzJIfILMkhMkty\niMySHCKzJIfILCl7FXct3px2in+d8b/9LsOSto+OptexaPPmGirpzG9u2VLLejwTmSU5RGZJDpFZ\nUiM+E9lweC8/zzSJZyKzJM9E9r5V18ypiKhlRakipP4XYXa6nRGxcKqFOvl6+BxJ35G0R9ILkj5f\nxs+XtFXSy+V+ZhmXpK9K2idpt6Qr8/ti1lydfCY6AdwVEb8CLALukHQ5sBLYFhELgG3lOcANVA1K\nFlC1xFpbe9VmDTJliCLiYEQ8Ux6/BewBZgFLgA1lsQ3AjeXxEuCBqGwHZki6pPbKzRrirM7OlXbC\nHwOeAi6OiINQBQ24qCw2C3it5W1jZcxsKHV8dk7SB6k6+XwhIo5VbbgnX3SSsdNOHLR2QDUbZB3N\nRJI+QBWgByPim2X40PhhWrk/XMbHgDktb58NHJi4ztYOqN0Wb9YEnZydE3A/sCci1rS8tAlYVh4v\nAx5vGb+lnKVbBLw5fthnNpQi4ow34LeoDsd2A7vKbTHwYaqzci+X+/PL8gL+gaoP93PAwg62Eb75\n1sDbjqn+7kaE/7PV7Azq+c9WMzszh8gsySEyS3KIzJIcIrOkpnyf6A3geLkfFhcwPPszTPsCne/P\nL3ayskac4gaQtGOYrl4Ypv0Zpn2B+vfHh3NmSQ6RWVKTQrSu3wXUbJj2Z5j2BWren8Z8JjIbVE2a\nicwGUt9DJGlU0t7S2GTl1O9oHkn7JT0naZekHWVs0kYuTSRpvaTDkp5vGRvYRjRt9meVpNfLn9Eu\nSYtbXru77M9eSZ866w12cql3r27ANKqvTMwHzgWeBS7vZ01d7sd+4IIJY18GVpbHK4G/7XedZ6j/\nWuBK4Pmp6qf6Gsy3qb7ysgh4qt/1d7g/q4A/n2TZy8vfu/OAeeXv47Sz2V6/Z6KrgH0R8WpEvAM8\nRNXoZBi0a+TSOBHxJHB0wvDANqJpsz/tLAEeioifRsQPgX1Ufy871u8QDUtTkwC2SNpZekdA+0Yu\ng2IYG9GsKIeg61sOr9P70+8QddTUZABcHRFXUvXcu0PStf0uqIcG9c9sLXAZMAIcBFaX8fT+9DtE\nHTU1abqIOFDuDwOPUR0OtGvkMihSjWiaJiIORcTJiDgF3MfPDtnS+9PvED0NLJA0T9K5wFKqRicD\nQ9J0SR8afwxcDzxP+0Yug2KoGtFM+Nx2E9WfEVT7s1TSeZLmUXXu/d5ZrbwBZ1IWAy9RnRW5p9/1\ndFH/fKqzO88CL4zvA20auTTxBmykOsT5P6p/mW9tVz9dNKJpyP78U6l3dwnOJS3L31P2Zy9ww9lu\nz1csmCX1+3DObOA5RGZJDpFZkkNkluQQmSU5RGZJDpFZkkNklvT/HbdiEgOssngAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f529d136eb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "observation = env.reset()\n",
    "print(\"Observation space shape: {}\".format(observation.shape))\n",
    "\n",
    "plt.figure()\n",
    "plt.imshow(env.render(mode='rgb_array'))\n",
    "\n",
    "env.step(2)\n",
    "plt.figure()\n",
    "plt.imshow(env.render(mode='rgb_array'))\n",
    "\n",
    "env.step(1)\n",
    "plt.figure()\n",
    "plt.imshow(env.render(mode='rgb_array'))\n",
    "\n",
    "env.render(close=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(160, 160, 3)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f52977f0c50>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQUAAAD8CAYAAAB+fLH0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADvRJREFUeJzt3X/sXXV9x/Hna0VwVFlbEdLRaovr3JhZRtOwTjdjZMPC\nGHWZJjVmEm1CluGm64wU+UP/MZG56WayYaqw4YIg80dsDDIaxJnFtQrITxGtVeErlaooOljEynt/\n3NPsfur3S9vvvefe75c9H8k3957PPfec9z33+331nHNP7ztVhSQd8gvTLkDSwmIoSGoYCpIahoKk\nhqEgqWEoSGr0FgpJNiW5P8neJNv7Wo+k8Uof1ykkWQJ8FfgDYAb4IvCaqvry2Fcmaaz62lM4C9hb\nVfuq6gngOmBzT+uSNEbH9bTc04AHh6ZngN+ea+alS5fWihUrjmrBMzMzo1UmPY2sWrXqqOedmZn5\nXlU990jz9RUKmWWsOU5JchFwEcDy5cvZtm3bUS34aOeT/j84lr+Hbdu2feto5uvr8GEGWD00vQp4\naHiGqtpRVRuqasPSpUt7KkPSseorFL4IrEuyNsnxwBZgZ0/rkjRGvRw+VNXBJG8E/h1YAlxVVff2\nsS5J49XXOQWq6gbghr6WL6kfXtEoqWEoSGoYCpIahoKkhqEgqWEoSGoYCpIahoKkhqEgqWEoSGoY\nCpIahoKkhqEgqWEoSGoYCpIahoKkhqEgqWEoSGoYCpIa8w6FJKuT3JLkviT3JnlTN74iya4kX+tu\nl4+vXEl9G2VP4SDw11X168BG4OIkZwDbgZurah1wczctaZGYdyhU1f6qur27/2PgPgbt4jYDV3ez\nXQ28ctQiJU3OWM4pJFkDnAnsAU6tqv0wCA7glDmec1GSW5Pc+thjj42jDEljMHIoJHkW8DHgzVX1\no6N9nm3jpIVppFBI8gwGgXBNVX28G344ycru8ZXAgdFKlDRJ8+4QlSTAlcB9VfWeoYd2AhcC7+pu\nPzlShYfZvWnTOBcnLWqf72GZo7SNewnwp8DdSe7oxt7GIAyuT7IVeAB49WglSpqkeYdCVf0nkDke\nPnu+y5U0XV7RKKlhKEhqGAqSGoaCpIahIKlhKEhqjHKdwlQ8+StHfSW1pHlwT0FSw1CQ1DAUJDUM\nBUkNQ0FSw1CQ1DAUJDUW3XUKj5z0+LRLkJ7W3FOQ1DAUJDUMBUmNcXzF+5IkX0ryqW56bZI9Xdu4\njyQ5fvQyJU3KOPYU3sSgO9QhlwPv7drG/QDYOoZ1SJqQUfs+rAL+EPhgNx3g5cBHu1lsGyctMqPu\nKfw98FbgyW76OcAPq+pgNz3DoL/kz7FtnLQwjdIM5nzgQFXdluRlh4ZnmbVme35V7QB2AKxevXrW\neWbzyK89cYyVSk9j3xv/IkdtBnNBkvOAZwInMdhzWJbkuG5vYRXw0OhlSpqUUVrRX1pVq6pqDbAF\n+ExVvRa4BXhVN9vY28ZJ6lcf1ylcAmxLspfBOYYre1iHpJ6M5f8+VNVngc929/cBZ41juZImzysa\nJTUMBUkNQ0FSY9F9n8KHn3zetEuQFoxzelimewqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKmx6K5T\neOK6d0y7BGnhOOfzY1+kewqSGoaCpIahIKlhKEhqGAqSGoaCpMaozWCWJflokq8kuS/J7yRZkWRX\n1zZuV5Ll4ypWUv9GvU7hH4Abq+pVXc/IE4G3ATdX1buSbAe2M/gy17H4zI0bx7UoadE7/5z3jH2Z\n895TSHIS8FK6b2uuqieq6ofAZgbt4sC2cdKiM8rhw+nAd4F/7rpOfzDJUuDUqtoP0N2eMoY6JU3I\nKKFwHLAeuKKqzgQeY3CocFTsJSktTKOEwgwwU1V7uumPMgiJh5OsBOhuD8z25KraUVUbqmrD0qVL\nRyhD0jiN0jbuO8CDSV7YDZ0NfBnYyaBdHNg2Tlp0Rv304S+Aa7pPHvYBr2cQNNcn2Qo8ALx6xHVI\nmqCRQqGq7gA2zPLQ2aMsV9L0eEWjpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKlh\nKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKkxatu4v0pyb5J7klyb5JlJ1ibZ07WN+0j3/Y2S\nFolROkSdBvwlsKGqXgQsAbYAlwPvrap1wA+AreMoVNJkjHr4cBzwi0mOY9BHcj/wcgY9IMC2cdKi\nM0rfh28Df8vga9z3A48CtwE/rKqD3WwzwGmjFilpckY5fFjOoJnsWuCXgaXAubPMWnM837Zx0gI0\nyuHD7wPfqKrvVtVPgY8DLwaWdYcTAKuAh2Z7sm3jpIVplFB4ANiY5MQk4f/axt0CvKqbx7Zx0iIz\nyjmFPQxOKN4O3N0tawdwCbAtyV7gOcCVY6hT0oSM2jbu7cDbDxveB5w1ynIlTY9XNEpqGAqSGoaC\npIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqS\nGkcMhSRXJTmQ5J6hsRVJdnWt4XZ1X/dOBt6XZG+Su5Ks77N4SeN3NHsK/wJsOmxsO3Bz1xru5m4a\nBn0f1nU/FwFXjKdMSZNyxFCoqs8Bjxw2vJlBSzhoW8NtBj5UA7sZ9IBYOa5iJfVvvucUTq2q/QDd\n7Snd+GnAg0Pz2TZOWmTGfaIxs4zZNk5aROYbCg8fOizobg904zPA6qH5bBsnLTLzDYWdDFrCQdsa\nbifwuu5TiI3Ao4cOMyQtDkfsEJXkWuBlwMlJZhh0hHoXcH2SrQx6Sr66m/0G4DxgL/A48PoeapbU\noyOGQlW9Zo6Hzp5l3gIuHrUoSdPjFY2SGoaCpIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaC\npIahIKlhKEhqGAqSGoaCpIahIKlhKEhqGAqSGoaCpMZ828a9O8lXutZwn0iybOixS7u2cfcneUVf\nhUvqx3zbxu0CXlRVvwl8FbgUIMkZwBbgN7rn/FOSJWOrVlLv5tU2rqpuqqqD3eRuBv0dYNA27rqq\n+klVfYPBtzqfNcZ6JfVsHOcU3gB8urtv2zhpkRspFJJcBhwErjk0NMtsto2TFpF5h0KSC4Hzgdd2\n/R7AtnHSojevUEiyCbgEuKCqHh96aCewJckJSdYC64AvjF6mpEmZb9u4S4ETgF1JAHZX1Z9V1b1J\nrge+zOCw4uKq+llfxUsav/m2jbvyKeZ/J/DOUYqSND1e0SipYShIahgKkhqGgqSGoSCpYShIahgK\nkhqGgqSGoSCpYShIahgKkhqGgqSGoSCpYShIahgKkhqGgqSGoSCpYShIahgKkhrz6iU59NhbklSS\nk7vpJHlf10vyriTr+yhaUn/m20uSJKuBPwAeGBo+l8HXuq8DLgKuGL1ESZM0r16SnfcCb6XtALUZ\n+FAN7AaWJVk5lkolTcR8m8FcAHy7qu487KGj7iVp2zhpYTpi34fDJTkRuAw4Z7aHZxmbtZdkVe0A\ndgCsXr161nkkTd4xhwLwAmAtcGfXHWoVcHuSsziGXpKSFqZjPnyoqrur6pSqWlNVaxgEwfqq+g6D\nXpKv6z6F2Ag8WlX7x1uypD4dzUeS1wL/BbwwyUySrU8x+w3APmAv8AHgz8dSpaSJmW8vyeHH1wzd\nL+Di0cuSNC1e0SipYShIahgKkhqGgqSGoSCpYShIahgKkhqGgqSGoSCpYShIahgKkhqGgqSGoSCp\nYShIasznm5fG7tElT/KpZf897TK0AO3e9HNfJD4WG2+8sZflTtqLb7pp7Mt0T0FSw1CQ1DAUJDUM\nBUkNQ0FSY0F8+iDN5enyKcFiksEXME+5iOS7wGPA96ZdC3Ay1jHMOlqLuY7nV9VzjzTTgggFgCS3\nVtUG67AO65huHZ5TkNQwFCQ1FlIo7Jh2AR3raFlH62lfx4I5pyBpYVhIewqSFoCph0KSTUnuT7I3\nyfYJrnd1kluS3Jfk3iRv6sbfkeTbSe7ofs6bQC3fTHJ3t75bu7EVSXYl+Vp3u7znGl449JrvSPKj\nJG+exPZIclWSA0nuGRqb9fVn4H3d78tdSdb3XMe7k3ylW9cnkizrxtck+Z+h7fL+nuuY831Icmm3\nPe5P8oqRC6iqqf0AS4CvA6cDxwN3AmdMaN0rgfXd/WcDXwXOAN4BvGXC2+GbwMmHjf0NsL27vx24\nfMLvy3eA509iewAvBdYD9xzp9QPnAZ8GAmwE9vRcxznAcd39y4fqWDM83wS2x6zvQ/c7eydwArC2\n+3taMsr6p72ncBawt6r2VdUTwHXA5kmsuKr2V9Xt3f0fA/cBp01i3UdpM3B1d/9q4JUTXPfZwNer\n6luTWFlVfQ545LDhuV7/ZuBDNbAbWJZkZV91VNVNVXWwm9wNrBrHuo61jqewGbiuqn5SVd8A9jL4\nu5q3aYfCacCDQ9MzTOEPM8ka4ExgTzf0xm538aq+d9s7BdyU5LYkF3Vjp1bVfhgEGHDKBOo4ZAtw\n7dD0pLcHzP36p/k78wYGeymHrE3ypST/keT3JrD+2d6HsW+PaYdCZhmb6MchSZ4FfAx4c1X9CLgC\neAHwW8B+4O8mUMZLqmo9cC5wcZKXTmCds0pyPHAB8G/d0DS2x1OZyu9MksuAg8A13dB+4HlVdSaw\nDfhwkpN6LGGu92Hs22PaoTADrB6aXgU8NKmVJ3kGg0C4pqo+DlBVD1fVz6rqSeADjLgrdjSq6qHu\n9gDwiW6dDx/aLe5uD/RdR+dc4PaqeriraeLbozPX65/470ySC4HzgddWdyDf7a5/v7t/G4Nj+V/t\nq4aneB/Gvj2mHQpfBNYlWdv9C7UF2DmJFScJcCVwX1W9Z2h8+Pj0j4F7Dn/umOtYmuTZh+4zOLF1\nD4PtcGE324XAJ/usY8hrGDp0mPT2GDLX698JvK77FGIj8Oihw4w+JNkEXAJcUFWPD40/N8mS7v7p\nwDpgX491zPU+7AS2JDkhydquji+MtLI+zp4e45nW8xic+f86cNkE1/u7DHaz7gLu6H7OA/4VuLsb\n3wms7LmO0xmcPb4TuPfQNgCeA9wMfK27XTGBbXIi8H3gl4bGet8eDEJoP/BTBv/ybZ3r9TPYXf7H\n7vflbmBDz3XsZXDMfuh35P3dvH/SvV93ArcDf9RzHXO+D8Bl3fa4Hzh31PV7RaOkxrQPHyQtMIaC\npIahIKlhKEhqGAqSGoaCpIahIKlhKEhq/C8HSPFtukRftgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f529d76bb00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(observation[34:-16,:,:].shape) # 210 - 34 - 16 = 160\n",
    "plt.imshow(observation[34:-16,:,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class StateProcessor():\n",
    "    \"\"\"\n",
    "    Processes a raw Atari iamges. Resizes it and converts it to grayscale.\n",
    "    图片的预处理。\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        # Build the Tensorflow graph\n",
    "        with tf.variable_scope(\"state_processor\"):\n",
    "            self.input_state = tf.placeholder(shape=[210, 160, 3], dtype=tf.uint8)\n",
    "            self.output = tf.image.rgb_to_grayscale(self.input_state)\n",
    "            self.output = tf.image.crop_to_bounding_box(self.output, 34, 0, 160, 160)\n",
    "            self.output = tf.image.resize_images(\n",
    "                self.output, [84, 84], method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)\n",
    "            self.output = tf.squeeze(self.output)\n",
    "\n",
    "    def process(self, sess, state):\n",
    "        \"\"\"\n",
    "        Args:\n",
    "            sess: A Tensorflow session object\n",
    "            state: A [210, 160, 3] Atari RGB State\n",
    "\n",
    "        Returns:\n",
    "            A processed [84, 84, 1] state representing grayscale values.\n",
    "        \"\"\"\n",
    "        return sess.run(self.output, { self.input_state: state })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep Q-Networks\n",
    "\n",
    "**带经验回放的DQN（Experience Replay in Deep Q-Networks）**：\n",
    "- 1.使用$\\epsilon-greedy$在policy中选择动作$a_t$。\n",
    "- 2.保存$(s_t, a_t, r_{t+1}, s_{t+1})$到回放记忆$D$。\n",
    "- 3.从$D$中抽一个mini-batch：$(s, a, r, s')$。\n",
    "- 4.使用老参数$w^-$计算Q-learning target（fixed Q-learning target）。\n",
    "- 5.优化Q-network和Q-learning之间的MSE：$L_i(w) = E_{s, a, r, s' \\sim D_i}[(r + \\gamma \\underset{a'}{max}Q(s', a'; w_i^-))^2]$\n",
    "- 两个Trick：**经验回放Experience Replay**，**fixed Q-learning target**。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Estimator():\n",
    "    \"\"\"Q-Value Estimator neural network.\n",
    "\n",
    "    This network is used for both the Q-Network and the Target Network.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, scope=\"estimator\", summaries_dir=None):\n",
    "        self.scope = scope\n",
    "        # Writes Tensorboard summaries to disk\n",
    "        self.summary_writer = None\n",
    "        with tf.variable_scope(scope):\n",
    "            # Build the graph\n",
    "            self._build_model()\n",
    "            if summaries_dir:\n",
    "                summary_dir = os.path.join(summaries_dir, \"summaries_{}\".format(scope))\n",
    "                if not os.path.exists(summary_dir):\n",
    "                    os.makedirs(summary_dir)\n",
    "                self.summary_writer = tf.summary.FileWriter(summary_dir)\n",
    "\n",
    "    def _build_model(self):\n",
    "        \"\"\"\n",
    "        Builds the Tensorflow graph.\n",
    "        \"\"\"\n",
    "\n",
    "        # Placeholders for our input\n",
    "        # Our input are 4 RGB frames of shape 160, 160 each\n",
    "        # 不需要再提取其他特征，使用图片与处理之后的像素点作为特征\n",
    "        self.X_pl = tf.placeholder(shape=[None, 84, 84, 4], dtype=tf.uint8, name=\"X\")\n",
    "        # The TD target value\n",
    "        self.y_pl = tf.placeholder(shape=[None], dtype=tf.float32, name=\"y\")\n",
    "        # Integer id of which action was selected\n",
    "        self.actions_pl = tf.placeholder(shape=[None], dtype=tf.int32, name=\"actions\")\n",
    "\n",
    "        X = tf.to_float(self.X_pl) / 255.0\n",
    "        batch_size = tf.shape(self.X_pl)[0]\n",
    "\n",
    "        # Three convolutional layers\n",
    "        conv1 = tf.contrib.layers.conv2d(\n",
    "            X, 32, 8, 4, activation_fn=tf.nn.relu)\n",
    "        conv2 = tf.contrib.layers.conv2d(\n",
    "            conv1, 64, 4, 2, activation_fn=tf.nn.relu)\n",
    "        conv3 = tf.contrib.layers.conv2d(\n",
    "            conv2, 64, 3, 1, activation_fn=tf.nn.relu)\n",
    "\n",
    "        # Fully connected layers\n",
    "        flattened = tf.contrib.layers.flatten(conv3)\n",
    "        fc1 = tf.contrib.layers.fully_connected(flattened, 512)\n",
    "        # 输出是不同的动作对应的TD-target\n",
    "        self.predictions = tf.contrib.layers.fully_connected(fc1, len(VALID_ACTIONS))\n",
    "\n",
    "        # Get the predictions for the chosen actions only\n",
    "        gather_indices = tf.range(batch_size) * tf.shape(self.predictions)[1] + self.actions_pl\n",
    "        self.action_predictions = tf.gather(tf.reshape(self.predictions, [-1]), gather_indices)\n",
    "\n",
    "        # Calcualte the loss\n",
    "        self.losses = tf.squared_difference(self.y_pl, self.action_predictions)\n",
    "        self.loss = tf.reduce_mean(self.losses)\n",
    "\n",
    "        # Optimizer Parameters from original paper\n",
    "        self.optimizer = tf.train.RMSPropOptimizer(0.00025, 0.99, 0.0, 1e-6)\n",
    "        self.train_op = self.optimizer.minimize(self.loss, global_step=tf.contrib.framework.get_global_step())\n",
    "\n",
    "        # Summaries for Tensorboard\n",
    "        self.summaries = tf.summary.merge([\n",
    "            tf.summary.scalar(\"loss\", self.loss),\n",
    "            tf.summary.histogram(\"loss_hist\", self.losses),\n",
    "            tf.summary.histogram(\"q_values_hist\", self.predictions),\n",
    "            tf.summary.scalar(\"max_q_value\", tf.reduce_max(self.predictions))\n",
    "        ])\n",
    "\n",
    "    def predict(self, sess, s):\n",
    "        \"\"\"\n",
    "        Predicts action values.\n",
    "\n",
    "        Args:\n",
    "          sess: Tensorflow session\n",
    "          s: State input of shape [batch_size, 4, 160, 160, 3]\n",
    "\n",
    "        Returns:\n",
    "          Tensor of shape [batch_size, NUM_VALID_ACTIONS] containing the estimated \n",
    "          action values.\n",
    "        \"\"\"\n",
    "        return sess.run(self.predictions, { self.X_pl: s })\n",
    "\n",
    "    def update(self, sess, s, a, y):\n",
    "        \"\"\"\n",
    "        Updates the estimator towards the given targets.\n",
    "\n",
    "        Args:\n",
    "          sess: Tensorflow session object\n",
    "          s: State input of shape [batch_size, 4, 160, 160, 3]\n",
    "          a: Chosen actions of shape [batch_size]\n",
    "          y: Targets of shape [batch_size]\n",
    "\n",
    "        Returns:\n",
    "          The calculated loss on the batch.\n",
    "        \"\"\"\n",
    "        feed_dict = { self.X_pl: s, self.y_pl: y, self.actions_pl: a }\n",
    "        summaries, global_step, _, loss = sess.run(\n",
    "            [self.summaries, tf.contrib.framework.get_global_step(), self.train_op, self.loss],\n",
    "            feed_dict)\n",
    "        if self.summary_writer:\n",
    "            self.summary_writer.add_summary(summaries, global_step)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cer/.local/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.          0.          0.06056029  0.00708057]\n",
      " [ 0.          0.          0.06056029  0.00708057]]\n",
      "99.9292\n"
     ]
    }
   ],
   "source": [
    "# For Testing....\n",
    "\n",
    "tf.reset_default_graph()\n",
    "global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n",
    "\n",
    "e = Estimator(scope=\"test\")\n",
    "sp = StateProcessor()\n",
    "\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Example observation batch\n",
    "    observation = env.reset()\n",
    "    \n",
    "    observation_p = sp.process(sess, observation)\n",
    "    observation = np.stack([observation_p] * 4, axis=2)\n",
    "    observations = np.array([observation] * 2)\n",
    "    \n",
    "    # Test Prediction\n",
    "    print(e.predict(sess, observations))\n",
    "\n",
    "    # Test training step\n",
    "    y = np.array([10.0, 10.0])\n",
    "    a = np.array([1, 3])\n",
    "    print(e.update(sess, observations, a, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class ModelParametersCopier():\n",
    "    \"\"\"\n",
    "    Copy model parameters of one estimator to another.\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, estimator1, estimator2):\n",
    "        \"\"\"\n",
    "        Defines copy-work operation graph.  \n",
    "        Args:\n",
    "          estimator1: Estimator to copy the paramters from\n",
    "          estimator2: Estimator to copy the parameters to\n",
    "        \"\"\"\n",
    "        e1_params = [t for t in tf.trainable_variables() if t.name.startswith(estimator1.scope)]\n",
    "        e1_params = sorted(e1_params, key=lambda v: v.name)\n",
    "        e2_params = [t for t in tf.trainable_variables() if t.name.startswith(estimator2.scope)]\n",
    "        e2_params = sorted(e2_params, key=lambda v: v.name)\n",
    "\n",
    "        self.update_ops = []\n",
    "        for e1_v, e2_v in zip(e1_params, e2_params):\n",
    "            op = e2_v.assign(e1_v)\n",
    "            self.update_ops.append(op)\n",
    "            \n",
    "    def make(self, sess):\n",
    "        \"\"\"\n",
    "        Makes copy.\n",
    "        Args:\n",
    "            sess: Tensorflow session instance\n",
    "        \"\"\"\n",
    "        sess.run(self.update_ops)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_epsilon_greedy_policy(estimator, nA):\n",
    "    \"\"\"\n",
    "    Creates an epsilon-greedy policy based on a given Q-function approximator and epsilon.\n",
    "\n",
    "    Args:\n",
    "        estimator: An estimator that returns q values for a given state\n",
    "        nA: Number of actions in the environment.\n",
    "\n",
    "    Returns:\n",
    "        A function that takes the (sess, observation, epsilon) as an argument and returns\n",
    "        the probabilities for each action in the form of a numpy array of length nA.\n",
    "\n",
    "    \"\"\"\n",
    "    def policy_fn(sess, observation, epsilon):\n",
    "        A = np.ones(nA, dtype=float) * epsilon / nA\n",
    "        q_values = estimator.predict(sess, np.expand_dims(observation, 0))[0]\n",
    "        best_action = np.argmax(q_values)\n",
    "        A[best_action] += (1.0 - epsilon)\n",
    "        return A\n",
    "    return policy_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def deep_q_learning(sess,\n",
    "                    env,\n",
    "                    q_estimator,\n",
    "                    target_estimator,\n",
    "                    state_processor,\n",
    "                    num_episodes,\n",
    "                    experiment_dir,\n",
    "                    replay_memory_size=500000,\n",
    "                    replay_memory_init_size=50000,\n",
    "                    update_target_estimator_every=10000,\n",
    "                    discount_factor=0.99,\n",
    "                    epsilon_start=1.0,\n",
    "                    epsilon_end=0.1,\n",
    "                    epsilon_decay_steps=500000,\n",
    "                    batch_size=32,\n",
    "                    record_video_every=50):\n",
    "    \"\"\"\n",
    "    Q-Learning algorithm for fff-policy TD control using Function Approximation.\n",
    "    Finds the optimal greedy policy while following an epsilon-greedy policy.\n",
    "\n",
    "    Args:\n",
    "        sess: Tensorflow Session object\n",
    "        env: OpenAI environment\n",
    "        q_estimator: Estimator object used for the q values\n",
    "        target_estimator: Estimator object used for the targets\n",
    "        state_processor: A StateProcessor object\n",
    "        num_episodes: Number of episodes to run for\n",
    "        experiment_dir: Directory to save Tensorflow summaries in\n",
    "        replay_memory_size: Size of the replay memory\n",
    "        replay_memory_init_size: Number of random experiences to sampel when initializing \n",
    "          the reply memory.\n",
    "        update_target_estimator_every: Copy parameters from the Q estimator to the \n",
    "          target estimator every N steps\n",
    "        discount_factor: Lambda time discount factor\n",
    "        epsilon_start: Chance to sample a random action when taking an action.\n",
    "          Epsilon is decayed over time and this is the start value\n",
    "        epsilon_end: The final minimum value of epsilon after decaying is done\n",
    "        epsilon_decay_steps: Number of steps to decay epsilon over\n",
    "        batch_size: Size of batches to sample from the replay memory\n",
    "        record_video_every: Record a video every N episodes\n",
    "\n",
    "    Returns:\n",
    "        An EpisodeStats object with two numpy arrays for episode_lengths and episode_rewards.\n",
    "    \"\"\"\n",
    "\n",
    "    Transition = namedtuple(\"Transition\", [\"state\", \"action\", \"reward\", \"next_state\", \"done\"])\n",
    "\n",
    "    # The replay memory\n",
    "    replay_memory = []\n",
    "    \n",
    "    # Make model copier object\n",
    "    estimator_copy = ModelParametersCopier(q_estimator, target_estimator)\n",
    "\n",
    "    # Keeps track of useful statistics\n",
    "    stats = plotting.EpisodeStats(\n",
    "        episode_lengths=np.zeros(num_episodes),\n",
    "        episode_rewards=np.zeros(num_episodes))\n",
    "    \n",
    "    # For 'system/' summaries, usefull to check if currrent process looks healthy\n",
    "    current_process = psutil.Process()\n",
    "\n",
    "    # Create directories for checkpoints and summaries\n",
    "    checkpoint_dir = os.path.join(experiment_dir, \"checkpoints\")\n",
    "    checkpoint_path = os.path.join(checkpoint_dir, \"model\")\n",
    "    monitor_path = os.path.join(experiment_dir, \"monitor\")\n",
    "    \n",
    "    if not os.path.exists(checkpoint_dir):\n",
    "        os.makedirs(checkpoint_dir)\n",
    "    if not os.path.exists(monitor_path):\n",
    "        os.makedirs(monitor_path)\n",
    "\n",
    "    saver = tf.train.Saver()\n",
    "    # Load a previous checkpoint if we find one\n",
    "    latest_checkpoint = tf.train.latest_checkpoint(checkpoint_dir)\n",
    "    if latest_checkpoint:\n",
    "        print(\"Loading model checkpoint {}...\\n\".format(latest_checkpoint))\n",
    "        saver.restore(sess, latest_checkpoint)\n",
    "    \n",
    "    # Get the current time step\n",
    "    total_t = sess.run(tf.contrib.framework.get_global_step())\n",
    "\n",
    "    # The epsilon decay schedule\n",
    "    epsilons = np.linspace(epsilon_start, epsilon_end, epsilon_decay_steps)\n",
    "\n",
    "    # The policy we're following\n",
    "    policy = make_epsilon_greedy_policy(\n",
    "        q_estimator,\n",
    "        len(VALID_ACTIONS))\n",
    "\n",
    "    # Populate the replay memory with initial experience\n",
    "    print(\"Populating replay memory...\")\n",
    "    state = env.reset()\n",
    "    state = state_processor.process(sess, state)\n",
    "    state = np.stack([state] * 4, axis=2)\n",
    "    for i in range(replay_memory_init_size):\n",
    "        # 1.使用epsilon-greedy在policy中选择动作a_t。\n",
    "        action_probs = policy(sess, state, epsilons[min(total_t, epsilon_decay_steps-1)])\n",
    "        action = np.random.choice(np.arange(len(action_probs)), p=action_probs)\n",
    "        next_state, reward, done, _ = env.step(VALID_ACTIONS[action])\n",
    "        next_state = state_processor.process(sess, next_state)\n",
    "        next_state = np.append(state[:,:,1:], np.expand_dims(next_state, 2), axis=2)\n",
    "        # 2.保存(s_t, a_t, r_{t+1}, s_{t+1})到回放记忆D。\n",
    "        replay_memory.append(Transition(state, action, reward, next_state, done))\n",
    "        if done:\n",
    "            state = env.reset()\n",
    "            state = state_processor.process(sess, state)\n",
    "            state = np.stack([state] * 4, axis=2)\n",
    "        else:\n",
    "            state = next_state\n",
    "\n",
    "\n",
    "    # Record videos\n",
    "    # Add env Monitor wrapper\n",
    "    env = Monitor(env, directory=monitor_path, video_callable=lambda count: count % record_video_every == 0, resume=True)\n",
    "\n",
    "    for i_episode in range(num_episodes):\n",
    "\n",
    "        # Save the current checkpoint\n",
    "        saver.save(tf.get_default_session(), checkpoint_path)\n",
    "\n",
    "        # Reset the environment\n",
    "        state = env.reset()\n",
    "        state = state_processor.process(sess, state)\n",
    "        state = np.stack([state] * 4, axis=2)\n",
    "        loss = None\n",
    "\n",
    "        # One step in the environment\n",
    "        for t in itertools.count():\n",
    "\n",
    "            # Epsilon for this time step\n",
    "            epsilon = epsilons[min(total_t, epsilon_decay_steps-1)]\n",
    "\n",
    "            # Maybe update the target estimator\n",
    "            if total_t % update_target_estimator_every == 0:\n",
    "                estimator_copy.make(sess)\n",
    "                print(\"\\nCopied model parameters to target network.\")\n",
    "\n",
    "            # Print out which step we're on, useful for debugging.\n",
    "            if t % 10 == 0:\n",
    "                print(\"\\rStep {} ({}) @ Episode {}/{}, loss: {}\".format(\n",
    "                    t, total_t, i_episode + 1, num_episodes, loss), end=\"\")\n",
    "                sys.stdout.flush()\n",
    "\n",
    "            # Take a step\n",
    "            action_probs = policy(sess, state, epsilon)\n",
    "            action = np.random.choice(np.arange(len(action_probs)), p=action_probs)\n",
    "            next_state, reward, done, _ = env.step(VALID_ACTIONS[action])\n",
    "            next_state = state_processor.process(sess, next_state)\n",
    "            next_state = np.append(state[:,:,1:], np.expand_dims(next_state, 2), axis=2)\n",
    "\n",
    "            # If our replay memory is full, pop the first element\n",
    "            if len(replay_memory) == replay_memory_size:\n",
    "                replay_memory.pop(0)\n",
    "\n",
    "            # Save transition to replay memory\n",
    "            # 更新回放记忆\n",
    "            replay_memory.append(Transition(state, action, reward, next_state, done))   \n",
    "\n",
    "            # Update statistics\n",
    "            stats.episode_rewards[i_episode] += reward\n",
    "            stats.episode_lengths[i_episode] = t\n",
    "\n",
    "            # Sample a minibatch from the replay memory\n",
    "            # 3.从D中抽一个mini-batch：(s, a, r, s')。\n",
    "            samples = random.sample(replay_memory, batch_size)\n",
    "            states_batch, action_batch, reward_batch, next_states_batch, done_batch = map(np.array, zip(*samples))\n",
    "\n",
    "            # Calculate q values and targets\n",
    "            # 4.使用老参数w计算Q-learning target（fixed Q-learning target）。\n",
    "            q_values_next = target_estimator.predict(sess, next_states_batch)\n",
    "            targets_batch = reward_batch + np.invert(done_batch).astype(np.float32) * discount_factor * np.amax(q_values_next, axis=1)\n",
    "\n",
    "            # Perform gradient descent update\n",
    "            # 5.优化Q-network和Q-learning之间的MSE\n",
    "            states_batch = np.array(states_batch)\n",
    "            loss = q_estimator.update(sess, states_batch, action_batch, targets_batch)\n",
    "\n",
    "            if done:\n",
    "                break\n",
    "\n",
    "            state = next_state\n",
    "            total_t += 1\n",
    "\n",
    "        # Add summaries to tensorboard\n",
    "        episode_summary = tf.Summary()\n",
    "        episode_summary.value.add(simple_value=epsilon, tag=\"episode/epsilon\")\n",
    "        episode_summary.value.add(simple_value=stats.episode_rewards[i_episode], tag=\"episode/reward\")\n",
    "        episode_summary.value.add(simple_value=stats.episode_lengths[i_episode], tag=\"episode/length\")\n",
    "        episode_summary.value.add(simple_value=current_process.cpu_percent(), tag=\"system/cpu_usage_percent\")\n",
    "        episode_summary.value.add(simple_value=current_process.memory_percent(memtype=\"vms\"), tag=\"system/v_memeory_usage_percent\")\n",
    "        q_estimator.summary_writer.add_summary(episode_summary, i_episode)\n",
    "        q_estimator.summary_writer.flush()\n",
    "        \n",
    "        yield total_t, plotting.EpisodeStats(\n",
    "            episode_lengths=stats.episode_lengths[:i_episode+1],\n",
    "            episode_rewards=stats.episode_rewards[:i_episode+1])\n",
    "\n",
    "    return stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/cer/.local/lib/python3.5/site-packages/tensorflow/python/ops/gradients_impl.py:91: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading model checkpoint /home/cer/Project/from_github/rl_learn/experiments/Breakout-v0/checkpoints/model...\n",
      "\n",
      "Populating replay memory...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-10-04 19:02:10,666] Starting new video recorder writing to /home/cer/Project/from_github/rl_learn/experiments/Breakout-v0/monitor/openaigym.video.1.3971.video000000.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 300 (44874) @ Episode 1/100, loss: 0.0029317396692931652\n",
      "Episode Reward: 3.0\n",
      "Step 160 (47175) @ Episode 11/100, loss: 0.0386776886880397858\n",
      "Episode Reward: 0.0\n",
      "Step 170 (49841) @ Episode 21/100, loss: 0.00122361839748919164\n",
      "Episode Reward: 0.0\n",
      "Step 150 (49995) @ Episode 22/100, loss: 0.0017673487309366465\n",
      "Copied model parameters to target network.\n",
      "Step 170 (52066) @ Episode 31/100, loss: 0.0017162094591185454\n",
      "Episode Reward: 0.0\n",
      "Step 210 (54363) @ Episode 41/100, loss: 0.0060867369174957275\n",
      "Episode Reward: 1.0\n",
      "Step 360 (56798) @ Episode 50/100, loss: 0.0476472973823547365"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-10-04 19:37:34,330] Starting new video recorder writing to /home/cer/Project/from_github/rl_learn/experiments/Breakout-v0/monitor/openaigym.video.1.3971.video000050.mp4\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step 340 (57138) @ Episode 51/100, loss: 0.0021206371020525694\n",
      "Episode Reward: 3.0\n",
      "Step 370 (59920) @ Episode 61/100, loss: 0.03912981599569321355\n",
      "Episode Reward: 4.0\n",
      "Step 70 (59999) @ Episode 62/100, loss: 0.0012430942151695495\n",
      "Copied model parameters to target network.\n",
      "Step 160 (62398) @ Episode 71/100, loss: 0.00104279548395425085\n",
      "Episode Reward: 0.0\n",
      "Step 170 (65048) @ Episode 81/100, loss: 0.00226859888061881077\n",
      "Episode Reward: 0.0\n",
      "Step 290 (67996) @ Episode 91/100, loss: 0.00119913835078477862\n",
      "Episode Reward: 2.0\n",
      "Step 0 (69991) @ Episode 99/100, loss: None00132140703499317175\n",
      "Copied model parameters to target network.\n",
      "Step 160 (70380) @ Episode 100/100, loss: 0.0230728760361671459"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-10-04 20:19:02,569] Finished writing results. You can upload them to the scoreboard via gym.upload('/home/cer/Project/from_github/rl_learn/experiments/Breakout-v0/monitor')\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph()\n",
    "\n",
    "# Where we save our checkpoints and graphs\n",
    "experiment_dir = os.path.abspath(\"./experiments/{}\".format(env.spec.id))\n",
    "\n",
    "# Create a glboal step variable\n",
    "global_step = tf.Variable(0, name='global_step', trainable=False)\n",
    "    \n",
    "# Create estimators\n",
    "q_estimator = Estimator(scope=\"q_estimator\", summaries_dir=experiment_dir)\n",
    "target_estimator = Estimator(scope=\"target_q\")\n",
    "\n",
    "# State processor\n",
    "state_processor = StateProcessor()\n",
    "\n",
    "# Run it!\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    count_ = 0\n",
    "    for t, stats in deep_q_learning(sess,\n",
    "                                    env,\n",
    "                                    q_estimator=q_estimator,\n",
    "                                    target_estimator=target_estimator,\n",
    "                                    state_processor=state_processor,\n",
    "                                    experiment_dir=experiment_dir,\n",
    "                                    num_episodes=100,\n",
    "                                    replay_memory_size=500000,\n",
    "                                    replay_memory_init_size=50000,\n",
    "                                    update_target_estimator_every=10000,\n",
    "                                    epsilon_start=1.0,\n",
    "                                    epsilon_end=0.1,\n",
    "                                    epsilon_decay_steps=500000,\n",
    "                                    discount_factor=0.99,\n",
    "                                    batch_size=32):\n",
    "\n",
    "        if count_ % 10 == 0:\n",
    "            print(\"\\nEpisode Reward: {}\".format(stats.episode_rewards[-1]))\n",
    "        count_ += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:py35]",
   "language": "python",
   "name": "conda-env-py35-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}