Ejercicio_Reinforcement_Learning_Pong.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aprendizaje por Refuerzo: el Pong"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El artículo completo en el blog [Aprende Machine Learning](http://www.aprendemachinelearning.com) en Español.\n",
    "\n",
    "Crearemos el juego del pong con interface gráfica de Matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "El Agente deberá aprender a jugar sólo mediante premios y castigos.\n",
    "\n",
    "Crearemos las clases:\n",
    "\n",
    "* Agente: implementará el algoritmo de QLearning\n",
    "* Environment: será nuestro tablero de juego\n",
    "\n",
    "Y crearemos una función para comenzar a jugar, donde entrenará iterando miles de partidas de pong.\n",
    "\n",
    "Finalmente, ejecutarmos 1 partida con el agente ya entrenado\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-27T10:21:01.578976Z",
     "start_time": "2020-12-27T10:20:58.063893Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Bad key savefig.frameon in file /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle, line 421 ('savefig.frameon : True')\n",
      "You probably need to get an updated matplotlibrc file from\n",
      "https://github.com/matplotlib/matplotlib/blob/v3.3.2/matplotlibrc.template\n",
      "or from the matplotlib source distribution\n",
      "\n",
      "Bad key verbose.level in file /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle, line 472 ('verbose.level  : silent      # one of silent, helpful, debug, debug-annoying')\n",
      "You probably need to get an updated matplotlibrc file from\n",
      "https://github.com/matplotlib/matplotlib/blob/v3.3.2/matplotlibrc.template\n",
      "or from the matplotlib source distribution\n",
      "\n",
      "Bad key verbose.fileo in file /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle, line 473 ('verbose.fileo  : sys.stdout  # a log filename, sys.stdout or sys.stderr')\n",
      "You probably need to get an updated matplotlibrc file from\n",
      "https://github.com/matplotlib/matplotlib/blob/v3.3.2/matplotlibrc.template\n",
      "or from the matplotlib source distribution\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead.\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The savefig.jpeg_quality rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The keymap.all_axes rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The animation.avconv_path rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In /Users/jbagnato/opt/anaconda3/envs/python36/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test.mplstyle: \n",
      "The animation.avconv_args rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from random import randint\n",
    "from time import sleep\n",
    "from IPython.display import clear_output\n",
    "from math import ceil,floor\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clase Agente"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-27T10:22:24.177788Z",
     "start_time": "2020-12-27T10:22:24.158177Z"
    }
   },
   "outputs": [],
   "source": [
    "class PongAgent:\n",
    "    \n",
    "    def __init__(self, game, policy=None, discount_factor = 0.1, learning_rate = 0.1, ratio_explotacion = 0.9):\n",
    "\n",
    "        # Creamos la tabla de politicas\n",
    "        if policy is not None:\n",
    "            self._q_table = policy\n",
    "        else:\n",
    "            position = list(game.positions_space.shape)\n",
    "            position.append(len(game.action_space))\n",
    "            self._q_table = np.zeros(position)\n",
    "        \n",
    "        self.discount_factor = discount_factor\n",
    "        self.learning_rate = learning_rate\n",
    "        self.ratio_explotacion = ratio_explotacion\n",
    "\n",
    "    def get_next_step(self, state, game):\n",
    "        \n",
    "        # Damos un paso aleatorio...\n",
    "        next_step = np.random.choice(list(game.action_space))\n",
    "        \n",
    "        # o tomaremos el mejor paso...\n",
    "        if np.random.uniform() <= self.ratio_explotacion:\n",
    "            # tomar el maximo\n",
    "            idx_action = np.random.choice(np.flatnonzero(\n",
    "                    self._q_table[state[0],state[1],state[2]] == self._q_table[state[0],state[1],state[2]].max()\n",
    "                ))\n",
    "            next_step = list(game.action_space)[idx_action]\n",
    "\n",
    "        return next_step\n",
    "\n",
    "    # actualizamos las politicas con las recompensas obtenidas\n",
    "    def update(self, game, old_state, action_taken, reward_action_taken, new_state, reached_end):\n",
    "        idx_action_taken =list(game.action_space).index(action_taken)\n",
    "\n",
    "        actual_q_value_options = self._q_table[old_state[0], old_state[1], old_state[2]]\n",
    "        actual_q_value = actual_q_value_options[idx_action_taken]\n",
    "\n",
    "        future_q_value_options = self._q_table[new_state[0], new_state[1], new_state[2]]\n",
    "        future_max_q_value = reward_action_taken  +  self.discount_factor*future_q_value_options.max()\n",
    "        if reached_end:\n",
    "            future_max_q_value = reward_action_taken #maximum reward\n",
    "\n",
    "        self._q_table[old_state[0], old_state[1], old_state[2], idx_action_taken] = actual_q_value + \\\n",
    "                                              self.learning_rate*(future_max_q_value -actual_q_value)\n",
    "    \n",
    "    def print_policy(self):\n",
    "        for row in np.round(self._q_table,1):\n",
    "            for column in row:\n",
    "                print('[', end='')\n",
    "                for value in column:\n",
    "                    print(str(value).zfill(5), end=' ')\n",
    "                print('] ', end='')\n",
    "            print('')\n",
    "            \n",
    "    def get_policy(self):\n",
    "        return self._q_table\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clase Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-27T10:22:39.529477Z",
     "start_time": "2020-12-27T10:22:39.493343Z"
    }
   },
   "outputs": [],
   "source": [
    "class PongEnvironment:\n",
    "    \n",
    "    def __init__(self, max_life=3, height_px = 40, width_px = 50, movimiento_px = 3):\n",
    "        \n",
    "        self.action_space = ['Arriba','Abajo']\n",
    "        \n",
    "        self._step_penalization = 0\n",
    "        \n",
    "        self.state = [0,0,0]\n",
    "        \n",
    "        self.total_reward = 0\n",
    "        \n",
    "        self.dx = movimiento_px\n",
    "        self.dy = movimiento_px\n",
    "        \n",
    "        filas = ceil(height_px/movimiento_px)\n",
    "        columnas = ceil(width_px/movimiento_px)\n",
    "        \n",
    "        self.positions_space = np.array([[[0 for z in range(columnas)] \n",
    "                                                  for y in range(filas)] \n",
    "                                                     for x in range(filas)])\n",
    "\n",
    "        self.lives = max_life\n",
    "        self.max_life=max_life\n",
    "        \n",
    "        self.x = randint(int(width_px/2), width_px) \n",
    "        self.y = randint(0, height_px-10)\n",
    "        \n",
    "        self.player_alto = int(height_px/4)\n",
    "\n",
    "        self.player1 = self.player_alto  # posic. inicial del player\n",
    "        \n",
    "        self.score = 0\n",
    "        \n",
    "        self.width_px = width_px\n",
    "        self.height_px = height_px\n",
    "        self.radio = 2.5\n",
    "\n",
    "    def reset(self):\n",
    "        self.total_reward = 0\n",
    "        self.state = [0,0,0]\n",
    "        self.lives = self.max_life\n",
    "        self.score = 0\n",
    "        self.x = randint(int(self.width_px/2), self.width_px) \n",
    "        self.y = randint(0, self.height_px-10)\n",
    "        return self.state\n",
    "\n",
    "    def step(self, action, animate=False):\n",
    "        self._apply_action(action, animate)\n",
    "        done = self.lives <=0 # final\n",
    "        reward = self.score\n",
    "        reward += self._step_penalization\n",
    "        self.total_reward += reward\n",
    "        return self.state, reward , done\n",
    "\n",
    "    def _apply_action(self, action, animate=False):\n",
    "        \n",
    "        if action == \"Arriba\":\n",
    "            self.player1 += abs(self.dy)\n",
    "        elif action == \"Abajo\":\n",
    "            self.player1 -= abs(self.dy)\n",
    "            \n",
    "        self.avanza_player()\n",
    "\n",
    "        self.avanza_frame()\n",
    "\n",
    "        if animate:\n",
    "            clear_output(wait=True);\n",
    "            fig = self.dibujar_frame()\n",
    "            plt.show()\n",
    "\n",
    "        self.state = (floor(self.player1/abs(self.dy))-2, floor(self.y/abs(self.dy))-2, floor(self.x/abs(self.dx))-2)\n",
    "    \n",
    "    def detectaColision(self, ball_y, player_y):\n",
    "        if (player_y+self.player_alto >= (ball_y-self.radio)) and (player_y <= (ball_y+self.radio)):\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "    \n",
    "    def avanza_player(self):\n",
    "        if self.player1 + self.player_alto >= self.height_px:\n",
    "            self.player1 = self.height_px - self.player_alto\n",
    "        elif self.player1 <= -abs(self.dy):\n",
    "            self.player1 = -abs(self.dy)\n",
    "\n",
    "    def avanza_frame(self):\n",
    "        self.x += self.dx\n",
    "        self.y += self.dy\n",
    "        if self.x <= 3 or self.x > self.width_px:\n",
    "            self.dx = -self.dx\n",
    "            if self.x <= 3:\n",
    "                ret = self.detectaColision(self.y, self.player1)\n",
    "\n",
    "                if ret:\n",
    "                    self.score = 10\n",
    "                else:\n",
    "                    self.score = -10\n",
    "                    self.lives -= 1\n",
    "                    if self.lives>0:\n",
    "                        self.x = randint(int(self.width_px/2), self.width_px)\n",
    "                        self.y = randint(0, self.height_px-10)\n",
    "                        self.dx = abs(self.dx)\n",
    "                        self.dy = abs(self.dy)\n",
    "        else:\n",
    "            self.score = 0\n",
    "\n",
    "        if self.y < 0 or self.y > self.height_px:\n",
    "            self.dy = -self.dy\n",
    "\n",
    "    def dibujar_frame(self):\n",
    "        fig = plt.figure(figsize=(5, 4))\n",
    "        a1 = plt.gca()\n",
    "        circle = plt.Circle((self.x, self.y), self.radio, fc='slategray', ec=\"black\")\n",
    "        a1.set_ylim(-5, self.height_px+5)\n",
    "        a1.set_xlim(-5, self.width_px+5)\n",
    "\n",
    "        rectangle = plt.Rectangle((-5, self.player1), 5, self.player_alto, fc='gold', ec=\"none\")\n",
    "        a1.add_patch(circle);\n",
    "        a1.add_patch(rectangle)\n",
    "        #a1.set_yticklabels([]);a1.set_xticklabels([]);\n",
    "        plt.text(4, self.height_px, \"SCORE:\"+str(self.total_reward)+\"  LIFE:\"+str(self.lives), fontsize=12)\n",
    "        if self.lives <=0:\n",
    "            plt.text(10, self.height_px-14, \"GAME OVER\", fontsize=16)\n",
    "        elif self.total_reward >= 1000:\n",
    "            plt.text(10, self.height_px-14, \"YOU WIN!\", fontsize=16)\n",
    "        return fig\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Juego"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-27T10:22:51.265359Z",
     "start_time": "2020-12-27T10:22:51.249319Z"
    }
   },
   "outputs": [],
   "source": [
    "def play(rounds=5000, max_life=3, discount_factor = 0.1, learning_rate = 0.1,\n",
    "         ratio_explotacion=0.9,learner=None, game=None, animate=False):\n",
    "\n",
    "    if game is None:\n",
    "        # si usamos movimiento_px = 5 creamos una tabla de politicas de 8x10\n",
    "        # si usamos movimiento_px = 3 la tabla sera de 14x17\n",
    "        game = PongEnvironment(max_life=max_life, movimiento_px = 3)\n",
    "        \n",
    "    if learner is None:\n",
    "        print(\"Begin new Train!\")\n",
    "        learner = PongAgent(game, discount_factor = discount_factor,learning_rate = learning_rate, ratio_explotacion= ratio_explotacion)\n",
    "\n",
    "    max_points= -9999\n",
    "    first_max_reached = 0\n",
    "    total_rw=0\n",
    "    steps=[]\n",
    "\n",
    "    for played_games in range(0, rounds):\n",
    "        state = game.reset()\n",
    "        reward, done = None, None\n",
    "        \n",
    "        itera=0\n",
    "        while (done != True) and (itera < 3000 and game.total_reward<=1000):\n",
    "            old_state = np.array(state)\n",
    "            next_action = learner.get_next_step(state, game)\n",
    "            state, reward, done = game.step(next_action, animate=animate)\n",
    "            if rounds > 1:\n",
    "                learner.update(game, old_state, next_action, reward, state, done)\n",
    "            itera+=1\n",
    "        \n",
    "        steps.append(itera)\n",
    "        \n",
    "        total_rw+=game.total_reward\n",
    "        if game.total_reward > max_points:\n",
    "            max_points=game.total_reward\n",
    "            first_max_reached = played_games\n",
    "        \n",
    "        if played_games %500==0 and played_games >1 and not animate:\n",
    "            print(\"-- Partidas[\", played_games, \"] Avg.Puntos[\", int(total_rw/played_games),\"]  AVG Steps[\", int(np.array(steps).mean()), \"] Max Score[\", max_points,\"]\")\n",
    "                \n",
    "    if played_games>1:\n",
    "        print('Partidas[',played_games,'] Avg.Puntos[',int(total_rw/played_games),'] Max score[', max_points,'] en partida[',first_max_reached,']')\n",
    "        \n",
    "    #learner.print_policy()\n",
    "    \n",
    "    return learner, game\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-27T10:24:06.516334Z",
     "start_time": "2020-12-27T10:22:51.808220Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Begin new Train!\n",
      "-- Partidas[ 500 ] Avg.Puntos[ 18 ]  AVG Steps[ 233 ] Max Score[ 120 ]\n",
      "-- Partidas[ 1000 ] Avg.Puntos[ 23 ]  AVG Steps[ 248 ] Max Score[ 170 ]\n",
      "-- Partidas[ 1500 ] Avg.Puntos[ 23 ]  AVG Steps[ 250 ] Max Score[ 200 ]\n",
      "-- Partidas[ 2000 ] Avg.Puntos[ 25 ]  AVG Steps[ 256 ] Max Score[ 200 ]\n",
      "-- Partidas[ 2500 ] Avg.Puntos[ 25 ]  AVG Steps[ 256 ] Max Score[ 240 ]\n",
      "-- Partidas[ 3000 ] Avg.Puntos[ 25 ]  AVG Steps[ 258 ] Max Score[ 240 ]\n",
      "-- Partidas[ 3500 ] Avg.Puntos[ 27 ]  AVG Steps[ 264 ] Max Score[ 240 ]\n",
      "-- Partidas[ 4000 ] Avg.Puntos[ 28 ]  AVG Steps[ 269 ] Max Score[ 310 ]\n",
      "-- Partidas[ 4500 ] Avg.Puntos[ 30 ]  AVG Steps[ 275 ] Max Score[ 490 ]\n",
      "Partidas[ 4999 ] Avg.Puntos[ 32 ] Max score[ 490 ] en partida[ 4202 ]\n"
     ]
    }
   ],
   "source": [
    "learner, game = play(rounds=5000, discount_factor = 0.2, learning_rate = 0.1, ratio_explotacion=0.85)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-27T10:35:16.554658Z",
     "start_time": "2020-12-27T10:25:44.533659Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATkAAAD4CAYAAACXIpFUAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAV00lEQVR4nO3de3SU9Z3H8feXcIskBkICBJIYFBAEQWoKAbbVctFU8LLasrIooYcKSFFcsUh3e1Zt/+lphdZzKC6x9RAKB4+7ICIVbBoKlsrFxAYkBUFXKCRpACEGUJHLb//IMJtAYm4TZubH53XOnJnn99w+k0M+PM8zMxlzziEi4qs24Q4gItKaVHIi4jWVnIh4TSUnIl5TyYmI19peyZ0lJSW5jIyMK7lLEbkKFBUVHXPOJdc174qWXEZGBoWFhVdylyJyFTCzg/XN0+mqiHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZVcPbZs2cLIkSNJSEggMTGRUaNG8e677wbnl5eXM23aNFJSUoiPj6d///4888wznD59GgDnHL/4xS/o27cvsbGxpKenM3/+fM6cORPcxtSpU2nfvj1xcXEkJiYybtw49u7dG5y/dOlSYmJiiIuLq3UrKyurM/OBAwe466676NKlCz169GD27NmcO3cOgBUrVtTaxjXXXIOZUVRUFMz79NNP07VrV7p27cq8efNo7Pd/bNq0idTU1DrnTZ06lR//+MfBfGZWK8eQIUOa/FzPnDnDtGnTuO6664iPj2fo0KGsX7++UVnl6qOSq0NVVRUTJkzgscce4/jx45SWlvLMM8/QoUMHAI4fP86IESP4/PPP2bp1KydPniQ/P5/Kyko++ugjAB5//HFyc3NZtmwZJ0+eZP369WzcuJGJEyfW2te8efM4deoUpaWl9OrVi2nTptWaP2LECE6dOlXr1rNnzzpzz5o1i27dulFeXk5xcTGbN29m8eLFAEyePLnWNhYvXsz111/P1772NQByc3NZs2YNO3fuZNeuXaxbt44lS5aE9Od6UWVlZTDHzp07m/xcz507R1paGps3b+bTTz/lpz/9KRMnTuTAgQOtkleim0quDvv27QNg0qRJxMTEEBsbyx133MHgwYMBWLhwIfHx8SxfvpyLfx8vLS2NF154gcGDB7N//34WL17MihUrGDFiBG3btmXgwIGsWrWKDRs2sHHjxsv2GRsby8SJEykuLm527o8//piJEyfSsWNHevToQXZ2NiUlJXUum5eXx5QpUzCz4PTcuXNJTU2lV69ezJ07l6VLlzY7S2vq1KkTzz77LBkZGbRp04YJEybQu3fv4FGpSE0quTr069ePmJgYcnJyWL9+PSdOnKg1/49//CP3338/bdrU/eMrKCggNTWVYcOG1RpPS0sjKyuL/Pz8y9Y5ffo0K1eupE+fPo3OOWvWLGbNmhWcnjNnDq+88gqfffYZpaWlrF+/nuzs7MvWO3jwIG+//TZTpkwJjpWUlARPHQGGDBlSb0GGw6XPtaaKigr27dvHwIEDr3AqiQYquTpce+21bNmyBTPjkUceITk5mXvuuYeKigoAPvnkE1JSUupd/9ixY/XOT0lJ4dixY8Hp559/ns6dOxMfH8+WLVv43e9+V2v5bdu20blz5+DthhtuCM5bvHhx8HQU4LbbbqOkpIRrr72W1NRUMjMzue+++y7LsGzZMr7xjW/Qu3fv4NipU6dISEgITickJHDq1KlGX5driqSkpODzef7554PjTXmuF509e5bJkyeTk5ND//79Q55Vol+jS87MYszsr2a2LjCdaGb5ZrY/cN+l9WJeeQMGDGDp0qUcPnyY3bt3U1ZWxhNPPAFA165dKS8vr3fdpKSkeueXl5eTlJQUnH7qqaeorKzkwIEDxMbG8sEHH9RaPisri8rKyuDt4jW/S124cIE777yT+++/n9OnT3Ps2DFOnDjB008/fdmyy5YtIycnp9ZYXFwcVVVVwemqqiri4uKCp7OhdOzYseDzeeqpp4LjjX2uF124cIGHH36Y9u3bs2jRopDnFD805UhuDrCnxvR8oMA51xcoCEx7qX///kydOpXdu3cDMHbsWF577TUuXLhQ5/KjR4/m0KFD7Nixo9b4oUOH2LZtG2PGjLlsnfT0dF544QXmzJnD559/3uSMx48f59ChQ8yePZsOHTrQtWtXvve97/Hmm2/WWu4vf/kLZWVlfOc736k1PnDgwFovAuzcuTOiT/+cc0ybNo2KigpWrVpFu3btwh1JIlSjSs7MUoHxwG9qDN8L5AUe5wH3hTRZGO3du5cFCxZw+PBhoLqcVq5cSVZWFgBPPvkkVVVV5OTkcPBg9Z+WLy0t5cknn2TXrl3069ePmTNnMnnyZLZt28b58+cpKSnhgQceYOzYsYwdO7bO/Y4bN46ePXuSm5vb5MxJSUn07t2bF198kXPnzlFZWUleXl6t62xQ/QLDAw88QHx8fK3xKVOmsHDhQkpLSykrK2PBggVMnTq1SRm++OKLWrfWONW96NFHH2XPnj288cYbxMbGttp+JPo19kjuV8A8oOahS3fnXDlA4L5bXSua2XQzKzSzwqNHj7Yk6xUTHx/P9u3bGT58OJ06dSIrK4tBgwaxYMECABITE3nnnXdo164dw4cPJz4+njFjxpCQkBB84WDRokV8//vf56GHHiIuLo7s7Gxuv/12Vq1a9ZX7/uEPf8jPf/7z4Pvptm7detl7xy6+X2/mzJnMnDkzuO7q1avZsGEDycnJ9OnTh7Zt2/LLX/4yOP+LL77g1VdfvexUFWDGjBncfffd3HzzzQwaNIjx48czY8aMRv/MSktLiY2NrXVr6HTzUo19rgcPHmTJkiUUFxfTo0eP4LIrVqxo0v7k6mAN/W9rZhOAu5xzs8zsduAp59wEM6t0znWusdwJ59xXXpfLzMx0+rYuEQk1MytyzmXWNa8xX0k4CrjHzO4COgLXmtlyoMLMUpxz5WaWAhwJXWQRkdBo8HTVOfcj51yqcy4DeBDY6Jx7CFgLXDzvyQFeb7WUIiLN1JL3yf0MGGdm+4FxgWkRkYjSmNPVIOfcJmBT4PEnwOXvhRARiSD6xIOIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5rsOTMrKOZ7TCznWZWYmbPBcYTzSzfzPYH7ru0flwRkaZpzJHcGWC0c24IcAuQbWZZwHygwDnXFygITIuIRJQGS85VOxWYbBe4OeBeIC8wngfc1xoBRURaolHX5MwsxsyKgSNAvnNuO9DdOVcOELjv1mopRUSaqVEl55w775y7BUgFhpnZoMbuwMymm1mhmRUePXq0mTFFRJqnbVMWds5VmtkmIBuoMLMU51y5maVQfZRX1zq5QC5AZmama2FekavamTNnKCgooKioiMLCIo6fOEGbNm1IS0tl2Ne/zsiRI7n11lsxs3BHjRiNeXU12cw6Bx7HAmOBvcBaICewWA7weitlFLnqVVRU8Pjjc+iRksLjT8xlXf6f+bLttSRdN4AuvfpRXnmGvJWrGD/hHvrd2J9FixZx9uzZcMeOCObcVx9cmdlgql9YiKG6FF91zv3EzLoCrwLpwN+B7zrnjn/VtjIzM11hYWFIgotcDZxz5L70EvPnz6fPTUMZdOtIOicmfeXypQc/onjbJtpynry8pQwbNuwKJg4PMytyzmXWNa/B01Xn3C5gaB3jnwBjWh5PROpy/vx5Hpk+nT/kFzD+Xx4huUfPBtcxM1Iz+tDruhvY9/573HHHneTmLmHixIlXIHFkatI1ORG5ch6dNYtNf36Hex96lPYdOjZpXTPjxsG3ktitBzNmPkqnTp0YP358KyWNbPpYl0gEWr16NWvWrCX7galNLriaknv04o5/fogpOTmUl5eHMGH0UMmJRJjTp0/zyPTpjL77QTp0bH7BXdQz/XpuvPnrzJ79WAjSRR+VnEiEycvLo3vPdFLSMkK2zaEjbucP+fkcPHgwZNuMFio5kQiz+MX/YsDQESHdZvsOHel/8628/PLLId1uNFDJiUSQzz77jA8/3E/P9OtDvu2U9BvYtHlzyLcb6VRyIhHk/fffp3uPnrRtG/o3PnTvmcauXbtCvt1Ip5ITiSBVVVV0iL2mVbbdMTaW06dPt8q2I5lKTiSCtG/fnvOt9HGsc2fP0r5d+1bZdiRTyYlEkAEDBlDxj1LchQsh3/axI+X06ds35NuNdCo5kQjSrVs3Oid05pOjFSHfdvmhA4waOTLk2410KjmRCDNp0oPs3bUjpNu8cOEC+3cXMXnyv4Z0u9FAJScSYebMmcPeXYWcPlkVsm3uLykmJaUHI0aE9v130UAlJxJhUlNTmT17Npve/G8a+lNojXGq6lPeKXiDxb9edFX+MU2VnEgE+slzzxHXsR3bN21oUdF9eeYL8tcs5/HHZjPyKrweByo5kYjUrl073nprA5VH/s6f33qNc814W0lV5XHWrljCmG99k2effTb0IaOESk4kQiUnJ7Nt61ZSu3fh1d8s5OP9f2vUUd25s1/y3tZN/M/LL/CDWTP4zUsv0abN1furrj+aKRLBEhISeH3NGn7/+98z54l/Y1vBOm64aSgpqdeR1L0nHa/phHOOk5+e4Og/Sik7+CH7SooZNWoURUWF9OnTJ9xPIewa/I6HUNJ3PIg0n3OO7du3s3z5Crbv2MHePX8LfkwrKbkbQwYP5rbbvsmUKVNIT08Pc9orq0Xf8SAikcHMyMrKIisrKzh28SDlanzVtLFUciJRTOXWsKv3aqSIXBVUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLitQZLzszSzOxPZrbHzErMbE5gPNHM8s1sf+C+S+vHFRFpmsYcyZ0D5jrnBgBZwA/M7CZgPlDgnOsLFASmRUQiSoMl55wrd869F3h8EtgD9ALuBfICi+UB97VSRhGRZmvSNTkzywCGAtuB7s65cqguQqBbPetMN7NCMys8evRoC+OKiDRNo0vOzOKAVcATzrmqxq7nnMt1zmU65zKTk5Obk1FEpNkaVXJm1o7qglvhnFsdGK4ws5TA/BTgSOtEFBFpvsa8umrAb4E9zrmFNWatBXICj3OA10MfT0SkZRrzbV2jgIeB982sODD278DPgFfNbBrwd+C7rZJQRKQFGiw559wWoL7vPRsT2jgiIqGlTzyIiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuK1BkvOzF42syNmtrvGWKKZ5ZvZ/sB9l9aNKSLSPI05klsKZF8yNh8ocM71BQoC0yIiEafBknPOvQ0cv2T4XiAv8DgPuC+0sUREQqO51+S6O+fKAQL33UIXSUQkdNq29g7MbDowHSC9J7DXWmdH/V3rbFdEolpzj+QqzCwFIHB/pL4FnXO5zrlM51xmsl6eEJErrLkltxbICTzOAV4PTRwRkdBqzFtIVgJbgRvN7LCZTQN+Bowzs/3AuMC0iEjEafCanHNuUj2zxoQ4i4hIyOkTDyLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJea1HJmVm2mX1gZh+a2fxQhRIRCZVml5yZxQC/Br4N3ARMMrObQhVMRCQUWnIkNwz40Dn3v865L4FXgHtDE0tEJDTatmDdXsChGtOHgeGXLmRm04HpAOnp6dD/YAt2KSLSNC05krM6xtxlA87lOucynXOZycnJLdidiEjTtaTkDgNpNaZTgbKWxRERCa2WlNy7QF8z621m7YEHgbWhiSUiEhrNvibnnDtnZrOBt4AY4GXnXEnIkomIhEBLXnjAOfcm8GaIsoiIhJw+8SAiXlPJiYjXVHIi4jWVnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJeU8mJiNdUciLiNZWciHhNJSciXlPJiYjXVHIi4jWVnIh4zZy77LtnWm9nZkeB1vq6riTgWCttu7VFa/ZozQ3Rmz1ac0PrZr/OOVfnN2Vd0ZJrTWZW6JzLDHeO5ojW7NGaG6I3e7TmhvBl1+mqiHhNJSciXvOp5HLDHaAFojV7tOaG6M0erbkhTNm9uSYnIlIXn47kREQuo5ITEa95UXJmlm1mH5jZh2Y2P9x56mNmL5vZETPbXWMs0czyzWx/4L5LODPWx8zSzOxPZrbHzErMbE5gPKLzm1lHM9thZjsDuZ8LjEd07ovMLMbM/mpm6wLT0ZL7gJm9b2bFZlYYGAtL9qgvOTOLAX4NfBu4CZhkZjeFN1W9lgLZl4zNBwqcc32BgsB0JDoHzHXODQCygB8Efs6Rnv8MMNo5NwS4Bcg2sywiP/dFc4A9NaajJTfAt5xzt9R4b1x4sjvnovoGjADeqjH9I+BH4c71FXkzgN01pj8AUgKPU4APwp2xkc/jdWBcNOUHrgHeA4ZHQ24gleoyGA2si6Z/L8ABIOmSsbBkj/ojOaAXcKjG9OHAWLTo7pwrBwjcdwtzngaZWQYwFNhOFOQPnPIVA0eAfOdcVOQGfgXMAy7UGIuG3AAO+IOZFZnZ9MBYWLK3vRI7aWVWx5jeF9NKzCwOWAU84ZyrMqvrxx9ZnHPngVvMrDPwmpkNCnOkBpnZBOCIc67IzG4Pc5zmGOWcKzOzbkC+me0NVxAfjuQOA2k1plOBsjBlaY4KM0sBCNwfCXOeeplZO6oLboVzbnVgOGryO+cqgU1UXxeN9NyjgHvM7ADwCjDazJYT+bkBcM6VBe6PAK8BwwhTdh9K7l2gr5n1NrP2wIPA2jBnaoq1QE7gcQ7V17oijlUfsv0W2OOcW1hjVkTnN7PkwBEcZhYLjAX2EuG5nXM/cs6lOucyqP43vdE59xARnhvAzDqZWfzFx8AdwG7ClT3cFyhDdJHzLmAf8BHwH+HO8xU5VwLlwFmqj0CnAV2pvri8P3CfGO6c9WT/J6ovA+wCigO3uyI9PzAY+Gsg927gPwPjEZ37kudwO///wkPE5wauB3YGbiUXfyfDlV0f6xIRr/lwuioiUi+VnIh4TSUnIl5TyYmI11RyIuI1lZyIeE0lJyJe+z8tWu9E8cevCgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 360x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner2 = PongAgent(game, policy=learner.get_policy())\n",
    "learner2.ratio_explotacion = 1.0  # con esto quitamos las elecciones aleatorias al jugar\n",
    "player = play(rounds=1, learner=learner2, game=game, animate=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}