Aprendizaje Reforzado/Cuadernos/ar_Metodo_Entropia_Cruzada_implementacion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5e6f8f0e",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<img src=\"../Imagenes/logo-final-ap.png\"  width=\"80\" height=\"80\" align=\"left\"/> \n",
    "</figure>\n",
    "\n",
    "# <span style=\"color:blue\"><left>Aprendizaje Profundo</left></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b718a9ce",
   "metadata": {
    "tags": []
   },
   "source": [
    "# <span style=\"color:red\"><center>Implementación del método entropía cruzada</center></span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "321bc449-4793-4745-af77-7306a1228453",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/trainer.png\" width=\"800\" height=\"800\" align=\"center\"/>\n",
    "</center>\n",
    "</figure>\n",
    "\n",
    "\n",
    "Fuente: Alvaro Montenegro"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0342c2-a624-40ab-a51b-1f1920795daf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "##   <span style=\"color:blue\">Autores</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9969c880-f590-43c2-a3c2-e9f2c48b5ab3",
   "metadata": {},
   "source": [
    "- Alvaro  Montenegro, PhD, ammontenegrod@unal.edu.co\n",
    "- Daniel  Montenegro, Msc, dextronomo@gmail.com "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3b93a6f-3ff7-4142-8e53-e48af5ea76e8",
   "metadata": {},
   "source": [
    "##   <span style=\"color:blue\">Asesora Medios y Marketing digital</span>\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c56a1c-ca23-4afb-a7f6-2c46ca4ba1bf",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "- Maria del Pilar Montenegro, pmontenegro88@gmail.com "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd06c80a-262c-4050-ab8d-923e80d1b66e",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Referencias</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75ac7d2",
   "metadata": {},
   "source": [
    "1. [Alvaro Montenegro y Daniel Montenegro, Inteligencia Artificial y Aprendizaje Profundo, 2021](https://github.com/AprendizajeProfundo/Diplomado)\n",
    "1. [Maxim Lapan, Deep Reinforcement Learning Hands-On: Apply modern RL methods to practical problems of chatbots, robotics, discrete optimization, web automation, and more, 2nd Edition, 2020](http://library.lol/main/F4D1A90C476A576238E8FE1F47602C67)\n",
    "1. [Richard S. Sutton, Andrew G. Barto, Reinforcement learning: an introduction, 2nd edition, 2020](http://library.lol/main/6502B74CE247C4CD4D4FB54747AD7C7E)\n",
    "1. [Praveen Palanisamy - Hands-On Intelligent Agents with OpenAI Gym_ Your Guide to Developing AI Agents Using Deep Reinforcement Learning, 2020](http://library.lol/main/E4FD128CF9B93E0F7A542B053330517A)\n",
    "1. [Turing Paper 1936](http://www.thocp.net/biographies/papers/turing_oncomputablenumbers_1936.pdf)\n",
    "1. [Solving a Reinforcement Learning Problem Using Cross-Entropy Method](https://towardsdatascience.com/solving-a-reinforcement-learning-problem-using-cross-entropy-method-23d9726a737)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffd63d24",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Contenido</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "683ee4c4",
   "metadata": {},
   "source": [
    "* [Introducción](#Introducción)\n",
    "* [Implementación básica del Método Entropia Cruzada](#Implementación-básica-del-Método-Entropia-Cruzada)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be027523",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Introducción</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaee1cf1-9a63-4312-a3b7-50c153ff1582",
   "metadata": {},
   "source": [
    "La entropía cruzada se considera un algoritmo evolutivo: algunos individuos se seleccionan de una población, y solo los considerados buenos o  `élite` gobiernan las características de las generaciones futuras.\n",
    "\n",
    "Esencialmente, lo que hace el método de `entropía-cruzada`(cross-entropy) es tomar un montón de entradas, ver las salidas producidas, elegir las entradas que han llevado a las mejores salidas y ajustar el agente hasta que estemos satisfechos con las salidas que vemos.\n",
    "\n",
    "Antes de describir el método técnicamente vamos a recordar los elementos básicos del aprendizaje reforzado, que debemos tener en cuenta en la implementación.\n",
    "\n",
    "En esta lección usaremos los ambientes *CartPole* y *FrozenLake* de [OpenAI Gym](https://github.com/openai/gym)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "679b4575-2a33-434c-b8d0-6fe0132a6d28",
   "metadata": {},
   "source": [
    "<figure>\n",
    "<center>\n",
    "<img src=\"../Imagenes/agente-ambiente.png\" width=\"600\" height=\"500\" align=\"center\"/>\n",
    "</center> \n",
    "</figure>\n",
    "\n",
    "\n",
    "Fuente: Alvaro Montenegro"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42f0ccb4-0bd5-488c-98b2-78a98edaaf18",
   "metadata": {},
   "source": [
    "- El ambiente es el objeto mas importante del aprendizaje reforzado. Debe definirse en cada problema.\n",
    "- El agente recibe un conjunto de `acciones` que se permite sean ejecutadas en el ambiente. Estas acciones son entregadas al agente por un selector de acciones, el se basa en el conjunto de acciones disponibles en el ambiente.\n",
    "- El ambiente provee el tamaño y bordes de las `observaciones` para el agente.\n",
    "- El ambiente dispone de un método *step* para ejecutar un acción. El método regresa la nueva observación, la `recompensa` y la indicación de si el `episodio` ha terminado (*done*).\n",
    "- Un método *reset* que retorna al ambiente a su *estado inicial* y entrega la primera observación."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf69bef-2f6f-47c2-bdca-d032646bdf9d",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Implementación básica del Método Entropia Cruzada</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5407d82a-0cf8-47e3-b9c2-7e5559ea4693",
   "metadata": {},
   "source": [
    "Primero demos una mirada al espacio de observaciones del ambiente. El método render de un objeto de tipo Env renderiza el espacio de acciones. Para interpretrar *S* es start, *F* es free, *H*  es hole y *G* es goal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2ebd074-a57d-481b-91a7-5056bd17ce53",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Importa librerías</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0d3b4116-1e15-4991-9c72-07f73cf5a1fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym, gym.spaces\n",
    "from collections import namedtuple\n",
    "import numpy as np\n",
    "\n",
    "#from torch.utils.tensorboard import SummaryWriter\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0e7c2a9-cd83-4d83-b920-515d1d7eb7a0",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Envuelve el espacio de acciones para el ambiente FrozenLake</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3027fd6-a44f-436f-8716-8a691141dd30",
   "metadata": {},
   "source": [
    "Con esta clase se envuelve el epacio de acciones para convertirlo en un espacio de tipo de tal manera que sea compatible con el tipo de espacio de CartPole, que vamos a estudiar en la siguiente lección. El nuevo tipo de observation será de  *Box* y contendrá un vector de tamaño 16, de tipo *onehot*. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "db5b9337-e20a-4af2-83ee-ec3eb36de48c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gym\n",
    "\n",
    "class DiscreteOneHotWrapper(gym.ObservationWrapper):\n",
    "    def __init__(self, env):\n",
    "        super(DiscreteOneHotWrapper, self).__init__(env)\n",
    "        assert isinstance(env.observation_space, gym.spaces.Discrete)\n",
    "        shape = (env.observation_space.n, )\n",
    "        self.observation_space = gym.spaces.Box(\n",
    "            0.0, 1.0, shape=shape, dtype=np.float32)\n",
    "        \n",
    "    def observation(self, observation):\n",
    "        res = np.copy(self.observation_space.low)\n",
    "        res[observation] = 1.0\n",
    "        return res\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3bc7bf6-5b47-4599-9d28-3c77d4fee683",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Jerarquía de clases  Net</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "91bb5232-37ed-406c-a71c-ce4a63de3527",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "class Net_basic(nn.Module):\n",
    "    def __init__(self, obs_size, hidden_size, n_actions):\n",
    "        super(Net_basic, self).__init__()\n",
    "        \n",
    "        # configura bloque secuencial\n",
    "        self.net = nn.Sequential(\n",
    "            nn.Linear(obs_size, hidden_size),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(hidden_size, n_actions)\n",
    "        )\n",
    "        \n",
    "        # configura optimizadores y función de pérdida\n",
    "        # debe implementarse en cada caso\n",
    "        self.optimizer = self.configure_optimizers()\n",
    "        self.loss = self.configure_looses()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.net(x)\n",
    " \n",
    "    def configure_optimizers(self):\n",
    "        raise NotImplementedError\n",
    "    \n",
    "    def configure_looses(self):\n",
    "        raise NotImplementedError\n",
    "    \n",
    "    def training_step(self, train_batch):\n",
    "        raise NotImplementedError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "89601d28-6f72-48c2-81bc-55ea8b4f852e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "class Net(Net_basic): \n",
    "    def __init__(self, obs_size, hidden_size, n_actions):\n",
    "        super(Net, self).__init__(obs_size, hidden_size, n_actions)\n",
    "           \n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)\n",
    "        return optimizer\n",
    "    \n",
    "    def configure_looses(self):\n",
    "        loss = torch.nn.CrossEntropyLoss()\n",
    "        return loss\n",
    "    \n",
    "    def training_step(self, train_batch):\n",
    "        x, y = train_batch\n",
    "        y_hat = self.net(x)\n",
    "        loss = self.loss(y_hat, y)\n",
    "        #self.log('train_loss', loss)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a543b172-5ac2-4686-b47a-5ca011c9c813",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Clase Trainer</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f760ebbc-db94-46ac-a9ed-7771c3eab6fd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "class Trainer(object):\n",
    "    def __init__(self, model, writer, verbose=True):\n",
    "        self.model = model\n",
    "        self.writer = writer\n",
    "        self.verbose = verbose\n",
    "        \n",
    "    def fit(self, iter_no, train_dataloader, reward_bound, reward_mean):\n",
    "        loss_l = []\n",
    "        for batch in train_dataloader:\n",
    "            self.model.optimizer.zero_grad()\n",
    "            loss = self.model.training_step(batch)\n",
    "            loss_l.append(loss.item())\n",
    "            loss.backward()\n",
    "            self.model.optimizer.step()\n",
    "            \n",
    "        mean_loss = np.mean(loss_l)\n",
    "        \n",
    "        # escribe en el log de writer\n",
    "        self.writer.add_scalar(\"perdida\", mean_loss, iter_no)\n",
    "        self.writer.add_scalar(\"recompensa_promedio\", reward_mean, iter_no)\n",
    "        self.writer.add_scalar(\"recompensa_frontera\", reward_bound, iter_no)\n",
    "        \n",
    "        # escribe en la pantalla\n",
    "        if self.verbose:\n",
    "            print(\"%d: pérdida promedio =%.3f, recompensa promedio=%.1f, cota recompensa=%.1f\" % (\n",
    "            iter_no, mean_loss, reward_mean, reward_bound))\n",
    "            \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf7ea8ee-8a23-4c6c-b74a-079f167ae858",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Clase iterable Batch</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a1159fc6-c604-4edd-baa1-b473fb157b3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "import numpy as np\n",
    "\n",
    "#  tupla para retornar todos los datos de un episodio completo\n",
    "Episode = namedtuple('Episode', field_names=['reward', 'steps'])\n",
    "# tupla para para almacenar las parejas (observación, acción) de cada paso \n",
    "# en un episodio\n",
    "EpisodeStep = namedtuple('EpisodeStep', field_names=['observation', 'action'])\n",
    "\n",
    "\n",
    "class Batch(object):\n",
    "    '''\n",
    "    Implementa la generación iterativa de lotes de  datos\n",
    "    '''\n",
    "    def __init__(self, env, net, batch_size):\n",
    "        self.env = env\n",
    "        self.net = net\n",
    "        self.batch_size = batch_size\n",
    "   \n",
    "    # hace la clase iterable       \n",
    "    def __iter__(self):\n",
    "        return self\n",
    "    \n",
    "    # define iterador\n",
    "    def __next__(self):\n",
    "        # lista que contendrá el lote  de datos a entregar\n",
    "        batch = [] \n",
    "        # recompensa de cada episodio\n",
    "        episode_reward = 0.0 \n",
    "        # lista de parejas (observación, acción) de cada episodio\n",
    "        episode_steps = [] \n",
    "        \n",
    "        # reinicia el ambiente, para empezar a generar datos\n",
    "        # recibe la primera observación\n",
    "        obs = self.env.reset()\n",
    "        # alias para la función softmax\n",
    "        sm = nn.Softmax(dim=1)\n",
    "        \n",
    "        # ciclo para generar lote(batch) de datos\n",
    "        while True:\n",
    "            # convierte obs a un tensor. debe pasar como lista\n",
    "            obs_v = torch.FloatTensor(np.array([obs]))\n",
    "            # calcula el tensor de puntaje para las acciones: self.net(obs_v)\n",
    "            # Transforma los puntajes entregado por la red en una distribución\n",
    "            # de probabilidad, la cual viene en un tensor\n",
    "            act_probs_v = sm(self.net(obs_v))\n",
    "            # extrae la distribución del tensor a d-array de  Numpy\n",
    "            act_probs = act_probs_v.data.numpy()[0]\n",
    "            # selecciona una acción aleatoriamente usando la distribución\n",
    "            action = np.random.choice(len(act_probs), p=act_probs)\n",
    "            # entrega la acción al ambiente y recibe respuesta del ambiete\n",
    "            next_obs, reward, is_done, _ = self.env.step(action)\n",
    "            # actualzia la recompensa\n",
    "            episode_reward += reward\n",
    "            # agrega la pareja (observación, acción) a la lista de pasos\n",
    "            episode_steps.append(EpisodeStep(observation=obs, action=action))\n",
    "            # al terminar el episodio\n",
    "            if is_done:\n",
    "                # agrega los datos al batch: (recompensa, lista de parejas (obs, acción))\n",
    "                batch.append(Episode(reward=episode_reward, steps=episode_steps))\n",
    "                # reinicia objetos para el siguiente episodio\n",
    "                episode_reward = 0.0\n",
    "                episode_steps = []\n",
    "                next_obs = env.reset()\n",
    "                # si completo el lote de datos, lo retorna y termina\n",
    "                if len(batch) == self.batch_size:\n",
    "                    return batch\n",
    "            obs = next_obs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f814382f-f0df-4475-9e73-dc0c639df154",
   "metadata": {},
   "source": [
    "#### Prueba del iterador de lotes usando el ambiente FrozenLake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "ebde585c-42fe-4faa-ac0e-b1ce6907b861",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Episode(reward=0.0, steps=[EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=2), EpisodeStep(observation=array([0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=2), EpisodeStep(observation=array([0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=1), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=2), EpisodeStep(observation=array([0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=2)]),\n",
       " Episode(reward=0.0, steps=[EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=1), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=1), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=2), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=2), EpisodeStep(observation=array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=2), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=0), EpisodeStep(observation=array([0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=3), EpisodeStep(observation=array([0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "       dtype=float32), action=1)])]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# selecciona un ambiente FrozenLake\n",
    "env = DiscreteOneHotWrapper(gym.make(\"FrozenLake-v1\"))\n",
    "\n",
    "# extrae tamaños de acciones y observaciones en el ambiente\n",
    "obs_size = env.observation_space.shape[0]\n",
    "n_actions = env.action_space.n\n",
    "# define tamaño capa oculta de la red\n",
    "HIDDEN_SIZE = 128\n",
    "#instancia un objeto Net\n",
    "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
    "\n",
    "# define tamaño de los lotes\n",
    "BATCH_SIZE = 2\n",
    "# instancia un iterador Batch\n",
    "batch = Batch(env, net, BATCH_SIZE)\n",
    "\n",
    "# extrae el primer lote de datos\n",
    "dato = next(batch)\n",
    "dato"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5c11ace-0bf3-4d89-9b88-e349b433216d",
   "metadata": {},
   "source": [
    "#### Prueba del iterador de lotes usando el ambiente CartPole"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7c0fe25a-0a54-49f6-9603-a1c5d06035b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Episode(reward=13.0, steps=[EpisodeStep(observation=array([-0.01702714,  0.04969298,  0.0171482 ,  0.00579553], dtype=float32), action=1), EpisodeStep(observation=array([-0.01603328,  0.24456486,  0.01726411, -0.28142798], dtype=float32), action=0), EpisodeStep(observation=array([-0.01114198,  0.04920096,  0.01163555,  0.01664959], dtype=float32), action=0), EpisodeStep(observation=array([-0.01015796, -0.14608592,  0.01196854,  0.31298083], dtype=float32), action=0), EpisodeStep(observation=array([-0.01307968, -0.3413763 ,  0.01822816,  0.6094141 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.01990721, -0.53674823,  0.03041644,  0.90778214], dtype=float32), action=0), EpisodeStep(observation=array([-0.03064217, -0.73226845,  0.04857208,  1.2098678 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.04528754, -0.9279828 ,  0.07276944,  1.5173677 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.0638472 , -0.73381263,  0.1031168 ,  1.2482586 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.07852345, -0.5401527 ,  0.12808196,  0.9895744 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.0893265 , -0.7367348 ,  0.14787346,  1.3195853 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.1040612 , -0.543759  ,  0.17426516,  1.0765972 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.11493638, -0.35131323,  0.1957971 ,  0.8432757 ], dtype=float32), action=1)]),\n",
       " Episode(reward=44.0, steps=[EpisodeStep(observation=array([ 0.01629445, -0.0020105 , -0.01588094,  0.02492443], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01625424, -0.19690114, -0.01538245,  0.31255475], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01231621, -0.3918006 , -0.00913136,  0.6003471 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.0044802 , -0.19655213,  0.00287559,  0.30480197], dtype=float32), action=0), EpisodeStep(observation=array([ 5.4915954e-04, -3.9171493e-01,  8.9716250e-03,  5.9839040e-01],\n",
       "       dtype=float32), action=1), EpisodeStep(observation=array([-0.00728514, -0.19671966,  0.02093943,  0.30854687], dtype=float32), action=0), EpisodeStep(observation=array([-0.01121953, -0.3921336 ,  0.02711037,  0.6077593 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.0190622 , -0.19740097,  0.03926556,  0.32373714], dtype=float32), action=1), EpisodeStep(observation=array([-0.02301022, -0.00285949,  0.0457403 ,  0.04369095], dtype=float32), action=1), EpisodeStep(observation=array([-0.02306741,  0.19157773,  0.04661412, -0.23421688], dtype=float32), action=1), EpisodeStep(observation=array([-0.01923586,  0.38600373,  0.04192978, -0.5118395 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.01151578,  0.19031703,  0.03169299, -0.2062437 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.00770944, -0.00524345,  0.02756812,  0.0962659 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.00781431, -0.20074946,  0.02949343,  0.39751747], dtype=float32), action=0), EpisodeStep(observation=array([-0.0118293 , -0.39627716,  0.03744378,  0.69935125], dtype=float32), action=1), EpisodeStep(observation=array([-0.01975484, -0.20169379,  0.05143081,  0.41868672], dtype=float32), action=0), EpisodeStep(observation=array([-0.02378872, -0.39750537,  0.05980454,  0.7271297 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.03173883, -0.203259  ,  0.07434714,  0.45385313], dtype=float32), action=1), EpisodeStep(observation=array([-0.03580401, -0.00926277,  0.0834242 ,  0.18550056], dtype=float32), action=0), EpisodeStep(observation=array([-0.03598926, -0.20547311,  0.08713421,  0.5032908 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.04009873, -0.01168036,  0.09720003,  0.23928952], dtype=float32), action=0), EpisodeStep(observation=array([-0.04033233, -0.2080468 ,  0.10198582,  0.5609807 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.04449327, -0.01449289,  0.11320543,  0.30209026], dtype=float32), action=1), EpisodeStep(observation=array([-0.04478313,  0.17884889,  0.11924724,  0.04714528], dtype=float32), action=1), EpisodeStep(observation=array([-0.04120615,  0.372077  ,  0.12019014, -0.20566288], dtype=float32), action=1), EpisodeStep(observation=array([-0.03376461,  0.5652935 ,  0.11607689, -0.45814592], dtype=float32), action=1), EpisodeStep(observation=array([-0.02245874,  0.75859964,  0.10691397, -0.71210384], dtype=float32), action=1), EpisodeStep(observation=array([-0.00728675,  0.9520915 ,  0.09267189, -0.9693116 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01175508,  0.7558559 ,  0.07328565, -0.6490147 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.0268722 ,  0.5597937 ,  0.06030536, -0.33418408], dtype=float32), action=0), EpisodeStep(observation=array([ 0.03806807,  0.3638676 ,  0.05362168, -0.02311015], dtype=float32), action=1), EpisodeStep(observation=array([ 0.04534543,  0.5581812 ,  0.05315948, -0.2984044 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.05650905,  0.7525067 ,  0.04719139, -0.57385933], dtype=float32), action=1), EpisodeStep(observation=array([ 0.07155918,  0.9469363 ,  0.03571421, -0.85131   ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.09049791,  1.1415536 ,  0.01868801, -1.132552  ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.11332899,  0.9461921 , -0.00396303, -0.8340669 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.13225283,  0.7511245 , -0.02064437, -0.542633  ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.14727531,  0.9465304 , -0.03149703, -0.8417484 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.16620593,  0.7518523 , -0.048332  , -0.5591346 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.18124297,  0.94761807, -0.05951469, -0.86664456], dtype=float32), action=1), EpisodeStep(observation=array([ 0.20019533,  1.1434972 , -0.07684758, -1.17743   ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.22306527,  1.3395286 , -0.10039619, -1.4931803 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.24985585,  1.5357186 , -0.1302598 , -1.8154502 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.2805702,  1.7320275, -0.1665688, -2.1456072], dtype=float32), action=1)]),\n",
       " Episode(reward=23.0, steps=[EpisodeStep(observation=array([-0.00571877, -0.00521932, -0.03667402,  0.01875161], dtype=float32), action=0), EpisodeStep(observation=array([-0.00582316, -0.19979666, -0.03629899,  0.2996416 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.00981909, -0.00417661, -0.03030616, -0.00426472], dtype=float32), action=1), EpisodeStep(observation=array([-0.00990262,  0.19136657, -0.03039145, -0.30635345], dtype=float32), action=1), EpisodeStep(observation=array([-0.00607529,  0.3869081 , -0.03651852, -0.60846406], dtype=float32), action=0), EpisodeStep(observation=array([ 0.00166287,  0.19231522, -0.0486878 , -0.32750335], dtype=float32), action=1), EpisodeStep(observation=array([ 0.00550918,  0.3880953 , -0.05523787, -0.63513404], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01327108,  0.19378555, -0.06794055, -0.36034563], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01714679, -0.00030818, -0.07514746, -0.08983587], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01714063, -0.19427706, -0.07694418,  0.17822373], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01325509, -0.3882184 , -0.0733797 ,  0.44567573], dtype=float32), action=1), EpisodeStep(observation=array([ 0.00549072, -0.19213916, -0.06446619,  0.13079411], dtype=float32), action=0), EpisodeStep(observation=array([ 0.00164794, -0.38628125, -0.06185031,  0.4024624 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.00607769, -0.58047396, -0.05380106,  0.675022  ], dtype=float32), action=0), EpisodeStep(observation=array([-0.01768717, -0.7748086 , -0.04030062,  0.95029235], dtype=float32), action=0), EpisodeStep(observation=array([-0.03318334, -0.96936554, -0.02129477,  1.2300457 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.05257065, -1.1642072 ,  0.00330614,  1.5159817 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.07585479, -0.9691254 ,  0.03362577,  1.2243326 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.0952373 , -1.1646639 ,  0.05811242,  1.5273587 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.11853058, -1.3604369 ,  0.0886596 ,  1.8375978 ], dtype=float32), action=0), EpisodeStep(observation=array([-0.14573932, -1.5564195 ,  0.12541156,  2.156449  ], dtype=float32), action=1), EpisodeStep(observation=array([-0.17686771, -1.3627316 ,  0.16854054,  1.9049733 ], dtype=float32), action=1), EpisodeStep(observation=array([-0.20412233, -1.1697847 ,  0.20664   ,  1.6689703 ], dtype=float32), action=0)]),\n",
       " Episode(reward=12.0, steps=[EpisodeStep(observation=array([-0.01044618, -0.02779534,  0.01137799,  0.04501402], dtype=float32), action=1), EpisodeStep(observation=array([-0.01100209,  0.16716163,  0.01227827, -0.24405743], dtype=float32), action=1), EpisodeStep(observation=array([-0.00765886,  0.36210608,  0.00739712, -0.53284234], dtype=float32), action=0), EpisodeStep(observation=array([-0.00041674,  0.16688088, -0.00325973, -0.23783782], dtype=float32), action=1), EpisodeStep(observation=array([ 0.00292088,  0.36204925, -0.00801648, -0.5315472 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.01016187,  0.55728304, -0.01864743, -0.8267453 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.02130753,  0.75265497, -0.03518233, -1.1252342 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.03636063,  0.9482199 , -0.05768701, -1.4287416 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.05532502,  1.1440048 , -0.08626185, -1.7388811 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.07820512,  1.3399972 , -0.12103947, -2.057106  ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.10500506,  1.1463023 , -0.16218159, -1.8041883 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.12793112,  1.342822  , -0.19826536, -2.142572  ], dtype=float32), action=1)]),\n",
       " Episode(reward=13.0, steps=[EpisodeStep(observation=array([-0.00936326,  0.0390394 ,  0.03533776, -0.03997233], dtype=float32), action=1), EpisodeStep(observation=array([-0.00858247,  0.23363726,  0.03453831, -0.32129976], dtype=float32), action=1), EpisodeStep(observation=array([-0.00390973,  0.4282508 ,  0.02811231, -0.60289365], dtype=float32), action=1), EpisodeStep(observation=array([ 0.00465529,  0.6229685 ,  0.01605444, -0.8865909 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.01711466,  0.42763233, -0.00167738, -0.5889046 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.02566731,  0.6227777 , -0.01345547, -0.8821154 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.03812286,  0.8180798 , -0.03109778, -1.1789979 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.05448446,  1.0135915 , -0.05467774, -1.4812648 ], dtype=float32), action=1), EpisodeStep(observation=array([ 0.07475629,  1.2093363 , -0.08430303, -1.7905104 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.09894301,  1.015255  , -0.12011324, -1.5251781 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.11924811,  0.82177037, -0.1506168 , -1.2722708 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.13568352,  0.62885684, -0.17606221, -1.0302906 ], dtype=float32), action=0), EpisodeStep(observation=array([ 0.14826065,  0.43645853, -0.19666803, -0.79764986], dtype=float32), action=0)])]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# selecciona un ambiente CartPole\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "\n",
    "# extrae tamaños de acciones y observaciones en el ambiente\n",
    "obs_size = env.observation_space.shape[0]\n",
    "n_actions = env.action_space.n\n",
    "# define tamaño capa oculta de la red\n",
    "HIDDEN_SIZE = 128\n",
    "#instancia un objeto Net\n",
    "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
    "\n",
    "# define tamaño de los lotes\n",
    "BATCH_SIZE = 5\n",
    "# instancia un iterador Batch\n",
    "batch = Batch(env, net, BATCH_SIZE)\n",
    "\n",
    "# extrae el primer lote de datos\n",
    "dato = next(batch)\n",
    "dato"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b75d234e-dabd-48d5-b462-9cc04a7c4084",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Clase Agent</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "77c3bc67-4b21-449a-bf40-0a22196e9e3e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from torch.utils.data import Dataset, TensorDataset, DataLoader\n",
    "\n",
    "class Agent:\n",
    "    def __init__(self, batch_iterator, percentile, batch_size=1):\n",
    "        self.batch_iterator =  batch_iterator\n",
    "        self.percentile = percentile\n",
    "        self.batch_size = batch_size # batch size para los dataloaders\n",
    "        \n",
    "    def take_action(self): \n",
    "        # toma un lote de datos del iterador de lotes\n",
    "        batch = next(self.batch_iterator)\n",
    "        # extrae todas las recompensas del batch y hace una lista con ellas\n",
    "        rewards = list(map(lambda s: s.reward, batch))\n",
    "        # calcula la cota inferior para extraer los episodios élite (por defecto percentil 70)\n",
    "        reward_bound = np.percentile(rewards, self.percentile)\n",
    "        # calcula la recompensa promedio del lote de datos\n",
    "        reward_mean = float(np.mean(rewards))\n",
    "        \n",
    "        # extrae las observaciones y las respectivas acciones de los episodios élite\n",
    "        train_obs = []\n",
    "        train_act = []\n",
    "        for example in batch:\n",
    "            if example.reward < reward_bound:\n",
    "                continue\n",
    "            # de cada episodio élite extrae todas las parejas (observación, acción)\n",
    "            # agregando las observaciones en la lista de observaciones\n",
    "            train_obs.extend(map(lambda step: step.observation, example.steps))\n",
    "            # y las acciones en la lista de acciones\n",
    "            train_act.extend(map(lambda step: step.action, example.steps))\n",
    "            \n",
    "        # convierte listas a tensores\n",
    "        #train_obs = np.array(train_obs, dtype=np.float32)\n",
    "        train_obs_v = torch.FloatTensor(train_obs)\n",
    "        #train_act= np.array(train_obs, dtype=np.int64)\n",
    "        train_act_v = torch.LongTensor(train_act)\n",
    "        # crea el dataset\n",
    "        train_dataset = TensorDataset(train_obs_v, train_act_v)\n",
    "        # crea el dataloader                      \n",
    "        train_dataloader = DataLoader(dataset = train_dataset, batch_size = self.batch_size)\n",
    "        # entrega los datos\n",
    "        return train_dataloader, reward_bound, reward_mean\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59800a93-8f57-4be3-b8ef-2bddac33eefb",
   "metadata": {},
   "source": [
    "####  Prueba del agente (CartPole)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e417050d-40fe-4d9f-bdd8-fc61d0615395",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<torch.utils.data.dataloader.DataLoader at 0x7ff6c8a7f160>,\n",
       " 30.299999999999997,\n",
       " 28.95)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# selecciona un ambiente CartPole\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "\n",
    "# extrae tamaños de acciones y observaciones en el ambiente\n",
    "obs_size = env.observation_space.shape[0]\n",
    "n_actions = env.action_space.n\n",
    "# define tamaño capa oculta de la red\n",
    "HIDDEN_SIZE = 128\n",
    "#instancia un objeto Net\n",
    "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
    "\n",
    "# define tamaño de los lotes para el iterador de lotes\n",
    "BATCH_SIZE = 20\n",
    "# instancia un iterador Batch\n",
    "batch = Batch(env, net, BATCH_SIZE)\n",
    "\n",
    "# define el percentil para los episodios élite\n",
    "PERCENTILE = 70\n",
    "\n",
    "# Instancia un agente\n",
    "agent = Agent(batch, PERCENTILE)\n",
    "\n",
    "# entrega un conjunto de datos de los episodios élite de un  lote\n",
    "dato = agent.take_action()\n",
    "dato"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61429604-59e9-4ff6-af6a-2720a30c44dc",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Entrenamiento CartPole</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "247e7077-64b5-4cc5-ab52-a9089a7634b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: pérdida promedio =0.682, recompensa promedio=22.0, cota recompensa=26.0\n",
      "2: pérdida promedio =0.673, recompensa promedio=20.9, cota recompensa=21.3\n",
      "3: pérdida promedio =0.669, recompensa promedio=25.9, cota recompensa=28.9\n",
      "4: pérdida promedio =0.641, recompensa promedio=40.2, cota recompensa=35.0\n",
      "5: pérdida promedio =0.627, recompensa promedio=66.6, cota recompensa=81.4\n",
      "6: pérdida promedio =0.619, recompensa promedio=77.5, cota recompensa=89.9\n",
      "7: pérdida promedio =0.626, recompensa promedio=84.2, cota recompensa=96.0\n",
      "8: pérdida promedio =0.628, recompensa promedio=85.4, cota recompensa=108.7\n",
      "9: pérdida promedio =0.612, recompensa promedio=105.2, cota recompensa=127.6\n",
      "10: pérdida promedio =0.616, recompensa promedio=120.1, cota recompensa=147.4\n",
      "11: pérdida promedio =0.610, recompensa promedio=123.8, cota recompensa=153.6\n",
      "12: pérdida promedio =0.613, recompensa promedio=134.2, cota recompensa=177.5\n",
      "13: pérdida promedio =0.602, recompensa promedio=162.8, cota recompensa=170.2\n",
      "14: pérdida promedio =0.601, recompensa promedio=152.3, cota recompensa=167.0\n",
      "15: pérdida promedio =0.580, recompensa promedio=192.2, cota recompensa=206.2\n",
      "16: pérdida promedio =0.579, recompensa promedio=219.9, cota recompensa=246.7\n",
      "Resuelto!\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.tensorboard import SummaryWriter\n",
    "# selecciona un ambiente CartPole\n",
    "env = gym.make(\"CartPole-v1\")\n",
    "\n",
    "# extrae tamaños de acciones y observaciones en el ambiente\n",
    "obs_size = env.observation_space.shape[0]\n",
    "n_actions = env.action_space.n\n",
    "# define tamaño capa oculta de la red\n",
    "HIDDEN_SIZE = 128\n",
    "#instancia un objeto Net\n",
    "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
    "\n",
    "# define tamaño de los lotes para el iterador de lotes\n",
    "BATCH_SIZE = 20\n",
    "# instancia un iterador Batch\n",
    "batch = Batch(env, net, BATCH_SIZE)\n",
    "\n",
    "# define el percentil para los episodios élite\n",
    "PERCENTILE = 70\n",
    "\n",
    "# Instancia un agente\n",
    "agent = Agent(batch, PERCENTILE)\n",
    "\n",
    "#instancia writer para tensorboard\n",
    "writer = SummaryWriter(comment=\"Entrenamiento de CartPole\")\n",
    "# agrega un grafo del modelo a tensorboard\n",
    "\n",
    "#episode = next(batch)\n",
    "#obs = episode.observation\n",
    "#writer.add_graph(net, obs)\n",
    "\n",
    "trainer = Trainer(model=net, writer=writer)\n",
    "\n",
    "# ciclo de entrenamiento\n",
    "min_reward = 200 # Para CartPole\n",
    "max_iterations = 300\n",
    "done = False\n",
    " \n",
    "iter_no = 0\n",
    "\n",
    "while not done:\n",
    "    iter_no += 1\n",
    "    # pide datos al agente\n",
    "    dataloader, reward_bound, reward_mean = agent.take_action()\n",
    "    # hace un paso de entrenamiento de la red\n",
    "      \n",
    "    trainer.fit(iter_no, dataloader, reward_bound, reward_mean)\n",
    "    #trainer.save_checkpoint()\n",
    "    #validation = trainer.validate(dataloaders=dataloader)\n",
    "    if reward_mean > min_reward:\n",
    "            print(\"Resuelto!\")\n",
    "            done = True\n",
    "    if iter_no == max_iterations:\n",
    "            print(\"Terminado por máximo número de iteraciones. No resuelto\")\n",
    "            done = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4908297b-689f-4671-8e6d-4b57c7b82fd7",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Accediendo a tensorboard</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "80cb67ad-c4ca-4b2b-82cb-5afde1dca6fb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow installation not found - running with reduced feature set.\n",
      "\n",
      "NOTE: Using experimental fast data loading logic. To disable, pass\n",
      "    \"--load_fast=false\" and report issues on GitHub. More details:\n",
      "    https://github.com/tensorflow/tensorboard/issues/4784\n",
      "\n",
      "Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n",
      "TensorBoard 2.8.0 at http://localhost:6006/ (Press CTRL+C to quit)\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "# indica donde se escribirá el log de tensorboard\n",
    "!tensorboard --logdir './runs'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2036ff7-f331-4a94-8251-59c2f37c4f94",
   "metadata": {},
   "source": [
    "### <span style=\"color:#4CC9F0\">Entrenamiento FrozenLake</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "35969654-1a8f-4b93-91fe-f2ba43760c1d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: pérdida promedio =1.389, recompensa promedio=0.0, cota recompensa=0.0\n",
      "2: pérdida promedio =1.386, recompensa promedio=0.0, cota recompensa=0.0\n",
      "3: pérdida promedio =1.382, recompensa promedio=0.0, cota recompensa=0.0\n",
      "4: pérdida promedio =1.382, recompensa promedio=0.0, cota recompensa=0.0\n",
      "5: pérdida promedio =1.382, recompensa promedio=0.0, cota recompensa=0.0\n",
      "6: pérdida promedio =1.379, recompensa promedio=0.0, cota recompensa=0.0\n",
      "7: pérdida promedio =1.364, recompensa promedio=0.0, cota recompensa=0.0\n",
      "8: pérdida promedio =1.356, recompensa promedio=0.0, cota recompensa=0.0\n",
      "9: pérdida promedio =1.367, recompensa promedio=0.0, cota recompensa=0.0\n",
      "10: pérdida promedio =1.368, recompensa promedio=0.0, cota recompensa=0.0\n",
      "11: pérdida promedio =1.352, recompensa promedio=0.0, cota recompensa=0.0\n",
      "12: pérdida promedio =1.347, recompensa promedio=0.0, cota recompensa=0.0\n",
      "13: pérdida promedio =1.351, recompensa promedio=0.0, cota recompensa=0.0\n",
      "14: pérdida promedio =1.349, recompensa promedio=0.0, cota recompensa=0.0\n",
      "15: pérdida promedio =1.348, recompensa promedio=0.0, cota recompensa=0.0\n",
      "16: pérdida promedio =1.366, recompensa promedio=0.0, cota recompensa=0.0\n",
      "17: pérdida promedio =1.336, recompensa promedio=0.0, cota recompensa=0.0\n",
      "18: pérdida promedio =1.354, recompensa promedio=0.0, cota recompensa=0.0\n",
      "19: pérdida promedio =1.359, recompensa promedio=0.0, cota recompensa=0.0\n",
      "20: pérdida promedio =1.357, recompensa promedio=0.0, cota recompensa=0.0\n",
      "21: pérdida promedio =1.346, recompensa promedio=0.0, cota recompensa=0.0\n",
      "22: pérdida promedio =1.353, recompensa promedio=0.0, cota recompensa=0.0\n",
      "23: pérdida promedio =1.341, recompensa promedio=0.0, cota recompensa=0.0\n",
      "24: pérdida promedio =1.342, recompensa promedio=0.0, cota recompensa=0.0\n",
      "25: pérdida promedio =1.335, recompensa promedio=0.0, cota recompensa=0.0\n",
      "26: pérdida promedio =1.319, recompensa promedio=0.0, cota recompensa=0.0\n",
      "27: pérdida promedio =1.314, recompensa promedio=0.0, cota recompensa=0.0\n",
      "28: pérdida promedio =1.339, recompensa promedio=0.0, cota recompensa=0.0\n",
      "29: pérdida promedio =1.338, recompensa promedio=0.0, cota recompensa=0.0\n",
      "30: pérdida promedio =1.338, recompensa promedio=0.0, cota recompensa=0.0\n",
      "31: pérdida promedio =1.332, recompensa promedio=0.0, cota recompensa=0.0\n",
      "32: pérdida promedio =1.340, recompensa promedio=0.0, cota recompensa=0.0\n",
      "33: pérdida promedio =1.329, recompensa promedio=0.0, cota recompensa=0.0\n",
      "34: pérdida promedio =1.340, recompensa promedio=0.0, cota recompensa=0.0\n",
      "35: pérdida promedio =1.333, recompensa promedio=0.0, cota recompensa=0.0\n",
      "36: pérdida promedio =1.320, recompensa promedio=0.0, cota recompensa=0.0\n",
      "37: pérdida promedio =1.313, recompensa promedio=0.0, cota recompensa=0.0\n",
      "38: pérdida promedio =1.322, recompensa promedio=0.0, cota recompensa=0.0\n",
      "39: pérdida promedio =1.317, recompensa promedio=0.0, cota recompensa=0.0\n",
      "40: pérdida promedio =1.283, recompensa promedio=0.0, cota recompensa=0.0\n",
      "41: pérdida promedio =1.282, recompensa promedio=0.0, cota recompensa=0.0\n",
      "42: pérdida promedio =1.285, recompensa promedio=0.0, cota recompensa=0.0\n",
      "43: pérdida promedio =1.294, recompensa promedio=0.0, cota recompensa=0.0\n",
      "44: pérdida promedio =1.291, recompensa promedio=0.0, cota recompensa=0.0\n",
      "45: pérdida promedio =1.289, recompensa promedio=0.0, cota recompensa=0.0\n",
      "46: pérdida promedio =1.281, recompensa promedio=0.0, cota recompensa=0.0\n",
      "47: pérdida promedio =1.291, recompensa promedio=0.0, cota recompensa=0.0\n",
      "48: pérdida promedio =1.292, recompensa promedio=0.0, cota recompensa=0.0\n",
      "49: pérdida promedio =1.315, recompensa promedio=0.0, cota recompensa=0.0\n",
      "50: pérdida promedio =1.334, recompensa promedio=0.0, cota recompensa=0.0\n",
      "51: pérdida promedio =1.334, recompensa promedio=0.0, cota recompensa=0.0\n",
      "52: pérdida promedio =1.325, recompensa promedio=0.0, cota recompensa=0.0\n",
      "53: pérdida promedio =1.324, recompensa promedio=0.0, cota recompensa=0.0\n",
      "54: pérdida promedio =1.311, recompensa promedio=0.0, cota recompensa=0.0\n",
      "55: pérdida promedio =1.313, recompensa promedio=0.0, cota recompensa=0.0\n",
      "56: pérdida promedio =1.322, recompensa promedio=0.0, cota recompensa=0.0\n",
      "57: pérdida promedio =1.317, recompensa promedio=0.0, cota recompensa=0.0\n",
      "58: pérdida promedio =1.296, recompensa promedio=0.0, cota recompensa=0.0\n",
      "59: pérdida promedio =1.322, recompensa promedio=0.0, cota recompensa=0.0\n",
      "60: pérdida promedio =1.316, recompensa promedio=0.0, cota recompensa=0.0\n",
      "61: pérdida promedio =1.303, recompensa promedio=0.0, cota recompensa=0.0\n",
      "62: pérdida promedio =1.295, recompensa promedio=0.0, cota recompensa=0.0\n",
      "63: pérdida promedio =1.308, recompensa promedio=0.0, cota recompensa=0.0\n",
      "64: pérdida promedio =1.337, recompensa promedio=0.0, cota recompensa=0.0\n",
      "65: pérdida promedio =1.328, recompensa promedio=0.0, cota recompensa=0.0\n",
      "66: pérdida promedio =1.313, recompensa promedio=0.0, cota recompensa=0.0\n",
      "67: pérdida promedio =1.296, recompensa promedio=0.0, cota recompensa=0.0\n",
      "68: pérdida promedio =1.272, recompensa promedio=0.0, cota recompensa=0.0\n",
      "69: pérdida promedio =1.291, recompensa promedio=0.0, cota recompensa=0.0\n",
      "70: pérdida promedio =1.276, recompensa promedio=0.0, cota recompensa=0.0\n",
      "71: pérdida promedio =1.287, recompensa promedio=0.0, cota recompensa=0.0\n",
      "72: pérdida promedio =1.291, recompensa promedio=0.0, cota recompensa=0.0\n",
      "73: pérdida promedio =1.316, recompensa promedio=0.0, cota recompensa=0.0\n",
      "74: pérdida promedio =1.318, recompensa promedio=0.0, cota recompensa=0.0\n",
      "75: pérdida promedio =1.322, recompensa promedio=0.0, cota recompensa=0.0\n",
      "76: pérdida promedio =1.297, recompensa promedio=0.0, cota recompensa=0.0\n",
      "77: pérdida promedio =1.260, recompensa promedio=0.0, cota recompensa=0.0\n",
      "78: pérdida promedio =1.251, recompensa promedio=0.0, cota recompensa=0.0\n",
      "79: pérdida promedio =1.258, recompensa promedio=0.0, cota recompensa=0.0\n",
      "80: pérdida promedio =1.224, recompensa promedio=0.0, cota recompensa=0.0\n",
      "81: pérdida promedio =1.238, recompensa promedio=0.0, cota recompensa=0.0\n",
      "82: pérdida promedio =1.211, recompensa promedio=0.0, cota recompensa=0.0\n",
      "83: pérdida promedio =1.181, recompensa promedio=0.0, cota recompensa=0.0\n",
      "84: pérdida promedio =1.190, recompensa promedio=0.0, cota recompensa=0.0\n",
      "85: pérdida promedio =1.184, recompensa promedio=0.0, cota recompensa=0.0\n",
      "86: pérdida promedio =1.135, recompensa promedio=0.0, cota recompensa=0.0\n",
      "87: pérdida promedio =1.159, recompensa promedio=0.0, cota recompensa=0.0\n",
      "88: pérdida promedio =1.087, recompensa promedio=0.0, cota recompensa=0.0\n",
      "89: pérdida promedio =1.145, recompensa promedio=0.0, cota recompensa=0.0\n",
      "90: pérdida promedio =1.115, recompensa promedio=0.0, cota recompensa=0.0\n",
      "91: pérdida promedio =1.161, recompensa promedio=0.0, cota recompensa=0.0\n",
      "92: pérdida promedio =1.129, recompensa promedio=0.0, cota recompensa=0.0\n",
      "93: pérdida promedio =1.087, recompensa promedio=0.0, cota recompensa=0.0\n",
      "94: pérdida promedio =1.122, recompensa promedio=0.0, cota recompensa=0.0\n",
      "95: pérdida promedio =1.131, recompensa promedio=0.0, cota recompensa=0.0\n",
      "96: pérdida promedio =1.112, recompensa promedio=0.0, cota recompensa=0.0\n",
      "97: pérdida promedio =1.067, recompensa promedio=0.0, cota recompensa=0.0\n",
      "98: pérdida promedio =1.116, recompensa promedio=0.0, cota recompensa=0.0\n",
      "99: pérdida promedio =1.069, recompensa promedio=0.0, cota recompensa=0.0\n",
      "100: pérdida promedio =1.048, recompensa promedio=0.0, cota recompensa=0.0\n",
      "Terminado por máximo número de iteraciones. No resuelto\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# selecciona un ambiente CartPole\n",
    "env = DiscreteOneHotWrapper(gym.make(\"FrozenLake-v1\"))\n",
    "\n",
    "# extrae tamaños de acciones y observaciones en el ambiente\n",
    "obs_size = env.observation_space.shape[0]\n",
    "n_actions = env.action_space.n\n",
    "# define tamaño capa oculta de la red\n",
    "HIDDEN_SIZE = 128\n",
    "#instancia un objeto Net\n",
    "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
    "\n",
    "# define tamaño de los lotes para el iterador de lotes\n",
    "BATCH_SIZE = 100\n",
    "# instancia un iterador Batch\n",
    "batch = Batch(env, net, BATCH_SIZE)\n",
    "\n",
    "# define el percentil para los episodios élite\n",
    "PERCENTILE = 70\n",
    "\n",
    "# Instancia un agente\n",
    "agent = Agent(batch, PERCENTILE)\n",
    "\n",
    "trainer = Trainer(net)\n",
    "\n",
    "# ciclo de entrenamiento\n",
    "min_reward = 0.8 # Para FrozenLake\n",
    "max_iterations = 100\n",
    "done = False\n",
    " \n",
    "iter_no = 0\n",
    "\n",
    "while not done:\n",
    "    iter_no += 1\n",
    "    # pide datos al agente\n",
    "    dataloader, reward_bound, reward_mean = agent.take_action()\n",
    "    # hace un paso de entrenamiento de la red\n",
    "      \n",
    "    trainer.fit(iter_no, dataloader, reward_bound, reward_mean)\n",
    "    #trainer.save_checkpoint()\n",
    "    #validation = trainer.validate(dataloaders=dataloader)\n",
    "    if reward_mean > min_reward:\n",
    "            print(\"Resuelto!\")\n",
    "            done = True\n",
    "    if iter_no == max_iterations:\n",
    "            print(\"Terminado por máximo número de iteraciones. No resuelto\")\n",
    "            done = True\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "881a1092-abc0-45d4-abdc-dfcab95cccca",
   "metadata": {
    "tags": []
   },
   "source": [
    "## <span style=\"color:blue\">Modificación para el  ambiente FrozenLake</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36701e57-aa10-44bb-ba00-a87c49dd159b",
   "metadata": {},
   "source": [
    "Más adelante en este curso volveremos a este ambiente para resolver las limitaciones del MEC con otros métodos de aprendizaje reforzado.\n",
    "\n",
    "De momento haremos unas mejoras que ayuden al MEC a estimar la distribución de las acciones en el espacio de las observaciones. Haremos los siguiente:\n",
    "\n",
    "* Lotes de episodios más largos. Pasaremos a 100 episodios por lote.\n",
    "* Aplicaremos  el factor de descuento $\\gamma$ a la recompensa. Así episodios más largos tendrán una menor recompensa y viceversa. Esto incrementa la variabilidad de la distribución de la recompensa.\n",
    "* Mantendremos episodios élite por más largo tiempo.\n",
    "* Decreceremos la rata de aprendizaje. Esto implica que la red neuronal tendrá más tiempo para ver en promedio mas muestras de entrenamiento.\n",
    "* Mucho mayor tiempo de entrenamiento. \n",
    "\n",
    "Por favor revise, corra y modifique si lo considera necesario, el siguiente código.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92e4d07b-a8cc-47da-af23-35dcd4ea7cd0",
   "metadata": {},
   "source": [
    "## <span style=\"color:blue\">Clase Agent2</span>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d78f6ea6-d903-4439-8d5e-adac83769468",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from torch.utils.data import Dataset, TensorDataset, DataLoader\n",
    "\n",
    "class Agent2(Agent):\n",
    "    def __init__(self, batch_iterator, percentile, batch_size=1, gamma=0.9):\n",
    "        super(Agent2, self).__init__(batch_iterator, percentile, batch_size)\n",
    "        \n",
    "        self.gamma = gamma\n",
    "        self.full_batch = []\n",
    "        \n",
    "    def take_action(self): \n",
    "        # toma un lote de datos del iterador de lotes\n",
    "        batch = next(self.batch_iterator)\n",
    "        # agrega los datos que tenga preservados del episodio anterior\n",
    "        batch = batch + self.full_batch \n",
    "        # filtro para modificar la recompensa. Episodios mas largos tiene mayor descuento \n",
    "        filter_fun = lambda s: s.reward * (self.gamma** len(s.steps))\n",
    "        # extrae todas las recompensas del batch y hace una lista con ellas\n",
    "        disc_rewards = list(map(filter_fun, batch))\n",
    "        # calcula la cota inferior para extraer los episodios élite (por defecto percentil 70)\n",
    "        reward_bound = np.percentile(disc_rewards, self.percentile)\n",
    "        # calcula la recompensa promedio del lote de datos\n",
    "        reward_mean = float(np.mean(disc_rewards))\n",
    "        \n",
    "        # extrae las observaciones y las respectivas acciones de los episodios élite\n",
    "        train_obs = []\n",
    "        train_act = []\n",
    "        elite_batch = []\n",
    "        \n",
    "        for example, discounted_reward in zip(batch, disc_rewards):\n",
    "            if discounted_reward > reward_bound:\n",
    "                train_obs.extend(map(lambda step: step.observation,\n",
    "                                     example.steps))\n",
    "                train_act.extend(map(lambda step: step.action,\n",
    "                                     example.steps))\n",
    "                elite_batch.append(example)\n",
    "        # guarda este batch élite para el siguiente episodio\n",
    "        self.full_batch = elite_batch[-500:] # conserva los últimos 500 datos\n",
    "            \n",
    "        # convierte listas a tensores\n",
    "        train_obs_v = torch.FloatTensor(train_obs)\n",
    "        train_act_v = torch.LongTensor(train_act)\n",
    "        # crea el dataset\n",
    "        train_dataset = TensorDataset(train_obs_v, train_act_v)\n",
    "        # crea el dataloader                      \n",
    "        train_dataloader = DataLoader(dataset = train_dataset, batch_size = self.batch_size)\n",
    "        # entrega los datos\n",
    "        return train_dataloader, reward_bound, reward_mean\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f06be220-d9a6-4ceb-9fdf-523110f7ab4c",
   "metadata": {},
   "source": [
    "### Re-entrenamiento FrozenLake"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "db086dc3-1e0e-4cf0-9f53-04df20139910",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1: pérdida promedio =1.360, recompensa promedio=0.0, cota recompensa=0.0\n",
      "2: pérdida promedio =1.340, recompensa promedio=0.0, cota recompensa=0.0\n",
      "3: pérdida promedio =1.369, recompensa promedio=0.0, cota recompensa=0.0\n",
      "4: pérdida promedio =1.344, recompensa promedio=0.0, cota recompensa=0.0\n",
      "5: pérdida promedio =1.319, recompensa promedio=0.0, cota recompensa=0.0\n",
      "6: pérdida promedio =1.310, recompensa promedio=0.0, cota recompensa=0.0\n",
      "7: pérdida promedio =1.288, recompensa promedio=0.0, cota recompensa=0.0\n",
      "8: pérdida promedio =1.277, recompensa promedio=0.0, cota recompensa=0.0\n",
      "9: pérdida promedio =1.270, recompensa promedio=0.1, cota recompensa=0.0\n",
      "10: pérdida promedio =1.264, recompensa promedio=0.1, cota recompensa=0.0\n",
      "11: pérdida promedio =1.263, recompensa promedio=0.1, cota recompensa=0.0\n",
      "12: pérdida promedio =1.258, recompensa promedio=0.1, cota recompensa=0.0\n",
      "13: pérdida promedio =1.245, recompensa promedio=0.1, cota recompensa=0.0\n",
      "14: pérdida promedio =1.234, recompensa promedio=0.1, cota recompensa=0.0\n",
      "15: pérdida promedio =1.222, recompensa promedio=0.1, cota recompensa=0.0\n",
      "16: pérdida promedio =1.229, recompensa promedio=0.1, cota recompensa=0.0\n",
      "17: pérdida promedio =1.217, recompensa promedio=0.1, cota recompensa=0.0\n",
      "18: pérdida promedio =1.211, recompensa promedio=0.1, cota recompensa=0.0\n",
      "19: pérdida promedio =1.205, recompensa promedio=0.1, cota recompensa=0.0\n",
      "20: pérdida promedio =1.199, recompensa promedio=0.1, cota recompensa=0.0\n",
      "21: pérdida promedio =1.195, recompensa promedio=0.1, cota recompensa=0.0\n",
      "22: pérdida promedio =1.192, recompensa promedio=0.1, cota recompensa=0.0\n",
      "23: pérdida promedio =1.190, recompensa promedio=0.1, cota recompensa=0.0\n",
      "24: pérdida promedio =1.176, recompensa promedio=0.1, cota recompensa=0.1\n",
      "25: pérdida promedio =1.167, recompensa promedio=0.1, cota recompensa=0.0\n",
      "26: pérdida promedio =1.168, recompensa promedio=0.1, cota recompensa=0.1\n",
      "27: pérdida promedio =1.166, recompensa promedio=0.1, cota recompensa=0.1\n",
      "28: pérdida promedio =1.163, recompensa promedio=0.2, cota recompensa=0.1\n",
      "29: pérdida promedio =1.160, recompensa promedio=0.2, cota recompensa=0.1\n",
      "30: pérdida promedio =1.151, recompensa promedio=0.2, cota recompensa=0.2\n",
      "31: pérdida promedio =1.139, recompensa promedio=0.2, cota recompensa=0.2\n",
      "32: pérdida promedio =1.133, recompensa promedio=0.2, cota recompensa=0.0\n",
      "33: pérdida promedio =1.129, recompensa promedio=0.2, cota recompensa=0.0\n",
      "34: pérdida promedio =1.132, recompensa promedio=0.2, cota recompensa=0.2\n",
      "35: pérdida promedio =1.131, recompensa promedio=0.2, cota recompensa=0.1\n",
      "36: pérdida promedio =1.127, recompensa promedio=0.2, cota recompensa=0.2\n",
      "37: pérdida promedio =1.118, recompensa promedio=0.2, cota recompensa=0.2\n",
      "38: pérdida promedio =1.119, recompensa promedio=0.2, cota recompensa=0.2\n",
      "39: pérdida promedio =1.114, recompensa promedio=0.2, cota recompensa=0.0\n",
      "40: pérdida promedio =1.107, recompensa promedio=0.2, cota recompensa=0.1\n",
      "41: pérdida promedio =1.101, recompensa promedio=0.2, cota recompensa=0.1\n",
      "42: pérdida promedio =1.105, recompensa promedio=0.2, cota recompensa=0.2\n",
      "43: pérdida promedio =1.103, recompensa promedio=0.2, cota recompensa=0.3\n",
      "44: pérdida promedio =1.111, recompensa promedio=0.2, cota recompensa=0.3\n",
      "45: pérdida promedio =1.109, recompensa promedio=0.2, cota recompensa=0.0\n",
      "46: pérdida promedio =1.109, recompensa promedio=0.2, cota recompensa=0.0\n",
      "47: pérdida promedio =1.111, recompensa promedio=0.2, cota recompensa=0.1\n",
      "48: pérdida promedio =1.105, recompensa promedio=0.2, cota recompensa=0.1\n",
      "49: pérdida promedio =1.109, recompensa promedio=0.2, cota recompensa=0.2\n",
      "50: pérdida promedio =1.102, recompensa promedio=0.2, cota recompensa=0.2\n",
      "51: pérdida promedio =1.110, recompensa promedio=0.2, cota recompensa=0.2\n",
      "52: pérdida promedio =1.109, recompensa promedio=0.2, cota recompensa=0.3\n",
      "53: pérdida promedio =1.103, recompensa promedio=0.2, cota recompensa=0.0\n",
      "54: pérdida promedio =1.100, recompensa promedio=0.2, cota recompensa=0.2\n",
      "55: pérdida promedio =1.097, recompensa promedio=0.2, cota recompensa=0.2\n",
      "56: pérdida promedio =1.098, recompensa promedio=0.2, cota recompensa=0.2\n",
      "57: pérdida promedio =1.106, recompensa promedio=0.2, cota recompensa=0.3\n",
      "58: pérdida promedio =1.107, recompensa promedio=0.2, cota recompensa=0.3\n",
      "59: pérdida promedio =1.107, recompensa promedio=0.2, cota recompensa=0.3\n",
      "60: pérdida promedio =1.084, recompensa promedio=0.2, cota recompensa=0.3\n",
      "61: pérdida promedio =1.083, recompensa promedio=0.2, cota recompensa=0.0\n",
      "62: pérdida promedio =1.064, recompensa promedio=0.2, cota recompensa=0.3\n",
      "63: pérdida promedio =1.060, recompensa promedio=0.2, cota recompensa=0.2\n",
      "64: pérdida promedio =1.058, recompensa promedio=0.2, cota recompensa=0.3\n",
      "65: pérdida promedio =1.066, recompensa promedio=0.2, cota recompensa=0.3\n",
      "66: pérdida promedio =1.071, recompensa promedio=0.2, cota recompensa=0.3\n",
      "67: pérdida promedio =1.074, recompensa promedio=0.2, cota recompensa=0.4\n",
      "68: pérdida promedio =1.059, recompensa promedio=0.2, cota recompensa=0.0\n",
      "69: pérdida promedio =1.054, recompensa promedio=0.2, cota recompensa=0.0\n",
      "70: pérdida promedio =1.035, recompensa promedio=0.2, cota recompensa=0.0\n",
      "71: pérdida promedio =1.040, recompensa promedio=0.2, cota recompensa=0.0\n",
      "72: pérdida promedio =1.049, recompensa promedio=0.2, cota recompensa=0.2\n",
      "73: pérdida promedio =1.047, recompensa promedio=0.2, cota recompensa=0.2\n",
      "74: pérdida promedio =1.045, recompensa promedio=0.2, cota recompensa=0.3\n",
      "75: pérdida promedio =1.038, recompensa promedio=0.2, cota recompensa=0.3\n",
      "76: pérdida promedio =1.048, recompensa promedio=0.2, cota recompensa=0.3\n",
      "77: pérdida promedio =1.046, recompensa promedio=0.2, cota recompensa=0.2\n",
      "78: pérdida promedio =1.045, recompensa promedio=0.2, cota recompensa=0.3\n",
      "79: pérdida promedio =1.048, recompensa promedio=0.2, cota recompensa=0.3\n",
      "80: pérdida promedio =1.053, recompensa promedio=0.2, cota recompensa=0.4\n",
      "81: pérdida promedio =1.047, recompensa promedio=0.2, cota recompensa=0.1\n",
      "82: pérdida promedio =1.052, recompensa promedio=0.2, cota recompensa=0.2\n",
      "83: pérdida promedio =1.050, recompensa promedio=0.2, cota recompensa=0.3\n",
      "84: pérdida promedio =1.050, recompensa promedio=0.2, cota recompensa=0.3\n",
      "85: pérdida promedio =1.065, recompensa promedio=0.2, cota recompensa=0.4\n",
      "86: pérdida promedio =1.031, recompensa promedio=0.2, cota recompensa=0.0\n",
      "87: pérdida promedio =1.014, recompensa promedio=0.2, cota recompensa=0.0\n",
      "88: pérdida promedio =1.008, recompensa promedio=0.2, cota recompensa=0.0\n",
      "89: pérdida promedio =1.017, recompensa promedio=0.2, cota recompensa=0.0\n",
      "90: pérdida promedio =1.005, recompensa promedio=0.2, cota recompensa=0.0\n",
      "91: pérdida promedio =1.011, recompensa promedio=0.2, cota recompensa=0.0\n",
      "92: pérdida promedio =1.006, recompensa promedio=0.2, cota recompensa=0.0\n",
      "93: pérdida promedio =1.005, recompensa promedio=0.2, cota recompensa=0.0\n",
      "94: pérdida promedio =1.022, recompensa promedio=0.2, cota recompensa=0.1\n",
      "95: pérdida promedio =1.033, recompensa promedio=0.2, cota recompensa=0.2\n",
      "96: pérdida promedio =1.030, recompensa promedio=0.2, cota recompensa=0.2\n",
      "97: pérdida promedio =1.017, recompensa promedio=0.2, cota recompensa=0.2\n",
      "98: pérdida promedio =1.007, recompensa promedio=0.2, cota recompensa=0.3\n",
      "99: pérdida promedio =1.006, recompensa promedio=0.2, cota recompensa=0.0\n",
      "100: pérdida promedio =1.010, recompensa promedio=0.2, cota recompensa=0.2\n",
      "101: pérdida promedio =0.997, recompensa promedio=0.2, cota recompensa=0.3\n",
      "102: pérdida promedio =0.993, recompensa promedio=0.2, cota recompensa=0.2\n",
      "103: pérdida promedio =0.999, recompensa promedio=0.2, cota recompensa=0.3\n",
      "104: pérdida promedio =1.017, recompensa promedio=0.2, cota recompensa=0.3\n",
      "105: pérdida promedio =1.021, recompensa promedio=0.2, cota recompensa=0.0\n",
      "106: pérdida promedio =0.994, recompensa promedio=0.2, cota recompensa=0.1\n",
      "107: pérdida promedio =1.011, recompensa promedio=0.2, cota recompensa=0.2\n",
      "108: pérdida promedio =1.008, recompensa promedio=0.2, cota recompensa=0.2\n",
      "109: pérdida promedio =1.009, recompensa promedio=0.2, cota recompensa=0.3\n",
      "110: pérdida promedio =1.017, recompensa promedio=0.2, cota recompensa=0.3\n",
      "111: pérdida promedio =1.021, recompensa promedio=0.2, cota recompensa=0.3\n",
      "112: pérdida promedio =1.018, recompensa promedio=0.2, cota recompensa=0.3\n",
      "113: pérdida promedio =1.017, recompensa promedio=0.2, cota recompensa=0.3\n",
      "114: pérdida promedio =1.016, recompensa promedio=0.2, cota recompensa=0.3\n",
      "115: pérdida promedio =1.019, recompensa promedio=0.2, cota recompensa=0.4\n",
      "116: pérdida promedio =1.019, recompensa promedio=0.2, cota recompensa=0.1\n",
      "117: pérdida promedio =1.014, recompensa promedio=0.2, cota recompensa=0.3\n",
      "118: pérdida promedio =1.016, recompensa promedio=0.2, cota recompensa=0.2\n",
      "119: pérdida promedio =1.009, recompensa promedio=0.2, cota recompensa=0.3\n",
      "120: pérdida promedio =1.010, recompensa promedio=0.2, cota recompensa=0.3\n",
      "121: pérdida promedio =1.008, recompensa promedio=0.3, cota recompensa=0.4\n",
      "122: pérdida promedio =1.009, recompensa promedio=0.3, cota recompensa=0.4\n",
      "123: pérdida promedio =1.012, recompensa promedio=0.2, cota recompensa=0.4\n",
      "124: pérdida promedio =0.997, recompensa promedio=0.2, cota recompensa=0.0\n",
      "125: pérdida promedio =0.981, recompensa promedio=0.2, cota recompensa=0.0\n",
      "126: pérdida promedio =0.977, recompensa promedio=0.2, cota recompensa=0.0\n",
      "127: pérdida promedio =0.996, recompensa promedio=0.2, cota recompensa=0.2\n",
      "128: pérdida promedio =0.998, recompensa promedio=0.2, cota recompensa=0.2\n",
      "129: pérdida promedio =0.997, recompensa promedio=0.2, cota recompensa=0.2\n",
      "130: pérdida promedio =1.001, recompensa promedio=0.2, cota recompensa=0.2\n",
      "131: pérdida promedio =1.007, recompensa promedio=0.2, cota recompensa=0.3\n",
      "132: pérdida promedio =0.998, recompensa promedio=0.2, cota recompensa=0.3\n",
      "133: pérdida promedio =1.005, recompensa promedio=0.3, cota recompensa=0.3\n",
      "134: pérdida promedio =1.016, recompensa promedio=0.3, cota recompensa=0.3\n",
      "135: pérdida promedio =1.023, recompensa promedio=0.3, cota recompensa=0.4\n",
      "136: pérdida promedio =1.026, recompensa promedio=0.2, cota recompensa=0.3\n",
      "137: pérdida promedio =1.018, recompensa promedio=0.2, cota recompensa=0.3\n",
      "138: pérdida promedio =1.022, recompensa promedio=0.2, cota recompensa=0.3\n",
      "139: pérdida promedio =1.018, recompensa promedio=0.3, cota recompensa=0.4\n",
      "140: pérdida promedio =1.004, recompensa promedio=0.3, cota recompensa=0.4\n",
      "141: pérdida promedio =0.999, recompensa promedio=0.3, cota recompensa=0.4\n",
      "142: pérdida promedio =1.002, recompensa promedio=0.3, cota recompensa=0.4\n",
      "143: pérdida promedio =1.004, recompensa promedio=0.3, cota recompensa=0.2\n",
      "144: pérdida promedio =1.005, recompensa promedio=0.3, cota recompensa=0.4\n",
      "145: pérdida promedio =1.005, recompensa promedio=0.3, cota recompensa=0.4\n",
      "146: pérdida promedio =1.008, recompensa promedio=0.3, cota recompensa=0.5\n",
      "147: pérdida promedio =1.006, recompensa promedio=0.2, cota recompensa=0.0\n",
      "148: pérdida promedio =1.001, recompensa promedio=0.2, cota recompensa=0.0\n",
      "149: pérdida promedio =0.978, recompensa promedio=0.2, cota recompensa=0.0\n",
      "150: pérdida promedio =0.968, recompensa promedio=0.2, cota recompensa=0.0\n",
      "151: pérdida promedio =0.959, recompensa promedio=0.2, cota recompensa=0.0\n",
      "152: pérdida promedio =0.935, recompensa promedio=0.2, cota recompensa=0.0\n",
      "153: pérdida promedio =0.940, recompensa promedio=0.2, cota recompensa=0.0\n",
      "154: pérdida promedio =0.953, recompensa promedio=0.2, cota recompensa=0.1\n",
      "155: pérdida promedio =0.950, recompensa promedio=0.2, cota recompensa=0.1\n",
      "156: pérdida promedio =0.944, recompensa promedio=0.2, cota recompensa=0.1\n",
      "157: pérdida promedio =0.950, recompensa promedio=0.2, cota recompensa=0.2\n",
      "158: pérdida promedio =0.948, recompensa promedio=0.2, cota recompensa=0.0\n",
      "159: pérdida promedio =0.966, recompensa promedio=0.2, cota recompensa=0.2\n",
      "160: pérdida promedio =0.966, recompensa promedio=0.2, cota recompensa=0.2\n",
      "161: pérdida promedio =0.953, recompensa promedio=0.2, cota recompensa=0.3\n",
      "162: pérdida promedio =0.947, recompensa promedio=0.2, cota recompensa=0.1\n",
      "163: pérdida promedio =0.951, recompensa promedio=0.2, cota recompensa=0.3\n",
      "164: pérdida promedio =0.953, recompensa promedio=0.2, cota recompensa=0.1\n",
      "165: pérdida promedio =0.955, recompensa promedio=0.2, cota recompensa=0.2\n",
      "166: pérdida promedio =0.958, recompensa promedio=0.2, cota recompensa=0.3\n",
      "167: pérdida promedio =0.960, recompensa promedio=0.2, cota recompensa=0.2\n",
      "168: pérdida promedio =0.956, recompensa promedio=0.2, cota recompensa=0.3\n",
      "169: pérdida promedio =0.960, recompensa promedio=0.2, cota recompensa=0.3\n",
      "170: pérdida promedio =0.964, recompensa promedio=0.2, cota recompensa=0.3\n",
      "171: pérdida promedio =0.961, recompensa promedio=0.2, cota recompensa=0.3\n",
      "172: pérdida promedio =0.967, recompensa promedio=0.2, cota recompensa=0.3\n",
      "173: pérdida promedio =0.965, recompensa promedio=0.2, cota recompensa=0.3\n",
      "174: pérdida promedio =0.964, recompensa promedio=0.2, cota recompensa=0.2\n",
      "175: pérdida promedio =0.969, recompensa promedio=0.2, cota recompensa=0.2\n",
      "176: pérdida promedio =0.953, recompensa promedio=0.3, cota recompensa=0.4\n",
      "177: pérdida promedio =0.948, recompensa promedio=0.2, cota recompensa=0.0\n",
      "178: pérdida promedio =0.941, recompensa promedio=0.2, cota recompensa=0.0\n",
      "179: pérdida promedio =0.946, recompensa promedio=0.2, cota recompensa=0.3\n",
      "180: pérdida promedio =0.944, recompensa promedio=0.2, cota recompensa=0.3\n",
      "181: pérdida promedio =0.946, recompensa promedio=0.2, cota recompensa=0.3\n",
      "182: pérdida promedio =0.945, recompensa promedio=0.2, cota recompensa=0.2\n",
      "183: pérdida promedio =0.945, recompensa promedio=0.2, cota recompensa=0.0\n",
      "184: pérdida promedio =0.943, recompensa promedio=0.2, cota recompensa=0.3\n",
      "185: pérdida promedio =0.958, recompensa promedio=0.3, cota recompensa=0.4\n",
      "186: pérdida promedio =0.975, recompensa promedio=0.2, cota recompensa=0.4\n",
      "187: pérdida promedio =0.957, recompensa promedio=0.2, cota recompensa=0.0\n",
      "188: pérdida promedio =0.946, recompensa promedio=0.2, cota recompensa=0.0\n",
      "189: pérdida promedio =0.943, recompensa promedio=0.2, cota recompensa=0.0\n",
      "190: pérdida promedio =0.928, recompensa promedio=0.2, cota recompensa=0.1\n",
      "191: pérdida promedio =0.925, recompensa promedio=0.2, cota recompensa=0.1\n",
      "192: pérdida promedio =0.928, recompensa promedio=0.2, cota recompensa=0.2\n",
      "193: pérdida promedio =0.927, recompensa promedio=0.3, cota recompensa=0.3\n",
      "194: pérdida promedio =0.931, recompensa promedio=0.2, cota recompensa=0.3\n",
      "195: pérdida promedio =0.928, recompensa promedio=0.2, cota recompensa=0.3\n",
      "196: pérdida promedio =0.948, recompensa promedio=0.3, cota recompensa=0.3\n",
      "197: pérdida promedio =0.940, recompensa promedio=0.3, cota recompensa=0.4\n",
      "198: pérdida promedio =0.943, recompensa promedio=0.3, cota recompensa=0.4\n",
      "199: pérdida promedio =0.941, recompensa promedio=0.3, cota recompensa=0.4\n",
      "200: pérdida promedio =0.932, recompensa promedio=0.2, cota recompensa=0.2\n",
      "201: pérdida promedio =0.943, recompensa promedio=0.3, cota recompensa=0.4\n",
      "202: pérdida promedio =0.959, recompensa promedio=0.3, cota recompensa=0.4\n",
      "203: pérdida promedio =0.948, recompensa promedio=0.2, cota recompensa=0.0\n",
      "204: pérdida promedio =0.961, recompensa promedio=0.3, cota recompensa=0.3\n",
      "205: pérdida promedio =0.968, recompensa promedio=0.3, cota recompensa=0.3\n",
      "206: pérdida promedio =0.963, recompensa promedio=0.3, cota recompensa=0.3\n",
      "207: pérdida promedio =0.957, recompensa promedio=0.3, cota recompensa=0.4\n",
      "208: pérdida promedio =0.954, recompensa promedio=0.3, cota recompensa=0.3\n",
      "209: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.4\n",
      "210: pérdida promedio =0.958, recompensa promedio=0.3, cota recompensa=0.4\n",
      "211: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.3\n",
      "212: pérdida promedio =0.962, recompensa promedio=0.3, cota recompensa=0.2\n",
      "213: pérdida promedio =0.948, recompensa promedio=0.3, cota recompensa=0.3\n",
      "214: pérdida promedio =0.963, recompensa promedio=0.3, cota recompensa=0.3\n",
      "215: pérdida promedio =0.966, recompensa promedio=0.3, cota recompensa=0.4\n",
      "216: pérdida promedio =0.967, recompensa promedio=0.3, cota recompensa=0.0\n",
      "217: pérdida promedio =0.967, recompensa promedio=0.3, cota recompensa=0.4\n",
      "218: pérdida promedio =0.967, recompensa promedio=0.3, cota recompensa=0.4\n",
      "219: pérdida promedio =0.964, recompensa promedio=0.3, cota recompensa=0.3\n",
      "220: pérdida promedio =0.967, recompensa promedio=0.3, cota recompensa=0.4\n",
      "221: pérdida promedio =0.967, recompensa promedio=0.3, cota recompensa=0.3\n",
      "222: pérdida promedio =0.970, recompensa promedio=0.3, cota recompensa=0.4\n",
      "223: pérdida promedio =0.970, recompensa promedio=0.3, cota recompensa=0.5\n",
      "224: pérdida promedio =0.929, recompensa promedio=0.2, cota recompensa=0.0\n",
      "225: pérdida promedio =0.919, recompensa promedio=0.2, cota recompensa=0.0\n",
      "226: pérdida promedio =0.906, recompensa promedio=0.2, cota recompensa=0.0\n",
      "227: pérdida promedio =0.898, recompensa promedio=0.2, cota recompensa=0.0\n",
      "228: pérdida promedio =0.901, recompensa promedio=0.2, cota recompensa=0.1\n",
      "229: pérdida promedio =0.909, recompensa promedio=0.2, cota recompensa=0.1\n",
      "230: pérdida promedio =0.925, recompensa promedio=0.2, cota recompensa=0.2\n",
      "231: pérdida promedio =0.937, recompensa promedio=0.2, cota recompensa=0.2\n",
      "232: pérdida promedio =0.933, recompensa promedio=0.2, cota recompensa=0.1\n",
      "233: pérdida promedio =0.932, recompensa promedio=0.2, cota recompensa=0.2\n",
      "234: pérdida promedio =0.940, recompensa promedio=0.2, cota recompensa=0.3\n",
      "235: pérdida promedio =0.942, recompensa promedio=0.2, cota recompensa=0.3\n",
      "236: pérdida promedio =0.957, recompensa promedio=0.2, cota recompensa=0.3\n",
      "237: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.3\n",
      "238: pérdida promedio =0.940, recompensa promedio=0.2, cota recompensa=0.0\n",
      "239: pérdida promedio =0.953, recompensa promedio=0.2, cota recompensa=0.1\n",
      "240: pérdida promedio =0.951, recompensa promedio=0.3, cota recompensa=0.3\n",
      "241: pérdida promedio =0.950, recompensa promedio=0.3, cota recompensa=0.3\n",
      "242: pérdida promedio =0.953, recompensa promedio=0.3, cota recompensa=0.3\n",
      "243: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.3\n",
      "244: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.4\n",
      "245: pérdida promedio =0.932, recompensa promedio=0.3, cota recompensa=0.1\n",
      "246: pérdida promedio =0.939, recompensa promedio=0.3, cota recompensa=0.3\n",
      "247: pérdida promedio =0.936, recompensa promedio=0.3, cota recompensa=0.3\n",
      "248: pérdida promedio =0.937, recompensa promedio=0.3, cota recompensa=0.4\n",
      "249: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.4\n",
      "250: pérdida promedio =0.946, recompensa promedio=0.3, cota recompensa=0.2\n",
      "251: pérdida promedio =0.943, recompensa promedio=0.3, cota recompensa=0.2\n",
      "252: pérdida promedio =0.939, recompensa promedio=0.3, cota recompensa=0.2\n",
      "253: pérdida promedio =0.953, recompensa promedio=0.3, cota recompensa=0.3\n",
      "254: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.3\n",
      "255: pérdida promedio =0.947, recompensa promedio=0.3, cota recompensa=0.4\n",
      "256: pérdida promedio =0.956, recompensa promedio=0.3, cota recompensa=0.4\n",
      "257: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.4\n",
      "258: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.4\n",
      "259: pérdida promedio =0.959, recompensa promedio=0.3, cota recompensa=0.4\n",
      "260: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.4\n",
      "261: pérdida promedio =0.957, recompensa promedio=0.3, cota recompensa=0.4\n",
      "262: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.4\n",
      "263: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.4\n",
      "264: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.4\n",
      "265: pérdida promedio =0.957, recompensa promedio=0.3, cota recompensa=0.5\n",
      "266: pérdida promedio =0.952, recompensa promedio=0.2, cota recompensa=0.0\n",
      "267: pérdida promedio =0.942, recompensa promedio=0.2, cota recompensa=0.0\n",
      "268: pérdida promedio =0.944, recompensa promedio=0.2, cota recompensa=0.0\n",
      "269: pérdida promedio =0.941, recompensa promedio=0.2, cota recompensa=0.0\n",
      "270: pérdida promedio =0.945, recompensa promedio=0.2, cota recompensa=0.0\n",
      "271: pérdida promedio =0.952, recompensa promedio=0.2, cota recompensa=0.1\n",
      "272: pérdida promedio =0.944, recompensa promedio=0.3, cota recompensa=0.2\n",
      "273: pérdida promedio =0.933, recompensa promedio=0.3, cota recompensa=0.3\n",
      "274: pérdida promedio =0.948, recompensa promedio=0.3, cota recompensa=0.3\n",
      "275: pérdida promedio =0.938, recompensa promedio=0.3, cota recompensa=0.3\n",
      "276: pérdida promedio =0.939, recompensa promedio=0.3, cota recompensa=0.2\n",
      "277: pérdida promedio =0.933, recompensa promedio=0.3, cota recompensa=0.3\n",
      "278: pérdida promedio =0.948, recompensa promedio=0.3, cota recompensa=0.3\n",
      "279: pérdida promedio =0.953, recompensa promedio=0.3, cota recompensa=0.2\n",
      "280: pérdida promedio =0.943, recompensa promedio=0.3, cota recompensa=0.3\n",
      "281: pérdida promedio =0.950, recompensa promedio=0.3, cota recompensa=0.3\n",
      "282: pérdida promedio =0.947, recompensa promedio=0.3, cota recompensa=0.3\n",
      "283: pérdida promedio =0.947, recompensa promedio=0.3, cota recompensa=0.3\n",
      "284: pérdida promedio =0.954, recompensa promedio=0.3, cota recompensa=0.4\n",
      "285: pérdida promedio =0.933, recompensa promedio=0.3, cota recompensa=0.0\n",
      "286: pérdida promedio =0.937, recompensa promedio=0.3, cota recompensa=0.1\n",
      "287: pérdida promedio =0.943, recompensa promedio=0.3, cota recompensa=0.3\n",
      "288: pérdida promedio =0.944, recompensa promedio=0.3, cota recompensa=0.3\n",
      "289: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.3\n",
      "290: pérdida promedio =0.943, recompensa promedio=0.3, cota recompensa=0.3\n",
      "291: pérdida promedio =0.942, recompensa promedio=0.3, cota recompensa=0.3\n",
      "292: pérdida promedio =0.942, recompensa promedio=0.3, cota recompensa=0.3\n",
      "293: pérdida promedio =0.949, recompensa promedio=0.3, cota recompensa=0.4\n",
      "294: pérdida promedio =0.946, recompensa promedio=0.3, cota recompensa=0.3\n",
      "295: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.3\n",
      "296: pérdida promedio =0.948, recompensa promedio=0.3, cota recompensa=0.4\n",
      "297: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.4\n",
      "298: pérdida promedio =0.962, recompensa promedio=0.3, cota recompensa=0.3\n",
      "299: pérdida promedio =0.947, recompensa promedio=0.3, cota recompensa=0.2\n",
      "300: pérdida promedio =0.949, recompensa promedio=0.3, cota recompensa=0.4\n",
      "301: pérdida promedio =0.950, recompensa promedio=0.3, cota recompensa=0.4\n",
      "302: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.1\n",
      "303: pérdida promedio =0.946, recompensa promedio=0.3, cota recompensa=0.3\n",
      "304: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.3\n",
      "305: pérdida promedio =0.948, recompensa promedio=0.3, cota recompensa=0.4\n",
      "306: pérdida promedio =0.948, recompensa promedio=0.3, cota recompensa=0.2\n",
      "307: pérdida promedio =0.942, recompensa promedio=0.3, cota recompensa=0.5\n",
      "308: pérdida promedio =0.941, recompensa promedio=0.3, cota recompensa=0.2\n",
      "309: pérdida promedio =0.938, recompensa promedio=0.3, cota recompensa=0.2\n",
      "310: pérdida promedio =0.935, recompensa promedio=0.3, cota recompensa=0.1\n",
      "311: pérdida promedio =0.932, recompensa promedio=0.3, cota recompensa=0.2\n",
      "312: pérdida promedio =0.931, recompensa promedio=0.3, cota recompensa=0.3\n",
      "313: pérdida promedio =0.937, recompensa promedio=0.3, cota recompensa=0.3\n",
      "314: pérdida promedio =0.938, recompensa promedio=0.3, cota recompensa=0.3\n",
      "315: pérdida promedio =0.938, recompensa promedio=0.3, cota recompensa=0.4\n",
      "316: pérdida promedio =0.951, recompensa promedio=0.3, cota recompensa=0.4\n",
      "317: pérdida promedio =0.953, recompensa promedio=0.3, cota recompensa=0.3\n",
      "318: pérdida promedio =0.949, recompensa promedio=0.3, cota recompensa=0.4\n",
      "319: pérdida promedio =0.951, recompensa promedio=0.3, cota recompensa=0.4\n",
      "320: pérdida promedio =0.951, recompensa promedio=0.3, cota recompensa=0.4\n",
      "321: pérdida promedio =0.954, recompensa promedio=0.3, cota recompensa=0.3\n",
      "322: pérdida promedio =0.949, recompensa promedio=0.3, cota recompensa=0.5\n",
      "323: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.3\n",
      "324: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.3\n",
      "325: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.3\n",
      "326: pérdida promedio =0.951, recompensa promedio=0.3, cota recompensa=0.3\n",
      "327: pérdida promedio =0.957, recompensa promedio=0.3, cota recompensa=0.4\n",
      "328: pérdida promedio =0.957, recompensa promedio=0.3, cota recompensa=0.4\n",
      "329: pérdida promedio =0.956, recompensa promedio=0.3, cota recompensa=0.4\n",
      "330: pérdida promedio =0.956, recompensa promedio=0.3, cota recompensa=0.4\n",
      "331: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.5\n",
      "332: pérdida promedio =0.955, recompensa promedio=0.3, cota recompensa=0.4\n",
      "333: pérdida promedio =0.958, recompensa promedio=0.3, cota recompensa=0.3\n",
      "334: pérdida promedio =0.952, recompensa promedio=0.3, cota recompensa=0.3\n",
      "335: pérdida promedio =0.954, recompensa promedio=0.3, cota recompensa=0.4\n",
      "336: pérdida promedio =0.947, recompensa promedio=0.3, cota recompensa=0.4\n",
      "337: pérdida promedio =0.950, recompensa promedio=0.3, cota recompensa=0.4\n",
      "338: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.5\n",
      "339: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.4\n",
      "340: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.5\n",
      "341: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.4\n",
      "342: pérdida promedio =0.945, recompensa promedio=0.3, cota recompensa=0.4\n",
      "343: pérdida promedio =nan, recompensa promedio=0.3, cota recompensa=0.5\n",
      "344: pérdida promedio =0.748, recompensa promedio=0.0, cota recompensa=0.0\n",
      "345: pérdida promedio =0.725, recompensa promedio=0.0, cota recompensa=0.0\n",
      "346: pérdida promedio =0.769, recompensa promedio=0.1, cota recompensa=0.0\n",
      "347: pérdida promedio =0.771, recompensa promedio=0.1, cota recompensa=0.0\n",
      "348: pérdida promedio =0.762, recompensa promedio=0.1, cota recompensa=0.0\n",
      "349: pérdida promedio =0.736, recompensa promedio=0.1, cota recompensa=0.0\n",
      "350: pérdida promedio =0.755, recompensa promedio=0.1, cota recompensa=0.0\n",
      "351: pérdida promedio =0.756, recompensa promedio=0.1, cota recompensa=0.0\n",
      "352: pérdida promedio =0.752, recompensa promedio=0.1, cota recompensa=0.0\n",
      "353: pérdida promedio =0.761, recompensa promedio=0.1, cota recompensa=0.0\n",
      "354: pérdida promedio =0.772, recompensa promedio=0.1, cota recompensa=0.0\n",
      "355: pérdida promedio =0.763, recompensa promedio=0.1, cota recompensa=0.0\n",
      "356: pérdida promedio =0.776, recompensa promedio=0.1, cota recompensa=0.1\n",
      "357: pérdida promedio =0.773, recompensa promedio=0.1, cota recompensa=0.1\n",
      "358: pérdida promedio =0.771, recompensa promedio=0.1, cota recompensa=0.1\n",
      "359: pérdida promedio =0.768, recompensa promedio=0.2, cota recompensa=0.1\n",
      "360: pérdida promedio =0.764, recompensa promedio=0.2, cota recompensa=0.1\n",
      "361: pérdida promedio =0.772, recompensa promedio=0.2, cota recompensa=0.1\n",
      "362: pérdida promedio =0.792, recompensa promedio=0.2, cota recompensa=0.2\n",
      "363: pérdida promedio =0.804, recompensa promedio=0.2, cota recompensa=0.2\n",
      "364: pérdida promedio =0.805, recompensa promedio=0.2, cota recompensa=0.2\n",
      "365: pérdida promedio =0.812, recompensa promedio=0.2, cota recompensa=0.2\n",
      "366: pérdida promedio =0.814, recompensa promedio=0.2, cota recompensa=0.3\n",
      "367: pérdida promedio =0.802, recompensa promedio=0.2, cota recompensa=0.3\n",
      "368: pérdida promedio =0.795, recompensa promedio=0.2, cota recompensa=0.1\n",
      "369: pérdida promedio =0.790, recompensa promedio=0.2, cota recompensa=0.1\n",
      "370: pérdida promedio =0.802, recompensa promedio=0.2, cota recompensa=0.3\n",
      "371: pérdida promedio =0.799, recompensa promedio=0.2, cota recompensa=0.3\n",
      "372: pérdida promedio =0.812, recompensa promedio=0.2, cota recompensa=0.3\n",
      "373: pérdida promedio =0.799, recompensa promedio=0.2, cota recompensa=0.1\n",
      "374: pérdida promedio =0.791, recompensa promedio=0.2, cota recompensa=0.2\n",
      "375: pérdida promedio =0.795, recompensa promedio=0.2, cota recompensa=0.2\n",
      "376: pérdida promedio =0.801, recompensa promedio=0.2, cota recompensa=0.3\n",
      "377: pérdida promedio =0.801, recompensa promedio=0.2, cota recompensa=0.3\n",
      "378: pérdida promedio =0.797, recompensa promedio=0.2, cota recompensa=0.3\n",
      "379: pérdida promedio =0.810, recompensa promedio=0.2, cota recompensa=0.3\n",
      "380: pérdida promedio =0.777, recompensa promedio=0.2, cota recompensa=0.0\n",
      "381: pérdida promedio =0.781, recompensa promedio=0.2, cota recompensa=0.0\n",
      "382: pérdida promedio =0.798, recompensa promedio=0.2, cota recompensa=0.1\n",
      "383: pérdida promedio =0.812, recompensa promedio=0.2, cota recompensa=0.2\n",
      "384: pérdida promedio =0.814, recompensa promedio=0.2, cota recompensa=0.2\n",
      "385: pérdida promedio =0.811, recompensa promedio=0.2, cota recompensa=0.3\n",
      "386: pérdida promedio =0.807, recompensa promedio=0.2, cota recompensa=0.2\n",
      "387: pérdida promedio =0.805, recompensa promedio=0.2, cota recompensa=0.3\n",
      "388: pérdida promedio =0.799, recompensa promedio=0.2, cota recompensa=0.3\n",
      "389: pérdida promedio =0.793, recompensa promedio=0.2, cota recompensa=0.2\n",
      "390: pérdida promedio =0.788, recompensa promedio=0.2, cota recompensa=0.3\n",
      "391: pérdida promedio =0.797, recompensa promedio=0.2, cota recompensa=0.4\n",
      "392: pérdida promedio =0.808, recompensa promedio=0.2, cota recompensa=0.0\n",
      "393: pérdida promedio =0.775, recompensa promedio=0.2, cota recompensa=0.0\n",
      "394: pérdida promedio =0.792, recompensa promedio=0.2, cota recompensa=0.1\n",
      "395: pérdida promedio =0.799, recompensa promedio=0.2, cota recompensa=0.1\n",
      "396: pérdida promedio =0.813, recompensa promedio=0.2, cota recompensa=0.2\n",
      "397: pérdida promedio =0.826, recompensa promedio=0.2, cota recompensa=0.2\n",
      "398: pérdida promedio =0.809, recompensa promedio=0.2, cota recompensa=0.3\n",
      "399: pérdida promedio =0.815, recompensa promedio=0.2, cota recompensa=0.3\n",
      "400: pérdida promedio =0.808, recompensa promedio=0.2, cota recompensa=0.3\n",
      "401: pérdida promedio =0.797, recompensa promedio=0.2, cota recompensa=0.3\n",
      "402: pérdida promedio =0.794, recompensa promedio=0.2, cota recompensa=0.2\n",
      "403: pérdida promedio =0.798, recompensa promedio=0.2, cota recompensa=0.3\n",
      "404: pérdida promedio =0.805, recompensa promedio=0.2, cota recompensa=0.3\n",
      "405: pérdida promedio =0.805, recompensa promedio=0.2, cota recompensa=0.3\n",
      "406: pérdida promedio =0.810, recompensa promedio=0.2, cota recompensa=0.3\n",
      "407: pérdida promedio =0.799, recompensa promedio=0.2, cota recompensa=0.3\n",
      "408: pérdida promedio =0.797, recompensa promedio=0.2, cota recompensa=0.4\n",
      "409: pérdida promedio =0.797, recompensa promedio=0.2, cota recompensa=0.3\n",
      "410: pérdida promedio =0.801, recompensa promedio=0.2, cota recompensa=0.3\n",
      "411: pérdida promedio =0.795, recompensa promedio=0.3, cota recompensa=0.4\n",
      "412: pérdida promedio =0.794, recompensa promedio=0.2, cota recompensa=0.3\n",
      "413: pérdida promedio =0.791, recompensa promedio=0.2, cota recompensa=0.3\n",
      "414: pérdida promedio =0.791, recompensa promedio=0.2, cota recompensa=0.4\n",
      "415: pérdida promedio =0.793, recompensa promedio=0.3, cota recompensa=0.4\n",
      "416: pérdida promedio =0.798, recompensa promedio=0.2, cota recompensa=0.0\n",
      "417: pérdida promedio =0.807, recompensa promedio=0.2, cota recompensa=0.0\n",
      "418: pérdida promedio =0.820, recompensa promedio=0.2, cota recompensa=0.2\n",
      "419: pérdida promedio =0.807, recompensa promedio=0.2, cota recompensa=0.2\n",
      "420: pérdida promedio =0.801, recompensa promedio=0.2, cota recompensa=0.2\n",
      "421: pérdida promedio =0.800, recompensa promedio=0.2, cota recompensa=0.3\n",
      "422: pérdida promedio =0.801, recompensa promedio=0.2, cota recompensa=0.3\n",
      "423: pérdida promedio =0.806, recompensa promedio=0.2, cota recompensa=0.3\n",
      "424: pérdida promedio =0.803, recompensa promedio=0.2, cota recompensa=0.3\n",
      "425: pérdida promedio =0.807, recompensa promedio=0.2, cota recompensa=0.3\n",
      "426: pérdida promedio =0.797, recompensa promedio=0.2, cota recompensa=0.3\n",
      "427: pérdida promedio =0.796, recompensa promedio=0.2, cota recompensa=0.2\n",
      "428: pérdida promedio =0.796, recompensa promedio=0.2, cota recompensa=0.3\n",
      "429: pérdida promedio =0.791, recompensa promedio=0.2, cota recompensa=0.1\n",
      "430: pérdida promedio =0.787, recompensa promedio=0.2, cota recompensa=0.3\n",
      "431: pérdida promedio =0.793, recompensa promedio=0.2, cota recompensa=0.3\n",
      "432: pérdida promedio =0.788, recompensa promedio=0.3, cota recompensa=0.3\n",
      "433: pérdida promedio =0.783, recompensa promedio=0.3, cota recompensa=0.4\n",
      "434: pérdida promedio =0.769, recompensa promedio=0.2, cota recompensa=0.1\n",
      "435: pérdida promedio =0.776, recompensa promedio=0.2, cota recompensa=0.3\n",
      "436: pérdida promedio =0.777, recompensa promedio=0.2, cota recompensa=0.3\n",
      "437: pérdida promedio =0.783, recompensa promedio=0.3, cota recompensa=0.3\n",
      "438: pérdida promedio =0.781, recompensa promedio=0.3, cota recompensa=0.3\n",
      "439: pérdida promedio =0.785, recompensa promedio=0.3, cota recompensa=0.4\n",
      "440: pérdida promedio =0.787, recompensa promedio=0.3, cota recompensa=0.4\n",
      "441: pérdida promedio =0.785, recompensa promedio=0.3, cota recompensa=0.4\n",
      "442: pérdida promedio =0.779, recompensa promedio=0.3, cota recompensa=0.4\n",
      "443: pérdida promedio =0.763, recompensa promedio=0.2, cota recompensa=0.1\n",
      "444: pérdida promedio =0.765, recompensa promedio=0.2, cota recompensa=0.2\n",
      "445: pérdida promedio =0.766, recompensa promedio=0.3, cota recompensa=0.3\n",
      "446: pérdida promedio =0.768, recompensa promedio=0.3, cota recompensa=0.3\n",
      "447: pérdida promedio =0.761, recompensa promedio=0.3, cota recompensa=0.2\n",
      "448: pérdida promedio =0.760, recompensa promedio=0.2, cota recompensa=0.3\n",
      "449: pérdida promedio =0.762, recompensa promedio=0.3, cota recompensa=0.3\n",
      "450: pérdida promedio =0.759, recompensa promedio=0.3, cota recompensa=0.3\n",
      "451: pérdida promedio =0.767, recompensa promedio=0.3, cota recompensa=0.4\n",
      "452: pérdida promedio =0.762, recompensa promedio=0.3, cota recompensa=0.3\n",
      "453: pérdida promedio =0.766, recompensa promedio=0.3, cota recompensa=0.4\n",
      "454: pérdida promedio =0.769, recompensa promedio=0.3, cota recompensa=0.4\n",
      "455: pérdida promedio =0.767, recompensa promedio=0.3, cota recompensa=0.3\n",
      "456: pérdida promedio =0.771, recompensa promedio=0.3, cota recompensa=0.3\n",
      "457: pérdida promedio =0.768, recompensa promedio=0.3, cota recompensa=0.4\n",
      "458: pérdida promedio =0.767, recompensa promedio=0.3, cota recompensa=0.4\n",
      "459: pérdida promedio =0.762, recompensa promedio=0.3, cota recompensa=0.3\n",
      "460: pérdida promedio =0.763, recompensa promedio=0.3, cota recompensa=0.3\n",
      "461: pérdida promedio =0.770, recompensa promedio=0.3, cota recompensa=0.4\n",
      "462: pérdida promedio =0.763, recompensa promedio=0.2, cota recompensa=0.1\n",
      "463: pérdida promedio =0.770, recompensa promedio=0.3, cota recompensa=0.4\n",
      "464: pérdida promedio =0.757, recompensa promedio=0.3, cota recompensa=0.5\n",
      "465: pérdida promedio =0.711, recompensa promedio=0.2, cota recompensa=0.0\n",
      "466: pérdida promedio =0.716, recompensa promedio=0.2, cota recompensa=0.0\n",
      "467: pérdida promedio =0.713, recompensa promedio=0.2, cota recompensa=0.0\n",
      "468: pérdida promedio =0.705, recompensa promedio=0.2, cota recompensa=0.0\n",
      "469: pérdida promedio =0.698, recompensa promedio=0.2, cota recompensa=0.0\n",
      "470: pérdida promedio =0.745, recompensa promedio=0.2, cota recompensa=0.1\n",
      "471: pérdida promedio =0.759, recompensa promedio=0.2, cota recompensa=0.1\n",
      "472: pérdida promedio =0.786, recompensa promedio=0.2, cota recompensa=0.2\n",
      "473: pérdida promedio =0.789, recompensa promedio=0.2, cota recompensa=0.2\n",
      "474: pérdida promedio =0.777, recompensa promedio=0.2, cota recompensa=0.2\n",
      "475: pérdida promedio =0.764, recompensa promedio=0.2, cota recompensa=0.2\n",
      "476: pérdida promedio =0.769, recompensa promedio=0.2, cota recompensa=0.2\n",
      "477: pérdida promedio =0.769, recompensa promedio=0.2, cota recompensa=0.3\n",
      "478: pérdida promedio =0.754, recompensa promedio=0.2, cota recompensa=0.3\n",
      "479: pérdida promedio =0.750, recompensa promedio=0.2, cota recompensa=0.2\n",
      "480: pérdida promedio =0.754, recompensa promedio=0.2, cota recompensa=0.3\n",
      "481: pérdida promedio =0.759, recompensa promedio=0.2, cota recompensa=0.0\n",
      "482: pérdida promedio =0.753, recompensa promedio=0.2, cota recompensa=0.2\n",
      "483: pérdida promedio =0.752, recompensa promedio=0.2, cota recompensa=0.3\n",
      "484: pérdida promedio =0.754, recompensa promedio=0.2, cota recompensa=0.2\n",
      "485: pérdida promedio =0.746, recompensa promedio=0.2, cota recompensa=0.4\n",
      "486: pérdida promedio =0.744, recompensa promedio=0.2, cota recompensa=0.3\n",
      "487: pérdida promedio =0.740, recompensa promedio=0.3, cota recompensa=0.4\n",
      "488: pérdida promedio =0.740, recompensa promedio=0.2, cota recompensa=0.3\n",
      "489: pérdida promedio =0.730, recompensa promedio=0.2, cota recompensa=0.4\n",
      "490: pérdida promedio =0.731, recompensa promedio=0.2, cota recompensa=0.2\n",
      "491: pérdida promedio =0.727, recompensa promedio=0.2, cota recompensa=0.2\n",
      "492: pérdida promedio =0.716, recompensa promedio=0.2, cota recompensa=0.3\n",
      "493: pérdida promedio =0.731, recompensa promedio=0.3, cota recompensa=0.3\n",
      "494: pérdida promedio =0.734, recompensa promedio=0.2, cota recompensa=0.3\n",
      "495: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.4\n",
      "496: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.4\n",
      "497: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.4\n",
      "498: pérdida promedio =0.725, recompensa promedio=0.3, cota recompensa=0.4\n",
      "499: pérdida promedio =0.712, recompensa promedio=0.2, cota recompensa=0.0\n",
      "500: pérdida promedio =0.712, recompensa promedio=0.2, cota recompensa=0.2\n",
      "501: pérdida promedio =0.727, recompensa promedio=0.3, cota recompensa=0.3\n",
      "502: pérdida promedio =0.736, recompensa promedio=0.3, cota recompensa=0.3\n",
      "503: pérdida promedio =0.736, recompensa promedio=0.3, cota recompensa=0.3\n",
      "504: pérdida promedio =0.742, recompensa promedio=0.3, cota recompensa=0.3\n",
      "505: pérdida promedio =0.735, recompensa promedio=0.3, cota recompensa=0.2\n",
      "506: pérdida promedio =0.734, recompensa promedio=0.3, cota recompensa=0.4\n",
      "507: pérdida promedio =0.732, recompensa promedio=0.3, cota recompensa=0.3\n",
      "508: pérdida promedio =0.730, recompensa promedio=0.3, cota recompensa=0.2\n",
      "509: pérdida promedio =0.724, recompensa promedio=0.3, cota recompensa=0.4\n",
      "510: pérdida promedio =0.729, recompensa promedio=0.3, cota recompensa=0.1\n",
      "511: pérdida promedio =0.719, recompensa promedio=0.3, cota recompensa=0.3\n",
      "512: pérdida promedio =0.723, recompensa promedio=0.3, cota recompensa=0.3\n",
      "513: pérdida promedio =0.726, recompensa promedio=0.3, cota recompensa=0.3\n",
      "514: pérdida promedio =0.730, recompensa promedio=0.3, cota recompensa=0.4\n",
      "515: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.4\n",
      "516: pérdida promedio =0.730, recompensa promedio=0.3, cota recompensa=0.4\n",
      "517: pérdida promedio =0.730, recompensa promedio=0.3, cota recompensa=0.4\n",
      "518: pérdida promedio =0.730, recompensa promedio=0.3, cota recompensa=0.4\n",
      "519: pérdida promedio =0.730, recompensa promedio=0.3, cota recompensa=0.4\n",
      "520: pérdida promedio =0.749, recompensa promedio=0.3, cota recompensa=0.5\n",
      "521: pérdida promedio =0.745, recompensa promedio=0.2, cota recompensa=0.0\n",
      "522: pérdida promedio =0.709, recompensa promedio=0.2, cota recompensa=0.0\n",
      "523: pérdida promedio =0.706, recompensa promedio=0.2, cota recompensa=0.1\n",
      "524: pérdida promedio =0.722, recompensa promedio=0.2, cota recompensa=0.2\n",
      "525: pérdida promedio =0.724, recompensa promedio=0.2, cota recompensa=0.2\n",
      "526: pérdida promedio =0.738, recompensa promedio=0.3, cota recompensa=0.3\n",
      "527: pérdida promedio =0.744, recompensa promedio=0.3, cota recompensa=0.3\n",
      "528: pérdida promedio =0.732, recompensa promedio=0.2, cota recompensa=0.1\n",
      "529: pérdida promedio =0.738, recompensa promedio=0.3, cota recompensa=0.3\n",
      "530: pérdida promedio =0.737, recompensa promedio=0.3, cota recompensa=0.3\n",
      "531: pérdida promedio =0.738, recompensa promedio=0.3, cota recompensa=0.3\n",
      "532: pérdida promedio =0.731, recompensa promedio=0.3, cota recompensa=0.4\n",
      "533: pérdida promedio =0.718, recompensa promedio=0.3, cota recompensa=0.4\n",
      "534: pérdida promedio =0.719, recompensa promedio=0.3, cota recompensa=0.2\n",
      "535: pérdida promedio =0.710, recompensa promedio=0.3, cota recompensa=0.3\n",
      "536: pérdida promedio =0.715, recompensa promedio=0.3, cota recompensa=0.4\n",
      "537: pérdida promedio =0.712, recompensa promedio=0.3, cota recompensa=0.4\n",
      "538: pérdida promedio =0.727, recompensa promedio=0.3, cota recompensa=0.4\n",
      "539: pérdida promedio =0.728, recompensa promedio=0.3, cota recompensa=0.3\n",
      "540: pérdida promedio =0.719, recompensa promedio=0.3, cota recompensa=0.3\n",
      "541: pérdida promedio =0.730, recompensa promedio=0.3, cota recompensa=0.3\n",
      "542: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.3\n",
      "543: pérdida promedio =0.725, recompensa promedio=0.3, cota recompensa=0.4\n",
      "544: pérdida promedio =0.723, recompensa promedio=0.3, cota recompensa=0.4\n",
      "545: pérdida promedio =0.725, recompensa promedio=0.3, cota recompensa=0.4\n",
      "546: pérdida promedio =0.726, recompensa promedio=0.3, cota recompensa=0.4\n",
      "547: pérdida promedio =0.724, recompensa promedio=0.3, cota recompensa=0.4\n",
      "548: pérdida promedio =0.722, recompensa promedio=0.3, cota recompensa=0.4\n",
      "549: pérdida promedio =0.722, recompensa promedio=0.3, cota recompensa=0.5\n",
      "550: pérdida promedio =0.743, recompensa promedio=0.3, cota recompensa=0.5\n",
      "551: pérdida promedio =0.723, recompensa promedio=0.3, cota recompensa=0.2\n",
      "552: pérdida promedio =0.712, recompensa promedio=0.2, cota recompensa=0.0\n",
      "553: pérdida promedio =0.728, recompensa promedio=0.3, cota recompensa=0.3\n",
      "554: pérdida promedio =0.731, recompensa promedio=0.3, cota recompensa=0.3\n",
      "555: pérdida promedio =0.735, recompensa promedio=0.3, cota recompensa=0.3\n",
      "556: pérdida promedio =0.727, recompensa promedio=0.3, cota recompensa=0.3\n",
      "557: pérdida promedio =0.731, recompensa promedio=0.3, cota recompensa=0.2\n",
      "558: pérdida promedio =0.723, recompensa promedio=0.3, cota recompensa=0.3\n",
      "559: pérdida promedio =0.736, recompensa promedio=0.3, cota recompensa=0.4\n",
      "560: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.3\n",
      "561: pérdida promedio =0.726, recompensa promedio=0.3, cota recompensa=0.4\n",
      "562: pérdida promedio =0.726, recompensa promedio=0.3, cota recompensa=0.4\n",
      "563: pérdida promedio =0.725, recompensa promedio=0.3, cota recompensa=0.4\n",
      "564: pérdida promedio =0.721, recompensa promedio=0.3, cota recompensa=0.5\n",
      "565: pérdida promedio =0.737, recompensa promedio=0.3, cota recompensa=0.5\n",
      "566: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.1\n",
      "567: pérdida promedio =0.723, recompensa promedio=0.3, cota recompensa=0.3\n",
      "568: pérdida promedio =0.728, recompensa promedio=0.3, cota recompensa=0.4\n",
      "569: pérdida promedio =0.739, recompensa promedio=0.3, cota recompensa=0.3\n",
      "570: pérdida promedio =0.736, recompensa promedio=0.3, cota recompensa=0.4\n",
      "571: pérdida promedio =0.735, recompensa promedio=0.3, cota recompensa=0.4\n",
      "572: pérdida promedio =0.734, recompensa promedio=0.3, cota recompensa=0.4\n",
      "573: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.4\n",
      "574: pérdida promedio =0.735, recompensa promedio=0.3, cota recompensa=0.5\n",
      "575: pérdida promedio =0.742, recompensa promedio=0.3, cota recompensa=0.4\n",
      "576: pérdida promedio =0.738, recompensa promedio=0.3, cota recompensa=0.4\n",
      "577: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.4\n",
      "578: pérdida promedio =0.733, recompensa promedio=0.3, cota recompensa=0.4\n",
      "579: pérdida promedio =0.741, recompensa promedio=0.3, cota recompensa=0.5\n",
      "580: pérdida promedio =0.740, recompensa promedio=0.3, cota recompensa=0.4\n",
      "581: pérdida promedio =0.740, recompensa promedio=0.3, cota recompensa=0.4\n",
      "582: pérdida promedio =0.740, recompensa promedio=0.3, cota recompensa=0.4\n",
      "583: pérdida promedio =0.740, recompensa promedio=0.3, cota recompensa=0.5\n",
      "584: pérdida promedio =0.740, recompensa promedio=0.3, cota recompensa=0.5\n",
      "585: pérdida promedio =nan, recompensa promedio=0.3, cota recompensa=0.5\n",
      "586: pérdida promedio =0.660, recompensa promedio=0.0, cota recompensa=0.0\n",
      "587: pérdida promedio =0.624, recompensa promedio=0.1, cota recompensa=0.0\n",
      "588: pérdida promedio =0.651, recompensa promedio=0.1, cota recompensa=0.0\n",
      "589: pérdida promedio =0.625, recompensa promedio=0.1, cota recompensa=0.0\n",
      "590: pérdida promedio =0.615, recompensa promedio=0.1, cota recompensa=0.0\n",
      "591: pérdida promedio =0.609, recompensa promedio=0.1, cota recompensa=0.0\n",
      "592: pérdida promedio =0.591, recompensa promedio=0.1, cota recompensa=0.0\n",
      "593: pérdida promedio =0.582, recompensa promedio=0.1, cota recompensa=0.0\n",
      "594: pérdida promedio =0.599, recompensa promedio=0.1, cota recompensa=0.0\n",
      "595: pérdida promedio =0.608, recompensa promedio=0.1, cota recompensa=0.1\n",
      "596: pérdida promedio =0.604, recompensa promedio=0.1, cota recompensa=0.1\n",
      "597: pérdida promedio =0.645, recompensa promedio=0.2, cota recompensa=0.1\n",
      "598: pérdida promedio =0.641, recompensa promedio=0.2, cota recompensa=0.2\n",
      "599: pérdida promedio =0.634, recompensa promedio=0.2, cota recompensa=0.2\n",
      "600: pérdida promedio =0.637, recompensa promedio=0.2, cota recompensa=0.2\n",
      "601: pérdida promedio =0.612, recompensa promedio=0.2, cota recompensa=0.2\n",
      "602: pérdida promedio =0.616, recompensa promedio=0.2, cota recompensa=0.1\n",
      "603: pérdida promedio =0.611, recompensa promedio=0.2, cota recompensa=0.1\n",
      "604: pérdida promedio =0.603, recompensa promedio=0.2, cota recompensa=0.2\n",
      "605: pérdida promedio =0.595, recompensa promedio=0.2, cota recompensa=0.3\n",
      "606: pérdida promedio =0.585, recompensa promedio=0.2, cota recompensa=0.2\n",
      "607: pérdida promedio =0.583, recompensa promedio=0.2, cota recompensa=0.3\n",
      "608: pérdida promedio =0.580, recompensa promedio=0.2, cota recompensa=0.3\n",
      "609: pérdida promedio =0.576, recompensa promedio=0.2, cota recompensa=0.2\n",
      "610: pérdida promedio =0.585, recompensa promedio=0.2, cota recompensa=0.3\n",
      "611: pérdida promedio =0.590, recompensa promedio=0.2, cota recompensa=0.2\n",
      "612: pérdida promedio =0.591, recompensa promedio=0.2, cota recompensa=0.2\n",
      "613: pérdida promedio =0.599, recompensa promedio=0.2, cota recompensa=0.3\n",
      "614: pérdida promedio =0.638, recompensa promedio=0.2, cota recompensa=0.3\n",
      "615: pérdida promedio =0.628, recompensa promedio=0.2, cota recompensa=0.0\n",
      "616: pérdida promedio =0.625, recompensa promedio=0.2, cota recompensa=0.1\n",
      "617: pérdida promedio =0.625, recompensa promedio=0.2, cota recompensa=0.2\n",
      "618: pérdida promedio =0.633, recompensa promedio=0.2, cota recompensa=0.3\n",
      "619: pérdida promedio =0.639, recompensa promedio=0.2, cota recompensa=0.3\n",
      "620: pérdida promedio =0.645, recompensa promedio=0.2, cota recompensa=0.3\n",
      "621: pérdida promedio =0.619, recompensa promedio=0.2, cota recompensa=0.0\n",
      "622: pérdida promedio =0.626, recompensa promedio=0.2, cota recompensa=0.0\n",
      "623: pérdida promedio =0.641, recompensa promedio=0.2, cota recompensa=0.1\n",
      "624: pérdida promedio =0.636, recompensa promedio=0.2, cota recompensa=0.2\n",
      "625: pérdida promedio =0.637, recompensa promedio=0.2, cota recompensa=0.2\n",
      "626: pérdida promedio =0.639, recompensa promedio=0.2, cota recompensa=0.3\n",
      "627: pérdida promedio =0.631, recompensa promedio=0.2, cota recompensa=0.3\n",
      "628: pérdida promedio =0.621, recompensa promedio=0.2, cota recompensa=0.2\n",
      "629: pérdida promedio =0.636, recompensa promedio=0.2, cota recompensa=0.3\n",
      "630: pérdida promedio =0.629, recompensa promedio=0.2, cota recompensa=0.3\n",
      "631: pérdida promedio =0.641, recompensa promedio=0.2, cota recompensa=0.4\n",
      "632: pérdida promedio =0.624, recompensa promedio=0.2, cota recompensa=0.0\n",
      "633: pérdida promedio =0.617, recompensa promedio=0.2, cota recompensa=0.0\n",
      "634: pérdida promedio =0.618, recompensa promedio=0.2, cota recompensa=0.0\n",
      "635: pérdida promedio =0.614, recompensa promedio=0.2, cota recompensa=0.1\n",
      "636: pérdida promedio =0.626, recompensa promedio=0.2, cota recompensa=0.2\n",
      "637: pérdida promedio =0.620, recompensa promedio=0.2, cota recompensa=0.2\n",
      "638: pérdida promedio =0.611, recompensa promedio=0.2, cota recompensa=0.2\n",
      "639: pérdida promedio =0.617, recompensa promedio=0.2, cota recompensa=0.2\n",
      "640: pérdida promedio =0.616, recompensa promedio=0.2, cota recompensa=0.3\n",
      "641: pérdida promedio =0.633, recompensa promedio=0.2, cota recompensa=0.3\n",
      "642: pérdida promedio =0.622, recompensa promedio=0.2, cota recompensa=0.3\n",
      "643: pérdida promedio =0.621, recompensa promedio=0.2, cota recompensa=0.3\n",
      "644: pérdida promedio =0.618, recompensa promedio=0.2, cota recompensa=0.3\n",
      "645: pérdida promedio =0.632, recompensa promedio=0.2, cota recompensa=0.4\n",
      "646: pérdida promedio =0.624, recompensa promedio=0.2, cota recompensa=0.2\n",
      "647: pérdida promedio =0.631, recompensa promedio=0.2, cota recompensa=0.2\n",
      "648: pérdida promedio =0.644, recompensa promedio=0.2, cota recompensa=0.3\n",
      "649: pérdida promedio =0.630, recompensa promedio=0.2, cota recompensa=0.3\n",
      "650: pérdida promedio =0.628, recompensa promedio=0.2, cota recompensa=0.3\n",
      "651: pérdida promedio =0.632, recompensa promedio=0.2, cota recompensa=0.4\n",
      "652: pérdida promedio =0.633, recompensa promedio=0.2, cota recompensa=0.3\n",
      "653: pérdida promedio =0.630, recompensa promedio=0.3, cota recompensa=0.4\n",
      "654: pérdida promedio =0.632, recompensa promedio=0.2, cota recompensa=0.3\n",
      "655: pérdida promedio =0.629, recompensa promedio=0.2, cota recompensa=0.4\n",
      "656: pérdida promedio =0.631, recompensa promedio=0.2, cota recompensa=0.4\n",
      "657: pérdida promedio =0.631, recompensa promedio=0.3, cota recompensa=0.4\n",
      "658: pérdida promedio =0.631, recompensa promedio=0.2, cota recompensa=0.4\n",
      "659: pérdida promedio =0.623, recompensa promedio=0.3, cota recompensa=0.4\n",
      "660: pérdida promedio =0.615, recompensa promedio=0.2, cota recompensa=0.0\n",
      "661: pérdida promedio =0.580, recompensa promedio=0.2, cota recompensa=0.0\n",
      "662: pérdida promedio =0.571, recompensa promedio=0.2, cota recompensa=0.0\n",
      "663: pérdida promedio =0.601, recompensa promedio=0.2, cota recompensa=0.1\n",
      "664: pérdida promedio =0.607, recompensa promedio=0.2, cota recompensa=0.1\n",
      "665: pérdida promedio =0.623, recompensa promedio=0.2, cota recompensa=0.2\n",
      "666: pérdida promedio =0.629, recompensa promedio=0.2, cota recompensa=0.2\n",
      "667: pérdida promedio =0.632, recompensa promedio=0.2, cota recompensa=0.3\n",
      "668: pérdida promedio =0.633, recompensa promedio=0.2, cota recompensa=0.3\n",
      "669: pérdida promedio =0.625, recompensa promedio=0.2, cota recompensa=0.3\n",
      "670: pérdida promedio =0.616, recompensa promedio=0.2, cota recompensa=0.3\n",
      "671: pérdida promedio =0.614, recompensa promedio=0.2, cota recompensa=0.3\n",
      "672: pérdida promedio =0.628, recompensa promedio=0.2, cota recompensa=0.3\n",
      "673: pérdida promedio =0.629, recompensa promedio=0.2, cota recompensa=0.3\n",
      "674: pérdida promedio =0.628, recompensa promedio=0.2, cota recompensa=0.3\n",
      "675: pérdida promedio =0.637, recompensa promedio=0.3, cota recompensa=0.4\n",
      "676: pérdida promedio =0.618, recompensa promedio=0.2, cota recompensa=0.1\n",
      "677: pérdida promedio =0.615, recompensa promedio=0.2, cota recompensa=0.2\n",
      "678: pérdida promedio =0.619, recompensa promedio=0.2, cota recompensa=0.2\n",
      "679: pérdida promedio =0.634, recompensa promedio=0.3, cota recompensa=0.3\n",
      "680: pérdida promedio =0.630, recompensa promedio=0.2, cota recompensa=0.2\n",
      "681: pérdida promedio =0.630, recompensa promedio=0.3, cota recompensa=0.3\n",
      "682: pérdida promedio =0.631, recompensa promedio=0.3, cota recompensa=0.4\n",
      "683: pérdida promedio =0.628, recompensa promedio=0.3, cota recompensa=0.4\n",
      "684: pérdida promedio =0.629, recompensa promedio=0.3, cota recompensa=0.3\n",
      "685: pérdida promedio =0.614, recompensa promedio=0.3, cota recompensa=0.4\n",
      "686: pérdida promedio =0.618, recompensa promedio=0.2, cota recompensa=0.0\n",
      "687: pérdida promedio =0.622, recompensa promedio=0.2, cota recompensa=0.1\n",
      "688: pérdida promedio =0.621, recompensa promedio=0.2, cota recompensa=0.2\n",
      "689: pérdida promedio =0.623, recompensa promedio=0.3, cota recompensa=0.3\n",
      "690: pérdida promedio =0.612, recompensa promedio=0.3, cota recompensa=0.3\n",
      "691: pérdida promedio =0.612, recompensa promedio=0.3, cota recompensa=0.3\n",
      "692: pérdida promedio =0.606, recompensa promedio=0.3, cota recompensa=0.4\n",
      "693: pérdida promedio =0.603, recompensa promedio=0.3, cota recompensa=0.4\n",
      "694: pérdida promedio =0.602, recompensa promedio=0.3, cota recompensa=0.4\n",
      "695: pérdida promedio =0.594, recompensa promedio=0.3, cota recompensa=0.4\n",
      "696: pérdida promedio =0.592, recompensa promedio=0.3, cota recompensa=0.2\n",
      "697: pérdida promedio =0.592, recompensa promedio=0.3, cota recompensa=0.3\n",
      "698: pérdida promedio =0.596, recompensa promedio=0.3, cota recompensa=0.3\n",
      "699: pérdida promedio =0.602, recompensa promedio=0.3, cota recompensa=0.4\n",
      "700: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.4\n",
      "701: pérdida promedio =0.586, recompensa promedio=0.3, cota recompensa=0.4\n",
      "702: pérdida promedio =0.586, recompensa promedio=0.3, cota recompensa=0.3\n",
      "703: pérdida promedio =0.584, recompensa promedio=0.3, cota recompensa=0.4\n",
      "704: pérdida promedio =0.583, recompensa promedio=0.3, cota recompensa=0.3\n",
      "705: pérdida promedio =0.580, recompensa promedio=0.3, cota recompensa=0.3\n",
      "706: pérdida promedio =0.581, recompensa promedio=0.3, cota recompensa=0.4\n",
      "707: pérdida promedio =0.583, recompensa promedio=0.3, cota recompensa=0.4\n",
      "708: pérdida promedio =0.615, recompensa promedio=0.3, cota recompensa=0.5\n",
      "709: pérdida promedio =0.562, recompensa promedio=0.2, cota recompensa=0.0\n",
      "710: pérdida promedio =0.575, recompensa promedio=0.2, cota recompensa=0.0\n",
      "711: pérdida promedio =0.570, recompensa promedio=0.2, cota recompensa=0.0\n",
      "712: pérdida promedio =0.550, recompensa promedio=0.2, cota recompensa=0.0\n",
      "713: pérdida promedio =0.543, recompensa promedio=0.2, cota recompensa=0.0\n",
      "714: pérdida promedio =0.577, recompensa promedio=0.2, cota recompensa=0.1\n",
      "715: pérdida promedio =0.568, recompensa promedio=0.2, cota recompensa=0.2\n",
      "716: pérdida promedio =0.578, recompensa promedio=0.2, cota recompensa=0.2\n",
      "717: pérdida promedio =0.587, recompensa promedio=0.2, cota recompensa=0.2\n",
      "718: pérdida promedio =0.608, recompensa promedio=0.2, cota recompensa=0.3\n",
      "719: pérdida promedio =0.596, recompensa promedio=0.2, cota recompensa=0.1\n",
      "720: pérdida promedio =0.598, recompensa promedio=0.2, cota recompensa=0.2\n",
      "721: pérdida promedio =0.595, recompensa promedio=0.2, cota recompensa=0.3\n",
      "722: pérdida promedio =0.597, recompensa promedio=0.2, cota recompensa=0.3\n",
      "723: pérdida promedio =0.586, recompensa promedio=0.2, cota recompensa=0.3\n",
      "724: pérdida promedio =0.595, recompensa promedio=0.2, cota recompensa=0.3\n",
      "725: pérdida promedio =0.590, recompensa promedio=0.2, cota recompensa=0.1\n",
      "726: pérdida promedio =0.596, recompensa promedio=0.2, cota recompensa=0.3\n",
      "727: pérdida promedio =0.595, recompensa promedio=0.2, cota recompensa=0.3\n",
      "728: pérdida promedio =0.592, recompensa promedio=0.2, cota recompensa=0.3\n",
      "729: pérdida promedio =0.589, recompensa promedio=0.2, cota recompensa=0.3\n",
      "730: pérdida promedio =0.586, recompensa promedio=0.2, cota recompensa=0.2\n",
      "731: pérdida promedio =0.582, recompensa promedio=0.3, cota recompensa=0.3\n",
      "732: pérdida promedio =0.582, recompensa promedio=0.2, cota recompensa=0.3\n",
      "733: pérdida promedio =0.589, recompensa promedio=0.2, cota recompensa=0.4\n",
      "734: pérdida promedio =0.592, recompensa promedio=0.2, cota recompensa=0.1\n",
      "735: pérdida promedio =0.594, recompensa promedio=0.2, cota recompensa=0.2\n",
      "736: pérdida promedio =0.589, recompensa promedio=0.2, cota recompensa=0.2\n",
      "737: pérdida promedio =0.570, recompensa promedio=0.3, cota recompensa=0.3\n",
      "738: pérdida promedio =0.592, recompensa promedio=0.3, cota recompensa=0.3\n",
      "739: pérdida promedio =0.588, recompensa promedio=0.3, cota recompensa=0.3\n",
      "740: pérdida promedio =0.585, recompensa promedio=0.3, cota recompensa=0.4\n",
      "741: pérdida promedio =0.594, recompensa promedio=0.3, cota recompensa=0.4\n",
      "742: pérdida promedio =0.597, recompensa promedio=0.3, cota recompensa=0.4\n",
      "743: pérdida promedio =0.586, recompensa promedio=0.2, cota recompensa=0.0\n",
      "744: pérdida promedio =0.549, recompensa promedio=0.2, cota recompensa=0.0\n",
      "745: pérdida promedio =0.563, recompensa promedio=0.2, cota recompensa=0.1\n",
      "746: pérdida promedio =0.565, recompensa promedio=0.2, cota recompensa=0.2\n",
      "747: pérdida promedio =0.572, recompensa promedio=0.2, cota recompensa=0.2\n",
      "748: pérdida promedio =0.557, recompensa promedio=0.2, cota recompensa=0.3\n",
      "749: pérdida promedio =0.556, recompensa promedio=0.2, cota recompensa=0.3\n",
      "750: pérdida promedio =0.557, recompensa promedio=0.2, cota recompensa=0.3\n",
      "751: pérdida promedio =0.569, recompensa promedio=0.2, cota recompensa=0.3\n",
      "752: pérdida promedio =0.575, recompensa promedio=0.3, cota recompensa=0.3\n",
      "753: pérdida promedio =0.574, recompensa promedio=0.3, cota recompensa=0.3\n",
      "754: pérdida promedio =0.581, recompensa promedio=0.2, cota recompensa=0.3\n",
      "755: pérdida promedio =0.569, recompensa promedio=0.3, cota recompensa=0.4\n",
      "756: pérdida promedio =0.566, recompensa promedio=0.2, cota recompensa=0.3\n",
      "757: pérdida promedio =0.588, recompensa promedio=0.3, cota recompensa=0.4\n",
      "758: pérdida promedio =0.585, recompensa promedio=0.2, cota recompensa=0.1\n",
      "759: pérdida promedio =0.594, recompensa promedio=0.3, cota recompensa=0.3\n",
      "760: pérdida promedio =0.592, recompensa promedio=0.3, cota recompensa=0.4\n",
      "761: pérdida promedio =0.601, recompensa promedio=0.3, cota recompensa=0.4\n",
      "762: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.2\n",
      "763: pérdida promedio =0.591, recompensa promedio=0.2, cota recompensa=0.2\n",
      "764: pérdida promedio =0.599, recompensa promedio=0.3, cota recompensa=0.2\n",
      "765: pérdida promedio =0.606, recompensa promedio=0.3, cota recompensa=0.3\n",
      "766: pérdida promedio =0.607, recompensa promedio=0.3, cota recompensa=0.3\n",
      "767: pérdida promedio =0.609, recompensa promedio=0.3, cota recompensa=0.3\n",
      "768: pérdida promedio =0.609, recompensa promedio=0.3, cota recompensa=0.3\n",
      "769: pérdida promedio =0.605, recompensa promedio=0.3, cota recompensa=0.3\n",
      "770: pérdida promedio =0.592, recompensa promedio=0.3, cota recompensa=0.4\n",
      "771: pérdida promedio =0.594, recompensa promedio=0.3, cota recompensa=0.4\n",
      "772: pérdida promedio =0.592, recompensa promedio=0.3, cota recompensa=0.3\n",
      "773: pérdida promedio =0.596, recompensa promedio=0.3, cota recompensa=0.4\n",
      "774: pérdida promedio =0.598, recompensa promedio=0.3, cota recompensa=0.4\n",
      "775: pérdida promedio =0.591, recompensa promedio=0.3, cota recompensa=0.4\n",
      "776: pérdida promedio =0.593, recompensa promedio=0.3, cota recompensa=0.4\n",
      "777: pérdida promedio =0.588, recompensa promedio=0.3, cota recompensa=0.4\n",
      "778: pérdida promedio =0.606, recompensa promedio=0.3, cota recompensa=0.5\n",
      "779: pérdida promedio =0.560, recompensa promedio=0.2, cota recompensa=0.0\n",
      "780: pérdida promedio =0.586, recompensa promedio=0.2, cota recompensa=0.0\n",
      "781: pérdida promedio =0.561, recompensa promedio=0.2, cota recompensa=0.1\n",
      "782: pérdida promedio =0.551, recompensa promedio=0.2, cota recompensa=0.2\n",
      "783: pérdida promedio =0.579, recompensa promedio=0.2, cota recompensa=0.2\n",
      "784: pérdida promedio =0.576, recompensa promedio=0.2, cota recompensa=0.2\n",
      "785: pérdida promedio =0.593, recompensa promedio=0.2, cota recompensa=0.3\n",
      "786: pérdida promedio =0.593, recompensa promedio=0.3, cota recompensa=0.3\n",
      "787: pérdida promedio =0.593, recompensa promedio=0.3, cota recompensa=0.2\n",
      "788: pérdida promedio =0.594, recompensa promedio=0.3, cota recompensa=0.3\n",
      "789: pérdida promedio =0.602, recompensa promedio=0.3, cota recompensa=0.3\n",
      "790: pérdida promedio =0.599, recompensa promedio=0.3, cota recompensa=0.3\n",
      "791: pérdida promedio =0.604, recompensa promedio=0.3, cota recompensa=0.3\n",
      "792: pérdida promedio =0.599, recompensa promedio=0.3, cota recompensa=0.3\n",
      "793: pérdida promedio =0.598, recompensa promedio=0.3, cota recompensa=0.4\n",
      "794: pérdida promedio =0.597, recompensa promedio=0.3, cota recompensa=0.2\n",
      "795: pérdida promedio =0.601, recompensa promedio=0.3, cota recompensa=0.2\n",
      "796: pérdida promedio =0.585, recompensa promedio=0.3, cota recompensa=0.3\n",
      "797: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.3\n",
      "798: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.3\n",
      "799: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.3\n",
      "800: pérdida promedio =0.593, recompensa promedio=0.3, cota recompensa=0.4\n",
      "801: pérdida promedio =0.591, recompensa promedio=0.3, cota recompensa=0.4\n",
      "802: pérdida promedio =0.592, recompensa promedio=0.3, cota recompensa=0.4\n",
      "803: pérdida promedio =0.585, recompensa promedio=0.3, cota recompensa=0.3\n",
      "804: pérdida promedio =0.586, recompensa promedio=0.3, cota recompensa=0.3\n",
      "805: pérdida promedio =0.585, recompensa promedio=0.3, cota recompensa=0.3\n",
      "806: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.4\n",
      "807: pérdida promedio =0.591, recompensa promedio=0.3, cota recompensa=0.2\n",
      "808: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.4\n",
      "809: pérdida promedio =0.583, recompensa promedio=0.3, cota recompensa=0.4\n",
      "810: pérdida promedio =0.587, recompensa promedio=0.3, cota recompensa=0.4\n",
      "811: pérdida promedio =0.591, recompensa promedio=0.3, cota recompensa=0.4\n",
      "812: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.3\n",
      "813: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.4\n",
      "814: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.3\n",
      "815: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.4\n",
      "816: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.4\n",
      "817: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.4\n",
      "818: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.4\n",
      "819: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.4\n",
      "820: pérdida promedio =0.590, recompensa promedio=0.3, cota recompensa=0.3\n",
      "821: pérdida promedio =0.602, recompensa promedio=0.3, cota recompensa=0.5\n",
      "822: pérdida promedio =0.581, recompensa promedio=0.3, cota recompensa=0.0\n",
      "823: pérdida promedio =0.594, recompensa promedio=0.3, cota recompensa=0.2\n",
      "824: pérdida promedio =0.596, recompensa promedio=0.3, cota recompensa=0.3\n",
      "825: pérdida promedio =0.604, recompensa promedio=0.3, cota recompensa=0.3\n",
      "826: pérdida promedio =0.595, recompensa promedio=0.3, cota recompensa=0.4\n",
      "827: pérdida promedio =0.591, recompensa promedio=0.3, cota recompensa=0.3\n",
      "828: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.3\n",
      "829: pérdida promedio =0.589, recompensa promedio=0.3, cota recompensa=0.3\n",
      "830: pérdida promedio =0.593, recompensa promedio=0.3, cota recompensa=0.4\n",
      "831: pérdida promedio =0.602, recompensa promedio=0.3, cota recompensa=0.4\n",
      "832: pérdida promedio =0.602, recompensa promedio=0.3, cota recompensa=0.3\n",
      "833: pérdida promedio =0.596, recompensa promedio=0.3, cota recompensa=0.3\n",
      "834: pérdida promedio =0.596, recompensa promedio=0.3, cota recompensa=0.4\n",
      "835: pérdida promedio =0.601, recompensa promedio=0.3, cota recompensa=0.4\n",
      "836: pérdida promedio =0.599, recompensa promedio=0.3, cota recompensa=0.3\n",
      "837: pérdida promedio =0.604, recompensa promedio=0.3, cota recompensa=0.5\n",
      "838: pérdida promedio =0.609, recompensa promedio=0.3, cota recompensa=0.3\n",
      "839: pérdida promedio =0.617, recompensa promedio=0.3, cota recompensa=0.3\n",
      "840: pérdida promedio =0.616, recompensa promedio=0.3, cota recompensa=0.4\n",
      "841: pérdida promedio =0.612, recompensa promedio=0.3, cota recompensa=0.4\n",
      "842: pérdida promedio =0.609, recompensa promedio=0.3, cota recompensa=0.2\n",
      "843: pérdida promedio =0.605, recompensa promedio=0.3, cota recompensa=0.3\n",
      "844: pérdida promedio =0.611, recompensa promedio=0.3, cota recompensa=0.4\n",
      "845: pérdida promedio =0.611, recompensa promedio=0.3, cota recompensa=0.4\n",
      "846: pérdida promedio =0.609, recompensa promedio=0.3, cota recompensa=0.4\n",
      "847: pérdida promedio =0.598, recompensa promedio=0.3, cota recompensa=0.5\n",
      "848: pérdida promedio =0.596, recompensa promedio=0.3, cota recompensa=0.3\n",
      "849: pérdida promedio =0.598, recompensa promedio=0.3, cota recompensa=0.4\n",
      "850: pérdida promedio =0.597, recompensa promedio=0.3, cota recompensa=0.4\n",
      "851: pérdida promedio =0.598, recompensa promedio=0.3, cota recompensa=0.5\n",
      "852: pérdida promedio =0.598, recompensa promedio=0.3, cota recompensa=0.4\n",
      "853: pérdida promedio =nan, recompensa promedio=0.3, cota recompensa=0.5\n",
      "854: pérdida promedio =0.514, recompensa promedio=0.0, cota recompensa=0.0\n",
      "855: pérdida promedio =0.469, recompensa promedio=0.1, cota recompensa=0.0\n",
      "856: pérdida promedio =0.444, recompensa promedio=0.1, cota recompensa=0.0\n",
      "857: pérdida promedio =0.443, recompensa promedio=0.1, cota recompensa=0.0\n",
      "858: pérdida promedio =0.452, recompensa promedio=0.1, cota recompensa=0.0\n",
      "859: pérdida promedio =0.441, recompensa promedio=0.1, cota recompensa=0.0\n",
      "860: pérdida promedio =0.435, recompensa promedio=0.1, cota recompensa=0.1\n",
      "861: pérdida promedio =0.432, recompensa promedio=0.1, cota recompensa=0.0\n",
      "862: pérdida promedio =0.443, recompensa promedio=0.1, cota recompensa=0.1\n",
      "863: pérdida promedio =0.445, recompensa promedio=0.1, cota recompensa=0.1\n",
      "864: pérdida promedio =0.477, recompensa promedio=0.2, cota recompensa=0.2\n",
      "865: pérdida promedio =0.488, recompensa promedio=0.2, cota recompensa=0.2\n",
      "866: pérdida promedio =0.483, recompensa promedio=0.2, cota recompensa=0.2\n",
      "867: pérdida promedio =0.491, recompensa promedio=0.2, cota recompensa=0.2\n",
      "868: pérdida promedio =0.494, recompensa promedio=0.2, cota recompensa=0.2\n",
      "869: pérdida promedio =0.509, recompensa promedio=0.2, cota recompensa=0.3\n",
      "870: pérdida promedio =0.539, recompensa promedio=0.2, cota recompensa=0.3\n",
      "871: pérdida promedio =0.532, recompensa promedio=0.2, cota recompensa=0.0\n",
      "872: pérdida promedio =0.516, recompensa promedio=0.2, cota recompensa=0.3\n",
      "873: pérdida promedio =0.521, recompensa promedio=0.2, cota recompensa=0.3\n",
      "874: pérdida promedio =0.506, recompensa promedio=0.2, cota recompensa=0.3\n",
      "875: pérdida promedio =0.491, recompensa promedio=0.2, cota recompensa=0.1\n",
      "876: pérdida promedio =0.506, recompensa promedio=0.2, cota recompensa=0.2\n",
      "877: pérdida promedio =0.505, recompensa promedio=0.2, cota recompensa=0.3\n",
      "878: pérdida promedio =0.498, recompensa promedio=0.2, cota recompensa=0.3\n",
      "879: pérdida promedio =0.494, recompensa promedio=0.2, cota recompensa=0.0\n",
      "880: pérdida promedio =0.481, recompensa promedio=0.2, cota recompensa=0.1\n",
      "881: pérdida promedio =0.478, recompensa promedio=0.2, cota recompensa=0.1\n",
      "882: pérdida promedio =0.489, recompensa promedio=0.2, cota recompensa=0.2\n",
      "883: pérdida promedio =0.501, recompensa promedio=0.2, cota recompensa=0.3\n",
      "884: pérdida promedio =0.497, recompensa promedio=0.2, cota recompensa=0.3\n",
      "885: pérdida promedio =0.498, recompensa promedio=0.2, cota recompensa=0.3\n",
      "886: pérdida promedio =0.491, recompensa promedio=0.2, cota recompensa=0.3\n",
      "887: pérdida promedio =0.495, recompensa promedio=0.2, cota recompensa=0.3\n",
      "888: pérdida promedio =0.487, recompensa promedio=0.2, cota recompensa=0.4\n",
      "889: pérdida promedio =0.469, recompensa promedio=0.2, cota recompensa=0.0\n",
      "890: pérdida promedio =0.495, recompensa promedio=0.2, cota recompensa=0.2\n",
      "891: pérdida promedio =0.495, recompensa promedio=0.2, cota recompensa=0.3\n",
      "892: pérdida promedio =0.482, recompensa promedio=0.2, cota recompensa=0.1\n",
      "893: pérdida promedio =0.488, recompensa promedio=0.2, cota recompensa=0.2\n",
      "894: pérdida promedio =0.492, recompensa promedio=0.2, cota recompensa=0.3\n",
      "895: pérdida promedio =0.489, recompensa promedio=0.2, cota recompensa=0.3\n",
      "896: pérdida promedio =0.482, recompensa promedio=0.2, cota recompensa=0.3\n",
      "897: pérdida promedio =0.488, recompensa promedio=0.2, cota recompensa=0.3\n",
      "898: pérdida promedio =0.486, recompensa promedio=0.2, cota recompensa=0.3\n",
      "899: pérdida promedio =0.487, recompensa promedio=0.2, cota recompensa=0.4\n",
      "900: pérdida promedio =0.480, recompensa promedio=0.2, cota recompensa=0.1\n",
      "901: pérdida promedio =0.476, recompensa promedio=0.2, cota recompensa=0.3\n",
      "902: pérdida promedio =0.476, recompensa promedio=0.2, cota recompensa=0.3\n",
      "903: pérdida promedio =0.478, recompensa promedio=0.2, cota recompensa=0.3\n",
      "904: pérdida promedio =0.477, recompensa promedio=0.2, cota recompensa=0.3\n",
      "905: pérdida promedio =0.482, recompensa promedio=0.2, cota recompensa=0.3\n",
      "906: pérdida promedio =0.478, recompensa promedio=0.2, cota recompensa=0.4\n",
      "907: pérdida promedio =0.490, recompensa promedio=0.2, cota recompensa=0.4\n",
      "908: pérdida promedio =0.439, recompensa promedio=0.2, cota recompensa=0.0\n",
      "909: pérdida promedio =0.400, recompensa promedio=0.2, cota recompensa=0.0\n",
      "910: pérdida promedio =0.402, recompensa promedio=0.2, cota recompensa=0.0\n",
      "911: pérdida promedio =0.411, recompensa promedio=0.2, cota recompensa=0.1\n",
      "912: pérdida promedio =0.435, recompensa promedio=0.2, cota recompensa=0.1\n",
      "913: pérdida promedio =0.442, recompensa promedio=0.2, cota recompensa=0.2\n",
      "914: pérdida promedio =0.438, recompensa promedio=0.2, cota recompensa=0.2\n",
      "915: pérdida promedio =0.464, recompensa promedio=0.2, cota recompensa=0.2\n",
      "916: pérdida promedio =0.480, recompensa promedio=0.2, cota recompensa=0.3\n",
      "917: pérdida promedio =0.477, recompensa promedio=0.2, cota recompensa=0.3\n",
      "918: pérdida promedio =0.498, recompensa promedio=0.2, cota recompensa=0.3\n",
      "919: pérdida promedio =0.496, recompensa promedio=0.2, cota recompensa=0.3\n",
      "920: pérdida promedio =0.496, recompensa promedio=0.2, cota recompensa=0.3\n",
      "921: pérdida promedio =0.488, recompensa promedio=0.2, cota recompensa=0.2\n",
      "922: pérdida promedio =0.496, recompensa promedio=0.2, cota recompensa=0.3\n",
      "923: pérdida promedio =0.497, recompensa promedio=0.3, cota recompensa=0.3\n",
      "924: pérdida promedio =0.480, recompensa promedio=0.3, cota recompensa=0.4\n",
      "925: pérdida promedio =0.484, recompensa promedio=0.2, cota recompensa=0.3\n",
      "926: pérdida promedio =0.478, recompensa promedio=0.2, cota recompensa=0.3\n",
      "927: pérdida promedio =0.474, recompensa promedio=0.3, cota recompensa=0.4\n",
      "928: pérdida promedio =0.485, recompensa promedio=0.3, cota recompensa=0.4\n",
      "929: pérdida promedio =0.465, recompensa promedio=0.2, cota recompensa=0.0\n",
      "930: pérdida promedio =0.481, recompensa promedio=0.2, cota recompensa=0.2\n",
      "931: pérdida promedio =0.477, recompensa promedio=0.2, cota recompensa=0.2\n",
      "932: pérdida promedio =0.474, recompensa promedio=0.3, cota recompensa=0.3\n",
      "933: pérdida promedio =0.477, recompensa promedio=0.3, cota recompensa=0.4\n",
      "934: pérdida promedio =0.477, recompensa promedio=0.3, cota recompensa=0.4\n",
      "935: pérdida promedio =0.467, recompensa promedio=0.3, cota recompensa=0.3\n",
      "936: pérdida promedio =0.469, recompensa promedio=0.3, cota recompensa=0.4\n",
      "937: pérdida promedio =0.477, recompensa promedio=0.3, cota recompensa=0.4\n",
      "938: pérdida promedio =0.476, recompensa promedio=0.3, cota recompensa=0.5\n",
      "939: pérdida promedio =0.383, recompensa promedio=0.2, cota recompensa=0.0\n",
      "940: pérdida promedio =0.379, recompensa promedio=0.2, cota recompensa=0.0\n",
      "941: pérdida promedio =0.365, recompensa promedio=0.2, cota recompensa=0.0\n",
      "942: pérdida promedio =0.367, recompensa promedio=0.2, cota recompensa=0.0\n",
      "943: pérdida promedio =0.394, recompensa promedio=0.2, cota recompensa=0.1\n",
      "944: pérdida promedio =0.401, recompensa promedio=0.2, cota recompensa=0.1\n",
      "945: pérdida promedio =0.410, recompensa promedio=0.2, cota recompensa=0.1\n",
      "946: pérdida promedio =0.409, recompensa promedio=0.2, cota recompensa=0.2\n",
      "947: pérdida promedio =0.401, recompensa promedio=0.2, cota recompensa=0.2\n",
      "948: pérdida promedio =0.396, recompensa promedio=0.2, cota recompensa=0.2\n",
      "949: pérdida promedio =0.394, recompensa promedio=0.2, cota recompensa=0.3\n",
      "950: pérdida promedio =0.401, recompensa promedio=0.2, cota recompensa=0.3\n",
      "951: pérdida promedio =0.391, recompensa promedio=0.2, cota recompensa=0.2\n",
      "952: pérdida promedio =0.383, recompensa promedio=0.2, cota recompensa=0.3\n",
      "953: pérdida promedio =0.388, recompensa promedio=0.2, cota recompensa=0.3\n",
      "954: pérdida promedio =0.380, recompensa promedio=0.2, cota recompensa=0.1\n",
      "955: pérdida promedio =0.405, recompensa promedio=0.2, cota recompensa=0.3\n",
      "956: pérdida promedio =0.396, recompensa promedio=0.2, cota recompensa=0.3\n",
      "957: pérdida promedio =0.390, recompensa promedio=0.2, cota recompensa=0.3\n",
      "958: pérdida promedio =0.386, recompensa promedio=0.2, cota recompensa=0.2\n",
      "959: pérdida promedio =0.400, recompensa promedio=0.3, cota recompensa=0.3\n",
      "960: pérdida promedio =0.398, recompensa promedio=0.2, cota recompensa=0.2\n",
      "961: pérdida promedio =0.402, recompensa promedio=0.2, cota recompensa=0.3\n",
      "962: pérdida promedio =0.416, recompensa promedio=0.3, cota recompensa=0.4\n",
      "963: pérdida promedio =0.418, recompensa promedio=0.2, cota recompensa=0.2\n",
      "964: pérdida promedio =0.422, recompensa promedio=0.2, cota recompensa=0.3\n",
      "965: pérdida promedio =0.409, recompensa promedio=0.2, cota recompensa=0.2\n",
      "966: pérdida promedio =0.432, recompensa promedio=0.3, cota recompensa=0.3\n",
      "967: pérdida promedio =0.423, recompensa promedio=0.2, cota recompensa=0.4\n",
      "968: pérdida promedio =0.420, recompensa promedio=0.3, cota recompensa=0.4\n",
      "969: pérdida promedio =0.436, recompensa promedio=0.3, cota recompensa=0.4\n",
      "970: pérdida promedio =0.403, recompensa promedio=0.2, cota recompensa=0.0\n",
      "971: pérdida promedio =0.371, recompensa promedio=0.2, cota recompensa=0.0\n",
      "972: pérdida promedio =0.382, recompensa promedio=0.2, cota recompensa=0.1\n",
      "973: pérdida promedio =0.396, recompensa promedio=0.2, cota recompensa=0.2\n",
      "974: pérdida promedio =0.407, recompensa promedio=0.2, cota recompensa=0.2\n",
      "975: pérdida promedio =0.418, recompensa promedio=0.2, cota recompensa=0.3\n",
      "976: pérdida promedio =0.423, recompensa promedio=0.2, cota recompensa=0.3\n",
      "977: pérdida promedio =0.414, recompensa promedio=0.2, cota recompensa=0.0\n",
      "978: pérdida promedio =0.425, recompensa promedio=0.2, cota recompensa=0.2\n",
      "979: pérdida promedio =0.417, recompensa promedio=0.2, cota recompensa=0.2\n",
      "980: pérdida promedio =0.411, recompensa promedio=0.2, cota recompensa=0.3\n",
      "981: pérdida promedio =0.414, recompensa promedio=0.3, cota recompensa=0.3\n",
      "982: pérdida promedio =0.417, recompensa promedio=0.3, cota recompensa=0.3\n",
      "983: pérdida promedio =0.409, recompensa promedio=0.3, cota recompensa=0.3\n",
      "984: pérdida promedio =0.403, recompensa promedio=0.3, cota recompensa=0.3\n",
      "985: pérdida promedio =0.413, recompensa promedio=0.3, cota recompensa=0.3\n",
      "986: pérdida promedio =0.415, recompensa promedio=0.3, cota recompensa=0.3\n",
      "987: pérdida promedio =0.430, recompensa promedio=0.3, cota recompensa=0.4\n",
      "988: pérdida promedio =0.420, recompensa promedio=0.3, cota recompensa=0.2\n",
      "989: pérdida promedio =0.432, recompensa promedio=0.3, cota recompensa=0.3\n",
      "990: pérdida promedio =0.427, recompensa promedio=0.3, cota recompensa=0.3\n",
      "991: pérdida promedio =0.425, recompensa promedio=0.3, cota recompensa=0.3\n",
      "992: pérdida promedio =0.423, recompensa promedio=0.3, cota recompensa=0.4\n",
      "993: pérdida promedio =0.420, recompensa promedio=0.3, cota recompensa=0.3\n",
      "994: pérdida promedio =0.419, recompensa promedio=0.3, cota recompensa=0.4\n",
      "995: pérdida promedio =0.416, recompensa promedio=0.3, cota recompensa=0.3\n",
      "996: pérdida promedio =0.407, recompensa promedio=0.3, cota recompensa=0.3\n",
      "997: pérdida promedio =0.409, recompensa promedio=0.3, cota recompensa=0.3\n",
      "998: pérdida promedio =0.412, recompensa promedio=0.3, cota recompensa=0.3\n",
      "999: pérdida promedio =0.409, recompensa promedio=0.3, cota recompensa=0.4\n",
      "1000: pérdida promedio =0.413, recompensa promedio=0.3, cota recompensa=0.4\n",
      "Terminado por máximo número de iteraciones. No resuelto\n"
     ]
    }
   ],
   "source": [
    "from torch.utils.tensorboard import SummaryWriter\n",
    "# indica donde se escribirá el log de tensorboard\n",
    "tensorboard --logdir=runs\n",
    "\n",
    "# selecciona un ambiente CartPole\n",
    "env = DiscreteOneHotWrapper(gym.make(\"FrozenLake-v1\"))\n",
    "\n",
    "# extrae tamaños de acciones y observaciones en el ambiente\n",
    "obs_size = env.observation_space.shape[0]\n",
    "n_actions = env.action_space.n\n",
    "# define tamaño capa oculta de la red\n",
    "HIDDEN_SIZE = 128\n",
    "#instancia un objeto Net\n",
    "net = Net(obs_size, HIDDEN_SIZE, n_actions)\n",
    "\n",
    "# define tamaño de los lotes para el iterador de lotes\n",
    "BATCH_SIZE = 100\n",
    "# instancia un iterador Batch\n",
    "batch = Batch(env, net, BATCH_SIZE)\n",
    "\n",
    "# define el percentil para los episodios élite\n",
    "PERCENTILE = 50\n",
    "\n",
    "# Instancia un agente\n",
    "agent = Agent2(batch, PERCENTILE)\n",
    "\n",
    "# instancia writer\n",
    "writer = SummaryWriter(comment=\"Entrenamiento de CartPole\")\n",
    "\n",
    "trainer = Trainer(model=net, writer=writer)\n",
    "\n",
    "# ciclo de entrenamiento\n",
    "min_reward = 0.8 # Para FrozenLake\n",
    "max_iterations = 1000\n",
    "done = False\n",
    " \n",
    "iter_no = 0\n",
    "\n",
    "while not done:\n",
    "    iter_no += 1\n",
    "    # pide datos al agente\n",
    "    dataloader, reward_bound, reward_mean = agent.take_action()\n",
    "    # hace un paso de entrenamiento de la red\n",
    "      \n",
    "    trainer.fit(iter_no, dataloader, reward_bound, reward_mean)\n",
    "    #trainer.save_checkpoint()\n",
    "    #validation = trainer.validate(dataloaders=dataloader)\n",
    "    if reward_mean > min_reward:\n",
    "            print(\"Resuelto!\")\n",
    "            done = True\n",
    "    if iter_no == max_iterations:\n",
    "            print(\"Terminado por máximo número de iteraciones. No resuelto\")\n",
    "            done = True\n",
    "\n",
    "# envia al write cualquier cálculo pendiente\n",
    "writer.flush()\n",
    "# cierra el writer\n",
    "close(writer)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}