OxML2020_RL2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "OxML2020_RL2.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPYf670QKmD2DXsoTIh49ei",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/OxML2020/practicals/blob/master/OxML2020_RL2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TqkRbG3SA1wn",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "#OxML 2020: Reinforcement Learning Part 2\n",
        "\n",
        "**Temporal-Difference Learning and Q-Learning**\n",
        "\n",
        "**content creator:** Fatemeh Abdollahi\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xai8BqPuNGBi",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Setup\n",
        "\n",
        "from __future__ import print_function\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NHv7Ua29BOZ5",
        "colab_type": "text"
      },
      "source": [
        "#Section 1: Introduction to TD Learning\n",
        "\n",
        "__MDPs:__\n",
        "\n",
        "Finite Markov decision processes (MDPs) are a classical formalization of sequential decision making, where actions influence not just immediate rewards, but also subsequent situations, or states, and future rewards.\n",
        "Thus MDPs involve delayed reward and the need to tradeoff immediate and delayed rewards. Whereas in bandit problems we estimated the value $q_{*}(a)$ of each action $a$, in MDPs we estimate the value $q_{*}(s, a)$ of each action $a$\n",
        "in each state $s$, or we estimate the value $v_{*}(s)$ of each state given optimal action selections.\n",
        "These state-dependent quantities are essential to accurately assigning credit for long-term consequences to individual action selections.\n",
        "MDPs are a mathematically idealized form of the reinforcement learning problem\n",
        "for which precise theoretical statements can be made.\n",
        "\n",
        "__Environment:__\n",
        "\n",
        "- In a finite MDP, the sets of states, actions, and rewards ($S, A$, and $R$) all have a finite number of elements. In this case, the random variables $R_{t}$ and $S_{t}$ have well defined discrete probability distributions dependent only on the preceding state and action. That is, for particular values of these random variables, $s' \\in S$ and $r \\in R$, there is a probability of those values occurring at time $t$, given particular values of the preceding state and action:\n",
        "\\begin{align}\n",
        "p(s', r|s, a) \\doteq \\Pr [ S_{t}=s', R_{t}=r \\;|\\; S_{t−1}=s, A_{t−1}=a]\n",
        "\\end{align}\n",
        "for all $s', s \\in S, r \\in R$, and $a \\in A(s)$. The function $p$ defines the dynamics of the MDP.\n",
        "- The agent experiences the environment in episodes or trials.\n",
        "- The agent's goal is to learn to predict expected rewards from each state in the trial. \n",
        "\n",
        "\n",
        "**Key Elements**\n",
        "\n",
        "* Return $G_{t}$: future cumulative reward, which can be written in arecursive form\n",
        "\\begin{align}\n",
        "G_{t} &= \\sum \\limits_{k = 0}^{\\infty} \\gamma^{k} R_{t+k+1} \\\\\n",
        "&= R_{t+1} + \\gamma G_{t+1}\n",
        "\\end{align}\n",
        "where $\\gamma$ is discount factor that controls the importance of future rewards, and $\\gamma \\in [0, 1]$. $\\gamma$ may also be interpreted as probability of continuing the trajectory.\n",
        "* Value funtion $V_{\\pi}(s_t=s)$: expecation of the return for a state $s$ under policy $\\pi$\n",
        "\\begin{align}\n",
        "V_{\\pi}(s_t=s) &= \\mathbb{E} [ G_{t}\\; | \\; s_t=s, a_{t:\\infty}\\sim\\pi] \\\\\n",
        "& = \\mathbb{E} [ R_{t+1} + \\gamma G_{t+1}\\; | \\; s_t=s, a_{t:\\infty}\\sim\\pi]\n",
        "\\end{align}\n",
        "With an assumption of **Markov process**, we thus have:\n",
        "\\begin{align}\n",
        "V_{\\pi}(s_t=s) &= \\mathbb{E} [ r_{t+1} + \\gamma V_{\\pi}(s_{t+1})\\; | \\; s_t=s, a_{t:\\infty}\\sim\\pi] \\\\\n",
        "&= \\sum_a \\pi(a|s) \\sum_{r, s'}p(s', r | s, a)[r + V_{\\pi}(s_{t+1}=s')]\n",
        "\\end{align}\n",
        "where $\\pi(a|s)$ is the probability of taking action $a$ in state $s$ under policy $\\pi$.\n",
        "\n",
        "**Temporal difference (TD) learning**\n",
        "\n",
        "* With a Markovian assumption, we can use $V(s_{t+1})$ as an imperfect proxy for the true value $G_{t+1}$ (Monte Carlo bootstrapping), and thus obtain the generalised equation to calculate TD-error:\n",
        "\\begin{align}\n",
        "\\delta_{t} = R_{t+1} + \\gamma V(S_{t+1}) - V(S_{t})\n",
        "\\end{align}\n",
        "\n",
        "* Value updated by using the learning rate constant $\\alpha$:\n",
        "\\begin{align}\n",
        "V(S_{t}) \\leftarrow V(S_{t}) + \\alpha \\delta_{t}\n",
        "\\end{align}\n",
        "\n",
        "\n",
        "\n",
        "__Definitions:__\n",
        "\n",
        "* TD-error:\n",
        "\\begin{align}\n",
        "\\delta_{t} = R_{t+1} + \\gamma V(S_{t+1}) - V(S_{t})\n",
        "\\end{align}\n",
        "\n",
        "* Value updates:\n",
        "\\begin{align}\n",
        "V(S_{t}) \\leftarrow V(S_{t}) + \\alpha \\delta_{t}\n",
        "\\end{align}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zXPyyuw67sAH",
        "colab_type": "text"
      },
      "source": [
        "##Exercise 1: Implementing TD Algorithm\n",
        "\n",
        "__Problem Definition:__\n",
        "\n",
        "This is an example found in the book Reinforcement Learning: An Introduction by Sutton and Barto. The basic idea is that you always start in state ‘C’, and you move randomly, with a 50% probability, to either the left or right until you reach the terminal or ending states ‘A’ or ‘E’. If you end in state ‘A’, you get a reward of 0, but if you end in state ‘E’, the reward is 1. There are no rewards for states ‘B’ through ‘D’.\n",
        "\n",
        "![random walk.JPG](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQEAeAB4AAD/4RDSRXhpZgAATU0AKgAAAAgABAE7AAIAAAAERkFCAIdpAAQAAAABAAAISpydAAEAAAAIAAAQwuocAAcAAAgMAAAAPgAAAAAc6gAAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFkAMAAgAAABQAABCYkAQAAgAAABQAABCskpEAAgAAAAM2NQAAkpIAAgAAAAM2NQAA6hwABwAACAwAAAiMAAAAABzqAAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMjAyMDowODoxOSAxMjozOTozNQAyMDIwOjA4OjE5IDEyOjM5OjM1AAAARgBBAEIAAAD/4QsWaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49J++7vycgaWQ9J1c1TTBNcENlaGlIenJlU3pOVGN6a2M5ZCc/Pg0KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyI+PHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj48cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0idXVpZDpmYWY1YmRkNS1iYTNkLTExZGEtYWQzMS1kMzNkNzUxODJmMWIiIHhtbG5zOmRjPSJodHRwOi8vcHVybC5vcmcvZGMvZWxlbWVudHMvMS4xLyIvPjxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSJ1dWlkOmZhZjViZGQ1LWJhM2QtMTFkYS1hZDMxLWQzM2Q3NTE4MmYxYiIgeG1sbnM6eG1wPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvIj48eG1wOkNyZWF0ZURhdGU+MjAyMC0wOC0xOVQxMjozOTozNS42NTI8L3htcDpDcmVhdGVEYXRlPjwvcmRmOkRlc2NyaXB0aW9uPjxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSJ1dWlkOmZhZjViZGQ1LWJhM2QtMTFkYS1hZDMxLWQzM2Q3NTE4MmYxYiIgeG1sbnM6ZGM9Imh0dHA6Ly9wdXJsLm9yZy9kYy9lbGVtZW50cy8xLjEvIj48ZGM6Y3JlYXRvcj48cmRmOlNlcSB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPjxyZGY6bGk+RkFCPC9yZGY6bGk+PC9yZGY6U2VxPg0KCQkJPC9kYzpjcmVhdG9yPjwvcmRmOkRlc2NyaXB0aW9uPjwvcmRmOlJERj48L3g6eG1wbWV0YT4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgPD94cGFja2V0IGVuZD0ndyc/Pv/bAEMABwUFBgUEBwYFBggHBwgKEQsKCQkKFQ8QDBEYFRoZGBUYFxseJyEbHSUdFxgiLiIlKCkrLCsaIC8zLyoyJyorKv/bAEMBBwgICgkKFAsLFCocGBwqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKioqKv/AABEIAJkEKAMBIgACEQEDEQH/xAAfAAABBQEBAQEBAQAAAAAAAAAAAQIDBAUGBwgJCgv/xAC1EAACAQMDAgQDBQUEBAAAAX0BAgMABBEFEiExQQYTUWEHInEUMoGRoQgjQrHBFVLR8CQzYnKCCQoWFxgZGiUmJygpKjQ1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4eLj5OXm5+jp6vHy8/T19vf4+fr/xAAfAQADAQEBAQEBAQEBAAAAAAAAAQIDBAUGBwgJCgv/xAC1EQACAQIEBAMEBwUEBAABAncAAQIDEQQFITEGEkFRB2FxEyIygQgUQpGhscEJIzNS8BVictEKFiQ04SXxFxgZGiYnKCkqNTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqCg4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2dri4+Tl5ufo6ery8/T19vf4+fr/2gAMAwEAAhEDEQA/APoKiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiisnxXq7aB4R1XVUAL2dpJKgPQsFOP1xSbsrjSu7E0+v6Pa6pHptzq1jDfyY2Wslyiyt9EJyfyrQrg/A2g2mo/CC1h1GJLmTWLQ3N5JIu5ppJAWLMepIyMHtgY6VP8I9dudf+G1hPqEjTXVuz2skjHJfy2IBJ7nGK0cbNrqiFK6T6M7WiiioKCiiigAorO1rXbPQreN7sySSzv5dvbQJvluH/uoo5J7nsBySBzVGHS/E+uL5mp3w8P2zfdtLALLcYz/HK4KqSMcIpxzhjQBv0Vkx+AdEWRpbh9Tu5GzlrnVblx+CmTaPwAph8CW1uGOjaxrWmyEggjUHuVGP9icuuPoBQLU2aK5ufVdZ8MHd4niivdNB51WxjK+SM9ZockqOmXUkdSQoroopY54UlgkWSORQyOjZVgehBHUUDHUUUUAFFFFABRRRQBDeXtrp1o91qFzDa28Yy808gRFHqSeBTNP1Kx1a0F1pd7b3tuxIE1tKsiEjr8ykiuDuLw678fItJudslnommG7SFuR9odgA+OmQp4z0yadqt3/wj/xx0RLUCOHxDZzQ3SKMB5IhuSQju2Plz6U4q9vO/wCF/wDIlu1/K342/wAz0OiiikUFFFFABRRRQAUUVz9x4hutQ1CbTPCdql9cwNtubyZitrat3VmHLvjnYvT+IrkUAdBRWGngu5vVLeIvEep3rOBmGzkNjCvrtEREmD/tO1Sv8PtAeJUCahGV6SR6tdLJ+LiTJ/EmmLU16KxJfDOtab+98Pa9NOqj/jy1b99G30lAEin3Jf6U7SvEIvL5tM1O1fTNWjUu1pKwYSIDjfE44kTpyMEZG4A8UhmzRRRQAUUUUAFFFFABWfY6/o+p3ctrpurWN3cQcSw29ykjx9vmUHI/GuP+L+qz2nh3TdJtpWhbW9ShsZHU4IiY/OAevIGPoTSfFe3i0P4eJq2kQxWtzoM0M1m0ahfLAdVKDH8JUkEdCKa7vvb8v8w1vZb2v/X3HoNFQ2Vyt5YW90gws0SyAexGf61NQ1Z2Yk01dBRRRSGFFFFABRRVHVtZstEsxcX8jAO4jijRS8kznoiKOWY+goAvUVz0Nj4p1/Et5cDw1ZNytvbhJrthx992BjQ9flUP1+9VtPAOjGXzbuXVL2U9WuNUuGH/AHwHCj8AKBGtRWM3gOyh3tpGqa1pkrDho9RkmVT7RzF0/DHSqs99r3hhTJr6Jq2mKfm1CyhKSwL/AHpYcnIHdk+u0DmgDo6KjtrmC8tY7m0mSeCVQ0ckbBlYHoQR1qSgYUUUUAFFFFABVe+1Cz0y0a61K7gs7dPvTXEgjRfqTwKsV51bXf8AwkHx6vbS6xJa+HtPQ28TDIWeUgmQD+8FO3PbmmtXYTdlc72xv7PU7RLrTbuC7t3+7NbyCRG+jDg1YrzyW6/sH48Wtna4jtfEGnO88SjAM8RJEh9yvGe/Gegr0Ojon3/zsF9Wu3+VwooopDCiiigAooooAKK5xtev9cuJbXwbbw3CROY5tUuiRbRMOCEA5mYHghSFHILAjFWR4IN4mdf1/V9QctuKwXLWUa+wWEqcf7zMfemI2qKyH+H+gtEiRjUoCgwJINWukf6lhJlj9c1DN4f8Q6Vul0DW21BBz9h1cBg3HRZ0AZfqwekBu0VkaN4hg1Weayngl0/VLYA3Fhc4EiA9GBBIdDzhlJHbggitegYUUUUAFFFFABRRXHfFfxBceGvhpqt/YuY7oosMTg8ozsF3D3AJNJ+Q1qzoY9f0eXVW0uLVrF9QT71otyhlX6pnP6VoVwniDw5Z2nwYnsrWJInsNP8AtMEqjDJOibxKD13bhnPXk1v+C9Zk8Q+CNI1Wf/XXVqjyY/v4w36g1bW/l/X6EJ3s+/8AwP8AM3KKKKkoKKKKACiiigAorN1nXrTRIYvtHmTXFw2y2tIF3zXD+ir/ADJwAOSQKpRaT4n1vEuq6iNBtm5FnpwWSfGf45nBUcdQi8dmNAG/RWTH4B0RXZ521K7kbq1zqtzJ+QMmB+AFRjwJbWik6LrOtadJkEE6hJdLx/sTlxj1xg+/SgWptUVzU+sav4XOfFkUNxpoOP7XskKrFz1miOTGOmXBZepO0V0iSJLGskTK6MAyspyCD0INAxaKKKACiiigAqC7vrTT4DPf3UNtEoJMk0gRQByeTU9ef/Gy1t3+FWs3DwRNOqRBZSgLAeavGevc/nR1Q0rux36sGUMpBBGQQetLUFj/AMg+3/65L/IVPTas7Exd0mFFFFIYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXH/EfVdT0vTtJGj38lhJdaj5EksUcbts8iZ8AOrD7yLziuwrhPin/AMeXh/8A7C3/ALaXFNbiexy/9ueKf+hs1D/wGs//AIxR/bnin/obNQ/8BrP/AOMVXoqhWLH9ueKf+hs1D/wGs/8A4xR/bnin/obNQ/8AAaz/APjFV6KAsej/AA/1K+1bwZb3Wq3TXd19ouommdFUuI7iRFyFAXO1R0ArpK5L4Yf8iFB/1+33/pZNXW1L3GtgoqlqWs6Zo1uZ9X1G1sYv79zMsY/Mmuc034qeEdZ8UW/h/R9U+3X1wWC+TExjG1C5+cgA8KemaLMLo7CiiikMKzvEGkrr3hvUdKkbYt7bPBu/u7lIzWjRSaurDTs7nnvhXxNZ+HvAVromsSLa65ptqbY6aT+/nZAQpiT70gYAEFQRz7Gtb4ZeG5/CvgCw0++ULeNunuFBztd2LFfwyB+FdZRVuTbb6shRskuiCiiipKCqmq6lb6PpVzqF6zCC3jLvtXcxx2AHUk8Adyat1z2uINU8V+H9Gdv3XmSalcRlciRINoQH0xLJE3/AKALPhbQJ1nbxD4hTdrd4mBGTlbCE8iBP03N/E3sAB09FFABRXz5448ceJIviBd+JdGvroeGvDN/b2F1bRyssdwzZ80lQcNg/Lz6rX0BDNHcW8c0LB45FDow6EEZBp291S/rv+KFf3uX+v6Q8gEEEZB6g1xK2v/CFeIILS348P6rKUgj5xY3Jydg9I5OcD+FuB94AdtWV4m0ca/4ZvtNzskmiPkyd4pR80bj3VwrD3FIZNRWb4d1Q634Z07U2jMT3dtHK8ZGCjFQSp+hyK0qACiuf1rx74V8PbxrGv2NvIg3ND5oeTH+4uW/SuJm/aD8LS6xa6do9tfX8lzMkSzeWIowWYL/Ed3f+7Ts2JySPVqKKKQzgNVs18M/FuLxXd/JpV/pxsrq5OdtrIrBlZz0VSBjJ4BHPWhraLxf8VtI1vTZEutJ0SzlK3kTBopp5flCow4bCgkkE4OBXf0U07W8r/jf/ADE1e/nb8Lf5BRRRSGFFeZav8dPD3h7xhe6BrdnfQtaS+WbmJVkQ/KDkjIYdewNdLovxL8HeINg0zxDZNJIcLDM/kyE+gV8E/lTsxcyOoooBDKCpyDyCO9FIZz2u3F1qmrQeGdJmaCW4j8+/uYzhra2zj5T2dyCqnth2/hrp9O0600jToLDTbdLe1gXZHEg4Uf4+/euf8DR/abXUtclUiXVb6RlJbOIYmMUQHoCqb8ern1rqaBBRXifhrw/f+OPGnjT7f4y8VWEWnas0FvDp2qNFGqEk42kHGMcYxTvG+n678JdOtfEui+MNc1a2S6jiu7DWroXKyoxP3SQNp+nPfPFUle3nb8dhNu7str/ge1Vk+IfD9v4h08RSu9vdQt5tpeRY822kHR1P6EdGBIOQa1I38yJHAI3KDg9qdU7D0aOZ8OatPqNnPb6lGsOp2Extr2Nfu7wAQ6/7LqVYezY6g1r1hasg0z4haZeIQqavbyWUw/vyxgyxH8FE4/L0rdoGFFQ3V5bWMBmvbiK3iXrJK4RR+JridZ+NPgTRsq+tx3smMhLFDNn/AIEPl/WnZsTaW53lFcP8PvijYfEW81KLS7C5tYrFY233DLufeW/hXOMbfXvXcUWsCaauji/ihoF1rPh6yvNOga5u9Hv4dQjt1+9MEPzIPcgnH0rN8d6lY+PPCDeHfDN5Fe3mozwxzRxNua0QOGdpR1jwFPDYJPA5r0aihaaPvf8AL/Ibvuv6/q5HbQJa2sVvEMJEgRR7AYFSUUUm76iSsrIKKwvGviiPwZ4RvNemtmuo7Qx7oUfaWDyKnB9t2fwrjtF/aA8EartW7ubnS5WONt3ASuf95Nwx7nFOzYOSTsz06is/SvEGj65F5mjapZ3y9zbzq+Prg8VoUhkF9e2+m6fcX17KsNtbRtLLI3RVUZJ/IVl+FtKuL2YeJtfhK39yv+iWsmD/AGfAeiD0kYYLn1+XOBUXiWL+09Q0TRCMw3l4JrkZxmGEeYR7guI1I7hjXX0CCiob28t9OsZ7y+mWC3t0MksjnARQMkmvN7Dxx438bxyXngDQtNs9JV9sN/r8kg+1AZBKRx8gZ7k4/HgC12Bu256dRXnNj8Qtb0LxRY+HviRpdnYzaidlnqenSs1rNJn7hD/Mh5A5zye1ejU7aXBO+hxN1ar4J12O4tF8vQNVuBHPCPu2dy5wsi/3UkYhSBwHZT/E1dPTtZ0uDW9EvNMuwfJu4WiYg4IyMZB7EdQfUVi+E9Rn1Twnp9zelGu/K8q6KHI85CUkx/wNWoA2KKKw9Z8a+GvD+RrOuWNo4GfLeYFz/wAAHzH8qQzcoryPWf2jfCNhldKgvtUfHDJH5Sfm+G/8dr1ym01uJNPYK8+ubePwl8W7rxFflYdJ1mxWGW8ckJbzxkY3t0VWXoTgZGO9eg0Ulo7jeqscDbWieKPi3beI7Jln0nSdOaGG6Q5SeaQnOxujAL1IJGTj1rvqKKfRL+u4rat/1tYKKK818RfG/QvCvja98O61Y3qG1MY+1QhXU741fJGQRjdjjPShJvYG0tz0qiuT0X4oeC9f2DT/ABDZiRzhYrhvJcn0Cvgn8K6tXV1DIwZSMgg5BpDvcWuc1X7R4i14eHLKaSC0hRZ9VuIyQ3ltnZArdmfBLEchR2LAjomYIhZzhVGSfQVkfD+ISeFY9XbmfW5G1KRiMErJzGv/AAGPy1/4DQI6K2toLK1jtrSFIIIlCRxRqFVFHQADoKlory/xD428av8AFmTwd4Ph0HCWC3fm6ok3rgjMbe4xx+NC1dgbsrnqFFeaXmtfFrQLWTUdU0bwzrNpCpaW20maeOfA6keZkHjPAGa7Lwp4o07xj4btta0dy1vOOUfho2HVGHYg07aXFza2G+JvDo1y1jmtJRaatZkyWN6BkxP6MB95G6Mvce4BEGgax/bWlieSE291DI0F3bt1hmQ4Zfcdwe4IPeuirkpEGl/EmRI1Kxa3YmdsAbfOgZUJ+rJIg+kVIZvUUVXvdQstNt2uNRu4LSFRlpZ5QigfUnFAyxRXn+tfG/wJo29f7X/tCVRny7GMy7vYNwn/AI9Wn8P/AIg2fxC0+9vdOsp7WG1nEI89huc7Qc4HA6+pp2e4uZXsdbXMfEbw1L4t8A6npFtj7TLGHgycAuhDKPxIx+NdPRUspOzPPdX8VWmsfDm60qwYN4gurD7L/ZGf9JildQhDR9VUE8sRjHOa6zwtoo8O+E9M0gMGNnbJEzDozAcn881rUVble/n/AF+pCVrLt/X6BRRXPeN/F0HgjwxJrd1ayXUMUqI0cbANhjjIzxUlbHQ0V5vovx58DavsWe+m0yVzjZewlQP+BLuUfiRXd6brOmazbifSNRtb6L+/bTLIPzBptNCTT2LtU9X1S30XR7nUbzcYrdC5VFyznoFUd2JwAO5Iq5XPavH/AGr4z0HSXUtbw+Zqk4x8reVtWJT/ANtJQ494/akMueFtBnt2fXNfVX1y9T94Ady2kfUQR+w7n+Jsn0A6Sio7iQxW0si4JRCwz7Ck3ZXYJX0JKK8h8JeKfiz4z8N2+uaVF4MitbhnVEuVu1cbWKnIUkdR610enf8AC2v7Ttv7W/4Qv7D5q/aPs32vzfLz82zdxuxnGeM1pytOzI5ux3TKHUq4DKwwQRkEVxNva/8ACE6/BpsO7/hH9TcrZqckWNxyfJB7RuASo/hYFRwygdvWP4s0htc8K39jC3l3LRb7aXbkxTod0bgeodVP4VBRZoqjoepx634f0/VIfuXttHOox03KD/Wr1AwooooAK85+OOrafa/DHU7C5vraK8uUjMNs8qiSUCVclVzk4x2r0aijqNOzuZugatp2saNBcaTfW17CqKjSW8qyBWCjKkg8Hnp1rSoopt3dyYqysFFFVdT1GDSNIvNSvCwt7OB7iUqMkIiljgd+BSGWqK5HRfip4K17aLHxDaLIxwIrljA5PoA+M/hXWo6yIHjYOrDIZTkGgV0xa5v4galfaT4MuLrSrprS6+0WsSzIisUElxGjYDArnax6g10lcl8T/wDkQp/+v2x/9LIaa3B7HE/254p/6GzUP/Aaz/8AjFH9ueKf+hs1D/wGs/8A4xVeiqFYsf254p/6GzUP/Aaz/wDjFH9ueKf+hs1D/wABrP8A+MVXooCx3fw41XU9U07VhrF/JfyWuo+RHLLHGjbPIhfBCKo+87c4rsK4T4Wf8eXiD/sLf+2lvXd1L3GtgooopDCiiigAooooAK4T4p/8eXh//sLf+2lxXd1wnxT/AOPLw/8A9hb/ANtLimtxPY4+iiimAUUUUASaTrnxB0zwKi+DvDFlqEAvb3bcyXW5/wDj6lzmL5O+QMM3Y+1eW+LviL8UDK8Ov3epaMjtgQxW5tBx2DABiPxP5V9GfDD/AJEKD/r9vv8A0smrqpYYriIxzxpLG3VXUEH8DT5rPYhxbW58D3FxPd3Dz3U0k8znLSSMWZj7k8muz+Db3kfxa0ZtMgguLoefsiuJzCjfuJM5cI5HGT90+nHWvpTWfhF4H1zJufD9tBITnzLPMBz6/JgH8QawvDXwM0rwj45sPEOjardlLQyE2tyqvuDxsmAwxjG7PIPSr500Z+zknc7H7Z4x/wCgFof/AIOpv/kWj7Z4x/6AWh/+Dqb/AORa6Cisjexz/wBs8Y/9ALQ//B1N/wDItH2zxj/0AtD/APB1N/8AItdBRQFjn/tnjH/oBaH/AODqb/5Fo+2eMf8AoBaH/wCDqb/5FroKKAsc/wDbPGP/AEAtD/8AB1N/8i0fbPGP/QC0P/wdTf8AyLXQUUBY5/7Z4x/6AWh/+Dqb/wCRag0afVJfiMg1uzsbOX+ypPJW1vWuN6iaPcfmijxglemeorp657X3XS/Eeg66yjyopnsLmTONkVxtAP8A39SEewJoA7CsLxt4ij8J+CtU1qXrawFoxn7znhB+LEVu1wHxL8Nav4w1Hw7o1vZ79CF8LrVZzKqgIn3Y9pO47snoDjiptze73He2vY818NeNfh3B8F7jwzrPiMHUdThllvXNlcMRcSc5JCEEqdozn+Gu9+AniceIfhfa28j7rnSWNnJ7qOUP02kD8DXpgAUAAYA6CvN9A8K634b+NuuajZ2e7w3rcCzSyrKgEVwP9ndu5O7kD+L2rXmUm/P9NvwuvuMuVpJ9v13/AB1PSKKKyPFWs/2D4Yvb6Nd9wqeXbRZwZZ3O2NB7lyo/GszU4KKx8a3Xg6EeDNX0qxUmbylubNmcfvXwQ+5lxjGPk/OvIPGnhH4xz+adbk1HVrYrhhY3G+Jh3/dJj/0GvpTQNKXQ/Dmn6VGxcWdtHBvbq5VQCx9yea0KpSsQ43Pga7s7qwuDBfW01tMvWOaMow/A81b8OCU+KdKFuyLKb2HYzqWUNvGCQCMjPbIr7kv9LsNVtzBqljb3kLDBjuIlkU/gRXHXXwY8ET6nBqFtpRsLmCVJUa0lZFypyPk5Xt6VftDL2Tvoa32Xxx/0GfD/AP4KZ/8A5Jo+y+OP+gz4f/8ABTP/APJNdLRWdzexzX2Xxx/0GfD/AP4KZ/8A5Jo+y+OP+gz4f/8ABTP/APJNdLRRcLHNfZfHH/QZ8P8A/gpn/wDkmj7L44/6DPh//wAFM/8A8k10tFFwsfFvxTW8T4oa2upSwTXQnHmPbxGNGOxeilmI/M1ysMMtxMsNvG8srnCoilmY+wFfY178IfB+qeJrrXdW0+S+vLqTzHE0zeWDgDAVcAjjvmun0vQtJ0SHytH0y0sU/u28Cpn8hzWntLIx9k27ny14M8IfFmNom8NxatpcG4kefObeIe5jcjcP+AmvfvB2m/EW0SL/AIS/XdJu0B+dI7RmkI9N6lFB99p/rXbUVDlc0jBRMn4dB1+Gvh9Zs+athEJdxyd+0bsnuc5rpa5XwRL9j/tbw/Kx83Tb2SSMMeWt52MsbD2G50+sZrqqkpbHhPg7Q/FGreOvHj+GfF//AAj8cesuJY/7MiuvNOWwcuRjHoKd8QfC/i3QrO08UeKdfh8a6Zo8yzzabPZiyUc4DgRnDEE/xDpnrWlott4+8GeLvFdxpvgX+2bTV9Sa5hm/teC3wuTj5Tk8574q14itviL8RtJfw/e+GrXwnp1w6fbLqXUku5HjByVRYxweB1qle0eXol+S/r8iZW5pc3Vv838v61PS9H1SDW9DstUs93kXkCTx7hghWUEZ/OrtVNK0230fR7TTbJStvZwpDGD12qAB/KrdErcz5dhxvyq+5yfjkFrzwusLiO5OsfuGZdyhvstxksMgkbd3AI5xzXBeMtB+MV3uOieItNaDaf3VnF9lkP0L7sf99iu91GQar8RLK2jw0Wi2z3Mx25xPMNkYB9RH5pI9HX1rdpXsFrnxX4o8J+O7Scz+KtN1eZlBJuJy06gf9dAWH61ydfoBWDrPgfwx4h3HWNCsbl2GDKYQsn/fYw361oqhk6XZng/7O0Wty3Wv/wBg3mn2pCW/m/bbR593MmNu2RMd+ufwr3H7L44/6DPh/wD8FM//AMk1H4R+HegeB7y+n8OxTwC+VFkieYuo2bsYzz/EepNdTUSd3c0jGyszmvsvjj/oM+H/APwUz/8AyTR9l8cf9Bnw/wD+Cmf/AOSa6WilcqxzX2Xxx/0GfD//AIKZ/wD5Jo+y+OP+gz4f/wDBTP8A/JNdLRRcLHknxit/FSfCfWW1bUtHntB5HmR22nSxSH9/HjDNOwHOP4Txn618tV91eKPDdl4t8OXOiaq0y2lyYzJ5LBWOx1cDJB6lQD7Z6daxdG+E/gjQtps/D1rLIp3CS6BnYH1G/OPwxVxmkjKdNyZ8h6Joeu6vcq3h/Tr+7lRwA9pC7bG/3lHy/WvbvBnhf4323l51kWFuGGY9VuFuMr7DDkfTK/hXvkcaRRhIkVEUYCqMAfhTqTnccadupx+nQ6vB8RdAHiG8tLu4/su/CyWls0Kg+Za8EM7Z4B6Y+ld/XH+KpTplxpGvbisOnXgF1gA/uJVMbE+gUsjk+iGuwqTRHmf7QN1NbfB+/EDlPOmhicg9VLgkfpXd+HbKDTvDOmWdmgSCC1jSNR2AUVQ8deFo/GfgvUdClfyjcx/upP7kikMpPtkDPtXDeG/ihD4N0K20P4oW15omoWCC3W6e2klgu1UYVkdA2Tgc/wCQCPwyXnf8Lf16il8Sl0s/6/rseg+I/Cei+LbSC28QWX2uK3mE8Q814yjgEAgoQe9bA4FeMeI57j42axpWlaFZXcXhSzuVu73VLmBoVucD5UiDYLZBPOO4P19mACqABgAYAp2ajr/Xn/XYLpy0/ry/ruLXldhaeLrnw5f/APCHapplkDq2peWt5aO5/wCPybGHDEAe2w16Jr2rxaDoF7qlwCyWsTOEUZaRv4UA7ljgAepFZfhfTZtI8L6fZXbB7mOEG4ZRgNK3zOfxYsaQz548Z+FPjTNvOq3N9qtuFOf7OuQUb28pNpP/AHzXkd7YXmm3Bt9RtJ7SYDJinjKMB9DzX3xVe90+y1K3a31G0gu4WGGiniDqR9CMVanYylSv1Pgevtv7L44/6DPh/wD8FM//AMk1ia18EPAms72/sj+z5WGPMsZDFt9wvKf+O16BSlK44Qcdzmvsvjj/AKDPh/8A8FM//wAk0fZfHH/QZ8P/APgpn/8Akmuloqbmljmvsvjj/oM+H/8AwUz/APyTR9l8cf8AQZ8P/wDgpn/+Sa6Wii4WOa+y+OP+gz4f/wDBTP8A/JNfLfxiS+T4sayurTW892PI8yS2haKM/uI8YVmYjjH8R5z9K+ya4nVfhH4S17xVd+INbs5r67umRmSSdljXYioAFXGRhQTknkntxVRlZ6kThzKyPjdEeWRY41Z3YgKqjJJPYCvQ/B3hH4qKyP4YtdY06Lf96SU20Z99rkBh+Br6p0nw5ougx7NF0mzsAev2eBULfUgZP41pU3UJVK3U810XS/ifa6DdjxRr2j3KiCTCC1aSQjaeN67AD74b8a9H8ONG3hbSmhz5ZsoSmTnjYMU8gMpVhkEYIPesbwBL5Hhz+xJBtn0OVrBkyT+7T/Utz6xFD9c+lRuarQ6ivJIf+TsLj/sAj/0IV63Xhuv+KtG8H/tNz6l4ivPsdodFWISeU8nzEggYQE9j2oj8a+f5MU/hfy/NHuXXrXkPwPza6/490yHizttZYwqBwpJcED8FWtO9+OXha5s5YvB73fiDVmUi3srWxmBZugLFlAC5Iya0fhN4PvfCXhOVtc2nWdUuGvb7BB2u3Rcjrgde2SaqKau32t+K/wAhSalZLvf8H/md1XK+K1uX8Q6CmnSxQ3Z+07JJojIijy+dyhlJGdvGRXVVyAddX+JNxcR/NDodn9jDhuPPnKySLj1VI4f++6gs878aaB8aLrzTpWv2UtsVwIdOxayfgW5B/wC2leD+JvDPjGwuXuPFWm6pvUfNc3KvIuP+unIP519uUEZGD0q1OxnKmn1Pz/r6A/Z7h8QyeFtVOhX2mW0QvRvW8spJmLbByCsqYGO2DXrmtfDzwl4h3nVtAsZZHGGlSLy5D/wNcN+tP8IeB9H8D2t1baAk0cF1KJWjlk37TgDgnnHHcmnKd0TGm4u5H9l8cf8AQZ8P/wDgpn/+SaPsvjj/AKDPh/8A8FM//wAk10tFRc1sc19l8cf9Bnw//wCCmf8A+SaPsvjj/oM+H/8AwUz/APyTXS0UXCxzX2Xxx/0GfD//AIKZ/wD5Jrz/AONkHiiP4X3jaxqOkXFr58O5LXT5YnJ3jGGaZh+ley1j+KfC+n+MNCfSNY802kkiOwifaTtOQM46UJ6icbqx8L1teH/D/iTVLqObw1puozyq3yT2kb/If98cD86+utF+F/gvQNh0/wAPWZkQ5WW4TznB9Qz5I/CurVVRQqKFUDAAGAK0dTsZKj3Z4Z4L8N/G218n7VrkVpag/NFqsoumI/AM34bhXqOgx6hF43ZdbuLa5vP7KX95bW7QoR5xyAGZj/dzz6cCujrnNckGk+LdB1x+IC8mmXTFsKiT7TGx/wC2sca/9tDWbdzVKyOyqG8/48Z/+ubfyqaobz/jxn/65t/Ks5/AzSPxI8N+EniXxtp3w10+20LwB/bFijzeXef2zDb+ZmRifkYZGDke+K9G0HxN421DWoLbXPh//ZFi+7zL3+2oZ/LwpI+RRk5OB+Nec/CT4s+CvC/w10/Sdd1r7LfQvMZIvskz7d0jMOVQjoR3rvLH42fD7UtQt7Gy8Qebc3MqwxJ9iuBudjgDJjwOT3rqmm5uyuc8WktXY7yiisXxfqsmj+Fb25tgHvHTyLSMn788h2Rr/wB9MPwzWBscb4T1DxSvhTTxpmg6VcWflZgmfV3jMiZO07RAwGR6E/Wtj+0vGn/QtaP/AODuT/5GrZ0bTYtG0Ox0y2GIrO3jgT6KoUfyq5TFZnNf2l40/wCha0f/AMHcn/yNR/aXjT/oWtH/APB3J/8AI1dLRQFjmv7S8af9C1o//g7k/wDkaj+0vGn/AELWj/8Ag7k/+Rq6WigLHNf2l40/6FrR/wDwdyf/ACNR/aXjT/oWtH/8Hcn/AMjV0tFAWOa/tLxp/wBC1o//AIO5P/kasTxrqHi1/APiBbvw9pUMDaZciWSPWHdkXymyQv2cZIHbIz6ivQKz/EGlnXPDOp6SJfIN/Zy2wlK7tm9Cu7GRnGc4oTE07HwfWppHiXXNAcNourXtjg7tsE7KpPuoOD+NfR2jfs4eE7Ha2rXV9qj45UuIYz+C/N/49XoGjeB/DHh7adH0KxtnUYEohDSf99nLfrWrmjBUpHiPgz4ifGG98sW+iPrkB5826s/JBHtINi/nmvSPF17rt78NZpPEWkW+lz/bLH93Deef/wAvcPXCgD8zXolcl8T/APkQp/8Ar9sf/SyGs73ZtZpbnA0UUUFBRRRQB2Hws/48vEH/AGFv/bS3ru64T4Wf8eXiD/sLf+2lvXd0nuC2CiiikMKKKKACiiigArh/ijBcS6ZosltZ3V0INU8yVbW2edlX7NOu4qgJxllGcdxXcUUCZ4f5lz/0B9c/8E13/wDG6PMuf+gPrn/gmu//AI3XuFFVcLM8P8y5/wCgPrn/AIJrv/43R5lz/wBAfXP/AATXf/xuvcKKLhZnK/DaCe38CWyXVvPbSNdXcnlXETROFa6lZSVYAjKkHkd66qiipGtgooooAKKKKACiiigAooooAKKKKACq+oWFvqmm3FjfRiW3uIzHIh7gjFWKKAMXwzrNxaXK+GvEMpOowJ/ot1JgDUIR/GP9sDG9fXkcGuprC1jRbHXbH7LqUPmIGDo6sUeJx0dHHKsOxBzWbFP4r0LKFIvElkoO1i6294o7A5/dyH3zH+PWgR19FcwfHNtDGpvtE1+2dhyg0yScqfQmEOP1xQ/i+6nKrpHhnVrln6SXKJaxpx/EZGD/AJIaAudLJIkUbSSsqIgLMzHAUDqSa4u0uT411u31ZUddC09y+n7xj7ZNgr5+P7gBITPUkt02mpG0LU/EEiyeMLqF7ZTuXSbLP2fPH+tc/NNgjoQq/wCyetdEqhFCqAqgYAA4AoAWiiigYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBgeILO9tL638RaJC1xe2aGOe0VsG8tycsgzxvBG5M98jgMTXQaPrFjr2lxahpc4mt5QcHGCpHBVgeVYHgg8gjFJWDfeHJE1GXVfDd7/ZWpS4Mx8vzILrAwPNjyMnp8ylW4644oEdbRXLReJtasmEet+Gp36/6TpUy3EZHqVYrID7BW+tPXx1YvIYo9K15pRkFTpE6Dj/AG2UKfzoGdNWP4h8QR6JbpHDH9r1K5ytnYq2Hnf+ijqzHgD8BWTLrfijVlMej6MmjRtkfa9WdZHX0KwRMc/8Cdcehqzo/h6DS7ia9mnm1DUrgYnvrogyMP7q4ACIOyqAO/J5oEHh3R5NJ0+Q3kq3Go3kpub64UYEsrAA4HZQAFUdlUVrUUUDCiiigAooooAKKKKACiiigAooooAKKKKAIrm2hvLSW2uo1lgmQxyRsMhlIwQfwrF8NalNolzH4Y16bLplNLvJG/4/YgOEJP8Ay1QcEdWA3Dvjfqnqmk2WtWDWep26zwsQwB4KsOQykcqwPII5FAG3RXHxHxVoB2QPH4jsB91LiQQ3kY54342S9h82w+rGrP8AwnMMMStqGg69aOeDGNOe4IP1g3j8c0COnpCQqksQABkk9q5lvGM06qNJ8N6xdM/Rp4VtUX3bzSrD8FJ9qpT6Lq/iXA8W3UUNgR82kWDN5cntNKQGkH+yAqnuGoAYbn/hONcgmg58OaZN5kchHGoXKn5WX1ijPIP8T4I4XJ6imxxpFGscSKiKMKqjAA9AKdQMKKKKACiiigAooooAKKKKACiiigAooooAK53WYrrRNZXxNpcMlwoiEOp2kS5aeEElZEA5MiEsQB95Sw67a6KigCewv7XVLGK90+4jubaZd0csbZVhViuQn8OXNjqMupeFL4abcTNuuLSSPzLW5bPLMgIKuf76EE/xBqmh8VatbMItc8L3kbbebjTpUuoSfblZPzT8aBHU0VzKeOrKYlbfSdekcdVbSJ4v/HnVV/Wq76r4r1lQmn6bF4fgYfNcag6z3C8/wxRkoDjuXOD/AAmgLl7xP4kOkiHTtLjW71u+BW0tuyjvLJj7sa5yT34AySBRoOkJoekR2gme4ly0k9xIBunlY5dzjuSTx24Ham6NoFroqyyRtLc3lxg3N7ctvmnI6Fj0A9FACjsBWnQMKKKKACiiigAooooAKKKKACiiigAooooAKq6ppttrGlXOnX6F7e5jMcgBwcHuD2I6g9jVqigDG8M69Ol2fDfiFyNXt0LQzMMLqEI481P9ocB16g89CDXUVg6votlrdqsF/EW2NvilRikkLjo6MOVYeorOhufFehL5csUXiSzRflkV1t7wADowOI5D15yn0piOvormG8c2sKKb3RdftnPVBpck5B9CYQ4/XFJJ4vvLjauj+GNUuWcZEl0EtY0443bzv/JDSC50s00dvA808ixRRqWd3OFUDkknsK460kfxdr0GtOrpo1gW/s1HGPtUhGDckddoBIT1DM3dacdA1HXpUm8Y3cU0KMHTSrMMLZSOnmE/NMQfUBeny5Ga6MAAAAYA6AUDCiiigAooooAKKKKACiiigAooooAKKKKACuV+JME9x4EuUtbee5kW6tJPKt4mlcqt1EzEKoJOFBPA7V1VFAM8P8y5/wCgPrn/AIJrv/43R5lz/wBAfXP/AATXf/xuvcKKq4rM8P8AMuf+gPrn/gmu/wD43R5lz/0B9c/8E13/APG69woouFmcP8LoLiLTNakubO6tRPqnmRLdWzwMy/ZoF3BXAOMqwzjsa7iiipBBRRRQMKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKyYPFGj3Pim48OQ3m7VrWETzW/lONqHGDuxtP3hwDnmtajzAKKKKACiismDxRo9z4puPDkN5u1a1hE81v5Tjahxg7sbT94cA55o30A1qKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAqC9vINOsJ728cR29vG0sjn+FVGSfyFT1R1rSodc0G+0q5ZlivYHgdl6gMCMj86TvbQatfU8sX4jfEvxNCNS8C+C7V9HZmEU1/Mu+YA43AeYmOnQA/Wun8EfESfxfbarY3GkvpniHSwRNYTNkEkHaQTjgkc/Uc85rkNKsfix8NrGPS9NsNP8AFGjW7MIdrbJkTOQOSCOTnGHxzzjFdH8PvHOj+KfE19BdeGD4f8UpCGulngAlljBA+/tVjj5eGA7YzitbJpqO34mN5Jpy3/A8v0vVviWvxp1S6tvD2mv4jeyRbqzaRfKjh/d4YHzsZ4X+I9TxXqnjv4l3XhzXLLw34c0ZtZ8QXsfmLBu2pEvOCfXoeMjAGSayND/5Ok8Rf9ghP/aNS+LPHsGn/ET+yfCHg+LXvFEcOJ7oIqNCmM7TJjOMEZyQBkDrRe8YK3R/qOzi5O/b8kUJfiZ8RPCUa33xC8GW0WlGRUe40+YExZ4yQJHz2xnb9ea9B8V+JX0n4e3/AIi0fyZ2is/tNv5qko4IBGQCDjB9RXkPxEuvilqXw31KXxfY6NpGmx7PNhtyXml/eLtAIdwBk56jpXc+Jf8Ak3Gb/sBRf+i1qZ/w5S6r/IcX76XR9/Vf5mTpPxP8Z+LtP06TwZ4YgvQqxjU72c+XCkhxvSJWdSdueuT9O9cNperfEtfjTql1beHtNfxG9ki3Vm0i+VHD+7wwPnYzwv8AEep4r2X4TQRW/wAJ/D6wIqBrUOwA6sSST+JNcvof/J0niL/sEJ/7RrVWjWaS7/kQrypXb7fmj1hc7Ru4OOaWiisDYKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACs/XtPn1Xw/fWFneTWNxcQMkVzC5V4mI4YEc8GtCik1dWGtHc8V03x38SfB2nxaPr/AID1DxBPbgouoW0zyecoPBZlR8n3JB9RW14C0HxLq3j6+8deMbBNJlltRaWdgrZZUyCWb8u/PJ4HFeoUVpza36mfJ0voeLeJj4n8GfG688UaX4WvNfstRsktwLQMdmAgOSqtggp3HINGt2Piz4ffFHU/FmgeHpfEOm6xEomgt8mWJuOPlBYcjOdpGDz0r2mikpWS8rr5PoOUea9+tvvR4j4tt/iH8T/B14s2gDw9p8UYnjsZJPMub2QchTkDaOpwQDkDr22I5tc1/wDZ3v7O88O3thqMNj9jjs2icyz7FUBwhUMM88YPTqa9WopNpxcejBR95SvqjmPhtaXNj8NdCtb63ltriK0VZIZkKOh9CDyDXnviY+J/BnxuvPFGl+FrzX7LUbJLcC0DHZgIDkqrYIKdxyDXtNFVzfvOf1/EFG0OT+tBFOVBIwSOnpS0UVBQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAH/2Q==)\n",
        "\n",
        "Using reinforcement learning on this example is to see if we can accurately predict each of these states' values through a model-free approach. The ground truth values are simply the probabilities for each state to gain a reward w.r.t. ending in state 'E'. The table below indicates the true value for each of the states that we'll measure our estimates against. Let's talk about our models and what temporal difference is now that we know our data/environment and the target values.\n",
        "\n",
        "States | $V_{\\pi}$\n",
        "---|:---:\n",
        "A|$\\frac{1}{6}$\n",
        "B|$\\frac{2}{6}$\n",
        "C|$\\frac{3}{6}$\n",
        "D|$\\frac{4}{6}$\n",
        "E|$\\frac{5}{6}$\n",
        "\n",
        "Now, let's implement the TD-Learning algorithm."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kSa_xY5DAv9G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 0 is the left terminal state \n",
        "# 6 is the right terminal state \n",
        "# 1 ... 5 represents A ... E\n",
        "states = np.zeros(7)\n",
        "states[1:6] = 0.5\n",
        "# For convenience, we assume all rewards are 0\n",
        "# and the left terminal state has value 0, the right terminal state has value 1\n",
        "# This trick has been used in Gambler's Problem\n",
        "states[6] = 1\n",
        "\n",
        "# set up true state values\n",
        "trueValue = np.zeros(7)\n",
        "trueValue[1:6] = np.arange(1, 6) / 6.0\n",
        "trueValue[6] = 1\n",
        "\n",
        "ACTION_LEFT = 0\n",
        "ACTION_RIGHT = 1\n",
        "\n",
        "# @states: current states value, will be updated\n",
        "# @alpha: step size\n",
        "def temporalDifference(states, alpha=0.05):\n",
        "    #starting from C\n",
        "    state = 3\n",
        "    trajectory = [state]\n",
        "    rewards = [0]\n",
        "    while True:\n",
        "        oldState = state\n",
        "        #walking randomly\n",
        "        if np.random.binomial(1, 0.5) == ACTION_LEFT:\n",
        "            state -= 1\n",
        "        else:\n",
        "            state += 1\n",
        "        # Assume all rewards are 0\n",
        "        reward = 0\n",
        "        trajectory.append(state)\n",
        "        # TD update\n",
        "        states[oldState] = ...\n",
        "        if state == 6 or state == 0:\n",
        "            break\n",
        "        rewards.append(reward)\n",
        "    return trajectory, rewards\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9JoqiBMBNnHG",
        "colab_type": "code",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "cf0a6636-ad7a-467b-e8aa-6e4b70ca4f15"
      },
      "source": [
        "#@title State Values\n",
        "\n",
        "def stateValue():\n",
        "    episodes = [0,1, 10, 100]\n",
        "    currentStates = np.copy(states)\n",
        "    plt.figure(1)\n",
        "    axisX = np.arange(0, 7)\n",
        "    for i in range(0, episodes[-1] + 1):\n",
        "        if i in episodes:\n",
        "            plt.plot(axisX, currentStates, label=str(i) + ' episodes')\n",
        "        temporalDifference(currentStates)\n",
        "    plt.plot(axisX, trueValue, label='true values')\n",
        "    plt.xlabel('state')\n",
        "    plt.legend()\n",
        "\n",
        "stateValue()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEGCAYAAAB1iW6ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd1gU19fA8e8svUkXBERQiiJFFFCj2LBrNJrElqIxiTEa9Y2/mB5bmiVdE00zxm7UWJIYYyzYxRY7KIgoIL132N37/oEhoGCl6v08D88uM3dmzi569u6dO2cUIQSSJElSw6eq6wAkSZKk6iETuiRJ0gNCJnRJkqQHhEzokiRJDwiZ0CVJkh4QunV1YBsbG+Hi4lJXh5ckSWqQjh8/niqEsK1sXZ0ldBcXF44dO1ZXh5ckSWqQFEW5UtU6OeQiSZL0gJAJXZIk6QEhE7okSdIDos7G0CtTUlJCXFwchYWFdR2KdJ2hoSFOTk7o6enVdSiSJN1GvUrocXFxmJmZ4eLigqIodR3OQ08IQVpaGnFxcbi6utZ1OJIk3cZth1wURVmiKEqyoihnq1ivKIrylaIoUYqinFYUpe29BlNYWIi1tbVM5vWEoihYW1vLb0yS1EDcyRj6UqDvLdb3A9yv/4wDFt1PQDKZ1y/y7yFJDcdtE7oQYi+Qfosmg4FlotRhwEJRlCbVFaAkSdKDoiArmx/HTmfv5jU1sv/qmOXiCMSW+z3u+rKbKIoyTlGUY4qiHEtJSamGQ1e/bdu24enpiZubG3PmzKmWfR47dozJkyff937GjBnD+vXrqyEiSZJq25XD0ayY+heF+t24vOtcjRyjVk+KCiG+A74DCAgIqHd31tBoNEycOJG///4bJycnAgMDGTRoEF5eXve134CAAAICAqopSkmSGpKi/BL2Lj7ExYtqDNUaCk2/5KVPaqZjVh099Higabnfna4va3COHDmCm5sbzZs3R19fnxEjRrB58+ab2l26dIm+ffvSrl07goODiYiIAEp70OPHjycgIAAPDw9+//13AEJDQxk4cCAAe/bsoU2bNrRp0wZ/f39ycnIQQjBt2jS8vb3x8fFh7dq1QOksk1deeQVPT0969uxJcnJyWQzHjx+na9eutGvXjj59+pCQkADAV199hZeXF76+vowYMaJG3y9Jkm4t+mQKK9/YzcULxVgn/83elh/h3/cxdGtoGnB19NC3AK8oirIGaA9kCSES7nens347x/lr2fcdXHleDo2Y8WjrKtfHx8fTtOl/n01OTk6EhYXd1G7cuHEsXrwYd3d3wsLCmDBhArt27QIgJiaGI0eOcOnSJbp3705UVFSFbT/55BO+/vprOnXqRG5uLoaGhvz666+cPHmSU6dOkZqaSmBgIF26dOHQoUNcuHCB8+fPk5SUhJeXF2PHjqWkpIRJkyaxefNmbG1tWbt2Le+88w5Llixhzpw5XL58GQMDAzIzM6vpnZMk6W7kZxezb1U4USfTMMlNwlH7B3P7nGFqphndeo+usePeNqErirIa6AbYKIoSB8wA9ACEEIuBrUB/IArIB56rqWDrg9zcXA4ePMiTTz5ZtqyoqKjs+bBhw1CpVLi7u9O8efOy3vu/OnXqxNSpU3nqqacYOnQoTk5O7N+/n5EjR6Kjo4OdnR1du3bl6NGj7N27t2y5g4MDPXr0AODChQucPXuWXr16AaVDRU2alJ6H9vX15amnnuKxxx7jscceq+m3Q5KkcoQQXDySxL41ERTnF+Ma8ydGXQRTHM/ySkY2Th0/R6VTcxfo3zahCyFG3ma9ACZWW0TX3aonXVMcHR2Jjf3v/G5cXByOjhXP72q1WiwsLDh58mSl+7hxmt+Nv7/55psMGDCArVu30qlTJ/7666+7jlMIQevWrTl06NBN6/744w/27t3Lb7/9xocffsiZM2fQ1a1X149J0gMpJ72QPasvcOVMGua5V2kb+yuN3h7NyJQ5tM0toWWBNx2Ce9VoDLKWSzmBgYFERkZy+fJliouLWbNmDYMGDarQplGjRri6urJu3TqgNLmeOnWqbP26devQarVcunSJ6OhoPD09K2x/6dIlfHx8eOONNwgMDCQiIoLg4GDWrl2LRqMhJSWFvXv3EhQURJcuXcqWJyQksHv3bgA8PT1JSUkpS+glJSWcO3cOrVZLbGws3bt3Z+7cuWRlZZGbm1uTb5kkPfSEVnB2bzyrZ4cRdzYF96j1dMz7Dc9lX/BW8Rr01Go+Sk5Fv+fMGr+uQ3bdytHV1WXhwoX06dMHjUbD2LFjad365m8KK1eu5OWXX+aDDz6gpKSEESNG4OfnB4CzszNBQUFkZ2ezePFiDA0NK2z7xRdfsHv3blQqFa1bt6Zfv37o6+tz6NAh/Pz8UBSFefPmYW9vz5AhQ9i1axdeXl44OzvTsWNHAPT19Vm/fj2TJ08mKysLtVrN//3f/+Hh4cHTTz9NVlYWQggmT56MhYVFzb9xkvSQykzKZ/eKCK5FZmJDEu6Hv8a+T2fsZ33GvNNfEJEewZdJaZwy6EPfwKAaj0cpHTGpfQEBAeLGG1yEh4fTqlWrOomnOowZM4aBAwfyxBNP1HUo1aqh/10kqbppNVpO7ojlyO+X0VEJ3OP+oPGFv7B/+y0sR45kT9weJu2axKMllrwbG8HZJ/YQ5FM9/4cURTkuhKh0HrTsoUuSJN2F1Lgcdi2LIOVqDk3tNTTb+iHGBlocl/2Mcdu2JOUl8d6B9/AwdWbmmf1sMRvF494tayU2mdCr0dKlS+s6BEmSaoimRMuxP2M4se0KBia6dLCJxGjNFxi3a4fj55+h17gxGq2Gt/a/RZGmiGnJuWSLRrg8+mat1USSCV2SJOk2EqOz2LUsnIzEfDzaWOJy4BvUR/dj+fTT2L0+DUVfH4Afz/7I0cSjTHcdTofI+SyznMCzns1qLU6Z0CVJkqpQXKgmbEs0p3fHYWppQO8BZvDJq2iysnCYNxfzcrPgTiaf5JuT39DPpR/dwjZyRdsYv8derdV4ZUKXJEmqROz5dHavjCAnrRDvro600p4m/e330bWzw2X1KgzLTRTILs7m9b2v08SkCa8ZtMQ271u+tXuHl1wa12rMMqFLkiSVU5hXwsENUYQfTMDCzpjBk71RrfqKtPUbMAkOxnH+PHTKTQcWQjDz4ExS8lNY1usHDH8axRmtC50fe7HWY5cXFt1g7NixNG7cGG9v72rb57Vr16plKuPMmTP55JNPqiEiSZIqE/1PCqtnhRFxOJG2fZox9HknSqZPJGv9BqxfHk/TxYsqJHOA9ZHr+fvK30xqO4kWFw/QqDiRXU0n0trRstbjlwn9BmPGjGHbtm3Vuk8HBwdZx1yS6rG8rCK2fXeGP789g7G5Pk++GYCvfTKxw4dRHBOD09cLaTxlCoqOToXtojKimHtkLh2bdGRM88dQ9s1nn9aHfoNuWTGlxsiEfoMuXbpgZWV1yzYpKSk8/vjjBAYGEhgYyIEDB4DSHvQzzzxDx44dcXd35/vvvwdKKzD+2+M/d+4cQUFBtGnTBl9fXyIjIwH47LPP8Pb2xtvbmy+++KLsWB9++CEeHh507tyZCxculC2vqoTvunXr8Pb2xs/Pjy5dulTfGyNJDyAhBBGHE1g9K4yY02l0eKw5j7/RDtXfv3D1+RfQsbbCZd0vmIWE3LRtobqQaXunYaJnwkfBH1EY+gVG6mwONZ+Mh51ZHbya+jyG/uebkHimevdp7wP97v8uRFOmTOHVV1+lc+fOXL16lT59+hAeHg7A6dOnOXz4MHl5efj7+zNgwIAK2y5evJgpU6bw1FNPUVxcjEaj4fjx4/z000+EhYUhhKB9+/Z07doVrVbLmjVrOHnyJGq1mrZt29KuXTug6hK+s2fP5q+//sLR0VGWz5WkW8hOK2DPygtcPZ+OfXNzejzbkkamkPDaa+Rs24ZZ3744fPgBKhOTSrf/5NgnRGVGsbjnYmxKiik5upjNmk48MaB/Lb+S/9TfhF6P7dixg/Pnz5f9np2dXVYEa/DgwRgZGWFkZET37t05cuQIbdq0KWvbsWNHPvzwQ+Li4hg6dCju7u7s37+fIUOGYHL9H87QoUPZt28fWq2WIUOGYGxsDFBWKOxWJXw7derEmDFjGDZsGEOHDq3ZN0KSGqB/i2kd2ngJAQQP98CnqyPFMTHEvDCJ4ujLNJ42Dauxz1V5QdDOKztZe2EtY1qPoZNjJ/LXT0RXq+Zcy0kMtjWt3RdUTv1N6NXQk64pWq2Ww4cP31R4C25fPnfUqFG0b9+eP/74g/79+/Ptt9/e0/GrKuG7ePFiwsLC+OOPP2jXrh3Hjx/H2tr6ro8hSQ+ijMQ8di+PIOFSFk29rOg2ypNGNkbk7NjBtTfeRNHXx3nJj5h06FDlPhJyE5h+cDqtrVsz2X8ypFzA8Owqlml780y/rrX4am4mx9DvQe/evVmwYEHZ7+UT6+bNmyksLCQtLY3Q0FACAwMrbBsdHU3z5s2ZPHkygwcP5vTp0wQHB7Np0yby8/PJy8tj48aNBAcH06VLFzZt2kRBQQE5OTn89ttvwK1L+F66dIn27dsze/ZsbG1tK9R3l6SHlUaj5fi2GNZ+cJT0hDxCRrfi0Ul+mFnqk/zFF8S9Mgl9V1dcN6y/ZTJXa9W8ue9N1Fo187rMQ09Hj/w/p5MvDIj3nUhTK+NafFU3q7899DoycuRIQkNDSU1NxcnJiVmzZvH8889XaPPVV18xceJEfH19UavVdOnShcWLFwOldwzq3r07qampvPfeezg4OBATE1O27S+//MLy5cvR09PD3t6et99+GysrK8aMGUNQUGl5zRdeeAF/f38Ahg8fjp+fH40bN67w4VBVCd9p06YRGRmJEIKQkJCysr6S9LBKuZrDruXhpMbm0qKtLcHDPTAxN0CTmUn8a9PI278f8ycex/6991AZGNxyX9+d/o4TySf4OPhjnBs5w9UwjKO38bl2GGN7B95y29ogy+dWo5kzZ2Jqasprr71W16FUq4b+d5EeTuoSDUf/iOGf7VcxNNWj60gPWviXXrlZGB5O3KTJqJOSsHvvXSyHDbvt/o4mHuWF7S8wsPlAPuz8IQhB4Xe9yL4WyQ/+G3j7sUor2lY7WT5XkqSHyrWoTHYvjyAzKZ+WjzSh0+NuGJroAZC1ZQsJ701Hx8KCZiuWY3QH32IzCzN5a99bNDVrytvt3y5deOFPDBOO8rF4gYkh1Xch4v2QCb0azZw5s65DkKSHWnGhmsMbL3FmTzxmVoY8OtkPZ6/SSQGiuJikefPJWLEC48BAHD//DF0bm9vuUwjB9IPTSStMY0X/FZjomYBGTdFfM4jXNsE4aAyNzW6eIFEXZEKXJOmBcPVcGrtXRpCbUYRvdyfaD26OvmFpiitJTib+/16l4MQJrMaMofFr/0O5w5unr7mwht2xu5kWMI3W1tdvSXlqNQYZF/mKqUzv7nnrHdQimdAlSWrQCvNK2L8ukguHE7G0N2boa+1o0sK8bH3+iRPETZmCNjcPh08/wfyGi/1u5UL6BT45+gnBjsE84/VM6cLifEp2fsBZrRtNOw3HykS/ul/SPZMJXZKkBkkIwaUTKexdc4GiPDXt+jUjoL8Luno6ZeszVq0i6eM56Dk64PzDjxh6etzx/vNL8nl97+s0MmjE+53e/++akiPfopeXyFeqmXzRpUVNvLR7JhO6JEkNTl5WEXtXXyT6ZAq2zmY8Orkltk3/q5+iLSwkccZMsjZvxrRbNxzmzUWnUaO7Osa8o/O4nHWZ73p/h7XR9Yvz8tPR7P2MUI0/bbsNxNxIrzpf1n2TFxbdoKryuenp6fTq1Qt3d3d69epFRkZGtRyvf//+911zJTQ0lIEDB1ZLPJJUnwkhCD94jdWzwrhyNo2OQ1rwxBvtKiTz4rg4YkaOImvLFmwmvYLTN1/fdTL/K+YvNkRuYKz3WDo0KXeh0b5PUYpzWKT7NM91dq2ul1VtZEK/QVXlc+fMmUNISAiRkZGEhIQwZ071lCbYunUrFjfUV5Yk6WbZqQVs+fIku5ZFYOVgwoj3gmjbpxkqnf/SWO7+A8Q8/gQl8fE4LfoG24kTUVR3l+bic+OZdXAWvra+TPSf+N+KzKtow75jgzqYXt26Y2pQ/wY4ZEK/QVXlczdv3szo0aMBGD16NJs2bbqpjUajYdq0aQQGBuLr61tWpyU0NJQuXbowYMAAPD09GT9+PFqtFgAXFxdSU1PJy8tjwIAB+Pn54e3tzdq1awHYuXMn/v7++Pj4MHbs2LIiXNu2baNly5a0bduWX3/9tSyGvLw8xo4dS1BQEP7+/mzevBmoumyvJNV3Wq3g1K5YVs8OI+lyNl1HejBkalss7P67zF5otaQu/pbYF19E184O1/XrMOvW7a6PVaIt4fW9ryMQzA2ei56q3JDK7o9RawVLDUbxbEeX+39hNaD+fcRcN/fIXCLSI6p1ny2tWvJG0Bv3tG1SUhJNmjQBwN7enqSkpJva/Pjjj5ibm3P06FGKioro1KkTvXv3BuDIkSOcP3+eZs2a0bdvX3799dcKdzHatm0bDg4O/PHHHwBkZWVRWFjImDFj2LlzJx4eHjz77LMsWrSI8ePH8+KLL7Jr1y7c3NwYPnx42X4+/PBDevTowZIlS8jMzCQoKIiePXtWWrZXkuq79Gt57F4RTmJ0Ns6tren2lCdmVhXnfGtyc7n25pvk7thJo4EDaTJ7Firje6upsujkIk6nnGZ+l/k4mTn9tyLxLOLUapaoB/BErw4Y6etUvZM6JHvo90BRlErLam7fvp1ly5bRpk0b2rdvT1paWllPOCgoiObNm6Ojo8PIkSPZv39/hW19fHz4+++/eeONN9i3bx/m5uZcuHABV1dXPDxKz8yPHj2avXv3EhERgaurK+7u7iiKwtNPP10hhjlz5tCmTRu6detGYWEhV69epWPHjnz00UfMnTuXK1euYGRkVIPvkCTdH41Gy7Gtl1n70REykvLp+ZwXA1/xvSmZF0VFEfPkMHJ3h2L39ls4zJ93z8k8LCGMH878wFD3ofR17Vthndg5izzFhI3GTzIyyPmeX1dNq7c99HvtSdcUOzs7EhISaNKkCQkJCTRufPPdvIUQLFiwgD59+lRYHhoaetuyuh4eHpw4cYKtW7fy7rvvEhISwuDBg+86TiEEGzZswNOz4sUOrVq1uqlsb48ePe56/5JU05KvZLNrWQRp8bm4tWtM8HAPjBvdPNc7+6/tJLz1FoqREc4/LcHkenG7e5FemM5b+97CxdyFNwJvyD2X96FEbmdByUie6eOPoV797J2D7KHfsUGDBvHzzz8D8PPPP1eabPv06cOiRYsoKSkB4OLFi+Tl5QGlQy6XL19Gq9Wydu1aOnfuXGHba9euYWxszNNPP820adM4ceIEnp6exMTEEBUVBcDy5cvp2rUrLVu2JCYmhkuXLgGwevXqCjEsWLCAf4uu/fPPP0DlZXslqT5RF2s4+GsU6+ccoyC3mH7jfejzovdNyVyo1SR/8gnxU6Zg4O6O668b7iuZCyF478B7ZBVlMb/LfIz1jMuvROyYQYrKhr/NBjMsoOk9H6dWCCFu+wP0BS4AUcCblax3BnYD/wCngf6322e7du3Ejc6fP3/Tsto2YsQIYW9vL3R1dYWjo6P44YcfhBBCpKamih49egg3NzcREhIi0tLSbtpWo9GIt956S3h7e4vWrVuLbt26iczMTLF7924RHBws+vfvLzw8PMRLL70kNBqNEEKIZs2aiZSUFLFt2zbh4+Mj/Pz8REBAgDh69KgQQogdO3aINm3aCG9vb/Hcc8+JwsJCIYQQf/75p/D09BT+/v5i8uTJYsCAAUIIIfLz88W4ceOEt7e38PLyKlv+8ccfCy8vL+Hn5yf69OlTafxVqQ9/F+nBFn8xXSx/76BY+NJOsWvZeVGYV1xpu5L0dBEzZow479lSXJsxQ2iKiu772MvPLRfeS73FyvMrb155dqMQMxqJ195+Taw9evW+j1UdgGOiqlxd1QrxX7LWAS4BzQF94BTgdUOb74CXrz/3AmJut9/6mtBrwu7du8sSa0P0oP5dpLpXlF8iQldGiIUv7RTL3jkgroZX3dHIP31GXOzeXYT7+IqM9Ruq5fjnU88L/2X+4pWdrwitVltxpbpYaL/0F5dneYse83aIErWmWo55v26V0O9kDD0IiBJCRAMoirIGGAycL9dGAP/O3DcHrt39dwVJkh4mMWdS2bPqArmZRfiFNKX9oOboGdw8Pi1KSsj45ReS585Dx8aaZqtWYeTd+r6Pn1+Sz7S907A0tOT9R96/eaLDiWUo6Zd4v/h/TB7UEl2d+j9CfScJ3REofx+zOKD9DW1mAtsVRZkEmAA9K9uRoijjgHEAzs7190xxdevWrRvd7mFOrCQ9iApyi9n/SyQXjyRh2cSEx6d5Y9/c/KZ2QqMh67ffSP1mESVXr2LyyCM4fPoJupaW1RLHR2EfcTX7Kj/2+RELwxsu7ivKRYTO4YyOF1etuzDQ16FajlnTqmuWy0hgqRDiU0VROgLLFUXxFkJoyzcSQnxH6fAMAQEBdXOrJEmS6oQQgqjjyexbe5GiPDUBA1wI6OuCjl7Fnq/Qasn+809SF35N8eXLGHi1wmnRN5h261bpdOF78Uf0H2y+tJmXfF8i0L6SW8cd/gYlL5mZRROY+pgnOqrqOW5Nu5OEHg+UP7XrdH1Zec9TeuIUIcQhRVEMARsguTqClCSpYcvLLCJ01QViTqfSuJkZg6a0wsbJtEIbodWS8/cOUhcuoCgyCgN3dxy/+hKzXr2qLZEDxGbH8v7h92nbuC3j/cZXEmwq4sCX7NPpQKF9AH1a21fbsWvanST0o4C7oiiulCbyEcCoG9pcBUKApYqitAIMgZTqDFSSpIZHCEH4gQQObIhCo9byyONu+PVwqlB/RQhB7u5QUhYsoCg8HP3mzXH87FPM+va96zost1OiKb20X6WomBM8B11VJSlwzzxESQEzC5/gnSc8UDWQ3jncQUIXQqgVRXkF+IvSGS9LhBDnFEWZTenZ1i3A/4DvFUV5ldITpGOun42VJOkhlZWSz+4VF4i/kIGDuwXdn2mJReNy9VeEIG//flK+WkDhmTPoOTvjMHcOjQYORNGpmYt3FvyzgLNpZ/m82+c0MW1yc4P0aMSxJWxRhdDIyYseLW++gLA+u6MxdCHEVmDrDcuml3t+HuhUvaHVjbFjx/L777/TuHFjzp49W7Y8PT2d4cOHExMTg4uLC7/88guWlpYIIZgyZQpbt27F2NiYpUuX0rZt2/uOY/r06XTp0oWePSs9v3zHTE1Nyc3Nve94JOlOabWC07tiCdscjaKj0O0pT7w6OaCU6+nmHT5MypdfUfDPP+g5ONDkg/cxHzwYRa/m6osfjD/IT+d+YpjHMHo2q+L/1a4P0aDDR3mD+WS4R7UO9dSG+j8Pp5bdbfncP//8k8jISCIjI/nuu+94+eWXqyWO2bNn33cyl6Talhafy4Z5xzmwPgrHlpaMmtGe1sGOZck8/9gxrjw7mqtjnqPk2jXsZ86gxbY/sXjiiRpN5qkFqby9/23cLNyYFjit8kbXTsLZ9SynPy4uLQh2v/0NpOsbmdBvcLflczdv3syzzz6Loih06NCBzMxMEhISbtp+xYoVZeVrX3rppbJqh6amprz66qu0bt2akJAQUlJKTz2MGTOG9evXA/Dmm2/i5eWFr68vr732GgAxMTH06NEDX19fQkJCuHr1KgCXL1+mY8eO+Pj48O6771aIYf78+WWlfWfMmAFQZdleSbobGrWWI79f5pePjpKdWkCv570YMMEXU8vSYloFJ09ydezzXHn6GYouR2P3zju02P4XliNGoOjX7D05tULLO/vfIbckl/ld5mOoa1h5wx0zKNSz4LP8/kzt3fB651CPi3MlfvQRReHVWz7XoFVL7N9++562rap8bnx8PE2b/jcJyMnJifj4+LK2AOHh4axdu5YDBw6gp6fHhAkTWLlyJc8++yx5eXkEBATw+eefM3v2bGbNmsXChQvLtk1LS2Pjxo1ERESgKErZ3Y0mTZrE6NGjGT16NEuWLGHy5Mls2rSJKVOm8PLLL/Pss8/y9ddfl+1n+/btREZGcuTIEYQQDBo0iL1795KSknJT2V5JuhtJMdnsWhZO+rU83APtCB7mjpFZaZIuOHuOlAVfkbdnLzqWljR+/XUsR45AVYvVPpedW8bBawd5r8N7uFm6Vd7o0i6IDmWhMgZft6Z0aG5da/FVp3qb0OuzqsrnVmXnzp0cP36cwMDS+a4FBQVl1RpVKlVZPfOnn36aoUOHVtjW3NwcQ0NDnn/+eQYOHFh2q7lDhw6V3djimWee4fXXXwfgwIEDbNiwoWz5G2+UVo7bvn0727dvx9/fH4Dc3FwiIyMJDg7mf//7H2+88QYDBw4kODj4nt4T6eFTUqzhyJZoTu2MxdjcgP4TfHH1LR2mKLxwgZQFC8jdsROVuTm2U6di9dQoVCYmtRrj2dSzfHniS3o168WTHk9W3kirhb9nkG3owHeZ3Vndy7Pydg1AvU3o99qTrilVlc91dHQkNva/C2nj4uJwdHSssK0QgtGjR/Pxxx/f9jg3flDo6upy5MgRdu7cyfr161m4cCG7du26q338G8Nbb73FSy+9dNO6G8v2Tp8+/aY2klRe3IUMdq+IIDulgNbBDnQc6oaBkS5FUVGkLPyanG3bUJmaYjPpFaxGj0bH1PT2O61mucW5vL73dWyMbZjRcUbVnbCzGyDxNB8zmU6eDrRrVj1XotYFOYZ+h6oqnzto0CCWLVuGEILDhw9jbm5eYbgFICQkhPXr15OcXHqdVXp6OleuXAFAq9WWjZWvWrXqprK6ubm5ZGVl0b9/fz7//HNOnToFwCOPPMKaNWsAWLlyZVnPulOnThWW/6tPnz4sWbKkbMZLfHw8ycnJlZbtlaSqFBWo2b0ygs2fl5ZlHvyqP92eagmJscRPe53oRweRt3cv1i+Px23nDmwnTqyTZC6E4IOwD4jPjWdu8FzMDW4uLQCAugh2zSbFxJ01hUFMbcC9c+DOyufWxE99rbZ4t+VztVqtmDBhgmjevJggeUQAACAASURBVLnw9vYuK3t7ozVr1gg/Pz/h4+Mj2rZtKw4dOiSEEMLExES8+uqronXr1qJ79+4iOTlZCCHE6NGjxbp168S1a9dEYGCg8PHxEd7e3mLp0qVCCCFiYmJE9+7dhY+Pj+jRo4e4cuWKEEKI6Oho0aFDB+Ht7S3eeecdYWJiUhbDF198Iby9vYW3t7fo0KGDiIqKqrJsb3n14e8i1b3oUynip9f3ia/H7xT710eK4iK1KLp6VcS/+ZY479VahLfxF0nz54uS9PS6DlVsitwkvJd6i0UnF9264aFFQsxoJMZNnyPGLav8/259wy2qLSqijq7/CQgIEMeOHauwLDw8nFatWtVJPHWlIcwTfxj/LtJ/CnKK2fdLJJFHk7B2NKH7M62w0s8lddFiMjduRFGpsBw5EusXX0DXpu6n+sVkxTDs92G0tm7ND71/QEdVxUVKhdnwVRtidJvTPeX/+HNKF1raN6q8bT2iKMpxIURAZevq7Ri6JEl1SwhB5NEk9q2NpLhQTdCjrvi0MSLzxy+5tG4dArAcNgzrl15Cz65+XFFZrCnm9b2vo6+jz8fBH1edzAEOfgX5aUzT/o+Bvo4NIpnfjkzoday+986lh1NuRiF7Vl0g5kwadq6N6DLQHrFlOTFvr0VoNFgMGYLNy+PRc6hfZWU/P/454enhfNX9K+xNblFUKycRDn3NOcueHE904e+e7rUXZA2SCV2SpDJCKzi3/xoHf41CaAQdBzjiGPEb6U+vQhQVYT54MDYTXka/af27t+beuL2sCF/BqJaj6O7c/daN98xFaIr5v5SBDPF3ooVt7Z+4rQkyoUuSBEBmUj67V0RwLTITxxZm+IqjlMz4H+kFBTQaOBCbCS9j4Opa12FWKjk/mXf3v4unpSdTA6beunFqJBz/mSPWj3H5mh0/hjwYvXOQCV2SHnpajZZTO+MI+y0aHR2FAMdrmP/yFUU5OZj17YvtKxMxcKviCst6QKPV8Pa+tynUFDKv6zwMdAxuvcHO2Wh1DZmS0JsnA5xwtja+dfsGRCZ0SXqIpcblsnt5OMlXcnA0z6X5/gXopcVhEhKC7aRXMGzZsq5DvK2fzv1EWGIYsx6ZRXPz5rduHHsUwrewy24s6fnmvNLjwemdg0zoFWRmZrJq1SomTJhQ16EAsHTpUo4dO1ahtoskVQdNiZZj22I48ecV9FRqfK6sx+byXky7dsF20hfVchPm2nAy+SQL/1lIX5e+DHEbcuvGQsCOGWiMbHg1NpiR7ZviaFF7NWVqg0zo5WRmZvLNN99UmtDVajW6uvLtkhq+xOgsdi07T0ZiAU0yT+F2diUWQb7YfLQK4+u1fhqC7OJs3tj7BvYm9kzvOP329ZUit8OVA2y0/z+Kc4yY2L3+DiPdK3npfzlvvvkmly5dok2bNkybNo3Q0FCCg4MZNGgQXl5exMTE4O3tXdb+k08+YebMmQBcunSJvn370q5dO4KDg4mIqFgpUqvV4uLiUlYtEcDd3Z2kpCR+++032rdvj7+/Pz179iyr5Fhe+XK6UHpB0r9kWVzpTpQUadi/JoIN845REJuI7+lvaKf3D24/LcJ5yZIGlcyFEMw+NJuk/CTmdpmLmb7ZrTfQamDHTIrNXXj7Slue7diMxo2qKKPbgNXbLue+Xy6SGlu9c7RtmpoSPMyjyvVz5szh7NmznDx5EoDQ0FBOnDjB2bNncXV1JSYmpsptx40bx+LFi3F3dycsLIwJEyZUKKKlUqkYPHgwGzdu5LnnniMsLIxmzZphZ2dH586dOXz4MIqi8MMPPzBv3jw+/fTTO3pNsiyudCdizyaz68dT5Bbo4Bi/l9Yml3H49H8Yd+jQIOt+/xr5K3/F/MWUtlPws/W7/Qan10LyeZY3mYGungHju7ao+SDrQL1N6PVFUFAQrreZqpWbm8vBgwd58sn/ynMWFRXd1G748OHMnj2b5557jjVr1pSVzY2Li2P48OEkJCRQXFx82+OVJ8viSrdSmFNE6Je7uRSnj1F+Kh2KD+H5+pOYdO7cIBM5QHRmNHOOzKF9k/aM9R57+w1KCmHXhxTY+vH+ZQ8mdnfB2vQ2M2EaqHqb0G/Vk65NJuXqN+vq6qLVast+LywsBEqHUywsLMp69lXp2LEjUVFRpKSksGnTprI7Ck2aNImpU6cyaNAgQkNDy4Zxyit/bK1WS3FxMSDL4kpVO7diN4dCcyjWMaJ53nE6PNsOi15fNthEDlCkKWLa3mkY6xnzceePUSl3MGp85DvIjuMbh//DzECPF4NvMxOmAZNj6OWYmZmRk5NT5Xo7OzuSk5NJS0ujqKiI33//HYBGjRrh6urKunXrgNIk+2+Z2/IURWHIkCFMnTqVVq1aYW1deleUrKysshrq/5bovZGLiwvHjx8HYMuWLZSUlACyLK50s5zYFDa9spLQ/QJ9bT79umvpu+x/WPbu0aCTOcCnxz7lYsZF3u/0PrbGtrffoCAD9n1KjlM3FkQ78EJwcyyMa/aWd3Wp3vbQ64K1tTWdOnXC29ubfv36MWDAgArr9fT0mD59OkFBQTg6OtKy3BzdlStX8vLLL/PBBx9QUlLCiBEj8PO7eWxv+PDhBAYGsnTp0rJlM2fO5Mknn8TS0pIePXpw+fLlm7Z78cUXGTx4MH5+fvTt27fsm0Pv3r0JDw+nY8eOQOnJ0hUrVhAVFcW0adNQqVTo6emxaNGi6niLpHpMCMHpJTsJO1SIWmWLt10ij7w9FD2jB+Pk366ru1gdsZpnvJ6hi1OXO9to/+dQmMV8zQgsjPUY29mlRmOsa7J8rnRb8u9S/2VcTmLn/N0kaRtjUZxAyFgf7Dv71nVY1SYxL5EnfnsCBxMHVvRfgb7OHfSys+Lgq7akuQyg3bkneL2vJxO6NfypirJ8riQ9oIRWcGzx3xz/RwOY4980ifZvPImOwYMxrFCoLuTPy3+y5OwSijXFzO86/86SOUDox4Bgdt5gbEz1GfOIS02GWi/IhC5JDVRqRDw7vjxAmrDBRpNAj/GB2Ab1q+uwqkVCbgJrL6xlQ+QGMosycbNw4/Nun9OsUbM720FyOJxcxbWWz7H5Hz3eHdACY/0HP93Vu1cohGjwJ24eJHU1JCdVTaPWELbwb06dV1BpjQlqkUy710ah0tOr69DuixCCY0nHWB2xmp1XdwLQo2kPRrUaRYBdwN3lhR2zEPqmvJPWG7tGujzd4Q4/CBq4epXQDQ0NSUtLw9raWib1ekAIQVpaGoaGD8ZJtQdB4qkr7Fh0jCwssVNfJmRKZyz9GvaNjQvUBWyN3sqqiFVczLiIuYE5Y1qPYbjncBxM7+EGGlcOwsU/ifH7H7vDNLw/uCWGere4c9EDpF4ldCcnJ+Li4khJSanrUKTrDA0NcXJyquswHnolxWoOfPE35y/poleiQyevFPxeHYOi03AT1bXca6y5sIZfI38lqygLT0tPZj0yi/6u/THUvcdOhBDw9wyEWRNej3sERwsVwwLr3804akq9Suh6enp3dZWkJD0MYo9El162r5jjpI6k22shmHs1zNkaQgiOJh5lVcQqdsfuRkGhh3MPRrUcRTu7dvf/zTzid4g7QnjABxzdX8Tcx30w0G24H3p3q14ldEmS/lNcUMKeT7dzMdYAw2It3dqk4jXpRRRVw7seML8knz8u/8Gq8FVEZUZhYWDBWO+xDPccfut7f94Njbp07NzGg9ejfGhmLRja9uH6dnlHCV1RlL7Al4AO8IMQYk4lbYYBMwEBnBJCjKrGOCXpoRK95yKhqyIowBgX9QW6vj0AU3eXug7rrsXlxJXNVskpzqGVVStmPzKbfq797n1YpSonV0BaJCc6LORsaB6fDfNDT6fhffjdj9smdEVRdICvgV5AHHBUUZQtQojz5dq4A28BnYQQGYqiNK6pgCXpQVaQW8Tu+X9zOckY48J8enYsxmPcyw2qVy6EICwxjFXhqwiNDUWlqOjZrCejWo7Cv7F/zUx4KM6D3R8jnNrz5rmmtLCFwW0cq/849dyd9NCDgCghRDSAoihrgMHA+XJtXgS+FkJkAAghkqs7UEl60F3Yfp59G6IpxgA37Vm6zB6CUbOGc0IvvySf36N/Z1X4Ki5lXcLK0IoXfF5gmOew6htWqcrhRZCbyL4284nckcfCUf7oqB6+mXJ3ktAdgdhyv8cB7W9o4wGgKMoBSodlZgohtt24I0VRxgHjAJydne8lXkl64ORlFLBz/i5i040wLcigezcTmj83qcFM3Y3NiWVNxBo2Rm4kpyQHL2svPuj0AX1d+97+hs3VIS8NDnyJ1qMfM042oqW9iv7eTWr+uPVQdZ0U1QXcgW6AE7BXURQfIURm+UZCiO+A76C0lks1HVuSGiQhBGd/O8OhP+LRCF1acorOHw3HwOke5l7XMiEEhxIOsSp8FXvj9qKj6NCrWS9GtRqFn61f7X4Y7fsEinPZ3mQ8l0/n8d0z7VA9hL1zuLOEHg+U/97ndH1ZeXFAmBCiBLisKMpFShP80WqJUpIeMFnJeez4JJTEbCPM8xII7mOD89P/V+975fkl+Wy5tIVVEau4nHUZK0MrxvmOY5jnMBob18Gps4wYOPI9Gr+neD9Mi6+TOb287Go/jnriThL6UcBdURRXShP5CODGGSybgJHAT4qi2FA6BBNdnYFK0oNAaAX/rD/JkZ3JoFXho3uSDvOfQr9J/U5CV7OvsjpiNZuiNpFbkou3tTcfdf6IPi597rxYVk3Y/RGodNhs8Qzxmal8OMS73n8o1qTbJnQhhFpRlFeAvygdH18ihDinKMps4JgQYsv1db0VRTkPaIBpQoi0mgxckhqa9Lhs/v58H6l5RljnXCZ4cFMchr1abxOQVmg5dO0QK8NXsj9+PzoqHXo3681TrZ7C17YelOZNOA2nf0H9yBTmHcylXTNLunrcwU0vHmB3NIYuhNgKbL1h2fRyzwUw9fqPJEnlaDRajq46wT/7M1CptfgbniDwy9Ho2dbP5JNXksfmqM2sjlhNTHYM1obWjPcbz5MeT97ZXYJqy46ZYGjOGr2hJGbH8dnwWh67r4fklaKSVIOSozP4+6tDZBYa0jgrnC4jPLF77LW6DqtSMVkxrI5YzeZLm8krycPXxpc5wXPo3aw3ejr1rJJjdChc2klxyGy+2JNCx+bWPNLCpq6jqnMyoUtSDVAXazj08zHOHMtBr7iIILNz+C96AV1Ly7oOrQKt0HIg/gArI1ZyIP4Auipd+rr0ZVTLUfjY+tR1eJXTauHvGWDelJ9LepGae5nFT7et66jqBZnQJamaxYensGPRMXKLDXDIPEnw6DbY9J9W12FVkFOcw+aozay5sIYr2VewNbJlYpuJPOHxBDZG9byne34jJJykcODXfLM1jq4etgS4WNV1VPWCTOiSVE2KC9Xs//Eo4WcKMCzIoZP1GXzmjEPHwqKuQysTnRXN6vDVbLm0hXx1Pn62fkzsMpGezj3r37BKZdTFsPN9aNyaHzIDyMi/xNReHnUdVb0hE7okVYOYk4ns/uEk+SW6OKcfpdO4Dlj1GlbXYQGlwyr74vaxKmIVB68dRE+lRz/XfoxqNYrW1q3rOry7c+JnyLhM3uOr+XbDFXp52eHXtP58YNY1mdAl6T4U5pYQ+l0Yly4WY5yXTheHOLzmT0SnUaM6jSunOIfEvEQOXTvEmgtriM2JpbFRYyb5T+Jx98exNrKu0/juSVEO7JkLLsEsvtacnELZO7+RTOiSdA+EEEQdjmfP8nMUq1W4Zhyi0ys9MO9a81WjizRFJOUlkZCXQGJeIol5iaXP8xNJyksiMS+R3JLcsvb+jf2Z3HYyIc4h6KkawLBKVQ4uhLwUsoasYMmyGAb4NKFVk7r94KxvZEKXpLuUl1XErm/CuHpFjVlOAp1cUvD4bCo6pqb3vW+1Vk1qQWrFRF3ueVJ+EumF6TdtZ2Vohb2JPc5mzgTZB9HEpAn2Jva4WbjhZtkw725UQW4yHFwAXoP55qI5BSVpvNrLva6jqndkQpekOySE4HzoFQ78chGNWotHxn46TB2A2SNj7nj7jKKMCom6Qk87P5GU/BQ0QlNhO1M9U+xN7LE3sae1TWvsjUuf/5u07UzsaqeqYW3QaiE/FXISITfp+mMiRO8BdSFpHd7k5+9jGNzGEbfGZnUdbb0jE7ok3YHs1AJ2fB1GQoIW88wY2nvm0OKLt1CZmJS1yS3OLUvMlSXtpPwkijRFFfarr9IvS9ZB9kHYGdvRxLRJWdK2N7HHTP8BSFzq4tIEXT5J5yRVfMxNLv254QMNAENz6P42C04KSjSCKSGyd14ZmdAl6Ra0WsGpv6IJ2xwN6hLc03cinnVmd3MzVp/+tELSzinJqbCtSlFha2SLvYk9raxb0cO5R2mSNrbH3rT00crQqmFfrl6UWzFJ5ybf0Lu+/lhw8zARKGBiC2Z2YGoP9j6lj2b2YGr336OpHegZci2zgFXzQ3mirRMuNiaV7E+SCV164AkhUGvVFGgKKFIXUagupEBTQKG6kCJNEQXq0ueFmkIK1YXkleSVjlVfy8U+tCUGhQ5YpUUQZ7yGt/pnUZylwD9gaWCJvYk9Tc2aEmgfWGEYxN7YHltjW3RVDfC/mBBQkHFzUq7ssTj35u1VetcTsh1YuoJzh+uJ2q7io4kt6Nz5+7NwdxQCwaSQB+CcQA1pgP/apAeFEIISbclNCbXs8Xri/TcJF2oKy9qWJeJybSt9fv3xxnHpW1FpVTwS1w/vuB7oqgtpmrqGjKea4N52Gl9fT9p2xnbVf5PjmqbVQF7K7ZN0bhJoim/eXs/kv2TcxLeSJH39ubEVVPO3jtj0fH45GsvIIGecLI2rdd8PEpnQ69CxLYtQXdh6+4b1lECQptIQp1tMvG4JeYqGYkVQpAiK//1BW2FZiaItfU7p7+Ie/t/rCDAQCvpChb5Q0EcpfRQqDISClVDQEwoGKOgLHfSF6fX1CgZCVa59xe30hYKm0J6kuD4UCDsapxzHVe8vVB0MsL+SAFdOAJB2/ach0BNFmGvSMFenY6bJRIX2pjZ5KjOydK3I1rEmS9eTLLNHyv1uTZaOFVm61hSpyiXSous/Fd6IQiDm+k/1upyah0qlMLG77J3fikzodSQ1MRav4zPIV4zIVdX/ubQ5KojRgyv61x+vPy8odzN6PQEGWjAQoC9KHw0EGGqgUbnfDcq1K/+jX8ly/Ura6wAgKC29Xz3UWl0uJvUjRtsD/eJsfOO/xbHVOVTWCpRQ+tMAlaBLgsqKcypX0nUtSVesSFdZXH+0JF2xpES54QYVgkpesxaoZHilFk3r7Ym9eQP7VlTLZEKvI1EbPyCQYlJHbcfFo01dh1OmUF1IdFY0kRmRpT+ZpY8pBSllbRrpN8Ld0p3BFu64W7rjYelBC4sWDXI2hhCCS+v3sG97Bvk65jhmHqd9iC12Y5aj0q/DO/FI0j2QCb0OJMdfxj9xAycs+xJYR8lco9UQmxNLZGYkURlRZYn7as5VtKL0a7m+Sp8WFi3o6NAR9+vJ293SHVsj24Y9MwMQWi1pf/zNwXURxBr7YKTR0sM/A8/nJ6MyeEDmdEsPHZnQ68DljbNpixbHwTNr/FhCCFILUiv0tiMzI4nOjKZQUwiAgoJzI2fcLNzo69q3LHk7mzmjo9Kp8Rhrk1Cryf7jD8KX7eCsWTeKjLxp2ayI4EkD0DeVX+elhk0m9FqWcOUC/imbOWEzkPauLat133kleRUSd1RmFJEZkWQWZZa1sTGywd3CnSc9n8TdonS4pLlFc4x0jao1lvpGW1xM1sZNJCxZyXnjjiTZDcHcTMuAl9rSxK1+3XRCku6VTOi1LHbTLKxRaPbY9Ns3rkKJtoSYrJibknd8bnxZG2NdY9ws3QhxDikdKrne67Y0fLiSl7aggMx160j9cQnXtI5EthpHiY4RAf2aEdDPFR091e13IkkNhEzotSgu6ixt0//keOOhtG96++lXQgiu5V0rG+O+mHGRyIxIYrJjUGvVAOgquriYu+Br48vj7o/jZuGGu6U7DqYOqJSHN1lpcnPJWLWa9KVLyc/VEBU0niRdZ2ybmdHjmVbYON1/IS1Jqm9kQq9FiVtmYo0uLYbc3DvPLMysMMb9b687rySvrE0Tkya4W7rT1akrbpZuuFu442ruir6OnI3xL3VGBhnLl5O+YiWa7GzSuowm3CAILSoeebQ5fiFOqHQe3g866cEmE3otuRJxgrZZOzjSZBQdHJoBsPPqTn658EuV0wIfbf5o2cwSNwu3BjktsLaoU1JI+2kpGWvWIPLzUfUcTITdABLiS3BwsaD7My2xaCyvMJQebDKh15LU32dhgwEeQ98BIDk/mbf2vYWVoVWFaYFuFm40Nm7c4KcF1paSa9dI++FHMtevR6jVmPUfQGK7ERw/mI2SpqXbU554dXJAUcn3U3rwyYReCy6dOUy73FAOOT1Hx8aOAHx14ivUWjXf9/qepo2a1nGEDU9xTAyp339P1uYtoChYPDYYZfAz7NuRTfKeTJr5WNNtlCemlnIqovTwkAm9FmT/OYtsjPF6vLR3fj7tPFsubWF069Eymd+lwgsXSfv2W7K3bUPR08NyxAgsxjzH6ZNFHP8pBn0jXXo974V7gJ38liM9dGRCr2GR/+zFP/8gh5qNp6OVLUII5h+dj4WBBeN8x9V1eA1GwZkzpC7+ltydO1EZG2M99jmsxowhLUefjUvCSb+Wh3ugHcHD3DEykyeJpYeTTOg1rOCv2WRiivfQNwDYdXUXx5KO8W77d+VJzjuQf/QoqYu/Je/AAVTm5thMnIjVM0+jNTYjbEs0p3bGYmxuQP8Jvrj62tR1uJJUp2RCr0ERYdvxLTzK4eaT6WBuRYmmhE+Pf0oL8xY87vF4XYdXbwkhyNt/gNTFiyk4fhwda2sav/Y/LEaMRMfUhLgLGexeHkZ2aiGtgx3oONQNAyP5T1mS5P+CGqTZ+QFpmOM79DUAVkWsIjYnlkU9FzXMO9nUMKHVkrtrF6mLFlN47hy69vbYvfMOFk8+gcrQkKICNXtXRnB+3zUa2Rox+FV/nDwfritfJelWZFapIWcP/IZ38SkOe7xGB1NzMgoz+PbUt3Ry7ERnx851HV69IjQasv/cRtq3iymKjELP2Rn792djMXgwyvUStpdPp7JnZQT52cW06eVM0KOu6Ok/WIXDJOl+3VFCVxSlL/AlpfcW+EEIMaeKdo8D64FAIcSxaouygRFaLTqhH5GMFW2GTAVg0alF5Kvzea3da3UcXf0hiovJ2rKF1O+/p+TKVfTdWuAwfz6N+vVF0S39p1mQU8y+XyKJPJqElYMJ/cb7Yuda/28IIkl14bYJXVEUHeBroBcQBxxVFGWLEOL8De3MgClAWE0E2pCc2fMrviXnCfN6h/ZGJkRnRvPLhV94wuMJ3CzlLbS0hYVkrt9A2o8/ok5IwNDLi8YLvsIsJARFVXpZvhCCyKNJ7FsbSXGhmqBHXWnbpxk6uvKyfUmqyp300IOAKCFENICiKGuAwcD5G9q9D8wFplVrhA2M0GoxOjCHBGzxHzwZgE+OfYKRrhET2kyo4+jqliY3j8y1a0j7aSma1FSM2ralyexZmHTuXGHOeE56IXtWX+DKmTTsXBvR/ZmWWDvIYlqSdDt3ktAdgdhyv8cB7cs3UBSlLdBUCPGHoihVJnRFUcYB4wCcnZ3vPtoG4OSOVfirIznqO5smBoYcjD/Ivvh9TG03FStDq7oOr05osrJIX76C9OXL0WZlYfLII1h/9inGgYEVErnQCs7tv8bBX6MQGkGnJ9zw7dEUlbxsX5LuyH2fFFUURQV8Boy5XVshxHfAdwABAQHifo9d32g1GswPzydOaYL/oy+j1qqZf2w+jqaOPNXqqboOr9ap09JIX7qUjFWr0eblYdqjBzbjX8LI1/emtplJ+exeEcG1yEwcPS3p/nRLzG0f7JtuSFJ1u5OEHg+Uvz7d6fqyf5kB3kDo9d6WPbBFUZRBD9uJ0X/+WkY7bQzH2s7FSU+fdRfXEZUZxaddP32oStyWJCaS9uMSMtetQxQV0ahfX6xfeglDT8+b2mo1Wk7tjCPst2h0dBS6P92SVp2ayMv2Jeke3ElCPwq4K4riSmkiHwGM+nelECILKLtET1GUUOC1hy2Za9RqbI59SoyqKf79XyC3OJeF/yykbeO29GrWq67DqxXFV6+S9v33ZG7aDEJgPmgQ1i++gIGra6XtU+Ny2b08nOQrObj42tB1pCemlvIGzZJ0r26b0IUQakVRXgH+onTa4hIhxDlFUWYDx4QQW2o6yIbgn63fE6CN5UT7L3DR1eX749+TXpjONyHfPPC9TW1+PokffkjWxk0ourpYPvkE1s8/j56jY6XtNSVajv0Zw4ltVzAw0aX3C61xaydLBkvS/bqjMXQhxFZg6w3LKr0pphCi2/2H1bCUFBdh/8+XXNJxpU3vZ4nLiWP5+eU82vxRWtu0ruvwapQ6NZXYlydQeO4cVs88g9XzY9Fr3LjK9onRWexaHkFGQh4e7e3o/KQ7RqYPz3CUJNUkeaVoNfjnt0UEiQROdliESkeHL058gY6iw+S2k+s6tBpVFH2Z2HHjUKem4rRwAWY9elTZtqRIQ9jmaE7tjsXUwoABE31x8ZHFtCSpOsmEfp+KCvNxPrOQi7oe+IWM4J/kf/gr5i/G+43H3sS+rsOrMfnHjxM3YSLo6tJs2c+Vzlz5V2xEOqErIshOLeT/27vv+KjKfI/jn196D+mkEHqvCaGJgA1FZcFyXfWKlSsq6KqIimXFZV1F8O7asKAoCCw2EFhFXNeEKiq9hpBQkwBJIBASEtLmuX9k9EYECWGGk5n83q+XL6ecmfmeaL4cznnmeboMjKff9a3x0cm0lHI4/a06TxsXvkEfCsi/+GWMwJQ1U4j2j+buzndbHc1pjn/9NQeeHI93XBzN3puGO9sGZgAAF+5JREFUT7PTL9JRXlrJqnlZpK86SGi0P9c/lkRcW51MSyln0UI/DydLS2iZ/g7p3p3pOvB6vtqzmC2Ht/BC/xcI8Ha/BYmNMRR+8CH5U6bg37MnCW++gVfY6Qt698YCls3NoOx4BUlXJtJ7aEu8dDItpZxKC/08bFzwD/pSSMElb3LSVs6r616lY3hH/tD6D1ZHczhTXU3eiy9xdM4cgq8eQtykSXj4/naIYenxCpZ/vJNd6/OJiA/i2tHdiG6uk2kpdSFooddTaUkRbXe+x1bfHnTpfy3vbnqXvNI8Jg2YhIe41wRStrIych8bR0lqKuH33EP0uMd+mUTrZ8YYdv54iBWfZVJZXk2fYa1IuioRT0/3+lko1ZBpodfTpvmv0I8iCi5/lvzSfKZvnc4ViVeQ0jTF6mgOVXXkSM2wxK1bifnzs4Tf9tspDIoLT7J0Tgb7tx2haasQLr29I+GxgRakVapx00Kvh+KiQjru/oBNfr3o3nswz616jkpbJY/2fNTqaA51tmGJxmbYujyX1V/swhjDxTe1peulCTqZllIW0UKvh63zJtGPEgqGTCD9SDoLshZwR6c7SAxxnxkkfxmW6Ol52mGJx/JKSZ2VzsGsIhI61EymFRKpk2kpZSUt9HNUdCSPLvs+YkNgf3p0v5iR/x5JqG8oo7qPsjqawxxfsoQDTzx52mGJtmobG/+TzU//2oOXjweX3dGBDv10Mi2lGgIt9HO0ff6L9JMyQq+ZQFp2GmsOreHpPk8T4uP6IzmMMRR+OIP8yZPxT04mYeqbvxqWWJBdTNqsHRTsL6ZVjygG3tqOwFCdTEuphkIL/RwU5ufSPWcu60IupVvHZB5aeB2tQltxU7ubrI523n41LHHIEOJe/v9hiVWV1az9ai/r/70fvyBvhozqQuvkM8/XopSyhhb6Odg5/2/0ooLIoc8zd8dc9hfvZ+rlU/HycO0fo62sjNxxj1Py3Xe/GZZ4cFcRabPSOXqolA59m9L/prb4BXpbnFgpdTqu3UQX0OED++hx8FPWN7mSts1b8M4Xo7ko7iIGxA+wOtp5+WVY4pYtxDz7LOEjaoYlVpys4oeFu9myNIegMF/+8FB3EjtHWJxWKfV7tNDraNcXE+lJNXHDJ/D2prc5UXmCcSnjXPpiYPmePWSPuo+qgoKaYYmXXw7A/u1HWDo7g+LCk3S9JIG+17XCx0//V1GqodPf0jo4tD+TpPwFrA+/hsgIfz5Z+Qk3tr2RtmFtrY5Wb6Xr15PzwOiaYYkzZ+DfvTsnT1Sy6vNMdqw+RJOYAK4fl0xcmyZWR1VK1ZEWeh3sWzCRcCDx+ud5Ye3f8ffyZ0yPMVbHqrfjS77hwBNP4B0bWzMsMTGRXRvyWT53J2UllSQPaU6va1vg5a2TaSnlSrTQzyJ3dzrJR75ifdRwbF6HWZazjEd7PkqEv+udTzbGUDhjZs2wxB49SHhrKuUeAaS+u4VdGwqIbBbE0Ae7E5UYbHVUpVQ9aKGfxYGFzxOBB82vf5bRax8nPiie2zr+dj6Ths5UV5P30iSOzp5N8FVXETvpJTI3HmPlZ1uoqrDR97pW9Bisk2kp5cq00H/H/p0bST72DWua3kLOiQ1kHs3klUGv4OvpWl+m+dWwxLvvxu+eMXz1XgbZ2wuJbR3Kpbd3IKypTqallKvTQv8d+f+aSCQ+xA17lCdXjiIpOokrm19pdaxzUntYYvTTz5DbbBCrX1gLwICb29F1UDyik2kp5Ra00M9gz/Y1JB9P5cf4O/gpbwmFJwuZevlUlxqmWHtYYvDfXmfZrjAOfp9JYqdwBt3WnpAInUxLKXeihX4GR7/6C5H4EXb1SD5aeg9DWw2lS2QXq2PV2c/DEm2e3hz701ukpZ7E2+cEl9/VkfZ9mrrUH0xKqbrRQj+NrE0rST6xgtWJ9zJ/92w8xIOHkx+2Olad/Twssax5dzKSR3HkpzJaJ0cx8Jb2BIT4WB1PKeUkWuincWLJRIoIxFw6jCUrxnBft/toGtjU6lhn9fOwxINT/k5O77vYHdAD/zIYcl8XWifpZFpKuTst9FPsWPsd3ct+ZFXL0byV/i5R/lHc0+Ueq2OdlamuJm/Sy+xZ9D0ZA1/gBEF06BtL/xvb6GRaSjUSWuinqPz2rxQSQn6vnmxe8yUTL5pIgHeA1bF+l62sjH3jnmLT/lByk8YSHOHHsNs60KxTuNXRlFIXkBZ6Ldu+X0zX8g0sb/Mwb21/l47hHRneZrjVsX5XVWEh6x96kc3eF1MeH0a3SxPoM1wn01KqMdLfejtjsyFpf6OAMLZ2jObQ1kO8ePGLeEjD/eZk0fYsUv/2FQeCryE02HDt/SnEtg61OpZSyiJa6HZbVy6ka+VW/t3hEWbs+IjLml1Gr6a9rI51WsYY0uf9xKqv86gM6kT3ZD/63d0XT++G+4ePUsr56lToIjIEeA3wBN43xkw65fmxwP8AVUABcI8xZp+DszqNsdnwWf4Sh4hkeVwVlfsqGZsy1upYp3WiqJzvXl1B9kEPQmwlDP2fjsT2ct1pfJVSjnPWQhcRT2AqMBjIAdaIyCJjzPZam20AUowxpSLyADAZuNkZgZ1hU9qn9KjK4PNOD7FozyJGdBpB85DmVsf6FWMMO1YfZMWc7VRXVNGhYgMD/vc+fCL1wqdSqkZdjtB7A1nGmN0AIvIxMBz4pdCNMWm1tv8BGOHIkM5kbDaCvp9MjsSwOOQAobZQ7ut2n9WxfuX44TLSZqeTs+MYocd2kxK1l3ZvPIeHn5/V0ZRSDUhdCj0eyK51Pwfo8zvbjwS+Pt0TIjIKGAWQmJhYx4jOtfHbWSRV72Ja51GsyVvCU72fItS3YVxYtNkMW5bm8MOCXZiKctrtnEfXK1oRM/6FXxZxVkqpnzn0oqiIjABSgEGne94YMw2YBpCSkmIc+dn1UV1VRdiPr5DlkcBCz+20DG3JTe1vsjoWAIUHTpA2O51Du48TVbmftuveo/nY+wm/43aroymlGqi6FHou0KzW/QT7Y78iIlcAzwCDjDHljonnXBuWfECKbT8vdRnB/uLlTL18Kt4e1n6rsrraxoZv9rFm8V68vYWu+f8ialca8a9MIWTwYEuzKaUatroU+hqgrYi0pKbIbwH+u/YGIpIEvAsMMcbkOzylE1RVVhCz7h9s9mrOl5Ub6RfbjwHxAyzNlL/vOKkf7eBIbgktW3uTuPAv+FSX0mzmDPx79LA0m1Kq4TvriVhjTBXwIPANkA58aozZJiITRWSYfbMpQBDwmYhsFJFFTkvsIOu/fJdm5gAz2ydTUlnCuF7jLJtStqqimu/nZ/H5pLWUlVRwSZ9qWs1+CL9AT1p8PFfLXClVJ3U6h26MWQwsPuWx52rdvsLBuZyqovwkCZvfYKlva1LLNnFD2xtoF9bOkiwHMo+SOmsHRflldOofS8eKNRydPAm/7t1JePstvMLCLMmllHI9jfKbohsWvUkfk8f41l3xrcplTI8xFzxDRVkVq7/YxdbluYRE+jHsoW54L5zG0Y9mETx4MHFTJuuwRKXUOWl0hX6y7AQttr3FZ4Ht2FC2k4eTHybSP/KCZti75TDL/plBybFyul/WjF5XxVHw7HiOfvsfwu+8g+gnnkA8PS9oJqWU62t0hb5xwWv04ggzEloT5+3B7Z0u3DDAspIKVn6Wyc4f8wiLDeTGJ7oQGVpNzqiRlG3eTMzTTxF+xx0XLI9Syr00qkIvO1FMm4x3eatJe/ZXHGJK3yn4evo6/XONMWSty2fFJzspP1FFr2tb0HNIC6py9rH31vupOpRH/GuvEnLllU7PopRyX42q0Dd98QpdpYhPIuPpEd6Dq1pc5fTPPHGsnKX/zGDv5sNENw9m+CMdCQvz4MjbUzkyfToeAQEkzviQgKQkp2dRSrm3RlPoJceP0j5rOi9FtaeoupjHez3u1GGKxhjSVx1k1bwsqqtsXHRjG7pdGk/JN0vYNeUVqg4dIuSaa4h+fBzesbFOy6GUajwaTaFvmf8yzb1KWRxcwTUtrqFbVDenfVZRQSlpszPIzThKfLsmXDKiA74Fe8i+807K1q3Dt1NH4l+ZQkBKitMyKKUan0ZR6EVHD9N570c81rQNHh42Hkl+xCmfY7MZNqdm8+PC3Xh4Cpfc1p52HXw5/PpLHPx8Hp5hYTT960Sa3HCDjmJRSjlcoyj09Hkv4u9byQ/+ZYzqPIrYIMef4jiSW0LqrB3k7z1Oi64RDPyvVlR8PY/dY9/CVlZG+J13Ejn6ATxDQhz+2UopBY2g0I8WHKRz9j+5vVlzIv2DGdllpEPfv7rKxrol+1j39V58/L0YPLITsWU7yb/rZir27CFwwABinhqPb6tWDv1cpZQ6ldsX+o75L1AYKOzyLmdi0jMEeAc47L3z9h4n9aN0Cg+coG2vGHr39aX49efJWbYMn+bNSXjnbYIGDbJsjhilVOPi1oV++FA27Q9+xg2J8XQIb8Ow1sPO/qI6qKyo5qdFu9n0XTYBob5cfU9bApd+zMFbZuHh40P0448TfvsIxMfHIZ+nlFJ14daFnjX/r2wK9eOIZxWTUx7H0+P8L0TmZBwlbVY6xw+fpNPFsXT22s6xR5+ksLCQ0BuuJ/qRR/CKinJAeqWUOjduW+h5ObtodngBDybGcmmzQfSO7X1e71deVsX387PYvuIAIVH+DBkahMeHf+bItm349+hBzDvv4N+1i4PSK6XUuXPbQt/7xUS+DAuiykMY23Pseb3Xns2HWTZnB6XHK+h2UQSJm+dSOm4hXtHRxE2ZTMjQoXqeXCllObcs9AN7Mwg8/jUL42O4rcOttAhtUa/3KSuuYMWnmWSuySM8NoB+MZmYf4yjrLqaiPvvI/Lee/EIDHRseKWUqie3LPTsBROYFt6EIJ9g7u9+/zm/3hhD5po8VnySScXJKrp3rCZ60XNU52YTPHgw0U8+gU9CghOSK6VU/bldoWdnbaGkfBk/hUUyPmkMob6h5/T64sKTLJubwb4tR4hq6kOnwoV4vv0NXm3bEj/jQwL79nVScqWUOj9uV+i5iybw9/AmNAuI54/t/1jn1xmbYdvKA3w/PwtTbegWspuIT1/FMziIqD8/S9jNNyNebvfjUkq5EbdqqH3p68g0P7HfJ4w3+o7H28O7Tq87lldK2uwdHMg8RkyTctqseh3fwmzCbr2FyAfH6LqeSimX4F6Fvvg53g4LpWdEEoMSBp11e1u1jY3fZfPTv/bggY0uR78laukCAvv2JWb6a/i1s2bhaKWUqg+3KfRdm79ntfdWSjxCeOqiZ846jPBwTglps9LJ31dMUzlA61VvEhQdTPQbrxN8xRU6DFEp5XLcptAzvn2OuRHBDG1+Le3D259xu+pKG2u/3sv6Jfvwlkq6ZMwh+tgWokbfR/jdd+Hh6/wl6ZRSyhncotB3rl/GEv89eBHM2L7jzrjdod1FpM5K5+jBUmKPbaLN1jlEXHMZ0Y9Nxjsm5gImVkopx3OLQl+39FnSIgK4v9NIIv0jf/N8ZXk1Py7czabUbPxsJXTfOpP4OA9iZr2na3kqpdyGyxf6th++Zl5QPuGEMjLp3t88n51eSNrMbRQfqyQ+dxntilYR99hoQq+/DvHwsCCxUko5h8sXetoPE8gI8+GFPuPx8/L75fHy0kpWfrqTHT/kEXCygKTMubS+rj+RDyzAMyjIwsRKKeUcLl3oa5Z/xrzgElpJNMPaX//L47s25LNs5lZOltlIzP4PXROLiZvzKj4tWlgXVimlnMxlC93YbHy56SUOh3gy+bKXERFKj1ewdPp69mSUElSSQ9/iNNo9dQ9BAwZYHVcppZzOZQs99bsP+CqonN6erUmJ70162h5WfpZJZRW0PvgfUm7oTMSIaYh33b4tqpRSrs4lC93YbMzLfAPjL4zrM5kvnvmGg4U+hBTtZ2CLw7R+6Um8IiKsjqmUUhdUnQpdRIYArwGewPvGmEmnPO8LfAT0BI4ANxtj9jo26v+bt/gVVgQYRu68gWWr94IxdKpaTZ9nriOgS2dnfaxSSjVoZy10EfEEpgKDgRxgjYgsMsZsr7XZSOCoMaaNiNwCvAzc7IzA1VVVpG5ewuisB7H5tKFJ2W4GDo8n9san9ev6SqlGrS5H6L2BLGPMbgAR+RgYDtQu9OHA8/bbnwNviogYY4wDswIw98m/0qX4STw8KukZl0vPcf+NZ0CAoz9GKaVcTl0KPR7IrnU/B+hzpm2MMVUiUgREAIdrbyQio4BRAImJifULHOWB7+FtDHv8D0R1aVuv91BKKXd0QS+KGmOmAdMAUlJS6nX0fsv4CQ7NpJRS7qIu333PBZrVup9gf+y024iIFxBKzcVRpZRSF0hdCn0N0FZEWoqID3ALsOiUbRYBd9pv/xeQ6ozz50oppc7srKdc7OfEHwS+oWbY4gfGmG0iMhFYa4xZBEwHZolIFlBITekrpZS6gOp0Dt0YsxhYfMpjz9W6fRK4ybHRlFJKnQudP1YppdyEFrpSSrkJLXSllHITWuhKKeUmxKrRhSJSAOyr58sjOeVbqC5M96XhcZf9AN2Xhup89qW5MSbqdE9YVujnQ0TWGmNSrM7hCLovDY+77AfovjRUztoXPeWilFJuQgtdKaXchKsW+jSrAziQ7kvD4y77AbovDZVT9sUlz6ErpZT6LVc9QldKKXUKLXSllHITLlfoIjJERDJEJEtExludp75E5AMRyReRrVZnOR8i0kxE0kRku4hsE5GHrc5UXyLiJyI/icgm+778xepM50tEPEVkg4h8aXWW8yEie0Vki4hsFJG1VuepLxFpIiKfi8gOEUkXkX4OfX9XOoduX7B6J7UWrAZuPWXBapcgIgOBEuAjY0wXq/PUl4jEArHGmPUiEgysA65z0f8mAgQaY0pExBtYCTxsjPnB4mj1JiJjgRQgxBgz1Oo89SUie4EUY4xLf7FIRGYCK4wx79vXlwgwxhxz1Pu72hH6LwtWG2MqgJ8XrHY5xpjl1Mwd79KMMQeNMevtt4uBdGrWmHU5pkaJ/a63/R/XOeI5hYgkANcC71udRYGIhAIDqVk/AmNMhSPLHFyv0E+3YLVLloc7EpEWQBLwo7VJ6s9+imIjkA98a4xx2X0BXgWeAGxWB3EAA/xbRNbZF5t3RS2BAuBD+2mw90Uk0JEf4GqFrhooEQkC5gGPGGOOW52nvowx1caYHtSsndtbRFzydJiIDAXyjTHrrM7iIBcbY5KBq4Ex9lOWrsYLSAbeNsYkAScAh14HdLVCr8uC1eoCs59vngfMMcbMtzqPI9j/KpwGDLE6Sz31B4bZzz1/DFwmIrOtjVR/xphc+7/zgS+oOf3qanKAnFp/6/ucmoJ3GFcr9LosWK0uIPuFxOlAujHm71bnOR8iEiUiTey3/am5+L7D2lT1Y4x5yhiTYIxpQc3vSaoxZoTFsepFRALtF9yxn6K4EnC50WHGmENAtoi0tz90OeDQwQN1WlO0oTjTgtUWx6oXEZkLXAJEikgOMMEYM93aVPXSH7gd2GI/9wzwtH0dWlcTC8y0j6byAD41xrj0cD83EQN8UXPsgBfwT2PMEmsj1dtDwBz7Aelu4G5HvrlLDVtUSil1Zq52ykUppdQZaKErpZSb0EJXSik3oYWulFJuQgtdKaXchBa6apRE5BERCXDUdko1BDpsUTVKdZ29z11m+VONgx6hK7dn/6bhV/Z5zreKyAQgDkgTkTT7Nm+LyNra86CLyJ9Os92VIrJaRNaLyGf2OWyUahD0CF25PRG5ERhijLnXfj8U2EStI28RCTfGFNq/Jfod8CdjzObaR+giEgnMB642xpwQkScBX2PMRCv2S6lT6RG6agy2AINF5GURGWCMKTrNNn8UkfXABqAz0Ok02/S1P77KPs3BnUBzZ4VW6ly51FwuStWHMWaniCQD1wAviMh3tZ8XkZbAOKCXMeaoiMwA/E7zVkLNHOm3OjuzUvWhR+jK7YlIHFBqjJkNTKFmytJiINi+SQg1c1MXiUgMNXNu/6z2dj8A/UWkjf19A0Wk3QXYBaXqRI/QVWPQFZgiIjagEngA6AcsEZEDxphLRWQDNVPlZgOrar122inb3QXMFRFf+/PPUrPOrVKW04uiSinlJvSUi1JKuQktdKWUchNa6Eop5Sa00JVSyk1ooSullJvQQldKKTehha6UUm7i/wC6etKDi7SXlwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DAxclWhdZi4L",
        "colab_type": "text"
      },
      "source": [
        "#Section 2: Introduction to Q-Learning\n",
        "\n",
        "__Grid Worlds:__\n",
        "\n",
        "As pointed out, bandits only have a single state and immediate rewards for our actions. Many problems we are interested in have multiple states and delayed rewards, i.e. we won't know if the choices we made will pay off over time, or which actions we took contributed to the outcomes we observed.\n",
        "\n",
        "To explore these ideas, we turn a common problem setting: the grid world. Grid worlds are simple environments where each state corresponds to a tile on a 2D grid, and the only actions the agent can take are to move up, down, left, or right across the grid tiles. The agent's job is almost always to find a way to a goal tile in the most direct way possible while overcoming some maze or other obstacles, either static or dynamic.\n",
        "\n",
        "For our discussion, we will be looking at the classic Cliff World, or Cliff Walker, environment. This is a 4x12 grid with a starting position in the lower-left and the goal position in the lower-right. Every tile between these two is the \"cliff.\" If the agent enters the cliff, they will receive a -100 reward and be sent back to the starting position. Every tile other than the cliff produces a -1 reward when entered. The goal tile ends the episode after taking any action from it.\n",
        "\n",
        "<img alt=\"CliffWorld\" width=\"577\" height=\"308\" src=\"https://github.com/NeuromatchAcademy/course-content/blob/master/tutorials/W2D5_ReinforcementLearning/static/W2D5_Tutorial3_CliffWorld.png?raw=true\">\n",
        "\n",
        "Given these conditions, the maximum achievable reward is -12 (1 up, 11 right, 1 down). Using negative rewards is a common technique to encourage the agent to move and seek out the goal state as fast as possible.\n",
        "\n",
        "---\n",
        "__Q-Learning:__\n",
        "\n",
        "Now that we have our environment, how can we solve it? \n",
        "\n",
        "One of the most famous algorithms for estimating action values (aka Q-values) is the **Off-policy** Temporal Differences (TD) **control** algorithm known as *Q-learning* (Watkins, 1989). \n",
        "\n",
        "\\begin{align}\n",
        "Q(S_t,A_t) \\leftarrow Q(S_t,A_t) + \\alpha \\big(R_t + \\gamma\\max_\\limits{a} Q(S_{t+1},a) - Q(S_t,A_t)\\big)\n",
        "\\end{align}\n",
        "\n",
        "where $Q(s,a)$ is the value function for action $a$ at state $s$, $\\alpha$ is the learning rate, $r$ is the reward, and $\\gamma$ is the temporal discount rate.\n",
        "\n",
        "The expression $R_t + \\gamma\\max_\\limits{a} Q(S_{t+1},a)$ is referred to as the TD target while the full expression \n",
        "\\begin{align}\n",
        "R_t + \\gamma\\max_\\limits{a} Q(S_{t+1},a) - Q(S_t,A_t),\n",
        "\\end{align}\n",
        "i.e. the difference between the TD target and the current Q-value, is referred to as the TD error, or reward prediction error.\n",
        "\n",
        "Because the max operator used to select the optimal Q-value in the TD target, Q-learning directly estimates the optimal action value, i.e. the cumulative future reward that would be obtained if the agent behaved optimally, independent of the policy currently being followed by the agent. For this reason, Q-learning is referred to as an **off-policy** method."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZyApRXxM5Uzn",
        "colab_type": "text"
      },
      "source": [
        "##Exercise 2: Implementing Q-Learning\n",
        "\n",
        "In this exercise you will implement the Q-learning update rule described above. \n",
        "\n",
        "Please read the environment setup carefully."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P21tWpOnC7jU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Environment Setup\n",
        "\n",
        "# world height\n",
        "WORLD_HEIGHT = 4\n",
        "\n",
        "# world width\n",
        "WORLD_WIDTH = 12\n",
        "\n",
        "# probability for exploration\n",
        "EPSILON = 0.1\n",
        "\n",
        "# step size\n",
        "ALPHA = 0.5\n",
        "\n",
        "# gamma for Q-Learning and Expected Sarsa\n",
        "GAMMA = 1\n",
        "\n",
        "# all possible actions\n",
        "ACTION_UP = 0\n",
        "ACTION_DOWN = 1\n",
        "ACTION_LEFT = 2\n",
        "ACTION_RIGHT = 3\n",
        "actions = [ACTION_UP, ACTION_DOWN, ACTION_LEFT, ACTION_RIGHT]\n",
        "\n",
        "# initial state action pair values\n",
        "stateActionValues = np.zeros((WORLD_HEIGHT, WORLD_WIDTH, 4))\n",
        "startState = [3, 0]\n",
        "goalState = [3, 11]\n",
        "\n",
        "# reward for each action in each state\n",
        "actionRewards = np.zeros((WORLD_HEIGHT, WORLD_WIDTH, 4))\n",
        "actionRewards[:, :, :] = -1.0\n",
        "#walking thorough Cliff\n",
        "actionRewards[2, 1:11, ACTION_DOWN] = -100.0\n",
        "actionRewards[3, 0, ACTION_RIGHT] = -100.0\n",
        "\n",
        "# set up destinations for each action in each state\n",
        "#(determine next states after taking all posible actions)\n",
        "actionDestination = []\n",
        "for i in range(0, WORLD_HEIGHT):\n",
        "    actionDestination.append([])\n",
        "    for j in range(0, WORLD_WIDTH):\n",
        "        destinaion = dict()\n",
        "        destinaion[ACTION_UP] = [max(i - 1, 0), j]\n",
        "        destinaion[ACTION_LEFT] = [i, max(j - 1, 0)]\n",
        "        destinaion[ACTION_RIGHT] = [i, min(j + 1, WORLD_WIDTH - 1)]\n",
        "        if i == 2 and 1 <= j <= 10:\n",
        "            destinaion[ACTION_DOWN] = startState\n",
        "        else:\n",
        "            destinaion[ACTION_DOWN] = [min(i + 1, WORLD_HEIGHT - 1), j]\n",
        "        actionDestination[-1].append(destinaion)\n",
        "actionDestination[3][0][ACTION_RIGHT] = startState"
      ],
      "execution_count": 32,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AVrNhdTDkXeP",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title Helper Function\n",
        "\n",
        "from __future__ import print_function\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "# print optimal policy\n",
        "def printOptimalPolicy(stateActionValues):\n",
        "    optimalPolicy = []\n",
        "    for i in range(0, WORLD_HEIGHT):\n",
        "        optimalPolicy.append([])\n",
        "        for j in range(0, WORLD_WIDTH):\n",
        "            if [i, j] == goalState:\n",
        "                optimalPolicy[-1].append('G')\n",
        "                continue\n",
        "            bestAction = np.argmax(stateActionValues[i, j, :])\n",
        "            if bestAction == ACTION_UP:\n",
        "                optimalPolicy[-1].append('U')\n",
        "            elif bestAction == ACTION_DOWN:\n",
        "                optimalPolicy[-1].append('D')\n",
        "            elif bestAction == ACTION_LEFT:\n",
        "                optimalPolicy[-1].append('L')\n",
        "            elif bestAction == ACTION_RIGHT:\n",
        "                optimalPolicy[-1].append('R')\n",
        "    for row in optimalPolicy:\n",
        "        print(row)\n",
        "\n",
        "# choose an action based on epsilon greedy algorithm\n",
        "def chooseAction(state, stateActionValues):\n",
        "    if np.random.binomial(1, EPSILON) == 1:\n",
        "        return np.random.choice(actions)\n",
        "    else:\n",
        "        values_ = stateActionValues[state[0], state[1], :]\n",
        "        return np.random.choice([action_ for action_, value_ in enumerate(values_) if value_ == np.max(values_)])\n",
        "\n",
        "def takeAction(currentState, currentAction):\n",
        "  newState = actionDestination[currentState[0]][currentState[1]][currentAction]\n",
        "  reward = actionRewards[currentState[0], currentState[1], currentAction]\n",
        "  return newState, reward\n"
      ],
      "execution_count": 33,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0xrE9HJz05bL",
        "colab_type": "text"
      },
      "source": [
        "You can use ``` chooseAction(state, stateActionValues) ``` and ``` takeAction(state, action) ``` and ``` np.max([...]) ```.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2q0TueZB5QD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# an episode with Q-Learning\n",
        "# @stateActionValues: values for state action pair, will be updated\n",
        "# @stepSize: step size for updating\n",
        "# @return: total rewards within this episode\n",
        "def qLearning(stateActionValues, stepSize=ALPHA):\n",
        "    currentState = startState\n",
        "    rewards = 0.0\n",
        "    while currentState != goalState:\n",
        "        currentAction = ...\n",
        "        newState, reward = ...\n",
        "        rewards += reward\n",
        "        # Q-Learning update\n",
        "        stateActionValues[currentState[0], currentState[1], currentAction] = ...\n",
        "        #going to the nest state and continueing the episode\n",
        "        ...\n",
        "    return rewards\n"
      ],
      "execution_count": 34,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pbJvlJ_8ggBK",
        "colab_type": "text"
      },
      "source": [
        "#Section 3: Introduction to SARSA\n",
        "\n",
        "An alternative to Q-learning, the SARSA algorithm, also estimates action values. However, rather than estimating the optimal (off-policy) values, SARSA estimates the **on-policy** action value (On-policy TD Control), i.e. the cumulative future reward that would be obtained if the agent behaved according to its current beliefs.\n",
        "\n",
        "\\begin{align}\n",
        "Q(S_t,A_t) \\leftarrow Q(S_t,A_t) + \\alpha \\big(R_t + \\gamma Q(S_{t+1},A_{t+1}) - Q(S_t,A_t)\\big)\n",
        "\\end{align}\n",
        "\n",
        "where, once again, $Q(s,a)$ is the value function for action $a$ at state $s$, $\\alpha$ is the learning rate, $R-{t}$ is the reward, and $\\gamma$ is the temporal discount rate.\n",
        "\n",
        "In fact, you will notice that the *only* difference between Q-learning and SARSA is the TD target calculation uses the policy to select the next action (in our case, epsilon-greedy) than using the action that maximizes the Q-value."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2Y_Qdad9idSO",
        "colab_type": "text"
      },
      "source": [
        "## Exercise 2: Implementing the SARSA algorithm\n",
        "\n",
        "In this exercise you will implement the SARSA update rule described above. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Ky63cJyCL0T",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# an episode with Sarsa\n",
        "# @stateActionValues: values for state action pair, will be updated\n",
        "# @stepSize: step size for updating\n",
        "# @return: total rewards within this episode\n",
        "def sarsa(stateActionValues, stepSize=ALPHA):\n",
        "    currentState = startState\n",
        "    currentAction = ...\n",
        "    rewards = 0.0\n",
        "    while currentState != goalState:\n",
        "        newState, reward = ...\n",
        "        newAction = ...\n",
        "        rewards += reward\n",
        "        #Q(newState, newAction) using stateActionValues[...]\n",
        "        valueTarget = ...\n",
        "        valueTarget *= GAMMA\n",
        "        # Sarsa update\n",
        "        stateActionValues[currentState[0], currentState[1], currentAction] = ...\n",
        "        currentState = newState\n",
        "        currentAction = newAction\n",
        "    return rewards\n"
      ],
      "execution_count": 35,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Nf_e-pRkrHS",
        "colab_type": "text"
      },
      "source": [
        "Now, we have completed the SARSA and Q-Learning; let's see the difference between these two algorithms!\n",
        "To compare them, we run both algorithms 20 times; each run has 500 episodes. So we can have a smooth learning curve!\n",
        "\n",
        "Please run the code block below and try to ask these questions:\n",
        "\n",
        "1. Which one is faster?\n",
        "2. Which one is optimal?\n",
        "3. Which one comes up with a safer path?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n9njjTvSkVIJ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 447
        },
        "cellView": "form",
        "outputId": "218c2908-5e67-4667-ac80-054c507f0ba6"
      },
      "source": [
        "#@title Comparing Sarsa and Q-Learning on Cliff Walk\n",
        "def learning_curve():\n",
        "    # averaging the reward sums from 10 successive episodes\n",
        "    averageRange = 10\n",
        "\n",
        "    # episodes of each run\n",
        "    nEpisodes = 500\n",
        "\n",
        "    # perform 20 independent runs\n",
        "    runs = 20\n",
        "\n",
        "    rewardsSarsa = np.zeros(nEpisodes)\n",
        "    rewardsQLearning = np.zeros(nEpisodes)\n",
        "    for run in range(0, runs):\n",
        "        stateActionValuesSarsa = np.copy(stateActionValues)\n",
        "        stateActionValuesQLearning = np.copy(stateActionValues)\n",
        "        for i in range(0, nEpisodes):\n",
        "            # cut off the value by -100 to draw the figure more elegantly\n",
        "            rewardsSarsa[i] += max(sarsa(stateActionValuesSarsa), -100)\n",
        "            rewardsQLearning[i] += max(qLearning(stateActionValuesQLearning), -100)\n",
        "\n",
        "    # averaging over independt runs\n",
        "    rewardsSarsa /= runs\n",
        "    rewardsQLearning /= runs\n",
        "\n",
        "    # averaging over successive episodes\n",
        "    smoothedRewardsSarsa = np.copy(rewardsSarsa)\n",
        "    smoothedRewardsQLearning = np.copy(rewardsQLearning)\n",
        "    for i in range(averageRange, nEpisodes):\n",
        "        smoothedRewardsSarsa[i] = np.mean(rewardsSarsa[i - averageRange: i + 1])\n",
        "        smoothedRewardsQLearning[i] = np.mean(rewardsQLearning[i - averageRange: i + 1])\n",
        "\n",
        "    # display optimal policy\n",
        "    print('Sarsa Optimal Policy:')\n",
        "    printOptimalPolicy(stateActionValuesSarsa)\n",
        "    print('Q-Learning Optimal Policy:')\n",
        "    printOptimalPolicy(stateActionValuesQLearning)\n",
        "\n",
        "    # draw reward curves\n",
        "    plt.figure(1)\n",
        "    plt.plot(smoothedRewardsSarsa, label='Sarsa')\n",
        "    plt.plot(smoothedRewardsQLearning, label='Q-Learning')\n",
        "    plt.xlabel('Episodes')\n",
        "    plt.ylabel('Sum of rewards during episode')\n",
        "    plt.legend()\n",
        "\n",
        "learning_curve()\n",
        "plt.show()"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sarsa Optimal Policy:\n",
            "['R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'D']\n",
            "['U', 'U', 'R', 'R', 'U', 'R', 'U', 'U', 'U', 'U', 'R', 'D']\n",
            "['U', 'U', 'U', 'R', 'U', 'U', 'L', 'R', 'R', 'R', 'R', 'D']\n",
            "['U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'G']\n",
            "Q-Learning Optimal Policy:\n",
            "['R', 'U', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'D', 'D']\n",
            "['R', 'D', 'R', 'R', 'D', 'R', 'D', 'R', 'R', 'R', 'D', 'D']\n",
            "['R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'R', 'D']\n",
            "['U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'U', 'G']\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEGCAYAAACZ0MnKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3hUVdrAf28mCekJSYCQhNB7h9ARRbCjqNi72MtadtXVdXfVz3V1LWvvbcVesYvSBOnSpAYIhJIASUggvU3mfH+cO5lJn0AmCcn5Pc88M/fcO/e+mczc97z1iFIKg8FgMBg8wae5BTAYDAbD8YNRGgaDwWDwGKM0DAaDweAxRmkYDAaDwWOM0jAYDAaDx/g2twDeJDo6WnXr1q25xTAYDIbjijVr1hxSSnWoaV+rVhrdunVj9erVzS2GwWAwHFeIyJ7a9hn3lMFgMBg8xigNg8FgMHhMi1MaIvKUiCSJyAYRmS0iEW77HhCRZBHZJiKnNaecBoPB0BZpcUoDmAsMUkoNAbYDDwCIyADgEmAgcDrwiojYmk1Kg8FgaIO0OKWhlPpFKWW3NlcA8dbr6cAnSqkSpVQKkAyMbg4ZDQaDoa3S4pRGFWYCP1mv44B9bvtSrbFKiMiNIrJaRFZnZmY2gYgGg8HQdmiWlFsRmQfE1LDrQaXUN9YxDwJ24MOGnFsp9QbwBkBiYqJp4WswGAyNSLMoDaXU1Lr2i8g1wDRginL1bk8DurgdFm+NGQyGOvh9dzZKwejukc0tiqEV0OLcUyJyOnAfcI5SqtBt17fAJSLSTkS6A72BVc0ho8HQksnIK+bnzQdRSqGU4sLXlnPR68vZf6SouUVr8RwpLOX05xYz83+/U1bu8Pr1DheUMvnpXzn12UV8tHKv16/XGLQ4pQG8BIQCc0VkvYi8BqCU2gx8BmwB5gC3KaXKm09Mg6HlUVxWzunP/cZN76/hvWW72ZGRX7Hvx40HmlGy44Nv/9hP0sE8FiRlMHtd4zsylFIkZ+SRW1wGwEsLk0k5VMD29Hye+jmJ4rL6b2lKKTLyius9xlu0uDYiSqledex7DHisCcUxGJqMQ/klHC4oJauglLE9oo7qHKtSsskuKAXg6V+2E7l0N342oaxcsW7fkcYUt1Xy5ZpU+sWE4msTXl6YzKhukTw3bztxEYHcNbUP/r7HNs/+9o/93PnJegZ0DiM0wJeVKdlcOroLZw2O5Yq3V/LrtgxOH9S5xvcqpViVks2fP/uDtCNFfHXreEYktK923NLkQ1z/3mo+v3kcg+LCj0nemmhxSsNgOFaKSsvZk11Av5iw5hbFY7bsz+W8V5ZSYtcukfvP6MfNJ/as9fhyh+K1RTvZf6SIHRn5JHZtz/Un9GDe1nT8fX146OwBPDh7E/kldj64bgwfr9rLmt2HKSt34GfzvoPhm/VpvPnbLm6f3KvWm2BLIzkjjz9Sc/j7Wf3pGhXMDbNWM/npXyv2F5aW8/A5A2t9f0ZuMdf+73d8RLh4VBfG9oikxO5gQOcwRISM3GKen7cDgC0HcvERuO/0vtxwQg8AQtv5smh7Zq2f1z2fb+DLtakV2w98uZHdWQWs/NsUIoL8K8bf+m0XRWXlPPj1Jr6+dTwiciwfSzWM0jC0Ol5YsINXf93Jh9ePYUKv6OYWxyPmb02nxO5gYq9oVqVk8+7SFG44oQc2n5p/8EuSD/HUz9sqtlelZPPFmlQOF5Zy9pBYZoyIJz2nmDE9opjQK5rCUjs/bDzAk3OSePCsAV75G1IPF/Lhyr30iwnlb19tpKC0nFs+XMtbVyUypX8nr1yzMfl5czoA5wyNpUNoOy4dncCynYd47NzBLEjK4J2lKZzYpwOT+3Ws8f2P/biVzftzAdiYluMaP28QZXYHH6zcy57sQv5+Vn8+WrmX60/owWVjEiqOm9g7mo9X7WNHej63Te5V6Tq/786uUBjXTujG8p1ZJB3MA+CXLelclKhzhHKLy1i2M4tOYe249aTaJx3HgnjT99XcJCYmKtPltu1x7stLWb/vCP1iQnnjykQSooKaW6R6uXHWapIz8llwz0n8uPEAt364ljevSuSUAdVvtlsP5HLG878R5G9jzd9PYcuBHGa8uhyAPp1CmDVzDDHhAdXe989vNjFr+R5umtSDuVvTmTmhO1eM7doo8pfYy5n2wpKKGEpIO1++uX0Cd3y8juSMfL64eTzRof7EhAU06sx3T1YBN72/hpcuG06vjqENfn/KoQL+O3c7YQG+rErJxs/mw493nlDtuFK7g5OeWkj/zmG8fc2oavufm7ed5+bt4MZJPZgxIp7TnluMv68PpXZXMD3Y38arV4xkUp8aO47z245Mrnxb5/bEhAWw9P6TKyYNF7y6jH2HC/nXuYOZ0CuKlxcm8/LCnQD07hjC+9eN4bVFO1mZks3WA7l8fdsEhnWJqPE6niAia5RSiTXta4mBcIPhqHlh/g7WW777pIN5THpqIQdymjZrKD23mIM5dQcq3bGXO1i/70iF//nUAZ2Ibx/IG4t31nj8Byt01+o7p/Qm0N/GyK6RvDdzNJP6dOCLW8bXqDBAu7wigvx4ffEudmUW8Pma1GrHzN2Szv+WpvDest0UlNhrOIumxF5O0kFrVp2aw6xle9iRkc+9p/Vlct8OvHPNKHp2COHhcwZSYndw9ktLGPf4Am6YtYY9WQU1nm/t3sN1f1A18NnqfSQdzOOuT9d7HEQud7gmyo9+v4Xv/tjPhyv3siMjn5NrsSL8fX04c3BnfttxiJRDleUvdyhmLd/D0Phw/nxKH/rGhPLutaNY9bcpvH7lSLpFBXHvaX35/e9Ta1UYABN7RfPIOQO559Q+HMwtZugjv/Do91vYnp7H6j2HmTmhO6cM6ESQvy/XTexBSDtfZoyIZ2dmPuOemM//lu1GKcXpA2MYGt/4sQwnxj1laFLyS+wkHcglsVvj1ww4rB8vwEuXDef2j9YBcN7Ly/jlz5MIC/A7qvMqpeqdHecVl/HAVxuZObE7t36wlsJSOx9eP5bBHvx4v1iTSkZeCWcN0b5sX5sPV43ryr9/TCI5I59eHUMqjrWXO5iz6SDThnTmJreYx4l9OnBiHTckgCB/X/570VBW7z5MVn4pX65Npai0nL3ZhQT528gqKOWGWS7LfFdmPo9MH1TtPEkHc/nrFxv4IzWn0viAzmHcelJPRFy5LKO6RfL8JcMq3FXztqYzb2s6N5zQnW3p+eQUlnLqwBheX7ST3GK7xy7FJTsOkVVQwoIk3fVhU1ouD32zmbE9I7GXKy5M7FLtPfZyBzd/sIZ92UV8etNYSuwOFiRlcP7wODLzS1iafIjrJnav9ZrXTOjGl2tTue6935k1czRr9x4hMsgff18fsgtKeejsAQT46XZ4k/tq5XPawBhOG1hTHXN1RISrx3ej3KEIbufLil1ZvLs0hbeXpAAw1c3qjAz2Z+PDpyIijOkeyW/Jh7hpUg+vBL6ryWncU62f/BI7Ly7YwW2TezXoxvnRyr3ERgRwUt+aZ19HwwWvLmP1nsP8/uBUOoS2a5RzKqXYm13IiwuS+WJNKs9cOJQZI+NRSjF3Szo3vr+GO6f05q6pvRERNqXlsHhHJjdP6smnq/fx2qKdfHPbhErBRIDCUjtXv7OKP/blcNW4rtw5tTehtXx+P208wC0frgXAzyZEh7Qjv9jOW1cnMqZHFHuyCogI9Cc8SL/fXu7gwdmbWLbrEPuyixjVrT2f3TSuQjll5BXrmfkJPbj/jH4V11mafIjL31rJa1eMOKYA84KkdGb+bzXvXze6wiXiJDY8gPS8EjqFtuOXP5/I3qxCencKwc/mQ0ZuMWe+8BtFpeUUlJYzIiECXx8fwgJ9+ce0AXSNCq7xesVl5eQV2xn12LyKsbiIQNJqqB359MaxjKkje2z/kSLGP7GgYvuZC4fy3Yb9/LrN1Tbo2YuHct7weDJyi/nv3O2k5xazMS2HQ/k6s2xYlwj6dgrl09X7+PmuSfToEExhaTnhgXX/Pj5YsYe/f72p2nh0iD8L7jnpqCcmtfH0z9t4aWEy4YF+rP/nKY0e1K6NutxTxtJoA7y3bDevL9rFV2vTOHtILBcmxtMvJrTiC/jRyr306xxakb43b0s6D8zeSGZeCQNjwzixT4cav6yHC0p5dt52ukYFM6xLBIPjwnn4u82cPzyuRksip6iM1Xu0C2LFrizOHhp7zH/bgqR07vtiQ8XNICrYv2JGJiKcMqATCZFBPD9/B4t3ZPLv8wbz+E9bWZqcxaxleziYq91Id3+6nmcuGkZksFYcL87fwTNzt1dc560lKWTklfDCpcNrlMOZztqjQzDXjO/G1P6dmPbiEl5amMwnv+9j9ro0+nYK5ee7JwHw7tLdfLra1UrtqQuGVvqMO4YGMLlvB75am8r1J3QnKtif7IJSPl61l2B/2zEr8nE9ogn0s/HSguSKsbE9IvnHtAEMjA3n3aUpPPLdFgY99LPrM7gqkbeXpJBbZOeHOybSu5PnMYQAPxsBfjYW/OVErntvNSVl5fx232S+XJvKr9syGRAbxtLkQyzbmcWz87bzyY3jaj3XW7/pmfcFI+MZEh/O+SPi6N85jMggf04bFMOzc7dz96d/0D7In7lb0vnk930kRAZxKL+UzuEBPHhWf/76xQbW7ztCr44h9OkUgogQHli/t/6SUdqCWbvnMON7RVNcVs7KlGyuGte10RUGwOVjE/jk9338Z8bgJlMY9WEsjTbAdf/7nflJGdXGp/TryB1TejP95aUA7H7iLJRSTPnvInZlFhDSzpf8EjvdooJY8JeTWLQ9kzE9Igny13ONT1bt5f6vNlY771mDO/Py5SMqtvNL7Ow/UsTBnGKuekfPameMiOfRcwdWnCunqIxgfxu+9aSDztl0gC/XprH/SBHDukTw7fr9RIX40zk8kJFd23NhYny12W7KoQLmb03n9cW7UEpRYneQV+zy1w+OC2djWg7tg/z49vaJZBeUVnwmAI+fP5hv1+/n993ZrPzbFKJCKltIWw/kMvN/vxMbEciXt4yvGL/n8z/4okrcwBmgPOuF3wjws3HusFj8bD5cMjqBqszZdICbP1hbbfy2yT2597R+1cYbyt2frq8oYPvohjGM7+lyC+WX2Hn112Re/XUnbiEAbD7C387sX6cbpz4KSuz4iBDoX3llg5zCMh79YQuz16Wx4oEpNVqiJfZyEv81j8l9O9aqwIvLyjnrhd/Yd7iIUruDS0cn8Pj5g1mQlE7XqGB6dgghp7CMbzfs54xBMUSHNI7F25owlkYbxuHQBUEAQf42vrh5PH/9cgMFJXbmJ2VUUia3f7SW9fuOkHq4iBP7dOC2yb246PXl7M4q5KFvN/P+ij2c3K8j71jZIxustMLnLh7G3K3p/LDhAIF+NhbvyKxUD3Dbh2tZtD2Tmyb1QAROGxDDl2tTWZmSxW/3TWZvdiHTXljCtRO6MSQ+gj3ZhYztEUlxWTkju7oslsJSOw98tZHDhbqa1pne+OJlw+uceXePDub6E3owsXc0pz/3GwAPnNEPm4+QmV/C/af345v1+7nvyw2c8ORCAPxtPkwb2pmEyCAuHZ3AsC4RnPH8b/yw8QBXjetWSaaLXl9OoJ+Nv5/Vv9J1R3VrzxdrUpk5oTt3Tu1N4r/m8uPGA3QOD2Dz/lzuO70vV7qdqyon93P5sKOC/ckqKGVolwhuOanW+tcG8e/zBjM8IYKs/FLGdq/sDgpp58u9p/XjnlP7kplXwvp9R3ht0U5uPrEnp3roo6+N4HY133bCg/y4aVIPvliTymer93Hb5Op/53vLdpNXbOfc4bVbqQF+Np6YMYQLX9MZZTefqOsg3D/P8CA/rmykzLG2hlEarZzdWQXkldj593mDmTEyjna+Nr7700SKy8p59dedPD9fFxvFtw/k+w2uNhPTh8UyunskH10/hsvfXsn7VsbOgqQMDuQUERMWwNo9h5nQK4pzh8dx5uDOnD0klhJ7OXd+sp61ew5X+KUXbde+5tcX76JfTCh3ndKbOZsPknq4iLV7j/DYD1vIK7HzgpurxMnieyeTEBVE2pEiznt5KYcLyzhjUAw/bToIQFiAL+N6elY93S8mjPvP6MeG1COcMyyWzuGBFfvOHR7HoLhwLnljBe18fXj8/MGVMl36dw6jb6dQvl6XVqE0lFK8uTiFvGI7L102guFVqnPPGRqH3aG4YGQ87XxtjOsZzbyt6URZLrD6AqT+vj78dOcJhLTzpUtkEGXlDmwi+NRSu9FQAv1tlRRgTYgIHcMCOHVgzDErC0/o3SmUMd0jefqXbQxPiKhk/SRn5PHvH5OIDPavN1g+qlskP95xAocLS2uNsxiODqM0WjnOIqNhXSJo5+tyBwT42bj7lD4Et7Mxb0sG7147ipcWJhMV7M/KlOyKTJzxvaJ595pR+IgQGezPtBeX8MvmdOLbB5J0MI9HrApZf18fTh8UQ15xGX42YcG2DMb0iKKwtHLa5lXjutEvJow//nkqiY/N5b4v/mBnZgEn9+vIAsvqGZEQwdq9OkYw6amFdAxtR0ZeCf6+PjxyzkCuGNsVm4+usC0td1T6u+qjrirrXh1DmP+XE2nn61ORBePOucPj+M+cJPZkFdA1KpifN6fz7Dwd9xhTQwfZQH8bl49xzWZHd2vP4u2ZvL0khVHd2tOzQ0i191Slf2dXVXtTVHK3BN6+ZhQnP/0rz8/bwdjuUazYlUW36GAWWplSn9w41qP/+YDY46cjwPGEURqtlOKycn7efJDfdhwi2N9G704136BunNSTGyfpG+lfT9d+8uuttgZOnK4fpRRD4sN55ddkBnQOIzqkXaWKVoDQAD+GJ7RnxS7tElu8/RAA14zvxoDOYcwYqRdiDA/yY1LvDsxPyiC0nS/PXDiU4Y/OBeDLW8ZT7lBsS89j+ktLycgrAWDWzNGVejJ1DKu5HuFYqCt75pxhsfxnThIXvLacqf07Vlhmz1w4tEYlU5WBVjpkRl4J951+7DGJ1kpIO19uOrEnj36/hYn/WcD+nGKC/G1EhfjTp1MIfRoQgDc0Pm1j6tIGeeKnJO78ZD1frEllXM+oRpmligiPnTuYw4VlLNyWWet5h8aHs/VALrsy8/lw5R7aB/nx97P6c9GoLpXaYlw2JoFgfxsXJnahfbA/d0zpzb2n9UVE8LX5MDA2nKX3n8wNJ3Rn+rDYo27i11jERQRyx5TeCPDxqn3kFdv517mDKhRhfQyKdeXQnznY+66e45nLRuvvxv6cYsb1iKKwtJx92UXcNbVPc4vW5jHZU62QRdszufZdV+79sxcPY/qwaivjHjU/bjzAbR+t5ckZQ2osovr2j/3c8fG6iu3GyvZpKdjLHaQdKWLL/lxOHxTToFTIpcmH6N0xxCtWUmtjZ2Y+P244wM0n9bRqhgJrbKtiaHzqyp4ySuM4weFQfLhyD7PXpfHqFSPpVMtNZ+3ew1zw6jLi2gfywx0nEOzvW2vTu2MhI6+Y6OB2NQZlD+YUM/bx+QCc3K8jr1w+wiP3jcFgaBmYlNvjEKUUX6xJpbTcQWiAH3/5bD1l5VrBL9qWyUWjXDP83OIyfEQIaefLC/N3EB3Sju9vP8ErxUZOOobWPlOOCQ/ghUuHs3bPYR46e0CLKUoyGAzHTotTGiLyKDAdcAAZwDVKqf2i7zzPA2cChdZ49cqnVoBSio9W7eXB2a52BT06BHPFmK489uNW7vtyAwNiwxgUF87+I0Wc98pSAv1szL51Aqt3H+a84XEV7Sqai3OGxnJOI1R8GwyGlkVLDIQ/pZQaopQaBnwP/NMaPwO9Lnhv4Ebg1WaSz+v8tOkgD87eRFiAL7HhAfgIPHvRMGZO7E5nq4PprOW7WZWSzWVvriA9t4TdWYX8+8et5JfYGdm1+mpeBoPB0Bi0OEtDKZXrthkMOIMu04FZSgdhVohIhIh0Vkq1uoWP523Ri8F8dvM4ukUFk19ir2h18NoVI5n24hI+W53KZ6t1i4oLRsbz3R/7+XxNKgF+Ph4XuxkMBkNDaYmWBiLymIjsAy7HZWnEAfvcDku1xqq+90YRWS0iqzMzM6vubvEopViSfIhpQzrTLyaMAD9bpd44g+LCucBK8Qz2t/HutaP459kDKpYJ/b9zBtUaJDd4QEEWlOQ1txSGhrJjnv7fGbyOR0pDRAJFpG9jXVRE5onIphoe0wGUUg8qpboAHwK3N+TcSqk3lFKJSqnEDh3qXl+gJbIjI5+MvBJO6F17mwRn9fH9Z/Rjct+OhAX40T1at0rwtHe/oQaUgqd6wJsnN7ckhoawaxF8OAPemgIOR93HOhww5wFI39w0srVC6nVPicjZwNOAP9BdRIYB/6eUOudoL6qUmurhoR8CPwIPAWmAe1FAvDXWqliyQ1dQ19Vb54KR8Zw6IKZSsHvWzNHsyMhr9gB4nSx6EnL2wTkvNrckNXNou+u5rBj8jMXW4nGUwy8P6teHUyBzK3QaWPOxGUnwyhj9OukHuGtD08hYG/kZsOBfMOUhCD5+XMqeWBoPA6OBIwBKqfXA0fdFrgcR6e22OR1Isl5/C1wlmrFATmuMZyxIyqBHdDDx7Wtf11pEqimHLpFBlbp4tkgWPgZrZ0FhdnNLUplDO+DlMbD8ZdfY7281nzwGz9mzDA5uhBPv19uvjoeMrTUfu9Itd6ao4UvLNjpr3oO178FrEyBzW3NL4zGeKI0ypVROlTFvVgQ+YbmqNgCnAnda4z8Cu4Bk4E3gVi/K0Cxk5pWwfFcWpw1qJBeTvRQ+vgz2rar/WG+T42YUrn2v+eSoSspieCkRMpO0XDGDoe+ZMO+h2m8+R0tx1Z+R4ZjJtOaUI69xje1a5Hq94TPX/7HUbW3v8rJju25OGvzyj2M7z65f9XPeAXjrFNg8+9hkaiI8URqbReQywCYivUXkRWCZtwRSSs1QSg2y0m7PVkqlWeNKKXWbUqqnUmqwUqp1lHpbpB4u5Mb3V2PzEWaM8KyXUb0c3ADbfoAvr9Pby16EzO11v8db7F6in8PitElecKh55KjK2lmVtwfNgHNeArHBug8a7zrb5sB/usG+3xvvnK2V+uIS7hzaAf4hEBoDM962xrbB0ue1+2f2TbD0BWvc7btvL4Zye/Xz1cfGL2DuP+HZAbDsBdi7ouHnANi9FPYsgZP/DtfNhXYh8Pk18HXLnwt7ojT+BAwESoCPgVzgLm8K1dYoLLVz8tOLWLf3CI9OH0ivjvW3zPaI/Vb/p/Iy7RL65e/wvzMb59wNZftPENwRLvkQHHbY9lPzyFEVHyusd+F7cPGHMP4O7V+O7l35JnO0/HgffHAB/PwAKAds/PzYz9ma+eXv8EzfylZBXWTtgKheIAKDL4CweFj9jr6x//J3/Zkf2q4VUaUJk4Ijexom275V8NUNWiE5OfBHw87hZM27EBgJY2+DLqPhhgXgFwTrP4S8g0d3ziaiXqWhlCq0splGWVlJDyqliptCuLbCzowCSssdXDO+GxePqr7s51GTZhXM5x1wzWAKmiENWSntMug1FToPg/AESPq+aa69fx18cjnYS2ren58BscNh4LnQfxr4WD2yovscu5+53A6rXofkuZC9C0Jj9fbGL6of+/3d8L9pLcPXXpWyIh3jqe0zbCx2LtDWcEGG/oxSfqv7+JI8/R3v6LZiYoRbrsyGT/XzoR2Qtx/sRTDhTjjrGde4p5QWwuybtRJyJ22N5+dw4iiH5HnQ+1Twt2KXoTFw/Tz9uql+G0dJrUpDRL4TkW9rezSlkK2d7em6LuCKxlx+0uGAnfOh0yDrIm4z+6Mxy49FjuUvQ1E2dB6iZ4T9p8HOhU1TD/HdnfpHmFqLN7MgU1tAVYnuo2ei6z86uusmz4Pdbje9mMEw0/ofbPkG/vhE34hAK5TV7+jjt353dNfzJlu+gR/+Aj/82bvXWfYS+FtrZXx3B7x/Lnx0Cax9v+bj138ExUdg1A2usfAqXZfbhUFJDnxzm97uOQUGnq9fZzVAafz2NGTvhIs/gEEXwF2boP85sP8oOhllbNGTg15TKo93HKAnMIuf8dzSagbqsjSeBp4BUoAidPD5TSAf2Ol90doO2zPy8LMJXaNqz5hqMAfWQ366drdMrPJj31fFD+v0Ie9fr8361e/oWW9tM+2SPHjnjOo3YqVg1ZuQ5fb1WPW6KyUy2kqM6zcNykvg7dO8E2NZ9wHkWxaVTS+tysGNNR9bkAkhNdTz9D5FPzv94Q3hUDJ8MEPf9MQG578J1/wA7btBwjjY+q32tX94gXahbP7a9d6kH7Qr8af7G+fGUZLfsBhBTRzerZ83fgl56d5R9g4HpK2GQefrzwm0G3P7T/Dt7dWP3b9eP0JiIH6ka1/f012vOw2CKVZtsDPoHNULgiIhKEp/9lke3MqUgg2fQ+/ToP/ZcMHb2qKJG6E/m/qKCguzK0/UnG7j2OGVjxOB05/QVtEyD9PSldIP0P+Xn+6H3P2evfcoqVVpKKUWKaUWAROUUhcrpb6zHpcBJ3hVqjbG5rRcenUMbdzlPJ0ZI/GJ0Hmoa9w3UM9ynaz7AP6vPbx9KrxxovbXfn+3nvX+9ozrOKdrInsXPNUb9i7TCsad5S/Dj/fAiyPgramw9XuY94hrf7RVH5ow1pJxM8x/hEblyD49q/z8Gr3tzFjav676sQ6HZWnUoDTiE2HSfTqouu4DrQwPVV/DvEayd7le9z8bhlwEAdYCTD5upVF7luqbw/xHoONAGH4FpP6uP9eVr+rP71hwlMPjcfDV9Q1/b7kdUi3XizO2Yy+CZ/rAK+OPTS53lIIVr+m/uzgH4kbC9Fe0v3+6lQJd9f+z/Sf9Xd08GzpUWZRpwHkw9laY+TPcshQSZ7r2xQ6H0M769Qn36Oef/lq/jBlbIGcv9Dur8nicpaxq+m45sZfAk91hjnWdnQvg2z/p15E1LD2cMBa6ToAdv9Qvl71UF6K+OFLHQX75h/7erHil/vceA57cpYJFpGL9TxHpju4JZWgE7OUO1u49zKhujdxk8MheQLS53nMyDL4QJt6tb0zr3td+fodD57kD7FtZ/Rzb5ugg+up34F8d9Y/0+7v1zQP0Te+nv8Ks6Xrfwsdc77hsGNYAACAASURBVE39HT69Qs8WR16jM1zCrK4vPjaX9VNeWrP8JXnw5fUNq9y1l+jiQdCW1rxHXDe8rB3wcDj8+h/X8cVHtHw1uadA32SUQyuhH+/RN6rlHvwgnQHWSz6Gs/5beV/XCfp50r2Vx/udCTFDoTALtlje3/z0+q/lTk6atgCd1tuRvfp505c6dtMQNn4Gb52sXUOHdlSeFefsbdi56mLvCn1DfedUvZ0wFrpNgL+m6O/qhLug6Ehla8mZsWQvgg5VFvfy8YHTH3dNTHxsMPVh/d2/8Ve9H2DcrXDqv3S8KXl+3TJu+1E/9zm98ngHK5aSVcdkwmnh/v4WzHsY3j9Pb/c4ySVLVToO0J95bWsdZe+C987Wn9v+tdpttuMXV3JJ8oK6/55jxJOGhXcDv4rILkCArugus4ZGYOuBPApLy0nsFtm4Jz6yF8JiwddfP2ZYxWplRVBWqLM0Nn3puskCxI+GyO765tttor5RZu9yWQvO2Xt0H9fNeOVr+nnPcu1yCojQN+PEmVrZ9D4Vzn5eP9yZ+pA+d21uowX/0plG0X1qr/Ctyhsn6VkhQGk+LLFu2D2n6PgOwK//hpOsWZ/zbw+vZVXDhLH6B+wfome06z7QWVB9ToOoGmaJoC2SpO/BNwD6nqFdDu5MuheGXaaTExY/pccGTIcRV0OuVctSYllHDc3e2j5HW4CfXAp/WlP5/ftWaqvHU5wKa84D+rM86X7tDnKWaDkctd/0PMHhgB//or8jTiJ76v+3O2Fx4CjTytTpRkxbq7OkCrMgflT915p4d83jo2+CxU/rmE3V+AJoS2v7HFj8pLYqQqsUzwZHa8v9SB1K1D1QvuRZ/dznDLhoVs3HA3ToCyW52noI61x9/5wHdH1RymL9P93+s8t6CY3VFnxhtnbDeYF6lYZSao5Vpe1U6UlKKS+nUbQdft+tq6MbxdJwOPRNs9NAPQOJqCETyy9Q1yFsn6NdUDmp0PcsbXoPvcSVPeSMV6St1UrAnevnaX/7jl+0r3jzbK0wAK7+Vle6nv6EVkJxI6mV6D7av28vAV9XU0Y+vxY2f6Vfe5rhYi9xKQx3JtwF/sEupQHaDZKf6fqx1/Q5gf7R3bpcv/79LVfdRtIPMOGO6scf3q0VLbjSQKti84X2XaGdFfBFXDeQ4A7QYzIEhGmroa4ZLOgMo9/+Cyfep7O/0q31V7J26puGe0wqbU3DlIazjqY0DwLbw5ibtZJzWL75nH367zha/viossIA/R2s+pk5b5p5+7XS2L9ex+TG3KwVmf8xpKf7+uvv59r3oOt4COmoXajOScRbbj3Ipj5c/f0iOrZR1fJylGsrweZbWWkEd9DWZ3Rvfe3acMb+Dm2vrjSUgr3Wd9I3EE57vHLyxOgbtLtz/7qaFWEj4EnvKT/gJmCSNfSriLyulDrGkkoDwOo92cRFBNI5PLDhby4r0o+gSD0rWfWmzvJo31334Rl6ac3v8/HRwcbDu7XS6H82DL+88jGRlkfy65srjz94UCuegHDtdhp5jY6fZCbpm2/noTDNmuEPq+X6TuJGaPfPnqXQ0/qBHt6tFUZ0X33z3PgZdJ+kX3edoGd3VclOqf0G276r/nG580QVJRHhwc3P3f+89Dn92VYNoDtjRf6hMPC8us8XFAnj/6STApz4BcBVVlD82zvqTr0sK9axj9w0nSHW4yR9o/AP0ZbB9jl6OyRG33gaWoSWd1AH8X1scNLfIDBCb2Mpjdk3aes1/CgLUbd+pz/TxJmuRImqMQNwuTQzturv1tLndEbUCX9xU7zHQHRvPaGYfZPejhkCN1dJ9Y3sqb+DNREer/+WbXO0Bfq/abpor124rhvZ8KmeCAy/ArqMqZwSXBvO72NOavV9BYf0pGf0TTpWFtFFx8kcdj1BSrxWK420tV5TGp7Yl68CI4FXrMdIWvECSE3NhtQchiVEHN2bP7xQB9mydsIz/bTCAK0wBkyHE+sI8kUk6C9WeWn1NEWobNo64w8+flphVMUZqIxuYCPk7ifqG7p7od+Oufr5ko9cweNvb4fPrqoetHS2cHhlnM5Ecucfh7RFNfzK2t1PTgI9sPKcqcsT/6zjLc4bnZMj+7T7od80+FuqrvStj1P/5fK9VyW6j3a/uGfmbPgckiz/+rIXtcI46QFtCaYs1t+DoZfq9y78t1YcvU+BXqdo91RDWofnp2vXz1+2wRjLG+20QsO76NnuCrfbwIrXYOUb9Z8394C2Yveu0LP78bfDZZ9pl01NrqaYIfrvWfKstm63zdEZVo3lehl9I/i5hWidN+p8D+uZnJbXxxfr+OAeq/NBSQ6stirU40ZoBeKJwgBdswHahelO6mpXT7Tep+hkDYBrf4Jpz8Ipj+jvcswQnfLtJTxRGqOUUlcrpRZYj2sBDxyJhvoodygO5BTT7WhSbdO3uOoAVrxKtXZgUx7S8YnaiEjQrgdwpThWZeytepY59SG4LwXuqcXH7ry5xwz2VHqNf5C2MJJ+dAX90tZCSCcdM6h6401ZpN1QP9yjewo9FqN79jgD86ADz+e/BTY/GHGlfk4YB5MfhLu3aGUK2ifuxJM1zEM6wMM5+rMYNMPV36g4R/vFnxukW1NMfbhhn0FtOH37zloCpXQW1CeXajfk6re1Mph4t04rTp6r/eDtu2nFlrNPx676naXrYpRDx7Hm/x98MbPWy1aQd1D78N1vzs7Mr6u/07GeNLcahTl/hZ+qBPerkrkdXhiuW5gXH9FxM9Az9Ms+cSkld2y++nuYmaSD8vYiHStqLKJ6wt2bXBMe59/7i9t3z911WpWpD0OClU3mrAXpfWrlY2qyoOrCL1DHBt0rwwsO6c9t0ROW3L1c+7qMrpwl1m+aniQ0NPnBQzxRGuUiUmGbW5lU5V6Rpo2RnltMuUMRF2EpjaIj8N1d9f+zN8+GV8e5tn9/Uz/f7ZZp1L6eRsTufvyqwUcnpz/uChoHRdY+u3MGTeNG1H3Nmuh3JuSm6j5ZoFNcO/TVN/LY4XDHOohLhKmP6PTYf3XUf+9XN+hZXmqVZowjroIhF1Yes/lpv394nCvVctRMrUTuqCNdsjbad4f8g7qFxJM9YMGjrn1Of/SxEm3dFJxxCndXxd7lehY6+AJ9Q4sZApusGFD7rvom4iRhrHbr9JqqU6h/e0YnQNTXaTg/Xbu23DnBsjhDO2t3zYH1OjW3tj5i7tk/jnJtMfoF6HYtl3+pla8nOLOWnOnZcYmevc9TgiLhtpXaXZiVDP8dCBs+0dlRQy/VdTa1ETcSzrMsrsMpMOYWmPac3u46Ef6ZXXdcrzZCY/T/WCldGe/eIy12eO1xOLCUlPJaqx5PsqfuBRZWyZ661ivStDHSjugZcmyEtW7DnPvhDytQNu622t+4bQ4VAdSt3+osI7Fp/2qfM3S1aX2ZLe5f5JrcUw3BmbIaM6Th73XONtPW6vdnbq8cC4nsATdYQeyIBPiiyldvxtvw6+Nw5dd6Zm2rZz2RzkPg5qV6pny02T/OH+zrk1wylJfqgrHGIqKbvmktfR5GXFO5wM1Zpe5M340fpQvjnLI541Hgcr2NuqGyy2LHXBh6cc3XLi3UVkvVbKGJd7sykbqM0ZlzaWsqF/vZS11B3sfjrQaQL+iEiX0rde1F/2k0iLDOEDtCJ3dE99HxlcZGxPVZ5VoKevyfqsf6apTPzf058Fw9Obn4A22B1GQ9eUJojE4CmX2zVmCgv7POxIy66DRQx0WSfoCRVx/d9evAk+yp+Vb2lNNhvc1kTzUOaYeLeMD3Q/qnp0OvK13+6rpS+EDfIPqeCQPO0Y+xt7pM6Ms+qfu9TjoPc70+ltRJ0Isq7b3s6LJpwhN0eurGz/XNrjSvdstn0Pn6prhroU7JDY3Vs+3BF9R8fG3EDGq4nO5U9U0PmuGZi6sh+PjAmJvg+7t0seSRPfrzyd5lVTeLy/cdnwjOMpuIrlqWk/9ROWmgx4mVz79vRe1KI99yi1S1NNzpNUXHuLZW6SiUn64/n6ydOiC/9j0440mt2H0DPbcuqtJtglYaQ2qRuTEYdrm2wJa/pLdr+x5WxX2i4kwPb0imWk2Edtb/50NuGXCjPax0ENHZZe5u20bEk+ypC4E5SqkNIvJ3YISI/EspdRRNVwzuHDiQxi2+P8CCHyC2qys//8AG2LsSEsbo4N/eFdr3L6JvGlnJ+gvu5GjcQiJw5Wy08XiMBEc1fPboxMdHd/fcsxRmWYtBulewVyU+UT/6ngm2OnzN3sTdMvMNaHyF4cQZk3IWC17zAzw7UM+EQzq5blZOd1Tnoa5Z+KR7Kp/LL9BVOxPcURdf1kRJvo4TQXVLw52AcB1bWP2Odj05yTuglYZ7wdzaWfp6fc6oOZHCEybcDUHReoLkLUJj4LTHtAXlsLtchA2hMTK6QCvHjK36dz7qeh0v829A7HOc9z4nT9xT/1BKfS4iE4Ep6J5UrwJjvCZVG8Hh7i6YY+Wc95umzdF3ToWbFuugZVayzjApL9NZMzZ/XSB2rDjTXJsb/2Dd0NCJJ24uTwv+vEFEgs5aiup1dAq7IddxkjhTF2uGxeu6gNCYysdd/X39spz5jE6RTvpB11w4C8DKy3TaaP+zdRZWoRWjqMvSAJ395bQ0RlyllcOhHVqJpbsVbTpbYpzvQXZVbQRHwcQmWpHh+nnaledJVp2T6+ZBWSM2Gew5WT+cNERheBmPAuHW81nAm0qpH9DrhXsVEfmLiCgRiba2RUReEJFkEdkgIl78tXqfsnIHQYc2uQYObdcpiAPcll7/+jZX/cG69+HTy7VbocfkyjeN453LPoUL3Aq9Wvra3CK6sGzwBZXjB42Ne4bXNKua2OkCDI2tfGz3E7TyrQsfH22N9D5NZ1Nt/1mn4X54oY4Vrf+w8uqC9X3H3N2RE+7SGUi/PqEnOu6B2+S5urdWQFjd52spxA6v3ualPrqM0rUybQBPLI00EXkdOAX4j4i0wzNlc9SISBf0Uq/uzv0zgN7WYwzHubWzKS2H+KpLnE99RKcA2trpCuv0jToDI7Jb5VXk6nLfHI90Gqgf7nEWQ81Vw+276lTrY5k0xA7XSifpe13rsWuhHt82p7K1EuhBLcRFs3R2T1RPnaH25XWuCunwLq5WLUfrljK0ODy5+V8E/AycppQ6AkSiM6q8ybPAfVQuPpgOzLKWfV0BRIhIDY1Zjg92bljOKba1KF/L7Dzvdeg0QAe0Hzyoc/BBm+RVUwydGUetjaietfd0aqtc/CHMdOt4OuZm7e8ectHRn9PHR8egkufrqvGwOO1e2rfSlYRx1jOeJUgMmA5nWUWlvaZW3tf3DLjiS/26sJa0XMNxR62WhoiEKaVygQDgV2ssEr3sq9fW5xaR6UCaUuoPqRxgjAPcuuuRao1Vmq6LyI1YDRUTEhpxFbzGRCkuWK3TSmXQeTq1z331MR8fuPh9nfoYHFV5Scm7Ntado21oXVRNMIgZfGyxASf9psGqN7S10fNkncVWlK3X9ohL1MHXhhIYoRVH8jzd5mPin10r3dXW0sZw3FGXe+ojYBqwBj3jd7+DK+ConbkiMg+oyb5+EPgb2jV1VCil3gDeAEhMTKylt3DzUpiygoqw1sDzKisMJ36BLpO+02DdmKzXFKMwDI1D1wmujsTRfXWQHXTKc+IxlGFdNEuvAzL4Qpel8kBq5VYdhuOaWpWGUmqa9VxPaXHDUUpNrWlcRAYD3QGnlREPrBWR0UAa4J4gH2+NHV8UZpM67zXiVDuSrljNyN4eKAEfH6+m0BnaIDZf7ZJa8562DtxdUcOvOPrz+gdXr/9orDRUQ4vAk0A4InI+MBFtYfymlPq6nrccFUqpjUDFijgishtIVEodstYlv11EPkEHwHOUqhpJbuHkpMGzA+gDLAg5k8m9jrES22A4Fk59VD8AMpKaVxbDcYMnxX2vAL2Aj62hm0XkFKVUHX0uvMKPwJlAMlDI8djKxFqbO8XRifSxDyLeKgozGBpKTYv9GAw14ImlcTLQXyndfUxE3gMasAbn0aOU6ub2WgFNragal7S1lPv4c0rxU3yaEFv/8QZDU9EuDMbepnsnGQx14InSSAYSAKuXAV2sMUND2b+OjOA+lBf50i/mOCl0MrQNROD0fze3FIbjAE/qNEKBrSLyq4gsBLYAYSLyrRVnMHiCoxz2r2er9KJ7VDDB7TwKJxkMBkOLwpM71z+9LkVb4NB2KCtgsaMLA3oaK8NgMByfeNIafZGIdAV6K6XmiUgg4KuUyqvvvQY3lr8MwMLC7lwSG97MwhgMBsPRUa97SkRuAL4AXreG4gGvpNy2WhwO+ONjvmAKe1QM54+oZ81qg8FgaKF4EtO4DZgA5AIopXbgVkth8IDiI+Cws7kslgfP7E+nsBbexdVgMBhqwROlUaKUKnVuiIgvlRsJGurDWkM5S4XTN8ZUxxoMhuMXTwLhi0Tkb0CgiJwC3Ap8512xWhG/PqHXUQYOEUavjiHNLJDBYDAcPZ4ojfuB64CNwE3oyuy3vClUq+LXxyte5vpEEGNcUwaD4TjGk+wpB/Cm9TA0hPzMSpsB4Z3w8TGtQwwGw/GLV1fga/NkVO62EhHdipZoNRgMbRKjNLzJ/nUA2AdeSJrqQJdoEwQ3GAzHN6aXhTfZs4yCsB4MXHMecB4f9OvU3BIZDAbDMeFJa/TvqJ5im4Ne8vV1pVSxNwQ77ikvQ+1dzi9lYwFI7Nqe8T2jmlkog8FgODY8sTR2AR1wradxMZAH9EEHx6/0jmjHOcnzkJI8vi8dwttXJzKlv7EyDAbD8Y8nSmO8UmqU2/Z3IvK7UmqUiDTJuhrHJZtnU2AL53ffEbzep0NzS2MwGAyNgieB8BARqVjI2nrtrFArrfktR4+IPCwiaSKy3nqc6bbvARFJFpFtInJaY1+70VAKdi9hnW0wA+Oj8bWZfAODwdA68MTS+AuwRER2AgJ0B24VkWDgPS/J9axS6mn3AREZAFwCDARigXki0kcpVe4lGY6eI3shN4259lMZ2bV9c0tjMBgMjYYnxX0/ikhvoJ81tM0t+P2c1ySrznTgE6VUCZAiIsnAaGB5E8rgGQc3ALCuvAeP9De9HQ0GQ+vBU7/JSPQMfyhwkYhc5T2RALhdRDaIyDsi4pyqxwH73I5JtcYqISI3ishqEVmdmZlZdXeT4MhIAqAwrCdD4yOaRQaDwWDwBp6k3L4P9ATWA05XkAJmHe1FRWQeUFN59IPAq8Cj1jUeBZ4BZnp6bqXUG8AbAImJic3SjXf/jvWgovnTGcNM2xCDwdCq8CSmkQgMUEo12g1YKTXVk+NE5E3ge2szDejitjveGmtZKIUc/IN9tgSmDYltbmkMBoOhUfHEPbWJmq0CryAind02z7OuD/AtcImItBOR7kBvYFVTyeURSsGSZ4mz7+NAzGRsxsowGAytDE8sjWhgi4isAkqcg0qpc7wk05MiMgztntqNbseOUmqziHwGbAHswG0tLnNq1Rsw/xG2OhLYlzC9uaUxGAyGRscTpfGwt4VwRylVa4W5Uuox4LEmFKdhrH6H0tjRnLnrDh6NMAFwg8HQ+vAk5XZRUwhy3JOTCplJ/NTxVhQ+Zh1wg8HQKqlVaYjIEqXURBHJo3LDQgGUUirM69IdTxz4A4B39+nwT8fQds0pjcFgMHiFWpWGUmqi9WwWgfCEvIMAHFCRAHQMM0rDYDC0PurMnhIRm4gkNZUwxzX56SiELMI4d1gsnUKNe8pgMLQ+6lQaVnbSNveGhYZayE+nwBZOeHAQz10y3BT1GQyGVokn2VPtgc1Wym2Bc9CLKbfHJ3npZEl7ukUFN7ckBoPB4DU8URr/8LoUrYH8gxx0RBAXEdjckhgMBoPXMCm3jYTKPUCqvS9x7Y3SMBgMrZd624iISJ6I5FqPYhEpF5HcphDuuKEkH8k/yK7yTsbSMBgMrRpPLI2KlFsREfS6FmO9KdRxR/YuAFJUDInG0jAYDK2YBq1DqjRfAy13qdXmICsZgBTVmc7hJtXWYDC0XjxZT+N8t00fdKv04loOb5tk7QRgt+pEjGkfYjAYWjGeZE+d7fbaju48a1q4upO9k1z/jpTbgwgP9GtuaQwGg8FreBLTuLYpBDmuyUrmoG88ncLaocM+BoPB0Dqpq2Hhi1RuVFgJpdQdXpHoeCQrmT0y3rQOMRgMrZ66AuGrgTVAADAC2GE9hgH+3hftOKEwG4oOs8PeybRDNxgMrZ5alYZS6j2l1HvAEOAkpdSLSqkXgSloxeE1RORPIpIkIptF5Em38QdEJFlEtolIy8jgyjsAwI7icNPZ1mAwtHo87T0VBmRb2yHWmFcQkcnoQPtQpVSJiHS0xgcAlwADgVhgnoj0afYlX62W6PvKQulrLA2DwdDK8URpPAGsE5GF6AWYJuHdJWBvAZ5QSpUAKKUyrPHpwCfWeIqIJAOjgeVelKV+8rV4mUTQyVgaBoOhlVNvcZ9S6l1gDDAb+AoYZ7mtvEUf4AQRWSkii0RklDUeB+xzOy7VGmte8tMByFQRJhBuMBhaPZ5YGiilDgLfNNZFRWQeEFPDrgctmSLRrUpGAZ+JSI8GnPtG4EaAhIQmWAYkP4MyWxCFBNDRuKcMBkMrxyOl0dgopabWtk9EbgG+UkopYJWIOIBoIA3o4nZovDVW9dxvAG8AJCYm1poy3Gjkp1PoHwUFEGNaiBgMhlZOg3pPNRFfA5MBRKQPOr33EPAtcImItBOR7kBvYFWzSekkJ5XDtmiC/W2EtGsWHWwwGAxNhiet0XuKSDvr9UkicoeIRHhRpneAHiKyCfgEuNpqlLgZ+AzYAswBbmv2zCmArGRSbbGmRsNgMLQJPJkafwkkikgvtNvnG+Aj4ExvCKSUKgWuqGXfY8Bj3rjuUVF0GAoPsSs0xtRoGAyGNoEn7imHUsoOnAe8qJS6F+jsXbGOE6zutltKTDW4wWBoG3iiNMpE5FLgauB7a6xttnJd+G94eYxrOzsFgA2FkUZpGAyGNoEnSuNaYBzwmFIqxQpCv+9dsVooi/4DmUmu7dxUAPbY29Mx1LinDAZD68eT1uhbgDvctlOA/3hTqOOG3P3Y/UMpKA6ka1Rwc0tjMBgMXqeu1ugbqbs1+hCvSHQ8kbufPP9OAPTvHFrPwQaDwXD8U5elMc16vs16drqkrqAOZdKmyEklQ6IIC/AlLiKwuaUxGAwGr1Or0lBK7QEQkVOUUsPddv1VRNYC93tbuBZP7n72lg+nX+cws2KfwWBoE3gSCBcRmeC2Md7D97Vu7KVQkEFSYRgDOoc1tzQGg8HQJHhS3DcTeFdEwq3tI9ZY28XhqFh8aW95exJNPMNgMLQR6lQaImIDTlRKDXUqDaVUTpNI1pJx2CFX90o8qCLpF2MsDYPB0Dao081k9Xa61HqdYxSGhcMOufsB2K+iiDVBcIPB0EbwxD21VEReAj4FCpyDSqm1XpOqpWEvhSN7XdtulkamRBEV7N9MghkMBkPT4onSGGY9/5/bmAJObnxxWigLHoVlL7i2HXbIPUCxTxBBIRH4+JjMKYPB0DbwpCJ8clMI0qLJ2Fp521EOBZnk+Jh1wQ0GQ9vCo1WDROQsYCBQ0ZVPKfV/tb+jlRHZvfK2ww4FmWSpMDqYdcENBkMbwpNFmF4DLgb+BAhwIdDVy3K1LII7Vt522KEwi/TyUGNpGAyGNoUnRXrjlVJXAYeVUo+gO9728ZZAIvKpiKy3HrtFZL3bvgdEJFlEtonIad6SoRpVFwh02HHkZ3DAHkLXqKAmE8NgMBiaG0/cU0XWc6GIxAJZeHERJqXUxc7XIvIMkGO9HgBcgnaTxQLzRKRPUyz5qsrLqBTqLi9DCrPIJox+0SHevrzBYDC0GDyxNL631gR/ClgL7EYv9+pVRDdzugj42BqaDnyilCqx2rMnA6O9LQdAWVlp5YHCQ4gqJ0uF0bOjURoGg6Ht4En21KPWyy9F5HsgoImK/E4A0pVSO6ztOGCF2/5Ua8zrFBeXUKkSw2ohcljC6dLeFPYZDIa2Q71KQ0SWAIuA34CljaEwRGQeEFPDrgeVUt9Yry/FZWU05Nw3AjcCJCQkHLWM7hSVlFCpUUheOgDlgdH42kzvRoPB0HbwJKZxJXrWPwN4SkRKgN+UUncf7UWVUlPr2i8ivsD5wEi34TSgi9t2vDVW9dxvAG8AJCYmHvO6Hw6HorikpPKgZWk4gqKP9fQGg8FwXFHvNNmKH8wF5gOLgSCgv5flmgokKaVS3ca+BS4RkXbWOuW9gVVeloNL3ljB0u0HKw/m6W1bSAdvX95gMBhaFJ7UaewEvgY6AW8Dg5RSp3tZrkuo4ppSSm0GPgO2AHOA25oic2rV7mz8qHIZy9LwDzOWhsFgaFt44p56AZiIjjEMBxaJyGKl1E5vCaWUuqaW8ceAx7x13dqwiaOyHHkHOaJCiQwNbmpRDAaDoVnxxD31vFLqQrTLaA3wMLDdy3K1KPywVx7IP8ghFWa62xoMhjaHJ+6pZ0RkJbASGAL8Ex1PaDPYqGxpSEke2YQSFWJaiBgMhraFJ+6p5cCTSql0bwvTUqkW0wBtaYQYS8NgMLQtPCky+Ao4RUT+ASAiCSLSJJXYLQVbDUojS4UTHWwsDYPB0LbwRGm8jG5SeJm1nWeNtRl8a1QaxtIwGAxtD0/cU2OUUiNEZB2AUuqwiLSpu2WNSoMwIk0g3GAwtDE8sTTKRMSGXuIVEekAVSLDrRxfqa40Cn3bE+BnawZpDAaDofnwRGm8AMwGOorIY8AS4N9elaqFUZOlYQ+MagZJDAaDoXmp0z0lIj5ACnAfMAW9ct+5Sqmtdb2vtVGT0iDYtBAxGAxtjzqVhlLKISIvK6WGA0lNJFOLw7cGb5x/eMcajjQYDIbWjSfuqfkiMsNaFKlN4lulIrxM2QiLMJaGwWBoe3iiNG4CPgdKRCRXRPJEJNfLcrUoqlaE+pYrSgAAFc5JREFUHyaUzhFmbXCDwdD28GTlvtCmEKQl44edFEcnuvvoovgsFUZMeEAzS2UwGAxNj1l2zgNs4mC1oy+jil8BdAuRzkZpGAyGNohRGvXgI7r3VBk2DlmLvmYRRqcwozQMBkPbo1alYa2O1+bxtflgo5xybCh8KFdCtgoj2nS4NRgMbZC6LI0vAERkfhPJ0iLx8xH8KMeOrv5+uXw635aPJ9DfVIMbDIa2R12BcB8R+RvQR0T+XHWnUuq/3hBIRIYBrwEBgB24VSm1ykr5fR44EygErlFKrfWGDFXkweamNP5rv8jblzQYDIYWS12WxiVAOVqxhNbw8BZPAo8opYahF3x60ho/A734U2/gRuBVL8pQQYCfDV/KGd8nhnOHxTbFJQ0Gg6HFUquloZTaBvxHRDYopX5qQpkUWBFnCAf2W6+nA7OUUgpYISIRItJZKXXAm8KIgJ84GBgXyROThvD1+v31v8lgMBhaKZ60Rl8mIv8FJlnbi4D/U0rleEmmu4CfReRptCU03hqPA/a5HZdqjVVSGiJyI9oSISEh4ZiFUeXl+OAAmx9+NpNsZjAY2jae3AXfQS+8dJH1yAXePZaLisg8EdlUw2M6cAtwt1KqC3A38HZDzq2UekMplaiUSuzQ4dhbfYSoPP3Cx4bNR3dSOWtw52M+r8FgMByPeGJp9FRKzXDbfkRE1h/LRZVSU2vbJyKzgDutzc+Bt6zXaUAXt0PjrTHvcXg3s9VdOPDBJ16vcLv+n6cQ3M6Tj81gMBhaH55YGkUiMtG5ISITgCLvicR+4ETr9cnADuv1t8BVohkL5Hg7nkHGVtqTx49d74UeWqSIIH/jpjIYDG0WT6bMNwOzRCTc2j4MXO09kbgBeF5EfIFirPgE8CM63TYZnXJ7rRdl0BQdBmBv+GivX8pgMBiOBzxpWPgHMFREwqxtr3a4VUotAUbWMK6A27x57WoUHQGgxC+sngMNBoOhbeCxc97byqJFUnQYhxLsfiHNLYnBYDC0CIxzvi6KDpNDMDYf0zLEYDAYwCiNOlFFhzmigvHxabOLFhoMBkMl6nVPiYgNOAvo5n68t3pPtSQyMg6SQwi+RmkYDAYD4FlM4zt0FtNGqLLuaSvnwMED5BhLw2AwGCrwRGnEK6WGeF2SFkg4+eylIzYxSsNgMBjAs5jGTyJyqtclaYEESwn5KrCifYjBYDC0dTyxNFYAs0XEBygDBF020eqLF4IopoAAY2kYDAaDhSeWxn+BcUCQUipMKRXaFhQGDgdBlFBIO2NpGAwGg4UnSmMfsMmqyG472IvwEUWhCjCBcIPBYLDwxD21C/hVRH4CSpyDrT7ltrQQwLinDAaDwQ1PlEaK9fC3Hm0CVZqPAIWqHaaprcFgMGg8aVj4SFMI0tKwF+XhBxQSgI+xNAwGgwHwrCJ8IXrd7koopU72ikQtBEdJPoAJhBsMBoMbnrin7nF7HQDMAOzeEaflYC/Jpx1QoAKM0jAYDAYLT9xTa6oMLRWRVV6Sp8XgKC4AjHvKYDAY3PHEPRXptumDXiApvJbDjxkRGQq8BoQAu4HLnWt5iMgDwHVAOXCHUupnb8mhSl3uKUcbyzY2GJqbsrIyUlNTKS4ubm5RWjUBAQHEx8fj5+fn8Xs8cU+tQcc0BO2WSkHfuL3FW8A9SqlFIjITuBf4h4gMAC4BBgKxwDwR6aOUKveGEI5iS2moAErtbapPo8HQ7KSmphIaGkq3bt0QY+l7BaUUWVlZpKam0r17d4/f54l7yvOzNQ59gMXW67nAz8A/gOnAJ0qpEiBFRJKB0f/f3r0HV1HlCRz//u5NTFwgQcNTAgJFEINAFBwcsBQHZXB8MRANOCKoVdYgcWe3HFlcHwOUVLEsNYyIrkk5Iws+EVAhlIMI+IBhkARjhLAYXGFFAwgYEInk9ds/+iRzifeGa5KbS25+n6qudJ/uvn1+zaXPPae7zwG2RiITWlHbPJVAVY3VNIxpST/88IMVGBEmIqSkpPDNN9/8pP1CvoEgIleKSLeA5btF5C0RWVSvyaq57cIrIABuB3q6+R54b6fXOuDSziAi94tIvojk/9STEUgrywEoJ4GqaqtpGNPSrMCIvMac44ZeW8sBKtwHXwPMA5YCx4HcRuSvjoi8KyI7g0y3AfcCD4hIAdChNg/hUtVcVR2mqsM6d+7c6DxqVQXVKtTgo7LaahrGGAMNN0/5VfWYm88CclV1JbBSRAqbclBVvf4sm4wBEJH+eKMGAnzFP2odAKkuLSK06jSV7vRU1VhNw5i2aO7cubz88sv4/X58Ph85OTkMHz482tmKqgYLDRGJU9UqYDRwf5j7NYmIdFHVw64r9sfwnqQCWA28LCJ/xLsRngZE7NFfra6kwoVpNQ1j2p6tW7eSl5fHjh07SEhI4MiRI1RUhNfwUVVVRVxcxC6TUdVQVK8A74vIEaAc+BBARPrhNVFFyiQRme7mVwEvAKjqLhFZDhTjPcU1PVJPTgFodQUVeI+hDeoRsSeMjTFnMXvNLoq/PtGsn5l+URJ/uGVgg9uUlpbSqVMnEhISAOjUqRMAc+bMYc2aNZSXlzNixAhycnIQEUaNGkVGRgabN29m0qRJ9OrVi9mzZ+P3+0lOTuaDDz5g3759TJ48me+/9x60Wbx4MSNGjGjW2CItZKGhqnNFZAPQHXgnoGt0H/BgpDKkqk8BT4XKEzA3Usc+g2ue+lNWBtf0b/y9EWNM6zRmzBjmzJlD//79uf7668nKyuLaa68lOzubJ554AoDJkyeTl5fHLbfcAkBFRQX5+fkADBo0iHXr1tGjRw/KysoA6NKlC+vXrycxMZGSkhImTZpUt31r0WD9SVX/HiTts8hl5xxSXUml+umenBjtnBjTpp2tRhAp7du3p6CggA8//JBNmzaRlZXFvHnz6NChA/Pnz+fUqVMcO3aMgQMH1hUaWVlZdfuPHDmSqVOncscddzB+/HjAe2kxOzubwsJC/H4/n33W+i6nsdno1hxc81Sc9YtuTJvl9/sZNWoUo0aNYtCgQeTk5FBUVER+fj49e/Zk1qxZZ7y13q5du7r55557jm3btrF27VqGDh1KQUEBTz/9NF27duWTTz6hpqaGxMTW96PUroihVFdQSRxx1lmhMW3Snj17KCkpqVsuLCzkkksuAbz7GydPnmTFihUh9//8888ZPnw4c+bMoXPnznz55ZccP36c7t274/P5WLZsGdXVEbstGzFW0whBqiuoII44vxUaxrRFJ0+e5MEHH6SsrIy4uDj69etHbm4uHTt25LLLLqNbt25ceeWVIfd/+OGHKSkpQVUZPXo0Q4YM4YEHHmDChAksXbqUsWPHnlEzaS0klof+HjZsmDb2JtM3z/yS/YeOkTR9I/27dmjmnBljGrJ7924uvfTSaGejTQh2rkWkQFWHBdvemqdCkJpKKtSap4wxJpAVGiFIdaW7p2GnyBhjatkVMQRfjd3TMMaY+qzQCEFqXE3DCg1jjKljhUYIvtqnp6x5yhhj6tgVMQTRSirVahrGGBPICo0Q/O5GeLzVNIxpkw4cOMBtt91GWloaffv2JTs7m9OnT/9ouyVLlpCdnd1i+Vq9ejXz5s1rsePVZ1fEEEQrqZQ4zouzU2RMW6OqjB8/nnHjxlFSUkJJSQnl5eXMmDGjRY7f0Jvit956KzNnzmyRfARjb4SH4KupxBeXgN/e0zAmut6eCQc/bd7P7DYIbgz9a33jxo0kJiZyzz33AF4fVAsXLuTiiy9m7ty5tG/f/qyHePHFF1m0aBEVFRUMHz6cZ599Fr/fz7Rp09i+fTvl5eVkZmYye/ZsAHr37k1WVhbr169nxowZzJw5kylTprBmzRoqKyt5/fXXGTBgAEuWLCE/P5/FixczdepUkpKSyM/P5+DBg8yfP5/MzExqamrIzs5m48aN9OzZk/j4eO69914yMzObfOrsZ3QI/ppK4s9rfZ2JGWOabteuXQwdOvSMtKSkJHr37s3evXvPuv/u3bt57bXX2LJlS12Pti+99BLgjQaYn59PUVER77//PkVFRXX7paSksGPHDiZOnAh4fVzt2LGDadOmsWDBgqDHKi0tZfPmzeTl5dXVQFatWsW+ffsoLi5m2bJlbN26tVHnIRiraQRTU0McVZznBl8xxkRRAzWCc9WGDRsoKCio65uqvLycLl26ALB8+XJyc3OpqqqitLSU4uJiBg8eDJzZtTpQ16X60KFDWbVqVdBjjRs3Dp/PR3p6OocOHQJg8+bN3H777fh8Prp168Z1113XbLFFpaYhIreLyC4RqRGRYfXWPSIie0Vkj4j8MiB9rEvbKyKRbdCrqQQgIcFqGsa0Renp6RQUFJyRduLECQ4ePMiWLVvIyMggIyODr7/+Ouj+qsqUKVMoLCyksLCQPXv2MGvWLL744gsWLFjAhg0bKCoq4qabbgrZtTpQN2qg3++nqqoq6LESAn7ctkRfgtFqntoJjAc+CEwUkXRgIjAQGAs8KyJ+EfEDzwA3Aul4Q8KmRypzWuU9IZGQeH6kDmGMOYeNHj2aU6dOsXTpUsC7Mf3QQw+RnZ3N9OnT6wqDiy66KOT+K1as4PDhwwAcO3aM/fv3c+LECdq1a0dycjKHDh3i7bffjkj+R44cycqVK6mpqeHQoUO89957zfbZUSk0VHW3qu4Jsuo24FVVPa2qXwB7gZ+5aa+q/q+qVgCvum0j4vtyr+RPtELDmDZJRHjjjTdYsWIFaWlppKSk4PP5ePTRR4Nuv2TJElJTU+umpKQknnzyScaMGcPgwYO54YYbKC0tZciQIVx++eUMGDCAO++8k5EjR0Yk/xMmTCA1NZX09HTuuusurrjiCpKTk5vnw1U1ahPwHjAsYHkxcFfA8p+BTDc9H5A+GVgc4jPvB/KB/F69emljfHvksOb/561atGlFo/Y3xjRNcXFxtLNwhi1btmivXr20oKAg2lkJ23fffaeqqkeOHNG+fftqaWlp0O2CnWsgX0NctyN2I1xE3gW6BVn1qKq+FanjqmoukAveeBqN+YyOKZ0Z+vuIZdEY08qMGDGC/fv3RzsbP8nNN99MWVkZFRUVPP7443TrFuxy/NNFrNBQ1esbsdtXQM+A5VSXRgPpxhhj6mnO+xiBzrX3NFYDE0UkQUT6AGnAR8B2IE1E+ojIeXg3y1dHMZ/GmAjTGB5V9FzRmHMcrUdufy0iB4CfA2tFZB2Aqu4ClgPFwF+B6aparapVQDawDtgNLHfbGmNiUGJiIkePHrWCI4JUlaNHj5KY+NNeLbAxwo0x55zKykoOHDhwxjsMpvklJiaSmppKfHz8GekNjRFub4QbY8458fHx9OnTJ9rZMEGca/c0jDHGnMOs0DDGGBM2KzSMMcaELaZvhIvIN0BT3sjpBBxppuy0FhZz22Axtw2NjfliVe0cbEVMFxpNJSL5oZ4giFUWc9tgMbcNkYjZmqeMMcaEzQoNY4wxYbNCo2G50c5AFFjMbYPF3DY0e8x2T8MYY0zYrKZhjDEmbFZoGGOMCZsVGkGIyFgR2SMie0VkZrTz01xE5C8iclhEdgakXSgi60WkxP29wKWLiCxy56BIRK6IXs4bT0R6isgmESkWkV0i8juXHrNxi0iiiHwkIp+4mGe79D4iss3F9pobZgA3FMFrLn2biPSOZv6bQkT8IvKxiOS55ZiOWUT2icinIlIoIvkuLaLfbSs06hERP/AMcCOQDkwSkfTo5qrZLAHG1kubCWxQ1TRgg1sGL/40N90P/FcL5bG5VQEPqWo6cBUw3f17xnLcp4FfqOoQIAMYKyJXAf8BLFTVfsC3wH1u+/uAb136Qrdda/U7vOETarWFmK9T1YyA9zEi+90ONQ5sW53wxvhYF7D8CPBItPPVjPH1BnYGLO8Burv57sAeN58DTAq2XWuegLeAG9pK3MA/ATuA4XhvBse59LrvOd44NT9383FuO4l23hsRa6q7SP4CyAOkDcS8D+hULy2i322rafxYD+DLgOUDLi1WdVXVUjd/EOjq5mPuPLgmiMuBbcR43K6ZphA4DKwHPgfK1BvQDM6Mqy5mt/44kNKyOW4WfwJmADVuOYXYj1mBd0SkQETud2kR/W7beBqmjqqqiMTkM9gi0h5YCfyLqp4Qkbp1sRi3qlYDGSLSEXgDGBDlLEWUiNwMHFbVAhEZFe38tKCrVfUrEekCrBeR/wlcGYnvttU0fuwroGfAcqpLi1WHRKQ7gPt72KXHzHkQkXi8AuMlVV3lkmM+bgBVLQM24TXNdBSR2h+KgXHVxezWJwNHWzirTTUSuFVE9gGv4jVRPUVsx4yqfuX+Hsb7cfAzIvzdtkLjx7YDae6pi/OAicDqKOcpklYDU9z8FLw2/9r0u90TF1cBxwOqvK2GeFWKPwO7VfWPAatiNm4R6exqGIjI+Xj3cHbjFR6ZbrP6Mdeei0xgo7pG79ZCVR9R1VRV7Y33f3ajqv6GGI5ZRNqJSIfaeWAMsJNIf7ejfSPnXJyAXwGf4bUDPxrt/DRjXK8ApUAlXnvmfXjtuBuAEuBd4EK3reA9RfY58CkwLNr5b2TMV+O1+xYBhW76VSzHDQwGPnYx7wSecOl9gY+AvcDrQIJLT3TLe936vtGOoYnxjwLyYj1mF9snbtpVe62K9HfbuhExxhgTNmueMsYYEzYrNIwxxoTNCg1jjDFhs0LDGGNM2KzQMMYYEzYrNIwJg4hUu55Ea6cGez8Wkd+KyN3NcNx9ItKpqZ9jTHOxR26NCYOInFTV9lE47j685+mPtPSxjQnGahrGNIGrCcx3Yxp8JCL9XPosEfm9m/9n8cbzKBKRV13ahSLypkv7u4gMdukpIvKOeONgPI/3Qlbtse5yxygUkRzXKaFfRJaIyE6Xh3+NwmkwbYgVGsaE5/x6zVNZAeuOq+ogYDFeT6v1zQQuV9XBwG9d2mzgY5f278BSl/4HYLOqDsTrS6gXgIhcCmQBI1U1A6gGfoM3XkYPVb3M5eGFZozZmB+xXm6NCU+5u1gH80rA34VB1hcBL4nIm8CbLu1qYAKAqm50NYwk4BpgvEtfKyLfuu1HA0OB7a6H3vPxOqJbA/QVkaeBtcA7jQ/RmLOzmoYxTach5mvdhNfnzxV4F/3G/FgT4L/VG6EtQ1UvUdVZqvotMAR4D68W83wjPtuYsFmhYUzTZQX83Rq4QkR8QE9V3QT8G14X3O2BD/Gal3DjPxxR1RPAB8CdLv1G4AL3URuATDduQu09kYvdk1U+VV0JPIZXMBkTMdY8ZUx4zncj4dX6q6rWPnZ7gYgU4Y3NPanefn7gRRFJxqstLFLVMhGZBfzF7XeKf3RlPRt4RUR2AX8D/g9AVYtF5DG8Udp8eD0VTwfKgRdcGnjDExsTMfbIrTFNYI/EmrbGmqeMMcaEzWoaxhhjwmY1DWOMMWGzQsMYY0zYrNAwxhgTNis0jDHGhM0KDWOMMWH7f8A8yua/gBWPAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zysaW8JlxPxC",
        "colab_type": "text"
      },
      "source": [
        "What should immediately jump out is that Q-learning learned to go up, then quickly go to the right, skirting the cliff edge, until it hits the wall and goes down to the goal. The policy further away from the cliff is less certain.\n",
        "\n",
        "On the other hand, SARSA appears to avoid the cliff edge, going up one more tile before starting over to the goal side. This also clearly solves the challenge of getting to the goal but does so at an additional -2 cost over the truly optimal route.\n",
        "\n",
        "Why do you think these behaviours emerged the way they did?\n",
        "\n",
        "In video below, you can watch how Q-Learning algorithm learns step by step! Enjoy it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dxsKPFC_El86",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 321
        },
        "cellView": "form",
        "outputId": "ffe6bdda-f7ed-4fe4-8f44-c40d5c7ec6cf"
      },
      "source": [
        "#@title (Video) How Q-Learning is Learning?\n",
        "from IPython.display import YouTubeVideo\n",
        "YouTubeVideo('Vto8n9C7DSQ')"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "\n",
              "        <iframe\n",
              "            width=\"400\"\n",
              "            height=\"300\"\n",
              "            src=\"https://www.youtube.com/embed/Vto8n9C7DSQ\"\n",
              "            frameborder=\"0\"\n",
              "            allowfullscreen\n",
              "        ></iframe>\n",
              "        "
            ],
            "text/plain": [
              "<IPython.lib.display.YouTubeVideo at 0x7f50e91f90f0>"
            ],
            "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQEDEQH/xAAbAAADAQEBAQEAAAAAAAAAAAAAAQIDBQQGB//EAEQQAAIBAwEEBgUJBgQHAQAAAAABAgMEEQUWMZLSEiEzU1TRExcyQVEGIiNDYXFykbEUFTRSgcEkNXPxJUJiY4Lh8KH/xAAVAQEBAAAAAAAAAAAAAAAAAAAAAf/EABQRAQAAAAAAAAAAAAAAAAAAAAD/2gAMAwEAAhEDEQA/APz8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADqfuK67yj+b8h/uC67yj+b8gOUB1dn7vvKPE/Iez133lDifkByQOvs7ed5Q4n5D2bvO8ocT8gOOB2dmr3vKHE/IezF73tvxPyA4oHb2Xve9t+KXkPZW+7234peQHDA7uyl93tvxS8h7J3/e23FLyA4IHf2Rv++tuKXkPZDUO+tuKXKB8+B9BsfqHfW3FLlHsdqPfWvFLlA+eA+i2N1HvrXilyhsZqPfWvFLlA+dA+j2L1LvrXjlyj2K1LvrXjlygfNgfSbE6l39pxy5R7E6l39pxy5QPmgPpdiNT7+045cobD6n39pxy5QPmgPpth9T7+045cobD6n39pxy5QPmQPpthtT7+045cobDan39pxy5QPmQPpthtT7+045co9htT7+045coHzAH0+w2p9/accuUWw2p9/accuUD5kD6bYbU+/tOOXKGw2p9/accuUD5kD6bYbU+/tOOXKGw+p9/accuUD5kD6XYfU+/tOOXKGxGp9/accuUD5oD6XYjUu/tOOXKLYnUu/tOOXKB82B9JsVqXfWvHLlFsXqXfWvHLlA+cA+j2L1HvrXilyi2N1HvrXilygfOgfQ7Haj31rxS5RbH6h31txS5QPnwPoNkNQ7624pcotkb/AL624peQHAA72yd/3ttxS8hbKX/e2/FLyA4QHc2Vvu9t+KXkLZe97234peQHEA7WzF73tvxPyFs1e95Q4n5AcYDsbN3neUOJ+QtnbzvKHE/IDkAdbZ677yjxPyFs/d95R4n5AcoDq/uC67yj+b8hfuK6/no/m/ID6JIpIEikggSLSBIpIASLSBItIASKSBItIASLSBItIBJFpAkWkAkikhpFJACRSQ0ikgEkNIpIrAEpDwUkPACwPBWB4AnA8FYHgCcBgvAYAnAYLwGAJwGCsDwBGAwXgWAJwLBeAwBGBYLwGAM8CwaYFgDPAmjTAmgMsCaNMCaAzaJaNWiWgMmiWjVoloDJoho2aIaAyaIaNmiGgMmiGjZohoDFolo1aIaAyaIaNmiGgMmiGjVoloDJolo0aJaAaLSEkWkA0ikgSLSAEi0gSLSAEi0gSLSAEi0hJFpACRaQJFpAJItIEikgBIpIEi0gEkUkCRSQCSHgpIeAFgeB4HgBYHgaQ8ATgeCsBgCcDwVgMATgMFYDAE4DBWAwBOBYLwGAIwLBeAwBGBYLwLAEYFgvAsAQ0S0aNCaAyaFg0aJaAzaJaNWiWgMmiGjZohoDJoho2aIaAyaIaNWiGgMmiGjZohoDFoho2aIaAxaJaNWiGgMmiGjVohoCki0hJFpANItISRaQDSLSEkaJACRaQJFpACRaQJFpACRaQJFJACRSQJF4ASRSQ0ikgEkUkCRSQCSGkNIrACwPA8DwBOB4KwGAFgeB4DACwGCsBgBBgeAwAsBgeAwBOAwVgMATgWCsBgCMCwXgMARgTReBYAzwJo0wS0BDRLRo0S0Bm0S0atEtAZNEtGrRDQGTRLRq0S0Bi0Q0bNENAYtENGzRDQGLRDRs0ZtAZNENGrRDQGTRDRq0Q0BSRaRKRokA0i0hRRaQFJFpCSLSAaRaQki0gGkWkJItIBpFJAkWkAJFJAkUkAJFJAkUkAkikhpDSAWCkgwVgBYHgMDwAsDwPAwFgMDGAsAMeAJDBWAwBOAwVgMASBWBYAnAYKEBOBF4FgCRYKwICcEtF4E0BDRLRo0JoDNolo0aJaAzaJaNGiWgMmiWjRoloDJoho1aJaAxaIaNWiGgMmiGjVoiSAxaIaNmjNoDJoho1kjNoCkjRImKNIoBpGiRMUaJANItIUUWkBSRaQki0gGkWkJItIASLSEkWkAJFJAkUkAJFJAkNIASKSBDABpAkUAsDGAAA8DAWBgMBAMAEAwAQDABAMAJwGChASBQsASJorAgJwLBZLQEtEtFiwBDJaLaE0Bm0S0aNEsDNoho1aIaAzaIaNWiGgMpIho1aIaAxaIaNWiJIDJozaNWiGgMmjNo1aIkgKSLSJijSIFJFpEpGkUBSRaQolpAUkWkJItIBpFJCSLSAaRSQJFIASLSEkUgBIpIEMAKSBIYAMBpAGB4AYCGAwEMeAAQYGMBYDAwAWAwMAJwGChAIRQYAkRQgELBQgJEU0ICWiWWJoCGS0WJgQyWi2SwM2iWaMloDJolo0aIYGbREkaszaAyaIaNWiGgMWiJI1aM5IDJoho1kjOSAqKLRMTSKApI0RCLQFpGiRETSIFJFolFJgWi0REtAUikhItACKQikA0NAhgMZOQ6SAtDIU0WnkBjAAAYmxdICwEmMAAYmwACemCmBYgTGAgGACAYgEIonIAIYASIoQEiKZLYCZLHkAIaJZbJYEMllSZDkgEyGisiYGbRDRoyGBm0ZyNJMybAlmbRoyGBkyGaszkA4mkSYloB5UYtyaSSyyYXFGbSjVg29yyKv2FT8D/Q41r/ABNL8SA+jgaoypmyAeM+9E9LHvRndVXQo+k/lks/duKmk0px64y68gbwl0iVdUXU9GqsOnnGM+8ig/n9F7zj7tVf+t/cD6SJojOmaoA/rgzo3FKs8UqsZ4+DLluf3HD0J4uZ/hA+hiUTAsDOUW9zRDp1Pdj8zx6104UYunOUHnrcXg89rUq/uK4nKpNzUniTk8rcB1Y06n2fmVQqxqLMJxkl1dTyfI/tF06j/wATWx8Omz6DQYYt55/mA6yBjSFIDNqUtxLpVN+U/wCpzNcv3SpRo0JuMn7WOpo+apXVzRrqpQqz9JnqSbefsx7wPuYyaeH1M3juPHXquN9Rp49qDbXwPXDcBRnLL6kaHP1hzjaZpylCWd8Xhgeh06vuS/MUadVb0vzPka11eR3XVZf+bOtplSvZ6ZUvrutUqzqfNpQnJtf/AD/RAd2DfvNDy2DlO2pzm8yay2esAEMAEAxATIyU4yqOCkukllrPWaTOfavOrXP4IgdBDIhuLAQiiWBEmYTn170jWo8I8aXpa6j7sgelKSWXjBR4aV6qmrVLfK6HR6K/Et/9/wAj2R+ADZMiyJ7gPPNvJn6Oo1lYx954tYnKPo1Ftbzy6pVqUvk/QlTqShJzSbi8P3gdf0dRfD8x9a3nyOn3F3XrOmqtWfVn2n1H0unyrO2ar5bUsJve0B6GZzNWZVNwHnl0pPEVlkulU9+F/U8VxfSpVpwS3MjWLitCxoSpScVUx0pL7twHv6ElvaJaOFaOrVkvQyamvtwdyLk4LprEsdf3gTIzkayIYBE0iTEuIEV+xqfgf6HFtv4in+JHbr9hU/A/0OJbdvT/ABID6OluRujGkvmo3QHk1P8Agav9P1RhpF0pxdtUf2w8j0ap/AVf/H9UcKDlCcZweJReUwPpqccVlnejiT/zOX+s/wBTtW1WNzTp1o9T3SXwZxZ/5nL/AFX+oH0lHrijdGND2UboCJ7n9xwdE/iZfhO/NdT+44OiL/Ey/CB9FA0RENxogObq6zRR5KK/4Fc/if8AY9mq9keSiv8AgVx+J/2A5FNfOPpNHWKLR89TXWfRaQvogOkKW4ZM9zA+b1WhVur1UqEHOb//AD7WXSt7HQ8Vbqar3f8Aywj/AMv3ebPfqNSrZWsnaUZ1K9Te4RbaPknZ6jUqupVtrmUm8tunLyA+g066ne37r1Op4wl8EfQx3HzWiUK1GovS0alNf9UWj6WO4Cjwat122D3ng1bsGB85Rt/2u8p0FulLr+xe892sVVUvYW0OqnQSWFuz/tg10OmoSuLqe6EcL9X/AGOdByq13Ul7U5dJgfT2CxbQXwR6TCz7CJuAAAAAAAGdTcc6z69UufwI6VT2Wc2x/wA0ufwID2wl1I0TPHF1Euznws2hKXvjJfegNyXuGhS3AeS6n0YvG8xi5W9nWrwpyqTjF9CEYtuT+5faVc/OqqKPLrF5WsoUaNrJRm+uTwn1f7gcPTaWo06npKlpdKcZ9PMqMln4+4+t34lhpSW5rDPmlq2oqS6VdY/BHyO1pd3K7t5Kq06kJdbxjq9wHtInuLW4mQHC1n2oBXnaQ0Sg72jKrT6WFGL9/X9qHrS+dAx1GlUq6BbxpU5VH008Qi2/eArDVNLt+lG0tZUnLflJt/1ydSlcQuoekpyyk+uLWMHzumaPd16+Z0Z04R3uccfqd6zsnZ+kzLPSx1AehmVXcbGNX2WB81qU+hct4zkLfVoQpegr01UpbsSRtdVYurKNS3pTw97zn9TPVdNhKlSuLCi+hJfOjHLf2MK9FvV05S6dOLpN/fjyPf1YynlHMttHrK3i6nzZPr6PwOhRpOjRjTby4hAyJGkjOQDiaRIiaIAnTVSDi20msdR4f2OlSrx6KfU/ezoo89VfTxA9tNdSNYkQ3I0QGdxQjXpShJvovfg4tahGlUcY5x9p9A1804t0vpmBNtcVbVt0msS3prqHbR9JdKc+tuWX95ng9NmvpkB3qaxFGqIh7KLQEzWYvDx1HOsLaFC4l0E/h1nUe48dsv8AEyA98UUJFAc7VezOUq9SNtO3WPRzeX1dZ2NTi5U+pHHdOS3oDGEcHe0nsjkKnJ7kzs6XFxp9aA6BMllFA9wHFvtRurep0abg1/1RPJ++r/8A7XB/7N9TX0x4eiB77S7r3lxT9O4/MeV0Vg7sVhHB0xYro74AeHVFmge48moxcqHUgOHC5q0repbxa6FTf1dZlSjiSNJwcX1oIL5yA+itOwibGNp2ETYAAACgAAIUllHAvqtW0vHVoNKTWHlZTR9AcLVF9MB5v31ff9rg/wDZ7bDULm5n0ajgl9kTl9FHv0pfSgdtLqE0UDA4epSnSuYVKcmpR/I59xUnc1nVqY6T6updR0tWXz0c3AGMqaOlo9LoVJTi2m1h/A8WDqaUt4HSwS0WSwONrNPpQX2HNoX93bU/R05RcVuUlnB2NVXzDjOIG8dVvG8P0fCdS2lOrTUpvLOIl1o7dn2KA1ZlUXzWbMzn7LA+avoP07wRb3dxbdVOS6P8rXUeq8WazPK4ge6hf168+jKMEvsyezHUc2zX0p03uAzZnI1ZmwCJoiImkQLR56vbxPQjCr20QPbDcjVGcNyNEBT9k5dWl6ScmvcdR7jy0FmvJActx6Lwz0WXbI0vaHQl0kuozs+2QH0EPZRaIh7JaAb9lnjtv4mX3nsfss8dt/Ey+8DoIYkMBSgpLDWTzXdGCpNpHrRhedkwIs6MHSTcUeuMFFYSwYWXZI9IAN7gB7gODqXbHjPZqXbnjA9mm9ujvHC03t0d0AE0mutZGAVx9UhGM1hYMLGg6tVfA9OqrM0erTqPQpZe9hHrhFRikvcUABSAYAIAAIDh6p2x3Dh6p2wHgPfpfangPfpfagdsQxAcbVvbRzDqar7aOd6OT9wEM6mlbmc1wkt6OlpXvA6TENiA5mq+wcdnY1X2DjsAW87Vn2KOKt6O3adigNWZz9lmjM5+ywODd9szzM9N32zPOwNrPtTps5tn2p0nuAzZnI1ZnIAiaRM4mkQNInnqr6eJ6EPoJyyBrDcjREIuIFvceW2X08j1IcKaTyt4Cr0lUptM5tCk6dyk/idhE+hh0+ljrA1h7KLRKKQDfss8dsv8RI9oo04qWUusDRDEMCkYXi+iZuhSipLDAxs+yR6SYRUVhFAMAADh6lFutuPF0X8D6WdCnN5kif2Wl/KBydNi1X3HcM4UKdN5jE0AYgGFc2+g514rB76UejTS+wbhGTTa60MIYAAUAAgAAAAOLqcJOrlJnaInShP2kEfNejl8Ge7TINVetHU/Zqf8pUKUIPKQFiGAHKvqbqV4rB66dtBQScUbuEW8tdYAeW4toSpPETzafBwlJNHSfWQoKLykA2SUyWBz9Tg5Q6kcd05fBn0s4qSw1kydCH8oHBp0JyklhnZoQ6FJI0VKMdyGwIZEutFslgca+oSU20jxOEvgfQzipb0YuhD4Ac6zoyUuk0e5ldFR3EyAzkZyNJGcgCJpEziaRA0iXEhFoDRGiM0WgNEWjNFoC0UiUUgKRSJRSApFIlFIBjEMCkMSGAwAAGAAAAAAAAAUwAAAAABDAAAQxAAAAAAAEAAACABAAmMlgITGIBMllMlgSxMbJYEsllMlgQyWUyWBmyJFsiQGbIkaSM5AZszkaSM5AETRGSNEBpE0RnE0QGiLRmi0Boi0Qi0BaKRCLQFIpEopAUhoSGBQxDAaKJQwGMQwGAhgAAAAAAADEAUwEADAAABAAAAAEAAAAACABDEAEsYmAhDEBLExsTAlkspksCWSymQwJZDLZDAhkSKZLAzZmzRmbAhmbLZmwCJojKJpEDWJaM4loDVFozRaA0RaIRSA0RSIRaApFIlFICkUiUMCkUShgUMkaAoYgAYxAAwAAAAAAAAAAAAoAAAAAAgAAAAAAAQAACAAExDZIASxiYCZLGxMCWSymSwJZLKZDAlkMtkSAhkMpkMCGZs0kZyAhmci2ZyAUTSJnEuIGsS4mcTRAaRNEZItAaotGaLQGiKRCKQFopEIpAWiiEUgKKRI0BQyRgUMkYFAIYDAQwAAAAAAAAAAAAAAAAAAAAABAAAAgAQCYAIBABLGyQBksbJYCZLKZDATJY2SwJZEimQwIZDKZDAmRmy5GbAiRnIuRnIBI0ifFL5V3y+qtuGXmUvlbfr6m24ZeYH26NEfCr5YagvqbbhlzDXyy1FfU2vDLmA+8Roj4FfLTUV9Ta8MuYa+WupL6i04JcwH6Ai0z89Xy31JfUWnBLmHtzqfcWnBLmA/REUj86261TuLTglzD281TuLPglzAfo6KR+b7e6p3FnwS5h7far4ez4JcwH6Sho/Ntv8AVfD2fBLmD1gat4ey4J8wH6Who/NPWDq3h7LgnzB6wtW8PZcE+YD9NGfmXrD1bw9lwT5g9Ymr+HsuCfMB+nDPzH1iav4ey4J8wesXV/D2PBPmA/TwPzD1i6v4ex4J8w/WNq/hrHgnzAfp4H5h6xtX8NY8E+YPWNq/hrHgnzAfp4z8v9Y2seGseCfMHrH1jw1jwT5gP1AD8v8AWPrHhrHgnzD9Y+seGseCfMB+ngfmHrH1jw1jwT5g9Y+seGseCfMB+ngfmHrH1jw1jwT5g9Y+seGseCfMB+ngfmHrH1jw1jwT5hesfWPDWPBPmA/UBH5h6xtY8NY8E+YPWNrHhrHgnzAfp4H5h6xtX8NY8E+YPWNq/hrHgnzAfpwH5j6xtX8NY8E+YXrF1fw9jwT5gP04R+ZesTV/D2XBPmD1iav4ey4J8wH6YI/M/WJq/h7LgnzB6w9W8PZcE+YD9KYj819YWreHsuCfMHrB1bw9lwT5gP0hiZ+b+sDVvD2XBPmDb/VfD2fBLmA/RmSz862+1Xw9nwS5hbe6p3FnwS5gP0Nks/PdvNU7iz4JcwtutU7iz4JcwH6AyGfA7c6n3FpwS5hP5b6k/qLTglzAfdsiR8M/lrqT+otOCXMJ/LPUX9Ta8MuYD7dmbPin8stRf1Nrwy5hP5Yag/qbbhlzAfZMzkfHv5W37+ptuGXmJ/Ku+f1Vtwy8wOEAAFAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB//9k=\n"
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IAiBC-Afzvi1",
        "colab_type": "text"
      },
      "source": [
        "#References\n",
        "\n",
        "1. Neuromatch [RL Tutorial](https://github.com/NeuromatchAcademy/course-content/tree/master/tutorials/W2D5_ReinforcementLearning)\n",
        "2. [RL an introduction -solutions](https://github.com/JaeDukSeo/reinforcement-learning-an-introduction/blob/master/chapter02/TenArmedTestbed.py)\n",
        "3. [RL an introduction, second edition](http://www.incompleteideas.net/book/the-book.html)"
      ]
    }
  ]
}