11_NLGeneration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center>\n",
    "<h2>Applying Imitation Learning to Natural Language Generation</h2>\n",
    "<h3>[Lampouras and Vlachos 2016](https://aclweb.org/anthology/C/C16/C16-1105.pdf)</h3>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Natural Language Generation <i> (concept-to-text)</i> ###### \n",
    "\n",
    "The natural language processing task of generating text from a meaning representation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "<div>\n",
    "    <div style=\"display:inline-block;\">\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{blue}{\\text{type = \"hotel\"}}\\\\\n",
    "        & \\color{green}{\\text{count = \"182\"}}\\\\\n",
    "        & \\color{red}{\\text{dogs_allowed = dont_care}}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "    </div>\n",
    "        <font size=\"300\">&#10143;</font>\n",
    "    <div style=\"display:inline-block; vertical-align: middle;\" >\n",
    "        <p style=\"display:block; width:400px; word-wrap:break-word;\">\n",
    "        There are <font color=\"green\"><u>182</u></font> <font color=\"blue\"><u>hotels</u></font> if you <font color=\"red\"><u>do not care</u></font> whether dogs are allowed.\n",
    "        </p>\n",
    "    </div>\n",
    "</div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Transition system? ###\n",
    "\n",
    "<b>State:</b> meaning representation, incomplete sentence.\n",
    "<br>\n",
    "<br>\n",
    "<span class=\"fragment\" data-fragment-index=\"1\"><b>Action space:</b></span>\n",
    "<ul>\n",
    "<li class=\"fragment\" data-fragment-index=\"2\"><b>What to say?</b>: content prediction actions a<sub>c</sub>, and</li>\n",
    "<li class=\"fragment\" data-fragment-index=\"2\"><b>How to say it?</b>: word prediction actions a<sub>w</sub> .</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### NLG formulation ### \n",
    "\n",
    "\n",
    "<p style=\"border:3px; border-radius: 25px; background-color:lightgrey; border-style:solid; border-color:black; padding: 0.5em; font-size: 70%\">\n",
    "\\begin{align}\n",
    "& \\textbf{Input:} \\; \\text{meaning representation} \\; MR \\; \\text{with set of attributes} \\; C, \\; \\\\ \n",
    "& \\quad \\quad \\quad \\text{word action dictionaries} \\; D_c,\\;\\forall c \\in C\\\\\n",
    "& \\textbf{Output:} \\; \\text{action sequence} \\; A \\\\\n",
    "& \\; \\\\\n",
    "& \\;\\;  \\quad \\mathbf{do} \\\\\n",
    "& \\;\\;  \\quad \\quad \\text{predict attribute} \\; c ∈ C ∪ {END_{attr}} \\\\\n",
    "& \\;\\;  \\quad \\quad \\text{append} \\; a_{c} \\; \\text{to} \\; A_{c} \\\\\n",
    "& \\;\\;  \\quad \\quad \\text{remove} \\; c \\; \\text{from} \\; C \\\\\n",
    "& \\;\\;  \\quad \\mathbf{while} \\; ac ≠ END_{attr} \\\\\n",
    "& \\;\\;  \\quad \\mathbf{for} \\; a_{c} \\; in \\; A_{c} \\; \\mathbf{do} \\\\\n",
    "& \\;\\;  \\quad \\quad \\mathbf{do} \\\\\n",
    "& \\;\\;  \\quad \\quad \\quad \\text{predict word} \\; w ∈ D_{c} ∪ {END_{word}} \\\\\n",
    "& \\;\\;  \\quad \\quad \\quad \\text{append} \\;a_{w} \\; \\text{to} \\; A_{w} \\\\\n",
    "& \\; \\quad \\quad \\mathbf{while} \\; a_{w} ≠ END_{word} \\\\\n",
    "& \\; \\quad A = (A_{c}, A_{w}) \\\\\n",
    "\\end{align}\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps; color: white\">...</span></h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step1.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "    <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "    \\begin{align}\n",
    "    & \\text{Predicate: INFORM}\\\\\n",
    "    & \\text{______________________}\\\\\n",
    "    & \\text{name = X-name-1}\\\\\n",
    "    & \\text{serves = food}\\\\\n",
    "    & \\text{eattype = restaurant}\\\\\n",
    "    & \\text{near = riverside}\\\\\n",
    "    & \\text{area = X-area-1, X-area-2}\n",
    "    \\end{align}\n",
    "    </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Content</span>: name</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step2.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "    <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "    \\begin{align}\n",
    "    & \\text{Predicate: INFORM}\\\\\n",
    "    & \\text{______________________}\\\\\n",
    "    & \\color{red}{\\textbf{name}} \\text{ = X-name-1}\\\\\n",
    "    & \\text{serves = food}\\\\\n",
    "    & \\text{eattype = restaurant}\\\\\n",
    "    & \\text{near = riverside}\\\\\n",
    "    & \\text{area = X-area-1, X-area-2}\n",
    "    \\end{align}\n",
    "    </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Content</span>: eattype</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step3.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name}}\\text{ = X-name-1}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype}}\\text{ = restaurant}\\\\\n",
    "        & \\text{near = riverside}\\\\\n",
    "        & \\text{area = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\"></span>and so on</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step4.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name}}\\text{ = X-name-1}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype}}\\text{ = restaurant}\\\\\n",
    "        & \\color{red}{\\textbf{near}}\\text{ = riverside}\\\\\n",
    "        & \\color{red}{\\textbf{area}}\\text{ = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Word:</span> X-name-1</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step5.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name = X-name-1}}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype}}\\text{ = restaurant}\\\\\n",
    "        & \\color{red}{\\textbf{near}}\\text{ = riverside}\\\\\n",
    "        & \\color{red}{\\textbf{area}}\\text{ = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Word: EoS</span></h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step6.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name = X-name-1}}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype}}\\text{ = restaurant}\\\\\n",
    "        & \\color{red}{\\textbf{near}}\\text{ = riverside}\\\\\n",
    "        & \\color{red}{\\textbf{area}}\\text{ = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Word:</span> is</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step7.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name = X-name-1}}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype}}\\text{ = restaurant}\\\\\n",
    "        & \\color{red}{\\textbf{near}}\\text{ = riverside}\\\\\n",
    "        & \\color{red}{\\textbf{area}}\\text{ = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Word:</span> a</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step8.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name = X-name-1}}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype}}\\text{ = restaurant}\\\\\n",
    "        & \\color{red}{\\textbf{near}}\\text{ = riverside}\\\\\n",
    "        & \\color{red}{\\textbf{area}}\\text{ = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Word:</span> restaurant</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step9.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name = X-name-1}}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype = restaurant}}\\\\\n",
    "        & \\color{red}{\\textbf{near}}\\text{ = riverside}\\\\\n",
    "        & \\color{red}{\\textbf{area}}\\text{ = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\">Word: EoS</span></h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq_step10.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name = X-name-1}}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype = restaurant}}\\\\\n",
    "        & \\color{red}{\\textbf{near}}\\text{ = riverside}\\\\\n",
    "        & \\color{red}{\\textbf{area}}\\text{ = X-area-1, X-area-2}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><h3><span style=\"font-variant: small-caps;\"></span>and so on</h3>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq1.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{red}{\\textbf{name = X-name-1}}\\\\\n",
    "        & \\text{serves = food}\\\\\n",
    "        & \\color{red}{\\textbf{eattype = restaurant}}\\\\\n",
    "        & \\color{red}{\\textbf{near = riverside}}\\\\\n",
    "        & \\color{red}{\\textbf{area = X-area-1, X-area-2}}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Word actions are chosen based on content actions.<ul><li>Content actions precede word actions.</li><li class=\"fragment\" data-fragment-index=\"5\">The length of the transition sequence is variable .</li><li class=\"fragment\" data-fragment-index=\"6\">Content is bound, but words are unbounded.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<ul><li>Instead, alternate between content and word actions?</li></ul>\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"images/NLG_altActionSeq.png\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<span>Different classifiers are required for content and word actions.</span><ul><li class=\"fragment\" data-fragment-index=\"3\">Actually, a different word classifier per content action.</li><li class=\"fragment\" data-fragment-index=\"4\">Doesn't affect imitation learning!</li></ul>\n",
    "<br>\n",
    "<br>\n",
    "<span class=\"fragment\" data-fragment-index=\"5\">Action space:</span><ul><li class=\"fragment\" data-fragment-index=\"5\">Content actions limited by the MR.</li><li class=\"fragment\" data-fragment-index=\"6\">Word actions limited to dataset vocabulary.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Loss function? ###\n",
    "\n",
    "<b>BLEU:</b> % of n-grams predicted present in the gold standard,<br> i.e. $L=1-BLEU(s_{final}, \\mathbf{y})$</span></li>\n",
    "\n",
    "Same as NLG's evaluation metric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Content actions are ignored by loss function.\n",
    "- Indirectly costed by their impact on word actions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "<img src=\"images/toBeAnimated/altContent1.png\" width=\"60%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "<img src=\"images/toBeAnimated/altContent2.png\" width=\"60%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "<img src=\"images/toBeAnimated/altContent3.png\" width=\"60%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "No explicit supervision on how each word is aligned to which attribute.\n",
    "- Evaluate how good the complete final sentence is.\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq1.png\" width=\"130%\">\n",
    "<img src=\"images/toBeAnimated/NLG_actionSeq2.png\" width=\"130%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "The loss function also penalizes undesirable behaviour: \n",
    "<ul>\n",
    "<li>Repeating the same word, etc.</li>\n",
    "</ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Expert policy? ### \n",
    "\n",
    "For word actions:\n",
    "- Which word would make sentence best match gold?\n",
    "- Only consider words in gold standard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><b>Gold standard:</b> \"X-name-1 is at the side of the river\"\n",
    "<br>\n",
    "<br>\n",
    "<img src=\"images/toBeAnimated/nlgWordRolloutsPre.png\" width=\"70%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><b>Gold standard:</b> \"X-name-1 is at the side of the river\"\n",
    "<br>\n",
    "<br>\n",
    "<img src=\"images/toBeAnimated/nlgWordRollouts.png\" width=\"70%\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If content actions:<ul><li class=\"fragment\" data-fragment-index=\"2\">which content actions to include,</li><li class=\"fragment\" data-fragment-index=\"2\">and in what order.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<span>Need alignments between MR and words in gold standard.</span><ul><li class=\"fragment\" data-fragment-index=\"4\">Loss requires no explicit supervision on alignments.</li><li class=\"fragment\" data-fragment-index=\"5\">Can employ heuristics on the gold standard!</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "<center>\n",
    "    <div style=\"display:inline-block;\">\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{blue}{\\text{type = \"hotel\"}}\\\\\n",
    "        & \\color{green}{\\text{count = \"182\"}}\\\\\n",
    "        & \\color{red}{\\text{dogs_allowed = dont_care}}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "    </div>\n",
    "        <font size=\"300\">&#10143;</font>\n",
    "    <div style=\"display:inline-block; vertical-align: middle;\" >\n",
    "        <p style=\"display:block; width:400px; word-wrap:break-word;\">\n",
    "        There are <font color=\"green\"><u>182</u></font> <font color=\"blue\"><u>hotels</u></font> if you <font color=\"red\"><u>do not care</u></font> whether dogs are allowed.\n",
    "        </p>\n",
    "    </div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Suboptimal expert policy ### \n",
    "\n",
    "The expert policy depends on naive heuristics.\n",
    "<ul><li>mistakes happen...</li></ul>\n",
    "<br>\n",
    "<center>\n",
    "    <div style=\"display:inline-block;\">\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\color{blue}{\\text{type = \"hotel\"}}\\\\\n",
    "        & \\color{green}{\\text{count = \"182\"}}\\\\\n",
    "        & \\color{red}{\\text{dogs_allowed = false}}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "    </div>\n",
    "        <font size=\"300\">&#10143;</font>\n",
    "    <div style=\"display:inline-block; vertical-align: middle;\" >\n",
    "        <p style=\"display:block; width:400px; word-wrap:break-word;\">\n",
    "        There are <font color=\"green\"><u>182</u></font> <font color=\"blue\"><u>hotels</u></font> if you are traveling without animals.\n",
    "        </p>\n",
    "    </div>\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Dealing with suboptimal ### \n",
    "\n",
    "Imitating the suboptimal expert policy is not enough.\n",
    "<ul><li>Should not rely heavily on expert policy during rollin/rollouts.</li><li class=\"fragment\" data-fragment-index=\"2\">e.g. DAgger uses expert exclusively to cost actions.</li></ul>\n",
    "<br>\n",
    "<br>\n",
    "<span class=\"fragment\" data-fragment-index=\"3\">We need to learn a better classifier than the suboptimal expert.</span><ul><li class=\"fragment\" data-fragment-index=\"4\">Use rollouts that mix classifier and expert policy.</li><li class=\"fragment\" data-fragment-index=\"5\">We will be using LOLS here.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<h3>Locally Optimal Learning to Search</h3>\n",
    "\n",
    "<p style=\"border:3px; border-radius: 25px; background-color:lightgrey; border-style:solid; border-color:black; padding: 0.3em; font-size: 70%\">\n",
    "\\begin{align}\n",
    "& \\textbf{Input:} \\; D_{train} = \\{(\\mathbf{x}^1,\\mathbf{y}^1)...(\\mathbf{x}^M,\\mathbf{y}^M)\\}, \\; expert\\; \\pi^{\\star}, \\; loss \\; function \\; L\\\\\n",
    "& \\text{set} \\; training\\; examples\\; \\cal E = \\emptyset\\\\\n",
    "& \\color{red}{\\text{initialize a classifier } H_{0}}\\\\\n",
    "& \\mathbf{for}\\; i = 0 \\;\\mathbf{to} \\; N\\; \\mathbf{do}\\\\\n",
    "& \\quad \\color{red}{\\text{set} \\; rollin \\; policy \\; \\pi^{in} = H_i}\\\\\n",
    "& \\quad \\color{red}{\\text{set} \\; rollout \\; policy \\; \\pi^{out} = mix(H_i,\\pi^{\\star})}\\\\\n",
    "& \\quad \\mathbf{for} \\; (\\mathbf{x},\\mathbf{y}) \\in D_{train} \\; \\mathbf{do}\\\\\n",
    "& \\quad \\quad \\text{rollin to predict} \\; \\hat \\alpha_1\\dots\\hat \\alpha_T  = \\pi^{in}(\\mathbf{x},\\mathbf{y})\\\\\n",
    "& \\quad \\quad \\mathbf{for} \\; \\hat \\alpha_t \\in \\hat \\alpha_1\\dots\\hat \\alpha_T \\; \\mathbf{do}\\\\\n",
    "& \\quad \\quad \\quad \\text{rollout to obtain costs}\\; c \\; \\text{for all possible actions using}\\; L\\;  \\\\\n",
    "& \\quad \\quad \\quad \\text{extract features}\\; feat=\\phi(\\mathbf{x},S_{t-1}) \\\\\n",
    "& \\quad \\quad \\quad \\cal E = \\cal E \\cup (feat,c)\\\\\n",
    "& \\quad \\color{red}{\\text{learn classifier} \\; H_{i+1} \\; \\text{from}\\; \\cal E}\\\\\n",
    "& \\color{red}{H = \\; \\mathbf{avg}\\{H_{0}\\mathbf{\\dots}H_{N}\\}}\\\\\n",
    "\\end{align}\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Performing the rollin ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS_1.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Performing the rollin ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS_2.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "    <div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Explore actions ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS_explore1.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Explore actions ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS_explore2.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "    <div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Performing the rollout ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS_3.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Performing the rollout ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS_4.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Performing the rollout ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS_5.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Performing the rollout ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/toBeAnimated/NLG_LOLS.png\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{eattype = restaurant}\\\\\n",
    "        & \\text{food = chinese}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<h3>Locally Optimal Learning to Search</h3>\n",
    "\n",
    "<p style=\"border:3px; border-radius: 25px; background-color:lightgrey; border-style:solid; border-color:black; padding: 0.3em; font-size: 80%\">\n",
    "\\begin{align}\n",
    "& \\textbf{Input:} \\; D_{train} = \\{(\\mathbf{x}^1,\\mathbf{y}^1)...(\\mathbf{x}^M,\\mathbf{y}^M)\\}, \\; expert\\; \\pi^{\\star}, \\; loss \\; function \\; L\\\\\n",
    "& \\quad \\quad \\quad \\color{red}{learning\\; rate\\; p}\\\\\n",
    "& \\text{set} \\; training\\; examples\\; \\cal E = \\emptyset\\\\\n",
    "& \\text{initialize a policy } H_{0}\\\\\n",
    "& \\mathbf{for}\\; i = 0 \\;\\mathbf{to} \\; N\\; \\mathbf{do}\\\\\n",
    "& \\quad \\color{red}{\\beta = (1 - p)^i}\\\\\n",
    "& \\quad \\text{set} \\; rollin \\; policy \\; \\pi^{in} = H_i\\\\\n",
    "& \\quad \\text{set} \\; rollout \\; policy \\; \\pi^{out} = mix(H,\\pi^{\\star})\\\\\n",
    "& \\quad \\mathbf{for} \\; (\\mathbf{x},\\mathbf{y}) \\in D_{train} \\; \\mathbf{do}\\\\\n",
    "& \\quad \\quad \\text{rollin to predict} \\; \\hat \\alpha_1\\dots\\hat \\alpha_T  = \\pi^{in}(\\mathbf{x},\\mathbf{y})\\\\\n",
    "& \\quad \\quad \\mathbf{for} \\; \\hat \\alpha_t \\in \\hat \\alpha_1\\dots\\hat \\alpha_T \\; \\mathbf{do}\\\\\n",
    "& \\quad \\quad \\quad \\text{rollout to obtain costs}\\; c \\; \\text{for all possible actions using}\\; L\\;  \\\\\n",
    "& \\quad \\quad \\quad \\text{extract features}\\; feat=\\phi(\\mathbf{x},S_{t-1}) \\\\\n",
    "& \\quad \\quad \\quad \\cal E = \\cal E \\cup (feat,c)\\\\\n",
    "& \\quad \\text{learn classifier} \\; H_{i+1} \\; \\text{from}\\; \\cal E\\\\\n",
    "& H = \\; \\mathbf{avg}\\{H_{0}\\mathbf{\\dots}H_{N}\\}\\\\\n",
    "\\end{align}\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### LOLS modifications ###\n",
    "\n",
    "<b>Exponential decay schedule</b><br>\n",
    "Introduced in SEARN ([Daumé III et al., 2009](http://hunch.net/~jl/projects/reductions/searn/searn.pdf))<center>$\\beta = (1 - p)^i$</center>\n",
    "\n",
    "<ul><li class=\"fragment\" data-fragment-index=\"4\">Gradual move away from the (suboptimal) expert.</li><li class=\"fragment\" data-fragment-index=\"5\">The costs are adjusted to the classifiers' predictions, thus teaching them to predict actions jointly.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "LOLS uses the same policy throughout each rollout.\n",
    "- But potentially a different policy for each examined action at the same time-step.\n",
    "<center>\n",
    "<img src=\"images/perStepLOLS_3.png\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Here, we also use the same policy for all actions at the same time-step.\n",
    "<center>\n",
    "<img src=\"images/perStepLOLS_4.png\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Rollin noise ### \n",
    "\n",
    "Rollin with the classifiers: <ul><li>exposes them to their own actions,</li><li>learn to recover from errors.</li></ul>\n",
    "<br><br>\n",
    "<span class=\"fragment\" data-fragment-index=\"4\">But may cause noise in feature vectors of later time-steps.</span>\n",
    "<ul><li class=\"fragment\" data-fragment-index=\"4\">Especially, for tasks that are context-dependent.</li></ul>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sequence correction ### \n",
    "<center>\n",
    "<img src=\"images/seqCorrection3.png\" width=\"70%\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{dogs_allowed = yes}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sequence correction ### \n",
    "<center>\n",
    "<img src=\"images/seqCorrection4.png\" width=\"70%\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{dogs_allowed = yes}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Sequence correction ### \n",
    "\n",
    "<center>\n",
    "<img src=\"images/seqCorrection5.png\" width=\"70%\">\n",
    "</center>\n",
    "<br>\n",
    "<br>\n",
    "<div style=\"display:inline-block;\">\n",
    "<small>\n",
    "        <p style=\"border:2px; border-radius: 15px; background-color:white; border-style:solid; border-color:black; padding: 0.5em; font-size: 80%; display: inline-block\">\n",
    "        \\begin{align}\n",
    "        & \\text{Predicate: INFORM}\\\\\n",
    "        & \\text{______________________}\\\\\n",
    "        & \\text{name = X-name-1}\\\\\n",
    "        & \\text{dogs_allowed = yes}\n",
    "        \\end{align}\n",
    "        </p>\n",
    "</small>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "After a suboptimal action, we apply <b>sequence correction</b>:\n",
    "- Correct all already explored time-steps using $\\pi^{\\star}$.\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"images/seqCorrection0.png\" width=\"70%\">\n",
    "</center>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "After a suboptimal action, we apply <b>sequence correction</b>:\n",
    "- Correct all already explored time-steps using $\\pi^{\\star}$.\n",
    "- And re-rollin the rest of the sequence using $H$.\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"images/seqCorrection1.png\" width=\"70%\">\n",
    "</center>\n",
    "<br>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "After a suboptimal action, we apply <b>sequence correction</b>:\n",
    "- Correct all already explored time-steps using $\\pi^{\\star}$.\n",
    "- And re-rollin the rest of the sequence using $H$.\n",
    "- i.e. expose classifier to its actions again.\n",
    "<br>\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"images/seqCorrection2.png\" width=\"70%\">\n",
    "</center>\n",
    "\n",
    "If suboptimal actions are encountered further in the new sequence, sequence correction may again be performed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Before <b>sequence correction</b>, allow at most $E$ actions after the suboptimal one.\n",
    "- i.e. allow the classifiers to learn how to recover from mistakes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### NLG dev results<h5>[Lampouras and Vlachos 2016](https://aclweb.org/anthology/C/C16/C16-1105.pdf)</h5>\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"images/seqCorrectResults.png\" style=\"width:30%; float: left; transform: translateY(60%);\">\n",
    "<br>\n",
    "<img src=\"images/NLG_policyResults.jpg\" style=\"width:60%; float: right;\" class=\"fragment\" data-fragment-index=\"4\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Comparison against state of the art<h5>[Lampouras and Vlachos 2016](https://aclweb.org/anthology/C/C16/C16-1105.pdf)</h5>\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"images/NLGResults.png\">\n",
    "</center>"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "livereveal": {
   "height": 768,
   "start_slideshow_at": "selected",
   "theme": "solarized",
   "transition": "fade",
   "width": 1024
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}