Tarea_4_HMM__MEMM__CRF__CNN__RNN_(1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tarea 4 - HMM, MEMM, CRF, CNN, RNN.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4lL5hGw07yP"
      },
      "source": [
        "# **Tarea 4 - CC6205 Natural Language Processing 📚**\n",
        "\n",
        "**Integrantes: Vicente Ardiles y Rodrigo Oportot**\n",
        "\n",
        "**Fecha límite de entrega 📆:** Martes 22 de junio.\n",
        "\n",
        "**Tiempo estimado de dedicación: 8 hrs**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P6jB5fLGMCaI"
      },
      "source": [
        "Bienvenid@s a la cuarta tarea del curso de Natural Language Processing (NLP). \n",
        "En esta tarea estaremos tratando el problema de **tagging** (generación de secuencias de etiquetas del mismo largo que la secuencia de input), el uso de **Convolutional Neural Networks** y **Recurrent Neural Networks**, e implementaremos una red usando PyTorch. \n",
        "\n",
        "Usen $\\LaTeX$ para las fórmulas matemáticas. En la parte de programación pueden usar lo que quieran, pero la [Axiliar 3](https://www.youtube.com/watch?v=r1pt9v_vuy8&list=PLwITEwTGVF6jXRu6l55W4OedqDiLgU9OM&index=2&t=5s) les puede ser de *gran ayuda*.\n",
        "\n",
        "**Instrucciones:**\n",
        "- La tarea se realiza en grupos de **máximo** 2 personas. Puede ser invidivual pero no es recomendable.\n",
        "- La entrega es a través de u-cursos a más tardar el día estipulado arriba. No se aceptan atrasos.\n",
        "- El formato de entrega es este mismo Jupyter Notebook.\n",
        "- Al momento de la revisión tu código será ejecutado. Por favor verifica que tu entrega no tenga errores de compilación.\n",
        "- En el horario de auxiliar pueden realizar consultas acerca de la tarea a través del canal de Discord del curso.\n",
        "\n",
        "Si aún no han visto las clases, se recomienda visitar los links de las referencias.\n",
        "\n",
        "**Referencias:**\n",
        "\n",
        "- [Tagging, and Hidden Markov Models ](http://www.cs.columbia.edu/~mcollins/cs4705-spring2019/slides/tagging.pdf) (slides by Michael Collins), [notes](http://www.cs.columbia.edu/~mcollins/hmms-spring2013.pdf), [video 1](https://youtu.be/-ngfOZz8yK0), [video 2](https://youtu.be/Tjgb-yQOg54), [video 3](https://youtu.be/aaa5Qoi8Vco), [video 4](https://youtu.be/4pKWIDkF_6Y)       \n",
        "- [MEMMs and CRFs](slides/NLP-CRF.pdf) | ([tex source file](slides/NLP-CRF.tex)), [notes 1](http://www.cs.columbia.edu/~mcollins/crf.pdf), [notes 2](http://www.cs.columbia.edu/~mcollins/fb.pdf), [video 1](https://youtu.be/qlI-4lSUDkg), [video 2](https://youtu.be/PLoLKQwkONw), [video 3](https://youtu.be/ZpUwDy6o28Y)\n",
        "- [Convolutional Neural Networks](slides/NLP-CNN.pdf) | ([tex source file](slides/NLP-CNN.tex)), [video](https://youtu.be/lLZW5Fn40r8)\n",
        "- [Recurrent Neural Networks](slides/NLP-RNN.pdf) | ([tex source file](slides/NLP-RNN.tex)), [video 1](https://youtu.be/BmhjUkzz3nk), [video 2](https://youtu.be/z43YFR1iIvk), [video 3](https://youtu.be/7L5JxQdwNJk)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bWXD3D7RYKJ-"
      },
      "source": [
        "# Hidden Markov Models (HMM), Maximum Entropy Markov Models (MEMM) and Conditional Random Field(CRF)\n",
        "\n",
        "### Pregunta 1 (1 pt)\n",
        "Para un problema de POS tagging se define el conjunto de etiquetas $S = \\{ \\text{DET}, \\text{NOUN}, \\text{VERB}, \\text{ADP} \\}$ y se tiene un Hidden Markov Model con los siguientes parámetros estimados a partir de un corpus de entrenamiento:\n",
        "\n",
        "\\begin{equation}\n",
        "\\begin{split}\n",
        "q(\\text{NOUN}| \\text{ VERB}, \\text{DET}) &= 0.3 \\\\\n",
        "q(\\text{NOUN}|\\ w, \\text{DET}) &= 0 \\qquad \\forall w \\in S, w \\neq \\text{VERB} \\\\\n",
        "q(\\text{DET}| \\text{ VERB}, \\text{NOUN}) &= 0.4 \\\\\n",
        "q(\\text{DET}|\\ w, \\text{NOUN}) &= 0 \\qquad \\forall w \\in S, w \\neq \\text{VERB} \\\\\n",
        "e(the|\\text{ DET}) &= 0.5 \\\\\n",
        "e(pasta|\\text{ NOUN}) &= 0.6\n",
        "\\end{split}\n",
        "\\end{equation}\n",
        "\n",
        "Luego para la oración: `the man is pouring sauce on the pasta`, se tiene una tabla de programación dinámica con los siguientes valores:\n",
        "\\begin{equation}\n",
        "\\begin{split}\n",
        "\\pi(7,\\text{DET},\\text{DET})&=0.1\\\\\n",
        "\\pi(7,\\text{NOUN},\\text{DET})&=0.2\\\\\n",
        "\\pi(7,\\text{VERB},\\text{DET})&=0.01\\\\\n",
        "\\pi(7,\\text{ADP},\\text{DET})&=0.5\n",
        "\\end{split}\n",
        "\\end{equation}\n",
        "\n",
        "Con esta información, calcule el valor de $\\pi(8,\\text{DET},\\text{NOUN})$. Puede dejar el resultado expresado como una fracción.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5UvNmJMvi83q"
      },
      "source": [
        "**Respuesta**\n",
        "\n",
        "Por simplicidad llamaremos a los elementos de $S$ en base a sus iniciales. $(D, N, V, A)$\n",
        "\n",
        "Por definicion tenemos que:\n",
        "\n",
        "\\begin{equation}\n",
        "\\pi(8,\\text{DET},\\text{NOUN}) = \\underset{w \\epsilon S_{6}}{max}\\, (\\pi(7,w,D) \\times q(N|w,D) \\times e(pasta|N))\n",
        "\\end{equation}\n",
        "\n",
        "Donde $k = 8, u = DET, v = NOUN, x_{8} = pasta.$\n",
        "\n",
        "Del enuncido tenemos que :\n",
        "\n",
        "\\begin{equation}\n",
        "\\begin{split}\n",
        "&e(pasta|\\text{NOUN}) = 0.6 \\\\\n",
        "&q(\\text{NOUN}| \\text{ VERB}, \\text{DET}) &= 0.3\n",
        "\\end{split}\n",
        "\\end{equation}\n",
        "\n",
        "Las otras etiquetas $w \\neq VERB$ en $q$ no maximizan el producto debido a que $q(\\text{NOUN}|\\ w, \\text{DET})$ es 0, entonces no queda otra opcion que $w = VERB$ y por lo tanto buscamos $\\pi(7,\\text{VERB},\\text{DET})$\n",
        "\n",
        "Reemplazando nos queda el calculo buscado:\n",
        "\\begin{equation}\n",
        "\\begin{split}\n",
        "\\pi(8,\\text{DET},\\text{NOUN}) &= \\pi(7,\\text{VERB},\\text{DET}) \\times q(\\text{NOUN}| \\text{ VERB}, \\text{DET}) \\times e(pasta|\\text{NOUN}) \\\\\n",
        "&= 0.01 \\times 0.3 \\times 0.6 \\\\\n",
        "\\pi(8,\\text{DET},\\text{NOUN}) &= 0.0018\n",
        "\\end{split}\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oiwJb_vmkKLZ"
      },
      "source": [
        "### Pregunta 2 (0.5 pts)\n",
        "Comente  sobre las similitudes o diferencias entre los HMMs, MEMMs y CRFs. Para esto, responda las siguientes preguntas.\n",
        "\n",
        "#### 2.1. ¿Para qué tipo de tarea sirven? Dé dos ejemplo de este tipo de tarea y descríbalos brevemente. (0.1 pts)\n",
        "\n",
        "**Respuesta:** \n",
        "\n",
        "Estos modelos sirven para asignarles etiquetas (discretas) a las palabras de una oracion,\n",
        "problema que es conocido como Sequence labeling o tagging.\n",
        "\n",
        "Dos ejemplos de este tipo de problema son:\n",
        "\n",
        "- Part-of-speech tagging (POS) donde se etiqueta gramaticalmente cada palabra de una oracion con su clase sintaxtica.\n",
        "- Named Entity Recognition (NER) que busca reconocer entidades en una oracion como lo son personas, ciudades, intituciones, marcas, etc.\n",
        "\n",
        "#### 2.2. ¿Qué modelos usan features? ¿Qué ventajas conlleva esto? (0.1 pts)\n",
        "\n",
        "**Respuesta:** \n",
        "\n",
        "MEMMs y CRFs son modelos que usan features.\n",
        "La gran ventaja de las features es que uno puede codificar en el modelo la informacion sobre relaciones/eventos\n",
        "entre las etiquetas y las palabras. En otras palabras codificar eventos que ocurren dentro del input.\n",
        "\n",
        "#### 2.3. ¿Cómo maneja cada uno de los modelos las palabras con baja frecuencia en el set de train? (0.1 pts)\n",
        "\n",
        "**Respuesta:** \n",
        "\n",
        "- HMMs: Las palabras con baja frecuencia se les asigna/mapea una categoria que mejor represente la palabra en base a\n",
        "un conjunto finito de etiquetas previmente definido.\n",
        "\n",
        "- MEMMs: Como el entrenamiento consiste en ajustar los valores de $\\vec W$, las palabras de bajas frecuencias terminarian\n",
        "siendo eventos de muy poca frecuencia, por lo que se le asignarian pesos negativos o cercanos a 0. \n",
        "\n",
        "- CRFs: De forma similar que en MEMMs las palabras con baja frecuencia seguirian siendo eventos de baja frecuencia y por\n",
        "lo tanto a pesar de que el contexto de etiquetas sea mas grande, el peso asigndo sera bajo.\n",
        "\n",
        "#### 2.4. ¿Qué le permite a los CRF realizar decisiones globales? ¿Qué diferencia con respecto a los MEMMs permite lograr esto? ¿Por qué los HMMs tampoco son capaces de tomar decisiones globales? (0.1 pts)\n",
        "\n",
        "**Respuesta:** \n",
        "\n",
        "Que el $\\vec \\Phi$ es un vector de features globales, es decir, toma en cuenta la secuencia\n",
        "completa de etiquetas $S$, en cambio MEMMs esta restringido a ciertas posiciones de etiquetas por lo\n",
        "que los CRF tienen informacion de todo el contexto.\n",
        "HMMs tampoco tiene este comportamiento global debido a que es incluso mas restrictiva que las MEMMs\n",
        "debido a que las palabras solo estan condicionadas a una sola etiqueta $(P(x_{i} | S_{i}))$.\n",
        "\n",
        "#### 2.5 Dado una secuencia de $x_1, ..., x_m$ ¿Cuántas posibles secuencias de etiquetas se pueden generar para un conjunto de etiquetas $S$ con $|S|=k$ ? ¿Analizarlas todas sería computacionalmente tratable? (0.1 pts)\n",
        "\n",
        "**Respuesta:** \n",
        "\n",
        "Se pueden generar $k^{m}$ secuencias de etiquetas. Usando algoritmos de progamacion dinamica es posible que\n",
        "el analisis sobre esta secuencia se haga de forma eficiente y por lo tanto computacionalmente tratable."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClRAHR95Y8aB"
      },
      "source": [
        "# Convolutional Neural Networks\n",
        "### Pregunta 3 (1 pt)\n",
        "\n",
        "Considere la frase $w_{1..7}=$ `El agua moja y el fuego quema` $=[El, agua, moja, y, el, fuego, quema]$.\n",
        "\n",
        "La siguiente matriz de embeddings, donde la i-ésima fila corresponde al vector de embedding de la i-ésima palabra, ordenadas según aparecen en la frase. (vectores de largo 2).\n",
        "\\begin{equation}\n",
        "E = \\begin{pmatrix}\n",
        "2 & 2\\\\\n",
        "0 & -2\\\\\n",
        "0 & 1\\\\\n",
        "-2 & 1\\\\\n",
        "1 & 0\\\\\n",
        "-1 & 1\\\\\n",
        "1 & 1\n",
        "\\end{pmatrix}\n",
        "\\end{equation}\n",
        "\n",
        "Los siguientes 3 filtros\n",
        "\\begin{equation}\n",
        "U = \\begin{pmatrix}\n",
        "-1 & 1 & 0\\\\\n",
        "1 & 1 & 0\\\\\n",
        "0 & 0 & -1\\\\\n",
        "1 & -1 & -1\\\\\n",
        "-1 & -1 & 1\\\\\n",
        "1 & 0 & -1\n",
        "\\end{pmatrix}\n",
        "\\end{equation}\n",
        "\n",
        "Y la función de activación\n",
        "\\begin{equation}\n",
        "tanh = \\frac{e^{2x} - 1}{e^{2x} + 1}\n",
        "\\end{equation}\n",
        "\n",
        "Usando estos paramátros escriba los pasos para calcular la representación (vector) resultante de aplicar la operación de convolución (sin padding) + max pooling. ¿De qué tamaño sería la ventana que debemos usar?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SlQ30Arkq0u4"
      },
      "source": [
        "**Respuesta**\n",
        "\n",
        "Primero, consideremos que no habrá padding por lo que será una convolución del tipo narrow. Por esta razón, el vector $w_{1...7}$ se mantiene sin cambios. Usaremos ventanas de tamaño 3, por ejemplo, \"El agua moja\", luego \"agua moja y\", etc. \n",
        "\n",
        "- Paso 1: se concatenan las filas de la matriz de embeddings E correspondientes a cada palabra de la ventana:\n",
        "\n",
        "\\begin{equation}\n",
        "El \\ agua \\ moja  \\rightarrow (2, \\ 2, \\ 0, -2, \\ 0, \\ 1) \\\\\n",
        "agua \\ moja \\ y \\rightarrow (0, -2, \\ 0, \\ 1, -2 , \\ 1) \\\\\n",
        "moja \\ y \\ el \\rightarrow (0, \\ 1, -2, \\ 1, \\ 1, \\ 0) \\\\\n",
        "y \\ el \\ fuego \\rightarrow (-2, \\ 1, \\ 1, \\ 0, -1, \\ 1) \\\\\n",
        "el \\ fuego \\ quema \\rightarrow (1, \\ 0, -1, \\ 1, \\ 1, \\ 1) \\\\\n",
        "\\end{equation}\n",
        "\n",
        "Resultando en vectores de tamaño 6\n",
        "\n",
        "- Paso 2: Los 5 vectores resultantes del paso 1 son multiplicados, cada uno, con las filas del filtro U mediante producto punto, para luego operadar aesos resultados través de la función de activación tahn(x) y obtener valores escalares:\n",
        "\n",
        "\\begin{equation}\n",
        "\\vec{p_{1}} = tanh((2, \\ 2, \\ 0, -2, \\ 0, \\ 1) \\bullet U) =  tanh((-1, \\ 6, \\ 1)) \\approx ( -0.762, \\ 0.999, \\ 0.762) \\\\\n",
        "\\vec{p_{2}} = tanh((0, -2, \\ 0, \\ 1, -2 , \\ 1 ) \\bullet U) =  tanh(( 2, -1, -4)) \\approx ( 0.964, -0.762, -0.999) \\\\\n",
        "\\vec{p_{3}} = tanh((0, \\ 1, -2, \\ 1, \\ 1, \\ 0) \\bullet U) = tanh(( 1, -1, \\ 2)) \\approx ( 0.762, -0.762, \\ 0.964) \\\\\n",
        "\\vec{p_{4}} = tanh((-2, \\ 1, \\ 1, \\ 0, -1, \\ 1) \\bullet U) = tanh(( 5, \\ 0, -3)) \\approx ( 0.999, \\ 0, -0.995) \\\\\n",
        "\\vec{p_{5}} = tanh((1, \\ 0, -1, \\ 1, \\ 1, \\ 1) \\bullet U) = tanh((0, -1, \\ 0)) \\approx ( 0, -0.762, 0) \\\\\n",
        "\\end{equation}\n",
        "\n",
        "Paso 3: Por último, queda aplicar max pooling a la serie de vectores $\\vec{p_{i}}$ para obtener el resultado final:\n",
        "\n",
        "\\begin{equation}\n",
        "max(\\vec{p_{1}}(1), \\vec{p_{2}}(1), \\vec{p_{3}}(1), \\vec{p_{4}}(1), \\vec{p_{5}}(1)) = max( -0.762, \\ 0.964, \\ 0.762, \\ 0.999, \\ 0) = 0.999 \\\\\n",
        "max(\\vec{p_{1}}(2), \\vec{p_{2}}(2), \\vec{p_{3}}(2), \\vec{p_{4}}(2), \\vec{p_{5}}(2)) = max( 0.999, -0.762, -0.762, \\ 0 , -0.762) = 0.999 \\\\\n",
        "max(\\vec{p_{1}}(3), \\vec{p_{2}}(3), \\vec{p_{3}}(3), \\vec{p_{4}}(3), \\vec{p_{5}}(3)) = max( 0.762, -0.999, \\ 0.964, -0.995, \\ 0) = 0.964 \\\\\n",
        "\\end{equation}\n",
        "\n",
        "Por lo tanto, el vector final de representación es $\\vec{c} = (0.999, \\ 0.999, \\ 0.964)$\n",
        "\n",
        "Posteriormente puede aplicarse una función softmax y después cross entropy como loss function.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tj1V_sAzZCHY"
      },
      "source": [
        "# Recurrent Neural Networks\n",
        "### Pregunta 4 (1 pt)\n",
        "Usando los embeddings de dos dimensiones de la pregunta anteror, la oración `el fuego quema` la podemos representar por una secuencia de vectores $(\\vec{x}_1,\\vec{x}_2,\\vec{x}_3)$, con $\\vec{x}_i \\in \\mathbb{R}^{d_x}$ y $d_x=2$.\n",
        "\n",
        "Tenemos una red recurrente *Elman* definidad como: \n",
        "\\begin{equation}\n",
        "\\begin{split}\n",
        "\\vec{s}_i &= R_{SRNN}\\left (\\vec{x}_i, \\vec{s}_{i-1}\\right ) = g \\left (\\vec{s}_{i-1}W^s + \\vec{x}_i W^x + \\vec{b}\\right ) \\\\\n",
        "\\vec{y}_i &= O_{SRNN}\\left(\\vec{s}_i\\right) = \\vec{s}_i \\\\\n",
        "\\end{split}\n",
        "\\end{equation}\n",
        "donde\n",
        "\\begin{equation}\n",
        "\\vec{s}_i, \\vec{y}_i \\in \\mathbb{R}^{d_s}, \\quad W^x \\in \\mathbb{R}^{d_x \\times d_s}, \\quad W^s \\in \\mathbb{R}^{d_s \\times d_s}, \\quad \\vec{b} \\in \\mathbb{R}^{d_s},\n",
        "\\end{equation}\n",
        "y los vectores de estado $s_i$ son de tres dimensiones, $ds= 3$.\n",
        "\n",
        "Sea\n",
        "\\begin{equation}\n",
        "\\begin{split}\n",
        "\\vec{s}_0 &= [0,0,0]\\\\\n",
        "W^x &= \\begin{pmatrix}\n",
        "0 &  0 & 1\\\\\n",
        "1 & -1 & 0\n",
        "\\end{pmatrix} \\\\\n",
        "W^s &= \\begin{pmatrix}\n",
        "1 & 0 &  1\\\\\n",
        "0 & 1 & -1\\\\\n",
        "1 & 1 &  1\n",
        "\\end{pmatrix} \\\\\n",
        "\\vec{b} &= [0, 0, 0] \\\\\n",
        "g(x) &= ReLu(x) = max(0, x)\n",
        "\\end{split}\n",
        "\\end{equation}\n",
        "\n",
        "<br>\n",
        "\n",
        "Calcule manualmente los valores de los vectores $\\vec{s}_1, \\vec{s}_2,\\vec{s}_3$ y de $\\vec{y}_1, \\vec{y}_2,\\vec{y}_3$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7M7sqIQV-Q3a"
      },
      "source": [
        "**Respuesta**\n",
        "\n",
        "\\begin{equation}\n",
        "\\vec{s_{1}} = R_{SRNN}(\\vec{x_{1}}, \\vec{s_{0}}) = ReLu(\\vec{s_{0}}W^{s}+\\vec{x_{1}}W^{x}+\\vec{b}) = ReLu((0,0,0)W^{s} + (1,0)W^{x} + (0,0,0)) = ReLu((0,0,0) + (0,0,1) + (0,0,0)) = ReLu((0,0,1)) = max(0,(0,0,1)) = (0,0,1) \\\\\n",
        "\\vec{s_{2}} = R_{SRNN}(\\vec{x_{2}}, \\vec{s_{1}}) = ReLu(\\vec{s_{1}}W^{s}+\\vec{x_{2}}W^{x}+\\vec{b}) = ReLu((0,0,1)W^{s} + (-1,1)W^{x} + (0,0,0)) = ReLu((1,1,1) + (1,-1,-1) + (0,0,0)) = ReLu((2,0,0)) = max(0,(2,0,0)) = (2,0,0) \\\\\n",
        "\\vec{s_{3}} = R_{SRNN}(\\vec{x_{3}}, \\vec{s_{2}}) = ReLu(\\vec{s_{2}}W^{s}+\\vec{x_{3}}W^{x}+\\vec{b}) = ReLu((2,0,0)W^{s} + (1,1)W^{x} + (0,0,0)) = ReLu((2,0,2) + (1,-1,1) + (0,0,0) = ReLu((3,-1,3)) = max(0,(3,-1,3)) = (3,0,3) \\\\\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "\\vec{y_{1}} = O_{SRNN}(\\vec{s_{1}}) = \\vec{s_{1}} = (0,0,1) \\\\\n",
        "\\vec{y_{2}} = O_{SRNN}(\\vec{s_{2}}) = \\vec{s_{2}} = (2,0,0) \\\\\n",
        "\\vec{y_{3}} = O_{SRNN}(\\vec{s_{3}}) = \\vec{s_{3}} = (3,0,3) \\\\\n",
        "\\end{equation}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W4rAT6ELxRZW"
      },
      "source": [
        "### Pregunta 5 (0.5 pts)\n",
        "¿De qué forma las RNN y las CNN logran aprender representaciones específicas\n",
        "para la tarea objetivo? Compare la forma en que las RNN y las CNN aprenden con los modelos que usan *features* diseñadas manualmente."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6AXbQSgA_t8"
      },
      "source": [
        "**Respuesta**\n",
        "\n",
        "RNN y CNN consideran la extracción de features y la clasificación como un solo trabajo, uniendo ambos procesos en una gran tarea de entrenamiento para aprender las representaciones adecuadas de la tarea objetivo. Este acercamiento que toman RNN y CNN, siendo redes neuronales como tal, es característico de Deep Learning, diferenciándose de otros modelos de clasificación donde la extracción de features es diseñada previa y manualmente al entrenamiento, entregándolas luego como entradas a dichos modelos. Esto es propio, por ejemplo, de Machine Learning supervisado. \n",
        "\n",
        "CNN y RNN logran aprender las representaciones específicas para la tarea objetivo de manera no supervisada en el mismo proceso de entrenamiento, a través de transformaciones no lineales, donde el resultado de cada capa de la red es entregado a la siguiente como entrada. En la capa final, la iteración acaba, entregando el resultado para ser procesado con otros métodos adicionales, como pooling, aplicaciones de funciones softmax y de loss, e incluso conectado dicho resultado a otra arquitectura que siga trabajando la información. Por ejemplo, las RNN por sí solas no son de mucha utilidad, por lo que sus outputs son parte del entrenamiento de redes más grandes que las contengan.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qxQIuO8axTUa"
      },
      "source": [
        "# Redes neuronales con PyTorch\n",
        "### Pregunta 6 (2 pts)\n",
        "En esta parte van a tener que implementar una red neuronal Feed Forward. Además, deberán entrenar el modelo usando uno de los datasets de TorchText. En la sección de la respuesta hay un esqueleto de lo que deben hacer, deberán completar los metodos del modelo e implementar la parte asociada al entrenamiento. Como les mencionamos en la [Auxiliar 3](https://www.youtube.com/watch?v=r1pt9v_vuy8&list=PLwITEwTGVF6jXRu6l55W4OedqDiLgU9OM&index=2&t=5s), el proceso de entrenamiento es bastante estándar, así que se pueden guiar en gran medida por los ejemplos que ahí mostramos y los que vamos a ver en las próximas auxiliares.\n",
        "\n",
        "#### 6.1 Capa Convolucional (Opcional)\n",
        "Agregue a la arquitectura una capa convolucional. Para esto puede registrar el parametro $U$ en la red y realizar el computo de la convolución en el metodo forward de la red, o puede usar la clase [`torch.nn.Conv1d`](https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#conv1d) de `torch`.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LVKEaQXZ3eGl",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 823
        },
        "outputId": "5874c720-68fa-404c-c3de-6b81a70fd7b6"
      },
      "source": [
        "# %%capture\n",
        "# Nos aseguramos que instalemos la version anterior de torchtext 0.9.0\n",
        "!pip install torch==1.8.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html\n",
        "!pip install torchtext==0.9.0"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Looking in links: https://download.pytorch.org/whl/torch_stable.html\n",
            "Collecting torch==1.8.0+cu111\n",
            "\u001b[?25l  Downloading https://download.pytorch.org/whl/cu111/torch-1.8.0%2Bcu111-cp37-cp37m-linux_x86_64.whl (1982.2MB)\n",
            "\u001b[K     |█████████████▌                  | 834.1MB 62.9MB/s eta 0:00:19tcmalloc: large alloc 1147494400 bytes == 0x560d62e4a000 @  0x7f034a092615 0x560d28a58cdc 0x560d28b3852a 0x560d28a5bafd 0x560d28b4cfed 0x560d28acf988 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acf7f0 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28bd03e1 0x560d28b306a9 0x560d28a9bcc4 0x560d28a5c559 0x560d28ad04f8 0x560d28a5d30a 0x560d28acb3b5 0x560d28aca7ad 0x560d28a5d3ea 0x560d28acb3b5 0x560d28a5d30a 0x560d28acb3b5\n",
            "\u001b[K     |█████████████████               | 1055.7MB 1.3MB/s eta 0:11:57tcmalloc: large alloc 1434370048 bytes == 0x560da74a0000 @  0x7f034a092615 0x560d28a58cdc 0x560d28b3852a 0x560d28a5bafd 0x560d28b4cfed 0x560d28acf988 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acf7f0 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28bd03e1 0x560d28b306a9 0x560d28a9bcc4 0x560d28a5c559 0x560d28ad04f8 0x560d28a5d30a 0x560d28acb3b5 0x560d28aca7ad 0x560d28a5d3ea 0x560d28acb3b5 0x560d28a5d30a 0x560d28acb3b5\n",
            "\u001b[K     |█████████████████████▋          | 1336.2MB 1.3MB/s eta 0:08:23tcmalloc: large alloc 1792966656 bytes == 0x560d2c2d2000 @  0x7f034a092615 0x560d28a58cdc 0x560d28b3852a 0x560d28a5bafd 0x560d28b4cfed 0x560d28acf988 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acf7f0 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28bd03e1 0x560d28b306a9 0x560d28a9bcc4 0x560d28a5c559 0x560d28ad04f8 0x560d28a5d30a 0x560d28acb3b5 0x560d28aca7ad 0x560d28a5d3ea 0x560d28acb3b5 0x560d28a5d30a 0x560d28acb3b5\n",
            "\u001b[K     |███████████████████████████▎    | 1691.1MB 1.2MB/s eta 0:03:56tcmalloc: large alloc 2241208320 bytes == 0x560d970ba000 @  0x7f034a092615 0x560d28a58cdc 0x560d28b3852a 0x560d28a5bafd 0x560d28b4cfed 0x560d28acf988 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acf7f0 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28acb853 0x560d28b4de36 0x560d28bd03e1 0x560d28b306a9 0x560d28a9bcc4 0x560d28a5c559 0x560d28ad04f8 0x560d28a5d30a 0x560d28acb3b5 0x560d28aca7ad 0x560d28a5d3ea 0x560d28acb3b5 0x560d28a5d30a 0x560d28acb3b5\n",
            "\u001b[K     |████████████████████████████████| 1982.2MB 35.0MB/s eta 0:00:01tcmalloc: large alloc 1982251008 bytes == 0x560e1ca1c000 @  0x7f034a0911e7 0x560d28a8ef37 0x560d28a58cdc 0x560d28b3852a 0x560d28a5bafd 0x560d28b4cfed 0x560d28acf988 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28a5d30a 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28aca4ae\n",
            "tcmalloc: large alloc 2477817856 bytes == 0x560f07146000 @  0x7f034a092615 0x560d28a58cdc 0x560d28b3852a 0x560d28a5bafd 0x560d28b4cfed 0x560d28acf988 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acb60e 0x560d28a5d30a 0x560d28acb60e 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28aca4ae 0x560d28a5d3ea 0x560d28acc32a 0x560d28aca4ae 0x560d28a5da81\n",
            "\u001b[K     |████████████████████████████████| 1982.2MB 3.8kB/s \n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torch==1.8.0+cu111) (1.19.5)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.8.0+cu111) (3.7.4.3)\n",
            "\u001b[31mERROR: torchvision 0.10.0+cu102 has requirement torch==1.9.0, but you'll have torch 1.8.0+cu111 which is incompatible.\u001b[0m\n",
            "\u001b[31mERROR: torchtext 0.10.0 has requirement torch==1.9.0, but you'll have torch 1.8.0+cu111 which is incompatible.\u001b[0m\n",
            "Installing collected packages: torch\n",
            "  Found existing installation: torch 1.9.0+cu102\n",
            "    Uninstalling torch-1.9.0+cu102:\n",
            "      Successfully uninstalled torch-1.9.0+cu102\n",
            "Successfully installed torch-1.8.0+cu111\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "torch"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Collecting torchtext==0.9.0\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/36/50/84184d6230686e230c464f0dd4ff32eada2756b4a0b9cefec68b88d1d580/torchtext-0.9.0-cp37-cp37m-manylinux1_x86_64.whl (7.1MB)\n",
            "\u001b[K     |████████████████████████████████| 7.1MB 6.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from torchtext==0.9.0) (4.41.1)\n",
            "Requirement already satisfied: torch==1.8.0 in /usr/local/lib/python3.7/dist-packages (from torchtext==0.9.0) (1.8.0+cu111)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from torchtext==0.9.0) (1.19.5)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from torchtext==0.9.0) (2.23.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch==1.8.0->torchtext==0.9.0) (3.7.4.3)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.9.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.9.0) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.9.0) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->torchtext==0.9.0) (2021.5.30)\n",
            "Installing collected packages: torchtext\n",
            "  Found existing installation: torchtext 0.10.0\n",
            "    Uninstalling torchtext-0.10.0:\n",
            "      Successfully uninstalled torchtext-0.10.0\n",
            "Successfully installed torchtext-0.9.0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "torchtext"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dJ-wrzFO5mCC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c521ccde-a13e-4401-a921-ce14aa6779b3"
      },
      "source": [
        "# Trabajaremos con el dataset AG_NEWS de torchtext\n",
        "# https://pytorch.org/text/stable/datasets.html#ag-news\n",
        "import os\n",
        "import torch\n",
        "from random import choice\n",
        "from torchtext.datasets import AG_NEWS\n",
        "from torchtext.data.utils import get_tokenizer\n",
        "from torchtext.vocab import build_vocab_from_iterator\n",
        "from random import sample\n",
        "\n",
        "os.makedirs(\"data\", exist_ok=True)\n",
        "os.environ['CUDA_LAUNCH_BLOCKING'] = \"1\"\n",
        "train_data, test_data = AG_NEWS(root=\"data\")\n",
        "\n",
        "train_split = list(train_data)\n",
        "test_split = list(test_data)\n",
        "\n",
        "print(\"Algunos ejemplos del dataset:\")\n",
        "for example in sample(list(train_split), 3):\n",
        "    print(\"\\nEjemplo aleatorio:\\n\", example)\n",
        "\n",
        "# Informacion relevante del dataset\n",
        "tokenizer = get_tokenizer(\"basic_english\")\n",
        "vocab = build_vocab_from_iterator(tokenizer(x[1]) for x in train_split)\n",
        "num_classes = 4\n",
        "\n",
        "labels = list({doc[0] for doc in train_split})\n",
        "label_map = {label: index for index, label in enumerate(labels)}\n",
        "\n",
        "print(\"\\nTamanno del vocabulario:\", len(vocab))\n",
        "print(\"Algunas palabras del vocabulario:\", sample(vocab.get_itos(), 5))\n",
        "print(\"El label map: \", label_map)\n",
        "print(\"Los labels: \", labels)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "train.csv: 29.5MB [00:00, 69.7MB/s]\n",
            "test.csv: 1.86MB [00:00, 29.5MB/s]                  \n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Algunos ejemplos del dataset:\n",
            "\n",
            "Ejemplo aleatorio:\n",
            " (3, 'Treasuries Up, Indirect Bidders Buy Big  NEW YORK (Reuters) - U.S. Treasury prices built on early  gains on Wednesday after an auction of five-year debt drew  surprisingly strong demand from indirect bidders, which include  foreign central banks.')\n",
            "\n",
            "Ejemplo aleatorio:\n",
            " (3, 'Walkout goes on as GM job-cut talks begin RUESSELSHEIM - Top Opel management and labour officials began talks Monday on the plans by General Motors to axe 10,000 jobs at the carmaker, while workers at one of the company #39;s main plants in Bochum continued a walkout.')\n",
            "\n",
            "Ejemplo aleatorio:\n",
            " (2, 'Crawford overcomes Kenteris clamour Thursday was the night every Greek had marked down in their diary weeks and months ago. It was the night Kostas Kenteris was supposed to successfully defend his 200m crown in front of his adoring public.')\n",
            "\n",
            "Tamanno del vocabulario: 95810\n",
            "Algunas palabras del vocabulario: ['financial-services', 'methodically', 'nodong', 'handbells', 'afanasenkov']\n",
            "El label map:  {1: 0, 2: 1, 3: 2, 4: 3}\n",
            "Los labels:  [1, 2, 3, 4]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Vqfr5BTj2c8"
      },
      "source": [
        "## Modelo con capa de Embedding"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IXngUm9HxKvA"
      },
      "source": [
        "# De aca para abajo viene su respuesta, completen las funciones en la red\n",
        "# y luego entrenen el modelo y evaluenlo usando los dataset que acaban de\n",
        "# cargar\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "\n",
        "class CNNClassifier(nn.Module):\n",
        "    def __init__(self, vocab_size, embed_dim, num_class, hidden_size, \n",
        "                 use_cnn=False):\n",
        "        # Aca deben registrar sus parametros. A lo menos necesitan\n",
        "        # una capa de embedding y un MLP basico (una capa lineal + softmax)\n",
        "        super().__init__()\n",
        "\n",
        "        # capa de embedding\n",
        "        self.embedding = nn.Embedding(vocab_size, \n",
        "                                      embed_dim\n",
        "                                      )\n",
        "        # capa de la MLP\n",
        "        self.fc = nn.Linear(embed_dim, num_class)\n",
        "\n",
        "    def forward(self, batch): # Reemplacen el *args por sus argumentos\n",
        "        # Acá debe programar la pasada hacia adelante\n",
        "\n",
        "        # La representacion de un documento sera el promedio de los\n",
        "        # embeddings de sus palabras.\n",
        "        # (B, N, 1) -> (B, N, E)\n",
        "        h = self.embedding(batch)\n",
        "        # (B, N, E) -> (B, E)\n",
        "        h = h.mean(dim=1)\n",
        "        # h = self.embedding(batch)\n",
        "        \n",
        "        # computar las capas de la red MLP\n",
        "        h = self.fc(h)\n",
        "        # h = F.relu(self.fc1(h))\n",
        "        # h = self.fc2(h)\n",
        "        \n",
        "        return h"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QGQUVbJOkMgA"
      },
      "source": [
        "## Entrenamiento"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPHyqgId96ra"
      },
      "source": [
        "# El resto de su respuesta. Acá deben programar el entrenamiento de la red\n",
        "\n",
        "import sys\n",
        "from torch.optim import SGD\n",
        "from torch.utils.data import DataLoader\n",
        "from itertools import zip_longest\n",
        "import time\n",
        "\n",
        "unk_token = '<unk>'\n",
        "default_index = 0\n",
        "if unk_token not in vocab: vocab.insert_token(unk_token, 0)\n",
        "vocab.set_default_index(default_index)\n",
        "\n",
        "# creamos lista de tensores\n",
        "train_dataset, validation_dataset = [\n",
        "    [\n",
        "        (\n",
        "            label_map[item[0]],\n",
        "            torch.tensor([vocab[token] for token in tokenizer(item[1])]),\n",
        "        ) for item in split\n",
        "    ] for split in [train_split, test_split]\n",
        "]\n",
        "\n",
        "def generate_batch(batch):\n",
        "    # hint: si definen la capa de embedding del modelo usando nn.EmbeddingBag,\n",
        "    # les puede ayudar computar offsets para cada elemento del batch\n",
        "\n",
        "    return (\n",
        "        # En este caso como los labels son números, \n",
        "        # el tensor es de una sola dimension de tamanno batch_size\n",
        "        torch.tensor([item[0] for item in batch]),\n",
        "\n",
        "        # En este caso se retorna un tensor de 2 dimensiones, batch_size x N,\n",
        "        # donde N es mayor largo de los ejemplo en el batch. Aca se realiza\n",
        "        # padding de los ejemplos mas cortos.\n",
        "        torch.tensor(\n",
        "            list(\n",
        "                zip(\n",
        "                    *zip_longest(\n",
        "                        *[item[1] for item in batch], fillvalue=vocab[\"<pad>\"]\n",
        "                    )\n",
        "                )\n",
        "            )\n",
        "        ),\n",
        "    )"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OD33Me-jvxt-"
      },
      "source": [
        "# Acá deben programar el entrenamiento de la red.\n",
        "\n",
        "def train_func(train_dataset):\n",
        "\n",
        "    # Entranamos el modelo\n",
        "    train_loss = 0\n",
        "    train_acc = 0\n",
        "    data = DataLoader(\n",
        "        train_dataset,\n",
        "        batch_size=BATCH_SIZE,\n",
        "        shuffle=True,\n",
        "        collate_fn=generate_batch,\n",
        "    )\n",
        "    for i, (cls, text) in enumerate(data):\n",
        "        optimizer.zero_grad()\n",
        "        cls, text = cls.to(device), text.to(device)\n",
        "        output = model(text)\n",
        "        \n",
        "        loss = criterion(output, cls)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        train_loss += loss.item()\n",
        "        \n",
        "        train_acc += (output.argmax(1) == cls).sum().item()\n",
        "\n",
        "    # Ajustar el learning rate\n",
        "    # scheduler.step()\n",
        "\n",
        "    return train_loss / len(train_dataset), train_acc / len(train_dataset)\n",
        "\n",
        "def test(test_dataset):\n",
        "    test_loss = 0\n",
        "    acc = 0\n",
        "    data = DataLoader(\n",
        "        test_dataset, batch_size=BATCH_SIZE, collate_fn=generate_batch\n",
        "    )\n",
        "    for cls, text in data:\n",
        "        cls, text = cls.to(device), text.to(device)\n",
        "        with torch.no_grad():\n",
        "            output = model(text)\n",
        "            loss = criterion(output, cls)\n",
        "            test_loss += loss.item()\n",
        "            acc += (output.argmax(1) == cls).sum().item()\n",
        "\n",
        "    return test_loss / len(test_dataset), acc / len(test_dataset)"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o2lFI-Juvy21",
        "outputId": "ca72b6ef-58fe-4040-8194-d31cdac7f3e7"
      },
      "source": [
        "print(torch.__version__)\n",
        "\n",
        "N_EPOCHS = 10\n",
        "LEARN_RATE = 2.0\n",
        "STEP_SIZE = 1\n",
        "BATCH_SIZE = 16\n",
        "EMBED_DIM = 100\n",
        "HIDDEN_SIZE = 1024\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "model = CNNClassifier(\n",
        "    vocab_size=len(vocab),\n",
        "    embed_dim=EMBED_DIM,\n",
        "    num_class=len(labels),\n",
        "    hidden_size=HIDDEN_SIZE,\n",
        "    ).to(device)\n",
        "\n",
        "print(device)\n",
        "\n",
        "optimizer = SGD(model.parameters(), lr=LEARN_RATE)\n",
        "criterion = nn.CrossEntropyLoss().to(device)\n",
        "\n",
        "# Intenten superar un Accuracy de 90% en el conjunto de test\n",
        "for epoch in range(N_EPOCHS):\n",
        "\n",
        "    start_time = time.time()\n",
        "    train_loss, train_acc = train_func(train_dataset)\n",
        "    valid_loss, valid_acc = test(validation_dataset)\n",
        "\n",
        "    secs = int(time.time() - start_time)\n",
        "    mins = secs // 60\n",
        "    secs = secs % 60\n",
        "\n",
        "    print(\n",
        "        f\"Epoch: {epoch + 1}\", f\" | time in {mins} minutes, {secs} seconds\",\n",
        "    )\n",
        "    print(\n",
        "        f\"\\tLoss: {train_loss:.4f}(train)\\t|\"\n",
        "        f\"\\tAcc: {train_acc * 100:.1f}%(train)\"\n",
        "    )\n",
        "    print(\n",
        "        f\"\\tLoss: {valid_loss:.4f}(valid)\\t|\"\n",
        "        f\"\\tAcc: {valid_acc * 100:.1f}%(valid)\"\n",
        "    )\n",
        "\n"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.9.0+cu102\n",
            "cpu\n",
            "Epoch: 1  | time in 4 minutes, 0 seconds\n",
            "\tLoss: 0.0674(train)\t|\tAcc: 58.3%(train)\n",
            "\tLoss: 0.1253(valid)\t|\tAcc: 47.4%(valid)\n",
            "Epoch: 2  | time in 3 minutes, 59 seconds\n",
            "\tLoss: 0.0482(train)\t|\tAcc: 73.5%(train)\n",
            "\tLoss: 0.0803(valid)\t|\tAcc: 44.9%(valid)\n",
            "Epoch: 3  | time in 4 minutes, 1 seconds\n",
            "\tLoss: 0.0405(train)\t|\tAcc: 78.1%(train)\n",
            "\tLoss: 0.0309(valid)\t|\tAcc: 82.4%(valid)\n",
            "Epoch: 4  | time in 4 minutes, 0 seconds\n",
            "\tLoss: 0.0358(train)\t|\tAcc: 80.6%(train)\n",
            "\tLoss: 0.0387(valid)\t|\tAcc: 78.6%(valid)\n",
            "Epoch: 5  | time in 3 minutes, 58 seconds\n",
            "\tLoss: 0.0326(train)\t|\tAcc: 82.2%(train)\n",
            "\tLoss: 0.0263(valid)\t|\tAcc: 85.4%(valid)\n",
            "Epoch: 6  | time in 3 minutes, 57 seconds\n",
            "\tLoss: 0.0312(train)\t|\tAcc: 83.2%(train)\n",
            "\tLoss: 0.0333(valid)\t|\tAcc: 80.8%(valid)\n",
            "Epoch: 7  | time in 3 minutes, 58 seconds\n",
            "\tLoss: 0.0301(train)\t|\tAcc: 83.8%(train)\n",
            "\tLoss: 0.0289(valid)\t|\tAcc: 83.3%(valid)\n",
            "Epoch: 8  | time in 3 minutes, 57 seconds\n",
            "\tLoss: 0.0286(train)\t|\tAcc: 84.6%(train)\n",
            "\tLoss: 0.0296(valid)\t|\tAcc: 82.6%(valid)\n",
            "Epoch: 9  | time in 3 minutes, 58 seconds\n",
            "\tLoss: 0.0274(train)\t|\tAcc: 85.2%(train)\n",
            "\tLoss: 0.0220(valid)\t|\tAcc: 88.8%(valid)\n",
            "Epoch: 10  | time in 3 minutes, 58 seconds\n",
            "\tLoss: 0.0269(train)\t|\tAcc: 85.6%(train)\n",
            "\tLoss: 0.0294(valid)\t|\tAcc: 83.4%(valid)\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}