Notebooks/08_RNNs_LMs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9fa8620b",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "papermill": {
     "duration": 0.011894,
     "end_time": "2023-02-20T21:43:40.132407",
     "exception": false,
     "start_time": "2023-02-20T21:43:40.120513",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "<center>\n",
    "<img src='..\\assets\\1\\lang-pic.jpg' width=600>\n",
    "</center>\n",
    "    \n",
    "# 1. Introduction\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.1 NLP series</p>\n",
    "\n",
    "This is the **eighth in a series of notebooks** covering the **fundamentals of Natural Language Processing (NLP)**. I find that the best way to learn is by teaching others, hence why I am sharing my journey learning this field from scratch. I hope these notebooks can be helpful to you too. \n",
    "\n",
    "NLP series:\n",
    "\n",
    "1. [Tokenization](./01_Tokenization.ipynb)\n",
    "2. [Preprocessing](./02_Pre_Processing.ipynb)\n",
    "3. [Bag of Words and Similarity](./03_BOW_Similarity.ipynb)\n",
    "4. [TF-IDF and Document Search](./04_TFIDF_DocSearch.ipynb)\n",
    "5. [Naive Bayes Text Classification](./05_NaiveBayes_TextClf.ipynb)\n",
    "6. [LDA Topic Modelling](./06_LDA_TopicModelling.ipynb)\n",
    "7. [Word Embeddings](./07_Word_Embeddings.ipynb)\n",
    "8. Recurrent Neural Networks (RNNs) and Language Modelling\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/08_RNNs_LMs.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>\n",
    "[![Kaggle](https://kaggle.com/static/images/open-in-kaggle.svg)](https://kaggle.com/kernels/welcome?src=https://github.com/JUSTSUJAY/NLP_One_Shot/blob/main/Notebooks/08_RNNs_LMs.ipynb)\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">1.2 Outline</p>\n",
    "\n",
    "In this notebook, we're going to cover **Recurrent Neural Networks** (RNNs) to model the sequential nature of language. This is the first class of models that we've come across that **takes word order into account**. \n",
    "\n",
    "We'll also see how RNNs and it's variants like GRU and LSTM can be used to approach tasks like **Part-of-Speech Tagging**, **Named Entity Recognition** and **Language Modelling**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b9e72d7",
   "metadata": {
    "papermill": {
     "duration": 0.011377,
     "end_time": "2023-02-20T21:43:40.154275",
     "exception": false,
     "start_time": "2023-02-20T21:43:40.142898",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2. Recurrent Neural Networks\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.1 Drawbacks to non-sequential models</p>\n",
    "\n",
    "In earlier notebooks, we saw how to use **Bag-of-Words** like approaches to vectorize text. This worked well for simple applications but it does have a number of drawbacks. The main ones include:\n",
    "\n",
    "* There is no way to handle **Out-of-Vocabulary** (OOV) words. If a new word appears in a later document, it will just be dropped. \n",
    "* It creates **sparse matrices** which can be inefficient, although we can overcome this by using a dictionary representation. \n",
    "* It isn't able to capture similarity between **synonyms**. \n",
    "* Word order is lost so words have **no relationship** to each other. For example, \"man eats bread\" is very different to \"bread eats man\" but they would have the same representations.\n",
    "\n",
    "We also came across **static word embeddings**. Whilst this was an improvement to Bag-of-Words because of its ability to capture word meaning, we still had no way to model the sequential nature of language. \n",
    "\n",
    "The main challenge with taking word order into account is that sentences can be of **different lengths**. So we need a model that can **automically scale** depending on the sequence length. This is where RNNs come in. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.2 Recurrence</p>\n",
    "\n",
    "<center>\n",
    "<img src=\"../assets/8/RNN.png\" width=650>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "The **recurrence** in an RNN refers to the **repeating** design of the model. An input word vector ($x_t$) is passed through a **fully connected layer** to generate some activations ($a_t$), which can be used to make predictions ($y_t$) at that time step. But these activations are also passed to the next time step acting as a form of **memory**, which are combined to make the next prediction ($y_{t+1}$). This design is repeated at each time step, so there can be any number of units in the sequence. \n",
    "\n",
    "There are **3 weights matrices**, which are used repeatedly used in each cell. $W_x$ are the weights for the input, $W_y$ are the weights for the output and $W_h$ are the weights for the memory. They are combined using the following equations:\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\large\n",
    "h_t = f(W_x x_t + W_h h_{t-1} + b), \\qquad y_t = g(W_y a_t + c)\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "where $f$ and $g$ are **activation functions** like ReLU or tanh and $b$ and $c$ are the **biases**. Note that it is common to set $h_0$ to be the zero vector. And not every task requires a prediction at each time step ($y_t$), some only require one at the end of each sentence. \n",
    "\n",
    "As an example, if we pass the sentence \"She has a cat\" through an RNN for part-of-speech tagging; at $t_1$ a prediction is made using \"She\", at $t_2$ \"She has\" is used, similarly at $t_3$ \"She has a\" is used and at $t_4$ the whole sentence is used to make a prediction. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">2.3 Training RNNs</p>\n",
    "\n",
    "<center>\n",
    "<img src=\"../assets/8/BPTT.png\" width=600>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "Training an RNN is **trickier** than a standard feed-forward neural network because of the time aspect. The algorithm used is called **Backpropagation Through Time** (BPTT). The main idea is that when we compute gradients, we need to **reverse all the arrows** in the computation graph and multiply them using the chain rule. \n",
    "\n",
    "For example at the final time step, all the input vectors ($x_t$) and all the hidden states ($h_t$) were used to make this prediction so we have to update all of their corresponding weights. As they share weights matrices, we can add each component together. You can find more details in [this article](https://d2l.ai/chapter_recurrent-neural-networks/bptt.html), but luckily all major deep learning frameworks will **implement this for us**. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "90221baf",
   "metadata": {
    "papermill": {
     "duration": 0.009757,
     "end_time": "2023-02-20T21:43:40.174020",
     "exception": false,
     "start_time": "2023-02-20T21:43:40.164263",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3. Language Models\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.1 Word prediction</p>\n",
    "\n",
    "A **Language model** determines the **probability** of a sequence of words. For example, if you consider the incomplete sentence \"I'm going to walk the ...\", a reasonable guess for the next word would be \"dog\" as opposed to \"ice-cream\" or \"mountain\". We can see that language is **predictable** and some words are more likely to appear than others. \n",
    "\n",
    "Mathematically, we can use the **multiplication rule** to represent the probability of a sequence as a product of conditional probabilities:\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\large\n",
    "\\begin{align*}\n",
    "\\mathbb{P}(w_1,...,w_n) &= \\prod_{i=1}^{n} \\mathbb{P} (w_i | w_1,...,w_{i-1}) \\\\\n",
    "& = \\mathbb{P}(w_1) \\mathbb{P}(w_2|w_1) ... \\mathbb{P}(w_n|w_1,...,w_{n-1})\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "This is a **difficult task** because it is impossible to get a training corpus with every combination of words in a sequence. We will see that we can use an RNN to **train** a language model by using a clever trick. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.2 Training an RNN for language modelling</p>\n",
    "\n",
    "To train an RNN for language modelling, we start by **segmenting** a corpus into chunks, sentences or paragraphs to feed into the model. The longer the sequence length the more **computationally expensive** it is to train but the better the model is at capturing **long range dependencies**. \n",
    "\n",
    "We then use a **self-supervised** learning approach. At every time step, we get the RNN to try to **predict the next word** in the sentence in the training corpus. What is great is that we don't need to label the data, as we use the words as the labels themselves. \n",
    "\n",
    "The outputs $y_t$ are **probability distributions** over the whole vocabulary. They assign a probability to each word, which represents how likely that word is to appear next given the previous words. A **cross-entropy loss** can be used to measure how good each prediction is. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.3 Text generation</p>\n",
    "\n",
    "To **generate new text** after training, we modify the RNN to **sample** the output probability distribution at each time step and feed this to be **input** word for following time step, that is $x_{t+1} \\sim y_{t}$. \n",
    "\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"../assets/8/RNN-for-LM.jpg\" width=600>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "So given an input word (this can fixed or randomly sampled), we can generated any amount of text we like. It won't be perfect or grammatically correct by default but it will produce text that has a **similar distribution** to the one it was trained on.\n",
    "\n",
    "For example, if we use a training corpus comprising of the Lord of the Rings books, then the generated text will look somewhat similar to languge in these books. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">3.4 Evaluating language models</p>\n",
    "\n",
    "If we **measure** the output **probabilities** of a language model on a **test set**, we expect better models to produce **higher** probabilities for the correct words. \n",
    "\n",
    "A common **metric** to evaluate language models is called **Perplexity** and is defined as:\n",
    "\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\large\n",
    "\\text{Perplexity} = \\left(\\prod_{i=1}^{T} \\mathbb{P}_{LM}(w_i|w_1,...,w_{i-1})\\right)^{-1/T} = \\sqrt[T]{\\prod_{i=1}^{T} \\frac{1}{\\mathbb{P}_{LM}(w_i|w_1,...,w_{i-1})}}\n",
    "$$\n",
    "\n",
    "<br>\n",
    "\n",
    "Because of the inverse probabilities, the **lower** the perplexity score the **better** and the more natural the language produced. The T-th root is there to **control** for different test lengths.\n",
    "\n",
    "Note that **humans** have a certain perplexity score too, depending on the corpus, so getting a model's perplexity score too low below that **benchmark** may also produce unnatural text."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1eabd11",
   "metadata": {
    "papermill": {
     "duration": 0.009701,
     "end_time": "2023-02-20T21:43:40.193590",
     "exception": false,
     "start_time": "2023-02-20T21:43:40.183889",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 4. RNN variants\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.1 Drawbracks of simple RNNs</p>\n",
    "\n",
    "We've now seen the architecture of a **standard** recurrent neural network. Whilst they are big **improvement** in terms of dealing with word order, they do have some shortcomings in their current state. These include:\n",
    "\n",
    "* Struggling to capture **long range dependencies** because the signal gets **weakens** over time. Simple RNNs have no way of deciding what information is important to 'remember' and what can be 'forgetten'.\n",
    "* **Vanishing and/or exploding gradients** can occur during training because of the many components affecting each matrix of weights.\n",
    "\n",
    "There are some ways to **dampen** the affects of these problems, like **gradient clipping**, **batch normalization**, **weight initialization** etc. However, better **cell architectures** have been found, which avoid these problems and produce better results. These include **GRU** and **LSTM**, which we will discuss now. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.2 GRU</p>\n",
    "\n",
    "<center>\n",
    "<img src=\"../assets/8/gru-cell.png\" width=600>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "GRU stands for **Gated Recurrent Unit** and is a variant of a RNN where the cell/unit has the ability to **forget** information it deems irrelevant. As a result, it tends to do a better job at capturing **long range dependencies** at the cost of increasing the number of **parameters** in the model. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.3 LSTM</p>\n",
    "\n",
    "<center>\n",
    "<img src=\"../assets/8/lstm-cell.jpg\" width=500>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "LSTM stands for **Long Short Term Memory** and is similar to GRU but has more parameters and can manage more complex information transfer. In addition to the normal hidden state that gets passed between units, LSTM has an additional **cell state** that captures **long term memory**. The affect is that the model can make predictions using both short term and long term memory. \n",
    "\n",
    "LSTM is more complicated than GRU, so it is able to reach **better performance** on some tasks but again at the cost of increasing the number of **parameters** in the model and therefore taking **longer to train**. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.4 Deep RNNs</p>\n",
    "\n",
    "<center>\n",
    "<img src=\"../assets/8/deep-rnn.png\" width=250>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "There are several ways to **combine** recurrent neural networks. One way is to **stack** multiple RNNs on top of each other. We can do this since each RNN takes in a sequence and outputs another sequence of the same length. \n",
    "\n",
    "Just like adding another **layer** in a feed-forward neural network, stacking RNNs allows the model to learn more useful **representations** of the data. This can be useful for very **complex** tasks that consist a large number **interactions** in the data, for example machine translation. Again, the thing to keep in mind is that this takes much **longer** to train the model. \n",
    "\n",
    "<br>\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">4.5 Bidirectional RNNs</p>\n",
    "\n",
    "There are some tasks where it makes sense to use not only previous words in the sentence but also **future words** in the sentence. For example, the word \"India\" in the two sentences \"India has a population of 1.4 billion people\" and \"India got her bag ready to go to school\" has **different meaning** based on the words that come **after** it. \n",
    "\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"../assets/8/bidirectional-rnns.png\" width=500>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "**Bidirectional RNNs** work by adding a **second** RNN that processes the text in **reverse order**. This ensures that the model uses the **whole sentence** each time it makes a prediction. \n",
    "\n",
    "Note that the cells in this model can be either **simple RNN, GRU or LSTM units** depending on your preference. It is even possible to **stack** bidirectional RNNs to create **deep bidirectional RNNs**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6476dbd0",
   "metadata": {
    "papermill": {
     "duration": 0.00974,
     "end_time": "2023-02-20T21:43:40.213234",
     "exception": false,
     "start_time": "2023-02-20T21:43:40.203494",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 5. Application\n",
    "\n",
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">5.1 Part-of-Speech Tagging</p>\n",
    "\n",
    "We're going to build a **bidirectional LSTM** to perform **Part-of-Speech (PoS) tagging**. We'll use datasets from nltk to train our model\n",
    "<br>\n",
    "\n",
    "* Shakespeare Text - [Link](https://www.kaggle.com/datasets/adarshpathak/shakespeare-text)\n",
    "\n",
    "\n",
    "**Import libraries**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7c85f425",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:43:40.235536Z",
     "iopub.status.busy": "2023-02-20T21:43:40.234685Z",
     "iopub.status.idle": "2023-02-20T21:43:46.988087Z",
     "shell.execute_reply": "2023-02-20T21:43:46.987098Z"
    },
    "papermill": {
     "duration": 6.767505,
     "end_time": "2023-02-20T21:43:46.990620",
     "exception": false,
     "start_time": "2023-02-20T21:43:40.223115",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "sns.set(style='darkgrid', font_scale=1.6)\n",
    "import requests\n",
    "import nltk\n",
    "from nltk.corpus import treebank, brown, conll2000\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96e6df11",
   "metadata": {
    "papermill": {
     "duration": 0.009877,
     "end_time": "2023-02-20T21:43:47.010798",
     "exception": false,
     "start_time": "2023-02-20T21:43:47.000921",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Load data**\n",
    "\n",
    "<br>\n",
    "\n",
    "We'll use 3 PoS datasets together. Because they all have different tagging schemes, we also download the **universal tagging schema** to ensure they are all tagged **consistently**. Note that this **simplifies** some of the tags, for example example proper noun is converted to noun. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6c5622dd",
   "metadata": {
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2023-02-20T21:43:47.032897Z",
     "iopub.status.busy": "2023-02-20T21:43:47.031716Z",
     "iopub.status.idle": "2023-02-20T21:43:47.224885Z",
     "shell.execute_reply": "2023-02-20T21:43:47.223830Z"
    },
    "papermill": {
     "duration": 0.208074,
     "end_time": "2023-02-20T21:43:47.228874",
     "exception": false,
     "start_time": "2023-02-20T21:43:47.020800",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package treebank to /usr/share/nltk_data...\n",
      "[nltk_data]   Package treebank is already up-to-date!\n",
      "[nltk_data] Downloading package brown to /usr/share/nltk_data...\n",
      "[nltk_data]   Package brown is already up-to-date!\n",
      "[nltk_data] Downloading package conll2000 to /usr/share/nltk_data...\n",
      "[nltk_data]   Package conll2000 is already up-to-date!\n",
      "[nltk_data] Downloading package universal_tagset to\n",
      "[nltk_data]     /usr/share/nltk_data...\n",
      "[nltk_data]   Package universal_tagset is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# PoS datasets\n",
    "nltk.download('treebank')\n",
    "nltk.download('brown')\n",
    "nltk.download('conll2000')\n",
    "\n",
    "# Use universal tagging system\n",
    "nltk.download('universal_tagset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ffa2a839",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:43:47.250801Z",
     "iopub.status.busy": "2023-02-20T21:43:47.250516Z",
     "iopub.status.idle": "2023-02-20T21:44:03.731511Z",
     "shell.execute_reply": "2023-02-20T21:44:03.730410Z"
    },
    "papermill": {
     "duration": 16.494292,
     "end_time": "2023-02-20T21:44:03.733657",
     "exception": false,
     "start_time": "2023-02-20T21:43:47.239365",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset size: 72202\n",
      "[('Pierre', 'NOUN'), ('Vinken', 'NOUN'), (',', '.'), ('61', 'NUM'), ('years', 'NOUN'), ('old', 'ADJ'), (',', '.'), ('will', 'VERB'), ('join', 'VERB'), ('the', 'DET'), ('board', 'NOUN'), ('as', 'ADP'), ('a', 'DET'), ('nonexecutive', 'ADJ'), ('director', 'NOUN'), ('Nov.', 'NOUN'), ('29', 'NUM'), ('.', '.')]\n"
     ]
    }
   ],
   "source": [
    "# Retrieve all PoS-tagged sentences and join them in one list\n",
    "tagged_sentences = treebank.tagged_sents(tagset='universal') + brown.tagged_sents(tagset='universal') + conll2000.tagged_sents(tagset='universal')\n",
    "\n",
    "# Example\n",
    "print(\"Dataset size:\", len(tagged_sentences))\n",
    "print(tagged_sentences[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "821a83c1",
   "metadata": {
    "papermill": {
     "duration": 0.01016,
     "end_time": "2023-02-20T21:44:03.754320",
     "exception": false,
     "start_time": "2023-02-20T21:44:03.744160",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Separate features and labels**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9a23e70a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:03.788614Z",
     "iopub.status.busy": "2023-02-20T21:44:03.788146Z",
     "iopub.status.idle": "2023-02-20T21:44:10.089200Z",
     "shell.execute_reply": "2023-02-20T21:44:10.088029Z"
    },
    "papermill": {
     "duration": 6.324383,
     "end_time": "2023-02-20T21:44:10.092052",
     "exception": false,
     "start_time": "2023-02-20T21:44:03.767669",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Pierre', 'Vinken', ',', '61', 'years', 'old', ',', 'will', 'join', 'the', 'board', 'as', 'a', 'nonexecutive', 'director', 'Nov.', '29', '.']\n",
      "['NOUN', 'NOUN', '.', 'NUM', 'NOUN', 'ADJ', '.', 'VERB', 'VERB', 'DET', 'NOUN', 'ADP', 'DET', 'ADJ', 'NOUN', 'NOUN', 'NUM', '.']\n"
     ]
    }
   ],
   "source": [
    "# Initalize\n",
    "X, y = [], []\n",
    "\n",
    "# Split sentences into words and tags\n",
    "for s in tagged_sentences:\n",
    "    sentence, tags = zip(*s)\n",
    "    X.append(list(sentence))\n",
    "    y.append(list(tags))\n",
    "    \n",
    "# Example\n",
    "print(X[0])\n",
    "print(y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ead1b9d9",
   "metadata": {
    "papermill": {
     "duration": 0.010196,
     "end_time": "2023-02-20T21:44:10.113696",
     "exception": false,
     "start_time": "2023-02-20T21:44:10.103500",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Train-valid-test split**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "fcaed42f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:10.136143Z",
     "iopub.status.busy": "2023-02-20T21:44:10.135252Z",
     "iopub.status.idle": "2023-02-20T21:44:10.200843Z",
     "shell.execute_reply": "2023-02-20T21:44:10.199112Z"
    },
    "papermill": {
     "duration": 0.079429,
     "end_time": "2023-02-20T21:44:10.203479",
     "exception": false,
     "start_time": "2023-02-20T21:44:10.124050",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train size: 43320\n",
      "Valid size: 14441\n",
      "Test size: 14441\n"
     ]
    }
   ],
   "source": [
    "# Create train, validation and test sets\n",
    "X_tr, X_test, y_tr, y_test = train_test_split(X, y, test_size=0.2, shuffle=True, random_state=0)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X_tr, y_tr, test_size=0.25, shuffle=True, random_state=0) # 0.25 * 0.8 = 0.2\n",
    "\n",
    "# Print shapes\n",
    "print(\"Train size:\", len(X_train))\n",
    "print(\"Valid size:\", len(X_valid))\n",
    "print(\"Test size:\", len(X_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c3b14bf",
   "metadata": {
    "papermill": {
     "duration": 0.010294,
     "end_time": "2023-02-20T21:44:10.224632",
     "exception": false,
     "start_time": "2023-02-20T21:44:10.214338",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Tokenizers**\n",
    "\n",
    "We need to tokenize both the features and the labels using **two different tokenizers** as they are both sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5cba2c82",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:10.247278Z",
     "iopub.status.busy": "2023-02-20T21:44:10.246330Z",
     "iopub.status.idle": "2023-02-20T21:44:11.505165Z",
     "shell.execute_reply": "2023-02-20T21:44:11.502630Z"
    },
    "papermill": {
     "duration": 1.272957,
     "end_time": "2023-02-20T21:44:11.507907",
     "exception": false,
     "start_time": "2023-02-20T21:44:10.234950",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word vocabulary size: 46881\n",
      "Tag vocabulary size: 12\n",
      "\n",
      "Possible tags: {'noun': 1, 'verb': 2, '.': 3, 'adp': 4, 'det': 5, 'adj': 6, 'adv': 7, 'pron': 8, 'conj': 9, 'prt': 10, 'num': 11, 'x': 12}\n"
     ]
    }
   ],
   "source": [
    "# Define tokenizers\n",
    "word_tokenizer = keras.preprocessing.text.Tokenizer(oov_token='<OOV>')\n",
    "tag_tokenizer = keras.preprocessing.text.Tokenizer()\n",
    "\n",
    "# Fit tokenizers\n",
    "word_tokenizer.fit_on_texts(X_train)\n",
    "tag_tokenizer.fit_on_texts(y_train)\n",
    "\n",
    "# Print vocabulary sizes\n",
    "print(\"Word vocabulary size:\", len(word_tokenizer.word_index))\n",
    "print(\"Tag vocabulary size:\", len(tag_tokenizer.word_index))\n",
    "\n",
    "# Print set of tags\n",
    "print(\"\\nPossible tags:\", tag_tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "71d68f57",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:11.531227Z",
     "iopub.status.busy": "2023-02-20T21:44:11.530898Z",
     "iopub.status.idle": "2023-02-20T21:44:13.322905Z",
     "shell.execute_reply": "2023-02-20T21:44:13.321853Z"
    },
    "papermill": {
     "duration": 1.806274,
     "end_time": "2023-02-20T21:44:13.325533",
     "exception": false,
     "start_time": "2023-02-20T21:44:11.519259",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Map tokens to integer ids\n",
    "X_train_ids = word_tokenizer.texts_to_sequences(X_train)\n",
    "X_valid_ids = word_tokenizer.texts_to_sequences(X_valid)\n",
    "X_test_ids = word_tokenizer.texts_to_sequences(X_test)\n",
    "\n",
    "y_train_ids = tag_tokenizer.texts_to_sequences(y_train)\n",
    "y_valid_ids = tag_tokenizer.texts_to_sequences(y_valid)\n",
    "y_test_ids = tag_tokenizer.texts_to_sequences(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1c5ac4d",
   "metadata": {
    "papermill": {
     "duration": 0.010552,
     "end_time": "2023-02-20T21:44:13.347170",
     "exception": false,
     "start_time": "2023-02-20T21:44:13.336618",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Padding**\n",
    "\n",
    "<br>\n",
    "\n",
    "Padding is a way of making all input sequences the **same length**, by filling shorter sequences with 0's and truncating longer sequences. Although is isn't necessarily required for RNNs, it usually **speeds up training** because it is easier to **batch** training examples together. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2cae038a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:13.370643Z",
     "iopub.status.busy": "2023-02-20T21:44:13.369758Z",
     "iopub.status.idle": "2023-02-20T21:44:14.168119Z",
     "shell.execute_reply": "2023-02-20T21:44:14.167199Z"
    },
    "papermill": {
     "duration": 0.812726,
     "end_time": "2023-02-20T21:44:14.170593",
     "exception": false,
     "start_time": "2023-02-20T21:44:13.357867",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAokAAAEYCAYAAAAwBWxaAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABNy0lEQVR4nO3deXhTVf4/8PfN3qR7m7a0QFnTsrRQGBAEZVFWcRQU0ar80BmRmcEdvi7z1a/jghuKIz6oI4IywAjINiKggyggUFmE0tJCy1JogW7pkmZr2uT+/qjNENOWprRN0r5fz8NDe3Jy87knN/DJuWcRRFEUQURERER0FYm3AyAiIiIi38MkkYiIiIjcMEkkIiIiIjdMEomIiIjIDZNEIiIiInLDJJGIiIiI3DBJJCLqwDZt2oSEhARs2rTJ26G02HPPPYeEhAQUFBR4OxSiToVJIhG1OZPJhA8//BB33HEHBg8ejOTkZIwdOxazZ8/Ghx9+iNLS0naLpaCgAAkJCXjuuefa7TWpaXxPiHyTzNsBEFHHZjQacd999yEnJwc9evTA73//ewQHB+Py5cvIzs7G0qVLMWTIEERGRno7VCIiugqTRCJqU59//jlycnJwzz334JVXXoEgCC6P5+bmIjAw0EvRERFRY3i7mYjaVHp6OgAgNTXVLUEEgL59+6JLly5u5VlZWXj88cdx4403YuDAgbj11lvx7rvvwmQyudT7+eefkZCQgKVLl+LEiROYM2cOUlJSMHz4cDzzzDMoKytz1t20aRNuueUWAMDmzZuRkJDg/HP1eDej0YglS5Zg8uTJSEpKwg033IA///nPOHXqlFuc48ePx/jx42E0GvHaa69h9OjRSEpKwl133YUDBw402CbFxcV47bXXMGHCBCQlJWHEiBG4//77Gxw3+O233+LBBx/E0KFDkZycjDvvvBPr1q1r8LieSktLwyOPPIIbbrgBSUlJmDp1Kv7xj3+gtrbWpd7V4xr37duHWbNmYdCgQRg5ciRefvllWCwWt2ObTCYsWrQIo0ePRnJyMqZPn44dO3a4jZFs7nsCAKIoYtWqVZg0aZLzmvjiiy/cXru6uhqffvopbr/9dqSkpGDIkCGYNGkSnn/+eRQVFbVK2xF1BuxJJKI2FRISAgC4cOEC+vXr16zn/Oc//8FTTz0FuVyOW2+9FZGRkcjKysI//vEP/Pzzz1i9ejUUCoXLczIyMrB8+XKMHDkS9957L3755Rds27YNBQUF+PLLLyEIAvr164fZs2dj1apVSExMxK233up8fnBwMACgrKwMDzzwAM6ePYsbbrgBY8eORUVFBb799lscOHAAK1euREpKistr19TU4A9/+AOMRiMmT54Mg8GAb775BnPnzsVXX32FxMREZ92zZ89i9uzZKC0txYgRIzBp0iQYjUZkZWVh1apVmDFjhrPuW2+9hRUrViAuLg5Tp06FSqXCgQMH8NJLL+Hs2bN44YUXPHszrrJ69Wq89tprCAsLw/jx4xESEoKjR4/i3XffxYkTJ/Dhhx+6Pef777/H3r17ccsttyAlJQX79+/Hv/71L1RWVmLJkiXOena7HXPnzsWRI0cwcOBATJ8+HSUlJXj22WcxcuRIl2M25z2p9/bbb+Po0aMYO3YsRo0ahZ07d2LRokVQKBS47777nPUWLlyIb7/9FkOGDMGsWbMgCAIKCgrw3Xff4a677kJ0dHSL242oUxGJiNrQd999J+p0OjElJUV88803xQMHDogGg6HR+nq9XkxJSRHHjx8vFhYWujz26aefijqdTly+fLmzLC0tTdTpdKJOpxN37tzpLLfb7eLs2bNFnU4n/vLLL87y/Px8UafTic8++2yDr//UU0+JOp1O/Prrr13K8/LyxCFDhojTpk1zKR83bpyo0+nE+fPnizabzVm+adMmUafTiS+++KJL/enTpzd4fFEUxStXrjh/3rt3r6jT6cQ//elPotVqdZbbbDbxT3/6k6jT6cT09PQGz+FqGzduFHU6nbhx40ZnWU5Ojti/f39x5syZLu+Fw+EQ//a3v4k6nU7csWOH2zEGDBggHjt2zFlutVrFyZMniwkJCS7v1bp160SdTic+/vjjosPhcJYfPnxYTEhIcIvnWu/Js88+K+p0OnHChAliSUmJszwvL0/s37+/OHHiRGeZwWAQExISxL/85S9ux7FaraLJZLpWkxHRr3i7mYja1IQJE/D000/D4XBgxYoVmDNnDn73u99h8uTJeOutt1BYWOhSf+vWrTCZTFi4cKFbj8/DDz+M8PBwfPPNN26vM3z4cEyaNMn5u0QiwZ133gkAyMzMbFasZWVl2LFjB8aOHYtp06a5PBYfH4977rkHOTk5yMnJcXvus88+C7lc7vz99ttvh0wmc3nt9PR0nDx5EqNHj3Y7PgDExMQ4f16zZg0EQcCrr74KpVLpLJfL5XjyyScBADt27GjWef3WunXrUFtbixdffBFBQUHOckEQ8PTTT0MQBGzfvt3tedOmTcPgwYOdvyuVStx2220QRREnT550ln/99dcAgCeffNJliMHvfvc73HTTTS2KGQDmzZvnMsEpPj4eQ4YMQV5eHoxGo/McRFF0abOr41Wr1S1+faLOhrebiajNPfroo0hNTcWePXtw7NgxZGRkIDMzEytWrMCGDRvw6aefOm/h1o9hPHr0KHJzc92OJZPJcP78ebfyhm5l1yeZBoOhWXFmZGTA4XDAbDZj6dKlbo+fPXsWAHDu3DnodDpneXBwMLp27eoWZ0REhMtrZ2RkAABGjx59zVhOnDiBwMBArF271u2x+jGD586da8ZZuUtPT4cgCNi9ezd+/PFHt8dVKlWDx25uG586dQqhoaHo2bOnW/3Bgwdj7969LYq7f//+jb5+VVUVAgMDERgYiJtuugnbtm1DYWEhbr31VgwdOhQDBgyAVCpt0esSdVZMEomoXQQFBWHatGnOHjS9Xo+//e1v+Pbbb/HSSy85e58qKysBAKtWrfL4+L9VnxQ4HI5mHaP+tQ8dOoRDhw41Wu+3EzUaem2gLlG8+rWrqqoAAFFRUc2Kpba2tsGxgfXMZvM1j9PYsUVRxLJlyzw6dnPb2GQyIT4+vsHjRkREeBquU0Oz4GWyuv/G7Ha7s+zvf/87PvroI2zbtg1vvvkmACAsLAxz5szB3LlzIZHwJhpRczBJJCKviIiIwNtvv40ffvgBOTk5KC8vR1hYmDMR+O677xpNNNpK/WvPmzcPTz31VKsfv34iRnFxcbNiUSqVLe51u9axpVIpjh8/7jYBqDVoNBqXWeVX0+v1rf56Db3+ggULsGDBApw/fx4HDx7E6tWrsWTJEqhUKsyZM6fNYyDqCPh1ioi8RqFQOHuCRFEEACQlJQEAjh8/3iavWd/zdXXPU72kpCQIguC85d3a6s/tp59+albdoqIiXLlypU3isNvtOHHiRKsfGwASExNRUVHR4LCAht7Xpt6T69WzZ0+kpqZixYoVAOpmaBNR8zBJJKI2tW7dOpdJDVdbvXo1zGYzevbsifDwcADAXXfdBbVajcWLFyMvL8/tOQaDAVlZWS2Op743r6H18rRaLSZNmoSDBw82OBbQ4XA0eRv6WpKTkzFgwAD89NNP2LZtm9vjV8f04IMPAgD++te/Om9TXy0/P7/FexmnpqZCKpXilVdeabBXs7S01Dn+siVuu+02AHW3feuTfwA4cuQI9u3b51a/qffEU2VlZc6xn1er3/pRpVJd92sQdRa83UxEbWrPnj146aWX0LNnT6SkpCAqKgpVVVU4fvw4Tp48CaVSif/7v/9z1o+IiMDixYvx1FNPYdq0abj55psRHx8Pi8WC/Px8HD58GHfeeSdeeeWVFsWj0WiQlJSEw4cP469//Su6d+8OQRBw3333ISgoCC+//DLOnTuHv/3tb/jqq6+QlJQEtVqNK1eu4Pjx49Dr9Q0mIc31zjvv4MEHH8QzzzzjPL7JZEJ2djYsFgu2bNkCABgzZgweffRRfPLJJ5gwYQJGjx6NmJgYlJWV4dy5czh+/DjeffddtwkzzZGQkID//d//xauvvopJkyZhzJgxiIuLg8FgQF5eHn755Rc88cQT6N27d4vO8e6778aWLVuwY8cOFBQUYOTIkSgpKcH27dsxZswY/Pjjjy7jAq/1nniiqKgId999NxISEtC/f39otVqUlpbiP//5D6RSKWbPnt2icyLqjJgkElGbWrBgAVJSUnDgwAH8/PPPKCkpgUQiQUxMDGbNmoU5c+agV69eLs+55ZZbsGnTJixfvhwHDx7E3r17odFoEBsbi9mzZ2P69OnXFdNbb72FRYsWYefOnc6lU6ZOnYqgoCCEhYVh3bp1WLVqFXbu3ImtW7dCEARotVoMGTIEkydPvq7X7t27NzZv3oyPP/4Ye/bswZEjRxAUFIQ+ffq4jZV7+umnMXToUKxZswb79u2DyWRCWFgY4uPj8T//8z9uC1N7IjU1Ff369cPKlStx5MgR7Nq1CyEhIYiLi8Of/vSnBpfoaS6ZTIbly5fj/fffx86dO/HFF1+gV69eeOutt3DlyhX8+OOP0Gg0Ls9p6j3xRFxcHB577DEcPHgQP/30EyoqKhAZGYmRI0fij3/8IwYNGtTi8yLqbATx6nsBREREbWjhwoX497//jW+++QZ9+vTxdjhE1ASOSSQiolbX0FjHI0eOYPv27ejRo0eLb2UTUfvh7WYiImp1L774IoqLi5GUlITAwECcO3cOe/bsgUQiwV//+leXnViIyDfxdjMREbW6LVu24Msvv8T58+dhNBoRGBiIlJQUzJ07F0OGDPF2eETUDEwSiYiIiMgNxyQSERERkRsmiURERETkhhNX2kh5uQkOR+veyY+ICIReb2zVY3YUbJvGsW0ax7ZpHNumYWyXxrFtGufLbSORCAgL0zT4GJPENuJwiK2eJNYflxrGtmkc26ZxbJvGsW0axnZpHNumcf7YNl5PEk0mE1auXInMzExkZmaipKQEkyZNwgcffOBWd/z48bh06ZJbuUKhaHCbrJycHCxevBhHjx6FKIpISUnBwoULkZiY6Fa3oqIC7733Hr7//ntUVVWhT58+eOSRRzBlypTWOVEiIiIiP+L1JLG8vBxLly6FVqvFwIED8cMPPzRZv2vXrnj88cddyqRSqVu9vLw8pKamQqVSYd68eVAqlVizZg1SU1Oxfv16l5X+bTYbHnroIeTm5mL27Nno0aMHduzYgSeffBLV1dW48847W+VciYiIiPyF15PEqKgo7N27F9HR0QDqNp5vSkhICO64445rHve9996D2WzG2rVrodPpAABTpkzB5MmT8c477+CTTz5x1l2/fj2ysrLw8ssv47777gNQt0F9amoq3nzzTUyePBkqlaqlp0hERETkd7w+u1mhUDgTxOaqra11bgDfEJPJhN27d2PEiBHOBBEAtFotpkyZgn379qGiosJZvm3bNqjVatx1113OMolEgtTUVJSXl2P//v0exUdERETk77yeJHoqJycHgwcPxtChQzFs2DC88MILKCsrc6tTU1OD5ORkt+cPGjQIdrsd2dnZAACHw4Hs7Gz069cPCoXCrS6ABsc7EhEREXVkXr/d7Im+ffti5syZ6N27N6qrq7F//35s2rQJR44cwVdffYXg4GAA/91YPioqyu0Y9WVFRUUAgMrKSlit1mbV9URERKDHz2kOrTaoTY7bEbBtGse2aRzbpnFsm4axXRrHtmmcP7aNXyWJV48jBIDbb78diYmJeOONN/D55587J7RYLBYAcOsZvLqsvo7Vam20rlKpdKnjCb3e2OrT3bXaIJSUVLXqMduDRCIAaNvp//7aNu2BbdM4tk3j2DYNY7s0jm3TOF9uG4lEaLRjy+9uN//W7NmzoVarXcYN1id3NpvNrX59Wf1ElKbqVldXu9Qlz0kkAtKyipCWVeRMFomIiMj3+VVPYkMkEgm6dOniMhElJiYGwH9vO1+tvqx+skxoaCiUSmWTdRu6Fd3ZXJ3gXd0j2Fj51Yxm9wSciIiIfJvfJ4m1tbW4fPky+vfv7yzT6XSQy+U4ceKEW/309HRIpVL069cPQF2S2b9/f2RnZ8Nms7ncdk5PTwcAJCUltfFZ+Lb63kCj2YZAtQIj+kfD4RAbLSciIiL/5zdJYkVFBUJDQ93KP/74Y1gsFowdO9ZZptFoMHbsWOzevRu5ubno27cvAKCkpAQ7d+7EqFGjEBYW5qw/depUHDt2DBs3bnSuk+hwOLB27VqEhoZi1KhRbXpu/sBotsFgskFAXdKYV2jAibN6HM8tRaXRBhEitu0/DwAIUisQolGgb7dQ3JgU493AiYiIqEV8IklcvXo1DAaD8/dz585h2bJlAIBhw4Zh2LBh2LJlCzZu3IibbroJXbt2hc1mw08//YR9+/Zh4MCBePDBB12O+fTTT+PgwYN46KGHMGfOHCiVSqxevRp2ux0LFy50qTtr1ixs3LgRr7/+OgoKChAfH48dO3bg2LFjeOONNxAQEND2jeAnKkw2vPBJGgpK6tapVCtlUCmkCFDKEB2mhkMUUWWpQeb5MqRlFWHtrhz0iA7CkASty3Gac5uaiIiIvMcnksQVK1a47Mmcm5uLv//97wCA+fPnY9iwYUhKSkJaWhq2b9/uXBcxPj4ejz32GP74xz+6TS7p1asX1qxZg8WLF2PZsmXOvZuXLFnissA2UDd55fPPP8e7776LTZs2wWg0ok+fPliyZAmmTp3axmfvPy4WG3E8txQBCin6xYdhxIBo1NpFVBqrEaJRYMLw7s7b0N8duoiLRVUoNVhxKq8CxRUWDNFpER2m5m1qIiIiPyCIosj/ndtAR1oCRyIR8PmObOw5dhmxWg1uHhQLk6UGsVoNTJZaVBqrEavVQBAkqDJVIypcjZJyi7P8/GUDfjx2GRKJgPkzkjCgZzh2HcmHwWRDsEaBW3/X7brbypeXF/A2tk3j2DaNY9s0jO3SOLZN43y5bTr0EjjU9s5cqsS+9CsI1igwaXg3yKQNXzb14xbN1hqX8qgwNaaO6I7wICWWbjyBK2Xm9gibiIiIrgOTRGqS3eHAZ9uyEKCUYUT/aCjk0hYdJzpCjfFDuwIA3lrzC2pqHa0ZJhEREbUyJonUpLSTRbiiN2OILhJKRcsSxKsN1WlhMFZj34kr4EgHIiIi38UkkRpV63Bg3e4ziA5Xo3t06+w5GRGiwqjkLrhcasKZS4ZrP4GIiIi8gkkiNeqHXy7BaKnBsEQtBKH1ttTr3yMM3aMDcfpiOcqrqlvtuERERNR6mCRSg6y2Wmzbn4eY8AB0jWp41lNLCYKAG/pHQy6TYH/GFY5PJCIi8kFMEqlBh7KLUWWpQXLvyDY5vkohw6A+kagw2rBl37k2eQ0iIiJqOSaJ1KADGVcQE66GNlR17cotFBOuRu+4YOxIu4gLhb65fhQREVFnxSSR3BSXm5FTUInRyV1adSxiQ4botAhUy/H5jlOwOxyQSATnHyIiIvIeJonk5kBmIQQANw6MafPXUsqleGCiDheKqrDrSAHSsoqw60g+0rKKmCgSERF5EZNEcuEQRRzILET/HmEID267W81XG5YYhcF9IrF57zkUlppgMNlgNNva5bWJiIioYUwSyUXOxQqUVlpxY1KXdnk9AYBUKsGDkxMAAfglt6RdXpeIiIiaxiSRXBzILIRKIcUQnbZdXk+jluPgySIcyynBEJ0WF4uMKDNY2+W1iYiIqHFMEsnJIYo4fqYUKX0joWzhHs0tYTTbYDDZMKBnOAKUUpzMKwdEkZNYiIiIvEjm7QDId1worILRUoPkPpFeSc7kMgkG9YlE2skiFFVYcPBkEapM1QhUKzCifzQcDu71TERE1F7Yk0hOGWf1EABYbXbsOpKP42dK23wJnN/qHReCILUcP2cVw2Cq5iQWIiIiL2GSSAAAiURA5vky9IwNht3ugMFkg9la0/5xCAISuofCYLIhv8jY7q9PREREdZgkEiQSAT8ev4SzlyoRFRbQ7r2Hv9UlXI1gjQIn88ogirzFTERE5A1MEgkAcKagEiKAmAi1t0OBIAhI7h0BfaUVZVXV3g6HiIioU2KSSACAy6UmyGUSRIUFeDsUAEBC91Ao5VKcvVTp7VCIiIg6JSaJBIco4nKpCdpQFSRevtVcTy6TIKF7KArLLKg0ceIKERFRe2OSSMgvNsJqsyMqzPu3mq+m6xYKiQDk5FdwvUQiIqJ2xiSRkHOxAgCgDWmfvZqbK0ApQ5cIDfKuVGHfiStIyypiokhERNROmCQSzl6uRIBSCpWi/XZZaa7u0YGorrEjO6+M6yUSERG1IyaJhHOXDYgM8f7SNw2JDFEhSC3HmQJOYCEiImpPTBI7OYPZhuJyCyJ97FZzPUEQkBgfhsIyM6rYk0hERNRumCR2cucvGwDAZ5NEoG45HAHA2UsGb4dCRETUaTBJ7OTOXjZAIgiICPbdJDEwQI5YrQZnL1fC7nB4OxwiIqJOgUliJ3f+ciW6Rmkgk/n2pdAnLgSWajsyzpZ5OxQiIqJOwbczA2pTDlHEuSsG9IoN8XYo19RVGwiVQoq96Ze5ZiIREVE7YJLYiRXqzbBU29E7LtjboVyTRCIgMT4Ux3JL8O/957lmIhERURtjktiJnb1ct6xMbz/oSQSAxPgwiCK4ZiIREVE7YJLYiZ2/bECAUoaYCN/ajq8xoYFKRIUF4GKREaIoejscIiKiDo1JYidWNx4xGDKp/1wGfbuGwGStRXG5xduhEBERdWj+kx1Qq3KIIgpKTBBFEcfPlPrkbisN6R4dBLlUgpz8Cm+HQkRE1KExSeykisstcDhEKOVSmK013g6n2WRSCbpHB+JisRH6Squ3wyEiIuqwmCR2UgUlRgBAsFru5Ug817NLMCACu47mezsUIiKiDotJYidVUGyEACDQD5NEtUqGbtGB2HP8Mqy2Wm+HQ0RE1CExSeykLpWYEKSWQyrxz0ugX3wYzNZa7M8o9HYoREREHZJ/Zgh03QpKjQgNVHo7jBaLDFGhV2wwdh3Jh4PL4RAREbU6JomdUHWNHcVlFoQG+W+SKAgCJg3vjqJyC47llHg7HCIiog6HSWIndEVvggggNFDh7VCuy7DEKESFBeDrA3lcXJuIiKiVMUnshC6VmAAAIX58uxmo28/5thHxuFhkxNFTxS7l9X+IiIioZZgkdkKXSkyQSSUICvC/mc31BNQlg6OSuyA8WIn1u3IgCIBMJkFaVhF2HclHWlYRE0UiIqIWknk7AGp/BSVGxEVq/DqB0qjlOHiyCFWmagzqE4kffrmEtf/JQVKfSJgsNTCYbN4OkYiIyK+xJ7ETulRqQpxW4+0wrpvRbIPBZEPvuGAEKGVIP6v3q91jiIiIfBmTxE7GZK1BeVU1umoDvR1Kq5FJJejXIwzF5RZUGqu9HQ4REVGHwCSxk6mftNI1yv97Eq/Wr2c4BAE4eb7c26EQERF1CEwSO5lLpXVJYlxkx+lJBAC1So7YCA1OXSxHTa3D2+EQERH5PSaJnUxRmRkKuQRhwf69/E1DenQJgq3GgfNXDN4OhYiIyO8xSexkCsvMiA5TQybteG99eJASESEqnL5YzsW1iYiIrlPHyxSoURKJgLzCKggCcPxMKQTBf5fAaYggCBjYMxwVRhvKOYGFiIjoujBJ7ERq7Q5UmW1QyaUddqmYXnHBkAgCLpeavR0KERGRX2OS2ImUVFggioDGj3dauRalXIrYSA0ul5p4y5mIiOg6MEnsRAr1db1rgR04SQSA+JggWG12lFZavR0KERGR3/J6kmgymfDhhx9i3rx5GD16NBISEvD44483Wn/Xrl246667kJycjJEjR+KFF15AWVlZg3VzcnIwd+5cDB06FEOGDMEf/vAHnDp1qsG6FRUVeOmllzBq1CgkJydjxowZ2LFjR6uco68oLKtLEjUBHXs3xq5RGkgE4EJhFSQSwfmHiIiIms/rSWJ5eTmWLl2KzMxMDBw4sMm627dvx1/+8hcAwPPPP4/7778fO3fuxP/7f/8PVqtrr1FeXh5SU1ORlZWFefPm4cknn0RBQQFSU1Nx5swZl7o2mw0PPfQQNm3ahDvuuAP/+7//i5CQEDz55JPYsmVLq56vNxWWmaGUS6GQSb0dSptSyKSICgtAfrERBzILsetIPtKyipgoEhERecDrXUpRUVHYu3cvoqOjAQAJCQkN1qupqcGiRYvQtWtXrF69GgEBAQCApKQkzJ07F2vWrMEf/vAHZ/333nsPZrMZa9euhU6nAwBMmTIFkydPxjvvvINPPvnEWXf9+vXIysrCyy+/jPvuuw8AcPfddyM1NRVvvvkmJk+eDJVK1Sbn354K9WYEazr2reZ6sZEa/JJTivOXK6GUd+ykmIiIqC14vSdRoVA4E8SmHDp0CCUlJZg5c6YzQQSAMWPGID4+Htu2bXOWmUwm7N69GyNGjHAmiACg1WoxZcoU7Nu3DxUVFc7ybdu2Qa1W46677nKWSSQSpKamory8HPv377/Os/QNhWVmBKsV3g6jXcSEqSGVCLhQWOXtUIiIiPyS15PE5srMzAQADBo0yO2x5ORknD59GjabDUDdWMSamhokJye71R00aBDsdjuys7MBAA6HA9nZ2ejXrx8UCoVbXQDIyMho1XPxBkt1LSpNNgRrOkeSKJNJ0C0qEPnFRs5yJiIiagG/SRKLi4sB1N2e/q2oqCjY7XaUlpY2qy4AFBUVAQAqKythtVqbVdef1U9aCVJ3jtvNQN0sZ5O1FgZzx1wTkoiIqC15fUxic1ksFgBw6+0DAKVS6VKnqbr1ZfV16ie8NHXc306KaY6IiECPn9McWm1Qi553Mr+y7vkRGihkUqhUctghoFbENX9WqWTNruutY5istdBolC7l0ZEa7Dl+GWVGG3Q9FG32nviDll43nQHbpnFsm4axXRrHtmmcP7aN3ySJ9Qlb/S3lq1VX123BVj+5pKm69WXNqfvb43pCrzfC4Wjd25xabRBKSlo2xi43Tw8BgEICmEzVsKplsFpqm/WzFGKz63rrGIDgVh6iliEiRIX8QgMG9Qpvk/fEH1zPddPRsW0ax7ZpGNulcWybxvly20gkQqOdKH5zuzkmJgbAf28lX624uBhSqRRarbZZdQE4J8uEhoZCqVQ2WbehW9H+prDMjMhQFaQSv3nLW0VXbSAqjDZYqmu9HQoREZFf8ZuMYcCAAQCA9PR0t8cyMjLQt29f5y1jnU4HuVyOEydOuNVNT0+HVCpFv379ANTNYu7fvz+ys7PdehPrXyspKalVz8UbisosiA5XezuMdtdVqwEAXCoxeTkSIiIi/+I3SeLw4cOh1WqxYcMGlzGCe/bsQV5eHqZNm+Ys02g0GDt2LNLS0pCbm+ssLykpwc6dOzFq1CiEhYU5y6dOnQqz2YyNGzc6yxwOB9auXYvQ0FCMGjWqjc+ubYmiiMJyM2I6YZIYFqREgEKKghKjt0MhIiLyKx6NSTx8+DDi4uIQGxvbaJ3CwkLk5+dj2LBhzT7u6tWrYTAYnL+fO3cOy5YtAwAMGzYMw4YNg0KhwHPPPYdnnnkG999/P+6++27o9XqsXLkSffr0wQMPPOByzKeffhoHDx7EQw89hDlz5kCpVGL16tWw2+1YuHChS91Zs2Zh48aNeP3111FQUID4+Hjs2LEDx44dwxtvvOGyLqM/MphrUG2zIzqs8yWJgiAgOlyN/GIjbDV2yKR+872IiIjIqzxKEmfPno2//OUvmD9/fqN1/v3vf2PJkiXOdQibY8WKFbh06ZLz99zcXPz9738HAMyfP9+ZcE6bNg0KhQIff/wxFi1aBI1Gg4kTJ2LBggVuiVyvXr2wZs0aLF68GMuWLYMoikhJScGSJUtcFtgG6iavfP7553j33XexadMmGI1G9OnTB0uWLMHUqVObfR6+qri8bvmb6PAAFJdbvBxN+4sJVyOvsApZF8qR3CvC2+EQERH5BY+SxOYsSuxwOCAInu2Ru3v37mbXnThxIiZOnNisuomJiVi+fHmz6oaFheG1117Da6+91uxY/EV9YhgVpu6USWJEiAoyqYDjuaVMEomIiJqp1e+9nT59GsHBwa19WLoOReUWSAQBkSH+v/90S0glArpEaJB+ppS7rxARETXTNXsSZ8+e7fL75s2bcejQIbd6DocDRUVFKCgoaHZPH7WP4nIzIkKUnXo8XjetBgdOFiG/xIjuUf63oCkREVF7u2aSeHVCKAgCLl265DJ+sJ5EIkFISAgmTZqEF154oXWjpOtSXG5BVCectHK1vt1DceBkEb7en4fH7krulItqExEReeKaSeKpU6ecPycmJmL+/PlNTlwh3yKKIorKLRgR27mHAAQoZdCGqnD+suHalYmIiMiziSurVq1CXFxcW8VCbcBkrYWluhbRof69jE9riNMG4nhuKcqrrAjRKL0dDhERkU/zaJDa8OHDmST6maJfl7/p7Lebgbot+gDgeG6plyMhIiLyfR71JAKAzWbDrl27kJGRgaqqKtjtdrc6giBg0aJFrRIgXZ/6JW+iw9mTGBqoQGCAHMdySzFmML/sEBERNcWjJDE/Px8PP/wwCgoKmlxKhEmi7ygut0BAXU+iROLZ+pUdjSAI6BqlQVZeGczWGqhVcm+HRERE5LM8ShJff/115OfnY/r06ZgxYwaio6MhlUrbKjZqBcUVFmgC5Nhz/BKiwtUeL3Te0fSICcKpCxU4eroENw1qfHtJIiKizs6jJPHQoUMYPXo03njjjbaKh1pZcbkZgQFyGEw2BKrZcxYRrEJUWAAOZRcxSSQiImqCRxNXJBKJ277H5NuKyywIYnLoJAgCbugXjawL5ag02bwdDhERkc/yeHZzdnZ2W8VCrcxsrUGVpYZJ4lUEACMHxkAUgaOni70dDhERkc/yKEl87rnnkJ2djc8//7yNwqHWVPTrzOYgtcLLkfgOjVqO/BITQgMV2HWkoNNP5iEiImqMR2MSP/roI/Tt2xdvvfUW1q5di4SEBAQGBrrV4+xm31C//E1QAHsSr2Y02xATrsapixXQV1oRFsSFtYmIiH7LoyRx8+bNzp8vXryIixcvNliPSaJvKP51Ie1AtRxma62Xo/EtcVoNTl2swMGThZg6It7b4RAREfkcj5LE77//vq3ioDZQXG5BWJASMqlHowo6BY1KDm2oCgczC3HbyHgIggCHo/G1P4mIiDobj5JEbsnnX4oqLIgK404rjenXIwx7j1/B+h/OID4mGCP6RzNRJCIi+hW7mDqw4nILorlnc6N6x4ZAIhFw6kI5jGYuh0NERHQ1j3oSt2zZ0uy6d955p4ehUGuyVNfCYLKxJ7EJSoUU3aICcanEBDt7EImIiFx4lCQ+99xz19zWTRRFCILAJNHLSirqZjZHhwVw0egm9IoNxoXCKlwuNXk7FCIiIp/iUZLY2HZ8RqMRWVlZ+PrrrzFu3DiMHz++VYKjlqtf/iYqXM0ksQmxERoo5BKcv2zwdihEREQ+xaMkcfr06U0+ft9992H27Nm49957rysoun5Fvy5/ExUagNz8Cu8G48MkEgFdIwORV1gFo6UGaqVHHwkiIqIOq1UnriQnJ2P8+PH44IMPWvOw1ALF5RYEaxQIYNJzTd2iNHCIIo6cKoZEInAXFiIiInjYk9gc0dHR2L17d2sfljxUXM7lb5orWKNAeLASO36+iFq7A4FqBZfDISKiTq9VexJramqwf/9+aDSa1jwstUBxhQXRoUwSm0MQBOi6haKozIzLpSYuh0NERIRWWgLHbrejqKgIO3bswJkzZzgm0cuqa+wor6pGVDjXSGyuvl1D8PPJIhQUGxEbyS85RERErbIEjijW3ZYTBAETJ07Es88+2zrRUYtcvfwNNY8mQI4uEWrklxgxXIxyGZfI285ERNQZtcoSOIIgIDg4GAMGDEB0dHSrBEYt51z+hkmiR3rFhuCnjCswWGpw8GQRqkzVHJ9IRESdVqsugUO+wZkkckyiR7pFB0J6UkBOfgVCNEoYuL4kERF1Yty7uQMqLjcjMEAOtUru7VD8ikwqQZcINc5dMsDucHg7HCIiIq9q0RI4W7duxZYtW5CdnQ2j0YjAwED0798fd9xxB+64447WjpE8VFRu4XjEFuqqDURBiQmXSkwI0Si8HQ4REZHXeJQk2u12PPHEE/j+++8hiiKCgoLQrVs36PV6HDhwAAcPHsS3336LpUuXQiqVtlXMdA3F5RbouoVwYegWiAxVIUApxfkrBgzuE+ntcIiIiLzGo9vNX3zxBXbt2oUbb7wRW7ZsweHDh7Fjxw4cOnQIW7ZswahRo/DDDz/g888/b6Nw6Vpqah0oM1gRFaZGWlYRjp8pbXBGOjVMIgjoHReCghITamp5y5mIiDovj5LEzZs3o2/fvvj000+RmJjo8lhiYiL+8Y9/oE+fPo2up0htT2+wQkTdzGaj2QaztcbbIfmdvl1D4HCIuKI3eTsUIiIir/EoSbx48SLGjBkDiaThp0kkEowZMwYXL15sleDIc/VrJHJmc8tFhQUgKECOghImiURE1Hl5lCQqlUoYDIYm6xgMBiiVyusKilquPknUcuJKiwmCgB5dglFaaYW5utbb4RAREXmFR0liUlIStm/fjvPnzzf4+MWLF7F9+3YkJye3SnDkuZIKC+QyCWfmXqeescEAgAuFVV6OhIiIyDs8mt08b948zJkzB3fddRdmzZqF4cOHIyIiAnq9HocPH8b69ethsVjw6KOPtlW8dA0lFVZEhqg4WeU6hWgUCNEocP5K0z3nREREHZVHSeKwYcPwzjvv4KWXXsLKlStdZjGLoojAwEC8/fbbGDZsWGvHSc1UWmlBVFgAl75pBV21GpzMK0eh3swtDomIqNPxeDHtqVOn4uabb8auXbtw6tQp52LaiYmJuPXWWxEYGNgWcVIzCAJQqDcjQCnj0jetIC6yLkk8eLIQd4zu6e1wiIiI2lWzksQffvgBlZWVuO222yCXyxEYGIg777zTpY7NZsOOHTsQEhKCsWPHtkGodC0mSy1stQ7IpAKXvmkFKqUMMeFqpJ0sxO9H9WDSTUREnco1J65kZWXhL3/5C7KysiCXN74XsEKhQHZ2Nv785z/j1KlTrRokNU/xrzOb1Uru2dxaenQJQlG5BeevcAILERF1LtdMEjds2AClUon58+df82B//vOfoVKp8OWXX7ZKcOSZ0l+TRI2qRVtyUwO6RwVCJpXgQOYVb4dCRETUrq6ZJP78888YOXIkgoODr3mw4OBg3HjjjTh06FCrBEeecfYkMklsNQq5FMMSo3DwZCGsNq6ZSEREncc1k8QrV66gV69ezT5gfHw8Ll++fF1BUcuUVFigUkghk3q0/CU1QQBw67CusFTb8XNWkbfDISIiajfXzCYcDgdEUWyPWOg6lVRYEBjA8YitSaOWo6TCirAgJbYduADOXSEios7imkmiVqvF2bNnm33As2fPIjIy8rqCopZhktg2TJYadI8KhN5gRW5BpbfDISIiahfXTBKHDBmCgwcPoqjo2rfaioqKcODAAfzud79rleCo+ewOB/SV1UwS20icVgO5TILdRwu8HQoREVG7uGaSeO+996K6uhpPPPEEKisb70WprKzEE088gZqaGsyaNatVg6RrKzNUwyGKCFQzSWwLMqkEvWODcfhUMSpNNm+HQ0RE1OauOQ12yJAhuPfee/Hll19i6tSpuPfeezF8+HBER0cDqOs9/Pnnn7F+/XqUlpYiNTUVKSkpbR44uSr5dWYzexLbTkqCFqcvVuDzHdl46p7BcDg4VpeIiDquZq2V8tJLLyEgIABffPEFli1bhmXLlrk8LooiJBIJHn74YTzzzDNtEig17eokkclL2wjRKNCjSxBOni+DyVoDjaouIWd7ExFRR9SsJFEikeDZZ5/FrFmzsHnzZhw7dgylpaUAgMjISKSkpGD69Ono0aNHW8ZKTSipsEIqEaBWyWA0c0u+tjKwVwTOX6nCim9OoV98KALVCozoH81EkYiIOhyPVl3u0aMHnnrqqbaKha5DaaUF4cEqSLhGS5sKDVSiZ5cgZJwtRVSoytvhEBERtRmuutxB6A1WRIYwaWkPQxK0sNU6kFdo8HYoREREbYZJYgehr7Qigkliu9CGBiAuUoMzlwyw1di9HQ4REVGbYJLYAdTaHag02hARzCSxvQzuG4maWgdO5pV7OxQiIqI24dGYRG8rKCjALbfc0uBjo0ePxmeffeZStmvXLnz00UfIzc2FRqPBuHHjsGDBAoSHh7s9PycnB4sXL8bRo0chiiJSUlKwcOFCJCYmtsm5tKYygxUigMgQFarZs9UuwoNViIvU4NSFcpRXVSNEo/B2SERERK3Kr5LEemPGjMFtt93mUhYVFeXy+/bt2/HUU09h4MCBeP7556HX67FixQpkZGRgw4YNUKn+2+uWl5eH1NRUqFQqzJs3D0qlEmvWrEFqairWr1+PPn36tMt5tZS+0goAiAhR4XKpycvRdB6J8aG4ojdh60/nMXtSgrfDISIialV+mST27t0bd9xxR6OP19TUYNGiRejatStWr16NgIAAAEBSUhLmzp2LNWvW4A9/+IOz/nvvvQez2Yy1a9dCp9MBAKZMmYLJkyfjnXfewSeffNK2J3SdSg11SWIkk8R2pVHJ0bdrKPYev4yJw7ohJlzt7ZCIiIhajd+OSayurobFYmnwsUOHDqGkpAQzZ850JohAXQ9kfHw8tm3b5iwzmUzYvXs3RowY4UwQAUCr1WLKlCnYt28fKioq2uw8WkOZoRoC6m6BUvsa2CsccpkEm/ae83YoRERErcovk8Qvv/wSgwYNwuDBgzFu3Dh89NFHqK2tdT6emZkJABg0aJDbc5OTk3H69GnYbHX77+bk5KCmpgbJycludQcNGgS73Y7s7Ow2OpPWoa+0IiRQAZnUL99Ov6ZWyjD5hu44cqoYF4qqvB0OERFRq/Gr280SiQQjRozAhAkT0KVLF+j1emzduhXvv/8+srOz8cEHHwAAiouLAbiPU6wvs9vtKC0tRWxs7DXrAnX7U/syvYHL33iLRi2HQiGDUi7F8m1ZeP2REdx9hYiIOgS/ShJjY2PxxRdfuJTNnDkT8+fPx7fffou0tDSMGDHCeRtaoXCfcapUKgHAWaepuvVljd3WbkpERKDHz2kOrTbIrazCaEPfbqGIiAhEgFoBOwSoVPJf/5bBDgG1Ipxlnv7sD8cwWWuh0Si9E4e1Fv17hePY6RLk680YrHP/wuFtDV03VIdt0zi2TcPYLo1j2zTOH9vGr5LEhgiCgEcffRS7du3C/v37MWLECGciWH9L+WrV1dUA4Jzd3FTd+rKrZ0I3l15vbPUeJa02CCUlv7mlKQBF5WaEBynww+ELsFpqYDJVw6qWwWqphRQirJZalzJPf/aHYwCCV+OIDVcjN0COv687hknDuiFIo/SZPZ0bvG4IANumKWybhrFdGse2aZwvt41EIjTasdUhBrHFxcUBgHOCSUxMDID/3na+WnFxMaRSKbRabbPqAkB0dHSrx9xaKo02OBwiJBIBZmuNt8PptKQSAcP7R6G0worsC+Uwmt2/dBAREfmTDpEkXrhwAQAQEREBABgwYAAAID093a1uRkYG+vbt67yVrNPpIJfLceLECbe66enpkEql6NevX1uFft3q10hUK/2+U9jv9e0agrAgJU5dqIDd4fB2OERERNfFr5LE8nL3LdBqa2vx4YcfAgDGjRsHABg+fDi0Wi02bNgAq9XqrLtnzx7k5eVh2rRpzjKNRoOxY8ciLS0Nubm5zvKSkhLs3LkTo0aNQlhYWFud0nUrrawbLxnAJNHrBEHAEJ0W5upa5ORXejscIiKi6+JXmcWLL74Is9mMwYMHIyYmBnq9Ht988w1yc3ORmprqXPJGoVDgueeewzPPPIP7778fd999N/R6PVauXIk+ffrggQcecDnu008/jYMHD+Khhx7CnDlzoFQqsXr1atjtdixcuNAbp9psegN7En1JbKQG2lAVMs7pYbLWIEDB94WIiPyTX/0PNmbMGGzduhVffvklDAYDlEolEhIS8NZbb+HOO+90qTtt2jQoFAp8/PHHWLRoETQaDSZOnIgFCxa4LLANAL169cKaNWuwePFiLFu2zLl385IlS1wW2PZF+korFDIJZDK/6hTu0PrHh2FP+hV8dygf02/uBQA+MYGFiIjIE36VJM6cORMzZ85sdv2JEydi4sSJzaqbmJiI5cuXtzQ0rymttEITIPd2GHSVkEAlesUGY3vaBSjkEkSEBPjMTGciIqLmYveTn9NXWqFR+VWu3yn8LlGLmloHjueWcqYzERH5JSaJfkwURegN7En0ReHBKsTHBOHcZQOqbXZIJILzDxERkT9gF5QfM1fXwmqzQ6NikuiLBvWOwIXCKuReqsTBk0WoMlUjUK3grWciIvILTBL9WP0aiYEBfBt9UUigEnFaDTLP6dEnLgS2Gru3QyIiImo23m72Y/VJInsSfVdCt1DY7SKy8sq8HQoREZFHmCT6sdJf10jUsCfRZwUGyNG3WwhOX6yA1Vbr7XCIiIiajUmiH6tfI1Epl3o7FGrC0AQtHKKIMwUGb4dCRETUbEwS/ZjeYEV4sAqCwBmzvqx+3cS8oiqYrexNJCIi/8Ak0Y/pK62IDFF5OwxqhqReERBFESc5NpGIiPwEk0Q/pjdYEcEk0S8EqRXoFhWI3IJKlFdZvR0OERHRNTFJ9FPVNXZUmWuYJPoRXdcQiKKIbQcueDsUIiKia2KS6KfKfp3ZzNvN/kOtkqN3bAj2HL/kfP+IiIh8FZNEP1W/RmJEMJNEfzKwVzhEEfjmIHsTiYjItzFJ9FN6Z09igJcjIU8EBchx8+BY7E2/jNJKK/dzJiIin8Uk0U/pDVZIBAGhQQpvh0Ie0Kjl6BYVBEEA/vH1Sew6ko+0rCImikRE5HOYJPopfaUVYUFKSCV8C/2NKIpI7h2BMwWVuFBYBaPZ5u2QiIiI3DDD8FP6Si5/488G942EUiFFVl4ZRFH0djhERERumCT6Kb3Bykkrfkwhl2JQ7wjoDdW4VGLydjhERERumCT6IbvDgfIqGyJClN4Oha5D366hCAyQ4cjpElTX2L0dDhERkQsmiX6ovKoaDlFkT6Kfk0gEJPeOhNFSg3//dN7b4RAREblgkuiHnGskckyi34sMUaF3bDB2/nwRBcVGb4dDRETkxCTRD9WvkciexI5hiE4LtVKGL749BYeDk1iIiMg3MEn0Q9xtpWNRKaS4f6IOZy8Z8E0ad2IhIiLfwCTRD+kNVgSr5VDIpd4OhVqBRi0HBAE9uwRhy95zyMmv8HZIRERETBL9UXiwCkm9IrwdBrUik6UG/bqHIVAtx8dbT8LABbaJiMjLmCT6od+P6ok/TOvv7TColclkEtyU3AVGSw0+2XoStXaHt0MiIqJOjEkikQ+JCFbhoamJyL5QjtXf5UAQ6pbK4d7ORETU3mTeDoCI/kujlkMQJBjYMxx70y/DVutAn7hgBKoVGNE/mrOfiYio3TBJJPIxRrMNPbsEwWqrRdrJQtTW2pEYH+btsIiIqJNhkkjkgwRBwNghcag02fBLbikiuXA6ERG1M45JJPJRMqkEY1PioFJI8ePxyyipsHg7JCIi6kSYJBL5MJVChhv6RcHhELFkfTos1bWcxEJERO2CSaKfqp/xyoSh4wtSKzDphu4o1JvxyueHsT/jCt93IiJqc0wS/ZBEIiAtqwi7juTj+JlSCAITho4uTqvBjQNjUFRuwX8O5zuXxiEiImorTBL9lNFsg8Fkg9la4+1QqJ30jA1GYvdQ5BVW4b116Th4spCJIhERtRnObibyI327hkAmkyD9TCkAESMHxHg7JCIi6qDYk0jkRwRBwKikGPSKDUb6GT2+PXTR2yEREVEHxZ5EIj8jCAJGDohBTa0D/9qVC0t1LW6/sQfHphIRUatikkjkhyQSATcld0FeYRW27DsPo7kG997aFxImikRE1EqYJBL5KalEwNw7BiAwQI7vDuejpNKKubf3hyZAzj2eiYjounFMIpGf0qjl+DmrGNpQFcYNiUPGWT2e/fggvj6Qx1nPRER03diTSOTHjGYbqsw1SOgeCqVcih+PXcLmPWchAJgyojukEn4PJCKiluH/IEQdhDY0AGMGx6JnbDC++vEsFi47gG0H8rwdFhER+Sn2JBJ1IEq5FBOGdUNcZDnSsgrx1Y9nUWmuwW0juiM0UMmxikRE1GxMEok6oPiYIGhUUlzWm/H9kXzsPpKPfvFhmD05AZEhAQD+u60fE0ciImoIk0SiDkouk2LkwBgk9AjHwRNXkJVXhmc/PoghOi2mjIhHaaUVADCifzQTRSIicsMkkaiDCwlUIqVvJIYnRsFW68D3vxTg6OkSaENV6N8jHMP7RXk7RCIi8kFMEok6CW14AARBgttv7IHiCgsOZRdjz/HLOFNQiSkj4jFiQDQX4yYiIicmiUSdiNFsg6W6FgndQxEboUFhmQnHcvVYvi0L//o+B3eM7oUxg2KhVEh5C5qIqJNjkkjUSUkkAvp2C0WXCA1yCypwsciItf/JwcYfzyC5TyQevq0fAhR1/0QwYSQi6nyYJBJ1coIgIDpMjRSdFnlXqnAspwSHs4uRnluKvt1C8bvEKIxLiWOiSETUyTBJJCInbWgAbugfDYVcgsPZJTh5vgynLpTj/GUDxqbEokdMsMuWf0wciYg6LiaJROQmMjQANw+OxaViIy4WVyEtqxB70y+jZ5dgxMcEISY8AKFBKi6fQ0TUgTFJJKJGBarlGDkgBs/c1wU/pV/BD8cu4cdjlyCXStA9JhBqpRSJ3cMgk3KHTyKijoZJIhE1SaOW48TZMggCMOuWPjh5rgwn88pwsdCI99alQyGXoFeXYPSOC0FspAZRYQGIDAlAsEYOAVxSh4jIXzFJJKJrMpptMJhsCFTLER2uhkohRXR4AApKzMgvroLJUoudP1+E/Te3noM1CoT8+ic0SIkuEWrERmoQHx2E8GAVAI5rJCLyVUwSf2W32/HZZ59hw4YNuHLlCmJiYjBjxgzMnTsXMhmbiei3pFIJIkNUiI8JhCBIUGGwQKGQIb+oCiUVFijkUliq7TCYqlFaacW5KwaYrbXO52tUMnSJ1GBUUhf0iQtGl3CNy6QYIiLyLmY/v3r11Vfxr3/9C1OmTMHcuXORkZGBDz74AAUFBVi0aJG3wyPyaUazDSZrLUKClIjTBiIwQI5YrQYmSy0qjdXOn0srzFAqZCgoMeFSiRGXS034YscpAIBaKUPvuBD07RqCmHA1tKEBiAhRQa2UMXkkIvICJokATp8+jS+//BJTp07FkiVLAAAzZ85EUFAQli9fjnvvvRfJyclejpLI/8llUsREqBGkViA2Qo0ukWoUlVlgMNlQWGbBxaIqZJzTuz1PpZBCo5JDrZIhQCEFfrt9oCjCIQIOUYRUIkCpkEKlkCE8SImIENV//w5WITBAzu0HiYiagUkigO3bt0MURTz44IMu5bNnz8by5cvxzTffMEkkagOCICBIrUBMhBpx2kD0iQtGZKgKBnMtivQmSKUSGC01MJptkEgEmKy1qDTZoJBLYXeIsNsdzp9lUgGiCJhr7aiyCLBW22G02FBrdx3zKJUICA9WITxYiVhtINQKKQKUMijlUijkUijkEihkUijlEsikEufvCrkEUonkqtgBAXU71wi/Jp2iKEIQBEgEQCqRQCoVIL3qcSIif8IkEUBmZiYkEgkGDhzoUh4dHY3o6GhkZGR4fMy2uj0mkQiQSARow9RQq+QID1ZBrXIgQClr8GeVQtLk48352R+O4YAAwSF6PQ5fPEZTbeOL5xIZqkJokAPhQUpEhqpgtTlgNNuu+bNKIWng8WoEaRTQG6pRVmGBRCr5NdGshrW6Flf0JlQYquEQ23byjFQiQCIVIBXqE0cJpBJAKhUgk0ggkQj/TSqFuvGeUqkAmSCpe57k14y0BVr6L5FSKYfNVgsBdck8fg1BEIRf/4Yz+RV+LRAACBCcHb2Sq5539d8uxCZ/dddABbcit2M2fFTn7PsGYnOJGf89V7VKAWu1Db+p+Zuf3AsEt4M3EI+H9V000nCNnTsAtPZlHxAgh8VSA8C9s99tpYOmf21UkyG3Vxs08ZzGXksVIIfVXOPp4RAWpMQQndaD4DzXVL7CJBFAcXExwsLCoFAo3B6LiopCUVGRx8cMC9O0RmhuIiICAQDjh7fN8YmIiIgAgCvgArBYLA0miACgVCphtVrbOSIiIiIi72KSiLpE0GazNfhYdXU1lEplO0dERERE5F1MEgHExMSgvLy8wUSxuLgY0dHRXoiKiIiIyHuYJAIYMGAAHA4HMjMzXcqLiopQVFTkNqGFiIiIqKNjkghg6tSpEAQB//znP13KV61aBQCYNm2aN8IiIiIi8hrObgaQmJiIe+65B+vWrYMoihg1ahQyMzOxbt06TJ8+HYMGDfJ2iERERETtShDFNl4gzE/U1tY6924uLCxEVFQUZsyYgUcffRRyudzb4RERERG1KyaJREREROSGYxKJiIiIyA2TRCIiIiJyw4krPs5utzvHSl65cgUxMTGYMWMG5s6dC5ms4799J0+exNdff420tDQUFBSgpqYGvXv3xsyZM3Hvvfc691MFgPHjx+PSpUtux1AoFC3af9vXFRQU4JZbbmnwsdGjR+Ozzz5zKdu1axc++ugj5ObmQqPRYNy4cViwYAHCw8PbI9x2tXTpUnz44YeNPn7jjTdi5cqVADr2dWMymbBy5UpkZmYiMzMTJSUlmDRpEj744IMG63tyjeTk5GDx4sU4evQoRFFESkoKFi5ciMTExLY+revW3HYpKirC5s2bsXfvXly4cAGVlZXo0qULJk6ciD/96U8IDAx0qf/cc89h8+bNDb7munXrMHjw4LY6pVbjyTXj6WfHn68ZoPlt8/PPP2P27NmNHkcmk+HkyZPO3335uun4WYafe/XVV/Gvf/0LU6ZMwdy5c5GRkYEPPvgABQUFWLRokbfDa3PLly/HgQMHMGHCBMyaNQs2mw07duzAyy+/jKysLLz66qsu9bt27YrHH3/cpUwqlbZnyO1uzJgxuO2221zKoqKiXH7fvn07nnrqKQwcOBDPP/889Ho9VqxYgYyMDGzYsAEqlao9Q25zEyZMQPfu3d3Kf/zxR2zfvh1jxoxxKe+o1015eTmWLl0KrVaLgQMH4ocffmi0rifXSF5eHlJTU6FSqTBv3jwolUqsWbMGqampWL9+Pfr06dMep9dizW2X3bt348MPP8SYMWMwceJEqFQqHD58GJ999hn27duH9evXN/jZee2119y2em3oevRFnlwzQPM/O/5+zQDNb5vevXvj7bffdivPz8/H0qVL3f79qeeT141IPuvUqVNiQkKC+OSTT7qUv/3226JOpxPT09O9FFn7OXLkiGi1Wl3K7Ha7+MADD4g6nU7Myclxlo8bN06cPn16e4foNfn5+aJOpxPffPPNJuvZbDZx1KhR4vjx40Wz2ews//HHH0WdTicuX768rUP1GbNmzRIHDBgg6vV6Z1lHvm6qq6vFwsJC5+86nU587LHH3Op5eo089thjYr9+/cTTp087y4qLi8UhQ4aIc+fObYMzaV3NbZecnByxuLjYrfz9998XdTqduHr1apfyZ599VtTpdKLRaGz9oNtJc9tGFD377Pj7NSOKnrVNQ959911Rp9OJ//nPf1zKffm64ZhEH7Z9+3aIoogHH3zQpby+G/ubb77xRljtaujQoW57Z0skEkycOBFA3e2L36qtrYXRaGyX+HxFdXU1LBZLg48dOnQIJSUlmDlzJgICApzlY8aMQXx8PLZt29ZeYXrV+fPncezYMYwZM6bB26cd8bpRKBTN2lbUk2vEZDJh9+7dGDFiBHQ6nbNcq9ViypQp2LdvHyoqKlr1PFpbc9ulb9++0Gq1buWTJ08GAJw+fbrB54miCKPRCIfDcX2BekFz2+Zq1/rsdIRrBmhZ29RzOBzYunUrwsPDG+1J9MXrhkmiD8vMzIREInHbFjA6OhrR0dF+P17qehQXFwOA23/2OTk5GDx4MIYOHYphw4bhhRdeQFlZmTdCbDdffvklBg0ahMGDB2PcuHH46KOPUFtb63y8frvJhhaFT05OxunTpxvct7yj2bRpEwBgxowZbo91xuvmap5cIzk5OaipqUFycrJb3UGDBsFutyM7O7ttA/ayxv79qXfzzTdj6NChGDRoEP74xz8iKyurPcNrV8357PCaAfbv34/CwkL8/ve/b3TtZV+8bjgm0YcVFxcjLCzMbYwCUDfmrKioyAtReV9paSnWrVuHuLg4DB061Fnet29fzJw5E71790Z1dTX279+PTZs24ciRI/jqq68QHBzsxahbn0QiwYgRIzBhwgR06dIFer0eW7duxfvvv4/s7GznYOr6/9B+O06xvsxut6O0tBSxsbHtGn97cjgc+Pe//42IiAi3b/Gd7bppiCfXyLXqAujQ/zY5HA589NFHkEqlblu2RkZGYs6cORg4cCBUKhUyMzOxatUq3HfffVi9ejWSkpK8FHXbaO5np7NfMwCcE1Ma+pLqy9cNk0QfZrFYGkwQAUCpVMJqtbZzRN5ns9nw+OOPo6qqCu+//75L+3zyyScudW+//XYkJibijTfewOeff+42uNrfxcbG4osvvnApmzlzJubPn49vv/0WaWlpGDFihPM2dEPXUv2t/MZuVXcU9d/iH374YbdVATrbddMQT66RpurWl3Xk6+mdd97B0aNH8ec//9ltssWCBQtcfp8wYQJuvfVWzJo1C2+99RZWr17dnqG2ueZ+djr7NVNVVYVdu3ZhwIABSEhIcHvcl68b3m72YUqlstHbgNXV1W5j9Tq62tpaPPHEE/jll1/wt7/9DTfeeOM1nzN79myo1Wrs37+/HSL0PkEQ8OijjwKA85zrr5OGrqXq6moA6HCzm3+r/lv89OnTm1W/s103nlwjTdWtL+uo19Py5cuxYsUK3HHHHc3+8pCUlIRRo0bh6NGjHToRqtfQZ6czXzNA3fyB6urqBnsRG+Mr1w2TRB8WExOD8vLyBj9YxcXFLR5A64/sdjueeeYZ7N69G3/9619xzz33NOt5EokEXbp08YtB0a0lLi4OAJznHBMTA+C/t3yuVlxcDKlU2uDg/I6i/lv8wIEDXQbNN6WzXTeeXCPXqgugQ/7btGrVKrzzzjuYMmUK3njjDZc1Wq8lLi4ODocDBoOhDSP0DQ19djrrNVNv8+bNUCgUbsMTrsUXrhsmiT5swIABcDgczkHl9YqKilBUVOQ2oaWjcjgc+J//+R/s3LkTzz77rNts76bU1tbi8uXLiIiIaMMIfcuFCxcAwHnOAwYMAACkp6e71c3IyEDfvn0bHdbQEWzbts3jb/Gd7brx5BrR6XSQy+U4ceKEW9309HRIpVL069evbQNuZ2vXrsXrr7+OCRMmYPHixR6voXnx4kXIZDKEhoa2TYA+pKHPTme8ZuqdO3cOx48fx/jx4z1+/33humGS6MOmTp0KQRDwz3/+06V81apVAODxtxJ/5HA48Pzzz2Pbtm14+umn8fDDDzdYr7Een48//hgWiwVjx45tuyC9pLy83K2strbWudPIuHHjAADDhw+HVqvFhg0bXMax7tmzB3l5eR3+OmrqW3xnvG4a4sk1otFoMHbsWKSlpSE3N9dZXlJSgp07d2LUqFEICwtr1/jb0oYNG/DKK69g3LhxWLJkSaM7XZnN5gbv+qSlpeHAgQMYOXJkhxoi5Mlnp7NdM1dralUFwPevG05c8WGJiYm45557sG7dOoiiiFGjRiEzMxPr1q3D9OnTG1yuoqN5++23sWXLFiQlJSEmJgZbt251eXzIkCHo1q0btmzZgo0bN+Kmm25C165dYbPZ8NNPP2Hfvn0YOHCgR72P/uLFF1+E2WzG4MGDERMTA71ej2+++Qa5ublITU11Xh8KhQLPPfccnnnmGdx///24++67odfrsXLlSvTp0wcPPPCAl8+k7Zw9exbp6emYOnUqQkJC3B7vDNfN6tWrXW5XnTt3DsuWLQMADBs2DMOGDfP4Gnn66adx8OBBPPTQQ5gzZw6USiVWr14Nu92OhQsXtuv5tVRz2uX777/Hiy++iJCQENxyyy3Yvn27yzG6d++OlJQUAHU9+I888ghuvfVW9OjRAwqFAidPnsSWLVsQEhKC559/vv1O7jo1p208/ex0hGsGaF7b1KtfVSEqKgqjR49u8Hi+ft0IoiiKXo2AmlRbW+vcu7mwsBBRUVGYMWMGHn300UbXWupIHnzwQRw6dKjRx9944w3MmDEDR48exaeffopTp0451+iKj4/HpEmT8Mc//rFDDoresGEDtm7dinPnzsFgMECpVCIhIQH33HMP7rzzTrf63333HT7++GO3fXk78i3VxYsX49NPP8Wnn36Km2++2e3xznDdNLa/LgDMnz8fjz32mPN3T66RU6dOYfHixfjll1+c+/AuWLAA/fv3b7NzaU3NaZdr7QE+ffp0vPnmmwDqesXeeustZGRkoLi4GDU1Nc7kYN68eX61xFRz2qYlnx1/v2YAzz5Pe/fuxSOPPIJHHnnEbQZzPV+/bpgkEhEREZEbjkkkIiIiIjdMEomIiIjIDZNEIiIiInLDJJGIiIiI3DBJJCIiIiI3TBKJiIiIyA2TRCIiIiJywySRiIiIiNwwSSQiIiIiN0wSiYiIiMjN/wf/aQ+BeviQ3QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 720x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,4))\n",
    "sns.histplot([len(X_train_ids[i]) for i in range(len(X_train_ids))], binwidth=1, kde=True)\n",
    "plt.title('Sentence lengths')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "667abec1",
   "metadata": {
    "papermill": {
     "duration": 0.010879,
     "end_time": "2023-02-20T21:44:14.192789",
     "exception": false,
     "start_time": "2023-02-20T21:44:14.181910",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Since most sentences have less than 75 words, we'll **truncate** longer sentences to only include the first 75 words. A more sophisticated approach would be to truncate on a batch-by-batch basis to reduce the number of unnecessary 0's but this will still work well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5988269d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:14.215758Z",
     "iopub.status.busy": "2023-02-20T21:44:14.215448Z",
     "iopub.status.idle": "2023-02-20T21:44:14.666998Z",
     "shell.execute_reply": "2023-02-20T21:44:14.665897Z"
    },
    "papermill": {
     "duration": 0.465906,
     "end_time": "2023-02-20T21:44:14.669444",
     "exception": false,
     "start_time": "2023-02-20T21:44:14.203538",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[  10 6746   18 5574  491   25 1237  582    5 5575  393    2  101  375\n",
      "   27  441 5931  582    3 5574   14 9422  393    8  101  319    4    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "    0    0    0    0    0]\n"
     ]
    }
   ],
   "source": [
    "# Maximum length of sequences\n",
    "max_length = 75\n",
    "\n",
    "# Pad sequences\n",
    "X_train_pad = keras.preprocessing.sequence.pad_sequences(X_train_ids, padding='post', maxlen=max_length)\n",
    "X_valid_pad = keras.preprocessing.sequence.pad_sequences(X_valid_ids, padding='post', maxlen=max_length)\n",
    "X_test_pad = keras.preprocessing.sequence.pad_sequences(X_test_ids, padding='post', maxlen=max_length)\n",
    "\n",
    "y_train_pad = keras.preprocessing.sequence.pad_sequences(y_train_ids, padding='post', maxlen=max_length)\n",
    "y_valid_pad = keras.preprocessing.sequence.pad_sequences(y_valid_ids, padding='post', maxlen=max_length)\n",
    "y_test_pad = keras.preprocessing.sequence.pad_sequences(y_test_ids, padding='post', maxlen=max_length)\n",
    "\n",
    "# Example\n",
    "print(X_train_pad[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e5573ec",
   "metadata": {
    "papermill": {
     "duration": 0.010948,
     "end_time": "2023-02-20T21:44:14.691566",
     "exception": false,
     "start_time": "2023-02-20T21:44:14.680618",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Build model**\n",
    "\n",
    "<br>\n",
    "\n",
    "Our model will consist of an **embedding layer** followed by a **bidirectional LSTM**. \n",
    "\n",
    "Note that `mask_zero = True` tells the embedding layer to **ignore padding values** so the model won't make predictions at these time steps. This will make it easier to **properly evaluate** our model as we only care about its performance on non-padded values.\n",
    "\n",
    "Furthermore, since this is a sequence labelling task we want our model to make a **prediction at each time step**. Setting `return_sequences = True` accomplishes that. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "50bc8642",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:14.715683Z",
     "iopub.status.busy": "2023-02-20T21:44:14.714220Z",
     "iopub.status.idle": "2023-02-20T21:44:14.719426Z",
     "shell.execute_reply": "2023-02-20T21:44:14.718566Z"
    },
    "papermill": {
     "duration": 0.019015,
     "end_time": "2023-02-20T21:44:14.721361",
     "exception": false,
     "start_time": "2023-02-20T21:44:14.702346",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Input/output size (+ 1 for padding token)\n",
    "num_tokens = len(word_tokenizer.word_index) + 1\n",
    "num_classes = len(tag_tokenizer.word_index) + 1\n",
    "\n",
    "# Embedding dimension\n",
    "embedding_dim = 128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "1274ad74",
   "metadata": {
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:14.745264Z",
     "iopub.status.busy": "2023-02-20T21:44:14.744474Z",
     "iopub.status.idle": "2023-02-20T21:44:19.622316Z",
     "shell.execute_reply": "2023-02-20T21:44:19.620790Z"
    },
    "papermill": {
     "duration": 4.89249,
     "end_time": "2023-02-20T21:44:19.625113",
     "exception": false,
     "start_time": "2023-02-20T21:44:14.732623",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-02-20 21:44:14.878145: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:14.997184: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:14.998463: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:15.002001: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-02-20 21:44:15.002504: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:15.003702: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:15.004793: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:17.413636: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:17.414545: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:17.415216: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2023-02-20 21:44:17.415783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 15401 MB memory:  -> device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 75, 128)           6000896   \n",
      "_________________________________________________________________\n",
      "bidirectional (Bidirectional (None, 75, 256)           263168    \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 75, 13)            3341      \n",
      "=================================================================\n",
      "Total params: 6,267,405\n",
      "Trainable params: 6,267,405\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Build model\n",
    "model = keras.Sequential([\n",
    "\n",
    "    # Map integer ids to trainable word vectors\n",
    "    layers.Embedding(input_dim = num_tokens, \n",
    "                           output_dim = embedding_dim, \n",
    "                           input_length = max_length,\n",
    "                           mask_zero = True),\n",
    "    \n",
    "    # Bidirectional LSTM\n",
    "    layers.Bidirectional(layers.LSTM(128, return_sequences = True, \n",
    "                           kernel_initializer = tf.keras.initializers.random_normal(seed=0))),\n",
    "    \n",
    "    # Output layer\n",
    "    layers.Dense(num_classes, activation='softmax', \n",
    "                           kernel_initializer=tf.keras.initializers.random_normal(seed=0))\n",
    "])\n",
    "\n",
    "# Define optimizer, loss function and accuracy metric\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['sparse_categorical_accuracy'])\n",
    "\n",
    "# Model summary\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35e5e385",
   "metadata": {
    "papermill": {
     "duration": 0.01108,
     "end_time": "2023-02-20T21:44:19.648370",
     "exception": false,
     "start_time": "2023-02-20T21:44:19.637290",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Train model**\n",
    "\n",
    "<br>\n",
    "\n",
    "We'll use **early stopping** to prevent overfitting to the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6a1f9ebe",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:44:19.672129Z",
     "iopub.status.busy": "2023-02-20T21:44:19.671812Z",
     "iopub.status.idle": "2023-02-20T21:45:29.213318Z",
     "shell.execute_reply": "2023-02-20T21:45:29.210279Z"
    },
    "papermill": {
     "duration": 69.556226,
     "end_time": "2023-02-20T21:45:29.215812",
     "exception": false,
     "start_time": "2023-02-20T21:44:19.659586",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-02-20 21:44:19.737669: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n",
      "2023-02-20 21:44:26.926783: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8005\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "339/339 [==============================] - 18s 28ms/step - loss: 0.1973 - sparse_categorical_accuracy: 0.7855 - val_loss: 0.0382 - val_sparse_categorical_accuracy: 0.9555\n",
      "Epoch 2/20\n",
      "339/339 [==============================] - 7s 21ms/step - loss: 0.0272 - sparse_categorical_accuracy: 0.9682 - val_loss: 0.0291 - val_sparse_categorical_accuracy: 0.9648\n",
      "Epoch 3/20\n",
      "339/339 [==============================] - 7s 22ms/step - loss: 0.0181 - sparse_categorical_accuracy: 0.9785 - val_loss: 0.0273 - val_sparse_categorical_accuracy: 0.9679\n",
      "Epoch 4/20\n",
      "339/339 [==============================] - 8s 23ms/step - loss: 0.0139 - sparse_categorical_accuracy: 0.9837 - val_loss: 0.0275 - val_sparse_categorical_accuracy: 0.9682\n",
      "Epoch 5/20\n",
      "339/339 [==============================] - 7s 21ms/step - loss: 0.0111 - sparse_categorical_accuracy: 0.9874 - val_loss: 0.0286 - val_sparse_categorical_accuracy: 0.9678\n",
      "Epoch 6/20\n",
      "339/339 [==============================] - 7s 22ms/step - loss: 0.0088 - sparse_categorical_accuracy: 0.9900 - val_loss: 0.0301 - val_sparse_categorical_accuracy: 0.9674\n",
      "Epoch 7/20\n",
      "339/339 [==============================] - 7s 21ms/step - loss: 0.0070 - sparse_categorical_accuracy: 0.9923 - val_loss: 0.0320 - val_sparse_categorical_accuracy: 0.9671\n",
      "Epoch 8/20\n",
      "339/339 [==============================] - 7s 22ms/step - loss: 0.0054 - sparse_categorical_accuracy: 0.9942 - val_loss: 0.0339 - val_sparse_categorical_accuracy: 0.9669\n"
     ]
    }
   ],
   "source": [
    "# Early stopping\n",
    "early_stopping = keras.callbacks.EarlyStopping(monitor = 'val_loss', patience = 5)\n",
    "\n",
    "# Train model\n",
    "history = model.fit(\n",
    "    X_train_pad, y_train_pad,\n",
    "    validation_data = (X_valid_pad, y_valid_pad),\n",
    "    epochs = 20,\n",
    "    batch_size = 128,\n",
    "    callbacks = [early_stopping],\n",
    "    verbose = True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f05e396",
   "metadata": {
    "papermill": {
     "duration": 0.05747,
     "end_time": "2023-02-20T21:45:29.331674",
     "exception": false,
     "start_time": "2023-02-20T21:45:29.274204",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "The model reaches a reasonably **high accuracy** very **quickly**. After only a few epochs, early stopping kicks in."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e50c4b5a",
   "metadata": {
    "papermill": {
     "duration": 0.05736,
     "end_time": "2023-02-20T21:45:29.446181",
     "exception": false,
     "start_time": "2023-02-20T21:45:29.388821",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Evaluate on test set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "da9eefa0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:45:29.562865Z",
     "iopub.status.busy": "2023-02-20T21:45:29.562512Z",
     "iopub.status.idle": "2023-02-20T21:45:50.053147Z",
     "shell.execute_reply": "2023-02-20T21:45:50.052186Z"
    },
    "papermill": {
     "duration": 20.551779,
     "end_time": "2023-02-20T21:45:50.055290",
     "exception": false,
     "start_time": "2023-02-20T21:45:29.503511",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "452/452 [==============================] - 3s 6ms/step - loss: 0.0330 - sparse_categorical_accuracy: 0.9672\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.03304804116487503, 0.9672076106071472]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Evaluate model\n",
    "model.evaluate(X_test_pad, y_test_pad)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59d962f7",
   "metadata": {
    "papermill": {
     "duration": 0.060135,
     "end_time": "2023-02-20T21:45:50.176836",
     "exception": false,
     "start_time": "2023-02-20T21:45:50.116701",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Tag new samples**\n",
    "\n",
    "<br>\n",
    "\n",
    "The following function takes a list of sentences as input and returns the models **predicted tags** for each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c57f230e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:45:50.301572Z",
     "iopub.status.busy": "2023-02-20T21:45:50.299828Z",
     "iopub.status.idle": "2023-02-20T21:45:50.308008Z",
     "shell.execute_reply": "2023-02-20T21:45:50.307087Z"
    },
    "papermill": {
     "duration": 0.072267,
     "end_time": "2023-02-20T21:45:50.310024",
     "exception": false,
     "start_time": "2023-02-20T21:45:50.237757",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Tag new sentences\n",
    "def tag_new_sentences(sentences):\n",
    "    # Tokenize new sentences\n",
    "    X_new_ids = word_tokenizer.texts_to_sequences(sentences)\n",
    "    \n",
    "    # Pad sequences\n",
    "    X_new_pad = keras.preprocessing.sequence.pad_sequences(X_new_ids, padding = 'post', maxlen = max_length)\n",
    "    \n",
    "    # Make predictions\n",
    "    X_new_preds = model.predict(X_new_pad)\n",
    "    \n",
    "    # Retrieve most likely tag for each word\n",
    "    sentence_tags = []\n",
    "    for i, preds in enumerate(X_new_preds):\n",
    "        # Extract tags for only non-padded tokens\n",
    "        tags_seq = [np.argmax(p) for p in preds[:len(X_new_ids[i])]]\n",
    "        \n",
    "        # Convert ids back to tokens\n",
    "        words = [word_tokenizer.index_word[w] for w in X_new_ids[i]]\n",
    "        tags = [tag_tokenizer.index_word[t] for t in tags_seq]\n",
    "        \n",
    "        # zip words and tags together\n",
    "        sentence_tags.append(list(zip(words, tags)))\n",
    "\n",
    "    return sentence_tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "31703502",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:45:50.432820Z",
     "iopub.status.busy": "2023-02-20T21:45:50.431087Z",
     "iopub.status.idle": "2023-02-20T21:45:52.801967Z",
     "shell.execute_reply": "2023-02-20T21:45:52.801025Z"
    },
    "papermill": {
     "duration": 2.434741,
     "end_time": "2023-02-20T21:45:52.804588",
     "exception": false,
     "start_time": "2023-02-20T21:45:50.369847",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('newly', 'adv'), ('discovered', 'verb'), ('green', 'adj'), ('comet', 'noun'), ('comes', 'verb'), ('close', 'adv'), ('to', 'adp'), ('earth', 'noun')]\n",
      "\n",
      " [('<OOV>', 'adj'), ('mars', 'noun'), ('rover', 'noun'), ('perseverance', 'noun'), ('completes', 'verb'), ('rock', 'noun'), ('depot', 'noun')]\n"
     ]
    }
   ],
   "source": [
    "# Example sentences\n",
    "ex = [\n",
    "    \"Newly discovered green comet comes close to Earth.\",\n",
    "    \"Nasa's Mars rover Perseverance completes rock depot.\",\n",
    "]\n",
    "\n",
    "# Tag sentences\n",
    "ex_tagged = tag_new_sentences(ex)\n",
    "\n",
    "# Print results\n",
    "print(ex_tagged[0])\n",
    "print('\\n',ex_tagged[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b5aa9dc",
   "metadata": {
    "papermill": {
     "duration": 0.059952,
     "end_time": "2023-02-20T21:45:52.925453",
     "exception": false,
     "start_time": "2023-02-20T21:45:52.865501",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Nowadays, state-of-the-art PoS taggers use **transformer** models and much **more data** but it's still impressive that we very good results using a recurrent neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8af2923",
   "metadata": {
    "papermill": {
     "duration": 0.060883,
     "end_time": "2023-02-20T21:45:53.047355",
     "exception": false,
     "start_time": "2023-02-20T21:45:52.986472",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## <p style=\"font-family:JetBrains Mono; font-weight:normal; letter-spacing: 1px; color:#207d06; font-size:100%; text-align:left;padding: 0px; border-bottom: 3px solid #207d06;\">5.2 Language Modelling</p>\n",
    "\n",
    "We're now going to build a language model to **generate text like William Shakespear**. To train this model, we'll use a **text file** that contains the scripts for all of Shakespear's plays. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d3eec4aa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:45:53.170567Z",
     "iopub.status.busy": "2023-02-20T21:45:53.170177Z",
     "iopub.status.idle": "2023-02-20T21:45:53.196166Z",
     "shell.execute_reply": "2023-02-20T21:45:53.195232Z"
    },
    "papermill": {
     "duration": 0.089296,
     "end_time": "2023-02-20T21:45:53.198243",
     "exception": false,
     "start_time": "2023-02-20T21:45:53.108947",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n",
      "All:\n",
      "We know't, we know't.\n",
      "\n",
      "First Citizen:\n",
      "Let us\n"
     ]
    }
   ],
   "source": [
    "# Load text file\n",
    "text = open('/kaggle/input/shakespeare-text/text.txt', 'r').read()\n",
    "\n",
    "# Preview text\n",
    "print(text[:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e1c0e16",
   "metadata": {
    "papermill": {
     "duration": 0.059912,
     "end_time": "2023-02-20T21:45:53.318636",
     "exception": false,
     "start_time": "2023-02-20T21:45:53.258724",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We're actually going to build a **character level** model because this will be **easier to train** since the target space is significantly smaller compared to a word prediction model.\n",
    "\n",
    "A downside to predicting characters is the additional challenge of **learning to spell** words correctly. It does make it easier to deal with out of vocabulary words though so there is a trade-off."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a28df01a",
   "metadata": {
    "papermill": {
     "duration": 0.059786,
     "end_time": "2023-02-20T21:45:53.438673",
     "exception": false,
     "start_time": "2023-02-20T21:45:53.378887",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Tokenizer**\n",
    "\n",
    "<br>\n",
    "\n",
    "Notice how some **special characters** get included in the vocabulary but all letters has been **lower cased**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5b22b081",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:45:53.561875Z",
     "iopub.status.busy": "2023-02-20T21:45:53.560501Z",
     "iopub.status.idle": "2023-02-20T21:45:53.841351Z",
     "shell.execute_reply": "2023-02-20T21:45:53.840163Z"
    },
    "papermill": {
     "duration": 0.347668,
     "end_time": "2023-02-20T21:45:53.846587",
     "exception": false,
     "start_time": "2023-02-20T21:45:53.498919",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size: 39\n",
      "\n",
      "Vocabulary: {' ': 1, 'e': 2, 't': 3, 'o': 4, 'a': 5, 'i': 6, 'h': 7, 's': 8, 'r': 9, 'n': 10, '\\n': 11, 'l': 12, 'd': 13, 'u': 14, 'm': 15, 'y': 16, 'w': 17, ',': 18, 'c': 19, 'f': 20, 'g': 21, 'b': 22, 'p': 23, ':': 24, 'k': 25, 'v': 26, '.': 27, \"'\": 28, ';': 29, '?': 30, '!': 31, '-': 32, 'j': 33, 'q': 34, 'x': 35, 'z': 36, '3': 37, '&': 38, '$': 39}\n"
     ]
    }
   ],
   "source": [
    "# Tokenize individual letters\n",
    "tokenizer = keras.preprocessing.text.Tokenizer(char_level=True)\n",
    "\n",
    "# Fit tokenizer\n",
    "tokenizer.fit_on_texts([text])\n",
    "\n",
    "# Print vocabulary size\n",
    "print(\"Vocabulary size:\", len(tokenizer.word_index))\n",
    "\n",
    "# Print vocabulary\n",
    "print(\"\\nVocabulary:\", tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "975ef978",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:45:54.045612Z",
     "iopub.status.busy": "2023-02-20T21:45:54.045207Z",
     "iopub.status.idle": "2023-02-20T21:45:54.384402Z",
     "shell.execute_reply": "2023-02-20T21:45:54.382923Z"
    },
    "papermill": {
     "duration": 0.437054,
     "end_time": "2023-02-20T21:45:54.386519",
     "exception": false,
     "start_time": "2023-02-20T21:45:53.949465",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of tokens: 1115393\n"
     ]
    }
   ],
   "source": [
    "# Vectorize characters to intger ids\n",
    "ids = tokenizer.texts_to_sequences([text])[0]\n",
    "\n",
    "# Print length of sequence\n",
    "print(\"Number of tokens:\", len(ids))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5be61b1a",
   "metadata": {
    "papermill": {
     "duration": 0.061727,
     "end_time": "2023-02-20T21:45:54.509991",
     "exception": false,
     "start_time": "2023-02-20T21:45:54.448264",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Chunking**\n",
    "\n",
    "<br>\n",
    "\n",
    "We now need to **segment** the sequence of ids into **chunks** that will be our training examples. We do this by converting into a **tensorflow dataset** object and applying a **windowing function**. \n",
    "\n",
    "We need to **add 1** to the input sequence length because we are going to use the **next character** as the label for the current character. This will ensure we have the same input and target sequence lenghts. The `shift` parameter determines how many characters we shift to the right before creating a new chunk, i.e. it controls how much the chunks **overlap** (if at all). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8d13b0d5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:45:54.635099Z",
     "iopub.status.busy": "2023-02-20T21:45:54.634703Z",
     "iopub.status.idle": "2023-02-20T21:45:59.747504Z",
     "shell.execute_reply": "2023-02-20T21:45:59.745024Z"
    },
    "papermill": {
     "duration": 5.184406,
     "end_time": "2023-02-20T21:45:59.757130",
     "exception": false,
     "start_time": "2023-02-20T21:45:54.572724",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[19  7  5 ... 17  1  4]\n",
      " [13 27 11 ...  1 21  5]\n",
      " [ 8  3  9 ...  1 15  2]\n",
      " ...\n",
      " [ 5  6 10 ...  1 28  5]\n",
      " [10  4 17 ... 15 16  1]\n",
      " [18  1  6 ... 33  4  6]], shape=(32, 101), dtype=int32)\n",
      "tf.Tensor(\n",
      "[[ 3  4  9 ... 20  1 16]\n",
      " [ 1  5 20 ... 33  4  7]\n",
      " [13  6 13 ... 15  2 24]\n",
      " ...\n",
      " [11  5  1 ... 16  4 14]\n",
      " [30 11  7 ... 24 11 19]\n",
      " [14 10 19 ... 13  1 10]], shape=(32, 101), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# Tensorflow dataset\n",
    "slices = tf.data.Dataset.from_tensor_slices(ids)\n",
    "\n",
    "# Create chunks\n",
    "input_timesteps = 100\n",
    "window_size = input_timesteps + 1\n",
    "windows = slices.window(window_size, shift=50, drop_remainder=True)  # bigger shift reduces number of chunks\n",
    "\n",
    "# Convert window objects to tensors\n",
    "dataset = windows.flat_map(lambda window: window.batch(window_size))\n",
    "\n",
    "# Divide dataset into batches for training\n",
    "batch_size = 32\n",
    "batches = dataset.shuffle(10000).batch(batch_size)\n",
    "\n",
    "# Print example batches\n",
    "for b in batches.take(2):\n",
    "    print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c190a53",
   "metadata": {
    "papermill": {
     "duration": 0.093933,
     "end_time": "2023-02-20T21:45:59.957769",
     "exception": false,
     "start_time": "2023-02-20T21:45:59.863836",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Features and labels**\n",
    "\n",
    "<br>\n",
    "\n",
    "Now that we have our chunks, we need to **split** them into training features and labels. Remember that we are using a **self-supervised** approach, so the label is always the character that comes after the current one. We are trying to teach our model to predict the **next character** in the sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6902f283",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:46:00.147264Z",
     "iopub.status.busy": "2023-02-20T21:46:00.146832Z",
     "iopub.status.idle": "2023-02-20T21:46:02.612300Z",
     "shell.execute_reply": "2023-02-20T21:46:02.611283Z"
    },
    "papermill": {
     "duration": 2.563001,
     "end_time": "2023-02-20T21:46:02.614763",
     "exception": false,
     "start_time": "2023-02-20T21:46:00.051762",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X1 length:  100\n",
      "X1:  [ 5 14 10  3 24 11  4 18  1  3  4  1 17  7  5  3  1 23 14  9 23  4  8  2\n",
      "  1 13  4  8  3  1  3  7  4 14  1  7  4  5  9 13  1  3  7 16  1 17  4  9\n",
      " 13  8 18 11  3  7  5  3  1  3  7  4 14  1  9  2  3 14  9 10 28  8  3  1\n",
      " 10  4  1 21  9  2  2  3  6 10 21  1  3  4  1  3  7 16  1 20  9  6  2 10\n",
      " 13  8 30 11]\n",
      "\n",
      "y1 length:  100\n",
      "y1:  [14 10  3 24 11  4 18  1  3  4  1 17  7  5  3  1 23 14  9 23  4  8  2  1\n",
      " 13  4  8  3  1  3  7  4 14  1  7  4  5  9 13  1  3  7 16  1 17  4  9 13\n",
      "  8 18 11  3  7  5  3  1  3  7  4 14  1  9  2  3 14  9 10 28  8  3  1 10\n",
      "  4  1 21  9  2  2  3  6 10 21  1  3  4  1  3  7 16  1 20  9  6  2 10 13\n",
      "  8 30 11 11]\n"
     ]
    }
   ],
   "source": [
    "# Split features and labels\n",
    "Xy_batches = batches.map(lambda batch: (batch[:, :-1], batch[:, 1:]))\n",
    "\n",
    "# Print first training example\n",
    "for b in Xy_batches.take(1):\n",
    "    print(\"X1 length: \", len(b[0][0].numpy()))\n",
    "    print(\"X1: \", b[0][0].numpy())\n",
    "    print(\"\\ny1 length: \", len(b[1][0].numpy()))\n",
    "    print(\"y1: \", b[1][0].numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "139eb2ef",
   "metadata": {
    "papermill": {
     "duration": 0.06022,
     "end_time": "2023-02-20T21:46:02.790521",
     "exception": false,
     "start_time": "2023-02-20T21:46:02.730301",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Vectorize inputs**\n",
    "\n",
    "<br>\n",
    "\n",
    "The last step is map our input integer ids to **vectors**. Since each token is just a character, it doesn't make much sense to use embeddings as there is no sense of character meaning. Instead, we'll just use **one-hot encoding** to be able to pass it to a RNN model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5876c1fe",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:46:02.912472Z",
     "iopub.status.busy": "2023-02-20T21:46:02.912115Z",
     "iopub.status.idle": "2023-02-20T21:46:02.942985Z",
     "shell.execute_reply": "2023-02-20T21:46:02.942088Z"
    },
    "papermill": {
     "duration": 0.094448,
     "end_time": "2023-02-20T21:46:02.945086",
     "exception": false,
     "start_time": "2023-02-20T21:46:02.850638",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Number of tokens (+1 of oov tokens)\n",
    "num_tokens = len(tokenizer.word_index) + 1\n",
    "\n",
    "# One-hot encode the input sequences\n",
    "Xy_batches = Xy_batches.map(lambda inputs, labels: (tf.one_hot(tf.cast(inputs, tf.int32), depth=num_tokens), labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edf3b3b6",
   "metadata": {
    "papermill": {
     "duration": 0.060612,
     "end_time": "2023-02-20T21:46:03.066611",
     "exception": false,
     "start_time": "2023-02-20T21:46:03.005999",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Prefetching **loads** the data for the **next batch**, while the model is training on the current batch. It can **speed** up the training process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "d0a3f637",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:46:03.190732Z",
     "iopub.status.busy": "2023-02-20T21:46:03.190373Z",
     "iopub.status.idle": "2023-02-20T21:46:03.195901Z",
     "shell.execute_reply": "2023-02-20T21:46:03.194816Z"
    },
    "papermill": {
     "duration": 0.069274,
     "end_time": "2023-02-20T21:46:03.198156",
     "exception": false,
     "start_time": "2023-02-20T21:46:03.128882",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Prefetch data\n",
    "Xy_batches = Xy_batches.prefetch(tf.data.AUTOTUNE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe30dc32",
   "metadata": {
    "papermill": {
     "duration": 0.061144,
     "end_time": "2023-02-20T21:46:03.320063",
     "exception": false,
     "start_time": "2023-02-20T21:46:03.258919",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Build model**\n",
    "\n",
    "<br>\n",
    "\n",
    "We're now ready to build our model. This time there will be **no imbedding layer** so the input vectors will be fed straight through to **2 stacked LSTM layers** followed by a **dense output layer**. As a result, the **number of parameters** in the model is much **lower** than before.\n",
    "\n",
    "Note that `recurrent_dropout` is added, which applies dropout **horizontally** across time steps. This will **prevent** the model from **memorizing** long strings of text in the training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "60da5dcd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:46:03.442904Z",
     "iopub.status.busy": "2023-02-20T21:46:03.442559Z",
     "iopub.status.idle": "2023-02-20T21:46:03.654704Z",
     "shell.execute_reply": "2023-02-20T21:46:03.653304Z"
    },
    "papermill": {
     "duration": 0.276605,
     "end_time": "2023-02-20T21:46:03.657368",
     "exception": false,
     "start_time": "2023-02-20T21:46:03.380763",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "lstm_1 (LSTM)                (None, None, 128)         86528     \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, None, 128)         131584    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, None, 40)          5160      \n",
      "=================================================================\n",
      "Total params: 223,272\n",
      "Trainable params: 223,272\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Build model\n",
    "model = keras.Sequential([\n",
    "\n",
    "    # LSTM layers\n",
    "    layers.LSTM(128, return_sequences=True, input_shape=[None, num_tokens], recurrent_dropout=0.2),\n",
    "    layers.LSTM(128, return_sequences=True, input_shape=[None, num_tokens], recurrent_dropout=0.2),\n",
    "    \n",
    "    # Output layer\n",
    "    layers.Dense(num_tokens, activation='softmax')\n",
    "])\n",
    "\n",
    "# Define optimizer and loss function\n",
    "model.compile(optimizer='adam',loss='sparse_categorical_crossentropy')\n",
    "\n",
    "# Model summary\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5fe94b3",
   "metadata": {
    "papermill": {
     "duration": 0.064249,
     "end_time": "2023-02-20T21:46:03.786055",
     "exception": false,
     "start_time": "2023-02-20T21:46:03.721806",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Train model**\n",
    "\n",
    "<br>\n",
    "\n",
    "This might take some time so we will **save the model** after **every epoch** if it reduces the loss. We'll also include **early stopping** and **learning rate scheduler** callbacks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8aba33d2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:46:03.909368Z",
     "iopub.status.busy": "2023-02-20T21:46:03.908997Z",
     "iopub.status.idle": "2023-02-20T21:46:03.915529Z",
     "shell.execute_reply": "2023-02-20T21:46:03.914480Z"
    },
    "papermill": {
     "duration": 0.070497,
     "end_time": "2023-02-20T21:46:03.917609",
     "exception": false,
     "start_time": "2023-02-20T21:46:03.847112",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Number of epochs\n",
    "num_epochs = 30\n",
    "\n",
    "# Early stopping\n",
    "early_stopping = keras.callbacks.EarlyStopping(\n",
    "    patience = 3,\n",
    "    min_delta = 0.0001,\n",
    "    monitor = 'loss',\n",
    "    restore_best_weights = True,\n",
    ")\n",
    "\n",
    "# Learning rate scheduler\n",
    "schedule = keras.optimizers.schedules.CosineDecay(initial_learning_rate = 0.002, decay_steps = num_epochs, alpha = 0.0001)\n",
    "scheduler = keras.callbacks.LearningRateScheduler(schedule, verbose = 0)\n",
    "\n",
    "# Save best model at every epoch\n",
    "checkpoint = keras.callbacks.ModelCheckpoint(\n",
    "    'stacked_lstm',\n",
    "    monitor = 'loss',\n",
    "    verbose = 1,\n",
    "    save_best_only = True,\n",
    "    save_weights_only = False,\n",
    "    mode = 'auto',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2c00a889",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-20T21:46:04.055448Z",
     "iopub.status.busy": "2023-02-20T21:46:04.054484Z",
     "iopub.status.idle": "2023-02-21T02:04:32.398308Z",
     "shell.execute_reply": "2023-02-21T02:04:32.397291Z"
    },
    "papermill": {
     "duration": 15508.419839,
     "end_time": "2023-02-21T02:04:32.400864",
     "exception": false,
     "start_time": "2023-02-20T21:46:03.981025",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "698/698 [==============================] - 486s 689ms/step - loss: 2.3170\n",
      "\n",
      "Epoch 00001: loss improved from inf to 2.31700, saving model to stacked_lstm\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-02-20 21:54:12.023820: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 2/30\n",
      "698/698 [==============================] - 481s 685ms/step - loss: 1.8156\n",
      "\n",
      "Epoch 00002: loss improved from 2.31700 to 1.81559, saving model to stacked_lstm\n",
      "Epoch 3/30\n",
      "698/698 [==============================] - 484s 690ms/step - loss: 1.6557\n",
      "\n",
      "Epoch 00003: loss improved from 1.81559 to 1.65568, saving model to stacked_lstm\n",
      "Epoch 4/30\n",
      "698/698 [==============================] - 484s 690ms/step - loss: 1.5750\n",
      "\n",
      "Epoch 00004: loss improved from 1.65568 to 1.57503, saving model to stacked_lstm\n",
      "Epoch 5/30\n",
      "698/698 [==============================] - 484s 690ms/step - loss: 1.5253\n",
      "\n",
      "Epoch 00005: loss improved from 1.57503 to 1.52534, saving model to stacked_lstm\n",
      "Epoch 6/30\n",
      "698/698 [==============================] - 486s 692ms/step - loss: 1.4917\n",
      "\n",
      "Epoch 00006: loss improved from 1.52534 to 1.49172, saving model to stacked_lstm\n",
      "Epoch 7/30\n",
      "698/698 [==============================] - 485s 691ms/step - loss: 1.4678\n",
      "\n",
      "Epoch 00007: loss improved from 1.49172 to 1.46779, saving model to stacked_lstm\n",
      "Epoch 8/30\n",
      "698/698 [==============================] - 486s 692ms/step - loss: 1.4489\n",
      "\n",
      "Epoch 00008: loss improved from 1.46779 to 1.44892, saving model to stacked_lstm\n",
      "Epoch 9/30\n",
      "698/698 [==============================] - 487s 694ms/step - loss: 1.4338\n",
      "\n",
      "Epoch 00009: loss improved from 1.44892 to 1.43379, saving model to stacked_lstm\n",
      "Epoch 10/30\n",
      "698/698 [==============================] - 487s 695ms/step - loss: 1.4220\n",
      "\n",
      "Epoch 00010: loss improved from 1.43379 to 1.42204, saving model to stacked_lstm\n",
      "Epoch 11/30\n",
      "698/698 [==============================] - 489s 697ms/step - loss: 1.4114\n",
      "\n",
      "Epoch 00011: loss improved from 1.42204 to 1.41137, saving model to stacked_lstm\n",
      "Epoch 12/30\n",
      "698/698 [==============================] - 495s 706ms/step - loss: 1.4027\n",
      "\n",
      "Epoch 00012: loss improved from 1.41137 to 1.40273, saving model to stacked_lstm\n",
      "Epoch 13/30\n",
      "698/698 [==============================] - 497s 709ms/step - loss: 1.3949\n",
      "\n",
      "Epoch 00013: loss improved from 1.40273 to 1.39492, saving model to stacked_lstm\n",
      "Epoch 14/30\n",
      "698/698 [==============================] - 497s 709ms/step - loss: 1.3888\n",
      "\n",
      "Epoch 00014: loss improved from 1.39492 to 1.38884, saving model to stacked_lstm\n",
      "Epoch 15/30\n",
      "698/698 [==============================] - 500s 714ms/step - loss: 1.3825\n",
      "\n",
      "Epoch 00015: loss improved from 1.38884 to 1.38253, saving model to stacked_lstm\n",
      "Epoch 16/30\n",
      "698/698 [==============================] - 496s 707ms/step - loss: 1.3778\n",
      "\n",
      "Epoch 00016: loss improved from 1.38253 to 1.37777, saving model to stacked_lstm\n",
      "Epoch 17/30\n",
      "698/698 [==============================] - 499s 712ms/step - loss: 1.3730\n",
      "\n",
      "Epoch 00017: loss improved from 1.37777 to 1.37298, saving model to stacked_lstm\n",
      "Epoch 18/30\n",
      "698/698 [==============================] - 499s 712ms/step - loss: 1.3690\n",
      "\n",
      "Epoch 00018: loss improved from 1.37298 to 1.36897, saving model to stacked_lstm\n",
      "Epoch 19/30\n",
      "698/698 [==============================] - 498s 710ms/step - loss: 1.3656\n",
      "\n",
      "Epoch 00019: loss improved from 1.36897 to 1.36557, saving model to stacked_lstm\n",
      "Epoch 20/30\n",
      "698/698 [==============================] - 502s 715ms/step - loss: 1.3622\n",
      "\n",
      "Epoch 00020: loss improved from 1.36557 to 1.36223, saving model to stacked_lstm\n",
      "Epoch 21/30\n",
      "698/698 [==============================] - 502s 715ms/step - loss: 1.3596\n",
      "\n",
      "Epoch 00021: loss improved from 1.36223 to 1.35963, saving model to stacked_lstm\n",
      "Epoch 22/30\n",
      "698/698 [==============================] - 505s 720ms/step - loss: 1.3570\n",
      "\n",
      "Epoch 00022: loss improved from 1.35963 to 1.35697, saving model to stacked_lstm\n",
      "Epoch 23/30\n",
      "698/698 [==============================] - 503s 718ms/step - loss: 1.3548\n",
      "\n",
      "Epoch 00023: loss improved from 1.35697 to 1.35477, saving model to stacked_lstm\n",
      "Epoch 24/30\n",
      "698/698 [==============================] - 504s 718ms/step - loss: 1.3536\n",
      "\n",
      "Epoch 00024: loss improved from 1.35477 to 1.35363, saving model to stacked_lstm\n",
      "Epoch 25/30\n",
      "698/698 [==============================] - 505s 719ms/step - loss: 1.3515\n",
      "\n",
      "Epoch 00025: loss improved from 1.35363 to 1.35155, saving model to stacked_lstm\n",
      "Epoch 26/30\n",
      "698/698 [==============================] - 507s 723ms/step - loss: 1.3505\n",
      "\n",
      "Epoch 00026: loss improved from 1.35155 to 1.35046, saving model to stacked_lstm\n",
      "Epoch 27/30\n",
      "698/698 [==============================] - 509s 726ms/step - loss: 1.3500\n",
      "\n",
      "Epoch 00027: loss improved from 1.35046 to 1.34996, saving model to stacked_lstm\n",
      "Epoch 28/30\n",
      "698/698 [==============================] - 508s 723ms/step - loss: 1.3488\n",
      "\n",
      "Epoch 00028: loss improved from 1.34996 to 1.34881, saving model to stacked_lstm\n",
      "Epoch 29/30\n",
      "698/698 [==============================] - 507s 723ms/step - loss: 1.3478\n",
      "\n",
      "Epoch 00029: loss improved from 1.34881 to 1.34785, saving model to stacked_lstm\n",
      "Epoch 30/30\n",
      "698/698 [==============================] - 508s 724ms/step - loss: 1.3480\n",
      "\n",
      "Epoch 00030: loss did not improve from 1.34785\n"
     ]
    }
   ],
   "source": [
    "# Train model\n",
    "history = model.fit(\n",
    "    Xy_batches,\n",
    "    epochs = num_epochs, \n",
    "    callbacks = [early_stopping, scheduler, checkpoint]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "feec7989",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-21T02:04:34.940273Z",
     "iopub.status.busy": "2023-02-21T02:04:34.939781Z",
     "iopub.status.idle": "2023-02-21T02:04:36.771649Z",
     "shell.execute_reply": "2023-02-21T02:04:36.770590Z"
    },
    "papermill": {
     "duration": 3.128176,
     "end_time": "2023-02-21T02:04:36.774179",
     "exception": false,
     "start_time": "2023-02-21T02:04:33.646003",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load best model after training completed\n",
    "model = keras.models.load_model('stacked_lstm')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91b4f91b",
   "metadata": {
    "papermill": {
     "duration": 1.244124,
     "end_time": "2023-02-21T02:04:39.160526",
     "exception": false,
     "start_time": "2023-02-21T02:04:37.916402",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**Generate new text**\n",
    "\n",
    "<br>\n",
    "\n",
    "We can generate new text using the following function. It takes in a portion of text to **kickstart** the model, which then makes predictions up to a **pre-defined maximum** number of characters. \n",
    "\n",
    "When sampling from the output distribution, we can adjust it using the **temperature parameter**. The **higher** the temperature the more **random** the predictions, whereas the **lower** the temperature the more **sharpened** the original distribution gets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "8252de1f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-21T02:04:41.564536Z",
     "iopub.status.busy": "2023-02-21T02:04:41.564172Z",
     "iopub.status.idle": "2023-02-21T02:04:41.572018Z",
     "shell.execute_reply": "2023-02-21T02:04:41.571071Z"
    },
    "papermill": {
     "duration": 1.24776,
     "end_time": "2023-02-21T02:04:41.574005",
     "exception": false,
     "start_time": "2023-02-21T02:04:40.326245",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def generate_text(model, tokenizer, seed_text, num_chars = 200, temperature = 1):\n",
    "    \n",
    "    text = seed_text\n",
    "    \n",
    "    for _ in range(num_chars):\n",
    "        \n",
    "        # Use 'input_timesteps' most recent characters as input for the model\n",
    "        X_new = np.array(tokenizer.texts_to_sequences([text[-input_timesteps:]]))\n",
    "        \n",
    "        # One-hot encode input\n",
    "        X_new = tf.one_hot(X_new, num_tokens)\n",
    "        \n",
    "        # Predict probability distribution for next character\n",
    "        preds = model.predict(X_new)[0, -1:, :]\n",
    "        \n",
    "        # Adjust probability distribution using temperature\n",
    "        preds = tf.math.log(preds) / temperature\n",
    "\n",
    "        # Sample next character\n",
    "        next_char_id = tf.random.categorical(preds, num_samples=1)\n",
    "        next_char = tokenizer.sequences_to_texts(next_char_id.numpy())[0]\n",
    "\n",
    "        # Add character to running text\n",
    "        text += next_char\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5c0e6b83",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-21T02:04:43.981679Z",
     "iopub.status.busy": "2023-02-21T02:04:43.981036Z",
     "iopub.status.idle": "2023-02-21T02:05:01.210462Z",
     "shell.execute_reply": "2023-02-21T02:05:01.209473Z"
    },
    "papermill": {
     "duration": 18.498006,
     "end_time": "2023-02-21T02:05:01.212633",
     "exception": false,
     "start_time": "2023-02-21T02:04:42.714627",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Romeo took a plane to visit his uncle,\n",
      "and so we will be so be so long and the country.\n",
      "\n",
      "clarence:\n",
      "i will not be the brother of the common of the poor\n",
      "of the prince of the state of the base of such a\n",
      "shall be the prince of the truth of t\n",
      "CPU times: user 16.8 s, sys: 375 ms, total: 17.2 s\n",
      "Wall time: 17.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "print(generate_text(model, tokenizer, \"Romeo took a plane to visit his uncle\", num_chars = 200, temperature = 0.2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "e3f9612b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-21T02:05:03.604559Z",
     "iopub.status.busy": "2023-02-21T02:05:03.604202Z",
     "iopub.status.idle": "2023-02-21T02:05:19.443309Z",
     "shell.execute_reply": "2023-02-21T02:05:19.442370Z"
    },
    "papermill": {
     "duration": 16.977556,
     "end_time": "2023-02-21T02:05:19.445414",
     "exception": false,
     "start_time": "2023-02-21T02:05:02.467858",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Juliet was enjoying her new job as a journalist,\n",
      "shall desire the people to say the necks,\n",
      "which she is a grave in a safe of the complexion.\n",
      "\n",
      "clown:\n",
      "which i was the man? o love a bear a band\n",
      "to be a speech in the shepherd of the father.\n",
      "\n",
      "provost:\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(model, tokenizer, \"Juliet was enjoying her new job as a journalist\", num_chars = 200, temperature = 0.5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "93d1e227",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-21T02:05:21.820915Z",
     "iopub.status.busy": "2023-02-21T02:05:21.819890Z",
     "iopub.status.idle": "2023-02-21T02:05:39.726079Z",
     "shell.execute_reply": "2023-02-21T02:05:39.725118Z"
    },
    "papermill": {
     "duration": 19.058321,
     "end_time": "2023-02-21T02:05:39.728244",
     "exception": false,
     "start_time": "2023-02-21T02:05:20.669923",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Macbeth put down the knife and walked away.\n",
      "\n",
      "coriolanus:\n",
      "which thou were in no face.\n",
      "\n",
      "lucio:\n",
      "and if you be speech!\n",
      "\n",
      "benvolio:\n",
      "were yell: it shall my father.\n",
      "\n",
      "prospira:\n",
      "what is touchest?\n",
      "\n",
      "warwick:\n",
      "this marchman short and tell on rutle;\n",
      "if sole\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(model, tokenizer, \"Macbeth put down the knife and walked away\", num_chars = 200, temperature = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5d4154d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-21T02:05:42.385290Z",
     "iopub.status.busy": "2023-02-21T02:05:42.384853Z",
     "iopub.status.idle": "2023-02-21T02:05:58.243649Z",
     "shell.execute_reply": "2023-02-21T02:05:58.242650Z"
    },
    "papermill": {
     "duration": 17.116949,
     "end_time": "2023-02-21T02:05:58.245725",
     "exception": false,
     "start_time": "2023-02-21T02:05:41.128776",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To eat chocolate or not to eat chocolate!--as sweetle,\n",
      "musern'm till's foor-deat our, 'haul we painted\n",
      "astopad allicdneg sern; dela, virtuis\n",
      "suquared seizen. sir; tybalt; boych dwerr,\n",
      "hat 'codanct,-fo horsumph:, knaggerenk-havknobiy'! we'll\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(model, tokenizer, \"To eat chocolate or not to eat chocolate\", num_chars = 200, temperature = 2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f06c044",
   "metadata": {
    "papermill": {
     "duration": 1.251965,
     "end_time": "2023-02-21T02:06:00.725653",
     "exception": false,
     "start_time": "2023-02-21T02:05:59.473688",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "We can see that as the **temperature increases**, the number of **spelling mistakes** increases and the text becomes closer to gibberish. That being said, it is still very impressive how the model learnt how to produce actual **words** in the style of **Shakespeare** by predicting a single character at a time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2aca62f8",
   "metadata": {
    "papermill": {
     "duration": 1.428463,
     "end_time": "2023-02-21T02:06:03.717117",
     "exception": false,
     "start_time": "2023-02-21T02:06:02.288654",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 6. Conclusion\n",
    "\n",
    "In this notebook we looked at how Recurrent Neural Networks and its variants can be used to model text as a **sequence of words**. This allowed to solve a new set of problems, namely **sequence labelling** and **language generation**.\n",
    "\n",
    "More generally, there are **different ways to set up RNNs** to solve different kinds of problems. \n",
    "\n",
    "<br>\n",
    "<center>\n",
    "<img src=\"..\\assets\\8\\rnn-setups.jpg\" width=600>\n",
    "</center>\n",
    "<br>\n",
    "\n",
    "* **one-to-one** - is equivalent to a feed-forward neural network.\n",
    "* **one-to-many** - can be used for captioning, decoders or music generation.\n",
    "* **many-to-one** - is used for classification tasks like sentiment analysis.\n",
    "* **many-to-many** - used for sequence labelling and language modelling tasks.\n",
    "* **staggered many-to-many** - (also known as sequence-to-sequence) its applications include translation, summarization and chatbots. \n",
    "\n",
    "We'll take a look at the **last type of RNN in more depth** in the **next notebook** where we'll build a translation model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2729179",
   "metadata": {
    "papermill": {
     "duration": 1.235795,
     "end_time": "2023-02-21T02:06:06.093893",
     "exception": false,
     "start_time": "2023-02-21T02:06:04.858098",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "**References:**\n",
    "* [NLP demystified](https://www.nlpdemystified.org/)\n",
    "\n",
    "### Coming UP\n",
    "#### [9. Machine Translation and Attention](./09_Machine_Translation_Attention.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36f86136",
   "metadata": {
    "papermill": {
     "duration": 1.234055,
     "end_time": "2023-02-21T02:06:08.469021",
     "exception": false,
     "start_time": "2023-02-21T02:06:07.234966",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Thanks for reading!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 15760.904062,
   "end_time": "2023-02-21T02:06:13.216096",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2023-02-20T21:43:32.312034",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}