PoemGeneration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9724a943",
   "metadata": {},
   "source": [
    "# Poem Generation using GPT\n",
    "\n",
    "In this notebook, we will generate a simple shakespearian poem using Generative Pretrained Transformers (GPT) that we implemented. This notebook will demonstrate poem generation character by character rather than the typical word by word generation. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62e21087",
   "metadata": {},
   "source": [
    "# Importing Libraries\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2466baa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "\n",
    "from gpt import GPT, GPTconfig\n",
    "from gpt import TrainingConfig, Trainer\n",
    "from gpt import sample_context\n",
    "# Could have imported all of them at once. Doesn't matter :)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17ae7a85",
   "metadata": {},
   "source": [
    "Setting Manual Seed to avoid varying results with every run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b21d6185",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gpt.utils.utils import seed_all\n",
    "\n",
    "seed_all(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7e79ba2",
   "metadata": {},
   "source": [
    "## Poem Dataset class \n",
    "\n",
    "We will now use the Dataset class from `torch.utils.data` to setup our own dataset class for the dataloader. This class is responsible for loading the data from disk and generating chunks of characters. The training data will be a chunk of characters where chunk is a block_size (T). The targets for training would be the same as the training data but offset by one character. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "863c6516",
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset\n",
    "\n",
    "class CharacterDataset(Dataset):\n",
    "        def __init__(self, data, block_size):\n",
    "            characters = sorted(list(set(data)))\n",
    "            data_size, vocab_size = len(data), len(characters)\n",
    "            \n",
    "            print(f\"Dataset has {data_size} characters. {vocab_size} of characters are unique.\")\n",
    "            \n",
    "            # char to idx mapping and vice-versa\n",
    "            self.stoi = {ch:i for i,ch in enumerate(characters)}\n",
    "            self.itos = {i:ch for i,ch in enumerate(characters)}\n",
    "            \n",
    "            self.block_size = block_size\n",
    "            self.vocab_size = vocab_size\n",
    "            self.data_size = data_size\n",
    "            self.data = data\n",
    "            \n",
    "        def __len__(self):\n",
    "            return self.data_size - self.block_size\n",
    "        \n",
    "        def __getitem__(self, idx):\n",
    "            # take a chunk of data from the given index from the dataset\n",
    "            chunk = self.data[idx : idx + self.block_size + 1]\n",
    "            \n",
    "            #convert the chunk to integers\n",
    "            data = [self.stoi[ch] for ch in chunk]\n",
    "            \n",
    "            # create x and y. \n",
    "            # x will contain every but the last character in the chunk.\n",
    "            # y will contain every but the first character in the chunk.\n",
    "            # Hence this will create an offset in targets by 1.\n",
    "            # Thus helps in language modelling. Given a character, the goal of the transformer would be to predict the next character in sequence.\n",
    "            \n",
    "            x = torch.tensor(data[:-1], dtype=torch.long) # nn.Embedding requires input data to be in torch.long\n",
    "            y = torch.tensor(data[1:], dtype=torch.long)\n",
    "            \n",
    "            return x,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5dab98bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_PATH = \"./data/input.txt\"\n",
    "BLOCK_SIZE = 128 # spatial context of the transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f5cf0d65",
   "metadata": {},
   "outputs": [],
   "source": [
    "file = open(DATA_PATH, \"r\")\n",
    "data = file.read(-1) # -1 means read the whole file. If file size is large, you may want to consider replacing it with the number of characters to be read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "db970d1a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset has 1115393 characters. 65 of characters are unique.\n"
     ]
    }
   ],
   "source": [
    "dataset = CharacterDataset(data = data, block_size = BLOCK_SIZE) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d8af62",
   "metadata": {},
   "source": [
    "### Visualizing the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "75dff3de",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mTraining Data : \u001b[0m\n",
      "]nFirst Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to \n",
      "\n",
      "\u001b[1mTargets : \u001b[0m\n",
      "\n",
      "irst Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to d\n"
     ]
    }
   ],
   "source": [
    "batch = dataset[0] # returns a tuple of tensors at idx = 0. Feel free to chnage the idx\n",
    "x, y = batch\n",
    "x = x.tolist()\n",
    "y = y.tolist()\n",
    "\n",
    "for i in range(len(x)):\n",
    "    x[i] = dataset.itos[x[i]]\n",
    "    y[i] = dataset.itos[y[i]]\n",
    "\n",
    "print(f\"\\033[1mTraining Data : \\033[0m\\n\\n{''.join(x)}\")\n",
    "print(f\"\\n\\033[1mTargets : \\033[0m\\n\\n{''.join(y)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f70981a",
   "metadata": {},
   "source": [
    "## Configuring GPT\n",
    "\n",
    "Now that we have finished setting up the dataset class, its now time to train our GPT model on this dataset. Before we start training, we will configure the GPT with appropriate model parameters. \n",
    "\n",
    "Because original GPT (referring to GPT3) requires huge computational resources, we will be using a smaller GPT model. This model, though being small, is by itself a very good model. A single layer model can learn to generate poems with fairly good accuracy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3b8169b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt_config = GPTconfig(num_layers = 2, \n",
    "                       n_heads = 12, \n",
    "                       embd_size = 768, \n",
    "                       vocab_size = dataset.vocab_size, \n",
    "                       block_size = dataset.block_size\n",
    "                      )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "cf2f5550",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Trainable Parameters :  14375489\n"
     ]
    }
   ],
   "source": [
    "model = GPT(gpt_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2d99847",
   "metadata": {},
   "source": [
    "## Training the GPT model\n",
    "\n",
    "GPT model has been configured. Now it is time to train it on our dataset. As mentioned earlier, training the model requires lot of computational time and resources. So the amount of time taken to train depends on the kind of system you have. The training loop is designed to work with multiple GPUs if you have access to them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "96b0ecaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_config = TrainingConfig(max_epochs = 2, \n",
    "                              batch_size = 256, \n",
    "                              lr_decay = True, \n",
    "                              lr = 6e-4,\n",
    "                              warmup_tokens = 512*20,\n",
    "                              final_tokens = 2 * len(dataset) * dataset.block_size,\n",
    "                              ckpt_path = \"./checkpoints/transformers.pt\"\n",
    "                             )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "797c0401",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "epoch 1 it: 4357 | loss: 0.93291 lr: 3.000169e-04: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4357/4357 [12:05<00:00,  6.01it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving model at ./checkpoints/transformers.pt.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "epoch 2 it: 4357 | loss: 0.69899 lr: 6.000000e-05: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4357/4357 [12:08<00:00,  5.98it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving model at ./checkpoints/transformers.pt.\n"
     ]
    }
   ],
   "source": [
    "trainer = Trainer(model = model, train_set = dataset, test_set = None, configs = train_config)\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24275f11",
   "metadata": {},
   "source": [
    "## Let's generate poems\n",
    "\n",
    "The model has been trained and it would have learnt the mappings of different sequences. Now, we will seed it with a starting context and ask the model to predict the next character seq by seq until we are done sampling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ab669e26",
   "metadata": {},
   "outputs": [],
   "source": [
    "seed_context = \"Help me!\"\n",
    "x = torch.tensor([dataset.stoi[s] for s in seed_context], dtype=torch.long, device=trainer.device)[None,...]\n",
    "y = sample_context(model=model, x=x, steps=10000, temperature=1.0, sample=True, top_k=10)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "db12530d",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = y.tolist()\n",
    "y = [dataset.itos[i] for i in y]\n",
    "y = \"\".join(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "46d27da7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mGenerated Data : \u001b[0m\n",
      "\n",
      "Help me!\n",
      "Let them not live their toward Tewksbury:\n",
      "What we have unfeignedly hast our closet?\n",
      "\n",
      "GLOUCESTER:\n",
      "No doubt, no doubt; and, I pray.\n",
      "\n",
      "LADY ANNE:\n",
      "Who would believe me?\n",
      "\n",
      "GLOUCESTER:\n",
      "I did not, my lord.\n",
      "\n",
      "LADY ANNE:\n",
      "If I thought that, I tell thee, homicide.\n",
      "\n",
      "GLOUCESTER:\n",
      "\n",
      "KING EDWARD IV:\n",
      "I prithee, may prove it me.\n",
      "\n",
      "LADY GREY:\n",
      "I know thee to bear where this once is dead.\n",
      "\n",
      "GLOUCESTER:\n",
      "Her husband, bade the court?\n",
      "\n",
      "QUEEN MARGARET:\n",
      "But didst thou so? thou speak again of such majesty\n",
      "As thou shalt wish for this pernicious mistress.\n",
      "\n",
      "LEONTES:\n",
      "O, would you have, friend. When did Camillo there;\n",
      "The innocent man come with a courteous honour,\n",
      "Whose uncle Clarence may swear as compell\n",
      "A, in the stocks redemption.\n",
      "\n",
      "MARCIUS:\n",
      "Nay, pray, get thee better?\n",
      "\n",
      "Messenger:\n",
      "They longer burns and go light it?\n",
      "\n",
      "LADY CAPULET:\n",
      "We two deep and cross in secret shall raise his head\n",
      "Which we will prove his letters play the orator.\n",
      "\n",
      "JULIET:\n",
      "Alas, get thy company, and old April on thee\n",
      "An o'er thief thy state bruising forth,\n",
      "Straing harsh divined words with the grossness;\n",
      "Show thy share his wife; but he, if thou slew'st,\n",
      "Without ripe moving full flowers, I could not\n",
      "say it was my neck: I have had an ill name\n",
      "But it-dot change; my spices o' the collar;\n",
      "The father, as I said, is gone, and full of weeping,\n",
      "Even for revenge! see when thou hast suck'd traitor,\n",
      "Which did not unturn my knee? What's thy way?\n",
      "\n",
      "DUKE OF AUMERLE:\n",
      "Where be thy brother!\n",
      "Yet will I make my wars a little mightend of day,\n",
      "To comfort the when I parted with him?\n",
      "\n",
      "Second Servingman:\n",
      "Here's one a side, and damnaked, from the earth,\n",
      "And let me see the issue of thy memory,\n",
      "To extraught this to his house, service\n",
      "To this female that down before: he doth shall some\n",
      "As those two eyes could give vent,\n",
      "Thy head of march with following hurts,\n",
      "And seem to be satisfied. We are the power\n",
      "Have swer'd, in forgiveness pardon all,\n",
      "And fearing ourselves, whose ribs are behold\n",
      "Is all to things extremity; but, in this view,\n",
      "And beat of wings forth great present death;\n",
      "My proper befal in thee, and thy bed;\n",
      "Thy beauty hath made me a power wear;\n",
      "For I would not stir a schoolmaster\n",
      "No, nor showers lambs about to be cordered,\n",
      "No crossing to kill thy warlike and means\n",
      "Both ransom the end of thy proper loins,\n",
      "The first o' the common ears: you might have me\n",
      "Begun, and so I'll be revenged on my side,\n",
      "And sleep out.\n",
      "\n",
      "LEONTES:\n",
      "How blessed be you?\n",
      "\n",
      "First Lord:\n",
      "I am great All tongues call'd me, do me king.\n",
      "\n",
      "PAULINA:\n",
      "If the law had been so brief wars to like a cold\n",
      "That he shall deposed, make the contrarious instrument.\n",
      "Stand fast; and we cannot mine enemy\n",
      "Because some owes that sad time so severe,\n",
      "And I for a wild contract my good friendly,\n",
      "And I'll be revenged for that enemies?\n",
      "O, coward! my son is your need of the cause,\n",
      "Why I should every day to lift our dial daughter:\n",
      "I saw the business in a house of them,\n",
      "Which now my other successive from breath. Prodigious but\n",
      "these greatness of his entertainments have redemption.\n",
      "\n",
      "MARCIUS:\n",
      "Come I too fond too, signified; no more remains,\n",
      "But thou shalt be blunt and not import of it.\n",
      "\n",
      "MISTRESS OVERDONE:\n",
      "Why, here be these gallants? who's at her that?\n",
      "\n",
      "LORD ROSS:\n",
      "A gentleman and a Roman: a weeping-will, words at home,\n",
      "Which five to cross my unto his father's falcon's tears,\n",
      "And bare an ill-brazed the enjoy'd:\n",
      "And with his own life or two kings did better:\n",
      "And thus far I resign to his with grief;\n",
      "It is the is the supposed king, the torches through\n",
      "And noble to be Richard in such a name\n",
      "This vault and so disgraced, and so did by\n",
      "The provost knowing the Duke of Norfolk, Thomas Mowbray?\n",
      "\n",
      "JOHN OF GAUNT:\n",
      "He that hath from Aufidius the greatest help!\n",
      "More of this own living reward, I charge thee,\n",
      "O, tenderly dear, and help to demand.\n",
      "\n",
      "FLORIZEL:\n",
      "I do;\n",
      "But, will not say 'tis now.\n",
      "\n",
      "LEONTES:\n",
      "Ha!\n",
      "\n",
      "ANTIGONUS:\n",
      "Widow, wife, you and your dancing. Stand, farewell.\n",
      "The most you have mutinous saying 'God save King Richard lewd\n",
      "I am too with that brother Clarence to your house:\n",
      "I'll try him dread left him down then before.\n",
      "\n",
      "GLOUCESTER:\n",
      "My lord of Ely!\n",
      "\n",
      "LADY ANNE:\n",
      "No fault on us,\n",
      "I hope to seek the day; I know 'tis not which\n",
      "Take into my prison, which else would laugh\n",
      "In secord, even such less as you, in a despite of\n",
      "you; you are heard of your life, a word, you\n",
      "shall should encounter you him.\n",
      "\n",
      "CORIOLANUS:\n",
      "Who should that do you for my men. I have\n",
      "not say 'Be blest you to marry,' and believe me\n",
      "how I may, but you will take it not.\n",
      "\n",
      "LEONTES:\n",
      "Though I am banish'd,\n",
      "Before this time we rejoice in thy triumph!\n",
      "\n",
      "First Senator:\n",
      "Farewell.\n",
      "\n",
      "SICINIUS:\n",
      "How! Was it we? we like to that?\n",
      "\n",
      "MENENIUS:\n",
      "Is this the chief?\n",
      "\n",
      "Servant:\n",
      "Not in good time, when rebellion wishing fire,\n",
      "More than for this tumult but by cheeks.\n",
      "\n",
      "Second Gentleman:\n",
      "These words are swell appears: she is well: I say it is\n",
      "cheap to pity, and I beat thee in my drum,\n",
      "while to him me.\n",
      "\n",
      "LEONTES:\n",
      "His princess,\n",
      "Loved is call'd him 'une' pardon' for right,\n",
      "And weep with our parliament be there?\n",
      "What stirr'd her for a pedlar thing?\n",
      "Ha! look'd thou but drawn awhile: so pause, farewell.\n",
      "\n",
      "HENRY BOLINGBROKE:\n",
      "Good aunt, stand up; you do as think, good night!\n",
      "You are too rash, that a man's son should bring ourselves;\n",
      "And that he utters for ever? I must answer\n",
      "Thy fault thou shalt be feel: therefore, like a tackled stair;\n",
      "And what thou dost love in sand farewell.\n",
      "When I was your father's standment to old;\n",
      "And you find example shall prove you violent\n",
      "Where I may be consul; and no better it must not.\n",
      "\n",
      "First Lord:\n",
      "There have their verdict and marks their father.\n",
      "\n",
      "Second Murderer:\n",
      "O excellent devise! make a swain as on thee,\n",
      "That a word of comfort, revenge my son,\n",
      "Do some flie in this one thing, being in case of charge.\n",
      "\n",
      "BRUTUS:\n",
      "O neither, I can be gone.\n",
      "Then, since we both in this prison, girl?\n",
      "How shall this night, do not but trouble you.\n",
      "\n",
      "MENENIUS:\n",
      "This business, did you never die.\n",
      "\n",
      "VOLUMNIA:\n",
      "Now, good sir, here's a woman would be a fool\n",
      "to the vengeance of her! Vnice gold\n",
      "To partial her! A prince, a rat, a word? what news?\n",
      "Or was't we pardon with you, sir.\n",
      "\n",
      "SICINIUS:\n",
      "Shall not:\n",
      "Methinks I see them both,\n",
      "And the true soul of the bosom, not their lives,\n",
      "Nor a month to sacred that was so full of him.\n",
      "\n",
      "Clown:\n",
      "Hath he with himself will make a cuckold of me.\n",
      "\n",
      "LUCIO:\n",
      "O this lady for it, God, I know you for your judgment,\n",
      "And, both by compliment and my banishment.\n",
      "\n",
      "DUCHESS OF YORK:\n",
      "At what that is your suit?\n",
      "\n",
      "QUEEN ELIZABETH:\n",
      "At least, that was true love's resort in the dust.\n",
      "\n",
      "KING RICHARD III:\n",
      "Then is our garments are not as oft\n",
      "That hard the villain last I live,\n",
      "Since thou didst preserve her confessence and record\n",
      "As thou overy good meat; yet do I but swear it;\n",
      "And so you are interrupt up\n",
      "To give us to pay for a palm.\n",
      "\n",
      "Second Servingman:\n",
      "An he had been long a warrant for it: for the\n",
      "world it counts not the shame instrument. Go to you,\n",
      "you thrive me and talk of peace, yet we have\n",
      "From received and those bloody strokes of are hearing:\n",
      "Therefore, bow, as thou hast done were in so disdain,\n",
      "That my request, to strike him, but stol'n\n",
      "And part the destroy of his appear again,\n",
      "'Tis part and supper while 'twere to be landed:\n",
      "We cannot judgely for 't.\n",
      "\n",
      "ANTIGONUS:\n",
      "Nor with my soul\n",
      "Make way to now remember mine eyes, that does me heaven!\n",
      "\n",
      "LEONTES:\n",
      "My brother's blessed cousin!\n",
      "\n",
      "HERMIONE:\n",
      "Hermione,\n",
      "How does my hands, by the earth to me;\n",
      "And she cannot prove a speed with man.\n",
      "\n",
      "LEONTES:\n",
      "Hermione?\n",
      "\n",
      "PAULINA:\n",
      "Her ancient kind uncivil.\n",
      "\n",
      "LEONTES:\n",
      "How does this good people?\n",
      "\n",
      "First Citizen:\n",
      "We have voiced am contents: the rest that time\n",
      "With scruples some more foreign to light letters to his\n",
      "The way; for he that he will still stand in narrow,\n",
      "That stays upon your reason her head,\n",
      "Should ask you that are our life,\n",
      "So that every day expedition came by,\n",
      "The precious storators of a goddess,--which no black words\n",
      "tatter than in bosom nice that doth give\n",
      "Which her aid all for Rome about me.\n",
      "\n",
      "CORIOLANUS:\n",
      "Why, that's my sacred clap she was but when you are\n",
      "ever looked for ever.\n",
      "\n",
      "CORIOLANUS:\n",
      "What news?\n",
      "\n",
      "LARTIUS:\n",
      "He did,\n",
      "How has never did thee hang'd us unprucely?\n",
      "\n",
      "COMINIUS:\n",
      "On's blood time second Sicilia told me of this country.\n",
      "\n",
      "BRUTUS:\n",
      "Call't not a plot\n",
      "That thee.\n",
      "\n",
      "CORIOLANUS:\n",
      "Why? A dozen times cold into the sea?\n",
      "Where is the best? Where is the nature, because his son\n",
      "Should choose. I am much before him, and mine, I fear.\n",
      "How now, my hardy, she fair son?\n",
      "\n",
      "Post:\n",
      "Boldly, good sir.\n",
      "\n",
      "GONZALO:\n",
      "I come to your royal grace!\n",
      "\n",
      "ANTONIO:\n",
      "What hast thou, Lord Grey rest?\n",
      "\n",
      "GONZALO:\n",
      "The gates are open rich, the like unscape of high.\n",
      "\n",
      "SEBASTIAN:\n",
      "An it like your honour.\n",
      "\n",
      "ANTONIO:\n",
      "Sir, you may not stand;\n",
      "You may chance to have stay with me to stay.\n",
      "\n",
      "SEBASTIAN:\n",
      "As rich good as the manners hall my stirrup soul,\n",
      "Lerceive leave to live, hence, I'll be warranted of you;\n",
      "How couldst go woo thou find mad a time to keep\n",
      "The treason was set of his foot and father; there is,\n",
      "and turn you to fly; and which I from this one offence\n",
      "And may enter breathe there in store-house,\n",
      "To wield my father's sight, I must away;\n",
      "But to be quickly moved in night;\n",
      "The crown I give thee to those will find\n",
      "Thy mother wall I wot that on my daughter:\n",
      "I, would I were as school-maid, he may chance to speak.\n",
      "\n",
      "Widow:\n",
      "Come on, and fell; come, sit down: but I speak a word\n",
      "Which to make you good congealed to worse than you\n",
      "And worship that seem'd for you; I will not\n",
      "Look on me; how the lambs or statue where he is\n",
      "For these traitors have known and their births.\n",
      "\n",
      "MENENIUS:\n",
      "Nay, temperately; you have made fault like\n",
      "upon my grave, shows fair dukedom from his trance:\n",
      "But look you, sir, he must be an importuned.\n",
      "Stand to your father, by time obedience?\n",
      "To prove forgot your pardon? your fellows presently;\n",
      "If no more of than nature, I pray,\n",
      "Because he be sworn to that perish.\n",
      "Is he, Montague, sit you to the majesty,\n",
      "That lives were the mortal blood of thine;\n",
      "The comfort of my spirits,\n",
      "My name was call'd Vincentio;\n",
      "And therefore direction cast off the made\n",
      "Shall be member what an aired will I live.\n",
      "Where is my page? Go, vile Paulina;\n",
      "For I will writ thee how to \n"
     ]
    }
   ],
   "source": [
    "print(f\"\\033[1mGenerated Data : \\033[0m\\n\\n{y}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91e93e98",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This notebook shows that transformers can learn to generate not just word by word but can also go one step further and generate character by character. Generating poems character by character is a hard task. The model must learn to recognise characters from scratch. Sequences of characters must be joined together to form meaning full sentences. \n",
    "\n",
    "Self attention modules in transformer architecture learn to pay different amounts of \"attention\" to different words (here characters). This helps the model to learn effectively and hence perform well in langauge modelling tasks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pytorch] *",
   "language": "python",
   "name": "conda-env-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}