Lightpost Test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lightpost Demo\n",
    "Welcome! Here we demo some basic tasks done in Lightpost, a lightweight, automated neural networks training framework written on top of the incredible PyTorch framework. It is made with extensibility and ease of use in mind.\n",
    "\n",
    "**Contents**\n",
    "* Basic Classification task\n",
    "* Text Classification Task\n",
    "* Deconstructing Lightpost ```Engine```s and ```Datapipe```s\n",
    "* Using your own PyTorch models with Lightpost\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Classification Task\n",
    "Here we'll test on the iris dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lightpost.datapipe import Datapipe\n",
    "from lightpost.estimators import MLPClassifier\n",
    "from lightpost.engine import Engine\n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "d = load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the basic ```Datapipe``` pipeline from the ```lightpost.datapipe``` API to load our data. It takes in two arrays: an array of features, and an array of targets. Since it's the most basic pipeline, it doesn't perform any preprocessing which other pipelines perform. It will then construct an iterator of randomized batches using the data. We batch it in groups of 32.\n",
    "\n",
    "We'll also use the basic ```MLPClassifier``` from the ```lightpost.estimators``` API. It's basically a series of fully-connected layers with ReLU activations. We give it the input dimensions and output dimensions of the iris dataset, and give it two hidden layers with 128 neurons each.\n",
    "\n",
    "Then we'll construct our training engine. We give it our data pipeline, our model, a loss function, and an optimizer.\n",
    "\n",
    "We can opt to use **Tensorboard**, TensorFlow's visualization tool using the ```use_tensorboard``` option. This will create a logging directory called ```runs``` on the same directory as your working files. To run Tensorboard, run ```tensorboard --logdir runs``` in a command line interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Datapipe(d.data, d.target, batch_size=32)\n",
    "model = MLPClassifier(input_dim=4, hidden_dim=128, output_dim=3, num_layers=2)\n",
    "engine = Engine(pipeline=pipe, model=model, criterion='cross_entropy', optimizer='adam', use_tensorboard=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll train the model using the engine by invoking the ```engine.fit()``` method. We train it for 1000 epochs and print the logs every 100 iterations. We also disable tqdm (the progress bars) so it doesn't overload our window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   100 | Train Loss 1.0537 | Train Acc 0.3438 | Val Loss 1.0853 | Val Acc 0.2240\n",
      "Epoch   200 | Train Loss 0.9970 | Train Acc 0.6562 | Val Loss 1.0201 | Val Acc 0.6458\n",
      "Epoch   300 | Train Loss 0.9270 | Train Acc 0.6562 | Val Loss 0.9430 | Val Acc 0.5938\n",
      "Epoch   400 | Train Loss 0.8650 | Train Acc 0.6641 | Val Loss 0.8377 | Val Acc 0.7448\n",
      "Epoch   500 | Train Loss 0.8255 | Train Acc 0.8672 | Val Loss 0.8062 | Val Acc 0.8073\n",
      "Epoch   600 | Train Loss 0.7927 | Train Acc 0.9688 | Val Loss 0.7889 | Val Acc 0.9844\n",
      "Epoch   700 | Train Loss 0.7790 | Train Acc 0.9766 | Val Loss 0.7666 | Val Acc 1.0000\n",
      "Epoch   800 | Train Loss 0.7652 | Train Acc 0.9375 | Val Loss 0.7482 | Val Acc 0.9844\n",
      "Epoch   900 | Train Loss 0.7438 | Train Acc 0.9766 | Val Loss 0.7395 | Val Acc 1.0000\n",
      "Epoch  1000 | Train Loss 0.7331 | Train Acc 0.9609 | Val Loss 0.7324 | Val Acc 0.9844\n"
     ]
    }
   ],
   "source": [
    "engine.fit(epochs=1000, print_every=100, disable_tqdm=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use our model to predict values. Here we'll use the ```engine.predict()``` method to make inferences from the entire iris dataset (which is stored in ```pipe.X```). The first five outputs are shown here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([150, 3])\n",
      "tensor([[0.9877, 0.0520, 0.0012],\n",
      "        [0.9788, 0.0789, 0.0023],\n",
      "        [0.9828, 0.0651, 0.0020],\n",
      "        [0.9761, 0.0842, 0.0027],\n",
      "        [0.9881, 0.0501, 0.0012]], grad_fn=<SliceBackward>)\n"
     ]
    }
   ],
   "source": [
    "predictions = engine.predict(pipe.X)\n",
    "print(predictions.shape)\n",
    "print(predictions[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also evaluate the model on a holdout testing set as needed using the ```engine.evaluate()``` method. For illustration, we'll use the entire iris dataset in this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Loss: 0.7283 | Total Accuracy: 0.6667\n"
     ]
    }
   ],
   "source": [
    "loss, acc = engine.evaluate_model(pipe.X, pipe.y)\n",
    "print('Total Loss: {:.4f} | Total Accuracy: {:.4f}'.format(loss, acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Text Classification\n",
    "For this example, we'll perform a simple text classification task on a subset of the Quora Insincere Comments dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lightpost.datapipe import Textpipe\n",
    "from lightpost.estimators import LSTMClassifier\n",
    "from lightpost.engine import Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For NLP tasks, we use a ```Textpipe``` from the ```lightpost.datapipe``` API. This pipeline will automatically do all the preprocessing for the dataset (tokenization, padding, truncation, vocabulary building, etc) as well as building the word embeddings, if one is provided. All corpus information are saved within the pipeline for use later on as needed. Please see the documentation (by typing ```Textpipe?``` in a Jupyter notebook cell) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 10000/10000 [00:00<00:00, 114142.21it/s]\n",
      "100%|██████████| 10000/10000 [00:00<00:00, 321222.92it/s]\n",
      "100%|██████████| 10000/10000 [00:00<00:00, 92524.95it/s]\n"
     ]
    }
   ],
   "source": [
    "pipe = Textpipe(path='quora/train_mini.csv', text='question_text', target='target', maxlen=50, \n",
    "                pretrained_embeddings=True, embed_path='quora/wiki.en.vec', embed_dim=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use an LSTM-based sequence classifier provided by the ```lightpost.estimators``` API. We'll pass it the embedding layer constructed by the pipeline, and give it 128 hidden units in it's one hidden layer. The ```LSTMClassifier``` is very flexible in terms of customization. For this example, we'll make it stack two recurrent, bidirectional layers, each with 0.2 recurrent dropout rate. The fully-connected layer will have a dropout of 0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LSTMClassifier(pretrained=pipe.embedding, embedding_dim=pipe.embed_dim, \n",
    "                       hidden_dim=128, output_dim=2, bidirectional=True, recur_layers=2, \n",
    "                       recur_dropout=0.2, dropout=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We construct the training engine by passing the pipeline and the model. In this example, we set a learning rate scheduer to decay the learning rate as the validation loss plateaus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine = Engine(pipeline=pipe, model=model, criterion='cross_entropy', optimizer='adam', scheduler='plateau')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training is done using the ```engine.fit()``` method. We train for five epochs and print the logs every 1 iteration. We also enable the progress bars, which will show the progress of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 235/235 [01:08<00:00,  3.95it/s]\n",
      "100%|██████████| 79/79 [00:06<00:00, 12.43it/s]\n",
      "  0%|          | 0/235 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch     1 | Train Loss 0.6931 | Train Acc 0.5043 | Val Loss 0.6931 | Val Acc 0.5103\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 235/235 [01:09<00:00,  3.92it/s]\n",
      "100%|██████████| 79/79 [00:06<00:00, 12.37it/s]\n",
      "  0%|          | 0/235 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch     2 | Train Loss 0.6932 | Train Acc 0.5049 | Val Loss 0.6930 | Val Acc 0.5162\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 235/235 [01:07<00:00,  3.95it/s]\n",
      "100%|██████████| 79/79 [00:06<00:00, 12.50it/s]\n",
      "  0%|          | 0/235 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch     3 | Train Loss 0.6720 | Train Acc 0.5627 | Val Loss 0.5991 | Val Acc 0.6962\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 235/235 [01:08<00:00,  3.86it/s]\n",
      "100%|██████████| 79/79 [00:06<00:00, 12.35it/s]\n",
      "  0%|          | 0/235 [00:00<?, ?it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch     4 | Train Loss 0.5824 | Train Acc 0.7192 | Val Loss 0.5649 | Val Acc 0.7555\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 235/235 [01:08<00:00,  3.94it/s]\n",
      "100%|██████████| 79/79 [00:06<00:00, 12.38it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch     5 | Train Loss 0.4926 | Train Acc 0.7805 | Val Loss 0.4798 | Val Acc 0.7860\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "engine.fit(epochs=5, print_every=1, disable_tqdm=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, we'll make inferences on some data. For this example, we'll feed the first five sequences of the dataset into the model to get it's log-softmax predictions (the ```LSTMClassifier``` uses log softmax on it's final layer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([5, 2])\n",
      "tensor([[-0.1055, -2.3010],\n",
      "        [-0.5954, -0.8015],\n",
      "        [-2.0665, -0.1354],\n",
      "        [-0.7965, -0.5995],\n",
      "        [-0.5310, -0.8868]], grad_fn=<LogSoftmaxBackward>)\n"
     ]
    }
   ],
   "source": [
    "predictions = engine.predict(pipe.X[:5])\n",
    "print(predictions.shape)\n",
    "print(predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save the model, we use the ```engine.save_weights()``` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "engine.save_weights('model.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Deconstruction\n",
    "As a framework made up of APIs and wrappers, Lightpost's idioms can be deconstructed and used as needed. Here' we can see the contents of a Text Pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1872,  4528,  4035,  ...,  3471,  3471,  3471],\n",
       "        [11941, 11758, 10017,  ...,  3471,  3471,  3471],\n",
       "        [ 8593,  2863, 15093,  ...,  3471,  3471,  3471],\n",
       "        ...,\n",
       "        [ 2076, 15957, 16461,  ...,  3471,  3471,  3471],\n",
       "        [ 1872,  7954, 14372,  ...,  3471,  3471,  3471],\n",
       "        [ 2076,  9782,  4226,  ...,  3471,  3471,  3471]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0, 1, 1,  ..., 0, 1, 0])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The vocabulary dictionary (and it's reverse) is also stored in the pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 'finding',\n",
       " 1: 'gentlemen',\n",
       " 2: 'coordinating',\n",
       " 3: 'Indonesians',\n",
       " 4: 'Rights',\n",
       " 5: 'footage',\n",
       " 6: 'hefemales',\n",
       " 7: 'Wildlife',\n",
       " 8: 'molar',\n",
       " 9: 'metallurgy',\n",
       " 10: 'rumours',\n",
       " 11: 'artistic',\n",
       " 12: 'substitutes',\n",
       " 13: 'mentioning',\n",
       " 14: 'alluring',\n",
       " 15: 'budget',\n",
       " 16: 'BC',\n",
       " 17: 'organs',\n",
       " 18: 'dreamworld',\n",
       " 19: 'criminalizing',\n",
       " 20: 'Asharam',\n",
       " 21: 'filter',\n",
       " 22: 'stolen',\n",
       " 23: 'tonality',\n",
       " 24: 'solution',\n",
       " 25: 'Rajput',\n",
       " 26: 'Beach',\n",
       " 27: '2005',\n",
       " 28: 'keep',\n",
       " 29: 'technically',\n",
       " 30: 'parenting',\n",
       " 31: 'involve',\n",
       " 32: 'Tirupathi',\n",
       " 33: 'CA',\n",
       " 34: 'sperm',\n",
       " 35: 'comfort',\n",
       " 36: 'spina',\n",
       " 37: 'States',\n",
       " 38: 'Gilgit',\n",
       " 39: 'esteem',\n",
       " 40: 'limiting',\n",
       " 41: 'disrespect',\n",
       " 42: 'hindni',\n",
       " 43: 'savvy',\n",
       " 44: 'obligations',\n",
       " 45: 'now',\n",
       " 46: 'consciousness',\n",
       " 47: 'cut',\n",
       " 48: 'elmo',\n",
       " 49: 'voluntarily',\n",
       " 50: 'switched',\n",
       " 51: 'drafted',\n",
       " 52: 'head',\n",
       " 53: 'earliest',\n",
       " 54: 'harm',\n",
       " 55: 'Eurocentrics',\n",
       " 56: 'Vegas',\n",
       " 57: 'lawfully',\n",
       " 58: 'benchmark',\n",
       " 59: 'secularism',\n",
       " 60: 'GoFundMe',\n",
       " 61: 'Good',\n",
       " 62: 'assistance',\n",
       " 63: 'mainstream',\n",
       " 64: 'allowed',\n",
       " 65: 'preparing',\n",
       " 66: 'molesting',\n",
       " 67: 'Reynes',\n",
       " 68: 'overpopulation',\n",
       " 69: 'TOTAL',\n",
       " 70: 'Rabindra',\n",
       " 71: 'associate',\n",
       " 72: 'MKS',\n",
       " 73: 'Vegeta',\n",
       " 74: 'Machedo',\n",
       " 75: 'fashion',\n",
       " 76: 'genitals',\n",
       " 77: 'slime',\n",
       " 78: 'Bowie',\n",
       " 79: 'truly',\n",
       " 80: 'ratio',\n",
       " 81: 'Quick',\n",
       " 82: 'PA',\n",
       " 83: 'fiction',\n",
       " 84: 'Ramsay',\n",
       " 85: 'pupils',\n",
       " 86: 'Baratheon',\n",
       " 87: 'myself',\n",
       " 88: 'senate',\n",
       " 89: 'unsuccesful',\n",
       " 90: 'pets',\n",
       " 91: 'DOJ',\n",
       " 92: 'unsaved',\n",
       " 93: 'whiteness',\n",
       " 94: 'flashing',\n",
       " 95: 'ACM',\n",
       " 96: 'worshippers',\n",
       " 97: 'riled',\n",
       " 98: 'auspicious',\n",
       " 99: 'Istanbul',\n",
       " 100: 'Anatolian',\n",
       " 101: 'southern',\n",
       " 102: 'scheduled',\n",
       " 103: 'Venezuela',\n",
       " 104: 'volunteered',\n",
       " 105: 'firm',\n",
       " 106: 'parameter',\n",
       " 107: 'protesters',\n",
       " 108: 'main',\n",
       " 109: 'Venkaiah',\n",
       " 110: 'premises',\n",
       " 111: 'toy',\n",
       " 112: 'excessive',\n",
       " 113: 'noticed',\n",
       " 114: 'wears',\n",
       " 115: 'evolutionists',\n",
       " 116: 'BA',\n",
       " 117: 'tone',\n",
       " 118: 'Aadhar',\n",
       " 119: 'Punjabi',\n",
       " 120: 'trader',\n",
       " 121: 'answred',\n",
       " 122: 'GO',\n",
       " 123: 'victimhood',\n",
       " 124: 'button',\n",
       " 125: 'alternate',\n",
       " 126: 'Siddaramaiah',\n",
       " 127: 'canvas',\n",
       " 128: 'believer',\n",
       " 129: 'followers',\n",
       " 130: 'cinemagraphs',\n",
       " 131: 'DBZ',\n",
       " 132: 'Studio',\n",
       " 133: 'aspie',\n",
       " 134: 'psychopath',\n",
       " 135: 'play',\n",
       " 136: 'vaccine',\n",
       " 137: 'polarise',\n",
       " 138: 'losers',\n",
       " 139: 'Bataclan',\n",
       " 140: 'wells',\n",
       " 141: 'mini',\n",
       " 142: 'match',\n",
       " 143: 'slavic',\n",
       " 144: 'Jadavpur',\n",
       " 145: 'Bajaj',\n",
       " 146: 'Empire',\n",
       " 147: 'teeth',\n",
       " 148: 'corals',\n",
       " 149: 'wounds',\n",
       " 150: 'chakra',\n",
       " 151: 'apps',\n",
       " 152: 'Economics',\n",
       " 153: 'Ect',\n",
       " 154: 'both',\n",
       " 155: 'republic',\n",
       " 156: 'toil',\n",
       " 157: 'Millenial',\n",
       " 158: 'Somali',\n",
       " 159: 'intent',\n",
       " 160: 'Kevin',\n",
       " 161: 'load',\n",
       " 162: 'applicate',\n",
       " 163: 'caliphate',\n",
       " 164: 'lover',\n",
       " 165: 'economy',\n",
       " 166: 'Franks',\n",
       " 167: 'Brampton',\n",
       " 168: 'For',\n",
       " 169: 'beautiful',\n",
       " 170: 'leaning',\n",
       " 171: 'Northwich',\n",
       " 172: 'riding',\n",
       " 173: 'Musk',\n",
       " 174: 'subconscious',\n",
       " 175: 'vintage',\n",
       " 176: 'effort',\n",
       " 177: 'Vault',\n",
       " 178: 'kyrgastan',\n",
       " 179: 'highest',\n",
       " 180: 'tanking',\n",
       " 181: 'advancement',\n",
       " 182: 'compound',\n",
       " 183: 'irresponsible',\n",
       " 184: 'clothes',\n",
       " 185: 'straightening',\n",
       " 186: 'in',\n",
       " 187: 'campaign',\n",
       " 188: 'coach',\n",
       " 189: 'upto',\n",
       " 190: '42',\n",
       " 191: 'enriching',\n",
       " 192: 'bishnois',\n",
       " 193: 'girly',\n",
       " 194: 'pies',\n",
       " 195: 'uninstalled',\n",
       " 196: 'MLP',\n",
       " 197: 'platfrom',\n",
       " 198: 'fulfil',\n",
       " 199: 'cant',\n",
       " 200: 'Mint',\n",
       " 201: 'Ghoul',\n",
       " 202: 'ahead',\n",
       " 203: 'Iranians',\n",
       " 204: 'total',\n",
       " 205: 'communist',\n",
       " 206: 'Sonos',\n",
       " 207: 'Gitmo',\n",
       " 208: 'saluting',\n",
       " 209: 'tits',\n",
       " 210: 'Trivandrum',\n",
       " 211: 'Madra',\n",
       " 212: 'work',\n",
       " 213: 'Arabia',\n",
       " 214: 'bigotry',\n",
       " 215: 'software',\n",
       " 216: 'forget',\n",
       " 217: 'lawless',\n",
       " 218: 'photosynthesis',\n",
       " 219: 'resent',\n",
       " 220: 'CEO',\n",
       " 221: 'Wars',\n",
       " 222: 'disqualify',\n",
       " 223: 'confederate',\n",
       " 224: 'honey',\n",
       " 225: 'targets',\n",
       " 226: 'step',\n",
       " 227: 'obsession',\n",
       " 228: 'Ground',\n",
       " 229: 'cheerful',\n",
       " 230: 'sycophant',\n",
       " 231: 'Bangalore',\n",
       " 232: 'DPRK',\n",
       " 233: 'm9760in',\n",
       " 234: '15n',\n",
       " 235: 'ambitious',\n",
       " 236: 'waging',\n",
       " 237: 'applying',\n",
       " 238: 'BELIEVE',\n",
       " 239: 'outrage',\n",
       " 240: 'respectfully',\n",
       " 241: '1916',\n",
       " 242: 'Parks',\n",
       " 243: 'embarrased',\n",
       " 244: 'COULD',\n",
       " 245: 'Rockies',\n",
       " 246: 'downloading',\n",
       " 247: 'WWII',\n",
       " 248: 'strike',\n",
       " 249: 'descended',\n",
       " 250: 'talk',\n",
       " 251: 'amongst',\n",
       " 252: 'radio',\n",
       " 253: 'ECT',\n",
       " 254: 'obnoxious',\n",
       " 255: 'preferably',\n",
       " 256: 'shouts',\n",
       " 257: 'SUGAR',\n",
       " 258: 'plane',\n",
       " 259: 'low',\n",
       " 260: 'duly',\n",
       " 261: 'collaborators',\n",
       " 262: 'but',\n",
       " 263: 'security',\n",
       " 264: 'cheesy',\n",
       " 265: 'Naukri',\n",
       " 266: 'Tyrion',\n",
       " 267: 'EMR',\n",
       " 268: 'news',\n",
       " 269: 'vibrator',\n",
       " 270: 'knowing',\n",
       " 271: 'admission',\n",
       " 272: 'challenge',\n",
       " 273: 'Patels',\n",
       " 274: 'teachers',\n",
       " 275: 'customize',\n",
       " 276: 'Denmark',\n",
       " 277: 'Angular',\n",
       " 278: 'spite',\n",
       " 279: 'friendliness',\n",
       " 280: 'Illuminism',\n",
       " 281: 'Bahadur',\n",
       " 282: 'attka',\n",
       " 283: 'phonetics',\n",
       " 284: 'selling',\n",
       " 285: 'reasoned',\n",
       " 286: 'Ministry',\n",
       " 287: 'weakened',\n",
       " 288: 'Europians',\n",
       " 289: 'shcool',\n",
       " 290: 'see',\n",
       " 291: 'batteries',\n",
       " 292: 'vandals',\n",
       " 293: 'alignment',\n",
       " 294: 'caught',\n",
       " 295: 'invento',\n",
       " 296: 'Soil',\n",
       " 297: 'officials',\n",
       " 298: 'colonists',\n",
       " 299: 'Lalu',\n",
       " 300: 'halmet',\n",
       " 301: 'CPEC',\n",
       " 302: 'Rwandan',\n",
       " 303: 'SWAT',\n",
       " 304: 'distracting',\n",
       " 305: 'silenced',\n",
       " 306: 'file',\n",
       " 307: 'ZsvGl',\n",
       " 308: 'motion',\n",
       " 309: 'original',\n",
       " 310: 'aid',\n",
       " 311: 'neither',\n",
       " 312: 'stunts',\n",
       " 313: 'consdier',\n",
       " 314: 'avenge',\n",
       " 315: 'creampie',\n",
       " 316: 'skeptics',\n",
       " 317: 'Millennials',\n",
       " 318: 'seductions',\n",
       " 319: '2010',\n",
       " 320: 'engineer',\n",
       " 321: 'FFFF0',\n",
       " 322: 'feral',\n",
       " 323: 'crusading',\n",
       " 324: 'seeds',\n",
       " 325: 'mediocrity',\n",
       " 326: 'icing',\n",
       " 327: 'annoyed',\n",
       " 328: 'creator',\n",
       " 329: 'Because',\n",
       " 330: 'satanist',\n",
       " 331: 'freshwater',\n",
       " 332: 'provider',\n",
       " 333: 'BITS',\n",
       " 334: '003',\n",
       " 335: 'weakest',\n",
       " 336: 'corazon',\n",
       " 337: 'Afghans',\n",
       " 338: 'History',\n",
       " 339: 'rishis',\n",
       " 340: 'siamese',\n",
       " 341: 'Kahala',\n",
       " 342: 'fortunate',\n",
       " 343: 'drunkards',\n",
       " 344: 'copies',\n",
       " 345: 'turks',\n",
       " 346: 'punk',\n",
       " 347: 'Just',\n",
       " 348: 'Siddiqui',\n",
       " 349: 'edible',\n",
       " 350: 'honorary',\n",
       " 351: 'Trinity',\n",
       " 352: 'realized',\n",
       " 353: 'heats',\n",
       " 354: 'bum',\n",
       " 355: 'president',\n",
       " 356: 'equipment',\n",
       " 357: 'easily',\n",
       " 358: 'prevents',\n",
       " 359: 'obey',\n",
       " 360: 'Gujarati',\n",
       " 361: 'LNMIIT',\n",
       " 362: 'seduce',\n",
       " 363: 'vacate',\n",
       " 364: 'confederacy',\n",
       " 365: 'FB',\n",
       " 366: 'EAD',\n",
       " 367: 'matrix',\n",
       " 368: 'minded',\n",
       " 369: 'kilogram',\n",
       " 370: 'shake',\n",
       " 371: 'ObamaCare',\n",
       " 372: 'honkongers',\n",
       " 373: 'NEVER',\n",
       " 374: 'Brexiteer',\n",
       " 375: 'engines',\n",
       " 376: '1960s',\n",
       " 377: 'yr10',\n",
       " 378: 'Burka',\n",
       " 379: 'Indiana',\n",
       " 380: 'graduating',\n",
       " 381: 'Priyanka',\n",
       " 382: 'Utah',\n",
       " 383: 'misconduct',\n",
       " 384: 'slaked',\n",
       " 385: 'again',\n",
       " 386: 'recommendation',\n",
       " 387: 'Peter',\n",
       " 388: 'maximum',\n",
       " 389: 'pant',\n",
       " 390: 'nearest',\n",
       " 391: 'excel',\n",
       " 392: 'Magnetic',\n",
       " 393: 'inclusion',\n",
       " 394: 'Trek',\n",
       " 395: 'Equestria',\n",
       " 396: 'MY',\n",
       " 397: 'beta',\n",
       " 398: 'volume',\n",
       " 399: 'fucked',\n",
       " 400: 'patients',\n",
       " 401: 'regain',\n",
       " 402: 'gadget',\n",
       " 403: 'EVMs',\n",
       " 404: 'percentage',\n",
       " 405: 'fundamentalists',\n",
       " 406: 'overused',\n",
       " 407: 'jewelry',\n",
       " 408: 'lock',\n",
       " 409: 'ideas',\n",
       " 410: 'CUNT',\n",
       " 411: 'Ayuthaya',\n",
       " 412: 'folks',\n",
       " 413: 'tennis',\n",
       " 414: 'district',\n",
       " 415: 'isle',\n",
       " 416: 'incentive',\n",
       " 417: 'humans',\n",
       " 418: 'legacies',\n",
       " 419: 'lawns',\n",
       " 420: 'shit',\n",
       " 421: 'stress',\n",
       " 422: 'kicks',\n",
       " 423: 'archaeological',\n",
       " 424: 'MEDIA',\n",
       " 425: 'damn',\n",
       " 426: 'problems',\n",
       " 427: 'chart',\n",
       " 428: 'decorations',\n",
       " 429: 'throwing',\n",
       " 430: 'substituted',\n",
       " 431: 'payments',\n",
       " 432: 'nor',\n",
       " 433: 'upset',\n",
       " 434: 'uniting',\n",
       " 435: 'boycotted',\n",
       " 436: '360',\n",
       " 437: 'prize',\n",
       " 438: 'rams',\n",
       " 439: 'Visa',\n",
       " 440: 'massive',\n",
       " 441: 'taking',\n",
       " 442: 'dildo',\n",
       " 443: '24fps',\n",
       " 444: 'imprison',\n",
       " 445: 'conditioning',\n",
       " 446: 'lestinians',\n",
       " 447: 'institute',\n",
       " 448: 'Issac',\n",
       " 449: 'winged',\n",
       " 450: 'ages',\n",
       " 451: 'BSc',\n",
       " 452: 'Brooke',\n",
       " 453: 'gerrymandered',\n",
       " 454: 'countries',\n",
       " 455: 'defecation',\n",
       " 456: 'rent',\n",
       " 457: 'countless',\n",
       " 458: 'Quran',\n",
       " 459: 'tuition',\n",
       " 460: 'jazz',\n",
       " 461: 'controlled',\n",
       " 462: 'apples',\n",
       " 463: 'erosion',\n",
       " 464: 'themselfs',\n",
       " 465: 'bullying',\n",
       " 466: 'lane',\n",
       " 467: 'counterfeiting',\n",
       " 468: 'sanin',\n",
       " 469: '3x',\n",
       " 470: 'NEWS',\n",
       " 471: 'Non',\n",
       " 472: 'casserole',\n",
       " 473: 'heterosexual',\n",
       " 474: 'Guwahati',\n",
       " 475: 'kidnapped',\n",
       " 476: 'anticodon',\n",
       " 477: 'ruled',\n",
       " 478: 'bravery',\n",
       " 479: 'supported',\n",
       " 480: 'Today',\n",
       " 481: 'surname',\n",
       " 482: 'strict',\n",
       " 483: 'tasks',\n",
       " 484: 'micelles',\n",
       " 485: 'derivative',\n",
       " 486: 'bribery',\n",
       " 487: 'Maz',\n",
       " 488: 'login',\n",
       " 489: 'Kurdish',\n",
       " 490: 'font',\n",
       " 491: 'forbidding',\n",
       " 492: 'autocorrect',\n",
       " 493: 'bitchy',\n",
       " 494: 'Christianty',\n",
       " 495: 'oral',\n",
       " 496: 'Encompass',\n",
       " 497: 'gullible',\n",
       " 498: 'skies',\n",
       " 499: 'objective',\n",
       " 500: 'Sphynx',\n",
       " 501: 'inferiority',\n",
       " 502: 'Defoe',\n",
       " 503: 'invention',\n",
       " 504: 'careless',\n",
       " 505: 'tags',\n",
       " 506: 'Other',\n",
       " 507: 'testicles',\n",
       " 508: 'Rohigya',\n",
       " 509: 'Awami',\n",
       " 510: 'rallies',\n",
       " 511: 'careing',\n",
       " 512: 'scripts',\n",
       " 513: 'toss',\n",
       " 514: 'dustbin',\n",
       " 515: 'gay',\n",
       " 516: 'while',\n",
       " 517: 'EKG',\n",
       " 518: 'BoM',\n",
       " 519: 'Rodham',\n",
       " 520: 'shares',\n",
       " 521: 'Parisian',\n",
       " 522: 'bcos',\n",
       " 523: 'differentiate',\n",
       " 524: 'Wipro',\n",
       " 525: 'acetylcholine',\n",
       " 526: 'brunettes',\n",
       " 527: 'disorders',\n",
       " 528: 'document',\n",
       " 529: 'electroluminescent',\n",
       " 530: 'petrol',\n",
       " 531: 'spread',\n",
       " 532: 'Jews',\n",
       " 533: 'families',\n",
       " 534: 'event',\n",
       " 535: 'rebel',\n",
       " 536: 'discusting',\n",
       " 537: 'civil',\n",
       " 538: 'projectile',\n",
       " 539: 'pages',\n",
       " 540: 'Nah',\n",
       " 541: 'being',\n",
       " 542: 'Dalits',\n",
       " 543: 'Claymore',\n",
       " 544: 'Chistians',\n",
       " 545: 'expects',\n",
       " 546: 'libtards',\n",
       " 547: 'appreciate',\n",
       " 548: 'equations',\n",
       " 549: 'pretty',\n",
       " 550: 'assume',\n",
       " 551: 'impossible',\n",
       " 552: 'archers',\n",
       " 553: 'Latinos',\n",
       " 554: 'handmade',\n",
       " 555: 'settling',\n",
       " 556: 'blonde',\n",
       " 557: 'L1',\n",
       " 558: 'filters',\n",
       " 559: 'insurance',\n",
       " 560: 'Cluster',\n",
       " 561: 'vinegar',\n",
       " 562: 'douche',\n",
       " 563: 'Aurangabad',\n",
       " 564: 'Homeland',\n",
       " 565: 'mylife',\n",
       " 566: 'formalties',\n",
       " 567: 'Italians',\n",
       " 568: 'Malu',\n",
       " 569: 'drink',\n",
       " 570: 'outside',\n",
       " 571: 'Chili',\n",
       " 572: 'MP',\n",
       " 573: 'tour',\n",
       " 574: 'nominations',\n",
       " 575: 'Overwatch',\n",
       " 576: 'assumption',\n",
       " 577: 'vascular',\n",
       " 578: 'Tanishq',\n",
       " 579: 'bitches',\n",
       " 580: 'Batty',\n",
       " 581: 'miss',\n",
       " 582: 'marked',\n",
       " 583: 'superstition',\n",
       " 584: 'grannies',\n",
       " 585: 'niches',\n",
       " 586: 'evoke',\n",
       " 587: 'trannies',\n",
       " 588: 'unequal',\n",
       " 589: 'similarities',\n",
       " 590: 'taproot',\n",
       " 591: 'Mace',\n",
       " 592: 'Japanese',\n",
       " 593: 'Pryor',\n",
       " 594: 'originally',\n",
       " 595: 'Gases',\n",
       " 596: 'displayed',\n",
       " 597: 'Puri',\n",
       " 598: 'flirtatious',\n",
       " 599: 'battleship',\n",
       " 600: 'Cr',\n",
       " 601: 'Stratocaster',\n",
       " 602: 'Carolina',\n",
       " 603: 'ICSE',\n",
       " 604: 'selfies',\n",
       " 605: 'constraint',\n",
       " 606: 'Am',\n",
       " 607: 'consecutive',\n",
       " 608: 'constructed',\n",
       " 609: 'argue',\n",
       " 610: 'apostates',\n",
       " 611: 'violating',\n",
       " 612: 'me',\n",
       " 613: 'Germans',\n",
       " 614: 'things',\n",
       " 615: 'Palestinians',\n",
       " 616: 'spreading',\n",
       " 617: 'exent',\n",
       " 618: 'generated',\n",
       " 619: 'KM',\n",
       " 620: 'Dance',\n",
       " 621: 'Bacardi',\n",
       " 622: 'vital',\n",
       " 623: 'shut',\n",
       " 624: 'taught',\n",
       " 625: '9050',\n",
       " 626: 'vomiting',\n",
       " 627: 'possibly',\n",
       " 628: 'Summit',\n",
       " 629: 'enemy',\n",
       " 630: 'up',\n",
       " 631: 'Thank',\n",
       " 632: 'thankless',\n",
       " 633: 'srilankan',\n",
       " 634: 'gaga',\n",
       " 635: 'Bulls',\n",
       " 636: 'through',\n",
       " 637: 'labels',\n",
       " 638: 'manner',\n",
       " 639: 'infectied',\n",
       " 640: 'Holdrege',\n",
       " 641: 'Gonen',\n",
       " 642: 'unconditionally',\n",
       " 643: 'fur',\n",
       " 644: 'MMA',\n",
       " 645: 'maintain',\n",
       " 646: 'Their',\n",
       " 647: 'soviets',\n",
       " 648: 'dues',\n",
       " 649: 'Jones',\n",
       " 650: 'confused',\n",
       " 651: 'hoshiar',\n",
       " 652: 'backwards',\n",
       " 653: 'productive',\n",
       " 654: 'wait',\n",
       " 655: 'effigies',\n",
       " 656: 'hood',\n",
       " 657: 'dragons',\n",
       " 658: 'weeds',\n",
       " 659: 'GN',\n",
       " 660: 'fiancée',\n",
       " 661: 'Solaris',\n",
       " 662: 'houris',\n",
       " 663: 'measuring',\n",
       " 664: 'favourite',\n",
       " 665: 'AEF',\n",
       " 666: 'Sal',\n",
       " 667: 'When',\n",
       " 668: 'revise',\n",
       " 669: 'trait',\n",
       " 670: 'nails',\n",
       " 671: 'denying',\n",
       " 672: 'Kingdom',\n",
       " 673: 'marrow',\n",
       " 674: 'lunarians',\n",
       " 675: 'believing',\n",
       " 676: 'islam',\n",
       " 677: 'swinger',\n",
       " 678: 'nut',\n",
       " 679: 'neutrality',\n",
       " 680: 'establish',\n",
       " 681: 'spiral',\n",
       " 682: 'pointing',\n",
       " 683: 'vindictive',\n",
       " 684: 'later',\n",
       " 685: 'genderist',\n",
       " 686: 'Office',\n",
       " 687: 'summer',\n",
       " 688: 'blind',\n",
       " 689: 'paw',\n",
       " 690: 'axis',\n",
       " 691: 'pension',\n",
       " 692: 'belongs',\n",
       " 693: 'groups',\n",
       " 694: 'documentaries',\n",
       " 695: 'Dem',\n",
       " 696: 'XL',\n",
       " 697: 'Chennaites',\n",
       " 698: 'takes',\n",
       " 699: 'coil',\n",
       " 700: 'aisha',\n",
       " 701: 'Porsche',\n",
       " 702: 'DIY',\n",
       " 703: 'tip',\n",
       " 704: 'bomber',\n",
       " 705: 'signs',\n",
       " 706: 'Santos',\n",
       " 707: 'appellations',\n",
       " 708: 'startup',\n",
       " 709: 'odia',\n",
       " 710: 'Islam',\n",
       " 711: 'femi',\n",
       " 712: 'indigenous',\n",
       " 713: 'absorption',\n",
       " 714: 'lest',\n",
       " 715: 'Benares',\n",
       " 716: 'asylum',\n",
       " 717: 'perpetuate',\n",
       " 718: 'MAMC',\n",
       " 719: 'Dubai',\n",
       " 720: 'honorable',\n",
       " 721: 'AmarNath',\n",
       " 722: 'Give',\n",
       " 723: 'wealthiest',\n",
       " 724: 'Kylie',\n",
       " 725: 'learner',\n",
       " 726: 'Hasn',\n",
       " 727: 'solidify',\n",
       " 728: 'when',\n",
       " 729: 'predate',\n",
       " 730: 'prevented',\n",
       " 731: 'Area',\n",
       " 732: 'YYY',\n",
       " 733: 'Italy',\n",
       " 734: 'veterans',\n",
       " 735: 'armor',\n",
       " 736: 'dim',\n",
       " 737: 'likely',\n",
       " 738: 'UNO',\n",
       " 739: 'magazines',\n",
       " 740: 'McGowan',\n",
       " 741: 'reaches',\n",
       " 742: 'Income',\n",
       " 743: 'Scythian',\n",
       " 744: 'splitsvilla',\n",
       " 745: 'stressful',\n",
       " 746: 'walkout',\n",
       " 747: 'enjoys',\n",
       " 748: 'distributed',\n",
       " 749: 'terrorized',\n",
       " 750: 'snobby',\n",
       " 751: '17',\n",
       " 752: 'nuclear',\n",
       " 753: 'Bollywood',\n",
       " 754: 'MTV',\n",
       " 755: 'attacks',\n",
       " 756: 'potentially',\n",
       " 757: 'duck',\n",
       " 758: 'electromagnetic',\n",
       " 759: 'unwelcome',\n",
       " 760: 'uncircumcised',\n",
       " 761: 'Malaysians',\n",
       " 762: 'biotechnology',\n",
       " 763: 'adivasis',\n",
       " 764: 'Sansa',\n",
       " 765: 'fecal',\n",
       " 766: 'Anglosphere',\n",
       " 767: 'misogynists',\n",
       " 768: 'normal',\n",
       " 769: 'clownfish',\n",
       " 770: 'modifier',\n",
       " 771: 'daydreamed',\n",
       " 772: 'confidently',\n",
       " 773: 'protests',\n",
       " 774: 'Industrial',\n",
       " 775: 'proposed',\n",
       " 776: 'hooker',\n",
       " 777: 'vermicompost',\n",
       " 778: 'letting',\n",
       " 779: 'TEDx',\n",
       " 780: 'travelling',\n",
       " 781: 'biologically',\n",
       " 782: 'iWatch',\n",
       " 783: 'nonlinear',\n",
       " 784: 'gunshots',\n",
       " 785: 'pools',\n",
       " 786: 'worried',\n",
       " 787: 'More',\n",
       " 788: '544',\n",
       " 789: 'indians',\n",
       " 790: 'champion',\n",
       " 791: 'stand',\n",
       " 792: 'school',\n",
       " 793: 'villages',\n",
       " 794: 'disappoint',\n",
       " 795: 'transfers',\n",
       " 796: 'ALICE',\n",
       " 797: 'uppity',\n",
       " 798: 'choices',\n",
       " 799: 'maids',\n",
       " 800: 'School',\n",
       " 801: 'deviance',\n",
       " 802: 'cube',\n",
       " 803: 'shaktas',\n",
       " 804: 'dlf',\n",
       " 805: 'theirs',\n",
       " 806: 'garage',\n",
       " 807: 'entirely',\n",
       " 808: 'Women',\n",
       " 809: 'scandals',\n",
       " 810: 'Ford',\n",
       " 811: 'codes',\n",
       " 812: 'MSE',\n",
       " 813: 'custody',\n",
       " 814: 'beard',\n",
       " 815: 'Moon',\n",
       " 816: 'RPN',\n",
       " 817: 'compelled',\n",
       " 818: 'xender',\n",
       " 819: 'Shang',\n",
       " 820: 'Calgary',\n",
       " 821: 'democracies',\n",
       " 822: 'leftism',\n",
       " 823: 'criticize',\n",
       " 824: 'humor',\n",
       " 825: 'shading',\n",
       " 826: 'usually',\n",
       " 827: 'deities',\n",
       " 828: 'aboriginals',\n",
       " 829: 'hospice',\n",
       " 830: 'feminine',\n",
       " 831: 'raciest',\n",
       " 832: 'quoras',\n",
       " 833: 'daycare',\n",
       " 834: 'behave',\n",
       " 835: 'douchey',\n",
       " 836: 'whereas',\n",
       " 837: 'sports',\n",
       " 838: 'getaway',\n",
       " 839: 'catholic',\n",
       " 840: 'confiscation',\n",
       " 841: 'carbonate',\n",
       " 842: 'masochist',\n",
       " 843: 'enlarged',\n",
       " 844: 'patch',\n",
       " 845: 'Arrowverse',\n",
       " 846: 'attend',\n",
       " 847: 'darker',\n",
       " 848: 'consideration',\n",
       " 849: 'brethren',\n",
       " 850: 'RC',\n",
       " 851: 'cbsc',\n",
       " 852: 'authenticate',\n",
       " 853: 'Yankees',\n",
       " 854: 'baned',\n",
       " 855: 'PCM',\n",
       " 856: 'equate',\n",
       " 857: 'LCM',\n",
       " 858: 'auspices',\n",
       " 859: 'conquistador',\n",
       " 860: 'laxative',\n",
       " 861: 'definition',\n",
       " 862: 'ISB',\n",
       " 863: 'remove',\n",
       " 864: 'fiddleheads',\n",
       " 865: 'welders',\n",
       " 866: 'impacted',\n",
       " 867: 'organized',\n",
       " 868: 'Re',\n",
       " 869: 'glue',\n",
       " 870: 'rental',\n",
       " 871: 'curriculum',\n",
       " 872: 'parent',\n",
       " 873: 'experiment',\n",
       " 874: 'tidal',\n",
       " 875: 'Oz',\n",
       " 876: 'presenting',\n",
       " 877: 'circumference',\n",
       " 878: 'noon',\n",
       " 879: 'violated',\n",
       " 880: 'scalar',\n",
       " 881: 'counterintuitive',\n",
       " 882: 'indentured',\n",
       " 883: 'tourists',\n",
       " 884: 'armrests',\n",
       " 885: 'alliance',\n",
       " 886: 'biased',\n",
       " 887: 'Selvan',\n",
       " 888: 'derogatory',\n",
       " 889: 'landed',\n",
       " 890: 'Net',\n",
       " 891: 'Kate',\n",
       " 892: 'abuser',\n",
       " 893: 'ck',\n",
       " 894: 'Oral',\n",
       " 895: 'jewconomy',\n",
       " 896: 'spectrum',\n",
       " 897: 'ignited',\n",
       " 898: 'massage',\n",
       " 899: 'deporting',\n",
       " 900: 'ruin',\n",
       " 901: 'bachelor',\n",
       " 902: 'pro',\n",
       " 903: 'rope',\n",
       " 904: 'excavating',\n",
       " 905: 'Rare',\n",
       " 906: 'Halloween',\n",
       " 907: 'unremoved',\n",
       " 908: 'classed',\n",
       " 909: 'Polestar',\n",
       " 910: 'condemning',\n",
       " 911: 'shipping',\n",
       " 912: 'judgement',\n",
       " 913: 'MBA',\n",
       " 914: 'Sad',\n",
       " 915: 'phone',\n",
       " 916: 'smug',\n",
       " 917: 'controversy',\n",
       " 918: '1400',\n",
       " 919: 'ghost',\n",
       " 920: 'congratulate',\n",
       " 921: 'destination',\n",
       " 922: 'therapy',\n",
       " 923: 'left',\n",
       " 924: 'wing',\n",
       " 925: 'Bank',\n",
       " 926: 'Walpole',\n",
       " 927: 'Simpson',\n",
       " 928: 'sincerely',\n",
       " 929: 'Romain',\n",
       " 930: 'retardation',\n",
       " 931: 'hairline',\n",
       " 932: '1945',\n",
       " 933: 'WordPress',\n",
       " 934: 'lately',\n",
       " 935: 'prod_',\n",
       " 936: 'promises',\n",
       " 937: 'witch',\n",
       " 938: '1951',\n",
       " 939: 'whistleblower',\n",
       " 940: 'PLC',\n",
       " 941: 'slut',\n",
       " 942: 'Boomer',\n",
       " 943: 'Rathi',\n",
       " 944: 'girlfriend',\n",
       " 945: 'Karna',\n",
       " 946: 'soft',\n",
       " 947: '115',\n",
       " 948: 'comparable',\n",
       " 949: 'feet',\n",
       " 950: 'joyful',\n",
       " 951: 'privilege',\n",
       " 952: '32',\n",
       " 953: 'wrestlers',\n",
       " 954: 'sucking',\n",
       " 955: 'tissue',\n",
       " 956: 'majored',\n",
       " 957: 'Aquaba',\n",
       " 958: 'nd',\n",
       " 959: 'cmd',\n",
       " 960: 'degradation',\n",
       " 961: 'grad',\n",
       " 962: 'Battlefield',\n",
       " 963: 'referring',\n",
       " 964: 'Baltistan',\n",
       " 965: 'Pets',\n",
       " 966: 'beaters',\n",
       " 967: 'bolivers',\n",
       " 968: 'empty',\n",
       " 969: 'commerce',\n",
       " 970: 'SVU',\n",
       " 971: 'imagine',\n",
       " 972: 'finished',\n",
       " 973: 'walmart',\n",
       " 974: 'alongwith',\n",
       " 975: 'perverted',\n",
       " 976: 'Rosenstein',\n",
       " 977: 'die',\n",
       " 978: 'Meth',\n",
       " 979: 'Refugees',\n",
       " 980: 'notice',\n",
       " 981: 'Rendia',\n",
       " 982: 'ovens',\n",
       " 983: 'segment',\n",
       " 984: 'shortcomings',\n",
       " 985: 'convicted',\n",
       " 986: 'electical',\n",
       " 987: 'requirement',\n",
       " 988: 'sequence',\n",
       " 989: 'child',\n",
       " 990: 'begin',\n",
       " 991: 'listed',\n",
       " 992: 'Koran',\n",
       " 993: 'scraping',\n",
       " 994: 'reboot',\n",
       " 995: 'salmon',\n",
       " 996: 'Karma',\n",
       " 997: 'Joe',\n",
       " 998: 'Moses',\n",
       " 999: 'privately',\n",
       " ...}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.vocab_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likewise, the ```Engine``` objects could also be deconstructed.\n",
    "\n",
    "Accessing ```engine.model``` returns a PyTorch ```nn.Module```-based model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LSTMClassifier(\n",
       "  (embedding): Embedding(16970, 300)\n",
       "  (rnn): LSTM(300, 128, num_layers=2, dropout=0.2, bidirectional=True)\n",
       "  (fc1): Linear(in_features=256, out_features=128, bias=True)\n",
       "  (fc2): Linear(in_features=128, out_features=2, bias=True)\n",
       "  (dropout): Dropout(p=0.5)\n",
       ")"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "engine.model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Extensibility\n",
    "It is easy to use your own PyTorch layers and models in Lightpost.\n",
    "\n",
    "Let's use the ```lightpost.engine``` API together with our own models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lightpost.datapipe import Datapipe\n",
    "from lightpost.estimators import MLPClassifier\n",
    "from lightpost.engine import Engine\n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "d = load_iris()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use PyTorch's ```Sequential``` interface to stack some layers together to use as our model. We'll use the iris dataset again so we'll pass in the dimensions of the iris dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "model = nn.Sequential(\n",
    "    nn.Linear(4, 128),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(128, 128),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(128, 128),\n",
    "    nn.ReLU(),\n",
    "    nn.Linear(128, 3),\n",
    "    nn.Sigmoid()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we'd like, we can subclass the ```torch.nn.Module``` class to construct a model we can parameterize and use it. The above cell is equivalent with this one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLP(nn.Module):\n",
    "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
    "        super(MLP, self).__init__()\n",
    "        self.fc1 = nn.Linear(input_dim, hidden_dim)\n",
    "        self.fc2 = nn.Linear(hidden_dim, hidden_dim)\n",
    "        self.fc3 = nn.Linear(hidden_dim, hidden_dim)\n",
    "        self.fc4 = nn.Linear(hidden_dim, output_dim)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        out = torch.relu(self.fc1(X))\n",
    "        out = torch.relu(self.fc2(out))\n",
    "        out = torch.relu(self.fc3(out))\n",
    "        out = torch.sigmoid(self.fc4(out))\n",
    "        return out\n",
    "    \n",
    "model = MLP(input_dim=4, hidden_dim=128, output_dim=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While more verbose, the above code is more flexible in terms of how tensors are propagated through the layers. We can control forward-propagation this way using control statements as needed, which is good if we'd like to implement complex models. Creating a class for our model is also good for experimenting on multiple instances of the model with different hyperparameters. Combining this with the ```lightpost.engine``` API makes testing multiple models easy. \n",
    "\n",
    "In truth, Lightpost was built with this quick prototyping task in mind! Case in point, we'll test it out here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Datapipe(d.data, d.target, batch_size=32)\n",
    "model = MLP(input_dim=4, hidden_dim=128, output_dim=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```Engine``` interface is also extensible. Here, we'll use our own optimizer and loss functions. For more information on how these are written, consult the PyTorch documentation!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "optimizer = optim.SGD(model.parameters(), lr=1e-4, momentum=0.9, nesterov=True)\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "engine = Engine(pipeline=pipe, model=model, criterion=criterion, optimizer=optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```Engine``` interface essentially becomes a training wrapper. Here we train the model using the engine object we created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   100 | Train Loss 1.0783 | Train Acc 0.6641 | Val Loss 1.0923 | Val Acc 0.5104\n",
      "Epoch   200 | Train Loss 1.0626 | Train Acc 0.6797 | Val Loss 1.0703 | Val Acc 0.5781\n",
      "Epoch   300 | Train Loss 1.0459 | Train Acc 0.6797 | Val Loss 1.0483 | Val Acc 0.6458\n",
      "Epoch   400 | Train Loss 1.0247 | Train Acc 0.6875 | Val Loss 1.0337 | Val Acc 0.5781\n",
      "Epoch   500 | Train Loss 1.0038 | Train Acc 0.6484 | Val Loss 1.0049 | Val Acc 0.6458\n",
      "Epoch   600 | Train Loss 0.9702 | Train Acc 0.6641 | Val Loss 0.9663 | Val Acc 0.7135\n",
      "Epoch   700 | Train Loss 0.9271 | Train Acc 0.6875 | Val Loss 0.9400 | Val Acc 0.6458\n",
      "Epoch   800 | Train Loss 0.8947 | Train Acc 0.6797 | Val Loss 0.9205 | Val Acc 0.5781\n",
      "Epoch   900 | Train Loss 0.8634 | Train Acc 0.6797 | Val Loss 0.8987 | Val Acc 0.5938\n",
      "Epoch  1000 | Train Loss 0.8377 | Train Acc 0.7422 | Val Loss 0.8483 | Val Acc 0.8594\n"
     ]
    }
   ],
   "source": [
    "engine.fit(epochs=1000, print_every=100, disable_tqdm=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Utilities\n",
    "Lightpost provides many utility functions in the form of the ```lightpost.utils``` interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lightpost.utils.general import split         # Splits a dataset into training and testing sets\n",
    "from lightpost.utils.text import tokenize_array   # Turns an array of untokenized sequences to an array of token sequences\n",
    "from lightpost.utils.text import pad              # Pads/Truncates a sequence of tokens according to a max length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explore the interface to see which you can use!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*The Lightpost Project (C) 2018 The DLSU Machine Learning Group.*\n",
    "\n",
    "*Lightpost is developed, maintained, and managed by the DLSU Machine Learning Group as an internal tool for use within the DLSU Center for Complexity and Emerging Technologies laboratory. Please report all bugs and issues encountered in the Lightpost GitHub repo, or contact the developers.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}