tutorials/blogposts/data_prep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition for Healthcare with SparkNLP NerDL and NerCRF - Data Preparation and Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why is Data Preparation Important?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's simple to use a pretrained named entity recognition model, but sometimes you need to train your own model to get the best results. This tutorial will show you how to prepare your healthcare training data and train your own NER model using Python and SparkNLP. SparkNLP NerDL has cutting edge scores with the BC2GM dataset (Micro-average F1: 0.87) and other benchmark datasets. You need to use licensed SparkNLP Clinical embeddings to get those cutting edge scores on healthcare data, but Glove embeddings still do great. I'll show you how to train and evaluate your NerCRF and NerDL models on the BC5CDR-Chem dataset using Glove embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Training Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a NerCRF or NerDL model, you will need to put your tokens and entity labels into a space-separated format called CoNLL. A CoNLL file puts each token of a sentence on a different line, and separates each sentence with an empty line. In the following Python example I will annotate one sentence and save it in CoNLL format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create some tokens\n",
    "tokens=['An', 'apple', 'a', 'day', 'keeps', 'the', 'doctor', 'away', '.']\n",
    "\n",
    "#Create part of speech labels or use a place-holder value like \"NN\".\n",
    "pos_labels=['DT', 'NN', 'DT', 'NN', 'VBZ', 'DT', 'NN', 'RB', '.']\n",
    "\n",
    "#Create some named entity labels. 'O' labels mean no named entity was found.\n",
    "entity_labels=['B-Treatment','I-Treatment','I-Treatment','I-Treatment','O','O','O','O','O']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please notice the entity labels above. When an entity has more than one word, the label for the first word should begin with \"B-\" and the label for the following words should begin with \"I-\". Now let's save the tokens, parts-of-speech, and entity labels in CoNLL format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "An DT DT B-Treatment\n",
      "apple NN NN I-Treatment\n",
      "a DT DT I-Treatment\n",
      "day NN NN I-Treatment\n",
      "keeps VBZ VBZ O\n",
      "the DT DT O\n",
      "doctor NN NN O\n",
      "away RB RB O\n",
      ". . . O\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "conll_lines=''\n",
    "\n",
    "for token,pos,label in zip(tokens,pos_labels,entity_labels):\n",
    "    \n",
    "    conll_lines+=\"{} {} {} {}\\n\".format(token, pos, pos, label)\n",
    "\n",
    "#Add another line break at the end of the sentence in order to create an empty line.\n",
    "conll_lines+='\\n'\n",
    "\n",
    "#For this example I will print the lines instead of writing a .txt file.\n",
    "print(conll_lines)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please see the printed CoNLL above. \"An\" is the first word in \"An apple a day\" so it is labelled \"B-Treatment\", while \"apple\",\"a\", and \"day\" are all labelled \"I-Treatment\". The words that are not \"Treatments\" are labelled with a capital \"O\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's another example of a sentence annotated in CoNLL format. The entity is \"blood pressure\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create some tokens\n",
    "tokens=['I','checked','my','blood','pressure','this','morning','.']\n",
    "\n",
    "#Create part-of-speech labels or use a place-holder value like 'NN'.\n",
    "pos_labels=['PRP', 'VBD', 'PRP', 'NN', 'NN', 'DT', 'NN', '.']\n",
    "\n",
    "#Create some named entity labels. 'O' labels mean no named entity was found\n",
    "entity_labels=['O','O','O','B-Test','I-Test','O','O','O']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I PRP PRP O\n",
      "checked VBD VBD O\n",
      "my PRP PRP O\n",
      "blood NN NN B-Test\n",
      "pressure NN NN I-Test\n",
      "this DT DT O\n",
      "morning NN NN O\n",
      ". . . O\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "conll_lines=''\n",
    "\n",
    "for token,pos,label in zip(tokens,pos_labels,entity_labels):\n",
    "    \n",
    "    conll_lines+=\"{} {} {} {}\\n\".format(token, pos, pos, label)\n",
    "\n",
    "#Add another line break at the end of the sentence in order to create an empty line.\n",
    "conll_lines+='\\n'\n",
    "\n",
    "#For this example I will print the lines instead of writing a .txt file.\n",
    "print(conll_lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, 'blood' is the first word in the entity, so it is labelled \"B-Test\", while \"pressure\" is the second word in the entity so it is labelled \"I-Test\". We do this so the model can tell that \"blood pressure\" is one whole entity, rather than the two separate entities \"blood\" and \"pressure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's work with some real datasets. First we have to load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-07-20 14:02:17--  https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/NCBIdisease.tsv\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.99.61\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.99.61|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1060328 (1.0M) [text/tab-separated-values]\n",
      "Saving to: ‘ncbi.tsv’\n",
      "\n",
      "ncbi.tsv            100%[===================>]   1.01M  1.23MB/s    in 0.8s    \n",
      "\n",
      "2020-07-20 14:02:18 (1.23 MB/s) - ‘ncbi.tsv’ saved [1060328/1060328]\n",
      "\n",
      "--2020-07-20 14:02:18--  https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/CRFtrain_dev.txt\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.99.61\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.99.61|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3347168 (3.2M) [text/plain]\n",
      "Saving to: ‘BC5CDRtrain.txt’\n",
      "\n",
      "BC5CDRtrain.txt     100%[===================>]   3.19M  9.50MB/s    in 0.3s    \n",
      "\n",
      "2020-07-20 14:02:19 (9.50 MB/s) - ‘BC5CDRtrain.txt’ saved [3347168/3347168]\n",
      "\n",
      "--2020-07-20 14:02:19--  https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/CRFtest.txt\n",
      "Resolving s3.amazonaws.com (s3.amazonaws.com)... 52.216.99.61\n",
      "Connecting to s3.amazonaws.com (s3.amazonaws.com)|52.216.99.61|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1760184 (1.7M) [text/plain]\n",
      "Saving to: ‘BC5CDRtest.txt’\n",
      "\n",
      "BC5CDRtest.txt      100%[===================>]   1.68M  8.63MB/s    in 0.2s    \n",
      "\n",
      "2020-07-20 14:02:19 (8.63 MB/s) - ‘BC5CDRtest.txt’ saved [1760184/1760184]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "! wget -O ncbi.tsv https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/NCBIdisease.tsv\n",
    "! wget -O BC5CDRtrain.txt https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/CRFtrain_dev.txt\n",
    "! wget -O BC5CDRtest.txt https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/resources/en/ner/conll-2003/CRFtest.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to Convert a Pandas Dataframe to CoNLL Format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next example I'll read from a Pandas dataframe and write a CoNLL file for NerDL. I'll use the sentence ID (sent_id) column to determine if I need to leave an empty line before a new sentence. Here are the first 5 lines of the dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "ncbi=pd.read_csv('ncbi.tsv',sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sent_id</th>\n",
       "      <th>token</th>\n",
       "      <th>entity_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Identification</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>of</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>APC2</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>,</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>a</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   sent_id           token entity_label\n",
       "0        1  Identification            O\n",
       "1        1              of            O\n",
       "2        1            APC2            O\n",
       "3        1               ,            O\n",
       "4        1               a            O"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ncbi.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For NerDL the part-of-speech column is not used, but a CoNLL must still have a part of speech column. Add a part-of-speech column with 'NN' or some other placeholder as the only value. If you already have a part of speech column, you don't need to take this step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "ncbi['pos']='NN'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "My Pandas dataframe is called 'ncbi' and I've added a part-of-speech column which I've called 'pos'. Now write a CoNLL file using the columns of the Pandas dataframe as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "conll_lines=\"-DOCSTART- -X- -X- -O-\\n\\n\"\n",
    "save=0\n",
    "\n",
    "for sent, token, pos, label in zip(ncbi['sent_id'],ncbi['token'],ncbi['pos'],ncbi['entity_label']): \n",
    "    \n",
    "# If the sentence ID has changed, that means we are starting a new sentence. We have to add an empty line.\n",
    "    \n",
    "    if save!=sent:\n",
    "        conll_lines+='\\n'\n",
    "    \n",
    "# Save the conll line\n",
    "    \n",
    "    conll_lines += \"{} {} {} {}\\n\".format(token, pos, pos, label)\n",
    "    \n",
    "    save=sent\n",
    "    \n",
    "\n",
    "# Now print all of the lines to a text file\n",
    "\n",
    "with open(file_loc,'w') as txtfile:\n",
    "        \n",
    "    for line in conll_lines:\n",
    "        txtfile.write(line)\n",
    "\n",
    "txtfile.close()\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look at the first 25 lines of the final CoNLL file, you'll see that rows containing only line breaks signal the beginning of a new sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['-DOCSTART- -X- -X- -O-\\n',\n",
       " '\\n',\n",
       " '\\n',\n",
       " 'Identification NN NN O\\n',\n",
       " 'of NN NN O\\n',\n",
       " 'APC2 NN NN O\\n',\n",
       " ', NN NN O\\n',\n",
       " 'a NN NN O\\n',\n",
       " 'homologue NN NN O\\n',\n",
       " 'of NN NN O\\n',\n",
       " 'the NN NN O\\n',\n",
       " 'adenomatous NN NN B-Disease\\n',\n",
       " 'polyposis NN NN I-Disease\\n',\n",
       " 'coli NN NN I-Disease\\n',\n",
       " 'tumour NN NN I-Disease\\n',\n",
       " 'suppressor NN NN O\\n',\n",
       " '. NN NN O\\n',\n",
       " '\\n',\n",
       " 'The NN NN O\\n',\n",
       " 'adenomatous NN NN B-Disease\\n',\n",
       " 'polyposis NN NN I-Disease\\n',\n",
       " 'coli NN NN I-Disease\\n',\n",
       " '( NN NN I-Disease\\n',\n",
       " 'APC NN NN I-Disease\\n',\n",
       " ') NN NN I-Disease\\n']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with open(file_loc,'r') as f:\n",
    "    lines=f.readlines()[0:25]\n",
    "f.close()\n",
    "lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see SparkNLPs cutting edge results! We'll train NerCRF and NerDL models on the BC5CDR-Chem benchmark dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Evaluating NerCRF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NerCRF is a named entity recognition model in the SparkNLP library which is based on Conditional Random Fields. It requires part-of-speech for model training. To train a model with NerCRF, first import SparkNLP and start your Spark session. Then load the CoNLL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://system76-pc.home:4041\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v2.4.4</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>Spark NLP</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x7f2f63fcb790>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sparknlp\n",
    "from pyspark.ml import Pipeline\n",
    "from pyspark.sql import SparkSession\n",
    "from sparknlp.annotator import *\n",
    "from sparknlp_jsl.annotator import *\n",
    "from sparknlp.base import *\n",
    "import sparknlp_jsl\n",
    "import pyspark.sql.functions as F\n",
    "\n",
    "spark = sparknlp.start()\n",
    "\n",
    "spark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sparknlp.training import CoNLL\n",
    "\n",
    "file_loc='BC5CDRtrain.txt'\n",
    "train = CoNLL().readDataset(spark, file_loc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+------+\n",
      "|ground_truth|count |\n",
      "+------------+------+\n",
      "|O           |221425|\n",
      "|B-CHEM      |10550 |\n",
      "|I-CHEM      |3648  |\n",
      "+------------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql import functions as F\n",
    "\n",
    "train.select(F.explode(F.arrays_zip('token.result','label.result')).alias(\"cols\")) \\\n",
    ".select(F.expr(\"cols['0']\").alias(\"token\"),\n",
    "        F.expr(\"cols['1']\").alias(\"ground_truth\")).groupBy('ground_truth').count()\\\n",
    "        .orderBy('count', ascending=False).show(100,truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I will add Glove embeddings to the dataset before Ner training, but if you want better results with your healthcare projects, use SparkNLP Clinical embeddings. First, set up your pipeline and fit your model to your training dataset. The fitting process could take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glove_100d download started this may take some time.\n",
      "Approximate size to download 145.3 MB\n",
      "[OK!]\n"
     ]
    }
   ],
   "source": [
    "word_embeddings = WordEmbeddingsModel.pretrained('glove_100d')\\\n",
    "          .setInputCols([\"document\", \"token\"])\\\n",
    "          .setOutputCol(\"embeddings\")\n",
    "\n",
    "nerTagger = NerCrfApproach()\\\n",
    "    .setInputCols([\"sentence\", \"token\", \"pos\",\"embeddings\"])\\\n",
    "    .setLabelColumn(\"label\")\\\n",
    "    .setOutputCol(\"ner\")\\\n",
    "    .setMaxEpochs(9)\\\n",
    "    \n",
    "ner_pipeline = Pipeline(stages=[\n",
    "          word_embeddings,\n",
    "          nerTagger\n",
    " ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "ner_model = ner_pipeline.fit(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next add word embeddings to your test dataset and make your predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sparknlp.training import CoNLL\n",
    "\n",
    "file_loc='BC5CDRtest.txt'\n",
    "test = CoNLL().readDataset(spark, file_loc)\n",
    "\n",
    "test_data = word_embeddings.transform(test)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = ner_model.transform(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see all of your input and output columns in the final \"predictions\" dataframe, but I'll focus on the 'ner' column which contains the prediction, and the 'label' column which contains the ground truth. You can use sklearn.metrics classification_report to check the accuracy of the predictions using these 2 columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "import pyspark.sql.functions as F\n",
    "\n",
    "preds = predictions.select(F.explode(F.arrays_zip('token.result','label.result','ner.result')).alias(\"cols\")) \\\n",
    "        .select(F.col('cols.0').alias(\"token\"),\n",
    "        F.col('cols.1').alias(\"label\"),\n",
    "        F.col('cols.2').alias(\"ner\"))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+------+------+\n",
      "|       token| label|   ner|\n",
      "+------------+------+------+\n",
      "|  dobutamine|B-CHEM|B-CHEM|\n",
      "|  dobutamine|B-CHEM|B-CHEM|\n",
      "|  dobutamine|B-CHEM|B-CHEM|\n",
      "|  Dubutamine|B-CHEM|B-CHEM|\n",
      "|           5|B-CHEM|B-CHEM|\n",
      "|           -|I-CHEM|I-CHEM|\n",
      "|fluorouracil|I-CHEM|I-CHEM|\n",
      "|           5|B-CHEM|B-CHEM|\n",
      "|           -|I-CHEM|I-CHEM|\n",
      "+------------+------+------+\n",
      "only showing top 9 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "preds.filter(\"ner!='O'\").show(9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Convert the Spark dataframe to a Pandas dataframe.\n",
    "import pandas as pd\n",
    "preds_df=preds.toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "      B-CHEM       0.91      0.85      0.87      5385\n",
      "      I-CHEM       0.72      0.84      0.77      1628\n",
      "           O       0.99      0.99      0.99    117737\n",
      "\n",
      "    accuracy                           0.98    124750\n",
      "   macro avg       0.87      0.89      0.88    124750\n",
      "weighted avg       0.98      0.98      0.98    124750\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print (classification_report(preds_df['label'], preds_df['ner']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Evaluating NerDL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NerDL is a deep learning named entity recognition model in the SparkNLP library which does not require training data to contain parts-of-speech. It is a Bidirectional LSTM-CNN. For a more detailed overview of training a model using NerDL, you can check out this [post](https://medium.com/r/?url=https%3A%2F%2Ftowardsdatascience.com%2Fnamed-entity-recognition-ner-with-bert-in-spark-nlp-874df20d1d77). We've already loaded the BC5CDR-Chem test and train datasets. Now I can show you how to add Glove embeddings and save the test data as a parquet file before NerDL model training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glove_100d download started this may take some time.\n",
      "Approximate size to download 145.3 MB\n",
      "[OK!]\n"
     ]
    }
   ],
   "source": [
    "word_embeddings = WordEmbeddingsModel.pretrained('glove_100d')\\\n",
    "          .setInputCols([\"document\", \"token\"])\\\n",
    "          .setOutputCol(\"embeddings\")\n",
    "\n",
    "test_data = word_embeddings.transform(test)\n",
    "\n",
    "test_data.write.parquet('../test.parquet')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next set up the rest of the pipeline by adding the location of the test data parquet file and the folder where your Tensorflow graphs are located. Using \".setEvaluationLogExtended(True)\" will output a more detailed model evaluation log. When you run the training, If you get an error for incompatible TF graph, use NerDL_Graph.ipynb located [here](https://github.com/JohnSnowLabs/spark-nlp-workshop/blob/master/tutorials/Certification_Trainings/Public/4.1_NerDL_Graph.ipynb) to create a graph using the parameters given in the error message. If you're having trouble with this part of NerDL model training, you should read this [post](https://medium.com/r/?url=https%3A%2F%2Ftowardsdatascience.com%2Fnamed-entity-recognition-ner-with-bert-in-spark-nlp-874df20d1d77)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "nerTagger = NerDLApproach()\\\n",
    "  .setInputCols([\"sentence\", \"token\", \"embeddings\"])\\\n",
    "  .setLabelColumn(\"label\")\\\n",
    "  .setOutputCol(\"ner\")\\\n",
    "  .setMaxEpochs(15)\\\n",
    "  .setLr(0.001)\\\n",
    "  .setPo(0.005)\\\n",
    "  .setBatchSize(32)\\\n",
    "  .setRandomSeed(0)\\\n",
    "  .setVerbose(1)\\\n",
    "  .setValidationSplit(0.2)\\\n",
    "  .setEvaluationLogExtended(True) \\\n",
    "  .setEnableOutputLogs(True)\\\n",
    "  .setIncludeConfidence(True)\\\n",
    "  .setGraphFolder('../tfgraphs')\\\n",
    "  .setTestDataset('../test.parquet')\n",
    "                  \n",
    "ner_pipeline = Pipeline(stages=[\n",
    "          word_embeddings,\n",
    "          nerTagger\n",
    " ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though the word_embeddings pipe is in a previous cell, it is still part of the pipeline. In the next cell I'll fit the model to the training set. This could take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 86.8 ms, sys: 34.4 ms, total: 121 ms\n",
      "Wall time: 12min 2s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "ner_model = ner_pipeline.fit(train)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find the final log at the top of the list here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 280\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 13529 Jun 13 18:07 NerDLApproach_004e6626b3bb.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 17997 Jun 12 17:51 NerDLApproach_15b6d84b808b.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  9819 Jun 11 20:00 NerDLApproach_31530d63198f.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  1092 Jun 11 19:52 NerDLApproach_e5b8f13159eb.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  1100 Jun 11 19:45 NerDLApproach_9e3eb5e1c0e9.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  2007 Jun 11 10:11 NerDLApproach_7e555f3bb935.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  2009 Jun 11 10:00 NerDLApproach_b61689d542fe.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 12650 Jun 10 17:38 NerDLApproach_3a19217cbe4b.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  1049 Jun  7 21:16 NerDLApproach_0560e20a620b.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 55543 May 16 09:57 NerDLApproach_70a344517ff7.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 55501 May 16 08:47 NerDLApproach_769f9d4b74d3.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 27771 May 16 08:15 NerDLApproach_acc76debb989.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 27750 May 16 07:07 NerDLApproach_764dcf2115db.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  2095 May 16 06:47 NerDLApproach_367a22b96935.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz 13374 May 16 06:40 NerDLApproach_7f8aa75af674.log\r\n",
      "-rw-rw-r-- 1 myilmaz myilmaz  1221 May 16 06:33 NerDLApproach_7676461e9e12.log\r\n"
     ]
    }
   ],
   "source": [
    "! cd ~/annotator_logs && ls -lt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each training epoch your extended log will print 2 sets of metrics, one for the validation dataset and one for the test dataset. (The metrics for the validation data is on the top). For each dataset there's a table showing true positives (tp), false positives (fp), false negatives (fn), precision, recall and f1 scores for each entity (except 'O'). Beneath this table you'll find the macro-average and micro-average precision, recall and f1 scores for the dataset. So if you're looking for the micro-average f1 score for the test data, you'll find it on the last line of the log for each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Name of the selected graph: /home/myilmaz/devel/mag/datasets/make_CoNLL/tutorial/tfgraphs/blstm_3_100_128_82.pb\r\n",
      "Training started, trainExamples: 7313, labels: 3 chars: 81, \r\n",
      "\r\n",
      "\r\n",
      "Epoch: 0 started, learning rate: 0.001, dataset size: 7313\r\n",
      "Done, 33.119487882 loss: 722.0155, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.284676026\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1572\t 294\t 508\t 0.8424437\t 0.75576925\t 0.7967562\r\n",
      "I-CHEM\t 349\t 85\t 501\t 0.8041475\t 0.41058823\t 0.5436137\r\n",
      "tp: 1921 fp: 379 fn: 1009 labels: 2\r\n",
      "Macro-average\t prec: 0.8232956, rec: 0.58317876, f1: 0.6827405\r\n",
      "Micro-average\t prec: 0.8352174, rec: 0.6556314, f1: 0.7346081\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.096200166\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 3946\t 743\t 1439\t 0.84154403\t 0.7327762\t 0.7834028\r\n",
      "I-CHEM\t 586\t 232\t 1042\t 0.71638143\t 0.35995087\t 0.47914964\r\n",
      "tp: 4532 fp: 975 fn: 2481 labels: 2\r\n",
      "Macro-average\t prec: 0.77896273, rec: 0.54636353, f1: 0.6422522\r\n",
      "Micro-average\t prec: 0.8229526, rec: 0.64622843, f1: 0.7239617\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 1 started, learning rate: 9.950249E-4, dataset size: 7313\r\n",
      "Done, 34.125822636 loss: 368.44656, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.058410662\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1638\t 170\t 442\t 0.90597343\t 0.7875\t 0.84259266\r\n",
      "I-CHEM\t 369\t 64\t 481\t 0.852194\t 0.43411765\t 0.5752144\r\n",
      "tp: 2007 fp: 234 fn: 923 labels: 2\r\n",
      "Macro-average\t prec: 0.87908375, rec: 0.61080885, f1: 0.72079307\r\n",
      "Micro-average\t prec: 0.8955823, rec: 0.68498296, f1: 0.7762522\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.118585091\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4101\t 448\t 1284\t 0.9015168\t 0.7615599\t 0.82564926\r\n",
      "I-CHEM\t 611\t 122\t 1017\t 0.8335607\t 0.3753071\t 0.5175773\r\n",
      "tp: 4712 fp: 570 fn: 2301 labels: 2\r\n",
      "Macro-average\t prec: 0.86753875, rec: 0.5684335, f1: 0.68683517\r\n",
      "Micro-average\t prec: 0.8920863, rec: 0.671895, f1: 0.76649046\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 2 started, learning rate: 9.90099E-4, dataset size: 7313\r\n",
      "Done, 35.640994426 loss: 270.21365, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.054015841\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1935\t 276\t 145\t 0.87516963\t 0.93028843\t 0.90188766\r\n",
      "I-CHEM\t 554\t 62\t 296\t 0.89935064\t 0.6517647\t 0.7557981\r\n",
      "tp: 2489 fp: 338 fn: 441 labels: 2\r\n",
      "Macro-average\t prec: 0.88726014, rec: 0.7910266, f1: 0.8363843\r\n",
      "Micro-average\t prec: 0.8804386, rec: 0.8494881, f1: 0.8646865\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.106889951\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4846\t 851\t 539\t 0.85062313\t 0.8999072\t 0.8745714\r\n",
      "I-CHEM\t 1037\t 155\t 591\t 0.86996645\t 0.6369779\t 0.73546094\r\n",
      "tp: 5883 fp: 1006 fn: 1130 labels: 2\r\n",
      "Macro-average\t prec: 0.8602948, rec: 0.7684425, f1: 0.81177866\r\n",
      "Micro-average\t prec: 0.8539701, rec: 0.83887064, f1: 0.846353\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 3 started, learning rate: 9.852217E-4, dataset size: 7313\r\n",
      "Done, 34.218184193 loss: 203.04968, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.053450374\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1934\t 188\t 146\t 0.9114043\t 0.92980766\t 0.920514\r\n",
      "I-CHEM\t 740\t 288\t 110\t 0.71984434\t 0.87058824\t 0.78807235\r\n",
      "tp: 2674 fp: 476 fn: 256 labels: 2\r\n",
      "Macro-average\t prec: 0.81562436, rec: 0.900198, f1: 0.85582685\r\n",
      "Micro-average\t prec: 0.8488889, rec: 0.912628, f1: 0.8796053\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.125469374\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4832\t 629\t 553\t 0.8848196\t 0.89730734\t 0.89101976\r\n",
      "I-CHEM\t 1342\t 582\t 286\t 0.6975052\t 0.8243243\t 0.7556306\r\n",
      "tp: 6174 fp: 1211 fn: 839 labels: 2\r\n",
      "Macro-average\t prec: 0.7911624, rec: 0.8608158, f1: 0.8245207\r\n",
      "Micro-average\t prec: 0.836019, rec: 0.880365, f1: 0.8576191\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 4 started, learning rate: 9.803922E-4, dataset size: 7313\r\n",
      "Done, 34.119872631 loss: 167.74683, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.071174386\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1919\t 107\t 161\t 0.9471866\t 0.92259616\t 0.93472975\r\n",
      "I-CHEM\t 719\t 94\t 131\t 0.88437885\t 0.84588236\t 0.8647024\r\n",
      "tp: 2638 fp: 201 fn: 292 labels: 2\r\n",
      "Macro-average\t prec: 0.9157827, rec: 0.88423926, f1: 0.8997346\r\n",
      "Micro-average\t prec: 0.9292004, rec: 0.9003413, f1: 0.9145433\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.152329663\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4714\t 372\t 671\t 0.92685807\t 0.87539464\t 0.9003915\r\n",
      "I-CHEM\t 1262\t 315\t 366\t 0.8002536\t 0.7751843\t 0.7875195\r\n",
      "tp: 5976 fp: 687 fn: 1037 labels: 2\r\n",
      "Macro-average\t prec: 0.86355585, rec: 0.8252895, f1: 0.84398913\r\n",
      "Micro-average\t prec: 0.89689326, rec: 0.8521318, f1: 0.87393975\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 5 started, learning rate: 9.756098E-4, dataset size: 7313\r\n",
      "Done, 34.174738547 loss: 137.58278, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.030198415\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1978\t 156\t 102\t 0.9268978\t 0.95096153\t 0.9387755\r\n",
      "I-CHEM\t 551\t 56\t 299\t 0.907743\t 0.6482353\t 0.75634867\r\n",
      "tp: 2529 fp: 212 fn: 401 labels: 2\r\n",
      "Macro-average\t prec: 0.9173204, rec: 0.79959846, f1: 0.8544236\r\n",
      "Micro-average\t prec: 0.92265594, rec: 0.8631399, f1: 0.89190614\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.163934411\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4829\t 506\t 556\t 0.90515465\t 0.8967502\t 0.9009328\r\n",
      "I-CHEM\t 971\t 151\t 657\t 0.8654189\t 0.59643734\t 0.7061819\r\n",
      "tp: 5800 fp: 657 fn: 1213 labels: 2\r\n",
      "Macro-average\t prec: 0.8852868, rec: 0.7465938, f1: 0.8100466\r\n",
      "Micro-average\t prec: 0.89825, rec: 0.8270355, f1: 0.861173\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 6 started, learning rate: 9.7087387E-4, dataset size: 7313\r\n",
      "Done, 34.302770625 loss: 116.390434, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.033750868\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1955\t 99\t 125\t 0.95180136\t 0.93990386\t 0.94581515\r\n",
      "I-CHEM\t 753\t 161\t 97\t 0.8238512\t 0.8858824\t 0.8537415\r\n",
      "tp: 2708 fp: 260 fn: 222 labels: 2\r\n",
      "Macro-average\t prec: 0.8878263, rec: 0.9128931, f1: 0.9001852\r\n",
      "Micro-average\t prec: 0.91239893, rec: 0.92423207, f1: 0.9182773\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.111024768\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4800\t 349\t 585\t 0.93221986\t 0.89136493\t 0.91133475\r\n",
      "I-CHEM\t 1327\t 445\t 301\t 0.7488713\t 0.81511056\t 0.78058827\r\n",
      "tp: 6127 fp: 794 fn: 886 labels: 2\r\n",
      "Macro-average\t prec: 0.8405456, rec: 0.85323775, f1: 0.8468442\r\n",
      "Micro-average\t prec: 0.8852767, rec: 0.8736632, f1: 0.87943155\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 7 started, learning rate: 9.6618367E-4, dataset size: 7313\r\n",
      "Done, 34.286044905 loss: 101.03544, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.052412068\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 2003\t 127\t 77\t 0.94037557\t 0.96298075\t 0.9515439\r\n",
      "I-CHEM\t 745\t 94\t 105\t 0.88796186\t 0.87647057\t 0.8821788\r\n",
      "tp: 2748 fp: 221 fn: 182 labels: 2\r\n",
      "Macro-average\t prec: 0.9141687, rec: 0.91972566, f1: 0.9169388\r\n",
      "Micro-average\t prec: 0.92556417, rec: 0.937884, f1: 0.93168336\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.147375342\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4931\t 464\t 454\t 0.91399443\t 0.91569173\t 0.9148423\r\n",
      "I-CHEM\t 1300\t 289\t 328\t 0.8181246\t 0.7985258\t 0.80820644\r\n",
      "tp: 6231 fp: 753 fn: 782 labels: 2\r\n",
      "Macro-average\t prec: 0.86605954, rec: 0.8571088, f1: 0.8615609\r\n",
      "Micro-average\t prec: 0.8921821, rec: 0.8884928, f1: 0.89033365\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 8 started, learning rate: 9.615385E-4, dataset size: 7313\r\n",
      "Done, 34.446644867 loss: 90.233, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.053225889\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 2004\t 122\t 76\t 0.9426152\t 0.9634615\t 0.95292443\r\n",
      "I-CHEM\t 738\t 76\t 112\t 0.9066339\t 0.8682353\t 0.8870192\r\n",
      "tp: 2742 fp: 198 fn: 188 labels: 2\r\n",
      "Macro-average\t prec: 0.92462456, rec: 0.9158484, f1: 0.92021555\r\n",
      "Micro-average\t prec: 0.93265307, rec: 0.9358362, f1: 0.9342419\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.123367152\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4922\t 493\t 463\t 0.9089566\t 0.9140204\t 0.91148144\r\n",
      "I-CHEM\t 1186\t 225\t 442\t 0.8405386\t 0.7285012\t 0.7805199\r\n",
      "tp: 6108 fp: 718 fn: 905 labels: 2\r\n",
      "Macro-average\t prec: 0.87474763, rec: 0.8212608, f1: 0.8471608\r\n",
      "Micro-average\t prec: 0.89481395, rec: 0.8709539, f1: 0.88272274\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 9 started, learning rate: 9.569379E-4, dataset size: 7313\r\n",
      "Done, 34.421332381 loss: 90.42398, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.009300916\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1993\t 93\t 87\t 0.95541704\t 0.9581731\t 0.95679307\r\n",
      "I-CHEM\t 738\t 55\t 112\t 0.93064314\t 0.8682353\t 0.8983566\r\n",
      "tp: 2731 fp: 148 fn: 199 labels: 2\r\n",
      "Macro-average\t prec: 0.9430301, rec: 0.9132042, f1: 0.92787755\r\n",
      "Micro-average\t prec: 0.94859326, rec: 0.93208194, f1: 0.9402651\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.167042729\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4832\t 310\t 553\t 0.93971217\t 0.89730734\t 0.91802037\r\n",
      "I-CHEM\t 1191\t 201\t 437\t 0.85560346\t 0.7315725\t 0.78874177\r\n",
      "tp: 6023 fp: 511 fn: 990 labels: 2\r\n",
      "Macro-average\t prec: 0.8976578, rec: 0.8144399, f1: 0.85402644\r\n",
      "Micro-average\t prec: 0.9217937, rec: 0.8588336, f1: 0.88920057\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 10 started, learning rate: 9.5238106E-4, dataset size: 7313\r\n",
      "Done, 34.059791336 loss: 71.982666, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.072081745\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1988\t 76\t 92\t 0.9631783\t 0.95576924\t 0.9594595\r\n",
      "I-CHEM\t 750\t 66\t 100\t 0.9191176\t 0.88235295\t 0.90036017\r\n",
      "tp: 2738 fp: 142 fn: 192 labels: 2\r\n",
      "Macro-average\t prec: 0.9411479, rec: 0.91906106, f1: 0.92997336\r\n",
      "Micro-average\t prec: 0.95069444, rec: 0.934471, f1: 0.9425129\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.178245287\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4787\t 320\t 598\t 0.9373409\t 0.88895077\t 0.91250473\r\n",
      "I-CHEM\t 1179\t 156\t 449\t 0.88314605\t 0.7242015\t 0.7958151\r\n",
      "tp: 5966 fp: 476 fn: 1047 labels: 2\r\n",
      "Macro-average\t prec: 0.9102435, rec: 0.80657613, f1: 0.8552799\r\n",
      "Micro-average\t prec: 0.9261099, rec: 0.8507058, f1: 0.8868079\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 11 started, learning rate: 9.478674E-4, dataset size: 7313\r\n",
      "Done, 33.961773914 loss: 65.852005, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.047509739\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 2015\t 105\t 65\t 0.9504717\t 0.96875\t 0.95952386\r\n",
      "I-CHEM\t 770\t 83\t 80\t 0.9026964\t 0.90588236\t 0.90428656\r\n",
      "tp: 2785 fp: 188 fn: 145 labels: 2\r\n",
      "Macro-average\t prec: 0.926584, rec: 0.9373162, f1: 0.9319192\r\n",
      "Micro-average\t prec: 0.93676424, rec: 0.95051193, f1: 0.943588\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.166007777\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4930\t 457\t 455\t 0.91516614\t 0.91550606\t 0.915336\r\n",
      "I-CHEM\t 1298\t 338\t 330\t 0.79339856\t 0.7972973\t 0.7953431\r\n",
      "tp: 6228 fp: 795 fn: 785 labels: 2\r\n",
      "Macro-average\t prec: 0.8542824, rec: 0.8564017, f1: 0.8553407\r\n",
      "Micro-average\t prec: 0.8868005, rec: 0.88806504, f1: 0.88743234\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 12 started, learning rate: 9.433963E-4, dataset size: 7313\r\n",
      "Done, 34.202173441 loss: 61.230732, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.072711542\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1981\t 88\t 99\t 0.9574674\t 0.95240384\t 0.95492893\r\n",
      "I-CHEM\t 748\t 41\t 102\t 0.9480355\t 0.88\t 0.9127516\r\n",
      "tp: 2729 fp: 129 fn: 201 labels: 2\r\n",
      "Macro-average\t prec: 0.9527514, rec: 0.91620195, f1: 0.9341193\r\n",
      "Micro-average\t prec: 0.95486355, rec: 0.93139935, f1: 0.94298553\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.196799511\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4801\t 331\t 584\t 0.9355027\t 0.8915506\t 0.91299796\r\n",
      "I-CHEM\t 1181\t 145\t 447\t 0.89064854\t 0.72542995\t 0.79959375\r\n",
      "tp: 5982 fp: 476 fn: 1031 labels: 2\r\n",
      "Macro-average\t prec: 0.9130756, rec: 0.8084903, f1: 0.8576061\r\n",
      "Micro-average\t prec: 0.92629296, rec: 0.8529873, f1: 0.88813007\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 13 started, learning rate: 9.389671E-4, dataset size: 7313\r\n",
      "Done, 33.89826042 loss: 62.6999, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.080093562\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1960\t 65\t 120\t 0.96790123\t 0.9423077\t 0.95493305\r\n",
      "I-CHEM\t 773\t 120\t 77\t 0.8656215\t 0.9094118\t 0.88697654\r\n",
      "tp: 2733 fp: 185 fn: 197 labels: 2\r\n",
      "Macro-average\t prec: 0.9167614, rec: 0.92585975, f1: 0.92128813\r\n",
      "Micro-average\t prec: 0.9366004, rec: 0.93276453, f1: 0.9346785\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.11080269\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4672\t 247\t 713\t 0.94978654\t 0.8675952\t 0.9068323\r\n",
      "I-CHEM\t 1303\t 283\t 325\t 0.82156366\t 0.80036855\t 0.8108276\r\n",
      "tp: 5975 fp: 530 fn: 1038 labels: 2\r\n",
      "Macro-average\t prec: 0.8856751, rec: 0.8339819, f1: 0.8590515\r\n",
      "Micro-average\t prec: 0.9185242, rec: 0.85198915, f1: 0.88400656\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 14 started, learning rate: 9.345794E-4, dataset size: 7313\r\n",
      "Done, 33.906427254 loss: 57.720154, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.069351368\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 2018\t 93\t 62\t 0.9559451\t 0.9701923\t 0.963016\r\n",
      "I-CHEM\t 777\t 141\t 73\t 0.8464052\t 0.91411763\t 0.8789593\r\n",
      "tp: 2795 fp: 234 fn: 135 labels: 2\r\n",
      "Macro-average\t prec: 0.90117514, rec: 0.942155, f1: 0.9212095\r\n",
      "Micro-average\t prec: 0.9227468, rec: 0.9539249, f1: 0.93807685\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.15805635\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4886\t 373\t 499\t 0.929074\t 0.90733516\t 0.918076\r\n",
      "I-CHEM\t 1330\t 296\t 298\t 0.8179582\t 0.8169533\t 0.8174554\r\n",
      "tp: 6216 fp: 669 fn: 797 labels: 2\r\n",
      "Macro-average\t prec: 0.8735161, rec: 0.86214423, f1: 0.8677929\r\n",
      "Micro-average\t prec: 0.90283227, rec: 0.8863539, f1: 0.8945172\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 15 started, learning rate: 9.302326E-4, dataset size: 7313\r\n",
      "Done, 34.075812956 loss: 46.7108, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.074266615\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 2004\t 107\t 76\t 0.9493131\t 0.9634615\t 0.956335\r\n",
      "I-CHEM\t 758\t 109\t 92\t 0.87427914\t 0.8917647\t 0.88293535\r\n",
      "tp: 2762 fp: 216 fn: 168 labels: 2\r\n",
      "Macro-average\t prec: 0.9117961, rec: 0.92761314, f1: 0.9196366\r\n",
      "Micro-average\t prec: 0.9274681, rec: 0.9426621, f1: 0.93500334\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.132328888\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4912\t 385\t 473\t 0.9273173\t 0.91216344\t 0.919678\r\n",
      "I-CHEM\t 1328\t 355\t 300\t 0.78906715\t 0.8157248\t 0.80217457\r\n",
      "tp: 6240 fp: 740 fn: 773 labels: 2\r\n",
      "Macro-average\t prec: 0.8581922, rec: 0.8639441, f1: 0.8610586\r\n",
      "Micro-average\t prec: 0.8939828, rec: 0.8897761, f1: 0.8918745\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 16 started, learning rate: 9.259259E-4, dataset size: 7313\r\n",
      "Done, 34.127444142 loss: 46.657284, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.030565497\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 2032\t 117\t 48\t 0.94555604\t 0.97692305\t 0.96098363\r\n",
      "I-CHEM\t 755\t 75\t 95\t 0.9096386\t 0.8882353\t 0.89880955\r\n",
      "tp: 2787 fp: 192 fn: 143 labels: 2\r\n",
      "Macro-average\t prec: 0.9275973, rec: 0.93257916, f1: 0.93008155\r\n",
      "Micro-average\t prec: 0.93554884, rec: 0.9511945, f1: 0.94330686\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.124047607\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4986\t 567\t 399\t 0.897893\t 0.9259053\t 0.91168404\r\n",
      "I-CHEM\t 1198\t 225\t 430\t 0.84188336\t 0.7358722\t 0.7853163\r\n",
      "tp: 6184 fp: 792 fn: 829 labels: 2\r\n",
      "Macro-average\t prec: 0.8698882, rec: 0.83088875, f1: 0.8499413\r\n",
      "Micro-average\t prec: 0.8864679, rec: 0.88179094, f1: 0.8841232\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 17 started, learning rate: 9.21659E-4, dataset size: 7313\r\n",
      "Done, 34.165765321 loss: 42.641945, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.056740516\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 2007\t 84\t 73\t 0.95982784\t 0.96490383\t 0.96235913\r\n",
      "I-CHEM\t 728\t 67\t 122\t 0.91572326\t 0.8564706\t 0.8851063\r\n",
      "tp: 2735 fp: 151 fn: 195 labels: 2\r\n",
      "Macro-average\t prec: 0.93777555, rec: 0.9106872, f1: 0.92403287\r\n",
      "Micro-average\t prec: 0.94767845, rec: 0.9334471, f1: 0.94050896\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.100443046\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4845\t 340\t 540\t 0.93442625\t 0.89972144\t 0.91674554\r\n",
      "I-CHEM\t 1202\t 185\t 426\t 0.8666186\t 0.73832923\t 0.7973466\r\n",
      "tp: 6047 fp: 525 fn: 966 labels: 2\r\n",
      "Macro-average\t prec: 0.9005224, rec: 0.81902534, f1: 0.8578426\r\n",
      "Micro-average\t prec: 0.92011565, rec: 0.8622558, f1: 0.89024657\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 18 started, learning rate: 9.174312E-4, dataset size: 7313\r\n",
      "Done, 34.125692016 loss: 38.628956, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.064173301\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1994\t 63\t 86\t 0.96937287\t 0.95865387\t 0.9639836\r\n",
      "I-CHEM\t 779\t 78\t 71\t 0.90898484\t 0.9164706\t 0.9127124\r\n",
      "tp: 2773 fp: 141 fn: 157 labels: 2\r\n",
      "Macro-average\t prec: 0.9391788, rec: 0.9375622, f1: 0.9383698\r\n",
      "Micro-average\t prec: 0.9516129, rec: 0.9464164, f1: 0.94900745\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.136078213\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4751\t 291\t 634\t 0.9422848\t 0.88226557\t 0.911288\r\n",
      "I-CHEM\t 1269\t 317\t 359\t 0.8001261\t 0.77948403\t 0.78967017\r\n",
      "tp: 6020 fp: 608 fn: 993 labels: 2\r\n",
      "Macro-average\t prec: 0.87120545, rec: 0.8308748, f1: 0.8505623\r\n",
      "Micro-average\t prec: 0.908268, rec: 0.8584058, f1: 0.88263327\r\n",
      "\r\n",
      "\r\n",
      "Epoch: 19 started, learning rate: 9.1324205E-4, dataset size: 7313\r\n",
      "Done, 34.520453505 loss: 34.90555, batches: 231\r\n",
      "Quality on validation dataset (20.0%), valExamples = 1828\r\n",
      "time to finish evaluation: 3.06872751\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 1996\t 72\t 84\t 0.96518373\t 0.9596154\t 0.96239144\r\n",
      "I-CHEM\t 527\t 74\t 323\t 0.8768719\t 0.62\t 0.72639555\r\n",
      "tp: 2523 fp: 146 fn: 407 labels: 2\r\n",
      "Macro-average\t prec: 0.9210278, rec: 0.7898077, f1: 0.8503855\r\n",
      "Micro-average\t prec: 0.94529784, rec: 0.86109215, f1: 0.90123236\r\n",
      "Quality on test dataset: \r\n",
      "time to finish evaluation: 8.12001153\r\n",
      "label\t tp\t fp\t fn\t prec\t rec\t f1\r\n",
      "B-CHEM\t 4763\t 296\t 622\t 0.9414904\t 0.88449395\t 0.91210264\r\n",
      "I-CHEM\t 946\t 175\t 682\t 0.84388936\t 0.5810811\t 0.6882503\r\n",
      "tp: 5709 fp: 471 fn: 1304 labels: 2\r\n",
      "Macro-average\t prec: 0.8926899, rec: 0.7327875, f1: 0.8048737\r\n",
      "Micro-average\t prec: 0.9237864, rec: 0.8140596, f1: 0.86545897\r\n"
     ]
    }
   ],
   "source": [
    "!cat ~/annotator_logs/NerDLApproach_15b6d84b808b.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall our NerDL and NerCRF models didn't do too bad with the BC5CDR-Chem benchmark dataset enriched with Glove embeddings. In the 11th epoch the NerDL model's macro-average f1 score on the test set was 0.86 and after 9 epochs the NerCRF had a macro-average f1 score of 0.88 on the test set. However, using Clinical embeddings instead of Glove will bring your NerDL micro-average F1 score from 0.887 up to 0.915, much closer to the best published score for this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}