Validate_GPT-3.5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.0 Introduction\n",
    "\n",
    "In this notebook we will perform all steps required to validate the finetuned GPT-3.5 Turbo model on our own custom dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Updated in latest version\n",
    "\n",
    "In this latest version (December 5th, 2023) of this notebook I have made the following updates:\n",
    "* Updated to the latest version of OpenAI (1.3.7) and modified the API calls accordingly.\n",
    "* Changed the model to the latest version: \"gpt-3.5-turbo-1106\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Modules\n",
    "import os\n",
    "import pandas as pd\n",
    "import time\n",
    "from sklearn.metrics import classification_report\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "# OpenAI\n",
    "from openai import OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Load Dataset\n",
    "\n",
    "We will reload the validation CSV file that was generated earlier with the notebook 'Prepare_Train_and_Validation_Datasets.ipynb'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1559, 11)\n"
     ]
    }
   ],
   "source": [
    "# Load Dataset\n",
    "val_df = pd.read_csv('./data/val_df.csv')\n",
    "\n",
    "# Summary\n",
    "print(val_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's review a small subset of the validation data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9266995</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-12-18</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/verdachte-dodelijke-steek...</td>\n",
       "      <td>188</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4130077</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-02-11</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/honderden-arrestaties-bij...</td>\n",
       "      <td>122</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11147268</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2019-01-20</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...</td>\n",
       "      <td>262</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10749100</td>\n",
       "      <td>Klaar voor de verdediging</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-16</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/klaar-voor-de-verdediging...</td>\n",
       "      <td>411</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10700707</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-05</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/windvlaag-grijpt-springma...</td>\n",
       "      <td>286</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0   9266995  Verdachte dodelijke steekpartijen Maastricht l...   \n",
       "1   4130077  Honderden arrestaties bij acties tegen mensen ...   \n",
       "2  11147268  Waarom de 'oudejaarsbonus' voor de jongeren va...   \n",
       "3  10749100                          Klaar voor de verdediging   \n",
       "4  10700707  Windvlaag grijpt springmatras en doodt 2-jarig...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Verdachte dodelijke steekpartijen Maastricht l...     /nieuws   2017-12-18   \n",
       "1  Honderden arrestaties bij acties tegen mensen ...     /nieuws   2017-02-11   \n",
       "2  Waarom de 'oudejaarsbonus' voor de jongeren va...       /home   2019-01-20   \n",
       "3  Klaar voor de verdedigingOver ruim een week be...     /nieuws   2018-10-16   \n",
       "4  Windvlaag grijpt springmatras en doodt 2-jarig...     /nieuws   2018-10-05   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0        ad     False  www.ad.nl/binnenland/verdachte-dodelijke-steek...   \n",
       "1        ad     False  www.ad.nl/buitenland/honderden-arrestaties-bij...   \n",
       "2     trouw      True  www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...   \n",
       "3        ad     False  www.ad.nl/binnenland/klaar-voor-de-verdediging...   \n",
       "4        ad     False  www.ad.nl/buitenland/windvlaag-grijpt-springma...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             188  Verdachte dodelijke steekpartijen Maastricht l...       0  \n",
       "1             122  Honderden arrestaties bij acties tegen mensen ...       0  \n",
       "2             262  Waarom de 'oudejaarsbonus' voor de jongeren va...       1  \n",
       "3             411  Klaar voor de verdedigingOver ruim een week be...       0  \n",
       "4             286  Windvlaag grijpt springmatras en doodt 2-jarig...       0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 Validate finetuned OpenAI GPT-3.5 Model\n",
    "\n",
    "In this section we will validate our finetuned GPT 3.5 model against our validation dataset and determine the validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OpenAI API Key\n",
    "client = OpenAI(api_key = os.environ[\"OPENAI_API_KEY\"])\n",
    "\n",
    "# OpenAI Model Identifier\n",
    "finetuned_model_id = 'ft:gpt-3.5-turbo-1106:lumi-ml-consulting::8SWUDZg1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_prompt(item_text, item_label = None, inference = False):\n",
    "    if inference:\n",
    "        # Base Prompt\n",
    "        prompt_text = [{\"role\": \"system\", \"content\": \"Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.\"}, {\"role\": \"user\", \"content\": \"\"}]\n",
    "\n",
    "        # Set Text and Label\n",
    "        prompt_text[1]['content'] = '### Tekst:\\n' + item_text\n",
    "    else:   \n",
    "        # Base Prompt\n",
    "        prompt_text = {\"messages\": [{\"role\": \"system\", \"content\": \"Je bent redacteur bij een krant. Je beoordeeld een krantenartikel of het politiek of neutraal is. Hieronder staat de tekst van het krantenartikel.\"}, {\"role\": \"user\", \"content\": \"\"}, {\"role\": \"assistant\", \"content\": \"\"}]}\n",
    "\n",
    "        # Set Text and Label\n",
    "        prompt_text['messages'][1]['content'] = '### Tekst:\\n' + item_text\n",
    "        prompt_text['messages'][2]['content'] = item_label\n",
    "    \n",
    "    return prompt_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Inference on Fine-Tuned OpenAI Model\n",
    "\n",
    "We loop through all the records in the validation dataframe. The fine-tuned model will respond with either 'Politiek' or 'Neutraal' to classify the newspaper article as partisan or neutral.\n",
    "\n",
    "For each response we determine the binary label to eventually create the Classification Report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "020a0b10755f422b9a8ae78e129630eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1559 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Placeholders\n",
    "gt_labels, pred_labels = [], []\n",
    "\n",
    "# Inference on Validation Dataframe\n",
    "for index, row in tqdm(val_df.iterrows(), total = val_df.shape[0]):\n",
    "    text = row['max_words_text']\n",
    "    label = row['labels']\n",
    "    gt_labels.append(label)\n",
    "\n",
    "    # Create Prompt\n",
    "    prompt = create_prompt(text, item_label = None, inference = True)\n",
    "\n",
    "    # Call API for inference on FineTuned Model\n",
    "    # Simple error handling .... earlier this summer I experienced occasional errors with the model not responding/not being available.. A simple wait period was enough to have it working.\n",
    "    # Recently no bad experiences with that...however I keep the error handling for it in-place\n",
    "    try:\n",
    "        completion = client.chat.completions.create(model = finetuned_model_id,\n",
    "                                                    messages = prompt)\n",
    "    except Exception as err:\n",
    "        print(err)\n",
    "        time.sleep(120.0)\n",
    "        completion = client.chat.completions.create(model = finetuned_model_id,\n",
    "                                                    messages = prompt)\n",
    "\n",
    "    # Response Determine label based on OpenAI ChatCompletion Response\n",
    "    predicted_label = -1\n",
    "    prediction_text = completion.choices[0].message.content\n",
    "    if prediction_text == 'Politiek':\n",
    "        predicted_label = 1\n",
    "    if prediction_text == 'Neutraal':\n",
    "        predicted_label = 0\n",
    "\n",
    "    # Store Result\n",
    "    pred_labels.append(predicted_label)\n",
    "    #print(f'Index: {index}   GT: {label}   Pred: {predicted_label}')\n",
    "\n",
    "    # Delay...To stay within the limits for tokens/requests per minute\n",
    "    time.sleep(0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Classification Results\n",
    "\n",
    "Below our classification results. The fine-tuned model (based on ) achieves a very nice accuracy of 89.4%. The previous fine-tuned model (based on GPT-3.5 Turbo 0613) achieved 90.8% accuracy.\n",
    "\n",
    "The recall and precision scores are also very balanced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Neutraal      0.939     0.839     0.886       765\n",
      "    Politiek      0.859     0.947     0.901       794\n",
      "\n",
      "    accuracy                          0.894      1559\n",
      "   macro avg      0.899     0.893     0.894      1559\n",
      "weighted avg      0.898     0.894     0.894      1559\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classification Results\n",
    "print(classification_report(gt_labels, pred_labels, target_names = ['Neutraal', 'Politiek'], digits = 3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}