main_tasks/sequence_classification/hf_seq_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine-tuning a Sequence Classification Model\n",
    "\n",
    "Hugging Face Transformers provides a Trainer class to help you fine-tune any of the pretrained models it provides on your dataset. Once you’ve done all the data preprocessing work in the last section, you have just a few steps left to define the `Trainer`. The hardest part is likely to be preparing the environment to run `Trainer.train()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from transformers import AutoTokenizer, DataCollatorWithPadding\n",
    "\n",
    "raw_datasets = load_dataset(\"glue\", \"mrpc\")\n",
    "checkpoint = \"bert-base-uncased\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_function(example):\n",
    "    return tokenizer(example[\"sentence1\"], example[\"sentence2\"], truncation=True)\n",
    "\n",
    "\n",
    "tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)\n",
    "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      ">>> [CLS] amrozi accused his brother, whom he called \" the witness \", of deliberately distorting his evidence. [SEP] referring to him as only \" the witness \", amrozi accused his brother of deliberately distorting his evidence. [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]\n",
      "Number of tokens: 59\n",
      "\n",
      ">>> [CLS] yucaipa owned dominick's before selling the chain to safeway in 1998 for $ 2. 5 billion. [SEP] yucaipa bought dominick's in 1995 for $ 693 million and sold it to safeway for $ 1. 8 billion in 1998. [SEP]\n",
      "Number of tokens: 59\n"
     ]
    }
   ],
   "source": [
    "samples = [tokenized_datasets[\"train\"][i] for i in range(2)]\n",
    "for sample in samples:  # remove the fields that cannot be collated\n",
    "    _ = sample.pop(\"sentence1\")\n",
    "    _ = sample.pop(\"sentence2\")\n",
    "\n",
    "for chunk in data_collator(samples)[\"input_ids\"]:\n",
    "    print(f\"\\n>>> {tokenizer.decode(chunk)}\")\n",
    "    print(f\"Number of tokens: {len(chunk)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model will input the sentence1 and sentence2 strings tokenized together\n",
      "Sentence1: Amrozi accused his brother , whom he called \" the witness \" , of deliberately distorting his evidence .\n",
      "Sentence2: Referring to him as only \" the witness \" , Amrozi accused his brother of deliberately distorting his evidence .\n",
      "And expects in the batch a 'label'. In this case: 1\n"
     ]
    }
   ],
   "source": [
    "sample = tokenized_datasets[\"train\"][0]\n",
    "print(\"The model will input the sentence1 and sentence2 strings tokenized together\")\n",
    "print(f\"Sentence1: {sample['sentence1']}\")\n",
    "print(f\"Sentence2: {sample['sentence2']}\")\n",
    "print(f\"And expects in the batch a 'label'. In this case: {sample['label']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation\n",
    "\n",
    "Let’s see how we can build a useful `compute_metrics()` function and use it when we train. The function must take an `EvalPrediction` object (which is a named tuple with a `predictions` field and a `label_ids` field) and will **return a dictionary mapping strings to floats** (the strings being the names of the metrics returned, and the floats their values)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "mockup_predictions = [1, 0, 0]\n",
    "mockup_labels = [1, 0, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'accuracy': 0.6666666666666666, 'f1': 0.6666666666666666}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import evaluate\n",
    "\n",
    "metric = evaluate.load(\"glue\", \"mrpc\")\n",
    "metric.compute(\n",
    "    predictions=mockup_predictions,\n",
    "    references=mockup_labels\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(eval_preds):\n",
    "    metric = evaluate.load(\"glue\", \"mrpc\")\n",
    "    logits, labels = eval_preds\n",
    "    predictions = logits.argmax(axis=1)\n",
    "    return metric.compute(predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training\n",
    "\n",
    "The first step before we can define our `Trainer` is to define a `TrainingArguments` class that will contain all the hyperparameters the Trainer will use for training and evaluation. The only argument you have to provide is a directory where the trained model will be saved, as well as the checkpoints along the way. For all the rest, you can leave the defaults, which should work pretty well for a basic fine-tuning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments\n",
    "\n",
    "output_dir = f\"tmp/seq_classification-{checkpoint}\"\n",
    "training_args = TrainingArguments(\n",
    "    output_dir,\n",
    "    eval_strategy=\"epoch\",\n",
    "    num_train_epochs=3,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second step is to define our model. We will use the `AutoModelForSequenceClassification` class, with two labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoModelForSequenceClassification\n",
    "\n",
    "model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You get a warning after instantiating this pretrained model. This is because BERT has not been pretrained on classifying pairs of sentences, so **the head of the pretrained model has been discarded and a new head suitable for sequence classification has been added** instead. The warnings indicate that some weights were not used (the ones corresponding to the dropped pretraining head) and that some others were randomly initialized (the ones for the new head). It concludes by encouraging you to train the model, which is exactly what we are going to do now.\n",
    "\n",
    "Once we have our model, we can define a `Trainer` by passing it all the objects constructed up to now — the model, the `training_args`, the training and validation `datasets`, our `data_collator`, and our `tokenizer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import Trainer\n",
    "\n",
    "trainer = Trainer(\n",
    "    model,\n",
    "    training_args,\n",
    "    train_dataset=tokenized_datasets[\"train\"],\n",
    "    eval_dataset=tokenized_datasets[\"validation\"],\n",
    "    data_collator=data_collator,\n",
    "    tokenizer=tokenizer,\n",
    "    compute_metrics=compute_metrics\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To fine-tune the model on our dataset, we just have to call the `train()` method of our `Trainer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: \u001b[33mWARNING\u001b[0m The `run_name` is currently set to the same value as `TrainingArguments.output_dir`. If this was not intended, please specify a different run name by setting the `TrainingArguments.run_name` parameter.\n",
      "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mmarioparreno\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "wandb version 0.17.6 is available!  To upgrade, please run:\n",
       " $ pip install wandb --upgrade"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.17.1"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>/home/maparla/Desktop/NLPJourney/wandb/run-20240812_183630-4q23u3x5</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href='https://wandb.ai/marioparreno/huggingface/runs/4q23u3x5' target=\"_blank\">tmp/seq_classification-bert-base-uncased</a></strong> to <a href='https://wandb.ai/marioparreno/huggingface' target=\"_blank\">Weights & Biases</a> (<a href='https://wandb.me/run' target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View project at <a href='https://wandb.ai/marioparreno/huggingface' target=\"_blank\">https://wandb.ai/marioparreno/huggingface</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       " View run at <a href='https://wandb.ai/marioparreno/huggingface/runs/4q23u3x5' target=\"_blank\">https://wandb.ai/marioparreno/huggingface/runs/4q23u3x5</a>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f0be4d8be67d41909035a3ba4b4bab20",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1377 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e721505b107f4d8f81e767f60fa6e015",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/51 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.620880126953125, 'eval_accuracy': 0.7058823529411765, 'eval_f1': 0.8214285714285714, 'eval_runtime': 1.6375, 'eval_samples_per_second': 249.16, 'eval_steps_per_second': 31.145, 'epoch': 1.0}\n",
      "{'loss': 0.617, 'grad_norm': 2.1467463970184326, 'learning_rate': 3.184458968772695e-05, 'epoch': 1.09}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "20b77f1d5d9e4379a0730c3e2f602849",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/51 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5247003436088562, 'eval_accuracy': 0.7426470588235294, 'eval_f1': 0.8372093023255814, 'eval_runtime': 1.493, 'eval_samples_per_second': 273.267, 'eval_steps_per_second': 34.158, 'epoch': 2.0}\n",
      "{'loss': 0.561, 'grad_norm': 7.290564060211182, 'learning_rate': 1.3689179375453886e-05, 'epoch': 2.18}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fe9618c6a7b54e498fc4adb7c3d5bf2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/51 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5071743130683899, 'eval_accuracy': 0.7818627450980392, 'eval_f1': 0.846286701208981, 'eval_runtime': 1.5385, 'eval_samples_per_second': 265.201, 'eval_steps_per_second': 33.15, 'epoch': 3.0}\n",
      "{'train_runtime': 112.2748, 'train_samples_per_second': 98.01, 'train_steps_per_second': 12.265, 'train_loss': 0.5668545835850978, 'epoch': 3.0}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=1377, training_loss=0.5668545835850978, metrics={'train_runtime': 112.2748, 'train_samples_per_second': 98.01, 'train_steps_per_second': 12.265, 'total_flos': 405114969714960.0, 'train_loss': 0.5668545835850978, 'epoch': 3.0})"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.train()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ml",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}