notebook_examples/data_labeling-text_classification.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "@notebook{data_labeling-text_classification.ipynb,\n",
    "    title: Text Classification with Data Labeling Service Integration,\n",
    "    summary: Use the Oracle Cloud Infrastructure (OCI) Data Labeling service to efficiently build enriched, labeled datasets for the purpose of accurately training AI/ML models. This notebook demonstrates operations that can be performed using the Advanced Data Science (ADS) Data Labeling module.,\n",
    "    developed_on: nlp_p37_cpu_v2,\n",
    "    keywords: data labeling, text classification,\n",
    "    license: Universal Permissive License v 1.0\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upgrade Oracle ADS to pick up latest features and maintain compatibility with Oracle Cloud Infrastructure.\n",
    "\n",
    "!pip install -U oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oracle Data Science service sample notebook.\n",
    "\n",
    "Copyright (c) 2021, 2022 Oracle, Inc. All rights reserved. Licensed under the [Universal Permissive License v 1.0](https://oss.oracle.com/licenses/upl).\n",
    "\n",
    "---\n",
    "\n",
    "# <font color=\"red\">Text Classification with Data Labeling Service Integration</font>\n",
    "<p style=\"margin-left:10%; margin-right:10%;\">by the <font color=\"teal\">Oracle Cloud Infrastructure Data Science Service.</font></p>\n",
    "\n",
    "---\n",
    "\n",
    "# Overview:\n",
    "\n",
    "Data labeling is the process of identifying specific properties or characteristics of raw data (text and images) and applying annotations called labels. Examples of meaningful labels are the topic of a news article, the sentiment of a tweet, the caption for an image, important words spoken in an audio recording, the genre of a video, etc. The purpose of data labeling is to create enriched, labeled datasets. These labeled datasets are required inputs for many machine learning applications.\n",
    "\n",
    "The Oracle Cloud Infrastructure (OCI) Data Labeling service enables customers to efficiently build enriched, labeled datasets for the purpose of accurately training AI/ML models. This notebook demonstrates operations that can be performed using the Advanced Data Science (ADS) Data Labeling module. The demonstrated operations are:\n",
    "\n",
    "* How to list datasets in the OCI Data Labeling service (DLS).\n",
    "* How to generate a snapshot of a dataset.\n",
    "* How to load a labeled dataset.\n",
    "\n",
    "The purpose of the `data_labeling` module is to provide an efficient and convenient way for users to utilize OCI DLS in a notebook session.\n",
    "\n",
    "Compatible conda pack: [Natural Language Processing](https://docs.oracle.com/iaas/data-science/using/conda-nlp-fam.htm) for CPU on Python 3.7 (version 2.0)\n",
    "\n",
    "---\n",
    "\n",
    "## Contents:\n",
    "\n",
    "- <a href=\"#intro\">Introduction</a>\n",
    "- <a href='#dls'>Data Labeling Service Class</a>\n",
    "    - <a href=\"#dls_list\">Listing Labeled Datasets</a>\n",
    "    - <a href='#dls_snapshot'>Export</a>\n",
    "- <a href='#text_classification'>Text Classification</a>\n",
    "    - <a href=\"#binary_class\">Binary Classification</a>\n",
    "        - <a href=\"#single_dataset\">Dataset</a>\n",
    "            - <a href=\"#single_loaddataset\">Load the Labeled Dataset</a>\n",
    "        - <a href='#single_train'>Decision Tree Classifier</a>\n",
    "        - <a href='#single_test'>Prediction</a>\n",
    "    - <a href=\"#imultilabel_class\">Multilabel Classification</a>\n",
    "        - <a href=\"#multi_dataset\">Dataset</a>\n",
    "            - <a href=\"#multi_loaddataset\">Load the Labeled Dataset</a>\n",
    "        - <a href='#multi_train'>Linear Support Vector Classifier</a>\n",
    "        - <a href='#multi_test'>Evaluation</a>   \n",
    "- <a href='#ref'>References</a>\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "Datasets are provided as a convenience.  Datasets are considered third-party content and are not considered materials \n",
    "under your agreement with Oracle.\n",
    "    \n",
    "You can access the `20 Newsgroups` dataset license [here](https://github.com/scikit-learn/scikit-learn/blob/master/COPYING).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "import logging\n",
    "\n",
    "logging.basicConfig(format=\"%(levelname)s:%(message)s\", level=logging.ERROR)\n",
    "\n",
    "import ads\n",
    "import nltk\n",
    "import oci\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "\n",
    "from ads.data_labeling import DataLabeling\n",
    "from nltk.corpus import stopwords\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "from sklearn.preprocessing import MultiLabelBinarizer\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"intro\"></a>\n",
    "# Introduction\n",
    "\n",
    "The OCI Data Labeling service (DLS) allows customers to create and browse datasets, view data records (text, images), and apply labels to build AI/ML models. The service also provides interactive user interfaces designed to aid in the labeling process. Once records are labeled, the dataset can be exported as line-delimited JSON for machine learning model development. \n",
    "\n",
    "Datasets are the core resource available within the Data Labeling service. Datasets contain records and their associated labels. A record represents a single image or text document. Records are stored by reference to their original source, such as a path on Object Storage. Customers can also upload records from local storage. Labels are annotations that describe a data record. In the case of text, labels may include the beginning and ending character position. Labeled datasets can be exported as a JSON manifest for use as an input to machine learning model development. \n",
    "\n",
    "Training datasets that are used in natural language processing applications require a corpus of raw text that is tagged. The tags indicate the important parts of the text, and labels are assigned to these tags. There are different types of labeling, such as parts of speech (POS) or named entities. POS would identify words or phrases such as a noun, verb, adverb, or adjective. Name entities would identify proper nouns and places within the text, such as Carey and Redwood City. The start and end positions are identified in the text, and a label is associated with that bounded region.\n",
    "\n",
    "<a id='dls'></a>\n",
    "# Data Labeling Service Class\n",
    "\n",
    "The main entry point to working with the DLS in ADS is the `DataLabeling` class. Generally, you will work with the DLS user interface in the console to label the data. The ADS library allows you to work with the labeled datasets within the notebook session efficiently.\n",
    "\n",
    "To obtain a handle to a `DataLabeling` object, call the `DataLabeling()` constructor. The default compartment is the same compartment as the notebook session but the parameter `compartment_id` can be used to select a different compartment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = DataLabeling()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"dls_list\"></a>\n",
    "## Listing Labeled Datasets\n",
    "\n",
    "The `list_dataset()` method provides a list of the available labeled datasets in the compartment set in the previous step when calling `DataLabeling()`. It returns a `DatasetList` object where each row is a dataset. It has the following columns:\n",
    "* `annotation_format`: The format of the annotation. Examples are ENTITY_EXTRACTION, MULTI_LABEL, etc.\n",
    "* `defined_tags`: Preset tags.\n",
    "* `display_name`: Name of the dataset as it is shown in the console.\n",
    "* `format_type`: The format of the dataset, for example, TEXT or IMAGE.\n",
    "* `freeform_tags`: User specified tags.\n",
    "* `id`: OCID of the dataset.\n",
    "* `lifecycle_state`: The lifecycle state of the dataset, such as ACTIVE, DELETED, etc.\n",
    "* `lifecycle_details`: Details about the lifecycle state.\n",
    "* `system_tags`: Tags that are defined by OCI.\n",
    "* `time_created`: The timestamp of when the dataset was created.\n",
    "* `time_updated`: The timestamp when the dataset was last updated. It will be None if the dataset has not been updated.\n",
    "\n",
    "The following cell will list the datasets that are available in the notebook session's compartment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls.list_dataset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='dls_snapshot'></a>\n",
    "## Export\n",
    "\n",
    "To work with the labeled data, a snapshot of the dataset is needed. The `export()` method will copy the labeled data from the DLS into a bucket in Object Storage. The `export()` method has the following parameters:\n",
    "\n",
    "* `dataset_id`: The OCID of the DLS dataset to take a snapshot of.\n",
    "* `path`: The object storage path to store the generated snapshot.\n",
    "\n",
    "The snapshot operation will create a unique file in the specified bucket. The `export()` method will return the key to the file that was created.\n",
    "\n",
    "This notebook does not assume that you have a labeled text dataset in the DLS service. Therefore, it will not attempt to create a snapshot.\n",
    "\n",
    "<a id='text_classification'></a>\n",
    "# Text Classification\n",
    "\n",
    "This section shows you how to build a binary and a multiclass classifier using two different annotated datasets that were labeled by the DLS.  \n",
    "\n",
    "\n",
    "<a id=\"binary_class\"></a>\n",
    "## Binary Classification\n",
    "To demonstrate a typical data science workflow, we will train a binary classifier using the text data as the source for generating features and the annotations to represent the response variable.\n",
    "\n",
    "<a id=\"single_dataset\"></a>\n",
    "### Dataset\n",
    "\n",
    "A subset of the 20 Newsgroups dataset is used in this notebook. The complete dataset is a collection of approximately 20,000 newsgroup documents partitioned across 20 different newsgroups. The dataset is popular for experiments where the machine learning application predicts which newsgroup a record belongs to. The newsgroups are treated as the categories/labels that you would like to predict. For the toy model created in this notebook, only the `rec.sport.baseball`, and `sci.space` newgroups are used, as this is a binary classification model.\n",
    "\n",
    "<a id='single_loaddataset'></a>\n",
    "#### Load the Labeled Dataset\n",
    "\n",
    "The Oracle Data Science service has created a labeled dataset based on the data described in the <a href=\"#single_dataset\">Dataset</a> section. It is stored in a publically accessible bucket and will be used here. The snapshot process creates a JSONL file that contains metadata about the labeled dataset. There is also a record JSONL file that has links to the original dataset and its label.\n",
    "\n",
    "Use the `read_labeled_data()` method to read in the metadata file, record file, and all the corpus documents. Only the metadata file needs to be specified as it contains references to the record and corpus documents. The `read_labeled_data()` method returns a dataframe that is easy to work with.\n",
    "\n",
    "The following cell loads a labeled dataset. It will return the text from each email and the labeled annotation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame.ads.read_labeled_data(\n",
    "    \"oci://hosted-ds-datasets@bigdatadatasciencelarge/DLS/text_single_label_20news/metadata.jsonl\",\n",
    "    auth={\"config\": oci.config.from_file(os.path.join(\"~/.oci\", \"config\"))},\n",
    "    materialize=True,\n",
    ")\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='single_train'></a>\n",
    "### Decision Tree Classifier \n",
    "\n",
    "The first step is to perform some preprocessing on the data. The following cell performs the following operations:\n",
    "* Converts the text to lower case.\n",
    "* Uses a Regular Expression (RegEx) command to remove any character that is not alphanumeric, underscore, or whitespace.\n",
    "* Replace the sequence of characters `\\n` with a space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"text_clean\"] = (\n",
    "    df[\"Content\"].str.lower().str.replace(r\"[^\\w\\s]+\", \"\").str.replace(\"\\n\", \" \")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The binary classifier model we will train is a decision tree where the features are based on n-grams of the words. We will use n-grams that are one, two, and three words long: unigrams, bigrams, and trigrams. The vectorizer will remove English stop words as they provide little value to the model being built. A weight will be assigned to these features using the term frequency-inverse document frequency (TF*IDF) approach. More details about TF*IDF can be found [here](https://en.wikipedia.org/wiki/Tf–idf).\n",
    "\n",
    "In this example, we will skip splitting the dataset into the training and test sets since our goal is to build a toy model for a demonstration purpose. In the following, we assign `0` for the `rec.sport.baseball` label and `1` for the `sci.space` label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = TfidfVectorizer(stop_words=\"english\", analyzer=\"word\", ngram_range=(1, 3))\n",
    "classifier = DecisionTreeClassifier()\n",
    "\n",
    "feature = vectorizer.fit_transform(df[\"text_clean\"])\n",
    "model = classifier.fit(feature, df[\"Annotations\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='single_test'></a>\n",
    "### Prediction\n",
    "\n",
    "Use the following code to predict the category for a given text data using the trained binary classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier.predict(vectorizer.transform([\"reggie jackson played right field\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"multilabel_class\"></a>\n",
    "## Multilabel Classification\n",
    "\n",
    "In the previous section, you saw how to build a binary text classifier. Building a multiclass text classifier is a similar process. However, a one-vs-the-rest (OvR) multiclass strategy will be employed. That is, you create one classifier for each class. The classifier will predict if the observation is in the class or not. If there are m classes then there will be m classifiers. Classification will be based on which classifier has the more confidence that an observation is in the class.\n",
    "\n",
    "<a id=\"multi_dataset\"></a>\n",
    "### Dataset\n",
    "\n",
    "A subset of the [Reuters Corpus](https://archive.ics.uci.edu/ml/datasets/reuters-21578+text+categorization+collection) dataset is used in this notebook. You will use scikit-learn and nltk to build a multilabel classifier. The Reuters data is a benchmark dataset for document classification. More precisely, it is a multilabel (each document can belong to many classes) dataset. It has 90 categories, 7,769 training documents, and 3,019 testing documents.\n",
    "\n",
    "<a id='multi_loaddataset'></a>\n",
    "#### Load the Labeled Dataset\n",
    "\n",
    "The following cell loads a multi-labeled dataset. It will return the text and the multi-labeled annotation in a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame.ads.read_labeled_data(\n",
    "    \"oci://hosted-ds-datasets@bigdatadatasciencelarge/DLS/text_multi_label_nltk_reuters/metadata.jsonl\",\n",
    "    auth={\"config\": oci.config.from_file(os.path.join(\"~/.oci\", \"config\"))},\n",
    "    materialize=True,\n",
    ")\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='multi_train'></a>\n",
    "### Linear Support Vector Classifier\n",
    "\n",
    "To demonstrate a typical data science workflow, you will train a `LinearSVC` classifier using the text data to generate features and annotations to represent the response variable.\n",
    "\n",
    "We utilize the `MultiLabelBinarizer()` method to convert the labels into the scikit-learn classification format during the dataset preprocessing. This transformer converts a list of sets or tuples into the supported multilabel format, a binary matrix of `samples*classes`. Further details about how this works can be found [here](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MultiLabelBinarizer.html).\n",
    "\n",
    "The next step is to vectorize the input text to feed it into a supervised machine learning system. In this example, TF*IDF vectorization is used in a manner that is similar to what was used in the <a href=\"#single_train\">Decision Tree Classifier</a> section.\n",
    "\n",
    "For performance reasons, the `TfidfVectorizer` is limited to 10,000 words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download(\"stopwords\")\n",
    "\n",
    "stop_words = stopwords.words(\n",
    "    \"english\"\n",
    ")  ## See scikit-learn documentation for what these words are\n",
    "vectorizer = TfidfVectorizer(stop_words=stop_words, max_features=10000)\n",
    "mlb = MultiLabelBinarizer()\n",
    "\n",
    "X_train = vectorizer.fit_transform(df[\"Content\"])  ## Vectorize the inputs with tf-idf\n",
    "y_train = mlb.fit_transform(df[\"Annotations\"])  ## Vectorize the labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A one-vs-rest (OvR) strategy is used in this model. This approach is a heuristic method that uses one class versus the rest of classes, in which we train m binary classifiers, where m is the number of classes in the training dataset. The data from the study class is treated as positive, and the data from all the other classes is treated as negative, [See here](https://probml.github.io/pml-book/book0.html).\n",
    "\n",
    "The following cell uses the scalable Linear Support Vector Machine, `LinearSVC`, for classification. It is quick to train and empirically adequate on NLP problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = OneVsRestClassifier(LinearSVC(class_weight=\"balanced\"), n_jobs=-1)\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how the model did! The next cell applies cross-validation to estimate the prediction error. `K` fold cross-validation works by partitioning a dataset into `K` splits. For the `k`th part, it fits the model to the other `K-1` splits of the data and calculates the prediction error. It uses the `k`th part to do this prediction. For more details about this process, see [here](https://en.wikipedia.org/wiki/Cross-validation_(statistics)) and specifically this [image](https://upload.wikimedia.org/wikipedia/commons/thumb/b/b5/K-fold_cross_validation_EN.svg/1920px-K-fold_cross_validation_EN.svg.png).\n",
    "\n",
    "\n",
    "<a id='multi_test'></a>\n",
    "### Evaluation\n",
    "\n",
    "By performing cross-validation, there will be five separate models trained on different train and test splits. The goal is to get an estimate of the error that is expected when the model is generalized to an independent dataset. The following cell uses the [`cross_val_score`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html) method to estimate the mean and standard deviation of errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_val_score(clf, X_train, y_train, cv=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref\"></a>\n",
    "# References\n",
    "\n",
    "- [ADS Library Documentation](https://accelerated-data-science.readthedocs.io/en/latest/index.html)\n",
    "- [Data Science YouTube Videos](https://www.youtube.com/playlist?list=PLKCk3OyNwIzv6CWMhvqSB_8MLJIZdO80L)\n",
    "- [OCI Data Science Documentation](https://docs.cloud.oracle.com/en-us/iaas/data-science/using/data-science.htm)\n",
    "- [Oracle Data & AI Blog](https://blogs.oracle.com/datascience/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "aee8b7b246df8f9039afb4144a1f6fd8d2ca17a180786b69acc140d282b71a49"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}