notebooks/Data-OOB-on-Magictelescope.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f7a2acc7",
   "metadata": {},
   "source": [
    "# DataOOB on Magictelescope dataset\n",
    "- This Jupyter notebook demonstrates Data-OOB and existing data valuation methods including leave-one-out, KNNShap, DataShap, and BetaShap. \n",
    "- We use the \"Magictelescope\" dataset. An instruction to download other datasets is available at `dataoob/preprocess`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c227ed9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import numpy as np\n",
    "import sys\n",
    "sys.path.append('../dataoob')\n",
    "import datasets\n",
    "from data_valuation import DataValuation\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "aa7f5130",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset='magictelescope'\n",
    "problem='clf'\n",
    "dargs = {'n_data_to_be_valued':300, \n",
    "        'n_val':30, \n",
    "        'n_test':3000,\n",
    "        'n_trees':800,\n",
    "        'openml_clf_path':'../dataoob/preprocess/tmp/dataset_clf_openml',\n",
    "        'is_noisy':0.1,\n",
    "        'model_family':'Tree',\n",
    "        'run_id':0}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebc18d7d",
   "metadata": {},
   "source": [
    "# Load dataset and prepare DataValuation engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "11c27bf7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------\n",
      "{'n_data_to_be_valued': 300, 'n_val': 30, 'n_test': 3000, 'n_trees': 800, 'openml_clf_path': '../dataoob/preprocess/tmp/dataset_clf_openml', 'is_noisy': 0.1, 'model_family': 'Tree', 'run_id': 0}\n",
      "--------------------------------------------------\n",
      "MagicTelescope\n",
      "--------------------------------------------------\n",
      "Train X: (300, 10)\n",
      "Val X: (30, 10)\n",
      "Test X: (3000, 10)\n",
      "------------------------------\n"
     ]
    }
   ],
   "source": [
    "# Load dataset\n",
    "(X, y), (X_val, y_val), (X_test, y_test), noisy_index=datasets.load_data(problem, dataset, **dargs)\n",
    "\n",
    "# instantiate data valuation engine\n",
    "data_valuation_engine=DataValuation(X=X, y=y, \n",
    "                                    X_val=X_val, y_val=y_val, \n",
    "                                    problem=problem, dargs=dargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14bb87ae",
   "metadata": {},
   "source": [
    "# Compute data values\n",
    "- `compute_marginal_contribution_based_methods` computes marginal-based methods, namely Leave-one-out, DataShap, KNNShap, and BetaShap. If the marginal contribution computation takes too long, you can skip it by feeding `betashap_run=False`. \n",
    "- `compute_oob_and_ame` computes Data-OOB and AME."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "96d7cf9b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Start: KNN_Shapley computation\n",
      "Done: KNN_Shapley computation\n",
      "Start: LOO computation\n",
      "Done: LOO computation\n",
      "Start: Beta_Shapley computation\n",
      "Start: marginal contribution computation\n",
      "Total number of random sets: 1000, GR_stat: 1.0071758936586486\n",
      "Done: marginal contribution computation\n",
      "Done: Beta_Shapley computation\n",
      "Start: OOB computation\n",
      "Done: OOB computation\n",
      "Start: AME computation\n",
      "Done: AME computation\n",
      "CPU times: user 3min 25s, sys: 242 ms, total: 3min 26s\n",
      "Wall time: 3min 26s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# compute data values\n",
    "data_valuation_engine.compute_marginal_contribution_based_methods(betashap_run=True)\n",
    "data_valuation_engine.compute_oob_and_ame()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3402df3",
   "metadata": {},
   "source": [
    "# Evaluate the quality of data values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b1801e9a",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data_valuation_engine.evaluate_data_values(noisy_index, X_test, y_test, removal_run=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "829dc0bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Meta_Data': ['Recall', 'Kmeans_label'],\n",
       " 'Results': {'KNN_Shapley': [0.3333333333333333, 0.31205673758865254],\n",
       "  'LOO_last': [0.2, 0.18181818181818182],\n",
       "  'Data_Shapley': [0.3333333333333333, 0.27692307692307694],\n",
       "  'Beta_Shapley(16,1)': [0.23333333333333334, 0.2962962962962963],\n",
       "  'Beta_Shapley': [0.26666666666666666, 0.30434782608695654],\n",
       "  'OOB': [0.4666666666666667, 0.43333333333333335],\n",
       "  'AME': [0.1, 0.17737003058103976]}}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The larger, the better\n",
    "data_valuation_engine.noisy_detect_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61da9e12",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80814a8b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "267907e7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}