notebooks/Propensity Model Selection.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Propensity Score Weighting in CausalTune \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<frozen importlib._bootstrap>:228: RuntimeWarning: scipy._lib.messagestream.MessageStream size changed, may indicate binary incompatibility. Expected 56 from C header, got 64 from PyObject\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import sys\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import warnings\n",
    "\n",
    "from scipy.stats import betabinom\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "root_path = root_path = os.path.realpath('../..')\n",
    "try:\n",
    "    import causaltune\n",
    "except ModuleNotFoundError:\n",
    "    sys.path.append(os.path.join(root_path, \"causaltune\"))\n",
    "\n",
    "from causaltune import CausalTune\n",
    "from causaltune.data_utils import CausalityDataset\n",
    "from causaltune.datasets import generate_non_random_dataset\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "CausalTune effect estimation consists of multiple models that can / need to be fitted.\n",
    "\n",
    "1. Propensity model to estimate treatment propensities from features $\\mathbb{E}[T|X,W]$.\n",
    "2. Outcome model to estimate outcomes from features $\\mathbb{E}[Y|X,W]$\n",
    "3. The final causal inference estimator which requires additional hyperparamter tuning.\n",
    "\n",
    "In this notebook, we focu on the Propensity Score Weighting (1.).\n",
    "\n",
    "There are four options to finding a propensity model.\n",
    "\n",
    "1. **[Default:] use a dummy estimator.**\n",
    "   - natural option for a computationally easy model / when perfect randomisation of the treatment is given \n",
    "\n",
    "\n",
    "2. **Letting AutoML fit the propensity model,**\n",
    "   - has all the advantages of using an elaborate propensity weighting model  \n",
    "\n",
    "\n",
    "3. **supply a custom sklearn-compatible prediction model,**\n",
    "   - for more flexibility in terms of propensity prediction model\n",
    "\n",
    "\n",
    "4. **supply an array of custom propensities to treat.** \n",
    "   - can be used, e.g. with custom propensities to treat based on an optimisation procedure such as Thompson sampling when there is an expected benefit from treating some subjects with higher propensity than others"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd = generate_non_random_dataset()\n",
    "cd.preprocess_dataset()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this synthetic dataset, the **true (constant) treatment effect** is $0.2$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CausalTune configuration\n",
    "components_time_budget = 40\n",
    "train_size = 0.7\n",
    "\n",
    "target = cd.outcomes[0]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. DEFAULT: Dummy propensity model\n",
    "\n",
    "\n",
    "The dummy propensity model identifies a constant propensity to treat given by $\\frac{\\text{Treatment Group Size}}{\\text{Total Sample Size}}  $."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting a Propensity-Weighted scoring estimator to be used in scoring tasks\n",
      "Initial configs: [{'estimator': {'estimator_name': 'backdoor.causaltune.models.NaiveDummy'}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.Dummy'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.SLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.DomainAdaptationLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.dr.ForestDRLearner', 'min_propensity': 1e-06, 'n_estimators': 100, 'min_samples_split': 5, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.econml.dml.CausalForestDML', 'drate': True, 'n_estimators': 100, 'criterion': 'mse', 'min_samples_split': 10, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'fit_intercept': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.TransformedOutcome'}}]\n"
     ]
    }
   ],
   "source": [
    "ct = CausalTune(\n",
    "    propensity_model='dummy',\n",
    "    components_time_budget=components_time_budget,\n",
    "    metric=\"energy_distance\",\n",
    "    train_size=train_size,\n",
    "    verbose=0\n",
    ")   \n",
    "ct.fit(data=cd, outcome=target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Propensity model: DummyClassifier()\n"
     ]
    }
   ],
   "source": [
    "print(f'Propensity model: {ct.propensity_model}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Difference in means estimate (naive ATE):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.30755\n"
     ]
    }
   ],
   "source": [
    "print(f'{ct.scorer.naive_ate(cd.data[cd.treatment], cd.data[target])[0]:.5f}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CausalTune ATE estimate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.26557\n"
     ]
    }
   ],
   "source": [
    "print(f'{ct.effect(ct.test_df).mean():.5f}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Propensity model estimation via AutoML\n",
    "\n",
    "The propensity score weighting estimation via AutoML is as simple as selecting `propensity_model='auto'`. \n",
    "\n",
    "The computational intensity can then be adapted via supplying a `components_budget_time`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting a Propensity-Weighted scoring estimator to be used in scoring tasks\n",
      "Initial configs: [{'estimator': {'estimator_name': 'backdoor.causaltune.models.NaiveDummy'}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.Dummy'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.SLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.DomainAdaptationLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.dr.ForestDRLearner', 'min_propensity': 1e-06, 'n_estimators': 100, 'min_samples_split': 5, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.econml.dml.CausalForestDML', 'drate': True, 'n_estimators': 100, 'criterion': 'mse', 'min_samples_split': 10, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'fit_intercept': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.TransformedOutcome'}}]\n",
      "Estimated ATE: 0.34498\n"
     ]
    }
   ],
   "source": [
    "ct = CausalTune(\n",
    "    propensity_model='auto',\n",
    "    components_time_budget=components_time_budget,\n",
    "    metric=\"energy_distance\",\n",
    "    train_size=train_size,\n",
    "    verbose=0\n",
    ")   \n",
    "ct.fit(data=cd, outcome=target)\n",
    "print(f'Estimated ATE: {ct.effect(ct.test_df).mean():.5f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Propensity model: AutoML(append_log=False, auto_augment=True, custom_hp={},\n",
      "       cv_score_agg_func=None, early_stop=False, ensemble=False,\n",
      "       estimator_list='auto', eval_method='holdout', fit_kwargs_by_estimator={},\n",
      "       hpo_method='auto', keep_search_state=False, learner_selector='sample',\n",
      "       log_file_name='', log_training_metric=False, log_type='better',\n",
      "       max_iter=None, mem_thres=4294967296, metric='auto',\n",
      "       metric_constraints=[('pred_time', '<=', 1e-05)], min_sample_size=10000,\n",
      "       model_history=False, n_concurrent_trials=1, n_jobs=-1, n_splits=5,\n",
      "       pred_time_limit=1e-05, preserve_checkpoint=True, retrain_full=True,\n",
      "       sample=True, skip_transform=False, split_ratio=0.30000000000000004, ...)\n"
     ]
    }
   ],
   "source": [
    "print(f'Propensity model: {ct.propensity_model}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Propensity model estimation with a custom model\n",
    "\n",
    "A custom propensity model that has an sklearn-style `fit` and `predict_proba` method can be supplied as a propensity model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting a Propensity-Weighted scoring estimator to be used in scoring tasks\n",
      "Initial configs: [{'estimator': {'estimator_name': 'backdoor.causaltune.models.NaiveDummy'}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.Dummy'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.SLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.DomainAdaptationLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.dr.ForestDRLearner', 'min_propensity': 1e-06, 'n_estimators': 100, 'min_samples_split': 5, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.econml.dml.CausalForestDML', 'drate': True, 'n_estimators': 100, 'criterion': 'mse', 'min_samples_split': 10, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'fit_intercept': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.TransformedOutcome'}}]\n",
      "Estimated ATE: 0.24008\n"
     ]
    }
   ],
   "source": [
    "propensity_model = RandomForestClassifier()\n",
    "\n",
    "ct = CausalTune(\n",
    "    propensity_model=propensity_model,\n",
    "    components_time_budget=components_time_budget,\n",
    "    metric=\"energy_distance\",\n",
    "    train_size=train_size,\n",
    ")   \n",
    "ct.fit(data=cd, outcome=target)\n",
    "print(f'Estimated ATE: {ct.effect(ct.test_df).mean():.5f}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Propensity model: RandomForestClassifier()\n"
     ]
    }
   ],
   "source": [
    "print(f'Propensity model: {ct.propensity_model}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Supplying individual treatment propensities\n",
    "\n",
    "In some settings such as uplift modelling, the experiment / study is based on heterogeneous treatment propensities known to the researcher / experimenter. An array of treatment propensities can be directly supplied to CausalTune in the data instantiation of the `CausalityDataset`. This can, e.g. be done by \n",
    "```\n",
    "cd = CausalityDataset(\n",
    "    ...\n",
    "    propensity_modifiers=[<individual_treatment_propensity_column_name>]\n",
    "    ...\n",
    ")\n",
    "```\n",
    "and then using the `passthrough_model` as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   T         Y  random        X1        X2        X3        X4        X5  \\\n",
      "0  0 -1.000312     0.0  0.259595 -0.994360  0.122632 -0.308056  2.110752   \n",
      "1  0  2.342408     1.0 -0.357165 -1.626471  0.768395  0.239236  0.874304   \n",
      "2  0 -1.087664     0.0 -0.780095 -1.917028 -0.156848  0.437076  0.516383   \n",
      "3  1  0.398676     1.0 -0.951582 -0.433123  1.299038  0.193750  1.311885   \n",
      "4  0  0.897118     1.0 -0.341460 -1.668032 -0.340667  0.548328  1.646835   \n",
      "\n",
      "   propensity  \n",
      "0    0.273852  \n",
      "1    0.148065  \n",
      "2    0.136952  \n",
      "3    0.213419  \n",
      "4    0.188777  \n",
      "True propensities to treat: ['propensity']\n",
      "Fitting a Propensity-Weighted scoring estimator to be used in scoring tasks\n",
      "Initial configs: [{'estimator': {'estimator_name': 'backdoor.causaltune.models.NaiveDummy'}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.Dummy'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.SLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.metalearners.DomainAdaptationLearner'}}, {'estimator': {'estimator_name': 'backdoor.econml.dr.ForestDRLearner', 'min_propensity': 1e-06, 'n_estimators': 100, 'min_samples_split': 5, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.econml.dml.CausalForestDML', 'drate': True, 'n_estimators': 100, 'criterion': 'mse', 'min_samples_split': 10, 'min_samples_leaf': 5, 'min_weight_fraction_leaf': 0.0, 'max_features': 'auto', 'min_impurity_decrease': 0.0, 'max_samples': 0.45, 'min_balancedness_tol': 0.45, 'honest': True, 'fit_intercept': True, 'subforest_size': 4}}, {'estimator': {'estimator_name': 'backdoor.causaltune.models.TransformedOutcome'}}]\n",
      "Estimated ATE: 0.27060\n"
     ]
    }
   ],
   "source": [
    "from causaltune.models.passthrough import passthrough_model\n",
    "\n",
    "print(cd.data.head())\n",
    "print(f'True propensities to treat: {cd.propensity_modifiers}')\n",
    "\n",
    "propensity_model=passthrough_model(\n",
    "    cd.propensity_modifiers, include_control=False\n",
    "    )\n",
    "\n",
    "ct = CausalTune(\n",
    "    propensity_model=propensity_model,\n",
    "    components_time_budget=components_time_budget,\n",
    "    metric=\"energy_distance\",\n",
    "    train_size=train_size,\n",
    "    verbose=0\n",
    ")   \n",
    "ct.fit(data=cd, outcome=target)\n",
    "print(f'Estimated ATE: {ct.effect(ct.test_df).mean():.5f}')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "causality",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}