Module-03g-Model-HyperParameterOpt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to parameter tuning\n",
    "\n",
    "**Hyper-parameters**\n",
    "\n",
    "A machine learning model is a mathematical formula with a number of parameters that are learnt from the data. That is the crux of machine learning: fitting a model to the data.\n",
    "\n",
    "However, there is another kind of parameters that cannot be directly learned from the regular training process. These parameters express “higher-level” properties of the model such as its complexity or how fast it should learn. They are called hyperparameters. Hyperparameters are usually fixed before the actual training process begins.\n",
    "\n",
    "So, how are hyperparameters decided?\n",
    "\n",
    "Broadly speaking, this is done by setting different values for those hyperparameters, training different models, and deciding which ones work best by testing them\n",
    "\n",
    "So, to summarize. Hyperparameters:\n",
    "\n",
    "- Define higher level concepts about the model such as complexity, or capacity to learn.\n",
    "- Cannot be learned directly from the data in the standard model training process and need to be predefined.\n",
    "- Can be decided by setting different values, training different models, and choosing the values that test better\n",
    "\n",
    "Some examples of hyperparameters:\n",
    "\n",
    "- Number of leaves or depth of a tree\n",
    "- Number of latent factors in a matrix factorization\n",
    "- Learning rate (in many models)\n",
    "- Number of hidden layers in a deep neural network\n",
    "- Number of clusters in a k-means clustering\n",
    "\n",
    "source: [Quora](https://www.quora.com/What-are-hyperparameters-in-machine-learning)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.style.use('fivethirtyeight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Read the data \n",
    "#Read the data\n",
    "df = pd.read_csv(\"data/historical_loan.csv\")\n",
    "\n",
    "# refine the data\n",
    "df.years = df.years.fillna(np.mean(df.years))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Setup the features and target\n",
    "X = df.iloc[:,1:]\n",
    "y = df.iloc[:,0]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Basic checks**\n",
    "\n",
    "Check if the columns are the same in train and test.\n",
    "\n",
    "What else will you check?  [**Discuss**]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['amount', 'grade', 'years', 'ownership', 'income', 'age'], dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['amount', 'grade', 'years', 'ownership', 'income', 'age'], dtype='object')"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6181, 6) (1546, 6)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train\n",
      "amount         int64\n",
      "grade         object\n",
      "years        float64\n",
      "ownership     object\n",
      "income       float64\n",
      "age            int64\n",
      "dtype: object\n",
      "\n",
      "test\n",
      "amount         int64\n",
      "grade         object\n",
      "years        float64\n",
      "ownership     object\n",
      "income       float64\n",
      "age            int64\n",
      "dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(\"train\")\n",
    "print(X_train.dtypes)\n",
    "print()\n",
    "print(\"test\")\n",
    "print(X_test.dtypes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The categorical data should be encoded.\n",
    "\n",
    "We saw LabelEncoder earlier. Now, we will use one-hot encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-hot encoding\n",
    "\n",
    "![](img/onehot.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_updated = pd.get_dummies(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6181, 6)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(6181, 15)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_updated.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "amount                14500.0\n",
       "years                    11.0\n",
       "income                64000.0\n",
       "age                      35.0\n",
       "grade_A                   1.0\n",
       "grade_B                   0.0\n",
       "grade_C                   0.0\n",
       "grade_D                   0.0\n",
       "grade_E                   0.0\n",
       "grade_F                   0.0\n",
       "grade_G                   0.0\n",
       "ownership_MORTGAGE        1.0\n",
       "ownership_OTHER           0.0\n",
       "ownership_OWN             0.0\n",
       "ownership_RENT            0.0\n",
       "Name: 303, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#print the first record\n",
    "X_train_updated.iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**\n",
    "Apply one-hot encoding to test dataset and store in test_updated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test_updated = pd.get_dummies(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1546, 6) (1546, 15)\n"
     ]
    }
   ],
   "source": [
    "print(X_test.shape, X_test_updated.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "amount                 3000.0\n",
       "years                     1.0\n",
       "income                49800.0\n",
       "age                      22.0\n",
       "grade_A                   1.0\n",
       "grade_B                   0.0\n",
       "grade_C                   0.0\n",
       "grade_D                   0.0\n",
       "grade_E                   0.0\n",
       "grade_F                   0.0\n",
       "grade_G                   0.0\n",
       "ownership_MORTGAGE        0.0\n",
       "ownership_OTHER           0.0\n",
       "ownership_OWN             0.0\n",
       "ownership_RENT            1.0\n",
       "Name: 2184, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#print the first record\n",
    "X_test_updated.iloc[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(6181, 15) (6181,)\n"
     ]
    }
   ],
   "source": [
    "print(X_train_updated.shape, y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Let's build random forest model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_rf = RandomForestClassifier(n_estimators=100,\n",
    "                                 criterion=\"gini\",\n",
    "                                 max_depth=5,\n",
    "                                 min_samples_split=2,\n",
    "                                 min_samples_leaf= 1,\n",
    "                                 oob_score=True,\n",
    "                                 n_jobs=-1\n",
    "                                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=5, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=-1,\n",
       "            oob_score=True, random_state=None, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_rf.fit(X_train_updated, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.63873159682899205"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_rf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's do cross validation and see what the generalization error is"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.metrics import roc_curve, auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_rf = RandomForestClassifier(n_estimators=100,\n",
    "                                 criterion=\"gini\",\n",
    "                                 max_depth=5,\n",
    "                                 min_samples_split=2,\n",
    "                                 min_samples_leaf= 1,\n",
    "                                 oob_score=True,\n",
    "                                 n_jobs=-1\n",
    "                                 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 112 ms, sys: 64.7 ms, total: 176 ms\n",
      "Wall time: 2.18 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "#Or use %%timeit -n1 -r1 to time the cell\n",
    "\n",
    "cross_val_score_rf = cross_val_score(model_rf, \n",
    "                                     X_train_updated, \n",
    "                                     y_train, scoring=\"roc_auc\",\n",
    "                                     cv=5,\n",
    "                                     n_jobs=-1\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.6969647 ,  0.68786796,  0.69946444,  0.69435555,  0.67146693])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score_rf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.69002391398907892"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#What is the average cross validation score?\n",
    "np.mean(cross_val_score_rf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### grid-search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above was for some arbitrary chosen parameter value.\n",
    "\n",
    "How do we run the model on various choices of hyper-parameters?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Tuning hyper-parameters for roc_auc\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'max_depth': 6, 'n_estimators': 100}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.684 (+/-0.022) for {'max_depth': 3, 'n_estimators': 50}\n",
      "0.684 (+/-0.022) for {'max_depth': 3, 'n_estimators': 100}\n",
      "0.687 (+/-0.018) for {'max_depth': 4, 'n_estimators': 50}\n",
      "0.687 (+/-0.022) for {'max_depth': 4, 'n_estimators': 100}\n",
      "0.687 (+/-0.016) for {'max_depth': 5, 'n_estimators': 50}\n",
      "0.690 (+/-0.021) for {'max_depth': 5, 'n_estimators': 100}\n",
      "0.691 (+/-0.022) for {'max_depth': 6, 'n_estimators': 50}\n",
      "0.692 (+/-0.020) for {'max_depth': 6, 'n_estimators': 100}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "AUC: 0.630219677953\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.63      0.71      0.67       807\n",
      "          1       0.64      0.55      0.59       739\n",
      "\n",
      "avg / total       0.63      0.63      0.63      1546\n",
      "\n",
      "\n",
      "1 loop, best of 1: 22.9 s per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n1 -r1\n",
    "\n",
    "# Set the parameters by cross-validation\n",
    "tuned_parameters = [{'n_estimators': [50,100], \n",
    "                     'max_depth': [3, 4, 5, 6]\n",
    "                    }]\n",
    "\n",
    "scores = ['roc_auc']\n",
    "\n",
    "for score in scores:\n",
    "    print(\"# Tuning hyper-parameters for %s\" % score)\n",
    "    print()\n",
    "\n",
    "    clf = GridSearchCV(RandomForestClassifier(n_jobs=-1), \n",
    "                       tuned_parameters, cv=5,\n",
    "                       scoring='%s' % score)\n",
    "    clf\n",
    "    clf.fit(X_train_updated, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print()\n",
    "    print(clf.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    print()\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "    print()\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    print()\n",
    "    print(\"The model is trained on the full development set.\")\n",
    "    print(\"The scores are computed on the full evaluation set.\")\n",
    "    print()\n",
    "    y_true, y_pred = y_test,  clf.predict(X_test_updated)\n",
    "    \n",
    "    false_positive_rate, true_positive_rate, thresholds = roc_curve(y_true, y_pred)\n",
    "    roc_auc = auc(false_positive_rate, true_positive_rate)\n",
    "    print(\"AUC:\", roc_auc)\n",
    "    \n",
    "    print(classification_report(y_true, y_pred))\n",
    "    print()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise**\n",
    "\n",
    "- For `max_depth` include - 6, 10\n",
    "- Add `min_samples_split`, `min_samples_leaf` to the grid search\n",
    "- In addition to `roc_auc`, add `precision` and `recall` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Challenges with `grid_search`**\n",
    "\n",
    "Discuss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Randomized Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "from scipy.stats import randint as sp_randint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Tuning hyper-parameters for roc_auc\n",
      "\n",
      "Best parameters set found on development set:\n",
      "\n",
      "{'bootstrap': False, 'criterion': 'gini', 'max_depth': None, 'max_features': 3, 'min_samples_leaf': 4, 'min_samples_split': 5, 'n_estimators': 50}\n",
      "\n",
      "Grid scores on development set:\n",
      "\n",
      "0.702 (+/-0.024) for {'bootstrap': True, 'criterion': 'gini', 'max_depth': None, 'max_features': 7, 'min_samples_leaf': 5, 'min_samples_split': 4, 'n_estimators': 100}\n",
      "0.686 (+/-0.020) for {'bootstrap': True, 'criterion': 'gini', 'max_depth': 4, 'max_features': 5, 'min_samples_leaf': 7, 'min_samples_split': 7, 'n_estimators': 50}\n",
      "0.687 (+/-0.016) for {'bootstrap': True, 'criterion': 'gini', 'max_depth': 4, 'max_features': 8, 'min_samples_leaf': 9, 'min_samples_split': 2, 'n_estimators': 100}\n",
      "0.685 (+/-0.018) for {'bootstrap': False, 'criterion': 'gini', 'max_depth': 4, 'max_features': 8, 'min_samples_leaf': 7, 'min_samples_split': 2, 'n_estimators': 100}\n",
      "0.690 (+/-0.019) for {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 6, 'max_features': 4, 'min_samples_leaf': 1, 'min_samples_split': 10, 'n_estimators': 50}\n",
      "0.685 (+/-0.019) for {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 6, 'min_samples_leaf': 3, 'min_samples_split': 2, 'n_estimators': 50}\n",
      "0.703 (+/-0.024) for {'bootstrap': True, 'criterion': 'entropy', 'max_depth': None, 'max_features': 1, 'min_samples_leaf': 4, 'min_samples_split': 5, 'n_estimators': 100}\n",
      "0.682 (+/-0.023) for {'bootstrap': False, 'criterion': 'entropy', 'max_depth': 3, 'max_features': 10, 'min_samples_leaf': 3, 'min_samples_split': 10, 'n_estimators': 100}\n",
      "0.697 (+/-0.023) for {'bootstrap': True, 'criterion': 'entropy', 'max_depth': None, 'max_features': 9, 'min_samples_leaf': 5, 'min_samples_split': 2, 'n_estimators': 50}\n",
      "0.684 (+/-0.018) for {'bootstrap': False, 'criterion': 'gini', 'max_depth': 3, 'max_features': 7, 'min_samples_leaf': 8, 'min_samples_split': 4, 'n_estimators': 100}\n",
      "0.686 (+/-0.019) for {'bootstrap': False, 'criterion': 'entropy', 'max_depth': 4, 'max_features': 5, 'min_samples_leaf': 7, 'min_samples_split': 7, 'n_estimators': 50}\n",
      "0.692 (+/-0.019) for {'bootstrap': True, 'criterion': 'gini', 'max_depth': 6, 'max_features': 8, 'min_samples_leaf': 9, 'min_samples_split': 3, 'n_estimators': 100}\n",
      "0.693 (+/-0.020) for {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 6, 'max_features': 4, 'min_samples_leaf': 7, 'min_samples_split': 8, 'n_estimators': 100}\n",
      "0.692 (+/-0.018) for {'bootstrap': True, 'criterion': 'gini', 'max_depth': 6, 'max_features': 5, 'min_samples_leaf': 7, 'min_samples_split': 5, 'n_estimators': 100}\n",
      "0.703 (+/-0.021) for {'bootstrap': False, 'criterion': 'entropy', 'max_depth': None, 'max_features': 2, 'min_samples_leaf': 6, 'min_samples_split': 6, 'n_estimators': 50}\n",
      "0.692 (+/-0.019) for {'bootstrap': True, 'criterion': 'gini', 'max_depth': 6, 'max_features': 8, 'min_samples_leaf': 3, 'min_samples_split': 10, 'n_estimators': 100}\n",
      "0.689 (+/-0.020) for {'bootstrap': False, 'criterion': 'entropy', 'max_depth': 6, 'max_features': 6, 'min_samples_leaf': 4, 'min_samples_split': 10, 'n_estimators': 50}\n",
      "0.701 (+/-0.022) for {'bootstrap': True, 'criterion': 'entropy', 'max_depth': None, 'max_features': 4, 'min_samples_leaf': 8, 'min_samples_split': 2, 'n_estimators': 100}\n",
      "0.704 (+/-0.026) for {'bootstrap': False, 'criterion': 'gini', 'max_depth': None, 'max_features': 3, 'min_samples_leaf': 4, 'min_samples_split': 5, 'n_estimators': 50}\n",
      "0.687 (+/-0.020) for {'bootstrap': True, 'criterion': 'entropy', 'max_depth': 4, 'max_features': 2, 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 100}\n",
      "\n",
      "Detailed classification report:\n",
      "\n",
      "The model is trained on the full development set.\n",
      "The scores are computed on the full evaluation set.\n",
      "\n",
      "1 loop, best of 1: 30.7 s per loop\n"
     ]
    }
   ],
   "source": [
    "%%timeit -n1 -r1\n",
    "\n",
    "# Set the parameters by cross-validation\n",
    "tuned_parameters = { \"n_estimators\": [50,100], \n",
    "                      \"max_depth\": [3, 4, 6, None],\n",
    "                      \"max_features\": sp_randint(1, 11),\n",
    "                      \"min_samples_split\": sp_randint(2, 11),\n",
    "                      \"min_samples_leaf\": sp_randint(1, 11),\n",
    "                      \"bootstrap\": [True, False],\n",
    "                      \"criterion\": [\"gini\", \"entropy\"]\n",
    "                    }\n",
    "\n",
    "scores = ['roc_auc']\n",
    "\n",
    "\n",
    "n_iter_search = 20\n",
    "\n",
    "for score in scores:\n",
    "    print(\"# Tuning hyper-parameters for %s\" % score)\n",
    "    print()\n",
    "\n",
    "    clf = RandomizedSearchCV(RandomForestClassifier(n_jobs=-1), \n",
    "                       param_distributions = tuned_parameters, \n",
    "                             n_iter = n_iter_search,\n",
    "                             n_jobs=-1,\n",
    "                             cv=5,\n",
    "                       scoring='%s' % score)\n",
    "    clf.fit(X_train_updated, y_train)\n",
    "\n",
    "    print(\"Best parameters set found on development set:\")\n",
    "    print()\n",
    "    print(clf.best_params_)\n",
    "    print()\n",
    "    print(\"Grid scores on development set:\")\n",
    "    print()\n",
    "    means = clf.cv_results_['mean_test_score']\n",
    "    stds = clf.cv_results_['std_test_score']\n",
    "    for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "        print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "              % (mean, std * 2, params))\n",
    "    print()\n",
    "\n",
    "    print(\"Detailed classification report:\")\n",
    "    print()\n",
    "    print(\"The model is trained on the full development set.\")\n",
    "    print(\"The scores are computed on the full evaluation set.\")\n",
    "    print()\n",
    "    y_true, y_pred = y_test,  clf.predict(X_test_updated)\n",
    "    \n",
    "    #false_positive_rate, true_positive_rate, thresholds = roc_curve(y_true, y_pred)\n",
    "    #roc_auc = auc(false_positive_rate, true_positive_rate)\n",
    "    #print(\"AUC:\", roc_auc)\n",
    "    \n",
    "    #print(classification_report(y_true, y_pred))\n",
    "    #print()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}