kaggle.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scikit-learn pipelines and pandas\n",
    "Typically, when you want to use the standard pandas/sklearn framework to tackle a machine learning or data analysis problem, you will start analysing the dataset using pandas. Once you've gotten some insights, you may derive a transformed data set using pandas and derive a model using a scikit-learn estimator like Linear or Ridge Regression. Typically, you will estimate the performance of such a model using cross validation.\n",
    "\n",
    "There are several things I don't like with this approach and want to raise awareness of:\n",
    "1. Typically, notebooks contain a lot of repetitive code\n",
    "2. The pandas and scikit-learn frameworks are somehow separated and a combined pipeline is not used\n",
    "3. Most importantly: As it is explained [here]http://scikit-learn.org/stable/modules/cross_validation.html#computing-cross-validated-metrics) the preprocessing step should be tested on a hold out set. This implies, that cross-validation on a dataset requires to perform preprocessing on the respective subsets constructed during cross-validation.\n",
    "    \n",
    "To solve these issues, we only have to find the answer to the following question: Is there a way to perform scikit-learn and pipeline compatible preprocessing using pandas?\n",
    "\n",
    "Luckily, all we need to do is to adhere to the scikit-learn transformers api. A transformer typically contains a transform and a fit method. Using the scikits [TransformerMixin-Class](http://scikit-learn.org/stable/modules/generated/sklearn.base.TransformerMixin.html#sklearn.base.TransformerMixin) a fit_transform function is constructed.\n",
    "\n",
    "Let us briefly describe how transformers are used:\n",
    "\n",
    "If the transformer needs to remember the state of the training data, e.g. the mean of a column, the fit method is used on the training data to store this state. Subsequently, the transform function is used on the train and test data. However, if not state preservation is needed, e.g. in the case of log transforming data, the fit functions may essentially do nothing, and we just use the transform function. Note that the fit function is **never** used on the test data\n",
    "\n",
    "So the agenda of this notebooks is as follows:\n",
    "\n",
    "1. Load and read the data\n",
    "2. Define transformer objects (feel free to skip this lengthy paragraph at first)\n",
    "3. Use these transformers to preprocess the data\n",
    "4. Use preprocessing and Ridge regression, gridsearch and crossvalidation to estimate the generalization performance\n",
    "6. Under the assumption, that the gridsearch parameters are stable between cross validation folds, retrain a model using gridsearch on all of the training data\n",
    "7. Further comments on additional transformers, next steps and references"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df_train = pd.read_csv(\"train.csv\")\n",
    "df_test = pd.read_csv(\"test.csv\")\n",
    "\n",
    "y_train = df_train.set_index(\"Id\")[\"SalePrice\"]\n",
    "X_train = df_train.set_index(\"Id\").iloc[:,:-1]\n",
    "X_test = df_test.set_index(\"Id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer Objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.base import TransformerMixin, BaseEstimator, clone\n",
    "\n",
    "class SelectColumnsTransfomer(BaseEstimator, TransformerMixin):\n",
    "    \"\"\" A DataFrame transformer that provides column selection\n",
    "    \n",
    "    Allows to select columns by name from pandas dataframes in scikit-learn\n",
    "    pipelines.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    columns : list of str, names of the dataframe columns to select\n",
    "        Default: [] \n",
    "    \n",
    "    \"\"\"\n",
    "    def __init__(self, columns=[]):\n",
    "        self.columns = columns\n",
    "\n",
    "    def transform(self, X, **transform_params):\n",
    "        \"\"\" Selects columns of a DataFrame\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : pandas DataFrame\n",
    "            \n",
    "        Returns\n",
    "        ----------\n",
    "        \n",
    "        trans : pandas DataFrame\n",
    "            contains selected columns of X      \n",
    "        \"\"\"\n",
    "        trans = X[self.columns].copy() \n",
    "        return trans\n",
    "\n",
    "    def fit(self, X, y=None, **fit_params):\n",
    "        \"\"\" Do nothing function\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : pandas DataFrame\n",
    "        y : default None\n",
    "                \n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        self  \n",
    "        \"\"\"\n",
    "        return self\n",
    "    \n",
    "\n",
    "class DataFrameFunctionTransformer(BaseEstimator, TransformerMixin):\n",
    "    \"\"\" A DataFrame transformer providing imputation or function application\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    impute : Boolean, default False\n",
    "        \n",
    "    func : function that acts on an array of the form [n_elements, 1]\n",
    "        if impute is True, functions must return a float number, otherwise \n",
    "        an array of the form [n_elements, 1]\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, func, impute = False):\n",
    "        self.func = func\n",
    "        self.impute = impute\n",
    "        self.series = pd.Series() \n",
    "\n",
    "    def transform(self, X, **transformparams):\n",
    "        \"\"\" Transforms a DataFrame\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : DataFrame\n",
    "            \n",
    "        Returns\n",
    "        ----------\n",
    "        trans : pandas DataFrame\n",
    "            Transformation of X \n",
    "        \"\"\"\n",
    "        \n",
    "        if self.impute:\n",
    "            trans = pd.DataFrame(X).fillna(self.series).copy()\n",
    "        else:\n",
    "            trans = pd.DataFrame(X).apply(self.func).copy()\n",
    "        return trans\n",
    "\n",
    "    def fit(self, X, y=None, **fitparams):\n",
    "        \"\"\" Fixes the values to impute or does nothing\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : pandas DataFrame\n",
    "        y : not used, API requirement\n",
    "                \n",
    "        Returns\n",
    "        ----------\n",
    "        self  \n",
    "        \"\"\"\n",
    "        \n",
    "        if self.impute:\n",
    "            self.series = pd.DataFrame(X).apply(self.func).copy()\n",
    "        return self\n",
    "    \n",
    "    \n",
    "class DataFrameFeatureUnion(BaseEstimator, TransformerMixin):\n",
    "    \"\"\" A DataFrame transformer that unites several DataFrame transformers\n",
    "    \n",
    "    Fit several DataFrame transformers and provides a concatenated\n",
    "    Data Frame\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    list_of_transformers : list of DataFrameTransformers\n",
    "        \n",
    "    \"\"\" \n",
    "    def __init__(self, list_of_transformers):\n",
    "        self.list_of_transformers = list_of_transformers\n",
    "        \n",
    "    def transform(self, X, **transformparamn):\n",
    "        \"\"\" Applies the fitted transformers on a DataFrame\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : pandas DataFrame\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        concatted :  pandas DataFrame\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        concatted = pd.concat([transformer.transform(X)\n",
    "                            for transformer in\n",
    "                            self.fitted_transformers_], axis=1).copy()\n",
    "        return concatted\n",
    "\n",
    "\n",
    "    def fit(self, X, y=None, **fitparams):\n",
    "        \"\"\" Fits several DataFrame Transformers\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : pandas DataFrame\n",
    "        y : not used, API requirement\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        self : object\n",
    "        \"\"\"\n",
    "        \n",
    "        self.fitted_transformers_ = []\n",
    "        for transformer in self.list_of_transformers:\n",
    "            fitted_trans = clone(transformer).fit(X, y=None, **fitparams)\n",
    "            self.fitted_transformers_.append(fitted_trans)\n",
    "        return self\n",
    "    \n",
    "\n",
    "class ToDummiesTransformer(BaseEstimator, TransformerMixin):\n",
    "    \"\"\" A Dataframe transformer that provide dummy variable encoding\n",
    "    \"\"\"\n",
    "    \n",
    "    def transform(self, X, **transformparams):\n",
    "        \"\"\" Returns a dummy variable encoded version of a DataFrame\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : pandas DataFrame\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        trans : pandas DataFrame\n",
    "        \n",
    "        \"\"\"\n",
    "    \n",
    "        trans = pd.get_dummies(X).copy()\n",
    "        return trans\n",
    "\n",
    "    def fit(self, X, y=None, **fitparams):\n",
    "        \"\"\" Do nothing operation\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        self : object\n",
    "        \"\"\"\n",
    "        return self\n",
    "\n",
    "\n",
    "class DropAllZeroTrainColumnsTransformer(BaseEstimator, TransformerMixin):\n",
    "    \"\"\" A DataFrame transformer that provides dropping all-zero columns\n",
    "    \"\"\"\n",
    "\n",
    "    def transform(self, X, **transformparams):\n",
    "        \"\"\" Drops certain all-zero columns of X\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : DataFrame\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        trans : DataFrame\n",
    "        \"\"\"\n",
    "        \n",
    "        trans = X.drop(self.cols_, axis=1).copy()\n",
    "        return trans\n",
    "\n",
    "    def fit(self, X, y=None, **fitparams):\n",
    "        \"\"\" Determines the all-zero columns of X\n",
    "        \n",
    "        Parameters\n",
    "        ----------\n",
    "        X : DataFrame\n",
    "        y : not used\n",
    "        \n",
    "        Returns\n",
    "        ----------\n",
    "        self : object\n",
    "        \"\"\"\n",
    "        \n",
    "        self.cols_ = X.columns[(X==0).all()]\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline, make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Area Columns\n",
    "We start with the columns describing some form of area. As  [Alexandre Paipu](https://www.kaggle.com/apapiu/house-prices-advanced-regression-techniques/regularized-linear-models) points out, skewed columns should be log transformed. Checking some data, we find that the skewed columns are essentially the area columns. \n",
    "\n",
    "We simply use a regular expression to filter the area columns, then we transform to float, impute by the mean (even if no missing values appear) and perform a log (x+1) transform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "area_cols = X_train.columns[X_train.columns.str.contains('(?i)area|(?i)porch|(?i)sf')].tolist()\n",
    "\n",
    "area_cols_pipeline = make_pipeline(  \n",
    "        SelectColumnsTransfomer(area_cols),\n",
    "        DataFrameFunctionTransformer(func = lambda x: x.astype(np.float64)),\n",
    "        DataFrameFunctionTransformer(func = np.mean, impute=True),\n",
    "        DataFrameFunctionTransformer(func = np.log1p) \n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Object Columns\n",
    "The object columns are the categorical columns. Reading the data set description, we see that NaN values are allowed for each column. Consequently, we assume that possible levels of the categorical values are known beforehand and that the NaNs are correctly encoded. So though we access the values of the test data, we don't use information not known before the analyzing the data\n",
    "\n",
    "So what we will do here is to determine all possible levels across all categories, construct dummy variables for all variables and levels (this is not efficient!) and then drop the combinations that do not occur. Particularly, when fitting the pipeline to the training data sets and applying to the test sets, we will only keep levels contained in the training data sets!\n",
    "\n",
    "We simply filter the columns by data type, construct the object levels, impute, convert to dummy notation and drop all zero columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "object_columns = X_train.columns[X_train.dtypes == object].tolist()\n",
    "object_levels = np.union1d(X_train[object_columns].fillna('NAN'), X_test[object_columns].fillna('NAN'))\n",
    "\n",
    "categorical_cols_pipeline = make_pipeline(\n",
    "        SelectColumnsTransfomer(object_columns),\n",
    "        DataFrameFunctionTransformer(lambda x:'NAN', impute=True),\n",
    "        DataFrameFunctionTransformer(lambda x:x.astype('category', categories=object_levels)),\n",
    "        ToDummiesTransformer(),\n",
    "        DropAllZeroTrainColumnsTransformer()\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remaining columns\n",
    "The remaining columns are mostly integer columns. However, if an integers column in a training set has a missing value in the test set the data type will be float in the test set. So, without actually using information beforehand, we just convert the remaining columns to float, store the mean and impute if necessary on train and test sets. \n",
    "\n",
    "Typically, integer columns provide some form of count data, hence we do not use a log transform here. Note however, that the GarageBltYear is also a remaining column. For simplicity, we treat it like the other count-like columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "remaining_cols = [x for x in X_train.columns.tolist() if x not in object_columns and x not in area_cols]\n",
    "\n",
    "remaining_cols_pipeline = make_pipeline(\n",
    "        SelectColumnsTransfomer(remaining_cols),\n",
    "        DataFrameFunctionTransformer(func = lambda x: x.astype(np.float64)),\n",
    "        DataFrameFunctionTransformer(func = np.mean, impute=True)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uniting the pipelines\n",
    "We put the pipelines together using the DataFrameFeatureUnion transformer. To demonstrate that we get a DataFrame we simply use fit_transform on the training set and show the first rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotArea</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>BsmtFinSF2</th>\n",
       "      <th>BsmtUnfSF</th>\n",
       "      <th>TotalBsmtSF</th>\n",
       "      <th>1stFlrSF</th>\n",
       "      <th>2ndFlrSF</th>\n",
       "      <th>LowQualFinSF</th>\n",
       "      <th>GrLivArea</th>\n",
       "      <th>...</th>\n",
       "      <th>HalfBath</th>\n",
       "      <th>BedroomAbvGr</th>\n",
       "      <th>KitchenAbvGr</th>\n",
       "      <th>TotRmsAbvGrd</th>\n",
       "      <th>Fireplaces</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>GarageCars</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>9.042040</td>\n",
       "      <td>5.283204</td>\n",
       "      <td>6.561031</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.017280</td>\n",
       "      <td>6.753438</td>\n",
       "      <td>6.753438</td>\n",
       "      <td>6.751101</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.444833</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2008.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9.169623</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.886532</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.652489</td>\n",
       "      <td>7.141245</td>\n",
       "      <td>7.141245</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.141245</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2007.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.328212</td>\n",
       "      <td>5.093750</td>\n",
       "      <td>6.188264</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.075346</td>\n",
       "      <td>6.825460</td>\n",
       "      <td>6.825460</td>\n",
       "      <td>6.765039</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.488294</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2008.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9.164401</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.379897</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.293419</td>\n",
       "      <td>6.629363</td>\n",
       "      <td>6.869014</td>\n",
       "      <td>6.629363</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.448916</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2006.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>9.565284</td>\n",
       "      <td>5.860786</td>\n",
       "      <td>6.486161</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.196444</td>\n",
       "      <td>7.044033</td>\n",
       "      <td>7.044033</td>\n",
       "      <td>6.960348</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.695758</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2008.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 304 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     LotArea  MasVnrArea  BsmtFinSF1  BsmtFinSF2  BsmtUnfSF  TotalBsmtSF  \\\n",
       "Id                                                                         \n",
       "1   9.042040    5.283204    6.561031         0.0   5.017280     6.753438   \n",
       "2   9.169623    0.000000    6.886532         0.0   5.652489     7.141245   \n",
       "3   9.328212    5.093750    6.188264         0.0   6.075346     6.825460   \n",
       "4   9.164401    0.000000    5.379897         0.0   6.293419     6.629363   \n",
       "5   9.565284    5.860786    6.486161         0.0   6.196444     7.044033   \n",
       "\n",
       "    1stFlrSF  2ndFlrSF  LowQualFinSF  GrLivArea   ...    HalfBath  \\\n",
       "Id                                                ...               \n",
       "1   6.753438  6.751101           0.0   7.444833   ...         1.0   \n",
       "2   7.141245  0.000000           0.0   7.141245   ...         0.0   \n",
       "3   6.825460  6.765039           0.0   7.488294   ...         1.0   \n",
       "4   6.869014  6.629363           0.0   7.448916   ...         0.0   \n",
       "5   7.044033  6.960348           0.0   7.695758   ...         1.0   \n",
       "\n",
       "    BedroomAbvGr  KitchenAbvGr  TotRmsAbvGrd  Fireplaces  GarageYrBlt  \\\n",
       "Id                                                                      \n",
       "1            3.0           1.0           8.0         0.0       2003.0   \n",
       "2            3.0           1.0           6.0         1.0       1976.0   \n",
       "3            3.0           1.0           6.0         1.0       2001.0   \n",
       "4            3.0           1.0           7.0         1.0       1998.0   \n",
       "5            4.0           1.0           9.0         1.0       2000.0   \n",
       "\n",
       "    GarageCars  MiscVal  MoSold  YrSold  \n",
       "Id                                       \n",
       "1          2.0      0.0     2.0  2008.0  \n",
       "2          2.0      0.0     5.0  2007.0  \n",
       "3          2.0      0.0     9.0  2008.0  \n",
       "4          3.0      0.0     2.0  2006.0  \n",
       "5          3.0      0.0    12.0  2008.0  \n",
       "\n",
       "[5 rows x 304 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessing_features = DataFrameFeatureUnion([area_cols_pipeline, categorical_cols_pipeline, remaining_cols_pipeline])\n",
    "preprocessing_features.fit_transform(X_train).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gridsearch and Crossvalidation\n",
    "We use nested cross validation to estimate the generalization performance. See the 3rd example [here](http://scikit-learn.org/stable/modules/grid_search.html#grid-search)\n",
    "\n",
    "Unfortunately, nested cross validation is not able to return the best model parameters for each fold ([and probably never will be](https://github.com/scikit-learn/scikit-learn/issues/6827)). However, for simplicity, we just assume that the model parameters are stable across the cross validation folds on the training sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV, cross_val_score\n",
    "from sklearn.linear_model import Ridge\n",
    "pipe_ridge = make_pipeline(preprocessing_features, Ridge())\n",
    "param_grid = {'ridge__alpha' : [0.05, 0.1, 0.3, 1, 3, 5, 10, 15, 30, 50, 75]}\n",
    "pipe_ridge_gs = GridSearchCV(pipe_ridge, param_grid=param_grid, scoring = 'neg_mean_squared_error', cv=3)\n",
    "result = np.sqrt(-cross_val_score(pipe_ridge_gs, X_train, np.log(y_train), scoring = 'neg_mean_squared_error', cv = 5))\n",
    "np.mean(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the result is relatively close to that of [Alexandra Papiu](https://www.kaggle.com/apapiu/house-prices-advanced-regression-techniques/regularized-linear-models). But I was not yet able to figure out where the differences come from exactly. One differnece is that I seem to construct more columns.\n",
    "\n",
    "Let me comment on the scoring parameter. Why 'neg_mean_squared_error'? This is a scikit learn convention that ensures that grid search and cross validation always [maximize](http://scikit-learn.org/stable/modules/model_evaluation.html#scoring-parameter) a specific score.\n",
    "\n",
    "Additionally, you might want to ask, why we don't use a custom scoring function and than use y_train instead of np.log(y_train). \n",
    "The reason is simply, that the loss functions in the algorithms are typically some form of squared loss. Using the np.log transformation, the optimization during model training directly optimizes the correct loss function. If we use a custom scorer object, the internal optimization is performed using standard quadratic error functional and we simply evaluate the results using that custom scorer. Those results do not necessarily have to be the same and perform worse in general (you can even test it here).\n",
    "\n",
    "For completeness, let us just fit an optimized model on the full training data and provide a data set for submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ridge__alpha': 5}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_ridge_gs.fit(X_train, np.log(y_train))\n",
    "predicted = np.exp(pipe_ridge_gs.predict(X_test))\n",
    "X_test[\"SalePrice\"] = predicted\n",
    "X_test[\"SalePrice\"].reset_index().to_csv('pipe_ridge_gs.csv', index=False)\n",
    "pipe_ridge_gs.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further comments on additional transformers, next steps and references\n",
    "* The FeatureUnion is not yet able to allow parallel processing\n",
    "* One might want to construct interaction terms only on e.g. the garage columns. If the year is not available, it typically means that no garage is present. So storing the NaNs of such a column and building interaction terms with e.g. the area might provide viable information. To this end, one would need a transformer that selects columns by a somewhat dynamical name pattern after a first FeatureUnion.\n",
    "* I discovered the [sklearn-pandas](https://github.com/paulgb/sklearn-pandas) package just recently. But it seems, that no dataframes but numpy arrays are returned. Thus, the previous idea will be really hard to implement. Nevertheless, it may be easier to just build a transformer\n",
    "* Unit tests are missing. Transformers should e.g. only accept DataFrame objects\n",
    "* I got several ideas from [Zac Stewart's](http://zacstewart.com/2014/08/05/pipelines-of-featureunions-of-pipelines.html) blog entry. However, I didn't find source codes for the transformer, so I constructed them on my own ."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}