11-categorical-encoders-and-benchmark.ipynb

{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\n\nInspired by this article and the repo, I have created the following kernel:\n\n- [Benchmarking Categorical Encoders](https://towardsdatascience.com/benchmarking-categorical-encoders-9c322bd77ee8)\n\n- [CategoricalEncodingBenchmark](https://github.com/DenisVorotyntsev/CategoricalEncodingBenchmark)\n\nLet's see how these methods work in this dataset.\n\n[Discussion](https://www.kaggle.com/c/cat-in-the-dat/discussion/112584)\n\n- no feature preprocessing\n- Use KFold(5) for CV (+ more fold get better score)\n- LR (C=0.1, solver=lbfgs)\n\n|Encoder|LB Score|\n|-|-|\n|TE|0.78018|\n|WOE|0.78861|\n|LOOE|0.79382|\n|James-Stein|0.77843|\n|Catboost|0.79164|\n|One-Hot(another my kernel)|0.77973|\n\n\n\n### Category-Encoders\n\n1. Label Encoder\n2. One-Hot Encoder\n3. Sum Encoder\n4. Helmert Encoder\n5. Frequency Encoder\n6. Target Encoder\n7. M-Estimate Encoder\n8. Weight Of Evidence Encoder\n9. James-Stein Encoder\n10. Leave-one-out Encoder\n11. Catboost Encoder\n---\n- Validation (Benchmark)\n    - single LR\n    - LR with Cross Validation\n\n- Submit"},{"metadata":{},"cell_type":"markdown","source":"__Note__: With no arguments passed in aforementioned encoders (__category\\_encoders__) library, only __string__ or __object__ type columns will undergo encoding process and numeric/integer columns will be left as it is. If we want to encode specific columns or even numeric/integer columns, then we have to pass list of columns as __cols__ variable while instantiating the encoder."},{"metadata":{},"cell_type":"markdown","source":"## Category-Encoders \n\nA set of scikit-learn-style transformers for encoding categorical variables into numeric by means of different techniques."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# If you want to test this on your local notebook\n# http://contrib.scikit-learn.org/categorical-encoding/\n# !pip install category-encoders","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import pandas as pd\n\nfrom category_encoders.ordinal import OrdinalEncoder\nfrom category_encoders.woe import WOEEncoder\nfrom category_encoders.target_encoder import TargetEncoder\nfrom category_encoders.sum_coding import SumEncoder\nfrom category_encoders.m_estimate import MEstimateEncoder\nfrom category_encoders.leave_one_out import LeaveOneOutEncoder\nfrom category_encoders.helmert import HelmertEncoder\nfrom category_encoders.cat_boost import CatBoostEncoder\nfrom category_encoders.james_stein import JamesSteinEncoder\nfrom category_encoders.one_hot import OneHotEncoder\n\nTEST = True","execution_count":35,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"read csv and doing some preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ntrain = pd.read_csv('/kaggle/input/cat-in-the-dat/train.csv')\ntest = pd.read_csv('/kaggle/input/cat-in-the-dat/test.csv')\ntarget = train['target']\ntrain_id = train['id']\ntest_id = test['id']\ntrain.drop(['target', 'id'], axis=1, inplace=True)\ntest.drop('id', axis=1, inplace=True)\nprint(train.dtypes)\ntrain.head()","execution_count":29,"outputs":[{"output_type":"stream","text":"bin_0     int64\nbin_1     int64\nbin_2     int64\nbin_3    object\nbin_4    object\nnom_0    object\nnom_1    object\nnom_2    object\nnom_3    object\nnom_4    object\nnom_5    object\nnom_6    object\nnom_7    object\nnom_8    object\nnom_9    object\nord_0     int64\nord_1    object\nord_2    object\nord_3    object\nord_4    object\nord_5    object\nday       int64\nmonth     int64\ndtype: object\nCPU times: user 2.34 s, sys: 69.8 ms, total: 2.41 s\nWall time: 2.39 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":29,"data":{"text/plain":"   bin_0  bin_1  bin_2 bin_3 bin_4  nom_0      nom_1    nom_2    nom_3  \\\n0      0      0      0     T     Y  Green   Triangle    Snake  Finland   \n1      0      1      0     T     Y  Green  Trapezoid  Hamster   Russia   \n2      0      0      0     F     Y   Blue  Trapezoid     Lion   Russia   \n3      0      1      0     F     Y    Red  Trapezoid    Snake   Canada   \n4      0      0      0     F     N    Red  Trapezoid     Lion   Canada   \n\n      nom_4  ...      nom_8      nom_9 ord_0        ord_1        ord_2  ord_3  \\\n0   Bassoon  ...  c389000ab  2f4cb3d51     2  Grandmaster         Cold      h   \n1     Piano  ...  4cd920251  f83c56c21     1  Grandmaster          Hot      a   \n2  Theremin  ...  de9c9f684  ae6800dd0     1       Expert     Lava Hot      h   \n3      Oboe  ...  4ade6ab69  8270f0d71     1  Grandmaster  Boiling Hot      i   \n4      Oboe  ...  cb43ab175  b164b72a7     1  Grandmaster     Freezing      a   \n\n  ord_4 ord_5 day month  \n0     D    kr   2     2  \n1     A    bF   7     8  \n2     R    Jc   7     2  \n3     D    kW   2     1  \n4     R    qP   7     8  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>T</td>\n      <td>Y</td>\n      <td>Green</td>\n      <td>Triangle</td>\n      <td>Snake</td>\n      <td>Finland</td>\n      <td>Bassoon</td>\n      <td>...</td>\n      <td>c389000ab</td>\n      <td>2f4cb3d51</td>\n      <td>2</td>\n      <td>Grandmaster</td>\n      <td>Cold</td>\n      <td>h</td>\n      <td>D</td>\n      <td>kr</td>\n      <td>2</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>T</td>\n      <td>Y</td>\n      <td>Green</td>\n      <td>Trapezoid</td>\n      <td>Hamster</td>\n      <td>Russia</td>\n      <td>Piano</td>\n      <td>...</td>\n      <td>4cd920251</td>\n      <td>f83c56c21</td>\n      <td>1</td>\n      <td>Grandmaster</td>\n      <td>Hot</td>\n      <td>a</td>\n      <td>A</td>\n      <td>bF</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>F</td>\n      <td>Y</td>\n      <td>Blue</td>\n      <td>Trapezoid</td>\n      <td>Lion</td>\n      <td>Russia</td>\n      <td>Theremin</td>\n      <td>...</td>\n      <td>de9c9f684</td>\n      <td>ae6800dd0</td>\n      <td>1</td>\n      <td>Expert</td>\n      <td>Lava Hot</td>\n      <td>h</td>\n      <td>R</td>\n      <td>Jc</td>\n      <td>7</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>F</td>\n      <td>Y</td>\n      <td>Red</td>\n      <td>Trapezoid</td>\n      <td>Snake</td>\n      <td>Canada</td>\n      <td>Oboe</td>\n      <td>...</td>\n      <td>4ade6ab69</td>\n      <td>8270f0d71</td>\n      <td>1</td>\n      <td>Grandmaster</td>\n      <td>Boiling Hot</td>\n      <td>i</td>\n      <td>D</td>\n      <td>kW</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>F</td>\n      <td>N</td>\n      <td>Red</td>\n      <td>Trapezoid</td>\n      <td>Lion</td>\n      <td>Canada</td>\n      <td>Oboe</td>\n      <td>...</td>\n      <td>cb43ab175</td>\n      <td>b164b72a7</td>\n      <td>1</td>\n      <td>Grandmaster</td>\n      <td>Freezing</td>\n      <td>a</td>\n      <td>R</td>\n      <td>qP</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"feature_list = list(train.columns) # you can custumize later.\nfeature_list","execution_count":4,"outputs":[{"output_type":"execute_result","execution_count":4,"data":{"text/plain":"['bin_0',\n 'bin_1',\n 'bin_2',\n 'bin_3',\n 'bin_4',\n 'nom_0',\n 'nom_1',\n 'nom_2',\n 'nom_3',\n 'nom_4',\n 'nom_5',\n 'nom_6',\n 'nom_7',\n 'nom_8',\n 'nom_9',\n 'ord_0',\n 'ord_1',\n 'ord_2',\n 'ord_3',\n 'ord_4',\n 'ord_5',\n 'day',\n 'month']"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"### notation\n\n- $y$ and $y+$ — the total number of observations and the total number of positive observations (y=1);\n- $x_i$, $y_i$ — the i-th value of category and target;\n- $n$ and $n+$ — the number of observations and the number of positive observations (y=1) for a given value of a categorical column;\n- $a$ — a regularization hyperparameter (selected by a user), prior — an average value of the target."},{"metadata":{},"cell_type":"markdown","source":"## 1. Label Encoder (LE), Ordinary Encoder(OE)\n\nOne of the most common encoding methods.\n\nAn encoding method that converts categorical data into numbers.\nThe code is very simple, and when you encode a specific column you can proceed as follows:\n\n``` python\nfrom sklearn.preprocessing import LabelEncoder\nlabel = LabelEncoder()\n\ntrain[column_name] = label.fit_transform(train[column_name])\n```\n\nThe simple idea is to convert the same category to a number with the same value.\n\nSo the range of numbers maps from 0 to n-1 as labels.\n\nThe disadvantage is that the labels are ordered randomly (in the existing order of the data), which can add noise while assigning an unexpected order between labels. In other words, the data becomes ordinary (ordinal, ordered) data, which can lead to unintended consequences.\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nLE_encoder = OrdinalEncoder(feature_list)\ntrain_le = LE_encoder.fit_transform(train)\ntest_le = LE_encoder.transform(test)\ntrain_le.head()","execution_count":5,"outputs":[{"output_type":"stream","text":"CPU times: user 6.33 s, sys: 533 ms, total: 6.86 s\nWall time: 5.63 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":5,"data":{"text/plain":"   bin_0  bin_1  bin_2  bin_3  bin_4  nom_0  nom_1  nom_2  nom_3  nom_4  ...  \\\n0      0      0      0      1      1      1      1      1      1      1  ...   \n1      0      1      0      1      1      1      2      2      2      2  ...   \n2      0      0      0      2      1      2      2      3      2      3  ...   \n3      0      1      0      2      1      3      2      1      3      4  ...   \n4      0      0      0      2      2      3      2      3      3      4  ...   \n\n   nom_8  nom_9  ord_0  ord_1  ord_2  ord_3  ord_4  ord_5  day  month  \n0      1      1      2      1      1      1      1      1    2      2  \n1      2      2      1      1      2      2      2      2    7      8  \n2      3      3      1      2      3      1      3      3    7      2  \n3      4      4      1      1      4      3      1      4    2      1  \n4      5      5      1      1      5      2      3      5    7      8  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>...</td>\n      <td>1</td>\n      <td>1</td>\n      <td>2</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>2</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>1</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>...</td>\n      <td>2</td>\n      <td>2</td>\n      <td>1</td>\n      <td>1</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>2</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n      <td>2</td>\n      <td>2</td>\n      <td>3</td>\n      <td>2</td>\n      <td>3</td>\n      <td>...</td>\n      <td>3</td>\n      <td>3</td>\n      <td>1</td>\n      <td>2</td>\n      <td>3</td>\n      <td>1</td>\n      <td>3</td>\n      <td>3</td>\n      <td>7</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>2</td>\n      <td>1</td>\n      <td>3</td>\n      <td>2</td>\n      <td>1</td>\n      <td>3</td>\n      <td>4</td>\n      <td>...</td>\n      <td>4</td>\n      <td>4</td>\n      <td>1</td>\n      <td>1</td>\n      <td>4</td>\n      <td>3</td>\n      <td>1</td>\n      <td>4</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>2</td>\n      <td>2</td>\n      <td>3</td>\n      <td>2</td>\n      <td>3</td>\n      <td>3</td>\n      <td>4</td>\n      <td>...</td>\n      <td>5</td>\n      <td>5</td>\n      <td>1</td>\n      <td>1</td>\n      <td>5</td>\n      <td>2</td>\n      <td>3</td>\n      <td>5</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 2. One-Hot Encoder (OHE, dummy encoder)\n\n\nSo what can you do to give values ​​by category instead of ordering them?\n\nIf you have data with specific category values, you can create a column. If the base Label Encoder label type is N, then OHE is the way to create N columns.\n\nSince only the row containing the content is given as 1, it is called one-hot encoding. Also called dummy encoding in the sense of creating a dummy.\n\n\nIn this competition:\n\n``` python\ntraintest = pd.concat([train, test])\ndummies = pd.get_dummies(traintest, columns=traintest.columns, drop_first=True, sparse=True)\ntrain_ohe = dummies.iloc[:train.shape[0], :]\ntest_ohe = dummies.iloc[train.shape[0]:, :]\ntrain_ohe = train_ohe.sparse.to_coo().tocsr()\ntest_ohe = test_ohe.sparse.to_coo().tocsr()\n```\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n# this method didn't work because of RAM memory. One-hot encoder blows up the dimensionality which could not be stored in memory\n# OHE_encoder = OneHotEncoder(feature_list)\n# train_ohe = OHE_encoder.fit_transform(train)\n# test_ohe = OHE_encoder.transform(test)","execution_count":6,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. Sum Encoder (Deviation Encoder, Effect Encoder)\n\n**Sum Encoder** compares the mean of the dependent variable (target) for a given level of a categorical column to the overall mean of the target. \n\nSum Encoding is very similar to OHE and both of them are commonly used in Linear Regression (LR) types of models.\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n# this method didn't work because of RAM memory. this encoder too blows up the dimensionality which could not be stored in memory\n# SE_encoder =SumEncoder(feature_list)\n# train_se = SE_encoder.fit_transform(train[feature_list], target)\n# test_se = SE_encoder.transform(test[feature_list])","execution_count":7,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Helmert Encoder\n\n**Helmert Encoding** is a third commonly used type of categorical encoding for regression along with OHE and Sum Encoding. \n\nIt compares each level of a categorical variable to the mean of the subsequent levels. \n\nThis type of encoding can be useful in certain situations where levels of the categorical variable are ordered. (not this dataset)\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"# %%time\n# this method didn't work because of RAM memory. this encoder too blows up the dimensionality which could not be stored in memory.\n# HE_encoder = HelmertEncoder(feature_list)\n# train_he = HE_encoder.fit_transform(train[feature_list], target)\n# test_he = HE_encoder.transform(test[feature_list])","execution_count":8,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Frequency Encoder\n\nThis method encodes by frequency.\n\nCreate a new feature with the number of categories from the training data.\n\nI will not proceed separately in this data."},{"metadata":{},"cell_type":"markdown","source":"## 6. Target Encoder\n\nThis is a work in progress for many kernels.\n\nThe encoded category values are calculated according to the following formulas:\n\n$$s = \\frac{1}{1+exp(-\\frac{n-mdl}{a})}$$\n\n$$\\hat{x}^k = prior * (1-s) + s * \\frac{n^{+}}{n}$$\n\n- mdl means **'min data in leaf'**\n- a means **'smooth parameter, power of regularization'**\n\nTarget Encoder is a powerful, but it has a huuuuuge disadvantage \n\n> **target leakage**: it uses information about the target. \n\nTo reduce the effect of target leakage, \n\n- Increase regularization\n- Add random noise to the representation of the category in train dataset (some sort of augmentation)\n- Use Double Validation (using other validation)\n\nLet's use while being careful about overfitting.\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# TE_encoder = TargetEncoder(cols= feature_list) #this will encode all the columns in feature_list including numeric/int64 columns.\nTE_encoder = TargetEncoder() #this will encode only string or object type columns\ntrain_te = TE_encoder.fit_transform(train[feature_list], target)\ntest_te = TE_encoder.transform(test[feature_list])\n\ntrain_te.head()","execution_count":32,"outputs":[{"output_type":"stream","text":"CPU times: user 12.9 s, sys: 1.24 s, total: 14.1 s\nWall time: 10.8 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":32,"data":{"text/plain":"   bin_0  bin_1  bin_2     bin_3     bin_4     nom_0     nom_1     nom_2  \\\n0      0      0      0  0.302537  0.290107  0.327145  0.360978  0.307162   \n1      0      1      0  0.302537  0.290107  0.327145  0.290054  0.359209   \n2      0      0      0  0.309384  0.290107  0.241790  0.290054  0.293085   \n3      0      1      0  0.309384  0.290107  0.351052  0.290054  0.307162   \n4      0      0      0  0.309384  0.333773  0.351052  0.290054  0.293085   \n\n      nom_3     nom_4  ...     nom_8     nom_9  ord_0     ord_1     ord_2  \\\n0  0.242813  0.237743  ...  0.372694  0.368421      2  0.403885  0.257877   \n1  0.289954  0.304164  ...  0.189189  0.076924      1  0.403885  0.326315   \n2  0.289954  0.353951  ...  0.223022  0.172414      1  0.317175  0.403126   \n3  0.339793  0.329472  ...  0.325123  0.227273      1  0.403885  0.360961   \n4  0.339793  0.329472  ...  0.376812  0.200000      1  0.403885  0.225214   \n\n      ord_3     ord_4     ord_5  day  month  \n0  0.306993  0.208354  0.401186    2      2  \n1  0.206599  0.186877  0.303880    7      8  \n2  0.306993  0.351864  0.206843    7      2  \n3  0.330148  0.208354  0.355985    2      1  \n4  0.206599  0.351864  0.404345    7      8  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.327145</td>\n      <td>0.360978</td>\n      <td>0.307162</td>\n      <td>0.242813</td>\n      <td>0.237743</td>\n      <td>...</td>\n      <td>0.372694</td>\n      <td>0.368421</td>\n      <td>2</td>\n      <td>0.403885</td>\n      <td>0.257877</td>\n      <td>0.306993</td>\n      <td>0.208354</td>\n      <td>0.401186</td>\n      <td>2</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.327145</td>\n      <td>0.290054</td>\n      <td>0.359209</td>\n      <td>0.289954</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.189189</td>\n      <td>0.076924</td>\n      <td>1</td>\n      <td>0.403885</td>\n      <td>0.326315</td>\n      <td>0.206599</td>\n      <td>0.186877</td>\n      <td>0.303880</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.309384</td>\n      <td>0.290107</td>\n      <td>0.241790</td>\n      <td>0.290054</td>\n      <td>0.293085</td>\n      <td>0.289954</td>\n      <td>0.353951</td>\n      <td>...</td>\n      <td>0.223022</td>\n      <td>0.172414</td>\n      <td>1</td>\n      <td>0.317175</td>\n      <td>0.403126</td>\n      <td>0.306993</td>\n      <td>0.351864</td>\n      <td>0.206843</td>\n      <td>7</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0.309384</td>\n      <td>0.290107</td>\n      <td>0.351052</td>\n      <td>0.290054</td>\n      <td>0.307162</td>\n      <td>0.339793</td>\n      <td>0.329472</td>\n      <td>...</td>\n      <td>0.325123</td>\n      <td>0.227273</td>\n      <td>1</td>\n      <td>0.403885</td>\n      <td>0.360961</td>\n      <td>0.330148</td>\n      <td>0.208354</td>\n      <td>0.355985</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.309384</td>\n      <td>0.333773</td>\n      <td>0.351052</td>\n      <td>0.290054</td>\n      <td>0.293085</td>\n      <td>0.339793</td>\n      <td>0.329472</td>\n      <td>...</td>\n      <td>0.376812</td>\n      <td>0.200000</td>\n      <td>1</td>\n      <td>0.403885</td>\n      <td>0.225214</td>\n      <td>0.206599</td>\n      <td>0.351864</td>\n      <td>0.404345</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(target.shape[0], target.sum(), target.shape[0]- target.sum())\n# train[feature_list].group_by(\"bin_0\")\ntrain_te.bin_4.value_counts()","execution_count":20,"outputs":[{"output_type":"stream","text":"300000 91764 208236\n","name":"stdout"},{"output_type":"execute_result","execution_count":20,"data":{"text/plain":"0.290107    191633\n0.333773    108367\nName: bin_4, dtype: int64"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 7. M-Estimate Encoder\n\n**M-Estimate Encoder** is a **simplified version of Target Encoder**. It has only one hyperparameter\n\n$$\\hat{x}^k = \\frac{n^+ + prior * m}{y^+ + m}$$\n\nThe higher value of m results into stronger shrinking. Recommended values for m is in the range of 1 to 100.\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nMEE_encoder = MEstimateEncoder()\ntrain_mee = MEE_encoder.fit_transform(train[feature_list], target)\ntest_mee = MEE_encoder.transform(test[feature_list])\ntest_mee.head()","execution_count":22,"outputs":[{"output_type":"stream","text":"CPU times: user 12.2 s, sys: 1.02 s, total: 13.2 s\nWall time: 10.4 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":22,"data":{"text/plain":"   bin_0  bin_1  bin_2     bin_3     bin_4     nom_0     nom_1     nom_2  \\\n0      0      0      1  0.302537  0.290107  0.241791  0.360976  0.319017   \n1      0      0      0  0.302537  0.333773  0.351051  0.338931  0.293085   \n2      1      0      1  0.309384  0.290107  0.241791  0.338931  0.245141   \n3      0      0      1  0.302537  0.290107  0.351051  0.310626  0.335367   \n4      0      1      1  0.309384  0.333773  0.351051  0.290055  0.245141   \n\n      nom_3     nom_4  ...     nom_8     nom_9  ord_0     ord_1     ord_2  \\\n0  0.242815  0.304164  ...  0.348078  0.187212      2  0.242055  0.288797   \n1  0.339792  0.304164  ...  0.223069  0.289258      1  0.355076  0.403125   \n2  0.311723  0.304164  ...  0.187194  0.108823      2  0.317175  0.225215   \n3  0.311723  0.304164  ...  0.359772  0.319457      1  0.278534  0.403125   \n4  0.311723  0.304164  ...  0.374739  0.294771      3  0.403884  0.403125   \n\n      ord_3     ord_4     ord_5  day  month  \n0  0.342474  0.324946  0.300590    5     11  \n1  0.379275  0.186884  0.244833    7      5  \n2  0.206602  0.236894  0.417684    1     12  \n3  0.220467  0.336262  0.365124    2      3  \n4  0.379275  0.409470  0.389782    4     11  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.241791</td>\n      <td>0.360976</td>\n      <td>0.319017</td>\n      <td>0.242815</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.348078</td>\n      <td>0.187212</td>\n      <td>2</td>\n      <td>0.242055</td>\n      <td>0.288797</td>\n      <td>0.342474</td>\n      <td>0.324946</td>\n      <td>0.300590</td>\n      <td>5</td>\n      <td>11</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.302537</td>\n      <td>0.333773</td>\n      <td>0.351051</td>\n      <td>0.338931</td>\n      <td>0.293085</td>\n      <td>0.339792</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.223069</td>\n      <td>0.289258</td>\n      <td>1</td>\n      <td>0.355076</td>\n      <td>0.403125</td>\n      <td>0.379275</td>\n      <td>0.186884</td>\n      <td>0.244833</td>\n      <td>7</td>\n      <td>5</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.290107</td>\n      <td>0.241791</td>\n      <td>0.338931</td>\n      <td>0.245141</td>\n      <td>0.311723</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.187194</td>\n      <td>0.108823</td>\n      <td>2</td>\n      <td>0.317175</td>\n      <td>0.225215</td>\n      <td>0.206602</td>\n      <td>0.236894</td>\n      <td>0.417684</td>\n      <td>1</td>\n      <td>12</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.351051</td>\n      <td>0.310626</td>\n      <td>0.335367</td>\n      <td>0.311723</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.359772</td>\n      <td>0.319457</td>\n      <td>1</td>\n      <td>0.278534</td>\n      <td>0.403125</td>\n      <td>0.220467</td>\n      <td>0.336262</td>\n      <td>0.365124</td>\n      <td>2</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.333773</td>\n      <td>0.351051</td>\n      <td>0.290055</td>\n      <td>0.245141</td>\n      <td>0.311723</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.374739</td>\n      <td>0.294771</td>\n      <td>3</td>\n      <td>0.403884</td>\n      <td>0.403125</td>\n      <td>0.379275</td>\n      <td>0.409470</td>\n      <td>0.389782</td>\n      <td>4</td>\n      <td>11</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 8. Weight of Evidence Encoder \n\n**Weight Of Evidence** is a commonly used target-based encoder in credit scoring. \n\nIt is a measure of the “strength” of a grouping for separating good and bad risk (default). \n\nIt is calculated from the basic odds ratio:\n\n``` python\na = Distribution of Good Credit Outcomes\nb = Distribution of Bad Credit Outcomes\nWoE = ln(a / b)\n```\n\nHowever, if we use formulas as is, it might lead to **target leakage**(and overfit).\n\nTo avoid that, regularization parameter a is induced and WoE is calculated in the following way:\n\n$$nomiinator = \\frac{n^+ + a}{y^+ + 2*a}$$\n\n$$denominator = ln(\\frac{nominator}{denominator})$$\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nWOE_encoder = WOEEncoder()\ntrain_woe = WOE_encoder.fit_transform(train[feature_list], target)\ntest_woe = WOE_encoder.transform(test[feature_list])\ntrain_woe.head()","execution_count":24,"outputs":[{"output_type":"stream","text":"CPU times: user 12.4 s, sys: 486 ms, total: 12.8 s\nWall time: 9.95 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":24,"data":{"text/plain":"   bin_0  bin_1  bin_2     bin_3     bin_4     nom_0     nom_1     nom_2  \\\n0      0      0      0 -0.015794 -0.075416  0.098327  0.248359  0.006058   \n1      0      1      0 -0.015794 -0.075416  0.098327 -0.075660  0.240683   \n2      0      0      0  0.016454 -0.075416 -0.323420 -0.075660 -0.060990   \n3      0      1      0  0.016454 -0.075416  0.205037 -0.075660  0.006058   \n4      0      0      0  0.016454  0.128285  0.205037 -0.075660 -0.060990   \n\n      nom_3     nom_4  ...     nom_8     nom_9  ord_0     ord_1     ord_2  \\\n0 -0.317803 -0.345614  ...  0.302749  0.333932      2  0.430146 -0.237516   \n1 -0.076148 -0.008087  ... -0.600377 -1.052362      1  0.430146  0.094611   \n2 -0.076148  0.217747  ... -0.417323 -0.607677      1  0.052724  0.426997   \n3  0.155252  0.108884  ...  0.096879 -0.338013      1  0.430146  0.248265   \n4  0.155252  0.108884  ...  0.321353 -0.468414      1  0.430146 -0.416062   \n\n      ord_3     ord_4     ord_5  day  month  \n0  0.005297 -0.514545  0.420532    2      2  \n1 -0.526005 -0.650766 -0.008737    7      8  \n2  0.005297  0.208660 -0.523234    7      2  \n3  0.111980 -0.514545  0.227089    2      1  \n4 -0.526005  0.208660  0.432324    7      8  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>-0.015794</td>\n      <td>-0.075416</td>\n      <td>0.098327</td>\n      <td>0.248359</td>\n      <td>0.006058</td>\n      <td>-0.317803</td>\n      <td>-0.345614</td>\n      <td>...</td>\n      <td>0.302749</td>\n      <td>0.333932</td>\n      <td>2</td>\n      <td>0.430146</td>\n      <td>-0.237516</td>\n      <td>0.005297</td>\n      <td>-0.514545</td>\n      <td>0.420532</td>\n      <td>2</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>-0.015794</td>\n      <td>-0.075416</td>\n      <td>0.098327</td>\n      <td>-0.075660</td>\n      <td>0.240683</td>\n      <td>-0.076148</td>\n      <td>-0.008087</td>\n      <td>...</td>\n      <td>-0.600377</td>\n      <td>-1.052362</td>\n      <td>1</td>\n      <td>0.430146</td>\n      <td>0.094611</td>\n      <td>-0.526005</td>\n      <td>-0.650766</td>\n      <td>-0.008737</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.016454</td>\n      <td>-0.075416</td>\n      <td>-0.323420</td>\n      <td>-0.075660</td>\n      <td>-0.060990</td>\n      <td>-0.076148</td>\n      <td>0.217747</td>\n      <td>...</td>\n      <td>-0.417323</td>\n      <td>-0.607677</td>\n      <td>1</td>\n      <td>0.052724</td>\n      <td>0.426997</td>\n      <td>0.005297</td>\n      <td>0.208660</td>\n      <td>-0.523234</td>\n      <td>7</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0</td>\n      <td>0.016454</td>\n      <td>-0.075416</td>\n      <td>0.205037</td>\n      <td>-0.075660</td>\n      <td>0.006058</td>\n      <td>0.155252</td>\n      <td>0.108884</td>\n      <td>...</td>\n      <td>0.096879</td>\n      <td>-0.338013</td>\n      <td>1</td>\n      <td>0.430146</td>\n      <td>0.248265</td>\n      <td>0.111980</td>\n      <td>-0.514545</td>\n      <td>0.227089</td>\n      <td>2</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.016454</td>\n      <td>0.128285</td>\n      <td>0.205037</td>\n      <td>-0.075660</td>\n      <td>-0.060990</td>\n      <td>0.155252</td>\n      <td>0.108884</td>\n      <td>...</td>\n      <td>0.321353</td>\n      <td>-0.468414</td>\n      <td>1</td>\n      <td>0.430146</td>\n      <td>-0.416062</td>\n      <td>-0.526005</td>\n      <td>0.208660</td>\n      <td>0.432324</td>\n      <td>7</td>\n      <td>8</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 9. James-Stein Encoder\n\n**James-Stein Encoder** is a target-based encoder.\n\nThe idea behind James-Stein Encoder is simple. Estimation of the mean target for category k could be calculated according to the following formula:\n\n$$\\hat{x}^k = (1-B) * \\frac{n^+}{n} + B * \\frac{y^+}{y} $$\n\nOne way to select B is to tune it like a hyperparameter via cross-validation, but Charles Stein came up with another solution to the problem:\n\n$$B = \\frac{Var[y^k]}{Var[y^k] + Var[y]}$$\n\nSeems quite fair, but James-Stein Estimator has a big disadvantage — it is defined only for normal distribution (which is not the case for any classification task). \n\nTo avoid that, we can either convert binary targets with a log-odds ratio as it was done in WoE Encoder (which is used by default because it is simple) or use beta distribution.\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nJSE_encoder = JamesSteinEncoder()\ntrain_jse = JSE_encoder.fit_transform(train[feature_list], target)\ntest_jse = JSE_encoder.transform(test[feature_list])\ntest_jse.head()","execution_count":25,"outputs":[{"output_type":"stream","text":"CPU times: user 12 s, sys: 417 ms, total: 12.5 s\nWall time: 9.88 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":25,"data":{"text/plain":"   bin_0  bin_1  bin_2     bin_3     bin_4     nom_0     nom_1     nom_2  \\\n0      0      0      1  0.302537  0.290107  0.241790  0.343763  0.315031   \n1      0      0      0  0.302537  0.333773  0.351052  0.328749  0.296876   \n2      1      0      1  0.309384  0.290107  0.241790  0.328749  0.262111   \n3      0      0      1  0.302537  0.290107  0.351052  0.309196  0.326306   \n4      0      1      1  0.309384  0.333773  0.351052  0.294730  0.262111   \n\n      nom_3     nom_4  ...     nom_8     nom_9  ord_0     ord_1     ord_2  \\\n0  0.260374  0.304449  ...  0.326349  0.234325      2  0.256848  0.293836   \n1  0.329339  0.304449  ...  0.260119  0.297338      1  0.342313  0.372130   \n2  0.309961  0.304449  ...  0.236455  0.155348      2  0.314323  0.247048   \n3  0.309961  0.304449  ...  0.331938  0.312710      1  0.285182  0.372130   \n4  0.309961  0.304449  ...  0.338701  0.300110      3  0.377845  0.372130   \n\n      ord_3     ord_4     ord_5  day  month  \n0  0.326330  0.316033  0.303194    5     11  \n1  0.346197  0.232549  0.272939    7      5  \n2  0.243675  0.266080  0.358623    1     12  \n3  0.253214  0.321938  0.334532    2      3  \n4  0.346197  0.358728  0.345933    4     11  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.241790</td>\n      <td>0.343763</td>\n      <td>0.315031</td>\n      <td>0.260374</td>\n      <td>0.304449</td>\n      <td>...</td>\n      <td>0.326349</td>\n      <td>0.234325</td>\n      <td>2</td>\n      <td>0.256848</td>\n      <td>0.293836</td>\n      <td>0.326330</td>\n      <td>0.316033</td>\n      <td>0.303194</td>\n      <td>5</td>\n      <td>11</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.302537</td>\n      <td>0.333773</td>\n      <td>0.351052</td>\n      <td>0.328749</td>\n      <td>0.296876</td>\n      <td>0.329339</td>\n      <td>0.304449</td>\n      <td>...</td>\n      <td>0.260119</td>\n      <td>0.297338</td>\n      <td>1</td>\n      <td>0.342313</td>\n      <td>0.372130</td>\n      <td>0.346197</td>\n      <td>0.232549</td>\n      <td>0.272939</td>\n      <td>7</td>\n      <td>5</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.290107</td>\n      <td>0.241790</td>\n      <td>0.328749</td>\n      <td>0.262111</td>\n      <td>0.309961</td>\n      <td>0.304449</td>\n      <td>...</td>\n      <td>0.236455</td>\n      <td>0.155348</td>\n      <td>2</td>\n      <td>0.314323</td>\n      <td>0.247048</td>\n      <td>0.243675</td>\n      <td>0.266080</td>\n      <td>0.358623</td>\n      <td>1</td>\n      <td>12</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.351052</td>\n      <td>0.309196</td>\n      <td>0.326306</td>\n      <td>0.309961</td>\n      <td>0.304449</td>\n      <td>...</td>\n      <td>0.331938</td>\n      <td>0.312710</td>\n      <td>1</td>\n      <td>0.285182</td>\n      <td>0.372130</td>\n      <td>0.253214</td>\n      <td>0.321938</td>\n      <td>0.334532</td>\n      <td>2</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.333773</td>\n      <td>0.351052</td>\n      <td>0.294730</td>\n      <td>0.262111</td>\n      <td>0.309961</td>\n      <td>0.304449</td>\n      <td>...</td>\n      <td>0.338701</td>\n      <td>0.300110</td>\n      <td>3</td>\n      <td>0.377845</td>\n      <td>0.372130</td>\n      <td>0.346197</td>\n      <td>0.358728</td>\n      <td>0.345933</td>\n      <td>4</td>\n      <td>11</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 10. Leave-one-out Encoder (LOO or LOOE)\n\n**Leave-one-out Encoding** is another example of target-based encoders.\n\nThis encoder calculate mean target of category k for observation j if observation j is removed from the dataset:\n\n$$\\hat{x}^k_i = \\frac{\\sum_{j \\neq i}(y_j * (x_j == k) ) - y_i }{\\sum_{j \\neq i} x_j == k}$$\n\nWhile encoding the test dataset, a category is replaced with the mean target of the category k in the train dataset:\n\n$$\\hat{x}^k = \\frac{\\sum y_j * (x_j == k)  }{\\sum x_j == k}$$\n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nLOOE_encoder = LeaveOneOutEncoder()\ntrain_looe = LOOE_encoder.fit_transform(train[feature_list], target)\ntest_looe = LOOE_encoder.transform(test[feature_list])\ntest_looe.head()","execution_count":33,"outputs":[{"output_type":"stream","text":"CPU times: user 13.5 s, sys: 810 ms, total: 14.3 s\nWall time: 11.8 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":33,"data":{"text/plain":"   bin_0  bin_1  bin_2     bin_3     bin_4     nom_0     nom_1     nom_2  \\\n0      0      0      1  0.302537  0.290107  0.241790  0.360978  0.319017   \n1      0      0      0  0.302537  0.333773  0.351052  0.338932  0.293085   \n2      1      0      1  0.309384  0.290107  0.241790  0.338932  0.245139   \n3      0      0      1  0.302537  0.290107  0.351052  0.310627  0.335367   \n4      0      1      1  0.309384  0.333773  0.351052  0.290054  0.245139   \n\n      nom_3     nom_4  ...     nom_8     nom_9  ord_0     ord_1     ord_2  \\\n0  0.242813  0.304164  ...  0.348315  0.181818      2  0.242055  0.288796   \n1  0.339793  0.304164  ...  0.222707  0.288889      1  0.355078  0.403126   \n2  0.311724  0.304164  ...  0.186667  0.090909      2  0.317175  0.225214   \n3  0.311724  0.304164  ...  0.360656  0.320000      1  0.278533  0.403126   \n4  0.311724  0.304164  ...  0.375000  0.294118      3  0.403885  0.403126   \n\n      ord_3     ord_4     ord_5  day  month  \n0  0.342476  0.324947  0.300588    5     11  \n1  0.379277  0.186877  0.244795    7      5  \n2  0.206599  0.236891  0.417726    1     12  \n3  0.220460  0.336264  0.365151    2      3  \n4  0.379277  0.409481  0.389864    4     11  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.241790</td>\n      <td>0.360978</td>\n      <td>0.319017</td>\n      <td>0.242813</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.348315</td>\n      <td>0.181818</td>\n      <td>2</td>\n      <td>0.242055</td>\n      <td>0.288796</td>\n      <td>0.342476</td>\n      <td>0.324947</td>\n      <td>0.300588</td>\n      <td>5</td>\n      <td>11</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.302537</td>\n      <td>0.333773</td>\n      <td>0.351052</td>\n      <td>0.338932</td>\n      <td>0.293085</td>\n      <td>0.339793</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.222707</td>\n      <td>0.288889</td>\n      <td>1</td>\n      <td>0.355078</td>\n      <td>0.403126</td>\n      <td>0.379277</td>\n      <td>0.186877</td>\n      <td>0.244795</td>\n      <td>7</td>\n      <td>5</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.290107</td>\n      <td>0.241790</td>\n      <td>0.338932</td>\n      <td>0.245139</td>\n      <td>0.311724</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.186667</td>\n      <td>0.090909</td>\n      <td>2</td>\n      <td>0.317175</td>\n      <td>0.225214</td>\n      <td>0.206599</td>\n      <td>0.236891</td>\n      <td>0.417726</td>\n      <td>1</td>\n      <td>12</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.351052</td>\n      <td>0.310627</td>\n      <td>0.335367</td>\n      <td>0.311724</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.360656</td>\n      <td>0.320000</td>\n      <td>1</td>\n      <td>0.278533</td>\n      <td>0.403126</td>\n      <td>0.220460</td>\n      <td>0.336264</td>\n      <td>0.365151</td>\n      <td>2</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.333773</td>\n      <td>0.351052</td>\n      <td>0.290054</td>\n      <td>0.245139</td>\n      <td>0.311724</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.375000</td>\n      <td>0.294118</td>\n      <td>3</td>\n      <td>0.403885</td>\n      <td>0.403126</td>\n      <td>0.379277</td>\n      <td>0.409481</td>\n      <td>0.389864</td>\n      <td>4</td>\n      <td>11</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## 11. Catboost Encoder\n\n**Catboost** is a recently created target-based categorical encoder. \n\nIt is intended to overcome target leakage problems inherent in LOO. \n\nIf you use `Category-Encoders` it will look like this code below."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nCBE_encoder = CatBoostEncoder()\ntrain_cbe = CBE_encoder.fit_transform(train[feature_list], target)\ntest_cbe = CBE_encoder.transform(test[feature_list])\ntest_cbe.head()","execution_count":27,"outputs":[{"output_type":"stream","text":"CPU times: user 19.9 s, sys: 1.12 s, total: 21 s\nWall time: 17.7 s\n","name":"stdout"},{"output_type":"execute_result","execution_count":27,"data":{"text/plain":"   bin_0  bin_1  bin_2     bin_3     bin_4     nom_0     nom_1     nom_2  \\\n0      0      0      1  0.302537  0.290107  0.241791  0.360976  0.319017   \n1      0      0      0  0.302537  0.333773  0.351051  0.338931  0.293085   \n2      1      0      1  0.309384  0.290107  0.241791  0.338931  0.245141   \n3      0      0      1  0.302537  0.290107  0.351051  0.310626  0.335367   \n4      0      1      1  0.309384  0.333773  0.351051  0.290055  0.245141   \n\n      nom_3     nom_4  ...     nom_8     nom_9  ord_0     ord_1     ord_2  \\\n0  0.242815  0.304164  ...  0.348078  0.187212      2  0.242055  0.288797   \n1  0.339792  0.304164  ...  0.223069  0.289258      1  0.355076  0.403125   \n2  0.311723  0.304164  ...  0.187194  0.108823      2  0.317175  0.225215   \n3  0.311723  0.304164  ...  0.359772  0.319457      1  0.278534  0.403125   \n4  0.311723  0.304164  ...  0.374739  0.294771      3  0.403884  0.403125   \n\n      ord_3     ord_4     ord_5  day  month  \n0  0.342474  0.324946  0.300590    5     11  \n1  0.379275  0.186884  0.244833    7      5  \n2  0.206602  0.236894  0.417684    1     12  \n3  0.220467  0.336262  0.365124    2      3  \n4  0.379275  0.409470  0.389782    4     11  \n\n[5 rows x 23 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>bin_0</th>\n      <th>bin_1</th>\n      <th>bin_2</th>\n      <th>bin_3</th>\n      <th>bin_4</th>\n      <th>nom_0</th>\n      <th>nom_1</th>\n      <th>nom_2</th>\n      <th>nom_3</th>\n      <th>nom_4</th>\n      <th>...</th>\n      <th>nom_8</th>\n      <th>nom_9</th>\n      <th>ord_0</th>\n      <th>ord_1</th>\n      <th>ord_2</th>\n      <th>ord_3</th>\n      <th>ord_4</th>\n      <th>ord_5</th>\n      <th>day</th>\n      <th>month</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.241791</td>\n      <td>0.360976</td>\n      <td>0.319017</td>\n      <td>0.242815</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.348078</td>\n      <td>0.187212</td>\n      <td>2</td>\n      <td>0.242055</td>\n      <td>0.288797</td>\n      <td>0.342474</td>\n      <td>0.324946</td>\n      <td>0.300590</td>\n      <td>5</td>\n      <td>11</td>\n    </tr>\n    <tr>\n      <td>1</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0</td>\n      <td>0.302537</td>\n      <td>0.333773</td>\n      <td>0.351051</td>\n      <td>0.338931</td>\n      <td>0.293085</td>\n      <td>0.339792</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.223069</td>\n      <td>0.289258</td>\n      <td>1</td>\n      <td>0.355076</td>\n      <td>0.403125</td>\n      <td>0.379275</td>\n      <td>0.186884</td>\n      <td>0.244833</td>\n      <td>7</td>\n      <td>5</td>\n    </tr>\n    <tr>\n      <td>2</td>\n      <td>1</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.290107</td>\n      <td>0.241791</td>\n      <td>0.338931</td>\n      <td>0.245141</td>\n      <td>0.311723</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.187194</td>\n      <td>0.108823</td>\n      <td>2</td>\n      <td>0.317175</td>\n      <td>0.225215</td>\n      <td>0.206602</td>\n      <td>0.236894</td>\n      <td>0.417684</td>\n      <td>1</td>\n      <td>12</td>\n    </tr>\n    <tr>\n      <td>3</td>\n      <td>0</td>\n      <td>0</td>\n      <td>1</td>\n      <td>0.302537</td>\n      <td>0.290107</td>\n      <td>0.351051</td>\n      <td>0.310626</td>\n      <td>0.335367</td>\n      <td>0.311723</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.359772</td>\n      <td>0.319457</td>\n      <td>1</td>\n      <td>0.278534</td>\n      <td>0.403125</td>\n      <td>0.220467</td>\n      <td>0.336262</td>\n      <td>0.365124</td>\n      <td>2</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <td>4</td>\n      <td>0</td>\n      <td>1</td>\n      <td>1</td>\n      <td>0.309384</td>\n      <td>0.333773</td>\n      <td>0.351051</td>\n      <td>0.290055</td>\n      <td>0.245141</td>\n      <td>0.311723</td>\n      <td>0.304164</td>\n      <td>...</td>\n      <td>0.374739</td>\n      <td>0.294771</td>\n      <td>3</td>\n      <td>0.403884</td>\n      <td>0.403125</td>\n      <td>0.379275</td>\n      <td>0.409470</td>\n      <td>0.389782</td>\n      <td>4</td>\n      <td>11</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 23 columns</p>\n</div>"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"## Validation\n\nValidation proceeds with single lr and lr with cv.\n\n- I will add OneHotEncoder, etc later.\n- More Fold get better score (my experience)\n- you can try another solver and another parameter"},{"metadata":{},"cell_type":"markdown","source":"### Single LR"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nimport gc\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import roc_auc_score as auc\nfrom sklearn.linear_model import LogisticRegression\n\nencoder_list = [ OrdinalEncoder(), WOEEncoder(), TargetEncoder(), MEstimateEncoder(), JamesSteinEncoder(), LeaveOneOutEncoder() ,CatBoostEncoder()]\n\nX_train, X_val, y_train, y_val = train_test_split(train, target, test_size=0.2, random_state=97)\n\nfor encoder in encoder_list:\n    print(\"Test {} : \".format(str(encoder).split('(')[0]), end=\" \")\n    train_enc = encoder.fit_transform(X_train[feature_list], y_train)\n    #test_enc = encoder.transform(test[feature_list])\n    val_enc = encoder.transform(X_val[feature_list])\n    lr = LogisticRegression(C=0.1, solver=\"lbfgs\", max_iter=1000)\n    lr.fit(train_enc, y_train)\n    lr_pred = lr.predict_proba(val_enc)[:, 1]\n    score = auc(y_val, lr_pred)\n    print(\"score: \", score)\n    del train_enc\n    del val_enc\n    gc.collect()\n\n","execution_count":34,"outputs":[{"output_type":"stream","text":"Test OrdinalEncoder :  ","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"score:  0.6058952245541614\nTest WOEEncoder :  score:  0.7814115175120097\nTest TargetEncoder :  score:  0.7790232741835907\nTest MEstimateEncoder :  score:  0.7792025608680453\nTest JamesSteinEncoder :  score:  0.7719623937439537\nTest LeaveOneOutEncoder :  score:  0.79576549204064\nTest CatBoostEncoder :  score:  0.7917094405644212\nCPU times: user 4min 55s, sys: 2.02 s, total: 4min 57s\nWall time: 3min 15s\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"### LR with CrossValidation"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"%%time\nfrom sklearn.model_selection import KFold\nimport numpy as np\n\n# CV function original : @Peter Hurford : Why Not Logistic Regression? https://www.kaggle.com/peterhurford/why-not-logistic-regression\n\ndef run_cv_model(train, test, target, model_fn, params={}, label='model'):\n    kf = KFold(n_splits=5)\n    fold_splits = kf.split(train, target)\n\n    cv_scores = []\n    pred_full_test = 0\n    pred_train = np.zeros((train.shape[0]))\n    i = 1\n    for dev_index, val_index in fold_splits:\n        print('Started {} fold {}/5'.format(label, i))\n        dev_X, val_X = train.iloc[dev_index], train.iloc[val_index]\n        dev_y, val_y = target[dev_index], target[val_index]\n        pred_val_y, pred_test_y = model_fn(dev_X, dev_y, val_X, val_y, test, params)\n        pred_full_test = pred_full_test + pred_test_y\n        pred_train[val_index] = pred_val_y\n        cv_score = auc(val_y, pred_val_y)\n        cv_scores.append(cv_score)\n        print(label + ' cv score {}: {}'.format(i, cv_score))\n        i += 1\n        \n    print('{} cv scores : {}'.format(label, cv_scores))\n    print('{} cv mean score : {}'.format(label, np.mean(cv_scores)))\n    print('{} cv std score : {}'.format(label, np.std(cv_scores)))\n    pred_full_test = pred_full_test / 5.0\n    results = {'label': label, 'train': pred_train, 'test': pred_full_test, 'cv': cv_scores}\n    return results\n\n\ndef runLR(train_X, train_y, test_X, test_y, test_X2, params):\n    model = LogisticRegression(**params)\n    model.fit(train_X, train_y)\n    pred_test_y = model.predict_proba(test_X)[:, 1]\n    pred_test_y2 = model.predict_proba(test_X2)[:, 1]\n    return pred_test_y, pred_test_y2\n","execution_count":36,"outputs":[{"output_type":"stream","text":"CPU times: user 107 µs, sys: 0 ns, total: 107 µs\nWall time: 116 µs\n","name":"stdout"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"if TEST:\n\n    lr_params = {'solver': 'lbfgs', 'C': 0.1}\n\n    results = list()\n\n    for encoder in  [ OrdinalEncoder(), WOEEncoder(), TargetEncoder(), MEstimateEncoder(), JamesSteinEncoder(), LeaveOneOutEncoder() ,CatBoostEncoder()]:\n        train_enc = encoder.fit_transform(train[feature_list], target)\n        test_enc = encoder.transform(test[feature_list])\n        result = run_cv_model(train_enc, test_enc, target, runLR, lr_params, str(encoder).split('(')[0])\n        results.append(result)\n    results = pd.DataFrame(results)\n    results['cv_mean'] = results['cv'].apply(lambda l : np.mean(l))\n    results['cv_std'] = results['cv'].apply(lambda l : np.std(l))\n    results[['label','cv_mean','cv_std']].head(8)","execution_count":37,"outputs":[{"output_type":"stream","text":"Started OrdinalEncoder fold 1/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"OrdinalEncoder cv score 1: 0.5478087630083479\nStarted OrdinalEncoder fold 2/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"OrdinalEncoder cv score 2: 0.5785368145198041\nStarted OrdinalEncoder fold 3/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"OrdinalEncoder cv score 3: 0.5794711812720389\nStarted OrdinalEncoder fold 4/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"OrdinalEncoder cv score 4: 0.5776419118948795\nStarted OrdinalEncoder fold 5/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"OrdinalEncoder cv score 5: 0.5530765797041557\nOrdinalEncoder cv scores : [0.5478087630083479, 0.5785368145198041, 0.5794711812720389, 0.5776419118948795, 0.5530765797041557]\nOrdinalEncoder cv mean score : 0.5673070500798453\nOrdinalEncoder cv std score : 0.01388216518854039\nStarted WOEEncoder fold 1/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"WOEEncoder cv score 1: 0.8296010840892494\nStarted WOEEncoder fold 2/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"WOEEncoder cv score 2: 0.8276778463251253\nStarted WOEEncoder fold 3/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"WOEEncoder cv score 3: 0.8343609323413513\nStarted WOEEncoder fold 4/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"WOEEncoder cv score 4: 0.8315878216378239\nStarted WOEEncoder fold 5/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"WOEEncoder cv score 5: 0.8307581723870281\nWOEEncoder cv scores : [0.8296010840892494, 0.8276778463251253, 0.8343609323413513, 0.8315878216378239, 0.8307581723870281]\nWOEEncoder cv mean score : 0.8307971713561155\nWOEEncoder cv std score : 0.002213045618434726\nStarted TargetEncoder fold 1/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"TargetEncoder cv score 1: 0.8228825225372465\nStarted TargetEncoder fold 2/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"TargetEncoder cv score 2: 0.819640037288339\nStarted TargetEncoder fold 3/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"TargetEncoder cv score 3: 0.8270373654953843\nStarted TargetEncoder fold 4/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"TargetEncoder cv score 4: 0.8279529191167299\nStarted TargetEncoder fold 5/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"TargetEncoder cv score 5: 0.8270456093348808\nTargetEncoder cv scores : [0.8228825225372465, 0.819640037288339, 0.8270373654953843, 0.8279529191167299, 0.8270456093348808]\nTargetEncoder cv mean score : 0.8249116907545162\nTargetEncoder cv std score : 0.003169511807334321\nStarted MEstimateEncoder fold 1/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"MEstimateEncoder cv score 1: 0.8239353389041686\nStarted MEstimateEncoder fold 2/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"MEstimateEncoder cv score 2: 0.8225460353941083\nStarted MEstimateEncoder fold 3/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"MEstimateEncoder cv score 3: 0.8250872944298472\nStarted MEstimateEncoder fold 4/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"MEstimateEncoder cv score 4: 0.8290057049907481\nStarted MEstimateEncoder fold 5/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"MEstimateEncoder cv score 5: 0.8257572319360416\nMEstimateEncoder cv scores : [0.8239353389041686, 0.8225460353941083, 0.8250872944298472, 0.8290057049907481, 0.8257572319360416]\nMEstimateEncoder cv mean score : 0.8252663211309826\nMEstimateEncoder cv std score : 0.002164601755855072\nStarted JamesSteinEncoder fold 1/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"JamesSteinEncoder cv score 1: 0.825785651337012\nStarted JamesSteinEncoder fold 2/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"JamesSteinEncoder cv score 2: 0.8213704510704314\nStarted JamesSteinEncoder fold 3/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"JamesSteinEncoder cv score 3: 0.8279921606609264\nStarted JamesSteinEncoder fold 4/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"JamesSteinEncoder cv score 4: 0.8144299433590861\nStarted JamesSteinEncoder fold 5/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"JamesSteinEncoder cv score 5: 0.8265248336422055\nJamesSteinEncoder cv scores : [0.825785651337012, 0.8213704510704314, 0.8279921606609264, 0.8144299433590861, 0.8265248336422055]\nJamesSteinEncoder cv mean score : 0.8232206080139323\nJamesSteinEncoder cv std score : 0.004918616364474788\nStarted LeaveOneOutEncoder fold 1/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"LeaveOneOutEncoder cv score 1: 0.7870271221339217\nStarted LeaveOneOutEncoder fold 2/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"LeaveOneOutEncoder cv score 2: 0.7883871851867641\nStarted LeaveOneOutEncoder fold 3/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"LeaveOneOutEncoder cv score 3: 0.7940299539122266\nStarted LeaveOneOutEncoder fold 4/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"LeaveOneOutEncoder cv score 4: 0.7901332335441938\nStarted LeaveOneOutEncoder fold 5/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"LeaveOneOutEncoder cv score 5: 0.7883777050213145\nLeaveOneOutEncoder cv scores : [0.7870271221339217, 0.7883871851867641, 0.7940299539122266, 0.7901332335441938, 0.7883777050213145]\nLeaveOneOutEncoder cv mean score : 0.7895910399596842\nLeaveOneOutEncoder cv std score : 0.0024287055632732923\nStarted CatBoostEncoder fold 1/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"CatBoostEncoder cv score 1: 0.7297166731892779\nStarted CatBoostEncoder fold 2/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"CatBoostEncoder cv score 2: 0.7805421729205533\nStarted CatBoostEncoder fold 3/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"CatBoostEncoder cv score 3: 0.7839670066128355\nStarted CatBoostEncoder fold 4/5\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"},{"output_type":"stream","text":"CatBoostEncoder cv score 4: 0.7795226775861617\nStarted CatBoostEncoder fold 5/5\nCatBoostEncoder cv score 5: 0.786081368408643\nCatBoostEncoder cv scores : [0.7297166731892779, 0.7805421729205533, 0.7839670066128355, 0.7795226775861617, 0.786081368408643]\nCatBoostEncoder cv mean score : 0.7719659797434942\nCatBoostEncoder cv std score : 0.021255246501015294\n","name":"stdout"},{"output_type":"stream","text":"/opt/conda/lib/python3.6/site-packages/sklearn/linear_model/logistic.py:947: ConvergenceWarning: lbfgs failed to converge. Increase the number of iterations.\n  \"of iterations.\", ConvergenceWarning)\n","name":"stderr"}]},{"metadata":{"trusted":true},"cell_type":"code","source":"results","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submit"},{"metadata":{},"cell_type":"markdown","source":"Even CVs did not solve the target based encoder's overfit problem."},{"metadata":{"trusted":true},"cell_type":"code","source":"if TEST:\n    for idx, label in enumerate(results['label']):\n        sub_df = pd.DataFrame({'id': test_id, 'target' : results.iloc[idx]['test']})\n        sub_df.to_csv(\"LR_{}.csv\".format(label), index=False)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":4}