Understanding_fairnesss_indicators.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Titanic had three cabin classes: first class was the most\n",
    "expensive, second class was in the middle, and third class, or\n",
    "steerage, was the least expensive and in the lower decks. It is\n",
    "well documented that most passengers who survived were\n",
    "female and in first-class cabins. We also know that gender and\n",
    "class played an important role in the selection process for getting\n",
    "on the lifeboats. That selection process prioritized women\n",
    "and children over men. Because this background is so well\n",
    "known, this dataset is suitable as a didactic example to investigate\n",
    "model bias."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Follow through from Author *KC Tung*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import Libraries\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import layers\n",
    "import tensorflow_model_analysis as tfma\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from google.protobuf import text_format\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>survived</th>\n",
       "      <td>627.0</td>\n",
       "      <td>0.387560</td>\n",
       "      <td>0.487582</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>1.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>627.0</td>\n",
       "      <td>29.631308</td>\n",
       "      <td>12.511818</td>\n",
       "      <td>0.75</td>\n",
       "      <td>23.0000</td>\n",
       "      <td>28.0000</td>\n",
       "      <td>35.0000</td>\n",
       "      <td>80.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>n_siblings_spouses</th>\n",
       "      <td>627.0</td>\n",
       "      <td>0.545455</td>\n",
       "      <td>1.151090</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>1.0000</td>\n",
       "      <td>8.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parch</th>\n",
       "      <td>627.0</td>\n",
       "      <td>0.379585</td>\n",
       "      <td>0.792999</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>5.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fare</th>\n",
       "      <td>627.0</td>\n",
       "      <td>34.385399</td>\n",
       "      <td>54.597730</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>15.0458</td>\n",
       "      <td>31.3875</td>\n",
       "      <td>512.3292</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    count       mean        std   min      25%      50%  \\\n",
       "survived            627.0   0.387560   0.487582  0.00   0.0000   0.0000   \n",
       "age                 627.0  29.631308  12.511818  0.75  23.0000  28.0000   \n",
       "n_siblings_spouses  627.0   0.545455   1.151090  0.00   0.0000   0.0000   \n",
       "parch               627.0   0.379585   0.792999  0.00   0.0000   0.0000   \n",
       "fare                627.0  34.385399  54.597730  0.00   7.8958  15.0458   \n",
       "\n",
       "                        75%       max  \n",
       "survived             1.0000    1.0000  \n",
       "age                 35.0000   80.0000  \n",
       "n_siblings_spouses   1.0000    8.0000  \n",
       "parch                0.0000    5.0000  \n",
       "fare                31.3875  512.3292  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_file_path = r'C:\\Users\\DELL\\.keras\\datasets\\train.csv'\n",
    "test_file_path = r'C:\\Users\\DELL\\.keras\\datasets\\eval.csv'\n",
    "\n",
    "titanic_df = pd.read_csv(train_file_path, header='infer')\n",
    "test_df = pd.read_csv(test_file_path, header='infer')\n",
    "\n",
    "titanic_df.describe().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read CSV data with TF\n",
    "LABEL_COLUMN = 'survived'\n",
    "LABELS = [0, 1]\n",
    "\n",
    "train_ds = tf.data.experimental.make_csv_dataset(\n",
    "    train_file_path, batch_size=3, label_name=LABEL_COLUMN, na_value='?',\n",
    "    num_epochs=1, ignore_errors=True\n",
    ")\n",
    "\n",
    "test_ds = tf.data.experimental.make_csv_dataset(\n",
    "    test_file_path, batch_size=3, label_name=LABEL_COLUMN, na_value='?',\n",
    "    num_epochs=1, ignore_errors=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sex :  ['male' 'female']\n",
      "class :  ['Third' 'First' 'Second']\n",
      "deck :  ['unknown' 'C' 'G' 'A' 'B' 'D' 'F' 'E']\n",
      "embark_town :  ['Southampton' 'Cherbourg' 'Queenstown' 'unknown']\n",
      "alone :  ['n' 'y']\n"
     ]
    }
   ],
   "source": [
    "# designate columns with Feature-columns\n",
    "feature_columns = []\n",
    "\n",
    "# numeric columns\n",
    "for header in ['age', 'n_siblings_spouses', 'parch', 'fare']:\n",
    "    feature_columns.append(tf.feature_column.numeric_column(header))\n",
    "\n",
    "\n",
    "# binning the AGE column based on the quartiles in DESC above\n",
    "age = tf.feature_column.numeric_column('age')\n",
    "age_buckets = tf.feature_column.bucketized_column(age, boundaries=[23, 28, 35])\n",
    "\n",
    "# checking the Unique values in the Categorical values\n",
    "h = {}\n",
    "for col in titanic_df:\n",
    "    if col in ['sex', 'class', 'deck', 'embark_town', 'alone']:\n",
    "        print(col, ': ', titanic_df[col].unique())\n",
    "        h[col] = titanic_df[col].unique()\n",
    "\n",
    "\n",
    "# how to enocde CAT columns & OHE it\n",
    "sex_type = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    key='Type', vocabulary_list=['male', 'female']\n",
    ")\n",
    "sex_type_one_hot = tf.feature_column.indicator_column(sex_type)\n",
    "\n",
    "\n",
    "\n",
    "# a better way to encode categorical values\n",
    "sex_type = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'Type', h.get('sex').tolist()\n",
    ")\n",
    "sex_type_one_hot = tf.feature_column.indicator_column(sex_type)\n",
    "\n",
    "\n",
    "class_type = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'Type', h.get('class').tolist()\n",
    ")\n",
    "class_type_one_hot = tf.feature_column.indicator_column(class_type)\n",
    "\n",
    "deck_type = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'Type', h.get('deck').tolist()\n",
    ")\n",
    "deck_type_one_hot = tf.feature_column.indicator_column(deck_type)\n",
    "\n",
    "embark_town_type = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'Type', h.get('embark_town').tolist()\n",
    ")\n",
    "embark_town_type_one_hot = tf.feature_column.indicator_column(embark_town_type)\n",
    "\n",
    "alone_type = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'Type', h.get('alone').tolist()\n",
    ")\n",
    "alone_type_one_hot = tf.feature_column.indicator_column(alone_type)\n",
    "\n",
    "\n",
    "# DECK column has 8 unique value more than others, so reduce to 3 dimenstions\n",
    "deck = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'deck', titanic_df.deck.unique()\n",
    ")\n",
    "deck_embedding = tf.feature_column.embedding_column(deck, dimension=3)\n",
    "\n",
    "# another way to reduce dimension\n",
    "class_hashed = tf.feature_column.categorical_column_with_hash_bucket(\n",
    "    'class', hash_bucket_size=4\n",
    ")\n",
    "\n",
    "# how to create a HASH Bucket\n",
    "cross_type_feature = tf.feature_column.crossed_column(['sex', 'class'], hash_bucket_size=5)\n",
    "feature_columns = [] # list to hold features to use\n",
    "\n",
    "# append numeric columns\n",
    "for header in ['age', 'n_siblings_spouses', 'parch', 'fare']:\n",
    "    feature_columns.append(tf.feature_column.numeric_column(header))\n",
    "\n",
    "age = tf.feature_column.numeric_column('age')\n",
    "age_buckets = tf.feature_column.bucketized_column(age, boundaries=[23, 28, 35])\n",
    "feature_columns.append(age_buckets)\n",
    "\n",
    "# append categorical columns\n",
    "indicator_column_names = ['sex', 'class', 'deck', 'embark_town', 'alone']\n",
    "for col_name in indicator_column_names:\n",
    "    categorical_column = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "        col_name, titanic_df[col_name].unique()\n",
    "    )\n",
    "    indicator_column = tf.feature_column.indicator_column(categorical_column)\n",
    "    feature_columns.append(indicator_column)\n",
    "\n",
    "# append embedding columns\n",
    "deck = tf.feature_column.categorical_column_with_vocabulary_list(\n",
    "    'deck', titanic_df.deck.unique()\n",
    ")\n",
    "deck_embedding = tf.feature_column.embedding_column(deck, dimension=3)\n",
    "feature_columns.append(deck_embedding)\n",
    "\n",
    "# append crossed columns\n",
    "feature_columns.append(tf.feature_column.indicator_column(cross_type_feature))\n",
    "\n",
    "\n",
    "# read the CSV files\n",
    "train_df = pd.read_csv(r'C:\\Users\\DELL\\.keras\\datasets\\train.csv')\n",
    "test_df = pd.read_csv(r'C:\\Users\\DELL\\.keras\\datasets\\eval.csv')\n",
    "\n",
    "\n",
    "# now create a Feature Layer which would serve as the first input layer\n",
    "feature_layer = layers.DenseFeatures(feature_columns)\n",
    "\n",
    "val_df, test_df = train_test_split(test_df, test_size=0.4)\n",
    "batch_size = 32\n",
    "labels = train_df.pop('survived')\n",
    "working_ds = tf.data.Dataset.from_tensor_slices((dict(train_df), labels))\n",
    "working_ds = working_ds.shuffle(buffer_size=len(train_df))\n",
    "train_ds = working_ds.batch(batch_size)\n",
    "\n",
    "# functino to convert Pandas DF to Tensor slices\n",
    "def pandas_to_dataset(df, shuffle=True, batch_size=32):\n",
    "    df = df.copy()\n",
    "    labels = df.pop('survived')\n",
    "    ds = tf.data.Dataset.from_tensor_slices((dict(df), labels))\n",
    "    if shuffle:\n",
    "        ds = ds.shuffle(batch_size=len(df))\n",
    "    ds = ds.batch(batch_size)\n",
    "    return ds\n",
    "\n",
    "val_ds = pandas_to_dataset(val_df, shuffle=False, batch_size=batch_size)\n",
    "test_ds = pandas_to_dataset(test_df, shuffle=False, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 1/20 [>.............................] - ETA: 23s - loss: 1.0704 - accuracy: 0.3750WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20/20 [==============================] - 2s 22ms/step - loss: 0.9034 - accuracy: 0.6364 - val_loss: 0.7880 - val_accuracy: 0.7025\n",
      "Epoch 2/10\n",
      "20/20 [==============================] - 0s 2ms/step - loss: 0.8887 - accuracy: 0.6332 - val_loss: 0.5642 - val_accuracy: 0.7025\n",
      "Epoch 3/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.7167 - accuracy: 0.6651 - val_loss: 0.5604 - val_accuracy: 0.7089\n",
      "Epoch 4/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.6207 - accuracy: 0.7161 - val_loss: 0.5440 - val_accuracy: 0.7215\n",
      "Epoch 5/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.5886 - accuracy: 0.7065 - val_loss: 0.5390 - val_accuracy: 0.6835\n",
      "Epoch 6/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.5810 - accuracy: 0.7129 - val_loss: 0.4862 - val_accuracy: 0.7278\n",
      "Epoch 7/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.5372 - accuracy: 0.7289 - val_loss: 0.5099 - val_accuracy: 0.7595\n",
      "Epoch 8/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.5942 - accuracy: 0.7145 - val_loss: 0.5103 - val_accuracy: 0.7722\n",
      "Epoch 9/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.5755 - accuracy: 0.7496 - val_loss: 0.5070 - val_accuracy: 0.8101\n",
      "Epoch 10/10\n",
      "20/20 [==============================] - 0s 3ms/step - loss: 0.5457 - accuracy: 0.7544 - val_loss: 0.5143 - val_accuracy: 0.7722\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x295acbdc898>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = tf.keras.Sequential([\n",
    "    feature_layer,\n",
    "    layers.Dense(128, activation='relu'),\n",
    "    layers.Dense(128, activation='relu'),\n",
    "    layers.Dropout(.1),\n",
    "    layers.Dense(1)\n",
    "])\n",
    "model.compile(\n",
    "    optimizer='adam', loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
    "    metrics=['accuracy']\n",
    ")\n",
    "model.fit(train_ds, validation_data=val_ds, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continue from here for Fairness Indicator Tutorial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layers in a Sequential model should only have a single input tensor, but we receive a <class 'dict'> input: {'sex': <tf.Tensor 'ExpandDims_8:0' shape=(None, 1) dtype=string>, 'age': <tf.Tensor 'ExpandDims:0' shape=(None, 1) dtype=float64>, 'n_siblings_spouses': <tf.Tensor 'ExpandDims_6:0' shape=(None, 1) dtype=int64>, 'parch': <tf.Tensor 'ExpandDims_7:0' shape=(None, 1) dtype=int64>, 'fare': <tf.Tensor 'ExpandDims_5:0' shape=(None, 1) dtype=float64>, 'class': <tf.Tensor 'ExpandDims_2:0' shape=(None, 1) dtype=string>, 'deck': <tf.Tensor 'ExpandDims_3:0' shape=(None, 1) dtype=string>, 'embark_town': <tf.Tensor 'ExpandDims_4:0' shape=(None, 1) dtype=string>, 'alone': <tf.Tensor 'ExpandDims_1:0' shape=(None, 1) dtype=string>}\n",
      "Consider rewriting this model with the Functional API.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 1.2949723 ],\n",
       "       [ 1.0698959 ],\n",
       "       [-0.84793216],\n",
       "       [-0.31190944],\n",
       "       [-1.2318875 ]], dtype=float32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction_raw = model.predict(test_ds)\n",
    "prediction_raw[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>predicted</th>\n",
       "      <th>survived</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>n_siblings_spouses</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>class</th>\n",
       "      <th>deck</th>\n",
       "      <th>embark_town</th>\n",
       "      <th>alone</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>205</th>\n",
       "      <td>1.294972</td>\n",
       "      <td>0</td>\n",
       "      <td>female</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>14.4542</td>\n",
       "      <td>Third</td>\n",
       "      <td>unknown</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1.069896</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7500</td>\n",
       "      <td>Third</td>\n",
       "      <td>unknown</td>\n",
       "      <td>Queenstown</td>\n",
       "      <td>y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>229</th>\n",
       "      <td>-0.847932</td>\n",
       "      <td>0</td>\n",
       "      <td>male</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2250</td>\n",
       "      <td>Third</td>\n",
       "      <td>unknown</td>\n",
       "      <td>Cherbourg</td>\n",
       "      <td>y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>223</th>\n",
       "      <td>-0.311909</td>\n",
       "      <td>0</td>\n",
       "      <td>male</td>\n",
       "      <td>18.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11.5000</td>\n",
       "      <td>Second</td>\n",
       "      <td>unknown</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-1.231887</td>\n",
       "      <td>0</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "      <td>Third</td>\n",
       "      <td>unknown</td>\n",
       "      <td>Southampton</td>\n",
       "      <td>y</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     predicted  survived     sex   age  n_siblings_spouses  parch     fare  \\\n",
       "205   1.294972         0  female  18.0                   0      1  14.4542   \n",
       "78    1.069896         1  female  22.0                   0      0   7.7500   \n",
       "229  -0.847932         0    male  28.0                   0      0   7.2250   \n",
       "223  -0.311909         0    male  18.0                   0      0  11.5000   \n",
       "0    -1.231887         0    male  35.0                   0      0   8.0500   \n",
       "\n",
       "      class     deck  embark_town alone  \n",
       "205   Third  unknown    Cherbourg     n  \n",
       "78    Third  unknown   Queenstown     y  \n",
       "229   Third  unknown    Cherbourg     y  \n",
       "223  Second  unknown  Southampton     y  \n",
       "0     Third  unknown  Southampton     y  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# convert Predictions to Python list and append to TEST_DF\n",
    "prediction_list = prediction_raw.squeeze().tolist()\n",
    "test_df['predicted'] = prediction_list # adding Predicted Column to Dataframe\n",
    "\n",
    "# put the PREDICTED column as the first column to compare with Survived\n",
    "cols = list(test_df.columns)\n",
    "cols = [cols[-1]] + cols[:-1]\n",
    "test_df = test_df[cols]\n",
    "\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": "\n        if (typeof window.interactive_beam_jquery == 'undefined') {\n          var jqueryScript = document.createElement('script');\n          jqueryScript.src = 'https://code.jquery.com/jquery-3.4.1.slim.min.js';\n          jqueryScript.type = 'text/javascript';\n          jqueryScript.onload = function() {\n            var datatableScript = document.createElement('script');\n            datatableScript.src = 'https://cdn.datatables.net/1.10.20/js/jquery.dataTables.min.js';\n            datatableScript.type = 'text/javascript';\n            datatableScript.onload = function() {\n              window.interactive_beam_jquery = jQuery.noConflict(true);\n              window.interactive_beam_jquery(document).ready(function($){\n                \n              });\n            }\n            document.head.appendChild(datatableScript);\n          };\n          document.head.appendChild(jqueryScript);\n        } else {\n          window.interactive_beam_jquery(document).ready(function($){\n            \n          });\n        }"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:root:Make sure that locally built Python SDK docker image has Python 3.7 interpreter.\n",
      "WARNING:apache_beam.io.tfrecordio:Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.\n",
      "WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: \n",
      "WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: \n",
      "WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: \n",
      "WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: \n",
      "WARNING:apache_beam.io.filebasedsink:Deleting 1 existing files in target path matching: \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Users\\DELL\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\tensorflow_model_analysis\\writers\\metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use eager execution and: \n",
      "`tf.data.TFRecordDataset(path)`\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Users\\DELL\\AppData\\Local\\Programs\\Python\\Python37\\lib\\site-packages\\tensorflow_model_analysis\\writers\\metrics_plots_and_validations_writer.py:113: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use eager execution and: \n",
      "`tf.data.TFRecordDataset(path)`\n"
     ]
    }
   ],
   "source": [
    "# define the Evaluation configuration\n",
    "eval_config = text_format.Parse(\"\"\"\n",
    "    model_specs {\n",
    "        prediction_key: 'predicted',\n",
    "        label_key: 'survived'\n",
    "    }\n",
    "    metrics_specs {\n",
    "        metrics {class_name: 'AUC'}\n",
    "        metrics {\n",
    "            class_name: 'FairnessIndicators'\n",
    "            config: '{\"thresholds\": [0.1, 0.50, 0.90]}'\n",
    "        }\n",
    "        metrics {class_name: 'ExampleCount'}\n",
    "    }\n",
    "\n",
    "    slicing_specs {\n",
    "        feature_keys: ['sex', 'class']\n",
    "    }\n",
    "\n",
    "    slicing_specs {}\n",
    "\"\"\", tfma.EvalConfig())\n",
    "\n",
    "# specify path to output Model\n",
    "OUTPUT_PATH = r'C:\\Users\\DELL\\Desktop\\Learning projects'\n",
    "\n",
    "eval_result = tfma.analyze_raw_data(\n",
    "    data=test_df, eval_config=eval_config, output_path=OUTPUT_PATH\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Enabling notebook extension tensorflow_model_analysis...\n",
      "      - Validating: problems found:\n",
      "        - require?  X tensorflow_model_analysis\n"
     ]
    }
   ],
   "source": [
    "!jupyter nbextension enable tensorflow_model_analysis --user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2baad84ec88c4d7a984aee1aed46caf9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FairnessIndicatorViewer(slicingMetrics=[{'sliceValue': 'Overall', 'slice': 'Overall', 'metrics': {'example_cou…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# rendering the Fairness Indicators\n",
    "\n",
    "tfma.addons.fairness.view.widget_view.render_fairness_indicator(eval_result)\n",
    "# this Widget correctly displayed on Jupyter notebook but won't on GitHub. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different parameters where clicked and analyzed."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "2b7b5a710781a87842cfe8010c06442799678ef166246941c85da77d9a8410c1"
  },
  "kernelspec": {
   "display_name": "Python 3.7.1 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}