03_basics_of_keras_sequential_api.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%HTML\n",
    "<link rel=\"stylesheet\" type=\"text/css\" href=\"//fonts.googleapis.com/css?family=Quicksand:300\" />\n",
    "<link rel=\"stylesheet\" type=\"text/css\" href=\"custom.css\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tell Keras to use tensorflow as backend\n",
    "\n",
    "At the moment is the default option"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ['KERAS_BACKEND'] = 'tensorflow'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import inspect\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "\n",
    "from utils import plot_training_summary\n",
    "from utils import TimeSummary\n",
    "from utils import set_seed\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = 15, 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Basics of Keras sequential API\n",
    "\n",
    "- Rodrigo Agundez\n",
    "- Amsterdam @ Booking.com\n",
    "- Friday May 25th, 2018\n",
    "\n",
    "![footer_logo](images/logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Goal**\n",
    "\n",
    "- Get familiar with the Keras sequential API\n",
    "- Build a feed forward neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import `Sequential` model API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import predefined `layers`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from keras import layers\n",
    "[layer for layer in dir(layers) if not layer.startswith('_')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import `optimizers`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras import optimizers\n",
    "[opt for opt in dir(optimizers) if not opt.startswith('_')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build dummy  model\n",
    "\n",
    "![simple nn](images/model_diagram.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "first initialize the model with the sequential API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential(name='DummyModel')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can immidiatelly add hidden layers using `model.add()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(model.add)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for each layer you can define the metadata parameters of it. For example the `Dense()` layer has name, number of units, its activation function, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(layers.Dense)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "create the firts hidden layer of 3 units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this should raise a ValueError\n",
    "model.add(layers.Dense(name='FullyConnected_1', units=3, activation='relu'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras takes a simple approach and defines the input layer together with the first hidden layer via the parameter `input_dim` or `input_shape`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(layers.Dense(name='FullyConnected_1', units=3, activation='relu', input_dim=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's see the structure of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question:** why are there 9 parameters?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### extra\n",
    ">\n",
    "> Which other activations could you use? Check out the [list of activations](https://keras.io/activations/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now we can add the next hidden layer of 2 units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(layers.Dense(name='FullyConnected_2', units=2, activation='relu'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "add the output layer of a single unit and use a `sigmoid` activation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(layers.Dense(name='FullyConnected_OutputLayer', units=1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Models have to be compiled before training, we need to add:\n",
    "\n",
    "- optimizer\n",
    "- loss function\n",
    "- metrics\n",
    "\n",
    "The loss function defines the goal of our model. In this case binary classification.\n",
    "\n",
    "The metric(s) set can be used to evaluate over the test dataset, but also, if at trainining time we define `validation_split`, a validation test will be performed over each epoch. This is very helpful to asses the health of our model (overfitting for example)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer='Adam',\n",
    "    loss='binary_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### extra\n",
    ">\n",
    "> - Which other optimizers are available? Check the [list of optimizers](https://keras.io/optimizers/)\n",
    "> - How many losses are there avilable? Check the [list of loss functions](https://keras.io/losses/)\n",
    "> - What other metrics? [List of metrics](https://keras.io/metrics/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you really want the complete picture, check the configuration. Don't worry if you don't understand what all the parameters mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.get_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model is ready be trained but where is the data?\n",
    "\n",
    "Normally you would look at the data frist of course before creating the model, but this notebook focuses on the Keras API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "moons = pd.read_csv('data/moons.csv')\n",
    "print('(rows, columns):', moons.shape)\n",
    "moons.sample(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "non-linearly separable data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.lmplot(data=moons, x='x1', y='x2', hue='y', fit_reg=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "separate in train and test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_index = moons.sample(frac=0.8, random_state=21).index\n",
    "X_train, y_train = moons.iloc[train_index][['x1', 'x2']], moons.iloc[train_index]['y']\n",
    "X_test, y_test = moons.drop(index=train_index)[['x1', 'x2']], moons.drop(index=train_index)['y']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add custom callback to collect execution time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "time_summary = TimeSummary()\n",
    "print(inspect.getsource(TimeSummary))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> #### extra\n",
    ">\n",
    "> Which default callbacks are available? Check the [list of callbacks](https://keras.io/callbacks/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary = model.fit(\n",
    "    X_train, y_train,\n",
    "    batch_size=100,\n",
    "    epochs=10,\n",
    "    validation_split=0.1,\n",
    "    verbose=0,\n",
    "    callbacks=[time_summary]\n",
    ")\n",
    "score = model.evaluate(X_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "plot_training_summary(summary, time_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explicit creation of layers\n",
    "\n",
    "The model we created looks like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.Dense(name='FullyConnected_1', units=3, activation='relu', input_dim=2))\n",
    "model.add(layers.Dense(name='FullyConnected_2', units=2, activation='relu'))\n",
    "model.add(layers.Dense(name='FullyConnected_OutputLayer', units=1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we could have defined explicitly each of the activation functions components as an extra layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "# input layer transformations (none in this case)\n",
    "\n",
    "# 1st hidden layer\n",
    "model.add(layers.Dense(name='HiddenLayer_1', units=3, input_dim=2))\n",
    "model.add(layers.Activation(name='ReLu_1', activation='relu'))\n",
    "\n",
    "# 2nd hidden layer\n",
    "model.add(layers.Dense(name='HiddenLayer_2', units=2))\n",
    "model.add(layers.Activation(name='ReLu_2', activation='relu'))\n",
    "# output layer\n",
    "model.add(layers.Dense(name='OutputLayer', units=1))\n",
    "model.add(layers.Activation(name='Sigmoid_3', activation='sigmoid'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "this is very important to understand as it allows for flexibility on the layer order when customizing a deep neaural network.\n",
    "\n",
    "NOTE: we use this explicit layer declaration in exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1\n",
    "\n",
    "Change the parameters:\n",
    "\n",
    "- batch_size\n",
    "- epochs\n",
    "\n",
    "and observe what happens to the execution time and learning curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_model():\n",
    "    model = Sequential()\n",
    "    model.add(layers.Dense(name='FullyConnected_1', units=3, activation='relu', input_dim=2))\n",
    "    model.add(layers.Dense(name='FullyConnected_2', units=2, activation='relu'))\n",
    "    model.add(layers.Dense(name='FullyConnected_OutputLayer', units=1, activation='sigmoid'))\n",
    "    return model\n",
    "\n",
    "model = make_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seed(123) # for reproducibility\n",
    "\n",
    "model = make_model()\n",
    "\n",
    "model.compile(\n",
    "    optimizer='Adam',\n",
    "    loss='binary_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")\n",
    "summary = model.fit(\n",
    "    X_train, y_train,\n",
    "    batch_size=100,\n",
    "    epochs=10,\n",
    "    validation_split=0.1,\n",
    "    verbose=0,\n",
    "    callbacks=[time_summary]\n",
    ")\n",
    "score = model.evaluate(X_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "plot_training_summary(summary, time_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2 (solution in `utils.py`)\n",
    "\n",
    "Make an overkill model for this data\n",
    "\n",
    "- 3 hidden dense layers\n",
    "- use a [`BatchNormalization`](https://keras.io/layers/normalization/) layer after each neuron layer before activation\n",
    "- add a `Dropout` layers\n",
    "- use `relu` and `tanh` or other [activation functions](https://keras.io/activations/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_overkill_model():\n",
    "    model = Sequential()\n",
    "    # input layer batch normalization (what does this do qualitatively speaking?)\n",
    "    \n",
    "    # 1st hidden\n",
    "    \n",
    "    # 2nd hidden\n",
    "    \n",
    "    # 3rd hidden\n",
    "    \n",
    "    # output layer\n",
    "    \n",
    "    return model\n",
    "\n",
    "model = make_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set_seed(123) # for reproducibility\n",
    "\n",
    "model = make_overkill_model()\n",
    "\n",
    "model.compile(\n",
    "    optimizer='Adam',\n",
    "    loss='binary_crossentropy',\n",
    "    metrics=['accuracy']\n",
    ")\n",
    "summary = model.fit(\n",
    "    X_train, y_train,\n",
    "    batch_size=100,\n",
    "    epochs=10,\n",
    "    validation_split=0.1,\n",
    "    verbose=0,\n",
    "    callbacks=[time_summary]\n",
    ")\n",
    "score = model.evaluate(X_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "plot_training_summary(summary, time_summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#  [Next: Build your first convolutional neural network](04_build_your_first_convolutional_neural_network.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:pydata]",
   "language": "python",
   "name": "conda-env-pydata-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}