08 Overfitting Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overfitting Tutorial\n",
    "\n",
    "Many deep learning models run the danger of overfitting on the training set. When this happens, the model fails to generalize its performance to unseen data, such as a separate validation set. Here we present a simple tutorial on how to recognize overfitting using our visualization tools, and how to apply Dropout layers to prevent overfitting.\n",
    "\n",
    "We use a simple network of convolutional layers on the CIFAR-10 dataset, a dataset of images belonging to 10 categories. \n",
    "\n",
    "The code below will build the model and train on the CIFAR-10 dataset for 25 epochs (~2 minutes on Titan X GPUs), displaying both the training cost as well as the cost on the validation set.\n",
    "\n",
    "Note: We highly recommend users run this model on Maxwell GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.initializers import Gaussian\n",
    "from neon.optimizers import GradientDescentMomentum, Schedule\n",
    "from neon.layers import Conv, Dropout, Activation, Pooling, GeneralizedCost\n",
    "from neon.transforms import Rectlin, Softmax, CrossEntropyMulti, Misclassification\n",
    "from neon.models import Model\n",
    "from neon.data import CIFAR10\n",
    "from neon.callbacks.callbacks import Callbacks\n",
    "from neon.backends import gen_backend\n",
    "\n",
    "be = gen_backend(batch_size=128, backend='gpu')\n",
    "\n",
    "# hyperparameters\n",
    "learning_rate = 0.05\n",
    "weight_decay = 0.001\n",
    "num_epochs = 25\n",
    "\n",
    "print \"Loading Data\"\n",
    "dataset = CIFAR10(path='data/', normalize=False, \n",
    "                  contrast_normalize=True, whiten=True, \n",
    "                  pad_classes=True)  # CIFAR10 has 10 classes, network has 16 outputs, so we pad some extra classes.\n",
    "train_set = dataset.train_iter\n",
    "valid_set = dataset.valid_iter\n",
    "\n",
    "print \"Building Model\"\n",
    "init_uni = Gaussian(scale=0.05)\n",
    "opt_gdm = GradientDescentMomentum(learning_rate=float(learning_rate), momentum_coef=0.9,\n",
    "                                  wdecay=float(weight_decay),\n",
    "                                  schedule=Schedule(step_config=[200, 250, 300], change=0.1))\n",
    "\n",
    "relu = Rectlin()\n",
    "conv = dict(init=init_uni, batch_norm=False, activation=relu)\n",
    "convp1 = dict(init=init_uni, batch_norm=False, activation=relu, padding=1)\n",
    "convp1s2 = dict(init=init_uni, batch_norm=False, activation=relu, padding=1, strides=2)\n",
    "\n",
    "layers = [\n",
    "          Conv((3, 3, 64), **convp1),\n",
    "          Conv((3, 3, 64), **convp1s2),\n",
    "          Conv((3, 3, 128), **convp1),\n",
    "          Conv((3, 3, 128), **convp1s2),\n",
    "          Conv((3, 3, 128), **convp1),\n",
    "          Conv((1, 1, 128), **conv),\n",
    "          Conv((1, 1, 16), **conv),\n",
    "          Pooling(8, op=\"avg\"),\n",
    "          Activation(Softmax())]\n",
    "\n",
    "cost = GeneralizedCost(costfunc=CrossEntropyMulti())\n",
    "\n",
    "mlp = Model(layers=layers)\n",
    "\n",
    "\n",
    "# configure callbacks\n",
    "callbacks = Callbacks(mlp, output_file='data.h5', eval_set=valid_set, eval_freq=1)\n",
    "\n",
    "print \"Training\"\n",
    "mlp.fit(train_set, optimizer=opt_gdm, num_epochs=num_epochs, cost=cost, callbacks=callbacks)\n",
    "\n",
    "print('Misclassification error = %.1f%%' % (mlp.eval(valid_set, metric=Misclassification())*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overfitting\n",
    "You should notice that in the logs above, after around Epoch 15, the model begins to overfit. Even though the cost on the training set continues to decrease, the validation loss flattens (even increasing slightly). We can visualize these effects using the code below.\n",
    "\n",
    "Note: The same plots can be created using our `nvis` command line utility (see: http://neon.nervanasys.com/docs/latest/tools.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from neon.visualizations.figure import cost_fig, hist_fig, deconv_summary_page\n",
    "from neon.visualizations.data import h5_cost_data, h5_hist_data, h5_deconv_data\n",
    "from bokeh.plotting import output_notebook, show\n",
    "\n",
    "cost_data = h5_cost_data('data.h5', False)\n",
    "output_notebook()\n",
    "show(cost_fig(cost_data, 400, 800, epoch_axis=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "This situation illustrates the importance of plotting the validation loss (blue) in addition to the training cost (red). The training cost may mislead the user into thinking that model is continuing to perform well, but we can see from the validation loss that the model has begun to overfit.\n",
    "\n",
    "## Dropout layers\n",
    "\n",
    "To correct overfitting, we introduce `Dropout` layers to the model, as shown below. `Dropout` layers randomly silence a subset of units for each minibatch, and are an effective means of preventing overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "layers = [\n",
    "          Conv((3, 3, 64), **convp1),\n",
    "          Conv((3, 3, 64), **convp1s2),\n",
    "          Dropout(keep=.5),   # Added Dropout\n",
    "          Conv((3, 3, 128), **convp1),\n",
    "          Conv((3, 3, 128), **convp1s2),\n",
    "          Dropout(keep=.5),   # Added Dropout\n",
    "          Conv((3, 3, 128), **convp1),\n",
    "          Conv((1, 1, 128), **conv),\n",
    "          Conv((1, 1, 16), **conv),\n",
    "          Pooling(8, op=\"avg\"),\n",
    "          Activation(Softmax())]\n",
    "\n",
    "cost = GeneralizedCost(costfunc=CrossEntropyMulti())\n",
    "\n",
    "mlp = Model(layers=layers)\n",
    "\n",
    "\n",
    "# configure callbacks\n",
    "callbacks = Callbacks(mlp, output_file='data.h5', eval_set=valid_set, eval_freq=1)\n",
    "\n",
    "print \"Training\"\n",
    "mlp.fit(train_set, optimizer=opt_gdm, num_epochs=num_epochs, cost=cost, callbacks=callbacks)\n",
    "\n",
    "print('Misclassification error = %.1f%%' % (mlp.eval(valid_set, metric=Misclassification())*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then plot the results of the training run below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from neon.visualizations.figure import cost_fig, hist_fig, deconv_summary_page\n",
    "from neon.visualizations.data import h5_cost_data, h5_hist_data, h5_deconv_data\n",
    "from bokeh.plotting import output_notebook, show\n",
    "\n",
    "cost_data = h5_cost_data('data.h5', False)\n",
    "output_notebook()\n",
    "show(cost_fig(cost_data, 400, 800, epoch_axis=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the dropout layers in place, the model is now able to continue performing well on the validation set beyond epoch 15. The validation loss (blue) is not shifted downwards compared to the previous figure, and the model reaches better validation performance. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}