chapter03_deep-neural-networks/mlp-dropout-gluon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout regularization with ``gluon``\n",
    "\n",
    "In [the previous chapter](./mlp-dropout-scratch.ipynb), \n",
    "we introduced Dropout regularization, implementing the algorithm from scratch. \n",
    "As a reminder, Dropout is a regularization technique \n",
    "that zeroes out some fraction of the nodes during training. \n",
    "Then at test time, we use all of the nodes, but scale down their values,\n",
    "essentially averaging the various dropped out nets. \n",
    "If you're approaching this chapter out of sequence,\n",
    "and aren't sure how Dropout works, it's best to take a look at the implementation by hand\n",
    "since ``gluon`` will manage the low-level details for us.\n",
    "\n",
    "Dropout is a special kind of layer because it behaves differently \n",
    "when training and predicting. \n",
    "We've already seen how ``gluon`` can keep track of when to record vs not record the computation graph.\n",
    "Since this is a ``gluon`` implementation chapter,\n",
    "let's get into the thick of things by importing our dependencies and some toy data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import mxnet as mx\n",
    "import numpy as np\n",
    "from mxnet import nd, autograd, gluon\n",
    "ctx = mx.gpu() if mx.test_utils.list_gpus() else mx.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MNIST dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "num_inputs = 784\n",
    "num_outputs = 10\n",
    "def transform(data, label):\n",
    "    return data.astype(np.float32)/255, label.astype(np.float32)\n",
    "train_data = mx.gluon.data.DataLoader(mx.gluon.data.vision.MNIST(train=True, transform=transform),\n",
    "                                      batch_size, shuffle=True)\n",
    "test_data = mx.gluon.data.DataLoader(mx.gluon.data.vision.MNIST(train=False, transform=transform),\n",
    "                                     batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model\n",
    "\n",
    "Now we can add Dropout following each of our hidden layers. \n",
    "Setting the dropout probability to .6 would mean that 60% of activations are dropped (set to zero) out and 40% are kept.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_hidden = 256\n",
    "net = gluon.nn.Sequential()\n",
    "with net.name_scope():\n",
    "    ###########################\n",
    "    # Adding first hidden layer\n",
    "    ###########################\n",
    "    net.add(gluon.nn.Dense(num_hidden, activation=\"relu\"))\n",
    "    ###########################\n",
    "    # Adding dropout with rate .5 to the first hidden layer\n",
    "    ###########################\n",
    "    net.add(gluon.nn.Dropout(.5))\n",
    "    \n",
    "    ###########################\n",
    "    # Adding second hidden layer\n",
    "    ###########################\n",
    "    net.add(gluon.nn.Dense(num_hidden, activation=\"relu\")) \n",
    "    ###########################\n",
    "    # Adding dropout with rate .5 to the second hidden layer\n",
    "    ###########################\n",
    "    net.add(gluon.nn.Dropout(.5))\n",
    "    \n",
    "    ###########################\n",
    "    # Adding the output layer\n",
    "    ###########################\n",
    "    net.add(gluon.nn.Dense(num_outputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter initialization\n",
    "\n",
    "Now that we've got an MLP with dropout layers, let's register an initializer \n",
    "so we can play with some data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net.collect_params().initialize(mx.init.Xavier(magnitude=2.24), ctx=ctx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train mode and predict mode\n",
    "\n",
    "Let's grab some data and pass it through the network.\n",
    "To see what effect dropout is having on our predictions, \n",
    "it's instructive to pass the same example through our net multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x, _ in train_data:\n",
    "    x = x.as_in_context(ctx)\n",
    "    break\n",
    "print(net(x[0:1]))\n",
    "print(net(x[0:1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we got the exact same answer on both forward passes through the net!\n",
    "That's because by, default, ``mxnet`` assumes that we are in predict mode. \n",
    "We can explicitly invoke this scope by placing code within a ``with autograd.predict_mode():`` block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with autograd.predict_mode():\n",
    "    print(net(x[0:1]))\n",
    "    print(net(x[0:1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unless something's gone horribly wrong, you should see the same result as before. \n",
    "We can also run the code in *train mode*.\n",
    "This tells MXNet to run our Blocks as they would run during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with autograd.train_mode():\n",
    "    print(net(x[0:1]))\n",
    "    print(net(x[0:1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing ``is_training()`` status\n",
    "\n",
    "You might wonder, how precisely do the Blocks determine \n",
    "whether they should run in train mode or predict mode?\n",
    "Basically, autograd maintains a Boolean state \n",
    "that can be accessed via ``autograd.is_training()``. \n",
    "By default this value is ``False`` in the global scope.\n",
    "This way if someone just wants to make predictions and \n",
    "doesn't know anything about training models, everything will just work.\n",
    "When we enter a ``train_mode()`` block, \n",
    "we create a scope in which ``is_training()`` returns ``True``. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with autograd.predict_mode():\n",
    "    print(autograd.is_training())\n",
    "    \n",
    "with autograd.train_mode():\n",
    "    print(autograd.is_training())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integration with ``autograd.record``\n",
    "\n",
    "When we train neural network models,\n",
    "we nearly always enter ``record()`` blocks.\n",
    "The purpose of ``record()`` is to build the computational graph.\n",
    "And the purpose of ``train`` is to indicate that we are training our model.\n",
    "These two are highly correlated but should not be confused.\n",
    "For example, when we generate adversarial examples (a topic we'll investigate later)\n",
    "we may want to record, but for the model to behave as in predict mode.\n",
    "On the other hand, sometimes, even when we're not recording,\n",
    "we still want to evaluate the model's training behavior.\n",
    "\n",
    "A problem then arises. Since ``record()`` and ``train_mode()``\n",
    "are distinct, how do we avoid having to declare two scopes every time we train the model?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "##########################\n",
    "#  Writing this every time could get cumbersome\n",
    "##########################\n",
    "with autograd.record():\n",
    "    with autograd.train_mode():\n",
    "        yhat = net(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make our lives a little easier, record() takes one argument, ``train_mode``,\n",
    "which has a default value of True.\n",
    "So when we turn on autograd, this by default turns on train_mode\n",
    "(``with autograd.record()`` is equivalent to\n",
    "``with autograd.record(train_mode=True):``).\n",
    "To change this default behavior\n",
    "(as when generating adversarial examples),\n",
    "we can optionally call record via\n",
    "(``with autograd.record(train_mode=False):``)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Softmax cross-entropy loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "softmax_cross_entropy = gluon.loss.SoftmaxCrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': .1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_accuracy(data_iterator, net):\n",
    "    acc = mx.metric.Accuracy()\n",
    "    for i, (data, label) in enumerate(data_iterator):\n",
    "        data = data.as_in_context(ctx).reshape((-1, 784))\n",
    "        label = label.as_in_context(ctx)\n",
    "        output = net(data)\n",
    "        predictions = nd.argmax(output, axis=1)\n",
    "        acc.update(preds=predictions, labels=label)\n",
    "    return acc.get()[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 10\n",
    "smoothing_constant = .01\n",
    "\n",
    "for e in range(epochs):\n",
    "    for i, (data, label) in enumerate(train_data):\n",
    "        data = data.as_in_context(ctx).reshape((-1, 784))\n",
    "        label = label.as_in_context(ctx)\n",
    "        with autograd.record():\n",
    "            output = net(data)\n",
    "            loss = softmax_cross_entropy(output, label)\n",
    "            loss.backward()\n",
    "        trainer.step(data.shape[0])\n",
    "\n",
    "        ##########################\n",
    "        #  Keep a moving average of the losses\n",
    "        ##########################\n",
    "        curr_loss = nd.mean(loss).asscalar()\n",
    "        moving_loss = (curr_loss if ((i == 0) and (e == 0)) \n",
    "                       else (1 - smoothing_constant) * moving_loss + (smoothing_constant) * curr_loss)\n",
    "\n",
    "    test_accuracy = evaluate_accuracy(test_data, net)\n",
    "    train_accuracy = evaluate_accuracy(train_data, net)\n",
    "    print(\"Epoch %s. Loss: %s, Train_acc %s, Test_acc %s\" %\n",
    "          (e, moving_loss, train_accuracy, test_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Now let's take a look at how to build convolutional neural networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "[Introduction to ``gluon.Block`` and ``gluon.nn.Sequential``](../chapter03_deep-neural-networks/plumbing.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}