02-03-classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets('MNIST_data', one_hot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = tf.InteractiveSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST\n",
    "\n",
    "MNIST is a database of handwritten digits. The images are 28x28 greyscale images (encoded as 784-dimensional vectors in row-major order). There are 60,000 images in the training set, and there are 10,000 images in the test set.\n",
    "\n",
    "Why is there a train-test split? We care about how our function generalizes, and so we want to benchmark its performance on a set of data that it hasn't seen before. Otherwise, a \"perfect\" learning algorithm could just memorize all the data points, but this algorithm wouldn't generalize well.\n",
    "\n",
    "Let's see what one of the MNIST images looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(mnist.test.images[0].reshape(28, 28), cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the label of this image. The MNIST labels are encoded in one-hot format. There are 10 possible labels, and the vector with label $i$ is the $i$-dimensional vector that has the entry $1$ in the $i$th position and $0$s elsewhere."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist.test.labels[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.argmax(mnist.test.labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's design a fully-connected neural network to classify MNIST digits. It should take a 784-dimensional input and give a 10-dimensional one-hot encoded probability distribution as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = tf.placeholder(tf.float32, (None, 28*28)) # batch of inputs\n",
    "y_ = tf.placeholder(tf.float32, (None, 10)) # batch of corresponding labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this corresponds with the model of a neuron in [ 02-01-notes ]\n",
    "# except this is describing an entire layer, not a single neuron\n",
    "# and we're not including the activation function inside here\n",
    "\n",
    "def fully_connected(x, input_dimension, output_dimension):\n",
    "    raise NotImplementedError"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a really simple neural network with only one fully-connected layer (the output layer) with 10 neurons.\n",
    "\n",
    "See [ 02-04-notes ] for an architecture diagram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# y = ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, $y$ is a 10-dimensional vector, but it's not a probability distribution. We can fix that by applying the softmax function to the logits $y$:\n",
    "\n",
    "$$\\sigma(y)_i = \\frac{e^{y_i}}{\\sum_j e^{y_j}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can define loss as the cross entropy between the true probability distribution (the labels) $p$ and the predicted probability distribution $q$:\n",
    "\n",
    "$$H(p, q) = - \\sum_i p(x) \\log q(x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In TensorFlow, we can do both of these in a single step (also needed for numerical stability):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.5).minimize(loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 60000 training points, so we'll be doing minibatch stochastic gradient descent to train our network (instead of computing gradients over all 60000 data points)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 100\n",
    "ITERATIONS = 1000\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "for i in range(ITERATIONS):\n",
    "    x_batch, y_batch = mnist.train.next_batch(BATCH_SIZE)\n",
    "    sess.run(optimizer, {x: x_batch, y_: y_batch})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's evaluate the accuracy of our network over the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def accuracy(predictions, labels):\n",
    "    return np.mean(np.argmax(predictions, 1) == np.argmax(labels, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = y.eval({x: mnist.test.images})\n",
    "accuracy(predictions, mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep fully-connected network\n",
    "\n",
    "Will adding a ton of parameters help us find a better solution? Let's use a deep fully-connected network using layers with 2000, 1000, and 100 neurons in the hidden layers and then 10 neurons in the output layer. Let's use ReLU activation for all the hidden layers. See [ 02-05-notes ] for an architecture diagram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# y = ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our initial network had ~8,000 parameters. The above network has ~3.5 million parameters, which is over 400x the capacity of the first one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use a fancier optimizer this time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 50\n",
    "ITERATIONS = 5000 # this takes ~ 2 minutes on my laptop\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "for i in range(ITERATIONS):\n",
    "    x_batch, y_batch = mnist.train.next_batch(BATCH_SIZE)\n",
    "    l, _ = sess.run([loss, optimizer], {x: x_batch, y_: y_batch})\n",
    "    if (i+1) % 100 == 0:\n",
    "        print('iteration %d, batch loss %f' % (i+1, np.mean(l)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = y.eval({x: mnist.test.images})\n",
    "accuracy(predictions, mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional neural network\n",
    "\n",
    "Let's design a convolutional neural network to classify MNIST digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_image = tf.reshape(x, (-1, 28, 28, 1)) # turn our 784-dimensional vector into a 28x28 image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a convolutional layer\n",
    "\n",
    "def convolve(x, kernel_height, kernel_width, input_channels, output_channels):\n",
    "    w = tf.Variable(tf.random_normal((kernel_height, kernel_width, input_channels, output_channels)))\n",
    "    b = tf.Variable(tf.random_normal((output_channels,)))\n",
    "    \n",
    "    return tf.nn.conv2d(x, w, strides=(1, 1, 1, 1), padding='SAME')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a 2x2 max pooling layer\n",
    "\n",
    "def pool(x):\n",
    "    return tf.nn.max_pool(x, (1, 2, 2, 1), (1, 2, 2, 1), padding='SAME')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network architecture\n",
    "\n",
    "Let's design an architecture with the following layers:\n",
    "\n",
    "* convolution layer with 25 3x3 filters, relu activation\n",
    "* 2x2 max pooling layer\n",
    "* convolution layer with 50 3x3 filters, relu activation\n",
    "* 2x2 max pooling layer\n",
    "* fully-connected layer with 1000 neurons, relu activation\n",
    "* fully-connected output layer (10 neurons)\n",
    "\n",
    "See [ 02-06-notes ] for an architecture diagram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO\n",
    "# y = ???"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This network has approximately 2.5 million parameters. Note that this is about 1 million parameters _fewer_ than the deep fully-connected network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss = tf.nn.softmax_cross_entropy_with_logits(logits=y, labels=y_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer().minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 50\n",
    "ITERATIONS = 5000 # this takes ~ 3.5 minutes on my laptop\n",
    "\n",
    "sess.run(tf.global_variables_initializer())\n",
    "\n",
    "for i in range(ITERATIONS):\n",
    "    x_batch, y_batch = mnist.train.next_batch(BATCH_SIZE)\n",
    "    l, _ = sess.run([loss, optimizer], {x: x_batch, y_: y_batch})\n",
    "    if (i+1) % 100 == 0:\n",
    "        print('iteration %d, batch loss %f' % (i+1, np.mean(l)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions = y.eval({x: mnist.test.images})\n",
    "accuracy(predictions, mnist.test.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see what some intermediate activations look like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv1_, conv2_ = sess.run([conv1, conv2], {x: mnist.test.images[0:1]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(2, 2)\n",
    "\n",
    "ax[0, 0].imshow(conv1_[0,:,:,0], cmap='gray')\n",
    "ax[0, 1].imshow(conv1_[0,:,:,1], cmap='gray')\n",
    "ax[1, 0].imshow(conv1_[0,:,:,2], cmap='gray')\n",
    "ax[1, 1].imshow(conv1_[0,:,:,3], cmap='gray')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(2, 2)\n",
    "\n",
    "ax[0, 0].imshow(conv2_[0,:,:,0], cmap='gray')\n",
    "ax[0, 1].imshow(conv2_[0,:,:,1], cmap='gray')\n",
    "ax[1, 0].imshow(conv2_[0,:,:,2], cmap='gray')\n",
    "ax[1, 1].imshow(conv2_[0,:,:,3], cmap='gray')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}