mnist_interpretation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "mnist_interpretation.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ezGUEDo1vG4F",
        "colab_type": "text"
      },
      "source": [
        "## In this notebook, we will train a CNN model on the [MNIST](https://en.wikipedia.org/wiki/MNIST_database) dataset and use *activation maximization* to visualize the features that the trained model has learnt. \n",
        "## Before starting this exercise, please go to:\n",
        "### Runtime -> Change runtime type\n",
        "## and make sure that the notebook is set to use *Python 3* and *GPU*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OF5RFj1dwo4e",
        "colab_type": "text"
      },
      "source": [
        "### Step 1 Install and import all dependencies.<br/> "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E2m0dmAw2Eq-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "!pip install --quiet --force-reinstall git+https://github.com/raghakot/keras-vis.git\n",
        "!pip install --quiet --force-reinstall scipy==1.2"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Sek3y7-ENweQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "from vis.utils import utils\n",
        "from vis.visualization import visualize_activation, visualize_cam\n",
        "import keras\n",
        "from keras.datasets import mnist\n",
        "from keras.models import Sequential, Model\n",
        "from keras.layers import Dense, Dropout, Flatten, Activation, Input\n",
        "from keras.layers import Conv2D, MaxPooling2D\n",
        "from keras import backend as K\n",
        "from keras import activations"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wvlnsYYY3cs4",
        "colab_type": "text"
      },
      "source": [
        "### **Step 2** Download the data. For this exercise, we are using the MNIST dataset. This is a dataset of handwritten digits from 0 to 9."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QxhtJqXxNO5S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# input image dimensions\n",
        "img_rows, img_cols = 28, 28\n",
        "\n",
        "\n",
        "# Download the data, shuffled and split between train and test sets\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "if K.image_data_format() == 'channels_first':\n",
        "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
        "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
        "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "# convert class vectors to binary class matrices\n",
        "num_classes = 10\n",
        "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mkeTWL_sIW4z",
        "colab_type": "text"
      },
      "source": [
        "### **Step 3** Visualize a sample in MNIST dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Y50MHSDIkOW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "fig = plt.figure(figsize=(2, 2))\n",
        "plt.imshow(x_train[0][:,:,0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SqgKOmJE4YE4",
        "colab_type": "text"
      },
      "source": [
        "### **Step 4** Architect and compile your deep learning model. This is a multi-layer CNN architecture. The last layer consists of 10 filters, where each filter corresponds to a number from 0 to 9. The model classifies the input image of a handwritten digit into a number from 0 to 9, depending on which filter maximizes the output activation.\n",
        "### Model below has been adapted from [Keras MNIST CNN documentation](https://keras.io/examples/mnist_cnn/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DOyzSFf3OGJm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = Sequential()\n",
        "model.add(Conv2D(32, kernel_size=(3, 3),\n",
        "                 activation='relu',\n",
        "                 input_shape=input_shape))\n",
        "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Dropout(0.25))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(128, activation='relu'))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(num_classes, activation='softmax'))\n",
        "\n",
        "model.compile(loss=keras.losses.categorical_crossentropy,\n",
        "              optimizer=keras.optimizers.Adadelta(),\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fdl0W0qPyFZS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# summary of model's architecture\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KjuarLwhSFap",
        "colab_type": "text"
      },
      "source": [
        "### Step 5 Train your model with the loaded data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bd8WzbPoSL9L",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 128\n",
        "epochs = 2\n",
        "\n",
        "model.fit(x_train, y_train,\n",
        "        batch_size=batch_size,\n",
        "        epochs=epochs,\n",
        "        verbose=1,\n",
        "        validation_data=(x_test, y_test))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ud1WdmQSja4",
        "colab_type": "text"
      },
      "source": [
        "### Step 6 Evaluate model accuracy with test data.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LKiNKACnUcP_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q5lSz0q8UXEU",
        "colab_type": "text"
      },
      "source": [
        "### Step 7 Use *activation maximization* to visualize what the first filter in the last layer of our model has learnt."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c70QhniZUXZg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "fig = plt.figure(figsize=(2, 2))\n",
        "\n",
        "# index of last layer\n",
        "layer_idx = 7\n",
        "\n",
        "# modifying activation of last layer to linear \n",
        "model.layers[layer_idx].activation = activations.linear\n",
        "model = utils.apply_modifications(model)\n",
        "\n",
        "\n",
        "# visualize_activation is a function in keras-vis that runs activation maximization \n",
        "# on your model and outputs an images producing maximum activation for a filter\n",
        "img = visualize_activation(model, layer_idx, filter_indices=0, max_iter=1000, tv_weight=1., lp_norm_weight=0.)\n",
        "plt.imshow(img[:,:,0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-16QwD7GS33F",
        "colab_type": "text"
      },
      "source": [
        "### Step 8 Use activation maximization to visualize the images that maximize output of each filter in the last layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T7vobERBoXBZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "columns = 5\n",
        "rows = 2\n",
        "fig = plt.figure(figsize=(16, 16))\n",
        "\n",
        "for i in range(0,10):\n",
        "  img = visualize_activation(model, layer_idx, filter_indices=[i], max_iter=1000, tv_weight=1., lp_norm_weight=0.)\n",
        "  fig.add_subplot(rows, columns, i+1)\n",
        "  plt.imshow(img[:,:,0])\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bj39GhNCfynK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}