COMP6915_a4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "test of COMP6915 a5.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "ruPAwKRBTda9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# **Computer Science 6915 - Winter 2019 Assignment 4**\n",
        "\n",
        "\n",
        "This Assignement is using TensorFlow create a deep neural network model to classify house numbers. For this task, use the images provided in the the Street View House Numbers (SVHN) Dataset (Format 2) available at http://ufldl.stanford.edu/housenumbers/. Use the train data set to train your network and the test dataset to evaluate your network performance.\n",
        "\n",
        "Import all the required packages:"
      ]
    },
    {
      "metadata": {
        "id": "oajubaxz3J1F",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from scipy.ndimage import gaussian_filter\n",
        "from skimage import color\n",
        "from matplotlib import pyplot as plt\n",
        "from scipy.io import loadmat\n",
        "import tensorflow as tf\n",
        "import tensorflow.keras as keras\n",
        "from keras import backend as K, regularizers, optimizers\n",
        "from keras.models import load_model, Sequential\n",
        "from keras.layers import MaxPooling2D, Convolution2D, Activation, Dropout, Flatten, Dense, InputLayer\n",
        "from keras.layers.normalization import BatchNormalization\n",
        "from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img\n",
        "from keras.callbacks import EarlyStopping, ModelCheckpoint\n",
        "from google.colab import files"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "ygzMa84MUnAW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Define all the constants will be used in the network:\n",
        "\n",
        "Image dimensions: height is 32, width is 32, and channel is 3.\n",
        "\n",
        "Model selection: range between 1~3.\n",
        "\n",
        "Number of epochs: 50.\n",
        "\n",
        "Number of class number: 10.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "grtchbsq3QKC",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "img_dims = (32,32,3)\n",
        "model_option = 2\n",
        "all_models = False\n",
        "epochs = 50\n",
        "NUM_MLP_LAYERS = 2\n",
        "train_url = \"http://ufldl.stanford.edu/housenumbers/train_32x32.mat\"\n",
        "test_url = \"http://ufldl.stanford.edu/housenumbers/test_32x32.mat\"\n",
        "optimizer = 'adam'\n",
        "loss=\"sparse_categorical_crossentropy\"\n",
        "metrics=['accuracy']\n",
        "es = keras.callbacks.EarlyStopping(\n",
        "      monitor='val_loss', verbose=0, patience=3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "EjcVJulKU3f0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Functions define:**\n",
        "\n",
        "Models comparison:\n",
        "\n",
        "When all_models parameter is turned on, show all validation accuracy/training accuracy graphy of different models for comparison.\n",
        "\n",
        "Dataset augmentation:\n",
        "\n",
        "\n",
        "We apply gaussian filter to blure the original image as the preprocessing function  and rotation to augment the data. For each orignial image, we generate another two augmented images. Finally, the training data consists of orignal image following with two augmented images if the augment option is set to True."
      ]
    },
    {
      "metadata": {
        "id": "O-3e_4Ia6scb",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "# for plotting multiple models - from https://www.tensorflow.org/tutorials/keras/overfit_and_underfit\n",
        "def plot_history(histories, key='acc'):\n",
        "  plt.figure(figsize=(16,10))\n",
        "    \n",
        "  for historyPair in histories:\n",
        "    name = list(historyPair.keys())[0]\n",
        "    history = historyPair.get(name)\n",
        "    val = plt.plot(history.epoch, history.history['val_'+key],\n",
        "                   '--', label=name.title()+' Val')\n",
        "    plt.plot(history.epoch, history.history[key], color=val[0].get_color(),\n",
        "             label=name.title()+' Train')\n",
        "\n",
        "  plt.xlabel('Epochs')\n",
        "  plt.ylabel(key.replace('_',' ').title())\n",
        "  plt.legend()\n",
        "  plt.xlim([0,max(history.epoch)])\n",
        "  \n",
        "  # for plotting predictions - from https://www.tensorflow.org/tutorials/keras/basic_classification\n",
        "def plot_image(i, predictions_array, true_label, img):\n",
        "  predictions_array, true_label, img = predictions_array[i], true_label[i], img[i]\n",
        "  plt.grid(False)\n",
        "  plt.xticks([])\n",
        "  plt.yticks([])\n",
        "  \n",
        "  plt.imshow(img, cmap=plt.cm.binary)\n",
        "\n",
        "  predicted_label = np.argmax(predictions_array)\n",
        "  if predicted_label == true_label:\n",
        "    color = 'blue'\n",
        "  else:\n",
        "    color = 'red'\n",
        "  \n",
        "  plt.xlabel(\"{} {:2.0f}% ({})\".format((predicted_label+1)%10,\n",
        "                                100*np.max(predictions_array),\n",
        "                                (true_label+1)%10),\n",
        "                                color=color)\n",
        "\n",
        "# for plotting array of predictions - from https://www.tensorflow.org/tutorials/keras/basic_classification\n",
        "def plot_value_array(i, predictions_array, true_label):\n",
        "  predictions_array, true_label = predictions_array[i], true_label[i]\n",
        "  plt.grid(False)\n",
        "  plt.xticks([])\n",
        "  plt.yticks([])\n",
        "  thisplot = plt.bar(range(10), predictions_array, color=\"#777777\")\n",
        "  plt.ylim([0, 1]) \n",
        "  predicted_label = np.argmax(predictions_array)\n",
        " \n",
        "  thisplot[predicted_label].set_color('red')\n",
        "  thisplot[int(true_label)].set_color('blue')\n",
        "\n",
        "# Define gaussian filter to blure the image\n",
        "def imgBlur(image):\n",
        "  result = gaussian_filter(image, sigma=1)\n",
        "  return result\n",
        "\n",
        "augment = ImageDataGenerator(\n",
        "    rotation_range=10,\n",
        "    shear_range=10.0,\n",
        "    brightness_range=[0.9,0.95],\n",
        "    zoom_range=0.1,\n",
        "    preprocessing_function=imgBlur\n",
        ")\n",
        "\n",
        "# From each original image, generate a dataset with original image\n",
        "# followed by two other augmented images\n",
        "def img_augmenting(img):\n",
        "  images = []\n",
        "  height, width, channels = img.shape\n",
        "  img = img.reshape(1, height, width, channels)\n",
        "  images.append(img)\n",
        "  for idx, batch in enumerate(augment.flow(img, batch_size=2)):\n",
        "    images.append(batch)\n",
        "    if idx >= 2: # Only two augmented images will be generated\n",
        "      break\n",
        "  return images\n",
        "\n",
        "# Load data by using url with the option of turn on/off the dataset augmentation\n",
        "def load_url(url, augment):\n",
        "  repo = np.DataSource()\n",
        "  file = repo.open(url)\n",
        "  mat_data = loadmat(file.name) # Load the mat file data\n",
        "  X = np.moveaxis(mat_data['X'], -1, 0) # Get features from dataset\n",
        "  y = mat_data['y'].flatten()-1 # Get class from dataset\n",
        "  \n",
        "  # If augment option is turned on, \n",
        "  # insert augmentation data to original data and class set\n",
        "  if augment:\n",
        "    dataList = []\n",
        "    classList = []\n",
        "    for img_element, class_element in zip(X, y):\n",
        "      aug_images = img_augmenting(img_element)\n",
        "      for aug in aug_images:\n",
        "        dataList.append(aug)\n",
        "        classList.append(class_element)\n",
        "    X = np.vstack(dataList)\n",
        "    y = np.vstack(classList).flatten()\n",
        "  X = np.true_divide(X, 255.0)\n",
        "  return X, y\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VIVHJnZ7EZ_H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Set the dataset format to image channels as the last data dimension, and load trainning and test data by using URL. "
      ]
    },
    {
      "metadata": {
        "id": "nOjMzkdL3Ubj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# enforce image format\n",
        "keras.backend.set_image_data_format('channels_last')\n",
        "# Load datasets with or without image augmentation\n",
        "train_data, train_class = load_url(train_url, True)\n",
        "test_data, test_class = load_url(test_url, False)    # don't augment the test set\n",
        "\n",
        "if len(np.unique(train_class)) != len(np.unique(test_class)):\n",
        "  print(\"The class number between training data and test data are not same\")\n",
        "class_num = len(np.unique(train_class))\n",
        "\n",
        "# Display the data and class set shape for training and test dataset.\n",
        "print(\"train data shape: \", train_data.shape)\n",
        "print(\"train class shape: \", train_class.shape)\n",
        "print(\"test data shape: \", test_data.shape)\n",
        "print(\"test class shape: \", test_class.shape)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "zjqSB6SZISsM",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Display some sample data and it's corresponding class. The value is randomly selected."
      ]
    },
    {
      "metadata": {
        "id": "NblIsmQF3dfG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10,10))\n",
        "for i in range(25):\n",
        "    rnd = np.random.randint(0,train_data.shape[0])\n",
        "    plt.subplot(5, 5, i+1)\n",
        "    plt.xticks([])\n",
        "    plt.yticks([])\n",
        "    plt.grid(False)\n",
        "    plt.imshow(train_data[rnd])\n",
        "    plt.xlabel((train_class[rnd]+1)%10)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "mJb3XohTLLNu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "**Neural Network models:**\n",
        "\n",
        "One multilayer perceptron network and two different structured Covelutional Neural Networks are built to compare the results. \n",
        "\n",
        "Create multilayer perceptron network with two regular densely-connected NN layers between the input and output layer. "
      ]
    },
    {
      "metadata": {
        "id": "Ly8Qc2FH4baN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# MLP\n",
        "mlpModel = keras.models.Sequential()\n",
        "mlpModel.add(keras.layers.Flatten(input_shape=img_dims))\n",
        "for n in range(NUM_MLP_LAYERS):\n",
        "    mlpModel.add(keras.layers.Dense(512, activation='relu'))\n",
        "mlpModel.add(keras.layers.Dense(class_num, activation='softmax'))\n",
        "mlpModel.compile(optimizer=optimizer, loss=loss, metrics=metrics)\n",
        "\n",
        "mlpModel.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "rIaj8RQmMV9O",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Create first Convolutional Neural Network with batch normalization as preprocessing function,  three different size of 2D Convelution layers, and two Densely-connected layers."
      ]
    },
    {
      "metadata": {
        "id": "emZBZnYJAyR8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# cnn1\n",
        "cnnModel = keras.models.Sequential()\n",
        "cnnModel.add(keras.layers.InputLayer(input_shape=img_dims))\n",
        "cnnModel.add(keras.layers.BatchNormalization())\n",
        "\n",
        "# Add three 2D Convolutional layers\n",
        "cnnModel.add(keras.layers.Conv2D(64, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.Conv2D(64, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.Conv2D(64, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.Conv2D(64, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.MaxPooling2D((2, 2)))\n",
        "\n",
        "cnnModel.add(keras.layers.Conv2D(128, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.Conv2D(128, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.Conv2D(128, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.Conv2D(128, kernel_size=(3, 3), padding='same',\n",
        "                 activation='relu'))\n",
        "cnnModel.add(keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "\n",
        "cnnModel.add(keras.layers.Conv2D(128, kernel_size=(5, 5), padding='same',\n",
        "              activation='relu'))\n",
        "cnnModel.add(keras.layers.MaxPooling2D((2, 2)))\n",
        "\n",
        "cnnModel.add(keras.layers.LocallyConnected2D(32, (3, 3), input_shape=img_dims))\n",
        "\n",
        "cnnModel.add(keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
        "\n",
        "cnnModel.add(keras.layers.Flatten())\n",
        "\n",
        "# Add densely-connected NN layers and apply L1 regulation\n",
        "cnnModel.add(keras.layers.Dense(128, activation='relu',\n",
        "               activity_regularizer=keras.regularizers.l1(0.003)))\n",
        "\n",
        "cnnModel.add(keras.layers.Dense(128, activation='relu',\n",
        "               activity_regularizer=keras.regularizers.l1(0.003)))\n",
        "\n",
        "# cnnModel2.add(keras.layers.Dropout(0.3))\n",
        "\n",
        "cnnModel.add(keras.layers.Dense(class_num, activation='softmax'))\n",
        "cnnModel.compile(optimizer=optimizer, loss=loss, metrics=metrics)\n",
        "\n",
        "cnnModel.summary()\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Ac1G2U43F653",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Create second Convolutional Neural Network with four different size of 2D Convolution layers, and two Densely-connected layers. In addition, a dropout is used as network modifying regularization."
      ]
    },
    {
      "metadata": {
        "id": "p158r1TDZVjv",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# cnn2\n",
        "cnnModel2 = keras.models.Sequential()\n",
        "cnnModel2.add(keras.layers.Conv2D(32, (3, 3), padding='same',\n",
        "              activation='relu', input_shape=img_dims))\n",
        "cnnModel2.add(keras.layers.MaxPooling2D((2, 2)))\n",
        "cnnModel2.add(keras.layers.Conv2D(64, (3, 3), padding='same',\n",
        "              activation='relu'))\n",
        "cnnModel2.add(keras.layers.MaxPooling2D((2, 2)))\n",
        "cnnModel2.add(keras.layers.Conv2D(128, (3, 3), padding='same',\n",
        "              activation='relu'))\n",
        "cnnModel2.add(keras.layers.MaxPooling2D((2, 2)))\n",
        "cnnModel2.add(keras.layers.Conv2D(128, (3, 3), padding='same',\n",
        "              activation='relu'))\n",
        "cnnModel2.add(keras.layers.MaxPooling2D((2, 2)))\n",
        "cnnModel2.add(keras.layers.Flatten())\n",
        "cnnModel2.add(keras.layers.Dropout(0.5))\n",
        "cnnModel2.add(keras.layers.Dense(512, activation='relu'))\n",
        "cnnModel2.add(keras.layers.Dense(128, activation='relu'))\n",
        "cnnModel2.add(keras.layers.Dense(class_num, activation='softmax'))\n",
        "cnnModel2.compile(optimizer=optimizer, loss=loss, metrics=metrics)\n",
        "\n",
        "cnnModel2.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2D0qYLrEGcUe",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Train the data with different Neural Network models and show the accuracy diagram with possible train all the models and compare the results."
      ]
    },
    {
      "metadata": {
        "id": "61QmHGRHbnzP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "if not all_models:\n",
        "  if model_option == 1:\n",
        "    model = mlpModel\n",
        "  elif model_option == 2:\n",
        "    model = cnnModel\n",
        "  elif model_option == 3:\n",
        "    model = cnnModel2\n",
        "\n",
        "  modelhistory = model.fit(train_data,train_class,\n",
        "                      epochs=epochs, validation_data=(test_data,test_class),\n",
        "                      batch_size = 512, callbacks=[es])\n",
        "  plt.plot(modelhistory.history['val_acc'])\n",
        "  plt.plot(modelhistory.history['acc'])\n",
        "  plt.legend(['validation accuracy', 'training accuracy'])\n",
        "  plt.title(\"Accuracy\")\n",
        "  pass\n",
        "else:\n",
        "  \n",
        "  model3 = mlpModel\n",
        "  model1 = cnnModel\n",
        "  model2 = cnnModel2\n",
        "  \n",
        "  #model1 run cnn1\n",
        "  model1history = model1.fit(train_data,train_class,\n",
        "                    epochs=epochs, validation_data=(test_data,test_class),\n",
        "                    batch_size = 512, callbacks=[es])\n",
        "  # model2 run cnn2\n",
        "  model2history = model2.fit(train_data,train_class,\n",
        "                      epochs=epochs, validation_data=(test_data,test_class),\n",
        "                      batch_size = 512, callbacks=[es])\n",
        "  \n",
        "  # model 3 run mlp\n",
        "  model3history = model3.fit(train_data,train_class,\n",
        "                      epochs=epochs, validation_data=(test_data,test_class),\n",
        "                      batch_size = 512, callbacks=[es])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GBuTPK2aZfNU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "  # add models to this list to add to history plot\n",
        "  history_list = []\n",
        "  if all_models:\n",
        "    history_list.append({'MLP':model3history})\n",
        "    history_list.append({'cnn1':model1history})\n",
        "    history_list.append({'cnn2':model2history})\n",
        "  else:\n",
        "    history_list.append({'model':modelhistory})\n",
        "  plot_history(history_list)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "LdMkC01ba2lS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next 2 cells show predictions of model2 and graphs of random images showing percentage of predictions"
      ]
    },
    {
      "metadata": {
        "id": "9bugQ1SaaPQL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "predictions = model.predict(test_data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "E-4P4gRcay9N",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Plot the first X test images, their predicted label, and the true label\n",
        "# Color correct predictions in blue, incorrect predictions in red\n",
        "num_rows = 5\n",
        "num_cols = 5\n",
        "num_images = num_rows*num_cols\n",
        "plt.figure(figsize=(2*2*num_cols, 2*num_rows))\n",
        "for i in range(num_images):\n",
        "  j = np.random.randint(0,test_data.shape[0])\n",
        "  plt.subplot(num_rows, 2*num_cols, 2*i+1)\n",
        "  plot_image(j, predictions, test_class, test_data)\n",
        "  plt.subplot(num_rows, 2*num_cols, 2*i+2)\n",
        "  plot_value_array(j, predictions, test_class)\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "o-XKZJ7-HFJo",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model.save(\"CNN_SVHN.h5\")\n",
        "files.download(\"CNN_SVHN.h5\")"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}