Module1/01_practical_empty.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Dataflowr](https://raw.githubusercontent.com/dataflowr/website/master/_assets/dataflowr_logo.png)](https://dataflowr.github.io/website/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-SvfKxAGCdhF"
   },
   "source": [
    "# Practical 1: More dogs and cats!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VI1VQlwECdhG"
   },
   "source": [
    "This time, you are going to use the [Oxford-IIIT Pet Dataset](http://www.robots.ox.ac.uk/~vgg/data/pets/) by [O. M. Parkhi et al., 2012](http://www.robots.ox.ac.uk/~vgg/publications/2012/parkhi12a/parkhi12a.pdf) which features 12 cat breeds and 25 dogs breeds. You will need to adapt the code from lesson 1 to this new task, i.e. a classification with 37 categories."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h4dAhx8-CdhM"
   },
   "source": [
    "##  Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6oU4Z_DPCdhM"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import os\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torchvision\n",
    "from torchvision import models,transforms,datasets\n",
    "import time\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "A66_r51xCdhS"
   },
   "outputs": [],
   "source": [
    "torch.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KN3FFTFhHQyi"
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tuej9DPjCdhX"
   },
   "source": [
    "Check if GPU is available and if not change the [runtime](https://jovianlin.io/pytorch-with-gpu-in-google-colab/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "t56d0zbFCdhY"
   },
   "outputs": [],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "print('Using gpu: %s ' % torch.cuda.is_available())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dDLlOjT5Et4p"
   },
   "source": [
    "## Downloading the data\n",
    "\n",
    "The data given on the website [Oxford-IIIT Pet Dataset](http://www.robots.ox.ac.uk/~vgg/data/pets/) is made of two files: `images.tar.gz` and `annotations.tar.gz`. We first need to download and decompress these files.\n",
    "\n",
    "Depending if you use google colab or your own computer, you can adapt the code below to choose where to store the data.\n",
    "\n",
    "To see where you are, you can use the standard unix comands:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qEAr5HNeZCjm"
   },
   "outputs": [],
   "source": [
    "%pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JNQTp-x4ZCjo"
   },
   "source": [
    "If you want to change to a directory to store your data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q5UwzcZKZCjp"
   },
   "outputs": [],
   "source": [
    "%cd #path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ll1oQ5RbZCjr"
   },
   "outputs": [],
   "source": [
    "%pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Rnn5pLK6EyJK"
   },
   "outputs": [],
   "source": [
    "%mkdir data\n",
    "# the line below needs to be adapted if not running on google colab \n",
    "%cd ./data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XkQKDwnfZCjv"
   },
   "source": [
    "Now that you are in the right directory, you can download the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BTobJ9vTE37J"
   },
   "outputs": [],
   "source": [
    "!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/images.tar.gz\n",
    "!wget http://www.robots.ox.ac.uk/~vgg/data/pets/data/annotations.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jEHXmG2vZCjy"
   },
   "source": [
    "and uncompress it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XsMtmnCbCdhd"
   },
   "outputs": [],
   "source": [
    "!tar zxvf images.tar.gz\n",
    "!tar zxvf annotations.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tJXlQ2ojZCj0"
   },
   "source": [
    "Check that everything went correctly!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nUlawYthZCj1"
   },
   "outputs": [],
   "source": [
    "%ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MLio3SAbZCj3"
   },
   "source": [
    "## Warning\n",
    "\n",
    "If you are running this notebook on your own computer, you need to download the data only once. If you want to run this notebook a second time, you can safely skip this section and the section below as your dataset will be stored nicely on your computer.\n",
    "\n",
    "If you are running this notebook on google colab, you need to download the data and to do the data wrangling each time you are running this notebook as data will be cleared once you log off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RPiTo762ZCj3"
   },
   "source": [
    "## 1. Exercise: data wrangling\n",
    "\n",
    "You will first need to do a bit of [data wrangling](https://en.wikipedia.org/wiki/Data_wrangling) to organize your dataset in order to use the PyTorch `dataloader`.\n",
    "\n",
    "If you want to understand how the files are organized, have a look at the `README` file in the folder `annotations`.\n",
    "\n",
    "First, we need to split the dataset in a test set and train/validation set. For this, we can use the files `annotations/test.txt` and `annotations/trainval.txt` containing the names of images contained in the test and train/validation sets of the original paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EqNGCio0ZCj3"
   },
   "outputs": [],
   "source": [
    "!head annotations/test.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tejOC_TZZCj5"
   },
   "outputs": [],
   "source": [
    "!head annotations/trainval.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4O0BRsbmZCj7"
   },
   "source": [
    "Above you see that the authors of the original paper made a partition of the dataset: `./images/Abyssinian_201.jpg` is in the test set while `./images/Abyssinian_100.jpg` is in the train/validation set and so on.\n",
    "\n",
    "BTW, it you wonder what Abyssinian means, it is explained [here](https://en.wikipedia.org/wiki/Abyssinian_cat)\n",
    "\n",
    "We first create two directories where we will put images form the test and trainval sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wn-tDnZXZCj7"
   },
   "outputs": [],
   "source": [
    "%mkdir test\n",
    "%mkdir trainval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5LwIlPrBZCj9"
   },
   "outputs": [],
   "source": [
    "%ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PgorOf_CZCkA"
   },
   "source": [
    "Now it's your turn!\n",
    "\n",
    "All the images are in the `./images/` folder and you want to store the data according to the following structure:\n",
    "```bash\n",
    ".\n",
    "├── test\n",
    "|   └── Abyssinian # contains images of Abyssinian from the test set\n",
    "|   └── Bengal # contains images of Bengal from the test set\n",
    "|    ... \n",
    "|   └── american_bulldog # contains images of american bulldog from the test set\n",
    "|    ...\n",
    "├── trainval\n",
    "|   └── Abyssinian # contains images of Abyssinian from the trainval set\n",
    "|   └── Bengal # contains images of Bengal from the trainval set\n",
    "|    ...\n",
    "|   └── american_bulldog # contains images of american bulldog from the trainval set\n",
    "|    ...\n",
    "```\n",
    "\n",
    "Note that all images wiht a name starting with a majuscule is a cat and all images with a name starting with a minuscule is a dog.\n",
    "\n",
    "So here is one way to achieve your task: you will read the `./annotations/test.txt` file line by line; from each line, you will extract the name of the corresponding file and then copy it from the `./images/filename_##.jpg` to `./test/filename/filename_##.jpg`, where `##` is a number.\n",
    "\n",
    "Then you'll do the same thing for `trainval.txt` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cAjydL_HZCkA"
   },
   "source": [
    "Below is a little piece of code to show you how to open a file and read it line by line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-fiBsTj9ZCkA"
   },
   "outputs": [],
   "source": [
    "with open('./annotations/test.txt') as fp:\n",
    "    line = fp.readline()\n",
    "    while line:\n",
    "        f,_,_,_ = line.split(' ')\n",
    "        print(f)\n",
    "        line = fp.readline()\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PLTktYgZZCkC"
   },
   "source": [
    "In order to remove the `_201` in the example above, you can use the `re` [regular expression lib](https://docs.python.org/3.6/library/re.html) as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "sQ9j71IsZCkD"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "pat = re.compile(r'_\\d')\n",
    "res,_ = pat.split(f)\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1zVpVOpgZCkE"
   },
   "source": [
    "This small piece of code might be useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5YeBT3CbZCkF"
   },
   "outputs": [],
   "source": [
    "# create directory if it does not exist\n",
    "def check_dir(dir_path):\n",
    "    dir_path = dir_path.replace('//','/')\n",
    "    os.makedirs(dir_path, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q0WzlkPUZCkG"
   },
   "source": [
    "Some more hints:\n",
    "- for moving files around you can use the `shutil` lib, see [here](https://docs.python.org/3.6/library/shutil.html#shutil.copy)\n",
    "- you can use `os.path.join`\n",
    "- have a look at python [f-string](https://cito.github.io/blog/f-strings/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "qMHRHKe9ZCkH"
   },
   "outputs": [],
   "source": [
    "import shutil"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7ciu2GLSZCkJ"
   },
   "outputs": [],
   "source": [
    "# Here your code for test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "G4jCU3QSZCkK"
   },
   "outputs": [],
   "source": [
    "# Here your code for train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mkj3DZjpCdha"
   },
   "source": [
    "## Data processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iasXk_FKCdhy"
   },
   "outputs": [],
   "source": [
    "%cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "anZGQpsFZCkP"
   },
   "source": [
    "Now you are ready to redo what we did during lesson 1.\n",
    "\n",
    "Below, you give the path where the data is stored. If you are running this code on your computer, you should modify this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MJRnJgGOCdh4"
   },
   "outputs": [],
   "source": [
    "data_dir = '/content/data/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "U3lE0cvyCdh8"
   },
   "source": [
    "`datasets` is a class of the `torchvision` package (see [torchvision.datasets](http://pytorch.org/docs/master/torchvision/datasets.html)) and deals with data loading. It integrates a multi-threaded loader that fetches images from the disk, groups them in mini-batches and serves them continously to the GPU right after each _forward_/_backward_ pass through the network.\n",
    "\n",
    "Images needs a bit of preparation before passing them throught the network. They need to have all the same size $224\\times 224 \\times 3$ plus some extra formatting done below by the normalize transform (explained later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8t4vokNrF19p"
   },
   "outputs": [],
   "source": [
    "normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])\n",
    "\n",
    "vgg_format = transforms.Compose([\n",
    "                transforms.CenterCrop(224),\n",
    "                transforms.ToTensor(),\n",
    "                normalize,\n",
    "            ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l8LMReVECdh-"
   },
   "outputs": [],
   "source": [
    "dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), vgg_format)\n",
    "         for x in ['train', 'test']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "pMh7kjEBCdiC"
   },
   "outputs": [],
   "source": [
    "os.path.join(data_dir,'train')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q8uKAPGAZCkW"
   },
   "source": [
    "We now have 37 different classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "m9ifn_R7CdiH"
   },
   "outputs": [],
   "source": [
    "dsets['train'].classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TB6sTwFuCdiK"
   },
   "outputs": [],
   "source": [
    "dsets['train'].class_to_idx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WefhjZb2CdiQ"
   },
   "outputs": [],
   "source": [
    "dset_sizes = {x: len(dsets[x]) for x in ['train', 'test']}\n",
    "dset_sizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SCLV1YgaCdiT"
   },
   "outputs": [],
   "source": [
    "dset_classes = dsets['train'].classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Zy52-XhACdiX"
   },
   "source": [
    "The `torchvision` packages allows complex pre-processing/transforms of the input data (_e.g._ normalization, cropping, flipping, jittering). A sequence of transforms can be grouped in a pipeline with the help of the `torchvision.transforms.Compose` function, see [torchvision.transforms](http://pytorch.org/docs/master/torchvision/transforms.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tWnJQiWgGP_R"
   },
   "outputs": [],
   "source": [
    "loader_train = #your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "wN1BKHfDCdig"
   },
   "outputs": [],
   "source": [
    "loader_valid = #your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mQra6Q7CZCkj"
   },
   "source": [
    "Check your dataloader and everything is doing fine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Z4Be7lSLCdik"
   },
   "outputs": [],
   "source": [
    "count = 1\n",
    "for data in loader_valid:\n",
    "    print(count, end=',')\n",
    "    if count == 1:\n",
    "        inputs_try,labels_try = data\n",
    "    count += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3BvxQqfzCdiq"
   },
   "outputs": [],
   "source": [
    "labels_try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MLNsfqc8Cdis"
   },
   "outputs": [],
   "source": [
    "inputs_try.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vsaL21ouKwd9"
   },
   "source": [
    "A small function to display images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "346yl-gbcLLm"
   },
   "outputs": [],
   "source": [
    "def imshow(inp, title=None):\n",
    "#   Imshow for Tensor.\n",
    "    inp = inp.numpy().transpose((1, 2, 0))\n",
    "    mean = np.array([0.485, 0.456, 0.406])\n",
    "    std = np.array([0.229, 0.224, 0.225])\n",
    "    inp = np.clip(std * inp + mean, 0,1)\n",
    "    plt.imshow(inp)\n",
    "    if title is not None:\n",
    "        plt.title(title)\n",
    "    plt.pause(0.001)  # pause a bit so that plots are updated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IJbW-QzGCdiv"
   },
   "outputs": [],
   "source": [
    "# Make a grid from batch\n",
    "out = torchvision.utils.make_grid(inputs_try)\n",
    "\n",
    "imshow(out, title=[dset_classes[x] for x in labels_try])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2HvFNWJICdiz"
   },
   "outputs": [],
   "source": [
    "# Get a batch of training data\n",
    "inputs, classes = next(iter(loader_train))\n",
    "\n",
    "n_images = 8\n",
    "\n",
    "# Make a grid from batch\n",
    "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
    "\n",
    "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LOvkYiROCdi7"
   },
   "source": [
    "## 2. Exercise: modifying VGG Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BO5LAG4bCdi7"
   },
   "source": [
    "The torchvision module comes with a zoo of popular CNN architectures which are already trained on [ImageNet](http://www.image-net.org/) (1.2M training images). When called the first time, if `pretrained=True` the model is fetched over the internet and downloaded to `~/.torch/models`.\n",
    "For next calls, the model will be directly read from there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "K9PsHjXgCdi9"
   },
   "outputs": [],
   "source": [
    "model_vgg = #your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "p6QQhwruCdjI"
   },
   "outputs": [],
   "source": [
    "inputs_try , labels_try = inputs_try.to(device), labels_try.to(device)\n",
    "\n",
    "model_vgg = model_vgg.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "epMB0UF9CdjM"
   },
   "outputs": [],
   "source": [
    "outputs_try = model_vgg(inputs_try)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "dOlx7YcPCdjO"
   },
   "outputs": [],
   "source": [
    "outputs_try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OrweFfK9k0TW"
   },
   "outputs": [],
   "source": [
    "outputs_try.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f3Bktt5PCdjh"
   },
   "source": [
    "### Modifying the last layer and setting the gradient false to all layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "L8kr3-tjCdji"
   },
   "outputs": [],
   "source": [
    "print(model_vgg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IXoMW73MCdjl"
   },
   "source": [
    "We'll learn about what these different blocks do later in the course. For now, it's enough to know that:\n",
    "\n",
    "- Convolution layers are for finding small to medium size patterns in images -- analyzing the images locally\n",
    "- Dense (fully connected) layers are for combining patterns across an image -- analyzing the images globally\n",
    "- Pooling layers downsample -- in order to reduce image size and to improve invariance of learned features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hA2f5FRuCdjm"
   },
   "source": [
    "![vgg16](https://dataflowr.github.io/notebooks/Module1/img/vgg16.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SK6lfAzfCdjn"
   },
   "source": [
    "Here, our goal is to use the already trained model and just change the number of output classes. To this end we replace the last ```nn.Linear``` layer trained for 1000 classes to ones with 37 classes. In order to freeze the weights of the other layers during training, we set the field ```required_grad=False```. In this manner no gradient will be computed for them during backprop and hence no update in the weights. Only the weights for the 37-class layer will be updated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qAr5tj2NKwfF"
   },
   "source": [
    "PyTorch documentation for [LogSoftmax](https://pytorch.org/docs/stable/nn.html#logsoftmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rQwRKKC-Cdjo"
   },
   "outputs": [],
   "source": [
    "for param in model_vgg.parameters():\n",
    "    param.requires_grad = False\n",
    "# your code here\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jJ3OenJpCdjp"
   },
   "outputs": [],
   "source": [
    "print(model_vgg.classifier)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eisaEI7vZClA"
   },
   "source": [
    "Once you modified the architecture of the network, do not forget to put in onto the device!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9ITZFX2MCdju"
   },
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fU9vWWT2CdkN"
   },
   "source": [
    "## Training fully connected module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qqp2u3IXCdkO"
   },
   "source": [
    "### Creating loss function and optimizer\n",
    "\n",
    "PyTorch documentation for [NLLLoss](https://pytorch.org/docs/stable/nn.html#nllloss) and the [torch.optim module](https://pytorch.org/docs/stable/optim.html#module-torch.optim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oP1F4yb8CdkO"
   },
   "outputs": [],
   "source": [
    "criterion = nn.NLLLoss()\n",
    "lr = 0.001\n",
    "optimizer_vgg = torch.optim.SGD(model_vgg.classifier[6].parameters(),lr = lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tenuLj67CdkS"
   },
   "source": [
    "### Training the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7nNAUibjCdkS"
   },
   "outputs": [],
   "source": [
    "def train_model(model,dataloader,size,epochs=1,optimizer=None):\n",
    "    model.train()\n",
    "    \n",
    "    for epoch in range(epochs):\n",
    "        running_loss = 0.0\n",
    "        running_corrects = 0\n",
    "        for inputs,classes in dataloader:\n",
    "            inputs = inputs.to(device)\n",
    "            classes = classes.to(device)\n",
    "            outputs = model(inputs)\n",
    "            loss = criterion(outputs,classes)\n",
    "            optimizer.zero_grad()\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            _,preds = torch.max(outputs.data,1)\n",
    "            # statistics\n",
    "            running_loss += loss.data.item()\n",
    "            running_corrects += torch.sum(preds == classes.data)\n",
    "        epoch_loss = running_loss / size\n",
    "        epoch_acc = running_corrects.data.item() / size\n",
    "        print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
    "                     epoch_loss, epoch_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_Jts2jK1CdkV",
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "train_model(model_vgg,loader_train,size=dset_sizes['train'],epochs=2,optimizer=optimizer_vgg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xkMYIX21Hc1k"
   },
   "outputs": [],
   "source": [
    "def test_model(model,dataloader,size):\n",
    "    model.eval()\n",
    "    predictions = np.zeros(size)\n",
    "    all_classes = np.zeros(size)\n",
    "    all_proba = np.zeros((size,37))\n",
    "    i = 0\n",
    "    running_loss = 0.0\n",
    "    running_corrects = 0\n",
    "    #print(size)\n",
    "    for inputs,classes in dataloader:\n",
    "        inputs = inputs.to(device)\n",
    "        classes = classes.to(device)\n",
    "        outputs = model(inputs)\n",
    "        loss = criterion(outputs,classes)           \n",
    "        _,preds = torch.max(outputs.data,1)\n",
    "            # statistics\n",
    "        running_loss += loss.data.item()\n",
    "        running_corrects += torch.sum(preds == classes.data)\n",
    "        predictions[i:i+len(classes)] = preds.to('cpu').numpy()\n",
    "        all_classes[i:i+len(classes)] = classes.to('cpu').numpy()\n",
    "        all_proba[i:i+len(classes),:] = outputs.data.to('cpu').numpy()\n",
    "        i += len(classes)\n",
    "    epoch_loss = running_loss / size\n",
    "    epoch_acc = running_corrects.data.item() / size\n",
    "    print('Loss: {:.4f} Acc: {:.4f}'.format(\n",
    "                     epoch_loss, epoch_acc))\n",
    "    return predictions, all_proba, all_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "y3JjYayKCdkW"
   },
   "outputs": [],
   "source": [
    "predictions, all_proba, all_classes = test_model(model_vgg,loader_valid,size=dset_sizes['test'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Cn6cIwZNCdkY"
   },
   "outputs": [],
   "source": [
    "# Get a batch of training data\n",
    "inputs, classes = next(iter(loader_valid))\n",
    "\n",
    "out = torchvision.utils.make_grid(inputs[0:n_images])\n",
    "\n",
    "imshow(out, title=[dset_classes[x] for x in classes[0:n_images]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uimC9MvQZClO"
   },
   "source": [
    "Compute the predictions made by your network for `inputs[:n_images]` and the associated probabilities.\n",
    "\n",
    "Hint: use `torch.max` and `torch.exp`.\n",
    "\n",
    "Do not forget to put your inputs on the device!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ghkbm7ByCdke"
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "vals_try, preds_try = # your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8LkHVBLrZClQ"
   },
   "outputs": [],
   "source": [
    "preds_try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "F1qdUtvjCdkg"
   },
   "outputs": [],
   "source": [
    "classes[:n_images]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hJrSi9yiZClS"
   },
   "outputs": [],
   "source": [
    "torch.exp(vals_try)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R0n0hLxTHc1w"
   },
   "source": [
    "## Speeding up the learning by precomputing features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mZegyLN3Hc13"
   },
   "outputs": [],
   "source": [
    "def preconvfeat(dataloader):\n",
    "    conv_features = []\n",
    "    labels_list = []\n",
    "    for data in dataloader:\n",
    "        inputs,labels = data\n",
    "        inputs = inputs.to(device)\n",
    "        labels = labels.to(device)\n",
    "        \n",
    "        x = model_vgg.features(inputs)\n",
    "        conv_features.extend(x.data.cpu().numpy())\n",
    "        labels_list.extend(labels.data.cpu().numpy())\n",
    "    conv_features = np.concatenate([[feat] for feat in conv_features])\n",
    "    return (conv_features,labels_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UQiycgsqHc1_"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "conv_feat_train,labels_train = preconvfeat(loader_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "imSeW6-7Hc2D"
   },
   "outputs": [],
   "source": [
    "conv_feat_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "q3e5bjsuHc2F"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "conv_feat_valid,labels_valid = preconvfeat(loader_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "V0fpZByOHc2H"
   },
   "source": [
    "### Creating a new data generator\n",
    "\n",
    "We will not load images anymore, so we need to build our own data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hpx-uU90Hc2H"
   },
   "outputs": [],
   "source": [
    "dtype=torch.float\n",
    "datasetfeat_train = [[torch.from_numpy(f).type(dtype),torch.tensor(l).type(torch.long)] for (f,l) in zip(conv_feat_train,labels_train)]\n",
    "datasetfeat_train = [(inputs.reshape(-1), classes) for [inputs,classes] in datasetfeat_train]\n",
    "loaderfeat_train = torch.utils.data.DataLoader(datasetfeat_train, batch_size=128, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mhrencIvZClZ"
   },
   "source": [
    "Now you can train for more epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SFNC9PyGHc2P"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "train_model(model_vgg.classifier,dataloader=loaderfeat_train,size=dset_sizes['train'],epochs=80,optimizer=optimizer_vgg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vAV2Ld2TZClc"
   },
   "outputs": [],
   "source": [
    "datasetfeat_valid = [[torch.from_numpy(f).type(dtype),torch.tensor(l).type(torch.long)] for (f,l) in zip(conv_feat_valid,labels_valid)]\n",
    "datasetfeat_valid = [(inputs.reshape(-1), classes) for [inputs,classes] in datasetfeat_valid]\n",
    "loaderfeat_valid = torch.utils.data.DataLoader(datasetfeat_valid, batch_size=128, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4CwELM6KZCld"
   },
   "source": [
    "Now you can compute the accuracy on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2yg770yZHc2W"
   },
   "outputs": [],
   "source": [
    "predictions, all_proba, all_classes = test_model(model_vgg.classifier,dataloader=loaderfeat_valid,size=dset_sizes['test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IFEjbq3DZClf"
   },
   "source": [
    "## Confusion matrix\n",
    "\n",
    "For 37 classes, plotting a confusion matrix is useful to see the performance of the algorithm per class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Usnkk0pNZClf"
   },
   "outputs": [],
   "source": [
    "#!pip install -U scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jUpzljRGZClg"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "import itertools\n",
    "def make_fig_cm(cm):\n",
    "    fig = plt.figure(figsize=(12,12))\n",
    "    plt.imshow(cm, interpolation='nearest', cmap='Blues')\n",
    "    tick_marks = np.arange(37);\n",
    "    plt.xticks(tick_marks, dset_classes, rotation=90);\n",
    "    plt.yticks(tick_marks, dset_classes, rotation=0);\n",
    "    plt.tight_layout();\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        coeff = f'{cm[i, j]}'\n",
    "        plt.text(j, i, coeff, horizontalalignment=\"center\", verticalalignment=\"center\", color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.ylabel('Actual');\n",
    "    plt.xlabel('Predicted');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IY_3JDL9ZClh"
   },
   "outputs": [],
   "source": [
    "cm = confusion_matrix(all_classes,predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "rE8ARkONZCli"
   },
   "outputs": [],
   "source": [
    "make_fig_cm(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wbOU_eNnZClk"
   },
   "source": [
    "Here, you see that [american pit bull terrier](https://en.wikipedia.org/wiki/American_Pit_Bull_Terrier) are often predicted as [staffordshire bull terrier](https://en.wikipedia.org/wiki/Staffordshire_Bull_Terrier) but overall your algorithm should give pretty good results!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BweFfNHSZClk"
   },
   "source": [
    "## 3. Exercise: change of neural network model\n",
    "\n",
    "Great! So far, you reproduced the result of the first lesson. Now, you will need to take another model. I propose to take a `resnet34` from the list availabel [here](https://pytorch.org/docs/stable/torchvision/models.html)\n",
    "\n",
    "__Note__ the main purpose of this exercise is to lean how to modify a network. We do not really care about performances..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nxbW9gOIZClk"
   },
   "outputs": [],
   "source": [
    "model_resnet = # your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "F1ha20kmZCll"
   },
   "outputs": [],
   "source": [
    "print(model_resnet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GgbsUEn6ZClm"
   },
   "source": [
    "OK, the strucutre is quite different from the one of VGG we saw during the course but we still see that the last layer is a Linear layer taking a vector of size 512 and producing a vector of size 1000 (i.e. the number of classes in Imagenet). Hence you should be able to modify this network for your classification task with 37 classes!\n",
    "\n",
    "First freeze the weights of your model and replace the last layer by a Linear layer with the appropriate size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zcSq7M-YZClm"
   },
   "outputs": [],
   "source": [
    "# your code here\n",
    "#Hint print(model_resnet.fc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lYa2iLdqZClo"
   },
   "source": [
    "Now we need to add the LogSoftmax layer. Follow the instructions [here](https://discuss.pytorch.org/t/how-to-add-an-additional-layer-on-top-of-a-pretrained-model/21303/2) to add this layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "X_qqzHFpZClo"
   },
   "outputs": [],
   "source": [
    "model_resnet_lsm = # your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PfpFpvaeZClq"
   },
   "source": [
    "Check that everything is working fine!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nzt3RioOZClq"
   },
   "outputs": [],
   "source": [
    "inputs_try , labels_try = inputs_try.to(device), labels_try.to(device)\n",
    "model_resnet_lsm = model_resnet_lsm.to(device)\n",
    "outputs_try = model_resnet_lsm(inputs_try)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Yk0tmI72ZClr"
   },
   "outputs": [],
   "source": [
    "outputs_try.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ykvg9ZF3ZCls"
   },
   "source": [
    "Now you can start training.\n",
    "\n",
    "You will take the same loss as before: [NLLLoss](https://pytorch.org/docs/stable/nn.html#nllloss) but you need to modify the argument of the optimizer [torch.optim module](https://pytorch.org/docs/stable/optim.html#module-torch.optim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "i3-iINrTZClt"
   },
   "outputs": [],
   "source": [
    "#Hint print(model_resnet_lsm[0].fc)\n",
    "optimizer_resnet = #your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s0zNvKydZClu"
   },
   "source": [
    "Now you can use the same function as above for the training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "drFh4eYuZClu"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "train_model(model_resnet_lsm,loader_train,size=dset_sizes['train'],epochs=30,optimizer=optimizer_resnet)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iEIPGndIZClv"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "predictions, all_proba, all_classes = test_model(model_resnet_lsm,loader_valid,size=dset_sizes['test'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "DlR0KKJsZClw"
   },
   "outputs": [],
   "source": [
    "cm = confusion_matrix(all_classes,predictions)\n",
    "make_fig_cm(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "shC7RYg8ZClx"
   },
   "source": [
    "## Well done!\n",
    "\n",
    "What would you do to speed up the process in the last part?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "caVi9Q06ZCly"
   },
   "source": [
    "[![Dataflowr](https://raw.githubusercontent.com/dataflowr/website/master/_assets/dataflowr_logo.png)](https://dataflowr.github.io/website/)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "include_colab_link": true,
   "name": "01_practical_empty.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}