2-Spit_Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split polyps / non-polyps dataset\n",
    "\n",
    "Use `polyps` and `non_polyps` from `cropped` to copy files into `train` and `validation` from `data_polyps`. This folder should already exists and have both subfolders (or ajust the script to create them). The current **data_polyps** folder will be the dataset folder for the next deep learning classifications.\n",
    "\n",
    "Therefore, we shall use a dataset split percentage such as **75% train** and **25% test**.\n",
    "\n",
    "We need `os` and `shutil` to manage the files, `random` to randomly split the dataset in train and validation subsets. You should have a folder structure such as:\n",
    "\n",
    "* `./data_polyps/train/polyps`\n",
    "* `./data_polyps/train/non_polyps`\n",
    "* `./data_polyps/validation/polyps`\n",
    "* `./data_polyps/validation/non_polyps`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random\n",
    "import shutil"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These lines are the locations for the source files (the entire dataset) and the future locations of the splitted dataset in train and validation subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Source dataset: from where to copy the files\n",
    "sourceFolderClass1 = 'cropped/polyps'\n",
    "sourceFolderClass2 = 'cropped/non_polyps'\n",
    "# Destination folders: splitted dataset in train and validation for polyps and non-polyps\n",
    "destFolderClass1_tr  = 'data_polyps/train/polyps'\n",
    "destFolderClass2_tr  = 'data_polyps/train/non_polyps'\n",
    "destFolderClass1_val = 'data_polyps/validation/polyps'\n",
    "destFolderClass2_val = 'data_polyps/validation/non_polyps'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the list with all the files in the source folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Class 1 - polyps: 606\n",
      "Class 2 - non-polyps: 606\n"
     ]
    }
   ],
   "source": [
    "sourceFiles1 = os.listdir(sourceFolderClass1)\n",
    "sourceFiles2 = os.listdir(sourceFolderClass2)\n",
    "print(\"Class 1 - polyps:\", len(sourceFiles1))\n",
    "print(\"Class 2 - non-polyps:\", len(sourceFiles2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's suffle the listw with the source files using a random seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.seed(1)\n",
    "random.shuffle(sourceFiles1)\n",
    "random.shuffle(sourceFiles2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We shall define a number of files to copy in the `validation` subfolder for each class. If you want a different split, you should modify `val_files`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Validation files copied!\n"
     ]
    }
   ],
   "source": [
    "# No of file to copy in VALIDATION folder for each class\n",
    "val_files = 151\n",
    "\n",
    "# Copy the first 151 files for polyps and non-polyps into validation folders\n",
    "print('--> Validation split ...')\n",
    "for i in range(val_files):\n",
    "    # copy validation polyps\n",
    "    File1 = os.path.join(sourceFolderClass1,sourceFiles1[i])\n",
    "    File2 = os.path.join(destFolderClass1_val,  sourceFiles1[i])\n",
    "    shutil.copy(File1,File2)\n",
    "    # copy validation non-polyps\n",
    "    File1 = os.path.join(sourceFolderClass2, sourceFiles2[i])\n",
    "    File2 = os.path.join(destFolderClass2_val,   sourceFiles2[i])\n",
    "    shutil.copy(File1, File2)\n",
    "\n",
    "print('--> Done!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Train files copied!\n"
     ]
    }
   ],
   "source": [
    "# Copy polyps to train\n",
    "print('--> Train split ...')\n",
    "for i in range(val_files,len(sourceFiles1)):\n",
    "    File1 = os.path.join(sourceFolderClass1,  sourceFiles1[i])\n",
    "    File2 = os.path.join(destFolderClass1_tr, sourceFiles1[i])\n",
    "    shutil.copy(File1,File2)\n",
    "# copy non-polyps to train\n",
    "for i in range(val_files,len(sourceFiles2)):    \n",
    "    File1 = os.path.join(sourceFolderClass2,  sourceFiles2[i])\n",
    "    File2 = os.path.join(destFolderClass2_tr, sourceFiles2[i])\n",
    "    shutil.copy(File1, File2)\n",
    "\n",
    "print('--> Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a splitted dataset into train and validation subfolder with each class inside:\n",
    "* **1212** images in the entire dataset;\n",
    "* **910** images for training: 455 polyps + 455 non-polyps;\n",
    "* **302** images for validation: 151 polyps + 151 non-polyps.\n",
    "\n",
    "Let's check the composition of the subsets for the future classification:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--> Dataset: data_polyps\n",
      "> Train - polyps: 455\n",
      "> Train - non-polyps: 455\n",
      "> Validation - polyps: 151\n",
      "> Validation - non-polyps: 151\n"
     ]
    }
   ],
   "source": [
    "print('--> Dataset: data_polyps')\n",
    "print('> Train - polyps:', len(os.listdir(destFolderClass1_tr)))\n",
    "print('> Train - non-polyps:', len(os.listdir(destFolderClass2_tr)))\n",
    "print('> Validation - polyps:', len(os.listdir(destFolderClass1_val)))\n",
    "print('> Validation - non-polyps:', len(os.listdir(destFolderClass2_val)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are ready to use Deep Learning to find a classifier for polyps/non-polyps images. Remember you could modify the dataset splitting, remove manually a specific list of files, use different names for the folders, etc.\n",
    "\n",
    "Let's create some classifiers with the next script [3-Small_CNNs.ipynb](./3-Small_CNNs.ipynb).\n",
    "\n",
    "Have fun with DL! @muntisa"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acknowledgements\n",
    "\n",
    "I gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research ([https://developer.nvidia.com/academic_gpu_seeding](https://developer.nvidia.com/academic_gpu_seeding))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}