COMP9318-Specs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COMP-9318 Final Project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions:\n",
    "1. This note book contains instructions for **COMP9318 Final-Project**.\n",
    "\n",
    "* You are required to complete your implementation in a file `submission.py` provided along with this notebook.\n",
    "\n",
    "* You are not allowed to print out unnecessary stuff. We will not consider any output printed out on the screen. All results should be returned in appropriate data structures returned by corresponding functions.\n",
    "\n",
    "* This notebook encompasses all the requisite details regarding the project. Detailed instructions including **CONSTRAINTS**, **FEEDBACK** and **EVALUATION** are provided in respective sections. In case of additional problem, you can post your query @ Piazza.\n",
    "\n",
    "* This project is **time-consuming**, so it is highly advised that you start working on this as early as possible.\n",
    "\n",
    "* You are allowed to use only the permitted libraries and modules (as mentioned in the **CONSTRAINTS** section). You should not import unnecessary modules/libraries, failing to import such modules at test time will lead to errors.\n",
    "\n",
    "* You are **NOT ALLOWED** to use dictionaries and/or external data resources for this project.\n",
    "\n",
    "* We will provide you **LIMITED FEEDBACK** for your submission (only **15** attempts allowed to each group). Instructions for the **FEEDBACK** and final submission are given in the **SUBMISSION** section.\n",
    "\n",
    "* For **Final Evaluation** we will be using a different dataset, so your final scores may vary.  \n",
    "\n",
    "* Submission deadline for this assignment is **23:59:59 on 27-May, 2018**.\n",
    "* **Late Penalty: 10-% on day-1 and 20% on each subsequent day.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction:\n",
    "\n",
    "In this Project, you are required to devise an algorithm/technique to fool a binary classifier named `target-classifier`. In this regard, you only have access to following information:\n",
    "\n",
    "<br>\n",
    "1. The `target-classifier` is a binary classifier classifying data to two categories, $\\textit{i.e.}$, **class-1** and **class-0**.\n",
    "\n",
    "2. You have access to part of classifiers' training data, $\\textit{i.e.}$, a sample of 540 paragraphs. 180 for **class-1**, and 360 for **class-0**, provided in the files: `class-1.txt` and `class-0.txt` respectively.\n",
    "\n",
    "3. The `target-classifier` belong to the SVM family.\n",
    "\n",
    "4. The `target-classifier` allows **EXACTLY 20 DISTINCT** modifications in each test sample.\n",
    "5. You are provided with a test sample of **200** paragraphs from **class-1** (in the file: `test_data.txt`). You can use these test samples to get feedback from the target classifier (**only 15 attempts** allowed to each group.).\n",
    "6. **NOTE: You are not allowed to use the data `test_data.txt` for your model training (if any). VIOLATIONS in this regard will get ZERO score**.\n",
    "\n",
    "<br>\n",
    "### -to-do:\n",
    "* You are required to come up with an algorithm named `fool_classifier()` that makes best use of the above-mentioned information (**point 1-4**) to fool the `target-classifier`. By fooling the classifier we mean that your algorithm can help mis-classify a bunch of test instances (**point-5**) with minimal possible modifications (**EXACTLY 20 DISTINCT** modifications allowed to each test sample). \n",
    "\n",
    "* **NOTE::** We put a **harsh limit** on the number of modifications allowed for each test instance. You are only allowed to modify each test sample by **EXACTLY 20 DISTINCT tokens (NO MORE NO LESS)**.\n",
    "\n",
    "* **NOTE::** **ADDING** or **DELETING** one word at a time is **ONE** modification. Replacement will be considered as **TWO** modifications $(\\textit{i.e.,}$ **Deletion** followed by **Insertion**)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constraints\n",
    "\n",
    "Your implementation `submission.py` should comply with following constraints.\n",
    "\n",
    "1. You should implement your methodology using `Python3`.\n",
    "* You should implement your code in the function `fool_classifier()` in the file `submission.py`. \n",
    "* You are only allowed to use pre-defined class `strategy()` defined in the file: `helper.py` in order to train your models (if any). \n",
    "* You **should not** do any pre-processing on the data. We have already pre-processed the data for you.  \n",
    "* You are supposed to implement your algorithm using **scikit-learn (version=0.19.1)**. We will **NOT** accept implementations using other Libraries.\n",
    "\n",
    "* You are **not supposed to augment** the data using external/additional resources.  You are only allowed to use the partial training data provided to you ($\\textit{i.e.,} $ `class-1.txt` and `class-0.txt`).\n",
    "\n",
    "* You are **not** allowed to use the test samples ($\\textit{i.e.,}$ `test_data.txt`) for model training and/or inference building. You can only use this data for testing, $\\textit{i.e.,}$ calculating success %-age (as described in the **EVALUATION** section.). **VIOLATIONS IN THIS REGARD WILL GET ZERO SCORE**.\n",
    "\n",
    "* You are **not** allowed to hard code the ground truth and any other information into your implementation `submission.py`. \n",
    "\n",
    "* Considering the **RUNNING TIME**, your implementation is supposed to read the test data file ($\\textit{i.e.,}$ `test_data.txt` with 200 test samples), process it and write the modified file (`modified_data.txt`) within **12 Minutes**.\n",
    "\n",
    "* Each modified test sample in the modified file (`modified_data.txt`) should not differ from the original test sample corresponding to the file (`test_data.txt`) by more than 20 tokens.\n",
    "\n",
    "* **NOTE::** Inserting or Deleting a word is **ONE** modification. Replacement will be considered as **TWO** modifications $(\\textit{i.e.,}$ deletion followed by insertion)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission Instructions:\n",
    "\n",
    "* Please read these instructions **VERY CAREFULLY**.\n",
    "\n",
    "### FEEDBACK:\n",
    "* For this project, we will provide real-time feed-back on a test data ($\\textit{i.e.,}$ the file `test_data.txt` containing **200** test cases).\n",
    "* Each group is allowed to avail only **15 attempts in TOTAL**, so use your attempts **WISELY**.\n",
    "* We will only provide **ACCUMULATIVE FEEDBACK** ($\\textit{i.e.,}$ how many modified test samples out of **200** were classified as Class-0). We **WILL NOT** provide detailed feedback for individual test cases.\n",
    "* For the feedback, you are required to submit the modified text file ($\\textit{i.e.,}$ `modified_data.txt`) via the submission portal: http://kg.cse.unsw.edu.au:8318/project/ (using Group name and Group password).\n",
    "* **NOTE::** Please make sure that the modified text file is generated by your program `fool_classifier()`, and it obeys the modification constraints. We have provided a function named: `check_data()` in the class: `strategy()`to check whether the modified file: `modified_data.txt` obeys the constraints.\n",
    "\n",
    "3. Your algorithm should modify each test sample in `test_data.txt` by **EXACTLY 20 DISTINCT TOKENS**.\n",
    "\n",
    "### Final Submission:\n",
    "1. For final submission, you need to submit:\n",
    "    * Your code in the file `submission.py`\n",
    "    * A report (`report.pdf`) outlining your approach for this project.\n",
    "2. We will release the detailed instructions for the final submission submission via Piazza."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation Details\n",
    "\n",
    "1. In the file `submission.py`, you are required to implement a function named: `fool_classifier()` that reads a text file named: `test_data.txt` from Present Working Directory(PWD), and writes out the modified text file: `modified_data.txt` in the same directory.\n",
    "* We have provided the implementation of **strategy** class in a seperate file `helper.py`. You are supposed to use this class for your model training (if any) and inference building.\n",
    "\n",
    "* **Detailed description of input and/or output parts is given below:**\n",
    "\n",
    "### Input: \n",
    "* The function `fool_classifier()` reads a text files named `test_data.txt` having almost (500-1500) test samples. Each line in the input file corresponds to a single test sample.\n",
    "\n",
    "* **Note:** We will also provide the partial training data ($\\textit{(i)}$ `class-0.txt` and $\\textit{(ii)}$ `class-1.txt`) in the test environment. You can  access this data using the class: `strategy()`. \n",
    "\n",
    "### Output:\n",
    "* You are supposed to write down the modified file named `modified_data.txt` in the same directory, and in the same format as that of the `test_data.txt`. In addition, your program is supposed to return the instance of the `strategy` class defined in `helper.py`.\n",
    "\n",
    "\n",
    "* **Note:** Please make sure that the file: `modified_data.txt` is generated by your code, and it follows the **MODIFICATION RESTRICTIONS (ADD** and/or **DELETE EXACTLY 20 DISTINCT TOKENS)**. In case of **ERRORS**, we will **NOT** allow more feedback attempts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We have provided these implementations in the file helper.py, provided along with this project.\n",
    "## Please do not change these functions.\n",
    "###################\n",
    "class countcalls(object):\n",
    "    __instances = {}\n",
    "    def __init__(self, f):\n",
    "        self.__f = f\n",
    "        self.__numcalls = 0\n",
    "        countcalls.__instances[f] = self\n",
    "    def __call__(self, *args, **kwargs):\n",
    "        self.__numcalls += 1\n",
    "        return self.__f(*args, **kwargs)\n",
    "    @staticmethod\n",
    "    def count(f):\n",
    "        return countcalls.__instances[f].__numcalls\n",
    "    @staticmethod\n",
    "    def counts():\n",
    "        res = sum(countcalls.count(f) for f in countcalls.__instances)\n",
    "        for f in countcalls.__instances:\n",
    "            countcalls.__instances[f].__numcalls = 0\n",
    "        return res\n",
    "    \n",
    "## Strategy() class provided in helper.py to facilitate the implementation.\n",
    "class strategy:\n",
    "    ## Read in the required training data...\n",
    "    def __init__(self):\n",
    "        with open('class-0.txt','r') as class0:\n",
    "            class_0=[line.strip().split(' ') for line in class0]\n",
    "        with open('class-1.txt','r') as class1:\n",
    "            class_1=[line.strip().split(' ') for line in class1]\n",
    "        self.class0=class_0\n",
    "        self.class1=class_1\n",
    "    \n",
    "    @countcalls\n",
    "    def train_svm(parameters, x_train, y_train):\n",
    "        ## Populate the parameters...\n",
    "        gamma=parameters['gamma']\n",
    "        C=parameters['C']\n",
    "        kernel=parameters['kernel']\n",
    "        degree=parameters['degree']\n",
    "        coef0=parameters['coef0']\n",
    "        \n",
    "        ## Train the classifier...\n",
    "        clf = svm.SVC(kernel=kernel, C=C, gamma=gamma, degree=degree, coef0=coef0)\n",
    "        assert x_train.shape[0] <=541 and x_train.shape[1] <= 5720\n",
    "        clf.fit(x_train, y_train)\n",
    "        return clf\n",
    "    \n",
    "    ## Function to check the Modification Limits...(You can modify EXACTLY 20-DISTINCT TOKENS)\n",
    "    def check_data(self, original_file, modified_file):\n",
    "        with open(original_file, 'r') as infile:\n",
    "            data=[line.strip().split(' ') for line in infile]\n",
    "        Original={}\n",
    "        for idx in range(len(data)):\n",
    "            Original[idx] = data[idx]\n",
    "\n",
    "        with open(modified_file, 'r') as infile:\n",
    "            data=[line.strip().split(' ') for line in infile]\n",
    "        Modified={}\n",
    "        for idx in range(len(data)):\n",
    "            Modified[idx] = data[idx]\n",
    "\n",
    "        for k in sorted(Original.keys()):\n",
    "            record=set(Original[k])\n",
    "            sample=set(Modified[k])\n",
    "            assert len((set(record)-set(sample)) | (set(sample)-set(record)))==20\n",
    "        return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import helper\n",
    "def fool_classifier(test_data): ## Please do not change the function defination...\n",
    "    ## Read the test data file, i.e., 'test_data.txt' from Present Working Directory...\n",
    "    \n",
    "    \n",
    "    ## You are supposed to use pre-defined class: 'strategy()' in the file `helper.py` for model training (if any),\n",
    "    #  and modifications limit checking\n",
    "    strategy_instance=helper.strategy() \n",
    "    parameters={}\n",
    "    \n",
    "\n",
    "    ##..................................#\n",
    "    #\n",
    "    #\n",
    "    #\n",
    "    ## Your implementation goes here....#\n",
    "    #\n",
    "    #\n",
    "    #\n",
    "    ##..................................#\n",
    "    \n",
    "    \n",
    "    ## Write out the modified file, i.e., 'modified_data.txt' in Present Working Directory...\n",
    "    \n",
    "    \n",
    "    ## You can check that the modified text is within the modification limits.\n",
    "    modified_data='./modified_data.txt'\n",
    "    assert strategy_instance.check_data(test_data, modified_data)\n",
    "    return strategy_instance ## NOTE: You are required to return the instance of this class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " **NOTE:** \n",
    " 1. **You are required to return the instance of the class: `strategy()`, $\\textit{e.g.}$, `strategy_instance` in the above cell.**\n",
    " 2. **You are supposed to write out the file `modified_data.txt` in the same directory, and in the same format as that of `test_data.txt`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How we test your code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Success %-age = 89.5-%\n"
     ]
    }
   ],
   "source": [
    "import helper\n",
    "import submission as submission\n",
    "test_data='./test_data.txt'\n",
    "strategy_instance = submission.fool_classifier(test_data)\n",
    "\n",
    "########\n",
    "#\n",
    "# Testing Script.......\n",
    "#\n",
    "#\n",
    "########\n",
    "\n",
    "print('Success %-age = {}-%'.format(result))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EVALUATION:\n",
    "\n",
    "1. For evaluation, we will consider a bunch of test paragraphs having:\n",
    "    * Approximately 500-1500 test samples for class-1, with each line corresponding to a distinct test sample.The input test file will follow the same format as that of `test_data.txt`.\n",
    "    * We will consider the success rate of your algorithm for final evaluation. By success rate we mean %-age of samples miss-classified by the `target-classifier` ($\\textit{i.e.,}$  instances of `class-1`, classified as `class-0` after `20` distinct modifications). \n",
    "\n",
    "### Example:\n",
    "\n",
    "1. Consider 200 test-samples (classified as **class-1** by the `target-classifier`). \n",
    "2. For-Example, after modifying each test sample by (**20 DISTINCT TOKENS**) the `target-classifier` mis-classifies **100** test samples ($\\textit{i.e.,}$ 100 test samples are classified as **class-0** then your **success %-age** is:\n",
    "\n",
    "3. **success %-age** = (200-100) x 100/200 = **50%**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}