lab_exercises/lab3_3_LogisticRegression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Logistic regression\n",
    "\n",
    "Today's second exercise will involve classification using logistic regression with <a href=\"http://spark.apache.org/docs/latest/mllib-guide.html\">MLlib</a>.\n",
    "\n",
    "This exercise will be divided into four parts:\n",
    "+ #### 1. Importing and preparing the data\n",
    "+ #### 2. Logistic regression\n",
    "+ #### 3. Evaluating the results\n",
    "\n",
    "<br>\n",
    "In the following exercises, you will need to replace the code parts in the cell that starts with following comment: \"#Replace the `<INSERT>`\"\n",
    "\n",
    "To go through the notebook, fill in the `<INSERT>`:s with appropriate code in the cells. \n",
    "To run a cell, press Shift-Enter to run it and advance to the following cell or Ctrl-Enter to only run the code in the cell. You should do the exercises from the top to the bottom in this notebook, because following cells may depend on code in previous cells.\n",
    "\n",
    "## Description of the data set\n",
    "In this exercise, we will utilize the <a href=\"https://www.kaggle.com/c/stumbleupon\">StumbleUpon Evergreen Classification Challenge</a> data set:\n",
    "\n",
    ">StumbleUpon is a user-curated web content discovery engine that recommends relevant, high quality pages and media to its users, based on their interests. While some pages we recommend, such as news articles or seasonal recipes, are only relevant for a short period of time, others maintain a timeless quality and can be recommended to users long after they are discovered. In other words, pages can either be classified as \"ephemeral\" or \"evergreen\".\n",
    "\n",
    "We will try to accurately predict a page as either \"ephemeral\" or \"evergreen\", i.e. having long lasting value to the users.\n",
    "\n",
    "First we create a spark context:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark import SparkContext, StorageLevel\n",
    "from pyspark.sql import SQLContext\n",
    "sc = SparkContext(master=\"local[*]\")\n",
    "sqlContext = SQLContext(sc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Helper functions to check results\n",
    "import numpy as np\n",
    "\n",
    "def check(x,y,label):\n",
    "    if(x == y):\n",
    "        print(\"Yay, \"+label+\" is correct!\")\n",
    "    else:\n",
    "        print(\"Nay, \"+label+\" is incorrect, please try again!\")\n",
    "\n",
    "def checkArray(x,y,label):\n",
    "    if np.allclose(x,y):\n",
    "        print(\"Yay, \"+label+\" is correct!\")\n",
    "    else:\n",
    "        print(\"Nay, \"+label+\" is incorrect, please try again!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Importing and preparing the data\n",
    "\n",
    "The data set is currently saved as a text file in the <a href=\"https://en.wikipedia.org/wiki/Tab-separated_values\">tab separated values (TSV)</a> format, without a header, named \"evergreen.tsv\". We want to read in this textfile as separate lines into an rdd. A line is represented by the following fields, specified here: <a href=\"https://www.kaggle.com/c/stumbleupon/data\">data specification</a>.\n",
    "\n",
    "### 1.1 Creating the RDD\n",
    "Read in the CSV-file as an RDD:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rawLines = sc.textFile('evergreen.tsv').cache()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Checking out the RDD\n",
    "\n",
    "Run the following code to get a feel for the data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "numObs = rawLines.count()\n",
    "sampleObs = rawLines.take(1)\n",
    "sampleObs = sampleObs[0]\n",
    "numFeatures = len(sampleObs.split('\\t'))-1\n",
    "print(\"The number of observations: \"+str(numObs))\n",
    "print(\"The number of features: \"+str(numFeatures))\n",
    "print(\"One observation: \"+ str(sampleObs))\n",
    "print(len(sampleObs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Check if the observations are correct\n",
    "check(numObs, 7395, \"the number of observations\")\n",
    "check(numFeatures, 26, \"the number of features\")\n",
    "check(len(sampleObs), 6803, \"the first observation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Checking out the RDD some more\n",
    "Features number 4, 5, 18, and 21 in the eighth observation got missing values represented by the string \"?\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sampleObs = rawLines.take(8)[7]\n",
    "print(sampleObs)\n",
    "sampleObsVector = sampleObs.split('\\t')\n",
    "print(\"\\nThe missing values from features 4, 5, 18, and 21: \"+str([sampleObsVector [i] for i in [3,4,17,20]]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature number 4 seems to be categorical:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feature4 = rawLines.map(lambda x:x.split(\"\\t\")[3].strip(\"\\\"\"))\n",
    "print(feature4.take(2**5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Parsing the vectors\n",
    "This time we will wait with the creation of the LabeledPoints, because the data needs some preprocessing first. Firstly we will remove the three first features: the url of the page, the page ID, and the JSON representing the text of the page. \n",
    "\n",
    "The url may contain some information, i.e. pages with similar addresses, may have similar evergreen qualities, but will skip it for this exercise. The page ID should not contain any information, as it is only used for identifying the page, although it could contain some structure, e.g. if the distribution of evergreen pages is not uniform over time. The JSON-feature we will return to later in this exercise.\n",
    "\n",
    "Implement the function below that parses the lines of the TSV-file and returns a list of unicode tokens corresponding to feature 4 to 27. You will remove the leading and trailing quotes (\") using strip()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "def parseObsVec(line):\n",
    "    \"\"\"Creates a vector of the 4 to 27 features, from a line in the input file.\n",
    "\n",
    "    Args:\n",
    "        line (str)): A line from the input TSV-file.\n",
    "\n",
    "    Returns:\n",
    "        vector: A unicode list of the 4 to 27 features, with the quotation marks removed.\n",
    "    \"\"\"\n",
    "    wholeLine = line.split(\"\\t\")\n",
    "    vector = <INSERT>\n",
    "    return vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parsedVec = rawLines.map(lambda x: parseObsVec(x))\n",
    "exampleVec = parsedVec.take(1)[0]\n",
    "print(exampleVec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the parsing function is correct\n",
    "check(exampleVec, [u'business', u'0.789131', u'2.055555556', u'0.676470588', u'0.205882353', u'0.047058824', \n",
    "                   u'0.023529412', u'0.443783175', u'0', u'0', u'0.09077381', u'0', u'0.245831182', u'0.003883495',\n",
    "                   u'1', u'1', u'24', u'0',u'5424', u'170', u'8', u'0.152941176', u'0.079129575', u'0'], \n",
    "      \"the parsing function\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Creating a list of categories\n",
    "\n",
    "To be able to map the categories to one-hot encoded (OHE) features, we will firstly need the list of the distinct categories. The category is the first element of the parsed vector parsedVec.\n",
    "\n",
    "Create a list of the different categories from this first item in parsedVec:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "#Extract the first element\n",
    "parsedCat= parsedVec.<INSERT>\n",
    "#Collect all the distinct elements\n",
    "listOfCat = parsedCat.<INSERT>\n",
    "print(listOfCat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the list of categories is correct\n",
    "check(listOfCat, [u'recreation', u'business', u'computer_internet', u'culture_politics', u'law_crime', u'health', u'?',\n",
    "                   u'gaming', u'unknown', u'science_technology', u'sports', u'religion', u'weather',\n",
    "                   u'arts_entertainment'], \n",
    "      \"the list of categories\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 Creating a dictionary for the categories\n",
    "\n",
    "After creating the list we need to create a dictionary mapping the categories to indices. \n",
    "\n",
    "As you may have noticed one category is \"?\", this could be imputed by replacing it with the most frequent category or the \"unknown\" category, but because it is a categorical feature we can use it as it is.\n",
    "\n",
    "Create such a dictionary below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "OHEDict ={}\n",
    "for i in range(0, len(listOfCat)):\n",
    "    OHEDict[listOfCat[i]] = i\n",
    "print(OHEDict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the dictionary of categories is correct\n",
    "check(OHEDict, {u'gaming': 7, u'recreation': 0, u'business': 1, u'computer_internet': 2, u'unknown': 8,\n",
    "                u'culture_politics': 3, u'science_technology': 9, u'law_crime': 4, u'sports': 10, u'religion': \n",
    "                11, u'weather': 12, u'health': 5, u'?': 6, u'arts_entertainment': 13}, \n",
    "      \"the dictionary of categories\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.6 Extending the feature vector with one-hot encoded categories\n",
    "We now need to create the OHE features, i.e. extending the feature vector with a vector consisting out of 13 zeros and 1 one.\n",
    "\n",
    "Implement the following function that extends the feature vector with OHE encoded features. You will convert the category, which is the first element of rawVector and append it to end of rawVector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "def oneHotEncoding(rawVector, OHEDict):\n",
    "    \"\"\"Extends the feature vector with binary OHE features using a dictionary.\n",
    "\n",
    "    Args:\n",
    "        rawVector (list of str)): The features corresponding to a single observation.\n",
    "        OHEDict (dict): A mapping of the categories to a unique integer.\n",
    "\n",
    "    Returns:\n",
    "        vector: A unicode list of the 5 to 27 features, extended with 14 binary OHE features.\n",
    "    \"\"\"\n",
    "    catVector = np.zeros(len(OHEDict))\n",
    "    catVector[<INSERT>] = 1\n",
    "    rawVector.<INSERT>\n",
    "    vector = rawVector[1:]\n",
    "    return vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "parsedOHE = parsedVec.map(lambda x:  oneHotEncoding(x,OHEDict))\n",
    "sampleOHE = parsedOHE.take(1)[0]\n",
    "print(sampleOHE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the OHE features is correct\n",
    "check(sampleOHE, [u'0.789131', u'2.055555556', u'0.676470588', u'0.205882353', u'0.047058824', u'0.023529412',\n",
    "                u'0.443783175', u'0', u'0', u'0.09077381', u'0', u'0.245831182', u'0.003883495', u'1', u'1',\n",
    "                u'24', u'0', u'5424', u'170', u'8', u'0.152941176', u'0.079129575', u'0', 0.0, 1.0, 0.0, 0.0,\n",
    "                0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], \n",
    "      \"the OHE features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.7 Impute real valued feature\n",
    "The first feature of the current feature vector is real valued, and has got missing values, represented by \"?\" as seen before. To be able to use the feature we need to somehow impute the missing values. One way of doing this is replacing the missing values with the mean of the non-missing values.\n",
    "\n",
    "Implement the following function that takes the feature vector and replaces the \"?\" from the first feature in the vector with the mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "def imputeMean(vector):\n",
    "    \"\"\"Imputes the missing values from the first feature with the mean\n",
    "\n",
    "    Args:\n",
    "        vector RDD(list of str): The features corresponding to a single observation.\n",
    "\n",
    "    Returns:\n",
    "        vector: A list with the 36 features, with the first feature imputed\n",
    "    \"\"\"\n",
    "    meanVec = vector.map(<INSERT>).filter(<INSERT>)\n",
    "    mean = meanVec.map(lambda x:  float(x)).<INSERT>\n",
    "    vector = vector.map(lambda x: x if x[0] != \"?\" else [mean]+x[1:])\n",
    "    return vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "parsedMean = imputeMean(parsedOHE)\n",
    "sampleMean = parsedMean.take(8)[7]\n",
    "print(sampleMean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the real value imputation is correct\n",
    "check(sampleMean, [0.603334316623788, u'1.883333333', u'0.71969697', u'0.265151515', u'0.113636364', u'0.015151515', \n",
    "                   u'0.49934811', u'0', u'0', u'0.02661597', u'0', u'0.173745927', u'0.025830258', u'?', u'0', u'5',\n",
    "                   u'?', u'27656', u'132', u'4', u'0.068181818', u'0.148550725', u'0', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, \n",
    "                   1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], \n",
    "      \"the mean imputation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.8 Impute binary feature\n",
    "The 14:th and 17:th features of the current feature vector are binary valued, and have got missing values, represented by \"?\" as seen before. To use these features, we need a method to impute the missing values.\n",
    "\n",
    "One way of doing this is to replace the missing values with the mode of the non-missing values, i.e. the most frequent value. Utilize that the mode of a binary feature is its mean rounded to the nearest integer.\n",
    "\n",
    "Implement the following function (that is very similar to the imputeMean function) that takes the feature vector and replaces the \"?\" from the feature with index, in the vector with the mode:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def imputeMode(vector,index):\n",
    "    \"\"\"Imputes the missing values from the feature with index, with the mode (for binary features)\n",
    "\n",
    "    Args:\n",
    "        vector (list of str): The features corresponding to a single observation.\n",
    "        index (integer):\n",
    "\n",
    "    Returns:\n",
    "        vector: A list with the 36 features, with the index feature imputed\n",
    "    \"\"\"\n",
    "    meanVec = vector.map(lambda x:  x[index]).filter(lambda x:  x != \"?\")\n",
    "    mean = meanVec.map(lambda x:  float(x)).mean()\n",
    "    mode = round(mean)\n",
    "    vector = vector.map(lambda x: x if x[index] != \"?\" else x[0:index]+[mode]+x[index+1:])\n",
    "    return vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "parsedMode1 = imputeMode(parsedMean,13)\n",
    "parsedMode2 = imputeMode(parsedMode1,16)\n",
    "sampleMode = parsedMode2.take(8)[7]\n",
    "print(sampleMode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the binary mode imputation is correct\n",
    "check(sampleMode, [0.603334316623788, u'1.883333333', u'0.71969697', u'0.265151515', u'0.113636364', u'0.015151515',\n",
    "                   u'0.49934811', u'0', u'0', u'0.02661597', u'0', u'0.173745927', u'0.025830258', 1.0, u'0', u'5',\n",
    "                   0.0, u'27656', u'132', u'4', u'0.068181818', u'0.148550725', u'0', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,\n",
    "                   1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], \n",
    "      \"the mode imputation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.9 Remove the label from the vector\n",
    "To scale and normalize the features, we need to remove the labels from the feature vectors. The label is the 23:th feature in the vector.\n",
    "\n",
    "Create a list of the labels by removing them from the feature vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labels = parsedMode2.map(lambda x: x[22])\n",
    "features = parsedMode2.map(lambda x: x[0:22]+x[23:])\n",
    "sampleLabel = labels.take(1)[0]\n",
    "sampleFeatures = features.take(1)[0]\n",
    "print(sampleLabel)\n",
    "print(sampleFeatures)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the labels and feature vectors are correct\n",
    "check(sampleLabel, \"0\", \"the labels\")\n",
    "check(sampleFeatures, [u'0.789131', u'2.055555556', u'0.676470588', u'0.205882353', u'0.047058824', u'0.023529412', \n",
    "                   u'0.443783175', u'0', u'0', u'0.09077381', u'0', u'0.245831182', u'0.003883495', u'1', u'1', \n",
    "                   u'24', u'0', u'5424', u'170', u'8', u'0.152941176', u'0.079129575', 0.0, 1.0, 0.0, 0.0, 0.0,\n",
    "                   0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], \n",
    "      \"the features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.10 Scaling and normalizing the observations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Importing some important stuff\n",
    "from pyspark.mllib.feature import StandardScaler\n",
    "from pyspark.mllib.feature import Normalizer\n",
    "from pyspark.mllib.regression import LabeledPoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def scaleNorm(features,labels):\n",
    "    scaler = StandardScaler().fit(features)\n",
    "    scaledFeatures = scaler.transform(features)\n",
    "    norm = Normalizer()\n",
    "    normScaledFeatures = norm.transform(scaledFeatures)\n",
    "    normScaledObs = labels.zip(normScaledFeatures)\n",
    "    normScaledObs = normScaledObs.map(lambda x: LabeledPoint(x[0], x[1]))\n",
    "    return normScaledObs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "normScaledObs = scaleNorm(features,labels)\n",
    "sampleNormScaled = normScaledObs.take(1)[0]\n",
    "print(sampleNormScaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "checkArray(sampleNormScaled.features, [0.487082351628,0.0258987958334,0.361671922814,0.152373563046,0.0532495054536,\n",
    "                                       0.0351843677492,0.0084491817701,0.0,0.0,0.237864093861,0.0,0.508669847859,\n",
    "                                       0.000219746796182,0.0,0.229299339034,0.127813034757,0.0,0.0663708422856,\n",
    "                                       0.10287584824,0.268730211619,0.0906239738681,0.108465470686,0.0,0.335395658357\n",
    "                                       ,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0], \n",
    "      \"the normalized and scaled features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.11 Illustrating the features\n",
    "We will visualize the data set, by projecting it from 36 dimensions to 2. This dimensionality reduction is achieved by using <a href=\"https://en.wikipedia.org/wiki/Principal_component_analysis\">PCA</a>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "#Calculate the PCA of normScaledFeatures\n",
    "Y = TruncatedSVD(n_components=2).fit_transform(features.collect())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.figure(figsize=(15,10))\n",
    "axes = plt.gca()\n",
    "axes.set_xlim([-3000,220000])\n",
    "axes.set_ylim([-3000,5000])\n",
    "plt.scatter(Y[:, 0], Y[:, 1], c=labels.collect(),linewidths=.5, s=25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us zoom in on the large cluster of points:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15,10))\n",
    "axes = plt.gca()\n",
    "axes.set_xlim([-1000,50000])\n",
    "axes.set_ylim([-600,1000])\n",
    "plt.scatter(Y[:, 0], Y[:, 1], c=labels.collect(),linewidths=.5, s=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.12  Splitting the data into training, validation, and test sets \n",
    "Our last task with preparing the data set is to split it into a training, validation and test set. An usual split is 70%/15%/15%:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights = [.7, .15, .15]\n",
    "seed = 0\n",
    "trainObsNum, valObsNum, testObsNum = normScaledObs.randomSplit(weights, seed)\n",
    "\n",
    "trainObsNum.cache()\n",
    "valObsNum.cache()\n",
    "testObsNum.cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numTrain = trainObsNum.count()\n",
    "numVal = valObsNum.count()\n",
    "numTest = testObsNum.count()\n",
    "\n",
    "print(\"Size of train set: \"+str(numTrain))\n",
    "print(\"Size of validation set: \"+str(numVal))\n",
    "print(\"Size of test set: \"+str(numTest))\n",
    "print(\"Size of total data set: \"+str(+numTrain+numVal+numTest))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#Check if the data sets are correct\n",
    "check(numTrain, 5141, \"the train set\")\n",
    "check(numVal, 1111, \"the validation set\")\n",
    "check(numTest, 1143, \"the test set\")\n",
    "check(numTrain+numVal+numTest, 7395, \"the total data set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Logistic regression\n",
    "\n",
    "### 2.1 Logistic regression with limited-memory BFGS\n",
    "MLlib's <a href=\"https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.classification.LogisticRegressionWithLBFGS\">LogisticRegressionWithLBFGS</a> utilizes logistic regression with <a href=\"https://en.wikipedia.org/wiki/Limited-memory_BFGS\">limited-memory Broyden–Fletcher–Goldfarb–Shanno</a> as an optimization algorithm. There exists a SGD variant of logistic regression in MLlib also, but the LBFGS tends to be both faster and use less memory.\n",
    "\n",
    "Create the following logistic regression model with the regParam option set to regPar and the rest the default values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Import even more important stuff\n",
    "from pyspark.mllib.classification import LogisticRegressionWithLBFGS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "regPar = 0.00001\n",
    "modelNum = <INSERT>\n",
    "weights = modelNum.weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the model is correct\n",
    "checkArray(weights, [-0.593982441776,-0.485583947393,0.983432367551,0.243023024636,2.697369678,-0.970078920158,\n",
    "                   -1.15629521729,-0.679875196911,0.0,-2.63052337353,-0.344456478847,0.401163670025,-1.09054556395,\n",
    "                   0.0,0.130032949101,-4.43478759595,-0.879216191225,-1.27835993825,1.4469798125,-0.583934656104,\n",
    "                   -0.0292248086351,-0.869472496836,5.33744180816,4.8052916109,-0.400227340539,1.44351290076,\n",
    "                   0.449742429556,2.51770631199,3.50314642457,0.0629487776013,0.119936170972,1.06368150779,\n",
    "                   -0.828227386113,0.383527091966,-8.34051003105,1.0660670245], \n",
    "      \"weights of the model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Evaluating the logistic regression\n",
    "\n",
    "A common metric to evaluate binary classification is the <a href=\"https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve\">area under the receiver operating characteristic (AUROC)</a>, which is also used as the evaluation metric by <a href=\"https://www.kaggle.com/c/stumbleupon/details/evaluation\">StumbleUpon Evergreen Classification Challenge</a>. A model that assigns random labels to observations will get an AUROC of 0.5, so hopefully we will beat that! ;)\n",
    "\n",
    "You do not need to calculate this yourself, use instead the following MLlib class: <a href=\"https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.evaluation.BinaryClassificationMetrics\">BinaryClassificationMetrics</a>. To get the raw percentages from the predict method, you need to use <a href=\"https://spark.apache.org/docs/latest/api/python/pyspark.mllib.html#pyspark.mllib.classification.LogisticRegressionModel.clearThreshold\">clearThreshold()</a> instead of the binary classification.\n",
    "\n",
    "Calculate the AUROC for the validation data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.mllib.evaluation import BinaryClassificationMetrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Replace the <INSERT>\n",
    "modelNum.<INSERT>\n",
    "predLabel = valObsNum.map(lambda lp:(modelNum.predict(lp.features), lp.label))\n",
    "metrics = <INSERT>\n",
    "AUROC = metrics.<INSERT>\n",
    "print(AUROC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the AUROC on the validation set is correct\n",
    "check(round(AUROC,3), 0.736, \"AUROC of the validation set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Evaluating the results\n",
    "\n",
    "### 3.1 Evaluate the model on the test set\n",
    "Let us try out the model on the test set, and see how they fare:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "predLabel = testObsNum.map(lambda lp:(modelNum.predict(lp.features), lp.label))\n",
    "metrics = BinaryClassificationMetrics(predLabel)\n",
    "AUROC = metrics.areaUnderROC\n",
    "print(AUROC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Check if the numeric model's AUROC on the test set is correct\n",
    "check(round(AUROC,3), 0.710, \"AUROC of the validation set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Post mortem discussion\n",
    "If you check <a href=\"https://www.kaggle.com/c/stumbleupon/leaderboard\">the leaderboard</a> on Kaggle, you can see that the winner had an AUROC of 0.88906, although not directly comparable with our result on the test set, because they used another test set, but still it gives an indication of the quality of the result.\n",
    "\n",
    "A lower bound should be an AUROC of 0.5, because a model that assigns random labels to observations will get it, although 10 participants in the Kaggle challenge did not get better than that.\n",
    "\n",
    "If we would have gotten the same result on their test set, i.e. 0.710 we would have placed on the 540:th place out of 625 participants. We would not have beaten Random Forest Benchmark 0.768, in which they applied random forest to the provided features.\n",
    "\n",
    "But you will improve on this result in this week home assignment, by using the JSON-feature!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}