lec8-ml.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapter 8: Introduction to machine learning\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear algebra: Terminology\n",
    "\n",
    "- $\\mathbf{X} \\in \\mathbb{R}^{n \\times m}$ is used to denote a $n$-by-$m$ matrix.\n",
    "- $\\mathbf{x} \\in \\mathbb{R}^n$ is used to denote a $n$-dimensional vector.\n",
    "- $\\mathbf{x}^{(i)}$ is used to denote the $i$-th row of the matrix $\\mathbf{X}$.\n",
    "- $x_j$ is used to denote the $j$-th element of the vector $\\mathbf{x}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## VC-dimension\n",
    "\n",
    "- Two points of two classes on a line can be surely separated by a point.\n",
    "- Three points of two classes on a line are not guaranteed to be separated.\n",
    "- On the 2-D space, 3 points of two classes can be surely separated by a line.\n",
    "- On the 2-D space, 4 points of two classes are not guaranteed to be separated.\n",
    "- ...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Supervised learning\n",
    "\n",
    "- With the input data $\\mathbf{y}$ and  $\\mathbf{X}$,  train and evaluate $f: \\mathbf{X} \\mapsto \\mathbf{y}$\n",
    "    * Regression:  $\\mathbf{y}$ is continuous variable.\n",
    "    * Classification: $\\mathbf{y}$ is categorical variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear regression\n",
    "\n",
    "- $\\mathbf{y} = \\mathbf{X w}$, where $\\mathbf{y} \\in \\mathbb{R}^n$, $\\mathbf{X} \\in \\mathbb{R}^{n \\times m}$\n",
    "- Loss function (squared loss): $L = \\frac{1}{2} (\\mathbf{y - Xw})^T (\\mathbf{y - Xw})$\n",
    "- Normal equation:\n",
    "    * $\\mathbf{w} = (\\mathbf{X^T X})^{-1} \\mathbf{X}^T \\mathbf{y}$\n",
    "    * When $\\mathbf{X}$ is not singular; that is, $\\mathbf{X}^T \\mathbf{X}$ is invertible\n",
    "\n",
    "- Regularization:\n",
    "    * Loss function: $L = \\frac{1}{2}(\\mathbf{y - Xw})^T (\\mathbf{y - Xw}) + \\lambda_1 \\lVert \\mathbf{w} \\rVert_2^2 + \\lambda_2 \\lVert \\mathbf{w} \\rVert_1$\n",
    "    * $\\ell_2$-regularization: ridge regression, $\\lambda_2 = 0$\n",
    "    * $\\ell_1$-regularization: LASSO (least absolute shrinkage and selection operator), $\\lambda_1 = 0$\n",
    "    * $\\ell_1$ + $\\ell_2$-regularization: Elastc net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear classifiers\n",
    "When the data is linearly separable:\n",
    "- Perceptron to Multiple-Layer Perceptron (MLP)\n",
    "- Logistic regression to Softmax classifier\n",
    "- Linear Support Vector Machines (SVM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Perceptron\n",
    "\n",
    "![](images/perceptron1.png)\n",
    "We can use a perceptron rule to classify the instances:\n",
    "$$\n",
    "\\begin{eqnarray}\n",
    "z &=& \\mathbf{w}^T \\mathbf{x}\\\\\n",
    "y &=& g(z) = \\left\\{ \\begin{array}{ll}\n",
    "1 & \\text{if } z \\ge 0\\\\\n",
    "-1 & \\text{otherwise}\n",
    "\\end{array} \\right.\n",
    "\\end{eqnarray}\n",
    "$$\n",
    "\n",
    "This is the illustration of the perceptron:\n",
    "![](images/perceptron2.png)\n",
    "\n",
    "When $\\mathbf{y}$ is binary (dichotomous):\n",
    "\n",
    "1. Initiate $\\mathbf{w} \\in \\mathbb{R}^m$\n",
    "2. Compute $\\hat{\\mathbf{y}} = \\mathbf{Xw}$\n",
    "3. REPEAT\n",
    "    * $\\Delta \\mathbf{w} = \\eta (\\hat{y}^{(i)} - y^{i}) \\mathbf{x}^{(i)}$\n",
    "    * $\\mathbf{w} := \\mathbf{w} - \\Delta \\mathbf{w}$\n",
    "    * Update $\\mathbf{y}$\n",
    "4. UNTIL no update can be done to $\\mathbf{w}$\n",
    "5. Output $\\mathbf{w}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Logistic regression\n",
    "\n",
    "![](images/logistic1.png)\n",
    "Logistic regression can be used to predict the conditional probability $p(y=1 \\big| \\mathbf{x})$:\n",
    "- Odds ratio (OR): $\\frac{p(y=1)}{p(y=0)} = \\frac{p(y=1)}{1-p(y=1)}$\n",
    "- Logit function: $\\log \\frac{p(y=1)}{1-p(y=1)}$\n",
    "- Sigmoid function: $g(z) = \\frac{1}{1+\\exp(-z)}$\n",
    "- Logistic regression:\n",
    "    * $logit(\\mathbf{y}) = \\mathbf{Xw}$\n",
    "    * $p(y=1) = \\frac{1}{1+\\exp(-\\mathbf{w}^T \\mathbf{x})}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Logistic regression: Optimization\n",
    "\n",
    "- $p(y^{(i)} \\big| \\mathbf{x}^{(i)}, \\mathbf{w}) = p(y^{(i)}=1)^{y^{(i)}} p(y^{(i)}=0)^{1-y^{(i)}}$\n",
    "- Likelihood function: $L = \\prod_{i=1}^n p(y^{(i)} \\big| \\mathbf{x}^{(i)}, \\mathbf{w})$\n",
    "- Log-likelihood function: $\\ell(\\mathbf{w}) = \\log L = \\sum_{i=1}^n y^{(i)} \\mathbf{w}^T \\mathbf{x}^{(i)} - \\log(1 + \\exp(\\mathbf{w}^T \\mathbf{x}^{(i)}))$\n",
    "- $\\mathbf{w}^{*} = \\operatorname{argmax}_{\\mathbf{w}} \\ell(\\mathbf{w})$\n",
    "    * $\\ell(\\mathbf{w})$ is a concave function\n",
    "    * Gradient ascent can be used for finding the optimal value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Gradient ascent for logistic regression\n",
    "\n",
    "- Gradient: $\\nabla_{\\mathbf{w}} \\ell(\\mathbf{w}) = \\left[  \\frac{\\partial \\ell}{\\partial w_0}, \\dots,  \\frac{\\partial \\ell}{\\partial w_n}\\right]$\n",
    "- Update rule: $\\Delta \\mathbf{w} =  \\eta \\nabla_{\\mathbf{w}} \\ell(\\mathbf{w})$\n",
    "    * $\\mathbf{w}^{(t+1)} = \\mathbf{w}^{(t)} + \\eta \\frac{\\partial \\ell(\\mathbf{w})}{\\partial \\mathbf{w}} $\n",
    "- Conjugate gradient ascent approach can be much better.\n",
    "- Or we can try stochastic gradient ascent, rather than batch gradient ascent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Support vector machines (SVM)\n",
    "- Margin maximization: \n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\max_{\\mathbf{w, b}} & \\frac{2}{\\lVert  \\mathbf{w} \\rVert} \\\\\n",
    "\\textit{s.t.} & y_i (\\mathbf{w}^T \\mathbf{x}_i + b) \\ge 1\n",
    "\\end{array}\n",
    "$$\n",
    "- Converted to minimization problem:\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\min_{\\mathbf{w}, b} & \\frac{1}{2} \\lVert  \\mathbf{w} \\rVert^2 \\\\\n",
    "\\textit{s.t.} & y_i (\\mathbf{w}^T \\mathbf{x}_i + b) \\ge 1\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lagrangian optimization\n",
    "\n",
    "- Optimization problem of the form:\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\min_{\\mathbf{x}} &  f(\\mathbf{x}) \\\\\n",
    "\\textit{s.t.} & g(\\mathbf{x}) \\le 0 \\\\\n",
    "& h(\\mathbf{x}) = 0\n",
    "\\end{array}\n",
    "$$\n",
    "- Lagrangian function: $L(\\mathbf{x}, \\lambda, \\gamma) = f(\\mathbf{x}) + \\lambda g(\\mathbf{x}) + \\gamma h(\\mathbf{x}), \\lambda \\ge 0$\n",
    "- The original problem now can be converted into:\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\min_{\\mathbf{x}} L(\\mathbf{x}, \\lambda, \\gamma)\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Max-margin problem: Lagrangian method of multiplier (拉格朗日乘子法)\n",
    "- Original form:\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\min_{\\mathbf{w}, b} & \\frac{1}{2} \\lVert  \\mathbf{w} \\rVert^2 \\\\\n",
    "\\textit{s.t.} & y_i (\\mathbf{w}^T \\mathbf{x}_i + b) \\ge 1\n",
    "\\end{array}\n",
    "$$\n",
    "- Lagrangian form:\n",
    "$$\n",
    "L(\\mathbf{w}, b, \\mathbf{\\alpha}) = \\frac{1}{2} \\lVert \\mathbf{w} \\rVert^2 - \\sum_{i=1}^n  \\alpha_i  (y_i (\\mathbf{w}^T \\mathbf{x}_i + b) - 1), \\alpha_i \\ge 0\n",
    "$$\n",
    "- Primal problem (原问题):\n",
    "$$\n",
    "\\min_{\\mathbf{w}, b}\\,\\max_{\\mathbf{\\alpha}}\\; L(\\mathbf{w}, b, \\mathbf{\\alpha})\n",
    "$$\n",
    "- Dual problem (对偶问题):\n",
    "$$\n",
    "\\max_{\\mathbf{\\alpha}} \\,\\min_{\\mathbf{w}, b}\\; L(\\mathbf{w}, b, \\mathbf{\\alpha})\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Dual function of the Lagrangian function\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "L(\\mathbf{w}, b, \\mathbf{\\alpha}) &=& \\frac{1}{2} \\lVert \\mathbf{w} \\rVert^2 - \\sum_{i=1}^n \\alpha_i (y_i (\\mathbf{w}^T \\mathbf{x}_i + b) - 1) \\\\\n",
    "&=& \\frac{1}{2} \\mathbf{w}^T \\mathbf{w} - \\mathbf{w}^T \\sum_{i=1}^n \\alpha_i  y_i  \\mathbf{x}_i - b \\sum_{i=1}^n \\alpha_i y_i + \\sum_{i=1}^n \\alpha_i\n",
    "\\end{array}\n",
    "$$\n",
    "- Compute the partial derivatives with regard to $\\mathbf{w}$ and $b$:\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "\\frac{\\partial L}{\\partial \\mathbf{w}} &=& \\mathbf{w} - \\sum_{i=1}^n \\alpha_i y_i  \\mathbf{x}_i = 0\\\\\n",
    "\\frac{\\partial L}{\\partial b} &=& \\sum_{i=1}^n \\alpha_i y_i  = 0\n",
    "\\end{array}\n",
    "$$\n",
    "- Then we can get:\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "L(\\mathbf{\\alpha}) &=& \\sum_{i=1}^n \\alpha_i - \\frac{1}{2} (\\sum_{i=1}^n \\alpha_i y_i \\mathbf{x}_i)^T \\sum_{i=1}^n \\alpha_i y_i \\mathbf{x}_i \\\\\n",
    "&=& \\sum_{i=1}^n \\alpha_i  - \\frac{1}{2}\\sum_{i,j=1}^n \\alpha_i \\alpha_j y_i y_j \\mathbf{x}_i^T \\mathbf{x}_j\n",
    "\\end{array}\n",
    "$$\n",
    "- Therefore the problem is converted to maximization of $L(\\mathbf{\\alpha})$:\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\max_{\\mathbf{\\alpha}} &  \\sum_{i=1}^n \\alpha_i  - \\frac{1}{2}\\sum_{i=1}^n \\sum_{j=1}^n \\alpha_i \\alpha_j y_i y_j \\mathbf{x}_i^T \\mathbf{x}_j \\\\\n",
    "\\textit{s.t.} & \\sum_{i=1}^n \\alpha_i y_i = 0 \\\\\n",
    "& \\alpha_i \\ge 0, i=1, \\dots, n\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sequential minimal optimization (SMO)\n",
    "\n",
    "If we have multiple Lagrangian multipliers:\n",
    "- Choose two of the multipliers for optimization (say $\\alpha_1, \\alpha_2$), while fix the other multiliers\n",
    "\n",
    "- Iteration equation:\n",
    "$$\n",
    "g(\\mathbf{x}) = \\sum_{i=1}^n y_i \\alpha_i \\kappa(\\mathbf{x}_i, \\mathbf{x}) + b\n",
    "$$\n",
    "- Compute $\\eta$:\n",
    "$$\n",
    "\\eta = \\kappa(\\mathbf{x}_1, \\mathbf{x}_1) + \\kappa(\\mathbf{x}_2, \\mathbf{x}_2) - 2 \\kappa(\\mathbf{x}_1, \\mathbf{x}_2)\n",
    "$$\n",
    "- Compute $E_i$:\n",
    "$$\n",
    "E_i = g(\\mathbf{x}_i) - y_i = \\left( \\sum_{j=1}^n y_j \\alpha_j \\kappa(\\mathbf{x}_j, \\mathbf{x}_i) + b \\right) - y_i, i=1, 2\n",
    "$$\n",
    "- Update $\\alpha_j$:\n",
    "$$\n",
    "\\alpha_j^{\\text{new}} = \\alpha_j^{\\text{old}} + \\frac{y_j (E_i - E_j)}{\\eta}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linearly-separable SVM\n",
    "After getting $\\mathbf{\\alpha}^{*}$, we can compute\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "\\mathbf{w}^{*} &=& \\sum_{i=1}^n \\alpha_i^{*} y_i \\mathbf{x}_i \\\\\n",
    "b^{*}  &=&  y_j - \\sum_{i=1}^n \\alpha_i^{*} y_i \\mathbf{x}_i ^T \\mathbf{x}_j \n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "Then we can get the separating hyperplane:\n",
    "$$\n",
    "\\mathbf{w^{*}}^T \\mathbf{x} + b^{*} = 0\n",
    "$$\n",
    "\n",
    "And the decision function is:\n",
    "$$\n",
    "f(\\mathbf{x}) = \\operatorname{sign} \\left( \\mathbf{w^{*}}^T \\mathbf{x} + b^{*} \\right)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Soft-margin classifier: the introduction of slack variable\n",
    "\n",
    "When the data is not linearly-separable (i.e., there exist outliers), we can introduce the slack variables $\\xi_i \\ge 0$, such that the constaint turn into:\n",
    "$$\n",
    "y_i [\\mathbf{w}^T \\mathbf{x}_i + b] \\ge 1 - \\xi_i\n",
    "$$\n",
    "\n",
    "And the objective function becomes:\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\min_{\\mathbf{w}, b} & \\frac{1}{2} \\lVert \\mathbf{w} \\rVert^2 + C \\sum_{i=1}^n \\xi_i\\\\\n",
    "\\textit{s.t.} & y_i [\\mathbf{w}^T \\mathbf{x}_i + b] \\ge 1 - \\xi_i \\\\\n",
    "& \\xi_i \\ge 0\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Lagrangian function with slack variable\n",
    "\n",
    "- The Lagrangian function:\n",
    "$$\n",
    "L(\\mathbf{w}, b, \\mathbf{\\xi}, \\mathbf{\\alpha}, \\mathbf{\\mu}) = \\frac{1}{2} \\lVert \\mathbf{w} \\rVert^2 + C\\sum_{i=1}^n \\xi_i - \\sum_{i=1}^n \\alpha_i (y_i(\\mathbf{w}^T \\mathbf{x}_i + b - 1 + \\xi_i)) - \\sum_{i=1}^n \\mu_i \\xi_i \n",
    "$$\n",
    "\n",
    "- Compute the partial derivatives with regard to $\\mathbf{w}$, $b$ and $\\mathbf{\\xi}$:\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "\\frac{\\partial L}{\\partial \\mathbf{w}}  = 0 &\\Rightarrow & \\mathbf{w} = \\sum_{i=1}^n \\alpha_i y_i \\mathbf{x}_i\\\\\n",
    "\\frac{\\partial L}{\\partial b} = 0 & \\Rightarrow & \\sum_{i=1}^n \\alpha_i y_i = 0\\\\\n",
    "\\frac{\\partial L}{\\partial \\mathbf{\\xi}} = 0 & \\Rightarrow & C - \\alpha_i - \\mu_i = 0\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lagrangian method of multiplier\n",
    "\n",
    "- Primal problem\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\max_{\\mathbf{\\alpha}} & -\\frac{1}{2} \\sum_{i=1}^n \\sum_{j=1}^n \\alpha_i \\alpha_j y_i y_j \\mathbf{x}_i^T \\mathbf{x}_j + \\sum_{i=1}^n \\alpha_i \\\\\n",
    "\\textit{s.t.} & \\sum_{i=1}^n \\alpha_i y_i = 0 \\\\\n",
    "& C - \\alpha_i - \\mu_i = 0\\\\\n",
    "& \\alpha_i \\ge 0, \\mu_i \\ge 0, i=1, \\dots, n\n",
    "\\end{array}\n",
    "$$\n",
    "- Dual problem:\n",
    "$$\n",
    "\\begin{array}{ll}\n",
    "\\min_{\\mathbf{\\alpha}} & \\frac{1}{2} \\sum_{i=1}^n \\sum_{j=1}^n \\alpha_i \\alpha_j y_i y_j \\mathbf{x}_i^T \\mathbf{x}_j - \\sum_{i=1}^n \\alpha_i \\\\\n",
    "\\textit{s.t.} & \\sum_{i=1}^n \\alpha_i y_i = 0 \\\\\n",
    "& C - \\alpha_i - \\mu_i = 0\\\\\n",
    "& 0 \\le \\alpha_i \\le C, i=1, \\dots, n\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "- Solution:\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "\\mathbf{w}^{*} &=& \\sum_{i=1}^n \\alpha_i y_i \\mathbf{x}_i  \\\\\n",
    "b^{*} &=& \\frac{\\operatorname{max}_{i: y=-1} \\mathbf{w^*}^T \\mathbf{x}_i + \\operatorname{min}_{i: y=+1} \\mathbf{w^*}^T \\mathbf{x}_i }{2}\n",
    "\\end{array}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Nonlinear SVM: Kernel SVM\n",
    "\n",
    "When the data is not linearly separable in the original feature space, we can map them to a higher-dimensional space using the basis function $\\phi(\\cdot)$, to make them separable in the kernel space. We can replace $\\mathbf{x}_i, \\mathbf{x}_j$ in the previous equations with $\\phi(\\mathbf{x}_i), \\phi(\\mathbf{x}_j)$.\n",
    "\n",
    "- Linear kernel function: $\\kappa(\\mathbf{x}_i, \\mathbf{x}_j) = \\mathbf{x}_i^T \\mathbf{x}_j = \\mathbf{x}_i \\cdot \\mathbf{x}_j$\n",
    "- Multinomial kernel function: $\\kappa(\\mathbf{x}_i, \\mathbf{x}_j) = \\left( \\alpha \\lVert \\mathbf{x}_i - \\mathbf{x}_j \\rVert^a + \\gamma \\right)^b$\n",
    "- Gaussian kernel function (RBF): $\\kappa(\\mathbf{x}_i, \\mathbf{x}_j) = \\exp \\left( -\\frac{\\lVert \\mathbf{x}_i - \\mathbf{x}_j \\rVert^2}{2\\sigma^2} \\right)$\n",
    "- Sigmoid kernel function: $\\kappa(\\mathbf{x}_i, \\mathbf{x}_j) = \\operatorname{tanh}(\\gamma \\lVert \\mathbf{x}_i - \\mathbf{x}_j \\rVert^a + \\gamma)$\n",
    "\n",
    "We can compute the kernel matrix $\\mathbf{K}$ thus $k_{ij} = \\kappa(\\mathbf{x}_i, \\mathbf{x}_j)$:\n",
    "- $\\mathbf{K}$ is symmetric and positive semi-definite."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Loss function\n",
    "\n",
    "- Logistic loss function\n",
    "- 0/1 loss function\n",
    "- Hinge loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEACAYAAACj0I2EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VNXWwOHfSqiR3pGSIO0qqAGVjkaxgAqIFQuIFRQu\nWD4bKEXBaxe7gkhARfQiCggiKgREehMQLkUFUYr0KiXJ+v7YSQhxkkySmUzmZL3PM485M3tm1nbC\nmpN11tlHVBVjjDHhLyLUARhjjAkMS+jGGOMRltCNMcYjLKEbY4xHWEI3xhiPsIRujDEe4XdCF5EI\nEVkmIpN9PHa7iPyV8vgyEbkzsGEaY4zJTpEcjO0HrAHKZPL4eFXtm/eQjDHG5IZfe+giUhO4Eng/\nq2EBicgYY0yu+FtyeRV4BMjqtNJrRWSFiHyW8gVgjDEmH2Wb0EXkKmCHqq7A7YX72hOfDMSoaizw\nPTAmoFEaY4zJlmS3louIPAvcBiQCJYHSwERV7Z7J+Ahgj6qW8/GYLRxjjDG5oKrZlrWz3UNX1f6q\nWltVzwC6AjMzJnMRqZZuszPu4Glmr+fZ26BBg0Ieg83P5lfY5lYY5uevnHS5nEJEhgCLVfUroK+I\ndAJOAHuAHrl93XC2adOmUIcQVDa/8OXluYH35+evHCV0VZ0NzE75eVC6+/sD/QMbmjHGmJzI/zNF\nR4/O97fMLz169Ah1CEFl8wtfXp4beH9+/sr2oGhA30xEtVgxmDULWrXKt/c1xphwJiJoIA6KBtzx\n49ClC/z+e76/dbAlJCSEOoSgsvmFXkxMDCJiN4/eYmJi8vT7keuDornWrh18/z107gxz58Jpp+V7\nCMaEq82bN+eo68GEF5G8nXCf/yWX3buheXPYuBH+7//gxRfz7f2NCXcpf3qHOgwTJJl9vv6WXPI/\noavC2rUwdCi88w6UyWytL2NMRpbQvS2vCT0066GfeSZ8/LHnknk41GDzwuZnTMFmF7gwxhRo48aN\no3379rl6buPGjZkzZ06AIyq4QlNyMcbkSkEvudSpU4dRo0ZxySWX5Pt733HHHdSqVYunn346T6+z\nefNm6tSpQ2JiIhER+bvPG54lF1/+/hueeQaOHQt1JMaYQi6v3SahUnASerduMHAg9OwJBXgPJCte\nr8Ha/ExejBw5kvr161OpUiWuueYatm3blvbYjBkz+Ne//kX58uXp3bs3cXFxfPDBBwCMGTOGtm3b\npo198MEHqVq1KuXKlSM2NpY1a9YwcuRIPv74Y1544QXKlClD586dAfcXw8yZMwFITk7m2WefpV69\nepQtW5YLLriAP//8M0dzOH78OA888AA1atSgZs2aPPjgg5w4cQKA3bt307FjR8qXL0/FihW56KKL\n0p73/PPPU7NmTcqUKcOZZ57JrFmzcvc/MRsFJ6EPGABRUTBmDLzySqijMSZ8ifi+5WR8gM2cOZP+\n/fszYcIEtm3bRu3atenatSsAu3bt4oYbbuD5559n9+7dNGzYkPnz52cI0cU0Y8YM5s6dy8aNG9m3\nbx+ffvopFStW5J577uHWW2/l0Ucf5cCBA0yaNOkfMbz88st8+umnTJ8+nf379/PBBx8QFRWVo3kM\nHTqURYsWsXLlSn766ScWLVrE0KFD016/Vq1a7N69m7/++otnn30WgPXr1/PWW2+xdOlSDhw4wDff\nfJPnE4gyU3ASepMmMHas+/mRR2DatNDGkwtxcXGhDiGobH4mt8aNG8ddd93FueeeS9GiRfnPf/7D\nggUL+P333/n6669p3LgxnTt3JiIigr59+1K1alWfr1O0aFEOHjzImjVrUFUaNmyY6diMRo0axbBh\nw6hXrx4AZ599NuXLl8/xPAYNGkTFihWpWLEigwYN4sMPP0yLbdu2bfz2229ERkbSunVrACIjIzl+\n/DirV68mMTGR2rVrU6dOnRy9r78KTkIHuO46GDLElVxuvhnWrQt1RMaEH1Xft5yMD7CtW7cSHR2d\ntn3aaadRoUIF/vzzT7Zu3UqtWrVOGV+zpu+rWF588cX06dOH3r17U61aNXr16sWhQ4f8imHLli2c\nccYZuZ8Ebh61a9dO246Ojmbr1q0APPLII9StW5fLL7+cevXq8fzzzwNQt25dhg8fzuDBg6latSq3\n3HLLKeWmQCpYCR3gqafghhugfXvI8CEXdF6vwdr8TG6dfvrpbN68OW378OHD7N69mxo1alC9enW2\nbNlyyvg//vgj09fq06cPS5Ys4eeff2bdunW8mHK2eXYHMmvVqsUvv/ySh1lAjRo1TpnH5s2bOf30\n0wEoVaoUL730Er/88gtTpkzhlVdeSauVd+3alR9++CHtuY8//nie4shMwUvoIvDhhzB+vKupG2PC\nyvHjxzl27FjaLSkpiVtuuYXRo0ezcuVKjh07Rv/+/WnRogW1a9fmqquuYvXq1UyePJmkpCTefPNN\nduzY4fO1lyxZwqJFi0hMTKRkyZKUKFGCyMhIAKpWrcqvv/6aaVx33303Tz31FBs3bgRg1apV7N27\n1+dYVeXo0aOnzENV6dq1K0OHDmXXrl3s2rWLZ555hm7dugEwderUtC+MUqVKUaRIESIjI1m/fj2z\nZs3i+PHjFCtWjJIlS6bFHHA5uARSBLAMmOzjsWLAeGADMB+onclrqDEm9wr6v6GYmBiNiIjQiIgI\nFRGNiIjQp556SlVV33vvPa1bt65WrFhRO3bsqH/++Wfa87755htt0KCBlitXTnv37q2tWrXSjz76\nSFVV4+PjtW3btqqq+v333+s555yjpUuX1sqVK+ttt92mhw8fVlXVDRs2aGxsrJYvX167dOmiqqp1\n6tTR77//XlVVk5KSdNiwYVqnTh0tU6aMNmvW7JQYUm3atCltDunn8f333+uxY8e0b9++Wr16dT39\n9NP1gQce0GPHjqmq6quvvqoxMTFaqlQprVWrlg4bNkxVVVeuXKnNmjXTMmXKpM1927ZtPv//Zfb5\nptyfbZ72+8QiEXkQOA8oo6qdMjx2H3C2qt4vIjcBXVS1q4/XUH/fzxjzTwX9xKJAUFVq1qzJuHHj\nTmn9Kwzy5cQiEakJXAm8n8mQzsCYlJ8nAO38ed0cOXQIVqwI+MsGktdrsDY/EywzZsxg//79HDt2\njGHDhgHQokWLEEcVfvytob8KPAJktmtQA9gCoKpJwD4RqeBr4NGjOQ0R2LkTWrd2a6lnUSMzxoSn\n+fPnU7duXapUqcLUqVOZNGkSxYsXD3VYYSfbC1yIyFXADlVdISJxgK/d/oz3CZkk/+joHvToEUPJ\nkqSd6ZXa/5u6h/SP7bZtoWZNElauhHbtiPvpJyhTJvPxIdpOva+gxGPz8+b8vGjQoEEMGjQo+4GF\nREJCAvHx8QA5Ogkp2xq6iDwL3AYkAiWB0sBEVe2ebszXwGBVXSgikcA2Va3i47UUlLPOgunTc9iV\neOAAtGjh1lK/6iqYNAmCdaTYmAKqMNTQC7Og19BVtb+q1lbVM4CuwMz0yTzFFOD2lJ9vAGZm9nqN\nGsGaNS43r1yZ3bunU6YMTJ4MFSrA1Knw6KM5eHL+8PIeFNj8jCnoct2HLiJDROTqlM1RQCUR2QA8\nAGTaNf/DD3DhhbB1K7RtCzMzTf0+1KsHEydC0aKwdy8kJ+c2fGOM8ZyQrId+9Ch07w7//a/LzWPG\nuDP9/bZqFTRuHJRFhIwpyKzk4m1huR56iRLuRNAHHoATJ+CWW+Cll3KwhMTZZ1syN8aYDEJ26n9E\nBLz6Krz8stt+5BGX4JOSQhVR3nm9BmvzM6ZgC/laLg89BJ98AsWKweuvQ9euuexV37fP7e4bY0Jm\n7969dOnShVKlSlGnTh0++eSTf4zp2bMn77//Ptu3b6dz587UqFGDiIgIfv/99yxfO/3FKoxvIU/o\n4JL4N99A2bIwYQJcfrk75um3DRugeXPo0yekVzvy+nraNj+Tnfvvv58SJUqwc+dOPvroI+677z7W\nrl17ypjp06dz5ZVXEhERQYcOHZg4cWLYXvKtoCkQCR0gLs51wNSo4f7bpg1k84V90p49sHkzjBgB\nr70WzDCNMZk4cuQIEydOZOjQoZQsWZLWrVvTqVOntAtAgFvhsHz58px++ulUqVKFXr16cf755+f5\nQG9Wl7fzdck6gGnTptGoUSPKlClDrVq1eMUDV0rL9kzR/HT22TB/PnToAD//DC1bugsXnXtuNk9s\n3hzi412rzMMPQ/367uSjfJb+LEMvsvkVfDIkcHu6OihnSXb9+vUUKVKEunXrpt137rnnMmfOnLTt\nadOmcVWA/22mXt7uu+++46yzzuLhhx+ma9euzJ49+5RL1pUuXZp169ZRrlw5wC2nO2HCBFq1asX+\n/fv57bffAhpXKBSYPfRUtWrB3Llw0UUne9W//96PJ3btCoMHu970rl1da6MxJt8cOnSIsmXLnnJf\n2bJlOXjwYNr21KlTufLKKwP6vlld3i6rS9YVK1aMn3/+mYMHD1K2bFliY2MDGldI+LPGbqBu5GAt\n56NHVW+80V0Pq2hR1Y8/9uNJycmqXbu6Jw0d6vd7GRMucvJvKL8tX75cTzvttFPue/nll7VTp06q\nqrp3716tUqWKJicnnzImMTFRRUQ3b96c5evHxMSkrW2eXocOHfTtt98+5b5q1arpvHnzVFX1jTfe\n0PPOO0+rVKmiPXv21IMHD6qq6pIlS7Rz585avnx5jYuL0/nz5+dswkGQ2eeLn+uhF7g99FTFi7vu\nlwcfdM0rt94KL76YzTFPEfjgA/j0UxgwIN9iNcZAgwYNSExMPOUybz/99BONGjUC3BK57dq1C/gB\n0KwubweZX7LuvPPO48svv2Tnzp107tyZG2+8MaBxhUKBTejgetVfecXdwC3f0q9fNr3qJUtCiD4Y\nr/cx2/xMVqKiorj22msZOHAgR44c4ccff2Ty5Ml07+6WfvJVbjl27BhHU/qUUy/5lhV/L2/XsmVL\nateunekl606cOMG4ceM4cOAAkZGRlC5dmiJFCtQhxdzxZzc+UDfy8Ofi+PGqxYq5asp116n+/Xeu\nXypoZs2aFeoQgsrmF3p5+TeUH/bs2aPXXHONnnbaaRodHa3jx49Pe6x69eq6c+fOU8anXt4t/aXe\nMpOby9tldsm648ePa/v27bVChQpatmxZbdasWVqJJpQy+3wJ9CXoAiGvl6BLSIBrroH9+93B0i+/\ndIsv+iUpyZbbNWEvXNdyWbx4Mf/+979ZsGBBqEMp0MJyLZfciotzHTDpe9XTlc4yt2qV64lcsiTY\nIRpjMjFkyJBQh+B5YZXQwS2yuGCB++/ata5X/aefsnnSiBFucKdO8McfQYvN6zVYm5/JrQsuuIAr\nrrgi1GF4XtgldICaNd0e+kUXwbZtfvSqv/zyycGdOrkLThtjjMeEVQ09o2PH3Lrqn33m1lUfPdq1\nN/q0e7e7TNLGjdCxI3zxhdXUTdgJ1xq68U/Qa+giUlxEForIchFZJSL/uJKriNwuIn+JyLKU251+\nzyAPUnvVH3rI9arfdhu88EImveoVK7p1BCpUgClTYMaM/AjRGGPyjT/XFD0GXKyqTYBYoIOINPMx\ndLyqNk25fRDoQDMTEeEqKq++6s4reuyxLHrV69d3e+ZjxrgFYwLM6zVYm58xBZtfNXRVPZLyY3Hc\ngl6+9oFDuv7lAw+4qyAVKwZvvOHOLfr7bx8DL7zQ1WmMMcZj/Kqhi0gEsBSoC7ylqk9kePx24Flg\nJ7AeeEhV/9FOEugaui+zZ0Pnzq5XvXVrmDw5B73qxhRwVkP3trzW0P0611VVk4EmIlIG+FJEzlLV\nNemGTAbGqeoJEekJjAHa+XqtHj16EBMTA5C2PnHqkqWpf/LmdXvu3Dg6dIAff0wgNhZ++CGO6OjA\nvb5t23Yot8Nd48aNefvtt7nwwgtDHUqBlZCQQHx8PEBavvSLP6eTpr8BA3F74Jk9HgHsy+SxHJ8K\nm1tbtqg2buyWCqheXXX58iwGL1qk2r696oEDeXrPcDh1PC9sfqGXn/+GcsPXiojx8fHapk2bEEV0\nkojoL7/8EuowspTZ50ugVlsUkUoiUjbl55LApcD/Moyplm6zM5B+7z0kUnvV4+Jc+/mFF8J33/kY\nmJwM994L06e7ddQTE/M7VGM8ryBcYq4gxBBs/hwUrQ7MEpEVwELgG1WdJiJDROTqlDF9RWS1iCwH\n+gA9ghNuzpQr5/L0TTfBwYOuseWjjzIMioiA//73ZFvjAw/k+rqk4X61m+zY/EwgpL/Y85AhQ7jp\nppu4/fbbKVOmDGeffTbLli1LG7ts2TKaNm1K2bJlufHGG+natSsDBw5Me/yrr76iSZMmlC9fnjZt\n2rAqiwvbaCb/rlWVoUOHEhMTQ7Vq1ejRowcHDhwA3GqQ3bp1o1KlSpQvX57mzZuzc+dOAOLj46lb\nty5lypShbt26Pi+Ine/82Y0P1I0Q/bmYlKT68MOu/AKqzz3nroVxirlzTy7nOHx4SOI0JjvZ/RtK\n/R0PxC03fJVcRo8erW3btvU5ZvDgwVqyZEmdPn26Jicn6xNPPKEtWrRQVdXjx49rdHS0vvHGG5qY\nmKgTJ07UYsWKpa2wuHTpUq1SpYouXrxYk5OTdezYsRoTE6PHjx/3GVtmJZdRo0Zp/fr1ddOmTXr4\n8GG99tprtXv37qrqVnHs1KmTHj16VJOTk3XZsmV68OBBPXz4sJYpU0Y3bNigqqrbt2/XNWvW5O5/\nWjqZfb74WXIpFAk91fDhqiJu1n36qCYmZhgwbpx7MDJSNRe1tnCoweaFzS/0wiGhly5dWsuXL592\ni4qKyjKhX3bZZWmPrVmzRqOiolRVdfbs2VqzZs1TXr9NmzZpCf2+++7TgQMHnvJ4w4YNdc6cOT5j\nyyyht2vXTt9555207XXr1mmxYsU0KSlJP/jgA23durWuXLnylOccPnxYy5cvrxMnTtS/A7iWd14T\neliu5ZJb/fq5ixkVKwZvvgk33JChV/3mm+H5592gM84IWZzG5FYgU3puTZo0iT179qTd3n777SzH\nV6t28hBcVFQUR48eJTk5mW3btqVddShVrVq10n7evHkzL7/8MhUqVKBChQqUL1+eP/74g61bt+Yo\n3q1btxIdHZ22HR0dzYkTJ9ixYwfdunXjiiuuoGvXrtSsWZPHH3+cpKQkoqKi+PTTT3nnnXeoXr06\nHTt2ZN26dTl632AoVAkdXBKfMcPV17/4Ai691C3zkubRR+G663L12l6vwdr8jD80L98G6VSvXp0/\n//zzlPu2bNmS9nOtWrUYMGBA2hfH3r17OXToEDfddFOO3ifjJew2b95M0aJFqVq1KkWKFOGpp57i\n559/Zt68eUyZMoWxY8cCcNlllzFjxgy2b99Ow4YNueeee/Iw28AodAkd3MKLc+e6Tph589wJSJs2\nhToqYwyc/EJo2bIlkZGRvPXWWyQlJTFp0iQWLVqUNu6ee+7h3XffTbvv8OHDTJs2jcOHD2f62ukv\nX3fs2DGSk5O5+eabefXVV9m0aROHDh1iwIABdO3alYiICBISEli9ejXJycmUKlWKokWLEhkZyV9/\n/cWUKVM4cuQIRYsWpVSpUgXiEnaFMqEDNGoE8+e7616sW+fWVV+xIm+v6ZUTPzJj8zPZ8ac1MLsx\nqY8XLVqUiRMn8v7771O+fHnGjRtHx44dKV68OOAu8jxy5Ej69OlDhQoVaNCgAWPGjMnydRs3bkxU\nVBQlS5YkKiqK+Ph47rrrLrp168aFF15I3bp1iYqK4vXXXwdg+/btXH/99ZQtW5ZGjRpx8cUXc9tt\nt5GcnMzLL79MjRo1qFSpEnPmzMm2tJQfwnr53EDYvx+6dIFZs6B0afj8c7jssgyD5syBr75y9fUs\nfhkTEhI8/We7zS/0Cvup/y1atOC+++7j9ttvD3UoQZHXU/8LfUIHt656jx5uca8iReCDD6Bbt5QH\n9+2DmBiX+Z99Fp54IotXMia4CltCnzNnDg0bNqRSpUp89NFH3H///fz6669UrVo11KEFRaG6pmiw\nFC8OH38MjzziThTt3h2eey7lSH+5ci7Di0D//j7OTDLGBMu6des499xzKVeuHK+++iqff/65Z5N5\nINgeegavvQYPPuiS+f33w+uvp1zY6LXX3FmkRYu6008vueQfzw2HP9nzwuYXeoVtD72wsT30AOvX\nz13SrnhxePttuP76lF71fv1cpj9xwi22fvBgqEM1xphT2B56JubMceuq79sHrVq5ddUrlk+Gu+92\ni8PYFcxNCNgeurfZQdEg+vlnt6DXli3QsKGrtORkaWJjAs0SurdZySWIfPWqL1+e+Xiv9zHb/EIv\nOjoaEbGbR2/plyDIDUvo2ahRw62rfsklsH27W1f9229DHZUprDZt2pSrhfFmzZqVrwvx5ffNK/Pb\nlMdT1q3k4qfjx+GOO2DcOB+96tOmuV33AQNCGqMxxpv8LbmEfvGBMFGsGHz4odtjf/FF16v+xx/w\nePetyLXXurOTqlSBArBAjzGmcPLnEnTFRWShiCwXkVUiMsjHmGIiMl5ENojIfBGpHZxwQysiAl54\nwbWkS8p5Rr2HnU7Sa28CkNCrl9tb96hwqDHnhZfn5+W5gffn569sE7qqHgMuVtUmQCzQQUSaZRh2\nF7BHVesDw4EXAh5pAdK3r7tqXfHi8M47cP30u/n70UHu+qQ33ghLl4Y6RGNMIZSjGrqIRAFzgPtU\ndXG6+6cDg1R1oYhEAttVtbKP54dtDd2XH36ATp1cr3rLlsqUWr2p+Nk7EB0N69e7Oo0xxuSRvzV0\nv7pcRCRC3AWgtwPfpk/mKWoAWwBUNQnYJyIVchhz2GnbFn78EWrVgvnzhdYr3uK3K3u79V4smRtj\n8plfB0VVNRloIiJlgC9F5CxVXZNuSMZvDgF87or36NGDmJSzc8qVK0dsbGza+hmpdbBw216wII4O\nHWDlytc4b+f1fBfVhqYFKL5AbQ8fPtwTn1dhnF/6GnNBiMfml/184uPjAdLypT9y3LYoIgOBQ6r6\nSrr7vgYGpyu5bFPVKj6e66mSS3r798MllySwbFkcpUq5ddUvvzzUUQVWQhgsXpUXXp6fl+cG3p+f\nvyWXbBO6iFQCTqjqfhEpCXwDPKeq09KNuR9orKr3i0hX4BpV7erjtTyb0MH1qt95p1uKt0gRGDXK\ntTcaY0xeBLIPvTowRkQicDX3T1V1mogMARar6lfAKOBDEdkA7Ab+kcwLg2LFYOxY16v+wgtw++2u\nV/2Jup8hP6+Gp58OdYjGGC/Lz9Na3dt516xZs9J+fv11VRFVUO0V8a4mEqE6fHjogguA9PPzIi/P\nz8tzU/X+/FJyZ7Y51tZyCZJ///tkr/q7yT25js858sAT7jp3xhgTBLaWS5DNnet61ffuhRbMZ0qR\na6n09Ydw6aWhDs0YEyYCdlA0kApjQgdYuxbat1d+/11owDqmV+5Ond9nQ4kSoQ7NGBMGAnpikfFP\n+l7Y9M480514dO65ynoa0jJ5LsvWhF8yz2x+XuHl+Xl5buD9+fnLEno+Of10mDNHaNcOduwuykUX\nwTffhDoqY4yXWMkln2XsVR85Enr0CHVUxpiCzEouBVRqr/pjj0FiortoxrBhoMmF+4vOGJN3ltAD\nyN86XkQEPPccvPGGW1f9ySfhvobfk3gsKbgB5pHX65Renp+X5wben5+/LKGHUJ8+MGHUfopzlPc2\nXsp1DVdx5LDtqRtjcsdq6AXAj2+toGOf2uylAi1O38yUn6KpVCnUURljCgqroYeR1r1j+fGdVUSz\niQVbo2l15h5+/TXUURljwo0l9ADKSx3vzF4XMf/NZcSynA27KtCyRXKBu5Kd1+uUXp6fl+cG3p+f\nvyyhFyDVe1/L7LfXcGmLQ/y1M4KLLoLp00MdlTEmXFgNvQA6fhzuustdyS4yEt5/33rVjSnMrIYe\nxlJ71R9/HJKSXK/60KFg34XGmKxkm9BFpKaIzBSRNSKySkT6+hhzkYjsE5FlKbcngxNuwRbIOp4I\n/Oc/8Oab7uennoJevdzJSKHi9Tqll+fn5bmB9+fnL3/20BOBh1T1LKAl0FtE/uVj3BxVbZpyGxrQ\nKAux3r3h8x5TKMHfjBgB114LR46EOipjTEGUm4tEfwm8oarfp7vvIuD/VLVjNs+1GnpOJSXBFVcw\n7/sjdIyYyp7k8jRvDlOmQOXKoQ7OGJMfglJDF5EYIBZY6OPhFiKyXESmishZOXldk4XISPjiC1o1\nS+LH5JZEF/2ThQuhdWusV90Ycwq/E7qIlAImAP1U9VCGh5cC0araBHgT+DJwIYaPoNXxSpeGadP4\nV6MizD9xPrEl/8eGDdCyJSxZEpy39MXrdUovz8/LcwPvz89fRfwZJCJFcMn8Q1WdlPHx9AleVb8W\nkbdFpIKq7sk4tkePHsTExABQrlw5YmNjiYuLA05+KOG6vWLFiuC9fsWKJDz9NPTty+wTHbm+yQq+\nnbeYtm1h4sQ4OnQI8/kVgG2vz8+2w2c7ISGB+Ph4gLR86Q+/augiMhbYpaoPZfJ4VVXdkfJzM+Az\nVf1HFFZDD4AtW+DoUY5H1+fuu+HDD11VZuRI195ojPGegF1TVERaA3OAVYCm3PoD0YCq6ggR6Q3c\nB5wA/gYeVNV/1NktoQeWKgwY4NobAZ5+2i3FK9l+7MaYcBKwg6Kq+qOqRqpqrKo2SWlLnK6q76nq\niJQxb6lq45THW/lK5oVB6p9M+UUEnn0W3nrLrbE+cCD07Bm8XvX8nl9+8/L8vDw38P78/GVninrA\n/ffD548vpkQJZeRI6NIFDh8OdVTGmPxma7l4wUsvwSOPMK/903Rc9CR79oj1qhvjIbaWS2HSsiVE\nRdFq+kDmXT6EmBhl4UJo1Qp++SXUwRlj8osl9AAKWR2vdWuYPBmKF6fh+CHMv3wwTZooGze6XL94\ncWDexut1Si/Pz8tzA+/Pz1+W0L2iXTuYOBGKFqXaiKeZ3eU1Lr8cdu6EuDiYNi3UARpjgs1q6F7z\n+efwyCPw7bccr1WXe+5xS/FGRsKIEXDnnaEO0BiTUwHrQw8kS+j55NgxKF4ccL3qTz7p2hsBhgxx\nS/Far7q1NyU+AAAclklEQVQx4cMOioZAganjpSRzcIl72DB45x3Xqz5oUO571QvM/ILEy/Pz8tzA\n+/PzlyX0QqJXL1diL1EC61U3xqOs5FJYDBkCRYow/5IBdOwIu3dDs2auV71KlVAHZ4zJitXQzUkr\nV0JsrCuoP/ss6659gvbtYdMmqFsXpk+HevVCHaQxJjNWQw+BAlvHO+cciI93BfX+/Wn45fPMnw9N\nmrgTj1q18q9XvcDOL0C8PD8vzw28Pz9/WUIvLLp3h9GjXVJ//HGqffgis2dzSq/61KmhDtIYkxdW\ncilsRo+Gu+6C6GhYuZITJUpzzz0wZozrVX/vPfewMabg8Lfk4tcVi4yH3HGHa3Vp1QpKl6YoLsfX\nqgVDh8Ldd8Mff7ileK1X3ZjwYiWXAAqbOt7NN7s99BQi8Mwz8O67rld98GC4995/9qqHzfxyycvz\n8/LcwPvz81e2CV1EaorITBFZIyKrRKRvJuNeF5ENIrJCRGIDH6oJtp494YsvoGRJeP99uOYa61U3\nJpz4cwm6akA1VV0hIqWApUBnVf1fujEdgD6qepWINAdeU9UWPl7LaugFlSpMmgSdOzN/gaT1ql9w\nAXz1lfWqGxNKgbwE3XZVXZHy8yFgLVAjw7DOwNiUMQuBsiJSNcdRm9AZMsSdPtqrFy2bJzNvHtSp\n49oZW7WCjRtDHaAxJjs5qqGLSAwQC2S8ZmgNYEu67T/5Z9L3vLCu47Vo4Q6WjhgBd91Fg7pJzJsH\nTZue7FV/552EUEcZVGH9+WXDy3MD78/PX34n9JRyywSgX8qe+ikP+3iK1VbCSfv2rhE9KsqdhNS9\nO9UqJZKQAFdc4XrVH3zQetWNKcj8alsUkSK4ZP6hqk7yMeQPoFa67ZrAVl+v1aNHD2JiYgAoV64c\nsbGxxMXFASe/ZcN1O/W+ghJPjrcjIuA//yFuwAAYN46ExES47z6mTInj3ntdnu/UKYH33ovj7rsL\nQLz2+fm9HRcXV6DisfllvZ2QkEB8fDxAWr70h18nFonIWGCXqj6UyeNXAr1TDoq2AIbbQdEwtmAB\n9OnjDpLWcJUzVbf07jPPuCGDBrmb9aobE3wBOygqIq2BW4FLRGS5iCwTkfYi0lNE7gVQ1WnAbyKy\nEXgPuD+P8Yel1G/YsNeihTsaWuPkYRARuOSShLRe9SFD4J574MSJEMYZYJ75/Hzw8tzA+/PzV7Yl\nF1X9EYj0Y1yfgERkCoZMdr179oTq1aFrVxg1CrZuhc8+g1Kl8jk+Y8w/2Fouxn/Jya45vXJlFiyA\nq692m+ef7w6WWq+6McFhy+eawFKFhx+G886D9etp0YK0XvUlS6xX3ZiCwBJ6AHm6jnf0KAkzZsCW\nLdCmDSxfToMGMH++y/G//AItW8KiRaEONPe8/Pl5eW7g/fn5yxK68U/JkvDSS6cuoD53LlWrQkIC\ndOgAu3bBxRe7pQKMMfnPaugmZ44dg1tvhc8/d0l++nS48EJOnHAHTEePdl0w777rumCMMXlnNXQT\nHMWLw/jxcOedULs2NGoEQNGiruvlqafcsdN773V96vb9bUz+sYQeQF6v46XNr0gRt77ujz9CxYpp\nj4vA00+7qx5FRLif77orfHrVvfz5eXlu4P35+csSuskdkVOSeXr33gtffukqMqNHQ+fOcCjj6j/G\nmICzGroJrMREdytRgoULXa/6rl2uE2bqVKhqiyobk2NWQzf5T9UdGb3iCti3j+bNXa/6GWfA0qWu\nV33DhlAHaYx3WUIPIK/X8bKd3/bt8M03MGcOtG0Lf/xB/fouqZ9/Pvz6q0vqCzOupl9AePnz8/Lc\nwPvz85cldBM41au77H3mmbB6tTvT6OefqVoVZs2yXnVjgs1q6Cbw9uyBjh1dci9bFr79Fi64gBMn\noFcv+OAD1wXzzjvuAKoxJmtWQzehU6ECfPcdXHstxMTAv/4FuF7199+HgQNdr3rPnu5n+443JjAs\noQeQ1+t4OZpfyZLw3//CzJlQunTa3SJuLfURI9xe+jPPFJxedS9/fl6eG3h/fv6yhG6CJyLC7a37\ncM897oJIqb3qnTpZr7oxeZVtDV1ERgFXAztU9Rwfj18ETAJ+TblroqoOzeS1rIZu4O+/4eBBqFLF\netWN8UMga+ijgSuyGTNHVZum3Hwmc2MAVzy//XZo3hzWrEnrVa9b1/Wqt2wJ69eHOkhjwlO2CV1V\n5wJ7sxlmlwrG+3W8gMzv4EHYvBk2bXLZ+5tvTulV/+0316u+YEHe3yqnvPz5eXlu4P35+StQNfQW\nKReQnioiZwXoNY0XlS3rmtKvvx4OHIArr4RXXqFKZWXWLLe5ezdccglMmRLqYI0JL371oYtINDAl\nkxp6KSBZVY+ISAfgNVVtkMnr6O23305MTAwA5cqVIzY2lri4OODkt6xtF4Lt5GQS7rgDxo4lDmDE\nCBLq1ycpCT75JI5Ro0AkgQcegFdeKQDx2rZt5+N2QkIC8fHxAMTExDBkyBC/auh5Tug+xv4GnKeq\ne3w8ZgdFzakmTIC33oJp01zLC64vfcgQdwN48km3FK9YYc8UUoE+sUjIpE4uIlXT/dwM9yXxj2Re\nGKR+w3pVUOZ3/fWuVz0lmYNL3IMHw8iREBkJQ4e662kEu1fdy5+fl+cG3p+fv4pkN0BExgFxQEUR\n+R0YBBQDVFVHANeLyH3ACeBv4KbghWs8KZNd77vvdsvD3HgjxMfDtm1uh75UqfwNz5hwYWu5mILp\nwAG32MvDD7NoWRGuvtpdm7ppU9erXq1aqAM0Jv/YWi4mvPXoAY8/Du3a0az29rRe9WXLXFuj9aob\n80+W0API63W8fJ3fQw+5esucOdC0KfW2z2XePLjgguD1qnv58/Py3MD78/OXJXRTMLVp43bHL7rI\nFc8vvpgq44Yza6ae0qs+eXKoAzWm4LAauinYEhPhiSfgpZfg/vvhrbdITIT77nNL8UZEuK7HXr1C\nHagxweNvDd0SugkPM2a4vfXixQHXq/700669EWDAALcUr/WqGy+yg6Ih4PU6Xkjnd/nlackcXOIe\nNMjtpUdGwrBhcMcdeetV9/Ln5+W5gffn5y9L6Cas3dVmHZM/O0pUFIwZ4658d/BgqKMyJjSs5GLC\n17590KQJlCzJ4qcmc1W/etarbjzJSi7G+3bvhqgoWLuWC+48m/l9P6FePWXZMrcy77p1oQ7QmPxl\nCT2AvF7HK3Dzq1sXFi92FyU9epS6T93CvLPuptl5iWzaBK1bw/z5/r9cgZtfAHl5buD9+fnLEroJ\nb1FR7sjouHFQujSVJ3/AzMdmcNVVJ3vVv/wy1EEakz+shm68Y+NGd6bRQw+RmOja1keOdL3qb77p\neteNCUfWh24KPVXXmz5okNvu398txWu96ibc2EHREPB6HS/c5icCAwfCqFEQGak8+6xb8yuzXvVw\nm19OeHlu4P35+csSuvG8O89ZwhTtSFTEUcaOhauvtl51401WcjHeN2MG3HILS3bHcJVM4y+tQpMm\nyrRpYr3qJiwErOQiIqNEZIeIrMxizOsiskFEVohIbE6DNSaoLr8cVq/m/CurMk9bUo8NLF8utGyW\naL3qxlP8KbmMBq7I7EER6QDUVdX6QE/g3QDFFna8XscL6/lVqwZffUXd0U8xr0wHmkUuYdOWIrRq\nBfPmuSFhPb9seHlu4P35+SvbhK6qc4G9WQzpDIxNGbsQKJv+wtHGFBgi0KMHldfOYeYXB7j6atiz\nB9q1s1514w1+1dBFJBqYoqrn+HhsCvAfVZ2Xsv0d8KiqLvMxVpu+1zTvURsTAJoUye/jHmf33GtB\nknhleCIP9i2e/RONyWf+1tCLBOK9fNyX6bfEsreXQbmUjRJANaBOyvZvKf+1bdvOr+2610HRJ2HW\nMzzU7weWvLGIjz67HGkSm/ZnfFxcHIBt23a+bSckJBAfHw9ATEwM/grEHvq7wCxV/TRl+3/ARaq6\nw8dYXfLnEr+DCzdL5i3h/FbnhzqMoPHq/HpP683Cr85EJt+Gaju6M5aRfVdRbNggKFUq1OEFREJC\nQlri8CKvzy/Qe+iC7z1xgMlAb+BTEWkB7POVzFOdd/p5fr5l+DlY6aDNLwyVK1EOmsTTI6Ypn73d\nhrEnurPt9Rl8PuF8Sr/1HFxzTahDNMYv2e6hi8g4IA6oCOwABgHFAFXVESlj3gTaA4eBO3zVz1PG\nWR+6KXA6fNyB6RunM+2WaVTe34GrrjjBX3uKEstypnEl1Se9B506hTpMU4gFbA9dVW/xY0wffwMz\npqCJENfslazJnH8+zF9clPbtlRUbmtCyxHKm163Mv0IcozH+sFP/A8jrvbBenV9qQv9pwU8AnHEG\n/Pij0Lw5bD5ajdYXRvLjj6GMMO+8+tml8vr8/GUJ3RR6aXvoJKfdV7kyzJzprlG6Zw9ceil88UW6\nJ02eDMuX53OkxmTNEnoAefkoO3h3fpJyvL9xs8an3B8VBRMnQs+ecPQoXHcdvPUWsHcv3HknnHee\ne3BHpj0ABYZXP7tUXp+fvyyhm0IvfQ09oyJF4J133DrqqtCnDzzxTBTarTtERsKIEVCvHgwZAocO\n5XfoxpzCEnoAeb2O59X5pSb0VQtX+XxcBAYMgNGjXQ5/7tXi3L77FY4vWem6Xw4dgsGDoVu3fIw6\nZ7z62aXy+vz8ZQndFHqScgmj7Fpqe/SAr76C006DDz+Eq/7vTA58OAnmzIEWLeDRR/MhWmMyZ+uh\nm0Lvpgk38dnPnzH+uvHc1PimbMcvXQpXXgl//QXnngvTpsHp1dWubWeCxi5BZ4yfsqqh+3LeeTB/\nPtSvDz/9BC1bwtr/ZfJvbds21yIzc6YrwhsTRJbQA8jrdTyvzi81oa9etNrv55xxhltHvXlz+P13\naN0a373qw4fD99+7NXovvhhmzw5Q1Dnj1c8uldfn5y9L6KbQk0yXKcpapUpux7tTJ9fJ+I9edYAn\nn4Rhw6B8eZfM4+Jccv/55zzHbUxGltADyOu9sF6dX+oeesPzG+b4uVFR8Pnn0KvXyV71N99MN6B0\naejfH377zbU2li0Ls2a5dpl85NXPLpXX5+cvS+im0MtpDT2jIkXg7bfdjrgq/Pvf8PjjkJz+5cqW\nhYEDYdMmGDcO/mWrw5jAs4QeQF6v43l1fqkJfc3iNbl+DRG3Iz56tEvwzz8P3bvD8eMZBpYrB127\n+n6RNWvc7v3hw7mOIzNe/exSeX1+/rKEbgq91Bp6IFpq0/eqf/yxa288cMDPJ//nP273vmZN19O+\neXOe4zGFiyX0APJ6Hc+r80vdQ29wXoOAvN4VV7jjn1WrugaXCy+ErVv9eOJ117keyH374MUXXSvN\n9de7+nseefWzS+X1+fnLErop9PJaQ/cltVe9QYN0veprs3nSNde4XshFi+DWWyEiwu3ue+QyeCb4\n/EroItJeRP4nIutF5DEfj98uIn+JyLKU252BD7Xg83odz6vzSz31f93SdQF93Tp1XG96y5Yne9Xn\nzvXjiRdcAB995Eoun3zi1vLNI69+dqm8Pj9/ZZvQRSQCeBO4AmgE3Cwivg7Rj1fVpim3DwIcpzFB\nE4w99FSVKsF330Hnzid71SdO9PPJp58OXbr4fuyrr6BtW5f4//47YPGa8ObPHnozYIOqblbVE8B4\noLOPcYV+IQuv1/G8Or/UhF6vab2gvH5qr/p998GxY64sfkqvem6MGeN297t1gxo1oF8/WLEi0+UF\nvPrZpfL6/PzlT0KvAWxJt/1Hyn0ZXSsiK0TkMxGpGZDojMkHwdxDTxUZ6S6Okb5X/bHHMvSq58To\n0W4t9vPPd7v+r78OTZq4PXdTaGV7kWh873ln3A2YDIxT1RMi0hMYA7Tz9WI9evQgJiYGgHLlyhEb\nG5v27ZpaBwvX7eHDh3tqPoVlfqlti7M+mcU5f58TtPebPTuBVq0gPj6Ou++GF15IYOlSmDYtjmLF\ncvh6pUqRUL8+vPgicWXKwKhRJEycCMWK4UafOj59jTnU/7+Dse21+SUkJBAfHw+Qli/9oqpZ3oAW\nwPR0248Dj2UxPgLYl8lj6mWzZs0KdQhB5dX5PfD1A8pg9P4378+39/zmG9VSpVRBtV071X37AvCi\nycm+79+/X2ddeqnq55+rHjkSgDcqeLz6u5kqJXdmm6/9KbksBuqJSLSIFAO64vbI04hItXSbnYHc\nn3IXxlK/ab3Kq/NLLbmc0eSMfHvPyy/PZa96VjJbj33KFOK++871uVeu7M5U/fxzOHIkj29YcHj1\ndzOnsk3oqpoE9AFmAD/julnWisgQEbk6ZVhfEVktIstTxvYIVsDGBFp+1NB9adrU9ao3bAgrV7r2\nxjXB2BW68EK3FsH557tlBT791B2Z7d07CG9mQsmvPnRVna6qDVW1vqo+l3LfIFX9KuXn/qraWFWb\nqGo7VV0fzKALqvR1PC/y6vxS+9A3LtuY7+/tq1f9hx8C/Ca1apHQrBksXgy//govvOB63TNriQzD\nNkiv/m7mlJ0pagq91D10/cex/vxRsaIru1xzjTvr/7LLXEUkKOrUgUcecWejdurke0yXLtCokVsy\n8scfITExSMGYQLOEHkBer+N5dX6pCT0mNiZkMZQsCRMmwP33u171G25wnYiB4vdnd+IELFvmaj/P\nPw9t2rhvnC5dYOfOwAUUYF793cwpS+im0Avkaot5ERnpTjh69lnXq96vn1t0Mde96rlRtCj88Yc7\nvbVfP7cYzYEDkJDgrrpkCjRL6AHk9TqeV+eXuof+y7JfQhyJa1R54gl3ImiRIm7Rxdtuc3vteZGj\nz65YMXeZvOHDYd06d1GOzz5zAWW0eTOcfbY7wPrppwFo1ckdr/5u5pQldFPopXW5kL9dLlnp3h2m\nTnULLX7yiVtXff/+EAUTHe0K+77Mng2rV7tLNnXt6pYhqF8fhg7N3xgNYAk9oLxex/Pq/FITevS5\n0SGO5FSXXw5z5kC1au5i1BdeCH/+mbvXCtpnd9NNri1n2DC3EHypUrBxY+b19p07Yc+egIfh1d/N\nnLKEbgq91LbFUHW5ZKVJk3zqVc+t4sXdgdP+/WH6dLeuzOLFmfe4v/aaO8jaoIFbWOyNN1wnzaFD\n+Ru3R1lCDyCv1/G8Or/UPfTfluf9ykDBEBPjcl6rVrBlS+561fPtsytSxJ3A1CCTqz8dPgwlSsCG\nDW7p37593RfCxx/n6W29+ruZU5bQTaEXqjNFc6JiRdd4kr5XfcKEUEeVC6++6rpmlixxy0/ecYf7\nM+S883yPv+ceN9l+/eC999w32e7d+RtzGJH8bNUSEQ11a5gxGT039zme+P4JHmv9GM9d+lyow8lS\nUpLbqX37bdcR8+qrLtd5VkyM74tlL1wIzZr9837VzNe0CWMigqpmOzF/ls81xtMKSh+6P1J71WvV\ncu2NDzzg2saff95dgtRzZs1yXTRr1py8rV3rDir4ctZZruxTr96pt7ZtXTumx1lCD6CEhARPH233\n6vxSSy6bVmyCTLrzChIRd1Z+jRpw553w0ksuqcfHu2OUvoTtZ1enjrt17HjyPh974QkJCcS1bOn6\n5lXdl0B6hw75TuiffOIu9Rcd7f6HFi0ahEnkH0voptBLW8slDPbQ0+vWzbU0XncdjB8PO3bAF19A\n2bKhjizIMiupFC/uumx++cW1Tqbe9uyB00775/hDh+CWW05uR0S45H7GGe7M2Izvo+oWLouKCthU\nAs1q6KbQe2X+Kzw842EebPEgr1zxSqjDybEVK6BDB9i+3Z20+fXXbmfTZGPnTndA4vffXZ1+61aX\ntGvX9l2337HDfYOWLesSf/Xq7lavHgweHNRQrYZujJ/CocslK7Gxrle9QwdYtcr1qn/9tVsw0WSh\ncmVXckl14oSrXe3b53v8rl2ubLN/v7utXevub9jQd0L/5Rc491z3PulvDRu6AyAZJSa6vwBKlcr1\ngV2/DqOISHsR+Z+IrBeRx3w8XkxExovIBhGZLyK1cxVNmPN6L6xX55ea0H//6fcQR5J7qb3qrVu7\nXvU2bdxZpqm8+tmlCsj8ihZ19fomTXw/3qgRHD3qEvuqVfDNN+7AxVNP+R6/c6fru9+0yZ1sNW2a\nW6Tnv//1PX71aihTxn1pVK7sevmbN4devfyeQrZ76CISAbyJu+jzVmCxiExS1f+lG3YXsEdV64vI\nTcALuEvVFSorVqwIzwNPfvLq/FIT+l+//BXiSPKmQgX49lu49VZXS7/sMnfuzg03ePezS5Vv8xNx\nJwVUrAiNG2c9tnlzV9Pftcsl99RbqVK+xx8+7OrzR4645+za5e7PwYFaf0ouzYANqrrZzUfG464b\nmj6hdwYGpfw8AfcFUOjsy+xPNY/w6vxS2xaPHcrjkoYFQMmSbgewXz933s5NN7n1X7z62aUqkPMT\ngXLl3K1evezHt27tkvqxY67ss3evuxUt6q4w5Qd/EnoNYEu67T9wSd7nGFVNEpF9IlJBVQO/Co8x\nAZa6h374xGGWbl0a4mgCo8cTIGWr8uazNXnwQWjc9BD1WqwNdVhBs3LjTj6a7sX5lcvRaH8Suq/q\nfMZWlYxjxMcYz9u0aVOoQwgqr86vSIT7Z7B2w1rOH3l+iKMJoGJAl1th0gesXraLbh3ODHVEQXSY\nLz728vz8k23booi0AAaravuU7ccBVdXn0435OmXMQhGJBLapahUfr1XokrwxxgRCoNoWFwP1RCQa\n2IY72HlzhjFTgNuBhcANwMzcBmSMMSZ3sk3oKTXxPsAMXJvjKFVdKyJDgMWq+hUwCvhQRDYAuymE\nHS7GGBNq+XqmqDHGmODJ9/XZRORpEflJRJaLyHQRqZbfMQSTiLwgImtFZIWIfC4iZUIdUyCJyPUi\nslpEkkSkaajjCYTsTpwLZyIySkR2iMjKUMcSDCJSU0RmisgaEVklIn1DHVMgiUhxEVmYki9Xicig\nLMfn9x66iJRS1UMpP/8bOEtV78vXIIJIRC4FZqpqsog8hzuA7OM83/AkIg2BZOA94P9UdVmIQ8qT\nlBPn1pPuxDmga4YT58KWiLQBDgFjVfWcUMcTaCk7hNVUdYWIlAKWAp298vkBiEiUqh5JaTj5Eeir\nqot8jc33PfTUZJ7iNChAl1oPAFX9TjVtUZAFQM1QxhNoqrpOVTfgu501HKWdOKeqJ4DUE+c8QVXn\nAntDHUewqOp2VV2R8vMhYC3uvBjPUNUjKT8Wxx33zHQvPCRL4ovIUBH5HbgFGBiKGPLJncDXoQ7C\nZMnXiXOeSgiFhYjEALG4bjvPEJEIEVkObAe+VdXFmY0NSkIXkW9FZGW626qU/3YEUNUnVbU28DHw\n72DEEEzZzS9lzADghKqOC2GoueLP/DzEnxPnTAGXUm6ZAPTLUAUIe6qarKpNcH/tNxeRszIbG5Tl\nc1XV3+u+fAJMBQYHI45gyW5+InI7cCVwSf5EFFg5+Py84A8g/eqgNXG1dBMmRKQILpl/qKqTQh1P\nsKjqARFJANoDa3yNCUWXS/pVajrjal6eISLtgUeBTqoa/qs9Zc0LdfS0E+dEpBjuHIrJIY4p0ARv\nfFaZ+QBYo6qvhTqQQBORSiJSNuXnksClnLow4qnjQ9DlMgFogDsYuhnoparb8jWIIEo5uaoY7gQr\ngAWqen8IQwooEbkGeAOoBOwDVqhqh9BGlTcpX8KvcfLEuedCHFLAiMg4IA6oCOwABqnq6JAGFUAi\n0hqYA6zClcoU6K+q00MaWICIyNnAGNzvZgTwqaoOy3S8nVhkjDHeEJIuF2OMMYFnCd0YYzzCErox\nxniEJXRjjPEIS+jGGOMRltCNMcYjLKEbY4xHWEI3xhiP+H9r2U5etu8GQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4cf5c7fc90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import math\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "x = np.array(np.linspace(start=-3, stop=3, num=1001, dtype=np.float))\n",
    "y_logit = np.log(1 + np.exp(-x)) / math.log(2)\n",
    "y_01 = x < 0\n",
    "y_hinge = 1.0 - x\n",
    "y_hinge[y_hinge < 0] = 0\n",
    "plt.plot(x, y_logit, 'r--', label='Logistic Loss', linewidth=2)\n",
    "plt.plot(x, y_01, 'g-', label='0/1 Loss', linewidth=2)\n",
    "plt.plot(x, y_hinge, 'b-', label='Hinge Loss', linewidth=2)\n",
    "plt.grid()\n",
    "plt.legend(loc=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Decision tree\n",
    "\n",
    "- Shannon's Information Entropy (熵)\n",
    "$$\n",
    "H(D) = - \\sum_{k=1}^K  \\frac{\\lvert  C_k\\rvert}{\\lvert D \\rvert} \\log_2 \\frac{\\lvert  C_k\\rvert}{\\lvert D \\rvert}\n",
    "$$\n",
    "\n",
    "- Empirical Conditional entropy (经验条件熵)\n",
    "$$\n",
    "H(D \\big| A) = - \\sum_{i=1}^n \\frac{\\lvert D_i \\rvert}{\\lvert D \\rvert}  \\sum_{k=1}^K \\frac{\\lvert D_{ik}\\rvert}{\\lvert D_i \\rvert}\\log_2 \\frac{\\lvert D_{ik}\\rvert}{\\lvert D_i \\rvert}\n",
    "$$\n",
    "\n",
    "- Information gain (信息增益)\n",
    "$$\n",
    "g(D, A) = H(D) - H(D \\big| A)\n",
    "$$\n",
    "\n",
    "- Information gain rate (信息增益率)\n",
    "$$\n",
    "g_r(D, A) = g(D, A) / H(A)\n",
    "$$\n",
    "\n",
    "- Gini coefficient (基尼系数)\n",
    "$$\n",
    "Gini(\\mathbf{p}) = \\sum_{k=1}^K p_k (1- p_k) = 1- \\sum_{k=1}^K p_k^2 = 1 - \\sum_{k=1}^K \\left( \\frac{\\lvert C_k\\rvert}{\\lvert D \\rvert}\\right)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Basic Algorithm\n",
    "\n",
    "1. Start at the root node as parent node\n",
    "2. Split the parent node at the feature $x_i$ to maximize information gain (IG)\n",
    "3. Assign training samples to new child nodes\n",
    "4. Stop if leave nodes are pure or early stopping criteria is satisfied, else repeat steps 1 and 2 for each new child node\n",
    "\n",
    "### Stopping Rules\n",
    "\n",
    "1. The leaf nodes are pure\n",
    "2. A maximal node depth is reached\n",
    "3. A minimum node size is reached\n",
    "4. Splitting a note does not lead to an information gain*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Decision tree: An example\n",
    "\n",
    "![](images/decisiontree1.png)\n",
    "\n",
    "### <font color=\"red\">Exercise</font>\n",
    "\n",
    "Compute the information gain at each split in the above decision tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble Learning: Majority vote\n",
    "\n",
    "Ensemble learning works by combining multiple *weak* classifiers into a *strong classifier*  through majority vote or weighted majority vote.\n",
    "\n",
    "- **Bagging**: Bootstrapped Aggregating by resampling the training samples with replacement\n",
    "![](images/bagging.png)\n",
    "- **Boosting**: Boosting the performance of weaker learner through gradient, and etc.\n",
    "![](images/boosting.png)\n",
    "- Known ensemble learners:\n",
    "    * random forest (随机森林)\n",
    "    * adaptive boosting (AdaBoost)\n",
    "    * GBDT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assessment of the classfiers\n",
    "\n",
    "\n",
    "- For a binary classifier, both the prediction error (ERR) and accuracy (ACC) provide general information about how many samples are misclassified.\n",
    "\n",
    "- The **error** can be understood as the sum of all false predictions divided by the number of total predictions:\n",
    "$$\n",
    "ERR = \\frac{FP + FN}{FP + FN + TP + TN}\n",
    "$$\n",
    "\n",
    "(TP = true positives, FP = false positives, TN = true negatives, FN = false negatives)\n",
    "\n",
    "- The prediction **accuracy** can then be calculated directly from the error:\n",
    "$$\n",
    "ACC = \\frac{TP + TN}{FP + FN + TP + TN} = 1 - ERR\n",
    "$$\n",
    "\n",
    "- The true **\\textit{positive rate} (TPR)** and **\\textit{false positive rate} (FPR)** are performance metrics that are especially useful for imbalanced class problems:\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "FPR &=& \\frac{FP}{N} = \\frac{FP}{FP + TN}\\\\\n",
    "TPR &=& \\frac{TP}{P} = \\frac{TP}{FN+TP}\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "- **\\textit{Precision (PRE)}** and **\\textit{recall} (REC)** are performance metrics that are related to those true positive and true negative rates, and in fact, recall is synonymous to the true positive rate:\n",
    "$$\n",
    "\\begin{array}{lcl}\n",
    "PRE &=& \\frac{TP}{TP + FP} \\\\\n",
    "REC &=& TPR = \\frac{TP}{P} = \\frac{TP}{FN + TP}\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "In practice, often a combination of precision and recall is used, the so-called **\\textit{F1-score}**:\n",
    "$$\n",
    "\\text{F1} = 2 \\times \\frac{PRE \\times REC}{PRE + REC}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## No free lunch for machine learning algorithms\n",
    "\n",
    "- Both logistic regression and SVMs work great for linear problems, logistic regression may be preferable for very noisy data\n",
    "- Naive Bayes may work better than logistic regression for small training set sizes; the former is also pretty fast, e.g., if you have a large multi-class problem, you'd only have to train one classifier whereas you'd have to use One-vs-Rest or One-vs-One with in SVMs or logistic regression (alternatively, you could implement multinomial/softmax regression though); another point is that you don't have to worry so much about hyperparameter optimization -- if you are estimating the class priors from the training set, there are actually no hyperparameters\n",
    "- Kernel SVM/logistic regression is preferable for nonlinear data vs. the linear models\n",
    "- KNN can also work quite well in practice for datasets with large number of samples and relatively low dimensionality\n",
    "- Random Forests & Extremely Randomized trees are very robust and work well across a whole range of problems -- linear and/or nonlinear problems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-Layer Neural Networks (Deep learning)\n",
    "\n",
    "We will illustrate this in next chapter."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}