Perceptron/Perceptron Code from scratch.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "\n",
    "def perceptron_learning_algorithm(X, y, w):\n",
    "    misclassified_examples = predict(hypothesis, X, y, w) \n",
    "    while misclassified_examples.any(): # execute if any misclassified\n",
    "        x, expected_y = pick_one_from(misclassified_examples, X, y)\n",
    "        w = w + x * expected_y  # update rule\n",
    "        misclassified_examples = predict(hypothesis, X, y, w) \n",
    "    return w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **perceptron_learning_algorithm** uses several functions. The **hypothesis** function is just 'h(x)' written in Python code; as we saw before, it is the function that returns the label $y_{i}$ predicted for an example $x_{i}$ when classifying with the hyperplane defined by **w**. The **predict** function applies the hypothesis for every example and returns the ones that are misclassified. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hypothesis(x, w):\n",
    "    return np.sign(np.dot(w, x)) # return 1,-1\n",
    "\n",
    "\n",
    "# Make predictions on all data points \n",
    "# and return the ones that are misclassified.\n",
    "def predict(hypothesis_function, X, y, w):\n",
    "    predictions = np.apply_along_axis(hypothesis_function, 1, X, w)\n",
    "    misclassified = X[y != predictions]\n",
    "    return misclassified "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pick one misclassified example randomly \n",
    "# and return it with its expected label. \n",
    "def pick_one_from(misclassified_examples, X, y):\n",
    "    np.random.shuffle(misclassified_examples)\n",
    "    x = misclassified_examples[0]\n",
    "    index = np.where(np.all(X == x, axis=1))\n",
    "    return x, y[int(index[0])] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It returns weight: \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([-1,  3,  9])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[8,1,7], [4,-3,9], [-5,-2,6]])\n",
    "w = np.array([4, 5, 3]) # slope values of 3 features\n",
    "expected_y = [1,1,1] \n",
    "print('It returns weight: ')\n",
    "perceptron_learning_algorithm(x,expected_y,w)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have made predictions with **predict**, we know which examples are misclassified, so we use the function **pick_one_from** to select one of them randomly "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then arrive at the heart of the algorithm: the update rule. For now, just remember that it changes the value of 'w'. Why it does this will be explained in detail later. We once again use the **predict** function, but this time, we give it the updated 'w'. It allows us to see if we have classified all data points correctly, or if we need to repeat the process until we do.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also note that sometimes updating the value of 'w' for a particular example 'x' changes the hyperplane in such a way that another example 'x*' previously correctly classified becomes misclassified. So, the hypothesis might become worse at classifying after being updated. which shows us the number of classified examples at each iteration step. One way to avoid this problem is to keep a record of the value of  before making the update and use the updated  only if it reduces the number of misclassified examples. This modification of the PLA is known as the **Pocket algorithm** (because we keep  in our pocket). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding the limitations of the PLA \n",
    "\n",
    "One thing to understand about the PLA algorithm is that because weights are randomly initialized and misclassified examples are randomly chosen, it is possible the algorithm will return a different hyperplane each time we run it.\n",
    "\n",
    "To train a model, we pick a sample of existing data and call it the training set. We train the model, and it comes up with a hypothesis (a hyperplane in our case). We can measure how well the hypothesis performs on the training set: we call this the in-sample error (also called training error). Once we are satisfied with the hypothesis, we decide to use it on unseen data (the test set) to see if it indeed learned something. We measure how well the hypothesis performs on the test set, and we call this the out-of-sample error (also called the generalization error). \n",
    " \n",
    "Our goal is to have the smallest out-of-sample error. \n",
    "\n",
    "When using the Perceptron with a linearly separable dataset, we have the guarantee of finding a hypothesis with zero in-sample error, but we have no guarantee about how well it will generalize to unseen data (if an algorithm generalizes well, its out-of-sample error will be close to its in-sample error).  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}