10. Intro to Deep Learning/tutorial/01. a-single-neuron.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "80d10fdf",
   "metadata": {
    "papermill": {
     "duration": 0.006244,
     "end_time": "2022-05-05T17:51:42.959708",
     "exception": false,
     "start_time": "2022-05-05T17:51:42.953464",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Welcome to Deep Learning! #\n",
    "\n",
    "Welcome to Kaggle's *Introduction to Deep Learning* course! You're about to learn all you need to get started building your own deep neural networks. Using Keras and Tensorflow you'll learn how to:\n",
    "- create a **fully-connected** neural network architecture\n",
    "- apply neural nets to two classic ML problems: **regression** and **classification**\n",
    "- train neural nets with **stochastic gradient descent**, and\n",
    "- improve performance with **dropout**, **batch normalization**, and other techniques\n",
    "\n",
    "The tutorials will introduce you to these topics with fully-worked examples, and then in the exercises, you'll explore these topics in more depth and apply them to real-world datasets.\n",
    "\n",
    "Let's get started!\n",
    "\n",
    "# What is Deep Learning? #\n",
    "\n",
    "Some of the most impressive advances in artificial intelligence in recent years have been in the field of *deep learning*. Natural language translation, image recognition, and game playing are all tasks where deep learning models have neared or even exceeded human-level performance.\n",
    "\n",
    "So what is deep learning? **Deep learning** is an approach to machine learning characterized by deep stacks of computations. This depth of computation is what has enabled deep learning models to disentangle the kinds of complex and hierarchical patterns found in the most challenging real-world datasets.\n",
    "\n",
    "Through their power and scalability **neural networks** have become the defining model of deep learning.  Neural networks are composed of neurons, where each neuron individually performs only a simple computation. The power of a neural network comes instead from the complexity of the connections these neurons can form.\n",
    "\n",
    "# The Linear Unit #\n",
    "\n",
    "So let's begin with the fundamental component of a neural network: the individual neuron. As a diagram, a **neuron** (or **unit**) with one input looks like:\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/mfOlDR6.png\" width=\"250\" alt=\"Diagram of a linear unit.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>The Linear Unit: $y = w x + b$\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "The input is `x`. Its connection to the neuron has a **weight** which is `w`. Whenever a value flows through a connection, you multiply the value by the connection's weight. For the input `x`, what reaches the neuron is `w * x`. A neural network \"learns\" by modifying its weights.\n",
    "\n",
    "The `b` is a special kind of weight we call the **bias**. The bias doesn't have any input data associated with it; instead, we put a `1` in the diagram so that the value that reaches the neuron is just `b` (since `1 * b = b`). The bias enables the neuron to modify the output independently of its inputs.\n",
    "\n",
    "The `y` is the value the neuron ultimately outputs. To get the output, the neuron sums up all the values it receives through its connections. This neuron's activation is `y = w * x + b`, or as a formula $y = w x + b$.\n",
    "\n",
    "<blockquote style=\"margin-right:auto; margin-left:auto; background-color: #ebf9ff; padding: 1em; margin:24px;\">\n",
    "    <strong>Does the formula $y=w x + b$ look familiar?</strong><br>\n",
    "It's an equation of a line! It's the slope-intercept equation, where $w$ is the slope and $b$ is the y-intercept. \n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ace11730",
   "metadata": {
    "papermill": {
     "duration": 0.004767,
     "end_time": "2022-05-05T17:51:42.969669",
     "exception": false,
     "start_time": "2022-05-05T17:51:42.964902",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Example - The Linear Unit as a Model #\n",
    "\n",
    "Though individual neurons will usually only function as part of a larger network, it's often useful to start with a single neuron model as a baseline. Single neuron models are *linear* models. \n",
    "\n",
    "Let's think about how this might work on a dataset like [80 Cereals](https://www.kaggle.com/crawford/80-cereals). Training a model with `'sugars'` (grams of sugars per serving) as input and `'calories'` (calories per serving) as output, we might find the bias is `b=90` and the weight is `w=2.5`. We could estimate the calorie content of a cereal with 5 grams of sugar per serving like this:\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/yjsfFvY.png\" width=\"1000\" alt=\"Computing with the linear unit.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>Computing with the linear unit.\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "And, checking against our formula, we have $calories = 2.5 \\times 5 + 90 = 102.5$, just like we expect.\n",
    "\n",
    "# Multiple Inputs #\n",
    "\n",
    "The *80 Cereals* dataset has many more features than just `'sugars'`. What if we wanted to expand our model to include things like fiber or protein content? That's easy enough. We can just add more input connections to the neuron, one for each additional feature. To find the output, we would multiply each input to its connection weight and then add them all together.\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/vyXSnlZ.png\" width=\"300\" alt=\"Three input connections: x0, x1, and x2, along with the bias.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>A linear unit with three inputs.\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "The formula for this neuron would be $y = w_0 x_0 + w_1 x_1 + w_2 x_2 + b$. A linear unit with two inputs will fit a plane, and a unit with more inputs than that will fit a hyperplane.\n",
    "\n",
    "# Linear Units in Keras #\n",
    "\n",
    "The easiest way to create a model in Keras is through `keras.Sequential`, which creates a neural network as a stack of *layers*. We can create models like those above using a *dense* layer (which we'll learn more about in the next lesson).\n",
    "\n",
    "We could define a linear model accepting three input features (`'sugars'`, `'fiber'`, and `'protein'`) and producing a single output (`'calories'`) like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "00ffc8c4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-05T17:51:42.981345Z",
     "iopub.status.busy": "2022-05-05T17:51:42.980950Z",
     "iopub.status.idle": "2022-05-05T17:51:49.595090Z",
     "shell.execute_reply": "2022-05-05T17:51:49.594047Z"
    },
    "papermill": {
     "duration": 6.622935,
     "end_time": "2022-05-05T17:51:49.597581",
     "exception": false,
     "start_time": "2022-05-05T17:51:42.974646",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-05-05 17:51:49.526369: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "# Create a network with 1 linear unit\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(units=1, input_shape=[3])\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "823009aa",
   "metadata": {
    "papermill": {
     "duration": 0.005333,
     "end_time": "2022-05-05T17:51:49.609261",
     "exception": false,
     "start_time": "2022-05-05T17:51:49.603928",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "With the first argument, `units`, we define how many outputs we want. In this case we are just predicting `'calories'`, so we'll use `units=1`. \n",
    "\n",
    "With the second argument, `input_shape`, we tell Keras the dimensions of the inputs. Setting `input_shape=[3]` ensures the model will accept three features as input (`'sugars'`, `'fiber'`, and `'protein'`).\n",
    "\n",
    "This model is now ready to be fit to training data!\n",
    "\n",
    "<blockquote style=\"margin-right:auto; margin-left:auto; background-color: #ebf9ff; padding: 1em; margin:24px;\">\n",
    "    <strong>Why is <code>input_shape</code> a Python list?</strong><br>\n",
    "The data we'll use in this course will be tabular data, like in a Pandas dataframe. We'll have one input for each feature in the dataset. The features are arranged by column, so we'll always have <code>input_shape=[num_columns]</code>.\n",
    "\n",
    "The reason Keras uses a list here is to permit use of more complex datasets. Image data, for instance, might need three dimensions: <code>[height, width, channels]</code>.\n",
    "</blockquote>\n",
    "\n",
    "# Your Turn #\n",
    "\n",
    "[**Define a linear model**](https://www.kaggle.com/kernels/fork/11887334) for the *Red Wine Quality* dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29a71cf3",
   "metadata": {
    "papermill": {
     "duration": 0.005218,
     "end_time": "2022-05-05T17:51:49.620088",
     "exception": false,
     "start_time": "2022-05-05T17:51:49.614870",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/intro-to-deep-learning/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 19.692016,
   "end_time": "2022-05-05T17:51:52.457795",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-05-05T17:51:32.765779",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}