10. Intro to Deep Learning/tutorial/03. stochastic-gradient-descent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f5d2e1bc",
   "metadata": {
    "papermill": {
     "duration": 0.010959,
     "end_time": "2022-05-05T17:51:08.731570",
     "exception": false,
     "start_time": "2022-05-05T17:51:08.720611",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction #\n",
    "\n",
    "In the first two lessons, we learned how to build fully-connected networks out of stacks of dense layers. When first created, all of the network's weights are set randomly -- the network doesn't \"know\" anything yet. In this lesson we're going to see how to train a neural network; we're going to see how neural networks *learn*.\n",
    "\n",
    "As with all machine learning tasks, we begin with a set of training data. Each example in the training data consists of some features (the inputs) together with an expected target (the output). Training the network means adjusting its weights in such a way that it can transform the features into the target. In the *80 Cereals* dataset, for instance, we want a network that can take each cereal's `'sugar'`, `'fiber'`, and `'protein'` content and produce a prediction for that cereal's `'calories'`. If we can successfully train a network to do that, its weights must represent in some way the relationship between those features and that target as expressed in the training data.\n",
    "\n",
    "In addition to the training data, we need two more things:\n",
    "- A \"loss function\" that measures how good the network's predictions are.\n",
    "- An \"optimizer\" that can tell the network how to change its weights.\n",
    "\n",
    "# The Loss Function #\n",
    "\n",
    "We've seen how to design an architecture for a network, but we haven't seen how to tell a network *what* problem to solve. This is the job of the loss function.\n",
    "\n",
    "The **loss function** measures the disparity between the the target's true value and the value the model predicts. \n",
    "\n",
    "Different problems call for different loss functions. We have been looking at **regression** problems, where the task is to predict some numerical value -- calories in *80 Cereals*, rating in *Red Wine Quality*. Other regression tasks might be predicting the price of a house or the fuel efficiency of a car.\n",
    "\n",
    "A common loss function for regression problems is the **mean absolute error** or **MAE**. For each prediction `y_pred`, MAE measures the disparity from the true target `y_true` by an absolute difference `abs(y_true - y_pred)`.\n",
    "\n",
    "The total MAE loss on a dataset is the mean of all these absolute differences.\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/VDcvkZN.png\" width=\"500\" alt=\"A graph depicting error bars from data points to the fitted line..\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>The mean absolute error is the average length between the fitted curve and the data points.\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "Besides MAE, other loss functions you might see for regression problems are the mean-squared error (MSE) or the Huber loss (both available in Keras).\n",
    "\n",
    "During training, the model will use the loss function as a guide for finding the correct values of its weights (lower loss is better). In other words, the loss function tells the network its objective.\n",
    "\n",
    "# The Optimizer - Stochastic Gradient Descent #\n",
    "\n",
    "We've described the problem we want the network to solve, but now we need to say *how* to solve it. This is the job of the **optimizer**. The optimizer is an algorithm that adjusts the weights to minimize the loss.\n",
    "\n",
    "Virtually all of the optimization algorithms used in deep learning belong to a family called **stochastic gradient descent**. They are iterative algorithms that train a network in steps. One **step** of training goes like this:\n",
    "1. Sample some training data and run it through the network to make predictions.\n",
    "2. Measure the loss between the predictions and the true values.\n",
    "3. Finally, adjust the weights in a direction that makes the loss smaller.\n",
    "\n",
    "Then just do this over and over until the loss is as small as you like (or until it won't decrease any further.)\n",
    "\n",
    "<figure style=\"padding: 1em;\">\n",
    "<img src=\"https://i.imgur.com/rFI1tIk.gif\" width=\"1600\" alt=\"Fitting a line batch by batch. The loss decreases and the weights approach their true values.\">\n",
    "<figcaption style=\"textalign: center; font-style: italic\"><center>Training a neural network with Stochastic Gradient Descent.\n",
    "</center></figcaption>\n",
    "</figure>\n",
    "\n",
    "Each iteration's sample of training data is called a **minibatch** (or often just \"batch\"), while a complete round of the training data is called an **epoch**. The number of epochs you train for is how many times the network will see each training example.\n",
    "\n",
    "The animation shows the linear model from Lesson 1 being trained with SGD. The pale red dots depict the entire training set, while the solid red dots are the minibatches. Every time SGD sees a new minibatch, it will shift the weights (`w` the slope and `b` the y-intercept) toward their correct values on that batch. Batch after batch, the line eventually converges to its best fit. You can see that the loss gets smaller as the weights get closer to their true values.\n",
    "\n",
    "## Learning Rate and Batch Size ##\n",
    "\n",
    "Notice that the line only makes a small shift in the direction of each batch (instead of moving all the way). The size of these shifts is determined by the **learning rate**. A smaller learning rate means the network needs to see more minibatches before its weights converge to their best values.\n",
    "\n",
    "The learning rate and the size of the minibatches are the two parameters that have the largest effect on how the SGD training proceeds. Their interaction is often subtle and the right choice for these parameters isn't always obvious. (We'll explore these effects in the exercise.)\n",
    "\n",
    "Fortunately, for most work it won't be necessary to do an extensive hyperparameter search to get satisfactory results. **Adam** is an SGD algorithm that has an adaptive learning rate that makes it suitable for most problems without any parameter tuning (it is \"self tuning\", in a sense). Adam is a great general-purpose optimizer.\n",
    "\n",
    "## Adding the Loss and Optimizer ##\n",
    "\n",
    "After defining a model, you can add a loss function and optimizer with the model's `compile` method:\n",
    "\n",
    "```\n",
    "model.compile(\n",
    "    optimizer=\"adam\",\n",
    "    loss=\"mae\",\n",
    ")\n",
    "```\n",
    "\n",
    "Notice that we are able to specify the loss and optimizer with just a string. You can also access these directly through the Keras API -- if you wanted to tune parameters, for instance -- but for us, the defaults will work fine.\n",
    "\n",
    "<blockquote style=\"margin-right:auto; margin-left:auto; background-color: #ebf9ff; padding: 1em; margin:24px;\">\n",
    "    <strong>What's In a Name?</strong><br>\n",
    "The <strong>gradient</strong> is a vector that tells us in what direction the weights need to go. More precisely, it tells us how to change the weights to make the loss change <em>fastest</em>. We call our process gradient <strong>descent</strong> because it uses the gradient to <em>descend</em> the loss curve towards a minimum. <strong>Stochastic</strong> means \"determined by chance.\" Our training is <em>stochastic</em> because the minibatches are <em>random samples</em> from the dataset. And that's why it's called SGD!\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "851c4a25",
   "metadata": {
    "papermill": {
     "duration": 0.010151,
     "end_time": "2022-05-05T17:51:08.752265",
     "exception": false,
     "start_time": "2022-05-05T17:51:08.742114",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Example - Red Wine Quality #\n",
    "\n",
    "Now we know everything we need to start training deep learning models. So let's see it in action! We'll use the *Red Wine Quality* dataset.\n",
    "\n",
    "This dataset consists of physiochemical measurements from about 1600 Portuguese red wines. Also included is a quality rating for each wine from blind taste-tests. How well can we predict a wine's perceived quality from these measurements?\n",
    "\n",
    "We've put all of the data preparation into this next hidden cell. It's not essential to what follows so feel free to skip it. One thing you might note for now though is that we've rescaled each feature to lie in the interval $[0, 1]$. As we'll discuss more in Lesson 5, neural networks tend to perform best when their inputs are on a common scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "60c07c35",
   "metadata": {
    "_kg_hide-input": true,
    "execution": {
     "iopub.execute_input": "2022-05-05T17:51:08.775018Z",
     "iopub.status.busy": "2022-05-05T17:51:08.774551Z",
     "iopub.status.idle": "2022-05-05T17:51:08.845162Z",
     "shell.execute_reply": "2022-05-05T17:51:08.844128Z"
    },
    "lines_to_next_cell": 0,
    "papermill": {
     "duration": 0.084257,
     "end_time": "2022-05-05T17:51:08.847376",
     "exception": false,
     "start_time": "2022-05-05T17:51:08.763119",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed acidity</th>\n",
       "      <th>volatile acidity</th>\n",
       "      <th>citric acid</th>\n",
       "      <th>residual sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free sulfur dioxide</th>\n",
       "      <th>total sulfur dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>quality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1109</th>\n",
       "      <td>10.8</td>\n",
       "      <td>0.470</td>\n",
       "      <td>0.43</td>\n",
       "      <td>2.10</td>\n",
       "      <td>0.171</td>\n",
       "      <td>27.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>0.99820</td>\n",
       "      <td>3.17</td>\n",
       "      <td>0.76</td>\n",
       "      <td>10.8</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1032</th>\n",
       "      <td>8.1</td>\n",
       "      <td>0.820</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.10</td>\n",
       "      <td>0.095</td>\n",
       "      <td>5.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.99854</td>\n",
       "      <td>3.36</td>\n",
       "      <td>0.53</td>\n",
       "      <td>9.6</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1002</th>\n",
       "      <td>9.1</td>\n",
       "      <td>0.290</td>\n",
       "      <td>0.33</td>\n",
       "      <td>2.05</td>\n",
       "      <td>0.063</td>\n",
       "      <td>13.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0.99516</td>\n",
       "      <td>3.26</td>\n",
       "      <td>0.84</td>\n",
       "      <td>11.7</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>487</th>\n",
       "      <td>10.2</td>\n",
       "      <td>0.645</td>\n",
       "      <td>0.36</td>\n",
       "      <td>1.80</td>\n",
       "      <td>0.053</td>\n",
       "      <td>5.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.99820</td>\n",
       "      <td>3.17</td>\n",
       "      <td>0.42</td>\n",
       "      <td>10.0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
       "1109           10.8             0.470         0.43            2.10      0.171   \n",
       "1032            8.1             0.820         0.00            4.10      0.095   \n",
       "1002            9.1             0.290         0.33            2.05      0.063   \n",
       "487            10.2             0.645         0.36            1.80      0.053   \n",
       "\n",
       "      free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
       "1109                 27.0                  66.0  0.99820  3.17       0.76   \n",
       "1032                  5.0                  14.0  0.99854  3.36       0.53   \n",
       "1002                 13.0                  27.0  0.99516  3.26       0.84   \n",
       "487                   5.0                  14.0  0.99820  3.17       0.42   \n",
       "\n",
       "      alcohol  quality  \n",
       "1109     10.8        6  \n",
       "1032      9.6        5  \n",
       "1002     11.7        7  \n",
       "487      10.0        6  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "import pandas as pd\n",
    "from IPython.display import display\n",
    "\n",
    "red_wine = pd.read_csv('../input/dl-course-data/red-wine.csv')\n",
    "\n",
    "# Create training and validation splits\n",
    "df_train = red_wine.sample(frac=0.7, random_state=0)\n",
    "df_valid = red_wine.drop(df_train.index)\n",
    "display(df_train.head(4))\n",
    "\n",
    "# Scale to [0, 1]\n",
    "max_ = df_train.max(axis=0)\n",
    "min_ = df_train.min(axis=0)\n",
    "df_train = (df_train - min_) / (max_ - min_)\n",
    "df_valid = (df_valid - min_) / (max_ - min_)\n",
    "\n",
    "# Split features and target\n",
    "X_train = df_train.drop('quality', axis=1)\n",
    "X_valid = df_valid.drop('quality', axis=1)\n",
    "y_train = df_train['quality']\n",
    "y_valid = df_valid['quality']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9113833",
   "metadata": {
    "papermill": {
     "duration": 0.010785,
     "end_time": "2022-05-05T17:51:08.870188",
     "exception": false,
     "start_time": "2022-05-05T17:51:08.859403",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "How many inputs should this network have? We can discover this by looking at the number of columns in the data matrix. Be sure not to include the target (`'quality'`) here -- only the input features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9ef4df3f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-05T17:51:08.894606Z",
     "iopub.status.busy": "2022-05-05T17:51:08.894218Z",
     "iopub.status.idle": "2022-05-05T17:51:08.899338Z",
     "shell.execute_reply": "2022-05-05T17:51:08.898198Z"
    },
    "lines_to_next_cell": 2,
    "papermill": {
     "duration": 0.019742,
     "end_time": "2022-05-05T17:51:08.900932",
     "exception": false,
     "start_time": "2022-05-05T17:51:08.881190",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1119, 11)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f4685c",
   "metadata": {
    "papermill": {
     "duration": 0.011457,
     "end_time": "2022-05-05T17:51:08.924283",
     "exception": false,
     "start_time": "2022-05-05T17:51:08.912826",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Eleven columns means eleven inputs.\n",
    "\n",
    "We've chosen a three-layer network with over 1500 neurons. This network should be capable of learning fairly complex relationships in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e4a96768",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-05T17:51:08.949955Z",
     "iopub.status.busy": "2022-05-05T17:51:08.949534Z",
     "iopub.status.idle": "2022-05-05T17:51:14.571233Z",
     "shell.execute_reply": "2022-05-05T17:51:14.570223Z"
    },
    "papermill": {
     "duration": 5.636819,
     "end_time": "2022-05-05T17:51:14.573031",
     "exception": false,
     "start_time": "2022-05-05T17:51:08.936212",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-05-05 17:51:14.489438: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "model = keras.Sequential([\n",
    "    layers.Dense(512, activation='relu', input_shape=[11]),\n",
    "    layers.Dense(512, activation='relu'),\n",
    "    layers.Dense(512, activation='relu'),\n",
    "    layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51116cd6",
   "metadata": {
    "papermill": {
     "duration": 0.013131,
     "end_time": "2022-05-05T17:51:14.599721",
     "exception": false,
     "start_time": "2022-05-05T17:51:14.586590",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Deciding the architecture of your model should be part of a process. Start simple and use the validation loss as your guide. You'll learn more about model development in the exercises.\n",
    "\n",
    "After defining the model, we compile in the optimizer and loss function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bc991c75",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-05T17:51:14.626399Z",
     "iopub.status.busy": "2022-05-05T17:51:14.626146Z",
     "iopub.status.idle": "2022-05-05T17:51:14.638882Z",
     "shell.execute_reply": "2022-05-05T17:51:14.638090Z"
    },
    "papermill": {
     "duration": 0.027452,
     "end_time": "2022-05-05T17:51:14.640479",
     "exception": false,
     "start_time": "2022-05-05T17:51:14.613027",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer='adam',\n",
    "    loss='mae',\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b519d2c",
   "metadata": {
    "papermill": {
     "duration": 0.011486,
     "end_time": "2022-05-05T17:51:14.664076",
     "exception": false,
     "start_time": "2022-05-05T17:51:14.652590",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Now we're ready to start the training! We've told Keras to feed the optimizer 256 rows of the training data at a time (the `batch_size`) and to do that 10 times all the way through the dataset (the `epochs`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "201c9910",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-05T17:51:14.689035Z",
     "iopub.status.busy": "2022-05-05T17:51:14.688785Z",
     "iopub.status.idle": "2022-05-05T17:51:17.800254Z",
     "shell.execute_reply": "2022-05-05T17:51:17.799259Z"
    },
    "papermill": {
     "duration": 3.126783,
     "end_time": "2022-05-05T17:51:17.802722",
     "exception": false,
     "start_time": "2022-05-05T17:51:14.675939",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-05-05 17:51:14.768185: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "5/5 [==============================] - 1s 73ms/step - loss: 0.2626 - val_loss: 0.1382\n",
      "Epoch 2/10\n",
      "5/5 [==============================] - 0s 24ms/step - loss: 0.1382 - val_loss: 0.1243\n",
      "Epoch 3/10\n",
      "5/5 [==============================] - 0s 17ms/step - loss: 0.1246 - val_loss: 0.1180\n",
      "Epoch 4/10\n",
      "5/5 [==============================] - 0s 17ms/step - loss: 0.1141 - val_loss: 0.1096\n",
      "Epoch 5/10\n",
      "5/5 [==============================] - 0s 29ms/step - loss: 0.1127 - val_loss: 0.1091\n",
      "Epoch 6/10\n",
      "5/5 [==============================] - 0s 24ms/step - loss: 0.1096 - val_loss: 0.1044\n",
      "Epoch 7/10\n",
      "5/5 [==============================] - 0s 22ms/step - loss: 0.1049 - val_loss: 0.1025\n",
      "Epoch 8/10\n",
      "5/5 [==============================] - 0s 17ms/step - loss: 0.1054 - val_loss: 0.1021\n",
      "Epoch 9/10\n",
      "5/5 [==============================] - 0s 21ms/step - loss: 0.1016 - val_loss: 0.1010\n",
      "Epoch 10/10\n",
      "5/5 [==============================] - 0s 17ms/step - loss: 0.1004 - val_loss: 0.0975\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "    X_train, y_train,\n",
    "    validation_data=(X_valid, y_valid),\n",
    "    batch_size=256,\n",
    "    epochs=10,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2f2cb27",
   "metadata": {
    "papermill": {
     "duration": 0.020275,
     "end_time": "2022-05-05T17:51:17.843301",
     "exception": false,
     "start_time": "2022-05-05T17:51:17.823026",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "You can see that Keras will keep you updated on the loss as the model trains.\n",
    "\n",
    "Often, a better way to view the loss though is to plot it. The `fit` method in fact keeps a record of the loss produced during training in a `History` object. We'll convert the data to a Pandas dataframe, which makes the plotting easy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "90d01429",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-05-05T17:51:17.884506Z",
     "iopub.status.busy": "2022-05-05T17:51:17.884122Z",
     "iopub.status.idle": "2022-05-05T17:51:18.091248Z",
     "shell.execute_reply": "2022-05-05T17:51:18.090377Z"
    },
    "lines_to_next_cell": 2,
    "papermill": {
     "duration": 0.230129,
     "end_time": "2022-05-05T17:51:18.093226",
     "exception": false,
     "start_time": "2022-05-05T17:51:17.863097",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAh4ElEQVR4nO3de3CV933n8fdX5+h+PYC4SToCAjjGCVgHmST1xnGwm+A0he42ztpptnE3M+5MStLdNLPjrXeTGaedycTpbtvEk9p1kzZJU9a4TpZN8Tpe200vqRsEAtuAMTJGSFzFRQJJ6Hb03T/OIziI2xE64tw+rxmG566vzsDnec7veX7Pz9wdERHJX0WZLkBERGaWgl5EJM8p6EVE8pyCXkQkzynoRUTyXDjTBUw2Z84cX7RoUabLEBHJKdu3bz/p7vVXWpd1Qb9o0SLa2toyXYaISE4xs86rrVPTjYhInlPQi4jkOQW9iEieU9CLiOQ5Bb2ISJ5T0IuI5DkFvYhInsuboO8dHOFPX9rP6919mS5FRCSrpBT0ZrbOzPaZWYeZPXKF9V80sz1m9pqZvWRmzUnromb2UzPbG2yzKI31XxAqMv7n/3uLl948PhOHFxHJWdcNejMLAU8A9wErgAfNbMWkzdqBVndfCTwLfD1p3feAx939VmANcCIdhU9WXVbM8rnV7DjUOxOHFxHJWalc0a8BOtz9gLuPAJuADckbuPsr7j4YzL4KNAIEJ4Swu78YbNeftF3axZrraD90hvFxjZolIjIhlaBvALqS5ruDZVfzWeD5YHo50Gtmz5lZu5k9HnxDuISZPWxmbWbW1tPTk2rtl4lFI5wbGuPtnv4bPoaISL5J681YM/s00Ao8HiwKAx8EvgTcASwBHpq8n7s/5e6t7t5aX3/Fl6+lJNYcAWDHoTM3fAwRkXyTStAfBpqS5huDZZcws3uBR4H17j4cLO4GdgbNPmPAj4HYtCq+hiVzKqmrKGZHZ+9M/QgRkZyTStBvA5aZ2WIzKwEeALYkb2BmLcCTJEL+xKR968xs4jJ9LbBn+mVfmZnR0lTHdl3Ri4hccN2gD67ENwIvAHuBZ9x9t5k9Zmbrg80eB6qAzWa208y2BPvGSTTbvGRmrwMG/PkM/B4XxKIROk700zc4OpM/RkQkZ6Q08Ii7bwW2Tlr25aTpe6+x74vAyhstcKom2unbu85w9y1zb9aPFRHJWnnTM3bCqqY6igw9Ty8iEsi7oK8qDbN8XjXtaqcXEQHyMOgBVjdH2HmoVx2nRETI06CPRSOcGx5j/wl1nBIRyc+gV8cpEZEL8jLoF82uYFZlCds7FfQiInkZ9BMdp3RFLyKSp0EPieabAz0D9A6OZLoUEZGMytugb4nWAdCu5+lFpMDlbdCvapzoOKXmGxEpbHkb9JWlYW5dUKOgF5GCl7dBD4nn6Xce6iWujlMiUsDyO+ib6xgYifPW8XOZLkVEJGPyO+ijiY5Tep5eRApZXgd9dFYFsytL1E4vIgUtr4PezGiJRvSIpYgUtJSC3szWmdk+M+sws0eusP6LZrbHzF4zs5fMrHnS+hoz6zazb6Wr8FTFmut45+QApwfUcUpECtN1g97MQsATwH3ACuBBM1sxabN2oNXdVwLPAl+ftP6rwD9Mv9ypm2in1/vpRaRQpXJFvwbocPcD7j4CbAI2JG/g7q+4+2Aw+yrQOLHOzFYD84CfpqfkqVnVWEeoyNROLyIFK5WgbwC6kua7g2VX81ngeQAzKwL+iMQA4VdlZg+bWZuZtfX09KRQUurKS0KsWFDDjs7etB5XRCRXpPVmrJl9GmgFHg8WfQ7Y6u7d19rP3Z9y91Z3b62vr09nSQDEonXs6u5lLD6e9mOLiGS7VIL+MNCUNN8YLLuEmd0LPAqsd/fhYPEHgI1mdhD4BvCbZva1aVV8A2LNEQZH4rx5TB2nRKTwhFPYZhuwzMwWkwj4B4BPJW9gZi3Ak8A6dz8xsdzdfyNpm4dI3LC97KmdmZZ8Q/Y9DbU3+8eLiGTUda/o3X0M2Ai8AOwFnnH33Wb2mJmtDzZ7HKgCNpvZTjPbMmMV34DGSDlzqkrZoefpRaQApXJFj7tvBbZOWvblpOl7UzjGXwJ/ObXy0sPMiEU14pSIFKa87hmbLNYcofPUICf7h6+/sYhIHimYoF/dPNFO35vZQkREbrKCCfr3NtQSVscpESlABRP0ZcUhbltYww69slhECkzBBD1ASzTCru5eRtVxSkQKSEEFfaw5wtDoOG8eVccpESkchRX00ToAtdOLSEEpqKBvqCtnbnWpgl5ECkpBBX2i41REQS8iBaWggh4Sz9N3nT5Pzzl1nBKRwlBwQR9rrgPUTi8ihaPggv62hbUUh9RxSkQKR8EFfaLjVK06TolIwSi4oIfE++lf6+5jZEwdp0Qk/xVm0DfXMTw2zt6jZzNdiojIjEsp6M1snZntM7MOM7tshCgz+6KZ7TGz18zsJTNrDpbfbmb/Yma7g3X/Pt2/wI2YGHFK7fQiUgiuG/RmFgKeAO4DVgAPmtmKSZu1kxgmcCXwLPD1YPkg8JvufhuwDvhjM6tLU+03bGFdOfNryjTilIgUhFSu6NcAHe5+wN1HgE3AhuQN3P0Vdx8MZl8lMYA47v6Wu+8Ppo8AJ4D6dBU/HaubI7ohKyIFIZWgbwC6kua7g2VX81ng+ckLzWwNUAK8fYV1D5tZm5m19fT0pFDS9LVE6zjce54TZ4duys8TEcmUtN6MNbNPA60kBgtPXr4A+D7wW+5+2aMu7v6Uu7e6e2t9/c254I81q51eRApDKkF/GGhKmm8Mll3CzO4FHgXWu/tw0vIa4O+AR9391emVmz63LayhJFTEdjXfiEieSyXotwHLzGyxmZUADwBbkjcwsxbgSRIhfyJpeQnwI+B77v5s+sqevtJwiPc01OiGrIjkvesGvbuPARuBF4C9wDPuvtvMHjOz9cFmjwNVwGYz22lmEyeCTwJ3AQ8Fy3ea2e1p/y1uUCwa4fXD6jglIvktnMpG7r4V2Dpp2ZeTpu+9yn4/AH4wnQJnUqw5wtP/9A67j/TREjxbLyKSbwqyZ+yEix2nejNbiIjIDCrooJ9fW0ZDXbmevBGRvFbQQQ+J5+nb9eSNiOSxgg/6WDTCkb4hjvadz3QpIiIzQkE/0XGqszezhYiIzJCCD/oVC2ooDRepnV5E8lbBB31JuIj3NtQq6EUkbxV80EOi+Wb34bMMj8UzXYqISNop6IFYtI6R+DhvHNaIUyKSfxT0XOw41a7mGxHJQwp6YG5NGY0RdZwSkfykoA/EohG2d57B3TNdiohIWinoA7FoHcfPDnOkTyNOiUh+UdAHLnacUvONiOQXBX3g1gU1lBWr45SI5J+Ugt7M1pnZPjPrMLNHrrD+i2a2x8xeM7OXzKw5ad1nzGx/8Ocz6Sw+nYpDRaxsqNMri0Uk71w36M0sBDwB3AesAB40sxWTNmsHWt19JfAs8PVg31nAV4D3AWuAr5hZ1o7w0dJcx54jfQyNquOUiOSPVK7o1wAd7n7A3UeATcCG5A3c/RV3HwxmXyUxgDjAR4EX3f20u58BXgTWpaf09FsdjTAad9443JfpUkRE0iaVoG8AupLmu4NlV/NZ4Pkb3DejLtyQVTu9iOSRlMaMTZWZfRpoBT40xf0eBh4GiEaj6SxpSuZUlRKdVcF2PXkjInkklSv6w0BT0nxjsOwSZnYv8Ciw3t2Hp7Kvuz/l7q3u3lpfX59q7TMiFk3ckFXHKRHJF6kE/TZgmZktNrMS4AFgS/IGZtYCPEki5E8krXoB+IiZRYKbsB8JlmWtWHOEnnPDdJ/RiFMikh+uG/TuPgZsJBHQe4Fn3H23mT1mZuuDzR4HqoDNZrbTzLYE+54GvkriZLENeCxYlrUmXnCmdnoRyRcptdG7+1Zg66RlX06avvca+34H+M6NFnizvXt+NeXFIdoP9bLh9qy9bywikjL1jJ0kHCpiZaNGnBKR/KGgv4LVzRH2HDmrjlMikhcU9FcQi0YYG3de61bHKRHJfQr6K2iJ1gHoeXoRyQsK+iuYXVXKotkVaqcXkbygoL+KWDRC+yGNOCUiuU9BfxUtzRFO9o/QdVodp0QktynoryIWtNOr+UZEcp2C/ipumVdNRUlIQS8iOU9BfxXhUBG3N9Up6EUk5ynoryEWjbD36DkGR8YyXYqIyA1T0F9DrLmO+Lizq0sdp0Qkdynor6GlSW+yFJHcp6C/hkhlCUvmVNKuoBeRHKagv46WaEQjTolITlPQX0esuY7TAyN0nhrMdCkiIjckpaA3s3Vmts/MOszskSusv8vMdpjZmJl9YtK6r5vZbjPba2Z/amaWruJvBo04JSK57rpBb2Yh4AngPmAF8KCZrZi02SHgIeCHk/b9JeBOYCXwHuAO4EPTrvomWj6vmqrSsIJeRHJWKkMJrgE63P0AgJltAjYAeyY2cPeDwbrxSfs6UAaUAAYUA8enXfVNFCqyRMepzt5MlyIickNSabppALqS5ruDZdfl7v8CvAIcDf684O57J29nZg+bWZuZtfX09KRy6JsqFq3jzWNn6R9WxykRyT0zejPWzJYCtwKNJE4Oa83sg5O3c/en3L3V3Vvr6+tnsqQb0tIcYdzhta7eTJciIjJlqQT9YaApab4xWJaKfwu86u797t4PPA98YGolZl5MHadEJIelEvTbgGVmttjMSoAHgC0pHv8Q8CEzC5tZMYkbsZc13WS72opi3lVfyY5DvZkuRURkyq4b9O4+BmwEXiAR0s+4+24ze8zM1gOY2R1m1g3cDzxpZruD3Z8F3gZeB3YBu9z9/8zA7zHjNOKUiOSqVJ66wd23AlsnLfty0vQ2Ek06k/eLA789zRqzwurmCJu3d/POyQGW1FdluhwRkZSpZ2yKYs0T7fS9mS1ERGSKFPQpWlpfRXWZOk6JSO5R0Keo6ELHKQW9iOQWBf0UxKIR9h0/x7mh0UyXIiKSMgX9FMSaI7ijEadEJKco6Kfg9qY6QB2nRCS3KOinoLa8mGVzqxT0IpJTFPRTtLo5QvuhXsbH1XFKRHKDgn6KYtEIfedHOXByINOliIikREE/RbHmOkDt9CKSOxT0U7RkThU1ZWE9Ty8iOUNBP0VFRUZLNKIrehHJGQr6GxCLRth/op+z6jglIjlAQX8DYs11uMNOveBMRHKAgv4G3N5Uh5luyIpIbkgp6M1snZntM7MOM3vkCuvvMrMdZjZmZp+YtC5qZj81s71mtsfMFqWp9oypLivmlnnVemWxiOSE6wa9mYWAJ4D7gBXAg2a2YtJmh4CHgB9e4RDfAx5391uBNcCJ6RScLVqCEafUcUpEsl0qV/RrgA53P+DuI8AmYEPyBu5+0N1fA8aTlwcnhLC7vxhs1+/ug+kpPbNi0TrODY3xdk9/pksREbmmVIK+AehKmu8OlqViOdBrZs+ZWbuZPR58Q7iEmT1sZm1m1tbT05PioTNrYsSp7XqeXkSy3EzfjA0DHwS+BNwBLCHRxHMJd3/K3VvdvbW+vn6GS0qPJXMqqaso1g1ZEcl6qQT9YaApab4xWJaKbmBn0OwzBvwYiE2pwixlZrQ01emGrIhkvVSCfhuwzMwWm1kJ8ACwJcXjbwPqzGziMn0tsGfqZWanWDRCx4l++gbVcUpEstd1gz64Et8IvADsBZ5x991m9piZrQcwszvMrBu4H3jSzHYH+8ZJNNu8ZGavAwb8+cz8KjffRDt9e5eab0Qke4VT2cjdtwJbJy37ctL0NhJNOlfa90Vg5TRqzFqrmuooMthxqJe7b5mb6XJERK5IPWOnoao0zC3za2jXDVkRyWIK+mmKRevYeaiXuDpOiUiWUtBPUywa4dzwGPtPnMt0KSIiV6Sgn6aJG7I7OnszW4iIyFUo6Kdp0ewKZlWWqOOUiGQtBf00Xew4paAXkeykoE+DWHOEAz0D9A6OZLoUEZHLKOjTIBYNOk7pdQgikoUU9GmwqqmWUJGp+UZEspKCPg0qSsK8e361gl5EspKCPk1i0Yg6TolIVlLQp0msuY6BkTj7jqnjlIhkFwV9mkzckFXzjYhkGwV9mkRnVTBbHadEJAsp6NPEzGiJRvSIpYhknZSC3szWmdk+M+sws0eusP4uM9thZmNm9okrrK8xs24z+1Y6is5Wq5sjvHNygNMD6jglItnjukFvZiHgCeA+YAXwoJmtmLTZIRKDfv/wKof5KvAPN15mbohF6wD0fnoRySqpXNGvATqCAb5HgE3AhuQN3P2gu78GjE/e2cxWA/OAn6ah3qy2srGOsDpOiUiWSSXoG4CupPnuYNl1mVkR8Eckxo3Ne+UlIW5dUMP2TgW9iGSPmb4Z+zlgq7t3X2sjM3vYzNrMrK2np2eGS5pZsWgdu7r6GItf9uVGRCQjUgn6w0BT0nxjsCwVHwA2mtlB4BvAb5rZ1yZv5O5PuXuru7fW19eneOjsFGuOcH40zmM/2cOJs0OZLkdEhHAK22wDlpnZYhIB/wDwqVQO7u6/MTFtZg8Bre5+2VM7+eSjt83n12ON/ODVTjZt6+JTa6L89oeWsKC2PNOliUiBuu4VvbuPARuBF4C9wDPuvtvMHjOz9QBmdoeZdQP3A0+a2e6ZLDqblRWH+KNPruLl37ubX7t9IT94tZMPff3vefRHr9N9ZjDT5YlIATL37HoJV2trq7e1tWW6jLTpOj3It3/2NpvbunCHX4818rkPv4vm2ZWZLk1E8oiZbXf31iuuU9DfHEd6z/NnP3ubTdu6iI87G25fyMYPL2VJfVWmSxORPKCgzyLHzw7x5M8O8MNfdDIyNs7HVy7k82uXsmxedaZLE5EcpqDPQj3nhnn6Hw/w/Vc7OT8a52PvWcDGtUu5dUFNpksTkRykoM9ipwdG+It/OsBf/byT/uExPrJiHl+4ZxnvaajNdGkikkMU9Dmgd3CE7/7zQb7zz+9wbmiMte+ey+fXLqUleM+9iMi1KOhzyNmhUb7384M8/U/v0Ds4ygeXzeF371lG66JZmS5NRLKYgj4H9Q+P8f1/6eTpfzzAqYERPrBkNl+4ZxnvXzILM8t0eSKSZRT0OWxwZIwf/ush/uxnBzjZP8yaRbP4wj3LuHPpbAW+iFygoM8DQ6NxNv0iEfjHzg7REq3jC/cs4+7l9Qp8EVHQ55PhsTib27r59t+/zeHe86xsrOXza5dx761zFfgiBUxBn4dGxsb5UXs333qlg67T57l1QQ1fWLuUj942n6IiBb5IoVHQ57HR+Dj/e+cRnnilg3dODrB8XhUb1y7jV967gJACX6RgKOgLQHzc+clrR/jmyx10nOhnSX0ln7t7KR9fuYCy4lCmyxORGaagLyDj487zbxzjmy/v581j56guC7N+1ULub21iVWOt2vFF8pSCvgCNjzuvHjjF5u3dPP/GUYZGx1k2t4pPtjbxay0N1FeXZrpEEUkjBX2BOzs0yk92HWXz9i7aD/USLjLuvmUun2xt5MPvnktxaKaHDhaRmTbtoDezdcCfACHgaXf/2qT1dwF/DKwEHnD3Z4PltwPfBmqAOPCH7v6/rvWzFPQzq+PEOTa3dfO3Ow5zsn+YOVUl/NrtDdzf2sQt8/WqZJFcNa2gN7MQ8Bbwy0A3iTFkH3T3PUnbLCIR5l8CtiQF/XLA3X2/mS0EtgO3unvv1X6egv7mGI2P87N9PWze3sVLe08wNu6saqzlE61NrF+1kNry4kyXKCJTcK2gT2Vw8DVAh7sfCA62CdgAXAh6dz8YrBtP3tHd30qaPmJmJ4B6oHdqv4KkW3GoiHtXzOPeFfM41T/Mj3ceYXNbF//9x2/wBz/Zw0dvm8/9rY3c+a45ei5fJMelEvQNQFfSfDfwvqn+IDNbA5QAb19h3cPAwwDRaHSqh5Zpml1Vymf/zWL+452LeOPwWTZv7+LH7YfZsusIC2vL+MTqRj6xuono7IpMlyoiNyCVoJ82M1sAfB/4jLuPT17v7k8BT0Gi6eZm1CSXMzPe21jLextr+f2P3cqLe46zeXs333ylgz99uYP3L5nF/aubuO+986kouSn/dEQkDVL533oYaEqabwyWpcTMaoC/Ax5191enVp5kSllxiF9dtZBfXbWQI73neW5HN5u3d/N7m3fxlS27+fjKBdzf2kgsGtGz+SJZLpWg3wYsM7PFJAL+AeBTqRzczEqAHwHfm7hBK7lnYV05G9cu43c+vJRtB8/wTFsXW3YdYdO2LpbUV3L/6ib+XayBeTVlmS5VRK4g1ccrP0bi8ckQ8B13/0Mzewxoc/ctZnYHiUCPAEPAMXe/zcw+DXwX2J10uIfcfefVfpaeuskN/cNjbH0t8Wz+toNnKDL40PJ67m9t4p5b51Ia1msXRG4mdZiSGXWgp59nt3fz3I7DHDs7RKSimA23N3B/ayO3LdQg5yI3g4Jebor4uPOP+3vYvL2bF3cfZyQ+zm0La1j77rnUlBVTURqiqjRMRUmYytIQlSVhKkuD6dIwlSVhvXFT5AZN9zl6kZSEglcr3H3LXM4MjLBl1xE2b+/imy93pHyMsuKiCyeAipLgxFAapqo0REVJODhRTJwYgr8n/pRcPGFMnDxKw0W6WSwFT1f0MuPi48750TgDw2P0D48xOBxP/D0SzI8k1g0MxxkYGQumxxiYWB78PRjsPzASJz6e2r/bUJFdOGHMqixhQW0Z82vLWFBbfmF6YW0582vL9DpnyWm6opeMChUZVaWJq/F5aTieuzM8Nn7xBHHh5BCfdIK4dPmpgRG6z5ynrfMMvYOjlx03UlF86Qmgrpz5NWUsqC1jQTBdXqKTgeQeBb3kHDOjrDhEWXGIWZUlN3SM8yNxjvad51jfEEf7hjjadz74e4gjfUPsOHSGM1c4GdQlnQwm/syvLWdh0jcFnQwk2yjopSCVl4RYUl/Fkvqqq25zfiTOsbPBSaB36JLpo31DtF/jZJD8TWBBTdI3hNoymiIVlIT1ami5eRT0IldRXhJi8ZxKFs+pvOo2Q6NxjvUNcWTSt4OJ6V3dfZweGLlkn3CRsWhOJbfMq2bZvCqWz6tm+bxqFs2uIKyxAWQGKOhFpqGsOMSiOZUsSuFkcLRviCO95zlwsp99x/p540gfW984ysTzECWhIpbUVwbBf/EE0DSrQo+dyrQo6EVm2LVOBudH4nSc6Oet4+cu/NneeYYtu44k7V/E0rlVLJ9bzfL5iZPAsrnVNNSV6xXSkhIFvUgGlZeELrwxNFn/8Bj7j59j//F+9gUngJ+/fYrn2i++T7CyJMTSedXcElz9L5tXzS3zqplXU6q+A3IJBb1IFqoqDdMSjdASjVyyvG9wlP0nzvHW8YvfAl5+s4dn2rovbFNdFg7a/xNX/xPTc6pKdAIoUAp6kRxSW1FM66JZtC6adcny0wMjvHX8HPuPnwu+AfTz/BtH+ZtfXHwqKFJRfKHd/131lRSHixiLO2Pjzlh8nLFxJ540nVjuxMfHGR134nFndHw8sU2wXTxpu7Hx8QvHi487o8nrx8eD/S/9GUVmLKwro3l2Jc2zKmieXUE0mG6IlGvg+jRR0IvkgVmVJbx/yWzev2T2hWXuTk//cKL559i5C98Eftx+mHPDY9c8XnHICBUZxUVFhEJGuKiIcFGwLFgXLioiHDLCRUY4VESoyCgJF1ERSmybWG6EioooDvYNB8eaOM5o3Ok+c57OUwP8w1s9DI9dHJcoVGQ01JUnwn/iJDCrkubZiWkNfpM6fVIiecrMmFtdxtzqMu5cOufCcnfnZP8I7n4hoMOTQjgTxsedE+eG6Tw1QOfpQQ6dGqTz9CCdpwb4yWtH6Tt/aZ+F+upSmmdVEJ1dQXPSCaB5diWRimI1UyVR0IsUGDOjvro002VcpqjImB/0MH5f0jeTCX2Do3SeHqDz1CCHTg9y8GTihPDzjlM8d/bSQe+qS8OJE0DwLWDR7OCEMLuSBTVlBfe0UkpBb2brgD8hMfDI0+7+tUnr7yIxMMlK4IHk0aTM7DPAfwtm/8Dd/yoNdYtIgamtKGZlRR0rG+suWzc0Gqfr9CCdpwY5eGqAQ8H03qPneHHPcUbjF1+CVxIqonFWeXBPIPFNYEFtOZWlISpKQpQXJ95+Wl6SeGNqRXEo508M1w16MwsBTwC/DHQD28xsi7vvSdrsEPAQ8KVJ+84CvgK0Ag5sD/Y9k57yRUQSfRWWBU8XTRYfd470nk98Czg1kGgSCpqFfvHOaQZG4ikcv4iKkjDlxaHgJJA4AVSUhKgoTUwnTgyJ12OXB+vKSxLjLkxMVwTTFcF0WfHNeY12Klf0a4AOdz8AYGabgA3AhaB394PBuvFJ+34UeNHdTwfrXwTWAX8z7cpFRFIQKjKaZlXQNKviknsVkLhfcWpghGN9QwyOxBkcGeP8SPzC9OCk6Yl1A8F2R/tGL7yC+/xInMHR1F+hDWBGcJJInABWNtbyrU/F0v0RpBT0DUBX0nw38L4Uj3+lfRsmb2RmDwMPA0Sj0RQPLSIyPWbGnKpS5lSl556FuzMSH2dwOBH650cmxluIc3406cQxPBasv/RE0lBXnpY6JsuKm7Hu/hTwFCQGHslwOSIiN8TMKA2HKA2HiFx/85smld4Ih4GmpPnGYFkqprOviIikQSpBvw1YZmaLzawEeADYkuLxXwA+YmYRM4sAHwmWiYjITXLdoHf3MWAjiYDeCzzj7rvN7DEzWw9gZneYWTdwP/Ckme0O9j0NfJXEyWIb8NjEjVkREbk5NDi4iEgeuNbg4HpjkIhInlPQi4jkOQW9iEieU9CLiOS5rLsZa2Y9QOc0DjEHOJmmcnKdPotL6fO4lD6Pi/Lhs2h29/orrci6oJ8uM2u72p3nQqPP4lL6PC6lz+OifP8s1HQjIpLnFPQiInkuH4P+qUwXkEX0WVxKn8el9HlclNefRd610YuIyKXy8YpeRESSKOhFRPJc3gS9ma0zs31m1mFmj2S6nkwysyYze8XM9pjZbjP73UzXlGlmFjKzdjP7SaZryTQzqzOzZ83sTTPba2YfyHRNmWRm/zn4f/KGmf2NmZVluqZ0y4ugTxrA/D5gBfCgma3IbFUZNQb8nruvAN4P/E6Bfx4Av0viNdsCfwL8X3d/N7CKAv5czKwB+ALQ6u7vAUIkxtzIK3kR9CQNYO7uI8DEAOYFyd2PuvuOYPocif/Il43VWyjMrBH4FeDpTNeSaWZWC9wF/AWAu4+4e29Gi8q8MFBuZmGgAjiS4XrSLl+CPqVByAuRmS0CWoB/zXApmfTHwH8BxjNcRzZYDPQA3w2asp42s8pMF5Up7n4Y+AZwCDgK9Ln7TzNbVfrlS9DLFZhZFfC3wH9y97OZricTzOzjwAl3357pWrJEGIgB33b3FmAAKNh7WsEQpxtInAAXApVm9unMVpV++RL0GoR8EjMrJhHyf+3uz2W6ngy6E1hvZgdJNOmtNbMfZLakjOoGut194hvesySCv1DdC7zj7j3uPgo8B/xShmtKu3wJ+ukMYJ53zMxItMHudff/kel6Msnd/6u7N7r7IhL/Ll5297y7YkuVux8DuszslmDRPcCeDJaUaYeA95tZRfD/5h7y8OZ0ONMFpIO7j5nZxADmIeA77r47w2Vl0p3AfwBeN7OdwbLfd/etmStJssjngb8OLooOAL+V4Xoyxt3/1cyeBXaQeFqtnTx8HYJegSAikufypelGRESuQkEvIpLnFPQiInlOQS8ikucU9CIieU5BLyKS5xT0IiJ57v8Dt3CeYUhTS1EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# convert the training history to a dataframe\n",
    "history_df = pd.DataFrame(history.history)\n",
    "# use Pandas native plot method\n",
    "history_df['loss'].plot();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7545fb3a",
   "metadata": {
    "papermill": {
     "duration": 0.031308,
     "end_time": "2022-05-05T17:51:18.156210",
     "exception": false,
     "start_time": "2022-05-05T17:51:18.124902",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Notice how the loss levels off as the epochs go by. When the loss curve becomes horizontal like that, it means the model has learned all it can and there would be no reason continue for additional epochs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "590cd28b",
   "metadata": {
    "papermill": {
     "duration": 0.031178,
     "end_time": "2022-05-05T17:51:18.221255",
     "exception": false,
     "start_time": "2022-05-05T17:51:18.190077",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Your Turn #\n",
    "\n",
    "Now, [**use stochastic gradient descent**](https://www.kaggle.com/kernels/fork/11887330) to train your network."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89455fd7",
   "metadata": {
    "papermill": {
     "duration": 0.020212,
     "end_time": "2022-05-05T17:51:18.262633",
     "exception": false,
     "start_time": "2022-05-05T17:51:18.242421",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/intro-to-deep-learning/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "cell_metadata_filter": "-all",
   "formats": "ipynb"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 20.247301,
   "end_time": "2022-05-05T17:51:21.181650",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-05-05T17:51:00.934349",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}