Hws/hw12/hw12.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [],
   "source": [
    "# Initialize Otter\n",
    "import otter\n",
    "grader = otter.Notebook(\"hw12.ipynb\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 12: Spam/Ham Classification\n",
    "\n",
    "## Feature Engineering, Logistic Regression\n",
    "## Due Date: Monday, November 29, 11:59 PM PDT\n",
    "\n",
    "**Collaboration Policy**\n",
    "\n",
    "Data science is a collaborative activity. While you may talk with others about\n",
    "the homework, we ask that you **write your solutions individually**. If you do\n",
    "discuss the assignments with others please **include their names** at the top\n",
    "of your notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Collaborators**: *list collaborators here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "proj2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "## This Assignment\n",
    "In this homework, you will use what you've learned in class to create a classifier that can distinguish spam (junk or commercial or bulk) emails from ham (non-spam) emails. In addition to providing some skeleton code to fill in, we will evaluate your work based on your model's accuracy and your written responses in this notebook.\n",
    "\n",
    "After this homework, you should feel comfortable with the following:\n",
    "\n",
    "- Feature engineering with text data\n",
    "- Using `sklearn` libraries to process data and fit models\n",
    "- Validating the performance of your model and minimizing overfitting\n",
    "- Generating and analyzing precision-recall curves\n",
    "\n",
    "## Warning\n",
    "This is a **real world** dataset – the emails you are trying to classify are actual spam and legitimate emails. As a result, some of the spam emails may be in poor taste or be considered inappropriate. We think the benefit of working with realistic data outweighs these innapropriate emails, and wanted to give a warning at the beginning of the homework so that you are made aware."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to suppress all FutureWarnings\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=FutureWarning)\n",
    "\n",
    "# more readable exceptions\n",
    "%pip install --quiet iwut\n",
    "%load_ext iwut\n",
    "%wut on"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Score Breakdown\n",
    "Question | Points\n",
    "--- | ---\n",
    "1 | 2\n",
    "2 | 3\n",
    "3 | 3\n",
    "4 | 2\n",
    "5 | 2\n",
    "6a | 1\n",
    "6b | 1\n",
    "6c | 2\n",
    "6d | 2\n",
    "6e | 1\n",
    "6f | 3\n",
    "Total | 22"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "p1",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Part I - Initial Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:41.341673Z",
     "start_time": "2019-04-03T20:17:41.330307Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "imports",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "import seaborn as sns\n",
    "sns.set(style = \"whitegrid\", \n",
    "        color_codes = True,\n",
    "        font_scale = 1.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "loading",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Loading in the Data\n",
    "\n",
    "In email classification, our goal is to classify emails as spam or not spam (referred to as \"ham\") using features generated from the text in the email. \n",
    "\n",
    "The dataset consists of email messages and their labels (0 for ham, 1 for spam). Your labeled training dataset contains 8348 labeled examples, and the unlabeled test set contains 1000 unlabeled examples.\n",
    "\n",
    "Run the following cells to load in the data into DataFrames.\n",
    "\n",
    "The `train` DataFrame contains labeled data that you will use to train your model. It contains four columns:\n",
    "\n",
    "1. `id`: An identifier for the training example\n",
    "1. `subject`: The subject of the email\n",
    "1. `email`: The text of the email\n",
    "1. `spam`: 1 if the email is spam, 0 if the email is ham (not spam)\n",
    "\n",
    "The `test` DataFrame contains 1000 unlabeled emails. You will predict labels for these emails and submit your predictions to the autograder for evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile('spam_ham_data.zip') as item:\n",
    "    item.extractall()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subject</th>\n",
       "      <th>email</th>\n",
       "      <th>spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Subject: A&amp;L Daily to be auctioned in bankrupt...</td>\n",
       "      <td>url: http://boingboing.net/#85534171\\n date: n...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Subject: Wired: \"Stronger ties between ISPs an...</td>\n",
       "      <td>url: http://scriptingnews.userland.com/backiss...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Subject: It's just too small                  ...</td>\n",
       "      <td>&lt;html&gt;\\n &lt;head&gt;\\n &lt;/head&gt;\\n &lt;body&gt;\\n &lt;font siz...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Subject: liberal defnitions\\n</td>\n",
       "      <td>depends on how much over spending vs. how much...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Subject: RE: [ILUG] Newbie seeks advice - Suse...</td>\n",
       "      <td>hehe sorry but if you hit caps lock twice the ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                            subject  \\\n",
       "0   0  Subject: A&L Daily to be auctioned in bankrupt...   \n",
       "1   1  Subject: Wired: \"Stronger ties between ISPs an...   \n",
       "2   2  Subject: It's just too small                  ...   \n",
       "3   3                      Subject: liberal defnitions\\n   \n",
       "4   4  Subject: RE: [ILUG] Newbie seeks advice - Suse...   \n",
       "\n",
       "                                               email  spam  \n",
       "0  url: http://boingboing.net/#85534171\\n date: n...     0  \n",
       "1  url: http://scriptingnews.userland.com/backiss...     0  \n",
       "2  <html>\\n <head>\\n </head>\\n <body>\\n <font siz...     1  \n",
       "3  depends on how much over spending vs. how much...     0  \n",
       "4  hehe sorry but if you hit caps lock twice the ...     0  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_training_data = pd.read_csv('train.csv')\n",
    "test = pd.read_csv('test.csv')\n",
    "\n",
    "# Convert the emails to lower case as a first step to processing the text\n",
    "original_training_data['email'] = original_training_data['email'].str.lower()\n",
    "test['email'] = test['email'].str.lower()\n",
    "\n",
    "original_training_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-34476156ed73b800",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "First, let's check if our data contains any missing values. We have filled in the cell below to print the number of NaN values in each column. If there are NaN values, we replace them with appropriate filler values (i.e., NaN values in the `subject` or `email` columns will be replaced with empty strings). Finally, we print the number of NaN values in each column after this modification to verify that there are no NaN values left.\n",
    "\n",
    "Note that while there are no NaN values in the `spam` column, we should be careful when replacing NaN labels. Doing so without consideration may introduce significant bias into our model when fitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.203231Z",
     "start_time": "2019-04-03T20:17:42.185104Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-b1fb39d9b651ca1b",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before imputation:\n",
      "id         0\n",
      "subject    6\n",
      "email      0\n",
      "spam       0\n",
      "dtype: int64\n",
      "------------\n",
      "After imputation:\n",
      "id         0\n",
      "subject    0\n",
      "email      0\n",
      "spam       0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print('Before imputation:')\n",
    "print(original_training_data.isnull().sum())\n",
    "original_training_data = original_training_data.fillna('')\n",
    "print('------------')\n",
    "print('After imputation:')\n",
    "print(original_training_data.isnull().sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "q1",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 1\n",
    "\n",
    "In the cell below, we have printed the text of the `email` field for the first ham and the first spam email in the original training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.247245Z",
     "start_time": "2019-04-03T20:17:42.228451Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q1-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "url: http://boingboing.net/#85534171\n",
      " date: not supplied\n",
      " \n",
      " arts and letters daily, a wonderful and dense blog, has folded up its tent due \n",
      " to the bankruptcy of its parent company. a&l daily will be auctioned off by the \n",
      " receivers. link[1] discuss[2] (_thanks, misha!_)\n",
      " \n",
      " [1] http://www.aldaily.com/\n",
      " [2] http://www.quicktopic.com/boing/h/zlfterjnd6jf\n",
      " \n",
      " \n",
      "\n",
      "<html>\n",
      " <head>\n",
      " </head>\n",
      " <body>\n",
      " <font size=3d\"4\"><b> a man endowed with a 7-8\" hammer is simply<br>\n",
      "  better equipped than a man with a 5-6\"hammer. <br>\n",
      " <br>would you rather have<br>more than enough to get the job done or fall =\n",
      " short. it's totally up<br>to you. our methods are guaranteed to increase y=\n",
      " our size by 1-3\"<br> <a href=3d\"http://209.163.187.47/cgi-bin/index.php?10=\n",
      " 004\">come in here and see how</a>\n",
      " </body>\n",
      " </html>\n",
      " \n",
      " \n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "first_ham = original_training_data.loc[original_training_data['spam'] == 0, 'email'].iloc[0]\n",
    "first_spam = original_training_data.loc[original_training_data['spam'] == 1, 'email'].iloc[0]\n",
    "print(first_ham)\n",
    "print(first_spam)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "Discuss one thing you notice that is different between the two emails that might relate to the identification of spam.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q1\n",
    "manual: True\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-78513403ef52a957",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "## Training-Validation Split\n",
    "The training data we downloaded is all the data we have available for both training models and **validating** the models that we train.  We therefore need to split the training data into separate training and validation datsets.  You will need this **validation data** to assess the performance of your classifier once you are finished training. Note that we set the seed (random_state) to 42. This will produce a pseudo-random sequence of random numbers that is the same for every student. **Do not modify this in the following questions, as our tests depend on this random seed.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.317970Z",
     "start_time": "2019-04-03T20:17:42.294532Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "cell-873194ed3e686dfb",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# This creates a 90/10 train-validation split on our labeled data\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, val = train_test_split(original_training_data, test_size = 0.1, random_state = 42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "feat-eng",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Basic Feature Engineering\n",
    "\n",
    "We would like to take the text of an email and predict whether the email is ham or spam. This is a *classification* problem, so we can use logistic regression to train a classifier. Recall that to train a logistic regression model we need a numeric feature matrix $X$ and a vector of corresponding binary labels $y$.  Unfortunately, our data are text, not numbers. To address this, we can create numeric features derived from the email text and use those features for logistic regression.\n",
    "\n",
    "Each row of $X$ is an email. Each column of $X$ contains one feature for all the emails. We'll guide you through creating a simple feature, and you'll create more interesting ones as you try to increase the accuracy of your model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q2",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 2\n",
    "\n",
    "Create a function called `words_in_texts` that takes in a list of `words` and a pandas Series of email `texts`. It should output a 2-dimensional NumPy array containing one row for each email text. The row should contain either a 0 or a 1 for each word in the list: 0 if the word doesn't appear in the text and 1 if the word does. For example:\n",
    "\n",
    "```\n",
    ">>> words_in_texts(['hello', 'bye', 'world'], \n",
    "                   pd.Series(['hello', 'hello worldhello']))\n",
    "\n",
    "array([[1, 0, 0],\n",
    "       [1, 0, 1]])\n",
    "```\n",
    "\n",
    "*The provided tests make sure that your function works correctly, so that you can use it for future questions.*\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q2\n",
    "points: 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.337281Z",
     "start_time": "2019-04-03T20:17:42.320567Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q2-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    },
    "tags": [
     "student"
    ]
   },
   "outputs": [],
   "source": [
    "def words_in_texts(words, texts):\n",
    "    '''\n",
    "    Args:\n",
    "        words (list): words to find\n",
    "        texts (Series): strings to search in\n",
    "    \n",
    "    Returns:\n",
    "        NumPy array of 0s and 1s with shape (n, p) where n is the\n",
    "        number of texts and p is the number of words.\n",
    "    '''\n",
    "    indicator_array = np.array(texts.apply(lambda s: [word in s for word in words]).to_list(), \n",
    "                               dtype=np.int64)     # boolean -> int64\n",
    "    return indicator_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0, 0],\n",
       "       [1, 0, 1]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "words_in_texts(['hello', 'bye', 'world'], pd.Series(['hello', 'hello worldhello']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q2</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q2 results: All test cases passed!"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "eda",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Basic EDA\n",
    "\n",
    "We need to identify some features that allow us to distinguish spam emails from ham emails. One idea is to compare the distribution of a single feature in spam emails to the distribution of the same feature in ham emails. If the feature is itself a binary indicator, such as whether a certain word occurs in the text, this amounts to comparing the proportion of spam emails with the word to the proportion of ham emails with the word.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "q3a",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "The following plot (which was created using `sns.barplot`) compares the proportion of emails in each class containing a particular set of words. \n",
    "\n",
    "![training conditional proportions](images/training_conditional_proportions.png)\n",
    "\n",
    "You can use DataFrame's `.melt` method to \"unpivot\" a DataFrame. See the following code cell for an example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:42.428419Z",
     "start_time": "2019-04-03T20:17:42.386697Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "> Our Original DataFrame has a `type` column and some columns corresponding to words. You can think of each row as a sentence, and the value of 1 or 0 indicates the number of occurences of the word in this sentence."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_1</th>\n",
       "      <th>word_2</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>ham</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   word_1  word_2  type\n",
       "0       1       0  spam\n",
       "1       0       1   ham\n",
       "2       1       0   ham\n",
       "3       0       1   ham"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "> `melt` will turn columns into entries in a variable column. Notice how `word_1` and `word_2` become entries in `variable`; their values are stored in the value column."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>type</th>\n",
       "      <th>variable</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>spam</td>\n",
       "      <td>word_1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>spam</td>\n",
       "      <td>word_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>ham</td>\n",
       "      <td>word_2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   type variable  value\n",
       "0  spam   word_1      1\n",
       "1   ham   word_1      0\n",
       "2   ham   word_1      1\n",
       "3   ham   word_1      0\n",
       "4  spam   word_2      0\n",
       "5   ham   word_2      1\n",
       "6   ham   word_2      0\n",
       "7   ham   word_2      1"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import display, Markdown\n",
    "df = pd.DataFrame({\n",
    "    'word_1': [1, 0, 1, 0],\n",
    "    'word_2': [0, 1, 0, 1],\n",
    "    'type': ['spam', 'ham', 'ham', 'ham']\n",
    "})\n",
    "display(Markdown(\"> Our Original DataFrame has a `type` column and some columns corresponding to words. You can think of each row as a sentence, and the value of 1 or 0 indicates the number of occurences of the word in this sentence.\"))\n",
    "display(df);\n",
    "display(Markdown(\"> `melt` will turn columns into entries in a variable column. Notice how `word_1` and `word_2` become entries in `variable`; their values are stored in the value column.\"))\n",
    "display(df.melt(\"type\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "### Question 3\n",
    "\n",
    "Create a bar chart like the one above comparing the proportion of spam and ham emails containing certain words. Choose a set of words that are different from the ones above, but also have different proportions for the two classes. Make sure to only consider emails from `train`.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q3\n",
    "manual: True\n",
    "format: image\n",
    "points: 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>subject</th>\n",
       "      <th>email</th>\n",
       "      <th>spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7657</th>\n",
       "      <td>7657</td>\n",
       "      <td>Subject: Patch to enable/disable log\\n</td>\n",
       "      <td>while i was playing with the past issues, it a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6911</th>\n",
       "      <td>6911</td>\n",
       "      <td>Subject: When an engineer flaps his wings\\n</td>\n",
       "      <td>url: http://diveintomark.org/archives/2002/10/...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6074</th>\n",
       "      <td>6074</td>\n",
       "      <td>Subject: Re: [Razor-users] razor plugins for m...</td>\n",
       "      <td>no, please post a link!\\n \\n fox\\n ----- origi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4376</th>\n",
       "      <td>4376</td>\n",
       "      <td>Subject: NYTimes.com Article: Stop Those Press...</td>\n",
       "      <td>this article from nytimes.com \\n has been sent...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5766</th>\n",
       "      <td>5766</td>\n",
       "      <td>Subject: What's facing FBI's new CIO? (Tech Up...</td>\n",
       "      <td>&lt;html&gt;\\n &lt;head&gt;\\n &lt;title&gt;tech update today&lt;/ti...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                            subject  \\\n",
       "7657  7657             Subject: Patch to enable/disable log\\n   \n",
       "6911  6911        Subject: When an engineer flaps his wings\\n   \n",
       "6074  6074  Subject: Re: [Razor-users] razor plugins for m...   \n",
       "4376  4376  Subject: NYTimes.com Article: Stop Those Press...   \n",
       "5766  5766  Subject: What's facing FBI's new CIO? (Tech Up...   \n",
       "\n",
       "                                                  email  spam  \n",
       "7657  while i was playing with the past issues, it a...     0  \n",
       "6911  url: http://diveintomark.org/archives/2002/10/...     0  \n",
       "6074  no, please post a link!\\n \\n fox\\n ----- origi...     0  \n",
       "4376  this article from nytimes.com \\n has been sent...     0  \n",
       "5766  <html>\\n <head>\\n <title>tech update today</ti...     0  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:43.145246Z",
     "start_time": "2019-04-03T20:17:42.430406Z"
    },
    "nbgrader": {
     "grade": true,
     "grade_id": "q3a-answer",
     "locked": false,
     "points": 2,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>apply</th>\n",
       "      <th>invest</th>\n",
       "      <th>therapy</th>\n",
       "      <th>calling</th>\n",
       "      <th>buy</th>\n",
       "      <th>spam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   apply  invest  therapy  calling  buy  spam\n",
       "0      0       0        0        0    0     0\n",
       "1      0       0        0        0    0     0\n",
       "2      0       0        0        0    0     0\n",
       "3      0       0        0        0    0     0\n",
       "4      0       0        0        0    0     0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We must do this in order to preserve the ordering of emails to labels for words_in_texts\n",
    "train = train.reset_index(drop=True) \n",
    "\n",
    "words = ['apply', 'invest', 'therapy', 'calling', 'buy']\n",
    "\n",
    "res = words_in_texts(words, train['email'])\n",
    "\n",
    "words_df = pd.DataFrame(res, columns=words)\n",
    "words_df = words_df.reset_index(drop=True)\n",
    "\n",
    "for_plot = pd.concat([words_df, train['spam']], axis=1)\n",
    "for_plot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "👋 Now, we need to `melt` the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEqCAYAAAAh0uZRAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABW/klEQVR4nO3deXwN1/vA8U92S1CJWBL7ckMFiYRQWwhCK63YSaUURe2qIVQVVcS+1RJE09QWtbQILbGUCo1S1dKqBLHFEkKErPP7I787X9dNuCE3IZ736+UVd+bMnGduJve5c+bMOSaKoigIIYQQRmaa3wEIIYR4PUjCEUIIkSck4QghhMgTknCEEELkCUk4Qggh8oQkHCGEEHlCEo4BFi1ahKOj41P/nTlzJr/DfG0oisKsWbNwd3fH2dmZ7777Tq/MyJEjcXR05M6dO3rrxo4di6OjIx9//LHeugcPHvDmm28yevRoo8QOcPToURwdHdm8efML78vR0ZFx48blQlSZEhMTmTt3Lu3ataNu3bq4ubnRo0cP1q9fT3p6eq7VYyxz5szhgw8+AHL/vcltrVq1eubnypo1a/IsnsuXL+Po6MiiRYuyfJ0bzHNtT6+BQYMGUbVq1SzX2dvb53E0r6/9+/ezcuVKPDw8aN26Na6urnplGjRoQHh4OKdOnaJFixY6644ePYqFhQXHjh0jPT0dMzMzdd2pU6dIT0/H3d3d6MeRGwIDA6lYsWKu7CsxMZHu3btz/fp1OnfuTNWqVXn48CGHDh1i0qRJ/PrrryxYsAATE5Ncqc8Yfv31V9q2bZvfYRisZMmSBAQEZLveyckpz2KxsbEhMDAQR0dHo9UhCScH3nrrrVfmg6gg++effwAYPXp0tn8cDRo0ANBLOBcuXODatWv4+PiwZcsW/vrrL+rWrauu//333wFo2LChscLPVe+9916u7Ss0NJT//vuPzZs3U7t2bXV53759mTx5MmvXruXgwYN6CfxlkZCQwN9//82kSZPyOxSDFSlSJFd/hy8iL2KRJjXxyklNTQWgaNGi2ZapUaMGJUuW5I8//tBZHhkZiampKYMGDcLExIQjR47orD9x4gSlS5emSpUquR/4S+7EiRO88cYbOslGS9tMdfLkyTyOynCRkZEUK1YsT68KRM5IwsllrVq14rPPPmP8+PHUqVOH5s2bEx8fD2T+Qfft2xcXFxdcXFz48MMPOXXqlN4+wsPD6dixI3Xr1qVDhw7s27ePfv360bt3b516Hn/9tOWG1NuqVSs+//xztm3bxjvvvEOdOnVo27ZtlvdH/vjjDwYMGECDBg1wd3fno48+Uq865syZg6OjI//995/ONhkZGTRt2pQRI0Y89f2LioqiT58+aqx+fn789ttvOnEuXrwYAE9PT1q1apXlfkxMTHB1deXPP//UWR4ZGUnNmjWpXLkyjo6OREZGqusUReHUqVM6Vzfp6emsXLkSLy8vnJycaNq0KZMmTVJ/p/C/ezJbtmzB29ubOnXqqM0kSUlJTJs2jaZNm+Ls7Mzw4cO5f/++XrzHjh3D19cXNzc3XFxc6NGjBxEREU99r0D/PoWjoyMrVqwgODiY1q1b4+TkhLe3N+Hh4c/cV9GiRbl79y67du3SW1e5cmX+/PNPnd9fq1atmDBhAmFhYXh6euLs7EyPHj103lPIfF/XrVtHly5dcHFxoU6dOrRr144VK1bw+MharVq1YsqUKYSFheHl5UXdunXp3Lkzp06d4ubNm4wYMQIXFxeaNWvGvHnzyMjI0Knn8OHDuLu7Y2qq+7G2Zs0aWrduTZ06dfD29mb37t0661NTU1m+fDnvvvsuzs7O1K1bl3fffZdNmzbpvddBQUGsWLECDw8P6tWrR+/evbl48SIxMTH069cPZ2dnWrVqRUhIyDPf75zQ3k/54YcfmDlzJm+99RYuLi58/PHHxMfHc+rUKXr06EG9evXw8vJi586dOtvfvHmTKVOm4OnpiZOTE66urvj5+XH8+HG9Op52z+aff/6hX79+NGrUiHr16uHj46P3Pj2NJJwcuH//PvHx8Xr/tN+4tXbs2MHZs2eZMGEC3bp1w8bGhsOHD9O7d2/u37/PiBEjGDx4MFevXsXX15eoqCh1261btzJy5EjMzc359NNPadCgAcOHD9f7ADeUofUC/PLLL0ybNg0vLy8CAgIoXLgwU6ZM4cCBA2qZqKgofH19OX/+PP369WPw4MH8999/+Pn5cfnyZby9vQH0PuCOHTvGzZs36dChQ7ax7t27l969e3Pt2jUGDx7M4MGDuXbtGn369GHv3r0AjB8/njZt2gAQEBDA+PHjs91fgwYNuHv3LhcuXAAyP/iOHTumNos2atSI33//nZSUFADOnz9PQkICjRo1UvcxatQoZs2ahUajISAggHbt2rFp0yZ69uzJvXv3dOqbMmUKrq6ufPrpp3h6eqIoCoMGDSI0NBRPT0/GjBlDfHy83o3s6OhoBg4ciKIojBo1ijFjxvDw4UM+/vhjvd+RIdatW8eaNWvo1q0b/v7+JCUlMWrUKP7999+nbte5c2dMTEwYMWIEXbt2ZdmyZZw8eVLtLGBpaam3za+//sqUKVPw8vJixIgRxMfH079/f44dO6aWmT9/Pl988QXVq1cnICCA0aNHY2VlxZw5c9i6davO/vbu3cuCBQvo0qULQ4cOJTo6mmHDhtG3b19MTU0ZN24cGo2GZcuWsW3bNp1tjxw5QuPGjXWW7dq1i+DgYLp168Ynn3zC/fv3GTlyJH/99ZdaJiAggIULF9KwYUMmTJjA0KFDSUpKYsKECTrHAfDtt9/y/fff8+GHH9KnTx9+//13hg0bxgcffICDgwPjxo2jZMmSTJs2TW/brGRkZGT5mRIfH8/du3f1ys+ePZtjx44xdOhQfHx82Lt3L0OGDGHAgAG4uLjg7+8PgL+/PxcvXgTg0aNH+Pr6Eh4ejo+PD5MmTaJHjx6cPn2ajz76iMTExGfGCRAfH0+/fv24ceMGgwcPJiAgAGtrayZMmMCPP/5o0D5QxDMtXLhQ0Wg02f6LjIxUy7Zs2VKpWbOmcvHiRXVZenq64unpqfTo0UNJS0tTlz948EBp06aN8t577ymKoihpaWlK48aNlXbt2inJyclquZCQEEWj0Sjvv/++Tj2Pv85quaH1ardzdHRUzpw5oy67ceOG4ujoqIwePVpd1qVLF6VJkyZKfHy8uiw6OlqpWbOmMnPmTEVRFKVDhw5K+/btdeKaOHGi4urqqnNcj0tNTVWaN2+utGjRQrl//766PCEhQWnWrJnSrFkzJSUlRVGU//0+YmNjs9yX1unTpxWNRqNs3bpVURRFOXv2rKLRaJR9+/YpiqIoe/fuVTQajXLkyBFFURRl48aNikajUX93Bw4cUDQajfLll1/q7Hfnzp2KRqNRAgMDFUVRlMjISL3fj6IoSkREhKLRaJTg4GCd4/zggw8UjUajfP/994qiKMqKFSsUjUaj3L59Wy0XHx+vtG3bVgkJCXnqMWo0GmXs2LE6r52dnZUbN26oy06ePKloNBpl7ty5T92XoihKWFiY4uLionN+N2jQQJk4caISFxenU7Zly5aKRqNRfv75Z3XZ7du3FTc3N6Vbt26KoihKSkqKUr9+fWXUqFE6296/f19xcnJSBg4cqLM/R0dH5ezZs+qymTNnKhqNRhk5cqS67MGDB0rt2rV1zsvY2FhFo9EoMTExeu/FtWvX1GXHjh1TNBqNMn/+fEVR/neOz549Wye+8+fPKxqNRpk6darO/urVq6fcvHlTXTZ8+HBFo9Eos2bNUpdduHDBoPdb+/5l98/V1VXv+Jo3b648fPhQXd65c2dFo9Eoa9euVZcdPnxY0Wg0yoYNGxRFUZQdO3YoGo1GOXjwoE7969atUzQajbJ7926dOhYuXJjla+1+Tp06pe4jOTlZ8fHx0Xv/siOdBnJg7Nix1KxZU2/5k8sqVqyo03Po77//JjY2lp49e5KQkKBTtmXLlqxZs4br169z/fp1bt++zUcffaTzbbJ79+7P1TXR0HrLli0LQJUqVXSOxc7OjlKlSnHr1i0Abt++zZ9//knfvn0pWbKkWq5KlSp8//33lCtXDgBvb2/mzJnDv//+i0ajIS0tjZ9++ok2bdpk+S1ZG+v169cZM2YM1tbW6vLixYvz/vvvM2fOHE6fPo2Li4vBx1+rVi2KFSvGqVOneO+994iMjMTMzAw3Nzcg8wrIzMyMY8eOqVc7ZcuWVX932iatgQMH6uy3ffv2LFiwgD179vDpp5+qy5s2bapT7uDBg5iamtK1a1d1mbm5Ob6+vjr3jrTv/9SpU+nXrx9OTk6ULFlSr+nHUK6urtjZ2em8D5DZrPIsXbp0wcvLi59//pn9+/dz9OhR7t69y4YNG9i9ezdr166lWrVqavmqVavSunVr9bWNjQ3vvfceoaGh3L59G1tbW3799Ve9VoA7d+5gbW1NUlKSzvKKFSvqdATR3kvTXtVC5s1tW1tbneP59ddfsbe3p3Llyjr7q1+/vvr+AtSpUwdAPaft7Ow4fvy4TjOcoiikpaUBmd3kH+fi4kKpUqXU19r6Ho+vfPnyANy4cYNnKVWqFLNmzcpynbm5/sdzs2bNKFSokE79f/7551Prf/vtt2nUqJHO36z2qh7Q+x1kR/s+zpkzh6FDh+Li4oKlpWWOuvdLwsmB2rVrG9RLzdbWVuf1pUuXgMwurIGBgVluc+3aNa5duwZAhQoVdNZZWlrqLTOEofVqTyQbGxu99ZaWlmpb+ZUrV1AUhUqVKumVe/PNN9X/d+jQgblz57Jr1y40Gg2HDx/mzp07anNbVi5fvgyQ5c16bVf0q1ev5ijhmJqa4urqqnYciIyMxMnJSU1oxYoVo1atWmo79smTJ3V+v5cvX6Z48eI6HzBa1apV4+DBgzrLnnz/rly5gq2trV7nhie71rdr146ff/6ZnTt3snPnTuzs7GjRogU+Pj5qcsyJJ+PQJvkn73lkp1ixYnTq1IlOnTqRkZHB77//ztKlSzl06BDTp09n5cqVatnq1avrbV+pUiUURVGP38LCgv3797N3715iYmK4ePGi+gVIeWJ2lCf/drRd1p88JjMzM51tf/31V73mtKz2p/2wfjwBWlpa8sMPP3Do0CEuXLjAxYsX1UTzrPi0SeHx+LQxP7ltVqysrHjrrbeeWe5Z9T++XJs8H6/fxMSEFStWcOLECS5dusSlS5fU98DQ86J+/fr07t2b0NBQjhw5whtvvEHTpk3x9vbGw8PDoH1IwjGCx5/rgP/9QkeMGIGzs3OW21StWlVNOFmxsrIyqO7HH84ztF6tJ2+2Pkm7v2eVs7e3p379+oSHhzN8+HDCw8MpVarUU5P10/44tessLCyeWm9W3NzcWLBgAcnJyURFRdGzZ0+d9e7u7qxbt474+Hj1xq8hMWVkZOjF8+Tv3cTEROeb5OPbPs7CwoKFCxfyzz//8PPPP3Pw4EE2b97Mpk2b+OSTT/joo48MPl549u8nK3FxcXz77bc0b95cp9OEqakpbm5urFixgo4dO6rdxh+P/Unac1CbFD799FO2b9+Oq6srLi4udO/enQYNGqg93x6X1bd64KnP/iiKQmRkJJ999pneume9FykpKfTr14/jx4/j7u5O48aN6dOnDw0bNszyQ/R54stNz1P/lStX6N69O0lJSTRt2pS3336bWrVqoSgKQ4YMyVH9n332GX5+fuzevZuDBw+ye/dutm/fTvfu3ZkyZcqz489RbeK5ODg4AJlNAU9+mzl16hQJCQkUKlRIvTyPiYnR20dsbKxOc4Gpqaneh1laWhp37txRm4QMrddQ2iYz7c3Ix82aNYsSJUqoH44dOnRg8uTJREdHs2/fPry9vfU+kB+njTU6Olpvnfb9eLxpxFANGzYkNTWV8PBw7t27p5f0GjVqxKpVq9ixYweKouisd3Bw4NChQ9y6dUvvKicmJkZ9P7JToUIF9u/fT3x8vM434NjYWJ1yV69e5erVq7i5ueHo6MjQoUO5fv06H3zwAatWrcpxwnkeGRkZBAUFcfv27SyfQTIzM6NKlSrcvn1bZ7n2KvpxFy9exMzMjPLlyxMVFcX27dv5+OOPdXq4paWlcffu3ee6cn/SmTNnuHv3bpZXOM+yc+dOjh07xrRp0+jSpYu6PC4u7oXjelksXryY27dvEx4ervMZYvCN/v9369Ytzp07R+PGjRkwYAADBgzgzp07DBkyhI0bN/Lpp59SrFixp+5DeqnlAScnJ+zs7Pj222912oQTExMZOXIkAQEBmJmZUbNmTSpWrMj69et12lV37dql1x5cqlQpYmJiePTokbosIiKC5OTkHNdrqDJlylCzZk127Nih07MlNjaWkJAQtV0cMu9zWFhYsGjRIu7evfvU3mmQ2VxpZ2fHunXrdPadmJjI2rVrsbOze67nK2rXrk2RIkVYv349FhYW1K9fX2e9q6sr5ubmbNmyBXt7e50PQG2X6+XLl+tss2fPHmJiYp7ZjKBtV1+9erW6TFEU1q5dq1Nu2bJl9OnTR+dDrmzZspQpU+a5rlaeR7ly5XBzc+PHH3/UezYJMpsXDx8+jKenp87yP//8U+fZnFu3bvHDDz/QqFEjSpQoofa0erLpbePGjTx8+FC9V/IiDh8+jEaj0WtuMkR28Wm7NedGfPnt7t27FC5cWGc0lJSUFNavXw9g8JBFmzdvpk+fPjqPGpQsWZJKlSphYmJi0LkqVzh5wMLCgokTJzJy5Eg6depEly5dsLKyIiwsjKtXrzJ79mz1UvmLL75g4MCBdO/enc6dO3Pr1i2+/fZbvaaLDh06MHXqVPr378+7777LxYsX2bhxo3qlkNN6DRUQEED//v3p3LkzXbt2xdTUlNDQUIoXL86AAQPUciVLlqRJkybs3LmT8uXLZ9ukl1WsnTt3Vr9tbtq0iRs3brBw4cLn+vA1NzfHxcWFw4cP4+bmRuHChXXWFy1alDp16nDixAl8fHx01rVo0QJPT09CQkKIi4vD3d2dCxcusG7dOipUqKDXmeBJ7u7utG/fnqCgIG7evEndunWJiIjQ6ZIL4Ovry7Zt2/D19aV79+6UKFGCyMhIjh49yvDhw3N8zM/rq6++olevXnz44Ye0adOGBg0aUKhQIc6dO8eWLVuwsbFh1KhROttYWloyYMAAPvjgAwoVKsTatWvJyMhQu+e6uLhgbW3N9OnTuXr1KsWLF+fo0aPs3LkTKysrvZvyz+PIkSM5ug/yuLfeegtzc3P8/f3x9fXF3Nycffv2cejQISwsLHIlvqdJSkrS6979uFKlStGkSZMXqqN58+ZEREQwcOBA2rVrx/3799m6dat6dWroMXbs2JHg4GAGDRpEz549KVOmDKdPn2br1q34+Pg89UFsLUk4ecTLy4vVq1ezdOlSvv76a0xNTalRowZLly6lZcuWarkmTZqwatUq5s6dy5w5c7C3t2f69OnMnj1bZ3+9evXi7t27bNq0ialTp1KzZk0WL17M6tWrda6ODK3XUI0aNeKbb75h4cKFLFmyBCsrKxo0aMCnn36q0zMKMnur7d+//5lXN0/G+vXXX7NkyRLMzc2pV68e06ZNe66b51oNGjTg8OHDOs/XPM7d3Z0TJ07oNSWZmJiwYMECgoKC2Lp1KxEREdja2tK9e3eGDRtG8eLFn1n3rFmzqFKlClu2bCE8PBw3Nzfmzp1L37591TKOjo4EBwezZMkSVq9eTWJiIpUrV2bixIn4+vo+93HnVKVKldi+fTsrV67k4MGDHDp0iLS0NOzt7enRowcfffSRXpOJs7Mz77zzDl9//TX379/Hzc2NTz75RO3tWKpUKVasWMHs2bP5+uuvsbS0pEqVKsydO5dTp06pV8ZZdcwwREpKCsePH8/yfpAhNBoNCxcuZPHixcydO5eiRYtSo0YNgoODWbt2LceOHSM1NfW57h8a4s6dO2pyzkrDhg1fOOH06NGDe/fuERYWxpdffkmpUqVwdnZm8eLF6oO6ffr0eeZ+SpcuTUhICAsXLmT9+vXcvXsXBwcHhg4dqvNl82lMFEO6Uoh816pVKxwcHPj222/zOxSD7dy5k1GjRrFz506drrSiYHgVz0mRv+QejjAKRVFYv3499erVk2QjhACkSU3ksrS0NEaPHs21a9c4depUrs6lIYR4tUnCEbnK3NycixcvcvnyZYYOHfpKzU0ihDAuuYcjhBAiT8gVTjYyMjJ48OABFhYWL/UMh0II8bJQFIXU1FSKFi2a5WMMknCy8eDBg2cO5y6EEEKfRqPJctQBSTjZ0Pa712g02Y5wLIQQ4n9SUlL4999/s31uSRJONrTNaJaWlgYPnCmEECL7wUTlORwhhBB5QhKOEEKIPCEJRwghRJ6QhCOEECJPSMIRQgiRJ6SX2gu6d+8eN27c0Jkj/XVjYWFB6dKlDRquXwjx+pKE8wLu3btHXFwcDg4OFC5c+LUckUBRFB4+fMiVK1cAJOkI8QqJjIxk48aNdOvWLdv5onKTJJwXcOPGDRwcHChSpEh+h5JvTExMKFKkCA4ODuqMjkKIV8OaNWs4d+4cSUlJeZJw5B7OC0hNTdWbsvh1Vbhw4de6WVGIV5F2duDHZwk2Jkk4L+h1bEbLirwPQohnkYQjhBAiT0jCEUIIkSck4QghhMgTknCEEELkCUk4Qggh8oQkHCGEEHlCHvzMBwkJCUyfPp3IyEhu3bpF2bJlad++PUOHDsXKyorevXtjZmaGn58fs2fP5vLly1StWpXBgwfj5eWls69du3YRGhrKmTNnSE5OpnTp0rRr146RI0eqM5X27t2bIkWK0LVrVxYuXEhMTAwVK1bE398fV1dXAgMD2bVrF2ZmZrRp04bx48dTqFCh/HhrhBAFmCScfDBy5Ej+/vtv/Pz8KF26NCdOnGDFihXcvXuXqVOnAnD+/HmGDx9O586d6dGjB1u3bmX48OHMnj0bb29vAMLCwvjss89o1aoVY8aMITU1lZ9//plVq1ZRpEgRhg4dqtb5119/ceLECfz8/ChWrBjLly9n5MiR1KpVi8KFCzNq1CiioqLYsGEDpUuX1tlWCCFygyScPHb79m1+/fVX/P396devHwBdu3ZFURRiY2PVcjdu3CAgIIA+ffoA0K1bN959910CAwN55513MDU1ZfXq1bi4uPD111+rD1726tULT09Pdu/erZM0bt68ybJly2jZsiUA5ubmTJkyhfT0dFatWgVAjx49iIqK4tChQ5JwhBC5ThJOHitWrBhFihRh7dq1lC9fnmbNmlGkSBGmT5+uV65Xr17q60KFCtGzZ09mzJjB6dOnqVu3Lj/88AMPHz7Uecr/9u3bFC9eXG+oCisrK5o1a6a+rlKlCgCenp7qMhMTExwcHIiLi8vVYxZCCJCEk+csLS2ZMmUKEydOZPjw4VhaWtKwYUPatm1Lx44dsbKyAqBixYrqPRitSpUqAXDlyhXq1q2LhYUFv/32G9u3byc6OppLly5x+/ZtABwcHHS2feONNzA3/9+v28zMDABbW1udcmZmZiiKkrsHLYQQSC+1fOHt7c3+/fuZNm0aHh4enDx5ks8//5xu3bqRkpICZM4x86SMjAzgf8lizpw59O3blzNnzlCrVi2GDRvGtm3bcHNz09v28WTzOBkDTQiRV+QKJ489ePCAM2fOUKNGDbp06UKXLl1ISUlh1qxZhISEcOjQIQAuX76Moig6CeHChQtA5pXOlStXWLFiBe+99x6BgYE6ddy6dSvPjkcIIQwlVzh57Ny5c/j6+rJp0yZ1maWlJW+++Sbwv6uXW7duER4erpZ5+PAh69ato3Llyjg6OpKQkABA9erVdfZ/4MABLly4QFpamrEPRQghckSucPJYvXr1cHNzY968eVy7dg1HR0euXbtGaGgoVatWpXHjxqxcuRILCwsCAgL466+/KF26NN9//z1xcXEsW7YMyEw09vb2LFu2jOTkZMqWLcupU6fYsmULVlZWPHjwIJ+PVAghdEnCyWMmJiYsWbKExYsXs2/fPjZs2ECJEiVo27YtI0aMUDsKlC5dmvHjxzNz5kxu3rxJ7dq1CQ4OpkGDBkDmVdGKFSuYMWMGISEhKIpCxYoVGT9+PGlpaUybNo3Tp0/j5OSUn4crhBAqEyUHXZIUReHy5ctUqFABgJiYGDZu3Ii5uTmdOnVSu9oWBMnJyeoHtrbn2JO0N+tzW+/evbly5QoRERG5vm9jMtb7IYQwDj8/P65cuYKDgwMhISEvvL9nfW4afIVz/fp1+vXrh6WlJVu2bOHWrVt0796de/fuARAaGsp3332n3osQQgghHmdwp4G5c+dy7do1evbsCcDGjRu5d+8e8+fPZ+/evZQrV46FCxcaLVAhhBCvNoMTzuHDh/nggw/o1q0bABEREZQrV4527drh4OBAt27d+P33340WqBBCiKfLSEt9qesyuEnt/v37lC9fHsgcPuWvv/6ia9eu6vrChQs/V1fc7du3s3TpUmJjY3FwcGDgwIF07NjRoG1nzpzJmTNnWLNmjc7yqKgofH199cp7eHiwfPnyHMeY17799tv8DkEI8QoyNbfgeGB/g8sn34lTf+ZkOwBX/5U5Kg85SDj29vb8+++/AOzYsQNAHQgS4JdfflETkqHCw8MZM2YMfn5+NGvWjD179jB27FgKFSpEu3btnrptaGgoq1evpnHjxnrr/vnnH4oUKUJwcLDO8uLFi+coPiGEELnH4ITToUMHvv76ay5evMjRo0cpV64czZo149KlS3z11VccOHCAcePG5ajyuXPn0r59e8aPHw9As2bNSEhIYMGCBdkmnLi4OAIDA9m5cyfFihXLsszZs2epUaMGzs7OOYpHCCGE8Rh8D2fo0KEMGzaM2NhY6tevz9KlSzE3NycxMZGoqCgGDx7MBx98YHDFsbGxXLp0ibZt2+os9/LyIjo6Wmeo/sfNmzePv//+m+Dg4Gy74J45cwZHR0eDYxFCCGF8OXrwc/DgwQwePFhnWc2aNTly5EiWg00+TXR0NIDeszvaEZFjYmLU530e179/f6pWrYqpqSlLlizRW5+RkcG5c+coWbIkPj4+nDt3jlKlSuHn50ffvn1lsMrXSGRkJBs3bqRbt240atQov8MR4rX3wiMNmJqaYmqa8yHZ7t+/D4C1tbXO8qJFiwKQmJiY5XZPjh32pJiYGB49ekRMTAyjR4+mZMmS7N27l8DAQBITExk+fHiOYxWvpjVr1nDu3DmSkpIk4QjxEsg24dSsWTPHVwMmJib8/fffBpXVDnDwZB3a5c+TxADKlClDUFAQtWrVws7ODoDGjRvz6NEjgoKC+PDDD/WS3NOcPn0623Xm5uYyZtljUlJSOH78eH6Hobpz547682WKSwhjcXV1zdP6cvp3lW3C6dixo1Gbn7Q3/J+8ktF+gGfXIeBZrK2tad68ud5yDw8PwsLCiImJoU6dOgbv71lD22ivyETm+G716tXL7zBU2t+blZVVnv8hCvE6ePLvSju0TXayTTgzZszIvaiyoL13c+nSJZ0b/BcvXtRZn1P//PMPx48fp2vXrjr3lR49egRAyZIlnzdkg6WkpmNpYWb0eowZw4s8HyWEEFnJt9GiK1WqRPny5dm1axdt2rRRl//0009UrlwZe3v759rvxYsXmTx5MmXKlMHT01NdvnPnTsqXL6839bIxWFqY0cv/O6PX8zRrA/UffDXUizwfJYQQ2ck24dSqVYvAwEC8vb0Bw+7p5OQeDsCQIUMICAigRIkSeHh4EBERQXh4OPPmzQMgPj6eS5cuUb16dYPvu3h4eODk5MTEiROJj4+nbNmy/Pjjj0RERLBo0SLppWaA53k+SgghnuWp93AqVqyo8zq3P6w7depESkoKq1evJiwsjAoVKjBz5kzefvttAPbv309AQAAhISG4u7sbtE9LS0uCgoKYP38+ixcvJj4+nho1arB48WJat26dq/EXRNrno0aPHq2z3MvLi/DwcGJjY7Psri6EEM+SbcKZPn26zmtj3dPp0aMHPXr0yHJdp06d6NSpU7bbZjfmmI2NDVOmTMmV+F43z/t8lBBCPMvz9T3ORk6a08TL6XmfjxJCiGcxuNNAamoqK1as4KeffiIpKYmMjAx1XXp6Og8ePCAxMZEzZ84YJVCRN4z1fJQQQhj86TF//nwWLVpEQkIChQsX5sqVK5QrVw5zc3OuX79OamoqEyZMMGasIg8Y6/koIYQwOOHs2rWLhg0bEhERQVBQEACff/45u3fvZvny5aSlpeV4PDXx8nn8+ajHvejzUUIIYXDCiYuLo23btpiamlKmTBlsbW05ceIEAC1atMDHx4eNGzcaLVCRNx5/PupxL/p8lBDi5WNlbqrz09gMvodTqFAhnSuYihUrqhOyAdStW5fdu3fnbnQiXzzr+SghRMHQtnpJDsQk0KJKiTypz+CEU6tWLQ4ePEj37t0BqFq1qnqFA5lXQPJQZcHwrOejhBAFQy27ItSyK5Jn9RmccHx9fRk5ciS9evVixYoVvPPOO3z//fcEBARQtWpV1qxZIzNs/r+U1PQXGlomt2J4kbHUnvZ8lBBCPA+DE067du2YOnUqwcHBFC5cmLfeeosBAwaoHQjs7e0JCAgwWqCvkvweuPNliUEIIR6Xo8E7u3btSteuXdXXn3zyCT179iQhIYFq1aphaWmZ6wEKoZWRloqped70hMzLuoR4XbzwaNH29vbSc0nkCVNzC44H9je4fPKdOPVnTrYDcPVfmaPyQohnMzjhJCQkMHPmTA4fPszNmzfVJ88fl9PRooUQQrw+DE44X3zxBeHh4dSvXx93d3fMzOQegRBCCMMZnHB+/fVX3n//fT777DNjxiOEEKKAMvjxUgsLC6pWrWrMWIQQQhRgBiccHx8ftm3bRlpamjHjEUIIUUAZ3KQ2YsQIBg4ciJeXF82bN8fW1lavjImJCUOGDMnVAIUQQhQMBiec7du3c+TIETIyMli3bl2WZSThCCGEyI7BCWfx4sVUrFiRgIAAqlSpIr3UhBBC5IjBCefmzZuMGzeOFi1aGDOeAuFleEo9t2I4c+YMXbp0Ye/evZQtWzYXIhNCvK5yNFr0lStXjBlLgZHTJ+KNITeelI+OjmbgwIHSUUQIkSsM7qXm7+9PWFgYa9eu5caNG2RkZBgzLpGP0tLS+O677+jSpQvJycn5HY4QooAw+Apn0qRJAEydOpWpU6dmWUaGtikYjh8/zuzZs+nXrx9lypSRh32FELnC4IRTu3ZtnJycjBmLeElUq1aNPXv2YGtry+bNm/M7HCFEAWFwwpkxY4Yx4xAvkVKlSuV3CEKIAsjgezjPEh8fz2+//ZZbuxNCCFHAZJtwGjZsyK5du3SWpaSkEBISQlxcnF75Q4cO4efnl/sRCvGcrMxNdX4KIfJXtn+J9+7dIzU1VWfZgwcPmD59OtHR0UYPTIgX1bZ6SaqWLETb6iXzOxQhBM8x42dWE68J8TKqZVeEWnZF8jsMIcT/k7YGIYQQeSLHVzji2TLSUnPlSf8XjSG/h9cRQojHyRWOEbwMH/S5FUOnTp34559/ZBw1IcQLk4QjhBAiTzy1SS0qKor09HT19YMHDwA4fPiwXtfo33//3QjhCSGEKCiemnA2btzIxo0b1dfaHmorV67ExMREp6yiKHrLhBBCCK1sE8706dPzMg4hhBAFXLYJx8fHJy/jeGXJlV0meT5LCPEs0mngBZibm8vkZP8vLS0Nc3PpZS+EyJ4knBdQqFAhEhMT8zuMl8L9+/cpVKhQfochhHiJScJ5AXZ2dty8eZOkpKTXtklJURSSkpK4desWdnZ2+R2OEOIlJm0gL6BQoUKUKVOG69evv9ZTMVtZWVGmTBm5whFCPFW2CefTTz+la9euNGzYEICrV69iY2MjHypPKFGiBCVKlMjvMIQQ4qWXbZPa7t27uXjxovra09OTPXv25ElQQgghCp5sr3Ds7OwIDg4mJSWFokWLoigKv/322zN7ZXXs2DG3YxRCCFEAZJtwRo4cyfjx45k6dSoAJiYmbNiwgQ0bNmS7MxMTE0k4QgghspRtwvH29qZZs2bExMSQkpLCBx98wKBBg3jrrbfyMj4hhBAFxFN7qb3xxhu4uLgAmSMPtGzZknr16uVJYEIIIQoWg7tFa8dWO3/+PHv37uXq1atYWFhQrlw5WrRoQbVq1Z4rgO3bt7N06VJiY2NxcHBg4MCBBjfLzZw5kzNnzrBmzRqd5WlpaSxevJgtW7Zw9+5dateuzbhx46hbt+5zxSiEEOLF5eg5nNmzZ7N69WoyMjL0lvfp0wd/f/8cVR4eHs6YMWPw8/OjWbNm7Nmzh7Fjx1KoUCHatWv31G1DQ0NZvXo1jRs31ls3bdo0tmzZwpgxY7C3tyc4OJg+ffqwbds2KlSokKMYhRBC5A6DE05YWBgrV67Ew8ODwYMHU61aNTIyMoiOjiYoKIjg4GBq1KiRo0E/586dS/v27Rk/fjwAzZo1IyEhgQULFmSbcOLi4ggMDGTnzp0UK1ZMb/3ly5fZsGEDEydOpGfPngA0bdoULy8vVq5cyeTJkw2OTwghRO4xeGib0NBQ3N3dWbZsGfXq1cPa2prixYvj7OzMkiVLaNiwIaGhoQZXHBsby6VLl2jbtq3Oci8vL6Kjo4mNjc1yu3nz5vH3338THBxMrVq19NZHRkaSnp6Ol5eXuszS0hIPDw8OHjxocHxCCCFyl8EJJyYmRi85PK5t27ZER0cbXLG2bJUqVXSWV6pUSa0vK/3792fHjh00atQo2/2WKFECGxsbvf1evXqVR48eGRyjEEKI3GNwk1rRokW5efNmtutv3LiBlZWVwRXfv38fAGtra716gGxHYa5evfpT95uYmKi3z8f3++DBgxwNz3P69GmDywrjcnV1zdP6jh8/nqf1CfGiXva/EYMTTtOmTQkNDaVdu3bUrFlTZ92ZM2cIDQ3Fw8PD4Iq1oytnNVU1gKnp8w1knd2ozdnV9yxOTk45SqSi4MjrP14hXjVP/o0kJyc/9Uu6wQln1KhRHDp0iM6dO9O0aVO1KSw6OprDhw9TrFgxRo4caXCg2hv+T17JPHjwQGd9TllbW6v7yGq/WV39CCGEMD6DE469vT1hYWHMmTOHAwcOcODAAQAKFy5M69atGTNmTI66HGsT1qVLl3B0dFSXawcMffLejqGqVq3K3bt3SUhI0BnF+eLFi5QvXx5LS8vn2q8QQogXk6PncMqXL8+8efPIyMjgzp07KIqCjY3NczV/VapUifLly7Nr1y7atGmjLv/pp5+oXLky9vb2Od4noA69s3v3brp16wZASkoKBw4coGnTps+1TyGEEC/uuSZgMzU1xdbW9oUrHzJkCAEBAZQoUQIPDw8iIiIIDw9n3rx5AMTHx3Pp0iWqV69ucFOYg4MDPj4+fPnllyQlJVGpUiWCg4NJSEigf//+LxyzEEKI55OvM3526tSJlJQUVq9eTVhYGBUqVGDmzJm8/fbbAOzfv5+AgABCQkJwd3c3eL9TpkyhePHirFixgqSkJGrXrk1wcLDa5VoIIUTeM1Gy69b1mtP2tpBeai+X44F5c5Xq6r8yT+oRIrfl59/Isz43n6/vsRBCCJFDknCEEELkCUk4Qggh8oTBnQZSUlJYuHAhP/74I7dv3yY9PV2vjImJCX///XeuBiiEEKJgMDjhBAYGEhoaSrVq1XBzc5MHKIUQQuSIwQknPDyctm3bsnDhQmPGI4QQooAy+B7OgwcPaNasmTFjEUIIUYAZnHCcnJxkqH4hhBDPzeCEM3bsWHbt2sV3331HfHy8MWMSQghRABl8D8ff3x+AL7/8ki+//DLLMtJLTQghRHYMTjj16tXL8eRlQgghhJbBCWfGjBnGjEMIIUQBl+PRos+fP8/evXu5evUqFhYW2Nvb06JFC6pWrWqM+IQQQhQQOUo4s2fPZvXq1WRkZOgsnzVrFn369FHv8wghhBBPMjjhhIWFsXLlSjw8PBg8eDDVqlUjIyOD6OhogoKCCA4OpkaNGvj4+BgzXiGEEK8og7tFh4aG4u7uzrJly6hXrx7W1tYUL14cZ2dnlixZQsOGDQkNDTVmrEIIIV5hBiecmJgY2rZtm+36tm3bEh0dnStBCSGEKHgMTjhFixbl5s2b2a6/ceOGzIwphBAiWwYnnKZNmxIaGsrZs2f11p05c4bQ0FCaNGmSq8EJIYQoOAzuNDBq1CgOHTpE586dadq0KVWqVAEgOjqaw4cPU6xYMUaOHGmsOIUQQrziDE449vb2hIWFMWfOHA4cOMCBAwcAKFy4MK1bt2bMmDFUqFDBaIEKIYR4teXoOZzy5cszb948MjIyuHPnDoqiYGNjg6mpzFQthBDi6XI80gCAqakptra2uR2LEEKIAizbhFOrVi0CAwPx9vYGoGbNms8cvFNGixZCCJGdbBNOx44dqVixos5rGS1aCCHE88o24UyfPl3ntYwWLYQQ4kUYfLffz8+PI0eOZLs+IiKCd955J1eCEkIIUfBke4Xz8OFD7ty5o74+duwYbdq0oVKlSnplMzIyOHjwIJcvXzZOlEIIIV55T004HTt25P79+0Bmh4CvvvqKr776KsvyiqLISANCCJELIiMj2bhxI926daNRo0b5HU6uyTbh2NjYMGvWLP78808URWHJkiW0adMGR0dHvbKmpqbY2NhIk5oQQuSCNWvWcO7cOZKSkl6PhAPQokULWrRoAcDp06fp378/9erVy5PAhBDidZWUlKTzs6AwuNPA2bNnOXTokDFjEUIIUYAZnHDu3LmDnZ2dMWMRQghRgBmccLy9vdmwYYP0RBNCCPFcDB5LzdTUlOjoaLy8vKhYsSK2trZ6g3aamJjwzTff5HqQQgghXn0GJ5zDhw9TsmRJAJKTk7l69arRghJCCFHwGJxwIiIijBmHEEIUSCmp6VhamOV3GC+FHE9PkJ6ezunTp7ly5QqWlpaUK1eO2rVrGyM2IYR45VlamNHL/7scbXPrVuYD99dv3c/RtmsDfXNUT17LUcLZt28fkydPJi4uDkVRgMz7NqVLl2bSpEm0atXKKEEKIYR49RmccKKiohg2bBi2traMGjWKatWqoSgK0dHRrF27luHDhxMSEkL9+vWNGa8QQohXlMEJZ9GiRTg4OLBp0yaKFSums65Xr1507tyZpUuXEhQUlOtBCiGEePUZ/BzOqVOn6Nq1q16yAbC2tqZLly788ccfuRqcEEKIgsPghPMsJiYmpKam5tbuhBBCFDAGJ5x69eqxadOmLAeTS0xMJCwsjDp16uRqcEII8ToyMbPQ+VlQGHwPZ+jQofj5+dGhQwfef/99KleuDKB2GoiLi2Py5MnGilMIIV4b1vb1eXD9T4qWLVhf4g1OOG5ubixatIgpU6YQGBiIiYkJkDnxmp2dHfPmzStQ8zYIIUR+sSpRAasSFfI7jFyXo+dwPD098fDw4K+//lIH8XRwcKB27dqYm+f4GVIAtm/fztKlS4mNjcXBwYGBAwfSsWPHbMs/ePCA2bNn89NPP5GUlISbmxsTJkxQr7ggswu3r6/+A1AeHh4sX778ueIUQgjxYnKcJczMzChfvjympqbq/5832YSHhzNmzBj8/Pxo1qwZe/bsYezYsRQqVIh27dpluc2oUaP4888/8ff3p2jRoixevBg/Pz927Nih9qD7559/KFKkCMHBwTrbFi9e/LniFEII8eJylCmioqKYPXs2p06dUkcaMDMzo1GjRvj7+6PRaHJU+dy5c2nfvj3jx48HoFmzZiQkJLBgwYIsE05UVBQHDhwgKCiI5s2bA5lNfZ6enqxbt46PPvoIyJwsrkaNGjg7O+coHiGEEMZjcC+1o0eP0qdPH6Kjo+nVqxcTJkxg3LhxdO3alZMnT9KzZ0/+/fdfgyuOjY3l0qVLtG3bVme5l5cX0dHRxMbG6m1z+PBhihYtSpMmTdRlNjY2NGjQgIMHD6rLzpw5g6Ojo8GxCCGEMD6Dr3Dmz5+Pg4MD69atw8bGRmfdkCFD6NatG3PnzmXZsmUG7S86OhqAKlWq6CyvVKkSADExMVSoUEFvm0qVKmFmpjvyasWKFQkPDwcgIyODc+fOUbJkSXx8fDh37hylSpXCz8+Pvn37qp0dCqrIyEg2btxIt27dpBOHEOKlYnDCOXv2LCNGjNBLNgClSpWiV69eLF261OCK79/PHA3V2tpaZ3nRokWBzGd7npSYmKhXXruNtnxMTAyPHj0iJiaG0aNHU7JkSfbu3UtgYCCJiYkMHz7c4BhfRWvWrOHcuXMkJSVJwhFCvFQMTji2trbcvn072/XJyclZJoPsPD7adFbLn5xN9PF1WdGWL1OmDEFBQdSqVQs7OzsAGjduzKNHjwgKCuLDDz/MUZynT582uOzL4M6dO+rP48eP53M0ucvV1TVP6yto75/IH3l93ualnP6NGJxwBg0axNSpU3FxcdGbhuCPP/4gJCSETz75xOCKtT3KnrySefDggc76x1lbW6vdsZ/cRptErK2t1Q4Fj/Pw8CAsLIyYmJgcjYjg5OSElZWVweXzmzZWKyurAn2i5wV5/4R4uif/RpKTk5/6Jd3ghHPy5ElsbW0ZMmQIVatWpVq1alhYWBAbG8uff/6JpaUl27dvZ/v27eo2JiYmfPPNN1nuT3vv5tKlSzo3+C9evKiz/sltjhw5gqIoOldGFy9eVMv/888/HD9+nK5du2Jh8b9hIR49egSgTpMthBAibxncS+3XX38FoFy5cjx8+JDTp09z4sQJbt26Rbly5bC1teXy5cs6/7LqaaZVqVIlypcvz65du3SW//TTT1SuXBl7e3u9bZo2bcq9e/fUWADi4+OJiorirbfeAjKTz+TJk3V6rQHs3LmT8uXL4+DgYOghCyOJjIxk9OjRREZG5ncoQog8ZPAVTkRERK5XPmTIEAICAihRogQeHh5EREQQHh7OvHnzgMxkcunSJapXr461tTUNGjSgYcOGjB49mjFjxvDGG2+waNEiihUrRs+ePYHMpjMnJycmTpxIfHw8ZcuW5ccffyQiIoJFixYV+F5qrwLp2CDE6ynHQwSkp6dz+vRprly5gqWlJeXKlaN27drPVXmnTp1ISUlh9erVhIWFUaFCBWbOnMnbb78NwP79+wkICCAkJAR3d3cAFi9ezIwZMwgMDCQjIwNXV1fmz59PiRIlALC0tCQoKIj58+ezePFi4uPjqVGjBosXL6Z169bPFafIXdoRx7MaeVwIUXDlKOHs27ePyZMnExcXp9PLrHTp0kyaNEmvM4EhevToQY8ePbJc16lTJzp16qSzrESJEkyfPp3p06dnu08bGxumTJmS41iEEEIYj8EJJyoqimHDhmFra8uoUaOoVq0aiqKo0xMMHz6ckJAQ6tevb8x4hRBCvKIMTjiLFi3CwcGBTZs26XVZ7tWrF507d2bp0qUEBQXlepBCCCFefQb3Ujt16hRdu3bN9vmYLl268Mcff+RqcOLll5Kant8hCCFeEc83r0AWTExMSE1Nza3dif+XkZaKqXneTDP7PHVZWpjRy/+7HG1z61bmsEbXb93P0bZrA/XnOBJCvDoMTjj16tVj06ZN9OrViyJFiuisS0xMJCwsLEdP8AvDmJpbcDywv8Hlk+/EqT9zsh2Aq//KHJUXQoicMDjhDB06FD8/Pzp06MD777+vzrCp7TQQFxfH5MmTjRWnEEKIV5zBCcfNzY1FixYxZcoUAgMD1QcoFUXBzs6OefPmyUN8QgghsmVwwrlz5w6enp54eHjw119/qYNoOjg4ULt27eeeZloIIcTrweAs4ePjQ9euXRkyZAh169albt26xoxLFGAmZhY6P4UQrweDu0XHx8er88sI8SKs7etjYV0Wa3t5SFiI14nBCcfb25sNGzZkOR+NEDlhVaICNo5vY1WiwrMLCyEKDIOb1ExNTYmOjsbLy4uKFStia2urNyvn0+a/EUII8XozOOEcPnxYnbwsOTmZq1evGi0oIYQQBU++zocjhBDi9fHMhJOamsp///1HWloa1atXp3DhwnkRl3hOVuamOj+FEOJl8dSEs2bNGpYsWUJiYiKQOblZr169+OSTT+S5m5dU2+olORCTQIsqJfI7FCGE0JFt1ti6dSszZszAwcGB9957D1NTU44ePcqaNWtIT09n/PjxeRmnMFAtuyLUsivy7IJCCJHHsk04a9euxdnZmW+++QYrKysgcxibUaNGsWHDBsaMGYOlpWWeBSqEEOLVlm1D//nz5/H29laTDWR2e+7Tpw8pKSlER0fnSYBCCJGbIiMjGT16NJGRkfkdymsn2yuchw8fZjnZWvny5VEUhXv37hk1MCGEMIY1a9Zw7tw5kpKSZMDhPJbtFU5GRoY6IvTjzMzMAEhPl5kec0K+VQnxckhKStL5KfKOdDXLI/KtSgjxuntqwrl7967eiAIJCQlA5mCeWY02YG9vn4vhFRzyrUoI8bp7asL56quv+Oqrr7JcN2bMGL1lJiYm/P3337kTmRBCiAIl24Tj4+OTl3EIIYQo4LJNONOnT8/LOIQQQhRwMuDWc0hJlR56QgiRU9JL7TlYWpjRy/+7HG1z69Z9AK7fup+jbdcG+uaoHiGEeFnJFY4Q4pWWkZZaIOsqiOQKRwjxSjM1t+B4YH+DyyffiVN/5mQ7AFf/lTkqL3TJFY4QQog8IQknj5iYWej8FCIvydBK4mUgTWp5xNq+Pg+u/0nRsnXyOxTxGpKhlcTLQBJOHrEqUQGrEhXyOwzxmpKhlcTLQJrUhBBC5AlJOEKIl0Je3WeyMjfV+SnyjjSpCSFeCnl1n6lt9ZIciEmgRZUSRqtDZE0SjhCvoIy0VEzN86bHY17VlVf3mWrZFaGWXRGj1iGyJglHiFeQPOwoXkXSiCmEyHUywK3IilzhCCFynQxwK7IiVzhCCCHyhCQcIcRLQYZ/Kvgk4QjxGngVnj2xtq+PhXVZrO3r53cowkjkHo4Qr4FX4dkTGf6p4JOEI8RrQJ49ES+Dl/f6WgghRIGS7wln+/btvPPOO9StW5f27duzdevWp5Z/8OABkydPpkmTJri4uDBgwAAuXLigUyYtLY358+fTokUL6tWrR69evTh16pTxDkIIIcQz5WvCCQ8PZ8yYMTRp0oQlS5bQsGFDxo4dy65du7LdZtSoUezatYsxY8Ywc+ZM4uLi8PPz4/79+2qZadOmsWbNGgYMGMC8efMwMzOjT58+xMbG5sVhCWEwmRhNvE7y9R7O3Llzad++PePHjwegWbNmJCQksGDBAtq1a6dXPioqigMHDhAUFETz5s0BcHNzw9PTk3Xr1vHRRx9x+fJlNmzYwMSJE+nZsycATZs2xcvLi5UrVzJ58uS8O0AhnkEmRhOvk3y7womNjeXSpUu0bdtWZ7mXlxfR0dFZXo0cPnyYokWL0qRJE3WZjY0NDRo04ODBg0DmN8b09HS8vLzUMpaWlnh4eKhlhHhZyMRo4nWSb1c40dHRAFSpUkVneaVKlQCIiYmhQoUKettUqlQJMzMzneUVK1YkPDxcLVOiRAlsbGz09nv16lUePXpEoUKFnhmfoigApKSkZLm+eJG8eTgtOTkZChXLu7qeg7wXmVLT0rEwN3t2wccUK1aMkiVLUqxYsZzX+RK/FyDnxeNel/dC+3mp/fx8Ur4lHO09F2tra53lRYsWBSAxMVFvm8TERL3y2m205Z9WBjI7HRiScFJTUwH4999/s1w/wLvaM/eRG06fPg1N3s+7up6DvBfP78MPP3z+Ol/y90LOi/953d6L1NTULD9n8y3haDOgiYlJlstNTfVb+7LLmo+Xz65MdvVlp2jRomg0GiwsLAzeRgghXmeKopCamqp+wX9SviWcYsUyL/uevJJ58OCBzvrHWVtbc/nyZb3lDx48UK9qrK2t1X1ktd+srn6yYmpqmmUMQgghsve0FqR86zSgvXdz6dIlneUXL17UWf/kNrGxsXpXMRcvXlTLV61albt375KQkKBXpnz58lhaWubaMQghhDBcviWcSpUqUb58eb1nbn766ScqV66Mvb293jZNmzbl3r17/Prrr+qy+Ph4oqKieOuttwDUn7t371bLpKSkcODAAXWdEEKIvJevz+EMGTKEgIAASpQogYeHBxEREYSHhzNv3jwgM5lcunSJ6tWrY21tTYMGDWjYsCGjR49mzJgxvPHGGyxatIhixYqpz9w4ODjg4+PDl19+SVJSEpUqVSI4OJiEhAT698/Z1LpCCCFyj4nytDvxeWD9+vWsXr2aa9euUaFCBT766CM6duwIwObNmwkICCAkJAR3d3cAEhISmDFjBnv27CEjIwNXV1fGjRtH1apV1X2mpKQwe/Zstm/fTlJSErVr18bf35969erlxyEKIYTgJUg4QgghXg/5PninEEKI14MkHPHaMfZFvTQaCJE1STivuEWLFvHmm2/mdxh6WrVqxYQJE/I7DD3Lly9n1apVRtv/vn37GDt2LABHjx7F0dGRqKgoo9X3OnnyXH/8HLt8+TKOjo5s27Ytv8Izipf17+h5yYyfwigWL178Uj44u2DBAgYPHmy0/X/zzTekp6cbbf8ia6VLl2bDhg1UrFgxv0MRTyEJRxjFy3jVJQouS0tLnJ2d8zsM8QzSpGYkSUlJzJo1i7Zt2+Lk5ET9+vXp168fZ8+eBWDcuHH06dOH9evX07x5c1xcXPjoo490Rl5YtGgRbdq0Yc+ePbRt2xZnZ2fef//9bAfNCw0NxdHRUW9qh3Xr1uHk5MTdu3eNdrxP0jYFaJs6fvrpJ4YOHYqLiwsNGzZk4sSJPHz4EICAgACaNWtGRkaGzj7Gjx9Pq1at1Hsiv/32G76+vtSrVw93d3c+++wz7t27p5bPyMhg3rx5tGrVCicnJ1q1asXcuXPVgVgdHR1JT09n8eLFODo65vox9+7dmyNHjnDs2DGd/Z8/f56+fftSt25dmjRpwty5c3WugjIyMli2bBmtW7fGycmJdu3aERYWprfvsWPHMnToUOrXr8/w4cOBzGk+Pv30U5o2bUrt2rV56623GDdunM5IG61atWLhwoVMnToVV1dXGjVqxBdffKG+/3l13iiKwpo1a2jXrh1169bFy8uLb7/9Vl2/YcMGOnXqhLOzM3Xr1sXHx0fnAe6nebJJbfPmzdSpU4fff/+drl27UqdOHVq2bMnq1at1trt27RrDhg3Dzc2NRo0aMWfOHAICAujdu3euHHNuSElJYdKkSbi6uuLu7s4XX3yhDgmWVZPb5s2bcXR05Pr16+zbtw9HR0e9Cf4OHjyIo6Mj586dy7PjAEk4RuPv78/WrVsZOHAgq1evJiAggH/++YcxY8aoH6CnT59myZIlfPLJJ0ybNo3z58/j5+enMzfKrVu3mDBhAn369GH27NkkJSXh5+fH9evX9er09vbG0tJSrx1727ZttGrVijfeeMOox/w0n332GRUqVODrr7+mX79+hIWFsXz5cgDee+89bty4oXOvIyUlhT179uDt7Y2JiQm//fYbffv2pWjRoixYsAB/f3/2799Pv379SEtLAyAoKIh169YxdOhQVq9eTc+ePVm5cqVaz4YNGzAzM6NLly5s2LAh149x0qRJ1KlThzfffJMNGzaoHwrTpk2jYcOGLF++nHbt2rF8+XI2btyobvfFF1+wePFifHx8WLZsGS1btmTixIk6H8aQOR174cKFWbJkCT179uThw4e8//77XLhwgS+++IJVq1bRu3dvfvzxR/Xhaa1vv/2Wv//+m1mzZjF48GC2bt3Kp59+CuTdeRMYGEhgYCBt27Zl2bJleHt7M23aNL777jtCQkKYPHkybdu2Zfny5cyePRtzc3M++eQT4uLinqu+tLQ0Ro8ejbe3N0FBQdSvX5+ZM2dy5MgRIPMc69OnD3/99RdffPEFX3zxBXv37mX79u25cry5ZceOHURHRzN79myGDh3Ktm3b1Ekrn6V58+bY2dll+bt1cnKiRo0axgg5W9KkZgTJyck8fPiQiRMnqjOXNmzYkMTERGbMmMGdO3eAzCkaVq1apT6QWq1aNd577z22bNmCr68vkHmlNG3aNN5++20AXFxc8PT05JtvvlFvTmuVKFGC1q1b8+OPPzJ06FAALly4wIkTJ9QP3fzSsmVLNd7GjRtz+PBh9u/fz8iRI3F3d6dcuXLs2LGDhg0bAnDo0CESEhJ49913AZgzZw7VqlVj2bJl6sjgb775Jj4+PuzcuZN3332XY8eO4eTkRKdOnYDM97xw4cLqvSRtk0vZsmWN0vyiHREjPT0dZ2dnjh49CkDfvn3V+0aNGjVi7969REZG0rNnT2JiYti4cSP+/v7qVAVNmzYlPT2dBQsW0KVLFwoXLgyAubk5U6dOVQdH/Ouvv3BwcCAwMJDy5cur+//jjz/47bffdGIzMzNj5cqV6ii+ZmZmTJ06lXPnzlGjRg2jnzf37t0jJCSEPn36MHr0aCBzGKrr16/z22+/Ubp0afr378+gQYPUbRwcHOjUqRO///477du3z3GdGRkZDBs2jM6dOwNQv359fv75Z/bt20fjxo354YcfuHjxItu2bVOvSJ2dnWndunUuHHHusbGxISgoSP29m5ubM3nyZP77779nbmtmZkbHjh1Zt24dkyZNolChQiQmJrJnzx78/f2NHboeucIxAisrK1atWkW7du2Ii4sjMjKS9evXs2/fPuB/c+1UqFBBZ/QDR0dHKlWqpPNN38LCQmf2UltbW1xdXTl+/HiWdXfu3JkLFy7wxx9/ALB161bs7Oxo1qxZrh9nTtSvX1/nddmyZdUmHRMTE7y9vfnpp5/Uq5UdO3ZQu3ZtqlWrxsOHD/njjz/w8PAgIyODtLQ00tLSqFGjBvb29urYeu7u7hw+fJhevXqxcuVK/vvvP95//33ee++9vD3YJ7i5uan/NzExwcHBQZ0PKjIyEkVRaNmypXpcaWlptGrVivv373Pq1Cl124oVK+qMxFu7dm3Wrl2Lvb09Fy5c4MCBA6xatYro6Gj1HNNq1aqVzpDx2pl2teeasc+bkydPkpaWRps2bXSWf/nll8yfP5/x48czevRo7t27x8mTJ9m2bRvfffcdgN6x5MTj552lpSU2NjbqeRcZGUnlypV1mj/Lli2Li4vLc9dnDC1atND5vXt6eqIoCn/++adB23fu3JnExEQiIiIACA8PJyMjg3feecco8T6NXOEYyS+//MJXX31FdHQ0RYsWpWbNmhQpUgT433MapUuX1tvO1tZW576Era2t3gynNjY2WU7TAJnfGsuVK8e2bduoW7cuP/zwA++++67ePvLak0OWm5qa6tyzee+991ixYgVHjhyhQYMGREREMHLkSCDz27H2PseyZcv09n3jxg0A+vfvT9GiRfn++++ZPXs2s2bNokaNGnz22Wc0atTIeAf3DNorFK3Hj117f0R7Jfwk7bFB5rnwpODgYJYtW8bdu3cpVaoUTk5OFC5cWG/K6ifPNe2MuNpzzdjnjfY4szoGyBw1/vPPP+fIkSNYWFhQtWpVatasCbzYc01Pe+/v3LmjNzMwgJ2dHTdv3nzuOnNbqVKldF5rY3783HiaKlWq4OLiwrZt23j77bfztYldEo4RXLp0iSFDhtCmTRtWrFihTpX93Xff8csvv6jlsroZe/v2berWrfvMMtn94ZqamtKxY0fCwsLw9vbmypUrahPTy6x69erUrl2bXbt2kZiYSHJysvoNrGjRopiYmPDhhx9m2bSi/eZuamqKr68vvr6+3L59mwMHDrBs2TKGDx/O4cOHsbDIm2l+c0Lb3BcaGprlPCLaprKs/Pjjj8yYMQN/f398fHzUD6IRI0bw999/65R98jy6ffs28L8PL2OfN9rjjI+P1+m6HBsby7Vr15g4cSKFChVi06ZN1KpVC3Nzc/777z+jPldTunRpfv/9d73l2vfmZfHkVCva+EqWLAmg19nmyS8bkHmVM3nyZM6fP09UVFS+NbFLk5oRnD59muTkZAYNGqQmG0BNNtoT5MKFC1y4cEFdf/bsWS5evKjzbfzRo0fqTU7I7ERw/Pjxp35j79y5M7dv32bu3LnUqVOH6tWr59ahGdW7777LgQMHCA8Pp0mTJuo3O2tra958800uXLhAnTp11H9VqlRh/vz5ajNQr169+PLLL4HMb9KdOnXC19eXhIQEtRklq5lkc1NOrwi0zW0JCQk6x3bt2jUWLlyoxp2V48ePU7JkSfr166cmjgcPHnD8+HG9D6FffvlFba6EzOk7TExMdM4jY5439erVw8LCQm1W1lq6dCnjxo3jwoULdOvWjTp16mBunvk9+ODBg4DxRm5o0KABFy9e1LkXcvv2bU6ePGmU+p7XkSNHdHo1hoeHA5nxW1tbc+3aNZ3yWTW3v/3221hYWPDFF19QqlQpmjZtatygsyFXOEZQu3ZtzM3NmTVrFn369CE5OZnNmzezf/9+APVDJCMjg8GDBzNy5EjS0tKYM2cO1atXx9vbW2d/Y8eOZfTo0VhbW7N48WKsra2f2m2zQoUKNGjQgGPHjvH5558b7ThzW4cOHQgMDGTPnj3MnDlTZ92IESMYNGgQ48aN4+233yYlJYWgoCD+/fdftTNCw4YNCQoKolSpUri4uBAXF0dwcDCNGzemePHiABQvXpwTJ07w22+/4ebmluvThxcrVoyoqCiOHDmiN5ttVmrWrEmHDh0YP348sbGx1KpVi//++4+5c+dSu3btLOeF0qpbty7r1q0jMDAQDw8Prl+/zurVq7l165ZeU9GVK1cYOnQovXr1Ijo6mvnz59OlSxedL0TGPG9sbGx4//33WbVqFebm5ri5uXH8+HG2bNnC1KlT+frrrwkJCaF06dJYW1vzyy+/EBISAmT9jT03eHt7s3z5cvVv0MrKiiVLlpCSkvJSTSt//fp1Ro0aRc+ePTlz5gzz58+nU6dOVKlShZYtW7J8+XJWrFhB3bp1iYiI0OsCDZmtAF5eXmzZsoV+/frlWxO7JBwjqFSpEnPmzGHx4sUMGjSIEiVK4OzszLfffkvv3r3VG7UVKlSgV69eTJ48mZSUFDw8PBg/frzOrKRmZmYEBAQQGBjI3bt3cXd3Z+HCherldHY8PDw4efIkHTp0MOqx5qZSpUrRpEkToqKi9HoKtWjRgpUrV7J48WKGDRuGlZUVderUISQkBI1GA8CwYcMwNzfn+++/Z8mSJRQrVgxPT08++eQTdT/Dhg1j7ty5DBgwgF27dlG2bNlcPYa+ffvyxx9/MGDAAAIDAw3aZsaMGSxbtozQ0FDi4uIoVaoUXbp0UZ+1yY6Pjw+XL1/m+++/JzQ0lDJlytCiRQt69erFxIkTiYmJUWfC9fb2plChQowYMQJra2s+/PBDhgwZordPY543/v7+2NjYsHHjRlasWEGlSpX46quv8PHxwcnJiWnTpuHv74+lpSXVq1dn6dKlfPXVVxw/fpxevXrlejwWFhasXr2aqVOnMmHCBAoXLkyvXr2wsrJS77e+DHr27ElCQgIff/wxhQoVws/PjxEjRgAwcOBA4uPjWblyJampqXh4eDBt2rQsR9Pw8PBgy5Yt+drELtMT5JNx48Zx/Phxfv7552zLLFq0iKVLl+q1xxvCz88POzs75syZ8yJhigKgVatWNG7cmGnTpj2z7Ot03pw7d44LFy7o9JxLS0vDw8ODd955h4CAgHyMLvcFBAQQExPD+vXr8y0GucIpYBYvXsz58+c5duwYmzZtyu9wxCvidTxvEhMTGTZsGH369MHDw4Pk5GQ2btzIvXv36NatW36Hl2u++eYbzp8/z5YtW1i0aFG+xiIJp4CJiIggNjaWgIAAnJyc8jsc8Yp4Hc8bFxcX5syZw6pVq9iwYQOmpqY4OzsTGhpKtWrV8ju8XBMVFcWhQ4f48MMP9Z6DymvSpCaEECJPSLdoIYQQeUISjhBCiDwhCUcIIUSekIQjRC4aOXIkjo6O6ojgjxs7diyOjo58/PHHeusePHjAm2++qY6knNu0011v3rzZKPsXwhCScITIRQ0aNADQGeVZ6+jRo1hYWHDs2DG9aahPnTpFeno67u7ueRKnEPlBEo4QuSi7hHPhwgWuXbtGhw4duH//Pn/99ZfOeu0gktr5gIQoiCThCJGLatSoQcmSJdUBRbUiIyMxNTVl0KBBmJiY6AzICnDixAlKly6tDkUjREEkCUeIXGRiYoKrq6ve5FiRkZHUrFlTnfDr8QEWFUXh1KlT6tVNeno6K1euxMvLCycnJ5o2bcqkSZOIj49Xt9Hek9myZQve3t7UqVNHHYpFO0ts06ZNcXZ2Zvjw4eqEb487duwYvr6+uLm54eLiQo8ePdRJuoQwBkk4QuSyBg0acPfuXXXqCUVROHbsmHp/plGjRvz++++kpKQAcP78eRISEtSpAkaNGsWsWbPQaDQEBATQrl07Nm3aRM+ePXUm5wOYMmUKrq6ufPrpp+pMkIMGDSI0NBRPT0/GjBlDfHw848aN09kuOjqagQMHoigKo0aNYsyYMTx8+JCPP/5YZ8ZZIXKVIoTIVadPn1Y0Go2ydetWRVEU5ezZs4pGo1H27dunKIqi7N27V9FoNMqRI0cURVGUjRs3KhqNRrl48aJy4MABRaPRKF9++aXOPnfu3KloNBolMDBQURRFiYyMVDQajfL+++/rlIuIiFA0Go0SHBysLktNTVU++OADRaPRKN9//72iKIqyYsUKRaPRKLdv31bLxcfHK23btlVCQkJy9f0QQkuucITIZbVq1aJYsWJqx4HIyEjMzMzUydYaNGiAmZkZx44dAzI7DJQtW5aKFSuqTVoDBw7U2Wf79u2pUqUKe/bs0Vn+5ERaBw8exNTUlK5du6rLzM3N8fX11SmnnZZh6tSpnD59GsicQXL37t1PnWtJiBchCUeIXGZqaoqrq6vacSAyMhInJyesra2BzEnaatWqpc7MePLkSbW57fLlyxQvXlxvHnuAatWqcfXqVZ1lWU20Zmtrq067rVW1alWd1+3atcPLy4udO3fSuXNnmjZtyoQJE6Q5TRiVJBwhjMDNzY2zZ8+SnJxMVFSU3pTg7u7unDp1ivj4eGJiYtQOA8pTxtLNyMjAwsJCZ9mTMzeamJio94ae3PZxFhYWLFy4kB9++IFhw4Zhb2/P5s2b8fX1ZcWKFTk6ViEMJQlHCCNo2LAhqamphIeHc+/ePb0HOhs1akRSUhI7duxAURR1vYODA/fu3ePWrVt6+4yJiaFcuXJPrbdChQokJCTo9GgDiI2N1Xl99epVoqKicHR0ZOjQoWzcuJF9+/ZRuXJlVq1a9TyHLMQzScIRwghq165NkSJFWL9+PRYWFtSvX19nvaurK+bm5mzZsgV7e3sqVKgAZM7OCbB8+XKd8nv27CEmJgYPD4+n1qud72T16tXqMkVRWLt2rU65ZcuW0adPH+Li4tRlZcuWpUyZMpiayseCMA6ZgE0IIzA3N8fFxYXDhw/j5uZG4cKFddYXLVqUOnXqcOLECXx8fNTlLVq0wNPTk5CQEOLi4nB3d+fChQusW7eOChUq6HUmeJK7uzvt27cnKCiImzdvUrduXSIiIvRGNvD19WXbtm34+vrSvXt3SpQoQWRkJEePHmX48OG590YI8Rj5KiOEkWiHuXny/o2Wthnt8eFsTExMWLBgASNGjODs2bNMnz6dn376ie7du7Np0yaKFy/+zHpnzZrFxx9/zNGjR5k5cyaKojB37lydMo6OjgQHB1OpUiVWr17N1KlTOXfuHBMnTsxycFEhcoPM+CmEECJPyBWOEEKIPCEJRwghRJ6QhCOEECJPSMIRQgiRJyThCCGEyBOScIQQQuQJSThCCCHyhCQcIYQQeUISjhBCiDwhCUcIIUSe+D+U1CYQQ1MgnwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(data = for_plot.melt('spam'),      # melt !\n",
    "            x = 'variable',  y = 'value', \n",
    "            hue = 'spam')\n",
    "\n",
    "plt.xlabel(\"Words\")\n",
    "plt.ylabel(\"Proportion of Emails\")\n",
    "plt.title(\"Frequency of Words in Spam/ham Emails\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "q3b",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "\n",
    "\n",
    "When the feature is binary, it makes sense to compare its proportions across classes (as in the previous question). Otherwise, if the feature can take on numeric values, we can compare the distributions of these values for different classes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbgrader": {
     "grade": false,
     "grade_id": "classification",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "# Basic Classification\n",
    "\n",
    "Notice that the output of `words_in_texts(words, train['email'])` is a numeric matrix containing features for each email. This means we can use it directly to train a classifier!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q4",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 4\n",
    "\n",
    "We've given you 5 words that might be useful as features to distinguish spam/ham emails. Use these words as well as the `train` DataFrame to create two NumPy arrays: `X_train` and `Y_train`.\n",
    "\n",
    "`X_train` should be a matrix of 0s and 1s created by using your `words_in_texts` function on all the emails in the training set.\n",
    "\n",
    "`Y_train` should be a vector of the correct labels for each email in the training set.\n",
    "\n",
    "*The provided tests check that the dimensions of your feature matrix (X) are correct, and that your features and labels are binary (i.e. consists of only 0's and 1's). It does not check that your function is correct; that was verified in a previous question.*\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q4\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:43.726012Z",
     "start_time": "2019-04-03T20:17:43.498088Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q4-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    },
    "tags": [
     "student"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([[0, 0, 0, 0, 0],\n",
       "        [0, 0, 0, 0, 0],\n",
       "        [0, 0, 0, 0, 0],\n",
       "        [0, 0, 0, 0, 0],\n",
       "        [0, 0, 0, 1, 0]]),\n",
       " array([0, 0, 0, 0, 0]))"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "some_words = ['drug', 'bank', 'prescription', 'memo', 'private']\n",
    "\n",
    "X_train = words_in_texts(some_words, train['email'])\n",
    "Y_train = train['spam'].to_numpy()\n",
    "\n",
    "X_train[:5], Y_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q4</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q4 results: All test cases passed!"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q4\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q5",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 5\n",
    "\n",
    "Now that we have matrices, we can build a model with `scikit-learn`! Using the [`LogisticRegression`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier, train a logistic regression model using `X_train` and `Y_train`. Then, output the model's training accuracy below. You should get an accuracy of around $0.75$\n",
    "\n",
    "*The provided test checks that you initialized your logistic regression model correctly.*\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q5\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:17:44.593918Z",
     "start_time": "2019-04-03T20:17:43.783872Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q5-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    },
    "tags": [
     "student"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Accuracy:  0.7576201251164648\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train, Y_train)\n",
    "\n",
    "training_accuracy = model.score(X_train, Y_train)\n",
    "print(\"Training Accuracy: \", training_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q5</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q5 results: All test cases passed!"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating Classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't seem too shabby! But the classifier you made above isn't as good as the accuracy would make you believe. First, we are evaluating accuracy on the training set, which may provide a misleading accuracy measure. Accuracy on the training set doesn't always translate to accuracy in the real world (on the test set). In future parts of this analysis, we will hold out some of our data for model validation and comparison.\n",
    "\n",
    "Presumably, our classifier will be used for **filtering**, i.e. preventing messages labeled `spam` from reaching someone's inbox. There are two kinds of errors we can make:\n",
    "- False positive (FP): a ham email gets flagged as spam and filtered out of the inbox.\n",
    "- False negative (FN): a spam email gets mislabeled as ham and ends up in the inbox.\n",
    "\n",
    "To be clear, we label spam emails as 1 and ham emails as 0. These definitions depend both on the true labels and the predicted labels. False positives and false negatives may be of differing importance, leading us to consider more ways of evaluating a classifier, in addition to overall accuracy:\n",
    "\n",
    "**Precision** measures the proportion $\\frac{\\text{TP}}{\\text{TP} + \\text{FP}}$ of emails flagged as spam that are actually spam.\n",
    "\n",
    "**Recall** measures the proportion $\\frac{\\text{TP}}{\\text{TP} + \\text{FN}}$ of spam emails that were correctly flagged as spam. \n",
    "\n",
    "**False-alarm rate** measures the proportion $\\frac{\\text{FP}}{\\text{FP} + \\text{TN}}$ of ham emails that were incorrectly flagged as spam. \n",
    "\n",
    "The two graphics below may help you understand precision and recall visually:\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/2/26/Precisionrecall.svg/700px-Precisionrecall.svg.png\" width=\"500px\">\n",
    "\n",
    "Note that a true positive (TP) is a spam email that is classified as spam, and a true negative (TN) is a ham email that is classified as ham."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "grade": false,
     "grade_id": "q6",
     "locked": true,
     "schema_version": 2,
     "solution": false
    }
   },
   "source": [
    "### Question 6a\n",
    "\n",
    "Suppose we have a classifier `zero_predictor` that always predicts 0 (never predicts positive). How many false positives and false negatives would this classifier have if it were evaluated on the training set and its results were compared to `Y_train`? Fill in the variables below (feel free to hard code your answers for this part):\n",
    "\n",
    "*Tests in Question 6 only check that you have assigned appropriate types of values to each response variable, but do not check that your answers are correct.*\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6a\n",
    "points: 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:20:13.853633Z",
     "start_time": "2019-04-03T20:20:13.825724Z"
    },
    "nbgrader": {
     "grade": false,
     "grade_id": "q6a-answer",
     "locked": false,
     "schema_version": 2,
     "solution": true
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 1918)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "zero_predictor_fp = 0\n",
    "zero_predictor_fn = (Y_train == 1).sum()\n",
    "zero_predictor_fp, zero_predictor_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q6a</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q6a results: All test cases passed!"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q6a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "### Question 6b\n",
    "\n",
    "What is the accuracy and recall of `zero_predictor` (classifies every email as ham) on the training set? Do **NOT** use any `sklearn` functions.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6b\n",
    "points: 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:23:21.553134Z",
     "start_time": "2019-04-03T20:23:21.548219Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.7447091707706642, 0)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# acc = (tp + tn) / n, (tp + tn) = n - fn - fp\n",
    "zero_predictor_acc = (len(Y_train) - zero_predictor_fn - zero_predictor_fp) / len(Y_train)\n",
    "# recall = tp / (tp + fn), tp = 0\n",
    "zero_predictor_recall = 0           \n",
    "zero_predictor_acc, zero_predictor_recall"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q6b</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q6b results: All test cases passed!"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q6b\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "### Question 6c\n",
    "\n",
    "Comment on the results from 6a and 6b. For **each** of FP, FN, accuracy, and recall, briefly explain why we see the result that we do.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6c\n",
    "manual: True\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "### Question 6d\n",
    "\n",
    "Compute the precision, recall, and false-alarm rate of the `LogisticRegression` classifier created and trained in Question 5. Do **NOT** use any `sklearn` functions, with the exception of the `.predict` method of your model object.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6d\n",
    "points: 2\n",
    "-->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-03T20:37:54.875265Z",
     "start_time": "2019-04-03T20:37:54.720667Z"
    }
   },
   "outputs": [],
   "source": [
    "Y_pred = model.predict(X_train)\n",
    "\n",
    "tp = np.sum((Y_train == 1) & (Y_pred == 1))\n",
    "fp = np.sum((Y_train == 0) & (Y_pred == 1))\n",
    "tn = np.sum((Y_train == 0) & (Y_pred == 0))\n",
    "fn = np.sum((Y_train == 1) & (Y_pred == 0))\n",
    "\n",
    "logistic_predictor_precision = tp / (tp + fp)\n",
    "logistic_predictor_recall = tp / (tp + fn)\n",
    "logistic_predictor_far = fp / (fp + tn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<p><strong><pre style='display: inline;'>q6d</pre></strong> passed!</p>"
      ],
      "text/plain": [
       "q6d results: All test cases passed!"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check(\"q6d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "### Question 6e\n",
    "\n",
    "Are there more false positives or false negatives when using the logistic regression classifier from Question 5?\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6e\n",
    "manual: True\n",
    "points: 1\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "<!-- END QUESTION -->\n",
    "\n",
    "<!-- BEGIN QUESTION -->\n",
    "\n",
    "### Question 6f\n",
    "\n",
    "1. Our logistic regression classifier got 75.76% prediction accuracy (number of correct predictions / total). How does this compare with predicting 0 for every email?\n",
    "1. Given the word features we gave you above, name one reason this classifier is performing poorly. Hint: Think about how prevalent these words are in the email set.\n",
    "1. Which of these two classifiers would you prefer for a spam filter and why? Describe your reasoning and relate it to at least one of the evaluation metrics you have computed so far.\n",
    "\n",
    "<!--\n",
    "BEGIN QUESTION\n",
    "name: q6f\n",
    "manual: True\n",
    "points: 3\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "---\n",
    "\n",
    "To double-check your work, the cell below will rerun all of the autograder tests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "q2 results: All test cases passed!\n",
       "\n",
       "q4 results: All test cases passed!\n",
       "\n",
       "q5 results: All test cases passed!\n",
       "\n",
       "q6a results: All test cases passed!\n",
       "\n",
       "q6b results: All test cases passed!\n",
       "\n",
       "q6d results: All test cases passed!"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grader.check_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "source": [
    "## Submission\n",
    "\n",
    "Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output. The cell below will generate a zip file for you to submit. **Please save before exporting!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "deletable": false,
    "editable": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "            <p>Your submission has been exported. Click <a href=\"hw12_2022_04_16T16_33_46_315936.zip\" download=\"hw12_2022_04_16T16_33_46_315936.zip\" target=\"_blank\">here</a>\n",
       "            to download the zip file.</p>\n",
       "            "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Save your notebook first, then run this cell to export your submission.\n",
    "grader.export()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}