notebooks/2.3 Skills, Wrangling with Pandas.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.3 Skills/Wrangling with 🐼\n",
    "\n",
    "In this notebook, we will focus on one skill in data analysis, namely the ability to add select, aggregate and transform data in a dataframe."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load dataset"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read in a CSV file containing an export of [Elon Musk's tweets](https://twitter.com/elonmusk), exported from Twitter's API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_path = '../data/musk_tweets/elonmusk_tweets.csv'\n",
    "df = pd.read_csv(dataset_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2819 entries, 0 to 2818\n",
      "Data columns (total 3 columns):\n",
      " #   Column      Non-Null Count  Dtype \n",
      "---  ------      --------------  ----- \n",
      " 0   id          2819 non-null   int64 \n",
      " 1   created_at  2819 non-null   object\n",
      " 2   text        2819 non-null   object\n",
      "dtypes: int64(1), object(2)\n",
      "memory usage: 66.2+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>849636868052275200</td>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>b'And so the robots spared humanity ... https:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>848988730585096192</td>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>848943072423497728</td>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>848935705057280001</td>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>b'Stormy weather in Shortville ...'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>848416049573658624</td>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "      <td>b\"@DaveLeeBBC @verge Coal is dying due to nat ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   id           created_at  \\\n",
       "0  849636868052275200  2017-04-05 14:56:29   \n",
       "1  848988730585096192  2017-04-03 20:01:01   \n",
       "2  848943072423497728  2017-04-03 16:59:35   \n",
       "3  848935705057280001  2017-04-03 16:30:19   \n",
       "4  848416049573658624  2017-04-02 06:05:23   \n",
       "\n",
       "                                                text  \n",
       "0  b'And so the robots spared humanity ... https:...  \n",
       "1  b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exa...  \n",
       "2      b'@waltmossberg @mims @defcon_5 Et tu, Walt?'  \n",
       "3                b'Stormy weather in Shortville ...'  \n",
       "4  b\"@DaveLeeBBC @verge Coal is dying due to nat ...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's give this dataset a bit more structure:\n",
    "- the `id` column can be transformed into the dataframe's index, thus enabling us e.g. to select a tweet by id;\n",
    "- `created_at` contains a timestamp, thus it can easily be converted into a `datetime` value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.set_index('id', drop=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['created_at'] = pd.to_datetime(df['created_at'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 2819 entries, 849636868052275200 to 15434727182\n",
      "Data columns (total 2 columns):\n",
      " #   Column      Non-Null Count  Dtype         \n",
      "---  ------      --------------  -----         \n",
      " 0   created_at  2819 non-null   datetime64[ns]\n",
      " 1   text        2819 non-null   object        \n",
      "dtypes: datetime64[ns](1), object(1)\n",
      "memory usage: 66.1+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selection"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Renaming columns"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An operation on dataframes that you'll find yourself doing very often is to rename the columns. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first way of renaming columns is by manipulating directly the dataframe's index via the `columns` property."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can change the column names by assigning to `columns` a list having as values the new column names.\n",
    "\n",
    "**NB**: the size of the list and new number of colums must match!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['created_at', 'text'], dtype='object')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.columns = ['created_at','text_tweet']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>text_tweet</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>b'And so the robots spared humanity ... https:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>b'Stormy weather in Shortville ...'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848416049573658624</th>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "      <td>b\"@DaveLeeBBC @verge Coal is dying due to nat ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "849636868052275200 2017-04-05 14:56:29   \n",
       "848988730585096192 2017-04-03 20:01:01   \n",
       "848943072423497728 2017-04-03 16:59:35   \n",
       "848935705057280001 2017-04-03 16:30:19   \n",
       "848416049573658624 2017-04-02 06:05:23   \n",
       "\n",
       "                                                           text_tweet  \n",
       "id                                                                     \n",
       "849636868052275200  b'And so the robots spared humanity ... https:...  \n",
       "848988730585096192  b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exa...  \n",
       "848943072423497728      b'@waltmossberg @mims @defcon_5 Et tu, Walt?'  \n",
       "848935705057280001                b'Stormy weather in Shortville ...'  \n",
       "848416049573658624  b\"@DaveLeeBBC @verge Coal is dying due to nat ...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check that the change did take place\n",
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second way of renaming colums is to use the method `rename()` of a dataframe. The `columns` parameter takes a dictionary of mappings between old and new column names.\n",
    "\n",
    "```python\n",
    "mapping_dict = {\n",
    "    \"old_column_name\": \"new_column_name\"\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's change column `text` => `tweet`\n",
    "df = df.rename(columns={\"text_tweet\": \"tweet\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>b'And so the robots spared humanity ... https:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>b'Stormy weather in Shortville ...'</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848416049573658624</th>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "      <td>b\"@DaveLeeBBC @verge Coal is dying due to nat ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "849636868052275200 2017-04-05 14:56:29   \n",
       "848988730585096192 2017-04-03 20:01:01   \n",
       "848943072423497728 2017-04-03 16:59:35   \n",
       "848935705057280001 2017-04-03 16:30:19   \n",
       "848416049573658624 2017-04-02 06:05:23   \n",
       "\n",
       "                                                                tweet  \n",
       "id                                                                     \n",
       "849636868052275200  b'And so the robots spared humanity ... https:...  \n",
       "848988730585096192  b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exa...  \n",
       "848943072423497728      b'@waltmossberg @mims @defcon_5 Et tu, Walt?'  \n",
       "848935705057280001                b'Stormy weather in Shortville ...'  \n",
       "848416049573658624  b\"@DaveLeeBBC @verge Coal is dying due to nat ...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Question**: in which cases is it more convenient to use the second method over the first?"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Selecting columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id\n",
       "849636868052275200   2017-04-05 14:56:29\n",
       "848988730585096192   2017-04-03 20:01:01\n",
       "848943072423497728   2017-04-03 16:59:35\n",
       "848935705057280001   2017-04-03 16:30:19\n",
       "848416049573658624   2017-04-02 06:05:23\n",
       "Name: created_at, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# this selects one single column and returns as a Series\n",
    "df[\"created_at\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df[\"created_at\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848416049573658624</th>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at\n",
       "id                                    \n",
       "849636868052275200 2017-04-05 14:56:29\n",
       "848988730585096192 2017-04-03 20:01:01\n",
       "848943072423497728 2017-04-03 16:59:35\n",
       "848935705057280001 2017-04-03 16:30:19\n",
       "848416049573658624 2017-04-02 06:05:23"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# whereas this syntax selects one single column\n",
    "# but returns a Dataframe\n",
    "df[[\"created_at\"]].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df[[\"created_at\"]])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transformation\n",
    "\n",
    "\n",
    "The two main functions used to manipulate and transform values in a dataframe are:\n",
    "- `map()`\n",
    "- `apply()`\n",
    "\n",
    "In this section we'll be using both to enrich our datasets with useful information (useful for exploration, for later visualizations, etc.)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add link to original tweet"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `map()` method can be called on a column, as well as on the dataframe's index.\n",
    "\n",
    "When passed as a parameter to `map`, the functional programming-stlye function `lambda` can be used to transform any value from that column into another one.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([849636868052275200, 848988730585096192, 848943072423497728,\n",
       "       848935705057280001, 848416049573658624, 848415731502923777,\n",
       "       848415356263702528, 848398971139629057, 848244577521647616,\n",
       "       848243350993895424,\n",
       "       ...\n",
       "       149441101684686848, 149439686702661632, 149436471764459520,\n",
       "       149435658115612672, 143171132814671872, 142881284019060736,\n",
       "       142880871391838208, 142188458125963264, 142179928203460608,\n",
       "              15434727182],\n",
       "      dtype='int64', name='id', length=2819)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['tweet_link'] = df.index.map(lambda x: f'https://twitter.com/i/web/status/{x}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "</style>\n",
       "<table id=\"T_5d664\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_5d664_level0_col0\" class=\"col_heading level0 col0\" >created_at</th>\n",
       "      <th id=\"T_5d664_level0_col1\" class=\"col_heading level0 col1\" >tweet</th>\n",
       "      <th id=\"T_5d664_level0_col2\" class=\"col_heading level0 col2\" >tweet_link</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th class=\"index_name level0\" >id</th>\n",
       "      <th class=\"blank col0\" >&nbsp;</th>\n",
       "      <th class=\"blank col1\" >&nbsp;</th>\n",
       "      <th class=\"blank col2\" >&nbsp;</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_5d664_level0_row0\" class=\"row_heading level0 row0\" >849636868052275200</th>\n",
       "      <td id=\"T_5d664_row0_col0\" class=\"data row0 col0\" >2017-04-05 14:56:29</td>\n",
       "      <td id=\"T_5d664_row0_col1\" class=\"data row0 col1\" >b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'</td>\n",
       "      <td id=\"T_5d664_row0_col2\" class=\"data row0 col2\" ><a target=\"_blank\" href=\"https://twitter.com/i/web/status/849636868052275200\">https://twitter.com/i/web/status/849636868052275200</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5d664_level0_row1\" class=\"row_heading level0 row1\" >848988730585096192</th>\n",
       "      <td id=\"T_5d664_row1_col0\" class=\"data row1 col0\" >2017-04-03 20:01:01</td>\n",
       "      <td id=\"T_5d664_row1_col1\" class=\"data row1 col1\" >b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"</td>\n",
       "      <td id=\"T_5d664_row1_col2\" class=\"data row1 col2\" ><a target=\"_blank\" href=\"https://twitter.com/i/web/status/848988730585096192\">https://twitter.com/i/web/status/848988730585096192</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5d664_level0_row2\" class=\"row_heading level0 row2\" >848943072423497728</th>\n",
       "      <td id=\"T_5d664_row2_col0\" class=\"data row2 col0\" >2017-04-03 16:59:35</td>\n",
       "      <td id=\"T_5d664_row2_col1\" class=\"data row2 col1\" >b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "      <td id=\"T_5d664_row2_col2\" class=\"data row2 col2\" ><a target=\"_blank\" href=\"https://twitter.com/i/web/status/848943072423497728\">https://twitter.com/i/web/status/848943072423497728</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5d664_level0_row3\" class=\"row_heading level0 row3\" >848935705057280000</th>\n",
       "      <td id=\"T_5d664_row3_col0\" class=\"data row3 col0\" >2017-04-03 16:30:19</td>\n",
       "      <td id=\"T_5d664_row3_col1\" class=\"data row3 col1\" >b'Stormy weather in Shortville ...'</td>\n",
       "      <td id=\"T_5d664_row3_col2\" class=\"data row3 col2\" ><a target=\"_blank\" href=\"https://twitter.com/i/web/status/848935705057280001\">https://twitter.com/i/web/status/848935705057280001</a></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_5d664_level0_row4\" class=\"row_heading level0 row4\" >848416049573658624</th>\n",
       "      <td id=\"T_5d664_row4_col0\" class=\"data row4 col0\" >2017-04-02 06:05:23</td>\n",
       "      <td id=\"T_5d664_row4_col1\" class=\"data row4 col1\" >b\"@DaveLeeBBC @verge Coal is dying due to nat gas fracking. It's basically dead.\"</td>\n",
       "      <td id=\"T_5d664_row4_col2\" class=\"data row4 col2\" ><a target=\"_blank\" href=\"https://twitter.com/i/web/status/848416049573658624\">https://twitter.com/i/web/status/848416049573658624</a></td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x146f0d7e0>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option(\"display.max_colwidth\", 10000)\n",
    "\n",
    "def make_clickable(val):\n",
    "    # target _blank to open new window\n",
    "    return '<a target=\"_blank\" href=\"{}\">{}</a>'.format(val, val)\n",
    "\n",
    "df.head().style.format({'tweet_link': make_clickable})\n",
    "\n",
    "# to apply the style to the entire dataframe just remove\n",
    "# `.head` from the line above"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add colums with mentions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def find_mentions(tweet_text):\n",
    "    handle_regexp = r'@[a-zA-Z0-9_]{1,15}'\n",
    "    if len(re.findall(handle_regexp, tweet_text)) > 0:\n",
    "        return \", \".join(re.findall(handle_regexp, tweet_text))\n",
    "    else:\n",
    "        return None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id\n",
       "849636868052275200                                           None\n",
       "848988730585096192    @ForIn2020, @waltmossberg, @mims, @defcon_5\n",
       "848943072423497728                @waltmossberg, @mims, @defcon_5\n",
       "848935705057280001                                           None\n",
       "848416049573658624                            @DaveLeeBBC, @verge\n",
       "                                         ...                     \n",
       "142881284019060736                                           None\n",
       "142880871391838208                                           None\n",
       "142188458125963264                                           None\n",
       "142179928203460608                                           None\n",
       "15434727182                                                  None\n",
       "Name: tweet, Length: 2819, dtype: object"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['tweet'].apply(find_mentions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['tweet_mentions'] = df['tweet'].apply(find_mentions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['n_mentions'] = df['tweet_mentions'].map(lambda x: len(x.split(',')) if x is not None else 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>week_day</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'</td>\n",
       "      <td>https://twitter.com/i/web/status/849636868052275200</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"</td>\n",
       "      <td>https://twitter.com/i/web/status/848988730585096192</td>\n",
       "      <td>@ForIn2020, @waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>4</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "      <td>https://twitter.com/i/web/status/848943072423497728</td>\n",
       "      <td>@waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>3</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>b'Stormy weather in Shortville ...'</td>\n",
       "      <td>https://twitter.com/i/web/status/848935705057280001</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848416049573658624</th>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "      <td>b\"@DaveLeeBBC @verge Coal is dying due to nat gas fracking. It's basically dead.\"</td>\n",
       "      <td>https://twitter.com/i/web/status/848416049573658624</td>\n",
       "      <td>@DaveLeeBBC, @verge</td>\n",
       "      <td>2</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "849636868052275200 2017-04-05 14:56:29   \n",
       "848988730585096192 2017-04-03 20:01:01   \n",
       "848943072423497728 2017-04-03 16:59:35   \n",
       "848935705057280001 2017-04-03 16:30:19   \n",
       "848416049573658624 2017-04-02 06:05:23   \n",
       "\n",
       "                                                                                                                                                                         tweet  \\\n",
       "id                                                                                                                                                                               \n",
       "849636868052275200                                                                                            b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'   \n",
       "848988730585096192  b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"   \n",
       "848943072423497728                                                                                                               b'@waltmossberg @mims @defcon_5 Et tu, Walt?'   \n",
       "848935705057280001                                                                                                                         b'Stormy weather in Shortville ...'   \n",
       "848416049573658624                                                                           b\"@DaveLeeBBC @verge Coal is dying due to nat gas fracking. It's basically dead.\"   \n",
       "\n",
       "                                                             tweet_link  \\\n",
       "id                                                                        \n",
       "849636868052275200  https://twitter.com/i/web/status/849636868052275200   \n",
       "848988730585096192  https://twitter.com/i/web/status/848988730585096192   \n",
       "848943072423497728  https://twitter.com/i/web/status/848943072423497728   \n",
       "848935705057280001  https://twitter.com/i/web/status/848935705057280001   \n",
       "848416049573658624  https://twitter.com/i/web/status/848416049573658624   \n",
       "\n",
       "                                                 tweet_mentions  n_mentions  \\\n",
       "id                                                                            \n",
       "849636868052275200                                         None           0   \n",
       "848988730585096192  @ForIn2020, @waltmossberg, @mims, @defcon_5           4   \n",
       "848943072423497728              @waltmossberg, @mims, @defcon_5           3   \n",
       "848935705057280001                                         None           0   \n",
       "848416049573658624                          @DaveLeeBBC, @verge           2   \n",
       "\n",
       "                   week_day_name  week_day  \n",
       "id                                          \n",
       "849636868052275200     Wednesday         2  \n",
       "848988730585096192        Monday         0  \n",
       "848943072423497728        Monday         0  \n",
       "848935705057280001        Monday         0  \n",
       "848416049573658624        Sunday         6  "
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add column with week day"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"week_day_name\"] = df['created_at'].dt.day_name()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"week_day\"] = df['created_at'].dt.weekday"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>week_day</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'</td>\n",
       "      <td>https://twitter.com/i/web/status/849636868052275200</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"</td>\n",
       "      <td>https://twitter.com/i/web/status/848988730585096192</td>\n",
       "      <td>@ForIn2020, @waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>4</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "      <td>https://twitter.com/i/web/status/848943072423497728</td>\n",
       "      <td>@waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>3</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "849636868052275200 2017-04-05 14:56:29   \n",
       "848988730585096192 2017-04-03 20:01:01   \n",
       "848943072423497728 2017-04-03 16:59:35   \n",
       "\n",
       "                                                                                                                                                                         tweet  \\\n",
       "id                                                                                                                                                                               \n",
       "849636868052275200                                                                                            b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'   \n",
       "848988730585096192  b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"   \n",
       "848943072423497728                                                                                                               b'@waltmossberg @mims @defcon_5 Et tu, Walt?'   \n",
       "\n",
       "                                                             tweet_link  \\\n",
       "id                                                                        \n",
       "849636868052275200  https://twitter.com/i/web/status/849636868052275200   \n",
       "848988730585096192  https://twitter.com/i/web/status/848988730585096192   \n",
       "848943072423497728  https://twitter.com/i/web/status/848943072423497728   \n",
       "\n",
       "                                                 tweet_mentions  n_mentions  \\\n",
       "id                                                                            \n",
       "849636868052275200                                         None           0   \n",
       "848988730585096192  @ForIn2020, @waltmossberg, @mims, @defcon_5           4   \n",
       "848943072423497728              @waltmossberg, @mims, @defcon_5           3   \n",
       "\n",
       "                   week_day_name  week_day  \n",
       "id                                          \n",
       "849636868052275200     Wednesday         2  \n",
       "848988730585096192        Monday         0  \n",
       "848943072423497728        Monday         0  "
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "###  Selecting rows\n",
    "\n",
    "Filtering rows in `pandas` is done by means of `[ ]`, which can contain the row number as well as a condition for the selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>week_day</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "      <td>https://twitter.com/i/web/status/848943072423497728</td>\n",
       "      <td>@waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>3</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>b'Stormy weather in Shortville ...'</td>\n",
       "      <td>https://twitter.com/i/web/status/848935705057280001</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "848943072423497728 2017-04-03 16:59:35   \n",
       "848935705057280001 2017-04-03 16:30:19   \n",
       "\n",
       "                                                            tweet  \\\n",
       "id                                                                  \n",
       "848943072423497728  b'@waltmossberg @mims @defcon_5 Et tu, Walt?'   \n",
       "848935705057280001            b'Stormy weather in Shortville ...'   \n",
       "\n",
       "                                                             tweet_link  \\\n",
       "id                                                                        \n",
       "848943072423497728  https://twitter.com/i/web/status/848943072423497728   \n",
       "848935705057280001  https://twitter.com/i/web/status/848935705057280001   \n",
       "\n",
       "                                     tweet_mentions  n_mentions week_day_name  \\\n",
       "id                                                                              \n",
       "848943072423497728  @waltmossberg, @mims, @defcon_5           3        Monday   \n",
       "848935705057280001                             None           0        Monday   \n",
       "\n",
       "                    week_day  \n",
       "id                            \n",
       "848943072423497728         0  \n",
       "848935705057280001         0  "
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[2:4]\n",
    "# equivalent to df.iloc[2:4]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or all rows up to a certain row number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>week_day</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'</td>\n",
       "      <td>https://twitter.com/i/web/status/849636868052275200</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"</td>\n",
       "      <td>https://twitter.com/i/web/status/848988730585096192</td>\n",
       "      <td>@ForIn2020, @waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>4</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "      <td>https://twitter.com/i/web/status/848943072423497728</td>\n",
       "      <td>@waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>3</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>b'Stormy weather in Shortville ...'</td>\n",
       "      <td>https://twitter.com/i/web/status/848935705057280001</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "849636868052275200 2017-04-05 14:56:29   \n",
       "848988730585096192 2017-04-03 20:01:01   \n",
       "848943072423497728 2017-04-03 16:59:35   \n",
       "848935705057280001 2017-04-03 16:30:19   \n",
       "\n",
       "                                                                                                                                                                         tweet  \\\n",
       "id                                                                                                                                                                               \n",
       "849636868052275200                                                                                            b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'   \n",
       "848988730585096192  b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"   \n",
       "848943072423497728                                                                                                               b'@waltmossberg @mims @defcon_5 Et tu, Walt?'   \n",
       "848935705057280001                                                                                                                         b'Stormy weather in Shortville ...'   \n",
       "\n",
       "                                                             tweet_link  \\\n",
       "id                                                                        \n",
       "849636868052275200  https://twitter.com/i/web/status/849636868052275200   \n",
       "848988730585096192  https://twitter.com/i/web/status/848988730585096192   \n",
       "848943072423497728  https://twitter.com/i/web/status/848943072423497728   \n",
       "848935705057280001  https://twitter.com/i/web/status/848935705057280001   \n",
       "\n",
       "                                                 tweet_mentions  n_mentions  \\\n",
       "id                                                                            \n",
       "849636868052275200                                         None           0   \n",
       "848988730585096192  @ForIn2020, @waltmossberg, @mims, @defcon_5           4   \n",
       "848943072423497728              @waltmossberg, @mims, @defcon_5           3   \n",
       "848935705057280001                                         None           0   \n",
       "\n",
       "                   week_day_name  week_day  \n",
       "id                                          \n",
       "849636868052275200     Wednesday         2  \n",
       "848988730585096192        Monday         0  \n",
       "848943072423497728        Monday         0  \n",
       "848935705057280001        Monday         0  "
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:4]\n",
    "# equivalent to df.iloc[:4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2819, 7)"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1674"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# equivalent of `df.query('n_mentions > 0')`\n",
    "\n",
    "len(df[df['n_mentions'] > 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2819, 7)"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['n_mentions'] >= 0].shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(426, 7)"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['week_day_name'] == 'Saturday'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(848, 7)"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df['week_day_name'].str.startswith('S')].shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multiple conditions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(187, 7)"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# AND condition with `&`\n",
    "\n",
    "df[\n",
    "    (df['week_day_name'] == 'Saturday') & (df['n_mentions'] == 0)\n",
    "].shape\n",
    "\n",
    "# equivalent to df.query(\"week_day_name == 'Saturday' and n_mentions == 0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1384, 7)"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# OR condition with `|`\n",
    "\n",
    "df[\n",
    "    (df['week_day_name'] == 'Saturday') | (df['n_mentions'] == 0)\n",
    "].shape"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add column with day hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.created_at.dt?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id\n",
       "849636868052275200    14\n",
       "848988730585096192    20\n",
       "848943072423497728    16\n",
       "848935705057280001    16\n",
       "848416049573658624     6\n",
       "Name: created_at, dtype: int32"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['created_at'].dt.hour.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"day_hour\"] = df['created_at'].dt.hour"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>week_day</th>\n",
       "      <th>day_hour</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  week_day  day_hour\n",
       "id                                                        \n",
       "849636868052275200 2017-04-05 14:56:29         2        14\n",
       "848988730585096192 2017-04-03 20:01:01         0        20\n",
       "848943072423497728 2017-04-03 16:59:35         0        16\n",
       "848935705057280001 2017-04-03 16:30:19         0        16"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_cols = ['created_at', 'week_day', 'day_hour']\n",
    "df[display_cols].head(4)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Aggregation\n",
    "\n",
    "(Figure taken from W. Mckinney's *Python for Data Analysis* 2013, p. 252)\n",
    "\n",
    "<img src='figures/groupby-mechanics.png' width='600px'>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['created_at', 'tweet', 'tweet_link', 'tweet_mentions', 'n_mentions',\n",
       "       'week_day_name', 'week_day', 'day_hour'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_mentions</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>2277.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.807733</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       n_mentions\n",
       "min      0.000000\n",
       "max      6.000000\n",
       "sum   2277.000000\n",
       "mean     0.807733"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.agg({'n_mentions': ['min', 'max', 'sum', 'mean']})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Grouping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.groupby?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "grp_by_day = df.groupby('week_day')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.groupby.generic.DataFrameGroupBy"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(grp_by_day)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The object returned by `gropuby` is a `DataFrameGroupBy` **not** a normal `DataFrame`.\n",
    "\n",
    "However, some methods of the latter work also on the former, e.g. `head` and `tail`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the head of a DataFrameGroupBy consists of the first\n",
    "# n records for each group (see `help(grp_by_day.head)`)\n",
    "\n",
    "grp_by_day.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`agg` is used to pass an aggregation function to be applied to each group resulting from `groupby`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>day_hour</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>week_day</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>315</td>\n",
       "      <td>315</td>\n",
       "      <td>315</td>\n",
       "      <td>315</td>\n",
       "      <td>315</td>\n",
       "      <td>315</td>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>385</td>\n",
       "      <td>385</td>\n",
       "      <td>385</td>\n",
       "      <td>385</td>\n",
       "      <td>385</td>\n",
       "      <td>385</td>\n",
       "      <td>385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>380</td>\n",
       "      <td>380</td>\n",
       "      <td>380</td>\n",
       "      <td>380</td>\n",
       "      <td>380</td>\n",
       "      <td>380</td>\n",
       "      <td>380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>361</td>\n",
       "      <td>361</td>\n",
       "      <td>361</td>\n",
       "      <td>361</td>\n",
       "      <td>361</td>\n",
       "      <td>361</td>\n",
       "      <td>361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>530</td>\n",
       "      <td>530</td>\n",
       "      <td>530</td>\n",
       "      <td>530</td>\n",
       "      <td>530</td>\n",
       "      <td>530</td>\n",
       "      <td>530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>426</td>\n",
       "      <td>426</td>\n",
       "      <td>426</td>\n",
       "      <td>426</td>\n",
       "      <td>426</td>\n",
       "      <td>426</td>\n",
       "      <td>426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>422</td>\n",
       "      <td>422</td>\n",
       "      <td>422</td>\n",
       "      <td>422</td>\n",
       "      <td>422</td>\n",
       "      <td>422</td>\n",
       "      <td>422</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          created_at  tweet  tweet_link  tweet_mentions  n_mentions  \\\n",
       "week_day                                                              \n",
       "0                315    315         315             315         315   \n",
       "1                385    385         385             385         385   \n",
       "2                380    380         380             380         380   \n",
       "3                361    361         361             361         361   \n",
       "4                530    530         530             530         530   \n",
       "5                426    426         426             426         426   \n",
       "6                422    422         422             422         422   \n",
       "\n",
       "          week_day_name  day_hour  \n",
       "week_day                           \n",
       "0                   315       315  \n",
       "1                   385       385  \n",
       "2                   380       380  \n",
       "3                   361       361  \n",
       "4                   530       530  \n",
       "5                   426       426  \n",
       "6                   422       422  "
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we are interested in how many tweets\n",
    "# there are for each group, so we pass `len()`\n",
    "\n",
    "grp_by_day.agg(len)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>week_day</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>315</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>530</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>426</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>422</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          count\n",
       "week_day       \n",
       "0           315\n",
       "1           385\n",
       "2           380\n",
       "3           361\n",
       "4           530\n",
       "5           426\n",
       "6           422"
      ]
     },
     "execution_count": 219,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# however, we are not interested in having the count for all\n",
    "# columns. rather we want to create a new dataframe with renamed\n",
    "# column names\n",
    "\n",
    "grp_by_day.agg({'tweet': len}).rename({'tweet': 'count'}, axis='columns')\n",
    "\n",
    "# equivalent: grp_by_day['tweet'].count().to_frame().rename({'tweet':'count'},axis='columns')\n",
    "# OR: grp_by_day[['tweet']].count().rename({'tweet':'count'},axis='columns')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### By label (column)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Previously we've added a column indicating on which day of the week a given tweet appeared."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [],
   "source": [
    "grpby_result_as_series = df.groupby('day_hour')['tweet'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "day_hour\n",
       "0     144\n",
       "1     183\n",
       "2     119\n",
       "3     104\n",
       "4      98\n",
       "5      81\n",
       "6      85\n",
       "7      94\n",
       "8      82\n",
       "9      55\n",
       "10     34\n",
       "11     15\n",
       "12     17\n",
       "13     23\n",
       "14    114\n",
       "15    190\n",
       "16    198\n",
       "17    190\n",
       "18    162\n",
       "19    154\n",
       "20    176\n",
       "21    192\n",
       "22    168\n",
       "23    141\n",
       "Name: tweet, dtype: int64"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grpby_result_as_series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "grpby_result_as_df = df.groupby('day_hour')[['tweet']]\\\n",
    "    .count()\\\n",
    "    .rename({'tweet': 'count'}, axis='columns')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>day_hour</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          tweet\n",
       "day_hour       \n",
       "0           144\n",
       "1           183\n",
       "2           119\n",
       "3           104\n",
       "4            98"
      ]
     },
     "execution_count": 178,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grpby_result_as_df.head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### By series or dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.groupby?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 <class 'pandas.core.frame.DataFrame'>\n",
      "2 <class 'pandas.core.frame.DataFrame'>\n",
      "3 <class 'pandas.core.frame.DataFrame'>\n",
      "4 <class 'pandas.core.frame.DataFrame'>\n",
      "5 <class 'pandas.core.frame.DataFrame'>\n",
      "6 <class 'pandas.core.frame.DataFrame'>\n",
      "7 <class 'pandas.core.frame.DataFrame'>\n",
      "8 <class 'pandas.core.frame.DataFrame'>\n",
      "9 <class 'pandas.core.frame.DataFrame'>\n",
      "10 <class 'pandas.core.frame.DataFrame'>\n",
      "11 <class 'pandas.core.frame.DataFrame'>\n",
      "12 <class 'pandas.core.frame.DataFrame'>\n",
      "13 <class 'pandas.core.frame.DataFrame'>\n",
      "14 <class 'pandas.core.frame.DataFrame'>\n",
      "15 <class 'pandas.core.frame.DataFrame'>\n",
      "16 <class 'pandas.core.frame.DataFrame'>\n",
      "17 <class 'pandas.core.frame.DataFrame'>\n",
      "18 <class 'pandas.core.frame.DataFrame'>\n",
      "19 <class 'pandas.core.frame.DataFrame'>\n",
      "20 <class 'pandas.core.frame.DataFrame'>\n",
      "21 <class 'pandas.core.frame.DataFrame'>\n",
      "22 <class 'pandas.core.frame.DataFrame'>\n",
      "23 <class 'pandas.core.frame.DataFrame'>\n",
      "24 <class 'pandas.core.frame.DataFrame'>\n",
      "25 <class 'pandas.core.frame.DataFrame'>\n",
      "26 <class 'pandas.core.frame.DataFrame'>\n",
      "27 <class 'pandas.core.frame.DataFrame'>\n",
      "28 <class 'pandas.core.frame.DataFrame'>\n",
      "29 <class 'pandas.core.frame.DataFrame'>\n",
      "30 <class 'pandas.core.frame.DataFrame'>\n",
      "31 <class 'pandas.core.frame.DataFrame'>\n"
     ]
    }
   ],
   "source": [
    "for group, rows in df.groupby(df['created_at'].dt.day):\n",
    "    print(group, type(rows))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>created_at</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            tweet\n",
       "created_at       \n",
       "1             107\n",
       "2             107\n",
       "3             114\n",
       "4              97\n",
       "5             117"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we pass the groups as a series\n",
    "df.groupby(df['created_at'].dt.day).agg({'tweet':len}).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>created_at</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>97</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>117</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            tweet\n",
       "created_at       \n",
       "1             107\n",
       "2             107\n",
       "3             114\n",
       "4              97\n",
       "5             117"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we pass the groups as a series\n",
    "df.groupby(df['created_at'].dt.day)[['tweet']].count().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>created_at</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>183</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>119</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>104</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>98</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            tweet\n",
       "created_at       \n",
       "0             144\n",
       "1             183\n",
       "2             119\n",
       "3             104\n",
       "4              98"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby(df['created_at'].dt.hour)[['tweet']].count().head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could save the count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df.groupby(df['created_at'].dt.hour)[['tweet']].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='created_at'>"
      ]
     },
     "execution_count": 232,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAigAAAG0CAYAAAAYQdwgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA7+0lEQVR4nO3dd3hUdfr//9ekB0gCCYQkEpLQBJYqJUYpQVHagiiroriAIqCCCFgwa6FYwLJ2hJVVsICg30UUWHHpRQEpAhZEShCUBBCEEDABkvv3Bz/m45iAmZAxJ/H5uK5zXTnnfeaee1JmXnmfM2dcZmYCAABwEL/SbgAAAOC3CCgAAMBxCCgAAMBxCCgAAMBxCCgAAMBxCCgAAMBxCCgAAMBxAkq7geLIz8/Xvn37FBYWJpfLVdrtAACAIjAzHTt2THFxcfLzO/8cSZkMKPv27VN8fHxptwEAAIph7969qlGjxnn3KZMBJSwsTNKZBxgeHl7K3QAAgKLIyspSfHy8+3X8fMpkQDl7WCc8PJyAAgBAGVOU0zM4SRYAADgOAQUAADgOAQUAADhOmTwHpajy8vJ06tSp0m6j3AkKCvrdt4cBAHAhymVAMTNlZmbqyJEjpd1KueTn56ekpCQFBQWVdisAgHKqXAaUs+EkOjpaFSpU4GJuJejsRfIyMjJUs2ZNvrcAAJ8odwElLy/PHU6ioqJKu51yqVq1atq3b59Onz6twMDA0m4HAFAOlbsTCc6ec1KhQoVS7qT8OntoJy8vr5Q7AQCUV14FlPHjx6tVq1YKCwtTdHS0evbsqW3btnnsk5OToyFDhigqKkqVKlVSr169tH//fo999uzZo27duqlChQqKjo7W/fffr9OnT1/4o/kVDj34Dt9bAICveRVQli9friFDhmjNmjVauHChTp06pauvvlrHjx937zNixAjNnTtX77//vpYvX659+/bpuuuuc4/n5eWpW7duOnnypD777DO9+eabmjZtmh599NGSe1QAAKBMc5mZFffGBw8eVHR0tJYvX6527drp6NGjqlatmmbMmKG//e1vkqRvv/1WDRo00OrVq3XppZfq448/1l//+lft27dP1atXlyRNnjxZo0aN0sGDBwt9Z0hubq5yc3Pd62ev5X/06NECl7rPyclRenq6kpKSFBISUtyHhvPgewwAKI6srCxFREQU+vr9Wxd0kuzRo0clSZGRkZKkDRs26NSpU+rYsaN7n/r166tmzZrugLJ69Wo1btzYHU4kqVOnTrrzzjv19ddfq3nz5gXuZ/z48Ro7duyFtCpJSnxw/gXXKKrdE7r9YfcFAEB5U+yTZPPz8zV8+HBdfvnlatSokaQzb+8NCgpS5cqVPfatXr26MjMz3fv8OpycHT87Vpi0tDQdPXrUvezdu7e4bTtaamqqhg8fXqo9JCYm6oUXXijVHgAAKPYMypAhQ/TVV19p1apVJdlPoYKDgxUcHOzz+wEAAM5QrBmUoUOHat68eVq6dKlq1Kjh3h4TE6OTJ08WuILr/v37FRMT497nt+/qObt+dp8/o/79+2v58uV68cUX5XK55HK5VLVqVT377LPufXr27KnAwEBlZ2dLkn744Qe5XC7t2LFD0plzde677z5ddNFFqlixopKTk7Vs2TKP+1m1apXatm2r0NBQxcfHa9iwYe6TnFNTU/X9999rxIgR7h4AACgNXgUUM9PQoUP1wQcfaMmSJUpKSvIYb9GihQIDA7V48WL3tm3btmnPnj1KSUmRJKWkpOjLL7/UgQMH3PssXLhQ4eHhatiw4YU8ljLtxRdfVEpKigYOHKiMjAxlZGTo73//uztgmJlWrlypypUru2etli9frosuukh16tSRdCY4rl69WjNnztSWLVt0/fXXq3Pnztq+fbskaefOnercubN69eqlLVu2aNasWVq1apWGDh0qSZo9e7Zq1KihcePGuXsAUP4kPji/SAtQmrwKKEOGDNE777yjGTNmKCwsTJmZmcrMzNQvv/wiSYqIiNCAAQM0cuRILV26VBs2bNCtt96qlJQUXXrppZKkq6++Wg0bNtTf//53bd68WZ988okefvhhDRky5E99GCciIkJBQUGqUKGCYmJiFBMToyuuuEKrVq1SXl6etmzZoqCgIPXp08cdWpYtW6b27dtLOnNtmalTp+r9999X27ZtVbt2bd13331q06aNpk6dKunMycZ9+vTR8OHDVbduXV122WV66aWX9NZbbyknJ0eRkZHy9/dXWFiYuwcAAEqDV+egTJo0SdKZQwG/NnXqVPXv31+S9Pzzz8vPz0+9evVSbm6uOnXqpFdffdW9r7+/v+bNm6c777xTKSkpqlixovr166dx48Zd2CMph9q2batjx47piy++0Geffab27dsrNTVVEyZMkHRmBuX++++XJH355ZfKy8tTvXr1PGrk5ua6L/m/efNmbdmyRdOnT3ePm5ny8/OVnp6uBg0a/EGPDACA8/MqoBTlkikhISGaOHGiJk6ceM59EhIS9N///tebu/5Tqly5spo2baply5Zp9erVuuqqq9SuXTvdeOON+u6777R9+3b3DEp2drb8/f21YcMG+fv7e9SpVKmSe5/Bgwdr2LBhBe6rZs2avn9AAAAUUbn7sMCyLCgoqMDn27Rv315Lly7V559/rieeeEKRkZFq0KCBnnjiCcXGxrpnTJo3b668vDwdOHBAbdu2LbT+JZdcom+++cZ9zkpRewAA4I9W7j4ssCxLTEzU2rVrtXv3bv3000/Kz89XamqqPvnkEwUEBKh+/fqSzhximz59unv2RJLq1aunPn36qG/fvpo9e7bS09P1+eefa/z48Zo//8zJbqNGjdJnn32moUOHatOmTdq+fbs+/PBD90myZ3tYsWKFfvzxR/30009/7DcAAID/359qBsXpV3e977771K9fPzVs2FC//PKL0tPT1bZtW+Xn53uEkdTUVL344ouFngv0+OOP695779WPP/6oqlWr6tJLL9Vf//pXSVKTJk20fPlyPfTQQ2rbtq3MTLVr19aNN97orjFu3DgNHjxYtWvXVm5ubpEO6wEAUNIu6LN4Ssv5ruXP58T4Ht9joGwr6luInf5PHcoebz6Lh0M8AADAcQgoAADAcf5U56AAAOBk3lzBt7wfgmMGBQAAOE65DSj5+fml3UK5VQbPqwYAlDHl7hBPUFCQ/Pz8tG/fPlWrVk1BQUF8Km8JMjMdPHhQLpdLgYGBpd0OAKCcKncBxc/PT0lJScrIyNC+fftKu51yyeVyqUaNGgUuqQ+g5JWVcxJ46zJKWrkLKNKZWZSaNWvq9OnTXLbdBwIDAwknAACfKpcBRZL7EASHIQAAKHvK7UmyAACg7CKgAAAAxyGgAAAAxyGgAAAAxyGgAAAAxyGgAAAAxyGgAAAAxyGgAAAAxyGgAAAAxym3V5ItKXy+BAAAfzxmUAAAgOMQUAAAgOMQUAAAgOMQUAAAgOMQUAAAgOMQUAAAgOMQUAAAgOMQUAAAgOMQUAAAgOMQUAAAgON4HVBWrFih7t27Ky4uTi6XS3PmzPEYd7lchS7PPPOMe5/ExMQC4xMmTLjgBwMAAMoHrwPK8ePH1bRpU02cOLHQ8YyMDI/ljTfekMvlUq9evTz2GzdunMd+d999d/EeAQAAKHe8/rDALl26qEuXLuccj4mJ8Vj/8MMP1aFDB9WqVctje1hYWIF9AQAoS/hAWd/x6Tko+/fv1/z58zVgwIACYxMmTFBUVJSaN2+uZ555RqdPnz5nndzcXGVlZXksAACg/PJ6BsUbb775psLCwnTdddd5bB82bJguueQSRUZG6rPPPlNaWpoyMjL03HPPFVpn/PjxGjt2rC9bBQAADuLTgPLGG2+oT58+CgkJ8dg+cuRI99dNmjRRUFCQBg8erPHjxys4OLhAnbS0NI/bZGVlKT4+3neNAwCAUuWzgLJy5Upt27ZNs2bN+t19k5OTdfr0ae3evVsXX3xxgfHg4OBCgwsAACiffHYOyuuvv64WLVqoadOmv7vvpk2b5Ofnp+joaF+1AwAAyhCvZ1Cys7O1Y8cO93p6ero2bdqkyMhI1axZU9KZQzDvv/++/vnPfxa4/erVq7V27Vp16NBBYWFhWr16tUaMGKFbbrlFVapUuYCHAgAAyguvA8r69evVoUMH9/rZc0P69eunadOmSZJmzpwpM9NNN91U4PbBwcGaOXOmxowZo9zcXCUlJWnEiBEe55gAAIA/N68DSmpqqszsvPsMGjRIgwYNKnTskksu0Zo1a7y9WwAA8CfCZ/EAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADH8eml7gEAQOkqq5+4zAwKAABwHAIKAABwHAIKAABwHAIKAABwHAIKAABwHAIKAABwHAIKAABwHAIKAABwHAIKAABwHAIKAABwHC51DwBwnKJenl1y3iXaUTKYQQEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI5DQAEAAI7DpxkDAP4UivoJyXw6sjMwgwIAAByHgAIAABzH64CyYsUKde/eXXFxcXK5XJozZ47HeP/+/eVyuTyWzp07e+xz+PBh9enTR+Hh4apcubIGDBig7OzsC3ogAACg/PA6oBw/flxNmzbVxIkTz7lP586dlZGR4V7effddj/E+ffro66+/1sKFCzVv3jytWLFCgwYN8r57AABQLnl9kmyXLl3UpUuX8+4THBysmJiYQse2bt2qBQsWaN26dWrZsqUk6eWXX1bXrl317LPPKi4uztuWAABAOeOTc1CWLVum6OhoXXzxxbrzzjt16NAh99jq1atVuXJldziRpI4dO8rPz09r164ttF5ubq6ysrI8FgAAUH6VeEDp3Lmz3nrrLS1evFhPPfWUli9fri5duigvL0+SlJmZqejoaI/bBAQEKDIyUpmZmYXWHD9+vCIiItxLfHx8SbcNAAAcpMSvg9K7d2/3140bN1aTJk1Uu3ZtLVu2TFdeeWWxaqalpWnkyJHu9aysLEIKAADlmM/fZlyrVi1VrVpVO3bskCTFxMTowIEDHvucPn1ahw8fPud5K8HBwQoPD/dYAABA+eXzgPLDDz/o0KFDio2NlSSlpKToyJEj2rBhg3ufJUuWKD8/X8nJyb5uBwAAlAFeH+LJzs52z4ZIUnp6ujZt2qTIyEhFRkZq7Nix6tWrl2JiYrRz50498MADqlOnjjp16iRJatCggTp37qyBAwdq8uTJOnXqlIYOHarevXvzDh4AACCpGDMo69evV/PmzdW8eXNJ0siRI9W8eXM9+uij8vf315YtW9SjRw/Vq1dPAwYMUIsWLbRy5UoFBwe7a0yfPl3169fXlVdeqa5du6pNmzZ67bXXSu5RAQCAMs3rGZTU1FSZ2TnHP/nkk9+tERkZqRkzZnh71wAA4E+Cz+IBAACOQ0ABAACOQ0ABAACOQ0ABAACOU+JXki1NiQ/OL/K+uyd082EnAADgQjCDAgAAHIeAAgAAHIeAAgAAHIeAAgAAHKdcnSQLAAB8r6hvSrmQN6QwgwIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAAByHgAIAABzH64CyYsUKde/eXXFxcXK5XJozZ4577NSpUxo1apQaN26sihUrKi4uTn379tW+ffs8aiQmJsrlcnksEyZMuOAHAwAAygevA8rx48fVtGlTTZw4scDYiRMntHHjRj3yyCPauHGjZs+erW3btqlHjx4F9h03bpwyMjLcy9133128RwAAAMqdAG9v0KVLF3Xp0qXQsYiICC1cuNBj2yuvvKLWrVtrz549qlmzpnt7WFiYYmJivL17AADwJ+Dzc1COHj0ql8ulypUre2yfMGGCoqKi1Lx5cz3zzDM6ffr0OWvk5uYqKyvLYwEAAOWX1zMo3sjJydGoUaN00003KTw83L192LBhuuSSSxQZGanPPvtMaWlpysjI0HPPPVdonfHjx2vs2LG+bBUAADiIzwLKqVOndMMNN8jMNGnSJI+xkSNHur9u0qSJgoKCNHjwYI0fP17BwcEFaqWlpXncJisrS/Hx8b5qHQAAlDKfBJSz4eT777/XkiVLPGZPCpOcnKzTp09r9+7duvjiiwuMBwcHFxpcAABA+VTiAeVsONm+fbuWLl2qqKio373Npk2b5Ofnp+jo6JJuBwAAlEFeB5Ts7Gzt2LHDvZ6enq5NmzYpMjJSsbGx+tvf/qaNGzdq3rx5ysvLU2ZmpiQpMjJSQUFBWr16tdauXasOHTooLCxMq1ev1ogRI3TLLbeoSpUqJffIAABAmeV1QFm/fr06dOjgXj97bki/fv00ZswYffTRR5KkZs2aedxu6dKlSk1NVXBwsGbOnKkxY8YoNzdXSUlJGjFihMc5JgAA4M/N64CSmpoqMzvn+PnGJOmSSy7RmjVrvL1bAADwJ8Jn8QAAAMchoAAAAMchoAAAAMfx6ZVkUbjEB+cXab/dE7r5uBMAAJyJGRQAAOA4BBQAAOA4BBQAAOA4BBQAAOA4BBQAAOA4BBQAAOA4BBQAAOA4BBQAAOA4XKgNAEoQF2IESgYzKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHEIKAAAwHECSrsBXLjEB+cXed/dE7r5sBMAAEoGMygAAMBxCCgAAMBxvA4oK1asUPfu3RUXFyeXy6U5c+Z4jJuZHn30UcXGxio0NFQdO3bU9u3bPfY5fPiw+vTpo/DwcFWuXFkDBgxQdnb2BT0QAABQfngdUI4fP66mTZtq4sSJhY4//fTTeumllzR58mStXbtWFStWVKdOnZSTk+Pep0+fPvr666+1cOFCzZs3TytWrNCgQYOK/ygAAEC54vVJsl26dFGXLl0KHTMzvfDCC3r44Yd1zTXXSJLeeustVa9eXXPmzFHv3r21detWLViwQOvWrVPLli0lSS+//LK6du2qZ599VnFxcRfwcAAAQHlQouegpKenKzMzUx07dnRvi4iIUHJyslavXi1JWr16tSpXruwOJ5LUsWNH+fn5ae3atYXWzc3NVVZWlscCAADKrxINKJmZmZKk6tWre2yvXr26eywzM1PR0dEe4wEBAYqMjHTv81vjx49XRESEe4mPjy/JtgEAgMOUiXfxpKWl6ejRo+5l7969pd0SAADwoRINKDExMZKk/fv3e2zfv3+/eywmJkYHDhzwGD99+rQOHz7s3ue3goODFR4e7rEAAIDyq0QDSlJSkmJiYrR48WL3tqysLK1du1YpKSmSpJSUFB05ckQbNmxw77NkyRLl5+crOTm5JNsBAABllNfv4snOztaOHTvc6+np6dq0aZMiIyNVs2ZNDR8+XI8//rjq1q2rpKQkPfLII4qLi1PPnj0lSQ0aNFDnzp01cOBATZ48WadOndLQoUPVu3dv3sHjIEW9fD6XzgcA+ILXAWX9+vXq0KGDe33kyJGSpH79+mnatGl64IEHdPz4cQ0aNEhHjhxRmzZttGDBAoWEhLhvM336dA0dOlRXXnml/Pz81KtXL7300ksl8HAAAEB54HVASU1NlZmdc9zlcmncuHEaN27cOfeJjIzUjBkzvL1rAADwJ1Em3sUDAAD+XAgoAADAcQgoAADAcQgoAADAcbw+SRYoz3h7NQA4AzMoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQgoAADAcQJKuwGguBIfnF/kfXdP6ObDTgAAJY0ZFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DicJAv4WFFP5uVEXgD4P8ygAAAAx2EGBX8YZhIAAEXFDAoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHCcEg8oiYmJcrlcBZYhQ4ZIklJTUwuM3XHHHSXdBgAAKMNK/Doo69atU15ennv9q6++0lVXXaXrr7/evW3gwIEaN26ce71ChQol3QYAACjDSjygVKtWzWN9woQJql27ttq3b+/eVqFCBcXExBS5Zm5urnJzc93rWVlZF94oAABwLJ+eg3Ly5Em98847uu222+Ryudzbp0+frqpVq6pRo0ZKS0vTiRMnzltn/PjxioiIcC/x8fG+bBsAAJQyn17qfs6cOTpy5Ij69+/v3nbzzTcrISFBcXFx2rJli0aNGqVt27Zp9uzZ56yTlpamkSNHutezsrIIKQAAlGM+DSivv/66unTpori4OPe2QYMGub9u3LixYmNjdeWVV2rnzp2qXbt2oXWCg4MVHBzsy1YBAICD+OwQz/fff69Fixbp9ttvP+9+ycnJkqQdO3b4qhUAAFDG+CygTJ06VdHR0erW7fyfTLtp0yZJUmxsrK9aAQAAZYxPDvHk5+dr6tSp6tevnwIC/u8udu7cqRkzZqhr166KiorSli1bNGLECLVr105NmjTxRSsAAKAM8klAWbRokfbs2aPbbrvNY3tQUJAWLVqkF154QcePH1d8fLx69eqlhx9+2BdtAACAMsonAeXqq6+WmRXYHh8fr+XLl/viLgEAQDnCZ/EAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHIaAAAADHCSjtBgB4J/HB+UXed/eEbj7sBAB8hxkUAADgOAQUAADgOAQUAADgOAQUAADgOCUeUMaMGSOXy+Wx1K9f3z2ek5OjIUOGKCoqSpUqVVKvXr20f//+km4DAACUYT6ZQfnLX/6ijIwM97Jq1Sr32IgRIzR37ly9//77Wr58ufbt26frrrvOF20AAIAyyidvMw4ICFBMTEyB7UePHtXrr7+uGTNm6IorrpAkTZ06VQ0aNNCaNWt06aWX+qIdAABQxvhkBmX79u2Ki4tTrVq11KdPH+3Zs0eStGHDBp06dUodO3Z071u/fn3VrFlTq1evPme93NxcZWVleSwAAKD8KvGAkpycrGnTpmnBggWaNGmS0tPT1bZtWx07dkyZmZkKCgpS5cqVPW5TvXp1ZWZmnrPm+PHjFRER4V7i4+NLum0AAOAgJX6Ip0uXLu6vmzRpouTkZCUkJOi9995TaGhosWqmpaVp5MiR7vWsrCxCCgAA5ZjP32ZcuXJl1atXTzt27FBMTIxOnjypI0eOeOyzf//+Qs9ZOSs4OFjh4eEeCwAAKL98HlCys7O1c+dOxcbGqkWLFgoMDNTixYvd49u2bdOePXuUkpLi61YAAEAZUeKHeO677z51795dCQkJ2rdvn0aPHi1/f3/ddNNNioiI0IABAzRy5EhFRkYqPDxcd999t1JSUngHDwAAcCvxgPLDDz/opptu0qFDh1StWjW1adNGa9asUbVq1SRJzz//vPz8/NSrVy/l5uaqU6dOevXVV0u6DQAAUIaVeECZOXPmecdDQkI0ceJETZw4saTvGgAAlBN8Fg8AAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHCcgNJuAEDpS3xwfpH22z2hm487AYAzmEEBAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOQ0ABAACOU+IBZfz48WrVqpXCwsIUHR2tnj17atu2bR77pKamyuVyeSx33HFHSbcCAADKqBIPKMuXL9eQIUO0Zs0aLVy4UKdOndLVV1+t48ePe+w3cOBAZWRkuJenn366pFsBAABlVEBJF1ywYIHH+rRp0xQdHa0NGzaoXbt27u0VKlRQTExMkWrm5uYqNzfXvZ6VlVUyzQIAAEfy+TkoR48elSRFRkZ6bJ8+fbqqVq2qRo0aKS0tTSdOnDhnjfHjxysiIsK9xMfH+7RnAABQukp8BuXX8vPzNXz4cF1++eVq1KiRe/vNN9+shIQExcXFacuWLRo1apS2bdum2bNnF1onLS1NI0eOdK9nZWURUgAAKMd8GlCGDBmir776SqtWrfLYPmjQIPfXjRs3VmxsrK688krt3LlTtWvXLlAnODhYwcHBvmwVAAA4iM8O8QwdOlTz5s3T0qVLVaNGjfPum5ycLEnasWOHr9oBAABlSInPoJiZ7r77bn3wwQdatmyZkpKSfvc2mzZtkiTFxsaWdDsAcE6JD84v0n67J3TzcScAfqvEA8qQIUM0Y8YMffjhhwoLC1NmZqYkKSIiQqGhodq5c6dmzJihrl27KioqSlu2bNGIESPUrl07NWnSpKTbAQAAZVCJB5RJkyZJOnMxtl+bOnWq+vfvr6CgIC1atEgvvPCCjh8/rvj4ePXq1UsPP/xwSbcCAADKKJ8c4jmf+Ph4LV++vKTvFgAAlCN8Fg8AAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgoAAHAcn35YIACUlKJell7i0vRAecAMCgAAcBwCCgAAcBwO8QDwCT4pGMCFYAYFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4DgEFAAA4TqkGlIkTJyoxMVEhISFKTk7W559/XprtAAAAhyi1gDJr1iyNHDlSo0eP1saNG9W0aVN16tRJBw4cKK2WAACAQ5RaQHnuuec0cOBA3XrrrWrYsKEmT56sChUq6I033iitlgAAgEMElMadnjx5Uhs2bFBaWpp7m5+fnzp27KjVq1cX2D83N1e5ubnu9aNHj0qSsrKyPPbLzz1R5B5+e9tzKWrNotbzRU0e9x9fk8f9x9fkcf/xNXncf3zN8v64z66b2e/f2ErBjz/+aJLss88+89h+//33W+vWrQvsP3r0aJPEwsLCwsLCUg6WvXv3/m5WKJUZFG+lpaVp5MiR7vX8/HwdPnxYUVFRcrlc57xdVlaW4uPjtXfvXoWHh5dIL2WhZlno0Rc1y0KPvqhZFnr0Rc2y0KMvapaFHn1Rsyz06IuaZaFHb2qamY4dO6a4uLjfrVkqAaVq1ary9/fX/v37Pbbv379fMTExBfYPDg5WcHCwx7bKlSsX+f7Cw8NL7IdQlmqWhR59UbMs9OiLmmWhR1/ULAs9+qJmWejRFzXLQo++qFkWeixqzYiIiCLVKpWTZIOCgtSiRQstXrzYvS0/P1+LFy9WSkpKabQEAAAcpNQO8YwcOVL9+vVTy5Yt1bp1a73wwgs6fvy4br311tJqCQAAOESpBZQbb7xRBw8e1KOPPqrMzEw1a9ZMCxYsUPXq1UvsPoKDgzV69OgCh4fKe82y0KMvapaFHn1Rsyz06IuaZaFHX9QsCz36omZZ6NEXNctCj76q6TIrynt9AAAA/jh8Fg8AAHAcAgoAAHAcAgoAAHAcAgoAAHAcAgqKjPOpAQB/lDJxqfui+umnn/TGG29o9erVyszMlCTFxMTosssuU//+/VWtWrVS7rBsCw4O1ubNm9WgQYPSbuVPLSMjQ5MmTdKqVauUkZEhPz8/1apVSz179lT//v3l7+9f2i0CwAUrN28zXrdunTp16qQKFSqoY8eO7uup7N+/X4sXL9aJEyf0ySefqGXLliV6v3v37tXo0aP1xhtvFPk2v/zyizZs2KDIyEg1bNjQYywnJ0fvvfee+vbtW+R6W7du1Zo1a5SSkqL69evr22+/1Ysvvqjc3FzdcsstuuKKK4pcS5LH5x792osvvqhbbrlFUVFRkqTnnnvOq7q/dvz4cb333nvasWOHYmNjddNNN7nrFsXGjRtVpUoVJSUlSZLefvttTZ48WXv27FFCQoKGDh2q3r17e9XT3XffrRtuuEFt27b16na/55VXXtHnn3+url27qnfv3nr77bc1fvx45efn67rrrtO4ceMUEFC0/xXWr1+vjh07qk6dOgoNDdXq1at188036+TJk/rkk0/UsGFDLViwQGFhYSX6GACc2+eff17gH+OUlBS1bt26RO/n559/1ty5c716fTgrPz9ffn4FD5rk5+frhx9+UM2aNb2qZ2bavXu34uPjFRAQoJMnT+qDDz5Qbm6uunbtqqpVq3rdY2F3Ui4kJyfboEGDLD8/v8BYfn6+DRo0yC699NISv99NmzaZn59fkffftm2bJSQkmMvlMj8/P2vXrp3t27fPPZ6ZmelVvY8//tiCgoIsMjLSQkJC7OOPP7Zq1apZx44d7YorrjB/f39bvHixV4/J5XJZs2bNLDU11WNxuVzWqlUrS01NtQ4dOnhVs0GDBnbo0CEzM9uzZ48lJiZaRESEtWrVyiIjIy06Otp27dpV5HpNmjSxhQsXmpnZlClTLDQ01IYNG2aTJk2y4cOHW6VKlez111/3qsezP5O6devahAkTLCMjw6vbF+axxx6zsLAw69Wrl8XExNiECRMsKirKHn/8cXvyySetWrVq9uijjxa53uWXX25jxoxxr7/99tuWnJxsZmaHDx+2Zs2a2bBhw4rVa25urs2aNcuGDx9uvXv3tt69e9vw4cPtvffes9zc3GLVPJfMzEwbO3ZssW67d+9eO3bsWIHtJ0+etOXLl3td76effrIlS5a4fz8PHjxoEyZMsLFjx9o333xTrB5/Kykpyb777rsSqZWfn29Lliyx1157zebOnWsnT570usbevXvt4MGD7vUVK1bYzTffbG3atLE+ffoU+KT53/Pss8/a7t27ve7j98ydO9ceeeQRW7VqlZmZLV682Lp06WKdOnWyf/3rX8WqeeLECXv99dft1ltvtc6dO1vXrl1t6NChtmjRIq9r7d+/39q0aWMul8sSEhKsdevW1rp1a/dzfJs2bWz//v3F6rMw3r7emJkdPXrUrr/+egsJCbHo6Gh75JFH7PTp0+5xb19zzMy+/fZbS0hIMD8/P6tTp47t2rXLWrRoYRUrVrQKFSpY1apVS+T3vdwElJCQENu6des5x7du3WohISFe1/3www/Puzz//PNe/XB79uxp3bp1s4MHD9r27dutW7dulpSUZN9//72Zef/LkpKSYg899JCZmb377rtWpUoV+8c//uEef/DBB+2qq64qcj0zs/Hjx1tSUlKBYBMQEGBff/21V7XOcrlc7j/UPn362GWXXWZHjhwxM7Njx45Zx44d7aabbipyvdDQUPcTYvPmze21117zGJ8+fbo1bNjQ6x4XLVpk99xzj1WtWtUCAwOtR48eNnfuXMvLy/Oq1lm1a9e2//znP2Z25snF39/f3nnnHff47NmzrU6dOkWuFxoaajt37nSv5+XlWWBgoGVmZpqZ2f/+9z+Li4vzus/t27dbrVq1LCQkxNq3b2833HCD3XDDDda+fXsLCQmxOnXq2Pbt272uey7FeaLdt2+ftWrVyvz8/Mzf39/+/ve/ewSV4jzRrl271iIiIszlclmVKlVs/fr1lpSUZHXr1rXatWtbaGiobdiwocj1XnzxxUIXf39/S0tLc697o0uXLu6/lUOHDllycrK5XC6rVq2a+fn5Wf369e3AgQNe1WzdurXNnTvXzMzmzJljfn5+1qNHDxs1apRde+21FhgY6B4vCpfLZf7+/taxY0ebOXNmiQTayZMnW0BAgLVo0cLCw8Pt7bfftrCwMLv99ttt8ODBFhoaai+88IJXNbdv324JCQkWHR1t8fHx5nK5rFu3bpacnGz+/v52/fXX26lTp4pcr1evXpaSkmLffvttgbFvv/3WLrvsMvvb3/5W5HpHjx4977Jy5Uqvf8eHDRtm9erVs/fff9+mTJliCQkJ1q1bN/fPKDMz01wul1c1r7nmGuvRo4dt2bLFhg8fbg0aNLBrrrnGTp48aTk5Oda9e3e75ZZbvKpZmHITUBITE+3NN9885/ibb75pCQkJXtc9+1+1y+U65+LNL0x0dLRt2bLFvZ6fn2933HGH1axZ03bu3On1k2x4eLj7hSMvL88CAgJs48aN7vEvv/zSqlevXuR6Z33++edWr149u/fee93/oZVUQKlVq5b973//8xj/9NNPLT4+vsj1oqKibP369WZ25nu6adMmj/EdO3ZYaGhosXs8efKkzZo1yzp16mT+/v4WFxdn//jHP7x+kQ4NDXWHTzOzwMBA++qrr9zru3fvtgoVKhS5XkJCgvu/SbMzL9oul8tOnDhhZmbp6enFCuIdO3a0a665xo4ePVpg7OjRo3bNNdfY1VdfXeR6mzdvPu8ya9Ysr59o+/bta8nJybZu3TpbuHChtWjRwlq2bGmHDx82s+I90Xbs2NFuv/12y8rKsmeeecZq1Khht99+u3v81ltvtZ49exa5nsvlsho1alhiYqLH4nK57KKLLrLExERLSkryqsdf/17eeeed1rBhQ/ds4969e61FixZ2xx13eFWzYsWK7hrJyck2YcIEj/GXX37Zmjdv7lWPU6dOtWuuucYCAwMtKirK7rnnHvvyyy+96uvXGjZs6P7HY8mSJRYSEmITJ050j0+dOtUaNGjgVc0uXbrY4MGD3TPtEyZMsC5dupiZ2XfffWeJiYk2evToIterVKmSx/Ptb61fv94qVapU5HpnX0/OtXj7emNmVrNmTVu6dKl7/eDBg9a6dWu7+uqrLScnp1jBvlq1avbFF1+YmVl2dra5XC5buXKle/zTTz+1mjVrelWzMOUmoLzyyisWHBxsw4YNsw8//NDWrFlja9assQ8//NCGDRtmoaGhHr/cRRUXF2dz5sw55/gXX3zh1Q83LCys0GnjIUOGWI0aNWzFihVeB5QdO3a41ytVquTxH/bu3buL9YJldmZmo2/fvtakSRP78ssvLTAw8IICytn/8uLi4go8cXnb5y233GIDBgwwM7Prr7/eHn74YY/xJ5980ho3bux1j4VNx37//fc2evRo95SmN5KSkuzjjz82szNPgH5+fvbee++5x+fPn2+JiYlFrnfPPfdYo0aN7OOPP7YlS5ZYhw4dLDU11T2+YMECq127tlc9mp0JUud7MdmyZYtXge98wb64T7RxcXG2du1a9/rZ/9SaNWtmhw4dKtYTbZUqVdx/jydPnjQ/Pz+P+9iwYYNddNFFRa43ePBga9asWYG/8ZIK9xdffLF9+OGHHuOLFi3yOvRERETY5s2bzexMwD/79Vk7duzwKjj/usf9+/fbU089ZfXr1zc/Pz9r1aqVvfbaa5aVleVVj4WF+1//jqanp3vVo5lZhQoVPA495ObmWmBgoP30009mdmY2yZu/x6ioKFu2bNk5x5cuXWpRUVFFrhceHm5PPfWULVu2rNBlypQpXv+Oh4aGFjh8npWVZSkpKXbFFVfYrl27ilXz1z+bSpUqebwO7dmzx4KDg72qWZhyE1DMzGbOnGnJyckWEBDgfjIMCAiw5ORkmzVrVrFqdu/e3R555JFzjm/atMmr/9patWplb731VqFjQ4YMscqVK3v1y9KkSRP3C6DZmRmTX09Rrlixwusnr9969913rXr16ubn53dBT7KNGze25s2bW6VKlez//b//5zG+fPlyr14IfvzxR0tMTLR27drZyJEjLTQ01Nq0aWMDBw60du3aWVBQkM2fP9/rHs93vDg/P7/AzM/vefjhh61atWp2++23W1JSkj344INWs2ZNmzRpkk2ePNni4+NtxIgRRa537Ngxu+GGG9y/45dddpnHk88nn3ziEYCKKjY29rxT+h999JHFxsYWuV5UVJS9/vrrtnv37kKX+fPne/2kWLFixQLHtU+dOmU9e/a0Jk2a2JYtW4pVMz093b3+24D//fffex3wZ8+ebfHx8fbyyy+7t11oQDkb7qOjoz1m4MzOhHtvXwx69OhhDz74oJmZderUqcBhpylTpljdunW96rGwv50VK1ZYv379rGLFilaxYkWvejz7D5vZmb93l8vl8Te9bNkyq1Gjhlc14+LiPA7Z/fzzz+ZyudzhadeuXV59L++66y5LSEiw2bNne8w+Hj161GbPnm2JiYk2dOjQItdLTU21p5566pzj3r7emJ0JtYU9Fx47dsxSUlKsadOmXv/d1K5d22PG5NVXX/UIoBs2bLCYmBivahamXAWUs06ePGn79u2zffv2FesEsl9bsWKFRwD4rezs7PMm6N968skn3VOKhbnzzju9+gWcNGmSzZs375zjaWlp7pmGC7F3716bM2eOZWdnF+v2Y8aM8VgWLFjgMX7fffdZ7969var5888/26hRo6xhw4YWEhJiQUFBlpCQYDfffLOtW7fO6x4TExPd/0mVlLy8PHviiSfsr3/9qz355JOWn59v7777rsXHx1tUVJT179+/WN/TX375pdATRYvrkUcesSpVqthzzz1nmzdvtszMTMvMzLTNmzfbc889Z5GRkV5NfV999dX22GOPnXO8OE+0jRs3LhBszf4vpNSsWdPrJ9r69et7nGs1b9489+EyM7M1a9Z4/SJoZvbDDz/YFVdcYZ07d7aMjIwLDihdu3a1a6+91qpUqVIgSK5Zs8brw7jffPONRUVFWd++fe2xxx6zSpUq2S233GJPPPGE9e3b14KDg23q1KlFrufn53fecH/06NEC54n9niFDhljdunXt8ccft9atW1u/fv2sfv369vHHH9uCBQuscePGdtttt3lVs1+/fta+fXvbunWr7dq1y2688UaPQ1nLli3z6lBzTk6O3XHHHRYUFGR+fn4WEhJiISEh5ufnZ0FBQXbnnXdaTk5Okeu99tpr5z1HKTMz0+Mk+aK4++67z3keTFZWliUnJ3v9dzN48GCbMmXKOcfHjx9vXbt29apmYcplQAHgvQkTJlhsbKzHcXCXy2WxsbHn/a+uMLNnz7a33377nOOHDx+2adOmeVXzgQceOOd5MKdOnbIePXp4HXrGjBlj77777jnH//GPf9h1113nVc2z8vPz7cknn7SYmBjz9/cvdkDp37+/x/Lb2eD777/fOnXq5HXdHTt2WO/evS0sLMw94xwYGGiXXXaZffDBB17V+r3Zx+LIzs62gQMHWqNGjWzQoEGWm5trzzzzjAUFBZnL5bLU1FSv73P//v126aWXun/HExISPM4hef/99+2ll17yutejR4/akiVLbMaMGTZjxgxbsmRJoedzlYbDhw8XmHX7taysLK/+yS6KXbt2ebw7tbjKzXVQAJSM9PR0j+s5nL3WTGk7ffq0Tpw4ofDw8HOO//jjj0pISCix+zxx4oT8/f0VHBxc7BobNmzQqlWr1LdvX1WpUqXEejvr+PHj8vf3V0hISLFub2Y6cOCA8vPzVbVqVQUGBpZwhyUrJydHp06duqBr/Wzfvl25ubmqX79+ka9BhD8el7oH4CEpKUkpKSlKSUlxh5O9e/fqtttuK7H7KE69gICAc4YT6cwVdseOHXuhrXk4dOiQ7rzzzguq0aJFC91zzz2qUqVKiX8fJenw4cO66667in17l8ul6tWrKzY21h1OnPDzPpeQkBCFhYVdUM26deuqUaNGBcJJcWr+8ssvWrVqlb755psCYzk5OXrrrbdKtV5ZqlnABc/BACj3inPdkj+yXlmpWRZ69EXNstBjcWoWduHNH3/80T3u7TvLSvpCnmWpZmGY2wKgjz766Lzju3btKtV6ZaVmWejRFzXLQo++qDlq1Cg1atRI69ev15EjRzR8+HC1adNGy5Yt8/rS8eeqd/nllxe7XlmqWRjOQQEgPz8/uVyu835itcvlUl5eXqnUKys1y0KPvqhZFnr0Rc3q1atr0aJFaty4saQz5/Pcdddd+u9//6ulS5eqYsWKiouLK7V6ZalmYTgHBYBiY2M1e/Zs5efnF7ps3LixVOuVlZploUdf1CwLPfqi5i+//OJxHovL5dKkSZPUvXt3tW/fXt99912p1itLNQtDQAGgFi1aaMOGDecc/73/On1dr6zULAs9+qJmWejRFzXr16+v9evXF9j+yiuv6JprrlGPHj286q+k65WlmoW64LNYAJR5JX1BwpKuV1ZqloUefVGzLPToi5olfeHNkq5XlmoWhnNQAACA43CIBwAAOA4BBQAAOA4BBQAAOA4BBQAAOA4BBUC5MGbMGDVr1qy02wBQQggoAEqNk0OFy+XSnDlzSrsN4E+LgALAa6dOnSrtFgCUcwQUAJKk/Px8Pf3006pTp46Cg4NVs2ZNPfHEE9q9e7dcLpdmzZql9u3bKyQkRNOnT5ck/fvf/1aDBg0UEhKi+vXr69VXX/WoOWrUKNWrV08VKlRQrVq19Mgjj7jDzbRp0zR27Fht3rxZLpdLLpdL06ZNkyQdOXJEt99+u6pVq6bw8HBdccUV2rx5s0ftCRMmqHr16goLC9OAAQOUk5NT5Me6bt06XXXVVapataoiIiLUvn17j8ucJyYmSpKuvfZauVwu9zqAP9AFX+oNQLnwwAMPWJUqVWzatGm2Y8cOW7lypU2ZMsXS09NNkiUmJtp//vMf27Vrl+3bt8/eeecdi42NdW/7z3/+Y5GRkTZt2jR3zccee8w+/fRTS09Pt48++siqV69uTz31lJmZnThxwu699177y1/+YhkZGZaRkWEnTpwwM7OOHTta9+7dbd26dfbdd9/Zvffea1FRUXbo0CEzM5s1a5YFBwfbv//9b/v222/toYcesrCwMGvatGmRHuvixYvt7bfftq1bt9o333xjAwYMsOrVq1tWVpaZmR04cMAk2dSpUy0jI8MOHDhQgt9pAEVBQAFgWVlZFhwcbFOmTCkwdjagvPDCCx7ba9eubTNmzPDY9thjj1lKSso57+eZZ56xFi1auNdHjx5dIFSsXLnSwsPDLScnp8D9/etf/zIzs5SUFLvrrrs8xpOTk4scUH4rLy/PwsLCbO7cue5tkuyDDz4oVj0AFy7g/PMrAP4Mtm7dqtzcXF155ZXn3Kdly5bur48fP66dO3dqwIABGjhwoHv76dOnFRER4V6fNWuWXnrpJe3cuVPZ2dk6ffq0wsPDz9vL5s2blZ2draioKI/tv/zyi3bu3Onu94477vAYT0lJ0dKlS3//wUrav3+/Hn74YS1btkwHDhxQXl6eTpw4oT179hTp9gB8j4ACQKGhob+7T8WKFd1fZ2dnS5KmTJmi5ORkj/38/f0lSatXr1afPn00duxYderUSREREZo5c6b++c9/nvd+srOzFRsbq2XLlhUYq1y58u/2WRT9+vXToUOH9OKLLyohIUHBwcFKSUnRyZMnS6Q+gAtHQAGgunXrKjQ0VIsXL9btt9/+u/tXr15dcXFx2rVrl/r06VPoPp999pkSEhL00EMPubd9//33HvsEBQUpLy/PY9sll1yizMxMBQQEnPPk1AYNGmjt2rXq27eve9uaNWt+t++zPv30U7366qvq2rWrJGnv3r366aefPPYJDAws0BuAPw4BBYBCQkI0atQoPfDAAwoKCtLll1+ugwcP6uuvvz7nYZ+xY8dq2LBhioiIUOfOnZWbm6v169fr559/1siRI1W3bl3t2bNHM2fOVKtWrTR//nx98MEHHjUSExOVnp6uTZs2qUaNGgoLC1PHjh2VkpKinj176umnn1a9evW0b98+zZ8/X9dee61atmype+65R/3791fLli11+eWXa/r06fr6669Vq1atIj3eunXr6u2331bLli2VlZWl+++/v8AsUmJiohYvXqzLL79cwcHBqlKlSvG+uQCKp7RPggHgDHl5efb4449bQkKCBQYGWs2aNe3JJ590nyT7xRdfFLjN9OnTrVmzZhYUFGRVqlSxdu3a2ezZs93j999/v0VFRVmlSpXsxhtvtOeff94iIiLc4zk5OdarVy+rXLmy+10zZmdO2r377rstLi7OAgMDLT4+3vr06WN79uxx3/aJJ56wqlWrWqVKlaxfv372wAMPFPkk2Y0bN1rLli0tJCTE6tata++//74lJCTY888/797no48+sjp16lhAQIAlJCR48Z0EUBJcZmalHZIAAAB+jQu1AQAAxyGgACh3KlWqdM5l5cqVpd0egCLgEA+AcmfHjh3nHLvooouK9LZqAKWLgAIAAByHQzwAAMBxCCgAAMBxCCgAAMBxCCgAAMBxCCgAAMBxCCgAAMBxCCgAAMBx/j8K9Eft436YxQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x.plot(kind='bar')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### By multiple labels (columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>week_day</th>\n",
       "      <th>day_hour</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>849636868052275200</th>\n",
       "      <td>2017-04-05 14:56:29</td>\n",
       "      <td>b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'</td>\n",
       "      <td>https://twitter.com/i/web/status/849636868052275200</td>\n",
       "      <td></td>\n",
       "      <td>1</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848988730585096192</th>\n",
       "      <td>2017-04-03 20:01:01</td>\n",
       "      <td>b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"</td>\n",
       "      <td>https://twitter.com/i/web/status/848988730585096192</td>\n",
       "      <td>@ForIn2020, @waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>4</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848943072423497728</th>\n",
       "      <td>2017-04-03 16:59:35</td>\n",
       "      <td>b'@waltmossberg @mims @defcon_5 Et tu, Walt?'</td>\n",
       "      <td>https://twitter.com/i/web/status/848943072423497728</td>\n",
       "      <td>@waltmossberg, @mims, @defcon_5</td>\n",
       "      <td>3</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848935705057280001</th>\n",
       "      <td>2017-04-03 16:30:19</td>\n",
       "      <td>b'Stormy weather in Shortville ...'</td>\n",
       "      <td>https://twitter.com/i/web/status/848935705057280001</td>\n",
       "      <td></td>\n",
       "      <td>1</td>\n",
       "      <td>Monday</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>848416049573658624</th>\n",
       "      <td>2017-04-02 06:05:23</td>\n",
       "      <td>b\"@DaveLeeBBC @verge Coal is dying due to nat gas fracking. It's basically dead.\"</td>\n",
       "      <td>https://twitter.com/i/web/status/848416049573658624</td>\n",
       "      <td>@DaveLeeBBC, @verge</td>\n",
       "      <td>2</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "849636868052275200 2017-04-05 14:56:29   \n",
       "848988730585096192 2017-04-03 20:01:01   \n",
       "848943072423497728 2017-04-03 16:59:35   \n",
       "848935705057280001 2017-04-03 16:30:19   \n",
       "848416049573658624 2017-04-02 06:05:23   \n",
       "\n",
       "                                                                                                                                                                         tweet  \\\n",
       "id                                                                                                                                                                               \n",
       "849636868052275200                                                                                            b'And so the robots spared humanity ... https://t.co/v7JUJQWfCv'   \n",
       "848988730585096192  b\"@ForIn2020 @waltmossberg @mims @defcon_5 Exactly. Tesla is absurdly overvalued if based on the past, but that's irr\\xe2\\x80\\xa6 https://t.co/qQcTqkzgMl\"   \n",
       "848943072423497728                                                                                                               b'@waltmossberg @mims @defcon_5 Et tu, Walt?'   \n",
       "848935705057280001                                                                                                                         b'Stormy weather in Shortville ...'   \n",
       "848416049573658624                                                                           b\"@DaveLeeBBC @verge Coal is dying due to nat gas fracking. It's basically dead.\"   \n",
       "\n",
       "                                                             tweet_link  \\\n",
       "id                                                                        \n",
       "849636868052275200  https://twitter.com/i/web/status/849636868052275200   \n",
       "848988730585096192  https://twitter.com/i/web/status/848988730585096192   \n",
       "848943072423497728  https://twitter.com/i/web/status/848943072423497728   \n",
       "848935705057280001  https://twitter.com/i/web/status/848935705057280001   \n",
       "848416049573658624  https://twitter.com/i/web/status/848416049573658624   \n",
       "\n",
       "                                                 tweet_mentions  n_mentions  \\\n",
       "id                                                                            \n",
       "849636868052275200                                                        1   \n",
       "848988730585096192  @ForIn2020, @waltmossberg, @mims, @defcon_5           4   \n",
       "848943072423497728              @waltmossberg, @mims, @defcon_5           3   \n",
       "848935705057280001                                                        1   \n",
       "848416049573658624                          @DaveLeeBBC, @verge           2   \n",
       "\n",
       "                   week_day_name  week_day  day_hour  \n",
       "id                                                    \n",
       "849636868052275200     Wednesday         2        14  \n",
       "848988730585096192        Monday         0        20  \n",
       "848943072423497728        Monday         0        16  \n",
       "848935705057280001        Monday         0        16  \n",
       "848416049573658624        Sunday         6         6  "
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"display.max_rows\", None, \"display.max_columns\", None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>tweet</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>week_day</th>\n",
       "      <th>day_hour</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   tweet\n",
       "week_day day_hour       \n",
       "0        0            11\n",
       "         1            23\n",
       "         2            17\n",
       "         3             7\n",
       "         4            12"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we group based on the values of two columns\n",
    "# instead of one\n",
    "\n",
    "df.groupby(['week_day', 'day_hour'])[['tweet']].count().head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aggregation methods\n",
    "\n",
    "**Summary**:\n",
    "\n",
    "- `count`: Number of non-NA values\n",
    "- `sum`: Sum of non-NA values\n",
    "- `mean`: Mean of non-NA values\n",
    "- `median`: Arithmetic median of non-NA values\n",
    "- `std`, `var`: standard deviation and variance\n",
    "- `min`, `max`: Minimum and maximum of non-NA values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['n_mentions'].max()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or also as aggregation functions within a groupby:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"6\" halign=\"left\">n_mentions</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>min</th>\n",
       "      <th>max</th>\n",
       "      <th>std</th>\n",
       "      <th>var</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>week_day</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>315</td>\n",
       "      <td>1.282540</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0.643077</td>\n",
       "      <td>0.413548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>385</td>\n",
       "      <td>1.184416</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0.590521</td>\n",
       "      <td>0.348715</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>380</td>\n",
       "      <td>1.168421</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.474192</td>\n",
       "      <td>0.224858</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>361</td>\n",
       "      <td>1.199446</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.531350</td>\n",
       "      <td>0.282333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>530</td>\n",
       "      <td>1.245283</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0.633222</td>\n",
       "      <td>0.400970</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>426</td>\n",
       "      <td>1.166667</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>0.497444</td>\n",
       "      <td>0.247451</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>422</td>\n",
       "      <td>1.251185</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0.571231</td>\n",
       "      <td>0.326305</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         n_mentions                                      \n",
       "              count      mean min max       std       var\n",
       "week_day                                                 \n",
       "0               315  1.282540   1   5  0.643077  0.413548\n",
       "1               385  1.184416   1   6  0.590521  0.348715\n",
       "2               380  1.168421   1   4  0.474192  0.224858\n",
       "3               361  1.199446   1   4  0.531350  0.282333\n",
       "4               530  1.245283   1   5  0.633222  0.400970\n",
       "5               426  1.166667   1   5  0.497444  0.247451\n",
       "6               422  1.251185   1   6  0.571231  0.326305"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.groupby('week_day').agg(\n",
    "    {\n",
    "        # each key in this dict specifies\n",
    "        # a given column\n",
    "        'n_mentions':[\n",
    "            # the list contains aggregation functions\n",
    "            # to be applied to this column\n",
    "            'count',\n",
    "            'mean',\n",
    "            'min',\n",
    "            'max',\n",
    "            'std',\n",
    "            'var'\n",
    "        ]\n",
    "    }\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sorting"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To sort the values of  a dataframe we use its `sort_values` method:\n",
    "- `by`: specifies the name of the column to be used for sorting\n",
    "- `ascending` (default = `True`): specifies whether the sorting should be *ascending* (A-Z, 0-9) or `descending` (Z-A, 9-0) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 237,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df.sort_values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>week_day</th>\n",
       "      <th>day_hour</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15434727182</th>\n",
       "      <td>2010-06-04 18:31:57</td>\n",
       "      <td>b'Please ignore prior tweets, as that was someone pretending to be me :)  This is actually me.'</td>\n",
       "      <td>https://twitter.com/i/web/status/15434727182</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Friday</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142179928203460608</th>\n",
       "      <td>2011-12-01 09:55:11</td>\n",
       "      <td>b\"Went to Iceland on Sat to ride bumper cars on ice!  No, not the country, Vlad's rink in Van Nuys. Awesome family fun :) http://t.co/rBQXJ9IT\"</td>\n",
       "      <td>https://twitter.com/i/web/status/142179928203460608</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142188458125963264</th>\n",
       "      <td>2011-12-01 10:29:04</td>\n",
       "      <td>b'I made the volume on the Model S http://t.co/wMCnT53M go to 11.  Now I just need to work in a miniature Stonehenge...'</td>\n",
       "      <td>https://twitter.com/i/web/status/142188458125963264</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142880871391838208</th>\n",
       "      <td>2011-12-03 08:20:28</td>\n",
       "      <td>b'Great Voltaire quote, arguably better than Twain. Hearing news of his own death, Voltaire replied the reports were true, only premature.'</td>\n",
       "      <td>https://twitter.com/i/web/status/142880871391838208</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>142881284019060736</th>\n",
       "      <td>2011-12-03 08:22:07</td>\n",
       "      <td>b'That was a total non sequitur btw'</td>\n",
       "      <td>https://twitter.com/i/web/status/142881284019060736</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "15434727182        2010-06-04 18:31:57   \n",
       "142179928203460608 2011-12-01 09:55:11   \n",
       "142188458125963264 2011-12-01 10:29:04   \n",
       "142880871391838208 2011-12-03 08:20:28   \n",
       "142881284019060736 2011-12-03 08:22:07   \n",
       "\n",
       "                                                                                                                                                              tweet  \\\n",
       "id                                                                                                                                                                    \n",
       "15434727182                                                         b'Please ignore prior tweets, as that was someone pretending to be me :)  This is actually me.'   \n",
       "142179928203460608  b\"Went to Iceland on Sat to ride bumper cars on ice!  No, not the country, Vlad's rink in Van Nuys. Awesome family fun :) http://t.co/rBQXJ9IT\"   \n",
       "142188458125963264                         b'I made the volume on the Model S http://t.co/wMCnT53M go to 11.  Now I just need to work in a miniature Stonehenge...'   \n",
       "142880871391838208      b'Great Voltaire quote, arguably better than Twain. Hearing news of his own death, Voltaire replied the reports were true, only premature.'   \n",
       "142881284019060736                                                                                                             b'That was a total non sequitur btw'   \n",
       "\n",
       "                                                             tweet_link  \\\n",
       "id                                                                        \n",
       "15434727182                https://twitter.com/i/web/status/15434727182   \n",
       "142179928203460608  https://twitter.com/i/web/status/142179928203460608   \n",
       "142188458125963264  https://twitter.com/i/web/status/142188458125963264   \n",
       "142880871391838208  https://twitter.com/i/web/status/142880871391838208   \n",
       "142881284019060736  https://twitter.com/i/web/status/142881284019060736   \n",
       "\n",
       "                   tweet_mentions  n_mentions week_day_name  week_day  \\\n",
       "id                                                                      \n",
       "15434727182                  None           0        Friday         4   \n",
       "142179928203460608           None           0      Thursday         3   \n",
       "142188458125963264           None           0      Thursday         3   \n",
       "142880871391838208           None           0      Saturday         5   \n",
       "142881284019060736           None           0      Saturday         5   \n",
       "\n",
       "                    day_hour  \n",
       "id                            \n",
       "15434727182               18  \n",
       "142179928203460608         9  \n",
       "142188458125963264        10  \n",
       "142880871391838208         8  \n",
       "142881284019060736         8  "
      ]
     },
     "execution_count": 238,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by='created_at', ascending=True).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 239,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>tweet</th>\n",
       "      <th>tweet_link</th>\n",
       "      <th>tweet_mentions</th>\n",
       "      <th>n_mentions</th>\n",
       "      <th>week_day_name</th>\n",
       "      <th>week_day</th>\n",
       "      <th>day_hour</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>752721031735812096</th>\n",
       "      <td>2016-07-12 04:27:33</td>\n",
       "      <td>b\"@kumailn @RealDaveBarton @JuddApatow @SiliconHBO @FredericLambert @MikeJudge can't wait to see the costume\"</td>\n",
       "      <td>https://twitter.com/i/web/status/752721031735812096</td>\n",
       "      <td>@kumailn, @RealDaveBarton, @JuddApatow, @SiliconHBO, @FredericLambert, @MikeJudge</td>\n",
       "      <td>6</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303244501957365760</th>\n",
       "      <td>2013-02-17 20:48:17</td>\n",
       "      <td>b'RT @LawrenceChanin: @TeslaRoadTrip @avantgame @TEDchris @elonmusk @nytimes   \\nThe driving public is the winner thanks to the efforts of  ...'</td>\n",
       "      <td>https://twitter.com/i/web/status/303244501957365760</td>\n",
       "      <td>@LawrenceChanin, @TeslaRoadTrip, @avantgame, @TEDchris, @elonmusk, @nytimes</td>\n",
       "      <td>6</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>845289977051148289</th>\n",
       "      <td>2017-03-24 15:03:29</td>\n",
       "      <td>b'@faultywarrior @matt_trulli @FredericLambert @TimShelton @JimPengelly Let me just go fetch my magic wand ...'</td>\n",
       "      <td>https://twitter.com/i/web/status/845289977051148289</td>\n",
       "      <td>@faultywarrior, @matt_trulli, @FredericLambert, @TimShelton, @JimPengelly</td>\n",
       "      <td>5</td>\n",
       "      <td>Friday</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>672792504895434753</th>\n",
       "      <td>2015-12-04 15:00:07</td>\n",
       "      <td>b\"RT @WSJLife: Supermodel @KarlieKloss blasts off to the future at @elonmusk's @spacex HQ https://t.co/PfE2bWcQwM @wsjmag https://t.co/zc7QM9\\xe2\\x80\\xa6\"</td>\n",
       "      <td>https://twitter.com/i/web/status/672792504895434753</td>\n",
       "      <td>@WSJLife, @KarlieKloss, @elonmusk, @spacex, @wsjmag</td>\n",
       "      <td>5</td>\n",
       "      <td>Friday</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>191005784862236672</th>\n",
       "      <td>2012-04-14 03:31:42</td>\n",
       "      <td>b'RT @SethGreen: My love @ClareGrant &amp; @elonmusk partied with the space cheese on our tour of @SpaceX &amp; @TeslaMotors ...which was AWESO\\xe2\\x80\\xa6 h ...'</td>\n",
       "      <td>https://twitter.com/i/web/status/191005784862236672</td>\n",
       "      <td>@SethGreen, @ClareGrant, @elonmusk, @SpaceX, @TeslaMotors</td>\n",
       "      <td>5</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            created_at  \\\n",
       "id                                       \n",
       "752721031735812096 2016-07-12 04:27:33   \n",
       "303244501957365760 2013-02-17 20:48:17   \n",
       "845289977051148289 2017-03-24 15:03:29   \n",
       "672792504895434753 2015-12-04 15:00:07   \n",
       "191005784862236672 2012-04-14 03:31:42   \n",
       "\n",
       "                                                                                                                                                                         tweet  \\\n",
       "id                                                                                                                                                                               \n",
       "752721031735812096                                               b\"@kumailn @RealDaveBarton @JuddApatow @SiliconHBO @FredericLambert @MikeJudge can't wait to see the costume\"   \n",
       "303244501957365760            b'RT @LawrenceChanin: @TeslaRoadTrip @avantgame @TEDchris @elonmusk @nytimes   \\nThe driving public is the winner thanks to the efforts of  ...'   \n",
       "845289977051148289                                             b'@faultywarrior @matt_trulli @FredericLambert @TimShelton @JimPengelly Let me just go fetch my magic wand ...'   \n",
       "672792504895434753  b\"RT @WSJLife: Supermodel @KarlieKloss blasts off to the future at @elonmusk's @spacex HQ https://t.co/PfE2bWcQwM @wsjmag https://t.co/zc7QM9\\xe2\\x80\\xa6\"   \n",
       "191005784862236672  b'RT @SethGreen: My love @ClareGrant & @elonmusk partied with the space cheese on our tour of @SpaceX & @TeslaMotors ...which was AWESO\\xe2\\x80\\xa6 h ...'   \n",
       "\n",
       "                                                             tweet_link  \\\n",
       "id                                                                        \n",
       "752721031735812096  https://twitter.com/i/web/status/752721031735812096   \n",
       "303244501957365760  https://twitter.com/i/web/status/303244501957365760   \n",
       "845289977051148289  https://twitter.com/i/web/status/845289977051148289   \n",
       "672792504895434753  https://twitter.com/i/web/status/672792504895434753   \n",
       "191005784862236672  https://twitter.com/i/web/status/191005784862236672   \n",
       "\n",
       "                                                                                       tweet_mentions  \\\n",
       "id                                                                                                      \n",
       "752721031735812096  @kumailn, @RealDaveBarton, @JuddApatow, @SiliconHBO, @FredericLambert, @MikeJudge   \n",
       "303244501957365760        @LawrenceChanin, @TeslaRoadTrip, @avantgame, @TEDchris, @elonmusk, @nytimes   \n",
       "845289977051148289          @faultywarrior, @matt_trulli, @FredericLambert, @TimShelton, @JimPengelly   \n",
       "672792504895434753                                @WSJLife, @KarlieKloss, @elonmusk, @spacex, @wsjmag   \n",
       "191005784862236672                          @SethGreen, @ClareGrant, @elonmusk, @SpaceX, @TeslaMotors   \n",
       "\n",
       "                    n_mentions week_day_name  week_day  day_hour  \n",
       "id                                                                \n",
       "752721031735812096           6       Tuesday         1         4  \n",
       "303244501957365760           6        Sunday         6        20  \n",
       "845289977051148289           5        Friday         4        15  \n",
       "672792504895434753           5        Friday         4        15  \n",
       "191005784862236672           5      Saturday         5         3  "
      ]
     },
     "execution_count": 239,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.sort_values(by='n_mentions', ascending=False).head()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save\n",
    "\n",
    "Before continuing with the plotting, let's save our enhanced dataframe, so that we can come back to it without having to redo the same manipulations on it.\n",
    "\n",
    "`pandas` provides a number of handy functions to export dataframes in a variety of formats."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we use `to_pickle` to serialize the dataframe into a binary format, by using behind the scenes Python's `pickle` library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pickle(\"./musk_tweets_enhanced.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 240,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--   1 npedrazzini  staff     622249 Jun 28 16:25 musk_tweets_enhanced.pkl\n"
     ]
    }
   ],
   "source": [
    "ls -al | grep pkl"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "384px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}