modern_1_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Effective Pandas\n",
    "\n",
    "## Introduction\n",
    "\n",
    "This series is about how to make effective use of [pandas](http://pandas.pydata.org), a data analysis library for the Python programming language.\n",
    "It's targeted at an intermediate level: people who have some experience with pandas, but are looking to improve.\n",
    "\n",
    "## Prior Art\n",
    "\n",
    "There are many great resources for learning pandas; this is not one of them.\n",
    "For beginners, I typically recommend [Greg Reda's](https://twitter.com/gjreda) [3-part introduction](http://gregreda.com/2013/10/26/intro-to-pandas-data-structures/), especially if they're familiar with SQL. Of course, there's the pandas [documentation](http://pandas.pydata.org/) itself. I gave [a talk](https://www.youtube.com/watch?v=otCriSKVV_8) at PyData Seattle targeted as an introduction if you prefer video form. Wes McKinney's [Python for Data Analysis](http://shop.oreilly.com/product/0636920023784.do) is still the goto book (and is also a really good introduction to NumPy as well). Jake VanderPlas's [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do), in early release, is great too.\n",
    "Kevin Markham has a [video series](http://www.dataschool.io/easier-data-analysis-with-pandas/) for beginners learning pandas.\n",
    "\n",
    "With all those resources (and many more that I've slighted through omission), why write another? Surely the law of diminishing returns is kicking in by now.\n",
    "Still, I thought there was room for a guide that is up to date (as of March 2016) and emphasizes idiomatic pandas code (code that is *pandorable*).\n",
    "This series probably won't be appropriate for people completely new to python\n",
    "or NumPy and pandas.\n",
    "By luck, this first post happened to cover topics that are relatively introductory,\n",
    "so read some of the linked material and come back, or [let me know](https://twitter.com/tomaugspurger) if you\n",
    "have questions.\n",
    "\n",
    "## Get the Data\n",
    "\n",
    "We'll be working with [flight delay data](http://www.transtats.bts.gov/databases.asp?Mode_ID=1&Mode_Desc=Aviation&Subject_ID2=0) from the BTS (R users can install Hadley's [NYCFlights13](https://github.com/hadley/nycflights13) dataset for similar data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import zipfile\n",
    "\n",
    "import requests\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "if int(os.environ.get(\"MODERN_PANDAS_EPUB\", 0)):\n",
    "    import prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "headers = {\n",
    "    'Referer': 'https://www.transtats.bts.gov/DL_SelectFields.asp?Table_ID=236&DB_Short_Name=On-Time',\n",
    "    'Origin': 'https://www.transtats.bts.gov',\n",
    "    'Content-Type': 'application/x-www-form-urlencoded',\n",
    "}\n",
    "\n",
    "params = (\n",
    "    ('Table_ID', '236'),\n",
    "    ('Has_Group', '3'),\n",
    "    ('Is_Zipped', '0'),\n",
    ")\n",
    "\n",
    "with open('modern-1-url.txt', encoding='utf-8') as f:\n",
    "    data = f.read().strip()\n",
    "\n",
    "os.makedirs('data', exist_ok=True)\n",
    "dest = \"data/flights.csv.zip\"\n",
    "\n",
    "if not os.path.exists(dest):\n",
    "    r = requests.post('https://www.transtats.bts.gov/DownLoad_Table.asp',\n",
    "                      headers=headers, params=params, data=data, stream=True)\n",
    "\n",
    "    with open(\"data/flights.csv.zip\", 'wb') as f:\n",
    "        for chunk in r.iter_content(chunk_size=102400): \n",
    "            if chunk:\n",
    "                f.write(chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That download returned a ZIP file.\n",
    "There's an open [Pull Request](https://github.com/pydata/pandas/pull/12175) for automatically decompressing ZIP archives with a single CSV,\n",
    "but for now we have to extract it ourselves and then read it in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 450017 entries, 0 to 450016\n",
      "Data columns (total 33 columns):\n",
      "fl_date                  450017 non-null datetime64[ns]\n",
      "unique_carrier           450017 non-null object\n",
      "airline_id               450017 non-null int64\n",
      "tail_num                 449378 non-null object\n",
      "fl_num                   450017 non-null int64\n",
      "origin_airport_id        450017 non-null int64\n",
      "origin_airport_seq_id    450017 non-null int64\n",
      "origin_city_market_id    450017 non-null int64\n",
      "origin                   450017 non-null object\n",
      "origin_city_name         450017 non-null object\n",
      "dest_airport_id          450017 non-null int64\n",
      "dest_airport_seq_id      450017 non-null int64\n",
      "dest_city_market_id      450017 non-null int64\n",
      "dest                     450017 non-null object\n",
      "dest_city_name           450017 non-null object\n",
      "crs_dep_time             450017 non-null int64\n",
      "dep_time                 441476 non-null float64\n",
      "dep_delay                441476 non-null float64\n",
      "taxi_out                 441244 non-null float64\n",
      "wheels_off               441244 non-null float64\n",
      "wheels_on                440746 non-null float64\n",
      "taxi_in                  440746 non-null float64\n",
      "crs_arr_time             450017 non-null int64\n",
      "arr_time                 440746 non-null float64\n",
      "arr_delay                439645 non-null float64\n",
      "cancelled                450017 non-null float64\n",
      "cancellation_code        8886 non-null object\n",
      "carrier_delay            97699 non-null float64\n",
      "weather_delay            97699 non-null float64\n",
      "nas_delay                97699 non-null float64\n",
      "security_delay           97699 non-null float64\n",
      "late_aircraft_delay      97699 non-null float64\n",
      "unnamed: 32              0 non-null float64\n",
      "dtypes: datetime64[ns](1), float64(15), int64(10), object(7)\n",
      "memory usage: 113.3+ MB\n"
     ]
    }
   ],
   "source": [
    "zf = zipfile.ZipFile(\"data/flights.csv.zip\")\n",
    "fp = zf.extract(zf.filelist[0].filename, path='data/')\n",
    "df = pd.read_csv(fp, parse_dates=[\"FL_DATE\"]).rename(columns=str.lower)\n",
    "\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing\n",
    "\n",
    "Or, *explicit is better than implicit*.\n",
    "By my count, 7 of the top-15 voted pandas questions on [Stackoverflow](http://stackoverflow.com/questions/tagged/pandas?sort=votes&pageSize=15) are about indexing. This seems as good a place as any to start.\n",
    "\n",
    "By indexing, we mean the selection of subsets of a DataFrame or Series.\n",
    "`DataFrames` (and to a lesser extent, `Series`) provide a difficult set of challenges:\n",
    "\n",
    "- Like lists, you can index by location.\n",
    "- Like dictionaries, you can index by label.\n",
    "- Like NumPy arrays, you can index by boolean masks.\n",
    "- Any of these indexers could be scalar indexes, or they could be arrays, or they could be `slice`s.\n",
    "- Any of these should work on the index (row labels) or columns of a DataFrame.\n",
    "- And any of these should work on hierarchical indexes.\n",
    "\n",
    "The complexity of pandas' indexing is a microcosm for the complexity of the pandas API in general.\n",
    "There's a reason for the complexity (well, most of it), but that's not *much* consolation while you're learning.\n",
    "Still, all of these ways of indexing really are useful enough to justify their inclusion in the library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slicing\n",
    "\n",
    "Or, *explicit is better than implicit*.\n",
    "\n",
    "By my count, 7 of the top-15 voted pandas questions on [Stackoverflow](http://stackoverflow.com/questions/tagged/pandas?sort=votes&pageSize=15) are about slicing. This seems as good a place as any to start.\n",
    "\n",
    "Brief history digression: For years the preferred method for row and/or column selection was `.ix`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fl_date</th>\n",
       "      <th>tail_num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N3LGAA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N368AA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N3DDAA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N332AA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N327AA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N3LBAA</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fl_date tail_num\n",
       "10 2014-01-01   N3LGAA\n",
       "11 2014-01-01   N368AA\n",
       "12 2014-01-01   N3DDAA\n",
       "13 2014-01-01   N332AA\n",
       "14 2014-01-01   N327AA\n",
       "15 2014-01-01   N3LBAA"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.ix[10:15, ['fl_date', 'tail_num']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However this simple little operation hides some complexity. What if, rather than our default `range(n)` index, we had an integer index like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fl_date</th>\n",
       "      <th>unique_carrier</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>airline_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>19393</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>WN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19690</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>HA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19790</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>DL</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19805</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>AA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19930</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>AS</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              fl_date unique_carrier\n",
       "airline_id                          \n",
       "19393      2014-01-01             WN\n",
       "19690      2014-01-01             HA\n",
       "19790      2014-01-01             DL\n",
       "19805      2014-01-01             AA\n",
       "19930      2014-01-01             AS"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first = df.groupby('airline_id')[['fl_date', 'unique_carrier']].first()\n",
    "first.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you predict ahead of time what our slice from above will give when passed to `.ix`?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fl_date</th>\n",
       "      <th>tail_num</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>airline_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [fl_date, tail_num]\n",
       "Index: []"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first.ix[10:15, ['fl_date', 'tail_num']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprise, an empty DataFrame! Which in data analysis is rarely a good thing. What happened?\n",
    "\n",
    "We had an integer index, so the call to `.ix` used its label-based mode. It was looking for integer *labels* between 10:15 (inclusive). It didn't find any. Since we sliced a range it returned an empty DataFrame, rather than raising a KeyError.\n",
    "\n",
    "By way of contrast, suppose we had a string index, rather than integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fl_date</th>\n",
       "      <th>tail_num</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>UA</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N14214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>US</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N650AW</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VX</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N637VA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WN</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N412WN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  fl_date tail_num\n",
       "unique_carrier                    \n",
       "UA             2014-01-01   N14214\n",
       "US             2014-01-01   N650AW\n",
       "VX             2014-01-01   N637VA\n",
       "WN             2014-01-01   N412WN"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first = df.groupby('unique_carrier').first()\n",
    "first.ix[10:15, ['fl_date', 'tail_num']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "And it works again! Now that we had a string index, `.ix` used its positional-mode. It looked for *rows* 10-15 (exclusive on the right).\n",
    "\n",
    "But you can't reliably predict what the outcome of the slice will be ahead of time. It's on the *reader* of the code (probably your future self) to know the dtypes so you can reckon whether `.ix` will use label indexing (returning the empty DataFrame) or positional indexing (like the last example).\n",
    "In general, methods whose behavior depends on the data, like `.ix` dispatching to label-based indexing on integer Indexes but location-based indexing on non-integer, are hard to use correctly. We've been trying to stamp them out in pandas.\n",
    "\n",
    "Since pandas 0.12, these tasks have been cleanly separated into two methods:\n",
    "\n",
    "1. `.loc` for label-based indexing\n",
    "2. `.iloc` for positional indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fl_date</th>\n",
       "      <th>tail_num</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AA</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N338AA</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AS</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N524AS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DL</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>N911DL</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  fl_date tail_num\n",
       "unique_carrier                    \n",
       "AA             2014-01-01   N338AA\n",
       "AS             2014-01-01   N524AS\n",
       "DL             2014-01-01   N911DL"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first.loc[['AA', 'AS', 'DL'], ['fl_date', 'tail_num']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fl_date</th>\n",
       "      <th>airline_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>AA</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>19805</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>AS</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>19930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DL</th>\n",
       "      <td>2014-01-01</td>\n",
       "      <td>19790</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  fl_date  airline_id\n",
       "unique_carrier                       \n",
       "AA             2014-01-01       19805\n",
       "AS             2014-01-01       19930\n",
       "DL             2014-01-01       19790"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first.iloc[[0, 1, 3], [0, 1]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.ix` is still around, and isn't being deprecated any time soon. Occasionally it's useful. But if you've been using `.ix` out of habit, or if you didn't know any better, maybe give `.loc` and `.iloc` a shot. For the intrepid reader, Joris Van den Bossche (a core pandas dev) [compiled a great overview](https://github.com/pydata/pandas/issues/9595) of the pandas `__getitem__` API.\n",
    "A later post in this series will go into more detail on using Indexes effectively;\n",
    "they are useful objects in their own right, but for now we'll move on to a closely related topic.\n",
    "\n",
    "## SettingWithCopy\n",
    "\n",
    "Pandas used to get *a lot* of questions about assignments seemingly not working. We'll take [this StackOverflow](http://stackoverflow.com/q/16553298/1889400) question as a representative question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a   b\n",
       "0  1  10\n",
       "1  2  20\n",
       "2  3  30\n",
       "3  4  40\n",
       "4  5  50"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f = pd.DataFrame({'a':[1,2,3,4,5], 'b':[10,20,30,40,50]})\n",
    "f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The user wanted to take the rows of `b` where `a` was 3 or less, and set them equal to `b / 10`\n",
    "We'll use boolean indexing to select those rows `f['a'] <= 3`,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a   b\n",
       "0  1  10\n",
       "1  2  20\n",
       "2  3  30\n",
       "3  4  40\n",
       "4  5  50"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ignore the context manager for now\n",
    "with pd.option_context('mode.chained_assignment', None):\n",
    "    f[f['a'] <= 3]['b'] = f[f['a'] <= 3 ]['b'] / 10\n",
    "f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And nothing happened. Well, something did happen, but nobody witnessed it. If an object without any references is modified, does it make a sound?\n",
    "\n",
    "The warning I silenced above with the context manager links to [an explanation](http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy) that's quite helpful. I'll summarize the high points here.\n",
    "\n",
    "The \"failure\" to update `f` comes down to what's called *chained indexing*, a practice to be avoided.\n",
    "The \"chained\" comes from indexing multiple times, one after another, rather than one single indexing operation.\n",
    "Above we had two operations on the left-hand side, one `__getitem__` and one `__setitem__` (in python, the square brackets are syntactic sugar for `__getitem__` or `__setitem__` if it's for assignment). So `f[f['a'] <= 3]['b']` becomes\n",
    "\n",
    "1. `getitem`: `f[f['a'] <= 3]`\n",
    "2. `setitem`: `_['b'] = ...`  # using `_` to represent the result of 1.\n",
    "\n",
    "In general, pandas can't guarantee whether that first `__getitem__` returns a view or a copy of the underlying data.\n",
    "The changes *will* be made to the thing I called `_` above, the result of the `__getitem__` in `1`.\n",
    "But we don't know that `_` shares the same memory as our original `f`.\n",
    "And so we can't be sure that whatever changes are being made to `_` will be reflected in `f`.\n",
    "\n",
    "Done properly, you would write"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>40.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   a     b\n",
       "0  1   1.0\n",
       "1  2   2.0\n",
       "2  3   3.0\n",
       "3  4  40.0\n",
       "4  5  50.0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f.loc[f['a'] <= 3, 'b'] = f.loc[f['a'] <= 3, 'b'] / 10\n",
    "f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this is all in a single call to `__setitem__` and pandas can ensure that the assignment happens properly.\n",
    "\n",
    "The rough rule is any time you see back-to-back square brackets, `][`, you're in asking for trouble. Replace that with a `.loc[..., ...]` and you'll be set.\n",
    "\n",
    "The other bit of advice is that a SettingWithCopy warning is raised when the *assignment* is made.\n",
    "The potential copy could be made earlier in your code.\n",
    "\n",
    "## Multidimensional Indexing\n",
    "\n",
    "MultiIndexes might just be my favorite feature of pandas.\n",
    "They let you represent higher-dimensional datasets in a familiar two-dimensional table, which my brain can sometimes handle.\n",
    "Each additional level of the MultiIndex represents another dimension.\n",
    "The cost of this is somewhat harder label indexing.\n",
    "\n",
    "My very first bug report to pandas, back in [November 2012](https://github.com/pydata/pandas/issues/2207),\n",
    "was about indexing into a MultiIndex.\n",
    "I bring it up now because I genuinely couldn't tell whether the result I got was a bug or not.\n",
    "Also, from that bug report\n",
    "\n",
    "> Sorry if this isn't actually a bug. Still very new to python. Thanks!\n",
    "\n",
    "Adorable.\n",
    "\n",
    "That operation was made much easier by [this](http://pandas.pydata.org/pandas-docs/version/0.18.0/whatsnew.html#multiindexing-using-slicers) addition in 2014, which lets you slice arbitrary levels of a MultiIndex..\n",
    "Let's make a MultiIndexed DataFrame to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>airline_id</th>\n",
       "      <th>fl_num</th>\n",
       "      <th>origin_airport_id</th>\n",
       "      <th>origin_airport_seq_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th>origin</th>\n",
       "      <th>dest</th>\n",
       "      <th>tail_num</th>\n",
       "      <th>fl_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AA</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">ABQ</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DFW</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">N200AA</th>\n",
       "      <th>2014-01-06</th>\n",
       "      <td>19805</td>\n",
       "      <td>1662</td>\n",
       "      <td>10140</td>\n",
       "      <td>1014002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-27</th>\n",
       "      <td>19805</td>\n",
       "      <td>1090</td>\n",
       "      <td>10140</td>\n",
       "      <td>1014002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N202AA</th>\n",
       "      <th>2014-01-27</th>\n",
       "      <td>19805</td>\n",
       "      <td>1332</td>\n",
       "      <td>10140</td>\n",
       "      <td>1014002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">N426AA</th>\n",
       "      <th>2014-01-09</th>\n",
       "      <td>19805</td>\n",
       "      <td>1662</td>\n",
       "      <td>10140</td>\n",
       "      <td>1014002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-15</th>\n",
       "      <td>19805</td>\n",
       "      <td>1467</td>\n",
       "      <td>10140</td>\n",
       "      <td>1014002</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                airline_id  fl_num  \\\n",
       "unique_carrier origin dest tail_num fl_date                          \n",
       "AA             ABQ    DFW  N200AA   2014-01-06       19805    1662   \n",
       "                                    2014-01-27       19805    1090   \n",
       "                           N202AA   2014-01-27       19805    1332   \n",
       "                           N426AA   2014-01-09       19805    1662   \n",
       "                                    2014-01-15       19805    1467   \n",
       "\n",
       "                                                origin_airport_id  \\\n",
       "unique_carrier origin dest tail_num fl_date                         \n",
       "AA             ABQ    DFW  N200AA   2014-01-06              10140   \n",
       "                                    2014-01-27              10140   \n",
       "                           N202AA   2014-01-27              10140   \n",
       "                           N426AA   2014-01-09              10140   \n",
       "                                    2014-01-15              10140   \n",
       "\n",
       "                                                origin_airport_seq_id  \n",
       "unique_carrier origin dest tail_num fl_date                            \n",
       "AA             ABQ    DFW  N200AA   2014-01-06                1014002  \n",
       "                                    2014-01-27                1014002  \n",
       "                           N202AA   2014-01-27                1014002  \n",
       "                           N426AA   2014-01-09                1014002  \n",
       "                                    2014-01-15                1014002  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdf = df.set_index(['unique_carrier', 'origin', 'dest', 'tail_num', 'fl_date']).sort_index()\n",
    "hdf[hdf.columns[:4]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And just to clear up some terminology, the *levels* of a MultiIndex are the\n",
    "former column names (`unique_carrier`, `origin`...).\n",
    "The labels are the actual values in a level, (`'AA'`, `'ABQ'`, ...).\n",
    "Levels can be referred to by name or position, with 0 being the outermost level.\n",
    "\n",
    "Slicing the outermost index level is pretty easy, we just use our regular `.loc[row_indexer, column_indexer]`. We'll select the columns `dep_time` and `dep_delay` where the carrier was American Airlines, Delta, or US Airways."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>dep_time</th>\n",
       "      <th>dep_delay</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th>origin</th>\n",
       "      <th>dest</th>\n",
       "      <th>tail_num</th>\n",
       "      <th>fl_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AA</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">ABQ</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DFW</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">N200AA</th>\n",
       "      <th>2014-01-06</th>\n",
       "      <td>1246.0</td>\n",
       "      <td>71.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-27</th>\n",
       "      <td>605.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N202AA</th>\n",
       "      <th>2014-01-27</th>\n",
       "      <td>822.0</td>\n",
       "      <td>-13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">N426AA</th>\n",
       "      <th>2014-01-09</th>\n",
       "      <td>1135.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-15</th>\n",
       "      <td>1022.0</td>\n",
       "      <td>-8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">US</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">TUS</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">PHX</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">N824AW</th>\n",
       "      <th>2014-01-16</th>\n",
       "      <td>1900.0</td>\n",
       "      <td>-10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-20</th>\n",
       "      <td>1903.0</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">N836AW</th>\n",
       "      <th>2014-01-08</th>\n",
       "      <td>1928.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-29</th>\n",
       "      <td>1908.0</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N837AW</th>\n",
       "      <th>2014-01-10</th>\n",
       "      <td>1902.0</td>\n",
       "      <td>-8.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>139194 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                dep_time  dep_delay\n",
       "unique_carrier origin dest tail_num fl_date                        \n",
       "AA             ABQ    DFW  N200AA   2014-01-06    1246.0       71.0\n",
       "                                    2014-01-27     605.0        0.0\n",
       "                           N202AA   2014-01-27     822.0      -13.0\n",
       "                           N426AA   2014-01-09    1135.0        0.0\n",
       "                                    2014-01-15    1022.0       -8.0\n",
       "...                                                  ...        ...\n",
       "US             TUS    PHX  N824AW   2014-01-16    1900.0      -10.0\n",
       "                                    2014-01-20    1903.0       -7.0\n",
       "                           N836AW   2014-01-08    1928.0       18.0\n",
       "                                    2014-01-29    1908.0       -2.0\n",
       "                           N837AW   2014-01-10    1902.0       -8.0\n",
       "\n",
       "[139194 rows x 2 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdf.loc[['AA', 'DL', 'US'], ['dep_time', 'dep_delay']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, so good. What if you wanted to select the rows whose origin was Chicago O'Hare (`ORD`) or Des Moines International Airport (DSM).\n",
    "Well, `.loc` wants `[row_indexer, column_indexer]` so let's wrap the two elements of our row indexer (the list of carriers and the list of origins) in a tuple to make it a single unit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>dep_time</th>\n",
       "      <th>dep_delay</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th>origin</th>\n",
       "      <th>dest</th>\n",
       "      <th>tail_num</th>\n",
       "      <th>fl_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AA</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DSM</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DFW</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">N200AA</th>\n",
       "      <th>2014-01-12</th>\n",
       "      <td>603.0</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-17</th>\n",
       "      <td>751.0</td>\n",
       "      <td>101.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">N424AA</th>\n",
       "      <th>2014-01-10</th>\n",
       "      <td>1759.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-15</th>\n",
       "      <td>1818.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N426AA</th>\n",
       "      <th>2014-01-07</th>\n",
       "      <td>1835.0</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">US</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">ORD</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">PHX</th>\n",
       "      <th>N806AW</th>\n",
       "      <th>2014-01-26</th>\n",
       "      <td>1406.0</td>\n",
       "      <td>-4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N830AW</th>\n",
       "      <th>2014-01-28</th>\n",
       "      <td>1401.0</td>\n",
       "      <td>-9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N833AW</th>\n",
       "      <th>2014-01-10</th>\n",
       "      <td>1500.0</td>\n",
       "      <td>50.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N837AW</th>\n",
       "      <th>2014-01-19</th>\n",
       "      <td>1408.0</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N839AW</th>\n",
       "      <th>2014-01-14</th>\n",
       "      <td>1406.0</td>\n",
       "      <td>-4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5205 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                dep_time  dep_delay\n",
       "unique_carrier origin dest tail_num fl_date                        \n",
       "AA             DSM    DFW  N200AA   2014-01-12     603.0       -7.0\n",
       "                                    2014-01-17     751.0      101.0\n",
       "                           N424AA   2014-01-10    1759.0       -1.0\n",
       "                                    2014-01-15    1818.0       18.0\n",
       "                           N426AA   2014-01-07    1835.0       35.0\n",
       "...                                                  ...        ...\n",
       "US             ORD    PHX  N806AW   2014-01-26    1406.0       -4.0\n",
       "                           N830AW   2014-01-28    1401.0       -9.0\n",
       "                           N833AW   2014-01-10    1500.0       50.0\n",
       "                           N837AW   2014-01-19    1408.0       -2.0\n",
       "                           N839AW   2014-01-14    1406.0       -4.0\n",
       "\n",
       "[5205 rows x 2 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdf.loc[(['AA', 'DL', 'US'], ['ORD', 'DSM']), ['dep_time', 'dep_delay']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try to do any flight from ORD or DSM, not just from those carriers.\n",
    "This used to be a pain.\n",
    "You might have to turn to the `.xs` method, or pass in `df.index.get_level_values(0)` and zip that up with the indexers your wanted, or maybe reset the index and do a boolean mask, and set the index again... ugh.\n",
    "\n",
    "But now, you can use an `IndexSlice`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>dep_time</th>\n",
       "      <th>dep_delay</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th>origin</th>\n",
       "      <th>dest</th>\n",
       "      <th>tail_num</th>\n",
       "      <th>fl_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">AA</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DSM</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DFW</th>\n",
       "      <th rowspan=\"2\" valign=\"top\">N200AA</th>\n",
       "      <th>2014-01-12</th>\n",
       "      <td>603.0</td>\n",
       "      <td>-7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-17</th>\n",
       "      <td>751.0</td>\n",
       "      <td>101.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">N424AA</th>\n",
       "      <th>2014-01-10</th>\n",
       "      <td>1759.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-15</th>\n",
       "      <td>1818.0</td>\n",
       "      <td>18.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N426AA</th>\n",
       "      <th>2014-01-07</th>\n",
       "      <td>1835.0</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">WN</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DSM</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">MDW</th>\n",
       "      <th>N941WN</th>\n",
       "      <th>2014-01-17</th>\n",
       "      <td>1759.0</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N943WN</th>\n",
       "      <th>2014-01-10</th>\n",
       "      <td>2229.0</td>\n",
       "      <td>284.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N963WN</th>\n",
       "      <th>2014-01-22</th>\n",
       "      <td>656.0</td>\n",
       "      <td>-4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N967WN</th>\n",
       "      <th>2014-01-30</th>\n",
       "      <td>654.0</td>\n",
       "      <td>-6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N969WN</th>\n",
       "      <th>2014-01-19</th>\n",
       "      <td>1747.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>22380 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                dep_time  dep_delay\n",
       "unique_carrier origin dest tail_num fl_date                        \n",
       "AA             DSM    DFW  N200AA   2014-01-12     603.0       -7.0\n",
       "                                    2014-01-17     751.0      101.0\n",
       "                           N424AA   2014-01-10    1759.0       -1.0\n",
       "                                    2014-01-15    1818.0       18.0\n",
       "                           N426AA   2014-01-07    1835.0       35.0\n",
       "...                                                  ...        ...\n",
       "WN             DSM    MDW  N941WN   2014-01-17    1759.0       14.0\n",
       "                           N943WN   2014-01-10    2229.0      284.0\n",
       "                           N963WN   2014-01-22     656.0       -4.0\n",
       "                           N967WN   2014-01-30     654.0       -6.0\n",
       "                           N969WN   2014-01-19    1747.0        2.0\n",
       "\n",
       "[22380 rows x 2 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdf.loc[pd.IndexSlice[:, ['ORD', 'DSM']], ['dep_time', 'dep_delay']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `:` says include every label in this level.\n",
    "The `IndexSlice` object is just sugar for the actual python `slice` object needed to remove slice each level."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(slice(None, None, None), ['ORD', 'DSM'])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.IndexSlice[:, ['ORD', 'DSM']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `IndexSlice` since `hdf.loc[(:, ['ORD', 'DSM'])]` isn't valid python syntax.\n",
    "Now we can slice to our heart's content; all flights from O'Hare to Des Moines in the first half of January? Sure, why not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"0\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>dep_time</th>\n",
       "      <th>dep_delay</th>\n",
       "      <th>arr_time</th>\n",
       "      <th>arr_delay</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique_carrier</th>\n",
       "      <th>origin</th>\n",
       "      <th>dest</th>\n",
       "      <th>tail_num</th>\n",
       "      <th>fl_date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">EV</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">ORD</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DSM</th>\n",
       "      <th>NaN</th>\n",
       "      <th>2014-01-07</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N11121</th>\n",
       "      <th>2014-01-05</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N11181</th>\n",
       "      <th>2014-01-12</th>\n",
       "      <td>1514.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1625.0</td>\n",
       "      <td>-2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N11536</th>\n",
       "      <th>2014-01-10</th>\n",
       "      <td>1723.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1853.0</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N11539</th>\n",
       "      <th>2014-01-01</th>\n",
       "      <td>1127.0</td>\n",
       "      <td>127.0</td>\n",
       "      <td>1304.0</td>\n",
       "      <td>149.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">UA</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">ORD</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">DSM</th>\n",
       "      <th>N24212</th>\n",
       "      <th>2014-01-09</th>\n",
       "      <td>2023.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>2158.0</td>\n",
       "      <td>34.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N73256</th>\n",
       "      <th>2014-01-15</th>\n",
       "      <td>2019.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2127.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">N78285</th>\n",
       "      <th>2014-01-07</th>\n",
       "      <td>2020.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2136.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014-01-13</th>\n",
       "      <td>2014.0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>2114.0</td>\n",
       "      <td>-10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>N841UA</th>\n",
       "      <th>2014-01-11</th>\n",
       "      <td>1825.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>1939.0</td>\n",
       "      <td>19.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>153 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                dep_time  dep_delay  arr_time  \\\n",
       "unique_carrier origin dest tail_num fl_date                                     \n",
       "EV             ORD    DSM  NaN      2014-01-07       NaN        NaN       NaN   \n",
       "                           N11121   2014-01-05       NaN        NaN       NaN   \n",
       "                           N11181   2014-01-12    1514.0        6.0    1625.0   \n",
       "                           N11536   2014-01-10    1723.0        4.0    1853.0   \n",
       "                           N11539   2014-01-01    1127.0      127.0    1304.0   \n",
       "...                                                  ...        ...       ...   \n",
       "UA             ORD    DSM  N24212   2014-01-09    2023.0        8.0    2158.0   \n",
       "                           N73256   2014-01-15    2019.0        4.0    2127.0   \n",
       "                           N78285   2014-01-07    2020.0        5.0    2136.0   \n",
       "                                    2014-01-13    2014.0       -1.0    2114.0   \n",
       "                           N841UA   2014-01-11    1825.0       20.0    1939.0   \n",
       "\n",
       "                                                arr_delay  \n",
       "unique_carrier origin dest tail_num fl_date                \n",
       "EV             ORD    DSM  NaN      2014-01-07        NaN  \n",
       "                           N11121   2014-01-05        NaN  \n",
       "                           N11181   2014-01-12       -2.0  \n",
       "                           N11536   2014-01-10       19.0  \n",
       "                           N11539   2014-01-01      149.0  \n",
       "...                                                   ...  \n",
       "UA             ORD    DSM  N24212   2014-01-09       34.0  \n",
       "                           N73256   2014-01-15        3.0  \n",
       "                           N78285   2014-01-07       12.0  \n",
       "                                    2014-01-13      -10.0  \n",
       "                           N841UA   2014-01-11       19.0  \n",
       "\n",
       "[153 rows x 4 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hdf.loc[pd.IndexSlice[:, 'ORD', 'DSM', :, '2014-01-01':'2014-01-15'],\n",
    "        ['dep_time', 'dep_delay', 'arr_time', 'arr_delay']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll talk more about working with Indexes (including MultiIndexes) in a later post. I have an unproven thesis that they're underused because `IndexSlice` is underused, causing people to think they're more unwieldy than they actually are. But let's close out part one.\n",
    "\n",
    "## WrapUp\n",
    "\n",
    "This first post covered Indexing, a topic that's central to pandas.\n",
    "The power provided by the DataFrame comes with some unavoidable complexities.\n",
    "Best practices (using `.loc` and `.iloc`) will spare you many a headache.\n",
    "We then toured a couple of commonly misunderstood sub-topics, setting with copy and Hierarchical Indexing."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}