lessons/lesson11.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lesson 11 - Pandas Part I\n",
    "\n",
    "In this lesson we will cover some basic features of [Pandas](http://pandas.pydata.org).\n",
    "\n",
    "### Readings\n",
    "\n",
    "* McKinney: [Chapter 4. Numpy Basics: Arrays and Vectorized Computation](http://proquest.safaribooksonline.com/book/programming/python/9781491957653/numpy-basics-arrays-and-vectorized-computation/numpy_html)\n",
    "* McKinney: [Chapter 5. Getting Started with Pandas](http://proquest.safaribooksonline.com/book/programming/python/9781491957653/getting-started-with-pandas/pandas_html)\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "* [Series](#series)\n",
    "* [DataFrame](#dataframe)\n",
    "* [index, columns](#indexing)\n",
    "* [dtypes, info, describe](#attributes)\n",
    "* [read_csv](#readcsv)\n",
    "* [head, tail](#headtail)\n",
    "* [Indexing with bracket/dot notation, loc, iloc](#indexing2)\n",
    "* [transpose](#transpose)\n",
    "* [to_csv, to_excel, read_excel](#toread)\n",
    "* [to_datetime](#datetime)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"series\"></a>\n",
    "\n",
    "### Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cubs', 'pirates', 'giants', 'yankees', 'donkeys']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a list of strings\n",
    "my_list = ['cubs', 'pirates', 'giants', 'yankees', 'donkeys']\n",
    "my_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       cubs\n",
       "1    pirates\n",
       "2     giants\n",
       "3    yankees\n",
       "4    donkeys\n",
       "dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pandas Series from list\n",
    "series_from_list = pd.Series(my_list)\n",
    "series_from_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'yankees'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# indexing a Series is similar to lists and arrays\n",
    "series_from_list[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.67421844, 0.55303709, 0.56433867, 0.54957026, 0.44879646])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a numpy array\n",
    "my_array = np.random.rand(5)\n",
    "my_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.674218\n",
       "1    0.553037\n",
       "2    0.564339\n",
       "3    0.549570\n",
       "4    0.448796\n",
       "dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pandas Series from array\n",
    "series_from_array = pd.Series(my_array)\n",
    "series_from_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    0.553037\n",
       "3    0.549570\n",
       "dtype: float64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# indexing supports lists\n",
    "series_from_array[[1, 3]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    0.549570\n",
       "4    0.448796\n",
       "dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# indexing supports slices\n",
    "series_from_array[3:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"dataframe\"></a>\n",
    "\n",
    "### DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2D array to DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.30183528,  0.8527996 ,  1.292565  ,  0.77310374, -1.02036927],\n",
       "       [-0.21313282, -0.50346792, -1.65864876,  0.00325651,  0.06528936],\n",
       "       [-0.51409901,  0.19897581, -1.56643912,  2.9767219 ,  1.50389804],\n",
       "       [ 0.16888865,  1.00044238,  0.93583698,  0.07282361, -2.1335255 ],\n",
       "       [ 0.46805536, -0.42282092,  0.79700182,  0.04224115,  0.55716763]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a 2D numpy array\n",
    "my_2d_array = np.random.randn(5,5)\n",
    "my_2d_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.301835</td>\n",
       "      <td>0.852800</td>\n",
       "      <td>1.292565</td>\n",
       "      <td>0.773104</td>\n",
       "      <td>-1.020369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.213133</td>\n",
       "      <td>-0.503468</td>\n",
       "      <td>-1.658649</td>\n",
       "      <td>0.003257</td>\n",
       "      <td>0.065289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.514099</td>\n",
       "      <td>0.198976</td>\n",
       "      <td>-1.566439</td>\n",
       "      <td>2.976722</td>\n",
       "      <td>1.503898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.168889</td>\n",
       "      <td>1.000442</td>\n",
       "      <td>0.935837</td>\n",
       "      <td>0.072824</td>\n",
       "      <td>-2.133526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.468055</td>\n",
       "      <td>-0.422821</td>\n",
       "      <td>0.797002</td>\n",
       "      <td>0.042241</td>\n",
       "      <td>0.557168</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2         3         4\n",
       "0  0.301835  0.852800  1.292565  0.773104 -1.020369\n",
       "1 -0.213133 -0.503468 -1.658649  0.003257  0.065289\n",
       "2 -0.514099  0.198976 -1.566439  2.976722  1.503898\n",
       "3  0.168889  1.000442  0.935837  0.072824 -2.133526\n",
       "4  0.468055 -0.422821  0.797002  0.042241  0.557168"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make a DataFrame from the 2D numpy array\n",
    "pd.DataFrame(my_2d_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>col1</th>\n",
       "      <th>col2</th>\n",
       "      <th>col3</th>\n",
       "      <th>col4</th>\n",
       "      <th>col5</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>row1</th>\n",
       "      <td>0.301835</td>\n",
       "      <td>0.852800</td>\n",
       "      <td>1.292565</td>\n",
       "      <td>0.773104</td>\n",
       "      <td>-1.020369</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row2</th>\n",
       "      <td>-0.213133</td>\n",
       "      <td>-0.503468</td>\n",
       "      <td>-1.658649</td>\n",
       "      <td>0.003257</td>\n",
       "      <td>0.065289</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row3</th>\n",
       "      <td>-0.514099</td>\n",
       "      <td>0.198976</td>\n",
       "      <td>-1.566439</td>\n",
       "      <td>2.976722</td>\n",
       "      <td>1.503898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row4</th>\n",
       "      <td>0.168889</td>\n",
       "      <td>1.000442</td>\n",
       "      <td>0.935837</td>\n",
       "      <td>0.072824</td>\n",
       "      <td>-2.133526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>row5</th>\n",
       "      <td>0.468055</td>\n",
       "      <td>-0.422821</td>\n",
       "      <td>0.797002</td>\n",
       "      <td>0.042241</td>\n",
       "      <td>0.557168</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          col1      col2      col3      col4      col5\n",
       "row1  0.301835  0.852800  1.292565  0.773104 -1.020369\n",
       "row2 -0.213133 -0.503468 -1.658649  0.003257  0.065289\n",
       "row3 -0.514099  0.198976 -1.566439  2.976722  1.503898\n",
       "row4  0.168889  1.000442  0.935837  0.072824 -2.133526\n",
       "row5  0.468055 -0.422821  0.797002  0.042241  0.557168"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can set the index and column labels when we create the DataFrame\n",
    "# note that we can combine positional arguments (data) and keyword arguments\n",
    "# (index, columns) as long as positional arguments come first\n",
    "df_from_2d_array = pd.DataFrame(\n",
    "    my_2d_array,\n",
    "    index=['row1', 'row2', 'row3', 'row4', 'row5'],\n",
    "    columns=['col1', 'col2', 'col3', 'col4', 'col5'])\n",
    "df_from_2d_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# also note that commas and brackets/parentheses can precede a newline in python code\n",
    "mylist = [\n",
    "    0, 2, 4,\n",
    "    6, 8, 10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 2, 4, 6, 8, 10]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mylist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Converting multiple Series to a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cubs</td>\n",
       "      <td>pirates</td>\n",
       "      <td>giants</td>\n",
       "      <td>yankees</td>\n",
       "      <td>donkeys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.674218</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>0.54957</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0         1         2        3         4\n",
       "0      cubs   pirates    giants  yankees   donkeys\n",
       "1  0.674218  0.553037  0.564339  0.54957  0.448796"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# method 1: getting data as a list of series will orient them as rows\n",
    "# this is typically not ideal\n",
    "x = pd.DataFrame(data=[series_from_list, series_from_array])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    object\n",
       "1    object\n",
       "2    object\n",
       "3    object\n",
       "4    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# note that each column has dtype object (string)\n",
    "x.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.54957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         0         1\n",
       "0     cubs  0.674218\n",
       "1  pirates  0.553037\n",
       "2   giants  0.564339\n",
       "3  yankees   0.54957\n",
       "4  donkeys  0.448796"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# in this example, we need to transpose the table - we'll see this again later in the lesson\n",
    "x = x.transpose()\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    object\n",
       "1    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the transposed columns have dtype object - we'll see how to fix this below\n",
    "x.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         a         b\n",
       "0     cubs  0.674218\n",
       "1  pirates  0.553037\n",
       "2   giants  0.564339\n",
       "3  yankees  0.549570\n",
       "4  donkeys  0.448796"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# method 2: pass list/Series as value of dictionary\n",
    "y = pd.DataFrame({'a': series_from_list, 'b': series_from_array})\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a     object\n",
       "b    float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# importing the data as columns gives us the correct dtypes\n",
    "y.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         0         1\n",
       "0     cubs  0.674218\n",
       "1  pirates  0.553037\n",
       "2   giants  0.564339\n",
       "3  yankees  0.549570\n",
       "4  donkeys  0.448796"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# method 3: use pd.concat to combine series in column orientation\n",
    "df = pd.concat([series_from_list, series_from_array], axis=1)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     object\n",
       "1    float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# again, importing the data as columns gives us the correct dtypes\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"indexing\"></a>\n",
    "\n",
    "### index, columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=5, step=1)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# numeric indexes\n",
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=2, step=1)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# numeric column names\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random\n",
       "a     cubs  0.674218\n",
       "b  pirates  0.553037\n",
       "c   giants  0.564339\n",
       "d  yankees  0.549570\n",
       "e  donkeys  0.448796"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rename columns and indexes - method 1\n",
    "# set the index and column names to an existing DataFrame\n",
    "df.index = ['a', 'b', 'c', 'd', 'e']\n",
    "df.columns = ['team', 'random']\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['a', 'b', 'c', 'd', 'e'], dtype='object')"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# label (object) indexes\n",
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['team', 'random'], dtype='object')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# label (object) column names\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>integers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  integers\n",
       "a     cubs  0.674218         2\n",
       "b  pirates  0.553037         3\n",
       "c   giants  0.564339         5\n",
       "d  yankees  0.549570         8\n",
       "e  donkeys  0.448796        13"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add a new column to the end of a DataFrame\n",
    "df['integers'] = [2, 3, 5, 8, 13]\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>integers2</th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>integers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>2</td>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>3</td>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>5</td>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>8</td>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>13</td>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   integers2     team    random  integers\n",
       "a          2     cubs  0.674218         2\n",
       "b          3  pirates  0.553037         3\n",
       "c          5   giants  0.564339         5\n",
       "d          8  yankees  0.549570         8\n",
       "e         13  donkeys  0.448796        13"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add a new column at a specific position\n",
    "df.insert(0, 'integers2', [2, 3, 5, 8, 13])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>integers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  integers\n",
       "a     cubs  0.674218         2\n",
       "b  pirates  0.553037         3\n",
       "c   giants  0.564339         5\n",
       "d  yankees  0.549570         8\n",
       "e  donkeys  0.448796        13"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# delete a column\n",
    "df.drop('integers2', axis=1, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  fibonacci\n",
       "a     cubs  0.674218          2\n",
       "b  pirates  0.553037          3\n",
       "c   giants  0.564339          5\n",
       "d  yankees  0.549570          8\n",
       "e  donkeys  0.448796         13"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# rename columns - method 2\n",
    "# using dictionary mapping old names to new names\n",
    "df.rename(columns={'integers': 'fibonacci'}, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fibonacci</th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>2</td>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>3</td>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>5</td>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>8</td>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>13</td>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fibonacci     team    random\n",
       "a          2     cubs  0.674218\n",
       "b          3  pirates  0.553037\n",
       "c          5   giants  0.564339\n",
       "d          8  yankees  0.549570\n",
       "e         13  donkeys  0.448796"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reorder the columns by passing a list of columns in the desired order\n",
    "df[['fibonacci', 'team', 'random']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"attributes\"></a>\n",
    "\n",
    "### dtypes, info, describe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "team          object\n",
       "random       float64\n",
       "fibonacci      int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# gives the datatype of each column\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 3)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# shape (dimensions) of dataframe\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 5 entries, a to e\n",
      "Data columns (total 3 columns):\n",
      "team         5 non-null object\n",
      "random       5 non-null float64\n",
      "fibonacci    5 non-null int64\n",
      "dtypes: float64(1), int64(1), object(1)\n",
      "memory usage: 160.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "# information about index and columns\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.557992</td>\n",
       "      <td>6.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.079950</td>\n",
       "      <td>4.438468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.448796</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.549570</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.553037</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.564339</td>\n",
       "      <td>8.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.674218</td>\n",
       "      <td>13.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         random  fibonacci\n",
       "count  5.000000   5.000000\n",
       "mean   0.557992   6.200000\n",
       "std    0.079950   4.438468\n",
       "min    0.448796   2.000000\n",
       "25%    0.549570   3.000000\n",
       "50%    0.553037   5.000000\n",
       "75%    0.564339   8.000000\n",
       "max    0.674218  13.000000"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# basic statistics\n",
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"readcsv\"></a>\n",
    "\n",
    "### read_csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read_csv with defaults\n",
    "# by default column headers are the first row and row indexes are integers starting from zero\n",
    "df_sio = pd.read_csv('../data/scripps_pier_20151110.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/10/15 1:42</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.712</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/10/15 1:35</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.588</td>\n",
       "      <td>33.201</td>\n",
       "      <td>19.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11/10/15 1:29</td>\n",
       "      <td>22.305</td>\n",
       "      <td>3.541</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11/10/15 1:23</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.463</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11/10/15 1:17</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.471</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date  chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "0  11/10/15 1:42      22.307        3.712     33.199     19.95\n",
       "1  11/10/15 1:35      22.311        3.588     33.201     19.94\n",
       "2  11/10/15 1:29      22.305        3.541     33.200     19.95\n",
       "3  11/10/15 1:23      22.323        3.463     33.200     19.95\n",
       "4  11/10/15 1:17      22.316        3.471     33.199     19.95"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date            object\n",
       "chl (ug/L)     float64\n",
       "pres (dbar)    float64\n",
       "sal (PSU)      float64\n",
       "temp (C)       float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# by default, read_csv will infer the object types\n",
    "df_sio.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 66 entries, 0 to 65\n",
      "Data columns (total 5 columns):\n",
      "Date           66 non-null object\n",
      "chl (ug/L)     66 non-null float64\n",
      "pres (dbar)    66 non-null float64\n",
      "sal (PSU)      66 non-null float64\n",
      "temp (C)       66 non-null float64\n",
      "dtypes: float64(4), object(1)\n",
      "memory usage: 2.7+ KB\n"
     ]
    }
   ],
   "source": [
    "df_sio.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>66.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>22.349576</td>\n",
       "      <td>3.041818</td>\n",
       "      <td>33.199318</td>\n",
       "      <td>20.06697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.038988</td>\n",
       "      <td>0.254295</td>\n",
       "      <td>0.004959</td>\n",
       "      <td>0.06850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>22.305000</td>\n",
       "      <td>2.714000</td>\n",
       "      <td>33.184000</td>\n",
       "      <td>19.94000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>22.319000</td>\n",
       "      <td>2.813250</td>\n",
       "      <td>33.197000</td>\n",
       "      <td>20.04000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>22.333500</td>\n",
       "      <td>2.997000</td>\n",
       "      <td>33.199000</td>\n",
       "      <td>20.07000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>22.385000</td>\n",
       "      <td>3.215500</td>\n",
       "      <td>33.203000</td>\n",
       "      <td>20.10500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>22.426000</td>\n",
       "      <td>3.712000</td>\n",
       "      <td>33.206000</td>\n",
       "      <td>20.19000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "count   66.000000    66.000000  66.000000  66.00000\n",
       "mean    22.349576     3.041818  33.199318  20.06697\n",
       "std      0.038988     0.254295   0.004959   0.06850\n",
       "min     22.305000     2.714000  33.184000  19.94000\n",
       "25%     22.319000     2.813250  33.197000  20.04000\n",
       "50%     22.333500     2.997000  33.199000  20.07000\n",
       "75%     22.385000     3.215500  33.203000  20.10500\n",
       "max     22.426000     3.712000  33.206000  20.19000"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read_csv specifying dtype (for all columns), index_col, and header\n",
    "# sometimes it's better to specify the dtype as object and convert to int, float, etc. later\n",
    "df_sio2 = pd.read_csv('../data/scripps_pier_20151110.csv', \n",
    "                     dtype=object, index_col=None, header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           object\n",
       "chl (ug/L)     object\n",
       "pres (dbar)    object\n",
       "sal (PSU)      object\n",
       "temp (C)       object\n",
       "dtype: object"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio2.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read_csv specifying dtypes (per column), index_col, and header\n",
    "# this allows us to have more control over the dtype of each column\n",
    "df_sio3 = pd.read_csv('../data/scripps_pier_20151110.csv',\n",
    "                      dtype={'pres (dbar)': np.float64, 'temp (C)': str}, \n",
    "                      index_col=None, header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date            object\n",
       "chl (ug/L)     float64\n",
       "pres (dbar)    float64\n",
       "sal (PSU)      float64\n",
       "temp (C)        object\n",
       "dtype: object"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio3.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Changing dtype of columns after DataFrame is created"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# method 1: list comprehension (one column)\n",
    "df_sio['chl (ug/L)'] = [float(x) for x in df_sio['chl (ug/L)']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# method 2: pd.to_numeric (one column)\n",
    "df_sio['pres (dbar)'] = pd.to_numeric(df_sio['pres (dbar)'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# method 3: apply(pd.to_numeric) (multiple columns)\n",
    "df_sio[['sal (PSU)','temp (C)']] = df_sio[['sal (PSU)','temp (C)']].apply(pd.to_numeric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date            object\n",
       "chl (ug/L)     float64\n",
       "pres (dbar)    float64\n",
       "sal (PSU)      float64\n",
       "temp (C)       float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"headtail\"></a>\n",
    "\n",
    "### head, tail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/10/15 1:42</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.712</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/10/15 1:35</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.588</td>\n",
       "      <td>33.201</td>\n",
       "      <td>19.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11/10/15 1:29</td>\n",
       "      <td>22.305</td>\n",
       "      <td>3.541</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11/10/15 1:23</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.463</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11/10/15 1:17</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.471</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date  chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "0  11/10/15 1:42      22.307        3.712     33.199     19.95\n",
       "1  11/10/15 1:35      22.311        3.588     33.201     19.94\n",
       "2  11/10/15 1:29      22.305        3.541     33.200     19.95\n",
       "3  11/10/15 1:23      22.323        3.463     33.200     19.95\n",
       "4  11/10/15 1:17      22.316        3.471     33.199     19.95"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add a number to change the number of rows printed\n",
    "df_sio.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>11/9/15 19:22</td>\n",
       "      <td>22.418</td>\n",
       "      <td>3.316</td>\n",
       "      <td>33.202</td>\n",
       "      <td>19.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>11/9/15 19:16</td>\n",
       "      <td>22.410</td>\n",
       "      <td>3.209</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>11/9/15 19:10</td>\n",
       "      <td>22.426</td>\n",
       "      <td>3.328</td>\n",
       "      <td>33.203</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             Date  chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "63  11/9/15 19:22      22.418        3.316     33.202     19.96\n",
       "64  11/9/15 19:16      22.410        3.209     33.200     19.96\n",
       "65  11/9/15 19:10      22.426        3.328     33.203     19.95"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tail works the same way\n",
    "df_sio.tail(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/10/15 1:42</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.712</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/10/15 1:35</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.588</td>\n",
       "      <td>33.201</td>\n",
       "      <td>19.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11/10/15 1:29</td>\n",
       "      <td>22.305</td>\n",
       "      <td>3.541</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11/10/15 1:23</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.463</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11/10/15 1:17</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.471</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>11/10/15 1:11</td>\n",
       "      <td>22.315</td>\n",
       "      <td>3.476</td>\n",
       "      <td>33.198</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>11/10/15 1:05</td>\n",
       "      <td>22.310</td>\n",
       "      <td>3.448</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>11/10/15 0:59</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.377</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>11/10/15 0:53</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.338</td>\n",
       "      <td>33.200</td>\n",
       "      <td>20.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>11/10/15 0:47</td>\n",
       "      <td>22.322</td>\n",
       "      <td>3.325</td>\n",
       "      <td>33.201</td>\n",
       "      <td>20.01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>11/10/15 0:41</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.344</td>\n",
       "      <td>33.200</td>\n",
       "      <td>20.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11/10/15 0:35</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.217</td>\n",
       "      <td>33.201</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>11/10/15 0:29</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.265</td>\n",
       "      <td>33.199</td>\n",
       "      <td>20.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>11/10/15 0:23</td>\n",
       "      <td>22.320</td>\n",
       "      <td>3.220</td>\n",
       "      <td>33.197</td>\n",
       "      <td>20.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>11/10/15 0:17</td>\n",
       "      <td>22.322</td>\n",
       "      <td>3.211</td>\n",
       "      <td>33.199</td>\n",
       "      <td>20.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>11/10/15 0:11</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.134</td>\n",
       "      <td>33.197</td>\n",
       "      <td>20.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>11/10/15 0:05</td>\n",
       "      <td>22.315</td>\n",
       "      <td>3.018</td>\n",
       "      <td>33.197</td>\n",
       "      <td>20.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>11/9/15 23:59</td>\n",
       "      <td>22.319</td>\n",
       "      <td>3.031</td>\n",
       "      <td>33.198</td>\n",
       "      <td>20.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>11/9/15 23:53</td>\n",
       "      <td>22.327</td>\n",
       "      <td>3.118</td>\n",
       "      <td>33.199</td>\n",
       "      <td>20.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>11/9/15 23:47</td>\n",
       "      <td>22.332</td>\n",
       "      <td>2.962</td>\n",
       "      <td>33.199</td>\n",
       "      <td>20.13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>11/9/15 23:41</td>\n",
       "      <td>22.328</td>\n",
       "      <td>3.014</td>\n",
       "      <td>33.198</td>\n",
       "      <td>20.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>11/9/15 23:35</td>\n",
       "      <td>22.341</td>\n",
       "      <td>2.972</td>\n",
       "      <td>33.193</td>\n",
       "      <td>20.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>11/9/15 23:29</td>\n",
       "      <td>22.336</td>\n",
       "      <td>3.013</td>\n",
       "      <td>33.194</td>\n",
       "      <td>20.17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>11/9/15 23:23</td>\n",
       "      <td>22.337</td>\n",
       "      <td>3.011</td>\n",
       "      <td>33.194</td>\n",
       "      <td>20.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>11/9/15 23:17</td>\n",
       "      <td>22.334</td>\n",
       "      <td>2.875</td>\n",
       "      <td>33.194</td>\n",
       "      <td>20.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>11/9/15 23:11</td>\n",
       "      <td>22.337</td>\n",
       "      <td>2.809</td>\n",
       "      <td>33.194</td>\n",
       "      <td>20.16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>11/9/15 23:05</td>\n",
       "      <td>22.335</td>\n",
       "      <td>2.868</td>\n",
       "      <td>33.195</td>\n",
       "      <td>20.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>11/9/15 22:59</td>\n",
       "      <td>22.330</td>\n",
       "      <td>2.851</td>\n",
       "      <td>33.197</td>\n",
       "      <td>20.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>11/9/15 22:53</td>\n",
       "      <td>22.315</td>\n",
       "      <td>2.769</td>\n",
       "      <td>33.199</td>\n",
       "      <td>20.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>11/9/15 22:47</td>\n",
       "      <td>22.318</td>\n",
       "      <td>2.774</td>\n",
       "      <td>33.191</td>\n",
       "      <td>20.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>11/9/15 22:05</td>\n",
       "      <td>22.325</td>\n",
       "      <td>2.770</td>\n",
       "      <td>33.199</td>\n",
       "      <td>20.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>11/9/15 21:59</td>\n",
       "      <td>22.320</td>\n",
       "      <td>2.760</td>\n",
       "      <td>33.198</td>\n",
       "      <td>20.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>11/9/15 21:53</td>\n",
       "      <td>22.335</td>\n",
       "      <td>2.798</td>\n",
       "      <td>33.198</td>\n",
       "      <td>20.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>11/9/15 21:47</td>\n",
       "      <td>22.333</td>\n",
       "      <td>2.764</td>\n",
       "      <td>33.198</td>\n",
       "      <td>20.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>11/9/15 21:41</td>\n",
       "      <td>22.343</td>\n",
       "      <td>2.714</td>\n",
       "      <td>33.197</td>\n",
       "      <td>20.09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>11/9/15 21:35</td>\n",
       "      <td>22.351</td>\n",
       "      <td>2.821</td>\n",
       "      <td>33.195</td>\n",
       "      <td>20.06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>11/9/15 21:29</td>\n",
       "      <td>22.359</td>\n",
       "      <td>2.832</td>\n",
       "      <td>33.201</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>11/9/15 21:23</td>\n",
       "      <td>22.363</td>\n",
       "      <td>2.735</td>\n",
       "      <td>33.203</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>11/9/15 21:17</td>\n",
       "      <td>22.363</td>\n",
       "      <td>2.746</td>\n",
       "      <td>33.205</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>11/9/15 21:11</td>\n",
       "      <td>22.368</td>\n",
       "      <td>2.760</td>\n",
       "      <td>33.204</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>11/9/15 21:05</td>\n",
       "      <td>22.370</td>\n",
       "      <td>2.805</td>\n",
       "      <td>33.205</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>11/9/15 20:59</td>\n",
       "      <td>22.377</td>\n",
       "      <td>2.813</td>\n",
       "      <td>33.205</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>11/9/15 20:53</td>\n",
       "      <td>22.395</td>\n",
       "      <td>2.913</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>11/9/15 20:47</td>\n",
       "      <td>22.395</td>\n",
       "      <td>2.818</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>11/9/15 20:41</td>\n",
       "      <td>22.379</td>\n",
       "      <td>2.877</td>\n",
       "      <td>33.205</td>\n",
       "      <td>20.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>11/9/15 20:35</td>\n",
       "      <td>22.388</td>\n",
       "      <td>2.921</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>11/9/15 20:29</td>\n",
       "      <td>22.387</td>\n",
       "      <td>2.971</td>\n",
       "      <td>33.205</td>\n",
       "      <td>20.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>11/9/15 20:23</td>\n",
       "      <td>22.392</td>\n",
       "      <td>2.983</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>54</th>\n",
       "      <td>11/9/15 20:17</td>\n",
       "      <td>22.402</td>\n",
       "      <td>3.061</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>11/9/15 20:11</td>\n",
       "      <td>22.404</td>\n",
       "      <td>2.961</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>11/9/15 20:05</td>\n",
       "      <td>22.409</td>\n",
       "      <td>3.065</td>\n",
       "      <td>33.204</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>11/9/15 19:59</td>\n",
       "      <td>22.412</td>\n",
       "      <td>3.159</td>\n",
       "      <td>33.203</td>\n",
       "      <td>20.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>11/9/15 19:53</td>\n",
       "      <td>22.418</td>\n",
       "      <td>3.039</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>11/9/15 19:47</td>\n",
       "      <td>22.422</td>\n",
       "      <td>3.163</td>\n",
       "      <td>33.204</td>\n",
       "      <td>20.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>11/9/15 19:41</td>\n",
       "      <td>22.423</td>\n",
       "      <td>3.193</td>\n",
       "      <td>33.206</td>\n",
       "      <td>20.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>11/9/15 19:34</td>\n",
       "      <td>22.423</td>\n",
       "      <td>3.204</td>\n",
       "      <td>33.202</td>\n",
       "      <td>20.05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>11/9/15 19:28</td>\n",
       "      <td>22.413</td>\n",
       "      <td>3.254</td>\n",
       "      <td>33.200</td>\n",
       "      <td>20.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>11/9/15 19:22</td>\n",
       "      <td>22.418</td>\n",
       "      <td>3.316</td>\n",
       "      <td>33.202</td>\n",
       "      <td>19.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>11/9/15 19:16</td>\n",
       "      <td>22.410</td>\n",
       "      <td>3.209</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>11/9/15 19:10</td>\n",
       "      <td>22.426</td>\n",
       "      <td>3.328</td>\n",
       "      <td>33.203</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             Date  chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "0   11/10/15 1:42      22.307        3.712     33.199     19.95\n",
       "1   11/10/15 1:35      22.311        3.588     33.201     19.94\n",
       "2   11/10/15 1:29      22.305        3.541     33.200     19.95\n",
       "3   11/10/15 1:23      22.323        3.463     33.200     19.95\n",
       "4   11/10/15 1:17      22.316        3.471     33.199     19.95\n",
       "5   11/10/15 1:11      22.315        3.476     33.198     19.95\n",
       "6   11/10/15 1:05      22.310        3.448     33.199     19.96\n",
       "7   11/10/15 0:59      22.316        3.377     33.200     19.99\n",
       "8   11/10/15 0:53      22.311        3.338     33.200     20.00\n",
       "9   11/10/15 0:47      22.322        3.325     33.201     20.01\n",
       "10  11/10/15 0:41      22.311        3.344     33.200     20.04\n",
       "11  11/10/15 0:35      22.311        3.217     33.201     20.05\n",
       "12  11/10/15 0:29      22.307        3.265     33.199     20.08\n",
       "13  11/10/15 0:23      22.320        3.220     33.197     20.09\n",
       "14  11/10/15 0:17      22.322        3.211     33.199     20.08\n",
       "15  11/10/15 0:11      22.323        3.134     33.197     20.09\n",
       "16  11/10/15 0:05      22.315        3.018     33.197     20.09\n",
       "17  11/9/15 23:59      22.319        3.031     33.198     20.08\n",
       "18  11/9/15 23:53      22.327        3.118     33.199     20.11\n",
       "19  11/9/15 23:47      22.332        2.962     33.199     20.13\n",
       "20  11/9/15 23:41      22.328        3.014     33.198     20.16\n",
       "21  11/9/15 23:35      22.341        2.972     33.193     20.18\n",
       "22  11/9/15 23:29      22.336        3.013     33.194     20.17\n",
       "23  11/9/15 23:23      22.337        3.011     33.194     20.16\n",
       "24  11/9/15 23:17      22.334        2.875     33.194     20.16\n",
       "25  11/9/15 23:11      22.337        2.809     33.194     20.16\n",
       "26  11/9/15 23:05      22.335        2.868     33.195     20.18\n",
       "27  11/9/15 22:59      22.330        2.851     33.197     20.18\n",
       "28  11/9/15 22:53      22.315        2.769     33.199     20.18\n",
       "29  11/9/15 22:47      22.318        2.774     33.191     20.19\n",
       "..            ...         ...          ...        ...       ...\n",
       "36  11/9/15 22:05      22.325        2.770     33.199     20.07\n",
       "37  11/9/15 21:59      22.320        2.760     33.198     20.08\n",
       "38  11/9/15 21:53      22.335        2.798     33.198     20.08\n",
       "39  11/9/15 21:47      22.333        2.764     33.198     20.08\n",
       "40  11/9/15 21:41      22.343        2.714     33.197     20.09\n",
       "41  11/9/15 21:35      22.351        2.821     33.195     20.06\n",
       "42  11/9/15 21:29      22.359        2.832     33.201     20.05\n",
       "43  11/9/15 21:23      22.363        2.735     33.203     20.05\n",
       "44  11/9/15 21:17      22.363        2.746     33.205     20.05\n",
       "45  11/9/15 21:11      22.368        2.760     33.204     20.05\n",
       "46  11/9/15 21:05      22.370        2.805     33.205     20.05\n",
       "47  11/9/15 20:59      22.377        2.813     33.205     20.05\n",
       "48  11/9/15 20:53      22.395        2.913     33.206     20.05\n",
       "49  11/9/15 20:47      22.395        2.818     33.206     20.05\n",
       "50  11/9/15 20:41      22.379        2.877     33.205     20.07\n",
       "51  11/9/15 20:35      22.388        2.921     33.206     20.07\n",
       "52  11/9/15 20:29      22.387        2.971     33.205     20.07\n",
       "53  11/9/15 20:23      22.392        2.983     33.206     20.07\n",
       "54  11/9/15 20:17      22.402        3.061     33.206     20.07\n",
       "55  11/9/15 20:11      22.404        2.961     33.206     20.08\n",
       "56  11/9/15 20:05      22.409        3.065     33.204     20.05\n",
       "57  11/9/15 19:59      22.412        3.159     33.203     20.04\n",
       "58  11/9/15 19:53      22.418        3.039     33.206     20.04\n",
       "59  11/9/15 19:47      22.422        3.163     33.204     20.04\n",
       "60  11/9/15 19:41      22.423        3.193     33.206     20.03\n",
       "61  11/9/15 19:34      22.423        3.204     33.202     20.05\n",
       "62  11/9/15 19:28      22.413        3.254     33.200     20.02\n",
       "63  11/9/15 19:22      22.418        3.316     33.202     19.96\n",
       "64  11/9/15 19:16      22.410        3.209     33.200     19.96\n",
       "65  11/9/15 19:10      22.426        3.328     33.203     19.95\n",
       "\n",
       "[66 rows x 5 columns]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# if we view the whole dataframe, only the first 30 and last 30 rows are shown\n",
    "df_sio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<a id=\"indexing2\"></a>\n",
    "\n",
    "### Indexing with bracket/dot notation, loc, iloc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas has three indexing methods:\n",
    "\n",
    "* `[ ]` and `.` work on labels of columns\n",
    "* `.loc` works on labels of indexes and columns\n",
    "* `.iloc` works on the positions of indexes and columns (so it only takes integers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  fibonacci\n",
       "a     cubs  0.674218          2\n",
       "b  pirates  0.553037          3\n",
       "c   giants  0.564339          5\n",
       "d  yankees  0.549570          8\n",
       "e  donkeys  0.448796         13"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### brackets only -- column by header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a       cubs\n",
       "b    pirates\n",
       "c     giants\n",
       "d    yankees\n",
       "e    donkeys\n",
       "Name: team, dtype: object"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to get a column (Series), use the column header (don't need .loc, .iloc, or .ix)\n",
    "df['team']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random\n",
       "a     cubs  0.674218\n",
       "b  pirates  0.553037\n",
       "c   giants  0.564339\n",
       "d  yankees  0.549570\n",
       "e  donkeys  0.448796"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for multiple columns, put a list inside the brackets (so two sets of brackets)\n",
    "df[['team', 'random']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### dot-notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "a       cubs\n",
       "b    pirates\n",
       "c     giants\n",
       "d    yankees\n",
       "e    donkeys\n",
       "Name: team, dtype: object"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# if the column name has only alpha-numerics (including underscores), \n",
    "# we can use a dot instead of brackets and quotes\n",
    "df.team"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### loc -- row by index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "team             cubs\n",
       "random       0.674218\n",
       "fibonacci           2\n",
       "Name: a, dtype: object"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to get a row by name, use .loc with the row index\n",
    "df.loc['a']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  fibonacci\n",
       "a     cubs  0.674218          2\n",
       "d  yankees  0.549570          8"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for multiple rows, put a list inside the brackets (so two sets of brackets)\n",
    "df.loc[['a', 'd']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### iloc -- row (or column) by position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "team             cubs\n",
       "random       0.674218\n",
       "fibonacci           2\n",
       "Name: a, dtype: object"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to get a row by position, use .iloc with the row number\n",
    "df.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  fibonacci\n",
       "a     cubs  0.674218          2\n",
       "d  yankees  0.549570          8"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for multiple rows, put a list inside the brackets (so two sets of brackets)\n",
    "df.iloc[[0, 3]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  fibonacci\n",
       "c   giants  0.564339          5\n",
       "d  yankees  0.549570          8\n",
       "e  donkeys  0.448796         13"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or pass a slice\n",
    "df.iloc[2:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team  fibonacci\n",
       "a     cubs          2\n",
       "b  pirates          3\n",
       "c   giants          5\n",
       "d  yankees          8\n",
       "e  donkeys         13"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# iloc also works with columns\n",
    "df.iloc[:, [0, 2]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"transpose\"></a>\n",
    "\n",
    "### transpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>team</th>\n",
       "      <td>cubs</td>\n",
       "      <td>pirates</td>\n",
       "      <td>giants</td>\n",
       "      <td>yankees</td>\n",
       "      <td>donkeys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random</th>\n",
       "      <td>0.674218</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>0.54957</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fibonacci</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  a         b         c        d         e\n",
       "team           cubs   pirates    giants  yankees   donkeys\n",
       "random     0.674218  0.553037  0.564339  0.54957  0.448796\n",
       "fibonacci         2         3         5        8        13"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>team</th>\n",
       "      <td>cubs</td>\n",
       "      <td>pirates</td>\n",
       "      <td>giants</td>\n",
       "      <td>yankees</td>\n",
       "      <td>donkeys</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random</th>\n",
       "      <td>0.674218</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>0.54957</td>\n",
       "      <td>0.448796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fibonacci</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  a         b         c        d         e\n",
       "team           cubs   pirates    giants  yankees   donkeys\n",
       "random     0.674218  0.553037  0.564339  0.54957  0.448796\n",
       "fibonacci         2         3         5        8        13"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"toread\"></a>\n",
    "\n",
    "### to_csv, to_excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to_csv with defaults (sep=',')\n",
    "df.to_csv('teams.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the sep option if the separator is not a comma\n",
    "df.to_csv('teams.tsv', sep='\\t')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# with index label\n",
    "df.to_csv('teams.csv', index_label='index')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to_excel requires the openpyxl package\n",
    "df.to_excel('teams.xlsx', index_label='index')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### read_csv (revisited), read_excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>a</td>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b</td>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c</td>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>d</td>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>e</td>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  index     team    random  fibonacci\n",
       "0     a     cubs  0.674218          2\n",
       "1     b  pirates  0.553037          3\n",
       "2     c   giants  0.564339          5\n",
       "3     d  yankees  0.549570          8\n",
       "4     e  donkeys  0.448796         13"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read_csv with defaults\n",
    "pd.read_csv('teams.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          team    random  fibonacci\n",
       "index                              \n",
       "a         cubs  0.674218          2\n",
       "b      pirates  0.553037          3\n",
       "c       giants  0.564339          5\n",
       "d      yankees  0.549570          8\n",
       "e      donkeys  0.448796         13"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read_csv specifying first column of csv as index_col\n",
    "df1 = pd.read_csv('teams.csv', index_col=0)\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "team          object\n",
       "random       float64\n",
       "fibonacci      int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# default datatypes\n",
    "df1.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "# again, we can specify the dtypes when we read_csv\n",
    "df2 = pd.read_csv('teams.csv', index_col=0, dtype=object)\n",
    "df3 = pd.read_csv('teams.csv', index_col=0, \n",
    "                  dtype={'team': object, 'random': np.float, 'integers': np.int})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "team         object\n",
       "random       object\n",
       "fibonacci    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify datatypes: all object\n",
    "df2.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "team          object\n",
       "random       float64\n",
       "fibonacci      int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# specify datatypes: per column\n",
    "df3.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      team    random  fibonacci\n",
       "a     cubs  0.674218          2\n",
       "b  pirates  0.553037          3\n",
       "c   giants  0.564339          5\n",
       "d  yankees  0.549570          8\n",
       "e  donkeys  0.448796         13"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use the sep option if the separator is not a comma\n",
    "df4 = pd.read_csv('teams.tsv', index_col=0, sep='\\t')\n",
    "df4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>team</th>\n",
       "      <th>random</th>\n",
       "      <th>fibonacci</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>index</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>a</th>\n",
       "      <td>cubs</td>\n",
       "      <td>0.674218</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>pirates</td>\n",
       "      <td>0.553037</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>c</th>\n",
       "      <td>giants</td>\n",
       "      <td>0.564339</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>d</th>\n",
       "      <td>yankees</td>\n",
       "      <td>0.549570</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>donkeys</td>\n",
       "      <td>0.448796</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          team    random  fibonacci\n",
       "index                              \n",
       "a         cubs  0.674218          2\n",
       "b      pirates  0.553037          3\n",
       "c       giants  0.564339          5\n",
       "d      yankees  0.549570          8\n",
       "e      donkeys  0.448796         13"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# read_excel requires the xlrd package\n",
    "df5 = pd.read_excel('teams.xlsx', index_col=0)\n",
    "df5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"datetime\"></a>\n",
    "\n",
    "### to_datetime\n",
    "\n",
    "We will cover time series in greater detail in a future lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>11/10/15 1:42</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.712</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>11/10/15 1:35</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.588</td>\n",
       "      <td>33.201</td>\n",
       "      <td>19.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11/10/15 1:29</td>\n",
       "      <td>22.305</td>\n",
       "      <td>3.541</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11/10/15 1:23</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.463</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>11/10/15 1:17</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.471</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Date  chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "0  11/10/15 1:42      22.307        3.712     33.199     19.95\n",
       "1  11/10/15 1:35      22.311        3.588     33.201     19.94\n",
       "2  11/10/15 1:29      22.305        3.541     33.200     19.95\n",
       "3  11/10/15 1:23      22.323        3.463     33.200     19.95\n",
       "4  11/10/15 1:17      22.316        3.471     33.199     19.95"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date            object\n",
       "chl (ug/L)     float64\n",
       "pres (dbar)    float64\n",
       "sal (PSU)      float64\n",
       "temp (C)       float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   2015-11-10 01:42:00\n",
       "1   2015-11-10 01:35:00\n",
       "2   2015-11-10 01:29:00\n",
       "3   2015-11-10 01:23:00\n",
       "4   2015-11-10 01:17:00\n",
       "Name: Date, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time = pd.to_datetime(df_sio['Date'])\n",
    "time.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sio['Date'] = time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-11-10 01:42:00</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.712</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-11-10 01:35:00</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.588</td>\n",
       "      <td>33.201</td>\n",
       "      <td>19.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-11-10 01:29:00</td>\n",
       "      <td>22.305</td>\n",
       "      <td>3.541</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-11-10 01:23:00</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.463</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-11-10 01:17:00</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.471</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Date  chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "0 2015-11-10 01:42:00      22.307        3.712     33.199     19.95\n",
       "1 2015-11-10 01:35:00      22.311        3.588     33.201     19.94\n",
       "2 2015-11-10 01:29:00      22.305        3.541     33.200     19.95\n",
       "3 2015-11-10 01:23:00      22.323        3.463     33.200     19.95\n",
       "4 2015-11-10 01:17:00      22.316        3.471     33.199     19.95"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           datetime64[ns]\n",
       "chl (ug/L)            float64\n",
       "pres (dbar)           float64\n",
       "sal (PSU)             float64\n",
       "temp (C)              float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to do this in a single step, we can use read_csv's parse_dates keyword\n",
    "df_sio4 = pd.read_csv('../data/scripps_pier_20151110.csv', index_col=None,\n",
    "                      parse_dates=['Date'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-11-10 01:42:00</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.712</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-11-10 01:35:00</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.588</td>\n",
       "      <td>33.201</td>\n",
       "      <td>19.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-11-10 01:29:00</td>\n",
       "      <td>22.305</td>\n",
       "      <td>3.541</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-11-10 01:23:00</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.463</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-11-10 01:17:00</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.471</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Date  chl (ug/L)  pres (dbar)  sal (PSU)  temp (C)\n",
       "0 2015-11-10 01:42:00      22.307        3.712     33.199     19.95\n",
       "1 2015-11-10 01:35:00      22.311        3.588     33.201     19.94\n",
       "2 2015-11-10 01:29:00      22.305        3.541     33.200     19.95\n",
       "3 2015-11-10 01:23:00      22.323        3.463     33.200     19.95\n",
       "4 2015-11-10 01:17:00      22.316        3.471     33.199     19.95"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio4.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           datetime64[ns]\n",
       "chl (ug/L)            float64\n",
       "pres (dbar)           float64\n",
       "sal (PSU)             float64\n",
       "temp (C)              float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio4.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for maximal control, this can be combined with dtype\n",
    "df_sio5 = pd.read_csv('../data/scripps_pier_20151110.csv', index_col=None,\n",
    "                      parse_dates=['Date'], dtype={'temp (C)': str}).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>chl (ug/L)</th>\n",
       "      <th>pres (dbar)</th>\n",
       "      <th>sal (PSU)</th>\n",
       "      <th>temp (C)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2015-11-10 01:42:00</td>\n",
       "      <td>22.307</td>\n",
       "      <td>3.712</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2015-11-10 01:35:00</td>\n",
       "      <td>22.311</td>\n",
       "      <td>3.588</td>\n",
       "      <td>33.201</td>\n",
       "      <td>19.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015-11-10 01:29:00</td>\n",
       "      <td>22.305</td>\n",
       "      <td>3.541</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2015-11-10 01:23:00</td>\n",
       "      <td>22.323</td>\n",
       "      <td>3.463</td>\n",
       "      <td>33.200</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2015-11-10 01:17:00</td>\n",
       "      <td>22.316</td>\n",
       "      <td>3.471</td>\n",
       "      <td>33.199</td>\n",
       "      <td>19.95</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Date  chl (ug/L)  pres (dbar)  sal (PSU) temp (C)\n",
       "0 2015-11-10 01:42:00      22.307        3.712     33.199    19.95\n",
       "1 2015-11-10 01:35:00      22.311        3.588     33.201    19.94\n",
       "2 2015-11-10 01:29:00      22.305        3.541     33.200    19.95\n",
       "3 2015-11-10 01:23:00      22.323        3.463     33.200    19.95\n",
       "4 2015-11-10 01:17:00      22.316        3.471     33.199    19.95"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio5.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Date           datetime64[ns]\n",
       "chl (ug/L)            float64\n",
       "pres (dbar)           float64\n",
       "sal (PSU)             float64\n",
       "temp (C)               object\n",
       "dtype: object"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_sio5.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}