fall20/432_Week_03_InfoVis_R.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "432-Week-03-InfoVis-R.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOP6N0jTJLpvpxYgR6FzBW5",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "ir",
      "display_name": "R"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/cs432-websci-fall20/assignments/blob/master/432_Week_03_InfoVis_R.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3jPMf4rKiusJ",
        "colab_type": "text"
      },
      "source": [
        "# CS 432/532 R Tutorial\n",
        "\n",
        "These commands can also be run locally using RStudio.\n",
        "\n",
        "There are a ton of references available for R and since it's popular, you can pretty much search for whatever you want and find something close."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NL85DyaUv_vs",
        "colab_type": "text"
      },
      "source": [
        "## R Basics"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZaZCqRxcrbPo",
        "colab_type": "text"
      },
      "source": [
        "Using R as a calculator:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QOThdcJIrdYO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "1 / 200 * 30\n",
        "(59 + 73 + 2) / 3\n",
        "sin(pi/2)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IDdqZzbGrqVF",
        "colab_type": "text"
      },
      "source": [
        "**Important:** Variable assignment is done with `<-`, not with `=`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UWoOSe-Qrwjr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x <- 3*4\n",
        "x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VkpUrrcJwGyM",
        "colab_type": "text"
      },
      "source": [
        "Vectors are important in R.  Access is similar to Python notation for lists (but must include both ends of the range)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c0s5LReEwIyJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "b1<-c(1,2,3,4,5,6)\n",
        "b1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QrNzZ1URwMxo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "b2<-1:10\n",
        "b2\n",
        "b2[4]\n",
        "b2[0:4]\n",
        "b2[4:6]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lAeqZf2Bwqmz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "b2[:4]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7QZTcfDxkFH",
        "colab_type": "text"
      },
      "source": [
        "## Working with Data\n",
        "\n",
        "`read.csv()`\n",
        " * reads data as table and converts it to data frame\n",
        " * can read local file or file on the web\n",
        " * specify separator with `sep=`\n",
        " * specify if there's column titles with `header`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zXxq1h7IzRmx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mydata1 <- read.csv(\"https://www.cs.odu.edu/~mweigle/courses/cs795/mklein-IntroR/lecture/input1.dat\", sep=\"\\t\", header=F)\n",
        "mydata1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j1pNJmr6zd49",
        "colab_type": "text"
      },
      "source": [
        "You can also read in a datafile that has column names (use `header=T`). Columns are then addressable by `var$colname`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j8M8bEqpzy0k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mydata2<-read.table(\"https://www.cs.odu.edu/~mweigle/courses/cs795/mklein-IntroR/lecture/input2.dat\",sep=\"\\t\",header=T)\n",
        "mydata2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yO6NHRmu2M57",
        "colab_type": "text"
      },
      "source": [
        "Accessing columns -- using names or using column number\n",
        "\n",
        "`var[row,col]`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E6VbrJ1ez5_V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mydata2$id\n",
        "mydata2$val_A\n",
        "mydata2$val_B\n",
        "mydata2[,2]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RDbP1Q7z2ORv",
        "colab_type": "text"
      },
      "source": [
        "Accessing an individual cell (row 3 in column val_A) -- two different ways"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OKBOm4qs2Jag",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mydata2$val_A[3]\n",
        "mydata2[3,2]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KANQIsQD2ZDI",
        "colab_type": "text"
      },
      "source": [
        "Accessing a row (row 3)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T64dzlwr2LkP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mydata2[3,]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QBOBPDuG0JNo",
        "colab_type": "text"
      },
      "source": [
        "What if you have missing data and want to perform some mathematical functions?  \n",
        "\n",
        "Use `NA` in place of the missing data and use `na.rm=T` options in functions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZJi7yDrr0cg8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "d1<-c(1:3,rep(NA,4),8:10)\n",
        "d1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HTL6A9hl0jkg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mean(d1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jies3MhM0ePT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mean(d1,na.rm=T)\n",
        "median(d1,na.rm=T)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xczHK8LMyF6r",
        "colab_type": "text"
      },
      "source": [
        "## Plotting with ggplot2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wHCDAmic0jdK",
        "colab_type": "text"
      },
      "source": [
        "First we load the ggplot2 library. \n",
        "\n",
        "Then we're going to use some of the datasets included with the ggplot2 library.  You can see the list of these with the `data(package=\"ggplot2\")` command. For each dataset, use `?datasetName` to get more information about the dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WLqrr04g3Gvf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "library(ggplot2)\n",
        "theme_set(theme_bw())  # selects a black and white theme\n",
        "library(scales)        # allows us to format axes labels with commas\n",
        "options(scipen=999)    # prevent using scientific notation"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8oVYCjTtNCST",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data(package=\"ggplot2\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u3niOxn3NEES",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "?midwest"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2PZdx3QH3ShI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data(\"midwest\", package = \"ggplot2\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iQIOeo8O3T0n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "head(midwest, 6)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z3EwRtP5Bw1e",
        "colab_type": "text"
      },
      "source": [
        "### Scatterplot"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvV8zwaT5Pnv",
        "colab_type": "text"
      },
      "source": [
        "Here's a basic scatterplot, showing the percentage of college educated (mapped to the y-axis) vs. the total population (mapped to the x-axis) in each county in Ohio (state==OH).\n",
        "\n",
        "Notice the notation used to subset the dataset inside the `ggplot()` function.  `midwest$state` refers to the `state` column in the midwest dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2NNW4b-Z4BCi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gg <- ggplot(midwest[midwest$state==\"OH\",], aes(x=poptotal, y=percollege)) + \n",
        "  geom_point()\n",
        "plot(gg)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xw7fWVy1IzLz",
        "colab_type": "text"
      },
      "source": [
        "Because we saved the basic chart in a variable, we can reuse it and add options.  We add `scale_x_continuous(label=comma)` so that the numbers are comma-formatted and specify the chart labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F_sc51uNI4Lo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gg + \n",
        "  scale_x_continuous(label=comma) + \n",
        "  labs(y=\"% College Educated\", \n",
        "       x=\"Population\", \n",
        "       title = \"Ohio Counties\",\n",
        "       caption=\"Source: midwest\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LMr3DnSqBsDB",
        "colab_type": "text"
      },
      "source": [
        "### Bar Chart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IXz3s0p27ZZp",
        "colab_type": "text"
      },
      "source": [
        "For our bar chart, let's look at the total population in each state.  We can sum `poptotal` in each county."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cLwLsDgf7YjJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "state_pop <- aggregate(midwest$poptotal, by=list(midwest$state), FUN=sum)\n",
        "state_pop"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kEwym-Ap-rBN",
        "colab_type": "text"
      },
      "source": [
        "Then we'll change the column labels to something reasonable."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WtllCOkT8IHg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "colnames(state_pop) <- c(\"state\", \"poptotal\") \n",
        "state_pop"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iT7SSI7r-vJP",
        "colab_type": "text"
      },
      "source": [
        "We use `geom_bar()` to create a bar chart.  We want the chart to directly show the values in the table, so we use `stat=\"identity\"`.  \n",
        "\n",
        "We can also specify the width of the bars and also the color for all the bars. Note that this is not mapping an attribute to color, but coloring all bars regardless of value, since it's outside of the `aes()` function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "B_R5BUVH8XpB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ggplot(state_pop, aes(x=state, y=poptotal)) + \n",
        "  geom_bar(stat=\"identity\", width=.5, fill=\"tomato3\") + \n",
        "  scale_y_continuous(label=comma) + \n",
        "  labs(y=\"Total Population\", caption=\"source: midwest\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rt_CP8cxHdFY",
        "colab_type": "text"
      },
      "source": [
        "Next, let's sort this by population in descending order."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9rD0WysNK8XS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ordered <- state_pop[order(-state_pop$poptotal),]\n",
        "ordered$state <- factor(ordered$state, levels = ordered$state)  # to retain the order in plot\n",
        "ordered"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FhpTSzeVLko9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ggplot(ordered, aes(x=state, y=poptotal)) + \n",
        "  geom_bar(stat=\"identity\", width=.5, fill=\"tomato3\") + \n",
        "  scale_y_continuous(label=comma) + \n",
        "  labs(y=\"Total Population\", caption=\"source: mpg\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wtUa_B9AHlg1",
        "colab_type": "text"
      },
      "source": [
        "And then turn it sideways for a horizontal bar chart by just switching the x and y axes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "odcyzPbjMN4u",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ggplot(ordered, aes(y=state, x=poptotal)) + \n",
        "  geom_bar(stat=\"identity\", width=.5, fill=\"tomato3\") + \n",
        "  scale_x_continuous(label=comma) + \n",
        "  labs(x=\"Total Population\", caption=\"source: midwest\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VWA6kEaABvjQ",
        "colab_type": "text"
      },
      "source": [
        "### Line Chart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fh-3A1-y_QTR",
        "colab_type": "text"
      },
      "source": [
        "For a line chart, we need an ordered (but not necessarily quantitative) value for the x-axis.  Usually this is something like time.  So we need to load a different dataset.  The economics dataset is a time series dataset with various economic indicators from 1967-2015."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sYCNPNko_43X",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "?economics"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G0ZwGlfCA35a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "head(economics)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wZ_MbQvOCbB",
        "colab_type": "text"
      },
      "source": [
        "We use `geom_line()` to create the line chart."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PNM54A4N_yzP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gg <- ggplot(economics, aes(x=date)) + \n",
        "  geom_line(aes(y=unemploy)) + \n",
        "  scale_y_continuous(label=comma) + \n",
        "  labs(y=\"Number unemployed (thousands)\",\n",
        "    caption=\"Source: Economics\")\n",
        "plot(gg)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrDBL-XrOQJQ",
        "colab_type": "text"
      },
      "source": [
        "We can add points to the line chart just by adding a `geom_point()`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "es7Nq5M2OJ5-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "gg + geom_point(aes(y=unemploy))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fnReU5K2B4D9",
        "colab_type": "text"
      },
      "source": [
        "### Scatterplot Matrix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UpuX4CUfD9eT",
        "colab_type": "text"
      },
      "source": [
        "The simplest way to plot a scatterplot matrix is with the standard R function `pairs()` (not a part of ggplot2).  Instead of plotting the data in the diagonals, it lists the attribute name.\n",
        "\n",
        "The example here goes back to the midwest dataset, selects only columns 4:6, and sets the point mark to a filled dot (19)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KgebzZ8IDMPg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pairs(midwest[,4:6], pch=19)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JNVw0qWvB7Sg",
        "colab_type": "text"
      },
      "source": [
        "### Histogram"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-6h-A6VQGv0w",
        "colab_type": "text"
      },
      "source": [
        "For the histogram, we show the distribution of population per county.  Note that we're limiting this to counties that have less than 1 M people (in particular, Cook County, IL includes Chicago and has > 5 M people), so that skews the histogram.\n",
        "\n",
        "To create the histogram, we use the `geom_histogram()` function.  `binwidth` sets the size of each histogram bin (this one is set to 10,000)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PxalsgwfEyPu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ggplot(midwest[midwest$poptotal < 1000000,], aes(poptotal)) + \n",
        "  geom_histogram(binwidth=10000) +\n",
        "  scale_x_continuous(label=comma) +\n",
        "  labs(x = \"Population per County\", caption=\"source: midwest\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eM60n1ATB8rC",
        "colab_type": "text"
      },
      "source": [
        "### Boxplot"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sac3Y-iwIZyu",
        "colab_type": "text"
      },
      "source": [
        "We're again looking at the total population by county in the midwest (and again, only for counties with < 1M people).  This time, we'll use boxplots (`geom_boxplot()`) and create a separate boxplot for each state.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AsSxSQNEHYEh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ggplot(midwest[midwest$poptotal < 1000000,], aes(x=state, y=poptotal)) + \n",
        "  scale_y_continuous(label=comma) +\n",
        "  geom_boxplot()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fl8w-CRpB9rE",
        "colab_type": "text"
      },
      "source": [
        "### Empirical CDF (ECDF)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kY0LN8TII6UK",
        "colab_type": "text"
      },
      "source": [
        "For the empirical CDF (CDF), we just have to take the histogram code and replace `geom_histogram()` with `stat_ecdf()`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R2VRVLJlIsEp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ggplot(midwest[midwest$poptotal < 1000000,], aes(poptotal)) + \n",
        "  stat_ecdf() +\n",
        "  scale_x_continuous(label=comma) +\n",
        "  labs(x = \"Population per County\", y = \"Cumulative Distribution\", caption=\"source: midwest\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F94zbq3hPcWF",
        "colab_type": "text"
      },
      "source": [
        "## Output R Datasets to CSV\n",
        "\n",
        "So that we can use the same data in our Python examples, let's output the `midwest` and `economics` datasets to CSV files that we can load into our Python notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tg_BAQLwP0R-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "write.csv(midwest,\"midwest.csv\", row.names = TRUE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b_OZ0IM-P7xZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "write.csv(economics,\"economics.csv\", row.names = TRUE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "InBKp8qHQAKK",
        "colab_type": "text"
      },
      "source": [
        "Now we can refresh our file list and download these files locally (or save them to Google Drive) so that we load them in our Python notebook later."
      ]
    }
  ]
}