IMD_D2_M3_Data_Wrangling.Rmd

---
title: "Day 2 Module 3"
author: "Lauren Pandori & John Paul Schmit"
date: "12/15/2021"
output: html_document
---
#### Dplyr functions {.tabset}

<details open><summary class='drop'> Data Wrangling</summary>

In this section, we'll introduce functions in the dplyr package. This package is one of the packages that make up the tidyverse, so you won't have to install and call another package. The dplyr package is a group of functions that help to manipulate and summarize data. 
<https://www.rdocumentation.org/packages/dplyr/versions/0.7.8>

You can think of these functions like verbs that accomplish tasks, and are strung together by the pipe operator (%>% or |>).

We'll cover the following functions: 

  1. select() - selects or removes columns
  2. rename() - renames columns
  3. arrange() - arranges a data.frame by column values
  4. filter() - filters data for observations that meet specific criteria
  5. mutate() - adds new column(s) to a data.frame
  6. pull() %>% table() - isolates and summarizes by a column 
  7. group_by() %>% summarize() - summary calculations by groups/factor levels and returns data for each group
  8. group_by() %>% mutate() - summary calculations by groups/factor levels and returns data for each observation

The data we'll use for modules 3-5 is from the Tidy Tuesday community on GitHub. Each week on Tuesday, the community posts data for exploratory data analysis and visualization. You can add data from this page to your RStudio environment by copying + pasting the "Get the data!" code chunk into your script and running it in the console.

<https://github.com/rfordatascience/tidytuesday/tree/master/data/2019/2019-09-17>

The data we'll use is in csv format, so we'll need to call the readr package, as well as the tidyverse and janitor for this lesson (if you've already called tidyverse and janitor, don't run those lines).

```{r echo=TRUE, message = FALSE, warning = FALSE}
library(readr)
library(janitor)
library(tidyverse)
```

Next, we'll load the data from the R for Data Science Tidy Tuesday GitHub page. There are three lines which will load three data.frames.

  1. park_visits - which contains data on visits to US National Parks
  2. state_pop - which contains information on populations of US states
  3. gas_price - which contains information on gas prices over space and time
  
For now, we'll focus on the park_visits data.frame, and will circle back to the other two later. There are descriptions of each column for each data.frame on the Tidy Tuesday website linked above if you'd like more information. 

```{r echo=TRUE, message = FALSE, warning = FALSE}
park_visits <- read.csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-09-17/national_parks.csv")
state_pop <- read.csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-09-17/state_pop.csv")
gas_price <- read.csv("https://raw.githubusercontent.com/rfordatascience/tidytuesday/master/data/2019/2019-09-17/gas_price.csv")
```
</details>
<br>
<details open><summary class='drop'> Working with dplyr</summary>

Dplyr is a package that is part of the tidyverse, which is used with data.frames. It makes tasks like filtering and summarizing data easier. Many of the functions within the package are "verb-like" so they are easy to understand and remember. 

</details>
<br>
<details open><summary class='drop'> dplyr - select() and rename()</summary>

select() - keeps the columns you indicate and drops the rest. Since select() puts the columns in the order you type them, you can also use select() to reorder columns.

rename() - keeps all the columns, but renames the ones you indicate

Since we won't be using geographical information, other than state, in our module, we'll select some columns from the park_visits data to use today.

Note that, the name to the left of the arrow is the resulting data.frame, the information to the right that's connected by pipes are the steps to get to the result. I normally name the result something different than the original data.frame so that the original data aren't affected. However, since we won't need the geographical information later on, I will name this one "park_visits". 

```{r eval = TRUE, include = TRUE, message = FALSE, warning = FALSE}
# let's see the column names in park_visits
colnames(park_visits)

# we can also look at the first few rows by using head()

head(park_visits)

# you can either select for the columns you want
park_visits1 <- park_visits %>% select(year, parkname, unit_code, state, visitors)

# or against the columns you don't want, by using the minus sign before the column name
park_visits2 <- park_visits %>% select(-gnis_id, -geometry, -metadata, -number_of_records, -region, -unit_name, -unit_type)

# you can check if you have the same result by using colnames()
colnames(park_visits1)
colnames(park_visits2)

# to clear up our environment, we'll get rid of park_visits1 and 2 using remove()
remove(park_visits1, park_visits2)

# since we won't need all of the columns in park_visits, we'll change that file (note that park_visits is on both the left and right sides of the pipe)
park_visits <- park_visits %>% select(year, parkname, unit_code, state, visitors)
```

Taking a look at the park_visits data, there are some column name inconsistencies. For example, some are in snake case (unit_code), but some are not (parkname). To rename a column, use rename(). The structure of this argument is as follows: 

rename(dataset, new_name = old_name)
OR
dataset %>% rename(new_name = old_name)

```{r eval = TRUE, include = TRUE, message = FALSE, warning = FALSE}
# rename parkname column to make it snake case
park_visits <- park_visits %>% rename(park_name = parkname)

# we can check if we've done it correctly by looking at the column names
colnames(park_visits)
```

</details>
<br>
<details open><summary class='drop'> dplyr- filter()</summary>

filter() - filters out rows from a data.frame based on values in columns
can combine criteria using & (and), | (or), ! (not)

If you'd like to get visitation data for just parks in the state of Washington (WA), you can use the filter function. The structure is as follows: 
```{r eval=FALSE, include=TRUE}
wa_park_visits <- filter(
                        # name of data.frame
                        park_visits, 
                        # condition
                        state == 'WA'
                        )

# head() shows the first few rows of data
head(wa_park_visits)

# you can use the unique() to check you've done the filtering correctly
unique(wa_park_visits$state)
```

Now that you've got parks in the state of Washington, perhaps you'd like to know which parks in WA had over 5 million total visitors. This will require more intensive filtering. We'll have to filter for parks where state == 'WA' AND year == 'Total' AND visitation >= 5,000,000. 

```{r eval=FALSE, include=TRUE}
wa_park_visits <- park_visits %>%
                  # filter for parks in WA
                  filter(state == 'WA' &
                  # filter for total visitation
                  year == 'Total' & 
                  # filter for over 5 million visitors
                  visitors > 5000000)

# head() shows the first few rows of data
head(wa_park_visits)
```

You also might be wondering why we use "==" instead of "=". In R, the single equal sign "=" is used to assign something. For example, if you want to make some variable equal to a number, you could tell R "x = 5", and it would remember that x is equal to 5. To evaluate something as TRUE/FALSE, you use "==". If you type "5 == 3" into the console, it will return the answer "FALSE". The filtering argument uses this TRUE/FALSE evaluation - filtering for rows where the stated condition - such as year == 'Total" - is TRUE.

</details>
<br>
<details open><summary class='drop'> dplyr- challenge 1</summary>

Time for the first challenge. Using the park_visits data, create a new data.frame containing total visitation for a state of your choice. Note that some "states" aren't states. For example, VI refers to the Virgin Islands. How can you check your work?

</details>
<br>
<details><summary class='drop'> dplyr- solution 1</summary>

```{r eval=FALSE, include=TRUE}
pr_park_visits <- park_visits %>%
                  # filter for parks in Puerto Rico
                  filter(state == 'PR' &
                  # filter for total visitation
                  year == 'Total')

# head() shows the first few rows of data
head(pr_park_visits)
# San Juan National Historic Site (Puerto Rico) had > 55 million visitors!

# remove result from environment
remove(pr_park_visits)
```

</details>
<br>
<details open><summary class='drop'> dplyr - arrange()</summary>

arrange() reorders rows. You can think of this like "sort" in Excel. While this is generally not needed in R, it can be helpful for exploring your data, and when working with time series data and lagged values. 

As an example, we'll re-visit total park visitation in WA, sorting by the number of visitors. 

```{r eval = TRUE, include=TRUE}
wa_park_visits <- park_visits %>%
                  # filter for parks in WA
                  filter(state == 'WA' &
                  # filter for total visitation
                  year == 'Total' & 
                  # filter for over 5 million visitors
                  visitors > 5000000) %>%
                  # arrange by visitation - default is ascending order
                  arrange(visitors)

# look at result (df is short so head() not needed)
wa_park_visits

# to arrange by descending order, we can use desc()
wa_park_visits2 <- wa_park_visits %>%
                  arrange(desc(visitors))

# look at result 
wa_park_visits2

# remove one result
remove(wa_park_visits2)
```

</details>
<br>
<details open><summary class='drop'> dplyr - pull()</summary>

pull() takes data from a column and returns in in vector form

pull() %>%  table() is a handy way to see a summary of categorical data (e.g. how may observations for each state there are in your data)

```{r eval=FALSE, include=TRUE}
# two ways to get the number of observations per state

# option 1 - use pull() to isolate a column and table() to get # of observations
state_n <- park_visits %>%
           # pull the state column only
           pull(state) %>%
           # get # of observations by state in table output
           table()

# option 2 - use group_by() to isolate column(s) and tally() to get # of obs
state_n2 <- park_visits %>%
           # group by state
           group_by(state) %>%
           # get tally of observations by state in output
           tally()

# remove created objects
remove(state_n, state_n2)
```

</details>
<br>
<details open><summary class='drop'> dplyr - mutate()</summary>

mutate() makes new columns by doing calculations on old columns

As an example, let's calculate the total visitation for parks with over 5 million total visitors in WA in millions, by dividing the number of total visitors by 1,000,000.

```{r eval = TRUE, include = TRUE}
wa_park_visits_millions <- wa_park_visits %>%
                           mutate(visitors_mil = visitors/1000000)

wa_park_visits_millions
```

</details>
<br>
<details open><summary class='drop'> dplyr - group_by() and mutate()</summary>

group_by() indicates that data is in groups, so you can do calculations separately for each group

group_by() %>% mutate() will add a column and return the same number of rows as the original data

This is helpful when you want to get group summary information without sacrificing the original values. For example, if we want to know what percent of the total visitation of a park occurs in each year, and add this to the original data. 

```{r eval = TRUE, include = TRUE}
park_visits2 <- park_visits %>%
                # remove the rows with Totals per park
                filter(year != "Total")  %>% 
                # do calculations by park
                group_by(park_name) %>%
                # add visit_percent column
                # visits fore each year divided by the sum of total visits for each park
                # the na.rm = TRUE means that NA values are not included in calculations
                # round(2) is rounds result to 2 decimal places for easier reading
                mutate(visit_percent = (100*visitors/sum(visitors, na.rm = TRUE)) %>%
                         round(2)) 

# take a look at the result
head(park_visits2)

# remove from environment
remove(park_visits2)
```

</details>
<br>
<details open><summary class='drop'> dplyr - group_by() and summarize()</summary>

group_by() and  summarize() (or summarise()) also work with grouped data. Here you calculate a summary for each group. Output data frame will have the columns that
define the group and the summary data, will have one row for each group.

I find this combination most helpful for calculating means and standard deviations for data. You can also find minimum and maximum values. We'll filter for total visitation numbers for each park, then calculate a visitation summary for each state. 

```{r eval = TRUE, include = TRUE}
state_summary <- park_visits %>%
                # filter for total visitation numbers (to avoid double counts)
                filter(year == 'Total') %>%
                # do calculations by state
                group_by(state) %>%
                # calculate summaries
                summarize(
                  # mean number of total visitors across parks in each state
                  mean_visit = mean(visitors, na.rm = TRUE),
                  # sd of total visitors across parks in each state
                  sd_visit = sd(visitors, na.rm = TRUE),
                  # min total visitors across parks in each state
                  min_visit = min(visitors, na.rm = TRUE),
                  # max total visitors across parks in each state
                  max_visit = max(visitors, na.rm = TRUE),
                  # get number of park units w data in each state
                  n_parks = n()
                )

# take a look at the result
head(state_summary)

# if you want to continue analyzing this summarized data, you must first ungroup()
# for example:
state_summary <- state_summary %>%
                 ungroup() %>%
                 mutate(dif = max_visit - min_visit)

# remove from environment
remove(state_summary)
```

</details>
<br>
<details open><summary class='drop'> dplyr - challenge 2</summary>

For this second challenge, you'll start with the park_visits data. Calculate the average visitation divided by the number of parks for each state. The final data.frame should be arranged in descending order by a numeric column of your choice. 

</details>
<br>
<details open><summary class='drop'> dplyr - solution 2</summary>

```{r eval = TRUE, include = TRUE}
challenge2 <- park_visits %>%
                # filter for total visitation numbers (to avoid double counts)
                filter(year == 'Total') %>%
                # do calculations by state
                group_by(state) %>%
                # calculate summaries
                summarize(
                  # mean number of total visitors across parks in each state
                  mean_visit = mean(visitors, na.rm = TRUE),
                  # get number of park units w data in each state
                  n_parks = n()
                ) %>%
                # ungroup to do another calculation
                ungroup() %>%
                # calculate visit/n
                mutate(visit_per_park = mean_visit/n_parks) %>%
                # select relevant columns
                select(state, visit_per_park) %>%
                # arrange in descending order
                arrange(desc(visit_per_park))
                

# take a look at the result
head(challenge2)

# remove from environment
remove(challenge2)
```
</details>
<br>