02-starting-with-data.Rmd~

---
layout: topic
title: Starting with data
author: Data Carpentry contributors
minutes: 20
---


------------

> ## Learning Objectives
>
> * load external data (CSV files) in memory using the survey table
>  (`Ecoli_metadata.csv`) as an example
> * explore the structure and the content of the data in R
> * understand what are factors and how to manipulate them

------------

# Looking at Metadata

```{r, echo=FALSE, purl=TRUE}
# Looking at metadata
```

We are studying a population of Escherichia coli (designated Ara-3), which were propagated for more than 40,000 generations in a glucose-limited minimal medium. This medium was supplemented with citrate which E. coli cannot metabolize in the aerobic conditions of the experiment. Sequencing of the populations at regular time points reveals that spontaneous citrate-using mutants (Cit+) appeared at around 31,000 generations. This metadata describes information on the Ara-3 clones and the columns represent:

| Column           | Description                        |
|------------------|------------------------------------|
| sample           | clone name                         |
| generation       | generation when sample frozen      |
| clade            | based on parsimony-based tree      |
| strain           | ancestral strain                   |
| cit              | citrate-using mutant status        |
| run              | Sequence read archive sample ID    |
| genome_size      | size in Mbp                        |



The metadata file required for this lesson can be downlaoded by clicking on this [link](./data/Ecoli_metadata.csv)

- First, make sure you are in the correct working directory by typing `getwd()`.
- Second, create a new directory within this working directory called `data`
- Third, move the downloaded file into this directory

You are now ready to load the data. We are going to use the R function  `read.csv()` to load the data file into memory (as a `data.frame`):

```{r, eval=TRUE,  purl=FALSE}
metadata <- read.csv('data/Ecoli_metadata.csv')
```

This statement doesn't produce any output because assignment doesn't display
anything. If we want to check that our data has been loaded, we can print the
variable's value: `metadata`

Alternatively, wrapping an assignment in parentheses will perform the assignment
and display it at the same time.

```{r, eval = TRUE, purl = FALSE}
(metadata <- read.csv('data/Ecoli_metadata.csv'))
```

Wow... that was a lot of output. At least it means the data loaded properly. Let's check the top (the first 6 lines) of this `data.frame` using the function `head()`:

```{r, results='show', purl=FALSE}
head(metadata)
```

Let's now check the __str__ucture of this `data.frame` in more details with the
function `str()`:

```{r, purl=FALSE}
str(metadata)
```

# Inspecting `data.frame` objects

We already saw how the functions `head()` and `str()` can be useful to check the
content and the structure of a `data.frame`. Here is a non-exhaustive list of
functions to get a sense of the content/structure of the data.

* Size:
    * `dim()` - returns a vector with the number of rows in the first element, and
    the number of columns as the second element (the __dim__ensions of the object)
    * `nrow()` - returns the number of rows
    * `ncol()` - returns the number of columns
* Content:
    * `head()` - shows the first 6 rows
    * `tail()` - shows the last 6 rows
* Names:
    * `names()` - returns the column names (synonym of `colnames()` for `data.frame`
	objects)
   * `rownames()` - returns the row names
* Summary:
   * `str()` - structure of the object and information about the class, length and
	content of  each column
   * `summary()` - summary statistics for each column

Note: most of these functions are "generic", they can be used on other types of
objects besides `data.frame`.


### Challenge

Based on the give table of functions to asses data structure, can you answer the following questions?

* What is the class of the object `surveys`?
* How many rows and how many columns are in this object?
* How many citrate+ mutants have been recorded in this population?

```{r, echo=FALSE, purl=TRUE}
## Exercise
## Based on the output of `str(surveys)`, can you answer the following questions?
## * What is the class of the object `surveys`?
## * How many rows and how many columns are in this object?
## * How many species have been recorded during these surveys?
```

As you can see, many of the columns in our data frame are of a special class called
`factor`. Before we learn more about the `data.frame` class, we are going to
talk about factors. They are very useful but not necessarily intuitive, and
therefore require some attention.


## Factors

```{r, echo=FALSE, purl=TRUE}
### Factors
```

Factors are used to represent categorical data. Factors can be ordered or
unordered and are an important class for statistical analysis and for plotting.

Factors are stored as integers, and have labels associated with these unique
integers. While factors look (and often behave) like character vectors, they are
actually integers under the hood, and you need to be careful when treating them
like strings.

Once created, factors can only contain a pre-defined set values, known as
*levels*. By default, R always sorts *levels* in alphabetical order. For
instance, if you have a factor with 2 levels:

```{r, purl=TRUE}
sex <- factor(c("male", "female", "female", "male"))
```

R will assign `1` to the level `"female"` and `2` to the level `"male"` (because
`f` comes before `m`, even though the first element in this vector is
`"male"`). You can check this by using the function `levels()`, and check the
number of levels using `nlevels()`:

```{r, purl=FALSE, eval=FALSE}
levels(sex)
nlevels(sex)
```

Sometimes, the order of the factors does not matter, other times you might want
to specify the order because it is meaningful (e.g., "low", "medium", "high") or
it is required by particular type of analysis. Additionally, specifying the
order of the levels allows to compare levels:

```{r, purl=FALSE, error=TRUE, eval=FALSE}
expression <- factor(c("low", "high", "medium", "high", "low", "medium", "high"))
levels(expression)
expression <- factor(expression, levels=c("low", "medium", "high"))
levels(expression)
min(expression) ## doesn't work
expression <- factor(expression, levels=c("low", "medium", "high"), ordered=TRUE)
levels(expression)
min(expression) ## works!
```

In R's memory, these factors are represented by numbers (1, 2, 3). They are
better than using simple integer labels because factors are self describing:
`"low"`, `"medium"`, and `"high"`" is more descriptive than `1`, `2`, `3`. Which
is low?  You wouldn't be able to tell with just integer data. Factors have this
information built in. It is particularly helpful when there are many levels
(like the species in our example data set).

### Converting factors

If you need to convert a factor to a character vector, simply use
`as.character(x)`.

Converting a factor to a numeric vector is however a little trickier, and you
have to go via a character vector. Compare:

```{r, purl=TRUE, eval=FALSE}
f <- factor(c(1, 5, 10, 2))
as.numeric(f)               ## wrong! and there is no warning...
as.numeric(as.character(f)) ## works...
as.numeric(levels(f))[f]    ## The recommended way.
```

### Challenge

The function `table()` tabulates observations and can be used to create
bar plots quickly. For instance:

```{r wrong-order, results='show', purl=TRUE}
## Question: How can you recreate this plot but by having "control"
## being listed last instead of first?
exprmt <- factor(c("treat1", "treat2", "treat1", "treat3", "treat1", "control",
                   "treat1", "treat2", "treat3"))
table(exprmt)
barplot(table(exprmt))
```

<!---
```{r correct-order, purl=FALSE}
exprmt <- factor(exprmt, levels=c("treat1", "treat2", "treat3", "control"))
barplot(table(exprmt))
```
--->