06-upload-download.Rmd

```{r, include = FALSE}
ottrpal::set_knitr_image_path()
```

# File Upload and Download

```{r, echo=FALSE, fig.alt='We are on the sixth step of the pathway.', out.width = '80%', fig.align = 'center'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.gff2211b72f_1_252")
```

Exchanging files with the cluster is very important. You can imagine scenarios where:

- You want to download log files or output files
- You want to upload a custom `.sh` script file that you wrote on your laptop
- You want to upload other files

In this course, upload and download of files is performed using [Cyberduck](https://cyberduck.io/){target="_blank"}. Cyberduck is a tool that lets us connect to the cluster securely, browse files, and transfer files securely.

<div class = "warning">
If you are working with sensitive data (such as data with [PHI](https://www.hhs.gov/answers/hipaa/what-is-phi/index.html){target="_blank"} that requires [HIPAA compliance](https://www.hhs.gov/hipaa/index.html){target="_blank"}, you need to be extra cautious about transferring your data to the cluster. Your home directory is not an appropriate storage option for such data. Make sure you consider any stipulations in your data use agreements.
</div>

## Download Cyberduck

Download the latest version of Cyberduck [here](https://cyberduck.io/download/){target="_blank"}.

```{r, echo=FALSE, fig.alt='Downloads page for Cyberduck.', out.width = '100%', fig.align = 'center'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g16e400fe3e8_0_0")
```

<div class = "warning">
Note that the version of Cyberduck in the Software Center or Self Service might not be current, causing compatibility issues with some operating systems.
</div>

## Create Connection

Launch Cyberduck and click on "Open Connection".

- From the dropdown menu, select "SFTP (SSH File Transfer Protocol)"
- For Server, type "rhino.fhcrc.org"
- Fill in your HutchNetID for Username and fill in your password

Click "Connect"

```{r, echo=FALSE, fig.alt='Screenshot of Cyberduck “Open Connection” configuration.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g1579ffd7b01_12_28")
```

Click "Allow". You can also check the box to indicate "Always".

```{r, echo=FALSE, fig.alt='You can select allow when Cyberduck asks about an Unknown Fingerprint.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g1579ffd7b01_12_33")
```

You should see your script file "`01.sh`" and the log file.

```{r, echo=FALSE, fig.alt='Files, including the script we previously ran, can be accessed via Cyberduck.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g1579ffd7b01_12_37")
```

## Download and Edit the Script

- Right click on "`01.sh`" and select "Download"
- You will see a "Transfers" prompt open, and the `01.sh` file should now appear in your Downloads folder
- Open the `01.sh` in your Downloads folder

```{r, echo=FALSE, fig.alt='Screenshot of the script opened on the users local laptop.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g1579ffd7b01_12_41")
```

Edit the message to include your name and save the file. Rename the file `01-name.sh`.

```{r, echo=FALSE, fig.alt='Screenshot of the edited and renamed script file contents. The contents of the script now reads "Hello, Ava!".', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g1579ffd7b01_12_45")
```

## Upload the New Script

From your Downloads folder, simply drag the file to Cyberduck.
  
```{r, echo=FALSE, fig.alt='Screenshot of local and Cyberduck files, depicting the dragging that transfers the file over.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g1579ffd7b01_12_49")
```

You should now see the new script among your cluster files.

```{r, echo=FALSE, fig.alt='The new file is now listed in Cyberduck and is therefore on the cluster.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.g15cf3fa00a4_0_6")
```

## Run the New Script

Return to your Terminal. Submit a job with your new script by running the following. When you type `ls` you should see a new log file! 

```
sbatch 01-name.sh
```

```{r, echo=FALSE, fig.alt='After running the newly edited script, there is a new log file present on the cluster.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.gff2211b72f_1_0")
```

<div class = "notice">
The job numbers included in log file names generally increase in number. The greater the number, the more recently the job was run.
</div>

Use the `cat` command to inspect the log. Make sure you replace `[your-number-here]` to match your file. The message should show the new text that you added!

```
cat slurm-[your-number-here].out
```

```{r, echo=FALSE, fig.alt='The message "Hello, Ava!" has been printed to the Terminal using the cat command.', out.width = '100%'}
ottrpal::include_slide("https://docs.google.com/presentation/d/1BQxrVYdKZTbpCaF-i_q9w7s9x034lEXpQZDU-Sl09cs/edit#slide=id.gff2211b72f_1_6")
```