diff --git a/README.md b/README.md index 7df8198..f2e9152 100644 --- a/README.md +++ b/README.md @@ -20,10 +20,12 @@ This lesson teaches: | # | Episode | Time | Question(s) | |--:|:---------|:----:|:------------| -| 1 | [Neuroimaging Fundamentals][episode01] | 30 | What are the common neuroimaging modalities? | -| 2 | [Anatomy of a NIfTI][episode02] | 30 | How is MRI data organized in a NIfTI file? | -| 3 | [Brain Imaging Data Structure][episode03] | 30 | How can I organize my study? | -| 4 | [Open MRI Datasets][episode04] | 30 | How can I download and query an MRI dataset? | +| 1 | [Before we start][episode01] | 30 | What is Python and why should I learn it? | +| 2 | [From the scanner to our computer][episode02] | 30 | What are the main MRI modalities?
What’s the first step necessary to start working with MRI data? | +| 3 | [Anatomy of a NIfTI][episode03] | 25 | How are MRI data represented digitally? | +| 4 | [Data organization with BIDS][episode04] | 45 | | +| 5 | [Exploring open MRI datasets][episode05] | 45 | How does standardizing neuroimaging data ease the data exploration process | +| 6 | [BIDS derivatives][episode06] | 45 | | ## Contributing @@ -68,10 +70,12 @@ To cite this lesson, please consult with [CITATION](CITATION) [slack_channel_url]: https://swcarpentry.slack.com/messages/CCJBHKCHZ [binder_svg]: https://mybinder.org/badge_logo.svg [binder_url]: https://mybinder.org/v2/gh/carpentries-incubator/SDC-BIDS-IntroMRI/gh-pages?filepath=code%2F -[episode01]: https://github.com/conp-pcno-training/SDC-BIDS-IntroMRI/blob/gh-pages/files/neuroimaging_analysis_at_scale.pptx -[episode02]: https://github.com/conp-pcno-training/SDC-BIDS-IntroMRI/blob/gh-pages/code/02-anatomy-of-nifti/02-anatomy-of-nifti.ipynb -[episode03]: https://github.com/conp-pcno-training/SDC-BIDS-IntroMRI/blob/gh-pages/files/neuroimaging_analysis_at_scale.pptx -[episode04]: https://github.com/conp-pcno-training/SDC-BIDS-IntroMRI/blob/gh-pages/code/04-open-mri-datasets/04-open-mri-datasets.ipynb +[episode01]: https://carpentries-incubator.github.io/SDC-BIDS-IntroMRI/before-we-start/index.html +[episode02]: https://carpentries-incubator.github.io/SDC-BIDS-IntroMRI/scanner-to-computer/index.html +[episode03]: https://carpentries-incubator.github.io/SDC-BIDS-IntroMRI/anatomy-of-nifti/index.html +[episode04]: https://carpentries-incubator.github.io/SDC-BIDS-IntroMRI/data-organization-bids/index.html +[episode05]: https://carpentries-incubator.github.io/SDC-BIDS-IntroMRI/open-mri-datasets/index.html +[episode06]: https://carpentries-incubator.github.io/SDC-BIDS-IntroMRI/bids-derivatives/index.html [lesson-example]: https://carpentries.github.io/lesson-example [link_issues]: https://github.com/conp-pcno-training/SDC-BIDS-IntroMRI/issues [michael_joseph]: https://github.com/josephmje diff --git a/_episodes_md/00-before-we-start.md b/_episodes_md/00-before-we-start.md deleted file mode 100644 index 850ff70..0000000 --- a/_episodes_md/00-before-we-start.md +++ /dev/null @@ -1,114 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -This overview have been adapted from the [Data Carpentry Python Lessons with Ecological Data](https://datacarpentry.org/python-ecology-lesson/). - -An introduction to working with magnetic resonance imaging (MRI) data in Python. - -## About the Lesson - -This lesson teaches: - -- a (re?) introduction to MR nomenclature - with BIDS -- how neuroimaging data is stored -- "converting" your data to BIDS -- BIDS apps -- queueing up neuroimaging pipelines - -## Episodes - -| # | Episode | Time | Question(s) | -|--:|:---------|:----:|:------------| -| 1 | Neuroimaging Fundamentals | 30 | What are the common neuroimaging modalities? | -| 2 | Anatomy of a NIfTI | 30 | How is MRI data organized in a NIfTI file? | -| 3 | Brain Imaging Data Structure | 30 | How can I organize my study? | -| 4 | Open MRI Datasets | 30 | How can I download and query an MRI dataset? | - -## Why Python? - -Python is rapidly becoming the standard language for data analysis, visualization and automated workflow building. -It is a free and open-source software that is relatively easy to pick up by new programmers and is available on multiple operating systems. -In addition, with Python packages such as `Jupyter` one can keep an interactive code journal of analysis - this is what we'll be using in the workshop. -Using Jupyter notebooks allows you to keep a record of all the steps in your analysis, enabling reproducibility and ease of code sharing. - -Another advantage of Python is that it is maintained by a large user-base. -Anyone can easily make their own Python packages for others to use. -Therefore, there exists a *very* large codebase for you to take advantage of for your neuroimaging analysis; from basic statistical analysis, to brain visualization tools, to advanced machine learning and multivariate methods! - -## Knowing your way around Anaconda and Jupyter - -The [Anaconda][anaconda] distribution of Python includes a lot of its popular packages, such as the IPython console, and Jupyter Notebook. -Have a quick look around the Anaconda Navigator. -You can launch programs from the Navigator or use the command line. - -The [Jupyter notebook](https://jupyter.org) is an open-source web application that allows you to create -and share documents that allow one to create documents that combine code, graphs, and narrative text. -If you have not worked with Jupyter notebooks before, you might start with the [tutorial from Jupyter.org](https://jupyter.org/try) called “Try Classic Notebook”. - -Anaconda also comes with a package manager called [conda](https://conda.io/docs/), which makes it easy to install and update additional packages. - -## Organizing your working directory - -Using a consistent folder structure across your projects will help you keep things organized, and will also make it easy to find/file things in the future. -This can be especially helpful when you have multiple projects. -In general, you may wish to create separate directories for your scripts, data, and documents. - -This lesson uses the following folders to keep things organized: - -- **`data/`**: Use this folder to store your raw data. For the sake of transparency and provenance, -you should always keep a copy of your **raw data**. If you need to cleanup data, do it -programmatically (_i.e._with scripts) and make sure to separate cleaned up data from the raw data. -For example, you can store raw data in files `./data/raw/` and clean data in `./data/clean/`. - -- **`code/`**: Use this folder to store your (Python) scripts for data cleaning, analysis, and -plotting that you use in this particular project. This folder contains all of the Jupyter notebooks used in the lesson. - -- **`files/`**: Use this folder to store any miscellaneous outlines, drafts, and other text. This folder contains a set of Powerpoint slides that will be used for parts of the lesson. - -## How to learn more after the workshop? - -The material we cover during this workshop will give you an initial taste of how you can use Python -to analyze MRI data for your own research. -However, you will need to learn more to do advanced operations such as working with specific MRI modalities, cleaning your dataset, using statistical methods, or creating beautiful graphics. -The best way to become proficient and efficient at Python, as with any other tool, is to use it to address your actual research questions. -As a beginner, it can feel daunting to have to write a script from scratch, and given that many people make their code available online, modifying existing code to suit your purpose might make it easier for you to get started. - -## Seeking help - -- check under the *Help* menu -- type `help()` -- type `?object` or `help(object)` to get information about an object -- [Python documentation][python-docs] -- [NiBabel documentation][nibabel-docs] - -Finally, a generic Google or internet search "Python task" will often either send you to the -appropriate module documentation or a helpful forum where someone else has already asked your -question. - -I am stuck... I get an error message that I don’t understand. -Start by googling the error message. However, this doesn’t always work very well, because often, -package developers rely on the error catching provided by Python. You end up with general error -messages that might not be very helpful to diagnose a problem (e.g. "subscript out of bounds"). If -the message is very generic, you might also include the name of the function or package you’re using -in your query. - -You can also check [Stack Overflow][stack-overflow]. Most questions have already been answered, but the challenge is to use the right words in the search to find the answers. - -Another helpful resource specific to the neuroimaging community is [NeuroStars](https://neurostars.org/). - -[anaconda]: https://www.anaconda.com -[nibabel-docs]: https://nipy.org/nibabel/manual.html#manual -[python-docs]: https://www.python.org/doc -[stack-overflow]: https://stackoverflow.com diff --git a/_episodes_md/01-scanner-to-computer.md b/_episodes_md/01-scanner-to-computer.md deleted file mode 100644 index dc7c8da..0000000 --- a/_episodes_md/01-scanner-to-computer.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# From the scanner to our computer - -SMALL INTRO BLURB ABOUT DIFFERENT MR SCANS - -## Neuroimaging file formats - -|Format Name | File Extension | Origin | -|---|---|---| -| DICOM | none | ACR/NEMA Consortium | -| Analyze | .img/.hdr | Analyze Software, Mayo Clinic | -| NIfTI | .nii | Neuroimaging Informatics Technology Initiative | -| MINC | .mnc | Montreal Neurological Institute | -| NRRD | .nrrd | | - -From the MRI scanner, images are initially collected in the DICOM format and can be converted to these other formats to make working with the data easier. - -Drawing - -Let's download some example DICOM data to see what it looks like. -This data was generously shared publicly by the [Princeton Handbook for Reproducible Neuroimaging](https://brainhack-princeton.github.io/handbook/). - -```bash - -wget https://zenodo.org/record/3677090/files/0219191_mystudy-0219-1114.tar.gz -O data/0219191_mystudy-0219-1114.tar.gz -mkdir -p ../data/dicom_examples -tar -xvzf ../data/0219191_mystudy-0219-1114.tar.gz -C ../data/dicom_examples -gzip -d ../data/dicom_examples/0219191_mystudy-0219-1114/dcm/*dcm.gz -rm ../data/0219191_mystudy-0219-1114.tar.gz -``` - -NIfTI is one of the most ubiquitous file formats for storing neuroimaging data. -If you're interested in learning more about NIfTI images, we highly recommend [this blog post about the NIfTI format](http://brainder.org/2012/09/23/the-nifti-file-format/). -We can convert our DICOM data to NIfTI using [dcm2niix](https://github.com/rordenlab/dcm2niix). - -We can learn how to run `dcm2niix` by taking a look at its help menu. - -```bash - -dcm2niix -help -``` - -### Converting DICOM to NIfTI - - -**EXERCISE:** Convert the Princeton DICOM data to NIfTI - - -```python - -``` - -```bash solution2="hidden" - -mkdir -p ../data/dicom_examples/nii -dcm2niix \ - -z y \ - -o ../data/dicom_examples/nii \ - ../data/dicom_examples/0219191_mystudy-0219-1114/dcm -``` diff --git a/_episodes_md/02-anatomy-of-nifti.md b/_episodes_md/02-anatomy-of-nifti.md deleted file mode 100644 index ec8b0e7..0000000 --- a/_episodes_md/02-anatomy-of-nifti.md +++ /dev/null @@ -1,302 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# Anatomy of a NIfTI - ---- - -In the last lesson, we introduced the NIfTI. NIfTI is one of the most ubiquitous file formats for storing neuroimaging data. We'll cover a few details to get started working with them. If you're interested in learning more about NIfTI images, we highly recommend [this blog post about the NIfTI format](http://brainder.org/2012/09/23/the-nifti-file-format/). - -## Reading NIfTI images - -[NiBabel](http://nipy.org/nibabel/) is a Python package for reading and writing neuroimaging data. To learn more about how NiBabel handles NIfTIs, check out the [Working with NIfTI images](http://nipy.org/nibabel/nifti_images.html) page of the NiBabel documentation. - -```python -import nibabel as nib -``` - -First, use the `load()` function to create a NiBabel image object from a NIfTI file. We'll load in an example T1w image from the zip file we just downloaded. - -```python -t1_img = nib.load("../data/dicom_examples/nii/dcm_anat_ses-01_T1w_20190219111436_5.nii.gz") -``` - -Loading in a NIfTI file with `nibabel` gives us a special type of data object which encodes all the information in the file. Each bit of information is called an **attribute** in Python's terminology. To see all of these attributes, type `t1_img.` and Tab. -There are three main attributes that we'll discuss today: - -### 1. [Header](http://nipy.org/nibabel/nibabel_images.html#the-image-header): contains metadata about the image, such as image dimensions, data type, etc. - -```python tags=[] -t1_hdr = t1_img.header -print(t1_hdr) -``` - -`t1_hdr` is a Python **dictionary**. Dictionaries are containers that hold pairs of objects - keys and values. Let's take a look at all of the keys. -Similar to `t1_img` in which attributes can be accessed by typing `t1_img.` and hitting Tab, you can do the same with `t1_hdr`. In particular, we'll be using a **method** belonging to `t1_hdr` that will allow you to view the keys associated with it. - -```python -t1_hdr.keys() -``` - -Notice that **methods** require you to include `()` at the end of them whereas **attributes** do not. -The key difference between a method and an attribute is: - -- Attributes are *stored values* kept within an object -- Methods are *processes* that we can run using the object. Usually a method takes attributes, performs an operation on them, then returns it for you to use. - -When you type in `t1_img.` followed by Tab, you'll see that attributes are highlighted in orange and methods highlighted in blue . - - -The output above is a list of **keys** you can use from `t1_hdr` to access **values**. We can access the value stored by a given key by typing: - -```python -t1_hdr[''] -``` - - -**EXERCISE:** Extract the value of `pixdim` from `t1_hdr` - -```python solution2="hidden" -t1_hdr['pixdim'] -``` - -```python - -``` - -In addition to metadata embedded in the NIfTI header, the T1w image also has a corresponding JSON file with additional scan acquisition details. Using the JSON file to store this information is a concept added by BIDS (which we'll cover in the next lesson) to log the important bits of information that traditionally get excluded from the NIfTI header. - -Let's take a look at it below: - -```python -import json - -with open("../data/dicom_examples/nii/dcm_anat_ses-01_T1w_20190219111436_5.json", "r") as f: - t1_metadata = json.load(f) - -t1_metadata -``` - -The additional metadata are also in the form of a Python **dictionary**. - -**EXERCISE:** Extract the value of `SliceThickness` from `t1_metadata` similar to how you did previously for `t1_hdr`. - -```python solution2="hidden" -t1_metadata["SliceThickness"] -``` - -```python - -``` - -### 2. Data: PROVIDE DESCRIPTION - -As you've seen above, the header contains useful information that gives us information about the properties (metadata) associated with the MR data we've loaded in. Now we'll move in to loading the actual *image data itself*. We can achieve this by using the *method* called `t1_img.get_fdata()`. - -```python -t1_data = t1_img.get_fdata() -t1_data -``` - -What type of data is this exactly? We can determine this by calling the `type()` function on `t1_data`. - -```python -type(t1_data) -``` - -The data is a multidimensional **array** representing the image data. In Python, an array is used to store lists of numerical data into something like a table. - -**EXERCISE:** Let's check out some *attributes* of the array. How can we see the number of dimensions in the `t1_data` array? What about the how big each dimension is (shape)? Once again, all of the attributes of the array can be seen by typing `t1_data.` and Tab. - -```python solution2="hidden" -t1_data.ndim -``` - -```python - -``` - -`t1_data` contains 3 dimensions. You can think of the data as a 3D version of a picture (more accurately, a volume). - -Drawing - -While typical 2D pictures are made out of squares called **pixels**, a 3D MR image is made up of 3D cubes called **voxels**. - -Drawing - -```python -t1_data.shape -``` - -The 3 numbers given here represent the number of values *along a respective dimension (x,y,z)*. This brain was scanned in 192 slices with a resolution of 256 x 256 voxels per slice. That means there are: - -$$x * y * z = value$$ -$$ 192 * 256 * 256 = 12582912$$ voxels in total! - -Let's see the type of data inside of the array. - -```python -t1_data.dtype -``` - -This tells us that each element in the array (or voxel) is a floating-point number. -The data type of an image controls the range of possible intensities. As the number of possible values increases, the amount of space the image takes up in memory also increases. - -```python tags=[] -import numpy as np - -print(np.min(t1_data)) -print(np.max(t1_data)) -``` - -For our data, the range of intensity values goes from 0 (black) to more positive digits (whiter). - -How do we examine what value a particular voxel is? We can inspect the value of a voxel by selecting an **index** as follows: - -~~~python -data[x,y,z] -~~~ - -So for example we can inspect a voxel at coordinates (10,20,3) by doing the following: - -```python -t1_data[9, 19, 2] -``` - -**NOTE**: Python uses **zero-based indexing**. The first item in the array is item 0. The second item is item 1, the third is item 2, etc. - -This yields a single value representing the intensity of the signal at a particular voxel! Next we'll see how to not just pull one voxel but a slice or an *array* of voxels for visualization and analysis! - -## Working with image data - -Slicing does exactly what it seems to imply. Giving our 3D volume, we pull out a 2D slice of our data. Here's an example of slicing from left to right (**sagittal slicing**): - - - -This gif is a series of 2D images or **slices** moving from left to right. - -Let's pull the 50th slice in the x axis. - -```python -x_slice = t1_data[49, :, :] -``` - -This is similar to the indexing we did before to pull out a single voxel. However, instead of providing a value for each axis, the `:` indicates that we want to grab *all* values from that particular axis. - -**EXERCISE:** Now try selecting the 80th slice from the y axis. - -```python solution2="hidden" -y_slice = t1_data[:, 79, :] -``` - -**EXERCISE:** Finally try grabbing the 100th slice from the z axis. - -```python solution2="hidden" -z_slice = t1_data[:, :, 99] -``` - -We've been slicing and dicing brain images but we have no idea what they look like! In the next section we'll show you how you can visualize brain slices! - -## Visualizing - -We previously inspected the signal intensity of the voxel at coordinates (10,20,3). Let's see what out data looks like when we slice it at this location. We've already indexed the data at each x, y, and z axis. Let's use `matplotlib`, Python's standard plotting library. - -```python -import matplotlib.pyplot as plt -%matplotlib inline - -slices = [x_slice, y_slice, z_slice] - -fig, axes = plt.subplots(1, len(slices)) -for i, slice in enumerate(slices): - axes[i].imshow(slice.T, cmap="gray", origin="lower") -``` - -```python -from nilearn import plotting - -plotting.view_img(t1_img, black_bg=True, colorbar=False, cmap="gray", symmetric_cmap=False, vmin=0) -``` - -An upgrade from `matplotlib` is `nilearn`, which provides several advanced plotting features for neuroimaging data. - -```python -from nilearn import plotting - -plotting.view_img(t1_img, black_bg=True, colorbar=False, cmap="gray", symmetric_cmap=False, vmin=0) -``` - -Now, we're going to step away from discussing our data and talk about the final important attribute of a NIfTI. - -### 3. [Affine](http://nipy.org/nibabel/coordinate_systems.html): tells the position of the image array data in a *reference space* - -The final important piece of metadata associated with an image file is the **affine matrix**. Below is the affine matrix for our data. - -```python -t1_affine = t1_img.affine -t1_affine -``` - -To explain this concept, recall that we referred to coordinates in our data as (x,y,z) coordinates such that: - -- x is the first dimension of `t1_data` -- y is the second dimension of `t1_data` -- z is the third dimension of `t1_data` - -Although this tells us how to access our data in terms of voxels in a 3D volume, it doesn't tell us much about the actual dimensions in our data (centimetres, right or left, up or down, back or front). The affine matrix allows us to translate between *voxel coordinates (x,y,z)* and *world space coordinates* in (left/right,bottom/top,back/front). An important thing to note is that in reality in which order you have: - -- left/right -- bottom/top -- back/front - -Depends on how you've constructed the affine matrix, but for the data we're dealing with it always refers to: - -- Right -- Anterior -- Superior - -Applying the affine matrix (`t1_affine`) is done through using a *linear map* (matrix multiplication) on voxel coordinates (defined in `t1_data`). - -Drawing - -The concept of an affine matrix may seem confusing at first but an example might help gain an intuition: - -Suppose we have two voxels located at the the following coordinates: - -$$(15,2,90)$$ -$$(64,100,2)$$ - -And we wanted to know what the distances between these two voxels are in terms of real world distances (millimetres). This information cannot be derived from using voxel coordinates so we turn to the **affine matrix**. - -Now, the affine matrix we'll be using happens to be encoded in **RAS**. That means once we apply the matrix our coordinates are as follows: - -$$(\text{Right},\text{Anterior},\text{Superior})$$ - -So increasing a coordinate value in the first dimension corresponds to moving to the right of the person being scanned. - -Applying our affine matrix yields the following coordinates: - -$$(90.23,0.2,2.15)$$ -$$(10.25,30.5,9.2)$$ - -This means that: - -- Voxel 1 is $90.23-10.25= 79.98$ in the R axis. Positive values mean move right -- Voxel 1 is $0.2-30.5= -30.3$ in the A axis. Negative values mean move posterior -- Voxel 1 is $2.15-9.2= -7.05$ in the S axis. Negatve values mean move inferior - ---- - -This covers the basics of how NIfTI data and metadata are stored and organized in the context of Python. In the next segment we'll talk a bit about an increasingly important component of MR data analysis - data organization. This is a key component to reproducible analysis and so we'll spend a bit of time here. diff --git a/_episodes_md/02-anatomy-of-nifti_solutions.md b/_episodes_md/02-anatomy-of-nifti_solutions.md deleted file mode 100644 index e2624f9..0000000 --- a/_episodes_md/02-anatomy-of-nifti_solutions.md +++ /dev/null @@ -1,292 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# Anatomy of a NIfTI - ---- - -In the last lesson, we introduced the NIfTI. NIfTI is one of the most ubiquitous file formats for storing neuroimaging data. We'll cover a few details to get started working with them. If you're interested in learning more about NIfTI images, we highly recommend [this blog post about the NIfTI format](http://brainder.org/2012/09/23/the-nifti-file-format/). - - -## Reading NIfTI images - -[NiBabel](http://nipy.org/nibabel/) is a Python package for reading and writing neuroimaging data. To learn more about how NiBabel handles NIfTIs, check out the [Working with NIfTI images](http://nipy.org/nibabel/nifti_images.html) page of the NiBabel documentation. - -```python -import nibabel as nib -``` - -First, use the `load()` function to create a NiBabel image object from a NIfTI file. We'll load in an example T1w image from the zip file we just downloaded. - -```python -t1_img = nib.load("../data/dicom_examples/nii/dcm_anat_ses-01_T1w_20190219111436_5.nii.gz") -``` - -Loading in a NIfTI file with `nibabel` gives us a special type of data object which encodes all the information in the file. Each bit of information is called an **attribute** in Python's terminology. To see all of these attributes, type `t1_img.` and Tab. -There are three main attributes that we'll discuss today: - - -### 1. [Header](http://nipy.org/nibabel/nibabel_images.html#the-image-header): contains metadata about the image, such as image dimensions, data type, etc. - -```python tags=[] -t1_hdr = t1_img.header -print(t1_hdr) -``` - -`t1_hdr` is a Python **dictionary**. Dictionaries are containers that hold pairs of objects - keys and values. Let's take a look at all of the keys. -Similar to `t1_img` in which attributes can be accessed by typing `t1_img.` and hitting Tab, you can do the same with `t1_hdr`. In particular, we'll be using a **method** belonging to `t1_hdr` that will allow you to view the keys associated with it. - -```python -t1_hdr.keys() -``` - -Notice that **methods** require you to include `()` at the end of them whereas **attributes** do not. -The key difference between a method and an attribute is: -- Attributes are *stored values* kept within an object -- Methods are *processes* that we can run using the object. Usually a method takes attributes, performs an operation on them, then returns it for you to use. - -When you type in `t1_img.` followed by Tab, you'll see that attributes are highlighted in orange and methods highlighted in blue . - - -The output above is a list of **keys** you can use from `t1_hdr` to access **values**. We can access the value stored by a given key by typing: - -```python -t1_hdr[''] -``` - - -**EXERCISE:** Extract the value of `pixdim` from `t1_hdr` - -```python -t1_hdr['pixdim'] -``` - -In addition to metadata embedded in the NIfTI header, the T1w image also has a corresponding JSON file with additional scan acquisition details. Using the JSON file to store this information is a concept added by BIDS (which we'll cover in the next lesson) to log the important bits of information that traditionally get excluded from the NIfTI header. - -Let's take a look at it below: - -```python -import json - -with open("../data/dicom_examples/nii/dcm_anat_ses-01_T1w_20190219111436_5.json", "r") as f: - t1_metadata = json.load(f) - -t1_metadata -``` - -The additional metadata are also in the form of a Python **dictionary**. - - -**EXERCISE:** Extract the value of `SliceThickness` from `t1_metadata` similar to how you did previously for `t1_hdr`. - -```python -t1_metadata["SliceThickness"] -``` - -### 2. Data -As you've seen above, the header contains useful information that gives us information about the properties (metadata) associated with the MR data we've loaded in. Now we'll move in to loading the actual *image data itself*. We can achieve this by using the *method* called `t1_img.get_fdata()`. - -```python -t1_data = t1_img.get_fdata() -t1_data -``` - -What type of data is this exactly? We can determine this by calling the `type()` function on `t1_data`. - -```python -type(t1_data) -``` - -The data is a multidimensional **array** representing the image data. In Python, an array is used to store lists of numerical data into something like a table. - - -**EXERCISE:** Let's check out some *attributes* of the array. How can we see the number of dimensions in the `t1_data` array? What about the how big each dimension is (shape)? Once again, all of the attributes of the array can be seen by typing `t1_data.` and Tab. - -```python -t1_data.ndim -``` - -`t1_data` contains 3 dimensions. You can think of the data as a 3D version of a picture (more accurately, a volume). - -Drawing - -While typical 2D pictures are made out of squares called **pixels**, a 3D MR image is made up of 3D cubes called **voxels**. - -Drawing - -```python -t1_data.shape -``` - -The 3 numbers given here represent the number of values *along a respective dimension (x,y,z)*. This brain was scanned in 192 slices with a resolution of 256 x 256 voxels per slice. That means there are: - -$$x * y * z = value$$ -$$ 192 * 256 * 256 = 12582912$$ voxels in total! - - -Let's see the type of data inside of the array. - -```python -t1_data.dtype -``` - -This tells us that each element in the array (or voxel) is a floating-point number. -The data type of an image controls the range of possible intensities. As the number of possible values increases, the amount of space the image takes up in memory also increases. - -```python tags=[] -import numpy as np - -print(np.min(t1_data)) -print(np.max(t1_data)) -``` - -For our data, the range of intensity values goes from 0 (black) to more positive digits (whiter). - - -How do we examine what value a particular voxel is? We can inspect the value of a voxel by selecting an **index** as follows: - -~~~python -data[x,y,z] -~~~ - - -So for example we can inspect a voxel at coordinates (10,20,3) by doing the following: - -```python -t1_data[9, 19, 2] -``` - -**NOTE**: Python uses **zero-based indexing**. The first item in the array is item 0. The second item is item 1, the third is item 2, etc. - -This yields a single value representing the intensity of the signal at a particular voxel! Next we'll see how to not just pull one voxel but a slice or an *array* of voxels for visualization and analysis! - - -## Working with image data - -Slicing does exactly what it seems to imply. Giving our 3D volume, we pull out a 2D slice of our data. Here's an example of slicing from left to right (**sagittal slicing**): - - - -This gif is a series of 2D images or **slices** moving from left to right. - - -Let's pull the 50th slice in the x axis. - -```python -x_slice = t1_data[49, :, :] -``` - -This is similar to the indexing we did before to pull out a single voxel. However, instead of providing a value for each axis, the `:` indicates that we want to grab *all* values from that particular axis. - - -**EXERCISE:** Now try selecting the 80th slice from the y axis. - -```python -y_slice = t1_data[:, 79, :] -``` - -**EXERCISE:** Finally try grabbing the 100th slice from the z axis. - -```python -z_slice = t1_data[:, :, 99] -``` - -We've been slicing and dicing brain images but we have no idea what they look like! In the next section we'll show you how you can visualize brain slices! - - -## Visualizing - -We previously inspected the signal intensity of the voxel at coordinates (10,20,3). Let's see what out data looks like when we slice it at this location. We've already indexed the data at each x, y, and z axis. Let's use `matplotlib`. - -```python -import matplotlib.pyplot as plt -%matplotlib inline - -slices = [x_slice, y_slice, z_slice] - -fig, axes = plt.subplots(1, len(slices)) -for i, slice in enumerate(slices): - axes[i].imshow(slice.T, cmap="gray", origin="lower") -``` - -```python -from nilearn import plotting - -plotting.view_img(t1_img, black_bg=True, colorbar=False, cmap="gray", symmetric_cmap=False, vmin=0) -``` - -Now, we're going to step away from discussing our data and talk about the final important attribute of a NIfTI. - -### 3. [Affine](http://nipy.org/nibabel/coordinate_systems.html): tells the position of the image array data in a *reference space* - -The final important piece of metadata associated with an image file is the **affine matrix**. Below is the affine matrix for our data. - -```python -t1_affine = t1_img.affine -t1_affine -``` - -To explain this concept, recall that we referred to coordinates in our data as (x,y,z) coordinates such that: - -- x is the first dimension of `t1_data` -- y is the second dimension of `t1_data` -- z is the third dimension of `t1_data` - -Although this tells us how to access our data in terms of voxels in a 3D volume, it doesn't tell us much about the actual dimensions in our data (centimetres, right or left, up or down, back or front). The affine matrix allows us to translate between *voxel coordinates (x,y,z)* and *world space coordinates* in (left/right,bottom/top,back/front). An important thing to note is that in reality in which order you have: - -- left/right -- bottom/top -- back/front - -Depends on how you've constructed the affine matrix, but for the data we're dealing with it always refers to: - -- Right -- Anterior -- Superior - -Applying the affine matrix (`t1_affine`) is done through using a *linear map* (matrix multiplication) on voxel coordinates (defined in `t1_data`). - -Drawing - -The concept of an affine matrix may seem confusing at first but an example might help gain an intuition: - -Suppose we have two voxels located at the the following coordinates: - -$$(15,2,90)$$ -$$(64,100,2)$$ - -And we wanted to know what the distances between these two voxels are in terms of real world distances (millimetres). This information cannot be derived from using voxel coordinates so we turn to the **affine matrix**. - -Now, the affine matrix we'll be using happens to be encoded in **RAS**. That means once we apply the matrix our coordinates are as follows: - -$$(\text{Right},\text{Anterior},\text{Superior})$$ - -So increasing a coordinate value in the first dimension corresponds to moving to the right of the person being scanned. - -Applying our affine matrix yields the following coordinates: - -$$(90.23,0.2,2.15)$$ -$$(10.25,30.5,9.2)$$ - -This means that: - -- Voxel 1 is $90.23-10.25= 79.98$ in the R axis. Positive values mean move right -- Voxel 1 is $0.2-30.5= -30.3$ in the A axis. Negative values mean move posterior -- Voxel 1 is $2.15-9.2= -7.05$ in the S axis. Negatve values mean move inferior - - ---- - -This covers the basics of how NIfTI data and metadata are stored and organized in the context of Python. In the next segment we'll talk a bit about an increasingly important component of MR data analysis - data organization. This is a key component to reproducible analysis and so we'll spend a bit of time here. diff --git a/_episodes_md/03-data-organization-bids.md b/_episodes_md/03-data-organization-bids.md deleted file mode 100644 index a35da86..0000000 --- a/_episodes_md/03-data-organization-bids.md +++ /dev/null @@ -1,219 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# Data organization with BIDS - ---- - -## When NIfTI isn't enough - -So far, we've only worked with a single T1w scan from a single subject. However, neuroimaging research requires working with **different scan modalities** across **several subjects** (sometimes even **multiple timepoints**)! - - -**EXERCISE:** Let's take a look at the rest of our NIfTI dataset from the last lesson. Imagine you are a collaborator who's currently gotten your hands on this dataset for an analysis. What are some immediate questions you might have that aren't apparent at a first glance? - -``` -nii -├── dcm_anat_ses-01_T1w_20190219111436_5.json -├── dcm_anat_ses-01_T1w_20190219111436_5.nii.gz -├── dcm_func_ses-01_task-faces_run-01_20190219111436_13.json -├── dcm_func_ses-01_task-faces_run-01_20190219111436_13.nii.gz -├── dcm_func_ses-01_task-sound_run-01_20190219111436_6.json -├── dcm_func_ses-01_task-sound_run-01_20190219111436_6.nii.gz -├── dcm_func_ses-01_task-sound_run-01_20190219111436_7.json -├── dcm_func_ses-01_task-sound_run-01_20190219111436_7.nii.gz -├── dcm_func_ses-01_task-sound_run-01_20190219111436_8.json -├── dcm_func_ses-01_task-sound_run-01_20190219111436_8.nii.gz -├── dcm_func_ses-01_task-story_run-01_20190219111436_9.json -├── dcm_func_ses-01_task-story_run-01_20190219111436_9.nii.gz -├── dcm_func_ses-01_task-story_run-02_20190219111436_10.json -├── dcm_func_ses-01_task-story_run-02_20190219111436_10.nii.gz -├── dcm_func_ses-01_task-story_run-03_20190219111436_11.json -├── dcm_func_ses-01_task-story_run-03_20190219111436_11.nii.gz -├── dcm_func_ses-01_task-story_run-04_20190219111436_12.json -└── dcm_func_ses-01_task-story_run-04_20190219111436_12.nii.gz -``` - - - -- How many subjects were scanned? -- What are the demographics of the participants? -- What is the name of the dataset? -- Who are the authors of the dataset? -- Is there a citation I should use to give credit to the authors? -- Is there a more elaborate description of the fMRI tasks? -- Do any of the fMRI tasks have an event-related design? -- What do the fMRI task stimuli look like? - - -## Introducing BIDS - -The [Brain Imaging Data Structure (BIDS)](https://www.nature.com/articles/sdata201644) is a simple and intuitive way to organize and describe your neuroimaging and behavioural data. Neuroimaging experiments result in complicated data that can be arranged in several different ways. BIDS tackles this problem by suggesting a new standard (based on consensus from multiple researchers across the world) for the arrangement of neuroimaging datasets. Using the same organizational standard for all of your studies will also allow you to easily reuse your scripts and share data and code with other researchers. - -For this lesson, we'll be focusing on organizing MRI data. However, BIDS has since expanded with specifications for [MEG](https://www.nature.com/articles/sdata2018110), [EEG](https://www.nature.com/articles/s41597-019-0104-8), [iEEG](https://www.nature.com/articles/s41597-019-0105-7), [PET](https://journals.sagepub.com/doi/10.1177/0271678X20905433), and [genetics](https://academic.oup.com/gigascience/article/9/10/giaa104/5928221). - -For a more comprehensive overview, check out the [BIDS Starter Kit](https://github.com/bids-standard/bids-starter-kit/wiki). - - -Drawing - - -## Structure - -The current BIDS specification is [documented online](https://bids-specification.readthedocs.io/en/stable/). - - - -``` -dataset/ - rawdata/ - dataset_description.json - participants.tsv - CHANGES - README - task-rest.json - sub-01/ - ses-01/ - anat/ - func/ - sub-02/ - ... - sourcedata/ - ... - derivatives/ - pipeline_1/ - pipeline_2/ - ... - code/ - stimuli/ -``` - - -**EXERCISE**: The Case of Rea Seach - -> Hey Rea, -> Great to see you at the conference last month. Here's the data we talked about for that -new project. We tested 1 male participant with structural and a few of our favorite tasks. -> The data is organized according to modality. -> Looking forward to seeing how this project works out! -> Cole Labo - -Lets try converting the NIfTI data to BIDS manually. - -```bash - -mkdir -p ../data/dicom_examples/bids/sub-01/{anat,func} -``` - -```python -import json -from collections import OrderedDict - -data= OrderedDict() -data['Name'] = 'BIDS Test' -data['BIDSVersion'] = '1.6.0' -data['License'] = '' -data['Authors'] = [''] -data['Acknowledgements'] = '' -data['HowToAcknowledge'] = '' -data['Funding'] = ['','',''] -data['ReferencesAndLinks'] = ['','',''] -data['DatasetDOI'] = '' -dataset_json_name='../data/dicom_examples/bids/dataset_description.json' - -with open(dataset_json_name, 'w') as f: - json.dump(data, f, sort_keys=False, indent=2) -``` - -## Conversion methods - -- [Dcm2Bids](https://github.com/cbedetti/Dcm2Bids) -- [heudiconv](https://github.com/nipy/heudiconv) -- [bidskit](https://github.com/jmtyszka/bidskit) -- [dac2bids](https://github.com/dangom/dac2bids) -- [BIDScoin](https://github.com/Donders-Institute/bidscoin) -- and more! - -```bash - -dcm2bids_scaffold -o ../data/dicom_examples/dcm2bids/ -``` - -```bash - -dcm2bids_helper \ - -d ../data/dicom_examples/0219191_mystudy-0219-1114/dcm \ - -o ../data/dicom_examples/dcm2bids -``` - - -`Dcm2Bids` uses a configuration file in order to match scans with the appropriate BIDS filename. We will create this file together but it can also be revealed by clicking the toggle below. - - - -{ - "descriptions": [ - { - "dataType": "anat", - "modalityLabel": "T1w", - "criteria": { "SeriesDescription": "anat_ses-01_T1w" } - }, - { - "dataType": "func", - "modalityLabel": "bold", - "customLabels": "task-sound", - "criteria": { "SeriesDescription": "func_ses-01_task-sound*" } - }, - { - "dataType": "func", - "modalityLabel": "bold", - "customLabels": "task-story", - "criteria": { "SeriesDescription": "func_ses-01_task-story*" } - }, - { - "dataType": "func", - "modalityLabel": "bold", - "customLabels": "task-faces", - "criteria": { "SeriesDescription": "func_ses-01_task-faces*" } - } - ] -} - - -```bash - -mkdir -p ../data/dicom_examples/dcm2bids/rawdata -dcm2bids \ - -d ../data/dicom_examples/0219191_mystudy-0219-1114/dcm \ - -p 01 -s 01 \ - -c ../data/dicom_examples/dcm2bids/code/bids_config.json \ - -o ../data/dicom_examples/dcm2bids/rawdata -``` - -## BIDS validator - -Can be run [online](https://bids-standard.github.io/bids-validator) - -> Selecting a dataset only performs validation. Files are never uploaded. - -Select the data directory and wait for it to finish validation. - -View errors and warnings. - - ---- - -#### References -1. Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., … Poldrack, R. A. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3(1). https://doi.org/10.1038/sdata.2016.44 diff --git a/_episodes_md/03-data-organization-bids_solutions.md b/_episodes_md/03-data-organization-bids_solutions.md deleted file mode 100644 index 2a2b855..0000000 --- a/_episodes_md/03-data-organization-bids_solutions.md +++ /dev/null @@ -1,216 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# Data organization with BIDS - ---- - -## When NIfTI isn't enough - -So far, we've only worked with a single T1w scan from a single subject. However, neuroimaging research requires working with **different scan modalities** across **several subjects** (sometimes even **multiple timepoints**)! - - -**EXERCISE:** Let's take a look at the rest of our NIfTI dataset from the last lesson. Imagine you are a collaborator who's currently gotten your hands on this dataset for an analysis. What are some immediate questions you might have that aren't apparent at a first glance? - -``` -nii -├── dcm_anat_ses-01_T1w_20190219111436_5.json -├── dcm_anat_ses-01_T1w_20190219111436_5.nii.gz -├── dcm_func_ses-01_task-faces_run-01_20190219111436_13.json -├── dcm_func_ses-01_task-faces_run-01_20190219111436_13.nii.gz -├── dcm_func_ses-01_task-sound_run-01_20190219111436_6.json -├── dcm_func_ses-01_task-sound_run-01_20190219111436_6.nii.gz -├── dcm_func_ses-01_task-sound_run-01_20190219111436_7.json -├── dcm_func_ses-01_task-sound_run-01_20190219111436_7.nii.gz -├── dcm_func_ses-01_task-sound_run-01_20190219111436_8.json -├── dcm_func_ses-01_task-sound_run-01_20190219111436_8.nii.gz -├── dcm_func_ses-01_task-story_run-01_20190219111436_9.json -├── dcm_func_ses-01_task-story_run-01_20190219111436_9.nii.gz -├── dcm_func_ses-01_task-story_run-02_20190219111436_10.json -├── dcm_func_ses-01_task-story_run-02_20190219111436_10.nii.gz -├── dcm_func_ses-01_task-story_run-03_20190219111436_11.json -├── dcm_func_ses-01_task-story_run-03_20190219111436_11.nii.gz -├── dcm_func_ses-01_task-story_run-04_20190219111436_12.json -└── dcm_func_ses-01_task-story_run-04_20190219111436_12.nii.gz -``` - - -- How many subjects were scanned? -- What are the demographics of the participants? -- What is the name of the dataset? -- Who are the authors of the dataset? -- Is there a citation I should use to give credit to the authors? -- Is there a more elaborate description of the fMRI tasks? -- Do any of the fMRI tasks have an event-related design? -- What do the fMRI task stimuli look like? - - -## Introducing BIDS - -The [Brain Imaging Data Structure (BIDS)](https://www.nature.com/articles/sdata201644) is a simple and intuitive way to organize and describe your neuroimaging and behavioural data. Neuroimaging experiments result in complicated data that can be arranged in several different ways. BIDS tackles this problem by suggesting a new standard (based on consensus from multiple researchers across the world) for the arrangement of neuroimaging datasets. Using the same organizational standard for all of your studies will also allow you to easily reuse your scripts and share data and code with other researchers. - -For this lesson, we'll be focusing on organizing MRI data. However, BIDS has since expanded with specifications for [MEG](https://www.nature.com/articles/sdata2018110), [EEG](https://www.nature.com/articles/s41597-019-0104-8), [iEEG](https://www.nature.com/articles/s41597-019-0105-7), [PET](https://journals.sagepub.com/doi/10.1177/0271678X20905433), and [genetics](https://academic.oup.com/gigascience/article/9/10/giaa104/5928221). - -For a more comprehensive overview, check out the [BIDS Starter Kit](https://github.com/bids-standard/bids-starter-kit/wiki). - - -Drawing - - -## Structure - -The current spec https://bids-specification.readthedocs.io/en/stable/ - - - -``` -dataset/ - rawdata/ - dataset_description.json - participants.tsv - CHANGES - README - task-rest.json - sub-01/ - ses-01/ - anat/ - func/ - sub-02/ - ... - sourcedata/ - ... - derivatives/ - pipeline_1/ - pipeline_2/ - ... - code/ - stimuli/ -``` - - -**EXERCISE**: The Case of Rea Seach - -> Hey Rea, -> Great to see you at the conference last month. Here's the data we talked about for that -new project. We tested 1 male participant with structural and a few of our favorite tasks. -> The data is organized according to modality. -> Looking forward to seeing how this project works out! -> Cole Labo - -Lets try converting the NIfTI data to BIDS manually. - -```bash - -mkdir -p ../data/dicom_examples/bids/sub-01/{anat,func} -``` - -```python -import json -from collections import OrderedDict - -data= OrderedDict() -data['Name'] = 'BIDS Test' -data['BIDSVersion'] = '1.6.0' -data['License'] = '' -data['Authors'] = [''] -data['Acknowledgements'] = '' -data['HowToAcknowledge'] = '' -data['Funding'] = ['','',''] -data['ReferencesAndLinks'] = ['','',''] -data['DatasetDOI'] = '' -dataset_json_name='../data/dicom_examples/bids/dataset_description.json' - -with open(dataset_json_name, 'w') as f: - json.dump(data, f, sort_keys=False, indent=2) -``` - -## Conversion methods - -- [Dcm2Bids](https://github.com/cbedetti/Dcm2Bids) -- [heudiconv](https://github.com/nipy/heudiconv) -- [bidskit](https://github.com/jmtyszka/bidskit) -- [dac2bids](https://github.com/dangom/dac2bids) -- [BIDScoin](https://github.com/Donders-Institute/bidscoin) -- and more! - -```bash - -dcm2bids_scaffold -o ../data/dicom_examples/dcm2bids/ -``` - -```bash - -dcm2bids_helper \ - -d ../data/dicom_examples/0219191_mystudy-0219-1114/dcm \ - -o ../data/dicom_examples/dcm2bids -``` - -Create a config file: - -``` -{ - "descriptions": [ - { - "dataType": "anat", - "modalityLabel": "T1w", - "criteria": { "SeriesDescription": "anat_ses-01_T1w" } - }, - { - "dataType": "func", - "modalityLabel": "bold", - "customLabels": "task-sound", - "criteria": { "SeriesDescription": "func_ses-01_task-sound*" } - }, - { - "dataType": "func", - "modalityLabel": "bold", - "customLabels": "task-story", - "criteria": { "SeriesDescription": "func_ses-01_task-story*" } - }, - { - "dataType": "func", - "modalityLabel": "bold", - "customLabels": "task-faces", - "criteria": { "SeriesDescription": "func_ses-01_task-faces*" } - } - ] -} -``` - -```bash - -mkdir -p ../data/dicom_examples/dcm2bids/rawdata -dcm2bids \ - -d ../data/dicom_examples/0219191_mystudy-0219-1114/dcm \ - -p 01 -s 01 \ - -c ../data/dicom_examples/dcm2bids/code/config.json \ - -o ../data/dicom_examples/dcm2bids/rawdata -``` - -## BIDS validator - -Can be run [online](https://bids-standard.github.io/bids-validator) - -> Selecting a dataset only performs validation. Files are never uploaded. - -Select the data directory and wait for it to finish validation. - -View errors and warnings. - - ---- - -#### References -1. Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., … Poldrack, R. A. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3(1). https://doi.org/10.1038/sdata.2016.44 diff --git a/_episodes_md/04-open-mri-datasets.md b/_episodes_md/04-open-mri-datasets.md deleted file mode 100644 index f405e5c..0000000 --- a/_episodes_md/04-open-mri-datasets.md +++ /dev/null @@ -1,267 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# Open MRI datasets - ---- - -In this lesson, we will be using a subset of a publicly available dataset, **ds000030**, from [openneuro.org](https://openneuro.org/datasets/ds000030). All of the datasets on OpenNeuro are already structured according to BIDS. - - -## OpenNeuro - -- client-side BIDS validation -- resumable uploads - - -## Downloading data - -### DataLad - -Datalad is a great tool for "versioning" data. Downloading data using datalad has some advantages over downloading data via other ways: - -1. When you download the data, datalad is keeping track of exactly which version of the data you have. Meaning that if somebody updates/fixes that dataset later you can incorporate those fixes quickly. -2. As datasets are getting bigger, we are getting to the point where all the data may not fit on your computer. Datalad first downloads the "small" data first (instead of the whole thing). So you can see what is there, plan your analysis, and only download the files you need to your local system. -3. `datalad drop` let's you delete the files you are no longer using from your system - while holding on to the critical metadata that would allow you to report on the data, or re-download it to the same spot if you need it later. - -For more information about datalad - check out the [datalad handbook](http://handbook.datalad.org/en/latest/) or [this awesome tutorial video with rabbits!](https://www.youtube.com/watch?v=QsAqnP7TwyY). - -`DataLad` installs the data - which for a dataset means that we get the "small" data (i.e. the text files) and the download instructions for the larger files. We can now navigate the dataset like its a file system and plan our analysis. - -We'll switch to the terminal for this part. - -Navigate to the folder where you'd like to download the dataset. - -```python -!datalad clone https://github.com/OpenNeuroDatasets/ds000030.git ../data/ds000030 -``` - -`datalad get` downloads any of the "large" data (i.e. NIfTI files) to our filesystem according to the filepath or regular expression we use. These files can also be removed from our filesystem using `datalad drop`. - -"Getting" the data is what actually downloads the larger files. - -```python -!datalad get ../data/ds000030/sub-10788 -``` - -"Dropping" the data deletes the larger datafile - while holding onto enough metadata that confidenty get it back if you need it later. - -```python -!datalad drop ../data/ds000030/sub-10788 -``` - -#### if you need to deleted everything and restart... - -It happens. Removing the dataset entirely can be done with `datalad remove`. WARNING: Don't `datalad remove` during the workshop as you will need to redownload the dataset using the steps above. - - -## Exploring a BIDS dataset - -Below is a tree diagram showing the folder structure of single MR session within ds000030. - -```python -!tree ../data/ds000030 -``` - -### the participants.tsv - -The `participants.tsv` file is meant to describe some demographic information on each participant within your study (eg. age, handedness, sex, etc.) Let's take a look at the `participants.tsv` file to see what's been included in this dataset. - -In order to load the data into Python, we'll need to import the `pandas` package. The `pandas` **dataframe** is Python's equivalent to an Excel spreadsheet. - -```python -import pandas as pd -``` - -We'll use the `read_csv()` function. It requires us to specify the name of the file we want to import and the separator that is used to distinguish each column in our file (`\t` since we're working with a `.tsv` file). - -```python -participant_metadata = pd.read_csv('../data/ds000030/participants.tsv', sep='\t') -``` - -In order to get a glimpse of our data, we'll use the `head()` function. By default, `head` prints the first 5 rows of our dataframe. - -```python -participant_metadata.head() -``` - -We can view any number of rows by specifying `n=?` as an argument within `head()`. -If we want to select particular rows within the dataframe, we can use the `loc[]` function and identify the rows we want based on their index label (the numbers in the left-most column). - -```python -participant_metadata.loc[[6, 10, 12]] -``` - -**EXERCISE**: Select the first 5 rows of the dataframe using `loc[]`. - -```python solution2="hidden" -participant_metadata.loc[:4] -``` - -```python - -``` - -**EXERCISE:** How many participants do we have in total? - -```python solution2="hidden" -participant_metadata.shape -``` - -```python - -``` - -There are 2 different methods of selecting columns in a dataframe: -* participant_metadata[`''`] (this is similar to selecting a key in a Python dictionary) -* participant_metadata.`` - - -Another way to see how many participants are in the study is to select the `participant_id` column and use the `count()` function. - -```python -participant_metadata['participant_id'].count() -``` - -**EXERCISE:** Which diagnosis groups are part of the study? -*Hint: use the* `unique()` *function.* - -```python solution2="hidden" -participant_metadata['diagnosis'].unique() -``` - -```python - -``` - -If we want to count the number of participants in each diagnosis group, we can use the `value_counts()` function. - -```python -participant_metadata['diagnosis'].value_counts() -``` - -**EXERCISE:** How many males and females are in the study? How many are in each diagnosis group? - -```python solution2="hidden" -participant_metadata['gender'].value_counts() -``` - -```python - -``` - -```python -participant_metadata.groupby(['diagnosis', 'gender']).size() -``` - -When looking at the participant dataframe, we noticed that there is a column called `ghost_NoGhost`. We should look at the README file that comes with the dataset to find out more about this. - -```python -!cat ../data/ds000030/README -``` - -For this tutorial, we're just going to work with participants that are either CONTROL or SCHZ (`diagnosis`) and have both a T1w (`T1w == 1`) and rest (`rest == 1`) scan. - -**EXERCISE:** Filter `participant_metadata` so that only the above conditions are present. - -```python solution2="hidden" -participant_metadata = participant_metadata[(participant_metadata.diagnosis.isin(['CONTROL', 'SCHZ'])) & - (participant_metadata.T1w == 1) & - (participant_metadata.rest == 1)] -participant_metadata -``` - -```python - -``` - -## Querying a BIDS dataset with pyBIDS - - -[pybids](https://bids-standard.github.io/pybids/) is a Python API for querying, summarizing and manipulating the BIDS folder structure. - -```python -from bids.layout import BIDSLayout -``` - -```python -layout = BIDSLayout("../data/ds000030") -``` - -Indexing a database can take a really long time, especially if you have several subjects, modalities, scan types, etc. `pybids` has an option to save the indexed results to a SQLite database. This database can then be re-used the next time you want to query the same database. - -```python -layout.save("../data/ds000030/.db") -``` - -```python -layout = BIDSLayout("../data/ds000030", database_path = "../data/ds000030/.db") -``` - -The pybids layout object lets you query your BIDS dataset according to a number of parameters by using a `get_*()` method. -We can get a list of the subjects we've downloaded from the dataset. - -```python -layout.get_subjects() -``` - -To get a list of all of the files, just use `get()`. - -```python -layout.get() -``` - -There are many arguments we can use to filter down this list. Any BIDS-defined keyword can be passed on as a constraint. In `pybids`, these keywords are known as **entities**. For a complete list of possibilities: - -```python -layout.entities -``` - -For example, if we only want the file paths of all of our resting state fMRI scans, - -```python -layout.get(datatype="func", suffix="bold", task="rest", extension=[".nii.gz"], return_type="file") -``` - -**EXERCISE**: Retrieve the file paths of any scan where the subject is '10292' or '50081' and the `RepetitionTime` is 2 seconds. - -```python solution2="hidden" -layout.get(subject=["10159", "50081"], RepetitionTime=2, return_type="file") -``` - -```python - -``` - -Let's save the first file from our list of file paths to a variable and pull the metadata from its associated JSON file using the `get_metadata()` function. - -```python -fmri_file = layout.get(subject=["10159", "50081"], RepetitionTime=2, return_type="file")[0] -layout.get_metadata(fmri_file) -``` - -We can even collect the metadata for all of our fmri scans into a list and convert this into a dataframe. - -```python -import pandas as pd - -metadata_list = [] -all_fmri_files = layout.get(datatype="func", suffix="bold", return_type="file", extension=[".nii.gz"]) -for fmri_file in all_fmri_files: - fmri_metadata = layout.get_metadata(fmri_file) - metadata_list.append(fmri_metadata) -df = pd.DataFrame.from_records(metadata_list) -df -``` diff --git a/_episodes_md/04-open-mri-datasets_solutions.md b/_episodes_md/04-open-mri-datasets_solutions.md deleted file mode 100644 index abc6197..0000000 --- a/_episodes_md/04-open-mri-datasets_solutions.md +++ /dev/null @@ -1,241 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -# Open MRI datasets - ---- - -In this lesson, we will be using a subset of a publicly available dataset, **ds000030**, from [openneuro.org](https://openneuro.org/datasets/ds000030). All of the datasets on OpenNeuro are already structured according to BIDS. - - -## OpenNeuro - -- client-side BIDS validation -- resumable uploads - - -## Downloading data - -### DataLad - -Datalad is a great tool for "versioning" data. Downloading data using datalad has some advantages over downloading data via other ways: - -1. When you download the data, datalad is keeping track of exactly which version of the data you have. Meaning that if somebody updates/fixes that dataset later you can incorporate those fixes quickly. -2. As datasets are getting bigger, we are getting to the point where all the data may not fit on your computer. Datalad first downloads the "small" data first (instead of the whole thing). So you can see what is there, plan your analysis, and only download the files you need to your local system. -3. `datalad drop` let's you delete the files you are no longer using from your system - while holding on to the critical metadata that would allow you to report on the data, or re-download it to the same spot if you need it later. - -For more information about datalad - check out the [datalad handbook](http://handbook.datalad.org/en/latest/) or [this awesome tutorial video with rabbits!](https://www.youtube.com/watch?v=QsAqnP7TwyY). - -`DataLad` installs the data - which for a dataset means that we get the "small" data (i.e. the text files) and the download instructions for the larger files. We can now navigate the dataset like its a file system and plan our analysis. - -We'll switch to the terminal for this part. - -Navigate to the folder where you'd like to download the dataset. - -```python -!cd ../data && datalad install https://github.com/OpenNeuroDatasets/ds000030.git -``` - -"Getting" the data is what actually downloads the larger files. - -```python -!datalad get ../data/ds000030/sub-10788 -``` - -"Dropping" the data deletes the larger datafile - while holding onto enough metadata that confidenty get it back if you need it later. - -```python -!datalad drop ../data/ds000030/sub-10788 -``` - -#### if you need to deleted everything and restart... - -It happens. Removing the dataset entirely can be done with `datalad remove`. WARNING: Don't `datalad remove` during the workshop as you will need to redownload the dataset using the steps above. - - -## Exploring a bids dataset - -Below is a tree diagram showing the folder structure of single MR session within ds000030. - -```python -!tree ../data/ds000030 -``` -### the participants.tsv - -The `participants.tsv` file is meant to describe some demographic information on each participant within your study (eg. age, handedness, sex, etc.) Let's take a look at the `participants.tsv` file to see what's been included in this dataset. - -In order to load the data into Python, we'll need to import the `pandas` package. The `pandas` **dataframe** is Python's equivalent to an Excel spreadsheet. - -```python -import pandas as pd -``` - -We'll use the `read_csv()` function. It requires us to specify the name of the file we want to import and the separator that is used to distinguish each column in our file (`\t` since we're working with a `.tsv` file). - -```python -participant_metadata = pd.read_csv('../data/ds000030/participants.tsv', sep='\t') -``` - -In order to get a glimpse of our data, we'll use the `head()` function. By default, `head` prints the first 5 rows of our dataframe. - -```python -participant_metadata.head() -``` - -We can view any number of rows by specifying `n=?` as an argument within `head()`. -If we want to select particular rows within the dataframe, we can use the `loc[]` function and identify the rows we want based on their index label (the numbers in the left-most column). - -```python -participant_metadata.loc[[6, 10, 12]] -``` - -**EXERCISE**: Select the first 5 rows of the dataframe using `loc[]`. - -```python -participant_metadata.loc[:4] -``` - -**EXERCISE:** How many participants do we have in total? - -```python -participant_metadata.shape -``` - -There are 2 different methods of selecting columns in a dataframe: -* participant_metadata[`''`] (this is similar to selecting a key in a Python dictionary) -* participant_metadata.`` - - -Another way to see how many participants are in the study is to select the `participant_id` column and use the `count()` function. - -```python -participant_metadata['participant_id'].count() -``` - -**EXERCISE:** Which diagnosis groups are part of the study? -*Hint: use the* `unique()` *function.* - -```python -participant_metadata['diagnosis'].unique() -``` - -If we want to count the number of participants in each diagnosis group, we can use the `value_counts()` function. - -```python -participant_metadata['diagnosis'].value_counts() -``` - -**EXERCISE:** How many males and females are in the study? How many are in each diagnosis group? - -```python -participant_metadata['gender'].value_counts() -``` - -```python -participant_metadata.groupby(['diagnosis', 'gender']).size() -``` - -When looking at the participant dataframe, we noticed that there is a column called `ghost_NoGhost`. We should look at the README file that comes with the dataset to find out more about this. - -```python -!cat ../data/ds000030/README -``` - -For this tutorial, we're just going to work with participants that are either CONTROL or SCHZ (`diagnosis`) and have both a T1w (`T1w == 1`) and rest (`rest == 1`) scan. - - -**EXERCISE:** Filter `participant_metadata` so that only the above conditions are present. - -```python -participant_metadata = participant_metadata[(participant_metadata.diagnosis.isin(['CONTROL', 'SCHZ'])) & - (participant_metadata.T1w == 1) & - (participant_metadata.rest == 1)] -participant_metadata -``` - -## Querying a BIDS dataset with pybids - - -[pybids](https://bids-standard.github.io/pybids/) is a Python API for querying, summarizing and manipulating the BIDS folder structure. - -```python -from bids.layout import BIDSLayout -``` - -```python -layout = BIDSLayout("../data/ds000030") -``` - -Indexing a database can take a really long time, especially if you have several subjects, modalities, scan types, etc. `pybids` has an option to save the indexed results to a SQLite database. This database can then be re-used the next time you want to query the same database. - -```python -layout.save("../data/ds000030/.db") -``` - -```python -layout = BIDSLayout("../data/ds000030", database_path = "../data/ds000030/.db") -``` - -The pybids layout object lets you query your BIDS dataset according to a number of parameters by using a `get_*()` method. -We can get a list of the subjects we've downloaded from the dataset. - -```python -layout.get_subjects() -``` - -To get a list of all of the files, just use `get()`. - -```python -layout.get() -``` - -There are many arguments we can use to filter down this list. Any BIDS-defined keyword can be passed on as a constraint. In `pybids`, these keywords are known as **entities**. For a complete list of possibilities: - -```python -layout.entities -``` - -For example, if we only want the file paths of all of our resting state fMRI scans, - -```python -layout.get(datatype="func", suffix="bold", task="rest", extension=[".nii.gz"], return_type="file") -``` - -**EXERCISE**: Retrieve the file paths of any scan where the subject is '10292' or '50081' and the `RepetitionTime` is 2 seconds. - -```python -layout.get(subject="10159", RepetitionTime=2, return_type="file") -``` - -Let's save the first file from our list of file paths to a variable and pull the metadata from its associated JSON file using the `get_metadata()` function. - -```python -fmri_file = layout.get(subject="10159", RepetitionTime=2, return_type="file")[0] -layout.get_metadata(fmri_file) -``` - -We can even collect the metadata for all of our fmri scans into a list and convert this into a dataframe. - -```python -import pandas as pd - -metadata_list = [] -all_fmri_files = layout.get(datatype="func", suffix="bold", return_type="file", extension=[".nii.gz"]) -for fmri_file in all_fmri_files: - fmri_metadata = layout.get_metadata(fmri_file) - metadata_list.append(fmri_metadata) -df = pd.DataFrame.from_records(metadata_list) -df -``` diff --git a/_episodes_md/05-bids-derivatives.md b/_episodes_md/05-bids-derivatives.md deleted file mode 100644 index d2fb6e4..0000000 --- a/_episodes_md/05-bids-derivatives.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -jupyter: - jupytext: - formats: ipynb,md - text_representation: - extension: .md - format_name: markdown - format_version: '1.3' - jupytext_version: 1.13.8 - kernelspec: - display_name: Python 3 - language: python - name: python3 ---- - -## BIDS Apps - -[BIDS Apps](https://bids-apps.neuroimaging.io/) are containerized applications that run on BIDS data structures. - -Some examples include: -- mriqc -- fmriprep -- freesurfer -- ciftify -- SPM -- MRtrix3_connectome - -They rely on 2 technologies for container computing: -- **Docker** - - for building, hosting, and running containers on local hardware (Windows, Mac OS, Linux) or in the cloud -- **Singularity** - - for running containers on high performance compute clusters - -Drawing - -All BIDS Apps use the same command line format to run them: - -` /data /output participant [options]` - -Building a singularity container is as easy as: - - `singularity build mriqc-0.16.1.simg docker://poldracklab/mriqc:0.16.1` - -To run the container: - -``` -singularity run --cleanenv \ - -B bids_folder:/data \ - mriqc-0.16.1.simg \ - /data /data/derivatives participant -```