Deep Learning for Brain-Computer Interface (BCI)

Book authors: Dr. Xiang Zhang (xiang_zhang@hms.harvard.edu), Prof. Lina Yao (lina.yao@unsw.edu.au)

Update

The whole well-processed and DL-ready data of 109 subjects from EEGMMIDB are uploaded!

Overview

This tutorial contains implementable python and jupyter notebook codes and benchmark datasets to learn how to recognize brain signals based on deep learning models. This tutorial associates our survey on DL-based noninvasive brain signals and book on DL-based BCI: Representations, Algorithms and Applications.

• We present a new taxonomy of BCI signal paradigms according to the acquisition methods. ECOG: Electrocorticography, EEG: Electroencephalography, fNIRS: functional near-infrared spectroscopy, fMRI: functional magnetic resonance imaging, EOG: Electrooculography, MEG: Magnetoencephalography.

• We systemically introduce the fundamental knowledge of deep learning models. MLP: Multi-Layer Perceptron, RNN: Recurrent Neural Networks, CNN: Convolutional Neural Networks, LSTM: Long Short-Term Memory, GRU: Gated Recurrent Units, AE: Autoencoder, RBM: Restricted Boltzmann Machine, DBN: Deep Belief Networks, VAE: Variational Autoencoder, GAN: Generative Adversarial Networks. D-AE denotes Stacked-Autoencoder which refers to the autoencoder with multiple hidden layers. Deep Belief Network can be composed of AE or RBM, therefore, we divided DBN into DBN-AE (stacked AE) and DBN-RBM (stacked RBM).

• Moreover, the guidelines for the investigation and design of BCI systems are provided from the aspects of signal category, deep learning models, and applications. The following figures show the distribution on signals and DL models in frointer DL-based BCI studies.

Distribution on signals	Distribution on DL models

• Special attention has been given to the state-of-the-art studies on deep learning for EEG-based BCI research in terms of algorithms. Specically, we introduces a number of advanced deep learning algorithms and frameworks aimed at several major issues in BCI including robust brain signal representation learning, cross-scenario classification, and semi-supervised classification.

• Furthermore, several novel prototypes of deep learning-based BCI systems are proposed which shed light on real-world applications such as authentication, visual reconstruction, language interpretation, and neurological disorder diagnosis. Such applications can dramatically benefit both healthy individuals and those with disabilities in real life.

BCI Dataset

Collection of brain signals is both financially and temporally costly. We extensively explore the benchmark data sets applicable to rain signal research and provides 31 public data sets with download links that cover most brain signal types.

Brain Signals	Dataset	#-Subject	#-Classes	Sampling Rate (Hz)	#-Channels	Download Link
FM EcoG	BCI-C IV, Data set IV	3	5	1000	48 -- 64	Link
MI EcoG	BCI-C III  Data set I	1	2	1000	64	Link
Sleeping EEG	Sleep-EDF Telemetry	22	6	100	2 EEG, 1 EOG, 1 EMG	Link
Sleeping EEG	Sleep-EDF: Cassette	78	6	100, 1	2 EEG (100 Hz), 1 EOG (100 Hz), 1 EMG (1 Hz)	Link
Sleeping EEG	MASS-1	53	5	256	17/19 EEG, 2 EOG, 5 EMG	Link
Sleeping EEG	MASS-2	19	6	256	19 EEG, 4 EOG, 1EMG	Link
Sleeping EEG	MASS-3	62	5	256	20 EEG, 2 EOG, 3 EMG	Link
Sleeping EEG	MASS-4	40	6	256	4 EEG, 4 EOG, 1 EMG	Link
Sleeping EEG	MASS-5	26	6	256	20 EEG, 2 EOG, 3 EMG	Link
Sleeping EEG	SHHS	5804	N/A	125, 50	2 EEG (125 Hz), 1EOG (50 Hz), 1 EMG (125 Hz)	Link
Seizure EEG	CHB-MIT	22	2	256	18	Link
Seizure EEG	TUH	315	2	200	19	Link
MI EEG	EEGMMI	109	4	160	64	Link
MI EEG	BCI-C II, Data set III	1	2	128	3	Link
MI EEG	BCI-C III, Data set III a	3	4	250	60	Link
MI EEG	BCI-C III, Data set III b	3	2	125	2	Link
MI EEG	BCI-C III, Data set IV a	5	2	1000	118	Link
MI EEG	BCI-C III, Data set IV b	1	2	1001	119	Link
MI EEG	BCI-C III, Data set IV c	1	2	1002	120	Link
MI EEG	BCI-C IV, Data set I	7	2	1000	64	Link
MI EEG	BCI-C IV, Data set II a	9	4	250	22 EEG, 3 EOG	Link
MI EEG	BCI-C IV, Data set II b	9	2	250	3 EEG, 3 EOG	Link
Emotional EEG	AMIGOS	40	4	128	14	Link
Emotional EEG	SEED	15	3	200	62	Link
Emotional EEG	DEAP	32	4	512	32	Link
Others EEG	Open MIIR	10	12	512	64	Link
VEP	BCI-C II, Data set II b	1	36	240	64	Link
VEP	BCI-C III, Data set II	2	26	240	64	Link
fMRI	ADNI	202	3	N/A	N/A	Link
fMRI	BRATS	65	4	N/A	N/A	Link
MEG	BCI-C IV, Data set III	2	4	400	10	Link

In order to let the readers have a quick access of the dataset and can play around it, we provide the well-processed and ready-ro-use dataset of EEG Motor Movement/Imagery Database (EEGMMIDB). This dataset contains 109 subjects while the EEG signals are recorded in 64 channels with 160 Hz sampling rate. After our clearning and sorting, each npy file represents a subject: the data shape of each npy file is [N, 65], the first 64 columns correspond to 64 channel features, the last column denotes the class label. The N varies for different subjects, but N should be either 259520 or 255680. This is the inherent difference in the original dataset.

Running the code

In our tutorial files, you will learn the pipline and workflow of BCI system including data acquiction, pre-processing, feature extraction (optional), classification, and evaluation. We present necessary references and actionable codes of the most typical deep learning models (GRU, LSTM, CNN, GNN) while taking advantage of temporal, spatial, and topographical depencencies. We also provide python codes that are very handy. For example, to check the EEG classification performance of CNN, run the following code:

python 4-2_CNN.py 

For PyTorch beginners, we highly recommond Morvan Zhou's PyTorch Tutorials.

Chapter resources

For the algorithms and applications introduced in the book, we provide the necessary implementary codes (TensorFlow version):

• Adaptive feature learning (Chapter 7)
• MindID: EEG-based user identification (Chapter 9)
• Reconstruction of image based on visual evoked EEG (Chapter 10)
• Brain typing: convert EEG to text (Chapter 11)
• Neurological disorder (seizure) diagnosis (Chapter 13)

Citing

If you find our research is useful for your research, please consider citing our survey or book:

@article{zhang2020survey,
  title={A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers},
  author={Zhang, Xiang and Yao, Lina and Wang, Xianzhi and Monaghan, Jessica JM and Mcalpine, David and Zhang, Yu},
  journal={Journal of Neural Engineering},
  year={2020},
  publisher={IOP Publishing}
}

@book{zhang2021deep,
  title={Deep Learning for EEG-based Brain-Computer Interface: Representations, Algorithms and Applications},
  author={Zhang, Xiang and Yao, Lina},
  year={2021},
  publisher={World Scientific Publishing}
}

Requirements

The tutorial codes are tested to work under Python 3.7.

Recent versions of Pytorch, torch-geometric, numpy, and scipy are required. All the required basic packages can be installed using the following command: ''' pip install -r requirements.txt ''' Note: For toch-geometric and the related dependices (e.g., cluster, scatter, sparse), the higher version may work but haven't been tested yet.

Miscellaneous

Please send any questions you might have about the code and/or the algorithm to xiang_zhang@hms.harvard.edu.

License

This tutorial is licensed under the MIT License.