A key goal in neuroscience is to understand brain mechanisms of cognitive functions. An emerging approach is to study “brain states” dynamics using functional magnetic resonance imaging (fMRI). So far in the literature, brain states have typically been studied using 30 seconds of fMRI data or more, and it is unclear to which extent brain states can be reliably identified from very short time series. In this project, we applied graph convolutional networks (GCN) to decode brain activity over short time windows in a task fMRI dataset, i.e. associate a given window of fMRI time series with the task used. Starting with a populational brain graph with nodes defined by a parcellation of cerebral cortex and the adjacent matrix extracted from functional connectome, GCN takes a short series of fMRI volumes as input, generates high-level domain-specific graph representations, and then predicts the corresponding cognitive state. We investigated the performance of this GCN "cognitive state annotation" in the Human Connectome Project (HCP) database, which features 21 different experimental conditions spanning seven major cognitive domains, and high temporal resolution task fMRI data. Using a 10-second window, the 21 cognitive states were identified with an excellent average test accuracy of 89% (chance level 4.8%). As the HCP task battery was designed to selectively activate a wide range of specialized functional networks, we anticipate the GCN annotation to be applicable as a base model for other transfer learning applications, for instance, adapting to new task domains. Identifying brain networks involved in human cognition has been one of the main goals of neuroscience research. Modern imaging techniques, such as functional magnetic resonance imaging (fMRI), provide an opportunity to accurately map the neural substrates of human cognition. An emerging topic in the literature is the identification of "brain states", characterized by a canonical spatial pattern of functional activity, which were found to associate with specific cognitive states. A popular approach to identify these brain states, called multi-voxel pattern analysis (MVPA), uses machine learning tools to decode which task a subject performed based on recordings of brain activity in task fMRI (8) . But the algorithm is usually limited to specific cognitive domains and relies on long acquisition of brain activity with repeated blocks to accurately decode a brain state. In this project, we proposed a GCN architecture for annotating human brain activity on a cognitive battery of 21 task states. Instead of using the averaging BOLD signals or statistical constrast maps, GCN takes a short series of fMRI volumes as input, generates task-specific graph representations, and then predicts the corresponding cognitive labels. Comparing to the multi-class support vector machines classification, GCN achieved much higher performance in identifying a variety of cognitive states. 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada. In this project, we are using block-design task fMRI data from the Human Connectome Project (HCP) S1200 release. The minimal preprocessed fMRI data of the CIFTI format were used, which maps individual fMRI time series onto the standard surface template with 32k vertices per hemisphere. Further details on fMRI data acquisition, task design and preprocessing can be found in (2) and (5). The task fMRI data includes seven cognitive tasks, which are emotion, gambling, language, motor, relational, social, and working memory. In total, there are 23 different cognitive states. Considering the short event design nature of the gambling trials (1.5s for button press, 1s for feedback and 1s for ITI), we evaluated the decoding models with and without the two gambling conditions and found a much lower precision and recall scores for gambling task (average f1-score = 61%) than other cognitive domains (average f1-score > 91%). In the following experiments, we excluded the two gambling conditions and only reported results on the remaining 21 cognitive states. The detailed description of the tasks can be found in (2) . A summary table is also shown in Tab. 1.
Using a 10s window of fMRI signals, our GCN model identified 21 different task conditions from HCP dataset with a test accuracy of 89%.