Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding
  (Survey)
Can we obtain insights about the brain using AI models? How is the
information in deep learning models related to brain recordings? Can we improve
AI models with the help of brain recordings? Such questions can be tackled by
studying brain recordings like functional magnetic resonance imaging (fMRI). As
a first step, the neuroscience community has contributed several large
cognitive neuroscience datasets related to passive reading/listening/viewing of
concept words, narratives, pictures, and movies. Encoding and decoding models
using these datasets have also been proposed in the past two decades. These
models serve as additional tools for basic cognitive science and neuroscience
research. Encoding models aim at generating fMRI brain representations given a
stimulus automatically. They have several practical applications in evaluating
and diagnosing neurological conditions and thus may also help design therapies
for brain damage. Decoding models solve the inverse problem of reconstructing
the stimuli given the fMRI. They are useful for designing brain-machine or
brain-computer interfaces. Inspired by the effectiveness of deep learning
models for natural language processing, computer vision, and speech, several
neural encoding and decoding models have been recently proposed. In this
survey, we will first discuss popular representations of language, vision and
speech stimuli, and present a summary of neuroscience datasets. Further, we
will review popular deep learning based encoding and decoding architectures and
note their benefits and limitations. Finally, we will conclude with a summary
and discussion about future trends. Given the large amount of recently
published work in the computational cognitive neuroscience (CCN) community, we
believe that this survey enables an entry point for DNN researchers to
diversify into CCN research.
Subba Reddy Oota, Zijiao Chen, Manish Gupta, Raju S. Bapi, Gael Jobard, Frederic Alexandre, Xavier Hinaut
# Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)

###### Abstract

How does the brain represent different modes of information? Can we design a system that automatically understands what the user is thinking? Such questions can be answered by studying brain recordings like functional magnetic resonance imaging (fMRI). As a first step, the neuroscience community has contributed several large cognitive neuroscience datasets related to passive reading/listening/viewing of concept words, narratives, pictures and movies. Encoding and decoding models using these datasets have also been proposed in the past two decades. These models serve as additional tools for basic research in cognitive science and neuroscience. Encoding models aim at generating fMRI brain representations given a stimulus automatically. They have several practical applications in evaluating and diagnosing neurological conditions and thus also help design therapies for brain damage. Decoding models solve the inverse problem of reconstructing the stimuli given the fMRI. They are useful for designing brain-machine or brain-computer interfaces. Inspired by the effectiveness of deep learning models for natural language processing, computer vision, and speech, recently several neural encoding and decoding models have been proposed. In this survey, we will first discuss popular representations of language, vision and speech stimuli, and present a summary of neuroscience datasets. Further, we will review popular deep learning based encoding and decoding architectures and note their benefits and limitations. Finally, we will conclude with a brief summary and discussion about future trends. Given the large amount of recently published work in the 'computational cognitive neuroscience' community, we believe that this survey nicely organizes the plethora of work and presents it as a coherent story.

## 1 Introduction

Neuroscience is the field of science that studies the structure and function of the nervous system of different species. It involves answering interesting questions like the following1. (1) How learning occurs during adolescence, and how it differs from the way adults learn and form memories. (2) Which specific cells in the brain (and what connections they form with other cells), have a role in how memories are formed? (3) How animals cancel out irrelevant information arriving from the senses and focus only on information that matters. (4) How do humans make decisions? (5) How humans develop speech and learn languages. Neuroscientists study diverse topics that help us understand how the brain and nervous system work.

Footnote 1: [https://zuckermaninstitute.columbia.edu/file/5184/download?](https://zuckermaninstitute.columbia.edu/file/5184/download?) token=qzId8yR

**Motivation:** The central aim of neuroscience is to unravel how the brain represents information and processes it to carry out various tasks (visual, linguistic, auditory, etc.). Deep neural networks (DNN) offer a computational medium to capture the unprecedented complexity and richness of brain activity. _Encoding_ and _decoding_ stated as computational problems succinctly encapsulate this puzzle. As the previous surveys systematically explore the brain encoding and decoding studies with respect to only language [1, 1], this survey summarizes the latest efforts in how DNNs begin to solve these problems and thereby illuminate the computations that the unreachable brain accomplishes effortlessly.

**Brain encoding and decoding**: Two main tasks studied in cognitive neuroscience are brain encoding and brain decoding, as shown in Figure 1. Encoding is the process of learning the mapping \(e\) from the stimuli \(S\) to the neural activation \(F\). The mapping can be learned using features engineering or deep learning. On the other hand, decoding constitutes learning mapping \(d\), which predicts stimuli \(S\) back from the brain activation \(F\). However, in most cases, brain decoding aims at predicting a stimulus representation \(R\) rather than actually reconstructing \(S\). In both cases, the first step is to learn a semantic representation \(R\) of the stimuli \(S\) at the train time. Next, for encoding, a regression function \(e:R\to F\) is trained. For decoding, a function \(d:F\to R\) is trained. These functions \(e\) and \(d\) can then be used at test time to process new stimuli and brain activations, respectively.

**Techniques for recording brain activations**: Popular techniques for recording brain activations include single Micro
Electrode (ME), Micro-Electrode array (MEA), Electro-Cortico Graphy (ECoG), Positron emission tomography (PET), functional MRI (fMRI), Magneto-encephalography (MEG), Electro-encephalography (EEG) and Near-Infrared Spectroscopy (NIRS). These techniques differ in their spatial resolution of neural recording and temporal resolution.

fMRI enable high spatial but low time resolution. Hence, they are good for examining which parts of the brain handle critical functions. fMRI takes 1-4 seconds to complete a scan. This is far lower than the speed at which humans can process language. On the other hand, both MEG and EEG have high time but low spatial resolution. They can preserve rich syntactic information [1] but cannot be used for source analysis. fNIRS are a compromise option. Their time resolution is better than fMRI, and spatial resolution is better than EEG. However, this spatial and temporal resolution balance may not compensate for the loss in both.

**Stimulus Representations**: Neuroscience datasets contain stimuli across various modalities: text, visual, audio, video and other multimodal forms. Representations differ based on modality. Older methods for _text-based stimulus representation_ include text corpus co-occurrence counts, topic models, syntactic, and discourse features. In recent times, both semantic and experiential attribute models have been explored for text-based stimuli. Semantic representation models include distributed word embeddings, sentence representation models, recurrent neural networks (RNNs), and Transformer-based language models. Experiential attribute models represent words in terms of human ratings of their degree of association with different attributes of experience, typically on a scale of 0-6 or binary. Older methods for _visual stimulus representation_ used visual field filter bank and Gabor wavelet pyramid for visual stimuli, but recent methods use models like ImageNet-pretrained convolutional neural networks (CNNs) and concept recognition methods. For _audio stimuli_, phoneme rate and the presence of phonemes have been leveraged, besides deep learning models like SoundNet. Finally, for multimodal stimulus representations, researchers have used both early fusion and late fusion deep learning methods. In the early fusion methods, information across modalities is combined in the early steps of processing. While in late fusion, the combination is performed only at the end. We discuss stimulus representation methods in detail in Sec. 2.

**Naturalistic Neuroscience Datasets**: Several neuroscience datasets have been proposed across modalities (see Figure 2). These datasets differ in terms of the following criteria: (1) Method for recording activations: fMRI, EEG, MEG, etc. (2) Repetition time (TR), i.e. the sampling rate. (3) Characteristics of fixation points: location, color, shape. (4) Form of stimuli presentation: text, video, audio, images, or other multimodality. (5) Task that participant performs during recording sessions: question answering, property generation, rating quality, etc. (6) Time given to participants for the task, e.g., 1 minute to list properties. (7) Demography of participants: males/females, sighted/blind, etc. (8) Number of times the response to stimuli was recorded. (9) Natural language associated with the stimuli. We discuss details of proposed datasets in Sec. 3.

**Brain Encoding**: Other than using the standard stimuli representation architectures, brain encoding literature has focused on studying a few important aspects: (1) Which models lead

Figure 1: Computational Cognitive Neuroscience of Brain Encoding and Decoding: Datasets & Stimulus Representations
to better predictive accuracy across modalities? (2) How can we disentangle the contributions of syntax and semantics from language model representations to the alignment between brain recordings and language models? (3) Why do some representations lead to better brain predictions? How are deep learning models and brains aligned in terms of their information processing pipelines? (4) Does joint encoding of task and stimulus representations help? We discuss these details of encoding methods in Sec. 5.

**Brain Decoding**: Ridge regression is the most popular brain decoder. Recently, a fully connected layer [1] or multi-layered perceptrons (MLPs) [23] have also been used. While older methods attempted to decode to a vector representation using stimuli of a single mode, newer methods focus on multimodal stimuli decoding [13, 12]. Decoding using Transformers [11, 12, 13], and decoding to actual stimuli (word, passage, image, dialogues) have also been explored. We discuss details of these decoding methods in Sec. 6.

**Computational Cognitive Science (CCS) Research goals**: CCS researchers have primarily focused on two main areas [14] (also, see Figure 3). (1) Improving predictive Accuracy. In this area, the work is around the following questions. (a) Compare feature sets: Which feature set provides the most faithful reflection of the neural representational space? (b) Test feature decodability: "Does neu

Figure 3: Alignment between deep learning systems and human brains [12].

Figure 2: Representative Samples of Naturalistic Brain Dataset: (LEFT) Brain activity recorded when subjects are reading and listening to the same narrative [1], and (RIGHT) example naturalistic image stimuli from various public repositories: BOLD5000 [13], SSFMRI [1], and VIM-1 [14].
ral data Y contain information about features X?" (c) Build accurate models of brain data: The aim is to enable simulation of neuroscience experiments. (2) Interpretability. In this area, the work is around the following questions. (a) Examine individual features: Which features contribute most to neural activity? (b) Test correspondences between representational spaces: "CNNs vs ventral visual stream" or "Two text representations". (c) Interpret feature sets: Do features X, generated by a known process, accurately describe the space of neural responses Y? Do voxels respond to a single feature or exhibit mixed selectivity? (d) How does the mapping relate to other models or theories of brain function? We discuss some of these questions in Sections 5 and 6.

## 2 Stimulus Representations

In this section, we discuss types of stimulus representations that have been proposed in the literature across different modalities: text, visual, audio, video and other multimodal stimuli.

**Text Stimulus Representations**: Older methods for text-based stimuli representation include text corpus co-occurrence counts [16, 15, 17], topic models [15], syntactic features and discourse features [23]. In recent times, for text-based stimuli, both semantic models as well as experiential attribute models have been explored. Semantic representation models include word embedding methods [15, 16, 17, 18, 19, 20], sentence representations models (see Figure 4) [23, 24, 25], RNNs [10, 26] and Transformer methods [16, 17, 18, 19, 20, 21, 22]. Popular word embedding methods include textual (i.e., Word2Vec, fastText, and GloVe), linguistic (i.e., dependency), conceptual (i.e., RWSGwn and ConceptNet), contextual (i.e., ELMo). Popular sentence embedding models include average, max, content of avg and max, SIF, fairseq, skip, GenSen, InferSent, ELMo, BERT, RoBERTa, USE, QuickThoughts and GPT-2. Transformer-based methods include pretrained BERT with various NLU tasks, finetuned BERT, Transformer-XL, GPT-2, BART, BigBird, LED, and LongT5. Experiential attribute models represent words in terms of human ratings of their degree of association with different attributes of experience, typically on a scale of 0-6 [1, 1, 19, 17] or binary [1, 20].

**Visual Stimulus Representations**: For visual stimuli, older methods used visual field filter bank [14] and Gabor wavelet pyramid [15, 16]. Recent methods use models like CNNs [23, 24, 25, 26] and concept recognition models [1].

**Audio Stimuli Representations**: For audio stimuli, phoneme rate and presence of phonemes have been leveraged [17]. Recently, authors in [20] used features from an audio deep learning model called SoundNet for audio stimuli representation.

**Multimodal Stimulus Representations**: To jointly model the information from multimodal stimuli, recently, various multimodal representations have been used. These include processing videos using audio-image representations like VGG+SoundNet [20] or using image+text combination models like GloVe+VGG and ELMo+VGG in [26]. Recently, the usage of multimodal text+vision models like CLIP, LXMERT, and VisualBERT was proposed in [1].

## 3 Naturalistic Neuroscience Datasets

We discuss the popular text, visual, audio, video and other multimodal neuroscience datasets that have been proposed in the literature. Table 1 shows a detailed overview of brain recording type, language, stimulus, number of subjects (\(|\)S\(|\)) and the task across datasets of different modalities. Figure 2 shows examples from a few datasets.

**Text Datasets**: These datasets are created by presenting words, sentences, passages or chapters as stimuli. Some of the text datasets include Harry Potter Story [20], ZUCO EEG [12] and datasets proposed in [1, 19, 20, 21]. In [19, 20], participants were asked to verbally enumerate in one minute the properties (features) that describe the entities the words refer to. There were four groups of participants: 5 sighted individuals were presented with a pictorial form of the nouns, 5 sighted individuals with a verbal-visual (i.e., written Italian words) form, 5 sighted individuals with a verbal auditory (i.e., spoken Italian words) form, and 5 congenitally blind with a verbal auditory form. Data proposed by [1] contains 70 Italian words taken from seven taxonomic categories (abstract, attribute, communication, event/action, person/social role, location, object/tool) in the law and music domain. The word list contains concrete as well as abstract words. ZUCO dataset [12] contains sentences for which fMRIs were obtained for 3 tasks: normal reading of movie reviews, normal reading of Wikipedia sentences and task-specific reading of Wikipedia sentences. For this dataset curation, sentences were presented

Figure 4: Language Model
to the subjects in a naturalistic reading scenario. A complete sentence is presented on the screen. Subjects read each sentence at their own speed, i.e., the reader determines for how long each word is fixated and which word to fixate next.

**Visual Datasets**: Older visual datasets were based on binary visual patterns [13]. Recent datasets contain natural images. Examples include Vim-1 [11], BOLD5000 [15], Algonauts [10], NSD [14], Things-data[1], and the dataset proposed in [1]. BOLD5000 includes \(\sim\)20 hours of MRI scans per each of the four participants. 4,916 unique images were used as stimuli from 3 image sources. Algonauts contains two sets of training data, each consisting of an image set and brain activity in RDM format (for fMRI and MEG). Training set 1 has 92 silhouette object images, and training set 2 has 118 object images with natural backgrounds. Testing data consists of 78 images of objects on natural backgrounds. Most of the visual datasets involve passive viewing, but the dataset in [1] involved the participant doing the one-back repetition detection task.

**Audio Datasets**: Most of the proposed audio datasets are in English [11, 1, 12], while there is one [11] on Italian. The participants were involved in a variety of tasks while their brain activations were measured: Property generation [11, 12], passive listening [11, 12], question answering [1] and imagining themselves personally experiencing common scenarios [13]. In the last one, participants underwent fMRI as they reimagined the scenarios (e.g., resting, reading, writing, bathing, etc.) when prompted by standardized cues. Narratives [14] used 17 different stories as stimuli. Across subjects, it is 6.4 days worth of recordings.

**Video Datasets**: Recently, video neuroscience datasets have also been proposed. These include BBC's Doctor Who [15], Japanese Ads [15], Pippi

\begin{table}
\begin{tabular}{|p{11.4pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline \multicolumn{2}{|c|}{**Dataset**} & **Authors** & **Type** & **Lang.** & **Stimulus** & **S** & **Task** \\ \hline \multirow{10}{*}{} & Harry Potter & [14] & MRU & English & Reading Chapter 9 of Harry Potter and the Soreerer’s Stone & 9 & Story understanding \\ \cline{2-7}  & & MRU & MEG & & & & \\ \cline{2-7}  & & [11] & fMRI & Italian & Verbal, pictorial or auditory presentation of 40 concrete nouns, four times & 20 & Property Generation \\ \cline{2-7}  & & [11] & fMRI & Italian & Reading 70 concrete and abstract nouns from two music, five times & 7 & Imaging a situation with noun \\ \cline{2-7}  & ZuCo & [11] & fMRI & English & Reading 1107 sentences with 21,629 words from movie reviews & 12 & Rate movie quality \\ \cline{2-7}  & 240 Sentences with Content Words & [1] & fMRI & English & Reading 240 active voice sentences describing everyday situations & 14 & Passive reading \\ \cline{2-7}  & BCCW-EEG & [14] & EEG & Japanese & Reading 20 newspaper articles for \(\sim\)30-40 minutes & 40 & Passive reading \\ \cline{2-7}  & Subset Moh Radio Hour & [15] & fMRI & English & Reading 11 stories & 9 & Passive reading and Listening \\ \cline{2-7}  & & [11] & fMRI & - & Viewing routing edges (8 times), expanding/contracting rings (8 times), rotating 36 Gabor filters (4 times), grad (36 times) & 9 & Passive viewing \\ \cline{2-7}  & Vim-1 & [11] & MRU & - & Viewing sequences of 1870 natural photos & 2 & Passive viewing \\ \cline{2-7}  & Generic Object Decoder & [11] & fMRI & - & Viewing 1,200 images from 150 object categories; 50 images from 50 & 5 & Repetition detection \\ \cline{2-7}  & BOLD5000 & [15] & fMRI & - & Viewing 5254 images depicting real-world scenes & 4 & Passive viewing \\ \cline{2-7}  & Algonauts & [14] & fMRI & - & Viewing 92 silhouette object images and 118 images of objects on natural & 15 & Passive viewing \\ \cline{2-7}  & NSD & [14] & fMRI & - & Viewing 7300 natural images & 8 & Passive viewing \\ \cline{2-7}  & THINGS & [11] & fMRI & - & Viewing 31188 natural images & 8 & Passive viewing \\ \cline{2-7}  & THINGS & [11] & MRU & Italian & Verbal, pictorial or auditory presentation of 40 concrete nouns, 4 times & 20 & Property Generation \\ \cline{2-7}  & The Moth Radio Hour & [11] & fMRI & English & Listening eleven 10-minute stories & 7 & Passive Listening \\ \cline{2-7}  & The Moth Radio Hour & [11] & EEG & English & Listening Chapter of dance’s Adventures in Wonderland (2,129 & 33 & Question answering words in 84 sentences) as read by Kristen McQuillan & 26 & Inagine personal experiences \\ \cline{2-7}  & & [11] & fMRI & English & Listening one of 20 scenarios names, 5 times & 345 & Passive Listening \\ \cline{2-7}  & Narratives & [14] & fMRI & English & Listening 27 diverse naturalistic spoken stories. 891 functional scans & 345 & Passive Listening \\ \cline{2-7}  & Natural Stories & [14] & fMRI & English & Listening Mook-Radio-Hour naturalistic spoken stories. & 19 & Passive Listening \\ \cline{2-7}  & The Little Pinace & [14] & fMRI & English & Listening audiobook for about 100 minutes. & 112 & Passive Listening \\ \cline{2-7}  & MEG-MASC & [11] & MEG & English & Listening two hours of naturalistic stories. 208 MED sensors & 1 & Passive listening \\ \cline{2-7}  & BBC's Doctor Who & [15] & fMRI & English & Viewing spatiotemporal visual and auditory videos (30 episodes). 120.8 whole-brain volumes (\(\sim\)23) of using single-spreading and data, and 12 values (11 min) of repeated narrative short episodes. 22 repetitions & 1 & Passive viewing \\ \cline{2-7}  & Japanese Ads & [15] & fMRI & Japanese Video 88 and 2427 NYPT Japanese ad movies (15-308). 7200 train and 1200 test m/MRIs for web; MRIs & MRIs & MRIs & MRIs & MRIs \\ \cline{2-7}  & Pippi Langkous & [14] & ECG & Swedish Viewing 30 \(\sim\) 35 concepts of a feature film (in total, 6.5 min long), edited & 37 & Passive viewing \\ \cline{2-7}  & Algonauts & [14] & fMRI & English & Viewing 1000 short video clips (3 sec each) & 10 & Passive viewing \\ \cline{2-7}  & Natural Stories & [11] & fMRI & English & Weaching natural short movie clips & 5 & Passive viewing \\ \cline{2-7}  & Natural Stot Clips & [11] & fMRI & English & Watching 170 natural short video clips & 10 & Passive viewing \\ \cline{2-7}  & 60 Concrete Nouns & [14] & fMRI & English & Viewing 60 different word-picture pairs from 12 categories, 6 times each & 9 & Passive viewing \\ \cline{2-7}  & & [14] & MEG & English & Reading 0 concrete nouns along with line drawings. 20 questions per & 9 & Question answering \\ \cline{2-7}  & & [14] & MEG & English & Reading 0 concrete nouns (audisovisual word and picture stimuli: bunny, bear, hark, dog, mouth, food, hand, and nose; 12 times repeated) & 24 & Passive viewing and listening \\ \cline{2-7}  & & [14] & fMRI & English & Viewing 180 Words with Picture, Sentences, word clouds; reading 96 & 16 & Passive viewing and reading text passages; 72 passages. 3 times repeated. & 24 & Passive viewing and listening \\ \cline{2-7}  & & [14] & fMRI & Chinese & Viewing and listening 50 concrete nouns from 10 semantic categories. & 7 & Passive viewing and listening \\ \cline{2-7}  & Neuromond & [1] & fMRI & English & Watching TV series (Friends, MovieID) & 6 & Passive viewing and listening \\ \hline \end{tabular}
\end{table}
Table 1: Naturalistic Neuroscience Datasets
Langkous [1] and Algonauts [1]. Japanese Ads data contains data for two sets of movies were provided by NTT DATA Corp: web and TV ads. There are also four types of cognitive labels associated with the movie datasets: scene descriptions, impression ratings, ad effectiveness indices, and ad preference votes. Algonauts 2021 contains fMRIs from 10 human subjects that watched over 1,000 short (3 sec) video clips.

**Other Multimodal Datasets**: Finally, beyond the video datasets, datasets have also been proposed with other kinds of multimodality. These datasets are audiovisual [1, 2], words associated with line drawings [15, 16], pictures along with sentences and word clouds [12]. These datasets have been collected using a variety of methods like fMRIs [15, 12], MEG [10] and fNIRS [14, 15]. Specifically, in [16], subjects were asked to perform a QA task, while their brain activity was recorded using MEG. Subjects were first presented with a question (e.g., "Is it manmade?"), followed by 60 concrete nouns, along with their line drawings, in a random order. For all other datasets, subjects performed passive viewing and/or listening.

## 4 Evaluation Metrics

Two metrics are popularly used to evaluate brain encoding models: 2V2 accuracy [13, 14] and Pearson Correlation [17], as shown in Figure 5.

They are defined as follows. Given a subject and a brain region, let \(N\) be the number of samples. Let \(\{Y_{i}\}_{i=1}^{N}\) and \(\{\hat{Y}_{i}\}_{i=1}^{N}\) denote the actual and predicted voxel value vectors for the \(i^{th}\) sample. Thus, \(Y\in R^{N\times V}\) and \(\hat{Y}\in R^{N\times V}\) where \(V\) is the number of voxels in that region. **2V2 Accuracy** is computed as \(\frac{1}{N_{C_{2}}}\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}I[\{\text{cosD}(Y_{i},\hat{Y }_{i})+\text{cosD}(Y_{j},\hat{Y}_{j})\}<\{\text{cosD}(Y_{i},\hat{Y}_{j})+\text{ cosD}(Y_{j},\hat{Y}_{i})\}]\) where cosD is the cosine distance function. \(I[c]\) is an indicator function such that \(I[c]=1\) if \(c\) is true, else it is 0. The higher the 2V2 accuracy, the better. **Pearson Correlation** is computed as PC=\(\frac{1}{N}\sum_{i=1}^{n}corr[Y_{i},\hat{Y}_{i}]\) where corr is the correlation function.

Brain decoding methods are evaluated using popular metrics like pairwise and rank accuracy [12, 16]. Other metrics used for brain decoding evaluation include R\({}^{2}\) score, mean squared error, and using Representational Similarity Matrix [1, 15].

**Pairwise Accuracy** To measure the pairwise accuracy, the first step is to predict all the test stimulus vector representations using a trained decoder model. Let S = [S\({}_{0}\), S\({}_{1}\),\(\cdots\),S\({}_{n}\)], \(\hat{S}\) = [\(\hat{S}_{0}\), \(\hat{S}_{1}\),\(\cdots\),\(\hat{S}_{n}\)] denote the "true" (stimuli-derived) and predicted stimulus representations for \(n\) test instances resp. Given a pair \((i,j)\) such that \(0\leq i,j\leq n\), score is 1 if \(corr\)(S\({}_{i}\),\(\hat{S}_{i}\)) + \(corr\)(S\({}_{j}\),\(\hat{S}_{j}\)) > \(corr\)(S\({}_{i}\),\(\hat{S}_{j}\)) + \(corr\)(S\({}_{j}\),\(\hat{S}_{i}\)), else 0. Here, \(corr\) denotes the Pearson correlation. Final pairwise matching accuracy per participant is the average of scores across all pairs of test instances. For computing rank accuracy, we first compare each decoded vector to all the "true" stimuli-derived semantic vectors and ranked them by their correlation. The classification performance reflects the rank \(r\) of the stimuli-derived vector for the correct word/picture/stimuli: \(1-\frac{r-1}{\#instances-1}\). The final accuracy value for each participant is the average rank accuracy across all instances.

## 5 Brain Encoding

Encoding is the learning of the mapping from the stimulus domain to the neural activation. The quest in brain encoding is for "reverse engineering" the algorithms that the brain uses for sensation, perception, and higher-level cognition. Recent breakthroughs in applied NLP enable reverse engineering the language function of the brain. Similarly, pioneering results have been obtained for reverse engineering the function of ventral visual stream in object recognition founded on the advances and remarkable success of deep CNNs. The overall schema of building a brain encoder is shown in Figure 6.

Initial studies on brain encoding focused on smaller data sets and single modality of brain responses. Early models used word representations [13]. Rich contextual representations derived from RNNs such as LSTMs resulted in superior encoding models [17, 16] of narratives. The recent

Figure 5: Evaluation Metrics for Brain Encoding and Decoding. (LEFT) Pearson Correlation, (MIDDLE) 2V2 Accuracy [13], and (RIGHT) Pairwise Accuracy.
efforts are aimed at utilizing the internal representations extracted from transformer-based language models such as ELMo, BERT, GPT-2, etc for learning encoding models of brain activation [11, 12]. High-grain details such as lexical, compositional, syntactic, and semantic representations of narratives are factorized from transformer-based models and utilized for training encoding models. The resulting models are better able to disentangle the corresponding brain responses in fMRI [12]. Finally, is has been found that the models that integrate task and stimulus representations have significantly higher prediction performance than models that do not account for the task semantics [13, 14].

Similarly, in vision, early models focused on independent models of visual processing (object classification) using CNNs [23]. Recent efforts in visual encoding models focus on using richer visual representations derived from a variety of computer vision tasks [24]. Instead of feed-forward deep CNN models, using shallow recurrence enabled better capture of temporal dynamics in the visual encoding models [15, 16].

Table 2 summarizes various encoding models proposed in the literature related to textual, audio, visual, and multimodal stimuli. Figure 7 classifies the encoding literature along various stimulus domains such as vision, auditory, multimodal, and language and the corresponding tasks in each domain.

**Linguistic Encoding**: A number of previous works have investigated the alignment between pretrained language models and brain recordings of people comprehending language. Huth et al. [10] have been able to identify brain ROIs (Regions of Interest) that respond to words that have a similar meaning and have thus built a "semantic atlas" of how the human brain organizes language. Many studies have shown accurate results in mapping the brain activity using neural distributed word embeddings for linguistic stimuli [11, 12, 13, 14]. Unlike earlier models where each word is represented as an independent vector in an embedding space, [13] built encoding models using rich contextual representations derived from an LSTM language model in a story listening task. With these contextual representations, they demonstrated dissociation in brain activation - auditory cortex (AC) and Broca's area in shorter context whereas left Temporal-Parietal junction (TPJ) in longer context. [10] presents the first multimodal framework for evaluating six types of word embedding (Word2Vec, WordNet2Vec, GloVe, FastText, ELMo, and BERT) on 15 datasets, including eye-tracking, EEG and fMRI signals recorded during language processing. With the recent advances in contextual representations in NLP, few studies incorporated them in relating sentence embeddings with brain activity patterns [24, 11, 12].

More recently, researchers have begun to study the alignment of language regions of the brain with the layers of language models and found that the best alignment was achieved in the middle layers of these models [13, 14, 15]. Schrimpf et al. [16] examined the relationship between 43 diverse state-of-the-art language models. They also studied the behavioral signatures of human language processing in the form of self-paced reading times, and a range of linguistic functions assessed via standard engineering tasks from NLP. They found that Transformer-based models perform better than RNNs or word-level embedding models. Larger-capacity models perform better than smaller models. Models initialized with random weights (prior to training) perform surprisingly similarly in neural predictivity as compared to final trained models, suggesting that network architecture contributes as much or more than experience dependent learning to a model's match to the brain. Antonello et al. [16] proposed a "language representation embedding space" and demonstrated the effectiveness of the features from this embedding in predicting fMRI responses to linguistic stimuli.

**Disentangling the Syntax and Semantics**: The representations of transformer models like BERT, GPT-2 have been

Figure 6: Schema for Brain Encoding
shown to linearly map onto brain activity during language comprehension. Several studies have attempted to disentangle the contributions of different types of information from word representations to the alignment between brain recordings and language models. Wang et al. (2020) proposed a two-channel variational autoencoder model to dissociate sentences into semantic and syntactic representations and separately associate them with brain imaging data to find feature-correlated brain regions. To separate each syntactic feature, Zhang et al. (2022) proposed a feature elimination method, called Mean Vector Null space Projection. Compared with word representations, word syntactic features (parts-of-speech, named entities, semantic roles, dependencies) seem to be distributed across brain networks instead of a local brain region. In the previous two studies, we do not know whether all or any of these representations effectively drive the linear mapping between language models (LMs) and the brain. Toneva et al. (2022) presented an approach to disentangle supra-word meaning from lexical meaning in language models and showed that supra-word meaning is predictive of fMRI recordings in two language regions (anterior and posterior temporal lobes). Caucheteux et al. (2021) proposed a taxonomy to factorize the high-dimensional activations of language models into four combinatorial classes: lexical, compositional, syntactic, and semantic representations. They found that (1) Compositional representations recruit a more widespread cortical network than lexical ones, and encompass the bilateral temporal, parietal and prefrontal cortices. (2) Contrary to previous claims, syntax and semantics are not associated with separated modules, but, instead, appear to share a common and distributed neural substrate.

While previous works studied syntactic processing as cap

\begin{table}
\begin{tabular}{|l|l|l|l|l|l|l|l|} \hline
**Stimuli** & **Authors** & **Dataset** & **Lang.** & **Stimulus Representations** & \(|\)\(\mathbf{S}|\) & **Dataset** & **Model** \\ \hline \multirow{11}{*}{[1]} & [Main and Huh, 2018] & fMRI & English & LSTM & 6 & Subset Moth Radio Hour & Ridge \\ \cline{2-8}  & [Toneva and Webe, 2019] & fMRI / MEG & English & ELMo, BERT, Transformer-XL & 9 & Story understanding & Ridge \\ \cline{2-8}  & [Toneva _et al._, 2020] & MEG & English & BERT & 9 & Question-Answering & Ridge \\ \cline{2-8}  & [Schrimpf, 2021b] & fMRI/ECG & English & 43 language models (e.g. GloVe, ELMo, BERT, GPT-2, XLNET) & 20 & Neural architecture of language & Ridge \\ \cline{2-8}  & [Gautier and Levy, 2019] & fMRI & English & BERT, line-tuned NLP tasks (Sentiment, Natural language inference), Scrambling language model & 7 & Imagine a situation with the noun & Ridge \\ \cline{2-8}  & [Deniz _et al._, 2019] & fMRI & English & GloVe & 9 & Subset Moth Radio Hour & Ridge \\ \cline{2-8}  & [Jain _et al._, 2020] & fMRI & English & LSTM & 6 & Subset Moth Radio Hour & Ridge \\ \cline{2-8}  & [Caucheteux _et al._, 2021] & fMRI & English & GPT-2, Basic syntax features & 345 & Narratives & Ridge \\ \cline{2-8}  & [Antonello _et al._, 2021] & fMRI & English & GloVe, BERT, GPT-2, Machine Translation, POS tasks & 6 & Moth Radio Hour & Ridge \\ \cline{2-8}  & [Reddy and Wehee, 2021] & fMRI & English & Constitutive, Basic syntax features and BERT & 8 & Harry Potter & Ridge \\ \cline{2-8}  & [Goldstein _et al._, 2022] & fMRI & English & GloVe, GPT-2 next word, pre-onset, post-onset & 8 & FC6G & \\ \cline{2-8}  & [Goldstein _et al._, 2022] & fMRI & English & BERT and GLUE tasks & 82 & Pereira \& Narratives & Ridge \\ \cline{2-8}  & [Mona _et al._, 2022] & fMRI & English & ENSTM, ELMo, Longformer & 82 & Narratives & Ridge \\ \cline{2-8}  & [Merlin and Toneva, 2022] & fMRI & English & BERT, Next word prediction, multi-word semantics, & 8 & Harry Potter & Ridge \\ \cline{2-8}  & [Toneva _et al._, 2022] & fMRI/ MEG & English & ELMo, BERT, Context Residuals & 8 & Harry Potter & Ridge \\ \cline{2-8}  & [Aw and Toneva, 2022] & fMRI & English & BART, Longformer, Long-T5, BigBird, and corresponding Booksum models as well & 8 & Passive reading & Ridge \\ \cline{2-8}  & [Zhang _et al._, 2022b] & fMRI & English & Noise Count & 19, 12 & Zhang & Ridge \\ \cline{2-8}  & [Mona _et al._, 2023a] & fMRI & English & Constitutive, Dependency trees, Basic syntax features and BERT & 82 & Narratives & Ridge \\ \cline{2-8}  & [Oota _et al._, 2023b] & MEG & English & Basic syntax features, GloVe and BERT & 8 & MEG-MASC & Ridge \\ \cline{2-8}  & [Tickute _et al._, 2023] & fMRI & English & BERT-Large, GPT-2 XL & 12 & Reading Sentences & Ridge \\ \cline{2-8}  & [Kauf _et al._, 2023] & fMRI & English & BERT-Large, GPT-2 XL & 12 & Pereira & Ridge \\ \cline{2-8}  & [Singh _et al._, 2023] & fMRI & English & BERT-Large, GPT-2 XL, Text Perturbations & 5 & Pereira & Ridge \\ \cline{2-8}  & [Wang _et al._, 2019] & fMRI & & 21 downstream vision tasks & 4 & BOLD 5000 & Ridge \\ \cline{2-8}  & [Kublius _et al._, 2019] & fMRI & CNN models AlexNet, ResNet, DenseNet & 7 & Algonauts & Ridge \\ \cline{2-8}  & [Dwivedi _et al._, 2021] & fMRI & & 21 downstream vision tasks & 4 & BOLD 5000 & Ridge \\ \cline{2-8}  & [Mona and Webbe, 2022] & fMRI & CNN models AlexNet & 4 & BOLD 5000 & Ridge \\ \cline{2-8}  & [Conwell _et al._, 2023] & fMRI & & CNN models AlexNet & 4 & BOLD 5000 & Ridge \\ \cline{2-8}  & [Miller _et al._, 2022] & fMRI & English & WA2Vec2.0 & 345 & Narratives & Ridge \\ \cline{2-8}  & [Vaidya _et al._, 2022] & fMRI & English & ARC, AST, Wav2Vec2.0, and HuBERT & 7 & Moth Radio Hour & Ridge \\ \cline{2-8}  & [Tuckute _et al._, 2022] & fMRI & English & 19 Speech Models (e.g. DeepSpeech, WA2Vec2.0, and VQ-VAE) & 19 & Passive listening & Ridge \\ \cline{2-8}  & [Oota _et al._, 2023c] & fMRI & English & 5 basic and 25 deep learning based speech models & 6 & Moth Radio Hour & Ridge \\ \cline{2-8}  & [Tera, CPC, APC, WA2Vec2.0, HuBERT, DistilHung & & & & \\ \cline{2-8}  & [Oota _et al._, 2023d] & fMRI & English & Wav2Vec2.0 and SUPERB tasks & 82 & Narratives & Ridge \\ \cline{2-8}  & [Dong and Toneva, 2023] & fMRI & English & Merlo Reserve & 5 & Neuumod & Ridge \\ \cline{2-8}  & [Popham _et al._, 2021] & fMRI & English & 985D Semantic Vector & 5 & Moth Radio Hour \& Short Movie & Ridge \\ \cline{2-8}  & [Oota _et al._, 2022d] & fMRI & English & CLIP, VisualBERT, LXMERT, CNNs and BERT & 5, 82 & Periez \& Narratives & Ridge \\ \cline{2-8}  & [La _et al._, 2022] & fMRI & English & BiVL & 5 & Pereira \& Short Movie Clips & Ridge \\ \cline{2-8}  & [Tang _et al._, 2023] & fMRI & English & BridgeTower & 5 & Moth Radio Hour \& Short Movie & Ridge \\ \cline{2-8}  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Summary of Representative Brain Encoding Studies
tured through complexity measures (syntactic surprisal, node count, word length, and word frequency), very few have studied the syntactic representations themselves. Studying syntactic representations using fMRI is difficult because: (1) representing syntactic structure in an embedding space is a non-trivial computational problem, and (2) the fMRI signal

Figure 7: Brain Encoding Survey Tree
is noisy. To overcome these limitations, Reddy et al. [2021] proposed syntactic structure embeddings that encode the syntactic information inherent in natural text that subjects read in the scanner. The results reveal that syntactic structure-based features explain additional variance in the brain activity of various parts of the language system, even after controlling for complexity metrics that capture the processing load. Toneva et al. [2021] further examined whether the representations obtained from a language model align with different language processing regions in a similar or different way.

**Linguistic properties in LMs and brains**: Understanding the reasons behind the observed similarities between language comprehension in LMs and brains can lead to more insights into both systems. Several works [Schwartz _et al._2019, Kumar _et al._2022, Aw and Toneva, 2022, Merlin and Toneva, 2022, Oota _et al._2022b] have found that using a fine-tuned BERT leads to improved brain predictions. However, it is not clear what type of information in the fine-tuned BERT model led to the improvement. It is unclear whether and how the two systems align in their information processing pipeline. Aw and Toneva [2022] used four pre-trained large language models (BART, Longformer Encoder Decoder, Big-Bird, and LongT5) and also trained them to improve their narrative understanding, using the method detailed in Figure 8. However, it is not understood whether prediction of the next word is necessary for the observed brain alignment or simply sufficient, and whether there are other shared mechanisms or information that is similarly important. Merlin and Toneva [2022] proposed two perturbations to pretrained language models that, when used together, can control for the effects of next word prediction and word-level semantics on the alignment with brain recordings. Specifically, they find that improvements in alignment with brain recordings in two language processing regions-Inferior Frontal Gyrus (IFG) and Angular Gyrus (AG)-are due to next word prediction and word-level semantics. However, what linguistic information actually underlies the observed alignment between brains and language models is not clear. Recently, Oota et al. [2022e] tested the effect of a range of linguistic properties (surface, syntactic and semantic) and found that the elimination of each linguistic property results in a significant decrease in brain alignment across all layers of BERT.

**Visual Encoding**: CNNs are currently the best class of models of the neural mechanisms of visual processing [Du _et al._2020, Beliy _et al._2019, Oota _et al._2019, Nishida _et al._2020]. How can we push these deeper CNN models to capture brain processing even more stringently? Continued architectural optimization on ImageNet alone no longer seems like a viable option. Kubilius et al. [2019] proposed a shallow recurrent anatomical network CORnet that follows neuroanatomy more closely than standard CNNs, and achieved the state-of-the-art results on the Brain-score benchmark. It has four computational areas, conceptualized as analogous to the ventral visual areas V1, V2, V4, and IT, and a linear category decoder that maps from the population of neurons in the model's last visual area to its behavioral choices.

Despite the effectiveness of CNNs, it is difficult to draw specific inferences about neural information processing using CNN- derived representations from a generic object-classification CNN. Hence, Wang et al. [2019] built encoding models with individual feature spaces obtained from 21 computer vision tasks. One of the main findings is that features from 3D tasks, compared to those from 2D tasks, predict a distinct part of visual cortex.

**Auditory Encoding**: Speech stimuli have mostly been represented using encodings of text transcriptions [Huth _et al._2016] or using basic features like phoneme rate, the sum of squared FFT coefficients [Pandey _et al._2022], etc. Text transcription-based methods ignore the raw audio-sensory information completely. The basic speech feature engineering method misses the benefits of transfer learning from rigorously pretrained speech DL models.

Recently, several researchers have used popular deep learning models such as APC [Chung _et al._2020], Wav2Vec2.0 [Baevski _et al._2020], HuBERT [Hsu _et al._2021], and Data2Vec [Baevski _et al._2022] for encoding speech stimuli. Millet et al. [2022] used a self-supervised learning model Wav2Vec2.0 to learn latent representations of the speech waveform similar to those of the human brain. They find that the functional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in the brain, and reveals the whole-brain organisation of speech processing with an unprecedented clarity. This means that the first transformer layers map onto the low-level auditory cortices (A1 and A2), the deeper layers (orange and red) map onto brain regions associated with higher-level processes (e.g. STS and IFG). Vaidya et al. [2022] present the first systematic study to bridge the gap between recent four self-supervised speech representation methods (APC, Wav2Vec, Wav2Vec2.0, and HuBERT) and computational models of the human auditory system. Similar to [Millet _et al._2022], they find that self-supervised speech models are the best models of auditory areas. Lower layers best modeled low-level areas, and upper-middle layers were most predictive of phonetic and semantic areas, while layer representations follow the accepted hierarchy of speech processing. Tuckute et al. [2022] analyzed 19 different speech models and find that some audio models derived in engineering contexts (model applications ranged from speech recognition and speech enhancement to audio captioning and audio source separation) produce poor predictions of auditory cortical responses, many task-optimized audio speech deep learning models outpredict a standard spectrotemporal model of the auditory cortex and exhibit hierarchical layer-region correspondence with auditory cortex.

**Multimodal Brain Encoding**: Multimodal stimuli can be best encoded using recently proposed deep learning based multimodal models. Oota et al. [2022d] experimented with multimodal models like Contrastive Language-Image Pre-training (CLIP), Learning Cross-Modality Encoder Representations from Transformers (LXMERT), and VisualBERT and found VisualBERT to the best. Similarly, Wang et al. [2022] find that multimodal models like CLIP better predict neural responses in visual cortex, since image captions typically contain the most semantically relevant information in an image for humans. [Dong and Toneva, 2023] present a systematic approach to probe multi-modal video Transformer model by leveraging neuroscientific evidence of multimodal information processing in the brain. The authors find that in
termediate layers of a multimodal video transformer are better at predicting multimodal brain activity than other layers, indicating that the intermediate layers encode the most brain-related properties of the video stimuli. Recently, [10] investigated a multimodal Transformer as the encoder architecture to extract the aligned concept representations for narrative stories and movies to model fMRI responses to naturalistic stories and movies, respectively. Since language and vision rely on similar concept representations, the authors perform a cross-modal experiment in which how well the language encoding models can predict movie-fMRI responses from narrative story features (story \(\rightarrow\) movie) and how well the vision encoding models can predict narrative story-fMRI responses from movie features (movie \(\rightarrow\) story). Overall, the authors find that cross-modality performance was higher for features extracted from multimodal transformers than for linearly aligned features extracted from unimodal transformers.

## 6 Brain Decoding

Decoding is the learning of the mapping from neural activations back to the stimulus domain. Figure 9 depicts the typical workflow for building an image/language decoder.

**Decoder Architectures**: In most cases, the stimulus representation is decoded using typical ridge regression models trained on each voxel and its 26 neighbors in 3D to predict each dimension of the stimulus representation. Also, decoding is usually performed using the most informative voxels [17]. In some cases, a fully connected layer [1] or a multi-layered perceptron [20] has been used. In some studies, when decoding is modeled as multi-class classification, Gaussian Naive Bayes [21, 22] and SVMs [14] have also been used for decoding. Figure 10 summarizes the literature related to various decoding solutions proposed in vision, auditory, and language domains.

**Decoding task settings**: The most common setting is to perform decoding to a vector representation using a stimuli of a single mode (visual, text or audio). Initial brain decoding experiments studied the recovery of simple concrete nouns and verbs from fMRI brain activity [23] where the subject watches either a picture or a word. Sun et al. [24] used several sentence representation models to associate brain activities with sentence stimulus, and found InferSent to perform the best. More work has focused on decoding the text passages instead of individual words [20].

Some studies have focused on multimodal stimuli based decoding where the goal is still to decode the text representation vector. For example, Pereira et al. [20] trained the decoder on imaging data of individual concepts, and showed that it can decode semantic vector representations from imaging data of sentences about a wide variety of both concrete and abstract topics from two separate datasets. Further, Oota

Figure 8: Comparison of brain recordings with language models trained on web corpora (LEFT) and language models trained on book stories (RIGHT) [20].
et al. [2022c] propose two novel brain decoding setups: (1) multi-view decoding (MVD) and (2) cross-view decoding (CVD). In MVD, the goal is to build an MV decoder that can take brain recordings for any view as input and predict the concept. In CVD, the goal is to train a model which takes brain recordings for one view as input and decodes a semantic vector representation of another view. Specifically, they study practically useful CVD tasks like image captioning, image tagging, keyword extraction, and sentence formation.

To understand application of Transformer models for decoding better, Gauthier et al. [2019] fine-tuned a pre-trained BERT on a variety of NLU tasks, asking which lead to im

\begin{table}
\begin{tabular}{|p{56.9pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|p{113.8pt}|} \hline
**Stimuli** & **Authors** & **Dataset Type** & **Lang.** & **Stimulus Representations** & **S** & **Dataset** & **Model** \\ \hline \multirow{6}{*}{[30]} & Pereira _et al._, 2018 & fMRI & English & Word2Vec, GloVe, BERT & 17 & Pereira & Ridge \\ \cline{2-9}  & Wang _et al._, 2020 & fMRI & English & BERT, RoBERTa & 6 & Pereira & Ridge \\ \cline{2-9}  & Oota _et al._, 2022c & fMRI & English & GloVe, BERT, RoBERTa & 17 & Pereira & Ridge \\ \cline{2-9}  & Tang _et al._, 2022 & fMRI & English & GPT, fine-tuned GPT on Reddit comments and autobiographical stories & 7 & Moth Radio Hour & Ridge \\ \hline \multirow{6}{*}{[30]} & Beliy _et al._, 2019 & fMRI & \multicolumn{2}{p{113.8pt}|}{End-to-End Encoder-Decoder, Decoder-Encoder, AlexNet} & \multicolumn{1}{p{113.8pt}|}{} & \multicolumn{1}{p{113.8pt}|}{} & \multicolumn{1}{p{113.8pt}|}{} & \multicolumn{1}{p{113.8pt}|}{} \\ \cline{2-9}  & Takaei and Nishimoto, 2022 & fMRI & Latent Diffusion Model, CLIP & 4 & NSD & Ridge \\ \cline{2-9}  & Ozcelik and VanRullen, 2023 & fMRI & \multicolumn{2}{p{113.8pt}|}{VDVAE, Latent Diffusion Model} & \multicolumn{1}{p{113.8pt}|}{7} & NSD & \multicolumn{1}{p{113.8pt}|}{} \\ \cline{2-9}  & Chen _et al._, 2023b & fMRI & \multicolumn{2}{p{113.8pt}|}{Latent Diffusion Model, CLIP} & \multicolumn{1}{p{113.8pt}|}{3} & \multicolumn{1}{p{113.8pt}|}{HCP fMRI-Video-Dataset} & \multicolumn{1}{p{113.8pt}|}{} & \multicolumn{1}{p{113.8pt}|}{} \\ \cline{2-9}  & Defossez _et al._, 2022 & MEG,EEG & English & \multicolumn{1}{p{113.8pt}|}{MEL Spectrogram, Wav2Vec2.0} & \multicolumn{1}{p{113.8pt}|}{169} & MEG-MASC & Ridge, CLIP \\ \cline{2-9}  & (Gwilliams _et al._, 2022) & MEG & English & Phonemes & 7 & MEG-MASC & \\ \hline \end{tabular}
\end{table}
Table 3: Summary of Representative Brain Decoding Studies

Figure 10: Brain Decoding Survey Tree

Figure 9: Schema for Brain Decoding. LEFT: Image decoder _[Smith et al., 2011]_, RIGHT: Language Decoder _[Wang et al., 2019]_
provements in brain-decoding performance. They find that tasks which produce syntax-light representations yield significant improvements in brain decoding performance. Toneva et al. (2019) study how representations of various Transformer models differ across layer depth, context length, and attention type.

Some studies have attempted to reconstruct words (Affolter et al., 2020), continuous language (Tang et al., 2022), images (Du et al., 2020; Beliy et al., 2019; Fang et al., 2020; Lin et al., 2022), speech (Defossez et al., 2022) or question-answer speech dialogues (Moses et al., 2019) rather than just predicting a semantic vector representation. Lastly, some studies have focused on reconstructing personal imagined experiences (Berezutskaya et al., 2020) or application-based decoding like using brain activity scanned during a picture-based mechanical engineering task to predict individuals' physics/engineering exam results (Cetron et al., 2019) and reflecting whether current thoughts are detailed, correspond to the past or future, are verbal or in images (Smallwood and Schooler, 2015). Table 3 aggregates the brain decoding literature along different stimulus domains such as textual, visual, and audio.

## 7 Conclusion, Limitations, and Future Trends

**Conclusion** In this paper, we surveyed important datasets, stimulus representations, brain encoding and brain decoding methods across different modalities. A glimpse of how deep learning solutions throw light on putative brain computations is given.

**Limitations** Naturalistic datasets of passive reading/listening offer ecologically realistic settings for investigating brain function. However, the lack of a task (as in a controlled psycholinguistic experiment) that probes the participant's understanding of the narrative limits the inferences that can be made on what the participant's brain is actually engaged in while passively following the stimuli. This becomes even more important when multi-lingual, multispectral participants process stimuli in L2 language or script - it is unclear if the brain activity reflects the processing of L2 or active suppression L1 while focusing on L2 (Malik-Moraleda et al., 2022).

**Future Trends** Some of the future areas of work in this field are as follows: (1) While there is work on the text, understanding the similarity in information processing between visual/speech/multimodal models versus natural brain systems remains an open area. (2) Decoding to actual multimodal stimuli seems feasible thanks to recent advances in generation using deep learning models. (3) Deeper understanding of the degree to which damage to different parts of the human brain could lead to the degradation of cognitive skills. (4) How can we train artificial neural networks in novel self-supervised ways such that they compose word meanings or comprehend images and speech like a human brain? (5) How can we leverage improved neuroscience understanding to suggest changes in proposed artificial neural network architectures to make them more robust and accurate? We hope that this survey motivates research along the above directions.
