Promises and pitfalls of deep neural networks in neuroimaging-based
  psychiatric research
By promising more accurate diagnostics and individual treatment
recommendations, deep neural networks and in particular convolutional neural
networks have advanced to a powerful tool in medical imaging. Here, we first
give an introduction into methodological key concepts and resulting
methodological promises including representation and transfer learning, as well
as modelling domain-specific priors. After reviewing recent applications within
neuroimaging-based psychiatric research, such as the diagnosis of psychiatric
diseases, delineation of disease subtypes, normative modeling, and the
development of neuroimaging biomarkers, we discuss current challenges. This
includes for example the difficulty of training models on small, heterogeneous
and biased data sets, the lack of validity of clinical labels, algorithmic
bias, and the influence of confounding variables.
Fabian Eitel, Marc-André Schulz, Moritz Seiler, Henrik Walter, Kerstin Ritter
# Promises and pitfalls of deep neural networks in neuroimaging-based psychiatric research

###### Abstract

By promising more accurate diagnostics and individual treatment recommendations, deep neural networks and in particular convolutional neural networks have advanced to a powerful tool in medical imaging. Here, we first give an introduction into methodological key concepts and resulting methodological promises including representation and transfer learning, as well as modelling domain-specific priors. After reviewing recent applications within neuroimaging-based psychiatric research, such as the diagnosis of psychiatric diseases, delineation of disease subtypes, normative modeling, and the development of neuroimaging biomarkers, we discuss current challenges. This includes for example the difficulty of training models on small, heterogeneous and biased data sets, the lack of validity of clinical labels, algorithmic bias, and the influence of confounding variables.

keywords: Deep learning, Convolutional Neural Networks, Psychiatry, Neuroimaging, MRI +
Footnote †: journal: Experimental Neurology

## 1 Introduction

By setting new standards in image and speech recognition tasks, deep neural networks advanced to a key technology in research [1; 2]. In medical imaging, a number of applications have reached or even exceeded human
level performance, especially in cases where the sample size was rather large (\(N>15000\)) and the prediction task was well defined (i.e., the pathology is clearly identifiable in the imaging data). This includes, for example, the detection of diabetes from fundus images, skin cancer from photographs, and pneumonia from chest X-rays [3; 4]. Given these success stories, it has been asked to what extent deep neural networks are also capable of identifying brain diseases based on neuroimaging data, e.g., data obtained from magnetic resonance imaging (MRI; [5]). While most neurological diseases are associated with measurable brain damage, such as atrophy or lesions, that is visible in structural MRI, brain alterations in psychiatric disorders are considered to be more subtle, mostly functional and still under debate [6; 7]. Nevertheless, in the last two decades, neuroimaging has become one of the cornerstones in the search for biomarkers that explain neurobiological variance associated with psychiatric disease [7; 8]. Promising biomarkers include regional atrophy and reduced cortical thickness [9; 10], brain age [11] as well as alterations in task-induced functional MRI activity or resting-state connectivity [12]; for a review, see Lui et al. [7]. However, such biomarkers are not necessarily disease-specific and have a high overlap across psychiatric (and neurological) diseases [13].

In addition to traditional statistical analyses, where mean differences between groups (e.g., patients and controls) have been investigated, neuroimaging-based biomarkers have also been employed in machine learning analyses with the goal to draw conclusions about individual subjects [14; 15; 16; 17; 18]. By being settled in the framework of precision medicine, machine learning is considered to provide a huge promise for transforming healthcare in general [19] and psychiatry in particular [20]. While machine learning approaches can been applied to different kinds of data including deep phenotyping, genetics, and metabolomics, they are in particular a suitable candidate for analyzing neuroimaging data due to their ability to flexibly handle high-dimensional data where the number of variables (e.g., voxels or regional volumes) commonly exceeds the number of samples (see Table 1 for a brief description of machine learning related terms). The learning settings range from automatic disease diagnosis and prognosis to subtype discovery and prediction of treatment outcome [16; 21; 22; 23].

Most classical machine learning algorithms learn comparably simple input-output functions. They therefore have difficulties in processing raw MRI data and rely on hand-crafted features such as cortical thickness or connectivity matrices [1; 24; 25; 26]. In psychiatry, the design of those features is build on
neurobiological disease models and usually reflects only one part of disease pathology, but does not meet the multifactorial nature of many psychiatric diseases [13]. Deep learning approaches, on the other hand, can learn hierarchical representations directly from the raw data and thus are capable of solving more complex problems [1; 2]. Major breakthroughs have been achieved in natural image recognition by using a specialized deep learning architecture called convolutional neural networks (CNNs); which are artificial neural networks (ANNs) that exploit the structural dependencies of data coming from arrays such as image, audio, and video signals [1]. The key idea behind CNNs is inspired by the mechanism of receptive fields in the primate's visual cortex and relates to the application of local convolutional filters in combination with downsampling [27; 2]. In addition to their utilization in industry for diverse image and speech recognition tasks, they have advanced to a strong instrument for analyzing medical imaging data, including some as

\begin{table}
\begin{tabular}{p{142.3pt} p{142.3pt}} \hline
**Term** & **Description** \\ \hline \multicolumn{3}{p{142.3pt}}{**Machine learning**} \\ \multicolumn{3}{p{142.3pt}}{**Deep learning**} \\ Artificial neural network (ANN)/ & \\ Deep neural network (DNN) & \\ \multicolumn{3}{p{142.3pt}}{_Fully-connected neural network_} \\ \multicolumn{3}{p{142.3pt}}{_Convolutional neural network (CNN)_} \\ \multicolumn{3}{p{142.3pt}}{_Recurrent neural network (RNN)_} \\ \multicolumn{3}{p{142.3pt}}{_Recurrent neural network (RNN)_} \\ \multicolumn{3}{p{142.3pt}}{ANN which introduces recurrent connections between the artificial neurons; and is, thus, primarily used on sequential data.} \\ \hline \end{tabular}
\end{table}
Table 1: Description and interdependencies of central terms in machine learning and deep learning.
pects of neuroimaging data [5; 24; 28]. Besides promising preliminary results in the field of neurology and psychiatry [24; 29], a number of open challenges still exist. On the one hand, these challenges can be technology-related, such as the difficulty of deep neural networks to learn robust models on small, heterogeneous data sets or to give meaningful explanations for complex model decisions [30; 31; 32; 33]. On the other hand, these challenges are driven by factors that are controversial within psychiatry itself, e.g., the heterogeneity of psychiatric diseases in their clinical presentation and the reliance on clinical symptoms rather than neurobiological substrates for establishing disease categories [34; 35; 36].

In this review, we will first give a short introduction into MRI and its role in psychiatry (section 2) and then introduce the basic concepts of machine and deep learning with a focus on CNNs as the most popular type of deep neural network currently applied to neuroimaging data (section 3; please see Durstewitz et al. [20] for an introduction into recurrent neural networks for analyzing temporal data in psychiatry). Both concepts are needed in order to understand the following sections; readers interested in more general introductions of machine learning in medicine and psychiatry are referred to [17; 37; 38]. In section 4, we will outline three methodological promises for the use of deep learning in psychiatry, namely representation learning, transfer learning, and the use of model architectures incorporating neuroimaging-specific priors (a.k.a. inductive bias). In sections 5 and 6, we will first present promising application scenarios within psychiatry, namely automated diagnostics, subtype identification, and development of new biomarkers; and then review existing applications for major psychiatric disorders including Alzheimer's disease, schizophrenia, substance abuse, neurodevelopmental disorders, and internalizing disorders such as depression, anxiety, and obsessive-compulsive disorder. Finally, current challenges and implications in the use of deep neural networks will be discussed in section 7.

## 2 Magnetic Resonance Imaging (MRI) and its role in psychiatry

After a short introduction into the role of MRI in psychiatric research, we will present the most common MRI modalities (and preprocessing steps) that have been used at the intersection of psychiatry and machine learning. MRI is a non-invasive medical imaging technique which has become an important tool in the diagnosis and monitoring of neurological diseases including
stroke, multiple sclerosis and tumor detection [39; 40]. In psychiatry, MRI data is mainly used to rule out other causes (e.g., brain tumors) that might explain changes in behavior and feelings but also became a cornerstone for investigating neurobiological correlates in psychiatric diseases [7]. The focus in psychiatric research, using structural MRI (sMRI), is on high-resolution T1-weighted MRI data (e.g., Magnetization Prepared RApid Gradient Echo (MPRAGE) [41]) providing the best contrast between grey and white matter which is therefore useful for measuring cortical thickness and regional atrophy in the brain. White matter abnormalities can be identified using T2-weighted imaging (e.g., Fluid-Attenuated Inversion Recovery (FLAIR)), magnetization transfer imaging, and diffusion tensor imaging but thus far play only a minor role in machine learning analyses. Although structural changes in patients with psychiatric diseases are considered to be small and difficult to detect by visual inspection, some relative consistent findings have been reported, e.g., smaller hippocampi in patients with depression or larger ventricles in patients with schizophrenia [42; 43; 7].

Functional MRI (fMRI), in contrast to sMRI, measures brain activity by changes in blood oxygenation and blood flow as a result of neural activity (the so-called blood oxygen level dependent (BOLD) response [44]). Whereas in task-based fMRI, the activation during specific tasks (e.g., processing of emotional faces) is examined, in resting-state fMRI it is the intrinsic network activity during rest (i.e., no task) that is investigated. Specifically, resting-state functional connectivity matrices capturing spatial correlations in BOLD fluctuations are thought to give important insights about the functional organization of the healthy and diseased brains and have been employed in machine and deep learning [45; 46]. However, the within- and between-subject variability is considerable and makes finding robust functional biomarkers challenging [47].

Most studies applying CNNs on MRI so far have focused on sMRI (in particular T1-weighted MPRAGE) rather than fMRI. There are two main reasons for this focus. First, sMRI shares more properties with the modalities where CNNs have been successful (e.g., natural images); fMRI, in contrast, does not contain clearly visible hierarchical objects such as brain regions with high-frequency edges. Second, given that fMRI is 4-dimensional, it makes computation very expensive on the raw data (see section 7.2.3), or leads to the aggregation across time or space, potentially losing relevant information. In the future, advanced MRI techniques such as quantitative MRI may play an important role [28].
To study the same brain regions across subjects, MRI data are usually spatially normalized to a common space (e.g., the Montreal Neurological Institute [MNI] template) [48]. Deep learning, on the other hand, does not necessarily require spatial normalization but it may help in reducing variance in the data especially in light of small sample sizes. Therefore, most deep learning studies at least linearly register the data to MNI space [31]. Additionally, deep learning algorithms have also been used to perform nonlinear registration by themselves (for an overview of deep learning applications in MRI aside from clinical applications, see [28]).

To analyze diseases trans-diagnostically and on a larger scale, a number of large multi-site imaging cohorts have been elicited (e.g., HCP2, UK biobank3, ENIGMA4, and IMAGEN5). The existence of large data bases is essential for the success of machine and deep learning techniques in psychiatric research (see section 7).

Footnote 2: [http://www.humanconnectomeproject.org/](http://www.humanconnectomeproject.org/)

Footnote 3: [https://www.ukbiobank.ac.uk/](https://www.ukbiobank.ac.uk/)

Footnote 4: [http://enigma.ini.usc.edu/](http://enigma.ini.usc.edu/)

Footnote 5: [https://imagen-europe.com/resources/imagen-dataset/](https://imagen-europe.com/resources/imagen-dataset/)

## 3 Key ideas of machine and deep learning

To make the methodological promises and challenges of deep learning technology within psychiatric research more accessible, we first briefly introduce the reader to some fundamental concepts of the more general field of machine learning, including different learning settings and validation schemes. We then concentrate on the sub-field of deep learning, which is the focus of the later explanations. Therefore, we briefly describe the underlying principles of deep learning, namely artificial neural networks (ANNs), and convolutional neural networks (CNNs) in particular since this type of ANN is predominantly used to analyze image data, as well as methods to explain individual predictions of these types of models. While experienced readers might skip this section, novice readers are encouraged to consider [49; 50] for a comprehensive coverage of the fundamentals of machine learning and deep learning or [16; 17; 37; 38] for its applications to medicine.
### Machine learning

#### 3.1.1 Basic concepts

Machine learning is an interdisciplinary field at the intersection of computer science, statistics, mathematics, and others, which in general defines algorithms that learn from data. In contrast to traditional rule-based systems in artificial intelligence, where human knowledge is encoded in rigid rules, machine learning algorithms can learn to perform a specific task (e.g., discriminating between images of dogs and cats) without being explicitly programmed [49; 51] (see Figure 1). This flexibility allows learning algorithms to be applied to complex problems which are not feasible for hand-tuned approaches [52; 53; 54].

Most machine learning algorithms can be categorized into two main paradigms: supervised learning and unsupervised learning (see Figure 2). In supervised learning, a data set contains examples of input-output pairs, and a function that maps the input features to the output labels is learned. In psychiatry, the input features could be a set of biomarkers (e.g., volumes of particular brain structures), while the label is given by group membership (e.g., patients or healthy controls), disease severity (e.g., the extent of positive symptoms in schizophrenia), prognosis, or treatment outcome (e.g.,

Figure 1: Artificial intelligence became popular in the mid’ 1900s using large tables of hand designed decision strategies, which for example beat human players in chess. Starting in the 1980s the field of machine learning gained traction and showed that the previously hand designed strategies could be learned from data. The so far unpopular methods around ANNs had a breakthrough in the early 2010s when large data sets and computing power allowed them to become deeper and more complex, and thus outperformed other methods in many disciplines including image and speech recognition.
response to cognitive-behavioral psychotherapy). After learning this mapping, the algorithm can then be used to make statements about new, unseen subjects. If the output labels are discrete classes, the prediction task is called classification, while for continuous labels, it is known as regression. In contrast, in unsupervised learning, the data set only contains input features without any information about the labels. Often, the goal of this paradigm is to discover a compact and more informative representation of the data. An example here is the identification of disease clusters or subtypes solely based on neuroimaging data.

To enable a machine learning algorithm to learn performing such tasks from data, four components are usually used to define the algorithm: (1) a data set (which is split into training and test set), (2) a statistical model to learn an approximate representation of the data, (3) a loss function to measure the goodness of the model, and (4) an optimization procedure to alter the model parameters in a way that the loss is minimized. Concerning the statistical models, machine learning comprises a large number of model classes, of which in particular, support vector machines (SVMs), Gaussian processes, random forests, logistic regression, and ANNs are common choices

Figure 2: Examples of supervised and unsupervised machine learning. In supervised machine learning such as classification (left) each brain MRI is a data point (represented as circles and crosses) and has a pre-determined class such as patient or control. The task is to find a model (black line) that discriminates between those classes. In unsupervised machine learning such as clustering (right) each brain MRI is again a data point but conversely does not have a class label. Here, the task is to find clusters in the data which can be well separated. These clusters could for example represent subgroups of a disease.
in the neuroimaging domain [16; 49; 55].

#### 3.1.2 Validation and performance metrics

The central challenge in machine learning is the ability of the learning algorithm to perform well on new, unobserved data; this is also called generalization [56]. Therefore, a quantitative measure of predictive performance, such as the model error, is required to test this ability. Additionally, according to the 'No Free Lunch Theorem' [57] in machine learning, there is no single model which performs universally best on all possible data sets. Therefore, two different procedures are performed to identify the best performing model on a data set which furthermore generalizes well to unseen data: model selection and model assessment [56].

Model selection is the process of selecting the best of many competing models. Since most machine learning algorithms have hyperparameters which must be set manually, finding the optimal hyperparameter configuration is performed as part of this model selection step. This step is crucial to adjust the expressive capacity (often called expressivity) of the model to match the complexity of the task. A mismatch between model capacity and task complexity is a common problem in machine learning. In particular, overfitting, in which the model learns noisy and too complex patterns in the data set without improving the generalization, results from too much model capacity for the task, and therefore, requires regularization to penalize this capacity. In the model assessment, the generalization ability of a model is evaluated based on the generalization error of the model.

This generalization error of the learning algorithm must be approximated, which is usually done by the holdout validation method. The data set is split by randomly partitioning it into three disjoint subsets: a training set, a validation set, and a test set. These subsets are assumed to be independent and identically distributed (i.i.d.) samples from an unknown data generating distribution. The i.i.d. assumption is crucial to receive an unbiased estimate of the generalization error, although violations are common in neuroimaging-based machine learning [58], for example, resulting from data leakage by splitting clinical data containing repeated measurements on the observation-level rather than the subject-level [59]. Based on the training set, different models are fitted and evaluated on the validation set by using the model errors as an estimate of the generalization error. The best model is then selected and tested on the holdout test set using the test error as an estimate
of the generalization error of the model.6

Footnote 6: Please note that the model selection error on the validation set will underestimate the true error of the models and should therefore not be used as an estimate of the models’ generalization ability.

In practice, especially in clinical applications, the data set sizes are usually small such that the single train-validation-test split might result in an inaccurate estimate of the generalization error [56]. The most common technique to address this issue is cross-validation (CV). In CV, the data set is partitioned into several different subsets, so-called folds. By choosing one of the folds as the test set, while the others are used as a training set, it is possible to repeat this training and testing computation until each fold has been used as a test set. The estimate of the generalization error is then an average over the errors on the test set of each repetition. A special case of CV is leave-one-out CV, where the number of folds is equivalent to the number of samples in the data set. This form of CV has been used in clinical research (e.g., [60; 61]) due to particularly small data set sizes. However, Kohavi [62] has empirically shown that although leave-one-out estimates are almost unbiased, the variance of the estimates can be large. Besides, it has been argued that repeated random splits lead to more stable results [63]. More recently, in multi-site studies, so-called leave-one-site-out CV is used (e.g, [64; 65]). Here, the data from different sites are pooled so that the data of each site being a separate fold in the CV framework. If both model selection and model evaluation are performed, CV needs to be extended to receive an unbiased estimate of the generalization error [66]. This extension is called nested CV and is merely a nesting of two CV loops, performing model selection in the inner loop while evaluating the final model in the outer loop.

The predictive performance of the final model is then reported using a performance metric. Common metrics include (balanced) accuracy, sensitivity and specificity, area under the receiver operating characteristic curve (ROC AUC), F-1 score for classification, and mean-squared-error (MSE) for regression. While some metrics such as accuracy might be strongly influenced by the class distributions, others try to correct for that, such as balanced accuracy, ROC AUC and F-1 score.
### Deep learning

#### 3.2.1 Artificial neural networks

Deep learning is a particular subtype of machine learning which is based on ANNs. These ANNs describe a large class of models that consists of a collection of connected units or artificial neurons. Although these models have recently gained attraction after achieving state-of-the-art results in fields such as computer vision [67] or natural language processing [68], the origins of these models date back to the 1940s and were used to study biologically inspired representations of information processing [69]. To describe the working mechanisms of ANNs, we first introduce the most basic type of these networks, the multilayer perceptron (MLP) or fully-connected neural network. This type is a feed-forward neural network which is constructed by connecting groups of artificial neurons or units organized within layers (see Figure 3). Unlike recurrent neural networks [70; 71], feed-forward neural networks only allow the input information to flow in one direction, from input to output, without loops between the neuron connections. For this, an MLP consists of an input layer, one or more hidden layers and an output layer. While the input layer passes the input features to the network using one unit for each input feature (e.g., one unit for every voxel in an MR image), the output layer completes the task by outputting a prediction, with the number of units depending on the task (e.g., one unit for patients and one unit for healthy controls in a binary classification task). The hidden layers define the capacity of ANNs, with the number of units in a hidden layer defining the width and the number of hidden layers defining the depth of the network [50]. An ANN architecture using more than one hidden layer is called a deep neural network; the name deep learning is derived from this terminology and therefore serves as a collective term for ANNs with multiple hidden layers.

In an MLP, a fully-connected layer connects each artificial neuron in a layer with every artificial neuron in the previous layer, enabling the flow of information between sets of artificial neurons. Each unit thus receives an input from every artificial neuron in the previous layer connected to it to compute a weighted sum of these connections using the associated weight parameters. Therefore, the weights can be seen as the relative strength or importance of the connections between consecutive units. Furthermore, inside a unit, a nonlinear activation function (e.g., rectifier function [72] or logistic sigmoid function) is applied to the previously computed weighted sum to nonlinearly transform the input to an output of the unit. These outputs then serve as
inputs to the units in the subsequent layer, creating a nested chain structure of connected units between layers. Therefore, an MLP can be understood as a complex mathematical function composed of many simpler functions.

Regarding the expressive power, the Universal Approximation Theorem (e.g., [73]) states that a fully-connected neural network with one hidden layer and a sufficient number of hidden units can theoretically approximate any continuous function.7 However, the required width of the hidden layer may be infeasibly large such that the fully-connected neural network may fail to learn and generalize correctly. Therefore, the capacity of an ANN model is typically increased by adding depth to the network [75]. This leads to a series of nonlinear transformations which enable an ANN, or rather a deep neural network, to hierarchically learn multiple levels of representations from raw data to abstract representations. Unlike traditional machine learning methods, which often rely on the discriminatory power of handcrafted features and previous feature engineering, ANNs can learn feature representations from raw data by using a general learning procedure [76]. Examples of feature engineering include the delineation of lesions to calculate the total lesion load for use in a classifier for multiple sclerosis or computing a gray matter segmentation, dividing it into separate brain regions and using the mean gray matter density per region in a classifier for schizophrenia. Artificial neural networks, however, are able to process the raw images and learn similar or

Figure 3: Architecture of a fully-connected network with interconnected groups of neurons in the input layer, a single hidden layer and the output layer.
potentially more powerful features themselves.

To learn the optimal model parameters (i.e., weights), ANNs rely on gradient-based optimization using the backpropagation algorithm [77]. During the training phase, the backpropagation algorithm uses the chain-rule to compute the partial derivatives of the loss function with respect to the weight parameters to further optimize the weight parameters using a gradient-based optimization procedure. This whole procedure is then repeated many times to find the optimal parameter configuration of the artificial neural network model. An essential aspect of the tremendous success of ANNs is the efficient implementation of this learning procedure using highly optimized programming libraries such as PyTorch [78], Tensorflow [79] or Theano [80] and the support of graphical processing units (GPUs), which enable an efficient parallelized computation of this procedure [1].

Despite the success and the representational power of ANNs, the learning process introduces certain challenges to this class of machine learning algorithms. First, although attempts at automated approaches [81] exist, the specification of an ANN model requires manually determined design choices such as the number of hidden layers or hidden units in a layer, regularization techniques, as well as further hyperparameters. These choices are mostly guided by background knowledge and experimentation, making training an ANN quite an art. Due to this issue, artificial neural network models are usually overparametrized using thousands or even millions of parameters and are prone to overfit the training data. To address the risk of overfitting, regularization techniques such as dropout [82] or weight decay [83] are used in the training process to reduce the network's capacity. Second, the loss surface, especially of deep neural network models, is highly non-convex and possesses many local minima [84]. Thus, the gradient-based optimization algorithms do not guarantee to converge to the global minimum of the loss surface such that different starting configurations of the network are required. However, Choromanska et al. [85] have shown that although deep neural networks mostly converge to local minima, the resulting models often generalize well to new data, so the problem of local minima is negligible.

#### 3.2.2 Convolutional neural networks

Although a fully-connected neural network can theoretically approximate any continuous function, the wiring of the neurons can lead to certain drawbacks, especially on grid-like topologies such as MR images [86]. First, the number of parameters to be learned for this often high-dimensional input
data can be very large due to the many required connections between the neurons in the input and the subsequent hidden layer. Second, for grid-like data such as images, the pixels or voxels of the input images are treated independently, ignoring the spatial information in form of correlations between pixels and translation invariance of objects in the image. Convolutional neural networks (CNNs) [87] use the mathematical convolution operation within a convolutional layer to address these drawbacks by introducing a biologically inspired local receptive field [27], weight sharing, and downsampling.

A convolution is a linear mathematical operation (see Figure 4) which computes the weighted sum of an input and the weight parameters of a function, also known as a kernel or filter, at every location in the input space to produce a feature map. Thus, instead of connected groups of neurons, each unit in the feature map is only locally connected to a specific region, also known as a receptive field, in the input space. This enables the extraction of local features, and therefore, preserves the spatial structure of the input space. Moreover, the units in the feature map are restricted to share the same weight parameters for every region in the input space. As a result, the number of parameters to be learnt and thereby the complexity is reduced to a set of parameters equivalent to the size of the kernel. Since the convolution is a linear operation, activation functions are again used to nonlinearly transform the resulting feature maps. Typically, in a CNN, a convolution layer is followed by a pooling layer that replaces the values of a local region in the feature map with summary statistics to reduce the dimensionality of the input. This results in translation invariance, which is an important property when the presence of a particular feature is of importance but not its

Figure 4: Example of a 2-dimensional convolution operation.
location. In a typical CNN architecture, successive pairs of convolutional layers and pooling layers are used to learn different levels of representations of the data, e.g. from an MR image (from edges and blobs to more abstract concepts such as lesions or atrophy). The final layers of a CNN architecture are typically fully-connected layers to compute the output predictions for a specific task. Most studies applying neural networks for classification in MRI follow the architectures of successful computer vision models such as DenseNets [88], ResNets [89] or VGGNet [90]. Yet, recently researchers have started designing architectures specialized for the properties of raw data or functional connectivity matrices introducing inductive biases (see section 4.3). For applications of CNNs to neuroimaging data, a couple of specialized software libraries exist (e.g., PHOTONAI [91], DeepNeuro [92]).

#### 3.2.3 Explaining model predictions

Despite their recent success, ANNs, and in particular, deep neural networks, are often criticized for being black-boxes [93; 94]. Although deep neural networks are mostly deterministic models, the large parameter space of these models, often comprising thousands or millions of parameters, and highly nonlinear interactions make it difficult to understand the relationship between the inputs and the outputs from a human perspective. This is problematic particularly in risk-sensitive disciplines such as medicine, where a transparent and verifiable decision-making process is crucial [95]. To address this lack of transparency, different methodological approaches have been proposed to understand the behaviour of machine learning models in general (e.g., LIME [96], SHAP [97]) and neural networks in particular [98; 99; 100; 101].

Various attribution methods have been introduced to explain neural network predictions in image classification [102], which attempt to determine the attribution or relevance of each input feature to the predicted output of a neural network. The resulting attributions are then displayed in a heatmap to visualize which input features have positively or negatively influenced the prediction. The proposed attribution methods can be categorized into two different methodological approaches: perturbation-based methods and backpropagation-based methods. Perturbation-based methods assign attribution values to input features directly by modifying the input space through removing, masking, or altering features and measuring the difference between the predictions based on the modified and the original inputs. An example is the occlusion method proposed by Zeiler and Fergus [103], which system
atically masks a region in the input image to observe potential changes in the target class probability. Rieke et al. [104] extended this method to an atlas-based occlusion for MR images. Backpropagation-based methods, on the other hand, assign the attribution for all input features using the backpropagation algorithm to either compute partial derivatives of the outputs with respect to the input features [105; 106] or to backpropagate a relevance score through the network [107]. Although the visualizations do not directly relate to output variations, these approaches are computational less expensive compared to perturbation-based methods and thus make them more suitable for high-dimensional MR images [30]. A popular method of this kind is layer-wise relevance propagation (LRP; [108]), in which a relevance score is introduced to the output layer, which is then then propagated back to the input layer using a modified backpropagation algorithm.

## 4 Methodological promises

Deep neural networks present advantages over classical statistics and machine learning methods that are particularly promising for neuroimaging data in psychiatry. In this section, we discuss how deep neural networks can provide meaningful intermediate representations, which facilitate discovery of disease subtypes. Furthermore, we show how these intermediate representations enable transfer learning to deal with comparably small sample sizes in neuroimaging. Lastly, we discuss how specialized architectures (i.e., inductive biases) can more efficiently exploit the structure in neuroimaging data. These three advantages lay the methodological foundation for the application scenarios in section 5.

### Representation learning

A central advantage of deep neural networks is their ability to perform automatic feature engineering on raw or minimally processed data (e.g., spatially normalized MRI data) as shown in Figure 5. These automatically learned features can be interpreted as a new view or representation of the input data and can be extracted by reading out the activations of neurons in a given layer [76]. Thus, any given data point (e.g., an MR volume of a particular subject) can be described by the activation profile that it generates in a particular layer of a deep neural network.

In analogy to how a CNN trained to classify natural images will yield successive layers of intermediate representations detecting edges, textures,
and, finally, whole objects [109], one expects a deep neural network trained on neuroimaging data to provide a hierarchy of intermediate representations reflecting brain structure or function (e.g., low-level neural structures such as boundaries between grey and white matter, focal abnormalities such as lesions or atrophy, and disease profiles [110]).

To create representations that are useful for a specific research question, there exist two general approaches: supervised and unsupervised represen

Figure 5: Deep neural networks have the ability to automatically perform feature extraction (bottom). Automatic feature extraction allows the automatic design and selection (“learning”) of representative features. In contrast (top), traditional machine learning models mostly require manual extraction and selection of features. These features are typically designed (“engineered”) by human experts and therefore introduce a bias about the experts’ knowledge into the features. Furthermore, these methods require additional tools (e.g., Statistical Parametric Mapping (SPM) or Freesurfer) or manual effort (e.g., manual segmentation).
tation learning. In supervised representation learning, models are trained to predict a target variable, and successive layers of the model discard more and more prediction-irrelevant information while retaining and recombining prediction-relevant information into increasingly abstract representations of the input data (for a formal discussion, see e.g., Tishby and Zaslavsky [111]). In contrast, unsupervised representation learning operates exclusively on the input data. One example are autoencoders that transform the data to pass through a low-dimensional bottleneck and then try to reconstruct the original data from the compressed intermediate representation [112; 50]. This approach implicitly assumes that a compressed representation of the data is likely to mirror abstract higher-order concepts or latent variables that generated the data. Recently popularized "self-supervised" learning extends this approach to other auxiliary tasks for learning meaningful representations, such as predicting the next word in a sentence [113], color channels from a grayscale image [114], or anatomical segmentations from an MR image [115]. Intermediate representations in deep neural networks can be constrained to have potentially useful mathematical properties such as normality and independence [116], or to selectively disentangle factors of variation [117]. They are often designed to reduce the dimensionality of the input data (e.g., high-dimensional neuroimaging data) for further analyses, such as clustering (section 5.2) or to test scientific hypotheses (section 5.3).

### Transfer learning

In many domains, for instance in computer vision, there exist large general purpose data sets (e.g., ImageNet) while data sets for specific applications are often prohibitively small. Recently, a similar situation has arisen in neuroimaging, where small clinical studies are now augmented by large-scale data collection initiatives such as the UK Biobank or ENIGMA. The existence of intermediate representations in deep neural networks can allow for carrying insights gained in large general purpose data sets over to small sample clinical settings [118; 119]. Models can be split at the level of an intermediate representation, allowing to _pre-train_ the lower layers of the networks on a large data set, and then _fine-tune_ the higher layers on the target data set. This approach is called transfer learning [120].

The motivation for transfer learning in CNNs is simple. Instead of initializing weights randomly, they are initialized based on a related data set. Ideally, the data used for pre-training the weights should have common properties with the target data, such as objects with high frequency edges. If the
low-level representations of a CNN are responsible for detecting those edges, then the learned properties of edge detection can be transferred to the new task. The transferred weights can then either be frozen i.e. used as a fixed feature extractor in combination with a classifier (e.g. fully-connected layers), or can be fine-tuned to adapt to the target domain. This setup can permit the application of expressive deep learning models even in situations where the sample size of the target data set would be insufficient for training such a model from scratch.

For analyzing MRI data, transfer learning from natural images [121; 122] and from related MRI data (e.g., other MRI sequences; [123; 31]) has been suggested. Even though, the number of studies on transfer learning in neuroimaging is growing, studying, for instance, brain lesion segmentation [124; 125; 123; 126] or Alzheimer's disease classification [127; 128; 30], the most effective approach for transfer learning in clinical neuroimaging has not yet been established.

### Inductive bias

Every machine learning algorithm must make certain assumptions in how it processes the training data. For instance, (\(l_{1}\)-regularized) linear regression assumes a (sparse) linear relationship between input and target variable. The sum of assumptions of a given learning algorithm is called its inductive bias. Inductive biases that reflect the data-generating process or restrict the space of potential solutions to those that are a priori meaningful, can substantially reduce the number of training samples needed to achieve the prediction goal [129]. The flexible architecture of deep neural networks allows for the creation of complex inductive biases [130]. The success and subsequent (re-)popularisation of deep learning in the last two decades has been attributed mainly to advances in CNNs, i.e. the development of an inductive bias that allows efficient exploitation of translation invariance in natural images. Other examples include recurrent neural networks which assume time invariance [131], or graph neural networks which assume invariance to ordering (except for pair relations) [132].

In contrast to natural images, MRI brain images do not possess strong translation invariance properties. Due to a standard brain structure (e.g., hippocampus is at the same location in all humans) and further registration to a common template (e.g., MNI), the usefulness of translation invariance as an inductive bias for neuroimaging data has been questioned [32]. Recently, specialized architectures of deep neural networks have been proposed that are
specifically adapted to neuroimaging data. Kawahara et al. [133] proposed an architecture designed for connectivity matrices. Eitel et al. [134] restrict the translation invariance of CNNs to local patches, accounting for spatially normalized MRI data. Graph CNNs [135; 136; 137] have been adapted to neuroimaging data by e.g., Parisot et al. [138; 139]. Further advances in inductive biases for neuroimaging data may be an important prerequisite for fully utilizing deep neural networks in psychiatric research.

## 5 Application scenarios

Motivated by the ability of CNNs to extract hierarchical representations of complex image data, and several groundbreaking results for medical imaging applications where human-level performance has been surpassed [3; 4], we have selected three broad use cases which we determined as specifically promising for neuroimaging-based psychiatric research. The following section outlines these use cases (see also Figure 6), namely the automatic diagnosis of diseases and prognosis, the identification of disease subtypes and modeling normative distributions for detecting aberrations, and how deep learning can be used to develop new neuroscientific hypotheses.

### Automatic disease diagnosis and prognosis

The most apparent tasks for deep learning in neuroimaging-based psychiatric research are automatic disease diagnosis, prognosis, and prediction of treatment response. While signs of psychiatric disorders in neuroimaging data are very difficult to detect for human experts, several studies have suggested that there exist subtle, but measurable alterations in brain MRI data [7]. Hence, computer-assisted diagnosis using machine learning models that can pick up even very small and diffuse signals become highly promising. Most applications so far focused on binary classification tasks, such as patient vs. control [24; 59], converter vs. non-converter [140], and response vs. non-response to treatment [141], but the extension to multi-class classification tasks, such as differential diagnosis [142; 143; 144], is straightforward.

Since deep neural networks specifically profit from data which contains objects that are composed of a hierarchy of abstractions, deep neural networks are a particularly suitable method for sMRI data; the brain is composed of four main lobes, which are each composed of several brain regions and each regions is composed of edges and structures. The complexity of the neurobiology of psychiatric disorders makes nonlinear methods such as
deep learning more likely to successfully detect subtle disease-related alterations. Since the aim is to predict an outcome for each subject individually, rather than group level associations, machine learning in general may facilitate personalized medicine [145, 18]. Crucially, transfer learning of deep neural networks may help circumvent the problem of small sample sizes in psychiatric studies. Furthermore, automatic disease diagnosis is less prone to human bias [146, 147]. This is especially true for deep learning as it does not require manual feature extraction which can introduce specific biases such as the importance of some regions over others.

Beyond end-to-end learning, in which the model is fed with imaging data and outputs a class label, more indirect approaches exist. Here, the model output can include segmentation of white matter hyperintensities [148] or grey matter segmentation, which can form the basis for atrophy measurements [149]. These outputs can either be used to support a human expert's

Figure 6: Three application scenarios for deep learning in neuroimaging-based psychiatric research. The input is processed by a convolutional feature extractor (1.) and then further analyzed (2.) by one of the three described scenarios: (a) automatic disease diagnosis and prognosis, (b) subtype discovery and normative modeling, and (c) hypothesis generation and biomarker identification.
decision or to extract latent features for training another classifier [150].

When using longitudinal data, one can build deep learning models for prognosis, which have shown to outperform other methods in modeling disease progression [151]. From a clinical perspective, it is highly relevant to be able to assess time-to-event questions such as treatment outcome, survival prediction, relapse probability or potential disease onset. Current data sets are slowly approaching sample sizes that may allow for answering those questions. With future large-scale data sets, new milestones for clinical neuroimaging might be reached [152].

### Subtype discovery and normative modeling

Traditional diagnostic categories rely on identifying groups of frequently co-occurring symptoms, which are assumed to represent coherent disease entities. While successful in many areas of medicine, this approach tends to fail in psychiatry, where symptoms are far removed from the underlying pathomechanisms. Mounting evidence suggests that established disease categories (such as ICD-10 or DSM-V [153]) insufficiently match actual biological dysfunction [154; 155; 156; 35; 157; 36]. What appears as a coherent disease entity often turns out to be a mixture of distinct subtypes with heterogeneous biomarkers, treatment responses, and disease progressions. Consequently, robust biomarkers that allow for unambiguous diagnosis and predicting treatment response, common in other areas of medicine, remain elusive [36].

Increasingly available large-scale neuroimaging data sets allow for a new approach to disease categorization. Instead of using clusters of often qualitative symptoms as reference points in the search for the biological causes of psychiatric disease, it becomes possible to identify clusters directly from the quantitative neuroimaging data. This data-driven approach to disease categorization has multiple benefits. First, it prioritizes biomarkers. The whole process centers on quantitative neuroimaging data for identifying clusters of dysfunction and thus directly yields historically elusive neuroimaging biomarkers for psychiatric diseases. Second, by considering data that is closer to the level of biological dysfunction than self-reported or clinician-observed symptoms, disease categories may better match the underlying pathomechanism. Third, and consequently, such biomarkers and disease categories should better predict disease progression and treatment response [36].

The data-driven search for biologically meaningful disease categories, or disease subtypes (biotypes), relies on cluster analysis [56; 158; 159; 160] - a
class of machine learning algorithms that automatically subdivide data samples into distinct groups, called clusters, such that data points in the same group are maximally similar while data points assigned to different groups are maximally dissimilar. The question now becomes how to define similarity. Similarity in the space of voxels of a brain MRI is unlikely to yield clusters that are relevant for psychiatry for two reasons. First, a brain MRI is extremely high-dimensional, so that the variables, i.e. voxels, vastly outnumber the available data samples. This leads to the so-called curse of dimensionality; the higher the dimensionality, the more ways there are to be dissimilar and distance (or similarity) functions become increasingly meaningless. Second, similarity in the voxel-space of a brain MRI will be dominated by strong dimensions of variation such as sex and age, which tend to drown out the comparatively small dimensions of clinical relevance [161]. For neuroimaging-based clustering to be useful for psychiatry, the raw voxels of the brain MRI need to be transformed into a more meaningful representation which alleviates the curse of dimensionality and emphasizes clinically relevant variation over irrelevant confounders. Deep neural networks are particularly well suited to provide these highly abstract representations of neuroimaging data which are crucial for data-driven disease subtyping (see section 4.1).

Both promises and challenges of neuroimaging-driven subtyping can be illustrated by a study on depression subtypes by Drysdale et al. [162]. These authors used canonical correlation analysis (CCA) [163] to generate an intermediate representation, linking resting-state fMRI functional connectivity to the Hamilton Depression Rating. The low-dimensional intermediate representation emphasized specifically those aspects of the fMRI functional connectivity that pertain to the patients' self-reported depression symptoms. Cluster analysis on the intermediate representation identified four subtypes of depression, each with distinct connectivity and clinical symptom profiles, and which were predictive of patients' responses to transcranial magnetic stimulation. In a replication study, however, Dinga et al. [164] were unable to reproduce these results on independent data and emphasize that "[clustering algorithms] always yield clusters, regardless of the structure of the data", showing that comprehensive statistical analysis is necessary to ensure robust findings. The CCA that was employed by Drysdale et al. [162] can be seen as a linear precursor to more expressive models based on deep neural networks. Deep neural network extensions of CCA [165] and deep neural network architectures that are more finely tuned to the structure of neuroimaging data [133, 134, 139] may in the future unlock more intricate nonlinear interactions
in the data, which are inaccessible to classical models like CCA.

A related approach to utilizing intermediate (latent) representations of neuroimaging data for psychiatry is normative modeling [166; 21]. Instead of training models on data from diseased subjects to delineate disease profiles, normative modeling relies on data from healthy subjects to model the underlying dimensions of normal variation in neuroimaging data. The high-dimensional neuroimaging data is reduced to a low-dimensional representation, either via traditional machine learning models (see [166] for an overview), or more recently, by extracting the intermediate representations of deep neural networks (see section 4.1; [167; 168; 169]). In the space spanned by the new low-dimensional representation, the distribution of healthy subjects can be easily characterized, thus defining a normal range of variation. Subjects that fall outside of this normal range can be considered to have abnormal brain structure or function. There are different ways in which subjects can deviate from the normal range. Subjects may be normal in most dimensions of the model's intermediate representation and deviate in one or two specific dimensions, others may be outliers in a whole range of dimensions. Investigating these anomaly profiles could serve as an approach to disease classification in its own right [170].

### Hypothesis generation and biomarker identification using deep learning

In contrast to training disease models on a priori defined features (e.g., hippocampal volume in Alzheimer's disease [171]), deep learning allows for extracting its own representations (see section 4.1) which can be used to generate new scientific hypotheses and indicate directions for further study. One approach is to extract the learned representation of a CNN into a comprehensible format, such as a voxel-wise relevance heatmap (see section 3.2.3), in order to detect patterns across subjects. For instance, multiple studies proposed different ways to compute relevance heatmaps in Alzheimer's disease for explaining individual network decisions [104; 30; 172; 173]. In accordance with neurobiological evidence, all studies found the hippocampus and other temporal regions as primarily important. On the one hand, relevance scores can be used for model validation and the detection of data set bias when comparing regions with neurobiological evidence. On the other hand, new disease discriminating associations between certain brain regions and psychiatric disorders could be identified. These associations could also include common patterns of relationships between different brain regions, which only in combination elucidate predictiveness. This similarly applies to
multi-modal studies which integrate multiple imaging modalities or combine them with clinical and socio-demographic features [174; 175; 176; 141; 177]. Here, more complex relationships between various variables might be detected, such as the covariance of a brain region with the clinical history of a patient. In addition, biomarkers which most human experts would only identify in a specialized neuroimaging modality (e.g., diffusion tensor imaging) might have traces in a more common modality (e.g., T1-weighted MPRAGE). Those traces could be learned in a multi-modal fashion and then be used in cases where only the latter modality is available. It is essential that new hypothesis and biomarkers are tested on independent data, in order to avoid the so called "double dipping" in which one data set is used in circular analysis [178].

## 6 Applications in psychiatric research

In the last 1-2 decades, a number of neuroimaging-based machine learning studies in psychiatric research have been performed (for reviews, see [16; 23; 18; 179; 180]). Regardless of the difficulty of the application scenario, most of the initial studies were quite promising with accuracies above chance level for applications ranging from diagnosing patients with mental illness to identifying patients at risk and clinical subgroups to predicting treatment outcome [65; 181; 182; 183]. Although the data sets and methods used were quite diverse and difficult to benchmark, usually classical machine learning analyses consisting of some kind of feature extraction (e.g., grey matter density, volumes of brain structures, cortical thickness or functional connectivity) in combination with a linear or nonlinear machine learning algorithm (e.g., SVMs, logistic regression or random forest) have been employed [23]. However, most of the studies relied on relatively small sample sizes (\(N<100\)) and internal testing which often lead to overly optimistic performance metrics [184; 185; 23; 186] in addition to publication bias towards positive findings [187]. Even though deep learning studies within neuroimaging-based psychiatric research are still rare (with the exception of Alzheimer's disease), we review its application on several mental disorders, which have been employed in at least one of the applied machine learning settings described in section
### Alzheimer's disease

Most MRI-based machine and deep learning studies so far have been conducted on Alzheimer's disease (AD), a neuropsychiatric disorder which is the main cause of dementia in the elderly. AD is symptomatically characterized by loss of memory and other cognitive abilities to such an extent that it affects daily life [188]. In contrast to other psychiatric disorders, the neurobiological correlates are rather clear (i.e., neurodegeneration and resulting atrophy starting in the hippocampus). In addition, the existence of the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, a large open data base with easy access for (non-clinical) researchers, considerably boosted the number of machine learning studies [189; 23]. In a recent review of deep learning studies in AD, accuracies of up to 98% have been reported for the discrimination between patients with AD and healthy controls, and 83% for the conversion from mild-cognitive impairment (MCI) to AD [140]. However, Wen et al. [59] pointed out that a considerable number of those studies suffer from data leakage (see section 3.1.2) and therefore overestimate the prediction performance. From the 16 studies where data leakage could be excluded, accuracies between 76% and 91% for the discrimination of patients with AD and healthy controls and 65% to 83% for the discrimination between MCI and healthy controls have been reported. Whereas early studies mostly analyzed 2-dimensional slices of MRI data and have drawn on established architectures and pre-training techniques in computer vision (e.g., [127; 190]), most studies now analyze 3-dimensional data in either a ROI-based (e.g., hippocampus) or a whole-brain setting [30; 191; 59]. In Wen et al. [59], those different approaches and additional settings regarding pre-processing and pre-training have been benchmarked with accuracies of up to 88%. Deep learning approaches based on 3-dimensional data were superior to 2-dimensional approaches but not better than a standard machine learning analysis relying on SVMs. The models also generalized well between different data sets when the inclusion criteria and the underlying distributions of demographics was not too different. Only few studies so far have tried to understand and explain what the deep neural networks (i.e., CNNs) have learned during training (see section 3.2.3). Different attribution methods, indicating the importance of each voxel for the final classification decision, have been compared [104] and especially layer-wise relevance propagation (LRP) has been shown to explain CNN decisions well in AD classification [108; 30]. In Wood et al. [192], a recurrent visual attention model is suggested that processes the data iteratively and thus is easier to interpret than
CNN-based approaches.

### Schizophrenia

Schizophrenia is a mental disorder characterized by episodes of psychosis for which neurobiological abnormalities in sMRI and fMRI data with a focus on enlarged ventricles and an involvement of structures within temporal and frontal lobe have been reported [193; 194; 195; 196]. Based on diverse features ranging from grey matter density to volumes to functional correlates, a number of classical machine learning studies have been performed to either diagnose schizophrenia, predict first-episode psychosis, or identify subtypes [181; 197; 198; 199; 200; 143; 201] (for a recent review, see [202]). The classification accuracies varied here considerably between 60% and 95%; a tendency of lower accuracies for larger data sets. Independent testing suggested that reported accuracies should be taken with caution [23; 185]. Regarding deep learning, a first promising study using a deep belief network has shown that more depth resulted in a strong increase of accuracy (from around 60% to 90%) for separating patients with schizophrenia and healthy controls [110]. Another study using a deep belief network has shown a slightly higher accuracy than for a SVM (74% to 68%) [203]. Using CNNs and MRI data from patients with schizophrenia and healthy controls from 5 public data sets (\(N=873\)), Oh et al. [204] report AUCs of up to 0.90 on the left-out data set with a high relevance of the right temporal region. However, the importance of a similar training and test distribution was shown by a rather low AUC of 0.71 for an external data set with younger patients and a shorter illness duration. Notably, for clinicians (5 psychiatrists, 2 radiologists) the average accuracy was 62% on 100 randomly selected MRI volumes.

For resting-state and task-based fMRI data, diverse techniques have been employed to discriminate between patients with schizophrenia and healthy controls including different kinds of autoencoders [205; 206; 207], deep generative models [208], and CNNs [209; 210]. For functional connectivity matrices from a large multi-site sample (\(N=734\)), accuracies between 81% and 85% for leave-one-site-out classification have been reported [206]. In accordance with neurobiological findings, most important for the classification were connectivity values within the cortical-striatal-cerebellar circuit. Compared to whole brain images and graph-based metrics obtained from resting-state fMRI, functional connectivity measures have generally been shown to be more informative in a machine and deep learning framework [211]. In Plis et al. [212], a deep learning-based translation approach is suggested in
order to investigate the linkage between MRI-based structure (grey matter) and function (dynamic connectivity). For task-based fMRI data, similar accuracies (around 80% to 84%) have been reported [207; 210], relying on, for example, a sensorimotor task [207]. To exploit the temporal structure of fMRI data, Yan et al. [213] developed a multi-scale recurrent neural network and report accuracies between 80% and 83% in a multi-site setting (\(N=1100\)). In Pinaya et al. [167], a normative model based on a deep autoencoder was trained in a large cohort of healthy controls (\(N=1113\)) and then the reconstruction error as a measure of deviation was assessed in patients with schizophrenia.

### Internalizing disorders

Internalizing disorders describe a group of psychiatric disorders which are characterized by negative affectivity and include, for instance, depression, anxiety disorders, and post-traumatic stress disorder (PTSD). In contrast to AD and schizophrenia, internalizing disorders have been comparably less studied using classical machine learning and deep learning. On the one hand, as for most psychiatric disorders, the labels per se have been criticized for being mainly based on symptoms rather than neurobiological correlates and thus being too noisy and unspecific for the use in supervised machine learning studies [34]. On the other hand, neurobiological correlates obtained from neuroimaging data are less clear for internalizing disorders and might be mediated by underlying subtypes [214; 162].

For major depressive disorder (MDD), a recent review [215] summarized the results of 66 classical machine learning studies equally distributed on structural MRI, resting-state and task-based fMRI. In comparison of modalities, models using resting-state fMRI data tended to yield higher accuracies. Some studies also investigated machine learning algorithms in the context of predicting treatment outcome with a focus on fMRI connectivy as a biomarker. The sample size of studies were mostly below 200 and accuracies for discriminating MDD patients and healthy controls ranged from chance (approx. 50%) to unrealistic 100%, again showing the need for caution when interpreting results. Problems such as data-leakage or insufficiently sized test sets (see section 7.1.2) have most likely afflicted at least some of these studies [162; 216; 215]. For bipolar disorder, a large recent multi-site study [217] based on 13 cohorts from ENIGMA exists (\(N=3020\)) that report aggregated subject-level accuracies of about 65% using a combination of SVMs
and extracted MRI features (regional cortical thickness, surface area, and subcortical volumes).

Using CNNs and recurrent neural networks, Pominova et al. (2018) report AUCs of up to 0.66 for sMRI and 0.73 for fMRI for separating patients with MDD and healthy controls. In a large-scale machine learning challenge (PAC 20188, \(N=2240\)) a variety of simple and high-complex classifiers (CNNs etc.) were benchmarked by 49 teams for the classification between patients with MDD and healthy controls. However, none of the teams achieved an accuracy higher than 65% on the holdout set. Given that a number of studies have reported classification accuracies of over 90% for the classification of depression based on sMRI data (for an overview, see Wolfers et al. (2016), Gao et al. (2015)), those results seem rather surprising on a large well-controlled data set. A possible conclusion from these results is that PAC showed more realistic accuracies than published studies that suffer from overconfidence as previously explained. Based on the same data set, it has been shown that accuracies are systematically overestimated in random subsets (Luo et al., 2016) replicating small sample size effects, e.g., reported in (Kumar et al., 2016; Luo et al., 2016; Luo et al., 2016). In contrast to decoding clinical categories, Pervaiz et al. (2016) suggests to predict neurotisicsm as an underlying construct for the potential incidence of mood disorders. However, based on functional connectivity data from the UK biobank (\(N\approx 14000\)) and diverse optimised pipelines, the average correlation between predicted and true neuroticism score was below 0.2.

Footnote 8: [https://www.photon-ai.com/pac](https://www.photon-ai.com/pac)

Regarding anxiety disorders including obsessive-compulsive disorder (OCD) and PTSD, a review from 2015 included 8 studies which all had a rather low sample size and report comparable high accuracies (Wolfers et al., 2016). Additional studies investigated the prediction of treatment outcome in anxiety disorders (for a review, see Lueken et al. (2016)). For OCD, a systematic review found 12 studies with a wide range of accuracies based on different modalities but sample sizes were rather small (Kumar et al., 2016). None of those studies used deep learning.

### Neurodevelopmental disorders

Neurodevelopmental disorders describe a group of neurological and psychiatric conditions that originate in childhood. We will focus here on two diseases, namely attention-deficit/hyperactivity disorder (ADHD) and autism spectrum disorder (ASD), which both have been investigated in deep learning
frameworks [24; 220].

ADHD is characterized by an ongoing pattern of inattention, hyperactivity and/or impulsivity, which has been related to smaller brain volumes (e.g., in the dorsolateral prefrontal cortex) and altered functional connectivity [221; 222]. In a competition from 2012 based on the ADHD-200 data set9, the best predictive model relied only on personal characteristic data including age, gender, and several IQ scales, and led to an accuracy of 62.5%.10 The best image-based model resulted in an accuracy of 60.5%. In an extended ADHD-200 data set, it has been shown that models based on only personal characteristic data outperform models based on resting-state fMRI data (75.0 to 70.7%) [223]. However, since then a number of deep learning studies based on resting-state fMRI data have been performed and accuracies of up to 90% have been reported [224; 225; 226] (for an overview, see [24]). Recently, also based on the ADHD-200 data set, a spatio-temporal model and a multi-channel model for combining resting-state fMRI data as well as demographics have been shown to result in AUCs between 0.74 and 0.8 [227; 228].

Footnote 9: [http://fcon_1000.projects.nitrc.org/indi/adhd200/](http://fcon_1000.projects.nitrc.org/indi/adhd200/)

Footnote 10: [http://fcon_1000.projects.nitrc.org/indi/adhd200/results.html](http://fcon_1000.projects.nitrc.org/indi/adhd200/results.html)

Subjects with ASD are mainly characterized by repetitive behavior and difficulties in social interactions. A number of structural and functional abnormalities have been described including a slightly thinner temporal cortex, a thicker frontal cortex, reduced volumes of amygdala and nucleus accumbens, as well as a reduced functional connectivity [229; 230]. An overview over deep learning studies in ASD can be found in Khodatars et al. [220]. Based on sMRI and resting-state fMRI from the Autism Brain Imaging Data Exchange (ABIDE) initiative, accuracies of about 70% have been reported [231; 220].

### Substance abuse

Substance use disorders (SUDs) describe a class of mental disorders related to problematic consumption of alcohol, tobacco and illicit drug use affecting daily and working life. For alcohol use disorder (AUD), the most prominent SUD, sMRI and fMRI studies have revealed a number of neurobiological correlates including enlarged ventricles, grey and white matter loss in frontal and reward-related brain areas, as well as altered functional connectiv
ity in the amygdala and nucleus accumbens [232; 233; 234; 235; 236]. Classical machine learning models have been employed to identify AUD or predict alcohol consumption / binge drinking on different kinds of data including demographics, history of life events, personality traits, cognition, candidate genes, as well as brain structure and function [237; 238; 239; 240; 174]. For sMRI data, it has been shown that a computer-based classification approach performed better than a blinded radiologist in diagnosing alcohol dependence based on regional grey matter (74% to 66%) and predicting future alcohol consumption [237]. Compared to demographics, sMRI and task-based fMRI, Fede et al. [241] showed that resting-state connectivity resulted in the lowest root-mean-squared error (RMSE) for predicting alcohol severity measured by the Alcohol Use Disorders Identification Test (AUDIT; resting-state fMRI: 8.04, demographics: 9.76, sMRI: 8.11, task-based fMRI: 8.63). Alterations in the reward network and executive control network have been reported as informative for diagnosing AUD [240]. For predicting alcohol use during adolescence, thinner cortices and less brain activation in frontal and temporal areas have been found [242]. The largest classical machine learning study so far has been performed on the IMAGEN data set, as part of which Whelan et al. [174] report an AUC of 0.90-0.96 for the separation of 14-year old binge drinkers and 14-year old controls and an AUC of 0.75 for predicting binge drinking at 16 years. Here, a combination of history, personality traits, and brain features were most predictive, supporting the hypothesis that multiple causal factors shape later alcohol use. In another study, sex-specific psychosocial and neurobiological features have been identified for the initiation of cannabis use [243].

To the best of our knowledge, only three recent studies (from a single group) exist that have used CNN models to identify AUD [244; 245; 246]. They applied different CNN models to 2-dimensional slices and reported accuracies around 97% in discriminating patients with abstinent long-term chronic AUD (\(N=188\)) and healthy controls (\(N=191\)). Due to the excessive amounts of alcohol consumed, the CNN models presumably captured neurotoxic effects rather than neurobiological underpinnings explaining AUD.

### Brain age

As an overarching biomarker relevant in many diseases including AD, schizophrenia, and substance abuse, brain age as opposed to chronological age has been suggested [237; 247; 248; 249]. Using CNNs, highly successful and robust models for brain age have been developed on raw MRI data [11]
and parcellations [250] with average deviations (mean absolute error) of 4-6 years. However, similar deviations were achieved using Gaussian processes applied to grey matter volumetric maps [11]. Based on a large and heterogeneous multi-site cohort (\(N=11729\)), Bashyam et al. [248] have shown that moderately fitting brain aging models (based on 2-dimensional CNNs) might lead to a better separation between patients (with AD, MCI, schizophrenia or depression) and healthy controls than tightly-fitting models.

## 7 Challenges

Besides the potential of deep learning in neuroimaging-based psychiatric research outlined in sections 4 and 5 and early promising studies presented in section 6, the field faces some substantial challenges hindering immediate application in clinical routine.

Noise in MRI data and disease labels affects prediction performance and increases the number of data samples required for training deep neural networks (section 7.1). In section 7.2, we discuss difficulties of model choice and model comparisons within the neuroimaging domain. For instance, it is still contentious, under which circumstances deep neural networks are useful for neuroimaging data and whether neuroimaging data require nonlinear models (section 7.2.1). And even within different neural network architectures, comparing model performances is impeded by a lack of neuroimaging benchmark data sets (section 7.2.2). Furthermore, the computational expense of 3-dimensional CNNs hinders widespread application (section 7.2.3).

Even when a deep neural network achieves high prediction performance on an independent test set, pitfalls with respect to validity and explainability remain (section 7.3). Imbalances in the training data can lead to systematically misdiagnosing minorities (7.3.1). Correction for confounding variables becomes substantially more challenging than in classical statistics (7.3.2). In addition, complex machine learning models are hard to interpret and require sophisticated engineering to extract explanations for a models' predictions (7.3.3).

### Noise in MRI data

#### 7.1.1 Low signal-to-noise ratio

The most basic precondition for machine learning to succeed is the existence of exploitable mutual information between brain images and target variables. While this precondition is surely fulfilled for sex or age, brain
lesions or atrophy, the situation is less clear for more intricate variables representing aspects of human cognition and affectivity which are relevant in psychiatry [251; 23]. It is, for instance, controversial to what extent subtle traits such as intelligence are reflected in brain morphometry measured via currently available sMRI technology [252; 253]. Similarly, fMRI does not directly reflect neuronal activity and instead relies on the haemodynamic response as a proxy, giving it a temporal and spatial resolution that is far removed from the actual underlying phenomenon [254; 23]. The actual signal, for example the cognitive process in question, will realistically only occupy a small fraction of a diverse conglomerate of different signals [255; 256]. Unrelated anatomical differences, differences in physiology, unrelated physiological processes such as breathing or head movements, thermal and system noise from the scanner, even effects from unrelated cognitive processes may obscure a particular trait or cognitive process to the point of invisibility. In comparison to areas where deep learning is highly effective, such as natural language, natural images, and even some areas of biomedicine (e.g., histology [257], or dermatology [3]), neuroimaging data in the context of psychiatry contains high levels of noise and often little trace of the phenomenon under investigation. Low signal-to-noise poses challenges to the application of deep learning, requiring a higher amount of training data (7.1.2) and may impede the extraction of nonlinear structure (7.2.1).

#### 7.1.2 Lack of ground truth labels

The success of supervised machine learning approaches in neuroimaging-based psychiatric research may be limited by insufficiently valid and reliable labels. Although psychiatric research is not imaginable without clinical labels such as depression or schizophrenia, those diagnostic categories have been severely criticized for not incorporating underlying neurobiological correlates and for their limited ability to account for heterogeneity as well as comorbidities within and across clinical categories [157; 36; 35; 154]. Moreover, both low inter-rater and low test-retest reliability have been repeatedly documented [47]. To address this issue, the National Institute of Mental Health (NIMH) suggested to study mental health along the so-called Research Domain Criteria (RDoC), and by this to shift focus away from classical clinical labels towards carefully designed domains of neurocognitive functioning across different levels of analysis assumed to be relevant to mental health [157; 35].

In machine learning terminology, the field of psychiatry would be de
scribed as suffering from high "label noise". The higher the label noise, the more data samples are necessary to reliably characterize the statistical relationships between brain image and clinical label. This reduced sample efficiency, i.e. the reduced amount of information gained per sample, further exacerbates the problem of insufficient sample sizes in neuroimaging psychiatry, where generating additional training data is often prohibitively expensive. Especially for small sample sizes, high label noise increases the likelihood of unrepresentative, spurious high-accuracy results [184; 185]. These can arise not only by overfitting or data leakage, but also out of sheer coincidence; the particular subsample of data used for model evaluation may randomly contain more correct or more easy-to-classify labels than what would be representative for the data set [186].

### Model choice and model comparison

#### 7.2.1 Linear vs. nonlinear models

Even though deep learning has revolutionized the fields of computer vision and natural language processing, it is yet unclear if these successes will fully translate to neuroimaging. Results on deep learning based prediction of demographic or behavioral phenotypes have been mixed. While various early successes have been reported [110; 24], deep learning models have often failed to outperform linear baselines [32; 33]. The mixed literature results are mirrored in recent challenges and competitions for machine learning in neuroimaging. In a number of competitions, for instance in recent ABCD [252], PAC [186], and TADPOLE [258] challenges, classical machine learning models outperformed more complex deep learning approaches. In their review of deep learning in neuroimaging, Vieira et al. [24] conclude that "despite the success of [deep learning] in several scientific areas, the superiority of this analytical approach in neuroimaging is yet to be demonstrated".

One possible explanation is the comparatively low sample size of neuroimaging studies. While computer vision and natural language processing (NLP) models are regularly trained on millions of data samples, a typical neuroimaging study will have only hundreds or at most thousands of participants. Given the arguably lower signal-to-noise ratio of neuroimaging data, one may expect a need for larger rather then smaller sample sizes to extract complex nonlinear interactions from neuroimaging data. This view is supported by research indicating that the performance of linear models on neuroimaging data is not yet saturated at \(N\approx 10000\)[32]. If present sample sizes are insufficient to even fully characterize linear effects, then it seems
unlikely that more complex nonlinear interactions can be extracted without further efforts in large scale data collection.

An alternative explanation for the difficulty of applying deep learning to psychiatry pertains to the high levels of noise in neuroimaging data (see section 7.1.1). Schulz et al. [32] argue that high levels of additive noise can in certain instances linearize the true decision boundary. In this case, no amount of additional training samples would allow complex nonlinear models such as deep neural networks to outperform linear baselines. Some nonlinear relationships in neuroimaging data could be inaccessible in principle.

Whether inability to exploit nonlinear structure in some neuroimaging data is due to insufficient sample size, or inaccessible due to noise remains an open question. Nevertheless, recent research suggests that machine learning for psychiatry will not necessarily be solved by more and more complex and expressive models alone. Researchers may instead need to focus on stronger inductive biases, which impose structural priors on the data to more efficiently learn statistical relationships from limited data [130]. Inductive biases that are specifically adapted to neuroimaging data (see section 4.3) may unlock prediction successes in the future.

#### 7.2.2 Lack of benchmark data sets

The literature contains a plethora of different deep neural network architectures, and new designs are continually being invented. Thus, reliable model comparisons are necessary to make an informed choice about which particular deep neural network architecture (or even which linear model) to use on a given brain imaging data set. Standardized benchmark data sets are crucial for a reliable and fair comparison of models and methods, and are an integral part of machine learning research (see, e.g., Imagenet [259] or Cityscapes [260] for computer vision). However, to the best of our knowledge, no large-scale neuroimaging benchmark data sets for the detection of psychiatric disorders are openly available and Arbabshirani et al. [185] even state that comparison of accuracies between studies is "essentially meaningless". The necessity of benchmark data sets for model comparisons, particularly in neuroimaging-based psychiatric research, is due to broadly four reasons.

First, benchmark data sets can control for heterogeneity of the data sources. Cityscapes, for example, contains street scenery from 50 different cities, procured in different seasons and weather conditions. In multi-site neuroimaging data sets, heterogeneity is both a promise and a pitfall. Since the imaging site has a significant effect on prediction [261], it is beneficial for
generalization to include several imaging sites, but it is also required to have similar label distributions across sites and to have high counts of samples per site.

Second, no further pre-processing of the data is required or code is available for additional steps such as cropping or intensity harmonization. In neuroimaging, MRI pre-processing is considerably more complex than on natural images. Consequently, the selection of pre-processing steps and tools have a significant impact on task-driven neurobiological inference and patient-wise prediction [262]. Neuroimaging benchmark data sets therefore need to either come fully pre-processed or include code to repeat the pre-processing. The aim is not to find an optimal pre-processing strategy but rather to ensure replicability and comparability between studies.

Third, the same motivation holds for a unique split of the data into training and testing before publication. As the heterogeneity of a data set is proportional to its size \(N\)[263, 184], the prediction result will change drastically with different train and test splits. Therefore, in order to compare model performances between studies it is required that all studies use the same train and test split. Cityscapes has carefully designed a 3-way split of the data (train/validation/test) which attributes equal amounts of data based on geographical location, city size and season. Similar efforts would be suggested for neuroimaging data. Another downside of not sharing unique data splits is the risk of data leakage as described in section 3.1.2 which has been shown for AD classification in [59].

And fourth, a benchmark data set for neuroimaging would need to provide an adequate level of task difficulty. On the one hand, the prediction task needs to be difficult enough for a superior model to clearly and reliably outperform simpler baselines. If a task is too simple and models are approaching the tasks' irreducible error (which might be the case for sex or AD classification), then it is unlikely that differentiation of model performance can be reliably observed. On the other hand, the task cannot be too complex, so that models can be successfully trained with available sample sizes.

Creating large benchmark data sets typically requires the collaboration of several clinics to form a multi-site study. Even though, conducting multi-site machine learning studies seems highly promising, and is most likely a requirement for building generalizable models, properly designing multi-site studies remains a challenge. The main issue with multi-site data sets is that the data collected will have measurable differences between sites and perfectly matching participants between sites is not feasible. Most often,
different sites have a different ratio of cases and controls. When aggregating those sites, covariates such as age, sex, total intracranial volume etc. might correlate strongly with the case-control ratio. High capacity machine learning models will easily learn to pickup those correlations to differentiate the MR images, rather than using disease specifying properties (see section 7.3.1). Hence, benchmarks on multi-site data sets might be overly optimistic in their results. Investigating the issue could be done by training a machine learning model solely on those covariates. If the covariates lead to a result which is not significantly different than the neuroimaging based model, then the results of the latter might have no meaning beyond portraying covariate differences.

Even if not matching the criteria of benchmark data sets, the publication of large and open data sets is almost always valuable for machine learning research. However, for creating better benchmarks, study designers should ensure that the aforementioned points are met. This includes reporting any data source heterogeneity, the inclusion of a standardized processing pipeline, a pre-determined data set split and determination of clinically relevant questions to address, which have an appropriate difficulty. In addition, it is necessary to control for the correlation of covariates within and between sites.

#### 7.2.3 Computational challenges

Other impediments to the application of deep neural networks for neuroimaging pertain to computational cost. Although general processing GPUs (gpGPUs) have enabled deep learning to take off in many fields, computational resources remain expensive and especially neuroimaging-based deep learning is still limited by technical capacities. In contrast to other applications where cloud computing can be used, medical data typically needs to stay within the boundaries of the hospital and cannot be transferred to the cloud provider's network. Furthermore, as sMRI data is 3-dimensional it requires cubic operations leading to extensive GPU memory utilization. With images that often have more than a million voxels, the memory of a typical GPU is quickly exceeded when using very deep neural networks, large batch sizes or wide fully-connected layers. In fMRI, which has an additional time dimension, the issue becomes even more severe. Therefore, most researchers are highly limited in the amount of architecture search they can do. Many deep learning applications are extremely sensitive to hyperparameters. However, the optimization problem of finding hyperparameters is ill defined and often results in random searches. This means that usually the search space
of hyperparameters is underexplored and one cannot prove that no better settings exist. This issue is similarly pressing when training simpler baseline models and comparing the suggested model to those baselines. Here, benchmark data sets as described in section 7.2.2 could help by giving many researchers the opportunity to search for hyperparameters, allowing them to reference the current state-of-the-art results as baselines.

### Validity and explainability

#### 7.3.1 Algorithmic bias

Challenges arise not only in training deep neural networks for psychiatry, but also in generalisation. How do trained models deal with new data samples that were generated after training and validation was already completed? In day-to-day clinical use, new data samples may differ from the training data in subtle but critical ways. The MRI scanner may be of a different model, scanning protocols or pre-processing pipelines may differ, and certain ethnicities or comorbidities may not have been included in the training data sample. This can lead to critical errors. The predictions of a deep neural network are meaningful only for new data samples from the same distribution that generated the original training data [264]. Goodfellow et al. [265] argue that "classifiers based on modern machine learning techniques, even those that obtain excellent performance on the test set, are not learning the true underlying concepts that determine the correct output label. Instead, these algorithms have built a Potemkin village that works well on naturally occurring data, but is exposed as a fake when one visits points in space that do not have high probability in the data distribution". A deep neural network will still make predictions for out-of-distribution samples, but these prediction will often be meaningless or misleading.

Furthermore, the expressivity of deep neural networks exacerbates distortions arising from sampling biases and confounding variables in the data [266; 267]. A deep neural network has neither the goal nor the ability to prefer causal relationships over a spurious associations and will exploit whatever serves to predict the target variable [94]. Hence, biases that exist in the training data will be reproduced in the learned model [268]. This phenomenon is well studied and to some extent under control for simple linear models, where (known) confounders can be anticipated and corrected for [269]. However, the highly expressive deep neural networks may pick up highly complex and unexpected biases in the training data. For instance, a deep neural network trained to classify skin lesions learned that dermatologists tend to include a
ruler in the photos of skin lesions they where particularly concerned about it, and used the ruler as a proxy for malignancy [270]. Thus, one flip-side of the high expressive capacity of deep neural networks is the resulting difficulty to control for potential biases in the predictions. Such "algorithmic biases" have diverse practical impacts. Minorities, who are often underrepresented in training data sets, have a high chance of being systematically misclassified, with potentially severe clinical consequences [271; 272]. Deep neural network's ability to exploit the most subtle biases in the training data represents a significant problem for multi-site studies where deep learning may pick up on even the slightest sampling differences between sites and thus use measurement site as a proxy for the actually studied phenomenon [266], further corrupting predictions on new data.

#### 7.3.2 Confound modeling

While the preceding section was most concerned with prediction errors, similar complications arise in the interpretation of results. Frequently, researchers will not only attempt to predict a target variable but will investigate the relative contributions of different inputs on a model's accuracy, for instance to delineate neural correlates of a psychopathology from confounds like gender, age, or education which may influence the risk of disease.

The high expressivity of deep neural networks and the necessity of comparatively large sample sizes for training them may require novel approaches to disentangle the impact of confounds on prediction performance, as traditional approaches become increasingly impractical. For instance, a priori balancing of the training data (i.e., matched, case controlled study design) is costly, limiting the achievable sample size and thus the prediction performance. Post hoc balancing by subsampling [273; 274] similarly reduces the effective sample size available for training the neural network. Regressing out the influence of confounds directly from the neuroimaging data [275; 276; 277; 273] works for simple linear models, but will not completely eradicate complex nonlinear traces of the confounding variables from the input. Machine learning models, particularly the highly expressive deep neural networks, may still exploit residual information that could not be regressed out. In a model agnostic approach, Dinga et al. [278] engage this problem by controlling for confounds post hoc on the level of model predictions.

Deep learning does not only complicate confound modelling, but also allows for novel solutions to this problem. Some researchers propose using confounding variables as additional targets in a multi-task setting and to
subsequently zero the model weights which are important for the confounding variable but not the target variable [279]. Others advocate incorporating confounds directly into the model (e.g., by disentangling them from the intermediate representations [117; 167]), or even using adversarial optimisation objectives [280] to train deep neural networks that correct against the influence of confounding variables.

Although properties of deep neural networks may exacerbate problems of algorithmic bias and confound modelling, the flexibility of deep neural networks to incorporate and isolate confounding variables within the model may in the future offer uniquely powerful solutions to deal with heterogeneous biomedical imaging data.

#### 7.3.3 Explainability

Another way to check for biased or implausible predictions is to interrogate the models decision making process (see section 3.2.3). However, even the features of linear models are not unambiguous in neuroimaging [281; 282]. Since deep neural networks have orders of magnitudes more parameters than most other machine learning models, they are extremely hard to comprehend from a human perspective. While in computer vision we are able to visualize the filters of convolutional layers and to assign meaning to them (i.e. edge detectors or more abstract concepts such as eyes for face recognition [103]), this becomes more challenging in neuroimaging as sMRI is 3-dimensional and abstract concepts are less immediate (e.g., how would a filter for atrophy look like?).

Attribution methods provide a popular approach for reasoning about deep learning models. In high-risk settings such as medicine, the performance of an algorithm might have a vital impact on an individual. Hence, it has been argued that explainability of medical algorithms might become a prerequisite for clinical adoption. On the other side, it has been discussed (in, e.g., London [283]) whether explainability should be a requirement for machine learning systems that have a high accuracy, considering that the comprehension of many other causal mechanisms in medicine, such as the mechanisms of certain drugs or psychotherapy, is incomplete. In any case, Lapuschkin et al. [94] have shown how post-hoc analyses of models with high accuracy can identify artifacts in the training data which the models misuse for classification. This showed that attribution methods can help to identify bias or subpar learning strategies, which in neuroimaging could for example be a focus on scanner artifacts, human motion, or random correlations between brain features and
group membership. As a caveat, it was recently shown that most attribution methods only minimally change their produced heatmaps when several layers of the model are being randomized [284; 285]; how to quantitatively assess attribution methods remains a challenge. Here, neuroimaging creates an opportunity, as the quantitative validation of attribution maps using neurobiological knowledge is more objective than the sole visual inspection [30; 31; 286].

## 8 Conclusion

While machine and in particular deep learning approaches provide a huge potential for transforming neuroimaging-based psychiatric research towards precision psychiatry, the field is still at the very beginning. From a structural perspective, the further collection of large, multi-site, standardized, and open-source neuroimaging data sets with overlapping disease categories and thorough clinical and behavioral characterization is essential. For addressing questions about disease prognosis and treatment outcome, longitudinal data will be increasingly important. In addition, efforts should be made in order to make analyses between different groups comparable, e.g., by creating benchmark data sets. The key to success will then be whether clinically meaningful and transparent representations can be learned from neuroimaging data that additionally account for individual and site-specific covariates.

## 9 Acknowledgements

We acknowledge support from the German Research Foundation (DFG, 389563835; 402170461-TRR 265; 414984028-CRC 1404), the Deutsche Multiple Sklerose Gesellschaft (DMSG), the Manfred and Ursula-Muller Stiftung, and the Brain & Behavior Research Foundation (NARSAD grant; USA).
