Saboo et al. [62] developed a deep learning model for cognition prediction
over five years after the baseline using clinical and imaging features and
leveraged model explainability to identify certain brain structures
responsible for cognitive vulnerability or cognitive resilience. Specifically,
it identifies those predictive brain structures (medial temporal lobe, fornix,
and corpus callosum) that best explain the heterogeneity between cognitively
vulnerable and resilient subjects. To this end, the study used LIME to compute
the contributions of each imaging and clinical feature toward the predicted
future cognitive score. While LIME generated contributions of features matched
with some prior reports, earlier literature blamed LIME for producing unstable
contributions over multiple runs. Moreover, like most of the ongoing
interpretability practices, this study also did not justify the rationale of
using LIME.

Gaur et al. [64] proposed an explanation-driven dual-input CNN-based solution
to predict the status of brain tumors using brain MRI scans. The authors
adopted both SHAP and LIME to generate explanations because SHAP captures the
consistency and accuracy in the explanations and LIME captures the local
behavior of the model around the test example. The study reported a predictive
training accuracy of 94.64% as the performance of the model, which is not the
usual practice of reporting model performance. Furthermore, while the study
used two post hoc interpretability methods (LIME and SHAP), validation of the
generated explanations was not provided to support the study. The
explainability was not used as part of the model training, and hence we argue
that the model is inappropriately referred to as "explanation-driven." Rather,
the concept of XAI was used to analyze the model predictions in a post hoc
manner, and not to enhance the model’s performance.

SHapley Additive exPlanations (SHAP)
Ball et al. [225] used three different machine learning approaches to estimate
brain age based on cortical development as the variations in brain age and
chronological age have been linked to many psychiatric disorders. The authors
also used kernel SHAP to identify which features explain errors (brain age
delta) in brain age prediction. These explanations are consistent across the
models and previously reported brain development regional patterns. However,
no generic spatial association among individual feature estimates is found for
the brain age prediction error. That is, given the similar demographics and
prediction error, feature importance estimates between subjects may vary
widely. Moreover, brain age delta estimates for any of the models did not
associate noticeably with cognitive performance. Given the limitations of SHAP
in estimating feature importance and the lack of any objective validation for
these explanations, this study needs further investigation.

Lombardi et al. [56] segmented the T1-weighted brain into 68 cortical regions
and 40 sub-cortical regions using atlases to extract morphological features
from the scans. The authors developed a fully connected network and analyzed
the feature attribution performance of two model-agnostic interpretability
methods: LIME and SHAP. To assess the predictive performance and intra-
consistency of the interpretability methods, the model was trained multiple
times using different subsets of the training fold. SHAP, compared to LIME,
provided greater intra-consistency of feature attribution scores implying that
the method is comparatively robust to the training set variations. Inter-
similarity analysis across the subjects reveals that SHAP values can be better
partitioned into different age ranges. Furthermore, compared to LIME, feature
attribution scores from SHAP were highly correlated with brain age and the
features were more aligned with the previous reports of morphological
association with brain age. However, the obvious limitation of this study was
that the model was trained based on a smaller cohort of samples with a limited
age range. Furthermore, a quantitative validation approach is required to
objectively compare the performance of XAI methods. Figure 15 shows how LIME
and SHAP were used to identify brain regions associated with the chronological
age of the subjects.

Figure 15:  Brain regions with the most significant correlations between the
XAI scores of the ROI-specific morphological features and the chronological
age. For the SHAP method, the study reports that the average thickness,
folding, and curvature index statistical attributes calculated based on the
precentral gyrus and inferior and lateral occipital cortex were highly
correlated with brain age. Also, the SHAP scores of the cortical thickness
features of both hemispheres showed a significant correlation with age. These
findings are highly aligned with the previous reports. On the other hand, for
LIME, the study found a strong correlation between brain age and the features
related to WM volumes of the opercular and triangular parts of the inferior
frontal gyrus and inferior temporal gyrus. Overall, the LIME explanations were
not consistent and had a very low overlap with SHAP scores. (Image Courtesy:
[56])

###  10.5 Intrinsic Methods

Very few neuroimaging studies so far have considered interpretability as part
of the algorithmic aspect of the model from its inception. Such models in the
literature are called glass-box or transparent-box models. Biffi et al. [221]
proposed a deep generative model for transparent visualization of the
classification space. Some other neuroimaging studies [238, 239] considered
interpretable models based on their design transparency. We discuss some
studies that utilized the concept of intrinsic interpretability as part of the
model design or training.

Attention Mechanism
Lian et al. [227] proposed an attention-guided unified framework that uses a
convolutional neural network to work as the task-specific guidance and a
multi-branch hybrid network to perform disease diagnosis. The main motivation
behind this work is to extract multilevel information at interpersonal,
individual, local, and global scales. The first fully convolutional network
offers guidance via disease attention maps and thus assists in generating
global and individual features. In the second stage, the unified framework
leverages disease attention maps (DAMs) calculated using class activation
maps. In the second stage, the framework passes the DAMs over different
branches to capture patch-level and global-level information for actual
classification. The AD classification task was designed from the scratch,
whereas the MCI progression prediction model was built based upon the learned
parameters of the AD classification model. Compared to the existing competing
deep learning approaches, this framework offers little performance
improvement. The proposed framework identified different parts of the
hippocampus, frontal lobe, fusiform gyrus, amygdala, and ventricle in
association with AD or MCI. The discriminative power of these regions has been
supported by earlier studies conducted for Alzheimer’s disease. One of the
main limitations of this work is that DAMs cannot be considered as
explanations for the final predictive model. Because, a smaller part of the
predictive model was pre-trained earlier to generate the DAMs and subsequently
guide the predictive model for the intended classification task. Moreover,
DAMs as generated using class activation maps are high in semantics but low in
resolution, which may not reflect the fine details of the discriminative
regions in the input space.

Jin et al. [110] proposed a deep learning model combining ResNet and a 3D
attention network (3DAN) to integrate the identification of discriminative
regions and the classification of AD into a unified framework. The proposed
model can diagnose Alzheimer’s and simultaneously capture imaging biomarkers
responsible for the predictions. As reported, the 3DAN provides a strong
association between the model output and the clinical features of AD and MCI.
The study claimed the biomarkers (attention maps) to be generalizable,
reproducible, and neurobiologically plausible. The study validated these
claims by demonstrating strong correlations of the attention maps between
datasets, between the mean attention score and the T-map, between the
attention score for the regions and the MMSE scores based on the Brainnetome
Atlas, and between classification accuracy and the mean attention score of $K$
groups of regions. The attention mechanism was able to produce generalizable
and reproducible results as the findings correlated across the datasets.
Figure 16 shows various aspects of the analyses for the effectiveness of
attention maps in capturing significant brain regions associated with the
progression of AD.

Figure 16: Panel A: Attention maps highlight the temporal lobe, hippocampus,
parahippocampal gyrus, cingulate gyrus, thalamus, precuneus, insula, amygdala,
fusiform gyrus, and medial frontal cortex as discriminative regions of AD.
Noticeably, the attention patterns for both datasets were significantly
correlated. Panel B: Correlation analysis between T-map and the attention
scores of the regions. High correlation values between attention scores and
the group difference map for both datasets demonstrate the efficacy of the
model’s feature extraction. Panel C: Correlation analyses between the
attention scores and the Mini-Mental State Exam (MMSE) scores. Attention
scores of 81% and 77% brain regions correlated with MMSE significantly for In-
house and ADNI datasets, respectively. Panel D:  Correlation between mean
attention score of top $K$ regions and the classification performance. As can
be observed, the attention scores of top $K$ regions highly correlated with
the classification performance for both datasets. (Image Courtesy: [110])

Joint Training
Zhu et al. [94] combined interpretable feature learning and dynamic graph
learning modules into the Graph Convolutional Network module. These three
modules are jointly optimized to provide improved diagnosis and
interpretability simultaneously via learning only the essential features. This
study used feature summarization and used gray matter volumes from 90 ROIs as
features. Interpretability of the diagnosis was conducted based on the values
of the learned weights of each region. The predictive performance of the top
regions was higher compared to other feature selection methods. Moreover, the
proposed method consistently identified the middle temporal gyrus right,
hippocampal formation left, precuneus left, and uncus left associated with AD,
which aligned with the previous reports. While this validation seems
plausible, it is unclear if the underlying classification model for this
validation across the feature selection methods was the same or different.
Furthermore, uninterpretable parameter sensitivity is another concern, which
caused a significant drop in predictive performance.

Bass et al. [66] proposed a VAE-GAN-based approach, called ICAM (Interpretable
Classification via Disentangled Representations and Feature Attribution
Mapping), to learn a shared class-relevant attribute latent space, that is
simultaneously suitable for classification, and feature attribution. It can
also inform the difference within and between classes. The authors argued that
the post hoc methods are sub-optimal in finding all the discriminative regions
of a class and hence not suitable for medical imaging. Instead, this work
hypothesized that generative models are more useful to capture class-relevant
features. As demonstrated in the results, ICAM’s latent attribute space
achieved greater discriminative power compared to other approaches.
Furthermore, the attribution maps generated using this approach have increased
correlations with the ground truth disease maps compared to other popular post
hoc methods. Figure 17 shows comparative feature maps generated using ICAM,
VA-GAN, and other post-hoc approaches.

Figure 17:  Feature attribution (FA) maps using ICAM, VA-GAN and other post-
hoc methods for AD classification from ADNI dataset. ICAM better captures the
phenotypic variation in brain structure compared to other baseline methods. In
particular, ICAM, when compared with the ground-truth disease map, detects the
ventricles (blue arrows), cortex (green arrows), and hippocampus (pink
arrows). (Image Courtesy: [66])

Model Transparency
Qiu et al. [98] utilized multimodal inputs such as MRI and other clinical
features (age, gender, and Mini-mental State Examination score) to propose an
interpretable deep learning framework for Alzheimer’s disease. The proposed
framework improves predictive performance for disease diagnosis and identifies
disease-specific neuroimaging signatures. As part of the architecture, MRI
sub-volumes are passed to a fully convolutional neural network, and patient-
specific probability maps of the brain are generated. As estimated by the
probability maps, the high-risk voxels are passed to a fully connected network
for classification. Despite site-specific distinctions among the datasets, the
study was able to demonstrate performance consistency across the datasets when
training was conducted based on a single cohort. The study further provided
neuropathological and neurologist-level validation. For the neuropathological
validation, the high-probability brain regions were closely associated with
the locations and frequencies of amyloid-$\beta$ and tau pathologies. Figure
18 demonstrates how the extracted disease probability maps correspond to the
post-mortem findings of neuropathology examinations. For the neurologist-level
validation, eleven neurologists conducted diagnoses based on the same
multimodal inputs, and the average performance was compared to the model’s
performance. The model is transparent in the sense that it is able to directly
produce disease probability maps from the scans, which is indicative of the
disease-specific brain regions. While the preliminary results are promising,
the model only considered the subpopulation of binary scenarios, subjects with
AD and normal controls, and does not consider progressive stages. Hence, the
model is still not directly applicable to the clinical decision-making
process.

Figure 18: A: Overlap of the model’s predictions with the neuropathology for a
single subject (AD). The column (i) shows the MRI scans at different visual
planes (ii) depicts the predicted disease probability maps (iii) A thresholded
probability maps overlapped with the MRI scans, (iv) shows the segmented brain
masks with color-coding indicating different pathology levels, (v) shows an
overlay of MRI scan, thresholded probability maps, and the color-coded
pathology levels. B: For qualitative assessment, the model examined the
density of neurofibrillary tangles (NFT), diffuse senile (DP), and neuritic
(NPL) or compacted senile plaques in each brain region. As shown, the average
probabilities of the model predicted brain regions were consistently
correlated with a high grade of amyloid-$\beta$ and tau in different brain
regions. Biel = Bielschowsky stain; L = left; R = right. (Image Courtesy:
[98])

Ravi et al. [97] proposed a 4D-Degenerative Adversarial NeuroImage Net
(4D-DANINet) for generating realistic 3D brain images over time. 4D-DANINet is
a modular framework based on adversarial training and a set of spatiotemporal
and biological constraints. It can generate subject-specific longitudinal
scans that reflect disease stage and age. The key motivation behind this work
is to provide sufficient realistic samples for model validation and efficient
model training. The main components of the DANINet are as follows:

  * •

Conditional deep encoder: This module is a combination of an encoder that
embeds each slice to a latent space $z$ and a generator that generates samples
from this latent space conditioned on diagnosis and age. This module is
trained based on reconstruction loss that minimizes the difference between the
actual slice and the projected slice in time.

  * •

Discriminator networks: This module uses two discriminators $\mathcal{D}^{b}$
and $\mathcal{D}^{z}$. The $\mathcal{D}^{b}$ discriminates between real and
simulated brain images and the generator (G) generates realistic synthetic
images to fool the $\mathcal{D}^{b}$. The encoder (E) produces embeddings from
a uniform distribution to ensure smooth temporal progression. To do this, the
discriminator $\mathcal{D}^{z}$ is adversarially trained with the encoder (E).

  * •

Biological constraints: The 4D-DANNet imposes two separate voxel-level and
region-level losses $L^{\text{vox}}$ and $L^{\text{reg}}$ to capture smooth
intensity changes reflecting disease progression over time.

  * •

Profile weight functions dynamically determine appropriate weights for the
losses as required for efficient training.

The ablation study demonstrated the significance of training consistency (TC),
super-resolution (SR), and transfer learning (TR) blocks as adopted in the
framework to produce realistic synthetic MRI images. Figure 19 shows how
different components of the proposed framework have played their respective
roles in producing realistic synthetic MRI images. However, the model may
produce less effective brain images due to poorly representative training
sets, and cohort differences.

Figure 19: Ablation study demonstrating the performance of the proposed full
configuration L∗_TC_SR_TL in producing superior synthetic MRI. The L_TL
configuration lacks 3D consistency constraints and when training does not
converge, it results in artifacts appearing in sagittal and coronal axes
(yellow boxes). When configurations include 3D training consistency strategy
TC, such artifacts disappear from the resulting images (Refer to L_TC_TL and
L_TC_SR_TL configurations). Super-resolution does play an important role and
if not used, anatomical details are often not visible (red boxes). If SR only
model is used without TC, unrealistic images may appear for artifact super-
resolution (green boxes). If transfer learning TL procedure (fine-tuning) is
omitted during test time, lack of individualization may cause inaccurate
morphology (configuration L_TC_SR, blue boxes). (Image Courtesy: [97])

###  10.6 Feature Map Visualization

Biffi et al. [221] proposed a hierarchical deep generative model called ladder
variational autoencoder (LVAE). LVAE learns a hierarchy of conditional latent
variables to represent the population of anatomical segmentations. The latent
space representation in the highest level of the hierarchy can efficiently
discriminate clinical conditions. The proposed model performed two
classification tasks: 1) Hypertrophic cardiomyopathy (HCM) versus healthy 3D
left ventricular (LV) segmentations and 2) AD versus healthy control 3D
hippocampal segmentations. The model was predictive of clinical conditions and
offered suitable visualization and quantification of the anatomical shape
changes associated with those clinical conditions. This study used sampling in
the highest latent space to visualize the corresponding regions in the brain
space. The authors further claimed that the shape changes, as evident in the
visualization, agreed with the clinical literature.

Martinez-Murcia et al. [222] used a deep CNN autoencoder for an exploratory
data analysis of AD. The autoencoder demonstrates links between cognitive
symptoms and the underlying neurodegenerative process. The autoencoder model
uses a data-driven approach to extract imaging characteristics into low-
dimensional manifolds. The study further used regression analysis to show that
the neurons in the manifold space correlate well with the clinical and
neuropsychological test outcomes and diagnoses. Subsequently, the authors used
a novel visualization approach using a linear decomposition model to show the
brain regions highly influenced by each manifold coordinate, which provides
additional information about the association between structural degeneration
and the cognitive decline of dementia. Figure 20 shows the brain regions that
affect $14^{\text{th}}$ neuron (the most correlated with disease progression)
in the $z$-space using a linear decomposition method. The regression model for
different test clinical variables, such as ADAS-13, ADAS-11, Age, TAU, etc.,
was trained with GM maps.

Figure 20: Area of Influence of the $14^{\text{th}}$ neuron. The area of
influence highlighted parts of the temporal and parietal lobes, as well as a
shrinkage of the cerebellum. These brain regions are traditionally associated
with the progression of AD. Thus, the visualization confirms the underlying
association between anatomical changes of neurodegeneration and the magnitude
of the neuron in the $z$ space. (Image Courtesy: [222])

As brain dynamics change far before the changes happen in the anatomical
structure, functional neuroimaging is more powerful to understand brain
disorders [226]. Parmar et al., 2020 [226] proposed a modified 3D CNN that
directly works on the 4D resting-state fMRI data for a multiclass
classification task to diagnose different stages of Alzheimer’s disease. The
network achieved very high accuracy (93%) with only 30 subjects per class.
However, the data augmentation using temporal patches may prevent the spatio-
temporal characteristics of the underlying dynamics. The study also showed the
temporal features extracted from the first two convolutional layers as network
activation maps. As the temporal features move from lower to higher layers,
the authors reported that discriminative regions of interest gradually take a
definitive structure. The obvious limitation of this study is the lack of
interpretability. Moreover, the model was trained and evaluated on only one
dataset.

##  11 The Usage Trend of Interpretability Methods

While the neuroimaging community has used a larger collection of
interpretability methods, only a few are popular and considered important for
knowledge discovery or potential clinical deployment. Several interpretability
methods have often been used as experimental baselines, not for their
beneficial effects in this domain. In this section, we conducted an in-depth
analysis of the usage of all popular interpretability methods in neuroimaging
studies. Indeed, we investigated the usage of these methods in more than 300
neuroimaging papers and observed their usage trend as shown in Table
LABEL:usage_trend_table. As we found in our exploratory analysis, studies have
used the methods in the following order of frequency:  1) CAM/Grad-CAM/Grad-
CAM++/Guided Grad-CAM [139, 127, 140] 2) SHapley Additive exPlanations (SHAP)
[120] 3) Integrated Gradients [119] 4) Layer-wise Relevance Propagation [124]
5) Occlusion Sensitivity [116] 6) Guided Backpropagation [125] 7) Local
Interpretable Model-Agnostic Explanations (LIME) [121] 8) Gradients [117, 118]
9) DeepLIFT [122] and  10) Smoothgrad [126] This usage trend also reveals that
preference for "gradients" and "guided backpropagation" methods are receiving
less attention because of their limitations [126, 155], while the steeper rise
in the use of integrated gradients and SHAP are potentially due to their
strong theoretical foundations.

Table 2: The usage trend of popular post hoc interpretability methods in neuroimaging studies. Interpretabily Method & Citing Publications |  Usage Trend
---|---
Occlusion Sensitivity [116]: [240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 201, 258, 259, 260, 261, 262, 263, 58, 67, 264, 265, 266, 267, 268, 269, 270, 271, 66, 65, 272, 273, 80, 274, 275, 276, 68, 277, 278, 279, 280, 215, 87, 281] |
Gradients [117, 118]: [78, 282, 73, 69, 79, 283, 284, 285, 95, 286, 105, 88, 287, 288, 80, 274, 215, 289] |
CAM/Grad-CAM/Grad-CAM++/Guided Grad-CAM [139, 127, 140]: [290, 291, 292, 293, 294, 295, 296, 297, 282, 298, 299, 300, 301, 302, 303, 304, 305, 306, 244, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 81, 321, 69, 322, 129, 86, 323, 324, 325, 242, 326, 327, 328, 329, 330, 73, 331, 332, 333, 251, 334, 335, 78, 283, 336, 337, 338, 339, 284, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 286, 353, 354, 355, 356, 357, 358, 359, 360, 17, 218, 361, 362, 105, 58, 67, 17, 363, 364, 365, 366, 367, 368, 369, 261, 370, 371, 372, 260, 373, 95, 374, 375, 376, 377, 378, 271, 379, 380, 381, 382, 383, 384, 385, 386, 387, 388, 389, 390, 288, 391, 66, 75, 392, 393, 394, 395, 396, 397, 398, 399, 133, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411, 412, 413, 414, 415, 74, 416, 87, 417, 418, 419, 289, 420] |
Integrated Gradients [119]: [81, 315, 13, 421, 422, 423, 424, 69, 425, 426, 427, 428, 429, 430, 431, 432, 73, 102, 433, 79, 85, 84, 129, 251, 434, 435, 82, 67, 436, 83, 437, 66, 438, 439, 74] |
Smoothgrad [126]: [73, 69, 13, 89, 440, 441, 283, 88] |
Guided Backpropagation (GBP) [125]: [79, 440, 73, 242, 315, 250, 69, 251, 442, 443, 129, 374, 365, 201, 444, 445, 286, 283, 105, 446, 447, 67, 448, 378, 379, 449, 393, 450, 394, 133, 75, 66, 274, 68, 216, 215, 416, 415, 74] |
DeepLIFT [122]: [102, 69, 79, 129, 451, 452, 104, 105, 106, 285, 453, 287, 103] |
Layer-wise Relevance Propagation [124]: [442, 72, 454, 455, 456, 457, 458, 459, 79, 460, 461, 462, 129, 463, 464, 72, 107, 452, 69, 81, 465, 466, 262, 467, 468, 441, 469, 470, 471, 106, 446, 472, 473, 474, 475, 71, 476, 287, 75, 477, 478, 479, 480, 481, 32, 480, 216, 213, 68, 482, 70, 483, 484, 485, 274, 486, 487, 219, 488] |
SHapley Additive exPlanations (SHAP) [120]: [489, 490, 491, 492, 493, 494, 495, 496, 497, 64, 498, 499, 500, 498, 501, 502, 503, 504, 505, 506, 60, 507, 508, 509, 510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 56, 521, 522, 58, 523, 105, 57, 524, 225, 437, 59, 525, 526, 527, 528, 529, 530, 531] |
Local Interpretable Model-Agnostic Explanations (LIME) [121]: [491, 302, 62, 63, 64, 532, 489, 533, 534, 323, 523, 56, 535, 518, 536, 61, 537, 478, 538, 539, 540] |

##  12 Suggestions for Interpretable Models in Neuroimaging

In this section, we discuss the significant pitfalls of interpretability
research in neuroimaging. One of the obvious concerns in interpretable deep
learning models is that DL models are capable of learning in numerous ways for
the same input-output relationship [23] and in most cases the hidden factors
are not interpretable. Hence, the real challenge is to verify if the model
learned from the true evidence or relied on some unintended spurious
correlations [541, 44, 542] that could be entirely unknown to the humans. As
such, explanations vary widely among architectures, model initializations, and
interpretability methods.

Model architecture is also a major consideration. As different neural networks
may assign different regions as important for predictions, most of them would
tell about different aspects of the disorder because of their very nature of
different computations performed during training. Combining explanations from
different models and further analysis of these explanations in association
with medical experts may be useful in revealing undiscovered aspects of the
disease. To this end, a unified framework [32] in interpretable neuroimaging
research may be useful so that the findings across the studies can be directly
compared to share advancement benchmarks. Based on our analysis and review, we
recommend that we may focus on the following directions for useful DL-based
understanding of the disorders: 1) Objective quantification of the explanation
method’s performance, 2) Investigation of the explanation sensitivity to
interpretability parameters, 3) Understanding if the generated explanations
point to any causality or underlying mechanism of the disorder, 4)
Investigating the reliability of the underlying model via model debugging 5)
Combining various aspects revealed using multiple approaches, multiple model
initializations and model ensembles. This is important because even when
models use "true" evidence, explanations and their relative importance may be
different. So combining them in a faithful manner can reveal useful unknown
insights for the disorders. As Rieke et al. [215] pointed out that different
visualization methods (gradients or non-gradient approaches) vary widely, so
in line with other earlier studies, we suggest investigating multiple methods
instead of blindly relying on one interpretability method.

We also should be careful while choosing a particular interpretability method.
For example, while many earlier studies used the _occlusion sensitivity_
method to generate explanations, Yang et al., 2018 [87] pointed out several
limitations of the approach. For example, this approach uses semantically
meaningless neighborhoods and an unspecified way of choosing the grid size.
Moreover, the method is computationally very intensive. As no backpropagation
from the target score is involved during heatmap generation, this explanation
is considered to be limited [80]. Also, 3D-Grad-CAM can be useful if we need
to track the attention of the convolution layers, but Grad-CAM or CAM is not
useful for generating explanations in the input space and hence not suitable
for data interpretation. While LRP has been used extensively, it has inherent
limitations. LRP cannot maintain implementation invariance as it uses modified
back-propagation rules. For future interpretability practices, we leave the
following suggestions for the neuroimaging community:

  1. 1.

Be aware of shortcut learning: Deep learning models are very prone to fall
into the trap of shortcut learning because it always looks for the easiest
possible solution for the problem [44]. The understanding of the influence of
model architecture, training data, loss function, optimization parameters may
reveal the nature of shortcuts the model may learn and thus preclude the
possibility for model deployment in clinical practices or for guiding further
discovery. Moreover, expert knowledge in the neuroimaging domain may help
identify these undesirable behaviors.

  2. 2.

High accuracy does not necessarily indicate higher human comprehensibility: A
highly accurate model does not necessarily mean the model is more
interpretable or relies on correct hidden features. While it is always
preferable for clinicians, doctors, and scientists to find the true underlying
factors for a model’s prediction, it is very unlikely that model would rely on
the same set of features every time it starts with new initialization weights.
As indicated by Hinton in [23], with the deep learning models, it is not
trying to identify the correct “hidden factors” responsible for a particular
diagnosis. Instead, DL models could rely on a different set of hidden factors
in the data to model the relationship between input and output variables.
Hence, a highly accurate model may not have intelligible interpretations for
humans. Refer to Figure 3 and Figure 21 to get ideas that explanations can
immensely vary for different models even when the predictions remain the same.

Figure 21: Explanations generated using Grad-CAM, Guided Backpropagation, and
Guided Grad-CAM methods for _Inception v3_ (trained on ImageNet) model
predictions. Compared to Figure 3 explanations for _Resnet 50_ model, the
explanations for _Inception v3_ model are very different even for the same
predictions.

  3. 3.

Design strong out-of-distribution (o.o.d) tests: While the model may perform
well on the i.i.d test samples, it may not generalize well for the real-world
datasets because of associated distribution shifts [44]. Hence, for successful
deployment, we must design suitable o.o.d tests as suggested by Geirhos et al.
[44]. While it may be less hard for designing those tests for some problem
domains, building strong o.o.d test cases may not be so easy for neuroimaging.
However, multi-cohort datasets may help mitigate this problem.

  4. 4.

Check for stability in generated explanations: Even if the model architecture,
training data, and other hyperparameters remain same, each time the model is
refitted starting from different initialization, the model may learn a very
different set of features [23] which may or may not be desirable. This is
indeed a very similar effect which occurs for the o.o.d cases, where a
distribution-wise different test sample may end up with an untended behavior.
This instability issue for the same model architecture, on the other hand, may
happen during training time for different initializations.

  5. 5.

We need to be aware of the fragility of neural network interpretations: The
fundamental problem with the popular interpretability methods is their
robustness [543]. Ghorbani et al. [543] showed that interpretations based on
feature importance maps such as DeepLIFT, integrated gradients, and influence
functions are susceptible to adversarial attacks. Put another way, a
systematic perturbation of the input can lead to a very different
interpretation (heatmap) because of the complexity of input feature space in
deep neural networks. In neuroimaging, earlier studies, so far we are aware,
usually overlooked this fragility of the interpretations, which may lead to
misleading interpretations. Moreover, there is an inherent human bias to trust
the model as correct and look for interpretations only based on predictive
performance. While model inspection or debugging can be a hard problem in
neuroimaging, it should be an essential consideration for this safety-critical
domain.

  6. 6.

Lack of any guiding principle to select explanation methods: While studies
have leveraged different explanation methods for deep learning models, there
is little theoretical evidence or guiding principle to choose a method for a
particular study. Recently, Han et al. [25] demonstrated how different
explanation methods describe different neighborhoods and thus produce
different explanations. Some disagreement scenarios are common because there
could be differences in the underlying aspects the methods are investigating.
For example, permutation importance [544] and SHAP [120, 545] in case of model
overfitting may produce very different explanations. However, some
disagreement scenarios are not expected. For example, gradients and LIME
should produce similar interpretations because they both focus on local
neighborhoods. However, in practice, they produce very different explanations.
The authors in [25] also showed how some methods cannot recover the underlying
model and are entirely independent. The authors also provided valuable
suggestions on choosing interpretability methods based on the nature of the
data. They further suggested building an explanation method for the data for
which no explanation method from the literature is considered beneficial.

  7. 7.

Post hoc methods are blamed for being insufficient: As post hoc methods
heavily rely on the models they are applied to, the methods can only discover
the minimal discriminative parts sufficient for the prediction. For example,
while LRP and GBP have been shown to be able to identify homogeneous brain
regions, e.g., the hippocampus, they cannot identify heterogeneous regions,
e.g., cortical folds [68, 216].

  8. 8.

Attribution normalization and polarity considerations varied widely: For the
post-processing of different explanations, studies use an ad-hoc approach.
There has yet to be an agreement on how to post-process the heatmaps. This
agreement must correspond to the underlying model and the interpretability
method used. This necessity of the agreement is especially applicable to
gradient-based attribution methods. Studies used the sign information
differently to finalize the heatmaps. As the distribution of the values in the
explanation maps generated using different methods varies widely, the need for
agreed upon normalization has been a open research question [75].

  9. 9.

Studies generally use an ad-hoc approach to validate explanations: For the
validation of results, studies generally use informal and unreliable ways.
Sometimes they used intuitions, hypotheses, and earlier results to justify the
current attributions. These validation techniques are very susceptible and may
end up with misleading conclusions. As Levakov et al. [220] indicated, any
reasonable conclusions regarding the contributions should be made based on
common parts of the maps from multiple models. Furthermore, deep learning
models usually capture complex hierarchical and multivariate interactions.
Localizing the brain regions should only be considered as an approximation of
the significance. Even a small architectural modification can be a significant
determinant of model performance and feature attribution maps, as indicated by
Lin et al. [78].

  10. 10.

Validate explanations based on their predictability and expert evaluation:
While RAR [13] and ROAR [138] evaluations of the salient regions is promising
and may further enhance the trust in the significance of what the model has
learned, it may still need to be guaranteed that the model did not rely on
spurious correlations. The domain experts should confirm the validation of the
interpretations. Equivalently the explanations must match a significant
proportion of the expert-extracted knowledge. We suggest complementing
quantitative validation with neuro-scientifically valid explanations.

  11. 11.

Use structure-function fusion model for model diagnosis: Earlier studies, in
general, independently focused on the anatomical or functional aspects of the
dynamics. However, using both modalities simultaneously and corresponding
existing knowledge in each modality during explanation generation may provide
rigorous validation and bring trust in the explanations.

  12. 12.

Counterfactuals may reveal the underlying biological mechanism: Wachter et al.
[53] first introduced _counterfactual_ explanations to know about the
hypothetical reality that could alter the model’s decision. Dandi et al. [55]
refined the formulation to satisfy the different practical desiderata of
counterfactual explanations to make them useful in real-world applications. In
the context of neuroimaging, we believe _countefactual_ explanations may help
understand the underlying biological mechanism that potentially caused the
specific disorder in the first place. To our knowledge, no neuroimaging study
has ever used counterfactuals to understand the model’s decision-making
process.

  13. 13.

Layer-wise Relevance Propagation (LRP) needs further investigation: As seen
from the interpretability in neuroimaging literature, LRP has been widely
used, and its popularity is on an upward trend. However, the explanations
produced by LRP are not reliable. Indeed, Shrikumar et al. [122] showed a
strong connection between LRP and $gradient\odot input$, especially when all
the activations are piecewise linear as in ReLU or Leaky ReLU. Ancona et al.
[136] also showed that $\epsilon$-LRP is equivalent to the feature-wise
product of the input and the modified partial derivative. Kindermans et al.
[546] showed that DeConvNet, Guided BackProp, and LRP cannot produce the
theoretically correct explanation even for a linear model—the most
straightforward neural network.

  14. 14.

SHAP is popular, but it should not be trusted blindly: SHAP, though very
popular in the XAI community, has some issues. For example, SHAP assumes that
the features are independent, while they are very unlikely. While features may
be correlated, the algorithm may generate unrealistic observations (instances)
with permutations. Moreover, no explanation method produces explanations that
imply causality. SHAP indicates the importance of a feature based on the model
prediction, not the importance in the real world. Humans are very prone to
confirmation bias. It is not very uncommon that humans tend to create
narratives as a result of confirmation bias. The most important question is:
Did the model learn to predict for the right reasons? This question is vital
because machine learning models do not know about truths, and it only cares
about correlations, and proxy or secondary or less important variables may be
loosely or tightly correlated with the actual cause. They can be revealed as
very important features. Moreover, Kwon and Zou [547] recently showed that
SHAP is suboptimal in that it gives the same weight to all marginal
contributions for a feature ${\bm{x}}_{i}$, which may potentially lead to
attribution mistakes if different marginal contributions have different signal
and noise. The authors further proposed a simple modification of the original
SHAP, called WeightedSHAP, that estimates the weights automatically from the
data.

  15. 15.

Studies generally focused only on classification and regression tasks: While
many studies in interpretable deep learning models for general classification
tasks exist, further subgrouping into patient subtypes or clustering is still
a novel area. This lack of interpretability literature for clustering tasks is
equally true for neuroimaging and other domains. Very few studies did
projection transformation from the latent space to observe the area of
influence [222, 221].

  16. 16.

Effectiveness of transfer learning in neuroimaging needs justification: what
causes the increased accuracy? What knowledge does it transfer? Raghu et al.
[548] showed that transfer learning from natural images to medical images did
help little with performance. Instead, as the authors surmised, the slight
improvement may come from the over-parameterization of the standard models
trained on natural images. Moreover, studies are not certain about the aspects
of knowledge they are transferring from the natural image domain to the
medical image domain or from one disorder area to another.

##  13 Conclusion

This article comprehensively introduces the problem of _interpretability_ for
AI models and thus offers a field guide for future AI practitioners in the
neuroimaging domain. In the earlier sections, we discussed the philosophical
ground, dimensions, methods, and desirable axiomatic properties of model
interpretability for reliable knowledge discovery. We also provide a useful
taxonomy that directly points to all the major interpretability approaches and
their use cases in many neuroimaging studies. We further discuss different
sanity tests and evaluation metrics required to justify the validity of the
explanations generated by any post hoc method. In the later sections, we
discussed how deep learning approaches have been used widely in recent
neuroimaging studies. Indeed, we performed an in-depth analysis of usage
trends of the most prevailing interpretability methods. We reckon that these
analyses will be helpful for future neuroimaging practitioners looking for
ideas of how scientists are using these approaches for novel discoveries and
how model interpretability is changing the course of neuroimaging studies in
recent years. Lastly, we discuss different caveats of interpretability
practices and provide insights on how this specialized sub-field of AI can be
used wisely and meaningfully for better diagnosis, prognosis, and treatment of
brain disorders.

## References

  * [1] David P Goldberg and Peter Huxley.  Common mental disorders: a bio-social model. Tavistock/Routledge, 1992.
  * [2] Vince D Calhoun, Robyn Miller, Godfrey Pearlson, and Tulay Adalı.  The chronnectome: time-varying connectivity networks as the next frontier in fMRI data discovery.  Neuron, 84(2):262–274, 2014.
  * [3] Jing Sui, Rongtao Jiang, Juan Bustillo, and Vince Calhoun.  Neuroimaging-based individualized prediction of cognition and behavior for mental disorders and health: methods and promises.  Biological psychiatry, 2020.
  * [4] Peter C Mulders, Philip F van Eijndhoven, Aart H Schene, Christian F Beckmann, and Indira Tendolkar.  Resting-state functional connectivity in major depressive disorder: a review.  Neuroscience & Biobehavioral Reviews, 56:330–344, 2015.
  * [5] Julia M Sheffield and Deanna M Barch.  Cognition and resting-state functional connectivity in schizophrenia.  Neuroscience & Biobehavioral Reviews, 61:108–120, 2016.
  * [6] Eleni Zarogianni, Thomas WJ Moorhead, and Stephen M Lawrie.  Towards the identification of imaging biomarkers in schizophrenia, using multivariate pattern classification at a single-subject level.  NeuroImage: Clinical, 3:279–289, 2013.
  * [7] Petr Dluhovs, Daniel Schwarz, Wiepke Cahn, Neeltje van Haren, René Kahn, Filip vSpaniel, Jivrí Horávcek, Tomávs Kavspárek, and Hugo Schnack.  Multi-center machine learning in imaging psychiatry: a meta-model approach.  Neuroimage, 155:10–24, 2017.
  * [8] Hugo G Schnack.  Improving individual predictions: machine learning approaches for detecting and attacking heterogeneity in schizophrenia (and other psychiatric diseases).  Schizophrenia research, 214:34–42, 2019.
  * [9] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.  The elements of statistical learning: data mining, inference, and prediction, volume 2.  Springer, 2009.
  * [10] Ali Khazaee, Ata Ebrahimzadeh, and Abbas Babajani-Feremi.  Application of advanced machine learning methods on resting-state fMRI network for identification of mild cognitive impairment and Alzheimer’s disease.  Brain Imaging and Behavior, 10(3):799–817, Sep 2016.
  * [11] Sergey M Plis, Devon R Hjelm, Ruslan Salakhutdinov, Elena A Allen, Henry J Bockholt, Jeffrey D Long, Hans J Johnson, Jane S Paulsen, Jessica A Turner, and Vince D Calhoun.  Deep learning for neuroimaging: a validation study.  Frontiers in neuroscience, 8:229, 2014.
  * [12] Richard Bellman.  Adaptive control processes: a guided tour princeton university press.  Princeton, New Jersey, USA, page 96, 1961.
  * [13] Md Rahman, Usman Mahmood, Noah Lewis, Harshvardhan Gazula, Alex Fedorov, Zening Fu, Vince D Calhoun, Sergey M Plis, et al.  Interpreting models interpreting brain dynamics.  Scientific Reports, 12(1):1–15, 2022.
  * [14] Adrien Payan and Giovanni Montana.  Predicting alzheimer’s disease: a neuroimaging study with 3d convolutional neural networks.  arXiv preprint arXiv:1502.02506, 2015.
  * [15] Pinkal Patel, Priya Aggarwal, and Anubha Gupta.  Classification of schizophrenia versus normal subjects using deep learning.  In Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, page 28. ACM, 2016.
  * [16] Jyoti Islam and Yanqing Zhang.  Brain mri analysis for alzheimer’s disease diagnosis using an ensemble system of deep convolutional neural networks.  Brain informatics, 5(2):2, 2018.
  * [17] Xin Zhang, Liangxiu Han, Wenyong Zhu, Liang Sun, and Daoqiang Zhang.  An explainable 3d residual self-attention deep neural network for joint atrophy localization and alzheimer’s disease diagnosis using structural mri.  IEEE journal of biomedical and health informatics, 2021.
  * [18] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  nature, 521(7553):436–444, 2015.
  * [19] Steven A Hicks, Jonas L Isaksen, Vajira Thambawita, Jonas Ghouse, Gustav Ahlberg, Allan Linneberg, Niels Grarup, Inga Strümke, Christina Ellervik, Morten Salling Olesen, et al.  Explaining deep neural networks for knowledge discovery in electrocardiogram analysis.  Scientific Reports, 11(1):1–11, 2021.
  * [20] Amirata Ghorbani, David Ouyang, Abubakar Abid, Bryan He, Jonathan H Chen, Robert A Harrington, David H Liang, Euan A Ashley, and James Y Zou.  Deep learning interpretation of echocardiograms.  NPJ digital medicine, 3(1):1–10, 2020.
  * [21] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In International conference on machine learning, pages 1885–1894. PMLR, 2017.
  * [22] Ruth C Fong and Andrea Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In Proceedings of the IEEE international conference on computer vision, pages 3429–3437, 2017.
  * [23] Geoffrey Hinton.  Deep learning—a technology with the potential to transform health care.  Jama, 320(11):1101–1102, 2018.
  * [24] Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju.  The disagreement problem in explainable machine learning: A practitioner’s perspective.  arXiv preprint arXiv:2202.01602, 2022.
  * [25] Tessa Han, Suraj Srinivas, and Himabindu Lakkaraju.  Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations.  arXiv preprint arXiv:2206.01254, 2022.
  * [26] Sandra Vieira, Walter HL Pinaya, and Andrea Mechelli.  Using deep learning to investigate the neuroimaging correlates of psychiatric and neurological disorders: Methods and applications.  Neuroscience & Biobehavioral Reviews, 74:58–75, 2017.
  * [27] Li Zhang, Mingliang Wang, Mingxia Liu, and Daoqiang Zhang.  A survey on deep learning for neuroimaging-based brain disorder analysis.  Frontiers in neuroscience, 14, 2020.
  * [28] Daniel T Huff, Amy J Weisman, and Robert Jeraj.  Interpretation and visualization techniques for deep learning models in medical imaging.  Physics in Medicine & Biology, 66(4):04TR01, 2021.
  * [29] Elina Thibeau-Sutre, Sasha Collin, Ninon Burgos, and Olivier Colliot.  Interpretability of machine learning methods applied to neuroimaging.  arXiv preprint arXiv:2204.07005, 2022.
  * [30] Khansa Rasheed, Adnan Qayyum, Mohammed Ghaly, Ala Al-Fuqaha, Adeel Razi, and Junaid Qadir.  Explainable, trustworthy, and ethical machine learning for healthcare: A survey.  Computers in Biology and Medicine, page 106043, 2022.
  * [31] Zohaib Salahuddin, Henry C Woodruff, Avishek Chatterjee, and Philippe Lambin.  Transparency of deep neural networks for medical image analysis: A review of interpretability methods.  Computers in biology and medicine, 140:105111, 2022.
  * [32] Lada Kohoutová, Juyeon Heo, Sungmin Cha, Sungwoo Lee, Taesup Moon, Tor D Wager, and Choong-Wan Woo.  Toward a unified framework for interpreting machine-learning models in neuroimaging.  Nature protocols, 15(4):1399–1435, 2020.
  * [33] Carl G Hempel and Paul Oppenheim.  Studies in the logic of explanation.  Philosophy of science, 15(2):135–175, 1948.
  * [34] William Bechtel and Adele Abrahamsen.  Explanation: A mechanist alternative.  Studies in History and Philosophy of Science Part C: Studies in History and Philosophy of Biological and Biomedical Sciences, 36(2):421–441, 2005.
  * [35] David Lewis.  Philosophy papers (vol. 2), 1986.
  * [36] Zachary C Lipton.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  Queue, 16(3):31–57, 2018.
  * [37] Finale Doshi-Velez and Been Kim.  Towards a rigorous science of interpretable machine learning.  arXiv preprint arXiv:1702.08608, 2017.
  * [38] Tim Miller.  Explanation in artificial intelligence: Insights from the social sciences.  Artificial intelligence, 267:1–38, 2019.
  * [39] Daniel Kahneman.  Thinking, fast and slow.  Macmillan, 2011.
  * [40] Denis J Hilton.  Conversational processes and causal explanation.  Psychological Bulletin, 107(1):65, 1990.
  * [41] Maartje MA De Graaf and Bertram F Malle.  How people explain action (and autonomous intelligent systems should too).  In 2017 AAAI Fall Symposium Series, 2017.
  * [42] Tim Miller, Piers Howe, and Liz Sonenberg.  Explainable ai: Beware of inmates running the asylum or: How i learnt to stop worrying and love the social and behavioural sciences.  arXiv preprint arXiv:1712.00547, 2017.
  * [43] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross.  Gradient-based attribution methods.  In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pages 169–191. Springer, 2019.
  * [44] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann.  Shortcut learning in deep neural networks.  Nature Machine Intelligence, 2(11):665–673, 2020.
  * [45] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  ACM computing surveys (CSUR), 51(5):1–42, 2018.
  * [46] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.  Explaining explanations: An overview of interpretability of machine learning.  In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pages 80–89. IEEE, 2018.
  * [47] Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso.  Machine learning interpretability: A survey on methods and metrics.  Electronics, 8(8):832, 2019.
  * [48] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information fusion, 58:82–115, 2020.
  * [49] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis.  Explainable ai: A review of machine learning interpretability methods.  Entropy, 23(1):18, 2020.
  * [50] Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus-Robert Müller.  Explaining deep neural networks and beyond: A review of methods and applications.  Proceedings of the IEEE, 109(3):247–278, 2021.
  * [51] Gabrielle Ras, Ning Xie, Marcel van Gerven, and Derek Doran.  Explainable deep learning: A field guide for the uninitiated.  Journal of Artificial Intelligence Research, 73:329–397, 2022.
  * [52] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Methods for interpreting and understanding deep neural networks.  Digital Signal Processing, 73:1–15, 2018.
  * [53] Sandra Wachter, Brent Mittelstadt, and Chris Russell.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.  Harv. JL & Tech., 31:841, 2017.
  * [54] Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan.  Explaining machine learning classifiers through diverse counterfactual explanations.  In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 607–617, 2020.
  * [55] Susanne Dandl, Christoph Molnar, Martin Binder, and Bernd Bischl.  Multi-objective counterfactual explanations.  In International Conference on Parallel Problem Solving from Nature, pages 448–469. Springer, 2020.
  * [56] Angela Lombardi, Domenico Diacono, Nicola Amoroso, Alfonso Monaco, João Manuel RS Tavares, Roberto Bellotti, and Sabina Tangaro.  Explainable deep learning for personalized age prediction with brain morphology.  Frontiers in neuroscience, page 578, 2021.
  * [57] Shaker El-Sappagh, Jose M Alonso, SM Islam, Ahmad M Sultan, and Kyung Sup Kwak.  A multilayer multimodal detection and prediction model based on explainable artificial intelligence for alzheimer’s disease.  Scientific reports, 11(1):1–26, 2021.
  * [58] Yasemin Turkan and F Boray Tek.  Convolutional attention network for mri-based alzheimer’s disease classification and its interpretability analysis.  In 2021 6th International Conference on Computer Science and Engineering (UBMK), pages 1–6. IEEE, 2021.
  * [59] Ji Ye Chun, Mohammad SE Sendi, Jing Sui, Dongmei Zhi, and Vince D Calhoun.  Visualizing functional network connectivity difference between healthy control and major depressive disorder using an explainable machine-learning method.  In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 1424–1427. IEEE, 2020\.
  * [60] Sayan Ghosal, Qiang Chen, Giulio Pergola, Aaron L Goldman, William Ulrich, Daniel R Weinberger, and Archana Venkataraman.  A biologically interpretable graph convolutional network to link genetic risk pathways and neuroimaging markers of disease.  bioRxiv, pages 2021–05, 2022.
  * [61] Pavan Rajkumar Magesh, Richard Delwin Myloth, and Rijo Jackson Tom.  An explainable machine learning model for early detection of parkinson’s disease using lime on datscan imagery.  Computers in Biology and Medicine, 126:104041, 2020.
  * [62] Krishnakant V Saboo, Chang Hu, Yogatheesan Varatharajah, Scott A Przybelski, Robert I Reid, Christopher G Schwarz, Jonathan Graff-Radford, David S Knopman, Mary M Machulda, Michelle M Mielke, et al.  Deep learning identifies brain structures that predict cognition and explain heterogeneity in cognitive aging.  NeuroImage, 251:119020, 2022.
  * [63] Ma Magboo, A Sheila, and Vincent Peter C Magboo.  Explainable ai for autism classification in children.  In Agents and Multi-Agent Systems: Technologies and Applications 2022, pages 195–205. Springer, 2022.
  * [64] Loveleen Gaur, Mohan Bhandari, Tanvi Razdan, Saurav Mallik, and Zhongming Zhao.  Explanation-driven deep learning model for prediction of brain tumour status using mri image data.  Frontiers in Genetics, page 448, 2022.
  * [65] Anees Abrol, Manish Bhattarai, Alex Fedorov, Yuhui Du, Sergey Plis, Vince Calhoun, Alzheimer’s Disease Neuroimaging Initiative, et al.  Deep residual learning for neuroimaging: an application to predict progression to alzheimer’s disease.  Journal of neuroscience methods, 339:108701, 2020.
  * [66] Cher Bass, Mariana da Silva, Carole Sudre, Petru-Daniel Tudosiu, Stephen Smith, and Emma Robinson.  Icam: Interpretable classification via disentangled representations and feature attribution mapping.  Advances in Neural Information Processing Systems, 33:7697–7709, 2020.
  * [67] Cher Bass, Mariana da Silva, Carole Sudre, Logan ZJ Williams, Petru-Daniel Tudosiu, Fidel Alfaro-Almagro, Sean P Fitzgibbon, Matthew F Glasser, Stephen M Smith, and Emma C Robinson.  Icam-reg: Interpretable classification and regression with feature attribution for mapping neurological phenotypes in individual scans.  arXiv preprint arXiv:2103.02561, 2021.
  * [68] Fabian Eitel, Kerstin Ritter, Alzheimer’s Disease Neuroimaging Initiative (ADNI, et al.  Testing the robustness of attribution methods for convolutional neural networks in mri-based alzheimer’s disease classification.  In Interpretability of machine intelligence in medical image computing and multimodal learning for clinical decision support, pages 3–11. Springer, 2019.
  * [69] Armin W Thomas, Christopher Ré, and Russell A Poldrack.  Comparing interpretation methods in mental state decoding analyses with deep learning models.  arXiv preprint arXiv:2205.15581, 2022.
  * [70] Armin W Thomas, Hauke R Heekeren, Klaus-Robert Müller, and Wojciech Samek.  Analyzing neuroimaging data through recurrent deep learning models.  Frontiers in neuroscience, 13:1321, 2019.
  * [71] AI Korda, A Ruef, S Neufang, C Davatzikos, S Borgwardt, EM Meisenzahl, and N Koutsouleris.  Identification of voxel-based texture abnormalities as new biomarkers for schizophrenia and major depressive patients using layer-wise relevance propagation on deep learning decisions.  Psychiatry Research: Neuroimaging, 313:111303, 2021.
  * [72] Simon M Hofmann, Frauke Beyer, Sebastian Lapuschkin, Ole Goltermann, Markus Loeffler, Klaus-Robert Müller, Arno Villringer, Wojciech Samek, and A Veronica Witte.  Towards the interpretability of deep learning models for multi-modal neuroimaging: Finding structural changes of the ageing brain.  NeuroImage, 261:119504, 2022.
  * [73] Ramy A Zeineldin, Mohamed E Karar, Ziad Elshaer, Christian R Wirtz, Oliver Burgert, Franziska Mathis-Ullrich, et al.  Explainability of deep neural networks for mri analysis of brain tumors.  International Journal of Computer Assisted Radiology and Surgery, pages 1–11, 2022.
  * [74] Christian F Baumgartner, Lisa M Koch, Kerem Can Tezcan, Jia Xi Ang, and Ender Konukoglu.  Visual feature attribution using wasserstein gans.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8309–8319, 2018.
  * [75] Martin Dyrba, Arjun H Pallath, and Eman N Marzban.  Comparison of cnn visualization methods to aid model interpretability for detecting alzheimer’s disease.  In Bildverarbeitung für die Medizin 2020, pages 307–312. Springer, 2020.
  * [76] Tomonori Nakagawa, Manabu Ishida, Junpei Naito, Atsushi Nagai, Shuhei Yamaguchi, Keiichi Onoda, and Alzheimer’s Disease Neuroimaging Initiative.  Prediction of conversion to alzheimer’s disease using deep survival analysis of mri images.  Brain communications, 2(1):fcaa057, 2020.
  * [77] Christian Gerhard Tinauer.  Explainable Deep Learning of Multi-parametric MRI Data for Alzheimer’s Disease Classification.  PhD thesis, Graz University of Technology, 2020.
  * [78] Qiu-Hua Lin, Yan-Wei Niu, Jing Sui, Wen-Da Zhao, Chuanjun Zhuo, and Vince D Calhoun.  Sspnet: An interpretable 3d-cnn for classification of schizophrenia using phase maps of resting-state complex-valued fmri data.  Medical Image Analysis, 79:102430, 2022.
  * [79] Jian Cui and Bin Weng.  Towards best practice of interpreting deep learning models for eeg-based brain computer interfaces.  arXiv preprint arXiv:2202.06948, 2022.
  * [80] Kanghan Oh, Young-Chul Chung, Ko Woon Kim, Woo-Sung Kim, and Il-Seok Oh.  Classification and visualization of alzheimer’s disease using volumetric convolutional neural network and transfer learning.  Scientific Reports, 9(1):1–16, 2019.
  * [81] Di Wang, Nicolas Honnorat, Peter T Fox, Kerstin Ritter, Simon B Eickhoff, Sudha Seshadri, and Mohamad Habes.  Deep neural network heatmaps capture alzheimer’s disease patterns reported in a large meta-analysis of neuroimaging studies.  arXiv preprint arXiv:2207.11352, 2022.
  * [82] Hemanth Manjunatha and Ehsan T Esfahani.  Extracting interpretable eeg features from a deep learning model to assess the quality of human-robot co-manipulation.  In 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER), pages 339–342. IEEE, 2021.
  * [83] Kai-Yi Lin, Vincent Chin-Hung Chen, Yuan-Hsiung Tsai, Roger S McIntyre, and Jun-Cheng Weng.  Classification and visualization of chemotherapy-induced cognitive impairment in volumetric convolutional neural networks.  Journal of Personalized Medicine, 11(10):1025, 2021.
  * [84] Hyoungshin Choi, Kyoungseob Byeon, Jong-eun Lee, Seok-Jun Hong, Bo-yong Park, and Hyunjin Park.  Subgroups of eating behavior traits independent of obesity defined using functional connectivity and feature representation learning.  bioRxiv, 2022.
  * [85] Kaustubh Supekar, Carlo de Los Angeles, Srikanth Ryali, Kaidi Cao, Tengyu Ma, and Vinod Menon.  Deep learning identifies robust gender differences in functional brain organization and their dissociable links to clinical symptoms in autism.  The British Journal of Psychiatry, 220(4):202–209, 2022.
  * [86] Xiangjun Chen, Zhaohui Wang, Yuefu Zhan, Faouzi Alaya Cheikh, and Mohib Ullah.  Interpretable learning approaches in structural mri: 3d-resnet fused attention for autism spectrum disorder classification.  In Medical Imaging 2022: Computer-Aided Diagnosis, volume 12033, pages 611–618. SPIE, 2022.
  * [87] Chengliang Yang, Anand Rangarajan, and Sanjay Ranka.  Visual explanations from deep 3d convolutional neural networks for alzheimer’s disease classification.  In AMIA annual symposium proceedings, volume 2018, page 1571. American Medical Informatics Association, 2018.
  * [88] Patrick McClure, Dustin Moraczewski, Ka Chun Lam, Adam Thomas, and Francisco Pereira.  Improving the interpretability of fmri decoding using deep neural networks and adversarial robustness.  arXiv preprint arXiv:2004.11114, 2020.
  * [89] Caihua Wang, Yuanzhong Li, Yukihiro Tsuboshita, Takuya Sakurai, Tsubasa Goto, Hiroyuki Yamaguchi, Yuichi Yamashita, Atsushi Sekiguchi, and Hisateru Tachimori.  A high-generalizability machine learning framework for predicting the progression of alzheimer’s disease using limited data.  NPJ digital medicine, 5(1):1–10, 2022.
  * [90] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.  Datamodels: Understanding predictions with data and data with predictions.  In International Conference on Machine Learning, pages 9525–9587. PMLR, 2022.
  * [91] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [92] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [93] Guangqi Wen, Peng Cao, Huiwen Bao, Wenju Yang, Tong Zheng, and Osmar Zaiane.  Mvs-gcn: A prior brain structure learning-guided multi-view graph convolution network for autism spectrum disorder diagnosis.  Computers in Biology and Medicine, 142:105239, 2022.
  * [94] Yonghua Zhu, Junbo Ma, Changan Yuan, and Xiaofeng Zhu.  Interpretable learning based dynamic graph convolutional networks for alzheimer’s disease analysis.  Information Fusion, 77:53–61, 2022.
  * [95] Matthias Wilms, Pauline Mouches, Jordan J Bannister, Deepthi Rajashekar, Sönke Langner, and Nils D Forkert.  Towards self-explainable classifiers and regressors in neuroimaging with normalizing flows.  In International Workshop on Machine Learning in Clinical Neuroimaging, pages 23–33. Springer, 2021.
  * [96] David Wood, James Cole, and Thomas Booth.  Neuro-dram: a 3d recurrent visual attention model for interpretable neuroimaging classification.  arXiv preprint arXiv:1910.04721, 2019.
  * [97] Daniele Ravi, Stefano B Blumberg, Silvia Ingala, Frederik Barkhof, Daniel C Alexander, Neil P Oxtoby, Alzheimer’s Disease Neuroimaging Initiative, et al.  Degenerative adversarial neuroimage nets for brain scan simulations: Application in ageing and dementia.  Medical Image Analysis, 75:102257, 2022.
  * [98] Shangran Qiu, Prajakta S Joshi, Matthew I Miller, Chonghua Xue, Xiao Zhou, Cody Karjadi, Gary H Chang, Anant S Joshi, Brigid Dwyer, Shuhan Zhu, et al.  Development and validation of an interpretable deep learning framework for alzheimer’s disease classification.  Brain, 143(6):1920–1933, 2020.
  * [99] Eunho Lee, Jun-Sik Choi, Minjeong Kim, Heung-Il Suk, Alzheimer’s Disease Neuroimaging Initiative, et al.  Toward an interpretable alzheimer’s disease diagnostic model with regional abnormality representation via deep learning.  Neuroimage, 202:116113, 2019.
  * [100] Usman Mahmood, Zening Fu, Satrajit Ghosh, Vince Calhoun, and Sergey Plis.  Through the looking glass: Deep interpretable dynamic directed connectivity in resting fmri.  NeuroImage, 264:119737, 2022.
  * [101] Yu Zhao, Junwei Han, Lei Guo, and Tianming Liu.  Modeling task fmri data via deep convolutional autoencoder.  In Information Processing in Medical Imaging: 25th International Conference, IPMI 2017, Boone, NC, USA, June 25-30, 2017, Proceedings, volume 10265, page 411. Springer, 2017.
  * [102] Pindi Krishna Chandra Prasad, Yash Khare, Kamalaker Dadi, PK Vinod, and Bapi Raju Surampudi.  Deep learning approach for classification and interpretation of autism spectrum disorder.  In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2022.
  * [103] Sukrit Gupta, Yi Hao Chan, Jagath C Rajapakse, Alzheimer’s Disease Neuroimaging Initiative, et al.  Decoding brain functional connectivity implicated in ad and mci.  In International conference on medical image computing and computer-assisted intervention, pages 781–789. Springer, 2019.
  * [104] Sukrit Gupta, Yi Hao Chan, Jagath C Rajapakse, Alzheimer’s Disease Neuroimaging Initiative, et al.  Obtaining leaner deep neural networks for decoding brain functional connectome in a single shot.  Neurocomputing, 453:326–336, 2021.
  * [105] Theerasarn Pianpanit, Sermkiat Lolak, Phattarapong Sawangjai, Thapanun Sudhawiyangkul, and Theerawit Wilaiprasitporn.  Parkinson’s disease recognition using spect image and interpretable ai: A tutorial.  IEEE Sensors Journal, 21(20):22304–22316, 2021.
  * [106] James D Bennett, Sam E John, David B Grayden, and Anthony N Burkitt.  Universal neurophysiological interpretation of eeg brain-computer interfaces.  In 2021 9th International Winter Conference on Brain-Computer Interface (BCI), pages 1–6. IEEE, 2021.
  * [107] Min Zhao, Weizheng Yan, Na Luo, Dongmei Zhi, Zening Fu, Yuhui Du, Shan Yu, Tianzi Jiang, Vince D Calhoun, and Jing Sui.  An attention-based hybrid deep learning framework integrating brain connectivity and activity of resting-state functional mri data.  Medical image analysis, 78:102413, 2022.
  * [108] Dan Jin, Jian Xu, Kun Zhao, Fangzhou Hu, Zhengyi Yang, Bing Liu, Tianzi Jiang, and Yong Liu.  Attention-based 3d convolutional network for alzheimer’s disease diagnosis and biomarkers exploration.  In 2019 IEEE 16Th international symposium on biomedical imaging (ISBI 2019), pages 1047–1051. IEEE, 2019.
  * [109] Guanghui Fu, Jianqiang Li, Ruiqian Wang, Yue Ma, and Yueda Chen.  Attention-based full slice brain ct image diagnosis with explanations.  Neurocomputing, 452:263–274, 2021.
  * [110] Dan Jin, Bo Zhou, Ying Han, Jiaji Ren, Tong Han, Bing Liu, Jie Lu, Chengyuan Song, Pan Wang, Dawei Wang, et al.  Generalizable, reproducible, and neuroscientifically interpretable imaging biomarkers for alzheimer’s disease.  Advanced Science, 7(14):2000675, 2020.
  * [111] Huzheng Yang, Xiaoxiao Li, Yifan Wu, Siyi Li, Su Lu, James S Duncan, James C Gee, and Shi Gu.  Interpretable multimodality embedding of cerebral cortex using attention graph network for identifying bipolar disorder.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 799–807. Springer, 2019.
  * [112] Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele, and Trevor Darrell.  Generating visual explanations.  In European conference on computer vision, pages 3–19. Springer, 2016.
  * [113] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  From recognition to cognition: Visual commonsense reasoning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019.
  * [114] Hui Liu, Qingyu Yin, and William Yang Wang.  Towards explainable nlp: A generative explanation framework for text classification.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5570–5581, 2019.
  * [115] Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu.  Multiple object recognition with visual attention.  arXiv preprint arXiv:1412.7755, 2014.
  * [116] Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * [117] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller.  How to explain individual classification decisions.  The Journal of Machine Learning Research, 11:1803–1831, 2010.
  * [118] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv preprint arXiv:1312.6034, 2013.
  * [119] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  arXiv preprint arXiv:1703.01365, 2017.
  * [120] Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  Advances in neural information processing systems, 30, 2017.
  * [121] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  "Why should i trust you?" explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * [122] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In International Conference on Machine Learning, pages 3145–3153. PMLR, 2017.
  * [123] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller.  Explaining nonlinear classification decisions with deep taylor decomposition.  Pattern Recognition, 65:211–222, 2017.
  * [124] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * [125] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller.  Striving for simplicity: The all convolutional net.  arXiv preprint arXiv:1412.6806, 2014.
  * [126] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  arXiv preprint arXiv:1706.03825, 2017.
  * [127] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * [128] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje.  Not just a black box: Learning important features through propagating activation differences.  arXiv preprint arXiv:1605.01713, 2016.
  * [129] Kwanseok Oh, Jee Seok Yoon, and Heung-Il Suk.  Learn-explain-reinforce: Counterfactual reasoning and its guidance to reinforce an alzheimer’s disease diagnosis model.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [130] Carlo Abrate and Francesco Bonchi.  Counterfactual graphs for explainable classification of brain networks.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2495–2504, 2021.
  * [131] Matthias Wilms, Jordan J Bannister, Pauline Mouches, M Ethan MacDonald, Deepthi Rajashekar, Sönke Langner, and Nils D Forkert.  Invertible modeling of bidirectional relationships in neuroimaging with normalizing flows: Application to brain aging.  IEEE Transactions on Medical Imaging, 41(9):2331–2347, 2022.
  * [132] Kwanseok Oh, Da-Woon Heo, Ahmad Wisnu Mulyadi, Wonsik Jung, Eunsong Kang, Kun Ho Lee, and Heung-Il Suk.  Deep counterfactual-guided mri feature representation and quantitatively interpretable alzheimer’s disease prediction.  2022\.
  * [133] Maxim Kan, Ruslan Aliev, Anna Rudenko, Nikita Drobyshev, Nikita Petrashen, Ekaterina Kondrateva, Maxim Sharaev, Alexander Bernstein, and Evgeny Burnaev.  Interpretation of 3d cnns for brain mri data classification.  In International Conference on Analysis of Images, Social Networks and Texts, pages 229–241. Springer, 2020.
  * [134] Hristina Uzunova, Jan Ehrhardt, Timo Kepp, and Heinz Handels.  Interpretable explanations of black box classifiers applied on medical images by meaningful perturbations using variational autoencoders.  In Medical Imaging 2019: Image Processing, volume 10949, pages 264–271. SPIE, 2019.
  * [135] Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller.  Evaluating the visualization of what a deep neural network has learned.  IEEE transactions on neural networks and learning systems, 28(11):2660–2673, 2016.
  * [136] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross.  Towards better understanding of gradient-based attribution methods for deep neural networks.  arXiv preprint arXiv:1711.06104, 2017.
  * [137] Junghoon Seo, Jeongyeol Choe, Jamyoung Koo, Seunghyeon Jeon, Beomsu Kim, and Taegyun Jeon.  Noise-adding methods of saliency map as series of higher order partial derivative.  arXiv preprint arXiv:1806.03000, 2018.
  * [138] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim.  A benchmark for interpretability methods in deep neural networks.  In Advances in Neural Information Processing Systems, pages 9737–9748, 2019.
  * [139] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.
  * [140] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In 2018 IEEE winter conference on applications of computer vision (WACV), pages 839–847. IEEE, 2018.
  * [141] Haofan Wang, Zifan Wang, Mengnan Du, Fan Yang, Zijian Zhang, Sirui Ding, Piotr Mardziel, and Xia Hu.  Score-cam: Score-weighted visual explanations for convolutional neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 24–25, 2020.
  * [142] Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas, and Michael Terry.  Xrai: Better attributions through regions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4948–4957, 2019.
  * [143] Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci, Ben Wedin, Michael Terry, and Tolga Bolukbasi.  Guided integrated gradients: An adaptive path method for removing noise.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5050–5058, 2021.
  * [144] Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran.  Explainable deep learning: A field guide for the uninitiated.  2020\.
  * [145] Christoph Molnar.  Interpretable machine learning.  Lulu. com, 2020.
  * [146] Amir-Hossein Karimi, Gilles Barthe, Borja Balle, and Isabel Valera.  Model-agnostic counterfactual explanations for consequential decisions.  In International Conference on Artificial Intelligence and Statistics, pages 895–905. PMLR, 2020.
  * [147] Frank R Hampel, Elvezio M Ronchetti, Peter Rousseeuw, and Werner A Stahel.  Robust statistics: the approach based on influence functions.  Wiley-Interscience; New York, 1986.
  * [148] Samyadeep Basu, Xuchen You, and Soheil Feizi.  On second-order group influence functions for black-box predictions.  In International Conference on Machine Learning, pages 715–724. PMLR, 2020.
  * [149] Benbo Zha and Hong Shen.  Ifme: Influence function based model explanation for black box decision systems.  In International Symposium on Parallel Architectures, Algorithms and Programming, pages 299–310. Springer, 2020.
  * [150] Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov.  Explaining black box predictions and unveiling data artifacts through influence functions.  arXiv preprint arXiv:2005.06676, 2020.
  * [151] Vitaly Feldman and Chiyuan Zhang.  What neural networks memorize and why: Discovering the long tail via influence estimation.  Advances in Neural Information Processing Systems, 33:2881–2891, 2020.
  * [152] Amirata Ghorbani and James Zou.  Data shapley: Equitable valuation of data for machine learning.  In International Conference on Machine Learning, pages 2242–2251. PMLR, 2019.
  * [153] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos.  Towards efficient data valuation based on the shapley value.  In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167–1176. PMLR, 2019.
  * [154] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert.  Feature visualization.  Distill, 2(11):e7, 2017.
  * [155] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  Advances in neural information processing systems, 31:9505–9515, 2018.
  * [156] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv preprint arXiv:1412.6572, 2014.
  * [157] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199, 2013.
  * [158] Leila Arras, Ahmed Osman, and Wojciech Samek.  Clevr-xai: a benchmark dataset for the ground truth evaluation of neural network explanations.  Information Fusion, 81:14–40, 2022.
  * [159] Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Suggala, David I Inouye, and Pradeep K Ravikumar.  On the (in) fidelity and sensitivity of explanations.  Advances in Neural Information Processing Systems, 32:10967–10978, 2019.
  * [160] Pascal Sturmfels, Scott Lundberg, and Su-In Lee.  Visualizing the impact of feature attribution baselines.  Distill, 2020.  https://distill.pub/2020/attribution-baselines.
  * [161] Piotr Dabkowski and Yarin Gal.  Real time image saliency for black box classifiers.  Advances in neural information processing systems, 30, 2017.
  * [162] Yao Rong, Tobias Leemann, Vadim Borisov, Gjergji Kasneci, and Enkelejda Kasneci.  A consistent and efficient evaluation strategy for attribution methods.  In International Conference on Machine Learning, pages 18770–18795. PMLR, 2022.
  * [163] Erico Tjoa and Cuntai Guan.  A survey on explainable artificial intelligence (xai): Toward medical xai.  IEEE Transactions on Neural Networks and Learning Systems, 2020\.
  * [164] Cynthia Rudin.  Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1(5):206–215, 2019.
  * [165] David Leslie.  Understanding artificial intelligence ethics and safety.  arXiv preprint arXiv:1906.05684, 2019.
  * [166] Octavio Loyola-Gonzalez.  Black-box vs. white-box: Understanding their advantages and weaknesses from a practical point of view.  IEEE Access, 7:154096–154113, 2019.
  * [167] Cynthia Rudin and Joanna Radin.  Why are we using black box models in ai when we don’t need to? a lesson from an explainable ai competition.  Harvard Data Science Review, 1(2):10–1162, 2019.
  * [168] Himabindu Lakkaraju and Osbert Bastani.  " how do i fool you?" manipulating user trust via misleading black box explanations.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 79–85, 2020.
  * [169] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki.  The dangers of post-hoc interpretability: Unjustified counterfactual explanations.  arXiv preprint arXiv:1907.09294, 2019.
  * [170] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, and Marcin Detyniecki.  Issues with post-hoc counterfactual explanations: a discussion.  arXiv preprint arXiv:1906.04774, 2019.
  * [171] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju.  Fooling lime and shap: Adversarial attacks on post hoc explanation methods.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180–186, 2020.
  * [172] Jean-Marie John-Mathews.  Some critical and ethical perspectives on the empirical turn of ai interpretability.  Technological Forecasting and Social Change, 174:121209, 2022.
  * [173] Lok Chan.  Explainable ai as epistemic representation.  Overcoming Opacity in Machine Learning, 7, 2021.
  * [174] Thibault Laugel, Marie-Jeanne Lesot, Christophe Marsala, Xavier Renard, and Marcin Detyniecki.  Unjustified classification regions and counterfactual explanations in machine learning.  In Joint European conference on machine learning and knowledge discovery in databases, pages 37–54. Springer, 2019.
  * [175] Michele Loi, Andrea Ferrario, and Eleonora Viganò.  Transparency as design publicity: explaining and justifying inscrutable algorithms.  Ethics and Information Technology, 23(3):253–263, 2021.
  * [176] Jean-Marie John-Mathews.  Critical empirical study on black-box explanations in ai.  arXiv preprint arXiv:2109.15067, 2021.
  * [177] Joshua Hatherley, Robert Sparrow, and Mark Howard.  The virtues of interpretable medical artificial intelligence.
  * [178] Andrés Páez.  The pragmatic turn in explainable artificial intelligence (xai).  Minds and Machines, 29(3):441–459, 2019.
  * [179] Nishanth Arun, Nathan Gaw, Praveer Singh, Ken Chang, Mehak Aggarwal, Bryan Chen, Katharina Hoebel, Sharut Gupta, Jay Patel, Mishka Gidwani, et al.  Assessing the (un) trustworthiness of saliency maps for localizing abnormalities in medical imaging.  Radiology: Artificial Intelligence, page e200267, 2021.
  * [180] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim.  The (un) reliability of saliency methods.  In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pages 267–280. Springer, 2019.
  * [181] Bernease Herman.  The promise and peril of human evaluation for model interpretability.  arXiv preprint arXiv:1711.07414, 2017.
  * [182] Amina Adadi and Mohammed Berrada.  Peeking inside the black-box: a survey on explainable artificial intelligence (xai).  IEEE access, 6:52138–52160, 2018.
  * [183] John Ashburner and Karl J Friston.  Voxel-based morphometry—the methods.  Neuroimage, 11(6):805–821, 2000.
  * [184] Stefan Klöppel, Cynthia M Stonnington, Carlton Chu, Bogdan Draganski, Rachael I Scahill, Jonathan D Rohrer, Nick C Fox, Clifford R Jack Jr, John Ashburner, and Richard SJ Frackowiak.  Automatic classification of mr scans in alzheimer’s disease.  Brain, 131(3):681–689, 2008.
  * [185] Saima Farhan, Muhammad Abuzar Fahiem, and Huma Tauseef.  An ensemble-of-classifiers based approach for early diagnosis of alzheimer’s disease: classification using structural features of brain images.  Computational and mathematical methods in medicine, 2014, 2014.
  * [186] Hugo G Schnack, Mireille Nieuwenhuis, Neeltje EM van Haren, Lucija Abramovic, Thomas W Scheewe, Rachel M Brouwer, Hilleke E Hulshoff Pol, and René S Kahn.  Can structural mri aid in clinical classification? a machine learning study in two independent samples of patients with schizophrenia, bipolar disorder and healthy subjects.  Neuroimage, 84:299–306, 2014.
  * [187] Yuan Xiao, Zhihan Yan, Youjin Zhao, Bo Tao, Huaiqiang Sun, Fei Li, Li Yao, Wenjing Zhang, Shah Chandan, Jieke Liu, et al.  Support vector machine-based classification of first episode drug-naïve schizophrenia patients and healthy controls using structural mri.  Schizophrenia research, 214:11–17, 2019.
  * [188] Vaughn R Steele, Vikram Rao, Vince D Calhoun, and Kent A Kiehl.  Machine learning of structural magnetic resonance imaging predicts psychopathic traits in adolescent offenders.  Neuroimage, 145:265–273, 2017.
  * [189] Benoît Magnin, Lilia Mesrob, Serge Kinkingnéhun, Mélanie Pélégrini-Issac, Olivier Colliot, Marie Sarazin, Bruno Dubois, Stéphane Lehéricy, and Habib Benali.  Support vector machine-based classification of alzheimer’s disease from whole-brain anatomical mri.  Neuroradiology, 51(2):73–83, 2009.
  * [190] Daoqiang Zhang, Yaping Wang, Luping Zhou, Hong Yuan, Dinggang Shen, Alzheimer’s Disease Neuroimaging Initiative, et al.  Multimodal classification of alzheimer’s disease and mild cognitive impairment.  Neuroimage, 55(3):856–867, 2011.
  * [191] Barnaly Rashid, Mohammad R Arbabshirani, Eswar Damaraju, Mustafa S Cetin, Robyn Miller, Godfrey D Pearlson, and Vince D Calhoun.  Classification of schizophrenia and bipolar patients using static and dynamic resting-state fmri brain connectivity.  Neuroimage, 134:645–657, 2016.
  * [192] Samuel Iddi, Dan Li, Paul S Aisen, Michael S Rafii, Wesley K Thompson, Michael C Donohue, Alzheimer’s Disease Neuroimaging Initiative, et al.  Predicting the course of alzheimer’s progression.  Brain informatics, 6(1):6, 2019.
  * [193] Hui Shen, Lubin Wang, Yadong Liu, and Dewen Hu.  Discriminative analysis of resting-state functional connectivity patterns of schizophrenia using low dimensional embedding of fmri.  Neuroimage, 49(4):3110–3121, 2010.
  * [194] Archana Venkataraman, Thomas J Whitford, Carl-Fredrik Westin, Polina Golland, and Marek Kubicki.  Whole brain resting state functional connectivity abnormalities in schizophrenia.  Schizophrenia research, 139(1-3):7–12, 2012.
  * [195] Ling-Li Zeng, Huaning Wang, Panpan Hu, Bo Yang, Weidan Pu, Hui Shen, Xingui Chen, Zhening Liu, Hong Yin, Qingrong Tan, et al.  Multi-site diagnostic classification of schizophrenia using discriminant deep learning with functional connectivity mri.  EBioMedicine, 30:74–85, 2018.
  * [196] Benson Mwangi, Tian Siva Tian, and Jair C Soares.  A review of feature reduction techniques in neuroimaging.  Neuroinformatics, 12(2):229–244, 2014.
  * [197] Arthur Mensch, Julien Mairal, Danilo Bzdok, Bertrand Thirion, and Gaël Varoquaux.  Learning neural representations of human cognition across many fMRI studies.  In Advances in Neural Information Processing Systems, pages 5883–5893, 2017.
  * [198] Saman Sarraf and Ghassem Tofighi.  Classification of alzheimer’s disease using fmri data and deep learning convolutional neural networks.  arXiv preprint arXiv:1603.08631, 2016.
  * [199] Dong Nie, Han Zhang, Ehsan Adeli, Luyan Liu, and Dinggang Shen.  3d deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients.  In International conference on medical image computing and computer-assisted intervention, pages 212–220. Springer, 2016.
  * [200] Usman Mahmood, Md Mahfuzur Rahman, Alex Fedorov, Noah Lewis, Zening Fu, V. D. Calhoun, and Sergey Plis.  Whole MILC: generalizing learned dynamics across tasks, datasets, and populations.  In Proceedings of the 23rd international conference on medical image computing and computer assisted intervention (MICCAI), pages 407–417. Springer, 2020.
  * [201] Anees Abrol, Zening Fu, Mustafa Salman, Rogers Silva, Yuhui Du, Sergey Plis, and Vince Calhoun.  Deep learning encodes robust discriminative neuroimaging representations to outperform standard machine learning.  Nature communications, 12(1):1–17, 2021.
  * [202] Kyle Young, Gareth Booth, Becks Simpson, Reuben Dutton, and Sally Shrapnel.  Deep neural network or dermatologist?  In Interpretability of Machine Intelligence in Medical Image Computing and Multimodal Learning for Clinical Decision Support, pages 48–55. Springer, 2019.
  * [203] Alfredo Vellido.  The importance of interpretability and visualization in machine learning for applications in medicine and health care.  Neural computing and applications, pages 1–15, 2019.
  * [204] Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed.  On interpretability of deep learning based skin lesion classifiers using concept activation vectors.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–10. IEEE, 2020.
  * [205] Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Klaus-Robert Muller, and Wojciech Samek.  Analyzing classifiers: Fisher vectors and deep neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2912–2920, 2016.
  * [206] Danilo Bzdok, Michael Eickenberg, Olivier Grisel, Bertrand Thirion, and Gaël Varoquaux.  Semi-supervised factored logistic regression for high-dimensional neuroimaging data.  Advances in neural information processing systems, 28, 2015.
  * [207] Ms Aayushi Bansal, Dr Rewa Sharma, and Dr Mamta Kathuria.  A systematic review on data scarcity problem in deep learning: solution and applications.  ACM Computing Surveys (CSUR), 54(10s):1–29, 2022.
  * [208] Drew Landis, William Courtney, Christopher Dieringer, Ross Kelly, Margaret King, Brittny Miller, Runtang Wang, Dylan Wood, Jessica A Turner, and Vince D Calhoun.  Coins data exchange: An open platform for compiling, curating, and disseminating neuroimaging data.  NeuroImage, 124:1084–1088, 2016.
  * [209] Armin W Thomas, Klaus-Robert Müller, and Wojciech Samek.  Deep transfer learning for whole-brain fMRI analyses.  arXiv preprint arXiv:1907.01953, pages 59–67, 2019.
  * [210] Usman Mahmood, Md Mahfuzur Rahman, Alex Fedorov, Zening Fu, and Sergey Plis.  Transfer learning of fMRI dynamics.  arXiv preprint arXiv:1911.06813, 2019.
  * [211] Alejandro Newell and Jia Deng.  How useful is self-supervised pretraining for visual tasks?  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7345–7354, 2020.
  * [212] Esten H Leonardsen, Han Peng, Tobias Kaufmann, Ingrid Agartz, Ole A Andreassen, Elisabeth Gulowsen Celius, Thomas Espeseth, Hanne F Harbo, Einar A Høgestøl, Ann-Marie de Lange, et al.  Deep neural networks learn general and clinically relevant representations of the ageing brain.  NeuroImage, 256:119210, 2022.
  * [213] Fabian Eitel, Emily Soehler, Judith Bellmann-Strobl, Alexander U Brandt, Klemens Ruprecht, René M Giess, Joseph Kuchling, Susanna Asseyer, Martin Weygandt, John-Dylan Haynes, et al.  Uncovering convolutional neural network decisions for diagnosing multiple sclerosis on conventional mri using layer-wise relevance propagation.  NeuroImage: Clinical, 24:102003, 2019.
  * [214] Juan Miguel Valverde, Vandad Imani, Ali Abdollahzadeh, Riccardo De Feo, Mithilesh Prakash, Robert Ciszek, and Jussi Tohka.  Transfer learning in magnetic resonance brain imaging: a systematic review.  Journal of imaging, 7(4):66, 2021.
  * [215] Johannes Rieke, Fabian Eitel, Martin Weygandt, John-Dylan Haynes, and Kerstin Ritter.  Visualizing convolutional networks for mri-based diagnosis of alzheimer’s disease.  In Understanding and Interpreting Machine Learning in Medical Image Computing Applications, pages 24–31. Springer, 2018.
  * [216] Moritz Böhle, Fabian Eitel, Martin Weygandt, and Kerstin Ritter.  Layer-wise relevance propagation for explaining deep neural network decisions in mri-based alzheimer’s disease classification.  Frontiers in aging neuroscience, 11:194, 2019.
  * [217] Rembrandt Bakker, Paul Tiesinga, and Rolf Kötter.  The scalable brain atlas: instant web-based access to public brain atlases and related content.  Neuroinformatics, 13(3):353–366, 2015.
  * [218] Wenxing Hu, Xianghe Meng, Yuntong Bai, Aiying Zhang, Gang Qu, Biao Cai, Gemeng Zhang, Tony W Wilson, Julia M Stephen, Vince D Calhoun, et al.  Interpretable multimodal fusion networks reveal mechanisms of brain cognition.  IEEE transactions on medical imaging, 40(5):1474–1483, 2021.
  * [219] Weizheng Yan, Sergey Plis, Vince D Calhoun, Shengfeng Liu, Rongtao Jiang, Tian-Zi Jiang, and Jing Sui.  Discriminating schizophrenia from normal controls using resting state functional network connectivity: A deep neural network and layer-wise relevance propagation method.  In 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1–6. IEEE, 2017.
  * [220] Gidon Levakov, Gideon Rosenthal, Ilan Shelef, Tammy Riklin Raviv, and Galia Avidan.  From a deep learning model back to the brain—identifying regional predictors and their relation to aging.  Human brain mapping, 41(12):3235–3252, 2020.
  * [221] Carlo Biffi, Juan J Cerrolaza, Giacomo Tarroni, Wenjia Bai, Antonio De Marvao, Ozan Oktay, Christian Ledig, Loic Le Folgoc, Konstantinos Kamnitsas, Georgia Doumou, et al.  Explainable anatomical shape analysis through deep hierarchical generative models.  IEEE transactions on medical imaging, 39(6):2088–2099, 2020.
  * [222] Francisco J Martinez-Murcia, Andres Ortiz, Juan-Manuel Gorriz, Javier Ramirez, and Diego Castillo-Barnes.  Studying the manifold structure of alzheimer’s disease: a deep learning approach using convolutional autoencoders.  IEEE journal of biomedical and health informatics, 24(1):17–26, 2019.
  * [223] Matthew Leming, Juan Manuel Górriz, and John Suckling.  Ensemble deep learning on large, mixed-site fmri datasets in autism and other tasks.  International journal of neural systems, 30(07):2050012, 2020.
  * [224] Ziqi Tang, Kangway V Chuang, Charles DeCarli, Lee-Way Jin, Laurel Beckett, Michael J Keiser, and Brittany N Dugger.  Interpretable classification of alzheimer’s disease pathologies with a convolutional neural network pipeline.  Nature communications, 10(1):1–14, 2019.
  * [225] Gareth Ball, Claire E Kelly, Richard Beare, and Marc L Seal.  Individual variation underlying brain age estimates in typical development.  Neuroimage, 235:118036, 2021.
  * [226] Harshit Parmar, Brian Nutter, Rodney Long, Sameer Antani, and Sunanda Mitra.  Spatiotemporal feature extraction and classification of alzheimer’s disease using deep learning 3d-cnn for fmri data.  Journal of Medical Imaging, 7(5):056001, 2020.
  * [227] Chunfeng Lian, Mingxia Liu, Yongsheng Pan, and Dinggang Shen.  Attention-guided hybrid network for dementia diagnosis with structural mr images.  IEEE transactions on cybernetics, 2020.
  * [228] Sartaj Bhuvaji, Ankita Kadam, Prajakta Bhumkar, Sameer Dedge, and Swati Kanchan.  Brain tumor classification (mri).  Kaggle, 10, 2020.
  * [229] Tal Yarkoni, Russell A Poldrack, Thomas E Nichols, David C Van Essen, and Tor D Wager.  Large-scale automated synthesis of human functional neuroimaging data.  Nature methods, 8(8):665–670, 2011.
  * [230] Markus Loeffler, Christoph Engel, Peter Ahnert, Dorothee Alfermann, Katrin Arelin, Ronny Baber, Frank Beutner, Hans Binder, Elmar Brähler, Ralph Burkhardt, et al.  The life-adult-study: objectives and design of a population-based cohort study with 10,000 deeply phenotyped adults in germany.  BMC public health, 15(1):1–14, 2015.
  * [231] Sina Farsiu, Stephanie J Chiu, Rachelle V O’Connell, Francisco A Folgar, Eric Yuan, Joseph A Izatt, Cynthia A Toth, Age-Related Eye Disease Study 2 Ancillary Spectral Domain Optical Coherence Tomography Study Group, et al.  Quantitative classification of eyes with and without intermediate age-related macular degeneration using optical coherence tomography.  Ophthalmology, 121(1):162–172, 2014.
  * [232] David W Shattuck, Mubeena Mirza, Vitria Adisetiyo, Cornelius Hojatkashani, Georges Salamon, Katherine L Narr, Russell A Poldrack, Robert M Bilder, and Arthur W Toga.  Construction of a 3d probabilistic atlas of human cortical structures.  Neuroimage, 39(3):1064–1080, 2008.
  * [233] Oskar Maier, Bjoern H Menze, Janina Von der Gablentz, Levin Häni, Mattias P Heinrich, Matthias Liebrand, Stefan Winzeck, Abdul Basit, Paul Bentley, Liang Chen, et al.  Isles 2015-a public evaluation benchmark for ischemic stroke lesion segmentation from multispectral mri.  Medical image analysis, 35:250–269, 2017.
  * [234] Nathalie Tzourio-Mazoyer, Brigitte Landeau, Dimitri Papathanassiou, Fabrice Crivello, Octave Etard, Nicolas Delcroix, Bernard Mazoyer, and Marc Joliot.  Automated anatomical labeling of activations in spm using a macroscopic anatomical parcellation of the mni mri single-subject brain.  Neuroimage, 15(1):273–289, 2002.
  * [235] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Visualizing higher-layer features of a deep network.  University of Montreal, 1341(3):1, 2009.
  * [236] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [237] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [238] Taban Eslami, Joseph S Raiker, and Fahad Saeed.  Explainable and scalable machine learning algorithms for detection of autism spectrum disorder using fmri data.  In Neural Engineering Techniques for Autism Spectrum Disorder, pages 39–54. Elsevier, 2021.
  * [239] Usman Mahmood, Zening Fu, Vince Calhoun, and Sergey Plis.  Deep dynamic effective connectivity estimation from multivariate time series.  arXiv preprint arXiv:2202.02393, 2022.
  * [240] Kichang Kwak, Marc Niethammer, Kelly S Giovanello, Martin Styner, Eran Dayan, Alzheimer’s Disease Neuroimaging Initiative, et al.  Differential role for hippocampal subfields in alzheimer’s disease progression revealed with deep learning.  Cerebral Cortex, 32(3):467–478, 2022.
  * [241] Kichang Kwak, William Stanford, Eran Dayan, and Alzheimer’s Disease Neuroimaging Initiative.  Identifying the regional substrates predictive of alzheimer’s disease progression through a convolutional neural network model and occlusion.  Human Brain Mapping, 2022.
  * [242] Jennifer S Polson, Haoyue Zhang, Kambiz Nael, Noriko Salamon, Bryan Y Yoo, Suzie El-Saden, Sidney Starkman, Namkug Kim, Dong-Wha Kang, William Speier, et al.  Deep learning approaches to identify patients within the thrombolytic treatment window.  medRxiv, 2022.
  * [243] Guowei Zheng, Yu Zhang, Ziyang Zhao, Yin Wang, Xia Liu, Yingying Shang, Zhaoyang Cong, Stavros I Dimitriadis, Zhijun Yao, and Bin Hu.  A transformer-based multi-features fusion model for prediction of conversion in mild cognitive impairment.  Methods, 204:241–248, 2022.
  * [244] Bin Lu, Hui-Xian Li, Zhi-Kai Chang, Le Li, Ning-Xuan Chen, Zhi-Chen Zhu, Hui-Xia Zhou, Xue-Ying Li, Yu-Wei Wang, Shi-Xian Cui, et al.  A practical alzheimer’s disease classifier via brain imaging-based deep learning on 85,721 samples.  Journal of Big Data, 9(1):1–22, 2022.
  * [245] Jeyeon Lee, Brian J Burkett, Hoon-Ki Min, Matthew L Senjem, Emily S Lundt, Hugo Botha, Jonathan Graff-Radford, Leland R Barnard, Jeffrey L Gunter, Christopher G Schwarz, et al.  Deep learning-based brain age prediction in normal aging and dementia.  Nature Aging, 2(5):412–424, 2022.
  * [246] Yuda Bi, Anees Abrol, Zening Fu, Jiayu Chen, Jingyu Liu, and Vince Calhoun.  Prediction of gender from longitudinal mri data via deep learning on adolescent data reveals unique patterns associated with brain structure and change over a two-year period.  arXiv preprint arXiv:2209.07590, 2022.
  * [247] Tim Hahn, Jan Ernsting, Nils R Winter, Vincent Holstein, Ramona Leenings, Marie Beisemann, Lukas Fisch, Kelvin Sarink, Daniel Emden, Nils Opel, et al.  An uncertainty-aware, shareable, and transparent neural network architecture for brain-age modeling.  Science advances, 8(1):eabg9471, 2022.
  * [248] Kaida Ning, Pascale B Cannon, Jiawei Yu, Srinesh Shenoi, Lu Wang, Joydeep Sarkar, Alzheimer’s Disease Neuroimaging Initiative, et al.  Characterizing brain imaging features associated with adas-cog13 sub-scores with 3d convolutional neural networks.  bioRxiv, 2022.
  * [249] Tarjni Vyas, Raj Yadav, Chitra Solanki, Rutvi Darji, Shivani Desai, and Sudeep Tanwar.  Deep learning-based scheme to diagnose parkinson’s disease.  Expert Systems, 39(3):e12739, 2022.
  * [250] David A Wood, Sina Kafiabadi, Ayisha Al Busaidi, Emily Guilhem, Antanas Montvila, Jeremy Lynch, Matthew Townend, Siddharth Agarwal, Asif Mazumder, Gareth J Barker, et al.  Accurate brain-age models for routine clinical mri examinations.  NeuroImage, 249:118871, 2022.
  * [251] Jennifer S Polson, Haoyue Zhang, Kambiz Nael, Noriko Salamon, Bryan Y Yoo, Suzie El-Saden, Sidney Starkman, Namkug Kim, Dong-Wha Kang, William F Speier IV, et al.  Identifying acute ischemic stroke patients within the thrombolytic treatment window using deep learning.  Journal of Neuroimaging, 2022.
  * [252] Valentina Bordin, Davide Coluzzi, Massimo W Rivolta, and Giuseppe Baselli.  Explainable ai points to white matter hyperintensities for alzheimer’s disease identification: a preliminary study.  In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 484–487. IEEE, 2022\.
  * [253] Kobra Etminani, Amira Soliman, Anette Davidsson, Jose R Chang, Begoña Martínez-Sanchis, Stefan Byttner, Valle Camacho, Matteo Bauckneht, Roxana Stegeran, Marcus Ressner, et al.  A 3d deep learning model to predict the diagnosis of dementia with lewy bodies, alzheimer’s disease, and mild cognitive impairment using brain 18f-fdg pet.  European journal of nuclear medicine and molecular imaging, 49(2):563–584, 2022.
  * [254] Kai Gao, Zhipeng Fan, Jianpo Su, Ling-Li Zeng, Hui Shen, Jubo Zhu, and Dewen Hu.  Deep transfer learning for cerebral cortex using area-preserving geometry mapping.  Cerebral Cortex, 32(14):2972–2984, 2022.
  * [255] Jeyeon Lee, Brian J Burkett, Hoon-Ki Min, Matthew L Senjem, Ellen Dicks, Nick Corriveau-Lecavalier, Carly T Mester, Heather J Wiste, Emily S Lundt, Melissa E Murray, et al.  Synthesizing images of tau pathology from cross-modal neuroimaging using deep learning.  bioRxiv, 2022.
  * [256] Gang Qu, Wenxing Hu, Li Xiao, Junqi Wang, Yuntong Bai, Beenish Patel, Kun Zhang, and Yu-Ping Wang.  Brain functional connectivity analysis via graphical deep learning.  IEEE Transactions on Biomedical Engineering, 69(5):1696–1706, 2021\.
  * [257] Kyle B See, David J Arpin, David E Vaillancourt, Ruogu Fang, and Stephen A Coombes.  Unraveling somatotopic organization in the human brain using machine learning and adaptive supervoxel-based parcellations.  NeuroImage, 245:118710, 2021.
  * [258] James Zou, David Park, Aubrey Johnson, Xinyang Feng, Michelle Pardo, Jeanelle France, Zeljko Tomljanovic, Adam M Brickman, Devangere P Devanand, José A Luchsinger, et al.  Deep learning improves utility of tau pet in the study of alzheimer’s disease.  Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, 13(1):e12264, 2021.
  * [259] Suriya Murugan, Chandran Venkatesan, MG Sumithra, Xiao-Zhi Gao, B Elakkiya, Muthuramalingam Akila, and S Manoharan.  Demnet: a deep learning model for early diagnosis of alzheimer diseases and dementia from mr images.  IEEE Access, 9:90319–90329, 2021.
  * [260] Matthew Leming and John Suckling.  Deep learning for sex classification in resting-state and task functional brain networks from the uk biobank.  NeuroImage, 241:118409, 2021.
  * [261] Kobra Etminani, Amira Soliman, Stefan Byttner, Anette Davidsson, and Miguel Ochoa-Figueroa.  Peeking inside the box: Transfer learning vs 3d convolutional neural networks applied in neurodegenerative diseases.  In 2021 International Conference on Computational Intelligence Methods for Bioinformatics and Biostatistics (CIBB 2021), 15–17 nov., 2021, 2021\.
  * [262] Martin Dyrba, Moritz Hanzig, Slawek Altenstein, Sebastian Bader, Tommaso Ballarini, Frederic Brosseron, Katharina Buerger, Daniel Cantré, Peter Dechent, Laura Dobisch, et al.  Improving 3d convolutional neural network comprehensibility via interactive visualization of relevance maps: evaluation in alzheimer’s disease.  Alzheimer’s research & therapy, 13(1):1–18, 2021.
  * [263] David A Wood, Sina Kafiabadi, Aisha Al Busaidi, Emily Guilhem, Antanas Montvila, Siddarth Agarwal, Jeremy Lynch, Matthew Townend, Gareth Barker, Sebastian Ourselin, et al.  Automated triaging of head mri examinations using convolutional neural networks.  In Medical Imaging with Deep Learning, pages 813–841. PMLR, 2021\.
  * [264] Kichang Kwak, Kelly S Giovanello, Andrea Bozoki, Martin Styner, Eran Dayan, Alzheimer’s Disease Neuroimaging Initiative, et al.  Subtyping of mild cognitive impairment using a deep learning model based on brain atrophy patterns.  Cell Reports Medicine, 2(12):100467, 2021.
  * [265] Lebo Wang, Kaiming Li, and Xiaoping P Hu.  Graph convolutional network for fmri analysis based on connectivity neighborhood.  Network Neuroscience, 5(1):83–95, 2021.
  * [266] Jinhyeong Bae, Jane Stocks, Ashley Heywood, Youngmoon Jung, Lisanne Jenkins, Virginia Hill, Aggelos Katsaggelos, Karteek Popuri, Howie Rosen, M Faisal Beg, et al.  Transfer learning for predicting conversion from mild cognitive impairment to dementia of alzheimer’s type based on a three-dimensional convolutional neural network.  Neurobiology of aging, 99:53–64, 2021.
  * [267] Rajat Mani Thomas, Selene Gallo, Leonardo Cerliani, Paul Zhutovsky, Ahmed El-Gazzar, and Guido Van Wingen.  Classifying autism spectrum disorder using the temporal statistics of resting-state functional mri data with 3d convolutional neural networks.  Frontiers in psychiatry, 11:440, 2020.
  * [268] Tahjid Ashfaque Mostafa and Irene Cheng.  Parkinson’s disease detection using ensemble architecture from mr images.  In 2020 IEEE 20th International Conference on Bioinformatics and Bioengineering (BIBE), pages 987–992. IEEE, 2020.
  * [269] Kichang Kwak, Kelly S Giovanello, Martin Styner, Eran Dayan, Alzheimer’s Disease Neuroimaging Initiative, et al.  Atrophy-centered subtyping of mild cognitive impairment.  medRxiv, 2020.
  * [270] Jihoon Oh, Baek-Lok Oh, Kyong-Uk Lee, Jeong-Ho Chae, and Kyongsik Yun.  Identifying schizophrenia using structural mri with a deep learning algorithm.  Frontiers in psychiatry, 11:16, 2020.
  * [271] Matthew Leming and John Suckling.  Stochastic encoding of graphs in deep learning allows for complex analysis of gender classification in resting-state and task functional brain networks from the uk biobank.  arXiv preprint arXiv:2002.10936, 2020.
  * [272] Christopher West, Sara Soltaninejad, and Irene Cheng.  Assessing the capability of deep-learning models in parkinson’s disease diagnosis.  In International Conference on Smart Multimedia, pages 237–247. Springer, 2019.
  * [273] Jinhyeong Bae, Jane Stocks, Ashley Heywood, Youngmoon Jung, Lisanne Jenkins, Aggelos Katsaggelos, Karteek Popuri, M Faisal Beg, Lei Wang, Alzheimer’s Disease Neuroimaging Initiative, et al.  Transfer learning for predicting conversion from mild cognitive impairment to dementia of alzheimer’s type based on 3d-convolutional neural network.  bioRxiv, 2019.
  * [274] Jyoti Islam and Yanqing Zhang.  Understanding 3d cnn behavior for alzheimer’s disease diagnosis from brain pet scan.  arXiv preprint arXiv:1912.04563, 2019.
  * [275] Tae-Eui Kam, Han Zhang, Zhicheng Jiao, and Dinggang Shen.  Deep learning of static and dynamic brain functional networks for early mci detection.  IEEE transactions on medical imaging, 39(2):478–487, 2019.
  * [276] Yuanyuan Liu, Zhouxuan Li, Qiyang Ge, Nan Lin, and Momiao Xiong.  Deep feature selection and causal analysis of alzheimer’s disease.  Frontiers in neuroscience, 13:1198, 2019.
  * [277] Soheil Esmaeilzadeh, Dimitrios Ioannis Belivanis, Kilian M Pohl, and Ehsan Adeli.  End-to-end alzheimer’s disease diagnosis and biomarker identification.  In International workshop on machine Learning in medical imaging, pages 337–345. Springer, 2018.
  * [278] Pál Vakli, Regina J Deák-Meszlényi, Petra Hermann, and Zoltán Vidnyánszky.  Transfer learning improves resting-state functional connectivity pattern analysis using convolutional neural networks.  GigaScience, 7(12):giy130, 2018.
  * [279] Manhua Liu, Danni Cheng, Weiwu Yan, and Alzheimer’s Disease Neuroimaging Initiative.  Classification of alzheimer’s disease by combination of convolutional and recurrent neural networks using fdg-pet images.  Frontiers in neuroinformatics, 12:35, 2018.
  * [280] Stanislas Chambon, Mathieu N Galtier, Pierrick J Arnal, Gilles Wainrib, and Alexandre Gramfort.  A deep learning architecture for temporal sleep stage classification using multivariate and multimodal time series.  IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(4):758–769, 2018.
  * [281] Benjamín Gutiérrez-Becker and Christian Wachinger.  Deep multi-structural shape analysis: application to neuroanatomy.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 523–531. Springer, 2018.
  * [282] Zülfikar Aslan and Mehmet Akin.  A deep learning approach in automated detection of schizophrenia using scalogram images of eeg signals.  Physical and Engineering Sciences in Medicine, 45(1):83–96, 2022\.
  * [283] Ksenia Sokolova.  Investigating brain aging with Magnetic Resonance Imaging and Convolutional Neural Networks.  PhD thesis, King’s College London, 2021.
  * [284] Iván Sánchez Fernández, Edward Yang, Marta Amengual-Gual, Cristina Barcia Aguilar, Paola Calvachi Prieto, and Jurriaan M Peters.  Convolutional neural networks to identify malformations of cortical development: A feasibility study.  Seizure, 91:81–90, 2021.
  * [285] Jinlong Hu, Lijie Cao, Tenghui Li, Shoubin Dong, and Ping Li.  Gat-li: a graph attention network based learning and interpreting method for functional brain network classification.  BMC bioinformatics, 22(1):1–20, 2021.
  * [286] Zixuan Liu, Ehsan Adeli, Kilian M Pohl, and Qingyu Zhao.  Going beyond saliency maps: Training deep models to interpret deep models.  In International Conference on Information Processing in Medical Imaging, pages 71–82. Springer, 2021.
  * [287] Alina Lopatina, Stefan Ropele, Renat Sibgatulin, Jürgen R Reichenbach, and Daniel Güllmar.  Investigation of deep-learning-driven identification of multiple sclerosis patients based on susceptibility-weighted images using relevance analysis.  Frontiers in neuroscience, 14:609468, 2020.
  * [288] Iván Sánchez Fernández, Edward Yang, Paola Calvachi, Marta Amengual-Gual, Joyce Y Wu, Darcy Krueger, Hope Northrup, Martina E Bebin, Mustafa Sahin, Kun-Hsing Yu, et al.  Deep learning in rare disease. detection of tubers in tuberous sclerosis complex.  PLoS One, 15(4):e0232376, 2020.
  * [289] Prabhat Garg, Elizabeth Davenport, Gowtham Murugesan, Ben Wagner, Christopher Whitlow, Joseph Maldjian, and Albert Montillo.  Using convolutional neural networks to automatically detect eye-blink artifacts in magnetoencephalography without resorting to electrooculography.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 374–381. Springer, 2017.
  * [290] Yosuke Fujiwara and Junichi Ushiba.  Deep residual convolutional neural networks for brain–computer interface to visualize neural processing of hand movements in the human brain.  Frontiers in Computational Neuroscience, 16, 2022.
  * [291] Daniela Ushizima, Yuheng Chen, Maryana Alegro, Dulce Ovando, Rana Eser, WingHung Lee, Kinson Poon, Anubhav Shankar, Namrata Kantamneni, Shruti Satrawada, et al.  Deep learning for alzheimer’s disease: Mapping large-scale histological tau protein for neuroimaging biomarker validation.  NeuroImage, 248:118790, 2022.
  * [292] Mianxin Liu, Han Zhang, Feng Shi, and Dinggang Shen.  Hierarchical graph convolutional network built by multiscale atlases for brain disorder diagnosis using functional connectivity.  arXiv preprint arXiv:2209.11232, 2022.
  * [293] Fan Zhang, Bo Pan, Pengfei Shao, Peng Liu, Shuwei Shen, Peng Yao, Ronald X Xu, Alzheimer’s Disease Neuroimaging Initiative, et al.  A single model deep learning approach for alzheimer’s disease diagnosis.  Neuroscience, 491:200–214, 2022.
  * [294] Chun-Chih Huang, Intan Low, Chia-Hsiang Kao, Chuan-Yu Yu, Tung-Ping Su, Jen-Chuen Hsieh, Yong-Sheng Chen, and Li-Fen Chen.  Meg-based classification and grad-cam visualization for major depressive and bipolar disorders with semi-cnn.  In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 1823–1826. IEEE, 2022\.
  * [295] Joan Prats-Climent, Maria Teresa Gandia-Ferrero, Irene Torres-Espallardo, Lourdes Álvarez-Sanchez, Begoña Martínez-Sanchis, Consuelo Cháfer-Pericás, Ignacio Gómez-Rico, Leonor Cerdá-Alberich, Fernando Aparici-Robles, Miquel Baquero-Toledo, et al.  Artificial intelligence on fdg pet images identifies mild cognitive impairment patients with neurodegenerative disease.  Journal of Medical Systems, 46(8):1–13, 2022.
  * [296] Francesca Inglese, Minseon Kim, Gerda M Steup-Beekman, Tom WJ Huizinga, Mark A Van Buchem, Jeroen De Bresser, Dae-Shik Kim, and Itamar Ronen.  Mri-based classification of neuropsychiatric systemic lupus erythematosus patients with self-supervised contrastive learning.  Frontiers in neuroscience, page 92, 2022.
  * [297] Linfeng Liu, Siyu Liu, Lu Zhang, Xuan Vinh To, Fatima Nasrallah, and Shekhar S Chandra.  Cascaded multi-modal mixing transformers for alzheimer’s disease classification with incomplete data.  arXiv preprint arXiv:2210.00255, 2022.
  * [298] Junhao Zhang, Vishwanatha M Rao, Ye Tian, Yanting Yang, Nicolas Acosta, Zihan Wan, Pin-Yu Lee, Chloe Zhang, Lawrence S Kegeles, Scott A Small, et al.  Detecting schizophrenia with 3d structural brain mri using deep learning.  arXiv preprint arXiv:2206.12980, 2022.
  * [299] Adrian Tousignant Duran.  Prediction of future multiple sclerosis disease progression using deep learning analysis of mri data.  2022\.
  * [300] Jiaguang Li, Ying Wei, Chuyuan Wang, Qian Hu, Yue Liu, and Long Xu.  3-d cnn-based multichannel contrastive learning for alzheimer’s disease automatic diagnosis.  IEEE Transactions on Instrumentation and Measurement, 71:1–11, 2022\.
  * [301] Lingling Ding, Ziyang Liu, Ravikiran Mane, Shuai Wang, Jing Jing, He Fu, Zhenzhou Wu, Hao Li, Yong Jiang, Xia Meng, et al.  Predicting functional outcome in patients with acute brainstem infarction using deep neuroimaging features.  European Journal of Neurology, 29(3):744–752, 2022.
  * [302] Burak Tasci and Irem Tasci.  Deep feature extraction based brain image classification model using preprocessed images: Pdrnet.  Biomedical Signal Processing and Control, 78:103948, 2022.
  * [303] Modupe Odusami, Rytis Maskeliūnas, and Robertas Damavsevivcius.  An intelligent system for early recognition of alzheimer’s disease using neuroimaging.  Sensors, 22(3):740, 2022.
  * [304] Yanjiao Ban, Xuejun Zhang, and Huan Lao.  Diagnosis of alzheimer’s disease using structure highlighting key slice stacking and transfer learning.  Medical Physics, 49(9):5855–5869, 2022.
  * [305] Lu Yu, Wei Xiang, Juan Fang, Yi-Ping Phoebe Chen, and Ruifeng Zhu.  A novel explainable neural network for alzheimer’s disease diagnosis.  Pattern Recognition, 131:108876, 2022.
  * [306] Wonjun Ko, Wonsik Jung, Eunjin Jeon, and Heung-Il Suk.  A deep generative–discriminative learning for multi-modal representation in imaging genetics.  IEEE Transactions on Medical Imaging, 2022.
  * [307] Marina Manso Jimeno, Keerthi Sravan Ravi, Zhezhen Jin, Dotun Oyekunle, Godwin Ogbole, and Sairam Geethanath.  Artifactid: Identifying artifacts in low-field mri of the brain using deep learning.  Magnetic resonance imaging, 89:42–48, 2022.
  * [308] Esra Zihni, Bryony McGarry, and John Kelleher.  Moving toward explainable decisions of artificial intelligence models for the prediction of functional outcomes of ischemic stroke patients.  Exon Publications, pages 73–90, 2022.
  * [309] Daniel J Delbarre, Luis Santos, Habib Ganjgahi, Neil Horner, Aaron McCoy, Henrik Westerberg, Dieter A Häring, Thomas E Nichols, and Ann-Marie Mallon.  Application of a convolutional neural network to the quality control of mri defacing.  bioRxiv, pages 2021–06, 2022.
  * [310] Sasmitha Dasanayaka, Vimuth Shantha, Sanju Silva, Dulani Meedeniya, and Thanuja Ambegoda.  Interpretable machine learning for brain tumour analysis using mri and whole slide images.  Software Impacts, 13:100340, 2022.
  * [311] Mohamed Shaban and Amy W Amara.  Resting-state electroencephalography based deep-learning for the detection of parkinson’s disease.  Plos one, 17(2):e0263159, 2022.
  * [312] Miguel Altuve and Ana Pérez.  Intracerebral hemorrhage detection on computed tomography images using a residual neural network.  Physica Medica, 99:113–119, 2022.
  * [313] Peixin Lu, Lianting Hu, Ning Zhang, Huiying Liang, Tao Tian, and Long Lu.  A two-stage model for predicting mild cognitive impairment to alzheimer’s disease conversion.  Frontiers in Aging Neuroscience, 14, 2022.
  * [314] Jingjing Gao, Mingren Chen, Die Xiao, Yue Li, Shunli Zhu, Yanling Li, Xin Dai, Fengmei Lu, Zhengning Wang, Shimin Cai, et al.  Classification of major depressive disorder using an attention-guided unified deep convolutional neural network and individual structural covariance network.  Cerebral Cortex, 2022.
  * [315] Wen Yu, Baiying Lei, Shuqiang Wang, Yong Liu, Zhiguang Feng, Yong Hu, Yanyan Shen, and Michael K Ng.  Morphological feature visualization of alzheimer’s disease via multidirectional perception gan.  IEEE Transactions on Neural Networks and Learning Systems, 2022\.
  * [316] Chuxin Huang, Weidao Chen, Baiyun Liu, Ruize Yu, Xiqian Chen, Fei Tang, Jun Liu, and Wei Lu.  Transformer-based deep-learning algorithm for discriminating demyelinating diseases of the central nervous system with neuroimaging.  Frontiers in immunology, 13, 2022.
  * [317] Xiang Guo, Jiehuan Wang, Xiaoqiang Wang, Wenjing Liu, Hao Yu, Li Xu, Hengyan Li, Jiangfen Wu, Mengxing Dong, Weixiong Tan, et al.  Diagnosing autism spectrum disorder in children using conventional mri and apparent diffusion coefficient based deep learning algorithms.  European Radiology, 32(2):761–770, 2022.
  * [318] Neelesh Kumar and Konstantinos P Michmizos.  A neurophysiologically interpretable deep neural network predicts complex movement components from brain activity.  Scientific reports, 12(1):1–12, 2022.
  * [319] Stefan Jonas, Michael Müller, Andrea O Rossetti, Stephan Rüegg, Vincent Alvarez, Kaspar Schindler, and Frédéric Zubler.  Diagnostic and prognostic eeg analysis of critically ill patients: A deep learning study.  NeuroImage: Clinical, 36:103167, 2022.
  * [320] Gulnaz Ahmed, Meng Joo Er, Mian Muhammad Sadiq Fareed, Shahid Zikria, Saqib Mahmood, Jiao He, Muhammad Asad, Syeda Fizzah Jilani, and Muhammad Aslam.  Dad-net: Classification of alzheimer’s disease using adasyn oversampling technique and optimized neural network.  Molecules, 27(20):7085, 2022.
  * [321] Jiajun Fu, Meili Lu, Yifan Cao, Zhaohua Guo, and Zicheng Gao.  Grad-cam based visualization of 3d cnns in classifying fmri.  In Fourteenth International Conference on Digital Image Processing (ICDIP 2022), volume 12342, pages 317–322. SPIE, 2022.
  * [322] Houliang Zhou, Lifang He, Yu Zhang, Li Shen, and Brian Chen.  Interpretable graph convolutional network of multi-modality brain imaging for alzheimer’s disease diagnosis.  In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1–5. IEEE, 2022.
  * [323] Michele Lo Giudice, Nadia Mammone, Cosimo Ieracitano, Maurizio Campolo, Arcangelo Ranieri Bruna, Valeria Tomaselli, and Francesco Carlo Morabito.  Visual explanations of deep convolutional neural network for eye blinks detection in eeg-based bci applications.  In 2022 International Joint Conference on Neural Networks (IJCNN), pages 01–08. IEEE, 2022.
  * [324] Ankit Kurmi, Shreya Biswas, Shibaprasad Sen, Aleksandr Sinitca, Dmitrii Kaplun, and Ram Sarkar.  An ensemble of cnn models for parkinson’s disease detection using datscan images.  Diagnostics, 12(5):1173, 2022.
  * [325] Brady J Williamson, David Wang, Vivek Khandwala, Jennifer Scheler, and Achala Vagal.  Improving deep neural network interpretation for neuroimaging using multivariate modeling.  SN Computer Science, 3(2):1–8, 2022.
  * [326] Andrea Termine, Carlo Fabrizio, Carlo Caltagirone, Laura Petrosini, and Frontotemporal Lobar Degeneration Neuroimaging Initiative.  A reproducible deep-learning-based computer-aided diagnosis tool for frontotemporal dementia using monai and clinica frameworks.  Life, 12(7):947, 2022.
  * [327] Sarmad Maqsood, Robertas Damavsevivcius, and Rytis Maskeliūnas.  Multi-modal brain tumor detection using deep neural network and multiclass svm.  Medicina, 58(8):1090, 2022.
  * [328] Bochong Li, Ryo Oka, Ping Xuan, Yuichiro Yoshimura, and Toshiya Nakaguchi.  Semi-automatic multiparametric mr imaging classification using novel image input sequences and 3d convolutional neural networks.  Algorithms, 15(7):248, 2022.
  * [329] Tehseen Zia, Shakeeb Murtaza, Nauman Bashir, David Windridge, and Zeeshan Nisar.  Vant-gan: adversarial learning for discrepancy-based visual attribution in medical imaging.  Pattern Recognition Letters, 156:112–118, 2022.
  * [330] Zhiwei Qin, Zhao Liu, Qihao Guo, and Ping Zhu.  3d convolutional neural networks with hybrid attention mechanism for early diagnosis of alzheimer’s disease.  Biomedical Signal Processing and Control, 77:103828, 2022.
  * [331] Muhammad Wildan Oktavian, Novanto Yudistira, and Achmad Ridok.  Classification of alzheimer’s disease using the convolutional neural network (cnn) with transfer learning and weighted loss.  arXiv preprint arXiv:2207.01584, 2022.
  * [332] Patrick H Luckett, Luigi Maccotta, John J Lee, Ki Yun Park, Nico UF Dosenbach, Beau M Ances, Robert Edward Hogan, Joshua S Shimony, and Eric C Leuthardt.  Deep learning resting state functional magnetic resonance imaging lateralization of temporal lobe epilepsy.  Epilepsia, 2022.
  * [333] Noriya Watanabe, Kosuke Miyoshi, Koji Jimura, Daisuke Shimane, Ruedeerat Keerativittayayut, Kiyoshi Nakahara, and Masaki Takeda.  Multi-modal deep neural decoding of visual object representation in humans.  bioRxiv, 2022.
  * [334] Venkata Ramana Murthy Oruganti, Dwarikanath Mahapatra, Ramanathan Subramanian, et al.  Outlier-based autism detection using longitudinal structural mri.  arXiv preprint arXiv:2202.09988, 2022.
  * [335] Melanie Garcia and Clare Kelly.  Towards 3d deep learning for neuropsychiatry: predicting autism diagnosis using an interpretable deep learning pipeline applied to minimally processed structural mri data.  medRxiv, 2022.
  * [336] Sarah C Brüningk, Felix Hensel, Louis P Lukas, Merel Kuijs, Catherine R Jutzeler, and Bastian Rieck.  Back to the basics with inclusion of clinical domain knowledge-a simple, scalable and effective model of alzheimer’s disease classification.  In Machine Learning for Healthcare Conference, pages 730–754. PMLR, 2021.
  * [337] Zhongxuan Wang, Letian Wu, and Xiangcheng Ji.  An interpretable deep learning system for automatic intracranial hemorrhage diagnosis with ct image.  In Proceedings of the 2021 International Conference on Bioinformatics and Intelligent Computing, pages 338–357, 2021.
  * [338] YL Wang, ZJ Zhao, SY Hu, and FL Chang.  Clcu-net: Cross-level connected u-shaped network with selective feature aggregation attention module for brain tumor segmentation.  Computer Methods and Programs in Biomedicine, 207:106154, 2021.
  * [339] Ethan Ocasio and Tim Q Duong.  Deep learning prediction of mild cognitive impairment conversion to alzheimer’s disease at 3 years after diagnosis using longitudinal and whole-brain 3d mri.  PeerJ Computer Science, 7:e560, 2021.
  * [340] Yu-Ching Ni, Fan-Pin Tseng, Ming-Chyi Pai, Ing-Tsung Hsiao, Kun-Ju Lin, Zhi-Kun Lin, Wen-Bin Lin, Pai-Yi Chiu, Guang-Uei Hung, Chiung-Chih Chang, et al.  Detection of alzheimer’s disease using ecd spect images by transfer learning from fdg pet.  Annals of Nuclear Medicine, 35(8):889–899, 2021.
  * [341] Emanuel A Azcona, Pierre Besson, Yunan Wu, Ajay S Kurani, S Kathleen Bandt, Todd B Parrish, Aggelos K Katsaggelos, Alzheimer’s Disease Neuroimaging Initiative, et al.  Analyzing brain morphology in alzheimer’s disease using discriminative and generative spiral networks.  bioRxiv, 2021.
  * [342] SV Mishinov, AI Demyanchuk, EV Pushkina, VV Stupak, TM Fatykhov, NE Russkikh, and DN Shtokalo.  Identification of enlargement of the ventricular system of the brain using machine learning.  Biomedical Engineering, 55(4):297–301, 2021.
  * [343] Merel Kuijs, Catherine R Jutzeler, Bastian Rieck, and Sarah C Brüningk.  Interpretability aware model training to improve robustness against out-of-distribution magnetic resonance images in alzheimer’s disease classification.  arXiv preprint arXiv:2111.08701, 2021.
  * [344] Nalin Mathur, Anubha Gupta, Snehlata Jaswal, and Rohit Verma.  Deep learning helps eeg signals predict different stages of visual processing in the human brain.  Biomedical Signal Processing and Control, 70:102996, 2021.
  * [345] Jeong-Won Jeong, Soumyanil Banerjee, Min-Hee Lee, Nolan O’Hara, Michael Behen, Csaba Juhasz, and Ming Dong.  Deep reasoning neural network analysis to predict language deficits from psychometry-driven dwi connectome of young children with persistent language concerns.  Human brain mapping, 42(10):3326–3338, 2021.