# A Comprehensive Review on Financial Explainable AI

Yeo Wei Jie<EMAIL_ADDRESS>0000-0001-5254-8359 Nanyang Technological
University (NTU)50 Nanyang AveSingapore639798 ,  van der Heever Wihan  Nanyang
Technological University (NTU)50 Nanyang AveSingapore639798
<EMAIL_ADDRESS>,  Mao Rui  0000-0002-1082-8755 Nanyang Technological
University (NTU)50 Nanyang AveSingapore639798<EMAIL_ADDRESS>,  Cambria
Erik  0000-0002-3030-1280 Nanyang Technological University (NTU)50 Nanyang
AveSingapore639798<EMAIL_ADDRESS>,  Satapathy Ranjan  0000-0002-0733-7381
(corresponding author) Institute of High Performance Computing (IHPC), Agency
for Science, Technology and Research (A*STAR)Fusionopolis Way, #16-16
Connexis, SingaporeRepublic of Singapore138632
<EMAIL_ADDRESS>and  Mengaldo Gianmarco
0000-0002-0157-5477 National University of Singapore (NUS); Asian Institute of
Digital Finance at NUS, Singapore; Honorary Research Fellow, Imperial College
London, United Kingdom9 Engineering Drive 1Singapore117575<EMAIL_ADDRESS>
(2023)

###### Abstract.

The success of artificial intelligence (AI), and deep learning models in
particular, has led to their widespread adoption across various industries due
to their ability to process huge amounts of data and learn complex patterns.
However, due to their lack of explainability, there are significant concerns
regarding their use in critical sectors, such as finance and healthcare, where
decision-making transparency is of paramount importance. In this paper, we
provide a comparative survey of methods that aim to improve the explainability
of deep learning models within the context of finance. We categorize the
collection of explainable AI methods according to their corresponding
characteristics, and we review the concerns and challenges of adopting
explainable AI methods, together with future directions we deemed appropriate
and important.

XAI, explainable AI, interpretable AI, finance, FinXAI

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††journal:
JACM††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth:
8††ccs: Computing methodologies Artificial intelligence††ccs: Computing
methodologies Machine learning

##  1\. Introduction

Finance is a constantly evolving sector that is deeply rooted in the
development of human civilization. One of the main tasks of finance is the
efficient allocation of resources, with a chief example being the handling of
capital flows between various entities with different needs. These entities
can be divided into individuals, companies, and countries, and lead to the
common categorization of personal, corporate, and government finance. The
sector can be traced back to 5000 years ago, in the agrarian societies that
had been established and developed for some thousand of years at the time.
Indeed, one of the first examples of banking, a central institution within
finance, can be attributed to the Babylonian empire. Since then, societal
development and technological advances have pushed the field to undergo
several changes. In the past two decades, these changes have been particularly
marked, due to the accelerating pace of technological development, especially
in the context of AI. The latter has started spreading across multiple
segments of finance, from digital transactions to investment management, risk
management, algorithmic trading, and more (Team, 2022). The use of novel AI-
and non-AI technologies to automate and improve financial processes is now
known as FinTech (Financial Technology), and its growth in the past two
decades has been remarkable (Mroczkowska, 2020). In this review, we focus on
AI-based technologies and machine learning for financial applications.s

Financial researchers and practitioners have been relying on supervised,
unsupervised, and semi-supervised machine learning methods as well as
reinforcement learning for tackling many different problems. Some examples
include credit evaluation, fraud detection, algorithmic trading, and wealth
management. In supervised-based machine learning methods it is common to use
e.g., neural networks to identify complex relationships hidden in the
available labeled data. The labels are usually provided by domain experts. For
instance, one can think of building a stock-picking system, where a domain
expert labels periods of positive and negative returns. The machine is then
tasked to build the relationship between (possibly) high-dimensional data, and
positive and negative returns of a given stock (or multiple stocks) and
generalize to unseen data to e.g., predict the future stock’s behavior. In
unsupervised-based machine learning methods, the task is instead to identify
data with similar characteristics that can therefore be clustered together,
without domain-expert labeling. For example, one can think of identifying all
stocks that have similar characteristics into clusters using some similarity
metrics, such as valuation, profitability and risk. Semi-supervised learning
is a middle ground between supervised and unsupervised learning, where only a
portion of the data is labeled. Finally, reinforcement learning aims to
maximize, through a set of actions, the cumulative reward specified by the
practitioners. Reinforcement learning is used in finance for e.g., portfolio
construction. Reinforcement learning is strictly related to Markov decision
processes and substantially differs from both supervised and unsupervised
learning.

Among supervised, unsupervised, and reinforcement learning methods, there is
vast heterogeneity in terms of complexity. Some methods are considered easier
to understand, hence to interpret by practitioners (also referred to as white-
box methods), while others are considered not interpretable (also referred to
as black-box methods). To this end, neural networks and deep learning
strategies, that underpin the majority (albeit not the entirety) of recent
machine learning methods for financial applications, are considered black-box
methods - i.e., the reason for a given prediction is not of easy access when
available). This constitutes a critical issue, especially in risky and highly
regulated sectors, such as healthcare and finance, where a wrong decision may
lead to catastrophic loss of life (healthcare) or capital (finance). Hence, it
was deemed important to understand the reasons (i.e., the data and patterns)
the machine used to make a given decision. This aspect encompasses the broad
field of AI transparency. The latter is composed of three pillars, (i) AI
awareness, (ii) AI model explainability, and (iii) AI outcome explainability.
The first is tasked to understand whether AI is involved in a given product.
The second is responsible to provide a detailed explanation of the AI model,
including its inputs and outputs. The third is responsible to provide a
granular explanation of the inputs’ contributions to the AI model’s outcomes.
To this last category, we find a vast array of post-hoc interpretability
methods. In this review, we assume that AI awareness is achieved, i.e., we
know that in a given financial process AI is involved, and focus on AI
explainability, also referred to as eXplainable AI or simply XAI. A further
distinction commonly made is between interpretability and explainability of an
AI model. These two terms, frequently used interchangeably, have subtle
differences. Interpretability refers to how and why a model works.
Explainability refers to the ability of explaining the results in human terms.

While deep learning methods are considered black-boxes, many other methods in
finance are considered white-box methods. The trade-off between complexity and
interpretability is perhaps one of the most debated aspects in the field of
financial AI. On the one hand, white-box methods are highly interpretable but
lack the ability to grasp complex relationships, frequently failing to meet
the desired performance. On the other hand, black-box methods are not
interpretable but usually (although not always) meet the desired performance.
Therefore, it is not surprising that there are significant efforts being
pushed forward in recent years to render black-box methods more interpretable,
where the primary example is the field of deep learning. In this paper, we
provide an extensive review of XAI methods for the financial field that we
name FinXAI. Although there have been a number of surveys on XAI methods
(Cambria et al., 2023; Sahakyan et al., 2021; Arrieta et al., 2020; Guidotti
et al., 2018; Molnar, 2020; Mueller et al., 2019; Rojat et al., 2021), these
papers are targeted towards general XAI, and are not specific to finance.
Hence, we conduct a review on explainability techniques exclusively related to
financial use cases.

To compile this review, we took into account 69 papers, focusing mainly,
though not exclusively on the third pillar, i.e., the explainability of the
inputs’ contributions to the AI model’s outcomes. To this end, we considered
both post-hoc interpretability methods applied to black-box deep learning
models, and inherently transparent models that do not require further post-hoc
interpretability. Despite the relatively small number of collected papers in
the field of XAI, it is important to note that our main objective is to focus
specifically on XAI techniques applicable to the financial industry. This
targeted approach will provide valuable insights for researchers in related
fields and will ultimately help drive innovation and progress in the financial
industry. With the growing need for transparency and accountability of deep
learning, the XAI community has seen increasing growth in the number of works
published, we focus here instead only on works concerning financial use cases.
Notably, FinXAI is but a small subset of the general field of XAI and, thus we
take a holistic approach to assembling existing studies with the goal of
keeping up to date with the current approaches. The papers were queried from
both Google Scholar and Scopus where we searched using a set of keywords
relating to works that have applied explainable AI techniques in financial use
cases, the set of keywords include “XAI, explainable AI, finance, financial
sector, financial field, explainable ML”. We try to collect a diverse set of
papers that covers each category sufficiently well, and summarized in tables
LABEL:tab:_XAI_CE, LABEL:tab:_XAI_FP, LABEL:tab:_XAI_FM. We also noticed the
majority of explanation types were limited to fact-based explanations, hence
we explicitly search for techniques explaining in the form of counterfactuals.
Counterfactual explanations are deemed as a desirable form of explanation as
the receiver tends to prefer understanding why a certain prediction was made
instead of the opposing.

The main contributions of our work as such:

  * •

We provide an extensive study on consolidating XAI methods in the field of
finance (FinXAI), for researchers interested in prioritizing transparency in
their solutions.

  * •

We frame the FinXAI process as a sequential flow of decision-making processes
(see Fig 4), where we place importance on aligning the XAI technique with the
target audience. The objective of this framework is to produce explanations
that are both goal-oriented and audience-centric.

  * •

We review current FinXAI techniques, analyze their technical contributions to
ethical goals, and list down a number of key challenges faced in implementing
XAI as well as important directions to be improved for the future.

The remainder of the review paper is organized as follows: Section 2 describes
the definitions, reasons, and brief overview of FinXAI. Subsequently, we
explain the methodology of FinXAI, starting from numerical in Section 3,
textual in Section 4, hybrid analysis in Section 5 and ending with transparent
models in Section 6. In Section 7, we analyze how the reviewed FinXAI methods
contribute to ethical goals. Section 8 discusses key challenges of adopting
explainable models and future directions for research. Finally, Section 9
offers concluding remarks.

##  2\. FinXAI: Definition, Reason, Approach

This section details the definition, purpose, and approaches that have been
taken in improving the transparency of AI models. A collection of existing
literature reviews are analyzed and collated to give the reader a better
understanding of commonly used terminologies in the XAI field, as well as the
interlink between AI, linguistics, and social sciences which is essential to
provide a solid understanding of the subject.

###  2.1. Definition of FinXAI

FinXAI or explainable AI methods in finance is inherently linked to the
broader concept of AI transparency. As mentioned in section 1, this term
encompasses under a single framework three key steps: AI awareness, AI model
explainability, and AI outcome explainability. In this review, we focus on the
latter two aspects, model and outcome explainability. Model explainability
means that the inner workings of a given AI solution are interpretable, and
therefore the results may be interpreted by humans. This is typically the case
for models with reduced complexity (i.e., white-box models), such as linear
and logistic regression, and decision trees.

Outcome explainability means that the inner workings of a given AI solution
are not fully interpretable, and therefore the results may not be fully
understandable by humans, unless some interpretability tools are applied to
explain the AI outcomes. This is the case for complex models (i.e., black-box
models), such as deep neural networks. In these cases, it is common to apply
model agnostic post-hoc (and other) interpretability tools to understand the
results the AI provided in human terms.

Correspondingly, XAI models may be cast into two broad categories:
intrinsically explainable due to their highly interpretable nature (e.g.,
linear and logistic regression), and extrinsically explainable, hence
requiring an external tool to make them interpretable. In turn, these two
categories of models lead to different classes of model transparency:
simulatability, decomposability, and algorithmic transparency (Arrieta et al.,
2020). Each of these three classes inherits the preceding class’ properties,
that is, if a model is decomposable, it is also simulatable, and if a model is
algorithmically transparent is also decomposable and simulatable. In simple
terms, simulatability refers to the model’s ability to allow a human observer
to simulate a thought process over the inner workings of the model.
Decomposability entails that interpretability is available at every segment of
the model, including inputs, outputs as well as model inner workings and
parameters. Algorithmic transparency largely deals with the human user being
able to understand how the model reacts with varying inputs and more
importantly the ability to reason about errors the model produces.

A transparent model is an interpretable model which exhibits the ability to
provide human-understandable explanations. Areas of transparency not only lie
within the region of the model but also the data and design process of the
end-product (van den Berg and Kuiper, 2020). The EU High-Level Expert Group on
AI (HLEG, 2019) states that the data the model interacted with should be
traceable by human users at any given time. In addition, the design process of
the system must be clear and explainable in a manner comprehensible to related
stakeholders. The list of information types, regarded as explainable can even
be extended to include principles and guidelines in the development of the AI
system, as well as personnel involved in the implementation and development
process (Kuiper et al., 2022).

A key rationale for explainability is to gain the trust of affected
stakeholders. Examples of such stakeholders include regulators, board members,
auditors, end-users and developers (Yeong Zee Kin, 2023). To this end, the
format and degree of explanation vary among audiences. The key message is
usually conveyed in reports customized to the suitability of the receiving
audience. It is common knowledge that financial service providers are
regularly audited by supervisory authorities to ensure adherence to
regulations and to prevent potential fraud from taking place. The level of
scrutiny expected of the authorities is much higher than what the service
providers expect. A study conducted by (Kuiper et al., 2022) involves a
preliminary investigation to identify the types of information that are deemed
necessary, in the perspective of banks and supervisory authorities. The result
was that supervisory authorities identify all forms of information types as
relevant while banks only consider a subset of them. As such, there exists a
gap between each organization’s understanding of necessity, more often than
not leading to the delay in approving the deployment of financial services.

As mentioned, the constitution of a good explanation is largely subjective.
The amount of required information usually increases in a hierarchical manner
starting from the audience to the regulatory authorities, as depicted in
Figure 1. Here, scrutiny refers to the amount of information regarded as
essential. On the one hand, end-users typically require the least amount of
explanations (cause of outcome, data security) since they are usually only
interested in resolving their practical concerns. On the other hand, external
regulators require explanations about the end-product from head-to-toe
(overall guideline in design process, accountable and involved personnel,
deployment process, training structure of organization), including the end-
users requirements.

Proximity refers to the region of explanation provided by the XAI technique
and can be classified under local (reasons about a particular outcome) and
global (view of the underlying reasoning and mechanics of the AI model). End-
users tend to be concerned with how the outcome affecting them is provided
(local proximity). For example, a person whose credit card application was
rejected would want to know the underlying reason behind it. In contrast, the
solution providers and regulators tend to focus on the internal operations and
design workflow of the product, for reasons related to performance
enhancement, fairness in the model’s sense of judgment, and identification of
biases in the prediction (global proximity).

Figure 1. Levels of explanation requirements by different audiences,
categorized by explanation proximity, and ordered by scrutiny level. Local
proximity refers to explanations concerned about a specific outcome. Global
proximity refers to the underlying reasoning and mechanics of an AI model).
End-users typically require are satisfied with local-proximity explanations,
and the level of scrutiny is low. Developers, domain experts and regulatory
authorities require global-proximity explanations instead, and the level of
scrutiny is much higher.

###  2.2. Reasons for FinXAI

As mentioned in the previous section, various stakeholders lean towards
different forms of explanation, naturally leading to different sets of goals
the explanation can provide. A paramount reason for adopting explainable
models is to ensure that financial solutions adhere to ethical standards set
forth in the financial sector. The Monetary Authority of Singapore (MAS)
stipulates that AI solutions should be developed in accordance with the
Fairness, Ethics, Accountability and Transparency (FEAT) principles (of
Singapore, 2021). EU’s General Data Protection Regulation (GDPR) (Goodman and
Flaxman, 2017) in 2018 announced a law referred as “right to explanation”,
dictating that individuals affected by automated decision-making solutions
have a right to ask for an explanation of the outcome made for them.

The rising call for explainable models is mainly influenced by the rapid
advancement of AI solutions and the increasing complexity surrounding them.
More importantly, public cases of AI model displaying biases in their
prediction magnifies the urge for explainable solutions. A famous example is
Google’s image recognition software, that accidentally labels dark-skin humans
as gorillas (VINCENT, 2018). Such biases can damage the company’s reputation,
and lead to profit losses.

The financial sector has its own set of ethics that should be upheld along
with the desirable principles of AI ethics. These set of financial ethics
often overlap with AI principles. An experiment involving 8 financial experts
to investigate the relationship between the aforementioned sets was carried
out in (Rizinski et al., 2022). The results show that financial ethics
(integrity, objectivity, competence, fairness, confidentiality,
professionalism, diligence) has significant similarities with AI ethics
(growth and sustainable development, human-centered values and fairness,
transparency and explainability, safety and accountability). The strength of
the links between each element was assessed, with integrity and fairness
having the strongest relationship with AI ethics. Indeed, this is
understandable given that AI solutions should naturally embody these
qualities, regardless of the industry taken into account.

Figure 2. Ethical goals are classified under three broad audiences: end-users,
developers/domain experts, and internal/external regulatory authorities. Some
ethical goals are shared by the three different audiences considered, such as
informativeness. (Arrieta et al., 2020)

As mentioned, the ethical goals set forth by XAI solutions differ among
audiences, similar to the explanation types desired. Each audience is likely
to be more affected by one than the other (Arrieta et al., 2020; Mohseni et
al., 2021b). Figure 2 list the ethical goals supported towards each set of
audiences, and shows that there are some overlapping ethical goals across
audiences. Referring to (Arrieta et al., 2020), we provide a brief explanation
of each ethical goal reported in Figure 2, taking a financial perspective.

  * •

Trustworthiness: Defined as instilling trust into users affected by the
decisions of the AI model. Trustworthiness can be achieved when there is a
high level of confidence in the model to constantly behave in the intended
manner (Ribeiro et al., 2016). Trust is also sustained if the services
provided are transparent and enables affected user to maintain their faith in
the service providers. However, trust is a highly subjective quality and hence
difficult to quantify. Judging if an explanation instills trust is mostly
subjected to the affected users’ opinions.

  * •

Fairness: Refers to delivering AI solutions and explanations to every user and
stakeholder equally, removing possible biases. Indeed, bias mitigation is a
key constituent of fairness. The transparency of the AI model allows for a
fair and socially ethical analysis, where any form of biases existing in the
product chain are eliminated. In the financial markets, users tend to use the
services provided by a firm if they are assured of fair and unbiased
treatment.

  * •

Informativeness: One important objective of AI models is to provide assistance
to human counterparts in making decisions. Therefore, it is vital that the
problem statement is made clear at all times. By providing explanations, the
model benefits both from a social perspective as well as a performance
standpoint, since knowing what is being done opens up opportunities for
further refinement. Most of the papers in the literature dealing with this
aspect aim at identifying relevant features which equates to highlighting
parts of the input data the model is paying attention to. This can assist in
debugging and allows pruning of unnecessary features which may cause
overfitting.

  * •

Accessibility: The main personnel interacting with algorithms are usually
restricted to AI developers or domain experts, providing accessibility could
allow for non-experts to get involved. This can be seen as an important
stepping stone for making AI prevalent and well-accepted by the general
society. Likewise, complicated algorithms deter financial companies from
adopting such solutions, since extensive training is required while having to
fear potential repercussions in the case of any unintended wrongdoings. If a
model is able to relate its mechanisms in easily understandable terms, it can
ease the fear of users and encourage more organizations to adopt such
practices.

  * •

Privacy Awareness: Not knowing the full limits of accessibility in the data
can result in a breach of privacy. Likewise, such an issue triggers concerns
within the overall design workflow. Accountable personnel in the designing
process should ensure third parties are only allowed restricted access to the
end-users data and prevent any misuse which can disrupt data integrity.
Privacy awareness is especially important in the financial sector due to the
amount and sensitivity of the information being captured.

  * •

Confidence: The AI model should provide not only an outcome but also the
confidence it has in the decision-making process, allowing domain experts to
identify uncertainty in both model’s results as well as the region of data
captured. Stability in the prediction can be used to access a model’s
confidence while explanations provided by the model should only be trusted if
it produces results that are consistent across different data inputs.

  * •

Causality: It is usually in the interest of developers or experts to
understand the causality between data features. However, proving it is a
difficult task that requires extensive experimenting. Correlation can be
involved in assessing causality, though it is frequently not representative of
causality. Since AI models only discover correlations among the data they
learn from, domain experts are usually required to perform a deeper analysis
of causal relationships.

  * •

Transferability: Allowing for the distillation of knowledge learned from AI
models is an extensive area of research, a notable benefit is that it allows
for the reusability of different models and averts endless hours of re-
training. However, the complexity of the algorithms limits experts from
deploying trained models in different domains. For example, a model trained to
forecast future stock prices can likely be used to predict other financial
variables such as bond price, market volatility, or creditworthiness, if the
model behavior in these circumstances is known. Delivering an intuition of the
inner workings can ease the burden of experts to facilitate adapting the
knowledge learned, reducing the effort required for fine-tuning.
Transferability is arguably one of the essential properties for the
improvement of future AI models.

###  2.3. Approach of XAI

Figure 3. Different stages where interpretability can be injected into the
design workflow. (Mellon, 2021)

The review provided in this paper aims to give the readers an overall view of
the XAI methodologies developed thus far in the financial industry. We note
that explainability can be injected across different stages of the development
cycle. These stages include: pre-modeling, modeling, and post-modeling
(Mellon, 2021). Pre-modeling stage refers to the process chain before the
designing stage of the AI model, this can include preliminary procedures which
focus on identifying salient features by accessing readily available domain
knowledge (Islam et al., 2019). The modeling phase includes any adjustment to
the model’s architecture or optimization objective. As a start, simpler
transparent models should be preferred over complex black-box models if the
problem at hand is not too complicated. Most of the papers in the review focus
on the post-modeling stage, mainly due to the flexibility and ease of
designing explainability techniques. Since the outcome is provided, it
provides developers with more information to design an appropriate explanation
method towards the form of data interacted (See Figure 3). Most XAI techniques
tend to focus on one stage of the modeling process, though it is possible to
do so in two or more.

Table 1. Classification of papers relating to credit evaluation. The papers reviewed are split by task category and subsequently categorized by entailed properties. Missing options are either not stated or non-applicable. Paper |  Trans -parency | Proximity | Explanation Procedure | Audience |  Data Analysis |  Expla -nation Type |  Expl Eval
---|---|---|---|---|---|---|---
|  Intrinsic |  Post-hoc |  Local |  Global |  Textual |  Visual |  By example |  Simplification |  Feature relevance |  End-user |  Developer |  Domain expert |  Regulatory |  Numerical |  Text |  Factual |  Counterfactual |
(Dikmen and Burns, 2022) |  |  ✓ |  ✓ |  |  |  |  |  |  ✓ |  ✓ |  |  |  |  ✓ |  |  ✓ |  |
(Gramespacher and Posth, 2021; Chen and Ye, 2022) |  ✓ |  |  |  ✓ |  |  |  |  |  |  |  |  |  ✓ |  ✓ |  ✓ |  ✓ |  |  ✓
(Misheva et al., 2021; Serengil et al., 2022) |  |  ✓ |  ✓ |  ✓ |  |  |  |  ✓ |  ✓ |  |  ✓ |  |  |  ✓ |  |  ✓ |  |
(Bussmann et al., 2021, 2020) |  |  ✓ |  ✓ |  ✓ |  |  |  |  |  ✓ |  ✓ |  ✓ |  |  |  ✓ |  |  ✓ |  |
(Rizinski et al., 2022) |  |  ✓ |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  ✓ |  |  |  ✓ |  |  |  ✓ |  ✓
(Müller et al., 2022) |  |  ✓ |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  |  |  ✓ |  ✓ |  |  ✓ |  |
(Zhang et al., 2022d) |  |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  |  ✓ |  |  |  |  ✓ |  ✓ |  |  ✓ |  ✓ |
(Biecek et al., 2021) |  |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  |  ✓ |  ✓ |  |  |  ✓ |  ✓ |  |  ✓ |  |
(Crosato et al., 2021) |  |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  |  ✓ |  ✓ |  |  |  |  ✓ |  |  ✓ |  |
(Davis et al., 2022) |  |  ✓ |  ✓ |  ✓ |  |  |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  ✓ |  ✓ |  |  ✓ |  |
(Sudjianto and Zhang, 2021; Dumitrescu et al., 2022) |  ✓ |  |  ✓ |  ✓ |  |  |  |  |  |  |  |  |  ✓ |  ✓ |  |  ✓ |  |
(Bueff et al., 2022) |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  |  |  ✓ |  ✓ |  |  ✓ |  ✓ |
(Fritz-Morgenthal et al., 2022; Tran et al., 2022) |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  ✓ |  ✓ |  |  ✓ |  |  ✓ |  |
(Srinivasan et al., 2019) |  ✓ |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  ✓ |  |  |  ✓ |  ✓ |  |
(Grath et al., 2018) |  |  ✓ |  ✓ |  |  |  |  |  |  ✓ |  ✓ |  |  |  |  ✓ |  |  |  ✓ |  ✓
(Adams and Hagras, 2020) |  ✓ |  |  ✓ |  ✓ |  |  |  |  |  |  |  |  |  ✓ |  ✓ |  ✓ |  ✓ |  |
(Demajo et al., 2020) |  |  ✓ |  ✓ |  ✓ |  |  |  ✓ |  |  ✓ |  ✓ |  |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  ✓
(Luo et al., 2018) |  |  ✓ |  ✓ |  |  |  ✓ |  |  |  |  |  |  ✓ |  |  |  ✓ |  ✓ |  |
(Zhang et al., 2022a) |  |  ✓ |  ✓ |  |  |  ✓ |  |  |  |  ✓ |  |  ✓ |  |  ✓ |  |  |  ✓ |  ✓

The focused regions of finance can be broadly categorized under three sections
(Bahrammirzaee, 2010): credit evaluation (peer-to-peer lending, credit
assessment, credit risk management, credit scoring, accounting anomalies),
financial prediction (Asset allocation, stock index prediction, market
condition forecasting, volatility forecasting, algorithmic trading, financial
growth rate, economic crisis forecast, bankruptcy prediction, fraud detection,
mortgage default) and financial analytics (financial text classification,
spending behavior, financial corporate social responsibility (CSR), customer
satisfaction). Following the task classification, we further differentiate the
studies based on the underlying characteristics of the XAI technique as shown
in Table LABEL:tab:_XAI_CE, LABEL:tab:_XAI_FP, LABEL:tab:_XAI_FM.
Specifically, we seek to answer questions such as “What form of explanation is
provided?” (explanation procedure), “Who is the explanation intended for?”
(audience), “What kind of explanation is provided?” (proximity, explanation
type).

Table 2. Classification of papers relating to financial prediction. The papers reviewed are split by task category and subsequently categorized by entailed properties. Missing options are either not stated or non-applicable. Paper |  Trans -parency | Proximity | Explanation Procedure | Audience |  Data Analysis |  Expla -nation Type |  Expl Eval
---|---|---|---|---|---|---|---
|  Intrinsic |  Post-hoc |  Local |  Global |  Textual |  Visual |  By example |  Simplification |  Feature relevance |  End-user |  Developer |  Domain expert |  Regulatory |  Numerical |  Text |  Factual |  Counterfactual |
(Zhang et al., 2020b) |  |  ✓ |  |  ✓ |  |  ✓ |  |  |  |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Yang et al., 2018) |  |  ✓ |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  ✓ |  |  |  ✓ |  ✓ |  |
(Deng et al., 2019) |  |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  |  |  ✓ |  ✓ |  |
(Ghosh and Sanyal, 2021) |  |  ✓ |  ✓ |  ✓ |  |  |  |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  |  ✓ |  ✓ |  ✓ |  |
(Collaris et al., 2018) |  |  ✓ |  ✓ |  ✓ |  |  |  |  ✓ |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓ |  |  ✓
(Benhamou et al., 2021; Fior et al., 2022) |  |  ✓ |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Bracke et al., 2019) |  |  ✓ |  ✓ |  ✓ |  |  |  |  |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  |  ✓ |  |
(Nazemi et al., 2022) |  ✓ |  ✓ |  |  ✓ |  |  |  |  |  |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Carta et al., 2021) |  ✓ |  |  |  ✓ |  |  |  |  |  |  |  ✓ |  ✓ |  |  ✓ |  ✓ |  ✓ |  |
(Cong et al., 2021) |  |  ✓ |  |  ✓ |  |  |  |  ✓ |  ✓ |  |  |  ✓ |  |  ✓ |  ✓ |  ✓ |  |
(Yasodhara et al., 2021) |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓
(Park and Yang, 2022; Islam et al., 2019; Weng et al., 2022; Wand et al., 2022; Vivek et al., 2022) |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Babaei et al., 2022) |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  |  |  ✓ |  ✓ |  |  ✓ |  |
(Lin et al., 2021) |  ✓ |  |  ✓ |  |  |  ✓ |  |  |  |  |  |  ✓ |  |  |  ✓ |  ✓ |  |
(Gite et al., 2021; Bandi et al., 2021) |  |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  |  ✓ |  |  |  ✓ |  ✓ |  ✓ |  |
(Yan et al., 2019) |  |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  |  |  |  ✓ |  ✓ |  |  ✓ |  |
(Carta et al., 2022) |  |  ✓ |  ✓ |  |  |  |  |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓ |  |  ✓
(Kumar et al., 2022) |  |  ✓ |  ✓ |  |  |  |  |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Cho and Shin, 2023) |  |  ✓ |  ✓ |  |  |  |  |  |  ✓ |  ✓ |  |  |  |  ✓ |  |  ✓ |  ✓ |
(Shi et al., 2021; Kumar et al., 2017; Chen et al., 2020) |  |  ✓ |  ✓ |  |  |  ✓ |  |  |  |  ✓ |  ✓ |  |  |  ✓ |  |  ✓ |  |
(Achituve et al., 2019) |  |  ✓ |  ✓ |  |  |  ✓ |  |  |  |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Farzad, 2019) |  |  ✓ |  |  ✓ |  |  ✓ |  |  |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  ✓
(Ong et al., 2023) |  ✓ |  |  |  ✓ |  |  |  |  |  ✓ |  |  |  ✓ |  |  ✓ |  ✓ |  ✓ |  |
(Yuan and Zhang, 2020) |  |  ✓ |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  |  |  |  ✓ |  ✓ |  |  ✓

Table 3. Classification of papers relating to financial analytics. The papers reviewed are split by task category and subsequently categorized by entailed properties. Missing options are either not stated or non-applicable. There were no evaluation metrics present for these papers. Paper |  Trans -parency | Proximity | Explanation Procedure | Audience |  Data Analysis |  Explanation Type |  Expl Eval
---|---|---|---|---|---|---|---
|  Intrinsic |  Post-hoc |  Local |  Global |  Textual |  Visual |  By example |  Simplification |  Feature relevance |  End-user |  Developer |  Domain expert |  Regulatory |  Numerical |  Text |  Factual |  Counterfactual |
(Rallis et al., 2022) |  |  ✓ |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Zhang et al., 2022c) |  |  ✓ |  ✓ |  ✓ |  |  ✓ |  |  ✓ |  ✓ |  |  |  |  ✓ |  ✓ |  |  ✓ |  ✓ |
(Maree et al., 2020) |  ✓ |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  ✓ |  |  |  ✓ |  ✓ |  ✓ |  |
(Maree and Omlin, 2022a) |  |  ✓ |  |  ✓ |  |  |  |  |  |  ✓ |  |  |  ✓ |  ✓ |  |  ✓ |  |
(Maree and Omlin, 2022b) |  |  ✓ |  |  ✓ |  |  |  |  ✓ |  |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Lachuer and Jabeur, 2022) |  |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  ✓ |  |
(Gramegna and Giudici, 2020) |  |  ✓ |  |  ✓ |  |  ✓ |  |  |  ✓ |  |  ✓ |  ✓ |  |  ✓ |  |  ✓ |  |
(Liu et al., 2020) |  |  ✓ |  |  ✓ |  |  ✓ |  |  |  |  |  |  |  ✓ |  ✓ |  ✓ |  ✓ |  |
(Yang et al., 2020) |  |  ✓ |  ✓ |  |  |  |  |  |  ✓ |  |  |  |  ✓ |  |  ✓ |  |  ✓ |
(Ito et al., 2020) |  |  ✓ |  ✓ |  |  |  ✓ |  |  |  |  ✓ |  |  ✓ |  |  |  ✓ |  ✓ |  |
(Zhang et al., 2020a) |  |  ✓ |  ✓ |  |  ✓ |  |  |  |  ✓ |  |  ✓ |  |  ✓ |  ✓ |  ✓ |  ✓ |  |

  * •

Transparency: As mentioned in Section 2.3, interpretability of the model is
either derived via interpreting the internal mechanisms of the AI model or
through external techniques aimed at delivering some form of visualization or
intuition of how the model works. Most of the reviewed papers focus on post-
hoc explainability techniques, which we believe are preferred for a number of
reasons. Intrinsic models usually under-perform complex networks and as such,
producing explanations for an inaccurate prediction is pointless. We
additionally note that the method of conveying explanations for intrinsic
models is by definition model-specific. This means the same method cannot be
reused for a different model. While post-hoc techniques can be agnostic or
specific towards any single model.

  * •

Proximity: The explanations provided by XAI tools can seek to explain either
the derivation of an outcome, known as local explanation, or how the model
outputs on a global scale, referred to as global explanation. Global
explanations tend to provide information on how the model makes decisions
globally based on the learned weights, data features, and structure of the
network. Producing an acceptable global explanation tends to be difficult in
most cases (Molnar, 2020) as opposed to just a region of the input data. On
the other hand, local explanations focus on a specific region of the dataset
and seek to assist the receiver in understanding how a particular prediction
is made. Local explanation is more accurate for unique cases where the
dependency on input features is rarely captured by the AI model, which can
cause global explanations to ignore such dependency. End-users tend to prefer
local explanations as their concern lies with the explanation surrounding
their outcome. Regulators and financial experts, on the other hand, prefer
global explanations in order to have a complete understanding of the model.

  * •

Explanation Procedure: According to (Arrieta et al., 2020), the various forms
of post-hoc XAI techniques can be divided into several sections: text
explanation (TE), visual explanation (VE), explanation by example (EE),
explanation by simplification (ES) and feature relevance (FR). TE provides an
explanation via text generation. Natural language tends to be easily
understood by non-experts and is a common source of information in human
society. VE enables visual understanding of the model’s behavior, which may be
preferable for image features (Selvaraju et al., 2017), such methods comprise
graphical plots for both local and global explainability. EE captures a
smaller subset of examples which represents the correlations modeled by the
black-box model at a high level. ES techniques build a simpler surrogate model
to approximate the underlying black-box model with high fidelity yet being
interpretable. FR techniques aim to identify features deemed relevant for the
model’s prediction, by computing a relevance score for each feature. FR can
account for explainability at both local and global levels and constitutes the
largest share among the reviewed papers in our literature.

  * •

Audience: Since the quality of explanations is subjective, it is very
difficult to derive a one-fit-all explanation and hence, explanations should
be customized towards one’s needs. The examples of audiences are referenced
from Fig 1, while we further merge internal and external regulators together.
We highlight that aligning the objective of the explanation to the audience
receiving it is important (Tomsett et al., 2018). Determining if an
explanation is considered meaningful, is dependent on the target goals
respective of each audience. Financial regulators, for example, would not be
very concerned with understanding what sort of AI model or ML technique is
used, but rather on the aspect of data privacy, model biases, or unfair
treatment between affected end-users. It is uncommon for a single explanation
to be deemed acceptable to audiences holding different positions in a
financial company. An example is that the explanation produced for the
developer tends to require additional customization before submitting to the
immediate superior and the same applies to the proceeding higher-ups and
external end-users.

  * •

Data Type: The most commonly used forms of input data among the reviewed
papers consists of text, images, and numerical values. In terms of frequency
among the forms of available data, numerical features are the most common
source of information used in the financial industry. Images represent the
least utilized source, as they tend to be storage intensive and contain a
large amount of redundant information or are not applicable for most use
cases. We only found a single work using image features. (Chen et al., 2020)
performs classification of eight different candlestick patterns and the
explanation is delivered through monitoring changes in prediction after
applying adversarial attacks. Surprisingly, textual information is not used as
frequently as expected, albeit being a valuable source of information for
deriving market sentiment or understanding consumers’ emotions towards certain
aspects of the business product. It is also possible to unify multiple sources
of information, otherwise known as multi-model data. A boost in performance
can be achieved, for instance by combining the patterns learned from time-
series features and sentiment from textual features.

  * •

Explanation Type: A single explanation can be conveyed in various forms,
including factual, contrastive, and counterfactual explanations (Miller,
2019). Factual delivers straightforward explanations that seek to answer the
question “Why does X lead to Y” as opposed to contrastive “Why does X lead to
Y instead of Z”. Counterfactual instead reasons how the consequent can be
changed with respect to the antecedent, answering the question “how to achieve
Z by changing X”. Humans tend to prefer contrastive rather than factual
explanations since the latter can have multiple answers and referring to
(Miller, 2019), explanations are selective. As humans tend to ignore a large
portion of the explanations except for the important ones due to cognitive
bias. For example, if Person A’s loan application was rejected, there could be
numerous reasons for this, such as “Person A’s income was too low for the past
6 months”, “Person A’s only have 1 existing credit card”, “Person A has had a
credit default 3 months ago” and so on. Whereas a contrastive explanation can
instead involve comparing against another applicant whose outcome contrasts
the target applicant’s and an explanation can be made, highlighting the most
significant factor. As argued by (Lipton, 1990), contrastive explanations are
easier to deliver as one does not have to investigate the entire region of
causes but rather a subset of it. Counterfactual explanations then seek to
provide solutions for the contrastive explanation, commonly done by
identifying the smallest changes to the input features, such that the outcome
can be altered towards the alternative.

  * •

Explanation Evaluation: Despite the extensive studies carried out to
investigate what defines a good explanation, it is difficult to qualitatively
compare among interpretations. The quality of an explanation is mostly
subjective as a single explanation can be perceived with varying opinions
among audiences. Nonetheless, there exist a number of studies that provides a
quantitative approach to evaluating explanations. These measurements can be
derived from human experts (Yang et al., 2020), referencing financial ethical
goals (Adams and Hagras, 2020) or through statistical methods (Müller et al.,
2022). (Islam et al., 2019) conducted a comparison between feature importance
techniques in time series data and proposed a multivariate dataset that deals
with the inability of techniques that identify salient time-series features. A
vast majority of the reviewed papers focused only on evaluating the
performance of the prediction model and consider it as a proxy for the quality
of the explanation. We argue that such evaluation does not fully represent the
quality of the explanation and even if so, it may not be suitable for every
form of explanation procedure.

Selection Procedure: We design a framework shown in Figure 4, framing the
designing of the XAI solution as a sequential decision-making process. The
selection categories can be referenced from Tables LABEL:tab:_XAI_CE,
LABEL:tab:_XAI_FP, LABEL:tab:_XAI_FM. The sequential structure of the
framework ensures the explanation provided is tailored to the audience’s needs
while achieving the goal set out with respect to the target audience. We note
that certain properties of the XAI technique have inner dependencies with each
other, such as the relationship between explanation proximity and target
audience. The quality of the explanation is evaluated and serves as feedback
for any necessary adjustment, resulting in an audience-centric explanation.

Figure 4. XAI framework depicting a sequential flow of decision-making events.
The proximity (local/global) and explanation type (factual/counterfactual)
should be chosen in accordance with the target audience and data type
available. The choice of explanation property is assessed by an iterative
evaluation under the appropriate metric for both performance and explanation
conveyed.

##  3\. XAI with numerical features

Numerical features are a common source of information across all aspects of
data-driven methodologies. Financial tasks such as credit scoring of
individuals/firms and financial market forecasts commonly use a collection of
historical numerical features, such as stock price, trade volume, and
volatility, and apply various forms of data-driven models to make predictions.
These data-driven models may include supervised learning approaches, e.g.,
classification and regression tasks, and unsupervised learning approaches,
e.g., clustering tasks. The use of numerical features within the context of
finance is well established, hence it is not surprising that the majority of
reviewed studies focus on this area. In the following, we outline the main
approaches used for explainability in this context, namely visual explanation,
explanation by simplification, feature relevance, and explanation by example,
and conclude with a brief summary.

###  3.1. Visual Explanation

Visual explanatory (VE) techniques generate explanations of the underlying
model in the form of visuals. VE techniques can be both model-specific and
model-agnostic. The model-specific techniques reviewed are mainly constructed
to interpret image-based networks such as convolutional neural networks (CNN).
(Kumar et al., 2017) proposes to perform a deconvolution on the last layer
preceding the output to extract a visual attentive map. The approach named
CLass Enhanced Attentive Response (CLEAR) generates a graphical plot denoting
the timeframe to which the stock-picking agent pays the most attention, along
with a separate plot corresponding to the sentiment class of the stock. (Chen
et al., 2020) implements a CNN network to identify 8 common candlestick
patterns which are widely used for technical analysis in stock market trading.
The authors then perform an adversarial attack on regions of the feature
space, to demonstrate that the model is focusing on regions similar to how a
human would process the candlesticks. (Shi et al., 2021) employs a
reinforcement learning (RL) agent to optimize a portfolio of equities while
using a temporal CNN as a feature extractor. The dynamic asset allocation is
interpreted with Gradient-weighted Class Activation Mapping (Grad-CAM)
(Selvaraju et al., 2017), improving over simple deconvolutions by producing
class-discriminative explanations, and is applicable to any deep neural
networks. The proposed technique outputs a localization map using gradients
corresponding to the target label. The right plot in Figure 5 depicts a global
map highlighting each asset’s importance across the trading period.
Interestingly in the left plot, the agent focuses on the worst-performing
stock, GLID the most, rather than the high-performing stocks. Here, the agent
predicts the stock decline and reduces the allocation proportion, and
indirectly increases the weights of high-performing stocks which in this case
is the target stock, NVDA. (Achituve et al., 2019) proposes to use an
attention mechanism (Vaswani et al., 2017) to compute similarity scores of
possibly fraudulent transactions on both feature and temporal levels and in
return, allows for visualization at the top contributing features accounting
for the model’s prediction.

Figure 5. [left] shows a heatmap denoting the global attentiveness of
individual stocks in the overall portfolio. [right] correspondingly presents a
heatmap of individual assets. The agent chooses to allocate the most weight of
the portfolio to NVDA, while surprisingly focusing most on a declining stock,
GILD. The agent reduces the weightage of GILD and allocates to NVDA. Adapted
from (Shi et al., 2021).

Model-agnostic VE techniques can be integrated with any form of model
architecture and bear a similar resemblance with feature relevance techniques.
Both investigate the effects on the model’s output by adjusting the input
features. (Zhang et al., 2022d; Biecek et al., 2021; Zhang et al., 2022c;
Farzad, 2019) employ Partial Dependence Plots (PDP) to visualize the marginal
effects of features relating to corporate distress, credit scoring, and
detecting mortgage loans defaults. The generated plots can enable a way of
inferring if the underlying input-output relationship is linear or complex.
However, PDP has often been criticized for its assumption of independence
between features, evaluating unrealistic inputs, and also conceals any
heterogeneous effects of the input features. Accumulated Local Effects (ALE)
(Apley and Zhu, 2016) address the concerns of feature correlation by
considering the conditional distribution rather than the marginal one. In
particular, it accumulates differences between intervals within the feature
set to account for individual feature effects. (Crosato et al., 2021) employs
ALE on top of a tree ensemble model, XGBoost (Chen and Guestrin, 2016), as
well as with global Shapley values (Shapley et al., 1953) for better
scrutability. This work deduces that the increase in profit margin and
solvency ratio leads to lower debt default rates of small enterprises. (Zhang
et al., 2022c) evaluates across an arsenal of XAI techniques, encompassing the
aforementioned, and also includes Individual Conditional Expectation (ICE) for
financial auditing purposes. ICE differs subtly from PDP in that it considers
instance-based effects rather than averaging across all instances, making it a
local approach (see Figure 6). (Zhang et al., 2022a) generates counterfactual
explanations on credit loan applications by coupling unsupervised VAE with a
supervised probit regression. The combined model yields a discriminative
latent state, corresponding to class labels of either delinquency or non-
delinquency. The counterfactual is subsequently produced by a stepwise
manipulation function towards the opposite class label. The authors evaluate
the generated counterfactuals quantitatively using maximum mean discrepancy
(MMD) (Zhang et al., 2022b), which measures the number of successfully flipped
class labels as well as minimal feature changes.

Figure 6. [left] shows a PDP on averaged marginal effects of total assets on
the probability of statement restatement and [right] displays ICE, which
considers instance-level relationship. Both show a negative relationship.
(Zhang et al., 2022c).

###  3.2. Explanation by Simplification

Figure 7. LIME process: Predictions of black-box model are uninterpretable.
The local instance in the red box is the target to be explained. Subsets of
nonzero elements of the target instance are uniformly drawn to form a local
dataset on which the surrogate transparent model is trained on. The prediction
from the linear transparent model can then be interpreted by the user.
(Ribeiro et al., 2016).

The idea of Explanation by Simplification (ES) techniques is to introduce a
surrogate model performing uncomplicated operations. The purpose is to allow
the machine learning developer to formulate a mental model of the AI model’s
behavior. The surrogate model has to be interpretable and more importantly
capture the performance of the black-box model with high fidelity. The latter
property should be given a higher priority since there is little use for
interpreting a low-fidelity solution. ML techniques which apply linear
operations and rule extraction are applicable as surrogate models in place of
uninterpretable neural networks. These include decisions tree (DT) with
limited depth, linear/logistic regression, K-Nearest Neighbors (KNN), and
generalized linear models (GLM).

Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro et al., 2016):
is perhaps one of the most popular explanation techniques across various use
cases, including finance. LIME is a model-agnostic method that is used to
provide insight as to why a certain prediction was made and can be constituted
as an outcome explanation technique. Since LIME is a local-based technique, it
only has to approximate the data points within a defined neighborhood,
achieving a much more realistic goal instead of capturing an interpretable
representation of the entire dataset. On a high level, LIME can be implemented
as follows (see Fig 7):

  1. (1)

The target instance to be explained is denoted as $x\in\mathbb{R}^{d}$.
Uniformly sample $n$ random subsets of nonzero elements of $x$ to form local
training points, $z\in{z_{1},z_{2},...,z_{n}}$, where $z_{i}\in\mathbb{R}^{d}$
for $1\leq i\leq n$.

  2. (2)

Derive labels $f(z_{i})$ for each point using the black-box model $f$. The
surrogate model, $g$ is then trained on the derived dataset, $\\{z,f(z)\\}\in
Z^{n}$.

  3. (3)

Choose a transparent surrogate model, $g$ and train it on the dataset, $Z^{n}$
via Equation 1.

  4. (4)

Interpret the outputs of the transparent model on the target instance, $g(x)$.

LIME minimizes the following loss function to optimize for both fidelity of
the local model as well as minimal complexity.

(1) $\mathrm{explanation}(x)=\underset{g\in
G}{\mathrm{argmin}}[\>L(f,g,\pi_{x})+\Omega(g)]$

$L$ represents the loss function of the surrogate model $g$ on the labels $f$,
weighted by proximity $\pi_{x}$. $\Omega$ represents the complexity or number
of features in the surrogate model. $G$ is the set of all locally fitted
models, where each explanation is produced by an individual local model. The
authors additionally propose a sparse selection of features, named Submodular
Pick LIME (SP-LIME), to present the observer with a global view, based on an
allocated budget of maximal features to focus on. The method delivers diverse
representation by omitting redundancy. (Misheva et al., 2021; Serengil et al.,
2022) use LIME on top of tree ensembles to identify the contributions of
individual features pushing towards predicting a specific borrower as
defaulting or successfully paying off the loan. Such explanations can be
useful in preventing social bias by discovering any socially discriminative
features on which the model may be focused, thereby instilling trust in the
model’s usability. (Yan et al., 2019) extends LIME towards financial
regulators requiring commercial banks to adhere to a set of financial factors,
where they propose a method named LIMER (R stands for Regtech). The authors of
LIMER argue that high acceptance of financial solutions can be achieved if
such factors are integrated into the explainability design of the AI model.
(Collaris et al., 2018) implements model simplification by extracting logical
rules from a random forest and selecting top most relevant rules. The decision
rules are extracted from a local dataset, derived similarly to LIME without
weighting the proximity of each drawn sample. (Maree and Omlin, 2022b) trains
a recurrent neural network (RNN) to classify customer spending into five
categories and an interpretable linear regression model was subsequently
trained to predict the nodes formed by the RNN model. The authors then perform
inverse regression which provides a mapping from output space to state space
where the features responsible for categorizing customer spending can be
identified.

###  3.3. Feature Relevance

Feature relevance (FR) techniques account for the majority of the proposed
explanation methodology we reviewed. FR techniques revolve around computing a
relevance score for each feature, highlighting the respective contribution of
the target feature either at a global or local scale. SHapley Additive
exPlanations (SHAP) (Lundberg and Lee, 2017), motivated by the fair
distribution among players from game theory (Shapley et al., 1953) is a highly
popular FR technique, which seeks to estimate the fair value of each feature
in contributing towards the outcome, $f(x)$. The fair value, otherwise known
as shapley values are determined, based on estimating the difference between
the black-box function over feature subset $S$ with and without the target
feature, $f(x_{S\cup\\{i\\}})$ and $f(x_{S})$ respectively. The difference is
then averaged across all possible coalitions within the feature set $F$.

(2) $f(x)=g(x^{\prime})=\phi_{0}+\sum^{M}_{i=1}\phi_{i}x^{\prime}_{i}$

The final outcome is intuitively derived as an aggregate over all non-zero
shapely values. SHAP’s popularity stems from three attractive properties:
guaranteeing a complete approximation of the original model $f(x)$ through
additive feature attribution (see Equation 2), ensuring non-contributing
features have no impact on model output and consistency of feature values
tracking the outcome contribution. We notice a large subset of papers reviewed
has utilized SHAP as an explanation approach, likely given its flexibility
towards explaining the model at both local and global scales (see Figure 8).
(Dikmen and Burns, 2022) incorporates SHAP with additional credit knowledge
for the layperson to assess the logic of XGBoost’s decision in a peer-to-peer
lending scenario. (Müller et al., 2022) introduces RESHAPE, designed for
unsupervised deep learning networks, which provide explanations at the
attribute level. Such explanations can assist auditors in understanding why an
accounting statement is flagged as anomalous. The authors evaluated RESHAPE
against other variants of SHAP, based on metrics measuring fidelity,
stability, and robustness. Attributing to the recent frenzy in cryptocurrency
which has led to a number of studies attempting to predict movements in the
cryptocurrency market, (Fior et al., 2022) proposes an interactive dashboard
providing multiple graphical tools using SHAP for financial experts. (Babaei
et al., 2022) applies SHAP to explain predictions, generated by the popular
mean-variance Markowitz model (Markowitz, 1952) which is an optimization model
for establishing the optimal portfolio balancing between returns and risk. The
generated explanation provides regulators a means of asserting compliance of
algorithmic automized traders, otherwise known as robot-advisors, with
established rules and regulations. (Demajo et al., 2020) incorporates Global
Interpretation via Recursive Partitioning (GIRP) with SHAP as a global
interpretability technique. GIRP uses the importance values generated by SHAP
to further extract meaning insights from tree models, and the method is
compared against a boolean rule technique in a credit scoring use case.
(Bussmann et al., 2021) constructs a tree-like visual explanation with
TreeSHAP (Lundberg et al., 2018), specifically designed for ensemble trees
with an improvement in computational efficiency. The produced structure allows
users to visualize clusters of similar outcomes describing company default
risk. (Yasodhara et al., 2021) compares TreeSHAP against impurity metrics
using information gain, on ensemble tree models for investment quality
prediction. (Gramegna and Giudici, 2020) identifies relevant features leading
to consumers’ decision on purchasing insurance and further clusters them into
least to most likely groups with Shapley values. (Bussmann et al., 2020)
similarly implements SHAP to explain XGBoost’s classification of credit risk,
while comparing it against an interpretable logistic regression model. Other
studies include discovering the relationship between corporate social
responsibility and financial performance (Lachuer and Jabeur, 2022), customer
satisfaction (Rallis et al., 2022), GDP growth rates (Park and Yang, 2022),
stock trading (Benhamou et al., 2021; Kumar et al., 2022), financial distress
(Tran et al., 2022), market volatility forecast (Weng et al., 2022) and credit
evaluation (Rizinski et al., 2022; Bueff et al., 2022; Fritz-Morgenthal et
al., 2022).

Figure 8. [left] Values of feature importance at a global level for the ML
model’s decision in credit card approval. [right] An example of instance-
level, $E(f(X))$ represents the model’s base prediction if no features were
considered, and $f(x)$ represents the final prediction after summing the
contributing features ($\phi_{i}$) (Rizinski et al., 2022).

(Wand et al., 2022) performs K-means clustering on historical S&P 500 stock
information to identify dominant sector correlations which describe the state
of the market. This work applies Layer-wise Relevance Propagation (LRP) (Bach
et al., 2015), after transforming the clustering classifier into a neural
network since LRP is designed to work specifically with neural network
architectures. (Carta et al., 2022) prunes unimportant technical indicators
using different configurations of a permutation importance technique, before
implementing decision tree techniques for stock market forecasting. The
proposed technique was compared with LIME and demonstrated better reliability.
(Bracke et al., 2019) introduces a set of feature relevance techniques,
Quantitative Input Influence (QII) (Datta et al., 2016) to compute interaction
effects between influential features and additionally for each multi-class
label. The authors additionally evaluated the ability of the XAI technique
with five questions relating to each individual audience class. All of the XAI
methods shown thus far are implemented in the post-modeling stage, while
(Islam et al., 2019) is an example pertaining to pre-modeling where the
identification of relevant features takes place before constructing the black-
box model. This work explores the set of features relating to mortgage
bankruptcy and performs feature mapping against a set of widely-used credit
concepts. The utility of such an approach is confirmed through empirical
evaluations.

As pointed out before, contrastive explanations are usually preferred. End-
users subjected to an unfavorable AI model’s decision would prefer a solution
to the problem rather than a fact-based explanation which may present multiple
possible reasons, giving little use to the explanation receiver. An
explanation providing changes to be made such that the outcome can be reversed
towards the favorable is referred to as a counterfactual explanation.
Counterfactuals are derived by computing small changes to the input features
continuously until the outcome is altered to the target class. (Cho and Shin,
2023) first identifies significant features, attributing to bankruptcy through
SHAP, and subsequently generates an optimal set of counterfactuals using
Genetic Algorithm (GA). The loss optimized by GA composes of objectives
describing desirable properties of a good counterfactual outcome, including
minimizing the size of altered features and maximizing the feasibility of the
outcome. (Grath et al., 2018) additionally provides positive counterfactual
explanations, describing the required changes to the current inputs that would
instead reverse the loan approval to rejection. Such explanations can provide
some form of safety margin for the user to be mindful of. (Vivek et al., 2022)
used various technique from DiCE (De Bruin et al., 2009) to generate
counterfactuals under five different experimental conditions. The experiment
aims to identify and study the effects of the causal variables in the fraud
detection of ATM transactions.

###  3.4. Explanation by Example

Apart from techniques that identify feature relevance on varying scales or
approximate with a surrogate model, another form of explanation exists by
selecting representative samples to illustrate the model’s behavior. Such
techniques can be classified as Explanation by Example (EE). One such
technique includes prototype-based explanations. Prototypes can be regarded as
representatives of the entire dataset, chosen based on similarity and
importance in the overall decision-making of the model. (Demajo et al., 2020)
implements protodash (Gurumoorthy et al., 2019), a gradient-based algorithm in
a credit loan application to select top $m$ prototypes, of which the top two
are selected, with $m$ being 6. The resulting outcome is a number of
representative prototypes and each instance can be represented by either
generated prototype in the clusters. In this case, the proportions of
allocated instances were balanced between both prototypes. The number of
prototypes is a hyperparameter to be fine-tuned. A higher value of $m$ is
frequently used where the complexity of the problem is a concern, albeit
raises the risk of overfitting, while a lower value is used in simpler
scenarios but incurs the risk of underfitting. For example, in the credit loan
dataset, the selection of two prototypes was considered too little by domain
experts who instead prefer 3-4 as being sufficiently representative of the
evaluated dataset. (Davis et al., 2022) similarly extracts representative
instances using KNN and generates insights on out-of-sample instances by
looking for similarities with the representative points. Additionally, the
data points for computing the distance are instead replaced with Shapley
values, taking into account the importance of input features. Representative
samples are generally suitable if the user is interested in determining the
types of patterns or behavior found in the dataset while being relatively fast
and straightforward to implement.

###  3.5. Summary of numerical features

The above-mentioned approaches should be chosen according to the task at hand
and the target audience. VE techniques, such as deconvolution and Grad-CAM,
are less commonly used in the financial industry due to their limited
applicability to networks other than CNNs. However, ALE, PDP, and ICE can be
suitable approaches for financial analysts who might want to study the
relationship between individual features and the model’s outcome. ES is a
straightforward approach that delegates the interpretability problem to a less
complex surrogate model, though it incurs the additional cost of ensuring the
faithfulness of the surrogate model. FR techniques allow users to observe each
feature’s contribution to the black-box prediction. Both global and local
explanations serve different purposes for individual audiences. However, in
situations where each feature equally contributes to the model outcome, such
explanations might not be very helpful depending on the objective of the
explanation. For example, a declined credit loan approval may have multiple
features such as prior default, debt-to-income, and household capital
contributing equivalently to the outcome. Such an explanation does not offer
an obvious course of action for the applicant. EE techniques are particularly
useful when the user wants a small set of representative samples to explain
the model’s outcome. This can provide a fast and straightforward explanation,
but it has limited usefulness.

##  4\. XAI for Textual information

In this section, we review models operating with textual information. We note
that the papers pertaining to this area represent the minority in the overall
literature. When it comes to data preparation, textual data generally require
additional pre-processing works such as stop word removal, stemming,
lemmatizing, and tokenization. In terms of feature extraction, the semantics,
and syntax structure around text is important and have to be learned to fully
capture the information conveyed, unlike numerical features which are readily
usable. Models suitable for training from textual data are also limited to a
smaller subset of available techniques. Nevertheless, unstructured data such
as text are in abundance. If properly processed, textual data can be used to
derive informative signals such as market sentiment and emerging trends (Ma et
al., 2023). Textual data are commonly classified under alternative
information, which comes in a wide variety of sources including social media,
online reviews, blog posts, and news headlines (Kolanovic and Krishnamachari,
2017), in contrast to non-alternative information which refers to data
commonly utilized for financial analysis. Fortunately, a wide variety of
explanation techniques exist which are compatible with textual information.
Conveniently, textual information is applicable for XAI techniques delivering
explanation via text generation, which can be preferable for the layperson as
natural language provide an easier form of interpretation as compared to
statistical graphs.

###  4.1. Text Explanation

Text explanation techniques provide clarity in the form of generating
informative textual statements to assist in the understanding of the model’s
behavior. The generated text can either be re-generated text statements by
using some form of generative model or replacing selected words in the
original sentence. (Srinivasan et al., 2019) utilizes Generative Adversarial
Networks (GAN) to produce text statements that seek to align with user-defined
inputs. Specifically, the explanation can take two different objectives,
either converting actionable text to educational text or vice versa. The
actionable text briefs audiences on optimal actions to consider, based on
real-world responses from human responses on multiple loan application
scenarios while the latter seeks to educate the audience on reasons
attributing to the response. The transfer of objectives from one to another is
analogous to the implementation of style transfer on images, a popular
application of GANs that translates the style of an instance to the target
image while retaining the content. Figure 9 shows a snippet of an example, the
proposed method can identify the semantics behind the statement and relay the
relationship between consistency and time, while knowing if the current income
is above or below the required threshold.

Figure 9. [Top] Transfer of educating statement to actionable statements
advising applicant on actions to take such that the subsequent loan
application can be approved. [Bottom] Transfer of original statement
highlighting actions to educating statement conveying the reason for rejected
loan application. (Srinivasan et al., 2019).

(Yang et al., 2020) generates plausible counterfactual text sentences with a
transformer architecture, trained under contextual decomposition. The
explanation technique, derived from Sampling and Contextual Decomposition
(SCD) (Jin et al., 2019), performs different actions including inserting,
removing, or replacing words that are representative of the context in the
statement, based on the target objective. The high-level idea of
counterfactual generation involves identifying the most relevant word and
replacing it with an antonym from a reference dictionary and continues until
the outcome is reversed. The proposed transformer outperforms even human
experts in classifying financial articles on merger & acquisition event
outcomes. (Yuan and Zhang, 2020) generates text explanations using a state-of-
the-art natural language generation transformer decoder, GPT-2 (Radford et
al., 2019), while fulfilling soft constraints of including keywords. The
proposed technique, soft-constrained dynamic beam allocation (SC-DBA) extracts
keywords corresponding to various levels of predicted market volatility using
a separate network on harvested news titles. The quantitative measurement is
evaluated based on the fluency and utility of the explanation produced.

###  4.2. Visual Explanation for Text

Besides interpreting through text, users can understand through the form of
visuals, which makes the use of attention a particularly attractive option.
Attention was first introduced when it was used to consider correlations
between words in a sentence in a parallel fashion and is a primary component
in the transformer architecture (Vaswani et al., 2017). Transformers are
notably suitable for processing long sequences of text and through the use of
attention. They are computationally efficient compared to RNN-based models. It
so happens that, computing attention scores of each word serves as a natural
form of interpretation, by allowing users to visualize how the network is
capturing information from the input text (Han et al., 2022). Representative
works in this area employ attention to highlight regions of text sentences
that are deemed relevant for the output. (Yang et al., 2018) utilizes dual-
level attention with Gated-Recurrent Units (GRU) (Chung et al., 2014),
processing both inter-day and intra-day embedding of news titles relating to
S&P 500 companies. The attention module assigns a relevance score to each news
article and the authors additionally construct a knowledge graph conducting
concept mapping between relevant entities as a visual explanation.
Corresponding to dual-level attention, (Luo et al., 2018; Lin et al., 2021)
propose a hierarchical attention model at both the word and sentence level and
produced explanations in the form of a heatmap, highlighting relevant text.
The proposed method, FISHQA was trained to detect loan arrears from financial
text statements, similar to the compared baselines. The uniqueness of the
proposed method lies in providing FISHQA with additional user queries. The
model was able to highlight regions of the statement corresponding to the set
of expert-defined concepts. This form of explanation allows users to verify if
the model is focusing on the correct terms relating to the concept at hand
(refer to Figure 10).

Figure 10. FISQHA: hierarchical attention model, different colors relating to
different financial concepts, grey - company, light brown - executives, red -
financing, blue - litigation, teal - personnel. (Luo et al., 2018).

Along the lines of hierarchical attention, (Lin et al., 2021) introduces a
quantitative measure to evaluate the precision and recall of captured against
various lexicon dictionaries and expert annotated lists. The approach,
analogous to the former study can be seen as an extrinsic process of ensuring
the correctness of concept identification, by capturing words associated with
financial risk. (Deng et al., 2019) implements knowledge graphs to provide a
visual linkage between event entities extracted from stock news articles. The
approach offers users a visual understanding between the feature’s
relationship and the corresponding prediction. (Ito et al., 2020) introduces
GINN, an interpretable neural network, the network is designed in a way that
each layer represents different entities such as words and concepts at the
node level. The approach identifies words attributing to the predicted
sentiment labels, as well as the concepts it belongs to.

###  4.3. Summary of textual information

TE techniques aim to augment existing input text or generate new text based on
given inputs. Such explanations are commonly preferred since natural language
is easily understood by humans if the explanation is concise and accurate.
However, textual explanations may fail to capture the nuanced relationships
between input features and model decisions. This can be especially detrimental
and counterproductive to domain experts whose goal is to discover further
improvements based on the provided explanations. Textual explanations might
also require additional processing work to ensure fluency, coherence, and
unambiguity. VE techniques in these works leverage the utility of attention to
provide a glimpse into how the model is representing the input text, and the
use of hierarchical attention allows for a more refined analysis. However,
since attention captures the relationship between each word or sentence, such
explanations might be overwhelming if the set of explainable features is too
large. Audiences who are not well-versed in heatmaps or attention scores may
have difficulty understanding the provided visuals.

##  5\. XAI for Hybrid information

The remaining studies implementing post-hoc explanation techniques utilize a
combination of both textual and numerical/technical features. With respect to
instance-level explanations, (Bandi et al., 2021) combines the sentimental
analysis of text with technical analysis of historical stock prices to train a
random forest stock forecasting model, explained through LIME. The resulting
explanation computes a set of relevant feature values and news wording
corresponding to the respective outcome. Similarly, (Gite et al., 2021)
implements LIME with LSTM-CNN and accurately identifies attentive words in
consonant with the target sentiment. (Liu et al., 2020) predicts the
possibility of litigation on financial firms from examining 10-K financial
reports and numerical indicators concerning the firm’s accounting knowledge.
The author additionally carry out an ablation study on the utility of hybrid
information as opposed to individual and validated the initial approach.
Correspondingly, the explanation served to regulators is framed as the
identification of text leading to the suspicion of insider trading, with the
help of an attention mechanism. (Zhang et al., 2020a) adopted the practice of
shapley values and further integrate external knowledge regarding truth
factors, namely Truth Default Theory (TDT) (Levine, 2014) to detect
information fraud. The explanation module incorporates both shapley values and
TDT to generate a report highlighting numerical contributions of features as
well as a text explanation. A union of explanation by simplification and
feature relevance was proposed by (Cong et al., 2021; Ghosh and Sanyal, 2021).
(Ghosh and Sanyal, 2021) implements both LIME and SHAP, offering a global and
local explanation of market fear prediction in the Indian financial market.
(Maree et al., 2020) uses SHAP and identifies textual information to be more
important for classifying financial transactions and further performs
clustering to identify top contributing keywords. (Cong et al., 2021)
interprets an RL-trained agent’s behavior in algorithmic trading. The
resulting explanation enables experts to focus on time-dependent variables
alongside consideration of non-linearity effects, which are reduced to a small
subset of initial variables. The learned policy is simplified via policy
distillation, onto the space of linear regressions such that an interpretable
Lasso regression model can be used as an interpretable approximation.
Subsequently, $k$-degree polynomial analysis is conducted to select salient
features, with $k$ acting as an additional flexibility for the developer to
decide. (Ong et al., 2023) utilizes aspect-based sentiment analysis to study
the relationship between stock price movement and top relevant aspects
detected in tweets. The polarity of each aspect is derived from a SenticNet-
based graph convolutional network (GCN) (Liang et al., 2022). The proposed
method can be seen as analogous to the feature relevance technique, aimed at
deriving top contributing aspects with polarity values. The proposed work
focuses on the relationship between financial variables instead of making
financial predictions. Such information can allow for further analysis,
leveraging the relationship between the price movement of individual stocks
and individual sentiment of popular terms detected in tweets.

Hybrid information combines the utility of both numerical and textual
information, which can lead to better performance and an increase in the
number of compatible explanation techniques. For example, text generation
techniques can be used to generate natural language explanations for non-
technical audiences, facilitating ease of understanding, while feature
relevance approaches can be utilized to identify top contributing factors in
the feature domain for technical experts. Models working with both numerical
and textual information can also benefit from a performance point of view if
such information can be processed without the risk of overfitting. However, it
may be difficult for models to seamlessly perform with hybrid information, as
it ultimately depends on the task at hand and may require complex feature
engineering. For instance, the utility of text information largely depends on
the source and often requires a significant amount of preprocessing before the
data can be useful. The combination of both text and numerical features may
increase the complexity of the explanation and end up being counterproductive.
Such issues limit the inclusion of textual information in use cases such as
stock trading or market index predictions. Nonetheless, we note that
leveraging hybrid information to provide explanations can be a promising
approach if the aforementioned issues are addressed.

##  6\. Explainability in Transparent models

The remaining studies look at instigating explainability from inherently
transparent models. These models are typically restricted to ML models
performing linear operations or rule extraction. The medium of explanation in
transparent models is by nature model-specific, in the sense that the same
mode of explaining how a model functions to an audience likely cannot be
reused by a different model. The usability of transparent models in
complicated tasks is largely restricted due to their poor predictive strength.
Nevertheless, transparent models still remain an attractive option if
sufficient performance can be guaranteed.

Linear/Logistic Regression: Linear regression model is among the earliest ML
models to be used for quantitative analysis. The prediction outcome can be
easily derived as a weighted aggregate of input features. As such, the outcome
can naturally be interpreted by inferring from the coefficients, $W_{i}$ which
serves as a quantitative measure of feature importance for the outcome.
Attributing to the linearity assumption, the output $y$ can be derived as
such:

(3) $y=W_{0}+W_{1}x_{1}+W_{2}x_{2}+...W_{n}x_{n}+\epsilon$

One can easily interpret the outcome as “By increasing feature $x_{i}$ by one
unit, the output increases by $W_{i}$”. On the other hand, the logistic
regression model is interpreted in a slightly different manner, since the
output is bounded between [0,1], a logistic function is used. Logistic
regression looks at the probability ratio between both outputs: “Increasing
one unit of $x_{i}$ is equivalent to increasing $\frac{P(y=1)}{P(y=0)}$ by
$exp(W_{i})$” (Molnar, 2020). (Dumitrescu et al., 2022) addresses the trade-
off between accuracy and interpretability by incorporating decision trees with
logistic regression acting as the main operational backbone. The technique is
coined as Penalised Logistic Tree Regression (PLTR). PLTR extracts binary
variables from short-depth DTs, all the while establishing sparsity through a
penalized lasso operation. The proposed model is able to account for non-
linear effects in a credit-scoring dataset while retaining interpretability by
observing the top-selected rules.

Decision Trees: Decision trees are one of the most commonly used techniques in
machine learning problems due to their simplicity and easily understandable
structure. Unlike linear/logistic regression, DT can approximate nonlinear
relationships and yet remain interpretable via simple if-else logic. However,
the transparency of tree models diminishes with increasing depth, and popular
ensemble tree models such as XGBoost or gradient-boosting tree models
completely eliminate any form of interpretability. The user can interpret
decision trees by traversing through the root node and upon arrival at each
leaf node. The outcome can simply be explained as “if $x_{1}$ is ¿/¡
$threshold_{1}$ AND $x_{2}$ is ¿/¡ $threshold_{2}$, $\cdots$ , outputs $Y$”.
(Gramespacher and Posth, 2021) employs a single DT and frames the loan
approval task as one which maximizes profit for the lender firm. (Carta et
al., 2021) builds a lexicon dictionary associated with stock price variation,
extracted from a dataset comprising both news and historical stock prices. The
combined effort provides users with two forms of explanation, observed in a
sequential rule-based manner as well as words correlated with the predicted
market direction.

Others: (Adams and Hagras, 2020) constructs an interactive platform, Temenos
XAI using fuzzy logic to make financial predictions. The authors demonstrated
the efficacy and explainability in various downstream banking and trading
scenarios. The usage of fuzzy-logic accounts for uncertainty, which is
prevalent in the financial environment, and is especially useful for modeling
imprecise information. The platform allows users to interpret the model on a
global scale as well as at an instance-level, via observing the top
contributing rules. (Chen and Ye, 2022) builds on top of neural additive
models (NAM) (Agarwal et al., 2021) and introduces a generalized form of NAM,
GGNAMS which focuses on sparse nonlinear interactions. GGNAMS can be regarded
as an intermediate between fully connected networks and logistic/linear
regression with the intent being to retain linearity and minimize excessive
interactions among features while maximizing accuracy. The additive components
can then be interpreted similarly to LR. (Nazemi et al., 2022) similarly
implements NAMs and Explainable Boosting Machine (Nori et al., 2019) to
identify financial drivers leading to creditor recovery rates. (Dumitrescu et
al., 2022) proposes a hybrid approach of combining decision trees with
logistic regression, capturing nonlinear effects while retaining the
transparency of the model’s behavior. (Sudjianto and Zhang, 2021) advocates
for designing inherently transparent models in the pre-modeling/modeling
stages and suggested a qualitative template, describing properties of model
interpretability. The intention of the template is to allow researchers to
ensure model interpretability while designing the model architecture. As a
proof of concept, this work designs an interpretable ReLU network while
conforming to the proposed template, and evaluates the network in a credit
default classification task. The resulting network can be disentangled into a
set of local linear models whose inherent transparency can be visualized by
observing the local coefficients.

Transparent models have the advantage of being interpretable without requiring
additional approaches to interpret the model or outcome. However, there exists
a clear trade-off between desired performance and sufficient interpretability.
Certain works have introduced approaches that combine different transparent
models to achieve better performance while still retaining as much model
transparency as possible. Transparent models remain a popular choice in the
financial domain, as companies must undergo routine audits that require
audited firms to provide accountability for their algorithmic services offered
to end-users. Nonetheless, given the monotonic success of deep learning
models, companies seeking to maintain their competitive edge must either
improve on the existing transparent models or balance the performance-
interpretability trade-off. A recommended approach would be to stick with
transparent models if their performance proves sufficient and proceed with
less interpretable models otherwise. One could also break down the task in a
hierarchical manner, using interpretable models for lower-level tasks and
better-performing models for more complicated tasks.

##  7\. FinXAI and ethical goals

In Sections 3-6, we have reviewed different FinXAI methods and summarized
their technical strengths and weaknesses, based on representative papers over
the past years. In this section, we analyze the contributions of these FinXAI
techniques to the ethical goals that were set out in Section 2.2. We also
discuss some of the goals lacking sufficient study in current FinXAI
techniques. In this section, the goal of accessibility also encompasses the
fact that developers and domain experts can easily access the decision-making
mechanisms of complicated black-box models due to improved interpretability.
This is slightly different from the narrative that we introduced in Section
2.2, which mainly focuses on accessibility for non-expert users. Such an
extension can better explain the technical contributions of the reviewed works
to XAI.

Table 4. The contributions of current FinXAI techniques to ethical goals. VE, ES, FR, EE, and TE denote visual explanation, explanation by simplification, feature relevance, explanation by example, and text explanation, respectively. Method | Trust. | Fairness | Informative. | Accessibility | Privacy | Confidence | Caus. | Trans.
---|---|---|---|---|---|---|---|---
Num. Feat. | VE | (Zhang et al., 2022a) |  |  |  (Achituve et al., 2019) (Biecek et al., 2021) (Chen et al., 2020)
---
(Crosato et al., 2021) (Farzad, 2019) (Kumar et al., 2017)
(Shi et al., 2021) (Zhang et al., 2022c)
(Zhang et al., 2022d)
|  (Achituve et al., 2019) (Crosato et al., 2021) (Zhang et al., 2022c)
---
|  |  |
ES |  |  (Collaris et al., 2018) (Maree and Omlin, 2022b)
---
(Yan et al., 2019)
(Misheva et al., 2021) (Serengil et al., 2022) |  |  |  |  |  |
FR |  |  (Babaei et al., 2022) (Cho and Shin, 2023)
---
(Grath et al., 2018)
| (Bussmann et al., 2021) |  |  (Babaei et al., 2022) (Benhamou et al., 2021) (Bracke et al., 2019)
---
(Bueff et al., 2022) (Bussmann et al., 2020) (Carta et al., 2022)
(Demajo et al., 2020) (Dikmen and Burns, 2022) (Fior et al., 2022)
(Fritz-Morgenthal et al., 2022) (Gramegna and Giudici, 2020) (Islam et al.,
2019)
(Kumar et al., 2022) (Lachuer and Jabeur, 2022) (Müller et al., 2022)
(Park and Yang, 2022) (Rallis et al., 2022) (Rizinski et al., 2022)
(Tran et al., 2022) (Vivek et al., 2022)
(Wand et al., 2022) (Weng et al., 2022)
(Yasodhara et al., 2021)
|  | (Vivek et al., 2022) |
EE | (Davis et al., 2022) |  |  |  |  |  (Davis et al., 2022) (Demajo et al., 2020) |  |
Text. Info. | TE | (Yang et al., 2020) |  |  (Srinivasan et al., 2019) (Yuan and Zhang, 2020) |  (Srinivasan et al., 2019) (Yang et al., 2020) |  |  | (Yuan and Zhang, 2020) |
VE |  |  |  (Lin et al., 2021) (Luo et al., 2018) (Yang et al., 2018) |  (Deng et al., 2019) (Ito et al., 2020) (Lin et al., 2021) |  |  |  |
Hybr. Info. |  |  |  |  (Bandi et al., 2021) (Cong et al., 2021) (Ghosh and Sanyal, 2021)
---
(Gite et al., 2021) (Liu et al., 2020) (Maree et al., 2020)
(Zhang et al., 2020a)
|  (Bandi et al., 2021) (Cong et al., 2021) (Ghosh and Sanyal, 2021)
---
(Gite et al., 2021) (Maree et al., 2020) (Ong et al., 2023)
|  |  |
Tran. Mod. |  |  |  |  |  (Adams and Hagras, 2020) (Carta et al., 2021) (Chen and Ye, 2022)
---
(Dumitrescu et al., 2022) (Gramespacher and Posth, 2021) (Nazemi et al., 2022)
(Sudjianto and Zhang, 2021)
|  |  |

As seen in Table 4, different explainable methods contribute to the ethical
goals from different aspects. XAI for numerical features has proposed several
methods to approach ethical goals, regarding trustworthiness, fairness,
informativeness, accessibility, confidence, and causality. For VE methods,
(Zhang et al., 2022a) is one example that advocates for trustworthiness by
generating counterfactual explanations, while being informative by detecting
important features that can alter the prediction. Counterfactual explanations
reveal the slightest modifications that are necessary on the input data to
achieve an alternative outcome. It helps to earn trustworthiness from target
audiences because counterfactual explanations provide possible rescue measures
for them to achieve their targets with minimum effort, e.g., proposing
possible improvements to help borrowers pass the qualification review of
credit agencies. Counterfactual explanations can also help to justify
predictions besides factual explanations. Both merits of generating
counterfactual explanations improve the trustworthiness of audiences. Many VE-
based approaches improve informativeness by gaining insights into the
decision-making mechanisms of models and revealing feature correlations
(Achituve et al., 2019; Biecek et al., 2021; Chen et al., 2020; Crosato et
al., 2021; Farzad, 2019; Kumar et al., 2017; Shi et al., 2021; Zhang et al.,
2022c, d), because visualization takes the advantages of demonstrating
patterns and trends of data, e.g., model parameters and numerical features. VE
can be also used to discover valuable features (Achituve et al., 2019; Crosato
et al., 2021; Zhang et al., 2022c). It is easy to communicate with both
experts and non-domain experts by using graphical representations or visual
images whenever available. Thus, VE also improves accessibility for broader
audiences. For ES methods, (Collaris et al., 2018; Maree and Omlin, 2022b; Yan
et al., 2019) can explain why a certain prediction was made from outputs,
which helps to improve the trustworthiness of AI predictions. (Misheva et al.,
2021; Serengil et al., 2022) use LIME to detect socially discriminative
features to prevent social bias. ES is the only approach that was used for
improving fairness in finance. For FR methods, (Babaei et al., 2022) uses SHAP
to improve the trustworthiness of algorithmic traders in crypto markets. (Cho
and Shin, 2023; Grath et al., 2018) generate contrastive explanations to
explain required changes for certain predictions. (Bussmann et al., 2021)
visualizes similar outcomes that describe the risk of a company default with
SHAP, while most SHAP-based methods (Babaei et al., 2022; Benhamou et al.,
2021; Bracke et al., 2019; Bueff et al., 2022; Bussmann et al., 2020; Carta et
al., 2022; Demajo et al., 2020; Dikmen and Burns, 2022; Fior et al., 2022;
Fritz-Morgenthal et al., 2022; Gramegna and Giudici, 2020; Islam et al., 2019;
Kumar et al., 2022; Lachuer and Jabeur, 2022; Park and Yang, 2022; Müller et
al., 2022; Rizinski et al., 2022; Tran et al., 2022; Vivek et al., 2022; Wand
et al., 2022; Weng et al., 2022; Yasodhara et al., 2021) improve accessibility
for technical audiences by discovering important features. (Fior et al., 2022)
improves usability by constructing interactive graphical tools upon SHAP,
which likewise promotes accessibility. (Vivek et al., 2022) is one of the rare
works that study causal inference based on generated counterfactuals. For EE
methods, (Davis et al., 2022) generates counterfactuals to explain the
required changes, based on representative instances. The selected
representatives of similar instances by EE methods (Davis et al., 2022; Demajo
et al., 2020) can be used to select instances to represent a particular
cluster in the output space. Similar instances aligned to such representatives
can assure and improve the confidence of stakeholders.

XAI for textual information targets to improve trustworthiness,
informativeness, accessibility, and causality. For TE methods, (Yang et al.,
2020; Srinivasan et al., 2019) similarly improves trustworthiness with
counterfactual texts, with the latter providing alignment according to the
user’s prompt. The alignment from educational to actionable information
enhances information flow, especially for individuals not familiar with the
service interface. For VE techniques operating on text, attention weights are
widely used for interpretation purposes. Such techniques enhance
informativeness and accessibility by using the attention weights to understand
regions of focus by the underlying model (Lin et al., 2021; Luo et al., 2018;
Yang et al., 2018). On the other hand, (Deng et al., 2019; Ito et al., 2020)
improve the interpretability of graph neural networks in the financial domain.

XAI for hybrid information leverages both textual and numerical features to
improve informativeness and accessibility. These works interpret the black-box
model’s behavior (Bandi et al., 2021; Cong et al., 2021; Ghosh and Sanyal,
2021; Gite et al., 2021; Liu et al., 2020; Maree et al., 2020; Zhang et al.,
2020a), and provide textual evidence regarding predictions (Bandi et al.,
2021; Cong et al., 2021; Ghosh and Sanyal, 2021; Gite et al., 2021; Maree et
al., 2020; Ong et al., 2023). (Adams and Hagras, 2020; Carta et al., 2021;
Chen and Ye, 2022; Dumitrescu et al., 2022; Nazemi et al., 2022; Gramespacher
and Posth, 2021; Sudjianto and Zhang, 2021) implements transparent models,
mitigating the need for post-hoc analysis, and the simplicity of such models
improves upon the accessibility for non-expert users. Transparent models may
be a suitable choice if the performance is satisfactory and the outcome has to
be readily interpretable by non-technical stakeholders. Decision trees are one
model which can be easily communicated to audiences without a technical
background, given its easily understandable format.

From the above works, we can find that most of the XAI research lies in either
studying the underlying model’s behavior or identifying important features.
Notably, the connotations of informativeness and accessibility goals are rich
as seen in Table 4, no XAI technique can achieve all of the desired goals. It
is therefore imperative that the XAI design process is tailored towards the
desired goals of the target audience Fig. 2. Likewise, the format of
presenting explanations is equally important. Non-technical audiences would
very much prefer user-friendly visuals as compared to technical plots.

On the other hand, the ethical goal of preserving data-privacy has not been
well studied in the works reviewed. Privacy-preserving techniques are a
popular research direction, e.g., federated learning (Yang et al., 2019). It
is a decentralized learning method that allows parties to collaboratively
train a model within a local environment without sharing their data with each
other. Generating synthetic data in place of actual data for training models
can be one such approach. One example is LIME which generates a local dataset
given a target instance, without requiring access to other data instances.
This can help to minimize the amount of information being accessed outside of
the accountable circle. The understanding of how various features lead to a
certain output creates the opportunity of generating more synthetic data. This
can be seen as a form of self-supervised learning with the purpose of
preserving privacy. However, XAI techniques can also become a double-edged
sword, attributing to privacy leakage instead. Such concerns are especially
prevalent in techniques manipulating decision boundaries including SVM,
K-nearest neighbors, and counterfactual explanations (Sokol and Flach, 2019).
For example, a counterfactual explanation on reserving a loan application
might reveal a suite of sensitive information (location, 10-year income,
marital status) to be modified, even though such information is meant to be
anonymized. The leaked information can be accessed by third-party providers
who may be part of the product design or malicious hackers. A key challenge is
managing the balance between the fidelity of the delivered explanation and the
sensitive features altered. Data leakage goes against the privacy awareness
goal of XAI and such events are not rare in the financial sector where there
exists a constant supply of computerized bots looking to capitalize on these
openings. The consequences often affect a large group of public stakeholders
(Dellinger, 2019), and the affected firm has to pay large fines and incur a
loss of trust from their clients. In addition, overly-expressive explanations
may allow external competitors to reverse-engineer the models and potentially
replicate and improve upon them, thereby compromising the competitive edge a
company holds.

XAI techniques that improve transferability are another less frequently
studied area. In the field of general AI, transferable knowledge is usually
acquired through transfer learning (Neyshabur et al., 2020), multi-task
learning (Mao and Li, 2021), meta-learning (He et al., 2022b), and domain
adaptation (Xie et al., 2022). However, the main carrier of these learning
paradigms at present is usually deep neural networks. It is difficult to
acquire explainability for a deep neural network by using these learning
methods. In addition, knowledge forgetting also brings challenges to
traditional neural network-based learning methods (He et al., 2022a). Thus,
the old knowledge stored in the neural network is likely to be replaced by the
learned new knowledge, if the old knowledge is not retained together with the
new knowledge. In light of this, how can we leverage explainability and
transferability, simultaneously? One possible direction is to utilize neural
symbolic techniques. Neural symbolic AI has achieved significant impacts in
natural language processing (NLP), e.g., sentiment analysis (Cambria et al.,
2022) and metaphor processing (Mao et al., 2022). It takes the merits of both
neural networks and symbolic representations. For example, neural networks
have strong generalization ability in learning feature representations.
Symbolic reasoning enables human-understandable explanations of the system’s
decision-making process through transparency and interpretability. Since
symbolic knowledge can be readily stored in a knowledge base permanently, it
avoids the problem of knowledge forgetting in neural networks. A comprehensive
and accurate knowledge base can weaken the fitting ability of the neural
network. As a result, a more lightweight and transparent neural network can be
used in a neural symbolic system. However, developing domain-specific
knowledge for finance is costly. Besides, developing symbolic representations
for numerical data is also challenging.

Finally, improving fairness, confidence, and causality is also important for
ethical concerns. Whereas, the FinXAI research in these areas is very limited.
As noted in Table 4, there are not many explanation methods that approach
these goals, e.g., ES for fairness with numerical features; EE for confidence
with numerical features; and FR and TE for causality with numerical and
textual features, respectively. However, it is difficult for a one-fit-all
explanation. Hence, we highlight the importance of an audience-centric XAI
technique as a more realistic expectation.

##  8\. Challenges and Future directions

We exploit the knowledge and insights gained from the agglomeration of FinXAI
research conducted thus far and put forward a list of challenges and
directions we consider to be important for readers to consider. A few of these
limitations have been similarly considered in previous works (Chen and
Storchan, 2021), which have presented seven major challenges encountered in
the context of presenting explanations to stakeholders. Some of these
limitations are evident from the reviewed XAI methodologies and we further
elaborate on them and cater avenues for improvement.

###  8.1. Over-reliance

A means of interpreting the model can be helpful while transforming how users
interact with data. However, it can cause users to over-rely on possibly
inaccurate explanations. A survey was conducted to study how data scientists
perceive explanations provided by different XAI tools and found out a large
proportion tend to over-trust the explanations provided (Kaur et al., 2020),
especially the ones which have received widespread usage. The visual
explanations delivered by feature relevance techniques such as SHAP, tend to
be absorbed at face value, which can cause researchers to not question their
legitimacy. Concurrently, a data scientist who has spent an enormous amount of
time designing the AI model may already have prior beliefs on the outcome or
model and are more inclined to accept the explanation if aligned with their
initial beliefs (Hohman et al., 2019). Such an occurrence is commonly known as
confirmation bias. Over-trusting these explanations can be especially damaging
if conveyed to the layperson and can result in the spread of misinformation to
a wider audience.

###  8.2. Social aspects

As mentioned by (Miller, 2019), explanations are selective, the receiving
users tend to only take a minor subset of the entire set of explanations,
predominantly those that agree with their prior belief. This can sometimes
cause the affected receiver to lose sight of the bigger picture and arrive at
some misinterpreted conclusion. (Kaur et al., 2020) notes this as a mismatch
between the solution’s conceptual purpose and the receiver’s mental model. XAI
tools that produce a feature ranking figure may overcloud the users with
excessive information, thereby increasing their cognitive load and rendering
the tool counterproductive. It is also observed that the amount of trust is
correlated with the level of appreciation the receiver has in the explanation
(Mohseni et al., 2021a). Take for example the case of a rejected loan
application, an under-appreciated explanation would just result in the
applicant resubmitting the application to a different bank, without addressing
the underlying root cause. Humans also tend to prefer contrastive explanations
as opposed to visualizing a large number of probable causes, thus designing
the explanation to be counterfactual can reduce under-appreciation and
rejection of XAI tools. Future research on human-centric explanations can look
to draw inspiration from social sciences and the study of human psychology to
bridge the gap between the two ends of the explanation chain.

###  8.3. Explanation evaluation

It is evident from Table LABEL:tab:_XAI_CE, LABEL:tab:_XAI_FP,
LABEL:tab:_XAI_FM, only a small subset of reviewed works attempt to provide
some form of quantitative measurement of the proposed XAI technique. An even
smaller number performs a comparison between multiple XAI techniques, possibly
due to model incompatibility and differences between the explanatory structure
of individual XAI techniques. (Gurumoorthy et al., 2019) uses a variety of
evaluation approaches, grounded on both statistical and human knowledge and
involves experts and non-technical users, while admitting the limitation of
ambiguity and inconsistency in human judgment. In the case of surrogate model
explanations, the fidelity of the surrogate model can be used to measure the
accuracy of the approximation. However, such an approach cannot be used for
feature relevance tools like SHAP (Amparore et al., 2021). A common basis for
instilling interpretability in financial solutions goes beyond the social
responsibilities of the provider firms but also concerns the need to comply
with the rules and regulations laid out. Even so, the mismatch between each
party’s perception of explanation sufficiency extends beyond the model itself
and includes commonly neglected variables such as accountable personnel,
feedback process, and personnel training procedure (Kuiper et al., 2022).
(Hoffman et al., 2018) highlights that explanations can be seen as a dynamic
interaction between the conveyed message and the receiver’s thought process.
The effectiveness can be measured via goodness and satisfaction in the form of
feedback upon receiving the message. This further exemplifies the fact that
what makes an explanation good is largely subjective and coming to a consensus
on a suitable set of metrics is no trivial task. Among the set of reviewed
works in this paper, there exist two forms of evaluation, either through
statistical approaches (F1-score, accuracy, and t-test) or opinions of a human
expert. The latter is defined as plausibility and should be made distinct from
faithfulness, which reflects how the AI model reasons about its behavior.
(Jacovi and Goldberg, 2020) states that the assessment of faithfulness should
be independent of human judgment and a common ground can be established by
evaluating XAI techniques with respect to a pre-defined set of goals, rather
than on the basis of achieving universal satisfaction. One criticized flaw in
existing works on evaluating interpretation methods pertains to the
distributional shift between training and test sets, as well as the infeasible
requirement of retraining models. (Turbé et al., 2023) creates a set of
synthetic datasets with known discriminative features and additionally
develops two new metrics which account for identifying top relevant time steps
in terms of ranking and score. The proposed method takes into consideration
the temporal elements in time-series analysis. We further note that it is
imperative for future works on XAI evaluation criteria to precisely define the
objective and target audience of the explanation.

###  8.4. Trade-off between performance and interpretability

It is often common to ponder “shouldn’t we deploy more transparent models if
no interpretability enhancement work is required?”, such an initiative is
often plagued by the limited representativeness of transparent models. The
trade-off between performance and interpretability is quite commonly a major
cause of the dilemma in selecting between black-box models and inherently
transparent models. Though there exist studies that have shown that black-box
models performing more complex operations do not necessarily lead to better
performance (Rudin, 2019; Rudin and Radin, 2019), it is often the case for
unstructured information and noisy environments such as the financial markets.
XAI tools explaining through a surrogate model have to face the burden of
ensuring both the fidelity of the surrogate model and the effectiveness of the
underlying AI model, all the while matching the required goals toward the
receiving audience. In light of such a challenge, it highlights the necessity
for a consensus metric to serve as a quantitative assessment. In general, it
happens more often than not that the selected model at hand is more complex
than required, resulting in additional explainability engineering. Moving
forward, an efficient way of handling the trade-off is to prioritize the usage
of transparent models if the obtained performance is satisfactory and progress
to a more complex model when necessary.

###  8.5. Better transparent models

We note that transparent models refer to models which exhibit inherent
transparency without the need to apply post-hoc explainability techniques.
However, caution should be taken in the assumption of such model (Jacovi and
Goldberg, 2020). The inherent transparency is dependent on the achievable
explanation goals and the explanation receiver, while there is much doubt
surrounding truly inherently transparent models (Serrano and Smith, 2019).
Nevertheless, there are numerous studies advocating for a greater need in
adopting transparent models. A study by (Lipton, 2018) argues that transparent
models are essential for promoting fairness in machine learning, as they allow
for easier identification and mitigation of biases in the decision-making
process. A team of researchers (Rudin and Radin, 2019) participated in an
explainable machine learning challenge and concluded that transparent models
do not only sidestep the common issues of trust and misinterpretation but also
exhibit the potential to match complex models in terms of performance on
specific tasks. It can therefore be for the well-being of society that
researchers prioritize the development of more sophisticated and robust
transparent machine learning models that are able to balance the trade-off
between model accuracy and interpretability.

###  8.6. Human-centric XAI

We note that, in order to effectively support human decision-making and ensure
the interpretability of AI models, there is a growing need for human-centric
XAI tools that prioritize user understandability and usability. Explanations
are interactive and should be viewed as a bidirectional form of communication
(Kaur et al., 2020), with the XAI tool explaining to the user and the user
reciprocating back for clarity. Incorporating Human-Computer Interaction (HCI)
principles into the design of XAI systems is beneficial in translating
interpretability, as HCI principles embrace user-friendly interfaces that
enhance human comprehension and engagement. One important aspect of this is
the development of interactive systems that enable users to actively engage
with and explore XAI models, allowing them to gain a deeper understanding of
how the models work and how they can be understood. Several studies have
highlighted the importance of incorporating HCI principles into the
development of interactive XAI systems (Hohman et al., 2019). The results show
that users had more trust when presented with virtual interactive explanations
(Weitz et al., 2019). Some popular examples of interactive XAI toolkits
include Microsoft AI widgets (mic, 2021) and What-if tool by Google (Wexler et
al., 2019). Besides being an easily approachable and interpretable tool,
interactive systems improve system usability and entice users to frequent the
financial services provided, thus adding to the benefits of financial firms
prioritizing the development of human-centric, interactive XAI tools.

###  8.7. Multimodal XAI

A less discussed avenue for improvement is in the incorporation of multimodal
information, particularly natural language. Among the works reviewed, a large
subset of processed input data only involves numerical features, while the
inclusion of textual information remains a minority. An underlying reason
might be due to the redundancy of using textual data or the lack of
substantial increase in performance while requiring additional pre-processing
works due to the inclusion of such information. Nevertheless, there exist
benefits from a transparency point-of-view in incorporating textual
information. (Danilevsky et al., 2020) highlighted that the separation of the
underlying AI model from the explainability tool is less distinct since NLP
models, particularly through the use of attention can produce both the
prediction and explanation. Likewise, NLP-type explanations can be attractive
for the layperson since it exhibits a natural feel which makes the whole
process interactive and efficient (Cambria et al., 2023). Ultimately, the
incorporation of multimodal information entails more flexibility in crafting a
good explanation and is supported by the abundance of textual information
available. We believe the inclusion of NLP in XAI presents an exciting
opportunity to enhance our understanding of financial models and further
promote better transparency and trustworthiness in today’s AI models.

##  9\. Conclusion

Overall, explainability will continue to be a critical area of focus in
FinTech as companies seek to build trust and confidence with consumers and
regulators alike. To conclude our work, we have provided a comprehensive
review of XAI tools in the financial domain (FinXAI), highlighting the
significant progress made in recent years toward developing explainable AI
models for financial applications. This includes both inherently transparent
models and post-hoc explainability techniques, the former of which we advocate
for more improvements to be made. We provided a framework that establishes the
selection of appropriate FinXAI tools as a sequential decision-making process,
placing great emphasis on the audience and iterative assessment of produced
explanation. The reviewed works are categorized according to their respective
characteristics for ease of access by interested readers. We also examine the
contributions of current FinXAI to several ethical goals, e.g.,
trustworthiness, fairness, informativeness, accessibility, privacy,
confidence, causality, and transparency.

Though there have been many great works done thus far, the review also reveals
some limitations and challenges associated with FinXAI. This includes
appropriate metrics to measure both the faithfulness and plausibility of
explanations, as well as issues concerning the over-reliance on potentially
misleading explanations. Future research should focus on addressing these
challenges, as well as exploring new directions for FinXAI, including
integrating NLP into explanation-generating techniques and a greater focus on
inherently transparent models. Nevertheless, there is great potential for XAI
techniques to enhance transparency, trust, and accountability in the financial
domain. This underscores the importance of active research and development in
this field.

## References

  * (1)
  * mic (2021) 2021\.  _Microsoft Responsible AI Toolbox_.  Retrieved February 23, 2023 from https://github.com/microsoft/responsible-ai-toolbox
  * Achituve et al. (2019) Idan Achituve, Sarit Kraus, and Jacob Goldberger. 2019\.  Interpretable online banking fraud detection based on hierarchical attention mechanism. In _2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP)_. IEEE, 1–6.
  * Adams and Hagras (2020) Janet Adams and Hani Hagras. 2020.  A type-2 fuzzy logic approach to explainable AI for regulatory compliance, fair customer outcomes and market stability in the global financial sector. In _2020 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)_. IEEE, 1–8.
  * Agarwal et al. (2021) Rishabh Agarwal, Levi Melnick, Nicholas Frosst, Xuezhou Zhang, Ben Lengerich, Rich Caruana, and Geoffrey E Hinton. 2021.  Neural additive models: Interpretable machine learning with neural nets.  _Advances in Neural Information Processing Systems_ 34 (2021), 4699–4711.
  * Amparore et al. (2021) Elvio Amparore, Alan Perotti, and Paolo Bajardi. 2021\.  To trust or not to trust an explanation: using LEAF to evaluate local linear XAI methods.  _PeerJ Computer Science_ 7 (2021), e479.
  * Apley and Zhu (2016) Daniel W Apley and Jingyu Zhu. 2016.  Visualizing the effects of predictor variables in black box supervised learning models.  _arXiv preprint arXiv:1612.08468_ (2016).
  * Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. 2020\.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information fusion_ 58 (2020), 82–115.
  * Babaei et al. (2022) Golnoosh Babaei, Paolo Giudici, and Emanuela Raffinetti. 2022\.  Explainable artificial intelligence for crypto asset allocation.  _Finance Research Letters_ 47 (2022), 102941.
  * Bach et al. (2015) Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  _PloS one_ 10, 7 (2015), e0130140.
  * Bahrammirzaee (2010) Arash Bahrammirzaee. 2010\.  A comparative survey of artificial intelligence applications in finance: artificial neural networks, expert system and hybrid intelligent systems.  _Neural Computing and Applications_ 19, 8 (2010), 1165–1195.
  * Bandi et al. (2021) Harit Bandi, Suyash Joshi, Siddhant Bhagat, and Dayanand Ambawade. 2021.  Integrated Technical and Sentiment Analysis Tool for Market Index Movement Prediction, comprehensible using XAI. In _2021 International Conference on Communication information and Computing Technology (ICCICT)_. IEEE, 1–8.
  * Benhamou et al. (2021) Eric Benhamou, Jean-Jacques Ohana, David Saltiel, and Beatrice Guez. 2021.  Explainable AI (XAI) models applied to planning in financial markets.  (2021).
  * Biecek et al. (2021) Przemysław Biecek, Marcin Chlebus, Janusz Gajda, Alicja Gosiewska, Anna Kozak, Dominik Ogonowski, Jakub Sztachelski, and Piotr Wojewnik. 2021.  Enabling machine learning algorithms for credit scoring–explainable artificial intelligence (XAI) methods for clear understanding complex predictive models.  _arXiv preprint arXiv:2104.06735_ (2021).
  * Bracke et al. (2019) Philippe Bracke, Anupam Datta, Carsten Jung, and Shayak Sen. 2019\.  Machine learning explainability in finance: an application to default risk analysis.  (2019).
  * Bueff et al. (2022) Andreas C Bueff, Mateusz Cytryński, Raffaella Calabrese, Matthew Jones, John Roberts, Jonathon Moore, and Iain Brown. 2022\.  Machine learning interpretability for a stress scenario generation in credit scoring based on counterfactuals.  _Expert Systems with Applications_ 202 (2022), 117271.
  * Bussmann et al. (2020) Niklas Bussmann, Paolo Giudici, Dimitri Marinelli, and Jochen Papenbrock. 2020.  Explainable AI in fintech risk management.  _Frontiers in Artificial Intelligence_ 3 (2020), 26.
  * Bussmann et al. (2021) Niklas Bussmann, Paolo Giudici, Dimitri Marinelli, and Jochen Papenbrock. 2021.  Explainable machine learning in credit risk management.  _Computational Economics_ 57 (2021), 203–216.
  * Cambria et al. (2022) Erik Cambria, Qian Liu, Sergio Decherchi, Frank Xing, and Kenneth Kwok. 2022.  SenticNet 7: A commonsense-based neurosymbolic AI framework for explainable sentiment analysis. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_. 3829–3839.
  * Cambria et al. (2023) Erik Cambria, Lorenzo Malandri, Fabio Mercorio, Mario Mezzanzanica, and Navid Nobani. 2023\.  A survey on XAI and natural language explanations.  _Information Processing & Management_ 60, 1 (2023), 103111\.
  * Carta et al. (2022) Salvatore Carta, Alessandro Sebastian Podda, Diego Reforgiato Recupero, and Maria Madalina Stanciu. 2022\.  Explainable AI for financial forecasting. In _Machine Learning, Optimization, and Data Science: 7th International Conference, LOD 2021, Grasmere, UK, October 4–8, 2021, Revised Selected Papers, Part II_. Springer, 51–69.
  * Carta et al. (2021) Salvatore M Carta, Sergio Consoli, Luca Piras, Alessandro Sebastian Podda, and Diego Reforgiato Recupero. 2021\.  Explainable machine learning exploiting news and domain-specific lexicon for stock market forecasting.  _IEEE Access_ 9 (2021), 30193–30205.
  * Chen and Ye (2022) Dangxing Chen and Weicheng Ye. 2022.  Generalized Gloves of Neural Additive Models: Pursuing transparent and accurate machine learning models in finance.  _arXiv preprint arXiv:2209.10082_ (2022).
  * Chen and Storchan (2021) Jiahao Chen and Victor Storchan. 2021.  Seven challenges for harmonizing explainability requirements.  _arXiv preprint arXiv:2108.05390_ (2021).
  * Chen et al. (2020) Jun-Hao Chen, Samuel Yen-Chi Chen, Yun-Cheng Tsai, and Chih-Shiang Shur. 2020.  Explainable deep convolutional candlestick learner.  _arXiv preprint arXiv:2001.02767_ (2020).
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_. 785–794.
  * Cho and Shin (2023) Soo Hyun Cho and Kyung-shik Shin. 2023.  Feature-Weighted Counterfactual-Based Explanation for Bankruptcy Prediction.  _Expert Systems with Applications_ 216 (2023), 119390.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Collaris et al. (2018) Dennis Collaris, Leo M Vink, and Jarke J van Wijk. 2018\.  Instance-level explanations for fraud detection: A case study.  _arXiv preprint arXiv:1806.07129_ (2018).
  * Cong et al. (2021) Lin William Cong, Ke Tang, Jingyuan Wang, and Yang Zhang. 2021\.  AlphaPortfolio: Direct construction through deep reinforcement learning and interpretable AI.  _Available at SSRN_ 3554486 (2021).
  * Crosato et al. (2021) Lisa Crosato, Caterina Liberati, and Marco Repetto. 2021\.  Look Who’s Talking: Interpretable Machine Learning for Assessing Italian SMEs Credit Default.  _arXiv preprint arXiv:2108.13914_ (2021).
  * Danilevsky et al. (2020) Marina Danilevsky, Kun Qian, Ranit Aharonov, Yannis Katsis, Ban Kawas, and Prithviraj Sen. 2020\.  A survey of the state of explainable AI for natural language processing.  _arXiv preprint arXiv:2010.00711_ (2020).
  * Datta et al. (2016) Anupam Datta, Shayak Sen, and Yair Zick. 2016.  Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In _2016 IEEE symposium on security and privacy (SP)_. IEEE, 598–617.
  * Davis et al. (2022) Randall Davis, Andrew W Lo, Sudhanshu Mishra, Arash Nourian, Manish Singh, Nicholas Wu, and Ruixun Zhang. 2022.  Explainable machine learning models of consumer credit risk.  _Available at SSRN_ (2022).
  * De Bruin et al. (2009) Kelly C De Bruin, Rob B Dellink, and Richard SJ Tol. 2009\.  AD-DICE: an implementation of adaptation in the DICE model.  _Climatic Change_ 95 (2009), 63–81.
  * Dellinger (2019) AJ Dellinger. 2019\.  _Understanding The First American Financial Data Leak: How Did It Happen And What Does It Mean?_ Retrieved February 22, 2023 from https://www.forbes.com/sites/ajdellinger/2019/05/26/understanding-the-first-american-financial-data-leak-how-did-it-happen-and-what-does-it-mean/?sh=7716df86567f
  * Demajo et al. (2020) Lara Marie Demajo, Vince Vella, and Alexiei Dingli. 2020\.  Explainable AI for Interpretable Credit Scoring. In _CS & IT Conference Proceedings_, Vol. 10. CS & IT Conference Proceedings.
  * Deng et al. (2019) Shumin Deng, Ningyu Zhang, Wen Zhang, Jiaoyan Chen, Jeff Z Pan, and Huajun Chen. 2019\.  Knowledge-driven stock trend prediction and explanation via temporal convolutional network. In _Companion Proceedings of The 2019 World Wide Web Conference_. 678–685.
  * Dikmen and Burns (2022) Murat Dikmen and Catherine Burns. 2022.  The effects of domain knowledge on trust in explainable AI and task performance: A case of peer-to-peer lending.  _International Journal of Human-Computer Studies_ 162 (2022), 102792\.
  * Dumitrescu et al. (2022) Elena Dumitrescu, Sullivan Hué, Christophe Hurlin, and Sessi Tokpavi. 2022.  Machine learning for credit scoring: Improving logistic regression with non-linear decision-tree effects.  _European Journal of Operational Research_ 297, 3 (2022), 1178–1192.
  * Farzad (2019) Taghi Farzad. 2019\.  Determinants of Mortgage Loan Delinquency: Application of Interpretable Machine Learning.
  * Fior et al. (2022) Jacopo Fior, Luca Cagliero, and Paolo Garza. 2022\.  Leveraging Explainable AI to Support Cryptocurrency Investors.  _Future Internet_ 14, 9 (2022), 251.
  * Fritz-Morgenthal et al. (2022) Sebastian Fritz-Morgenthal, Bernhard Hein, and Jochen Papenbrock. 2022\.  Financial risk management and explainable, trustworthy, responsible AI.  _Frontiers in Artificial Intelligence_ 5 (2022), 5.