# How should AI decisions be explained?

Requirements for Explanations from the Perspective of European Law

Benjamin Fresz  Fraunhofer IPA and University of Stuttgart IFFNobelstrasse
12StuttgartGermany70569<EMAIL_ADDRESS>0009-0002-7463-8907 ,
Elena Dubovitskaya  University of GiessenLicher Straße 72GiessenGermany35394
<EMAIL_ADDRESS>0000-0002-0491-3670 ,  Danilo Brajovic
Fraunhofer IPA and University of Stuttgart IFFNobelstrasse
12StuttgartGermany70569<EMAIL_ADDRESS>0000-0002-3949-6870
,  Marco Huber  Fraunhofer IPA and University of Stuttgart IFFNobelstrasse
12StuttgartGermany70569<EMAIL_ADDRESS>0000-0002-8250-2092 and
Christian Horz  University of GiessenLicher Straße 72GiessenGermany35394
<EMAIL_ADDRESS>
###### Abstract.

This paper investigates the relationship between law and eXplainable
Artificial Intelligence (XAI). While there is much discussion about the AI
Act, for which the trilogue of the European Parliament, Council and Commission
recently concluded, other areas of law seem underexplored. This paper focuses
on European (and in part German) law, although with international concepts and
regulations such as fiduciary plausibility checks, the General Data Protection
Regulation (GDPR), and product safety and liability. Based on XAI-taxonomies,
requirements for XAI-methods are derived from each of the legal bases,
resulting in the conclusion that each legal basis requires different XAI
properties and that the current state of the art does not fulfill these to
full satisfaction, especially regarding the correctness (sometimes called
fidelity) and confidence estimates of XAI-methods.

Law, EU, European Law, German Law, eXplainable Artificial Intelligence (XAI),
XAI, Interpretable ML, norms, standards, regulation, safe ML

††copyright: none††ccs: Social and professional topics Governmental
regulations††ccs: Social and professional topics Management of computing and
information systems††ccs: Computing methodologies Machine learning††ccs:
Computing methodologies Artificial intelligence

##  1\. Introduction

The explainability of AI systems is crucial for their development and
deployment in sensitive domains. Therefore, the AI Act draft of the European
Union (EU) mandates that high-risk AI systems ensure “sufficient transparency”
for users to interpret and utilize the system’s results appropriately.
However, a challenge lies in the vagueness of these transparency requirements
and what exactly constitutes “sufficient transparency”. While more details on
this “sufficient transparency” will hopefully be specified in standards such
as the ISO/IEC CD TS 6254, other legal bases for the use of eXplainable
Artificial Intelligence (XAI) already exist. This paper reviews the legal
requirements for explainability of AI systems based on European (and in part
German) law, focusing on three legal domains and four application areas.
Therefore, our contributions can be summarized as follows:

  * •

We present an extended list of XAI properties which can be used to formulate
requirements.

  * •

We describe legal bases for the use of XAI and derive requirements such
methods need to fulfill in each of the considered areas of law.

  * •

We point out open research problems and a possible way forward for the field
of XAI by embracing interdisciplinarity.

For this, we first provide a brief overview of common XAI terminology in the
next section and establish the choice of areas of law. After the description
of related works in Chapter 2, the XAI properties used in this paper are
described in Chapter 3, with the distinction of method-specific properties and
process-specific properties. In Chapter 4, each of the considered areas of law
is introduced, accompanied by the formulation of the task for XAI in this
area, from which the XAI requirements are derived. The paper is closed with a
discussion in Chapter 5 and a summary in Chapter 6. The paper structure and
requirements for XAI are also replicated in short in Table 1 in Chapter 6.

###  1.1. Taxonomy

We start by summarizing the taxonomy of explainability methods for AI briefly.
For a full introduction we point the readers to (Burkart and Huber, 2021;
Molnar, 2022). On the highest level, explanations can be divided into local
and global ones. Local explanations explain a particular decision (e.g., the
loan was rejected due to a low income) while global explanations reveal how
decisions are made in general (e.g., every applicant with income above 100k is
given a loan). They are introduced in more detail below. It needs to be noted
that most XAI-methods provide non-causal explanations, although special causal
methods exist.

####  1.1.1. Local Explanations

Feature attributions are widely used to provide local explanations by
quantifying the contribution of specific features to a decision. For example,
in the loan granting scenario, feature attribution could reveal that a short
length of employment had the highest negative impact on the rejection, while
an income of 75k was the strongest positive feature. Methods like SHapley
Additive exPlanations (SHAP) (Lundberg and Lee, 2017) and Linear Interpretable
Model-Agnostic Explanations (LIME) (Ribeiro et al., 2016) are commonly used
for computing feature attributions. Saliency maps, on the other hand,
highlight important regions in images that contribute to specific decisions.
They can be created using methods like SHAP or LIME, although usually methods
specifically for neural networks—such as Grad-CAM (Selvaraju et al., 2020)—are
used. Counterfactual explanations are a type of local explanation that provide
insight into decision-making without revealing the detailed functioning of the
AI model (Wachter et al., 2018). These explanations show a change in input
features necessary to alter the outcome. For instance, a counterfactual
explanation for the loan example could be “If you had worked at least five
years, you would have been granted the loan.” Data attribution or sample-based
explanations involve identifying influential samples from the training set
that contributed to a decision. In the loan example, this could be interpreted
as “Your loan was rejected because similar applicants from the training data
defaulted on a loan,” potentially accompanied by such similar applicants.

####  1.1.2. Global Explanations

Global explanations provide information about the workings of a model across
its entire data range. The simplest form of global explainability is the usage
of so-called whitebox models such as decision trees. These models provide out-
of-the-box insights into the decision making process and, hence, do not need
any specific post-hoc explainability—given that they are small enough in size
to be understandable by humans. Providing global explanations for black-box
models is more difficult. Apart from partial dependence plots (PDPs),
surrogate models provide a way of opening the black box to some degree. For
this, either a black-box model is transformed into a white-box one (Nguyen et
al., 2020) or a white-box (surrogate) model is fitted on the output of the
black-box model that shall be explained, thus showing how similar decisions
could be made. As a further way to explain neural networks, especially in
computer-vision tasks, concept-based explanations can be used. They usually
provide an account of which concepts, e.g., certain patterns like stripes or
dots, are especially relevant for the classification of single images (local
concept-based explanations) or for the class or model under consideration in
general (global concept-based explanations) (Kim et al., 2018). Other methods
produce global feature attribution explanations (Covert et al., 2020),
highlighting features with high predictive power.

###  1.2. Considered Laws

This paper focuses on EU and German laws, although with a wide span of
possible stakeholders and reasons for the use of XAI (see Chapter 4). While
there is uncertainty about what XAI can achieve, the considered areas of
law—company law, the General Data Protection Regulation (GDPR), product safety
and liability for defect products—are among the most discussed and show the
different legal questions XAI could possibly answer. Additionally, these areas
bear similarity across many different jurisdictions, as the ISION-rule (as
used in Chapter 4.1) serves as a specification of fiduciary duties, which are
applicable to the directors and officers of nearly every corporation
worldwide. Moreover, the ISION-rule is closely connected to § 141(e) of the
Delaware General Corporation Law (Title 8, Chapter 1 of the Delaware Code) in
the US, addressing the reliance on expert opinions by board directors. The
GDPR (Chapter 4.2) was introduced as a de-facto international standard,
coining the term “Brussels-Effect” for EU laws that are adhered to even
outside the EU. Product safety and liability (Chapter 4.3) are harmonized
within the EU. This paper does not intend to provide an exhaustive list of
legal bases for the use of XAI, but aims to provide a starting point towards
such a list and shows how laws can inform the development of XAI techniques.

##  2\. Related Work

To date, not many works have concerned themselves with the link between law
and XAI, apart from works around the AI Act and the GDPR. In the following,
the related works and the understanding of transparency in law are summarized.
A broader discussion around the need for algorithmic transparency was started
by the provisions of the GDPR, especially about whether a “right to
explanation” exists (Wachter et al., 2017) and how much information would be
necessary to fulfill such a right. Wachter et al. (2018) devised
counterfactual explanations to give users actionable advise while keeping the
exact workings of an Machine Learning (ML) model a secret (see Section 1.1.1).
Vorras and Mitrou (2021) argue for a “right to explanation” with the
implication of a “qualified transparency mechanism”, i.e., adapting
explanations to the target audience and “considering the technical limitations
of AI interpretability, the underlying legal and regulatory barriers, as well
as the legitimate expectations of stakeholders”. A similar sentiment can be
found in (Błażej Kuźniacki, Marco Almada, Kamil Tyliński, Łukasz Górski, Beata
Winogradska, Reza Zeldenrust, 2022), where Kuzniacki et al. argue for the
construction of a global regulatory framework for the use of XAI in tax law.
They additionally recommend that “tax lawyers and computer scientists should
work together to help reach a consensus on legal and technical solutions that
secure XAI’s place in tax law”. While our work is not based on tax law, it
follows a similar idea as it aims to combine the information from legal
experts and computer scientists to make progress towards a structured use of
XAI for legal purposes. As of now, a gap between the legal and technical
understanding can be discerned in many works, as echoed by Gyevnar et al.
(2023), who argue that this difference in understanding can also be seen in
the EU AI Act. They note that there seem to be multiple understandings of
transparency and that the transparency mandated in the EU AI Act is different
from purely technical transparency, which is the aim of most works in XAI.
Also concerned with the different interpretations of transparency in the AI
Act are Madalina Busuioc, Deirdre Curtin, Marco Almada (2022). They stress
that the explainability of AI systems, i.e., the presentation of
specific—mediated—details of their workings, is different from the disclosure
of the model itself, where only the second option allows for the unbiased
scrutiny of such systems. Such explanations and thus, mediated information,
could be used maliciously, as shown by Zhou and Joachims (2023). Summarizing,
despite some works in that direction, the use of XAI for legal purposes is
still unclear.

##  3\. Properties of XAI-Methods (possibly) relevant for their legal use

To be able to formulate requirements for XAI systems, a taxonomy of possibly
necessary properties of such systems needs to be defined. In this work, we
primarily rely on the 12 Co-properties introduced by Nauta et al. (2023),
which were derived from scientific literature published until the end of 2020.
This comprehensive set of properties is described below. Additional to these
XAI-method specific properties, further ones are formulated in Section 3.2 for
the process in which XAI is used. A short overview of the used properties can
be found in Table 2 in the appendix.

###  3.1. 12 Co-Properties of Nauta

_Correctness_ describes whether—or to what extent—an explanation conforms to
the ML model the explanation was created for. While this seems like a very
basic feature of any explanation method, this property is not necessarily
fulfilled, for example for sample-based methods like LIME and SHAP.
_Completeness_ addresses how much of the explanandum—either a model itself or
a prediction—is captured within an explanation. The extreme cases would be to
either present the entire ML model with its mathematical operations or to just
highlight one input feature as the main cause of a specific outcome. White-box
models are by definition complete explanations of themselves. _Consistency_
demands that identical inputs provide identical explanations, as some
explanation methods are not deterministic in their behavior—especially sample-
based methods. Inconsistent explanations might also result from the
hyperparameter choices of specific methods (Tritscher et al., 2023).
_Continuity_ poses some similarity to Consistency, as it demands that small
changes in the input or output of the model should also only result in small
changes in the explanation. This can be illustrated for saliency maps for
images: If an image is overlaid with noise, but the prediction of the model
does not change, the explanation should also only change slightly.
_Contrastivity_ describes the discriminativeness of an explanation, as
explanations are supposed to show why a decision was made in contrast to a
different outcome that was not chosen, as argued by Miller (2019). _Covariate
Complexity_ shows how complex features and their interactions in an
explanation are. The simplest way to go about this would be to use the input
features and ignore all feature interactions (and thus possibly lose out on
Correctness), whereas for example simplified feature interactions such as
linear or monotonous interactions could be one step more complex.
_Compactness_ denotes the overall size of an explanation, as too large
explanations often overwhelm users, whereas too small explanations provide
less information about a decision or the model behavior (Hoffman et al.,
2023). _Compositionality_ describes how an explanation is presented,
especially its format, organization, and structure. Different ways of
presenting explanations can improve the way the explanation is received, for
example by providing higher-level information (Alvarez-Melis and Jaakkola,
2018) or different terminology (Swartout and Moore, 1993). _Confidence_
estimates can be used in two different ways to assist in the use of XAI—either
to show the confidence of the model in the original prediction as part of the
explanation or in denoting the confidence that a given explanation is correct
and thus, conforms to the decision/model at hand. _Context_ of an explanation,
especially of who the explainee is, could help to adapt an explanation to
specific user needs, thus improving its practical use. One of the earlier
mentions of context can be found in (Miller et al., 2017) which even described
the development of XAI methods only conforming to the needs of AI engineers
and scientists as “inmates running the asylum.” _Coherence_ addresses whether
an explanation conforms to prior knowledge—to that of an explainee, general
consensus or other beliefs—thus making it easier to understand. While humans
often use their prior knowledge to evaluate whether they deem an explanations
as trustworthy or correct, Coherence is distinctly different from Correctness
as it is described above, as an explanation can be correct (i.e., conform to
the logic of a model) without conforming to prior beliefs. _Controllability_
describes how and to what extent an explanation can be interacted with. Also
called interactivity, it seems as this can improve the understanding of an
explanation (Cheng et al., 2019) as well and also helps to address Context, as
users could adapt the explanation to their needs as they see fit. It should be
noted that there is currently no consensus on the best way to evaluate the
properties discussed, as different evaluation methods (with many listed in
(Nauta et al., 2023)) may contradict each other (Tomsett et al., 2019).
Therefore, this paper will not attempt to formulate specific requirements for
validating XAI systems, such as through specific technical means or user
studies.

###  3.2. Process Properties

The aforementioned properties are mainly formulated for XAI-methods
themselves, while for legal purposes, requirements could also be formulated
for properties of the implementation or of the process of creating
explanations. These are discussed in the following.

Since most current XAI methods do not guarantee Correctness, using multiple
methods can help, e.g., to spot errors in an application. Given their
_Consilience_ , this can also be an indication of Correctness. However, if
explanations do not align, there is no established way to resolve such
situations. The _Computations_ in real-time systems should also be considered.
In time-sensitive scenarios, explanations need to be provided from the start,
while in other cases, longer runtimes (e.g., hours of runtime on a GPU-server)
may be feasible. The _Coverage_ also needs to be considered, potentially being
single decisions (local), entire models (global) or the data. Local
explanations could be helpful to base a decision on, whereas data attributions
might be more helpful where the traceability of errors is relevant. While,
strictly speaking, not part of the process of creating and showing
explanations to users, it could also be a legal requirement to implement a way
for end-users to object to decisions made by a model or to object to
explanations which are conceived as false by the concerned user
(_Counterability_ of decisions/explanations). A further requirement can be
_Constancy_ , as it might be necessary to refer to a documented decision and
its explanation to avoid liability. In addition to just documenting an XAI-
explanation, in cases of ambiguous explanations (e.g., when multiple saliency
maps are used to try to find general rules about how a computer-vison model
decides) it might be necessary to not just document the explanation itself but
accompany it by the interpretation of the person who used it to make a
decision.

##  4\. Legal Bases for the Use of XAI

When formulating legal requirements for the explainability of decisions made
by ML-based applications, a stakeholder-based approach appears promising. This
approach focuses on groups of people with a legitimate interest in XAI.
Depending on the situation, different groups of people come into
consideration, whereby four groups can be identified:

  1. (1)

Persons who make a decision based on the recommendations/predictions of an ML
model,

  2. (2)

Persons whose data is processed by an ML application,

  3. (3)

Persons who are responsible for the errors of an ML application and

  4. (4)

Persons who want to assert claims against individuals responsible for the
errors.

Group 2 stakeholders will have an interest in questioning the
recommendations/prediction of the ML model, especially if it is
disadvantageous to them. The interests of Group 3 stakeholders are similar to
those of Group 1; here, too, liability should be avoided, but not for their
own decision, but for errors in the ML application. Finally, Group 4
stakeholders seek to determine if the provider of the ML application can be
held accountable for any errors. These definitions can be mapped to the
definition of stakeholders for whom explanations are created described by
Tomsett et al. (2018) as data-subjects and decision-subjects (group 1),
operators and executors (group 2) and creators and examiners of ML systems
(group 3 and 4). In the following, the legal situation of the respective
stakeholders and the corresponding XAI-requirements are described.

###  4.1. ML Prediction as the Basis for a Human Decision: Plausibility Check

People sometimes make decisions based on ML recommendations/predictions. This
is a challenge for various areas of law, such as criminal law (ML-supported
sentencing), labor law (ML-supported personnel recruitment and management),
administrative law, insurance law and others. The issues relating to company
law are outlined below. However, it can be assumed that the legal problems in
connection with ML-supported decision-making are comparable in all areas of
law.

Company law deals with corporate fiduciaries (directors and officers of a
company) who use ML applications when making decisions. From the legal point
of view, the fiduciaries remain solely responsible for the decision made. If
the decision is disadvantageous for the company, the fiduciaries are obliged
to compensate the company for any resulting damages (fiduciary duty claims). A
fiduciary can only avoid this liability if they prove that they did not breach
any fiduciary duties (duties of care and loyalty) when basing a decision on
the ML recommendation/prediction. Ways to achieve this are intensely discussed
in legal literature. Many scholars consider even the use of black box ML
applications by fiduciaries to be unproblematic (Petrin, 2019) while others
argue that the lack of explainability leads the fiduciary to make an unfounded
decision, thereby breaching the fiduciary duty of care (Cowger Jr., 2023). In
Germany, the fiduciary duty of care has been specified by the Federal Supreme
Court (BGH) in the so-called ISION judgement (BGH, 2011). The “ISION-Rule“
applies when the fiduciary seeks external advice due to a lack of in-house
expertise. It has three requirements: (1) the expert consulted must be an
independent professional who is qualified for the issue to be clarified, (2)
the fiduciary must provide the expert with a comprehensive presentation of the
company’s circumstances and disclose all necessary documents, (3) the
fiduciary must subject the expert’s advice to a careful plausibility check. A
breach of these requirements constitutes a breach of the corporate duty of
care. Comparable regulations exist in various jurisdictions; for example,
Section 141(e) of the Delaware General Corporation Law affords corporate
directors a defense against liability only when the expert is chosen “with
reasonable care”, and the director “reasonably believes” that the matter falls
within the expert’s competence, relying in “good faith” on the expert’s
advice.

In this context, the last requirement—the plausibility check—is particularly
important. There is a consensus in the legal discourse that the above
requirements should be applied to the use of an ML application (Buchholz and
Dubovitskaya, 2023). However, it is disputed how the plausibility check should
be carried out. Most authors seem to recognize that, at least in the case of
black box models, sufficient traceability is not given and a plausibility
check is therefore de facto impossible. For this reason, attempts are made to
replace the plausibility check with less stringent requirements, such as
requiring board members to familiarize themselves with the basic functions of
AI applications and to regularly monitor the operation of the applications
(Möslein, 2018). However, it remains unclear how such checks are to be carried
out without the traceability of an AI decision. The requirements would
therefore be reduced to the duty of fiduciaries to equip themselves with basic
technical knowledge of ML applications. This does not enable effective and
sufficient control of algorithmic decisions. Instead, decision-making
authority shifts from human to machine. This remains true even if one resorts
to the standard repertoire of organizational obligations, such as the duty to
document decisions, appoint a Chief Information Officer (CIO), establish
internal company rules for dealing with AI (e.g., through a special compliance
brochure), and provide training for managers and other staff.

In order to be able to use ML models for fiduciary decisions, a plausibility
check of their decisions needs to be possible and could be facilitated through
XAI. To enable such a plausibility check, XAI systems need to provide certain
properties, as discussed below. Note that properties, for which no relevant
requirements are seen for this case, are left out.

Correctness is crucial in the use of XAI as explanations that do not align
with the model’s decision do not offer a meaningful way to assess the
plausibility of an ML decision. While explanations without Correctness may
seem plausible, they can potentially increase automation bias (Vered et al.,
2023) or hinder proper scrutiny of the decision system (Zhou and Joachims,
2023). To evaluate a decision accurately, all factors influencing the decision
should be highlighted. To be able to reject a decision based on sensitive
attributes like gender, a combination of Correctness and Completeness is
necessary. Since most current XAI methods provide no guarantee for Correctness
(except for limited axiomatic properties in SHAP), Consilience of the
explanations of different methods could be used as a substitute. It is
important to note that even when multiple XAI methods agree, Correctness of
explanations cannot be guaranteed, just as human experts may make decisions
based on reasons beyond their explanations. In the case of human expert
advice, the fiduciary assesses the plausibility based on their understanding
of the expert’s explanations. Applied to ML, this implies two criteria:
explanations must be understandable to laypersons and to domain experts. This
also requires Consistency, as an inconsistent explanation method can create
confusion for non-experts who cannot resolve differing explanations and
hyperparameter choices. Contrastivity is a necessary precondition to enable a
user to discern between the prediction given by the model and relevant
alternatives for action. For Covariate Complexity, the same as for Correctness
can be stated: Ideally, the explanations should consider features and feature
interactions as complex as in the original model, while still being
understandable for laypersons. But given that simplifications are made
explicit and the person tasked with the plausibility check still considers an
explanation plausible, these requirements can be loosened. As described in
Section 3.1, insufficient Compactness can render explanations unintelligible.
On the other hand, explanations should be as complete as possible to highlight
all relevant features. To resolve this conflict, each user should be shown the
most complex explanation this user can still grasp, requiring an adaptation of
the explanation to the Context. One way to give an XAI system this
adaptability would be to make it controllable, allowing each user to adapt the
explanation to their needs, e.g., by limiting the number of features
highlighted in an explanation. Controllability can resolve such conflicts and
additionally increase the amount of time a user spends with an explanation,
increasing the explanation’s effectiveness (Cheng et al., 2019). Albeit
Coherence to prior knowledge is not a requirement for explanations for
plausibility checks, it will be a main part of the plausibility checks
themselves, as laypersons and domain experts do not have other meaningful ways
to assess ML decisions and explanations than comparing them to their prior
knowledge. As the “ISION-Rule“ states, the expert consulted must be an
independent professional qualified for the issue to be clarified. When
extended to ML systems, the qualification of such systems for a general use
case can be assessed via evaluation metrics such as accuracy, but the
qualification for a specific data point with unknown label (as in real-world
decisions) is more difficult to estimate. One main possibility to assess
whether a model is qualified to make a decision for a specific scenario is the
quantification of the Confidence of the model. Thus, a Confidence estimate of
the model prediction should accompany the explanation for a specific decision.
As explanations are not guaranteed to be correct, a Confidence estimate of the
explanation itself is necessary to assess whether it can be relied upon. To
facilitate the fiduciary’s defense in potential lawsuits, explanations should
be presented in a way that allows for permanent storage and thus, Constancy.
In the case of automatically generated ambiguous explanations, e.g., saliency
maps, fiduciaries are advised to document their interpretation to prevent
later disputes. Since the plausibility of a single decision needs to be
assessed, local explanations should be sufficient. To improve the
understanding of the limits of a ML system, it could be helpful to “calibrate
one’s trust” via continued interaction with a model, requiring more than one
local explanations (Ma et al., 2023). Legally, XAI could enable a
comprehensive review of AI decisions, surpassing the level of the ISION
principles. In legal discussions, there is debate about whether transparency
requirements for algorithmic decisions should be no higher than those for
human decisions. However, equal treatment of humans and machines is not
required by law. On the contrary: if it is technically feasible to provide
better explanations for algorithmic decisions compared to human decisions,
there is no reason not to utilize this greater efficiency and consequently
take it as a legal prerequisite.

Summarizing, when extending the ISION-principles to non-human explanations,
they do not limit the type of explanation that can be used to assess the
plausibility of a decision. While data attributions—showing similar decisions
that happened in the past—could also be helpful here, it can be argued that
such explanations are not specific enough to the decision at hand and require
too much interpretation themselves. Thus, the simplest way to aid in a
plausibility assessment is to use explanations such as feature importance with
axiomatic properties for Correctness and Completeness such as SHAP, although
the actual value of such properties is part of an ongoing discussion (Kumar et
al., 2020; Merrick and Taly, 2019; Sundararajan and Najmi, 2020). As the
requirement for Correctness is the most difficult one to fulfill, using
multiple explanation methods (and thus, doing a “plausibility check” of the
explanation itself) currently seems to be the best way to meet the legal
requirements.

###  4.2. ML Prediction for the Assessment of Human Characteristics: Right to
Explanation?

People are also increasingly becoming the recipients of algorithmic decisions,
for example when ML is used to decide whether someone receives a loan or is
invited to a job interview. As a result, decisions are being made on issues
that are of central importance to social life. Especially if the decision is
negative, the person concerned has an interest in finding out why. From a
legal perspective, the question arises as to whether the person concerned has
a right to an explanation of the decision and from which legal basis such a
right can be derived. Of course, this raises the question of the extent of the
legal guarantee, i.e., what must be disclosed/explained to the data subject.

It is being discussed whether the “right to an explanation” of algorithmic
decisions can be derived from the provisions of the GDPR, specifically Article
15(1)(h) and Article 22(3) (Wachter et al., 2017; Wachter and Mittelstadt,
2018). In our legal opinion, this is indeed correct. Article 15(1)(h) grants
the data subject the right to be informed of the existence of automated
decision-making, including profiling, and to receive meaningful information
about the logic, significance, and consequences of such processing. This right
is further supported by Article 22(3), which allows the data subject to
contest AI decisions. In the SCHUFA judgment, the Court of Justice of the
European Union (CJEU) emphasized the need to interpret EU law not only of its
wording but also of its context, objectives, and the purpose pursued by the
act of which it forms a part (CJEU, 2023). Given the GDPR’s goal of enabling
effective exercise of data subjects’ rights, the “right to an explanation” is
already inherent in the GDPR. The Advocate General at the CJEU Priit Pikamäe,
in his opinion on the SCHUFA case, shares a similar sentiment. He considers
that “the obligation to provide ’meaningful information about the logic
involved’ must be understood to include sufficiently detailed explanations of
the method used to calculate the score and the reasons for a certain result.
In general, the controller should provide the data subject with general
information, notably on factors taken into account for the decision-making
process and on their respective weight on an aggregate level, which is also
useful for him or her to challenge any ‘decision’ within the meaning of
Article 22(1) of the GDPR.”(Advocate General Pikamäe, 2023) Besides the GDPR,
the right to explanation of algorithmic decisions may find grounding in
fundamental rights. At the EU level, considerations of human dignity and the
right to the protection of personal data (Articles 1 and 8 of the EU Charter
of Fundamental Rights) come into play. Article 8(2) sentence 2 of the EU
Charter of Fundamental Rights affirms every person’s right to access data
collected about them and the right to have it rectified. However, it is
questionable whether this guarantee also includes the right to explanation of
decisions that are the result of ML-based data processing. The narrow
formulation of Article 8(2) sentence 2 may not explicitly encompass such
rights. Nevertheless, ML-based data processing, like any data processing,
constitutes an interference with the right to the protection of personal data,
necessitating justification. The proportionality test allows for weighing the
data subject’s interest in decision explanation against the ML application
operator’s interest in protecting trade secrets (Article 17(2) of the EU
Charter of Fundamental Rights, protection of intellectual property). The
principle of practical concordance mandates finding a balance that is as
gentle as possible, ensuring neither conflicting fundamental right takes
precedence over the other (akin to Pareto optimum in game theory). Thus, the
goal is to explain the ML prediction to the concerned individual without
revealing algorithmic components constituting trade secrets. However, the
challenge of reverse engineering, often permitted in some national laws (e.g.,
in the German Law on the Protection of Trade Secrets), must be addressed, e.g.
by new regulations.

The task for XAI regarding a “right to explanation” (independent of the legal
basis used) can be summarized as: Enable end-users of AI systems to object to
ML decisions and—as argued in (Wachter et al., 2018)—show ways to achieve a
different decision. As these tasks require local explanations as in Section
4.1 with mainly the same requirements—a plausibility check of the decision by
a layperson—only the differences are discussed here.

The GDPR grants users the right to object to a decision in Article 22(3) (here
called Counterability), thus from a practical standpoint, a process for
objecting automated decisions should be implemented. This could also entail
the option to object to explanations which are deemed unjust or incorrect,
especially since Correctness cannot be guaranteed. As much as a Confidence
estimate of a decision and its explanation could help users to assess whether
a model was certain in their case or whether it was more likely to be wrong,
it could also complicate the privacy of such systems by enabling membership
inference attacks (Shokri et al., 2016). To prevent issues such as this, other
measures could be taken, e.g., limiting the amount of explanation requests a
user can make. Similar problems arise for data influence explanations, as
sensitive training data (other than their own) should not be shown to end-
users. Compared to fiduciary decisions, explanations can be generated when
need arises, as it is not feasible (nor necessary) to compute explanations for
all possible users in advance. In our opinion, merely providing counterfactual
explanations, as recommended in (Wachter et al., 2018), may not be sufficient
to achieve the goal of enabling users to object to ML decisions and give them
the opportunity to alter the outcome. This viewpoint is underscored by the
statement of Advocate General Priit Pikamäe at the CJEU (Advocate General
Pikamäe, 2023). Because counterfactual explanations do only provide a very
limited account of how a decision was made (“This feature must change to
change the outcome, thus it is most likely important.”), further information
about the decision needs to be given, which could for example take the form of
feature importance explanations as in Section 4.1. Thus, to fulfill all aims
of the use of XAI regarding the ‘right to explanation’, several types of
explanations could therefore be jointly given to users.

###  4.3. ML Defects: Product Safety, Product Liability and Contractual
Liability for Defects

XAI-methods are often stated to help in debugging models or showing the
fairness of such systems (Deck et al., 2023; Langer et al., 2021), both
relevant aims in product safety, product liability and contractual liability
for defects. In the following, these areas and their relation to XAI are
described, followed by a discussion of properties XAI-methods need to fulfill
to truly aid in these cases.

####  4.3.1. Product Safety

Under certain conditions, developers of ML applications must comply with the
requirements of the Product Safety Law. The European Union’s regulations on
product safety are outlined in the General Product Safety Directive (GPSD) of
December 3, 2001, which is set to be superseded by the new General Product
Safety Regulation (GPSR). The GPSR came into effect on June 12, 2023, and its
application will commence on December 13, 2024. In Germany, product safety for
non-food products is governed by the German Product Safety Act (ProdSG). It
incorporates the regulations of European Product Safety Law and applies when
products are introduced to the market, distributed, or used for the first time
as part of a business activity (Section 1(1) of the Product Safety Act). A
product is a good, a substance or a mixture that has been manufactured through
a production process (Section 2 no. 21 of the Product Safety Act). According
to the prevailing view, a product requires a physical object, so that stand-
alone software is not a product within the meaning of this act. Thus, it only
becomes applicable to software when installed on data carriers or embedded in
a physical product. In this case, at least the general requirement applies
that the product must not endanger the safety and health of persons when used
as intended or in a foreseeable manner (Section 3(2) sentence 1 of the Product
Safety Act); if the product is subject to one or more special legal
regulations mentioned in Section 8 of the Product Safety Act, it must also
meet the requirements specified in these legal regulations. The person who
places the product on the national market is therefore obliged to ensure that
the product is sufficiently safe to exclude risks to the physical safety and
health of persons, with measures adapted to specific hazards. However,
absolute safety is not owed; rather, the manufacturer must (only) take the
measures that are objectively necessary and reasonable according to objective
standards in the circumstances of the specific case.

If an ML application is used in the manufacture of a product without becoming
part of this product (e.g., for quality control in product manufacturing
processes), the obligations under the Product Safety Act only apply to the
product manufacturer (supplier). A fault in the ML application can potentially
lead to a safety-relevant fault in the end product. In such a case, the
manufacturer will have an interest in recourse against the person who has
contractually provided them with the ML application. Even without a special
agreement, such a possibility of recourse will regularly arise from the
contractual obligation between the manufacturer and the supplier (Section
280(1) of the German Civil Code). Product manufacturers will likely seek to
negotiate contractual guarantees and liability clauses with their suppliers.
Then the supplier, who is the developer of the ML application, would have an
increased interest in ensuring that their ML application is free of defects
(given they are the manufacturer) or in agreeing contractual guarantees with
the supplier of the application.

####  4.3.2. Product Liability

Product liability in the European Union is currently governed by the (almost
40-year old) Directive 85/374/EEC on Liability for Defective Products,
implemented in Germany through the Product Liability Act (ProdHaftG). Civil
liability (liability for damages) under Section 1(1) of the Product Liability
Act applies if a product defect causes death, personal injury, or damage to
property other than the defective product. This requires that the product is
intended for private use or consumption and has been used accordingly. The
current interpretation of “products” includes movable objects and electricity
(Article 2 Product Liability Directive; Section 2 ProdHaftG), but it is
unclear if non-physical software is covered. German literature tends to
include software under product liability (as in the proposed new Product
Liability Directive).

Section 2(2) no. 2 of the Product Liability Act states that the manufacturer
must ensure that the product is free of defects when placed on the market.
According to Section 2(2) no. 5 of the Product Liability Act, the
manufacturer’s liability is excluded if the defect could not have been
recognized according to the state of the art in science and technology at the
time when the manufacturer placed the product on the market (so-called
development risk). However, the fact that the manufacturer of an ML
application cannot foresee its behavior in a specific hazardous situation does
not constitute a development risk. The blanket exoneration of manufacturers of
ML systems is therefore out of the question.

The proposed directive extends the requirements for freedom from defects:
According to Article 6(1)f, a product is also defective if it does not fulfill
the security-relevant cybersecurity requirements. At the same time, the
proposed directive specifies the requirements for the manufacturer. The
manufacturer cannot be held liable if they were unable to detect the defect
based on objective scientific and technical knowledge at the time of sale or
during their control of the product. However, the manufacturer must ensure
that both the software and products containing software are defect-free
throughout the duration of their control. During this period, the manufacturer
must also provide the updates and upgrades necessary to maintain safety
(Article 10(2) of the proposed directive). “Control of the manufacturer” means
the fact that the manufacturer of a product authorizes: a) the integration,
connection or supply of a component, including software updates or upgrades,
by a third party or b) the modification of the product (Article 4(5) of the
proposed directive).

In addition to the Product Liability Act, there is also tortious product
liability in Germany in accordance with Section 823(1) of the German Civil
Code. Here, the manufacturer is liable for damages to anyone whose legal
interests (in particular life, body, health and property) have been injured by
a defective product. This presupposes that the manufacturer has breached
marketing obligations (design, manufacturing, instruction, product monitoring,
warning and recall obligations). In the context of Section 823(1) of the
German Civil Code, it is recognized that the manufacturer is obliged to
constantly adapt the software to new knowledge about safety risks and to
exploit new technical possibilities. In contrast, the Product Liability Act
does not recognize obligations to monitor products, issue subsequent warnings
or recall defective products.

####  4.3.3. Liability for Defects (Seller’s Point of View)

The manufacturer or supplier of an ML application is liable to its contractual
partner (client, buyer, etc.) for defects in accordance with the provisions of
the German Civil Code on contractual liability for defects (warranty law). A
defect exists if the subject matter of the contract does not fulfill the
contractual requirements, considering the objective or average requirements in
the absence of specific contractual agreements. Therefore, liability extends
beyond damages to health or property, as in product liability law, and covers
any deviation from the contractually agreed quality. On the other hand,
liability for defects is narrower than product liability, as it applies only
to the contractual partner. For contractual liability, it does not matter
whether the ML application is integrated into a physical object or whether it
is stand-alone software.

Which provisions are relevant in the individual case depends on the respective
type of contract (purchase, contract for work, lease, etc.). In a purchase
agreement, for example, the seller’s liability for defects under Section 437
of the German Civil Code is applicable. In this case, the buyer is entitled to
special warranty rights such as the right to subsequent fulfillment,
withdrawal or reduction regardless of whether the supplier of the ML
application is responsible for its defectiveness. The buyer’s claims for
damages, however, require fault on the part of the seller. In the absence of
such special warranty provisions, liability arises from the general standard
of Section 280(1) of the German Civil Code, because the defect in the ML
application constitutes a breach of the obligations arising from the contract.
However, this general provision governs liability for damages and therefore
presupposes culpability (fault). The parties are free to agree on stricter or
more lenient liability terms. Special provisions apply if the supplier of the
ML application is an entrepreneur and its contractual partner is a consumer.
In this case, deviation from statutory provisions (Sections 327 et seq., 475b
et seq. of the German Civil Code) is only permitted under strict conditions
(Sections 327h, 327s, 476 of the German Civil Code).

The above three Sections 4.3.1 to 4.3.3 result in the same task for XAI: Find
and if possible rule out defects in the ML model. Different solutions can be
devised, ranging from global explainability of a model to rule out defects, to
using local explanations or data attributions to find defects, to possibly not
using XAI methods at all given sufficient testing and documentation. Ways in
which XAI can assist to find or rule out defects are covered in the following,
for a way to ensure safety by documentation see (Brajovic et al., 2023). In
all of these cases, explanations can be expected to be generated for expert
users.

To ensure freedom of defects, Correctness is at the heart of XAI use, as a
global explanation method that does not conform to the model might miss out on
defects. To date, no such global explanation method seems to be known and
computationally feasible, as highlighted in the closely related field of
formal verification of neural networks (Albarghouthi, 2021). The only way to
provide an explanation with such properties is to use
explainable/interpretable models, as demanded by Rudin (2018) for safety
critical use cases. To spot defects within black-box AI applications, local
explanations can help, where Correctness-problems could once again be
mitigated by using the Consilience of multiple XAI-methods, although this
would still not give providers of such system enough information to guarantee
freedom of defects. Since XAI in the near future seems more likely to be able
to show the presence of defects than to provide correct global explanation
methods on which a guarantee for the freedom of defects could be built, the
first case will be considered for the following properties.

As long as the explanations do show relevant defects, there is no need for
Complete explanations, as for instance even a very specific method which only
highlights whether a protected attribute is used in a decision could help to
spot unwanted behavior. The same can be argued for Compactness, as not entire
explanations need to be understood but they can be searched for parts where
defects are suspected. In practice, it could prove difficult to choose the
(number of) features for such explanations. In the search for defects,
inconsistent explanation methods can be used, as running them multiple times
seems feasible to potentially spot defects. Additionally, they do not
necessarily need to provide Continuity, unless a global understanding of a
model should be gained, which is only possible if explanations are continuous
and can thus be generalized to near data points. Since defects, for which
liability could become relevant, e.g., discriminating decisions, can also
result from the interaction of input features for instance via confounding
variables, used XAI-methods should provide the maximum understandable amount
of Covariate Complexity. Additional to using XAI, confounders could be
discovered via methods as described by Rogozhnikov et al. (2022) to ensure
that no sensitive attributes are reconstructed based on other input features
(Wachter and Mittelstadt, 2018). A Confidence estimate could itself show
unwanted behavior in a model, e.g., a model being uncertain about a safety-
critical decision which is expected to arise during its deployment. Because of
this, model Confidence should be considered. Since the Controllability of an
explanation improves the user’s understanding of the underlying model and
thus, the chance of finding relevant defects, explanations should be
interactive. To avoid liability claims and prove a freedom of defects (or the
safety of the ML system), the relevant explanation artifacts and the way in
which they were created need to provide Constancy.

Overall, there is no clear consensus on how XAI can help to spot or rule out
defects in ML applications. While a human intelligible and fully correct
global explanation seems ideal, it also seems unrealistic for some use cases
as of now—partly due to the currently available ML and XAI methods, but also
due to the complexity of some use cases. The closest to this ideal are
interpretable models, for which guarantees can be derived, given that they
reason on human-intelligible features. If no such model can be devised for a
certain use case, the best way forward seems to be a joint combination of
documentation and a variety of explanation methods for the model itself, its
explanations and the data used.

####  4.3.4. Liability for Defects (Buyer’s Point of View)

The buyer who receives a defective product and intends to take action against
the seller under Section 437 of the German Civil Code must prove that the
product was defective at the time of the transfer of risk, generally at the
time of delivery. Such a defect may be presumed only if the buyer is a
consumer (Section 477 of the German Civil Code) Furthermore, the buyer may
have an interest in determining whether the seller is responsible for the
defect, for instance, if the defect could have been discovered during
production. This information could be necessary when the buyer asserts claims
for damages.

This tasks XAI with showing the existence of a defect and finding the cause of
it, as this way the liability of the provider of an AI system (or even the
data provider) can be proven. This can be expected to be done by AI experts.
To find (reasons for) defects, all parts of an AI application can be
inspected—as in the use case above with local and global explanations. A
special focus for the liability from the buyer’s point of view lies on data
attribution, as for example local explanations can show the presence of
erroneous behavior, but not its cause, which is expected to originate in the
training data used or in the development of the model itself. While the second
case should be checked via documentation as proposed in (Brajovic et al.,
2023), the first one—attributing certain behavior to training data samples—can
be solved via data attribution.

To make sure that only true reasons for a defect are shown and thus, no false
liability claims are made, Correctness of explanations in combination with
Completeness is necessary to not miss out on causes of defects. Contrastivity
is necessary to be able to discern between different causes for defects,
otherwise the users of such systems might be burdened with the interpretation
of which possible cause for a defect was relevant for specific data points,
rendering the use of XAI pointless. As in the previous use cases, Covariate
Complexity needs to be considered in the explanations, as the interactions of
features can themselves highlight relevant causes for defects. Since
Confidence estimates can point towards problems such as unbalanced
datasets—which can lead to defects themselves—the model Confidence should be
considered for the ML model. Like in the previous cases, Controllability of
explanations helps to better understand the ML model and its explanations, and
thus, should be utilized.

Overall, the current state of the art of XAI is only partly suitable for
identifying the causes of defects, as most current explanations provide rather
superficial answers and require user interpretation to find the cause for
certain behavior. This can easily be described for saliency explanations:
While they can show whether or not something has been detected in an image,
they do not provide any information about why this has happened. The most
promising direction towards finding defect causes via XAI seem to be data
attribution explanations, where defects can be traced down to (possibly
erroneous) data samples. A further way to investigate such defects could be
causal attribution mechanisms in addition to XAI.

##  5\. Discussion

While the EU AI Act is seen as a breakthrough in AI regulation, it is
important to note that it is not expected to supersede all other relevant laws
influencing the use of AI (and thus, XAI). The laws described in Chapter 4,
from which the requirements were derived, mainly pertain to civil liability
and will coexist with the AI Act, which falls under public law. Although these
laws interact (unfulfilled requirements of the AI Act might be seen as defects
in the sense Sections 4.3.3 and 4.3.4), any further discussion on their
possible interactions is beyond the scope of this paper.

There has been much debate about the perceived conflict between regulation and
innovation in the context of the AI Act. However, it is important to note that
regulation itself does not stifle innovation; rather, unclear regulation or
instructions may impede progress. To avoid such ambiguity in the field of XAI,
this paper proposes a structured approach based on specific properties to
guide XAI use and development. Similarly, international or national norms
could recommend XAI properties or specific methods for specific use cases,
providing clear instructions on which methods to employ under different
circumstances. These norms should consider additional legal foundations for
XAI, such as criminal law, administrative law, and insurance law.

To promote advancements in the intersection of law and XAI, technical and
legal experts should actively exchange information. It is crucial to bridge
the gap between the two fields, as their current understanding of XAI differs
significantly. While legal experts often emphasize the need for transparency
to ensure safe AI usage and address liability issues, the current technical
possibilities do not fully support such a broad statement. Additionally, the
terminology used in both fields may not align, leading to communication
challenges (Schneeberger et al., 2023). While the collaboration and
integration of both fields are important, it is worth noting that the law
cannot fully dictate the use of technology. In the case of XAI, other
disciplines such as social sciences (examining the effectiveness of
explanations) and user interface design (determining how explanations should
be presented for desired effects) can contribute to filling gaps in legal
requirements. As the law provides minimum requirements, insights from these
fields become more crucial for properties not touched by legal requirements
(such as Compositionality and others) and when surpassing these minimum
standards.

##  6\. Summary

In this paper, the relationship between law and XAI was explored, deriving
requirements for different areas of law based on an extended taxonomy of XAI
properties. The areas of law considered were fiduciary decisions, GDPR, and
product safety and liability. These areas require different XAI properties and
methods. For fiduciary decisions, assessing plausibility is important, similar
to the GDPR, where users should be able to object to decisions and receive
actionable advice. In product safety, identifying defects in ML models is
crucial, while product liability focuses on demonstrating the causes of
defects. A tabular summary of the derived requirements and explanation
recommendations can be found in Table 1 as a guidance for working with XAI in
these areas.

Various potential aims for XAI have been proposed in the literature (see for
example Saeed and Omlin (2023)), but the practical utilization of XAI to
achieve these goals remains unclear. The lack of consensus regarding
evaluation metrics for the objectives and properties of XAI further
complicates the matter, as the reliable and valid assessment of XAI method
characteristics is a necessary precondition to meaningfully confirm or reject
specific methods for a given use case. Possible actions to address this could
involve developing new evaluation metrics or conducting studies to assess the
effectiveness of XAI in specific settings. However, while the legal practice
often requires the use of state of the art technologies, it appears that the
current state of XAI is not fully equipped to fulfill all legal requirements.
The most challenging XAI properties in this regard are Correctness
(explanations conforming to model logic), Confidence estimates of models and
explanations themselves, and the lack of global explanation methods,
indicating the need for further research in these areas. One ongoing study by
some of the authors focuses on the use of XAI in science and industry. It
remains to be seen if practitioners in these areas face similar challenges. To
inform future development, law can be used as an inspiration for XAI-methods
with specific properties or new ways to evaluate XAI. To this end, we urge
practitioners from both fields to collaborate, so that a common terminology
and a common understanding of the capabilities of XAI can be established.

The requirements for XAI presented in this paper serve as recommendations that
contribute to the ongoing discussion on the utilization und future development
of XAI. We eagerly anticipate the future advancements in this field, both from
a technical standpoint and within the legal and normative realms.

Table 1. Requirements for XAI derived from the different legal areas. Checkmarks denote that a property is necessary for that specific use case. (1) denotes that instead of Correctness, Consilience of explanations can be used. (2) denotes the Tradeoff between Completeness and Compactness, (3) between Completeness/Covariate Complexity and understandability, (4) between Confidence estimates and privacy. (5) denotes that multiple predictions could be used to ”calibrate one’s trust”, e.g. to learn when and when not to trust the ML model. Legal Basis |  Plausibility Check - Section 4.1 |  Right to Explanation - Section 4.2 |  Safety/Liability - Sections 4.3.1-4.3.3 |  Liability (Buyer) - Section 4.3.4
---|---|---|---|---
XAI-Task |  Check the ML-decision for plausibility |  Enable user to understand and object to decision |  Show the absence of errors (as far as possible) |  Show the cause of an error
Audience of Explanation |  Laypeople, domain experts |  Laypeople |  ML experts |  ML experts
Correctness |  $\checkmark(1)$ |  $\checkmark(1)$ |  $\checkmark(1)$ |  $\checkmark$
Completeness |  $\checkmark(2)$ |  $\checkmark(2)$ |  |  $\checkmark$
Consistency |  $\checkmark$ |  $\checkmark$ |  |
Continuity |  |  |  |
Contrastivity |  $\checkmark$ |  $\checkmark$ |  |  $\checkmark$
Covariate Complexity |  $\checkmark(3)$ |  $\checkmark(3)$ |  $\checkmark$ |  $\checkmark$
Compactness |  $\checkmark(2)$ |  $\checkmark(2)$ |  |
Compositionality |  |  |  |
Confidence |  $\checkmark$ (for prediction and explanation) |  $\checkmark$ (for prediction and explanation) (4) |  $\checkmark$ (for prediction) |  $\checkmark$ (for prediction)
Context |  $\checkmark$ |  $\checkmark$ |  |
Coherence |  |  |  |
Controllability |  $\checkmark$ |  $\checkmark$ |  $\checkmark$ |  $\checkmark$
Consilience |  $\checkmark(1)$ |  $\checkmark(1)$ |  $\checkmark(1)$ |
Computations |  |  |  |
Coverage |  Prediction (5), Data |  Prediction |  Model, Prediction, Data |  Model, Prediction, Data
Counterability |  |  $\checkmark$ |  |
Constancy |  $\checkmark$ |  |  $\checkmark$ |  $\checkmark$
Recommendation for Explanation Type |  Feature Importance (or similar) |  Counterfactuals and Feature Importance (or similar) |  Global/interpretable models |  Data attribution

###### Acknowledgements.

Parts of this paper were created with the help of a data-centric
implementation of ChatGPT (3.5 turbo). It was used to create LaTeX-code for
tables and figures and to refine the drafts of some sections. This paper is
funded in parts by the German Federal Ministry for Economic Affairs and
Climate Action under grant no. 19A21040B (project ”veoPipe”).

## References

  * (1)
  * Advocate General Pikamäe (2023) Advocate General Pikamäe. 16.03.2023.  SCHUFA Holding (Scoring).  Opinion, C‑634/21, ECLI EU:C:2023:220.
  * Albarghouthi (2021) Aws Albarghouthi. 2021.  Introduction to Neural Network Verification.  http://arxiv.org/pdf/2109.10317.pdf
  * Alvarez-Melis and Jaakkola (2018) David Alvarez-Melis and Tommi S. Jaakkola. 2018.  Towards Robust Interpretability with Self-Explaining Neural Networks.  http://arxiv.org/pdf/1806.07538.pdf
  * BGH (2011) BGH. 20.09.2011.  ISION.  Judgement, II ZR 234/09. ZIP 2011, 2097.
  * Błażej Kuźniacki, Marco Almada, Kamil Tyliński, Łukasz Górski, Beata Winogradska, Reza Zeldenrust (2022) Błażej Kuźniacki, Marco Almada, Kamil Tyliński, Łukasz Górski, Beata Winogradska, Reza Zeldenrust. 2022.  Towards eXplainable Artificial Intelligence (XAI) in Tax Law: The Need for a Minimum Legal Standard.
  * Brajovic et al. (2023) Danilo Brajovic, Niclas Renner, Vincent Philipp Goebels, Philipp Wagner, Benjamin Fresz, Martin Biller, Mara Klaeb, Janika Kutz, Jens Neuhuettler, and Marco F. Huber. 2023.  Model Reporting for Certifiable AI: A Proposal from Merging EU Regulation into AI Development.  http://arxiv.org/pdf/2307.11525.pdf
  * Buchholz and Dubovitskaya (2023) Annika Buchholz and Elena Dubovitskaya. 2023.  Die Geschäftsleitung und der Rat des Algorithmus.  _ZIP_ (2023), 63–73.
  * Burkart and Huber (2021) Nadia Burkart and Marco F. Huber. 2021.  A Survey on the Explainability of Supervised Machine Learning.  _Journal of Artificial Intelligence Research_ 70 (2021), 245–317.  https://doi.org/10.1613/jair.1.12228
  * Cheng et al. (2019) Hao Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, Max Harper, and Haiyi Zhu. 2019.  Explaining decision-making algorithms through UI: Strategies to help non-expert stakeholders. In _CHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(Conference on Human Factors in Computing Systems - Proceedings)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3290605.3300789
  * CJEU (2023) CJEU. 07.12.2023.  SCHUFA Holding (Scoring).  Judgement, C‑634/21, ECLI EU:C:2023:957.
  * Covert et al. (2020) Ian Covert, Scott Lundberg, and Su-In Lee. 2020.  Understanding Global Feature Contributions With Additive Importance Measures.  http://arxiv.org/pdf/2004.00668.pdf
  * Cowger Jr. (2023) Alfred R. Cowger Jr. 2022–2023.  Corporate Fiduciary Duty in the Age of Algorithms.  _Case Western Reserve Journal of Law, Technology & the Internet_ 14 (2022–2023), 136–207.
  * Deck et al. (2023) Luca Deck, Jakob Schoeffer, Maria De-Arteaga, and Niklas Kühl. 2023.  A Critical Survey on Fairness Benefits of XAI.  https://arxiv.org/pdf/2310.13007.pdf
  * Gyevnar et al. (2023) Balint Gyevnar, Nick Ferguson, and Burkhard Schafer. 2023.  Bridging the Transparency Gap: What Can Explainable AI Learn From the AI Act?  https://arxiv.org/pdf/2302.10766.pdf
  * Hoffman et al. (2023) Robert R. Hoffman, Shane T. Mueller, Gary Klein, and Jordan Litman. 2023.  Measures for explainable AI: Explanation goodness, user satisfaction, mental models, curiosity, trust, and human-AI performance.  _Frontiers in Computer Science_ 5 (2023), 1096257.  https://doi.org/10.3389/fcomp.2023.1096257
  * Kim et al. (2018) Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. 2018.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In _Proceedings of the 35th International Conference on Machine Learning_ (July 10-15, 2018) _(Proceedings of Machine Learning Research, Vol. 80)_ , Jennifer G. Dy and Andreas Krause (Eds.). PMLR, Stockholmsmässan, Stockholm, Sweden, 2673–2682.  http://proceedings.mlr.press/v80/kim18d.html
  * Kumar et al. (2020) I. Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle Friedler. 2020.  Problems with Shapley-value-based explanations as feature importance measures.  https://arxiv.org/pdf/2002.11097.pdf
  * Langer et al. (2021) Markus Langer, Daniel Oster, Timo Speith, Holger Hermanns, Lena Kästner, Eva Schmidt, Andreas Sesing, and Kevin Baum. 2021.  What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research.  , 103473 pages.  https://doi.org/10.1016/j.artint.2021.103473
  * Lundberg and Lee (2017) Scott M. Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ (Long Beach, California, USA) _(NIPS’17)_. Curran Associates Inc., Red Hook, NY, USA, 4768–4777.
  * Ma et al. (2023) Shuai Ma, Ying Lei, Xinru Wang, Chengbo Zheng, Chuhan Shi, Ming Yin, and Xiaojuan Ma. 2023.  Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 759, 19 pages.  https://doi.org/10.1145/3544548.3581058
  * Madalina Busuioc, Deirdre Curtin, Marco Almada (2022) Madalina Busuioc, Deirdre Curtin, Marco Almada. 2022.  Reclaiming transparency: contesting the logics of secrecy within the AI Act.  _European Law Open_ 2 (12 2022), 1–27.  https://doi.org/10.1017/elo.2022.47
  * Merrick and Taly (2019) Luke Merrick and Ankur Taly. 2019.  The Explanation Game: Explaining Machine Learning Models with Cooperative Game Theory.  https://arxiv.org/pdf/1909.08128.pdf
  * Miller (2019) Tim Miller. 2019.  Explanation in artificial intelligence: Insights from the social sciences.  _Artificial Intelligence_ 267 (2019), 1–38.  https://doi.org/10.1016/j.artint.2018.07.007
  * Miller et al. (2017) Tim Miller, Piers Howe, and Liz Sonenberg. 2017.  Explainable AI: Beware of Inmates Running the Asylum Or: How I Learnt to Stop Worrying and Love the Social and Behavioural Sciences.  http://arxiv.org/pdf/1712.00547v2
  * Molnar (2022) Christoph Molnar. 2022.  _Interpretable Machine Learning_ (2 ed.).  https://christophm.github.io/interpretable-ml-book
  * Möslein (2018) Florian Möslein. 2018.  Digitalisierung im Gesellschaftsrecht: Unternehmensleitung durch Algorithmen und künstliche Intelligenz?  _ZIP_ (2018), 204–212.
  * Nauta et al. (2023) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. 2023.  From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI.  _ACM Comput. Surv._ 55, 13s, Article 295 (jul 2023), 42 pages.  https://doi.org/10.1145/3583558
  * Nguyen et al. (2020) Duy T. Nguyen, Kathryn E. Kasmarik, and Hussein A. Abbass. 2020.  Towards Interpretable ANNs: An Exact Transformation to Multi-Class Multivariate Decision Trees.  https://doi.org/10.48550/arXiv.2003.04675
  * Petrin (2019) Martin Petrin. 2019.  Corporate Management in the Age of AI.  _Columbia Business Law Review_ 3 (2019), 965–1030.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  "Why Should I Trust You?". In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , Balaji Krishnapuram, Mohak Shah, Alex Smola, Charu Aggarwal, Dou Shen, and Rajeev Rastogi (Eds.). ACM, New York, NY, USA, 1135–1144.  https://doi.org/10.1145/2939672.2939778
  * Rogozhnikov et al. (2022) Alex Rogozhnikov, Pavan Ramkumar, Rishi Bedi, Saul Kato, and G. Sean Escola. 2022.  Hierarchical confounder discovery in the experiment-machine learning cycle.  _Patterns_ 3, 4 (2022), 100451.  https://doi.org/10.1016/j.patter.2022.100451
  * Rudin (2018) Cynthia Rudin. 2018.  Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead.  https://arxiv.org/pdf/1811.10154.pdf
  * Saeed and Omlin (2023) Waddah Saeed and Christian Omlin. 2023.  Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities.  _Knowledge-Based Systems_ 263 (2023), 110273.  https://doi.org/10.1016/j.knosys.2023.110273
  * Schneeberger et al. (2023) David Schneeberger, Richard Röttger, Federico Cabitza, Andrea Campagner, Markus Plass, Heimo Müller, and Andreas Holzinger. 2023.  The Tower of Babel in Explainable Artificial Intelligence (XAI). In _Machine Learning and Knowledge Extraction_ _(Lecture Notes in Computer Science)_ , Andreas Holzinger, Peter Kieseberg, Federico Cabitza, Andrea Campagner, A. Min Tjoa, and Edgar Weippl (Eds.). Springer Nature Switzerland and Imprint Springer, Cham, 65–81.  https://doi.org/10.1007/978-3-031-40837-3{_}5
  * Selvaraju et al. (2020) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2020.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  , 336–359 pages.  https://doi.org/10.1007/s11263-019-01228-7
  * Shokri et al. (2016) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2016.  Membership Inference Attacks against Machine Learning Models.  http://arxiv.org/pdf/1610.05820.pdf
  * Sundararajan and Najmi (2020) Mukund Sundararajan and Amir Najmi. 2020.  The Many Shapley Values for Model Explanation. In _Proceedings of the 37th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 119)_ , Hal Daumé III and Aarti Singh (Eds.). JMLR.org, 9269–9278.  https://proceedings.mlr.press/v119/sundararajan20b.html
  * Swartout and Moore (1993) William R. Swartout and Johanna D. Moore. 1993.  Second Generation Expert Systems.  https://doi.org/10.1007/978-3-642-77927-5
  * Tomsett et al. (2018) Richard Tomsett, Dave Braines, Dan Harborne, Alun Preece, and Supriyo Chakraborty. 2018.  Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems.  http://arxiv.org/pdf/1806.07552.pdf
  * Tomsett et al. (2019) Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. 2019.  Sanity Checks for Saliency Metrics.  https://arxiv.org/pdf/1912.01451
  * Tritscher et al. (2023) Julian Tritscher, Maximilian Wolf, Andreas Hotho, and Daniel Schlör. 2023.  Evaluating Feature Relevance XAI in Network Intrusion Detection. In _Explainable Artificial Intelligence_ _(Communications in Computer and Information Science)_ , Luca Longo (Ed.). Springer Nature Switzerland and Imprint Springer, Cham, 483–497.  https://doi.org/10.1007/978-3-031-44064-9{_}25
  * Vered et al. (2023) Mor Vered, Tali Livni, Piers Douglas Lionel Howe, Tim Miller, and Liz Sonenberg. 2023.  The effects of explanations on automation bias.  _Artificial Intelligence_ 322 (2023), 103952.  https://doi.org/10.1016/j.artint.2023.103952
  * Vorras and Mitrou (2021) Apostolos Vorras and Lilian Mitrou. 2021.  Unboxing the Black Box of Artificial Intelligence: Algorithmic Transparency and/or a Right to Functional Explainability.  In _EU Internet Law in the Digital Single Market_ , Tatiana-Eleni Synodinou, Philippe Jougleux, Christiana Markou, and Thalia Prastitou-Merdi (Eds.). Springer International Publishing, Cham, 247–264.  https://doi.org/10.1007/978-3-030-69583-5_10
  * Wachter et al. (2018) Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018.  Counterfactual Explanations Without Opening the Black Box: Automated Decisions and the GDPR.  _Harvard Journal of Law & Technology_ 31 (04 2018), 841–887.  https://doi.org/10.2139/ssrn.3063289
  * Wachter and Mittelstadt (2018) Sandra Wachter and Brent Daniel Mittelstadt. 2018.  A Right to Reasonable Inferences: Re-Thinking Data Protection Law in the Age of Big Data and AI.  _Columbia Business Law Review_ 2019 (2018), 494–620–494–620.  https://api.semanticscholar.org/CorpusID:226950761
  * Wachter et al. (2017) Sandra Wachter, Brent Daniel Mittelstadt, and Luciano Floridi. 2017.  Why a Right to Explanation of Automated Decision-Making Does Not Exist in the General Data Protection Regulation.  _International Data Privacy Law_ 7 (2017), 76–99.  https://doi.org/10.2139/SSRN.2903469
  * Zhou and Joachims (2023) Joyce Zhou and Thorsten Joachims. 2023.  How to Explain and Justify Almost Any Decision: Potential Pitfalls for Accountability in AI Decision-Making. In _2023 ACM Conference on Fairness, Accountability, and Transparency_. ACM, New York, NY, USA, 12–21.  https://doi.org/10.1145/3593013.3593972

##  Appendix A Summary of used XAI Properties

A short description of all used XAI properties is provided in Table 2. A more
detailed description can be found in Chapter 3.

Table 2. 12 Co-Properties of Nauta et al. (2023) and additional Process-Properties (table adopted from (Nauta et al., 2023)). Property |  Definition
---|---
Correctness |  Describes how faithful the explanation is w.r.t. the black box.
Completeness |  Describes how much of the black box behavior is described in the explanation.
Consistency |  Describes how deterministic and implementation-invariant the explanation method is.
Continuity |  Describes how continuous and generalizable the explanation function is.
Contrastivity |  Describes how discriminative the explanation is w.r.t. other events or targets.
Covariate Complexity |  Describes how complex the (interactions of) features in the explanation are.
Compactness |  Describes the size of the explanation.
Compositionality |  Describes the format and organization of the explanation.
Confidence |  Describes the presence and accuracy of probability information in the explanation.
Context |  Describes how relevant the explanation is to the user and their needs.
Coherence |  Describes how accordant the explanation is with prior knowledge and beliefs.
Controllability |  Describes how interactive or controllable an explanation is for a user.
Consilience |  Describes whether more than one XAI-method should be used to fulfill the legal requirements.
Computations |  Describes when an explanation needs to be available, e.g. right when a prediction is shown or at a later point of time.
Coverage |  Describes what should be explained, a single prediction (local), an entire model (global) or the influence of training data samples (for predictions or the overall model behavior).
Counterability |  Describes whether a process needs to be implemented which allows end-users to object to AI decisions or explanations.
Constancy |  Describes whether explanations need to be presented in a format that can be saved for later examination.
  *[XAI]: eXplainable Artificial Intelligence
  *[SHAP]: SHapley Additive exPlanations
  *[LIME]: Linear Interpretable Model-Agnostic Explanations
  *[GDPR]: General Data Protection Regulation
  *[ML]: Machine Learning
  *[GPSD]: General Product Safety Directive
  *[GPSR]: General Product Safety Regulation
  *[ProdSG]: German Product Safety Act
  *[ProdHaftG]: Product Liability Act