11institutetext: Siemens AG, Munich/Nuremberg, Germany 11email:
<EMAIL_ADDRESS>22institutetext: Technical University of
Munich, Munich, Germany 22email<EMAIL_ADDRESS>
# AI Hazard Management: A framework for the systematic management of root
causes for AI risks

Ronald Schnitzer 1122 Andreas Hapfelmeier 11 Sven Gaube 11 Sonja Zillner 1122

###### Abstract

Recent advancements in the field of Artificial Intelligence (AI) establish the
basis to address challenging tasks. However, with the integration of AI, new
risks arise. Therefore, to benefit from its advantages, it is essential to
adequately handle the risks associated with AI. Existing risk management
processes in related fields, such as software systems, need to sufficiently
consider the specifics of AI. A key challenge is to systematically and
transparently identify and address AI risks’ root causes - also called AI
hazards. This paper introduces the AI Hazard Management (AIHM) framework,
which provides a structured process to systematically identify, assess, and
treat AI hazards. The proposed process is conducted in parallel with the
development to ensure that any AI hazard is captured at the earliest possible
stage of the AI system’s life cycle. In addition, to ensure the AI system’s
auditability, the proposed framework systematically documents evidence that
the potential impact of identified AI hazards could be reduced to a tolerable
level. The framework builds upon an AI hazard list from a comprehensive state-
of-the-art analysis. Also, we provide a taxonomy that supports the optimal
treatment of the identified AI hazards. Additionally, we illustrate how the
AIHM framework can increase the overall quality of a power grid AI use case by
systematically reducing the impact of identified hazards to an acceptable
level.

###### Keywords:

Trustworthy AI AI risk management AI development

##  1 Introduction

Recent advancements in the field of Artificial Intelligence (AI) establish the
basis for solving challenging tasks. However, despite many opportunities,
using AI-based solutions introduces new risks. For instance, in the field of
computer vision, it has been shown that minimal changes in the pixels of an
image, so small that a human eye cannot recognize it, can yield an AI
classification system to deliver inconsistent results [14]. In medical
diagnosis support, this characteristic of AI can have massively negative
consequences [1].

To fully benefit from the advantages of AI, it is important to handle the
risks associated with AI systems adequately. To address the problem of how to
manage AI risks, the particularities of AI systems need to be reflected. Well-
known AI-related issues include opaqueness, unpredictability, high dependence
on data, or vulnerability against security threats of AI systems. Such
pitfalls can negatively affect trustworthiness requirements, including the
safety, privacy, and fundamental rights of affected persons, and are,
therefore, vital factors on AI risks. Addressing these issues is required to
ensure AI systems are developed and operated in compliance with future
regulations, such as the Artificial Intelligence Act (AIA) - published by the
European Commission [7]. The proposed regulation requires a comprehensive risk
management process for all critical AI systems, including safety-critical
applications.

To manage AI-related risks, it is required to identify, assess, and treat
their corresponding root causes - further also called AI hazards - in a
systematic way. This is a challenging task because AI hazards can appear at
different stages of the AI system’s lifecycle, manifest at different
components of the AI system, and might require various actions. To illustrate
this complexity, consider the example of using deep neural networks (DNNs) in
autonomous driving to identify pedestrians. A specific AI risk in this domain
is that the AI system may not be able to detect all types of pedestrians with
equal reliability. The authors in [47] present evidence that standard object
detection models perform significantly better in identifying light-colored
pedestrians than dark-colored ones. The root causes for this AI risk can stem
from different components and stages of the AI system’s life cycle. A possible
AI hazard is biased training data, which itself can be introduced at the data
collection and the data preparation stage. Another possible AI hazard may lie
within the model itself, possibly introduced by wrong design decisions,
including the so-called loss function or performance evaluation metrics. In
this example, providing non-discriminative outputs is an essential objective
of the AI system. Hence this should be reflected in the mentioned design
choices. Examples of how to encode fairness in relevant design choices are
given in [5].

Furthermore, risk management in general needs to ensure completeness in terms
of addressed risk sources. Several approaches, such as failure mode and effect
analysis (FMEA) and preliminary hazard analysis (PHA), introduce valuable
mechanisms to analyze hazards in a systematic manner. While the FMEA
investigates potential failure modes of a system, the PHA relies on a so-
called preliminary hazard list, including potential, significant hazards
associated with a system’s design [38]. As an example of such a hazard list
covering a whole sector, the machinery directive provides a list of machinery-
related hazards in Annex I [10].

In this work, we aim to transfer this idea to the context of AI applications
by providing a wide-ranging list of potential AI hazards that may occur in any
scenario. This list serves as a basis for the identification of relevant AI
hazards in a specific AI application.

The contribution of this paper is the AI Hazard Management framework that
builds upon this AI hazard list. The AIHM establishes means to systematically
and identify, assess, and treat AI hazards. To ensure that any AI hazard is
handled at the earliest possible stage in the life cycle of an AI system, the
AIHM Framework provides a process executed in parallel to the development of
AI systems. In addition, to fulfill the required level of auditability, the
process systematically documents evidence that the impact of identified AI
hazards has been reduced to a tolerable level if possible. Otherwise, the
framework alerts the developer about unavoidable shortcomings of the AI
system. In addition, we show in this paper along a use case from the field of
power grids how the AIHM framework can increase the overall quality of an AI
system by systematically reducing the impact of all identified hazards to a
tolerable level. Note, that due to the dominance of data-driven AI systems
utilizing Machine Learning (ML) techniques, this is also the focus of the AIHM
framework.

The paper is structured as follows. In Section 2, we present an overview of
similar approaches toward the management of AI risks and their root causes,
followed by Section 3 providing detailed definitions and clarification of used
terms and concepts. In Section 4, the state-of-the-art regarding AI hazards is
comprehensively analyzed, yielding a preliminary AI hazard list together with
a taxonomy of AI hazards. Building upon this, we present the AIHM and its
corresponding process in Section 5. In Section 6, we illustrate the
framework’s effectiveness along an industrial use case from the field of power
grids. We conclude the paper’s results and show possible future research
directions in Section 7.

##  2 Related Work

In this section, we provide an overview of the relevant literature concerning
the management of AI risks. However, the related work concerning particular AI
hazards in detail is described in Section 4.

International standards, such as the ISO IEC Guide 51 [23] or ISO 31000 [18],
provide general guidelines on managing risks independent of the application
domain or used technology. In addition, initial efforts exist toward
standardizing the management of AI risks. For instance, there is the AI Risk
Management Framework published by NIST [45]. Additionally, the ISO/IEC JTC SC
42 is currently working on several standards in the field of AI. This includes
documents providing general guidelines on the management of risks associated
with AI [20, 22]. However, these guidelines stay on a relatively abstract
level, which makes them challenging to operationalize.

Also, controlling risks is of essential importance in safety-critical
applications, particularly in the context of AI. To address safety-related AI
risks, safety assurance cases provide a structured argumentation for
sufficiently mitigating these risks. A specific framework for an AI safety
assurance case has been proposed in [15]. Several works also describe the
general concepts required for an AI safety assurance case, see e.g. [4, 33,
41]. However, while safety assurance cases provide a comprehensive approach to
addressing safety-related risks, they do not fully cover other types of risk,
including legal, economic, or reputational risks.

The topic of AI risks is also covered by the literature on trustworthy AI. The
concepts of trustworthy AI are built upon the works published by the European
High-Level Expert Group (HLEG), see [8, 9]. Although not explicitly addressing
AI risks, there is a direct connection between the principles of trustworthy
AI and the adequate handling of AI risks. This is also reflected in the
proposed European AI Act. Building upon the principles of trustworthy AI, the
proposed regulation requires an AI risk management process for every AI system
to be classified as ”high-risk” by the regulation. Therefore, approaches
assessing the trustworthiness of AI systems provide implicitly valuable
insight on how to manage AI risks, see e.g. [11, 39]. With this work, we aim
to build upon all mentioned approaches and propose an operationalizable
framework to manage AI risks.

##  3 Definitions and background information

To set the stage for the forthcoming discussion, it is essential to clarify
the concept of an AI hazard and offer the necessary context for related terms
and definitions. Also, these terms vary in their interpretation, depending on
the context and domain of use. However, due to a lack of published standards
in the field of AI, a common interpretation of these terms is still to be
established. Therefore, it is important to make clear how we interpret them in
the scope of this work. In this section, we first provide all normative
references and consequently declare our interpretation in the context of AI.

The first important term is AI system. For a definition, we refer to the
proposed AI Act, which defines an AI system as: ”software that is developed
with one or more of the techniques and approaches listed in Annex I and can,
for a given set of human-defined objectives, generate outputs such as content,
predictions, recommendations, or decisions influencing the environments they
interact with” [7]. Annex I of the AIA provides a list of technologies to be
considered AI, which the European Council proposed to narrow down to machine
learning and/or logic and knowledge-based approaches. Note that both, the
regulation itself and also the definition of AI system in the AIA, are still
under discussion at the time of writing.

While the AIA definition of an AI system restricts its focus to the software
aspects, AI systems might be integrated into a larger context, including the
system hardware and even the operational environment. We call this larger
context AI application in the following. Considering, for instance, an
autonomous driving car, the physical vehicle and the operating environment
determine the AI application’s scope. The AI system refers to the software
that is responsible for fulfilling the human-defined objectives, which in this
case can be aggregated to ”operate safely in the operational domain”. This
might include different AI models, such as lane segmentation, and pedestrian
detection.

For the scope of this work, it is also important to clarify the terms risk and
hazard, as well as their relationship and connection to AI systems. According
to the ISO IEC Guide 51, risk can be understood as a function of the
probability and severity associated with a specific hazard [23]. The term
hazard, as defined by the standard, refers to a potential source of harm [23].
While this definition is rather abstract, sector-specific standards adapt the
definition to their specific requirements. For instance, in the road vehicle
domain ISO 26262-1, provides a more tangible version, describing a hazard as a
potential source of harm caused by the behavioral malfunction of an item
regarding its intended design [17]. For the road vehicle domain, this
definition of hazard is appropriate because usually risk comes from non-
properly functioning subsystems of the vehicle.

Nonetheless, as previously noted, these definitions are not immediately
suitable to apply to AI systems. With the growing adoption of AI-based
solutions in the automotive industry, the SOTIF standard (ISO PAS 21448 [19])
was introduced as an extension to ISO 26262. The SOTIF methodology defines the
desired behavior of the AI system as intended functionality and aims to
minimize the operational scenarios in which this functionality cannot be
assured [19]. The specific limitations of the ML algorithms, including their
capability “to handle possible scenarios or non-deterministic behavior”,
typically fits in the scope of the SOTIF [29].

Inspired by these ideas, we aim to extend the concept to AI systems in
general. For the road vehicle domain, the intended functionality of an
autonomous vehicle is clearly to operate safely in any operational scenario.
To transfer this concept to a more general setting it is important to note
that the intended functionality is application dependent. For example, if an
AI system’s purpose is to address a classification problem, a violation of the
intended functionality at the instance level would be a misclassification.
However, the intended functionality might be more complex. For instance, a
classification system may additionally need to maintain similar accuracy for
different input groups. An example would be a CV screening tool that should
not discriminate against applications based on the gender of the applicant,
thus ensuring fairness across all submissions.

Due to the complexity and the fact that AI systems implicitly derive their
behavior from large amounts of data, a direct evaluation of the system in its
operational state is hindered, presenting one of the main challenges in
assessing the risks of AI systems.

Nonetheless, systematic problems persist across the entire lifecycle of AI
systems, which can adversely affect an AI system’s capacity to maintain the
intended functionality. We designate these issues as AI hazards and define
them as the underlying causes for the inability to uphold the intended
functionality, thereby contributing to AI risks. In essence, AI hazards
represent potential sources of harm instigated by AI systems.

##  4 AI Hazard Taxonomy and List

To efficiently identify, assess and treat AI hazards a systematic approach is
required. As a foundation for this, in this section, we introduce a taxonomy
for AI hazards. This taxonomy can then be used to understand when, how, and by
whom to treat an AI hazard during the development and operation of an AI
system. Also, we provide a comprehensive list of AI hazards identified in the
literature as well as a corresponding classification according to the proposed
taxonomy.

###  4.1 AI hazard taxonomy

AI hazards vary significantly across manifold dimensions, such as the life
cycle phase in which they emerge, the system components they impact, and the
stakeholders they involve. To ensure systematic management of these AI
hazards, we propose a taxonomy that characterizes them based on diverse
attributes. This taxonomy subsequently provides an indication of the optimal
methods for the detection, evaluation, and mitigation of a specific AI hazard,
as well as the responsible parties for executing these actions.

The taxonomy comprises three distinct axes: AI life cycle stage, AI hazard
mode, and AI hazard level. To clarify these axes, we provide a detailed
explanation and illustrative examples for each constituent in the subsequent
discussion. An overview of the taxonomy is given in Figure 1.

Figure 1: A taxonomy to classify AI hazards along the axes AI hazard level, AI
hazard mode, and AI life cycle stage.

1) AI life cycle - When to address AI hazards The first axis pertains to the
life cycle of the AI system, as AI hazards may materialize during various
phases of an AI system’s life cycle. For instance, issues triggered by bias in
training data emerge during the data collection and preparation stages. On the
other hand, data drift serves as an example of an AI hazard that arises during
the AI system’s operation. Additionally, certain AI hazards may span multiple
phases of the AI system, such as ”lack of data understanding”. This is because
a proper understanding of the data by the AI developer is required in the data
collection and preparation stage but also in the modeling stage.

To efficiently mitigate the overall risk posed by an AI hazard to the AI
system, it is crucial to identify and address these hazards as early as
possible during the AI life cycle.

However, to construct this axis, a life cycle model is required that
meaningfully subdivides the system’s life into distinct sub-phases. In the
literature, several different models exist describing the life cycle of AI
systems, see for instance [11, 21, 44]. While representing the same process
continuum, these models primarily differ in the granularity of their stages. A
majority of them possess an initial stage devoted to the planning and scoping
of the AI system. Subsequently, the development stage follows, which is
further subdivided. Considering the significance of data and the multitude of
AI hazards that can manifest in the corresponding stage, we decided to split
the development phase into the data preparation and modeling stages. Although
some of the cited models differentiate between the evaluation and deployment
stages, we opted to merge them into a singular stage, as the development of AI
systems typically entails continuous evaluation, and emergent AI hazards
transpire during the ultimate evaluation preceding deployment. Conclusively,
the AI life cycle model terminates with the maintenance and monitoring stage,
which aligns with the referenced models. Therefore, the used AI life cycle
model used in this paper is given: (1) Scoping, (2) data collection and
preparation, (3) modeling, (4) evaluation and deployment, (5) monitoring and
maintenance.

2) AI Hazard mode - How to address AI hazards The second axis of the taxonomy
pertains to the mode of an AI hazard, which determines with what methods to
assess and treat AI hazards. We distinguish among three distinct classes:
technological, socio-technological, and procedural.

Technical AI hazards are the root causes of technical deficiencies in the AI
system. An example of such an AI hazard is overfitting, which describes a
model’s excessive adaptation to the training dataset. Quantitative methods to
assess (metrics) and treat (mitigation means) exist for technical AI hazards,
which might be performed automatically.

In case of overfitting, metrics are based on the comparison of performance
between the training and validation datasets, and mitigation means may include
regularization techniques, among others.

In contrast to technical AI hazards, socio-technical hazards also require
human input related to social and cultural aspects [45]. Human judgment must
be employed when deciding on quantification and treatment methods. For
instance, AI hazards concerning discrimination and privacy, which are abstract
concepts lacking a uniform technical definition, further complicate a clear
quantification of the associated risks. Although quantitative methods exist to
assess and treat these AI hazards, they require coordination with social and
cultural values [27].

The third class encompasses procedural AI hazards. These pertain to issues
arising from processes and actions made by individuals involved in the
development process. Such hazards are not readily quantifiable and necessitate
alternative mitigation strategies. An example of such an AI hazard would be
”poor model design choices,” which could be expressed, for instance, through a
developer’s decision to select an unsuitable AI model for a given problem. Due
to the challenges in quantifying and mitigating these issues, qualitative
approaches must be employed. In the case of the aforementioned example, a
potential strategy might involve requiring the AI developer to provide a
documented rationale for their choice.

Classifying an AI hazard along this axis yields valuable information
concerning the proper evaluation and treatment of the hazard in question.

3) AI hazard level - Where to address AI hazards The third axis of the
taxonomy pertains to the level, which differentiates between the AI
application and system levels, as they are defined in Section 3.

Allocating an AI hazard to its level helps to determine the level at which an
action is required. This consequently sets the basis for who is supposed to
act. For instance, the main person responsible for an AI hazard manifesting on
the AI system level would be the AI developer, whereas an AI hazard affecting
the whole AI application requires a more diverse group, including domain
experts.

###  4.2 AI hazard list

In the following, we provide a list of AI hazards identified in the literature
and a classification according to the taxonomy introduced in Section 4.1. The
corresponding literature references for each AI hazard are displayed in Table
1.

To generate a comprehensive list of AI hazards for all types of AI systems, we
consider a wide range of stakeholders, such as AI developers and users,
industrial associations, standardization bodies, and civil society. To reflect
that AI systems are used in various fields and accomplish different tasks, the
consolidation of a holistic list of AI hazards needs to consider state-of-the-
art in different fields and disciplines. This includes various sectors (e.g.
autonomous (driving) systems, healthcare), different tasks (e.g.
classification, segmentation, clustering), backgrounds (e.g. safety, ethics,
security), and components (e.g. data, algorithms, development process). In the
following, we present an initial list of AI hazards that was consolidated
along the risk-related requirements of a concrete industrial AI use case
(Section 6) for the purpose of demonstrating the value of the overall AI
hazard management approach (Section 5). Due to the focus on the industrial use
case, this initial AI hazard list mainly covers DNN-related AI hazards and
will be extended in our future work. Also, in future work, we aim to enrich
the list by providing information regarding concrete methods on how to manage
particular AI hazards.

In the following, we provide the AI hazard list, together with a short
description of each AI hazard and the corresponding allocation according to
the taxonomy. This is indicated in the format [(AI life cycle steps), Mode,
Level]. Hereby the first entry is a set of numbers representing the
corresponding AI life cycle steps according to the AI life cycle model
described in Section 4.1. The second entry represents the AI hazard mode, with
abbreviations p = procedural, t = technical, s-t = socio-technical. Finally,
the third entry represents the AI hazard level with abbreviations sys = AI
system level, app = AI application level.

AIH 1: Inadequate specification of ODD [(1, 2, 4, 5), p, app] The operational
design domain (ODD) is a technical description of the application’s
operational environment, initially conceptualized for autonomous driving
systems. An inadequate specification of the ODD limits essential functions
such as testing the learned functionality and out-of-distribution detection.

AIH 2: Inappropriate degree of automation [(1, 4, 5), p, app] The AI
application’s degree of automation ranges from no automation to fully
autonomous. AI applications with a high degree of automation may exhibit
unexpected behaviour and pose risks in terms of their reliability and safety.

AIH 3: Inadequate planning of performance requirements [(1, 3, 4, 5), p, sys]
The expected performance of the AI system should be planned adequately.
Hereby, an important aspect is that chosen performance metrics are meaningful
for presenting the intended functionality. Otherwise, expectations and safety
requirements can be unfulfillable at later life cycle stages.

AIH 4: Insufficient AI development documentation [(1, 2, 3), p, sys]
Throughout the development of an AI system, it is vital to document every
decision and action taken. This is not only essential to optimize the
development process itself but also required for the auditability of the AI
system.

AIH 5: Inappropriate degree of transparency to end users [(1, 4, 5), p, app]
The transparency to end users of the AI system increases the user’s trust in
the AI application. If not adequately integrated into the design, this might
prevent the proper operation and cause potential misuse of the AI application.

AIH 6: Missing requirements for the implemented hardware [(1, 4, 5), p, app]
The development and operation of an AI system can require significant amounts
of (computational) power. If not considered in the hardware selection, this
can become an issue in development and operation.

AIH 7: Choice of untrustworthy data source [(2), p, sys] The choice of a
trustworthy data source is a first prerequisite in order to fulfill data
quality requirements. This is especially the case if third-party data sources
are used to develop the AI system.

AIH 8: Lack of data understanding [(2,3), p, sys] The correct understanding of
the used data for developing an AI system is a prerequisite to avoid data
shortcomings and hinders the development of an AI system which is best suiting
for the intended functionality.

AIH 9: Discriminative data bias [(2), s-t, sys] Discriminative data bias
describes the systematic discrimination of groups of persons in the form of
data shortcomings, such as distributional representation or incorrectness.
Data bias can manifest in the model and lead to unfair decisions if not
appropriately treated. Note, that the term bias is often used in other
contexts, such as data representation. However, these issues are treated by
other AI hazards in this list.

AIH 10: Harming users’ data privacy [(2), s-t, sys] Modern AI systems rely on
large amounts of data. If this includes personal data about individuals, the
risk of harming the privacy of persons arises.

AIH 11: Incorrect data labels [(2), t, sys] Data labels are essential for any
supervised learning algorithm since they preset the result of the learning
process. If the correctness of the data labels is not given, the AI system is
prevented from learning the ground truth and therefore the intended
functionality.

AIH 12: Data poisoning [(2), t, sys] Data poisoning describes an attack in the
form of an injection of malicious data into the training set. If not
prevented, this attack leads the AI system to learn unintended behavior.

AIH 13: Insufficient data representation [(2), t, sys] The distribution of the
data used for training a model should match the operational data´s
distribution while consisting of sufficiently many samples. An important
aspect of matching distributions between training and operational data is that
also data which is rarely confronting the AI system in operation is
represented in the training data.

AIH 14: Problems of synthetic data [(2), t, sys] In the case of sparse data
quantity, the simulation or generation of data is a valid alternative.
However, it is essential to make sure that the simulated data is sufficiently
similar to real data, especially in the way the AI system perceives them.
Otherwise, generalization to operational data and reliable operational
behavior can not be guaranteed.

AIH 15: Inappropriate data splitting [(2), p, sys] In data-driven AI
development, the annotated data set is commonly split into training,
validation, and test sets, whereby it is essential that the latter is not used
for development but only for evaluation. Using the test set for training
manipulates the testing strategy, which is the basis of the system’s quality
assurance.

AIH 16: Poor model design choices [(1,3), p, sys] The model specifications
have significant impact on the functionality of an AI system. The developer
making wrong decisions might cause the AI system to behave biased and
unreliable.

AIH 17: Over- and underfitting [(3), t, sys] Over- and underfitting describe
the over or insufficient adaption of a model to training data. Both phenomena
can cause an AI system to behave unreliably if confronted with operational
data.

AIH 18: Lack of explainability [(3), s-t, sys] The explainability of AI
systems based on so-called black-box models is often limited. This opaqueness
of AI systems can prevent developers from detecting shortcomings in the data
or the model itself and decrease the performance and safety levels of the AI
system.

AIH 19: Unreliability in corner cases [(3), t, sys] AI systems tend to show
unreliable behavior when confronted with rare or ambiguous input data, also
called corner cases. Therefore, the controlled behavior is required whenever
the AI system is faces a corner case.

AIH 20: Lack of robustness [(3), t, sys] Robustness characterizes the
resilience of an AI system’s output against minor changes in the input domain.
A great variation in an AI system’s response to small input changes indicates
unreliable outputs.

AIH 21: Uncertainty concerns [(3), t, sys] AI systems should be able not only
to return output for a given instance but also to provide a corresponding
level of confidence. If such a method is not implemented or not working
correctly, this can have a negative impact on performance and safety.

AIH 22: Operational data issues [(4), t, app] Until the deployment of the AI
application into its operational environment, the AI system has been tested
with a test set that aims to approximate the distribution of operational data.
However, an unexpected deviation in this approximation can cause an AI
application to behave unreliably. Therefore, its behavior under confrontation
with operational data needs to be evaluated.

AIH 23: Data drift [(5), t, sys] Data drift is a phenomenon in that
distribution of operational input data departs from those used during
training. This can cause a degradation in performance.

AIH 24: Concept drift [(5), t, sys] Concept drift refers to a change in the
relationship between input variables and model output. If not treated
appropriately, concept drift can reduce the reliability of AI systems.

Table 1: An overview of the AI hazards (AIH) together with their related
literature.

AIH sources AIH sources AIH sources AIH sources AIH 1 [25, 46] AIH 7 [26] AIH
13 [31, 46] AIH 19 [16, 36, 46] AIH 2 [25, 42] AIH 8 [24, 37] AIH 14 [2, 30]
AIH 20 [16, 25, 42, 46] AIH 3 [42, 48] AIH 9 [34, 42] AIH 15 [46, 48] AIH 21
[16, 48] AIH 4 [13, 32, 40] AIH 10 [34, 42] AIH 16 [16, 48] AIH 22 [44] AIH 5
[3, 42] AIH 11 [12, 46] AIH 17 [6, 48] AIH 23 [16, 25, 42, 46] AIH 6 [25, 42]
AIH 12 [42, 43] AIH 18 [3, 16, 25, 31] AIH 24 [16, 25, 46]

##  5 AI Hazard Management

In this section, the AI Hazard Management framework is presented. It builds
upon the preliminary AI hazard list shown in Section 4 to fulfill its primary
objectives, which are:

1) Systematically identify, assess, and treat all AI hazards.

2) Provide structured documentation to demonstrate that each identified AI
hazard has either been mitigated to an acceptable level or the developer has
been alerted to any shortcomings in the AI system caused by AI hazards.

With the AIHM framework, we aim to follow the general structure of a risk
management process, such as given in the ISO 31000 [18], to ensure that it is
compatible with existing risk management procedures in industry. The four main
components of the risk management process in ISO 31000 are: risk
identification, risk estimation, risk evaluation, and risk treatment [18]. We
found that in AI development, the risk estimation and evaluation steps are
highly overlapping, so we aggregated them into a single step called AI risk
assessment. Therefore, the three steps of the AIHM process are: 1) AI hazard
identification, 2) AI risk assessment, and 3) AI risk treatment. The proposed
process is depicted schematically in Figure 2, and details regarding the three
steps are outlined in the remaining part of the section. Additionally, as the
transparency and auditability of the process are key features, continuous
documentation of the process is required, which will be discussed at the end
of the section. The AIHM process is executed at every stage of the AI life
cycle.

Figure 2: A schematic representation of the AIHM framework comprising the
three main components: AI hazard identification, AI risk assessment, and AI
risk treatment.

###  5.1 AI hazard identification

The AI hazard identification is the initial step of the process and represents
the risk identification as outlined in ISO 31000. It aims to identify any
possible AI hazard potentially causing AI risk. In the AIHM, the outcome of
the AI hazard identification step is an instantiated AI hazard list, which is
the result of filtering the complete AI hazard list, as given in Section 4, in
two steps: Firstly, only AI hazards having the potential to occur at a
particular stage of the AI life cycle are selected. This information can be
obtained from the AI life cycle axis within the AI hazard list and taxonomy,
see Section 4. The second phase of the filtering process depends on the
specific use case, as the impact of AI hazards may vary based on various
factors, such as the AI technique (e.g., reinforcement learning vs. supervised
learning) and the context of the AI application (e.g., human resources vs.
industrial predictive maintenance). Therefore, this step is performed by a
team consisting of domain experts and data scientists to ensure the required
expertise. Reducing the instantiated AI hazard list to only the relevant AI
hazards aims to increase the efficiency of the whole process. Every decision
regarding including or excluding AI hazards in the AIHC must be documented and
justified with a clear argumentation.

###  5.2 AI risk assessment

The AI risk assessment represents the risk estimation and evaluation steps
outlined in ISO 31000. This step aims to assess the impact of AI hazards
identified in the previous step and evaluate whether it is tolerable in a
particular application. To accomplish this, a test strategy has to be
initialized, by determining appropriate metrics and corresponding acceptance
criteria. The exact execution of this step depends on the mode of the
respective AI hazards. While no quantitative methods exist for procedural AI
hazards, which rely on human errors, a possible form of assessment could be
the provision of process documentation in which the involved party justifies
all decisions and actions. A second person then evaluates this process
documentation for correctness and meaningfulness. In the case of socio-
technical or technical AI hazards, appropriate metrics, and acceptance
criteria must be established, such as thresholds related to the chosen
metrics. The specification of the acceptance criteria also is highly dependent
on the AI application’s context and potential technical limitations.
Therefore, having a team with a diverse background is beneficial for setting
the acceptance criteria to cover all relevant aspects. This includes domain
experts, data scientists, and business representatives. For example, metrics
such as false positive or negative rates can be utilized to evaluate the
quality of a classification algorithm. However, the consequences of a false
negative classification in a medical decision-making system could be more
severe than in a Human Resources application, classifying job applicants based
on their suitability for an open position. Hence, the acceptance criteria for
the metrics should also consider these different levels of consequences.

###  5.3 AI risk treatment

The objective of the third step is to reduce the impact of AI hazards that
have been evaluated as non-tolerable in the previous step. This corresponds
with risk treatment in ISO 31000. Suitable AI risk mitigation measures shall
be applied to reduce the impact of AI hazards. If several are available, a
team of data scientists and domain experts should hold a council on the most
suitable. For example, a common problem in supervised learning is that a model
might not be robust against small perturbations in the input space. Augmented
training is a common way to mitigate this problem, which means re-training the
supervised learning model with additional data gained by applying data
augmentation techniques, e.g., [49]. However, in some cases, the augmentation
of training data is too complex, and the additional data sampling is too
expensive. In such a case, a different mitigation technique, for instance,
using a different cost function (regularization), can be applied, see for
instance [35].

Further problems in the AI risk treatment step are interactions and trade-offs
between AI hazards. For example, although increasing the robustness of a
model, regularization techniques might decrease the overall performance of a
model, e.g., [50]. Consequently, acceptance criteria dependent on the overall
performance can be violated if mitigation techniques improving the robustness
of the model are applied. Therefore, such trade-offs should be considered in
the decision process regarding suitable mitigation techniques within a trade-
off analysis. Also, they need to be re-evaluated after applying a mitigation
technique.

###  5.4 Documentation

The above-described process is iterated until all acceptance criteria are
fulfilled or the completion of all acceptance criteria is evaluated as
impossible. In both cases, every step is to be documented in a structured
form, including every action taken and argumentation for any decision made.
The developer must be notified if the risk cannot be reduced to an acceptable
level. Consequently, additional safeguards shall be implemented.

##  6 Case Study

In this section, we demonstrate how the AIHM framework provides guidance for
risk management while developing an industrial AI application. As a case
study, we examined a protective solution for power grids to address the
challenge of detecting residual currents resulting from high-impedance ground
faults (HIGF). These faults can pose significant dangers, such as wildfires,
and are difficult to identify even with the expertise of domain specialists.
The proposed solution utilizes a deep neural network (DNN) to classify the
presence or absence of high-resistance faults in the distribution power grid
by analyzing time series data of voltage and current. For simplicity, we
assume the AI system to be a binary classifier, i.e. detecting the presence or
non-presence of a HIGF in the grid. Note that the AI system is scoped as a
decision support tool at the project’s current state. This means that if it
detects a HIGF, an alerting system informs a human user, who ultimately has
the decision power over consequent actions.

The application of the AIHM to the power grid use case was simulated by
conducting an interview with the lead data scientist and domain expert. This
interview was performed in three sessions, taking a total of 4 hours. In this
manner, decision-based steps in the AIHM could be simulated. The execution of
the AIHM framework is demonstrated by examining its implementation regarding
four AI hazards: discriminative data bias, lack of robustness, inappropriate
degree of transparency to end users, and data drift. These hazards are
representative of various stages in the AI life cycle. With this, the
different actions required to handle AI hazards are highlighted.
Discriminative data bias

AI risk identification Firstly, note that discriminative data bias relates to
the data collection and preparation stage in the AI life cycle, see Table 1.
There are various types of how data bias can manifest itself in the data,
including representation, sampling, aggregation, and omitted variable bias.
For a comprehensive overview of these different types, we refer to [34]. The
second filtering step described in Section 5.1 reviews the relevance of the AI
hazard for the particular use case. The only data used in developing the AI
system was time series data of voltage and current in the power grid.
Therefore, arguably any impact of discriminative data bias could be ruled out.
Hence, there is no need to consider this AI hazard in the subsequent
assessment and mitigation steps.

AI risk assessment and treatment no need for action

Documentation In summary, as the AI hazard was evaluated as irrelevant for
this application, the impact of discriminative data bias is tolerable.
Furthermore, this claim is supported by an argumentation regarding the data
properties, ensuring required auditability.
Lack of robustness

AI risk identification The AI hazard lack of robustness is concerned in the
modeling stage, see Section 4. The confrontation with adversarial data during
operation is evaluated as extremely unlikely due to an attacker’s extreme
effort to create and infiltrate them into the AI system. Hence, robustness
against adversarial attacks is not the developer’s main concern. Since the
application’s operational data potentially suffers the influence of noisy
measurements, the model’s robustness against input perturbations is essential.

AI risk assessment Consequently, suitable metrics and corresponding acceptance
criteria need to be determined. A possible way of assessing the robustness of
a model is to evaluate its performance against meaningful perturbations in the
input data. For example, there exist many techniques in the field of signal
processing, see e.g. [28] Concerning the data in this particular application,
more sophisticated data augmentation is required. Data scientists and domain
experts create an augmented data set, mainly using simulation-based
augmentation techniques. Regarding the acceptance criteria, a maximal
tolerable performance decrease on this perturbed data set compared to the
original one has to be determined. If this threshold is exceeded, the impact
of lack of robustness is evaluated as non-tolerable.

AI risk treatment Assuming to observe a non-tolerable impact of this AI
hazard, mitigation measures are required. For instance, the model could be re-
trained using augmented data. A possible negative side effect when increasing
the robustness of a model is that this potentially causes a decrease in the
model’s overall performance. This is called the robustness-accuracy trade-off
[50]. The model’s overall performance is re-evaluated to ensure the mitigation
technique has no adverse side effects.

AI risk assessment (revised) After re-training the model on the augmented
training set, its estimation metric is also to be re-evaluated to see if the
mitigation technique was sufficient. If evaluated as sufficient, the AI hazard
is considered tolerable.

Documentation Every step and preliminary result, including the choice of
estimation metrics, acceptance criteria, and evaluation results, shall be
documented to ensure the auditability of the process.
Inappropriate degree of transparency to end users

AI hazard identification The transparency of an AI system to the end user is
concerned in the scoping stage as well as at deployment and operation. The AI
system is based on a DNN and is considered a so-called black box model. This
means that determining an output for a given input is not human-
understandable, per se. However, since this decision-making has severe
consequences (e.g., shutting down the power grid), more transparency is
required. Hence, the AI hazard inappropriate degree of transparency to end
users is identified as relevant and needs to be considered in the assessment
and potentially in the mitigation step.

AI risk assessment Numerically quantifying transparency, in general, is not
viable. Therefore, a qualitative estimation and evaluation might oppose a
suitable solution. Determining what means are necessary to provide an
appropriate degree of transparency depends on the application and user
profile. In this application, the end-user are domain experts with a
particular level of domain knowledge, which should be acknowledged in what
kind of information the AI application needs to provide the user during
operation.

AI risk treatment To increase transparency during operation, a monitoring
system is implemented. This includes a dashboard informing the user about
activities made by the AI system. For instance, if the system predicts the
presence of a HIGF in the grid, this is reported to the user, who can see the
input data that led to this decision. By this means, a human is set in the
loop.

Documentation Again every result is to be documented.
Data drift

AI risk identification Data drift describes the change of input distribution
over time. If not managed properly, this issue can limit the performance of an
AI system. To understand whether this AI hazard is relevant for the power grid
application, consider the following observations. Electric grids differ in
size and location. Especially with the latter, there come differences in
climate-related factors such as temperature or humidity impacting the input
data. Therefore, assuming to deploy the AI system in different locations, it
is not guaranteed that it has equal performance. Also, factors such as climate
change can influence the functionality quality of the AI system in the long
run. Therefore, data drift is relevant in this application.

AI risk assessment Data drift is an issue known to manifest over time after
deployment. However, preventive activities can be executed during development.
This includes considering foreseeable distribution shifts. For instance, in
case of deploying the AI system in another power grid, the model can be
adapted by re-training with data gained from the new location.

AI risk reduction However, as not every shift in distribution can be certainly
predicted, a periodic distribution shift detection mechanism can be
implemented. This means the similarity between the training data and a sample
collected in operation is evaluated in a pre-defined frequency. If a
significant deviation in this distribution is detected, this indicates the
presence of data drift. In such a case, countermeasures can be initialized,
including re-training the model.

Documentation By laying out a concrete plan for handling data drift,
requirements towards auditability can be fulfilled.

##  7 Conclusion

In this paper, we proposed the AI Hazard Management framework, which
systematically manages AI risk’s root causes. Further, it delivers evidence
that their impact is reduced to a tolerable level. The application to a power
grid use case showed the framework’s effectiveness in managing AI hazards. The
framework builds upon a preliminary list of AI hazards and a taxonomy
supporting to manage the AI hazards. An extension of the AI hazard list is
subject to future work. Through the analysis of further AI use cases, we
expect to identify further relevant pitfalls of AI. In addition, we plan to
systematically investigate quantification and mitigation techniques for the
list of identified AI hazards.

####  7.0.1 Acknowledgement

This research has received funding from the Federal Ministry for Economic
Affairs and Climate Action (BMWK) under grant agreements 19I21039A.

## References

  * [1] Apostolidis, K.D., Papakostas, G.A.: A survey on adversarial deep learning robustness in medical image analysis 10(17), 2132
  * [2] Assefa, S.: Generating synthetic data in finance: Opportunities, challenges and pitfalls
  * [3] Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J.M.F., Eckersley, P.: Explainable machine learning in deployment. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
  * [4] Burton, S., Hellert, C., Hüger, F., Mock, M., Rohatschek, A.: Safety assurance of machine learning for perception functions. In: Fingscheidt, T., Gottschalk, H., Houben, S. (eds.) Deep Neural Networks and Data for Automated Driving. Springer International Publishing
  * [5] Caton, S., Haas, C.: Fairness in machine learning: A survey
  * [6] Dietterich, T.: Overfitting and undercomputing in machine learning 27(3), 326–327
  * [7] European Commission: Proposal for a regulation of the european parliament and the council laying down harmonized rules on artificial intelligence (ARTIFICIAL INTELLIGENCE ACT)
  * [8] European Commission, HLEG: Assessment list for trustworthy artificial intelligence (ALTAI) for self-assessment
  * [9] European Commission, HLEG: Ethics guidelines for TAI
  * [10] European Parliament: Directive 2006/42/EC of the european parliament and of the council of 17 may 2006 on machinery, and amending directive 95/16/EC
  * [11] Floridi, L., Holweg, M., Taddeo, M., Amaya Silva, J., et al.: capAI - a procedure for conducting conformity assessment of AI systems in line with the EU artificial intelligence act
  * [12] Frank, J., Rebbapragada, U., Bialas, J., Oommen, T., Havens, T.: Effect of label noise on the machine-learned classification of earthquake damage
  * [13] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J.W., et al.: Datasheets for datasets
  * [14] Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples
  * [15] Hawkins, R., Paterson, C., Picardi, C., Jia, Y., et al.: Guidance on the assurance of machine learning in autonomous systems (AMLAS)
  * [16] Houben, S., Abrecht, S., Akila, M., Bär, A., et al.: Inspect, understand, overcome: A survey of practical methods for AI safety. pp. 3–78
  * [17] ISO: ISO 26262-1: Road vehicles — functional safety — part 1: Vocabulary (2018)
  * [18] ISO: ISO 31000: Risk management — guidelines (2009)
  * [19] ISO: ISO PAS 23894: Road vehicles — safety of the intended functionality (2019)
  * [20] ISO/IEC: ISO IEC 23894: Artificial intelligence: Guidance on risk management (2022)
  * [21] ISO/IEC: ISO IEC 5338: AI system life cycle processes (2022)
  * [22] ISO/IEC: ISO IEC FDIS 42001: Information technology — artificial intelligence — management system
  * [23] ISO/IEC: ISO IEC GUIDE 51: Safety aspects: Guidelines for their inclusion in standards (2014)
  * [24] ISO/IEC: ISO IEC TR 24027: Artificial intelligence — bias in AI systems and AI-aided decision making (2022)
  * [25] ISO/IEC: ISO IEC TR 5469: Functional safety and AI systems (2022)
  * [26] ISO/IEC: ISO IEC TS 29119: Software and systems engineering - software testing - part 11: Testing of AI systems (2022)
  * [27] ISO/IEC: ISO/IEC TR 24368: Information technology — artificial intelligence — overview of ethical and societal concerns (2022)
  * [28] Iwana, B.K., Uchida, S.: An empirical survey of data augmentation for time series classification with neural networks
  * [29] Jenn, E., Albore, A., Mamalet, F., Flandin, G., et al.: Identifying challenges to the certification of machine learning for safety critical systems p. 10
  * [30] Johnstone, M., Peacock, M.: Seven pitfalls of using data science in cybersecurity. In: Sikos, L.F., Choo, K.K.R. (eds.) Data Science in Cybersecurity and Cyberthreat Intelligence, vol. 177, pp. 115–129. Springer International Publishing
  * [31] Kurd, Z., Kelly, T.: Establishing safety criteria for artificial neural networks. In: Palade, V., Howlett, R.J., Jain, L. (eds.) Knowledge-Based Intelligent Information and Engineering Systems. Springer Berlin Heidelberg
  * [32] Königstorfer, F., Thalmann, S.: AI documentation: A path to accountability
  * [33] McDermid, J., Jia, Y., Habli, I.: Towards a framework for safety assurance of autonomous systems
  * [34] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning
  * [35] Moradi, R., Berangi, R., Minaei, B.: A survey of regularization strategies for deep models
  * [36] Ouyang, T., Marco, V.S., Isobe, Y., Asoh, H., et al.: Corner case data description and detection
  * [37] Polyzotis, N., Roy, S., Whang, S.E., Zinkevich, M.: Data lifecycle challenges in production machine learning: A survey
  * [38] Popov, G., Lyon, B.K., Hollcroft, B.: Risk Assessment: A Practical Guide to Assessing Operational Risks. Wiley, 1 edn.
  * [39] Poretschkin, M., Schmitz, A., Akila, M., Adilova, L., et al.: Leitfaden zur gestaltung vertrauenswürdiger künstlicher intelligenz (KI-prüfkatalog) Publisher: Fraunhofer-Gesellschaft
  * [40] Raji, I.D., Smart, A., White, R.N., Mitchell, M., et al.: Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing
  * [41] Schwalbe, G., Schels, M.: A survey on methods for the safety assurance of machine learning based systems
  * [42] Steimers, A., Schneider, M.: Sources of risk of AI systems
  * [43] Steinhardt, J., Koh, P.W.W., Liang, P.S.: Certified defenses for data poisoning attacks
  * [44] Studer, S., Bui, T.B., Drescher, C., Hanuschkin, A., et al.: Towards CRISP-ML(q): A machine learning process model with quality assurance methodology
  * [45] Tabassi, E., NIST: AI risk management framework: AI RMF (1.0), https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf
  * [46] Willers, O., Sudholt, S., Raafatnia, S., Abrecht, S.: Safety concerns and mitigation approaches regarding the use of deep learning in safety-critical perception tasks
  * [47] Wilson, B., Hoffman, J., Morgenstern, J.: Predictive inequity in object detection
  * [48] Winter, P.M., Eder, S., Weissenböck, J., Schwald, C., et al.: Trusted artificial intelligence: Towards certification of machine learning applications
  * [49] Wong, S.C., Gatt, A., Stamatescu, V., McDonnell, M.D.: Understanding data augmentation for classification: when to warp?
  * [50] Zhang, H., Yu, Y., Jiao, J., Xing, E., Ghaoui, L.E., Jordan, M.: Theoretically principled trade-off between robustness and accuracy. In: Proceedings of the 36th International Conference on Machine Learning