Having posed this concern, it should be observed that the emergence of big data in some existing industrial applications may not extend to new highly automated processes involving human–autonomy teaming scenarios, as in algorithmic management approaches. That is, there may be an absence of empirical observation of novel work systems and, consequently, sparse data relative to the number of decision variables that must be considered in risk assessment and safety practice. In data analytics, this problem is referred as the "curse of dimensionality", where available process data records are sparse compared with the number of attributes being measured. The curse of dimensionality poses fundamental issues of sensitivity and reliability of any data analytics or statistical analysis. Thus, in some truly unique algorithmic management applications, there may also be a need to identify alternative AI or advanced analytical methods by which to generate field-relevant data/estimates as a basis for application of modeling and decision analysis tools.

It is anticipated that new computational capabilities and AI algorithms may also provide a platform for more computationally sophisticated (real-time) safety analysis methods. For example, more demanding modeling approaches, including high dimensional structures, can be processed by new AI supercomputers in a matter of milliseconds. However, such data-driven computational safety analysis methods may lack the scientific basis and systematic nature of traditional systems safety analysis (SSA) methods. For example, results of ML methods tend to be unique to a specific system, rather than generalisable across application domains. On the other hand, existing traditional SSA methods may not be able to accommodate big data sources. For example, traditional fault tree analysis (FTA) methods do not account for potential event consequences, which can be estimated based on big (process) data.

In general, there is a need for enhanced risk assessment methods to ensure that new management approaches and safety controls, as part of digital industrial processes, are compatible and jointly effective. The specific research question we seek to address is whether new advanced tools can be created for precise projections of safetycritical system outcomes, given high-dimensional decision spaces? Such spaces are characterised by sources of harm in a work system leading to numerous safety-related events (or mechanisms) to negative outcomes. Ultimately, we also need to investigate how such technologies can be exploited to better support work performance and safety.

#### **6.2 Background**

A fundamental challenge for safety science has been to determine how to minimise hazard exposure without compromising work productivity. Unfortunately, traditional SSA methods for mitigating exposure have trailed behind industry applications, not even considering the various forms of digitalisation identified by Le Coze and Antonsen [5]. In the early 1980s, the Air Force developed a collection of formal SSA methods as part of MIL-STD-882 [8]. These methods primarily focused on the frequency of hazard exposure vs. severity of outcomes or degree of loss. It took another 14 years for NIOSH (National Institute for Occupational Safety and Health) to develop a guide to further disseminate these methods [2]. Unfortunately, there have been limited advances in formal SSA methods since that time.

The persistence of workplace deaths and injuries underscores the need for enhancement of existing SSA methods to better support industry in precision loss assessment and control when applying digital technologies and process management. This research need is further motivated by economics. The Liberty Mutual (LM) 2019 Worker's Compensation Claims (WCC) Index revealed \$46.93 B per year in losses for US industry from the top-10 most disabling worker injuries. The insurance carrier estimated industry costs at over \$1 B per week [6]. This level of loss is unacceptable with respect to industry financial stability and US global competitiveness. Comprehensive and valid safety critical engineering methods represent one potential solution to reducing total incident rates (TIRs) in industry.

#### **6.3 Objective**

We present a new algorithm for advancing an existing SSA technique, specifically fault tree analysis (FTA), by integrating consideration of the severity of consequences of safety-critical system events for design and operational risk assessment. Such considerations can now be made based on big data sources in digital process scenarios. As the reader may be aware, traditional FTA only accounts for the likelihood of occurrence of system fault states but not the probability of various degrees of system or operational loss, given the occurrence of a fault [2]. Although FTA has realised substantial application in government/military and industrial applications since its conception, the method has always represented an incomplete risk assessment approach. Furthermore, it requires substantial analyst time and effort, even when addressing a single fault event. Another limitation of FTA is that the method considers only the outcome of an identified undesirable and credible ("top") fault event for which an analyst must have prior knowledge. There is no analysis of potential negative outcomes of predecessor (intermediate) events to the top event that are causal in nature. Consequently, any FTA provides an incomplete picture of the total level of system risk posed by a top event, intermediate events and basic sources of harm in a work environment.

The way that these FTA shortcomings have been addressed is by an analyst applying other safety analysis techniques, including but not limited to FMEA (Failure Modes and Effects Analysis (FMEA), Event-tree Analysis (ETA), Probabilistic Risk Assessment (PRA), Cause-consequence Analysis (CCA), etc., in conjunction with FTA [3]. Most of these additional methods make use of FTA outputs to facilitate evaluation of the severity of specific system faults. The common FTA outputs include identification of a minimum set of basic initiators and intermediate events guaranteeing occurrence of the top event [i.e., minimal cut sets (MCSs)] or the exact probability of a top event. Related to this, we only found one study in the literature that addressed severity of hazard exposure in application of FTA. Lindhe et al. [7] presented a dynamic fault tree (FT) with hazard risk determined by means of Monte Carlo simulation [7]. However, any intermediate events in the FT were not identified as having unique outcomes relative to the top event.

In summary, with the advent of digitalisation and big data sources, it may now be possible to predict system event consequences and probabilities for more accurate risk assessment methods. However, there remains an absence of systematic and structured approaches to account for this information as part of FTA. This methodological situation limits the accuracy of SSA as a basis for risk assessment and supporting system design, which should be particularly critical for highly automated and highrisk work environments.

#### **6.4 Review of Traditional FTA**

In the traditional FTA approach, we initially construct a FT diagram (see Fig. 6.1) and identify relationships among all process events deduced to contribute to the top event (fault state). Logic (AND/OR) gates are used to create connections between the events (see Fig. 6.2).

We subsequently assign probability values to the occurrence of basic sources of harm in the environment (or initiators). Typically, an analyst makes use of a historical database or safety records for probability data. In the case of novel human– autonomy teaming applications, such data may not be available and, consequently, expert judgements and advanced analytical methods may be necessary to generate probability estimates for analysis (we say more on this below). In traditional FTA, we

**Fig. 6.1** Example FT. *Note* Tree has five layers with eight basic events/initiators (*Bi* ), six intermediate events (*Ei* ), and one top event (*ET* ). An example AND gate integrates inputs from *E*1· *B*4· *E*3; an example OR gate integrates inputs from *E*2 + *B*3

**Fig. 6.2** Each intermediate event in a FT may represent a top event in another FT. For each intermediate event, we specify a set of negative outcomes and levels of associated severity as bases for the CSPIM analysis

calculate the probability of the top event and intermediate events, based on initiator probabilities *(P(Bi ))*, and use of Boolean algebra.

The value of the event probabilities is a sufficient basis for calculating the importance of initiating events and cuts sets (minimum combinations of events causing the top event). It is necessary to make use of a method to obtain cut sets (MOCUS) or Boolean equivalent tree construction to identify the MCSs for a FT (but event probabilities are not necessary). For the example tree in Fig. 6.1, we determined three MCSs, including: {*B*3, *B*4}, {*B*1, *B*2, *B*4, *B*5} and {*B*1, *B*2, *B*4, *B*6, *B*7, *B*8}. These are guaranteed pathways to the top event or system fault state and represent decision alternatives/priorities for a safety engineer. In traditional FTA, MCSs represent areas of greatest system vulnerability and the most likely targets for safety countermeasures.

For each of the MCSs, we determine a probability of occurrence as well as an importance value and ranking of the cut sets. Traditional FTA limits MCS importance to the impact on the top event probability of occurrence. The probability of a cut set can be calculated as the product of the probabilities of embedded initiators ( - *Pk* = *m B*=1 *P*B), where *Pk* is the *k* th cut set probability and *m B*=1 *P*B represents the product of the probabilities of *m* initiators in *k* th MCS. The importance is the ratio of the MCS probability to top event likelihood: *Ik* = *Pk P*T , where *P*T is the estimated probability of the top event (based on data or expert judgment) and *Ik* is the *k* th cut importance value.

On the basis of this analysis, cut sets can be ranked in terms of importance and the ranking can be used as a basis for safety resource allocation. The MCS with the greatest importance ratio is the most likely set of initiators to cause the top event. An analyst might recommend changing the system structure (or FT event relationships) to increase the number of initiators in the top-rank MCS and reduce the probability of its occurrence. An analyst might also recommend changing system components, materials or procedures to reduce the likelihood of initiators as part of the top-rank MCS.

These traditional FTA outcomes are all fine and good but, as we noted in the literature review, the method does not capture the contributions of initiators to the severity of consequences of the intermediate and top events, nor does it support event risk assessment; i.e., *Ri* = *P(Ei )* · *S*, where *Ri* is the hazard risk for event *Ei* and *S* is the severity of outcome of exposure. However, this kind of information is very important in most safety analyses as those initiators primarily contributing to the occurrence of an event may not be one in the same with those factors contributing to the degree of loss, given the occurrence of the event.

#### **6.5 A Consequence Severity-Probability Importance Measure Algorithm for FTA**

On the basis of the literature and review of traditional FTA, we propose an algorithmic extension of FTA with the purposes of: (a) analysis of the severity of consequences of both top and intermediate events as part of a FT and (b) risk assessment at both the event and cut set level. The ultimate objective of the algorithm is to provide a fine-grained analysis of FT event and cut set risk as a basis for precise and costeffective safety control measure prescription by practitioners. Here, it is important to note that we elect to extend the existing FTA method due to its scientific basis and systematic approach to generating results for specific system fault states that may generalise to other domains. These features are desirable relative to a ML/neural network approaches that can only reveal combinations of initiators guaranteeing a system fault but do not identify various event pathways to the fault.

The CSPIM algorithm begins just like traditional FTA, including constructing a FT diagram and assigning probability values to initiators. The FT identifies relationships among the top event and hardware failures, environmental factors, system command faults, etc. For initiator probabilities, we make use of any available system performance data, observations on legacy systems, benchmarking of competing technologies, or expert estimates. In the latter case, estimates are used to address the curse of dimensionality that can occur in FTA, specifically the number of pathways to a top event may exceed the data available for estimating the likelihood of events in a pathway. Consequently, there is a need for estimates and advanced analytical approaches to facilitate accurate decision making on safety controls. Lavasani et al. [4] showed how a fuzzy estimation approach can be applied to judgements of event possibilities by a group of experts. Linguistic terms (of probability) are translated to predetermined intervals of possibility values using fuzzy sets. Possibility estimates are aggregated across experts with mean values being defuzzified to crisp possibility scores. These scores are mathematically transformed to initiator probability values. Here, it is important to note that experts are typically senior systems operators, who are initially interviewed individually for event likelihood estimates. This step is followed by expert group review and discussion on likelihoods in which some individual estimates may be adjusted. Lastly, all estimates are transformed to probability scores, and aggregate values are calculated. This process is most applicable to unique work systems with limited data for safety decision analyses.

At this stage, CSPIM departs from traditional FTA. We identify consequences associated with each event (intermediate and top) and qualitatively categorise consequences according to levels of severity (*S*). (See Fig. 6.2 for a conceptual representation of this step, as an extension of traditional FTA.) For this purpose, we use the Hazard Severity Category (HSC) scheme provided in MIL-STD-882E (1984; *S*1 = "catastrophic"; *S*2 = "critical", *S*3 = "marginal"; *S*4 = "negligible"). An analyst may also use other severity ranking scales appropriate to the specific industry or develop a custom scale based on prior company safety experience/accidents. The consequences of *ET* and *Ei* may occur at many different levels of severity (*Sc*). Therefore, it is necessary to assign a set of conditional probabilities for levels of severity of outcome for each intermediate event (*P S*1|*E j* ; *P S*2|*E j* ; *P S*3|*E j* ; *P S*4|*E j* ) as well as the top event, producing a matrix of severity probabilities for the FT. Ordinarily, these data would also be based on prior system experience but expert judgements can be used as well for novel applications.

As with traditional FTA, the CSPIM algorithm requires that we use initiator probabilities and Boolean algebra to calculate the top event and intermediate event probabilities. These values are then used as bases for calculating and assessing composite event risk values. We calculate the product of the probability of the fault event, the probability of a specific severity of consequence (assuming fault occurrence) and the severity level, written as: *R j*c = *P E j* · *P S*c|*E j* · *S*c, where *c* is the level of consequence severity. As all the variables in this equation should actually appear in a risk matrix for all FT events and all consequence severity levels, the equation should be rewritten using matrix notation as: *RE* = *P(E)* · *P(S*|*E)* · *S*.

The composite risk value of events are compared with an established (or custom) Hazard Risk Index (HRI), such as that from MIL-STD-882E. This index facilitates classification of risk outcomes in terms of hazard exposure and severity of outcomes and identifies HRI levels. Using the risk assessment matrix, an analyst codes the composite risk values as representing "low", "medium", "serious" and "high" exposures. The MIL-STD index also provides for general safety control actions based on HRI levels.

At this stage, the analyst is looking for any events that represent serious or highlevel risks. According to the MIL-STD, high risks are "unacceptable" and serious risks are "undesirable", which means response countermeasures should be taken as soon as possible to mitigate the probability or severity of a specific outcome.

As with the traditional FTA method, we use MOCUS or Boolean equivalent tree construction, as part of the CSPIM algorithm, to identify the MCSs for the FT. It is also necessary for an analyst to identify any and all intermediate events triggered by initiators as part of different MCSs. In Fig. 6.3, we have marked in "red" the {*B*3, *B*4} MCS as well as the intermediate events triggered by this cut set. This step should also be applied to the other two identified MCSs. (It should be noted that this diagram does not represent a complete visualisation of the new CSPIM outcomes, as modelled in Fig. 6.2. A full CSPIM visualisation includes presentation of all negative consequences for each intermediate, and the top event, along with the severity of outcome indicators and probabilities for each consequence.)

Subsequently, we make a major departure from traditional FTA by determining a composite probability for each MCS, which includes the likelihoods for triggered

**Fig. 6.3** Minimal cut set {*B*3, *B*4} and triggered events (in "red") according to CSPIM method

intermediate and top events. (Here it should be noted that intermediate event probabilities account for the MCS initiator probabilities.) Related to this, the intermediate event probabilities will vary among cut sets depending on the different initiators included in the cut set and activated pathways to the intermediate event. Therefore, it is not appropriate to directly transfer the *P E j* value from one cut set to another. We define *P Ekj* to represent the probability of *E j* in the *k*th cut set. The composite probability value accounting for all events associated with the cut set that can lead to negative consequences is then defined as *P(Ek )* for all *j*.

We then move on to calculation of the composite risk associated with the MCS. This step requires risk values for all intermediate and top events for the cut set at each level of severity of outcome, given as *Rkj*c = *P Ekj* · *P S*c|*Ekj* · *S*c. For each specific level of severity (*c*), the composite risk of event *E j* in the *k*th cut set is noted as *Rkj* . We can then obtain the total composite risk for the *k*th cut set at a given severity level as: *Rk* = *mk j*=1*Rkj* , where *mk* is the number of events (intermediate and top) associated with *k*th cut set. These risk values are then used to determine and rank importance ratios for all MCSs.

The importance of each MCS will vary for different outcome severity levels; consequently, all of these values need to be calculated separately. We define the importance of the *k*th MCS at the *c*th severity level as the ratio of the composite risk value for the cut set to the total composite risk value for the given system with negative outcomes caused by all triggered events occurring at the *c*th severity level: *Ik*c = *Rk*c *R*c , where *R*c is the composite risk level for the system across all cut sets for the severity level *c* and *Ik*c is the importance of *k*th cut set at the *c*th severity level. On the basis of this analysis, cut sets can be ranked in terms of importance (as in traditional FTA) and the ranking can be used for safety resource allocation.

Following an almost identical methodology, an analyst can calculate and evaluate the importance of each initiator in an MCS. This additional analysis is used to identify primary risk factors contributing to negative system outcomes and to further refine risk mitigation strategies and target control measures.

As with the event risk analysis, the DoD HRI can be applied to the calculated cut set and initiator risk values. The values are classified as low, medium, serious and high risks, revealing initiators that should be a focal point for control measures to reduce the probability of occurrence or severity of associated event outcomes.

#### **6.6 Conclusions**

The contributions of this work include identification of the need for enhanced SSA methods for comprehensive risk analysis in digital industry processes, and a stepby-step algorithm extending traditional FTA. This new method can exercise big data and/or process estimates to account for both the likelihood of hazard exposure and severity of event outcomes and preserves the scientific and systematic nature of the original FTA method. The CSPIM algorithm involves determining risks for all basic, intermediate and top events in a FT based on additional data sources from new AI methods and/or expert judgements and advanced analytical techniques. Implicitly, the algorithm addresses all consequences of intermediate and top events and leads to an overall system risk assessment. The CSPIM algorithm generates new FTA outputs that are sensitive to the severity of event outcomes and allows for prioritisation of fault events for safety controls on the basis of potential risk. These capabilities are currently absent from traditional FTA and substantially enhance the method for new digital industry applications.

Such enhanced SSA methods facilitate higher resolution/precision risk assessment and control formulation maximising industry effectiveness in use of safety resources. At the same time, the CSPIM approach can minimise misallocation of design and administrative measures for safety-critical systems. Consequently, our response to the specific research question identified in the Motivation Section is that it is possible to create new advanced tools for comprehensive safety-critical systems analysis that can accommodate/exploit additional "big" data on event consequences and likelihoods by starting from existing formal SSA techniques.

Although there are advantages to the CSPIM algorithm, the approach is mathematically complex and computationally intensive. In this work, we have only considered application to a small FT; however, in actual industrial applications, such as offshore oil rig design, a FT may literally have thousands of intermediate events included in pathways to an undesired top event of platform failure. In this case, the risk calculations as part of the CSPIM algorithm could be extremely large and complex. Related to this, the recent advances in computational technologies and AI methods may support application of such analysis of high-dimensional system structures, even in the presence of sparse historical data for event probability assignment. As described, expert subjective judgements can be used to generate valuable estimates to support application of such methods when faced with the curse of dimensionality. As noted earlier, in real-world FTA applications, the number of decision variables, or pathways to a top event, can exceed the data available for estimating the frequency of occurrence of specific initiators or intermediate events comprising pathways as well as the data for estimates of event consequences and likelihoods of consequences. New advanced analytical and AI-based methods can be used to generate additional field-relevant data to limit the impact of the curse of dimensionality of safety risk analysis and mitigation strategy decisions.

In general, more SSA tools like the CSPIM algorithm need to be created to support safety engineers in dealing with the broad challenges of digitalisation and algorithmic management in industry, which Le Coze and Antonsen [5] have identified. This represents a challenge for the safety science community, which may be addressed through new computational systems and AI tools that are currently being developed by computer scientists for big data analysis. For example, new AI-based sensing methods may be effective for generating field-relevant data for analysis with advanced risk assessment methods. In addition, new AI-based simulation methods make possible the development of "digital-twins" for highly realistic models of actual systems and processes that support testing of safety controls in advance of actual implementation. These are new digitalisation trends that will have a major impact on the future application of systems safety analysis methods.

6 Considering Severity of Safety-Critical System Outcomes in Risk … 63

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 7 Are We Going Towards "No-Brainer" Safety Management?**

**Nicola Paltrinieri**

**Abstract** Industry is stepping into its 4.0 phase by implementing and increasingly relying on cyber-technological systems. Wider networks of sensors may allow for continuous monitoring of industrial process conditions. Enhanced computational power provides the capability of processing the collected "big data". Early warnings can then be picked and lead to suggestion for proactive safety strategies or directly initiate the action of autonomous actuators ensuring the required level of system safety. But have we reached these safety 4.0 promises yet, or will we ever reach them? A traditional view on safety defines it as the absence of accidents and incidents. A forward-looking perspective on safety affirms that it involves ensuring that "as many things as possible go right". However, in both the views there is an element of uncertainty associated to the prediction of future risks and, more subtly, to the capability of possessing all the necessary information for such prediction. This uncertainty does not simply disappear once we apply advanced artificial intelligence (AI) techniques to the infinite series of possible accident scenarios, but it can be found behind modelling choices and parameters setting. In a nutshell, any model claiming superior flexibility usually introduces extra assumptions ("there ain't no such thing as a free lunch"). This contribution will illustrate a series of examples where AI techniques are used to continuously update the evaluation of the safety level in an industrial system. This will allow us to affirm that we are not even close to a "no-brainer" condition in which the responsibility for human and system safety is entirely moved to the machine. However, this shows that such advanced techniques are progressively providing a reliable support for critical decision making and guiding industry towards more risk-informed and safety-responsible planning.

**Keywords** Uncertainty · Safety assessment · Machine learning · Decision-making

N. Paltrinieri (B)

Department of Mechanical and Industrial Engineering, NTNU, Trondheim, Norway e-mail<EMAIL_ADDRESS>
#### **7.1 Introduction**

At the beginning of the 90 s, Prof. Diekmann [7] stated the following. "New analysis tools are emerging, which have the potential to allow complex risk analyses to be performed simply. These new tools, which are underpinned by decision analysis and, lately, expert-systems technology, may lead to powerful, yet simple, approaches to the representation of risky problems". This optimistic prediction on the future of risk analysis was accompanied by the suggestion of a possible interdisciplinary direction: "Future approaches to risk analysis will certainly rely more on the advances being made in Artificial Intelligence (AI) and the cognitive sciences. New computer tools and knowledge-representation schemes will unquestionably lead to new techniques, insights and opportunities for risk analysis".

In the same decade (1997), the Russian chess grandmaster Garry Kimovich Kasparov (former World Chess Champion, ranked world No. 1 from 1984 until his retirement in 2005) lost a chess game with IBM's chess playing computer Deep Blue, which was an example of Good Old-Fashioned Artificial Intelligence (GOFAI) [16]. On that game, [17] later stated the following: "Deep Blue was intelligent the way your programmable alarm clock is intelligent. Not that losing to a 10-million-dollar alarm clock made me feel any better".

Industrial risk analysis and safety management have tried to make use of AI, but they have unevenly progressed since the described events. They neither respected Diekmann's prediction (methodological gaps are still present [24]), nor turned into "programmable-alarm-clock intelligence" thanks to the progressive refinement of machine learning models and the increase in available computing power [12].

This contribution aims to outline what AI can bring to risk analysis and safety management by illustrating a series of examples (with emphasis on benefits and limitations) where AI techniques are used to continuously update the evaluation of the safety level in an industrial system.

#### *7.1.1 Artificial Intelligence and Machine Learning*

AI is intelligence demonstrated by machines, and it is divided into subfields based on technical considerations, such as particular goals (e.g., "robotics" or "machine learning"), the use of particular tools ("logic" or artificial neural networks) or deep philosophical differences.

This contribution focuses on the subfield of machine learning (ML). ML refers to techniques aiming to program computers to learn from experience [32]. Some of its models (e.g., deep learning) aim to simulate the learning model of the human brain [12]. Such models are composed of multiple processing layers to learn representations of data with multiple levels of abstraction.

A computer may be trained to assess risk for safety-critical industries such as Oil and Gas through these learning techniques. This would allow processing a large amount of information in the form of indicators from normal operations and past unwanted events (from mishaps to major accidents), which would be used for training. Due to the subjectivity of the definition of risk [40], a risk level cannot be assigned to each event with certainty and expert supervision is needed. Once the model has learned risk categorisation, it uses its knowledge to evaluate real-time risk from the state of the monitored system.

#### *7.1.2 Monitoring of Early Deviations and Past Events*

Increasing attention has been dedicated to monitoring safety barrier performance through indicators, as a way to assess and control risk. Indicators may report a series of factors: physical conditions of a plant (equipment pressure and temperature), number of failures of an equipment, maintenance backlog, number of emergency preparedness exercises run, amount of overtime worked, etc. A number of indicator typologies are theorised and used in the literature [23]. Øien et al. [23] affirm that we can refer to risk indicators if: they provide numerical values (such as a number or a ratio), they are updated at regular intervals; they only cover some selected determinants of overall risk, in order to have a manageable set of them. That being said, the latter feature is quickly becoming outdated due to the extensive collection carried out in industry and the attempts to process large numbers of them [30].

Øien et al. [23], Paltrinieri et al. [25, 26] and Landucci et al. [19] have produced several reviews on risk and barrier indicators. They show that the definition and collection of risk indicators have become consolidated practices in "high-risk" sectors, such as the petroleum and chemical industries. For instance, the Norwegian Petroleum Safety Authority (PSA) requires indicators describing the technical performance of safety barriers within the Norwegian Oil and Gas industry since 1999 [31], while the European directive "Seveso III" [9] on the control of majoraccident hazards involving dangerous substances suggests their use for sites handling hazardous substances [10]. Such a trend towards the definition and collection of higher numbers of indicators [30] demonstrates the mentioned challenge on big data processing for risk level assessment.

#### **7.2 Examples of AI-Based Prediction**

Three examples of AI-based prediction with safety-related purposes are described in the following. The cases depict not only the application of machine learning techniques, but also the criticality of input data and implicitly the human efforts in preparing the data.

#### *7.2.1 Consequence Class Associated with a Hazardous Material Release*

ML techniques were applied by Paltrinieri et al. [28, 29] to a database of past accidents with the purpose of simulating its application on the national databases managed by the Seveso competent authorities. The data set used is the Major Hazard Incident Database (MHIDAS) [1] launched by the UK Health and Safety Executive in 1986 and developed by AEA Technology until the mid-1990s. The events included are based on public domain information sources, and their characteristics are registered using keywords.

MHIDAS includes about 8972 hazardous events from 1916 to 1992, with the attributes listed in Table 7.1. Some attributes use a taxonomy to systematically categorise the event. While the actual quality of the data could not be fully verified across the recorded hazardous events, this database is characterised by a high quality of data model, i.e., high semantic quality allowing for clear boundaries and relevant properties of the problem domain and the requirements of the task. Given that it takes a high amount of creativity and vision to design a solution that is robust, usable and can stand the test of time [15], the high semantic quality of MHIDAS could only be reached by significant knowledge and experience of the field.

The attributes listed in the upper part of Table 7.1 were used as inputs to the ML models to predict the consequences—lower part of Table 7.1. The details of data preprocessing are explained elsewhere [34]. The study focused on the number of

**Table 7.1** Attributes used to record hazardous events in MHIDAS [1]

Specific keywords are used to describe some of the attributes

people killed and aimed to predict the occurrence of a hazardous event within one of the severity categories listed in Table 7.2 based on the considered inputs. Only categorical data are used.

#### *7.2.2 Wellhead Damage Frequency in a Drilling Rig*

To avoid potential damage during drilling operations for a new offshore Oil and Gas well, a semisubmersible drilling unit should maintain its position above the wellhead. This is particularly critical if the platform is in shallow waters, where small changes of position lead to higher riser (pipe connecting the platform to the subsea drilling system) angles. Exceeding physical inclination limits may result in damages to the wellhead, Blowout Preventer (BOP—sealing the well) or Lower Marine Riser Package (LMRP—connecting riser and BOP) [5].

Platform position is maintained in an autonomous way (without mooring system) by a set of thrusters controlled by the Dynamic Positioning (DP) system. Input for the DP system is provided by the position reference system (Differential Global Positioning System—DGPS and Hydroacoustic Position Reference—HPR), environmental sensors, gyrocompass, radar and inclinometer [5]. A Dynamic Positioning Operator (DPO) located in the Marine Control Room (MCR) is responsible for constant monitoring of DP panels and screens and carrying out emergency procedures if needed [11]. Platform position may be lost due to several reasons.

In this case study, Paltrinieri et al. [29] assume that the platform thrusters exercise propulsion in a wrong direction, leading to a "drive-off" scenario. If the rig moves to an offset position, specific alarms turn on and suggest that the DPO stop the drive-off scenario by deactivating the thrusters and initiate the manual Emergency Disconnect Sequence (EDS) to disconnect the riser from the BOP. If the manual EDS fails, the automatic EDS activates at the ultimate position limit allowing for safe disconnection [5].

A number of works [21, 24, 26] address the details of occurrence and development of drive-off scenarios. Relevant indicators are defined to assess the performance of safety barriers and related systems. Examples of these indicators are the following.

Collection of a wide variety of indicators may lead to challenges related to data integrity. Lack of accurate data may be due to several reasons, such as time and financial constraints experienced by database managers responsible for recording relevant indicators. As companies are expected to do more with less, developers must make decisions about the extent to which they are going to implement and evaluate quality considerations [15].

Simulations of drive-off indicator trends for a period of 30 years can be found in the literature [24]. They are inspired by the typical bathtub curve for technical elements [41] and relevant expert judgement for the remaining elements.

As shown by Bucelli et al. [4], indicator values may be aggregated based on relative weights and hierarchical barrier models, in order to enable dynamic update of barrier failure probabilities. This can be used to update, in turn, occurrence frequencies of potential outcomes. Outcome frequencies are an expression of the scenario probability and, in turn, of the risk. If we assume that the other factors are constant, this represents a simplified risk model. However, Matteini [21] points out a certain complexity within the hierarchical barrier model, which may be due to a tangled structure and an unclear approach to assign relative weights to single model elements. For this reason, a machine learning approach bypassing the construction of such hierarchies and aggregation rules is suggested by Paltrinieri et al. [29].

#### *7.2.3 Alarm Chattering in an Ammonia Plant*

Alarm data from a section of an ammonia production process [39] are analysed by Tamascelli et al. [38]. Due to the large quantity of hazardous substances stored and handled during normal activity, the plant has been classified as an "upper tier" Seveso III establishment. Extensive use of methane, hydrogen and ammonia (anhydrous and aqueous solution) occurs in the plant section. Furthermore, due to the intrinsic properties of the processes involved, severe operating conditions (i.e., high pressure and high temperature) are often associated with corrosive substances. Additional information about ammonia production and the considered site can be found at: [2, 42].

The alarm database consists of alarm data collected during an observation period of more than four months. In this case, both data and data model are of high quality as they are acquired from consolidated monitoring systems. Human effort would instead reside in the interpretation and the definition of appropriate priorities among the provided data.

Each row of the database represents an alarm event (26,473 observations in total), and each column (36 in total) represents a piece of information (i.e., an "attribute") about the alarm. The most meaningful attributes are presented in Table 7.3.

**Table 7.3** Alarm database attributes

The Alarm Identifier (point 5. of the "Message" attribute) is a code that defines the alarm status. Examples of Alarm Identifiers are "HHH" (which means that the measured variable has exceeded the "high level" setpoint), "HTRP" (the measured variable has exceeded the "very high level" alarm setpoint and automatic block intervention procedures might be triggered), "IOP" (which indicates an instrumental failure or out-of-range measure), "LLL" and "LTRP" (same as "HHH" and "HTRP" but referring to a "low/very low level").

According to [18], an alarm event is uniquely identified by three attributes only: Time Stamp, Source, and Alarm Identifier (e.g., HHH, HTRP, LLL, LTRP, etc.). The combination of a "source" and an "alarm identifier" is called a "unique alarm".

More than 96% of the alarms registered in the database occurred within one month only, when a considerable number of floods and chattering alarms must have occurred. In fact, only ten alarm sources (out of 194 in total) were responsible for more than 80% of the alarms recorded.

Chattering alarms are alarms "that repeatedly transitions between active state and inactive state in a short period of time" [3]. Therefore, chattering alarms have the potential to produce a large count of alarms and reducing their number is a key step to improve the performance of the alarm system during alarm floods.

Kondaveeti et al. [18] proposed a method for quantifying alarm chatter based on run lengths distributions. Although effective, this technique produces static results (i.e., chattering is quantified based on historical alarm data, but no conclusion can be drawn about the alarm's future behaviour). This Chattering Index approach is modified by Tamascelli et al. [38] to predict chattering behaviour by means of standard ML models.

#### **7.3 Method**

ML classification models were used for the three examples in Sect. 7.2. Moreover, comparison among different ML models is also beneficial. Results from multiple linear regression (MLR) were compared to the relatively more sophisticated deep neural network (DNN) models.

Both MLR and DNN aim at modelling the relationship between two or more independent feature variables and a label dependent variable. While the former model fits a linear equation to observed data, the structure of the latter model is similar to the organisation of neurons in the brain, arguably the most powerful computational engine known today [20].

An algorithm uses part of the available data to train the ML model to predict the specific label variable based on the feature variables and test the result on the remaining data. Model performance needs to be evaluated before employing it for actual applications. The result might be far from perfect, and this may be due to poor data quality or indicate the need to tune the model to the actual application.

#### *7.3.1 Metrics*

The performance of the classification models used is assessed during the evaluation phase. As an example, consider a situation where accidents must be classified into two classes A or B. A positive prediction occurs when the model predicts the class A. Instead, a negative prediction occurs when the model predicts the class B. Whenever the model predicts the class of an object, there are four possible outcomes:

The sum of True Positives and True Negatives represents the number of correct predictions, while the sum of False Positives and False Negatives indicates the number of wrong predictions. True Positives, True Negatives, False Positives, and False Negatives are used to obtain three performance indicators:

$$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} \tag{7.1}$$

$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} \tag{7.2}$$

$$\text{Recall } = \frac{\text{TP}}{\text{TP } + \text{FN}} \tag{7.3}$$

Accuracy represents the fraction of objects that have been correctly classified. Precision indicates the success rate of a positive prediction. Recall denotes the fraction of actual positives that have been correctly identified.

It is worth mentioning that metrics and indicators depend on the probability threshold used by the classification models. For example, if the decision threshold is lowered, the model may produce more positive predictions. As a result, the Recall might increase, but the Precision might decrease [33]. In fact, actions aimed at increasing Recall often lower the Precision, and vice versa [13]. A convenient mean of displaying the effect of the decision threshold is the Precision-Recall curve, i.e., a plot where each point represents the couple Precision vs. Recall at a specific decision threshold [22]. A convenient mean of summarising the information in the Precision– Recall curve is the area under the curve (AUC P-R) [22], which takes values between 0 and 1. Being independent on the decision threshold, the AUC PR is considered a more comprehensive indicator of the model performance if compared with Accuracy, Precision and Recall. In general, a large AUC P-R value indicates good performance [33].

#### **7.4 Results and Discussion**

Table 7.4 summarises all the results from the examples described in Sect. 7.2. The results from the two approaches used (MLR and DNN) are directly compared to identify the best predictive performance. MLR shows a higher number of higher values in green cells (9) if compared to DNN (6).

However, this overall result cannot convey the message that MLR performs better than DNN as "there ain't no such thing as a free lunch". In fact, in these examples, DNN was applied with default parameters (e.g., number of layers and nodes

**Table 7.4** Summary of results from the representative examples of ML application for safety purposes

*C* stands for consequence class, *WDF* stands for wellhead damage frequency, *AC* stands for alarm chattering, and *PR AUC* stands for the area under the precision recall curve. Greed and Red cells respectively show higher and lower values when compared with the other approach

suggested by Tensorflow tutorials [13]). In addition, DNN is relatively more sensitive to poor quality of data [24].

Table 7.4 reports all the metrics discussed in Sect. 3.3. If we exclusively focus on accuracy, we notice that the highest value (0.99) is obtained for both MLR and DNN predictions of the consequence class 10–100 fatalities associated with a hazardous substance release. However, accuracy alone is not informative if the problem involves the identification of rare classes, i.e., when the dataset is class imbalanced [14].

Releases of hazardous substances with 10–100 fatalities are (fortunately) rare events as they represent about 1% of the records in the MHIDAS database. In this case, the models have learned that the result will be correct 99% of the times if they predict that this kind of event never occurs. If the cost of a False Negative is higher than the cost of a False Positive (such as the case of a release of hazardous substance with 10–100 fatalities), Recall is the most meaningful metric. In this context, a good model must produce high Recall, while low precision might be considered acceptable and, to a certain extent, conservative.

The prediction of events with a relatively higher frequency and lower consequence (e.g., a release of a hazardous substance with no fatalities, an increase of wellhead damage frequency or a chattering alarm) may instead benefit from higher precision at the expense of the recall value.

For this reason, rather than considering Precision and Recall individually, one may aggregate them into the so-called F-score [6], especially if the area under the precision recall curve indicates the potential of optimising the model by tuning the decision threshold probability. Human contribution would again come into play in the setting of the algorithm parameters, which would inevitably represent a form of subjective calibration. For this reason, the techniques used in the depicted examples require a deep understanding of their benefits, limitations and application boundaries.

This contribution aims to convey the message that AI-based techniques must be considered as tools supporting and not substituting decision making. Awareness and knowledge of these tools' properties by the user are essential to effectively exploit their results. The role of the human as user of these tools is even more central than before. AI should not be intended as a way to replace the human, but only as an improved approach assisting the human. This is compatible with the concept of trustworthy AI by the European Commission [8] promoting explainable AI (XAI), human centrality by means of interpretability, infobesity (overload of information) avoidance and transparency.

Embracing the principles of trustworthy AI and XAI will unlock the vast potential of machine learning in safety management, especially considering emerging variants of the traditional approaches described in this contribution, such as:

• **Meta-learning**, focus on the learning model and its optimisation towards new observations, in order to apprehend the emergence of unknown scenarios (e.g., unknown risks [35, 36]).

Machine learning has the potential to be eventually capable of supporting human users as [7] states. However, the author must admit the presence of another important challenge ahead that is yet to be fully overcome: ensuring appropriate safety culture by the user, i.e., foundations and motivations for which such advanced tools would be used. Once again, this challenge brings the discussion back to humans. Risk and safety analysts and managers would potentially have an advantage in the application of digitalised safety management due to their predefined state of mind, but only given their willingness to learn the basics and use of such advanced and promising techniques.

#### **7.5 Conclusion**

This contribution has illustrated examples of AI-based prediction used to continuously update the evaluation of the safety level in an industrial system. The examples refer to the impact prediction of a hazardous substance release in chemical industry, the wellhead damage frequency in offshore oil and gas and chattering alarms in ammonia production. The results can and must be read on different levels, carefully considering the available metrics based on the scenario addressed. This shows that we are not (and will not be in a near future) in a "no-brainer" condition in which the responsibility for human and system safety is entirely moved to the machine. At the same time, an understanding of digital solutions will be progressively required to guarantee their effective application. These advanced techniques have the potential to provide reliable support for critical decision making, guiding industry towards more risk-informed and safety-responsible planning.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 8 Looking at the Safety of AI from a Systems Perspective: Two Healthcare Examples**

**Mark A. Sujan**

**Abstract** There is much potential and promise for the use of artificial intelligence (AI) in healthcare, e.g., in radiology, mental health, ambulance service triage, sepsis diagnosis and prognosis, patient-facing chatbots, and drug and vaccine development. However, the aspiration of improving the safety and efficiency of health systems by using AI is weakened by a narrow technology focus and by a lack of independent real-world evaluation. It is to be expected that when AI is integrated into health systems, challenges to safety will emerge, some old, and some novel. Examples include design for situation awareness, consideration of workload, automation bias, explanation and trust, support for human–AI teaming, training requirements and the impact on relationships between staff and patients. The use of healthcare AI also raises significant ethical challenges. To address these issues, a systems approach is needed for the design of AI from the outset. Two examples are presented to illustrate these issues: 1. Design of an autonomous infusion pump and 2. Implementation of AI in an ambulance service call centre to detect out-of-hospital cardiac arrest.

**Keywords** Artificial intelligence · Healthcare · Safety · Systems perspective

#### **8.1 Introduction**

There is a lot of excitement about the potential benefits that the use of artificial intelligence1 (AI) can bring to healthcare. Health systems are struggling with rising costs, staff shortages and burnout, an increasingly elderly population with more complex health needs, and health outcomes that often fall short of expectations.

M. A. Sujan (B)

1 The term artificial intelligence, in the widest sense, refers to the science and engineering of intelligent computer systems. In this paper, the focus is mostly on AI applications that use machine learning. Machine learning refers to the use of computer algorithms that learn from data through supervised learning, unsupervised learning or reinforcement learning approaches.

Human Factors Everywhere Ltd., London, UK

AI is seen as the next step in addressing these challenges, with the hype so high to prompt leading US digital medicine researcher Eric Topol to compile an amusing list of "outlandish expectations" of AI, such as: the ability to diagnose the undiagnosable; treat the untreatable; predict the unpredictable; classify the unclassifiable; eliminate workflow inefficiencies; and cure cancer [29].

There is certainly a strong appetite among governments around the world to promote the use of AI in healthcare. In the UK, a dedicated body (NHSX2 ) has been set up with a remit to accelerate the digital transformation of the National Health Service (NHS) and to support the development and integration of AI applications into the NHS.

Examples of the potential benefit that AI can bring to healthcare can be found readily in news reports and in the scientific literature. Over 200 AI-based medical devices have already received regulatory approval in Europe and the USA [16], and there are many more AI applications that do not require such approvals (i.e., which fall outside the narrow definition of medical devices). The area of diagnostics is particularly strong with examples including AI applications to support identification of diabetic retinopathy [1], skin cancer [7], and, recently, to distinguish COVID-19 from other types of chest infections [12]. Other developments include, for example, ambulance service triage, sepsis diagnosis and prognosis, patient scheduling, and drug and vaccine development.

While these studies provide encouraging results, the evidence base remains weak for several reasons: the focus of the evaluation is usually on a narrowly defined task; the evaluation is typically undertaken retrospectively by the technology developer, and independent evaluation remains the exception; the number of human participants tends to be small; and prospective trials are still infrequent. Taken together, claims that AI outperforms humans are likely to be overstated given the limitations in the study design, reporting and transparency, and the high risk of study bias [17].

The real challenges for the adoption of AI in healthcare will arise when algorithms are integrated into health systems to deliver a service in collaboration with healthcare professionals as well as other technology. It is at this level of the wider system, where teams of consisting of healthcare professionals and AI applications cooperate and collaborate to provide a service, that safety challenges will need to be addressed [26].

The aim of this paper is to review and highlight some of the safety challenges at the system level relevant to the use of AI in healthcare settings by looking back at what we already know from the extensive research on automation, as well as what novel challenges might need further attention. Two examples are described (1. Design of an autonomous infusion pump and 2. Implementation of AI in an ambulance service call centre to detect out-of-hospital cardiac arrest) to illustrate the types of design and implementation issues that should be considered.

2 NHSX merged into NHS England's Transformation Directorate in February 2022.

#### **8.2 Challenges Old and New**

Many of the challenges with using AI in healthcare are actually very familiar. Back in the 1970s and 1980s, industrial systems saw the widespread introduction of automation to improve efficiency and to reduce failures attributed to human error. This was soon accompanied by research studying failures involving highly automated systems, which highlighted the potential for "automation surprises" [22] and the "ironies of automation" [2]. Problems with automation can arise because people are not actually eliminated from the system, but instead the automation changes the nature of the work that people do [18], often resulting in a set of tasks, which were left over by the developers of the automation. This can make the human role and the interaction between people and automation challenging, e.g., due to lengthy periods of monitoring, the need to respond to abnormal situations under time pressure, and the difficulty of building an understanding of different situations and strategies for their management.

However, modern AI systems (especially those that are increasingly autonomous) also present completely new challenges that were not as relevant in the design of traditional automated systems. AI systems can augment what people do in ways that were not possible when machines simply replaced physical work. The interaction with interconnected AI-based systems could potentially develop more into a relationship between people and the AI, especially where the AI has means of expressing something akin to a personality via its interfaces [11]. Social aspects will become much more relevant, as well as mutual understanding of expected behaviours and norms. We might think of, for example, the seemingly ubiquitous voice-enabled virtual assistants (e.g., Amazon's Alexa or Apple's Siri) that aim to deliver a realistic and natural social interaction experience. Examples from healthcare might include mental health chatbots and assistive robots. These relationships between people and technology, along with social, cultural and ethical aspects have much greater importance for future AI-based systems than for traditional automation. Healthcare professionals, patients and AI will increasingly collaborate as part of the wider clinical system.

The use of healthcare AI also raises ethical challenges on a much wider and more fundamental scale compared with traditional automation. For example, concerns about privacy and data protection have come to the fore, such as the controversy and subsequent litigation around the transfer of 1.6 m identifiable confidential electronic patient records from the Royal Free London NHS Foundation Trust to Google subsidiary Deep Mind in 2015. This data transfer was within the scope of a collaboration to develop a tool to support the identification of patients at risk of developing acute kidney injury (which was subsequently abandoned), but the data sharing agreement did not impose any explicit bounds on the use of these patient records. In addition, wider issues around fairness and impact on different stakeholder groups need to be considered, such as racial bias and disparities in accuracy across different population groups [30]. Many data sets are representative of more affluent health systems, and are, therefore, at risk of disadvantaging ethnic minority and vulnerable groups. Fairness at the health system level can also go beyond issues of bias in training data. For example, the use of AI-based patient-facing symptom checkers paired with remote consultations (such as the UK "GP at hand" service offered by Babylon Health) can potentially disadvantage elderly patients and those with significant healthcare needs by shifting and depleting the budget allocated to primary care: these services are typically attractive to younger, healthier populations, leaving traditional primary care services to care for more complex cases with a significantly reduced budget. It is important to note that addressing such concerns requires a broader range of expertise and a social and political dialogue to advance health equity in the age of AI [23].

Designers of AI and healthcare organisations deploying AI should be aware of these critical considerations at the systems level. Examples from the extensive literature have been summarised in a recent White Paper published by the UK Chartered Institute of Ergonomics and Human Factors and include (not an exhaustive list) [27]:

performance monitoring, back-up behaviour, adaptability, and team orientation) and supporting mechanisms (shared mental models, mutual trust, closed-loop communication) to enhance human–AI teaming. The design should consider how human team members can understand the AI's roles and responsibilities, and more challenging—how the AI can understand the human's roles and responsibilities, e.g., in dynamic AI applications that take over human tasks when people are at risk of being overloaded. It is also important that appropriate mental models are shared across human and AI team members.

Below, two examples are described to illustrate the type of considerations that follow from this line of systems thinking for the design and use of healthcare AI.

#### **8.3 Example 1: AI-based infusion pumps for IV medication administration**

This first illustrative example is taken from a project that studied safety assurance challenges of the use of autonomous infusion pumps (i.e., infusion pumps driven by AI) for intravenous (IV) medication administration in intensive care [25]. The purpose of the research was to make recommendations that could feed into the design and implementation of the autonomous technology in such a way that its use enhances rather than diminishes safety.

It has been estimated that as many as 237 million medication errors occur in England every year, and that these cause over 700 deaths [6]. Intravenous medication preparation and administration are particularly error prone. The introduction of highly automated and ultimately autonomous IV medication management systems might contribute to reducing these error rates by taking over functions previously carried out by clinicians, such as safety cross-checks (e.g., patient identity and prescription), calculating infusion rates and independently adjusting infusion parameters based on the patient's physiology. A large UK-US study found that whether or not infusion technology successfully improves patient safety depended largely on the specific context of implementation within the clinical system [3].

The project was carried out in an English NHS hospital, serving a population of 600,000. The intensive care unit (ICU) within the hospital has 16 beds and cares for 1300 patients annually. Patients on ICU are, by default, very ill. Patients can be on life support machines, such as ventilators, and they typically require a significant number of drugs. Some of these drugs are given intravenously via an infusion pump. The infusion pump controls the flow of the drug. The traditional set-up is that a doctor (or clinician with prescribing privileges) prescribes a drug as part of the patient's treatment plan, and a nurse then needs to prepare the drug syringe, load the infusion pump with the drug syringe and then program the infusion pump to run at the required infusion rate for a specific duration. This is the baseline scenario used for illustration in this paper. A more comprehensive description of the analysis is given in [8].

Interviews with patients, healthcare professionals and individuals with responsibility for procurement, IT integration and training were undertaken, as well as an analysis of existing working practices. The interviews and analysis helped to anticipate and explore potential implications for the design of the AI and the impact on the wider clinical system, such as:

These considerations at the level of the clinical system can support designers of AI applications in defining the operating environment and in understanding relevant interactions with people, other tools and systems, other tasks that might be relevant and the characteristics of the local work environment.

#### **8.4 Example 2: AI to support the recognition of out-of-hospital cardiac arrest**

The second example is concerned with the implementation of an AI support system in an NHS ambulance service to improve the recognition rate of and time to recognition of out-of-hospital cardiac arrest (OHCA). Currently in the UK, approximately 30,000 people sustain an out-of-hospital cardiac arrest (OHCA) annually, and survival to hospital discharge ranges from 2.2 to 12% [20]. Early defibrillation within the initial 3–5 min could deliver survival rates of 50–70%, but each minute of delay to defibrillation reduces the probability of survival by 10% [19]. Hence, speedy recognition of OHCA by the ambulance service call handler is crucial to support bystander cardiopulmonary resuscitation and to enable fast paramedic attendance at the scene. However, recognition of OHCA is difficult, because signs can be subtle, and the international evidence demonstrates that around 25% of OHCA are not picked up by call centre operators [5].

**Fig. 8.1** Different levels of automation and interaction in the implementation of AI support for the recognition of out-of-hospital cardiac arrest3

An AI system to support ambulance service call handlers in recognising OHCA has been developed by a Danish manufacturer, and initial independent retrospective evaluation with data from Copenhagen produced encouraging results demonstrating that the AI system had higher sensitivity than call handlers (84% vs. 75%), but slightly lower specificity (97% vs. 99%) [5]. However, a subsequent randomised controlled trial of the technology in use found that the AI support did not lead to improved recognition of OHCA [4].

Again, this reinforces the need for consideration of the wider system when designing and implementing AI technology. Taking a systems perspective can help understand the breadth of design decisions and their potential impact, especially when considering the implementation of the AI tool in a different context (in this case using the technology in a more rural environment as opposed to the urban environment where it was initially tested). For example, the interaction between ambulance service call handler and the AI can be designed to accommodate different levels of support (or levels of automation); see also Fig. 8.1:

• No AI support (current situation).

3 Figure from the white paper "Human Factors and Ergonomics in Healthcare AI", 2021, reproduced with permission from the UK Chartered Institute of Ergonomics and Human Factors. All rights reserved.

Interviews with ambulance service staff and a cognitive task analysis [14] of call handlers' tasks suggested that different types of interaction design might have far-reaching consequences that need to be considered:

In addition to the above questions about the interaction between the AI and the call handler, it is also not clear how the AI best augments what the call handlers do, i.e., how to support call handlers with difficult decisions. For example, the cognitive task analysis identified issues that make OHCA recognition more challenging as well as strategies that call handlers employ to overcome these difficulties:

#### **8.5 Conclusion**

The aspiration of using AI to improve the efficiency of health systems and to enhance patient safety requires a transition from the predominant technology-centric focus that contrasts people and AI ("humans vs. machines") towards a systems approach that considers AI as part of the wider health system.

Several lessons can be learned from research and practical experiences with the design and operation of highly automated systems. However, advanced AI systems also present novel challenges around social and relational aspects, and human–AI teaming. Addressing these requires a multidisciplinary approach as well as a broader political and societal dialogue around fairness and values in algorithms. This should be reflected in policies of research funding bodies and regulators, because funding specifications and regulatory frameworks frequently only reflect the technologycentric perspective of AI rather than reinforcing a systems approach.

There is a need to raise awareness of these issues among healthcare stakeholders, because Human Factors and Ergonomics (HF/E) and safety science, which advocate a systems approach, are currently not sufficiently well embedded in health systems.

**Acknowledgements** This work was supported by the Assuring Autonomy International Programme, a partnership between Lloyd's Register Foundation and the University of York (HF/ AI and ASSIST demonstrator grants).

**Ethics Statement** The study described in Example 1 received institutional approval at the participating NHS hospital as a service evaluation study. Prior to the interview, potential participants received a participant information leaflet. Interviews took place in a meeting room at the hospital (patients and hospital staff), over the telephone or at the business offices of the interview participant. Participation was voluntary, and all participants provided written consent. The study described in Example 2 was approved by the Health Research Authority and Health and Care Research Wales (IRAS reference 21/HCRW/0002).

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 9 Normal Cyber-Crises**

**Sarah Backman**

**Abstract** Despite an increasing scholarly interest in cyber-security issues, the phenomenon of large-scale cyber-crises affecting critical infrastructure is largely unexplored. While some characteristics of its consequence dynamics have been identified—prominently its transboundary features—the underlying conditions that allow such dynamics to unfold have not yet been thoroughly explored. This chapter aims to contribute to bridging this gap by applying the classical theoretical perspectives of Normal Accidents (NA) and High Reliability organisations (HRO) on the sociotechnical systems of modern critical infrastructure. It argues that NA characteristics (the combination of interactive complexity and tight coupling) can be found in multiple layers of critical infrastructure operations (technical, cognitive, organisational and macro). Implications are discussed in terms of its connection to transboundary crisis dynamics.

**Keywords** Cyber-security · Cyber-crises · Critical infrastructure · High reliability organisation theory

#### **9.1 Introduction**

Globalisation and fast-paced technological innovation have given rise to a worldwide digitalisation over the last 20 years. The impact of the internet and cyberspace on modern societies is so vast that some commentators argue that we have entered a new era and "cyber age", characterised by rapid change [1]. In this development, cyberspace has become an enabler of economic and social prosperity, but also of vulnerability and transnational security concerns. An especially serious concern has been centred around the increasing connectivity and interconnectedness of critical infrastructures. These worries have been enhanced by high-profile cyber-incidents

S. Backman (B)

J.-C. Le Coze and S. Antonsen (eds.), *Safety in the Digital Age*, SpringerBriefs in Safety Management, https://doi.org/10.1007/978-3-031-32633-2\_9

Department of Economic History and International Relations, Stockholm University, Stockholm, Sweden

e-mail<EMAIL_ADDRESS>
and crises, including the NotPetya attacks in 2017 affecting organisations in over 60 counties including the global transport and logistics firm Maersk, the pharmaceutical giant Merck and the National Bank of Ukraine, the WannaCry attacks on the UK National Health Services in 2017 and the ransomware attack on an American oil pipeline system in 2021.

Meanwhile, research contributions on the phenomenon of large-scale cyber-crises have been surprisingly thin and scattered within the field of crisis management research and beyond, reflecting the state of macro-level cyber-security research in general. Despite an increasing cyber-security interest within research fields dealing with national and international security affairs, attempts to theorise this essentially complex and multidisciplinary issue have been fragmented. As noted by Green [2], this fragmentation is problematic because of the "blind spots" of each perspective. "Thus lawyers have little idea of the technology that they are trying to regulate, strategists do not pay enough heed to the wider ethical and legal implications of acts of interstate cyber-aggression; and computer scientists delineate the intricacies of the technology with little focus on its political and strategic implications" [2, p. 3]. In other words, there is a shortage of comprehensive (although necessarily less detailed) scholarly perspectives on cyber-security issues in general, and cyber-crises in particular.

Although still relatively slim, the current body of literature focusing on cybersecurity at a macro-level has illuminated some important aspects of the consequence dynamics of cyber-crises. One is that cyber-crises (as cyber-security issues in general) tend to blur important dichotomies, including internal/external, technical/strategic and civilian/military, making them difficult to analyse, frame and conceptualise [3, 4]. This has resulted in, for example, early national cyber-crisis strategies that differed significantly from each other, despite facing largely the same challenges [5].

Another aspect identified in the literature is that the consequences and response efforts of cyber-crises tend to be characterised by transboundary-ness, to the extent that cyber-crises can be conceptualised specifically as transboundary crises [6, 7]. For example, this manifests in moments of rapid escalation after moments of slow development and a quick increase of involved and affected actors [8]. These consequence dynamics tend to put substantial stress on national crisis management structures that were not built with transboundary crises in mind [9, 10]. Previous cases have highlighted that response challenges that are challenging in any crisis (especially if the crisis shows transboundary features), like performing joint sense-making, coordination of response efforts and effective crisis communication, become even more challenging in cyber-crises. Not least because of the added complexity of a crisis which essentially involves technical matters, but has societal consequences [6, 7].

However, less attention has so far been placed on understanding the underlying conditions that allow this transboundary dynamic to unfold. This chapter begins to address this research gap, drawing upon the classic theoretical perspectives of Normal Accidents (Perrow) and High Reliability organisations [11, 12].

#### **9.2 Normal Accidents and High Reliability Organisations**

Since the publication of Charles Perrow's book Normal Accidents in 1984, the idea that tightly coupled complex systems will inevitably cause accidents (making them "normal") has been highly influential across many academic disciplines, especially those concerned with technology, risk, safety and crisis.

The idea of Normal Accidents (NA) is based upon the combination of two system conditions: interactive complexity and tight coupling. System in this sense is loosely defined and can refer to a computer system as well as an organisational system. Interactive complexity of a system starts with a lot of different components. These components can be technical parts (software or hardware, for instance), but they can also refer to procedures or human operators. Within this setting, failures among system components can interact in unexpected ways. Due to the complexity of the system and all its components, few if anyone (designers of the system included) can predict the many ways that failures in different components can interact and the consequences of these interactive failures [13]. In itself, interactive complexity is not a major problem, unless combined with what Perrow refers to as "tight coupling". If a system is complex but not tightly coupled, it means that even if failures interact in unexpected ways within this system, there is enough "slack" within the system to have time to figure out how to do things in other ways (where it is possible to do things/operate the system in other ways). When a system is both complex and tightly coupled, it means that the interactive failures, for some reason, cannot be isolated from each other, and that there is no alternative way of operating. This means that the disturbances within this system will spread quickly and "cascade" [14].

When the integral system characteristics of interactive complexity and tight coupling are present, accidents will inevitably (although perhaps rarely) happen due to multiple and unexpected interaction of failures. According to Perrow, neither new technological solutions nor better organisation can totally undo this dynamic, since the added complexity (either organisational or technological) from these "fixes" will then be part of the possible interactive failures within the system [14]. Decentralisation is required to deal with unexpected interactions in tightly coupled and complex systems. The problem is that systems cannot be decentralised and tightly coupled at the same time and there are strong economic incentives to keep and extend the tight coupling [14, 15].

While the perspective of NA drew attention and gained popularity both within and outside of academia in the 1990s, critical reactions and perspectives also emerged. One of the prominent stemmed from Todd La Porte and a group of Berkeley researchers. These scholars highlighted the fact that some organisations experience virtually no accidents despite the presence of interactive complexity and tight coupling (referred to as High Reliability organisations, or HROs), thus challenging the idea that organisational "fixes" cannot prevent accidents. A common finding of this research has been that HROs seem to allow flexible and decentralised decision making, have strong external preferences for failure-free operations and invest heavily in reliability improvement, including redundancy and training. The cost of failure is high in these organisations [15]. Bierly et al. [16] argue that HROs in general share two main characteristics, besides interactive complexity and tight coupling, that set them apart from other organisations: catastrophic potential (which increases scrutiny and expectations of accident-free operations) and accountability (linked to clear areas of responsibility, control and expectations of performance) [16].

In more recent contributions of NA application, two main trends can be identified. The first is the notion that despite some differences, the perspectives of NA and HRO can largely be viewed and used as complementary to understand the complex dynamics of accidents and safety in high-risk systems [15, 17]. The second departs from the observation that the world becomes ever more interconnected and complex, with global, multiorganisational, large-scale systems that are managed by a plethora of private and public actors. Thus, contributions within this trend aim to extend the classical theoretical arguments of both NA and HRO beyond technological systems and organisations to the macro-level, or the level of "organisation of organisations". This chapter draws on both trends.

The main argument of this chapter is that the consequence dynamics of largescale cyber-crises, characterised by their transboundary nature, can be explained by NA-dynamics in several layers of the sociotechnical systems that comprises modern critical infrastructure operation. Through five interviews with senior experts on cybersecurity and critical infrastructure and two case examples (the incident involving the Ukraine power grid in 2015 and the Kaseya incident in 2021), this chapter will explore how these dynamics can manifest on several layers of critical infrastructure operation. In doing so, this chapter aims to contribute to bridging the gap between the understanding of the consequence dynamics of cyber-crises and the structural conditions in sociotechnical systems of critical infrastructure that allow them to unfold and cascade as observed in previous research [3, 6, 7]. The interviewees included senior experts from Sweden, the UK and the USA, all with a background of working with national level cyber-security and critical infrastructure protection. Beyond the interview data, this chapter also used media reports and official reports as material.

The analysis in the following empirical sections will be loosely structured around four analytical categories, or layers, of NA application, identified by Le Coze [17]: 1. Technology, 2. Cognition, 3. Organisation and 4. Macro.

#### **9.3 Analysis**

#### *9.3.1 Technology*

The first category of NA application, as relevant for critical infrastructure operations, is technology. Perrow realised rather early the applicability of NA to the internet, which is essentially composed of technical systems (both hardware and software) with interdependent components in interactive complexity [see for example Perrow [18]]. However, the extent of digitalisation of society and the development of ICS (Industrial Control Systems) in critical infrastructure has come a long way since then. Today, an analysis of the NA dynamic applied to critical infrastructure calls for a focus on the problem of legacy code, systems and hardware.

A legacy system can be defined as "An information system that may be based on outdated technologies but is critical to day-to-day operations. As enterprises upgrade or change their technologies, they must ensure compatibility with old systems and data formats that are still in use" [19].

In modern critical infrastructure, it is not uncommon to have legacy systems and code more than 15 years old underpinning operations. Some legacy code is written in old and outdated programming languages, like COBOL, which relatively few programmers know today. Moreover, many legacy systems were built without security in mind, making security considerations an "add on" aspect, or afterthought. When the systems underpinning ICS in critical infrastructure are built upon layers upon layers of legacy code, and systems written in a variety of code languages, with numerous add-ons to make them compatible, interactive complexity is continuously built into the system as a whole. Thus, components with the potential to fail, and interact with other failing components in unexpected ways, are continuously added.

Getting rid of legacy code, systems and hardware is often exceedingly expensive, which is one of the factors why many of them operate way beyond their intended lifetime. As one of my interviewees explained: "There is lots of legacy code and legacy hardware out there. I've been to places where they can't find a vendor to replace the hardware any more, and places that buy things from online auction sites, because that's cheaper than to upgrade to something modern, and they just don't have the funding" (Interviewee 1).

NA are expected when the characteristic of interactive complexity is paired with tight coupling: an inability to isolate subsystems and interdependent systems from each other, or to stop them. This means that failures will cascade until a major part of the system, or all of it, fails [14]. Previous procedures to decrease the degree of tight coupling in Industrial Control Systems (ICS) of critical infrastructure, such as creating an "air gap" between the system and the internet, are made more difficult as the demand for digital transformation and efficiency of industrial organisations increases. Instead, modern ICS networks may be connected both to third parties and the wider organisation [20].

In the words of one expert commentator: "Legacy systems are often maintained only to ensure function, and their operations are often digitized with upgraded Internet of Things (IoT) functionality for the sole purpose of operability. OT maintenance may fail to consider the IT and cybersecurity perspective, seeking to make changes to improve systems without questioning if those systems remain secure. While these legacy systems may seem helpful after years of use, networked systems' prolonged exposure to these legacy devices proves time and time again the familiar adage: What can go wrong will go wrong" [21].

Moreover, as one interviewee highlighted, legacy technologies might be maintained because the processes they underpin are too important to risk being disrupted even for a short amount of time: "Many organisations are afraid of swapping out legacy technologies for new ones because they are afraid that it will disrupt the production process or cause it to fail" (Interviewee 4).

#### *9.3.2 Cognition*

A key problem connected to the cybersecurity of critical infrastructure operations is that the actual danger characterised by NA characteristics in large-scale systems is not always easily perceived. This difficulty is partly because the components interacting are not only technical but include organisational and human components too, making the complex interactions and interdependencies between components and subsystems more difficult to understand and estimate. In the words of Grabowski and Roberts: "In general, large complex systems are difficult to comprehend as a whole. Therefore, the tendency is to decompose or factor them into smaller subsystems, which can lead to the development of a large number of subsystem interfaces" [22]. One of the interviewees of this study highlighted the difficulty of getting a comprehensive picture of all the subsystems involved in critical infrastructure operations: "many operators and roles are very specialized now. You only understand one small part of the system and worry about that. However, all the parts must be included and compared to achieve a common model and understanding. Some parts may affect or even disturb other parts in unexpected ways. You must build your operation on a comprehensive analysis including supply chain dependencies. But this is currently lacking when it comes to critical infrastructure" (Interviewee 2).

According to the findings of HRO research, commitment to reliability is a key feature of managing the danger of the combination of interactive complexity and tight coupling, and this commitment is connected to a common understanding of the potential of catastrophic consequence [11]. As one interviewee argued, this commitment to reliability does not appear to be widespread when it comes to critical infrastructure operations and cyber: "It seems that when it comes to cybersecurity and digitalization of critical societal functionalities, we have not learned much from the high-risk industries. In those industries, we are happy to let security cost whatever is necessary. This is not the case with cybersecurity yet, despite that healthcare services (for example) could be disrupted nationwide due to a zero-day vulnerability. We are not yet at the point where we allow digitalization to be expensive due to security concerns" (Interviewee 3).

The difficulty of grasping cyber related vulnerability in critical infrastructure can also be enhanced by the fact that some systems (and system components) are critical and high risk, but many are not (and thus would not require strict security and safety measures in accordance with HRO). However, due to interactive complexity, system components that appear to be non-important could potentially be contributing to the failure of a system that is, thus making the distinction between critical and non-critical more challenging.

#### *9.3.3 Organisation*

A common finding of both classical HRO and NA research has been to point to the importance of reducing tight coupling in systems through "organisational slack". This can be done by achieving structural flexibility and redundancy, which involves duplication or multiple and independent ways of operating, communicating and making decisions [11].

The 2015 cyber-attack on the Ukraine power grid is an example of how centralisation can make accidents more consequential, but is also an example of how redundancy in organisations can reduce the same. In December 2015, one of Ukraine's power grid providers was taken down by a cyber-attack, leaving 230,000 customers across various areas without electricity for several hours. Through a sophisticated, long-term attack campaign earlier that year, including spear-phishing methods, hackers had succeeded in taking remote total control of the ICS of at least three energy distribution companies. In disrupting power through remotely switching breakers, the attackers also disabled backup power supplies to all but one distribution centre in order to hinder operators from reaching out and giving or receiving information about the evolving situation. Finally, they launched a distributed denial-of-service (DDoS) attack on the customer service centre.

Despite the sophistication and novelty of the attack, the operators were able to restore service within 3–6 h by moving over operations to manual control [23]. The possibility of going manual was highlighted in a later report as an important mitigation mechanism. It also highlighted that those utilities that are more reliant on automation might not be able to do this [24]. An interviewee echoed this: "The reason this attack was not more disruptive was that there were parts of the grid which was not digitalized, which means it was possible to move to manual operating mode" (Interviewee 5).

Applying the NA-dynamics perspective, interactive complexity allowed the hackers to gain total access and control over the ICS of the energy grid. They identified, attacked and exploited many individual technical and human components of the system in unexpected ways, and once they were in, they were able to "cascade" their access due to tight integration of the system [25]. However, this case is also an example of the impact of redundancy and organisational slack on limiting the tight coupling of the system and thus the full effects of the NA dynamics. By being able to operate the grid manually, the tight coupling of the system was reduced, and operational capacity was restored before the situation could develop into a serious, long-lasting energy crisis.

#### *9.3.4 Macro*

The interdependent linkages and interactive complexity that surround critical infrastructure services on a macro-level involve a complex ecology of supply chain actors and other critical infrastructure-sectors. These interdependencies may be difficult to analyse and regulate. This affects the ability to detect NA characteristics and possible cascade paths of disruptions. It also makes it difficult to implement coherent regulation across the transnational structures of critical infrastructure. As noted by Grabowski & Roberts: "In large-scale systems, subsystems are often characterized paradoxically by both autonomy and interdependence. At one level, subsystems exist and operate independently of other systems, resources, and interference, and they are often responsible for their own survival, success, and growth. Thus, they appear to be rather autonomous entities, requiring little coordination. At the same time, subsystems are also interdependent" [22].

This dynamic can be exemplified by the tendency of critical infrastructure services to be dependent on supply-chain actors for upholding its digital systems (including legacy systems). As one interviewee explained: "The overall fundamental security of the system will be dependent on the particular company running the legacy technology to be updating its software to be compatible with the latest operating systems from Microsoft and other companies. For example, if the technology only runs off Windows XP and cannot be upgraded to Windows 10, because the company that created it does not support that, or no longer exists, then you have a fundamentally vulnerable system" (Interviewee 4).

The combination of centralisation, interactive complexity and tight coupling in the systems of organisations that underpins the functionality of critical services (through, for example, the reliance on supply chain actors) as well as in the technological systems of critical services is continuously exploited by cyber-threat actors who use this dynamic to launch ransomware and supply-chain attacks. Interactive complexity creates the possibility for a cyber-exploit to spread quickly and unexpectedly, and centralisation in combination with tight coupling enhances the impact of the attack, putting the victim under more pressure to pay the ransom (especially if the victim provides a critical societal service such as energy, water or food distribution).

An example of this can be found in the case of the recent REvil (also known as Sodinokibi) ransomware attack, affecting at least hundreds of businesses worldwide, including the grocery chain COOP in Sweden, in early July 2021. The attackers managed to use a vulnerability in Kaseya's VSA software to bypass security measures and distribute malware (ransomware) to its customers [26]. Through the centralisation of IT-infrastructure service delivery, the attackers could focus on targeting this one business to get to connected businesses further down the line. In the case of COOP, there was no alternative way of operating without access to their digital payment system (no organisational slack/redundancy), and they had to simply close most of their 800 grocery stores in Sweden until the problem was solved [27].

In other words, REvil used the conditions of centralisation of service providers (making it possible to effectively spread malware to many businesses by exploiting a single supply-chain actor), interactive complexity (using different interactive components to achieve unexpected consequences) and tight coupling (leveraging the victim's dependency on the ransomed digital systems in order to force them to pay the ransom) to achieve their goals.

#### **9.4 Conclusion**

This chapter aimed to contribute to the understanding of the transboundary characteristics of large-scale cyber-crises by suggesting that they can be explained by the existence of NA dynamics (the combination of interactive complexity and tight coupling) in several layers of the sociotechnical systems that support modern critical infrastructure operations. With support from the insights of classical NA and HRO theory, it explored the application of these arguments. Analysing the technical layer, the chapter highlighted the problem of interactive complexity and tight coupling in the sociotechnical systems underpinning critical infrastructure, especially through legacy code, systems and hardware. Analysing the cognitive layer, it pointed to the difficulty of clearly perceiving the danger stemming from NA dynamics in largescale systems. At the organisational layer, it highlighted the 2015 cyber-attack on a Ukraine power grid as an example of how centralisation can make accidents more consequential, but also an example of the mitigating effect of operational redundancy measures to reduce tight coupling. Finally, analysing the macro-layer, the chapter used the case of the recent REvil attacks to discuss how the macro-NA dynamics including the reliance on supply-chain actors, can be exploited by cyber-threat actors and create cascading consequences and transboundary cyber-crises.

**Ethics Statement** This work adhered to the research ethics that are stipulated in the "Stockholm University research integrity and ethics policy" that complies with relevant legislation regarding ethical conduct of research. Informed consent was obtained from participants, and all data have been anonymised. Ethics board approval is not required for this kind of study in Sweden.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 10 Information Security Behaviour in an Organisation Providing Critical Infrastructure: A Pre-post Study of Efforts to Improve Information Security Culture**

#### **T.-O. Nævestad, J. Hovland Honerud, and S. Frislid Meyer**

**Abstract** The study examines whether information security behaviour (ISB) in an organisation providing critical infrastructure improved after systematic efforts to improve information security culture (ISC) through the implementation of an information security management system (ISMS). The data are based on quantitative surveys before (*N* = 323) and after (*N* = 446) efforts to improve ISC in the organisation. Qualitative interviews were also conducted before (*N* = 22) and after (*N* = 12). The study finds that the organisation has managed to improve its ISC through systematic efforts over a two-year period (2014–2016), and that this also has led to improvements in ISB among the personnel in the organisation. Multivariate regression analyses indicate that ISC is the most important variable influencing ISB, while ISMS measures is the most important variables influencing ISC. Thus, our results indicate that it is important to work with ISMS and ISC to increase IS in our increasingly digitalised society, especially in organisations providing critical infrastructure.

**Keywords** Information security management system · Information security culture · Critical infrastructure

T.-O. Nævestad (B) · S. F. Meyer Institute of Transport Economics, Oslo, Norway e-mail<EMAIL_ADDRESS>
J. H. Honerud University College of Southeast Norway, Kongsberg, Norway

#### **10.1 Introduction**

#### *10.1.1 Background*

One of the key aspects of increased digitalisation of society's functions, especially those related to critical infrastructure, is that it introduces new vulnerabilities, indicating the need for new types of protection. An important insight in this respect is the critical importance of human and cultural factors for the security level of critical infrastructure. According to Lim et al. [1], security is becoming more challenging in today's business because people are both a cause of information security (IS) incidents as well as a key part of the protection from them. In this context, physical and technological measures provide an insufficient strategy for protection. Several studies indicate the critical importance of information security behaviour (ISB) for IS in organisations, suggesting that these behaviours often reflect more general patterns of information security culture (ISC) in the organisations [1–3].

We define ISBs as behaviours that are relevant to IS. IS is often defined as protection against breaches of confidentiality, integrity and accessibility. This applies to information that is oral, written or electronic. Confidentiality refers to ensuring that only those who are authorised to access information, accesses it. Integrity refers to protecting the accuracy and entirety of information and processing methods. Accessibility refers to ensuring that authorised users have access to the information and associated equipment when necessary. We define ISC as shared and information security relevant ways of thinking or acting that are (re)created through the joint negotiation of people in social settings.

According to Chen et al. [2], previous studies have paid little attention to the important influence of ISC on ISB. Additionally, Chen et al. [2] state that there is little research on the relationship between comprehensive efforts to manage IS and ISC. We may also refer to such efforts as an information security management system (ISMS), which defines policies and procedures to ensure, manage, control and continuously improve IS in an organisation. One of the most prevalent ISMS is the ISO 27001 standard, which involves systematic efforts to ensure confidentiality, integrity and accessibility.

#### *10.1.2 Aims*

In this chapter, we address the research gap identified by Chen et al. [2] by studying whether the implementation of an ISO 27001 compliant ISMS (and additional measures) has led to changes in ISC and subsequently IS in an organisation providing critical infrastructure.

The aims of the study are to:

We compare implementation and effect in the six departments of the organisation.

#### *10.1.3 Previous Research*

#### **<IP_ADDRESS> Information Security Management System**

An ISMS consists of formal routines and measures that enable the organisation to work systematically to avoid breaches of confidentiality, integrity and accessibility, by establishing formal safety policies and goals, establishing important roles and responsibilities, conducting risk analyses systematically gathering information on incidents and dangers, developing countermeasures, monitoring the effects of these and adjusting measures if necessary (cf. Mitsch et al. [4]).

Other key elements in an ISMS are role descriptions with responsibilities, reporting systems, risk assessments, security training, security procedures, etc. The security policy states security goals and how these are to be achieved via ISMS. The procedures for achieving the goals are documented, along with who is responsible for doing what.

Although the implementation of an ISMS, e.g., ISO 27001, is often cited as an important way of establishing an ISC, there seems to be few studies which have actually examined this relationship (cf. [2]). The relationship between management systems and culture is, however, well established in the research on safety culture [5].

#### **<IP_ADDRESS> Information Security Culture**

While an ISMS refers to the formal aspects of IS management ("how things should be done"), as it is described in procedures, manuals, etc., the informal aspects of IS management generally refer to ISC ("how things are actually done") [6]. ISC is often studied using various concepts and models of organisational culture [1]. Although Ruighaver et al. [7] note that the organisational security culture concept has gained recognition, they also underline the lack of consensus on definitions and concepts (cf. [8]). Additionally, they assert that in spite of a large amount of research on organisational security and how it should be improved, this research only focuses on certain aspects of security, and not how these aspects can be analysed as part of a larger organisational culture [7]. Based on this understanding, they choose to draw on organisational culture research in their analysis of ISC. This approach is similar to that applied by scholars studying organisational safety culture, who analyse safety culture as a focused and safety-relevant aspect of the larger organisational culture (e.g., [6]). Based on this, we may also analyse ISC as "security-relevant" aspects of the larger organisational culture, defined and conceptualised using models of organisational culture. When studied qualitatively, ISC refers to common frames of reference that form the basis for interpretations of actions, hazards and our own identity, and which motivate and legitimise behaviour that affects IS (cf. [6, 9]). Such common frames of reference arise through interaction in groups. When studied quantitatively, ISC is measured as the way IS is valued in the organisation by managers and employees and their perceptions of the ISMS, or "the way things actually are done" when it comes to IS.

#### **<IP_ADDRESS> Information Security Behaviour**

Lim et al. [1] cite several studies indicating that IS problems in organisations have been linked to employee behaviour. These behaviours are typically different types of violations and non-compliance with IS procedures, indicating that it is not sufficient to have a formal system in place, if it is not supported by the ISC [1]. This indicates the importance of viewing IS behaviours as part of a larger ISC context.

#### **<IP_ADDRESS> Theoretical Model and Hypotheses**

Based on previous research, we have developed the following theoretical model and hypotheses:

#### **10.2 Methods**

#### *10.2.1 Qualitative Interviews*

We used a semistructured and relatively open interview guide, focusing on security work in the organisation since 2014. The interviews were built up around the following main topics: (a) IS measures since 2014 and ISMS implementation, (b) follow-up of the 2014 ISC survey and (c) managers and employees' perceptions of IS rules and measures.

#### *10.2.2 Quantitative Survey*

#### **<IP_ADDRESS> Survey Items**

The survey contains a set of background questions (e.g., gender, age, experience, education). The survey also includes questions measuring ISMS measures and implementation, e.g., questions about each department's follow-up of the survey results in 2014 and information about specific IS issues over the past two years (e.g., passwords, security policy for mobile units, policies for strangers in the premises) (cf. Sect. 3.1.2).

In this study, we choose to reformulate one of the few existing universal organisational safety culture scales, the GAIN scale [10] for safety culture, into an organisational security culture scale. The questionnaire contains 24 questions concerning, e.g., respondents' perceptions of management's and employees' focus on information security, reporting of information security issues. Respondents can rate the questions from 1 (totally disagree) to 5 (totally agree). Thus, a security culture index with a minimum value of 24 (1 × 24) and a maximum value of 120 (5 × 24) can be compared across companies and sectors.

We have one question measuring ISB: "When I am asked for information, I always think carefully about whether the information can be used for purposes other than its intended purpose".

#### *10.2.3 Samples*

We compare the results of two surveys done over a period of just over two years; the first in the spring of 2014 and the second in the autumn of 2016. Response rates are provided in Table 10.1.

**Table 10.1** Response rates

#### **10.3 Results**

#### *10.3.1 IS Management System Implementation*

#### **<IP_ADDRESS> Qualitative Results**

The study organisation is a provider of critical infrastructure in Norway. As a provider of critical infrastructure, it is obliged to follow the requirements of the Safety Act ("Sikkerhetsloven") when it comes to preventive safety work, which includes safety analyses, securing objects, IS and safety drill.

The study organisation decided to map and analyse its ISC in 2014, due to its legal obligations, work activities and engagement. The organisation used the measurement of ISC in 2014 as an indicator of the IS level in the organisation and as a basis for identification of critical areas (related to attitudes, knowledge, practices) in need of improvement. Based on this, future goals for improvement were established, both at a general level and at a more specific level. A score of 87 or higher on the ISC index was established as a general goal for all departments. The 2014 survey identified needs when it comes to, e.g., increasing IS knowledge, attitudes and engagement among the personnel.

Several measures were taken to improve IS. Department managers were given the task of presenting the results from the 2014 ISC survey to their employees, discuss the results and measures that could be implemented to improve the status on the specific challenges discussed. A number of actions were taken: digital and on-site intrusion test, strengthened physical access control in the facilities, access card pin codes, new password policy, VPN dashboard and stronger fire wall policy (including strengthened fire wall and two-factor authentication), new routines for use of access card and visitor registration, internal training on password handling, handling of physical documents in field operations, photography and social media, office and document access in-house and information on possible consequences of negligent information leaks relating to security critical objects.

Second, the organisation started to establish a basic safety organisation in 2014, describing roles and responsibilities, as well as principles and guidelines related to IS. This was developed in accordance with the ISO 27001 principles. The implemented ISMS is particularly linked to the administrative department's responsibilities, the role of security coordinators and systematic risk analyses on all processes and goals relating to IS. A security coordinator was appointed, with a special responsibility to provide systematic training of the personnel in IS issues, to coordinate and train security coordinators in each department and to further develop IS protocols and instructions. The organisation reviewed all information related to security critical objects, crisis plans, security certifications and critical ICT systems, defining information into three categories: (a) open, (b) internal, (c) sensitive and graded. Additionally, new policies were developed for personal information, graded information, acceptable use of information and security policy for mobile units. The organisation developed a security declaration for employees to sign, documenting that they had received all the required training and information. A new system for recording and dealing with non-conformities was developed. The organisation also started to arrange an annual security month, and information security became a mandatory theme in each manager meeting. In spite of all these measures, interviewees agreed that challenges remained and that there were needs for improvement and maturing of the ISMS.

#### **<IP_ADDRESS> Quantitative Results**

We asked the respondents three questions about the follow-up of the ISC measurement in 2014. We introduced the questions with the text: "We want to know a little about what actions your immediate supervisor has taken in your department/section after the evaluation of the ISC in 2014".

The head of my department/section has gone through the results of the evaluation with us

My department/section has taken steps to improve the ISC (e.g., focus on passwords and security-critical information) based on the results of the evaluation

I am satisfied with how my department/section has worked on IS over the past two years

We also asked the respondents three questions related to whether they have received useful information over the past two years that has increased their knowledge and awareness of IS. These questions measure training in IS:

During the past two years I have received useful information (e.g., from security coordinator, manager, intranet) about what a secure password is

During the past two years, I have received information (e.g., from security coordinator, manager, intranet) that made me more aware of strangers in our premises

During the past two years, I have received information (e.g., from the security coordinator, manager, intranet) that has given me more insight into what security-critical information is

We made two indexes based on these six questions (cf. Table 10.2). The first on follow-up, the second on training. All the questions include six response options: 1 = totally disagree and 5 = totally agree and 6 = have no knowledge about this. When we created the indexes, we removed the sixth response option.

Table 10.2 indicates the highest levels of follow-up in department 6, 1 and 5, and the highest levels of training in department 6, 5 and 1.

#### *10.3.2 Improvements in ISC*

We have combined the 24 statements with five response options on the five different aspects of IS in an ISC index. The indexes for the departments correspond to the average scores for the respondents. The minimum score is 24 (24 × 1), and the maximum score is 120 (24 × 5) (Table 10.3). Cronbach's Alpha for the 24 questions in the index was 0.913 in 2014, which means very good agreement between the questions and that the index is very good.

We see that the Department 6 (again) had the highest score in 2016 and 2014, followed by Department 2 and Department 1. All the departments saw an improvement on the ISC index in 2016, especially Department 1, which increased by 12 points on the index. The average improvement for all the departments from 2014 to 2016 was 9 points. This change is statistically significant at the 1% level. The

differences between the departments are significant at the 1% level, both in 2014 and 2016.

GAIN [10] defines different types of culture, based on the scores of the index. The limits for "positive culture" range from 88 to 120 points on the GAIN index. The moderate culture scale goes from 47 points to a maximum of 87 points, and scores below 46 points correspond to a poor culture. If we are to transfer the GAIN scale values from safety culture to ISC, we see that none of the departments had a positive ISC in 2014. However, in 2016 we find that Department 6, Department 2 and Department 1 were within the part of the scale that we refer to as a positive culture.

#### **<IP_ADDRESS> Which Factors Influence ISC?**

In Table 10.4, we examine the variables influencing respondents' ISC. We include variables measuring ISMS implementation and background variables.

Table 10.4 indicates two main results. The first is that the follow-up index is the strongest predictor of respondents' ISC. We see that the department variable ceases

**Table 10.4** Linear regression of factors influencing respondents' scores on the ISC index in 2016. Standardized beta coeffisients.

Dependent variable: ISC standardised beta coefficients

\**p* < 0.1; \*\**p* < 0.05; \*\*\**p* < 0.01

to contribute significantly from Model 4 to Model 5, when the follow up variable is included. This indicates that this variable is related to follow-up, i.e., that this department has a higher score on the ISC index, due to the follow-up of the 2014 survey of ISC. The second main result is that the training index also contributes significantly and positively, indicating that IS training is related to a higher score on the ISC index. The Adjusted *R*2 value in Model 6 is 0.641, indicating that the model explains 64% of the variation in the dependent variable.

#### *10.3.3 Improvements in Information Security Behaviour*

Table 10.5 shows results on the question: "When I am asked for information, I always think carefully about whether the information can be used for purposes other than its intended purpose", in 2014 and 2016.

Table 10.4 indicates the highest level of improvement in Departments 6 and 5, followed by Departments 4 and 1.

#### **<IP_ADDRESS> Which Factors Influence Information Security Behaviours?**

In Table 10.6, we examine the variables influencing respondents' ISB. We include variables measuring ISC, IS knowledge and background variables.

Table 10.6 indicates two main results. The first is that the ISC index is a strong and significant predictor of the ISB of the respondents. We see that the department variable ceases to contribute significantly from Model 4 to Model 5, when the ISC index is included. This indicates that this variable is related to ISC (i.e., that the ISC score is higher in this department). The second main result is that respondents' IS knowledge also is a strong and significant predictor of their ISB. Knowledge is measured as the degree of agreement with the statement: "I am well aware of which kind of information that is sensitive and security graded". The Adjusted *R*2 value in Model 6 is 0.207, indicating that the model explains 21% of the variation in the dependent variable.

**Table 10.5** ISB scores in

**Table 10.6** Linear regression of factors influencing respondents' ISB in 2016. Standardized beta coeffisients.

Dependent variable: ISB standardised beta coefficients \**p* < 0.1; \*\**p* < 0.05; \*\*\**p* < 0.01

#### **10.4 Discussion**

#### *10.4.1 The Implementation of ISMS*

The first aim of the study was to describe the efforts to improve information security culture through the implementation of an ISMS. The study organisation has implemented several efforts to manage IS following the first measurement of ISC in 2014. In accordance with the continuous improvement approach inherent in ISMS, the organisation used the 2014 measurement as a baseline for future improvement, treating ISC as a broad indicator of IS in the organisation. Improvement goals were agreed upon, which were going to be followed up in a new measurement. In the meantime, several organisational measures were implemented and/or improved, e.g., related to the security coordinator, training, risk analyses and procedures. These were developed in accordance with ISO 27001 principles. As a consequence, interviewees in 2016 agreed that they had established a basic "security organisation" through the implemented measures.

#### *10.4.2 How Can We Explain the Improvements in Information Security Culture?*

The second aim of the study was to compare information security culture in the organisation before (2014) and after (2016) efforts to improve organisational information security culture, and examine factors influencing ISC. Our analyses indicate a 12% increase in the score for ISC in the organisation, which is statistically significant at the 1% level. Multivariate analyses indicate that variables measuring ISMS implementation were the most important predictors of ISC. This is in accordance with Hypothesis 1, stating that implementation of ISMS measures will lead to improvements in ISC. The follow-up index, measuring the follow-up of the 2014 measurement of ISC in each department was the strongest predictor of the respondents' ISC. This indicates the importance of such group-wise processes of continuous improvement, when it comes to developing ISC. The rates of improvement in ISC also varied between the different departments; ranging from 2.5% to 15% improvement. In line with Hypothesis 2, results indicate that the departments with the best implementation had the best improvements in ISC. This especially applies to department 6. According to Chen et al. [2], there is little research on the relationship between comprehensive IS programs and ISC. In this study, we contribute to this knowledge gap by studying how the ISMS measures of the study organisation have contributed to improvements in ISC and subsequently ISB.

#### *10.4.3 How Can We Explain the Improvements in Information Security Behaviours?*

The third aim of the study was to compare information security behaviour in the organisation before (2014) and after (2016) efforts to improve organisational information security culture, and examine factors influencing ISB. Our analyses indicate an 8% increase in the average score in the examined ISB from 2014 to 2016, which is statistically significant at the 1% level. These changes were also significant when controlled for factors like the age and education of the respondents. Multivariate analyses indicate that the ISC index and IS knowledge were the most important predictors of ISB. This result is in line with Hypothesis 3, stating that improvements in ISC will lead to improvement in ISB. This is in accordance with previous research indicating a relationship between ISC and ISB [1–3]. The rates of improvements in behaviour varied among the different departments in the organisation, ranging from 3% improvement in Department 3 to 16% in Department 6, again indicating the importance of effective ISMS implementation for ISC and ISB results. The ISC index measures questions related to management and employee focus on IS, IS training, etc. Our results indicate that this is positively related to our measure of ISB, which is related to confidentiality: "always thinking carefully about whether requested information can be used for purposes other than its intended purpose".

#### *10.4.4 Safety Culture Versus Security Culture*

We used a modified organisational safety culture scale [10] to measure ISB. The scale was chosen as the research on organisational safety culture seems to have been through many of the challenges that the organisational security culture research now is facing (cf. Ruighaver et al. [7]). At the same time, the research on organisational safety culture seems to have matured a bit more conceptually and methodologically, as it has employed the culture perspective for a few more years than the field of security research. We therefore draw on the experiences of the safety culture literature.

There are however several important differences between safety culture and security culture and the applications of the concepts. First, the difference between sharp end and blunt end is harder to see in the field of security. All members of the organisation are in one sense in the security sharp end, as they are users of information, equipment and facilities. Additionally, the results of serious security incidents and non-conformities may remain unseen and unnoticed for a long time. The same applies to latent system failures, which may be exploited undetected for long periods by third parties. Security incidents are generally not physical accidents with immediate damages or injuries. A consequence of this is that it may be even more difficult to define a state of "security" (e.g., as the absence of serious security incidents) than it is to define a state of "safety" (e.g., as the absence of physical accidents).1 Thus, security is a more abstract state than safety, making security assessments and preventive efforts more challenging.

It could also be mentioned that in the field of security, the distinction between the private domain and work domain is blurred, as employees use digital workplace equipment (e.g., phones, tablets, computers) in their leisure time. Thus, they must also act according to their ISMS and ISC after working hours and in their private spheres. Thus, in contrast to the field of safety, the field of security requires employees to be always "at work". This has interesting implications for ISC management: it stretches into both the professional and the private sphere.

Another difference is related to intent: safety culture mainly concerns prevention of incidents related to combinations of technological, (*unintentional*) human and organisational risk factors, while security culture concerns prevention of incidents related to combinations of technological, (*intentional*) human and organisational risk factors. As the human component in the security field often deals with intentional actors with hostile intentions, the possibilities for failure (security breaches) are greater. In the case of safety, human risk factors are typically related to unintended errors, mistakes and violations, combined with technological and organisational weaknesses. In the case of security, these risk factors at the "victim end" are combined with the creativity, expertise and imagination of intentional human actors at the "offender end". An implication of this is that security management also is related to crime prevention.

1 This may also be the case with latent organisational or technological safety failures, which may exist undetected until they interact with active failures in ways that lead to accidents. Given the abstract character of security, preventing unwanted security incidents may have more to learn from research on safety culture in complex technological systems, where technological failures may act in unseen, unanticipated and incomprehensible manners due to "interactive complexity" (cf. Perrow [15]). Previous research indicates that cultural management is particularly important in these settings (cf. Weick et al. [16]).

Finally, the most important similarity between safety and security management is that physical and organisational measures are important for increasing both safety and security, but as long as human actors use these systems and relate to the physical measures, the state of safety and security is contingent on the behaviours and subsequently the culture of these actors. This indicates the crucial importance of ISC for IS and the importance of organisational safety culture for safety.

#### **10.5 Conclusion**