Simulation practitioners should use their limited resources to focus on their domain specific needs (simulation models, simulation data exchange models, simulation environment agreements, and verification methods) and benefit from existing tools and knowledge bases. I.e. focus on at least semantic interoperability between simulation models in a certain problem domain, and leverage existing standardised simulation middleware for the technical interoperability.

Acknowledgement and Disclaimer This chapter was derived from the FP7 project CIPRNet, which has received funding from the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 312450.

The contents of this chapter do not necessarily reflect the official opinion of the European Union. Responsibility for the information and views expressed herein lies entirely with the author(s).

### References

### 6 Federated Simulations 137

### Author Biographies

Wim Huiskamp is Chief Scientist Modelling, Simulation and Gaming in the M&S department at TNO Defence, Security and Safety in the Netherlands. His research areas include system architecture, distributed real-time simulation and C2-Simulation interoperability problems. Wim acted as project lead for several national and international simulation (interoperability) projects and he currently leads the national simulation research program carried out on behalf of the Dutch MoD. In recent years Wim was the chairman of the NATO Modelling and Simulation Group (NMSG) and formerly also the chairman of the NMSG M&S Standards Subgroup (MS3). Wim is the liaison of the NMSG to the Simulation Interoperability Standards Organization (SISO).

Tom van den Berg is a senior scientist in the Modeling, Simulation and Gaming department at TNO, The Netherlands. He holds an M.Sc. degree in Mathematics and Computing Science from Delft Technical University and has over 25 years of experience in distributed operating systems, database systems, and simulation systems. His research area includes simulation systems engineering, distributed simulation architectures, systems of systems, and concept development and experimentation.

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Chapter 7 Cyber Threats Impacting Critical Infrastructures

Michał Choraś, Rafał Kozik, Adam Flizikowski, Witold Hołubowicz and Rafał Renk

Abstract Nowadays it is important to note that security of critical infrastructures and enterprises consists of two factors, those are cyber security and physical security. It is important to emphasise that those factors cannot be considered separately and that the comprehensive cyber-physical approach is needed. In this paper we analyse different methods, methodologies and tools suits that allows modelling different cyber security aspects of critical infrastructures. Moreover, we provide an overview of goals an challenges, an overview of case studies (which show an increasing complexity of cyber physical systems), taxonomies of cyber threats, and the analysis of ongoing actions trying to comprehend and address cyber aspects.

### 1 Introduction

The CPS abbreviation stands for Cyber-Physical Systems and it refers to systems that have distributed natures, are comprised of physical elements that work in a real-time and are capable of communicating with each other by means of communication network (both wired and wireless, see Fig. 1). CPS integrate computational, communication and physical aspects in order to improve usability, efficiency, reliability, etc. However, such combinations, introduce a wide spectrum of risks related to cyber domain (e.g. privacy issues, cyber attacks).

The CPS are comprised of elements that allow for reading relevant information about controlled physical process (e.g. sensor) and elements that allows for influencing (via actuators) the behaviour of this process. The CPS are widely adapted in many critical sectors including energy, water, and transportation as well as in the area of smart houses, vehicles, etc.

Institute of Telecommunications and Computer Science,

UTP University of Science and Technology, Bydgoszcz, Poland e-mail<EMAIL_ADDRESS>
W. Hołubowicz R. Renk Adam Mickiewicz University, Poznań, Poland

M. Choraś (&) R. Kozik A. Flizikowski W. Hołubowicz R. Renk

© The Author(s) 2016

R. Setola et al. (eds.), Managing the Complexity of Critical Infrastructures, Studies in Systems, Decision and Control 90, DOI 10.1007/978-3-319-51043-9\_7

Fig. 1 Dependencies between complex systems comprising CPS

Currently, the CPS are on the direction to become an integral part of our lives, embracing in the near future such aspects as healthcare, disaster recovery, engineering, traffic control, robotic surgery, sea and space exploration, defence and military operations.

This paper is structured as follows: firstly, we analyse goals and challenges in area of cyber security of critical infrastructures, presenting case study overview and elaborating on the impact of the cyber domain on the real world. Next, we provide short overview of the taxonomies used to model and to analyse the cyber threats. Afterwards, we provide an general overview on how the cyber security life-cycle is modelled in terms of crisis management and critical infrastructures protection. Particularly, we focus on different approaches to cyber risk identification and cyber incidents handling. In the following section, we present different aspects of IT and physical networks that can be modelled with the formal tools and methodologies. The analysis of ongoing actions trying to comprehend and address the challenges of cyber aspects of critical infrastructures as well as the conclusions are given afterwards.

### 2 Goals and Challenges

Quantitative evaluation of cyber security is always a challenge in the area of computer science. For the CPS, the integration of ICT technologies with physical elements has introduced new threats. Currently, we may find many examples of the attackers have been able to compromise complex systems by finding vulnerable elements. In many cases those attacks have had direct impact on physical elements. Therefore, there is an ongoing effort to embrace the cyber aspects of CPS with comprehensive tools and methodologies that commonly leverage wide spectrum of technical and non-technical means. The current challenges related to CPS can fall into following groups of problems:

Of course, such problems should be handled in the holistic manner, e.g. by the THOR (Technical, Human, Regulatory, Organizational) approach and aspects as proposed by recently finished European projects [1, 2].

As the cyber security of CPS systems imposes a significant challenge, in this section, we particularly focus on different aspects of the cyber domain. We start with examples of case studies that in many cases reveal the complex nature of those systems and huge amount of interconnections that span across different levels of Critical Infrastructure management.

Afterwards, we provide examples of how the European Union addresses current problems and the challenges in the H2020 work program.

### 2.1 Cyber World and Real Impact—Selected Case Studies

Due to the fact that the energy sector is quickly evolving and it is widely adapting different ICT technologies, we are able to identify many high profile cyber incidents. One of the most important cyber attacks in history of Critical Infrastructures happened in 2012 [3], when Iran authorities announced that computers controlling one of its nuclear processing facilities had been infected with malicious software called Stuxnet. It was the first case in which industrial equipment was a target of computer attack. Since that date, the cyber community has realised that cyber weapon can be used "… to create physical destruction […] in someone else's critical infrastructure…" [4].

Also for the water sector, we are able to find relevant cyber incidents, which show a real and high impact of the cyber world on physical infrastructures. Similarly as for Energy sector, the cyber components for both drinking water and wastewater facilities include control systems known as Supervisory Control and Data Acquisition (SCADA) systems. Cyber attacks on such utilities may cause cascading effect on a public health, economics, and nations as whole. An example presented in [5] shows how the attacker can influence water treatment plants. According to IBTimes [5], attackers infiltrated the water plant and were able to change the level chemicals that were used to treat drinking water.

Healthcare industry is also an important part of critical infrastructure. It is also targeted by cyber criminals. As examples show [6, 7], cyber-attacks targeted at this sector can slow down hospitals and expose patients to danger.

Also the financial sector is struggling with cyber attack. According to [8] the activity of cyber criminals increased by 41% in recent years. Recent example of Bangladesh bank [9] show that attackers have effective tools and skills to infiltrate bank institutions and to steal serious amounts of money.

According to the [10], also the growth of the Internet of Things and complexity of industrial control systems will lead to more vulnerabilities in hardware systems. Many companies dealing with cyber security [1] have identified serious vulnerabilities in automotive systems and home-automotive systems. This shows that not only critical infrastructures but also citizens directly are currently impacted by the attackers as the cyber domain embraces increasing number of our lives.

### 2.2 The Coordinated Cyber Attack—Ukrainian Case

On the 23rd December 2015, the Ukrainian power distribution operator Prykarpattya Oblenergo was suffered attack on their ICT infrastructure performed by the third party. In effect of this breach, operation of a number of power substations were interrupted and about 80 thousands of customers from Ivano-Frankivsk region were suffered an outage for next three to six hours, according to the official information published through the operator website. At the same time, the operator informed publicity about other technical failure related to the operation of the call centre infrastructure. This caused impossibility for the customers to contact operator during the blackout and deepened the crisis.

The above described circumstances indicate that the energy operator faced the well-coordinated attack, that can be decomposed into three elements: a malware attack, a denial of service attack targeted at the call centre functionalities and the opening of substation breakers to cause the outage.

Firstly, the attackers infected the main servers controlling the electricity distribution process, they infiltrated in the victim's network (possibly using a malware backdoor) and issued a command to open breakers of various substations.

The goal of the cyber criminal was to enter the power grid system by infecting the victim's machines with malware software. They used macro script in Excel files to drop the malware. The infected Excel spreadsheets have been distributed during a spear-phishing campaign that targeted IT staff and system administrators working for multiple companies responsible for distributing electricity throughout Ukraine.

After the power was cut off, DoS attacks were launched to limit the target's awareness of the consequences of the attack—error messages did not reach service personnel what prevented from proper reaction on the crisis and delayed the recovering of the infrastructure operation.

The Ukrainian blackout case can be seen as the one of the first significant and publicly reported cyber attacks aimed at civil infrastructure and directly impacting civil population (e.g. in opposition to the Stuxnet, Iranian case, where industry/ military premises were infected). Ukrainian case shows that motivated attackers are able to cause serious damages to the economy and public safety of countries.

In case of the Ukrainian grid, luckily at that time, the manual mechanical reaction was possible. It would rather not be possible in case of the much modern and automated energy grids in some other countries.

### 2.3 Hybrid Conflicts

The Ukrainian case (described in the previous subsection) gives the short glance at the possible impacts of the successful cyber attack launched at the critical infrastructure such as energy grid. Unfortunately, due to the current geo-political situation and the current market of hackers (state and non-state), there is a significant threat that a country or its critical infrastructure can be attacked by another country or hackers organization working for another hostile country. It is worth to notice, that nowadays most hackers work for organizations rather than on their own (it changed significantly since in the past there were more freelancers than hackers working for organizations). In other words, cyber attacks might be a part of so called hybrid war or hybrid conflict, where (at least at the first stages) traditional military measures (such as soldiers) are not used, but the focus is on other destabilizing aspects like cyber attacks, cyber propaganda, influencing social media and electronic media etc. If the worst scenarios become reality, the successful coordinated cyber attacks launched at critical infrastructures such as banks (no possibility to draw money from ATM), energy (no electricity), transport, media etc. could paralyze societies, countries and create chaos.

Therefore, in order to avoid situation like in the Ukraine, the effective solutions and techniques to protect cyber physical systems are needed. The created recommendations and technologies should cover the wide spectrum of aspects, such as technological, organizational, human and regulatory (similarly to the THOR approach suggested by the new cyber roadmaps by projects like CAMINO, COURAGE and CyberRoad) [2, 11].

### 3 Cyber Threats Taxonomies

An important part of CPS cyber threats modelling is the taxonomy of cyber threats. To combat the cyber crime effectively, it is required to identify, define and classify the problem. It is not a trivial task, and currently even the spelling of the related words is not agreed, some use cybersecurity or cyber security or cyber-security. Similarly with other words like cybercrime, cyberterrorism etc.

A taxonomy is most often defined as a classification of terms and has close a relationship with the use of ontology. The primary purpose of ontologies and taxonomies is to use them as the basis for processing, communicating and reasoning about the cyber-related aspects and threats.

Also as noted by Furnell [12], having a consistent classification of cyber crime is beneficial for bodies and organisations interested in cyber counterterrorism. One of the earliest cyber crime classifications was established by UK Audit Commission and proposed in [13]. This categorisation identifies different groups of cyber crime activities, like: Fraud (for private gain of benefits), Viruses, Theft (of data or software), Use of unlicensed software, Hacking, Sabotage, Misuse of personal data, Introducing pornographic material.

In Furnell [12] proposed classification that is based on two major types of the cyber crime, namely computer-assisted and computer-focused. The computerassisted cybercrimes are these which use computer as supporting tool and where the target is not to be directly connected with the cyberspace (e.g. harassment). The computer-focused category of crimes includes these incidents that are almost entirely technical, associated with ICT systems and not (or weakly) connected to other sectors.

Similar dichotomized categorisation (as by Furnell) has been proposed by Gordon and Ford [14]. Authors divided cyber terrorism into two distinct classes, namely: (i) Type I Cybercrime, which is mostly technological in nature, (ii) Type II Cybercrime, which has a more pronounced human element.

Different classification is proposed in [15]. It is mainly focused on subject of criminal activity and defines following main categories, namely: against individual, against property, against organization, against society.

In opposite Walden [16] has postulated that there are five possible schemas of classification that overlap but are different in their perspective. These are: technologybased, motivation-based, outcome-based, communication-based and informationbased crimes.

According to [17] motivation-based classification schema provides more holistic perspective on the topic cybercrime. The proposed motivational model is composed of five major components: people, motivation, perpetration technology, security barrier and the target. The people in the model refer either to offenders or criminals. When individuals are exposed to a certain type of the factors they may become motivated to carry out particular behaviour and commit a crime. The motivation component refers to certain factors like unemployment, low median income, poverty, or social status that push the individual to carry out a cyber crime. The perpetration technology refers to technology used as a tool to commit a crime, while security barrier indicates components (firewalls, anti-virus software, etc.) that need to be comprised in order for crime attempt to be successful. The last component of the model indicated as Target refers to the people or organization that are being targeted by a criminal.

One of the approaches intending to comprehend cyber security aspects of critical infrastructures have been attempted by the European Union-sponsored project named Vital Infrastructure Threats and Assurance (VITA) [18]. One of the outcomes of the project was a generic threat taxonomy for Critical Infrastructures (CIs). It categorises such aspects as threat cause, human intent, threat, etc. It is emphasised by authors [19] of the taxonomy that terror, sabotage or activism are not threats but motivations.

In [20] authors adapted and extended the VITA threat taxonomy for Smart Grids. While identifying threats authors have addressed both the information and infrastructure dimension. Authors particularly wanted to identify how Smart Grid hardware may influence the resilience and reliability of energy grids.

Recently, the taxonomy of the cyber crime and cyber terrorism was discussed in [1].

### 4 CIP Cyber-Physical Security Life-Cycle Models

A wide spectrum of services provided by intelligent critical infrastructures (e.g. Smart Grids) heavily depend on Cyber-Physical Systems (CPS) that are able to monitor, share and manage information. On the other hand, an increasing number of cyber attacks and security breaches are part of rapidly expanding cyber threat, which in many cases has form of cyber terrorism.

The cyber-physical security can be analysed from classical crisis management point of view. In fact, most of incident management processes in the cyber domain follows the ITIL model that is depicted in Fig. 2. It focuses on incidents detection, diagnosis (e.g. identification of exploits that attacker exploited), repairmen (e.g. elimination of the software vulnerability that attacker exploited), recovery and restoration (e.g. to normal business operation status).

However, this type of model may not properly show the iterative nature of continuous improvement that usually are implemented after the crisis as an element of lessons learnt. Therefore, the model of cyber security life-cycle would be that one which is intended to define how to prevent, detect, respond to and recover from cyber crisis, and finally to avoid reoccurrence. Thus, we can define Cyber Attack Timeline, illustrated in Fig. 3, which is constituted of the following three phases:

Fig. 2 Incident management according to ITIL standard [35]

In this section, we further elaborate on different aspects related to cyber security of CPS systems that is embraced into crisis management phases namely: prevention, detection, containment, and post-incident.

# 4.1 Pre-crisis Phase

### 4.1.1 Prevention and Proactive Response

The cyber security prevention is an important aspect when it comes to cyber-physical systems and its impact on critical infrastructures. It requires some amount of the resources to be allocated, however, it is better than often costly recovery (or in worst case no recovery at all). As the value and importance of prevention is at least well acknowledged in the communities, it is still in many cases perceived as product that can be purchased and deployed in an organisation. In fact, the prevention is long-lasting and continuous process reaching far beyond technical problems embracing organisational, regulatory, and human aspects.

Fig. 4 Risk management information and communication flows (NIST SP 800-39)

Particularly, the cyber attack prevention requires (within the organisation) well established roles that will be responsible for containing the cyber attack and its causes. This implies that an organisation should define detailed cyber incident response plan that will describe how an incident should be reported, investigated and responded (Fig. 4). Moreover, when the cyber incident involves personal information, it implies various data privacy and security laws that may have different shape in different countries.

As mentioned in [21], it is very important for Critical Infrastructures operators to identify the risks posed by the communication networks and existence of dependencies with third party systems. This is even more important form wider perspective, because such risk anticipation can prevent the possibility of cascading failures causing catastrophic system damages.

The risk management cycle is a comprehensive process (Fig. 2) that requires organizations to:

Usually this requires effective communication and an iterative feedback loop, that will facilitate continuous improvement in the risk-related activities.

As it is suggested by ENISA [22], a good practice for well-suited prevention mechanisms is to subscribe to relevant information sources that would give up-to-date overview of current cyber threats and incidents reported. ENISA also stresses the importance of information sharing.

More local (service based) approach to risk modelling has been proposed by OWASP [23]. The approach follows the idea of decomposition of complex system to smaller components (see Fig. 5 Threat Risk Modelling proposed by OWASP). It is important to stress the fact that all key players (e.g. security officers, employees)

Fig. 5 Threat risk modelling proposed by OWASP [23]

need to understand the security objectives. Therefore, usually the complex system is broken down into objectives such us: reputation, availability, financial, etc. Other security objectives may be enforced by the law (financial or privacy laws), adapted standards (e.g. ISO).

The key element of this risk assessment methodology is the possible threats identification. Microsoft has suggested two different approaches to identify those threats. One is a threat graph (see Fig. 6), as shown in Fig. 2, and the other is a structured list.

### 4.1.2 Threat Detection

The capability of early detection of cyber threats is a very important element for good cyber crisis preparedness. Probably, one of the most classic way to categorise the cyber attack detection technique is to assign them into one of the following groups, namely: signature-based, anomaly-based or hybrid (Fig. 7).

Each of this class of algorithms has their drawbacks and advantages, and different approaches to identify attacks. Some of the methods have also different

methods for data aggregation (e.g. host-based or network-based) and traffic properties description (e.g. packet-based analysis or aggregated connections flows). All the above mentioned aspects are dissuaded in the consecutive subsections.

The Signature-based category of cyber attacks detection typically include Intrusion Prevention and Detection Systems (IDS and IPS) which use predefined set of patters (or rules) in order to identify an attack. The patterns (or rules) are typically matched against a content of a packet (e.g. TCP/UDP packet header or payload). Commonly IPS and IDS are designed to increase the security level of a computer network trough detection (in case of IDS) and detection and blocking (in case of IPS) of network attacks.

Commonly the patterns an attack for IPS and IDS software are provided by experts form a cyber community. Typically, for a deterministic attacks it is fairly easy to develop patterns that will clearly identify given attack. It often happens when given malicious software (e.g. worm) uses the same protocol to communicate trough network with command and control centre or other instance of such software. However, the problem of developing new signatures becomes more complicated when it comes to a polymorphic worms or viruses. Such software commonly modifies and obfuscates its code (without changing the internal algorithms) in order to be less predictive and easy to detect.

### 4.2 Crisis Phase

In this phase risk management is not important, because it gives priority to incident management in order to solve crisis and mitigate threats by adopting proper countermeasures. However, it is worth mentioning that the emergency and contingency procedures adopted during a Crisis Phase are developed during the Pre-Crisis phase. In other words, during the Crisis phase it is not only important to have an overall situational awareness picture, but also to have a strategy to recover form crisis in the most efficient way possible. There are different models for cyber incidents handling. For instance, ENISA defines (see Fig. 8) formal manner starting from incident reporting, going through analysis and recovery, and concluding with post-analysis followed by improvements proposal. This model of cyber crisis response is widely adapted by Emergency Response Teams (CERTs). According to definition provided by ENICS [24] CERTs are the key institutions that are obliged to receive, inform and respond to cyber security incidents. At the same time, they act as educational entities in order to raise the cyber-related awareness and provide primary security service for government and citizens. Every single country that is connected to the Internet should have capabilities to respond to cyber-related security incidents. Nevertheless, not every country has such capabilities. One of the

Fig. 8 ENISA incident handling model [22]

earliest CERT teams focused on critical infrastructures was the US ICS-CERT (Industrial Control Systems Cyber Emergency Response Team) that was established in 2009 [25]. This institution aims at reducing the impact of cyber attacks. In order to achieve this goal ICS-CERT takes preventative actions such as vulnerability monitoring and reporting (each year ICS-CERT releases annual reports in order to spread the information about the security incidents).

However, before the actual incident handling will take place, usually the incident is verified and pre-classified, in order to assess its significance, severity and time constrains required to resolve it. This activity is named triage and refers to situation in which there are limited resources and the decision maker has to decide on the priorities of actions relying on the severity of the particular cases.

An important thing, which is not directly reflected by the incident handling model, is fact that CERTs also collaborate with other Computer Emergency Response Teams that are part of international or private sector institutions. This cooperation allows the CERTs to share the information about control systems-related security incidents and mitigation measures.

### 4.3 Post-crisis Phase

The post crisis phase is the phase in which threat has been eliminated and system has been repaired, thus allowing the restoration of provided services and return to usual business activities.

As recent cyber incidents show, it is important for the Critical Infrastructure operators to have employees that would be educated and skilled in cyber security aspects. The post-crisis phase is important for an organisation to draw some conclusion after the crisis and use this time as an opportunity to increase the number of cyber security professionals at various levels of skill and competence, as well as to upgrade the competence levels of the already hired staff.

In fact, learning from previous experiences is a continuous process for the organisation. According to the terminology adapted in [26] this problem can be decomposed into:

Obviously, in order to address all of above mentioned aspects, it is necessary to have resources allowing for relevant data gathering and analysis. In many cases, dedicated tools facilitating the end-user with such functionalities are used. Particularly, in the post-crisis phase it is necessary to collect the lessons learnt and analyse the overall crisis scenario from wider perspective in order to identify root cause of the crisis and procedural pitfalls that may have been identified. In particular, a new risk analysis must be performed in order to evaluate if the previously defined security controls are still effective and to estimate whether risk levels have been changed.

### 5 Modelling Cyber Security Aspects

There are different approaches to model cyber security aspects. Depending on the goal of the modelling process one can divide these as problem of modelling the (Fig. 9):

### 5.1 Network Modelling

As for the Network modelling, one can use different network simulation tools (e.g. NS3, NS2, OPNET, NetSim) to analyse selected impacts of cyber attacks on modelled network. For instance, in [27] authors used NS2 simulator to predict the impact of malware propagation, Denial of Service and Man In The Middle attacks on SCADA systems. The authors measure the impact among others in terms of loss of control, Quality of Service (QoS), and number of dropped packets.

Different tools suits allow the user to model different aspects of telecommunication network with a varying granularity using different modelling techniques. In the NS3, the topology and the configuration of the simulation are provided either in \*.py (python) or in \*.cc (c/c++) files. Commonly, these files contain the following information:

Fig. 9 Different approaches to model cyber aspects

In NS3 the term node is used to name an abstract device connected to a network such as end-users hosts, end-systems, routers, switches, hubs etc. Since NS3 does not focus on Internet technologies only, it is the responsibility of simulation creator to define nodes properly by adding applications, protocols stack, etc. In NS3 the concept of application is defined as an element that runs the simulation. It is the basic abstraction of a user program, which generates some network traffic. The NS3 allows the user to use additional tools to visualise simulation at a runtime (see PyViz in Fig. 10) and to prototype the network topology with GUI-enabled software.

### 5.2 Cyber Risk Assessment

The goal of the tools and methods used for the modelling the cyber risk is similar to the previous approach, however the approach is substantially different. For instance, the aim of tools like Haruspex [28], is to evaluate the probability that an adversaries can implement successful cyber attack against a system. Haruspex implements the simulation as model comprising of threat agents and the attacks they convey. The system is modelled as a set of components interacting through channels. As a final result, the tool collects relevant statistical data from the simulations.

Similar approach to probability-based risk evaluation is presented in [29]. The authors have adapted an ontology to model the system, its key components and interaction between them. Main concepts, which compose main classes of proposed

ontology (see Fig. 11) are Assets (anything that has value to the organization), Vulnerabilities (include weaknesses of an asset or group of assets which can be exploited by threats), Threats (potential cause of an unwanted incident which may result in harm to a system or organization), Safeguards (practices, procedures or mechanisms that reduce vulnerabilities).

As argued by the authors, the ontology-based data models allows for addressing the complexity, diversity, and sparsity of dependencies. An example of instantiated ontology classes is shown in Fig. 12.

Fig. 12 Ontology-based data model describing elements and dependencies between elements

Fig. 13 Example of analysis conveyed by DAT tool [29]

The results of the analysis conveyed with this tool is an interactive security report (see Fig. 13). It allows the operator to go through the list of identified threats and get the detailed description accompanied with security counter measure that is likely to eliminate (or decrease) given risk.

### 5.3 System Behaviour and Attacks Modelling

The underlying motivation for system and attack modelling is the evolvement of tools and techniques in the area of artificial intelligence, data mining, and classification. Those techniques allow for automated data analysis, novelty and anomaly detection without extensive understanding of the underlying data content. The anomaly-based methods for a cyber attacks detection build a model that is intended to describe normal and abnormal behaviour of network traffic.

The approach to adopt these techniques is in many cases similar. Firstly, sensors collecting relevant data are deployed across network. Typically, these data require further processing in order to extract relevant information (average value of measured physical property or number of packet transmitted, see Fig. 14).

Commonly such methods uses two types of algorithms from machine learning theory, namely unsupervised and supervised approach.

Fig. 14 A conceptual overview of on-line analysis

For unsupervised learning commonly clustering approaches are used that usually adapt algorithms like k-means, fuzzy c-means, QT, and SVM. The clustered network traffic established using mentioned approaches commonly requires decision whenever given cluster should be indicated as a malicious or not. Pure unsupervised algorithms uses a majority rule telling that only the biggest clusters are considered normal. That means that network events that happens frequently have no symptoms of an attack. In practice, it is a human role to indicate which cluster should be considered as the abnormal one.

The supervised machine learning techniques requires at least one phase of learning in order to establish the model traffic. The learning is typically off-line one and is conducted on specially prepared (cleaned) traffic traces. One of the exemplar approaches to supervised machine learning for cyber attack detection use auto regression stochastic process (AR). In literature there are also methods using Kalman filters. Recently, more gaining in popularity are solutions adapting SVM, neural networks, and ID3-established decision trees.

### 6 Ongoing Efforts

### 6.1 H2020 Work Program View on CPS Aspects

The research topics defined for the security call in Horizon 2020 programme reflect the need for securing Critical Infrastructures—both physically as well as in digital domain, preventing them from cyber-attacks. For example, the topic CIP-01-2016-2017 entitled "Prevention, detection, response and mitigation of the combination of physical and cyber threats to the critical infrastructure of Europe" addresses aspects of cyber and physical security convergence to protect installations of the critical infrastructure of Europe. The challenge related to such protection is not only addressing separately physical threats to CI (such as bombing and other terrorists acts and natural-born threats as seismic activities or floods) and cyber threats, but establishment of security management paradigms that include the combinations of both group of threats, analysis of their interconnections and cascading effects resulting from cyber or physical damages. Also, it is expected that research initiatives acting under this topic will pursuit solutions related to sharing information with the public in the region of affected installations, and the protection of rescue teams, security teams and monitoring teams. The main expected results of the research in short- and medium-term perspectives include analysis of physical/cyber detection technologies and risk scenarios in the context of a specific critical infrastructure, analysis of physical-cyber vulnerabilities of a specific critical infrastructure, development of tools, concepts, and technologies for combating both physical and cyber threats to a specific critical infrastructure. These tools should be innovative, integrated, and dedicated to prevent, detect, respond and mitigate physical and cyber threats and enabling monitoring of the environment, communication with the inhabitants in the vicinity of the critical infrastructure. In long-term perspective, achievement of convergence of safety and security standards, and the establishment of relevant certification mechanisms are expected in this area.

Another example of topic in which the importance cyber-physical security is emphasized is DS-01-2016: "Assurance and Certification for Trustworthy and Secure ICT systems, services and components". In particular, specific nature of CPS systems (that smart meters are highly connected to) as evolving, complex and dynamically changing environment makes critical security-related decisions very challenging and demanding a technology-based support.

Moreover, topics from past security call (H2020 WP2014-15) also addressed problems of cyber-physical security convergence. One of examples was DRS-12-2015 topic, entitled "Critical Infrastructure smart grid protection and resilience under smart meters threats", under which physical safety (threat of undesired physical access to smart meters) was examined alongside other cyber threats.

### 6.2 Security Standards for Critical Infrastructures

In this section we provide the short overview of wide spectrum of different standards that address the cyber (as well as physical) security aspects of critical infrastructures.

The ISA99 committee addresses the cyber security of industrial automation and control systems by its ISA/IEC-62443 series of standards. The scope of the ISA99 standards is very broad, i.e. the committee does not limit application of its standards to the specific type of plants, facilities or systems. Manufacturing and control systems to which the ISA/IEC-62443 can be applicable include hardware and software systems such as DCS, PLC, SCADA, networked electronic sensing, monitoring and diagnostic systems as well as associated interfaces used to provide control, security, and continuity of industrial processes. In the ISA/IEC-62443 series of standards physical security is not directly addressed, despite the fact that physical security highly impacts the integrity of any control system environment [30].

The NERC (North American Electric Reliability Corporation) CIP plan is a set of requirements designed to secure the assets required for operating North America's bulk electric system. This set includes 9 security standards and 45 requirements and addresses security of electric systems and the protection of the critical cyber assets operating within these systems. Cyber security training, management and crisis recovery are also included. Physical security of Critical electric systems is covered by the CIP-006-1: Physical Security of Critical Cyber Assets sub-standard [31, 32].

The IEC 62210 technical report on "Data and Communications Security" can be applied to supervision, control, metering and protection systems in electrical utilities. The report covers a broad range of security related aspects such as definitions, prioritization of threats, consequence analysis, attacks, policies and "Common Criteria" protection profile. Communication protocols used within and between systems, secure use of the systems and access to them are also discussed. Consequence analysis was adopted in the report as the security methodology for prioritization of assets and threats to the security of the some industrial protocols e.g. TC 57 protocol used for power systems management and exchange of associated information. However, as it is stated in the report, the document does not include recommendations or criteria development related to physical security of critical systems [33]. In addition, IEC 62351 is a series of technical specifications covering aspects of information security for power system control operations. Selected aspects that are discussed in IEC 62351 are authentication of data exchange (digital signatures, certificates), security of TCP/IP (e.g. encryption), networks and systems security management and key management.

Also the IEEE 1402 standard applies to the power distribution and critical energy infrastructures protection, however in a contrary to above described IEC documents, this standard addresses aspects of physical security, especially in a context of unauthorized access to electric power substations. The document describes and guides a variety of methods to prevent such substations from human intrusion [34].

### 7 Conclusions

In this paper we have described various cyber security aspects related to the cyber-physical systems and critical infrastructures. We have described current challenges related to the technical aspects as well as the European vision on that matters. As we currently observe, due to the evolution of Internet and the wide adoption of the Internet of Things concept, we may expect that in the near future the cyber security of cyber-physical systems will become of even higher importance. As gradually increasing number of elements and aspects (smart devices, homes, cars, etc.) of our lives becomes connected to the Internet, it gives new opportunities and motivations for the cyber criminals to research and to exploit technological vulnerabilities in order to gain economical profits. Those attempts cannot be successful with regards to critical infrastructures and homeland security.

Therefore new technological and organizational solutions are needed for cyber physical systems protection. There are also many urgent questions and aspects to be addressed by nations and companies, such as if the standards and guidelines for cyber security should be obligatory and mandatory (which also involved controlling organizations and possible penalties), or if those should rather be voluntary. Moreover, the minimal security standards have to be defined. Another difficulty is to find the right balance for the appropriate level of details of recommendations and standards. Should those be rather general, universal and high level (for further customization for each organization), or should those be as detailed as possible mentioning particular technologies and solutions to be applied. At the nations level, the decision should be also made who (which organizations) should issue such standards and guidelines. Should those be sectorial organizations (e.g. for standards for energy, healthcare, financial sector etc.) or rather ministries covering wider range of applications?

However, the most crucial aspects now for protecting critical infrastructures is the awareness building. Without the understanding and awareness of all the actors (private CI owners, governments, managers, employees at all levels etc.) our critical infrastructures will be still endangered by the cyber and physical attacks.

Acknowledgement and Disclaimer This chapter was derived from the FP7 project CIPRNet, which has received funding from the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 312450. The contents of this chapter do not necessarily reflect the official opinion of the European Union. Responsibility for the information and views expressed herein lies entirely with the author(s).

The work is also funded by Polish National Centre for Research and Development (NCBiR) from funds for science in the years 2013–2016 allocated for the international projects.

### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Chapter 8 Verification and Validation for CIPRNet

Jeroen Voogd

Abstract In this chapter it is shown that if an appreciable risk is present in the use of Modelling and Simulation (M&S), Verification and Validation (V&V) should be employed to manage and mitigate that risk. The use of M&S in the domain of critical infrastructure (CI) will always be accompanied by such a risk. It is important that a structured approach to V&V is used in order to be more effective and more efficient than just testing without a clear plan. The Generic Methodology for V&V (GM-VV) is a recommended practise in the international M&S community and adopted by large organisations such as NATO. The GM-VV has a number of concepts that allow for a structured approach to V&V. A structured approach to V&V such as the GM-VV leads to a set of handles that allow the best choices for V&V techniques to employ. The choice for a specific technique is dependent on a number of factors such as the needed certainty, the expected residual uncertainty of the proposed technique and its requirements in terms of costs, real-world knowledge, etc. This chapter is divided in 4 parts. The first part has the take away message "You have to do Verification and Validation because there is risk involved", the second "You have to do it in a structured way if you want to do it more effective and more efficient" and the third "You have to choose the appropriate Verification and Validation technique to balance risk, effectiveness and efficiency." In the last part some conclusions are drawn.

### 1 Do V&V If There Is Risk Involved

In this section we first briefly explain what Modelling and Simulation (M&S) are. It will be made clear that if the M&S results are applied in the real world, M&S Use Risk has to be considered. To manage this risk it is required to have insight into the

J. Voogd (&)

Modelling, Simulation and Gaming, TNO, Oude Waalsdorperweg 63, PO Box 96864, 2597 AK The Hague, The Netherlands e-mail<EMAIL_ADDRESS>
© The Author(s) 2016 R. Setola et al. (eds.), Managing the Complexity of Critical Infrastructures, Studies in Systems, Decision and Control 90, DOI 10.1007/978-3-319-51043-9\_8

quality and associated risk of the M&S system over its entire life cycle. Verification and Validation are the two processes to obtain this insight. These processes are also briefly explained.

### 1.1 Modelling and Simulation

Modelling and simulation start—as all system engineering projects do—with a purpose. Then the modelling starts. A possible definition of a model is that it is an abstract representation or specification of a system. A model can represent a system that exists in our material world but can also represent not yet existing systems or combinations thereof. That part of (the imagined) reality that the model is supposed to represent is called the simuland. Then further abstractions are applied to the simuland in order to make the model suited for its purpose. Abstraction in this context is a process in which a relative sparse set of relevant (sub)systems, relationships and their inherent qualities are extracted or separated from the more complex (imagined) reality (Fig. 1).

In a simulation the model is used to replicate the simuland behaviour. Thus a simulation is a method, software framework or system to implement and evaluate a model over time i.e., it is a system in which a model is made to execute and is exercised. This model in its executable form is called the M&S system.

The M&S system is provided with input and its output is used within a certain context provided by a frame such as shown in Fig. 2 which is called the Simulation Frame. The model that is executed in the simulation is controlled and observed by means of its ports (ellipses in Fig. 2). Through these ports simulation data, stimuli or settings are entered into the model and simulation output leaving the executed model is observed. During the simulation the model behaves according to a dynamics that represent the state change and behavioural properties of the simuland. The notion of time, behavioural representation and frame are fundamental characteristics of a simulation.

To properly replicate the simuland for the intended use, the model is configured, controlled and stimulated by the Simulation Frame by means of input trajectories,

Fig. 1 Modelling is taking abstractions

Fig. 2 Relation between simulation frame and the M&S system

scenario's, parameters, environment variable settings and experimental control settings. Furthermore, environment disturbances coming from connections with live entities may impact the behaviour of the M&S system. During the execution of the model, human input can become part of the displayed behaviour. This can be from trainees, but also from operators such as opponent players to provide stimuli to the trainees or Subject Matter Experts (SMEs) that interfere with the execution of the simulation for some purpose dictated by the Simulation Frame (e.g., keeping the execution within a desired regime).

So, all in all the M&S process consists of cutting away all elements of the real and imaginary world that are not needed for the purpose at hand, then apply various abstraction techniques to make the model suited for use, then the model is executed in order to obtain results (e.g. a trained operator or an optimized CI configuration). These results are then applied in some form or another in the real world.

And that last part is exactly where the risk exists. When the M&S-based solutions to problems are applied in the real world there is a risk that those results are not fully appropriate. There can be a number of causes: the purpose for the M&S endeavour was not what was ultimately needed, maybe the simuland did not contain all needed elements of the real and imaginary world, maybe some abstractions were too large and important details were abstracted away, maybe the implementation and execution of the model or the interpretation of it's results introduced errors.

If the results of M&S are never used in the real world, e.g. if it is used for entertainment purposes or as a hobby, then there is no problem. But for CI this is not the case. The possible sources of errors may for example lead to operators of actual CI taking wrong actions if M&S was used for their training. If it is used for determining the best possible configuration of interconnecting CI, it may result in a system that performs less than desired.

The conclusion is that we need to be sure that the M&S results are fit for purpose before actually applying them to the real world. There are two processes that do exactly that: Verification and Validation. Therefore the take away message of this part is "You have to do Verification and Validation because there is risk involved".

### 1.2 Verification and Validation

There is no true consensus on the exact definitions of what Verification and Validation (V&V) are. Some definitions are:

Verification. The process of providing evidence justifying the M&S system's correctness [1]. Confirmation, through the provision of objective evidence that specified requirements have been fulfilled [2]. The process of determining that a model or simulation implementation and its associated data accurately represent the developer's conceptual description and specifications [3]. The process of determining the degree that a model, simulation, or data accurately represent its conceptual description and its specifications [4].

Correctness. The extent to which an M&S system implementation conforms to its specifications and is free of design and development errors [1].

Validation. The process of providing evidence justifying the M&S system's validity [1]. Confirmation, through the provision of objective evidence that the requirements for a specific intended use or application have been fulfilled [2]. The process of determining the degree to which a model or simulation and its associated data are an accurate representation of the real-world from the perspective of the intended uses of the model [3]. The process of determining the degree to which a model, simulation, or data is an accurate representation of the real world, from the perspective of the intended purpose of the model, simulation or data [4].

Validity. The property of an M&S system's representation of the simuland to correspond sufficiently enough with the referent for the intended use [1]. The property of a model, simulation or federation of models and simulations representations being complete and correct enough for the intended use [5].

A more intuitive explanation can be seen in Fig. 3. There the blue arrows indicate verification: starting from the specification of the M&S system, a simuland is made, which, after modelling, results in an implementation that can be executed to obtain M&S results. At each step one can check if the transformation has been done correctly and the goal is to show that the M&S system adheres to the specification. In literature one often finds that verification assesses if the M&S is built and used right.

Validation, which is the red arrow in Fig. 3, on the other hand, is making sure that the M&S results produced by the M&S system are fit for the customer's needs

Fig. 3 Verification (blue arrows) and Validation (red arrow)

in the real world. In literature one often finds that validation assesses if the right M&S is built or procured.

During the execution of V&V it may (and usually will) happen that elements of the M&S System or its use, are found that are not correct or that contribute negatively to the customer's need. Identifying these sources of risk are necessary in order to start managing them. In short: doing V&V provides insight into and advice on the quality of the M&S system over its entire life cycle, and the associated risks.

When studying literature on V&V another term is often found: accreditation. This is, however, a somewhat problematic concept. According to [3] accreditation is "The official certification that a model or simulation and its associated data are acceptable for use for a specific purpose." One problem is that in most countries and for most application domains of M&S there is no official body that can issue such M&S certificates. And besides often the word accreditation is reserved for the official recognition that an organization is allowed to issue certificates. The official certification is called just that: certification (and not accreditation).

In this text the word accreditation or certification is not used. What is assumed is that the result of doing V&V is a body of knowledge on the quality and deficiencies and their associated risks, based on which the customer can decide whether to accept the M&S system or not.

### 1.3 But How to Do V&V, and How Much?

As described above, for M&S applied to CI it is necessary to identify and manage risk. V&V can be used for that but the question is how should the V&V be approached and how much effort should be spend on it.

The second question is difficult to answer because there is no general answer. Doing V&V can be costly and it should be in balance with the M&S Use Risk involved. Another aspect that has to be considered is the risk tolerance, e.g. in the form of insurance, of the user. What is most important is that the cost spent on the V&V effort should be in balance with the possible costs associated with the risk. The cost of doing V&V should also be significantly less than the possible saving due to risk reduction.

The first question—how to do V&V—is easier to answer. In practice it is often observed that those who develop the simulation also perform the V&V activities. Although they often do a good job, the result does leave something to be desired. After the V&V activities it is not clear anymore which tests were performed and why. The documentation is more often than not a bunch of files on the developer's computer. If after some time things need to be changed in the M&S system and thus some additional V&V activities have to be performed, it is not clear which of the results from the initial V&V activities are still applicable and which tests need to be redone. In short: there is no traceable path from the user's goal to the tests to the results, and no re-usable documentation exists. An unstructured approach to doing V&V may be effective, but often this cannot be shown. It may also be efficient at first, but again it cannot be shown that the most efficient way of doing V&V tests has been chosen.

So, the question arises if there is a V&V approach that does work well. The first thing to look at is if there are appropriate standards for doing V&V. It turns out that there are a number of V&V approaches for M&S, but these are often domain specific, strongly tied to a specific technology or developer oriented. If that is what is needed, then use them. In general, however, it is not advised to use a developer-oriented approach because the link with the user's goal is not clear. If the V&V effort does not involve a specific domain or technology for which a V&V standard is available, then a more general V&V approach is required.

In order to make sure the V&V effort is effective, the starting point has to be the goal of the user, or to be more precise: the M&S Use Risk associated with the user's goal. From that starting point criteria need to be derived that show what needs to be tested. That derivation and choosing V&V techniques for doing the tests needs to be within the limits of the resources available for the V&V effort, which in practice is always rather limited. The results of the V&V effort need to be documented in such a way that all results can be traced back to the tests and the user's goal, and it should also be such that re-use at a later data is possible. In short: the V&V approach must result in the biggest bang for the buck as well as allow full traceability, otherwise serious questions can be raised about the effectiveness and efficiency of the V&V effort.

The take away message of this section is "You have to do V&V in a structured way if you want to do it more effective and more efficient".

### 2 Do V&V in a Structured Way to Be More Effective and Efficient

The choice of which V&V method works best in a given situation depends on the individual needs and constraints of an M&S organization, project, application domain or technology. Moreover, V&V usually requires a complex mixture of various activities, methods, tools, techniques and application domain knowledge, which are often tightly coupled with the M&S development process. Therefore, many different approaches to V&V exist that rely on a wide variety of different V&V terms, concepts, products, processes, tools or techniques. In many cases, the resulting proliferation restricts or even works against the transition of V&V results from one M&S organization, project, and technology or application domain to another. Furthermore, history shows that V&V is often more of an afterthought than a built-in part of an M&S development, employment and procurement policy.

The purpose of the Generic Methodology for V&V (GM-VV) is to address these issues by means of providing general applicable guidance for V&V that:

GM-VV is not aimed to replace the existing V&V approaches, methodologies, standards or policies of M&S organizations, technology and application domains; nor is GM-VV's intent to substitute common enterprise or project management practices prevalent within M&S client or supplier organizations. In addition, GM-VV is not intended to be prescriptive, in that it does not specify a single concrete or unique solution for all V&V applications. Rather, the GM-VV should be tailored to meet the needs of individual V&V applications.

The GM-VV provides a technical framework that focuses on M&S V&V practices. Though interrelated, acceptance decision processes and associated practices such as M&S accreditation and certification are outside the scope of the methodology. GM-VV attains its generic quality from a technical framework that consists of three subparts: the conceptual, implementation and tailoring framework (Fig. 4). This framework is rooted in established international standards and other related practices. The conceptual framework provides the terminology, concepts and principles to facilitate communication and a common understanding and execution of V&V within an M&S context. The implementation framework translates these concepts and principles into a set of generic building blocks to develop consistent V&V solutions for an individual M&S organization, project, and technology or application domain. GM-VV provides a tailoring framework that utilizes these building blocks to develop and cost-efficiently apply such V&V application instances. As such, the GM-VV provides a high-level framework for developing concrete V&V solutions and conducting V&V, into which lower-level practices (e.g., tools, techniques, tasks, acceptability criteria, documentation templates) native to each individual M&S organization, project, technology or application domain can easily be integrated.

Each of the three frameworks will be described in sections below.

Fig. 4 GM-VV technical framework design and operational use concept

### 2.1 Conceptual Framework

This section discusses the GM-VV conceptual framework. This framework provides fundamental and general applicable terminology, semantics, concepts and principles for V&V. The purpose of the framework is to facilitate communication, understanding and implementation of V&V across and between different M&S contexts (e.g., organizations, application domains, standards, technologies). The framework is the foundation upon which the GM-VV implementation framework rests.

### 2.1.1 Links to Systems Engineering

Within the GM-VV, M&S systems are considered to be systems of systems that have a lifecycle and are subject to system engineering practices. Moreover, models and simulations are considered to be part of a larger system in which they are used. From this perspective, M&S is a systems engineering specialization. V&V is an intrinsic part of the systems engineering process [6–9]. Therefore, the GM-VV considers the V&V of M&S as a specialization of systems engineering V&V. Hence, the GM-VV can be integrated with, complement or extend the V&V processes within such existing systems engineering methodologies or standards.

### 2.1.2 M&S-Based Problem Solving Approach

The basic premise of the GM-VV is that models and simulations are always developed and employed to fulfil the specific needs of their end users (e.g., trainers, analysts, decision makers). Modelling and simulation is thus considered to be a problem solving process that transforms a simple statement of an end user's need into an M&S-based solution for the problem implied in the need. The GM-VV assumes that V&V always takes place within such larger context. This context is abstracted by means of defining four interrelated worlds (Fig. 5). Together, these four worlds define a generic lifecycle and process view of M&S-based problem solving. A view that serves as a common basis, in which V&V for M&S (e.g., concepts, principles, processes, products, techniques) can be understood, developed or applied.

These four worlds can be described as follows:

Fig. 5 Four worlds view of M&S based problem solving

Stakeholders from within the Problem World are those parties involved in the complete solution (e.g., organizations) such as education centres and their trainers in case of training, analysts in case of an analysis problem. Stakeholders from the Real World or their experts are typically also involved in the Problem World.

When the M&S problem solving process described by the four-worlds view is properly executed, the resulting solution should satisfy the originally identified needs with a minimal level of (use) risk in the Real World.

The M&S system, M&S requirements, M&S results and other development artefacts (e.g., conceptual model, software design, code) are thus always directed toward contributing to and satisfying the Real World operational needs. The degree of success of such M&S in satisfying these needs depends on how well they are specified, designed, developed, integrated, tested, used, and supported. These M&S activities require the contribution of individuals or organizations that have a vested interest in the success of the M&S asset, either directly or indirectly. An individual or organization with such interest is referred to in GM-VV as a stakeholder. Stakeholders can play one or more roles in each of the four worlds such as M&S user/sponsor, supplier, project manager, software developer, operator, customer, or subject matter expert (SME). Depending upon their role, stakeholders may hold different responsibilities in the M&S life-cycle processes, activities or tasks.

### 2.1.3 V&V Problem Solving Approach

Within the four-world context, stakeholders exist who are responsible for making acceptance decisions on the use of M&S. Within the GM-VV, these stakeholders are referred as V&V User/Sponsor. In this context the V&V User/Sponsor could be an M&S User/Sponsor, Accreditation Authority or any other domain specific role that uses the outcomes of the V&V. V&V Users/Sponsors face the problem of having to make a judgment on the development and suitability of the M&S system or results for an intended use. The key issue here is that it is not possible to demonstrate with absolute certainty that the M&S system or results will meet the Real World needs prior to its actual use. Consequently, there is always a probability that the M&S-based solution is not successful when used (i.e., fails). Such a failure would result in an undesirable impact (i.e., a risk) on the operational environment. Therefore, an M&S system or result is only acceptable to the V&V User/Sponsor if he or she has sufficient confidence that the use of an M&S system or result satisfies the Real World needs without posing unacceptable risks (e.g., costs, liabilities). This M&S acceptability is something relative to different V&V Users/Sponsors: what is acceptable to one V&V User/Sponsor may not be acceptable for another. The V&V User/Sponsor's decision-making process therefore requires appropriate evidence-based arguments to justify his or her acceptance decision.

The basic premise of GM-VV is that V&V are performed to collect, generate, maintain and reason with a body of evidence in support of the V&V Users/Sponsors acceptance decision-making process. Here, validation is referred to as the process that establishes the V&V User/Sponsor's confidence as to whether or not they have built or procured the right M&S system or result for the intended use (i.e., M&S validity). In other words "Did we build the right M&S system?". To ensure that the M&S system or results at delivery can be demonstrated to be valid, it is necessary to ensure that the M&S system is built and employed in the right manner. Here verification is referred to as the process of establishing V&V User/Sponsors confidence in whether the evolving M&S system or result is built right (i.e., M&S correctness). In other words "Did we build the M&S system right?". The GM-VV considers V&V as a specific problem domain of M&S with its own needs, objectives and issues. This domain is referred to as the V&V World (Fig. 6).

The V&V world groups the products, processes and organizational aspects that are needed to develop an acceptance recommendation that can be used by the V&V User/Sponsor in his or her acceptance decision procedure(s). This recommendation included in a V&V report is the key deliverable of a V&V effort and contains evidence-based arguments regarding the acceptability of an M&S system or results. Here the GM-VV premise is that the acceptance decision itself is always the responsibility of the V&V User/Sponsor and decision procedure(s) may involve trade-off aspects beyond the V&V effort scope.

The development of an acceptance recommendation in the V&V world is driven by the V&V needs that are traceable to the V&V User/Sponsor's acceptance decision or procedure(s) needs (e.g., budget, responsibilities, risks, liabilities). Therefore, the extent, rigor and timeframe of a V&V effort depend on these needs. Depending on these needs, the V&V effort could span the whole or specific M&S lifecycle phase of the four worlds; could focus on one specific or multiple (intermediate) M&S products; and should match the development paradigm that was used (e.g., waterfall, spiral). Each case may require a separate acceptance recommendation with its own scope and development timeline. Moreover, the way the V&V effort interacts with the four M&S-based problem worlds also varies from

Fig. 6 V&V world and four-world interfacing

case to case. These mutual dependencies are depicted in Fig. 6 with bidirectional arrows that interface the V&V world with each of the four M&S-based problem solving worlds. Two classical types of V&V that can be identified based on the time frame of their execution are [6, 10–12]:

The GM-VV supports both V&V time frames but is not limited to these distinct types. A V&V effort can be post hoc, concurrent, iterative, recursive or even be a recurrent effort in the case where legacy M&S products are updated or reused for a different intended-use.

### 2.1.4 Acceptance Recommendation, Acceptability Criteria and Evidential Quality

The objective of a V&V effort is to develop evidence upon which an acceptance recommendation is based. This V&V objective is articulated as an acceptance goal. This high-level goal should be translated into a set of concrete and assessable acceptability criteria for the M&S system or result(s). Relevant and convincing

Fig. 7 Utility, validity, correctness and V&V quality criteria relationships

evidence should then be collected or generated to assess the satisfaction of these criteria. When it is convincingly demonstrated to what extent the M&S system or result(s) does or does not satisfy all these acceptability criteria, a claim can be made on whether or not the M&S system or result(s) is acceptable for its intended use (i.e., acceptance claim).

The GM-VV identifies three types of M&S properties for which acceptability criteria could be set (Fig. 7):

These three types of M&S properties include but not limited to capability, accuracy, usability and fidelity [13, 14]. To make an acceptance decision, the V&V User/Sponsor needs to know whether the M&S system or results are (un)acceptable, as well as the evidential value of this acceptance claim (i.e., strength). The required evidential strength to establish sufficient trust in such a claim depends on the use risks and the V&V User/Sponsor responsibilities (i.e., liability). The convincing force that can be placed on such a claim depends on the quality of the whole V&V effort. For this purpose, the GM-VV identifies quality properties that can be associated with identifying and defining the acceptability criteria; and developing convincing evidence for demonstrating their satisfaction (Fig. 7).

• V&V Quality: this property refers to how well the V&V effort is performed (e.g., rigor) with regard to developing the acceptability criteria, collecting evidence, and assessing to what extent the M&S satisfy the acceptability criteria (e.g., evidential value, strength).

Typical examples of V&V quality properties are the completeness, correctness, consistency, unambiguous and relevance of the acceptability criteria or their supporting items of evidence. In the process of collecting or generating evidence, quality properties could comprise independence of applied V&V techniques or persons, knowledge gaps and uncertainties of referent data for the simuland [15], skill level of V&V personnel, and reliability and repeatability of V&V techniques. Relevance and warrants for any assumption made in a V&V effort could also be addressed in the form of quality properties.

The defined acceptability criteria, the collected evidence and assessment of the satisfaction of these criteria are the basis for developing the arguments underlying the acceptance claim. This acceptance claim provides the V&V User/Sponsor with a recommendation regarding the acceptability of the M&S system or result for the intended use. In practice, an acceptance recommendation is not necessarily just a yes or no claim, in the sense that an M&S system or results can be accepted only if it meets all of the acceptability criteria and cannot be accepted if it does not. Meeting all the acceptability criteria means the claim can be made that the M&S system or result should be accepted to support the intended use without limitations. In case not all acceptability criteria are met, alternative weaker acceptance claims with underlying arguments can be constructed. Such alternative acceptance claims could, for example, provide recommendations regarding conditions or restrictions under which the M&S system or result can still be used (i.e., limit the domain of use); or on modifications that, when implemented, will lead to an unconditionally acceptable M&S system or results for the intended use. Another rationale for alternative acceptance claims is when convincing or sufficient evidence is lacking (e.g., access to data prohibited, or referent system unavailable for testing). In any case, an acceptance recommendation always requires well-structured supporting arguments and evidence for the V&V User/Sponsor to make the right acceptance decision. Depending on the identified M&S use risk, the V&V User/Sponsor can also decide not to take any actions when not all acceptability criteria are met by the M&S system. In that case, the V&V User/Sponsor simply accepts the risks associated with the M&S system use.

### 2.1.5 V&V Argumentation Approach: Structured Reasoning with Arguments

Developing an acceptance recommendation that meets the V&V User/Sponsor needs usually involves the identification and definition of many interdependent acceptability criteria, particularly for large-scale and complex M&S systems or for M&S-based solutions used in safety–critical, real-world environments. Demonstrating the satisfaction of acceptability criteria requires evidence. Collecting the appropriate evidence is not always simple and straightforward, or even not always possible due to various practical constraints (e.g., safety, security, costs, schedule). In many cases, the collected evidence comprises a large set of individual items or pieces of evidence that may be provided in different forms or formats, and may originate from various sources (e.g., historical, experimental data, SME opinion). Moreover, the strength of each item of evidence may vary and the total set of collected evidence may even contain contradicting items of evidence (i.e., counter evidence). The quality of this effort determines the value of an acceptance recommendation for the V&V User/Sponsor. Therefore, the arguments underlying an acceptance recommendation should be developed in a structured manner using a format where the reasoning is traceable, reproducible and explicit. Alternative approaches to implement such reasoning exist and may be incorporated within the GM-VV technical framework to tailor it the specific needs of an M&S organization or domain. An example of such an approach is the V&V goal-claim network approach (Fig. 8). A V&V goal-claim network is an information and argumentation structure rooted in both goal-oriented requirements engineering and claim-argument-evidence safety engineering principles [16–19].

Fig. 8 V&V goal—claim network

Figure 8 provides an abstract illustration of a V&V goal-claim network. The left part of the goal-claim network is used to derive the acceptability criteria from the acceptance goal; and deriving solutions for collecting evidence to demonstrate that the M&S asset satisfies these criteria as indicated by the top-down arrows. The acceptance goal reflects the V&V needs and scope (e.g., simuland, intended use). Evidence solutions include the specification of tests/experiments, referent for the simuland (e.g., expected results, observed real data), methods for comparing and evaluating the test/experimental results against the referent. Collectively, they specify the design of the V&V experimental frame used to assess the M&S system and its results. When implemented, the experimental frame produces the actual V&V results. After a quality assessment (e.g., for errors, reliability, strength), these results can be used as the items of evidence in the right part of the goal-claim network. These items of evidence support the arguments that underpin the acceptability claims. An acceptability claim states whether a related acceptability criterion has been met or not. Acceptability claims provide the arguments for assessing whether or to what extent the M&S system and its results are acceptable for the intended use. This assessment, as indicated by the bottom-up arrows in Fig. 8, results in an acceptance claim inside the V&V goal-claim network. As such a V&V goal-claim network encapsulates, structures and consolidates all underlying evidence and argumentation necessary for developing an appropriate and defensible acceptance recommendation. The circular arrows in Fig. 8 represent the iterative nature of developing a V&V goal-claim network during planning, execution and assessment phases of a V&V effort.

### 2.1.6 V&V Organizational and Management Approach

In order to facilitate efficient and high quality V&V, the V&V effort inside the V&V world should be executed in a controlled and organized way. The basic premise of the GM-VV is that the acceptance recommendation for an M&S asset is developed and delivered by means of a managed project. Moreover, GM-VV assumes that V&V is conducted by a person, a team of people or a dedicated organization with assigned responsibilities, obligations and functions. Therefore, GM-VV identifies three organizational levels at which V&V efforts can be considered. In order of the lowest to the highest organizational level these levels are:

The core GM-VV concept on the V&V project level is the concept of a managed project. A V&V project can be viewed as a unique process comprised of coordinated and controlled activities that address: V&V effort planning in terms like cost, timescales and milestones; measuring and checking progress against this planning; and selecting and taking corrective actions when needed. A V&V project could be a separate project alongside the M&S project of which the M&S asset is part, or be an integral part of this M&S project itself (e.g., subproject, work package). A separate V&V project is particularly relevant in the case when a level of independence must be established between the M&S development and V&V team/organization. On the V&V project level, GM-VV also provides derived concepts such as a V&V plan and report to manage the technical V&V work.

For CIPRNet all three levels are important. For CI it is important to have a good set of tools and techniques to do the technical V&V activities. Since with the application of M&S systems for serious CI application there is always M&S Use Risk involved, for each project run by the to be established EISAC (European Infrastructures Simulation and Analysis Centre), V&V activities should be executed. A project approach is suited for that. Doing V&V from within EISAC means that EISAC should have support for the V&V activities at the highest level: the enterprise level.

The core GM-VV concept on the V&V enterprise level is the concept of an enterprise entity. A V&V enterprise entity can be viewed as an organization that: establishes the processes and lifecycle models to be used by V&V projects; initiates or defers V&V projects; provides resources required (e.g., financial, human, equipment); retains reusable knowledge and information from current V&V projects; and leverages such knowledge and information from previous V&V projects. The V&V enterprise provides the environment in which V&V projects are conducted. GM-VV defines two types of enterprise entities:

A V&V agreement is arranged between a V&V client and V&V supplier to provide products and/or services that meet the V&V client's needs. Both these V&V entities could be organizations (e.g., companies) separate from the organization that develops or acquires M&S or it could be different units (e.g., department, division, group) within a single M&S supplier or client organization. Typically, a separate V&V supplier is an organization that has the provision of independent V&V products and services to external V&V clients as its core business. Though depending on their business model, an M&S supplier or client organization could have their own V&V supplier entity that may provide V&V services and products to internal and external V&V clients alike.

### 2.1.7 V&V Levels of Independence: Acceptance, Certification and Accreditation

An independent V&V (IV&V) authority is often described as an organization or a person that is employed to conduct V&V, independent of the developer's team or organization [6, 10, 12]. The need for IV&V is mostly driven by:

In practice however, it is highly incumbent upon the V&V User/Sponsor acceptance decision needs and complexity of the M&S system as to which parts and to what extent V&V should be conducted in an independent manner. Therefore, the GM-VV adopts a sliding scale of independence for V&V [15], which can be selected accordingly to match the V&V needs. The justification and selection of a proper level of independence is supported within GM-VV through the use of the V&V argumentation network. Within this sliding scale for independent V&V, certification and accreditation can be located in the right part of the scale (Fig. 9).

### 2.1.8 V&V Information and Knowledge Management

V&V of M&S is an information and knowledge intensive effort. In particular, during the V&V of large scale, distributed or complex M&S applications, care must be taken to preserve or reuse information and knowledge. Therefore, GM-VV

Fig. 9 Levels of independent V&V

applies the memory concept on both the V&V project and enterprise levels. A memory is viewed as a combination of an information and knowledge repository and a community of practice [20]. The repository is a physical place where information, knowledge objects, and artefacts are stored. The community of practice is composed of the people who interact with those objects to learn, understand context and make decisions.

The V&V project memory provides the means to manage information and knowledge produced and used during the lifetime of an individual V&V project. V&V is often an iterative and recurrent process linked to an M&S system's life-cycle, hence V&V products for an M&S system may have different configurations. Therefore, a V&V project memory may also retain records on possible different V&V product configurations. The V&V enterprise memory retains the total body of information and knowledge from past and current V&V projects to sustain and support the cost-effective execution of future V&V projects. Such reusable information could be, for example, M&S technology or domain specific recommended practices, acceptability criteria, V&V goal-claim network design patterns, V&V tools and techniques, or policies and standards. On a more strategic level, a V&V enterprise memory could retain information and knowledge on V&V project costs and maturity as well.

### 2.2 Implementation Framework

The GM-VV implementation framework translates the GM-VV basic concepts into a set of generic V&V building blocks (i.e., components). These may be used to develop a tailored V&V solution that fits the V&V needs of any particular M&S organization, project, application, and technology or problem domain. The implementation framework has three interrelated dimensions: product, process and organization (Fig. 10). The underlying principle of this framework is that the V&V needs of the V&V User/Sponsor in the M&S four-world view are addressed by one or more V&V products, those being the V&V report and possibly other custom V&V products the V&V User/Sponsor may need. These V&V products in general require intermediate products (i.e., information artefacts) and associated processes to produce them. The V&V processes are executed by a corresponding V&V organization that is responsible for the development and delivery of the V&V products. In general the V&V effort should result in a V&V report to be delivered to the customer containing one or more of the information artefacts. Individual needs will drive which V&V products are required.

As indicated in Fig. 10, the GM-VV implementation framework consists of three key dimensions:

The V&V effort culminates in a V&V report that is comprised of the information generated throughout the execution of the V&V and acceptance decision-support process (Fig. 6). The following sub-sections provide an overview of the information artefacts, activities and roles that are implemented or produced during this execution. They are ordered according to the GM-VV technical, project and enterprise levels.

It is important to re-emphasize the tailorable nature of the methodology. GM-VV provides all the elementary information artefacts, activities, tasks and roles to address the most common technical, project and enterprise level aspects of a V&V effort. Depending on the M&S project and organizational needs one could choose not to implement all GM-VV components or one could choose to adjust them accordingly. This is particularly relevant for M&S organizations that already have some project and enterprise level components in place, and only require technical level V&V (intermediate) products, processes and roles to conduct their V&V effort. The overall tailoring and application concepts of the GM-VV implementation framework are provided in the next section.

### 2.3 Tailoring Framework

GM-VV recognizes that a particular M&S organization, project, application, technology or problem domain may not need all these components or use them directly as-is. Therefore, the GM-VV components are intended to be selected, combined and modified accordingly, to obtain an effective and efficient V&V effort of sufficient rigor. This is particularly relevant for M&S projects and organizations that already have some project and enterprise level components in place, and only require technical level V&V (intermediate) products, processes and roles to conduct their V&V effort.

The basic premise of the GM-VV tailoring concept is that the GM-VV should first be cast into a concrete V&V method fit for an organization or application domain, and secondly this instance should be optimized for a V&V project. This tailoring concept is implemented by means of a framework that refers to all three levels of the GM-VV implementation framework. The objective of this GM-VV tailoring framework is to adapt each GM-VV (intermediate) product, process and role to satisfy the specific requirements and constraints of:

As described above tailoring is accomplished in two phases. In the first phase of the GM-VV tailoring framework, the implementation framework components are utilized to establish concrete V&V solution instances on one or more of the three organizational levels (i.e. a permanent V&V organization, V&V project or technical V&V approach). In here, the GM-VV recognizes that a particular M&S organization, project, technology or problem domain may not need all three organizational levels or all components on a single organizational level nor even use them directly as-is. Therefore, the GM-VV implementation framework organizational levels and components are selected, combined and modified accordingly, to obtain a concrete tailored V&V solution. For instance an M&S organization may already have an M&S project and enterprise level in place, and only require technical level V&V (intermediate) products, processes and roles to conduct their technical V&V work. Successful application of the first phase of the tailoring framework results in a modified or new V&V solution instance conforming to the GM-VV architectural templates (i.e. in a structure and organizational manner). Four tailoring approaches can be used for this: extension, omission, specialization and balancing, which are discussed below.

In the second phase these same tailoring approaches are applied throughout the operational lifetime (i.e. permanent organization or project) or execution (i.e. technical approach) of each V&V solution instance. This type of tailoring comprises run-time optimization of the instantiated V&V processes at all three organizational levels. At a technical level this could imply the application of a risk-based V&V approach, such as the MURM [21], to prioritize the acceptability criteria, allocate and specific V&V techniques and tools based on V&V User/Sponsor risk tolerance levels. On the project level this could be the alignment of technical V&V activities with the progress of the M&S system's life-cycle phases, balance and allocate the available V&V resources to each phase M&S life-cycle or (work) products. On the enterprise level this could mean balancing the cost-risk of new investments in training of personnel or V&V tool infrastructure development against a future V&V project order intake volume.

The GM-VV tailoring framework applies four basic tailoring approaches:

	- M&S use-risk tolerances and thresholds
	- criticality and scope of the acceptance decision
	- scale and complexity of the M&S system
	- information security, with

V&V project resource variables such as

Tailoring by these four approaches should be performed in accordance with the three dimension design principle of the GM-VV implementation framework (Fig. 10), to obtain a consistent and coherent V&V method and project. For example, each new or specialized product needs a corresponding process (activities, tasks) and role (responsibilities, obligations).

Successful application of the tailoring framework results in a modified or new V&V method instance conforming to the GM-VV. This consists of concrete V&V organization, products and processes, which should achieve the V&V objectives of an M&S organization, project, technology, or application domain.

The first three types of tailoring are mainly of importance at the start of a V&V effort. The tailoring by balancing is important during the V&V effort.

### 2.3.1 Risk Decomposition and Tailoring by Balancing

As described above it is advised to use a decomposition of the top goal into smaller and smaller goals up to the point that a test can be devised that is within resources and is likely to deliver suitable evidence. During the balancing tailoring during the execution of the V&V work priorities need to be determined. These priorities together with the resources available are used to decide which goals will be further expanded and which will be left undeveloped. The basis for that decision and thus for the prioritization is risk. What is needed is to determine the contribution of a goal to the overall M&S use-risk. If a goal has a high contribution of risk it must be taken into consideration in the V&V work. If it has a very low contribution it can. In that case it should be explicitly be recorded that that goal is not used in the rest of the V&V work such that at the end a feeling for how complete the V&V work is can be obtained.

An evidence solution for a goal with a (relatively) high contribution to the overall risk should likely result in a high confidence in the evidence. For a goal with a low contribution to the M&S use-risk risk it may be sufficient to have evidence that contains some uncertainty, i.e. if the evidence is just an indication that the goal is met it may already suffice.

To find the contribution to the overall M&S use risk for a node it is necessary to make a risk decomposition in the same way as the decomposition of the Acceptance Goal, see Fig. 8. In practice it is difficult to make an exact risk decomposition, therefore it is advised to use a somewhat simpler approach as indicated in Fig. 11. The red stands for high contribution to the overall M&S use risk, orange for medium contribution and green for low contribution. During the decomposition nodes with a low contribution to the overall use risk may be left undeveloped. At the bottom of Fig. 11 the contribution to the risk is an indication of how convincing the evidence should be which is important for specifying which type of tests are required.

If after evidence collection it turns out not all goals are met, the contribution to the overall risk may be used during the acceptance decision to decide what to do. If it concerns a node with low contribution to the overall M&S use risk, it may be decided to leave things as is and accept the small risk. If it is goal with a medium or

high contribution to risk it can be decided to either change the M&S system such that the identified problems are corrected, or the purpose for which the user intends to use the M&S System should be made smaller such that the current state of the M&S System will be fit for purpose.

### 2.4 Why Is This Structured Approach so Much More Effective and Efficient

The above-described structured approach to doing V&V has a number of advantages that make it more effective and more efficient than doing V&V in a less structured way. Below some of the key advantages are discussed.

### The right starting point for the V&V effort leads to more effective results

The V&V effort should start from the perspective of risk. Who runs the real risk in an M&S endeavour? It is not the modeller, not the implementer (maybe there is a risk of repetitive strain injury) and not the person who executes the simulation (maybe if there is a moving base simulator). In general the real M&S use risk is found when the M&S based results are applied in the real word. Therefore V&V processes that are developer oriented might miss the real risk. Also, when studying the 4-world view in Fig. 5 it may become clear that possibly many more aspects may need to be considered than just the domain knowledge as coded in a simulation. Thus organizational aspects that may make or break the use of simulation, the level of proficiency of all people involved, the processes used to derive the products such as the Operational Needs, etc. may all play a significant role and may need to be included in determining the overall utility and thus in the V&V approach. If such a very broad scope is used it becomes clear that a domain oriented V&V process may also miss some aspects. Therefore a general methodology that starts at the true M&S use risk and that can incorporate domain specific elements as well as other aspects will result in a more effective V&V result because the right starting point can be chosen and all relevant aspects included.

### Balancing resources with needs leads to efficiency and effectiveness

A structured decomposition of the Acceptance Goal into all aspects that are relevant and on top of that a decomposition of the contribution of the M&S use risk attached to the Acceptance Goal leads to the possibility to spend the available resources for the V&V effort wisely. Based on priorities related to the contribution to the overall M&S use risk it can be decided which parts of the decomposition requires more or less effort. When available resources do not allow testing all aspects to their maximum, i.e. in all practical situations, it can be decided to let the goals with low contribution to risk remain undeveloped. In that case it should be explicitly recorded that that goal is not used in the rest of the V&V work, see "Knowledge of the completeness of the V&V effort leads to effectiveness" below. If nodes are developed to the point where tests can be defined, the contribution to the M&S use risk can be used to make choices for tests. Low contribution to the risk allow for cheaper tests that may not give a high convincing force. A high contribution to the risk means that sufficient convincing force must be required of the evidence, possibly meaning more expensive tests need to be performed.

The structured approach to V&V makes it possible to balance the resources during the construction of the goal network and the evidence solutions. This means that the V&V effort uses the available resources in an efficient way, allowing for the best possible answer for the given resources, which means the highest possible effectiveness.

### Re-usable domain knowledge leads to more efficient and effective results

The top part of the decomposition of the Acceptance Goal, see Fig. 8, contains domain knowledge because it is the user's perspective that is encoded and the role of the M&S system in that domain. From an V&V enterprise point of view, see Sect. 2.1.6, this domain knowledge can be re-used if other V&V projects are executed on (almost) the same domain or for (almost) the same purpose. In that case the domain knowledge can be re-used and even extended to be more complete. Of course, for each new project in which existing domain knowledge is re-used it must be made sure that no irrelevant aspects are taken into account. Over the course of several projects the domain knowledge becomes more and more complete, which helps in not forgetting possibly important aspects. The re-use of domain knowledge thus leads to more to a more effective and more efficient V&V effort. It is, however, needed that a good discipline in documenting the V&V effort is used.

### Distribute the V&V work among experts leads to efficiency

In the lower part of the goal-network many different aspects covering many different disciplines can be found. The expansion—if needed—of these goals and the execution of associated tests likely requires different experts and facilities. Using the natural break up of a structured approach to V&V, e.g. the tree structure in Fig. 8, it becomes easier to assign experts to different groups of goals and tests. For CI simulation it may be that organizations do not wish to have other experts test their simulation assets, in that case each partner can be assigned a set of goals for which they need to provide evidence. It would be better, however, to have a certain level of independence (see Sect. 2.1.7). The structured approach leads to more efficient execution of the V&V effort by clearly indicating which expertise should be handled by which expertise.

### Complete one branch while waiting for others to complete leads to efficiency

In the structured approach as presented above, it becomes clear that if one branch of the tree structure is fully developed and ready for execution of the tests, there may be no need to wait for other parts to also become fully developed. The parts that are ready to go to the test phase can start independently of the rest. This may even lead to the discovery of problems with the M&S System that can already be corrected before tests of other branches are executed. This leads to a more overall efficient V&V effort.

### Knowledge of the completeness of the V&V effort leads to effectiveness

During the balancing of the resources in building the goal network and the specification of the evidence solutions the important decisions on when goals with a low contribution to the M&S use risk are left undeveloped and which tests are chosen in the specification of the evidence solutions should be unambiguously be recorded. That makes it possible to get a feeling for how complete the V&V effort as a whole is. This completeness should be translated into an uncertainty in the Acceptance Recommendation to the customer. Thus if insufficient resources were allocated to the V&V work, the conclusion might state that the available evidence indicates that the M&S system is fit for purpose, but that the V&V effort as a whole has left too many aspects out of consideration and that thus a high level of uncertainty is present in that statement.

The statement on completeness of the V&V effort will allow the decision maker to make a much better decision, which leads to better effectiveness of the use of the V&V results.

### Standardized documentation leads to efficiency

An often observed problem with unstructured V&V efforts is that it results in either very little documentation or it results in a lot of documents that are unorganized and scattered over different places, usually in the form of computer files that are difficult to find and for which it is hard to recall what its content means and in what piece of evidence it was used.

A structured approach should adopt some standard approach to documentation. This documentation should be such that the Acceptance Recommendation should be completely traceable through the claim network, via the evidence collection, through the goal network back to the Acceptance Goal. Also all decisions due to tailoring should be well documented and immediately clear where they influence the Acceptance Recommendation.

A standardized approach to documentation is also important on the V&V enterprise level where it can be expected that re use of previous V&V projects will lead to efficiency.

### Efficiency for recurrent testing

In practice it may occur that a M&S system had been used for some time and that subsystems are being replaced or upgraded. In that case the structured approach described above makes it immediately clear which parts of the goal network are affected by the change and which tests should be re-done for the new M&S system. This leads to a very efficient way of doing recurrent testing.

# 3 Choose the Appropriate Verification and Validation Technique

There are many different V&V techniques, see e.g. [22–25]. The V&V techniques in those references are categorized in Table 1.

Table 1 Examples of V&V techniques

(continued)

Table 1 (continued)

The four broad categories of V&V techniques can be described as:

It is difficult to state in general which V&V techniques (i.e. what type of tests) should be used. So in this text we provide a basis to choose the right V&V technique. There are a number of important aspects that determine which V&V techniques are appropriate for a given situation:

	- It is clear that a relatively high contribution to the M&S Use Risk requires evidence that can be trusted. This requires a rigorous V&V technique, i.e. one for which the expected residual uncertainty is low. When possible formal techniques should be used. In practice however, this is often prohibitively

### 8 Verification and Validation for CIPRNet 191

expensive and (combinations of) techniques have to be used that are with the available means but still deliver sufficiently trustworthy evidence.

	- The available means are a set of limiting factors such as budget, time, expert knowledge, access to testing facilities, etc. The whole V&V effort has to be run within these limits. That means that during the construction of the goal network only those criteria can be considered that contribute highly to the over M&S Use Risk and collectively are likely to remain within the available means. The collection of evidence solutions has to be chosen such that the expected results of executing the tests delivers the lowest overall residual uncertainty.
	- The Referent data is the knowledge of the real world. It is needed during the tests to compare the simulation results with. If no or little referent data is available only tests that do not (heavily) depend on referent data can be chosen, e.g. expert opinion or examination of the conceptual model.
	- For dynamic testing it is evident that (parts of) the M&S system itself has to be available. Some types of tests require access to M&S system internals in order to make "measurements" that are not visible to the end user. For other tests it is necessary to have access to development documents such as the conceptual model.

Summarizing: the tests all have different costs and different expected residual uncertainty. The contribution to the M&S User Risk should be the basis for choosing the best V&V techniques. A set of evidence solutions need to be chosen such that collectively the best possible result for the given available resources is obtained.

Take away message: You have to choose the appropriate Verification and Validation techniques to balance risk, effectiveness and efficiency.

### 4 Conclusion

As a very brief summary of the text above it can be stated that:

Acknowledgement and Disclaimer This chapter was derived from the FP7 project CIPRNet, which has received funding from the European Union's Seventh Framework Programme for research, technological development and demonstration under grant agreement no. 312450.

The contents of this chapter do not necessarily reflect the official opinion of the European Union. Responsibility for the information and views expressed herein lies entirely with the author(s).

### References

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Chapter 9 Design of DSS for Supporting Preparedness to and Management of Anomalous Situations in Complex Scenarios

### Antonio Di Pietro, Luisa Lavalle, Luigi La Porta, Maurizio Pollino, Alberto Tofani and Vittorio Rosato

Abstract Decision Support Systems (DSS) are complex technological tools, which enable an accurate and complete scenario awareness, by integrating data from both "external" (physical) situation and current behaviour and state of functioning of the technological systems. The aim is to produce a scenario analysis and to guess identify educated the most efficient strategies to cope with possible crises. In the domain of Critical Infrastructures (CI) Protection, DSS can be used to support strategy elaboration from CI operators, to improve emergency managers capabilities, to improve quality and efficiency of preparedness actions. For these reasons, the EU project CIPRNet, among others, has realised a new DSS designed to help operators to deal with the complex task of managing multi-sectorial CI crises, due to natural events, where many different CI might be involved, either directly or via cascading effects produced by (inter-)dependency mechanisms. This DSS, called CIPCast, is able to produce a real-time operational risk forecast of CI in a given area; other than usable in a real-time mode, CIPCast could also be used as scenario builder, by using event simulators enabling the simulation of synthetic events whose impacts on CI could be emulated. A major improvement of CIPCast is its capability of measuring societal consequences related to the unavailability of primary services such as those delivered by CI.

### 1 Introduction

The set of Critical Infrastructures (CI) constitutes nowadays an enabling pillar of societal life. They guarantee the supply of vital services (transport of energy products, telecommunication, drinkable water delivery, provide mobility functions)

© The Author(s) 2016 R. Setola et al. (eds.), Managing the Complexity of Critical Infrastructures, Studies in Systems, Decision and Control 90, DOI 10.1007/978-3-319-51043-9\_9

A. Di Pietro (&) L. Lavalle L. La Porta M. Pollino A. Tofani V. Rosato Laboratory for the Analysis and Protection of Critical Infrastructures, ENEA Casaccia Research Centre, Rome, Italy

e-mail<EMAIL_ADDRESS>
thus concurring to the achievement of citizens' (and societal as a whole) well-being. CI are complex technological or engineering systems: they are thus vulnerable as exposed to natural and anthropic-related events. Physical damages inflicted to CI elements might produce severe repercussions on their functioning which can reduce (or even reset) their functionality. Other than being individually wounded and functionally reset, they can propagate perturbations to other CI to whom they are functionally (inter-)connected. Connection and inter-connection are two relevant properties of systems of CI: connection indicates a one-direction supply mechanism, when one CI supplies a service to another. When such a service is no longer provided, the supplied CI may undergo a more or less severe perturbation. Inter-dependency indicates the presence of feedback loops: a CI might perturb other CI which, directly or through a further perturbation to other CI, could back-propagate the perturbation to the CI which initiates the perturbation cascade. This might produce a further functionality degradation which is amplify the negative feedback loop, by producing more and more serious effects. Cascading effects may spread perturbations on large geographical scales, on time scales ranging from a few second to days, producing reversible and, in some cases, irreversible societal effects.

Other than having repercussion on citizen activities, CI damages and the consequent services perturbation could affect the environment and produce large economic losses. Industrial activities are directly related to the supply of these services; their loss directly implies a lack of production and revenues contraction. In some cases, moreover, CI outages might produce environmental damages (gas release, spill of oil or other products, fires releasing toxic products, nanoparticles, ashes) that further increase the societal consequences.

As CI deliver relevant (in some cases, "vital") services to the citizens, their societal impact has increased significantly in the last century. For these reasons, significant efforts are going to be produced at the national and EU scales, either at the governance level1 and by deploying the most advanced technologies.

Major benefits for protecting CI and enhancing the continuity of services they deliver could come from the deployment of technological systems providing access to crisis related data, allowing their monitoring and, whenever possible, the prediction of their occurrence, allowing the setup of timely preparedness and mitigation actions. A relevant role in this context could be played by Decision Support Systems (DSS). These are technological tools that can be functional to support the whole risk analysis process up to crisis management, in the preparatory and the hot phases.

A new concept of DSS should account for, and support, all phases of the risk analysis process: event forecast (where applicable/predictable), prediction of reliable and accurate damage scenarios, estimate of the impact that expected damages could have on services (in terms of reduction or loss of the services) also accounting for perturbation spreading via cascading effects, estimate of the possible consequences to citizens and to other sectors of societal life. The complete DSS workflow

1 COUNCIL DIRECTIVE 2008/114/EC of 8 December 2008 on the identification and designation of European critical infrastructures and the assessment of the need to improve their protection.

should end up with the identification and definition of preparedness and emergency strategies that, taking into account the different phases of the expected crisis (event, damage, impact and consequence) could be adopted to reduce the impact, speed-up mitigation and healing procedures, ease the recovery phase, thus reducing as much as possible the extent, the severity and the duration of the crisis.

Such new concept of DSS can also be used as effective simulation tools to perform comprehensive stress tests on areas where the impact deriving from CI crises could be large and relevant. This activity would produce educated contingency plans based on the analysis of many (synthetic) crisis scenarios instead of being built upon (a few, when available) records of historical events. This will enhance their quality and adapt them to the effective current scenario conditions (in terms of infrastructures, assets, available technical tools, current settings of the crisis or emergency management etc.).

The EU project CIPRNet2 has thus devoted a considerable effort to realize an novel DSS enabling to tackle the entire workflow of risk forecast of CI, from event prediction to consequences analysis.

The DSS can be supported by a large database of information collected from public and private sources. Furthermore, the DSS collects many different real-time data from the field (meteorological stations, sensor networks, meteorological radars, etc.) and forecasts from several publicly available sources (Meteorological Office, Earthquake alerts systems, etc.) producing a comprehensive assessment of the physical state of the area (urban, district, regional up to a national scale).

The availability of the geographical position (in terms of geospatial data) of the CI elements and the networks would allow, through the correlation between the physical vulnerability to natural events and the strength of expected event manifestation, to formulate an educated guess of the probability that some of the CI elements could be physically damaged by a perturbation. This analysis would thus allow to produce a "damage scenario" containing location and probability of predicted faults.

Starting from the "damage scenario", the DSS will attempt, through the analysis of appropriate simulation tools, to emulate the outages on the affected networks and, through the CI dependency information, to reproduce the effects of the cascading events propagating faults from one network to the others. This task would result in the "impact scenario", i.e. the expected profile of services unavailability over time for all the considered infrastructures.

Such data would further allow to estimating the consequences that services unavailability might produce on the different societal sectors. This is the goal of the "consequence analysis" which is meant to transform the "impact scenario" into a prediction of the social severity of the crisis, by measuring, through appropriate metrics, the consequences associated to the population, the industrial activities, the primary services (Hospitals functionality, schools, public offices, public

2 CIPRNet, Critical Infrastructures Preparedness and Resilience research Network has been funded by EU FP7 under GA No. 312450.

transportation) and the environment (in the case when a CI crisis is associated to some type of an environmental damage).

The last step of the DSS would be the elaboration of an optimal strategies for the systems recovery by analysing possible "recovery sequences" of the different elements: the sequence "score" is evaluated in order to reduce as much as possible the social costs of the crisis.

The project CIPRNet has introduced all these elements into the DSS which has been designed and realized as one of the major outcomes of its joint technological activities. The DSS was called CIPCast. We will refer to this name in the course of this work when considering the CIPRNet DSS.

### 2 Design Study

In order to cast all the expected DSS properties and functions, we tried to translate the expected functionalities into a number of prescriptions, of practical issues and of technological requests to the DSS for enabling the implementation of those functions. These are the major key-words which have been translated into related functionalities of the DSS.

perturbation takes place on a longer time scale and, often, with a longer latency. The DSS should thus cope with multiple time scales and the geographical long-ranged perturbation spreading.

The design of CIPCast has taken into account all the issues that have been previously listed. Figure 1 shows the main functional blocks Bi of CIPCast and the relevant components i.e. the Database and the Graphic User Interface (GUI). In the

Fig. 1 Block diagram showing the main functionalities and the relevant components of CIPCast

following we will briefly describe the five functional blocks which will be better analysed in the further Sections.

Monitoring of Natural Phenomena (B1). In this block, the DSS acquires external data (real-time data, forecast) from many different sources: weather forecast and nowcasting data, seismic data, real time data coming from weather stations, hydrometer levels from water basins. These data are acquired to establish the current and predicted external conditions.

Prediction of Natural Events (B2). This block estimates the expected manifestation strength of all the predicted events. Predictions are made on different time scales: short time scale (up to 60 min from the current time), medium-long range time scale (within 48 h from the current time).

Prediction of Damage Scenarios (B3). In this block, the DSS correlates the strength of the expected manifestations with the vulnerability of the different CI elements present in the area where the events are predicted to occur, in order to estimate the probability that the manifestation could effectively damage (and, in the positive case, to what extent) the CI elements. At the end of B2 block, the DSS elaborates a "Damage Scenario" containing the information on which CI element (and to what extent) will be damaged in a specific time frame.

Prediction of Impact and Consequences (B4). This block converts the expected damages of CI elements into impact on the services the CI elements produce. This is the core of the prediction process as, in this block, the DSS transforms the expected punctual damages (to one or more CI) into a reduction (or loss) of services. To do that, CIPCast needs to deploy dependency data connecting the different CI in order to reproduce faults propagation. In addition, starting from the inoperability (or partial operability) of the different services, this block also estimates the consequences that the loss of services produces on citizens, public services, industrial activities and the environment. The consequences on each societal sector are estimated on the basis of specific metrics; a distinct "consequence score" on each societal life domain is presented separately (a unified score is not produced) in order to describe the severity of the expected crisis under many viewpoints.

Support of efficient strategies (B5). This block contains a number of applications which, taking into account the expected critical scenario, made by damages, impacts on services and weighted by the consequences estimate, will attempt to support operators and emergency managers to design and validate mitigation and healing procedures. At the current state of implementation, these supporting actions relates to:

In the following, we will describe, in some more detail, each of the relevant elements of the DSS and the technical contents of the different blocks Bni.

### 3 Database

The geospatial Database (DB) and the related modules (GIS Server and WebGIS application) has been implemented by adopting a client-server architecture, using Free/Open Source Software (FOSS) packages. Such architecture has been properly designed to allow the interchange of geospatial data and to provide to the CIPCast users a user-friendly application, characterised by accessibility and versatility. The DB is a PostgreSQL object-relational database with PostGIS spatial database extender. PostGIS adds support for geographic objects allowing location queries to be run in SQL. The DB can be used at various levels by exploiting the potential offered by GIS tools, starting with the effective support for the operational management in the frame of the risk assessment workflow.

Data contained in the DB are classified according the following scheme:

	- Static data
	- Dynamic data
	- Forecast
	- Damage scenario

Short term (<2 h) Medium term (>2 h and <24) Long term (>24 h)

– Impact scenario

Short term (<2 h) Medium term (>2 h and <24) Long term (>24 h)

– Consequence analysis

Short term (<2 h) Medium term (>2 h and <24) Long term (>24 h)

Concerning the Input data, the DB contains the following geographical information layers. Concerning with Static Data, the DB contains:

Concerning with Dynamic input data (i.e. data which are collected by field sensors, which are constantly updated), the DB contains:

Concerning with Forecast data, the DB contains:

The GIS Server represents the hardware/software environment that allows organizing information and making them accessible from the network. The GeoServer suite has been adopted, being a largely used open source application server, which plays a key role within the Spatial Data Infrastructure (SDI). It allows sharing and managing (by means of different access privileges) the information layers stored in the DB, according to the standards defined by the Open Geospatial Consortium (OGC), such as, for example, the Web Map Service (WMS). It also supports interoperability (e.g., reads and manages several formats of geospatial data) (Fig. 2).

The basic geospatial data and the results produced (i.e., scenarios) are stored and managed into the DB repository in order to be exploited in the different DSS blocks. To this end, the WebGIS application developed represents the natural geographical interface of CIPCast. Basic information, maps and scenarios can be visualized and queried via web, by means of standard Internet browsers and, consequently, the main results can be easily accessible to CIPCast users.

Fig. 2 Deployment diagram showing the connection among the GeoDatabase and the other DSS core components

### 4 Dynamic Data

In order to predict the external scenario, CIPCast has been configured to acquire external information by collecting real time data from field sensors. In particular, it acquires field data from:

Concerning seismic and earthquakes data, given as initial assumption that no real prediction can be achieved for these types of events, CIPCast receives data from the Agency committed to release this information (e.g., the National Institute of Geophysics and Volcanology INGV3 in Italy). In the INGV official site, by accessing the Italian Seismological Instrumental and Parametric Database (ISIDE) portal,4 information on the detected earthquakes are produced and released in real time. Upon a constant poll to the ISIDE portal, CIPCast receives the earthquakes data (within 1 min from the occurrence). Earthquake information consists of the GPS coordinates of the epicentre, its depth and the measured intensity (Richter scale). Figure 3 reports a typical snapshot of the ISIDe website.

Once earthquake data are issued, the CIPCast crawler picks them up and reports them into the synoptic chart of the DSS geographical interface (Fig. 4). The knowledge of the coordinates, the depth and the magnitude of the earthquake (basic

3 INGV: Italian National Institute of Statistics: http://www.istat.it/en/.

4 http://iside.rm.ingv.it/iside/standard/index.jsp?lang=en.

Fig. 3 Snapshot of the Italian ISIDe website providing real-time data on earthquake events occurred in the Italian territory. The portal is managed by the National Institute for Geophysics and Volcanology, INGV

Fig. 4 ISIDe data are immediately reported into the DSS DB and visualized on the GIS web interface

earthquake's features) are not sufficient to estimate the "physical manifestations" associated to the natural event. Indeed, an earthquake creates distinct types of waves with different velocities; when reaching seismic sensors, their different travel times allow to locate the source of the hypocentre:

• Primary waves (P-waves) are compressional waves that are longitudinal in nature and propagate faster than other waves through the earth to arrive at seismograph stations first (hence the name "Primary");

• Secondary waves (S-waves) are shear waves that are transverse in nature: following an earthquake event, S-waves reach seismograph stations after the faster-moving P-waves and displace the ground perpendicular to the direction of propagation.

In the case of local, or nearby, earthquakes, the difference in the arrival times of the P and S waves can be used to determine the distance of the event. Once ISIDe operators perform their validation procedures, data of the occurred earthquakes are immediately available: based on the basic earthquake's features, CIPCast is able to convert them into a Shake Map dataset which contains, for each spatial point of a given area (as large as that involved by the physical manifestations associated to the event), the Peak Ground Acceleration (PGA) distribution induced by the seismic event.

Figure 5 shows an example of a shake map.