Clocks in computer systems are usually realized by a digital counter register and a crystal oscillator whereas each oscillation generates a tick event that increments the counter register. The oscillator frequency output slightly deviates from its designed nominal frequency output, because of (1) mechanical imperfections introduced during manufacturing of the oscillator, (2) dynamic deviations caused by aging of the oscillator, and (3) environmental conditions (e.g., temperature, acceleration, humidity) acting on the oscillator. The static and dynamic deviations from the nominal frequency are the cause for clock drift: Any two clocks of the same design, even when perfectly started at the same instant, will eventually drift apart as time progresses.

For establishing a global time in Cyber-Physical Systems-of-Systems (CPSoSs), we need to periodically resynchronize the local clocks of the Constituent Systems (CSs) by external clock synchronization (e.g., synchronization with the GPS time source). The resynchronization is necessary to ensure a bounded offset, also called precision, among the clocks of the CSs. A critical parameter of clock synchronization is the resynchronization period which needs to be short enough to keep all clocks within the required precision. By actively compensating the clock drift, the duration between two resynchronization instants can be increased. This is of particular interest if the source for external synchronization is unavailable (e.g., losing the GPS signal when driving into a tunnel, turning the GPS receiver off to save power), while the synchronization precision of the clocks of the CSs has to be maintained until the external source becomes available again.

Drift Compensation Method. In order to compensate clock drift, the effects of internal and external sources of oscillator frequency deviations must be negated. This requires to measure these effects or know a-priori about them, but also to apply corrective actions on the clock. There are two options to apply corrections: (1) control the enclosing environment of the oscillator (e.g., oven-controlled or voltage-controlled oscillators) such that the output frequency is corrected, or (2) to periodically correct the counter register by adding each correction period a correction value that compensates for the frequency deviations.

The first approach requires, additionally to the measurement of the enclosing environment, possibly expensive and power-demanding actuation hardware (e.g., heat source), insulation, or is in case of some effects (e.g., acceleration) infeasible. However, for some effects (for example, temperature, humidity, pressure, aging) this approach is technically more simple, as it only requires to steer the oscillator frequency close to the external clock source and then maintain the same environmental conditions during holdover mode.

The clock drift compensation of the CDC module is based on the second correction approach which also depends on measuring the oscillator environment, but – besides that – can be realized purely in software. This software implements a compensation model which predicts for each correction period an accurate correction value in clock ticks. To achieve a high level of accuracy, the clock drift compensation model fuses several sources of information (a-priori knowledge, sensor observations).

Important parameters of our drift compensation method – besides the drift compensation model – are the tick rate of the external time signal which should be forwarded as the output tick by the CDC module during normal operation and generated when absent during holdover mode, the correction period, and the internal tick rate or oscillator frequency. The output tick rate determines the counter register size, i.e., this register needs to be able to count the number of internal ticks that correspond to one output tick. The output tick rate also determines the correction period, because corrections predicted by the compensation model should be applied for each output tick to avoid imprecision effects. Finally, in order to have only a small discretization error, the clock compensation method assumes that the output tick rate (e.g., 1 Hz) is much lower than the internal tick rate (e.g., at least a few kHz or MHz).

Compensation Model. The clock drift compensation model is based on (1) a-priori knowledge about the oscillator (e.g., manufacturing defects, aging behavior, known effects of environmental conditions), and (2) on the currently observed external time signal and the environmental conditions. Environmental conditions that can be observed by sensors are for example: temperature, barometric pressure, acceleration, air humidity.

Explicitly defining a compensation model would be a tremendous task, because it requires precise knowledge about all relevant physical properties of the crystal oscillator, their interrelationships, and the involved sensors which – similarly to the oscillator – also slightly deviate from their designed characteristics (measurement errors that depend on currently prevailing environmental conditions). Consequently, we focus on defining and parameterizing the compensation model by using classical machine learning techniques where, for example, regression (curve fitting) for independent input variables, or artificial neural networks for input variables with unknown dependency relationships are available. The selection of a concrete machine learning technique depends on the available sensors, available computational resources and the required compensation quality. Regardless of the technique, training of the compensation model is necessary by taking a set of input instances (e.g., an observation of the environmental conditions) and adjusting the model parameters such that it maps the input data to a known correction value for the counter register. The trained model is then available during holdover mode for predicting correction values for new input data where the correction value wasn't known before.

For model training there are two methods that should be applied in combination:

• Offline-Learning/Calibration: After manufacturing the CDC module, including its oscillator and the sensors, obtains training data by observing (using its own sensors) the controlled environmental conditions. Under these controlled environmental conditions, the oscillator frequency deviation is measured by external equipment (e.g., an oscilloscope that measures the difference of the external time signal with respect to the internally generated output tick during holdover). Training data is collected by doing various sweeps of the controlled environmental conditions through the range of expected environmental conditions and recording the sensor observations together with the necessary correction value. Offline learning initializes the compensation model with a-priori knowledge.

• Online-Learning/Adaptation: During operation of the CDC module the compensation model can be adjusted by constantly retraining it, when the external time signal is available. Online-learning compensates for aging effects of the oscillator and involved sensors. Also online-learning possibly allows for limiting the necessary offline-learning to a smaller sample size of a production series where only small model deviations are expected among individual CDC modules.

#### 3.2 Proof-of-Concept Prototype

The proof-of-concept prototype is based on an implementation of the CDC module on the SmartAP 2.01 , a small embedded system originally intended for auto-piloting small aircrafts. It consists of a STM32 ARM Cortex M4 microcontroller integrated with two external quartz oscillators (84 MHz, and 32.768 kHz), and sensors to measure acceleration (Invensense MPU-6050/ MPU-9150), barometric pressure (MS5611-01BA03). We customized the board by adding sensors to additionally measure temperature and air humidity (Sensirion SHT75). As an external time signal we used an UBLOX LEA-6H GPS receiver. For wirelessly communicating with the board we added a Bluetooth module (Microchip RN42).

The microcontroller implements a data recording functionality to obtain the measured training data, and the drift compensation method including a simple variant of a compensation model. In this prototype we did not implement online-learning, because the effects of online-learning are minor, if the compensation model is well calibrated and the CDC module prototype has not been left several months for aging between calibration and evaluation.

Compensation Model Implementation. Figure 2 illustrates a simple compensation model based on look-up tables where averaged training data can be deployed directly. For each environmental condition a look-up table exists from which the contribution of the measured parameter to the clock drift is obtained. Temperature is codependent with all other environmental conditions. Consequently, the lookup tables for pressure, humidity, and acceleration contain temperature dependent correction values. To estimate the clock drift for the current correction period, the individual contributions are summed up and added to the constant drift value of the crystal oscillator.

Training of the Compensation Model. The compensation model training data is obtained by placing the CDC module with its sensors in an experimental chamber within which the environmental conditions can be controlled. For each correction period an oscilloscope records the deviation of the internally generated output time signal from the external time signal (GPS receiver).

Figure 3 shows our prototype of such an experimental chamber. It consists of a thermally insulated box with a Peltier element for heating and cooling, an air pump to produce an over or under pressure inside, and a humidifier. To achieve a constant acceleration force on the oscillator crystal, the board is mounted on a plate that can be

1 http://sky-drones.com/autopilots/9-smartap-autopilot-20.html.

Fig. 2. Clock drift compensation model based on look-up tables.

Fig. 3. Experimental chamber for obtaining training data for the compensation model. On the right the prototype of the CDC module has been highlighted with a red circle and is mounted on the rotating plate.

rotated. Acceleration forces can be applied on the oscillator in X and Y directions, depending on how the board is mounted on the rotating plate.

By using this box, one environmental condition after the other is varied from its minimum value up to the maximum, and vice versa. For each set point the number of oscillations of the crystal oscillator, which drives the clock, is counted several times. The reading and resetting of this counter is triggered every second by the PPS time pulse originating from the GPS receiver.

From the recorded training data, the constant drift at certain environmental conditions – i.e., the zero conditions C0 – is obtained. C0 can be selected arbitrarily. Furthermore, for each condition that can be observed a table entry (see lookup-based compensation model in the previous section) is derived by averaging over the training data that have been recorded for the same environmental conditions. These table entries indicate the additional drift if the conditions deviate from the zero conditions C0.

#### 3.3 Evaluation

The evaluation of the CDC module is conducted using the abovementioned experimental chamber. In particular, only results for temperature and acceleration in X

Fig. 4. Frequency deviation with respect to temperature.

Fig. 5. Frequency deviation with respect to acceleration in X-axis at different temperatures.

direction for the 84 MHz quartz oscillator are shown here, while information also on other environmental conditions (Y-acceleration, air humidity, barometric pressure) and the second oscillator can be found in [15].

Obtaining Training Data. The investigated oscillator has a measured variance (i.e., variations in the number of oscillations when environmental conditions are stable) around its nominal frequency of approximately 0.02381 ppm. In the following we present frequency deviations of the oscillator when doing a temperature sweep from 0° C to 50° C and an acceleration sweep from 0 g to 5 g (g is the weight per mass unit).

Figure 4 depicts the frequency deviation over the temperature range. If, for instance, the frequency deviation is about 2.5 ppm (e.g., at 15° C), it means that in each second a clock driven by this oscillator deviates 2.5 µs from a clock with perfect accuracy. Without clock synchronization, this deviation sums up to 9 ms/h, or 0.216 s/day. In Fig. 4, a positive value of the deviation denotes an increase in the number of oscillations compared to the nominal value.

Figure 5 shows the frequency deviation of the oscillator concerning variations in the acceleration force and temperature.

Fig. 6. Three test sequences of temperature set points.

Fig. 7. Three test sequences of X-acceleration set points.

The results show that additionally to the significant correlation between the temperature and the frequency deviation, the oscillator exhibited deviations concerning changes in acceleration (and pressure and humidity, see [15]). All these deviations have been higher than the variance of the oscillator at stable environmental conditions.

Evaluation of Compensation Model. The evaluation of the compensation model carried out by recording the compensation performance over a set of test sequences where the controllable environmental conditions are varied. Each of the test sequences takes 240 min. Figures 6 and 7 show three of such sequences for the environmental conditions temperature and acceleration in X direction.

Figure 8 shows the compensation performance of the CDC module in holdover mode only for the first test sequence (1. Sequence), because all other test sequences gave similar results. In Fig. 8 we plotted the results for two different compensation models: basic compensation (only constant drift correction), and temperature, acceleration and pressure (T&A&P). Also for comparison reasons, the deviation is depicted, when no compensation mechanism is applied. Different effects during manufacturing or aging of the oscillator crystal lead to a permanent deviation from the nominal frequency (here it is faster than the reference), which is about 60 ppm.

Fig. 8. Frequency deviation under different compensation models.

Clearly, basic compensation has the largest corrective effect and improved the mean drift rate of the oscillator from an order of magnitude of about 10−5 to approximately 10−6 . The compensation model that regarded more of the investigated environmental conditions (T&A&P) gave a better result, as the oscillator's drift rate is now in the 10−7 order of magnitude.

Even more improved results should be easily achievable when using more sophisticated compensation models, sensors of better quality, and implementing online-learning for self-fine-tuning. Consequently, this evaluation gives strong support to the benefits of our proposed clock drift compensation method.

#### 3.4 Summary of Main Findings on the Clock Drift Compensation Module

The proof-of-concept prototype confirms that the compensation of frequency deviations of crystal oscillators by passive observation of the surrounding environmental conditions, and using a trained compensation model leads to a significant decrease of clock drift. The different environmental conditions indeed have an effect on the stability of the oscillator and some of these effects can be reduced when these conditions are known. Available protocols (e.g., the Network Time Protocol – NTP) are able to compensate constant drifts of the local clock of a computer system, if the environmental conditions are not changed after the reference clock is disconnected. However, many CPSoSs operate in environments, within which it is infeasible – or at least only with a considerable effort (e.g., by constantly heating the crystal oscillator) – to keep these conditions stable. In contrast, sensors to determine the environmental conditions are often already available in those systems, or can be installed at relatively low cost.

While the presented improvement by temperature, acceleration and pressure compensation are already promising, further experiments have to be performed with more sophisticated compensation models and a more advanced experimental equipment that allows higher ranges of pressure, the up- and down variation of humidity, as well as experiments under other environmental conditions (e.g., vibration, radiation, electromagnetic fields).

### 4 Reliable and Self-Aware Clock (R&SAClock) Module

The Reliable and Self Aware Clock (R&SAClock, [12]) is a software component that provides IEEE 1588 compliant techniques for the analysis and improvement of the synchronization quality among CSs interacting in SoS. The R&SAClock exploits statistical information in order to provide information about uncertainty of the current time view.

Generally, in several contexts such as industrial automation, telecommunication or energy distribution, SoSs require an accurate synchronization of their CSs in order to assure the adequate Quality of Service (QoS). The statistical information collected by R&SAClock to estimate synchronization uncertainty [8] is used as feedback about quality of synchronization. The CSs equipped with R&SAClock are continuously updated about the current synchronization performance.

#### 4.1 General Concepts on R&SAClock

A CS uses R&SAClock to acquire both the time value and synchronization uncertainty associated with the time value.

For clarity, we report basic notions on time and clocks that are used in the rest of this section. Noteworthy, the terminology is consistent with the terms defined in Chap. 1. Figure 9 below is introduced to better clarify relevant aspects.

The global time is an abstraction of physical time in a distributed computer system; it is the unique time view shared by the CSs. The reference clock is a working hypothesis for measuring the instant of occurrence of an event of interest: it is a clock that always holds the global time. We can say that the reference node is the CS that owns the reference clock. Also, given a local clock c and any time instant t, we define c (t) as the time value read by local clock c at time t.

The behavior of a local clock c is characterized by the quantities offset, accuracy and drift. The offset of two events denotes the duration between two events and the position of the second event with respect to the first event on the timeline; the offset Θc(t) = t − c(t) is the actual distance of local clock c of the CS node n from the global time at time. This distance may vary through time.

Accuracy Ac of clock c denotes the maximum offset of a given clock from the external time reference, measured by the reference clock. An upper bound of the offset adopted in the definition of system requirements and therefore targeted by clock synchronization mechanisms.

Fig. 9. Basic notions on time and clocks.

The precision π of an ensemble of synchronized clocks denotes the maximum offset of (distance) respective ticks of the global time of any two clocks of the considered clock ensemble.

The drift ρc(t) of a physical clock describes the frequency ratio between the physical clock and the reference clock i.e., the rate of deviation of a local clock c at time t from global time [10].

Synchronization uncertainty Uc(t) is defined as an adaptive and conservative evaluation of the offset Θc(t) at any time t; uncertainty is such that Ac ≥ Uc(t) ≥ | Θc(t)| ≥ 0 [8]. Hence, accuracy Ac is an upper bound of uncertainty Uc(t) and consequently of the absolute value of the offset Θc(t).

When a CS asks the current time to R&SAClock, the latter provides an enriched time value useful for time synchronization. The enriched time value is composed of a set of values: likelyTime, minTime, maxTime and FLAG. LikelyTime is the time value computed reading the local clock. minTime and maxTime represent left and right synchronization uncertainty margins with respect to likelyTime. They are based on synchronization uncertainty provided by the internal mechanisms of R&SAClock. Finally, the FLAG takes the value 1 if requirements on uncertainty are satisfied, 0 otherwise. Details on R&SAClock and its implementation can be found in [8, 11].

It is evident that the main core of R&SAClock is the uncertainty evaluation algorithm that equips R&SAClock with the ability to compute the uncertainty. Such an algorithm relies on the Statistical Predictor and Safety Margin (SPS) algorithm.

Each CS that uses the R&SAClock getTime method for getting synchronization information and each CS has the two main expectations: (i) a request for the time value

Table 1. Requirements for R&SAClock

should be satisfied quickly, and (ii) the enriched time value should include the correct real time. These are formally expressed by the two requirements in Table 1.

#### 4.2 The Statistical Predictor and Safety Margin (SPS)

In the following the SPS algorithm is briefly described for a local software clock c that is disciplined by an external clock synchronization mechanism. SPS computes the uncertainty at a time t with a coverage, intended as the probability that Ac ≥ Uc(t) ≥ |Θc(t)| ≥ 0 holds. The computed uncertainty is composed by three quantities: (i) the estimated offset, (ii) the output of a predictor function, P and (iii) the output of a safety margin function, SM. The computation of synchronization uncertainty requires a right uncertainty Ur(t) and a left uncertainty Ul(t): consequently, SPS has a right predictor with a right safety margin for right uncertainty, and a left predictor with a left safety margin for left uncertainty. The output of the SPS at t ≥ t0 is constituted by the two values:

$$U\_r(t) = \max(0, \tilde{\Theta}(t\_0)) + P\_r(t) + \mathcal{SM}\_r(t\_0) \tag{1}$$

$$U\_l(t) = \min(0, \tilde{\Theta}(t\_0)) + P\_l(t) + \mathcal{SM}\_l(t\_0) \tag{2}$$

The estimated offset H~ ðt0Þ is computed by the synchronization mechanism and can contain errors. If the estimated offset is positive, it influences the computation of an upper bound on the offset itself and consequently is considered in (1). If it is negative, it is ignored. A symmetric reasoning holds for (2).

The predictor functions, Prð Þt and Plð Þt , predict the behavior of the oscillator and continuously provide bounds (lower and upper) which constitute a safe (pessimistic) estimation of the oscillator drift and consequently a bound on the offset. The oscillator drift is modelled with the random walk frequency noise model, one of the five canonical models used to model oscillators (the power-law models [14]), that we considered as appropriate and used. Obviously the parameters of this random walk are unknown and depend on the specific oscillator used. They are computed resorting to the observation of the last m samples of the drift (where m smaller or equal to the set-up

Table 2. SPS parameters

parameter M), and using a safe bound on the population variance of the estimated drift values. The coverage of this safe bound depends on the set-up probabilities pds and pdv defined in Table 2 together other main quantities involved in the SPS algorithm.

The safety margin functions SMrðt0Þ and SMlðt0Þ aim at compensating possible errors in the prediction or in the offset estimation. The safety margin function is computed starting from the collection of the last n samples of the estimated offset (where n is smaller or equal to the set-up parameter N). A safe bound to the population variance of the estimated offset is computed. The coverage of this safe bound depends on the set-up probabilities pos and pov (see Table 2).

The parameter t0 is the time in which the most recent synchronization is performed. At time t0 the synchronization mechanism computes the estimated offset H~ ðt0Þ and possibly the estimated drift q~ð Þ t0 (if not provided by the mechanism, it can be easily computed by R&SAClock itself).

#### 4.3 Proof-of-Concept and Exemplary Runs

The R&SAClock has been implemented in the Beagle Bone [16] board with Debian OS as described in the AMADEOS deliverable D4.4 [15]. The R&SAClock uses GPS for clock synchronization; it acquires data on the estimated offset and drift from the Network Time Protocol (NTP) [9] component.

When the synchronization uncertainty exceeds a given threshold, the Checker module is notified by reading the FLAG field of the enriched time value. The communication channel between the Checker and the R&SAClock is socket-based.

Fig. 10. Exemplary run of R&SAClock on the proof-of-concept.

In the following we show an exemplary run with the R&SAClock executing on the Beagle Bone board proof-of-concept. We acknowledge that an extensive assessment activity is required [13] (i) to give evidence that the defined software executing on the proof-of-concepts satisfies the identified requirements, and (ii) to opportunely tune the R&SAClock parameters (Table 2). Still, we present this exemplary run because it explains intuitively the behavior of R&SAClock.

In the considered run, reported in Fig. 10, the x-axis (in seconds) corresponds to the likelyTime collected reading the local clock. The maxTime and minTime computed by R&SAClock are respectively the two lines above and below the x-axis. In fact, the yaxis (in milliseconds) shows the maxTime and minTime with respect to likelyTime i.e., maxTime-likelyTime and likelyTime - minTime. The FLAG value is not shown in this figure.

In the present run, the R&SAClock was already running for 30000 s (see x-axis). As long as the connection with the GPS signal is stable, NTP reliably disciplines the local clock: an accurate estimation of offset is provided, and synchronization uncertainty is a small interval (in the order of few microseconds or less). As it can be shown in the time interval between second 30000 and 30600, the synchronization uncertainty is slightly reduced through time. In fact, the R&SAClock "studies" the past behavior of the local clock, understand that it is overall stable and trustable, and reduces synchronization uncertainty.

Instead, at approximately second 30700, an instability in the local clock and NTP is detected, most likely due to the temporary unavailability of the GPS signal (signal loss). Synchronization uncertainty is increased, because no fresh information on the clock behavior w.r.t. the reference time is provided.

At approximately second 31900, the synchronization uncertainty steadily increases through time. In fact, GPS signal is lost and no fresh information from the time source is provided. There are no guarantees that the clock is disciplined correctly.

When the GPS signal is newly available (approximately at second 33200), a new accurate estimation of the offset is provided. Consequently, the synchronization uncertainty is reduced again. However, from now on, the synchronization with the GPS is unstable: there are only few, sparse synchronizations. This determines the behavior of the R&SAClock, which cannot trust the local clock and consequently the synchronization uncertainty grows.

#### 4.4 Summary of Main Findings on R&SAClock

The R&SAClock was initially proposed in [8, 11, 12] to monitor the software clock in distributed system. In such works, R&SAClock was implemented and exercised on a fixed node, and with the intention of supporting only the node itself.

Instead, in the RMC, the R&SAClock is intended to operate as a failure detector for a Master clock: in other words, the Checker module of the RMC can read the FLAG value of the R&SAClock, and decide if the RMC can act as a master clock or not.

In addition, we implemented the R&SAClock on a light board which has small requirements for power consumption. This improves the range of applicability of the R&SAClock w.r.t. the previous environments described in [12], which are distributed servers. The experiments, although still preliminary, confirm that the R&SAClock behaves as expected also confirming, in a different environment, the results shown in [12].

#### 5 Conclusions

This Chapter discussed the role of time in Systems-of-Systems. Building on the terms, definition and knowledge defined in Chap. 1, this Chapter identified motivation, with examples, for the prominent role of time and clocks in time-aware Systems-of-Systems. Further, the Chapter discussed the challenges of resilient time keeping and it presents the Resilient Master Clock (RMC), a hardware-software solution that acts as an accurate, fail-silent global time base which is externally synchronized to a satellite-based time source.

The design of the RMC is presented, its main algorithm illustrated including results from the execution on two different prototypes. Although significant work is still needed to consolidate results, the RMC appears a promising approach to provide a low-cost, low-power consumption solution for resilient time-keeping and resilient master clock in Systems-of-Systems.

#### References

Open Access This chapter is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the work's Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work's Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.

# Managing Dynamicity in SoS

Sara Bouchenak1(&) , Francesco Brancati2 , Andrea Ceccarelli2 , Sorin Iacob3 , Nicolas Marchand1 , Bogdan Robu1 , and Patrick De Oude3

1 Université Grenoble Alpes, Grenoble, France<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>2 Department of Mathematics and Informatics, University of Florence, Florence, Italy<EMAIL_ADDRESS><EMAIL_ADDRESS>3 Thales Netherlands B.V., Hengelo, The Netherlands<EMAIL_ADDRESS>
### 1 Introduction

SoS dynamicity refers to short-term changes in an SoS, which occur in response to changing environmental or operational parameters of the CSs. These changes may have different effects, such as SoS adaptation or the generation of emergent phenomena. This chapter starts by recalling the MAPE approach in Sect. 2 before to introduce existing monitoring approaches in Sect. 3. Finally, Sect. 4 overviews existing reconfiguration techniques for SoS dynamicity management, related to Analyzis, Planning and Execution phases and illustrates through an examples the possible implementations of dynamicity management with modelling and feedback control techniques.

#### 2 Overall MAPE Approach

We follow the classical MAPE-K control loop for designing an autonomic manager over managed elements (Fig. 1). This consists mainly in components to Monitor, Analyze, Plan and Execute the reconfiguration plan. When an SLA (Service Level Agreement) and its associated service level objectives are associated with the service of a managed element, the MAPE control loop guarantees that those service level objectives are met, and if it is not the case, a new plan is calculated and used to reconfigure the system.

© The Author(s) 2016

This work has been partially supported by the FP7-610535-AMADEOS project.

A. Bondavalli et al. (Eds.): Cyber-Physical Systems of Systems, LNCS 10099, pp. 186–206, 2016. DOI: 10.1007/978-3-319-47590-5\_7

Fig. 1. MAPE control loop

#### 2.1 SoS Management Infrastructure

In this section we present how to exploit the AMADEOS SysML profile described in Chap. 4 in order to build the infrastructure for MAPE purposes. To this end, we implemented through the profile six MAPE architectural patterns. Thus, for each of the six patterns, we realize a SysML block diagram built using the stereotypes defined in the profile.

Each CS is represented as a block and it has an interface, either a RUMI or a RUPI interface in order to enable the exchange of messages and physical entities respectively. Each CS may implement the SoS management activities, namely monitoring, analysis, planning and execution.

In the following we report the (1) Hierarchical Control, (2) Master/Slave, (3) Regional Planner, (4) Coordinated control, (5) Information sharing, (6) Atomic patterns and a conclusive analysis on their recursive inclusion in a SoS.

#### 2.2 Hierarchical Control Pattern

In the hierarchical control pattern, we have a managed CS which is controlled directly by a managing CS by means of their RUMI interfaces over which monitoring information and the enacting actions are transmitted. As we can notice (Fig. 2), the managing CS has been stereotyped with all the functions of the MAPE cycle. We present below the UML representation of such pattern.

#### 2.3 Master/Slave Pattern

In the master/slave pattern¸ a set of managing CSs shares part of the MAPE functions (Fig. 3). In our instantiation, master CS (Managing\_AP\_CS) performs analysis and planning activities and two slave CSs perform monitoring and execute functions

Fig. 2. Hierarchical control pattern

(Managing\_ME\_CSx). The slave CSs send monitored\_info to the master and they receive planned info from the master CS. This information is exchanged through the RUMI interfaces of slave CSs and the master CS.  The latter transmit monitored data to the slave CS, which in turn forwards back the enacting actions as planned by the master CS.

Fig. 3. Master-slave pattern

#### 2.4 Regional Planner Pattern

In the Regional Planner, a regional managing CS (Managing\_P\_CS\_x) implements only the Planning activity (Fig. 4). Instead, Analysis, Monitoring and Execution are

Fig. 4. Regional planner

delegated to other managing CSs (Managing\_MAE\_CS\_x). The regional managing CS is responsible for a region of CSs and it exchanges with other peer CSs the information on a regional basis. At the bottom level, each managed CSs (Managed\_CS\_x) send monitoring information to its corresponding managing CS (Managing\_MAE\_CS\_x), which performs analysis activities and then forwards the results to the regional CS (Managing\_P\_CS). The latter performs the Planning and forward enacting actions back to the managing CS. Finally, the managing CS enacts such actions towards the managed CSs.

#### 2.5 Coordinated Control Planner

In the following, we present the implementation for the non-formal hierarchy pattern Coordinated Control pattern (Fig. 5). The coordinated control pattern consists of a set of CS implementing all the MAPE phases (Managing\_CS\_x) and exchanging through their RUMIs the monitored, analysis, planned and execution information. Through RUMIs, the managing CSs can collect monitoring info from the managed CS (Managed\_CS\_x) and forward the actions to enact.

Fig. 5. Coordinated control

#### 2.6 Information Sharing Pattern

The information sharing pattern is another non-formal hierarchy pattern similar to the coordinated control pattern (Fig. 6). The only thing that differentiates the corresponding implementations is the nature of information which is exchanged through the RUMIs of the managing CSs. In the sharing information pattern only monitored data are exchanged while the rest of information is not shared among managing CSs (Managing\_CS\_x).

Fig. 6. Information sharing

#### 2.7 Atomic Pattern

In the following, we present the atomic pattern to enable the interaction of a CS with the physical environment (Fig. 7). To this end, the managing CS (Atomic\_CS) carries out the MAPE cycle once it has received monitoring information through its RUPI interface and it forwards physical signal over the same interface. Physical entities received through the RUPI interface come from the afferent environment while the outgoing flow of physical entities is forwarded to the efferent environment.

Fig. 7. Atomic pattern

#### 2.8 Recursive Inclusion of Patterns for SoS Design

At SoS level, the MAPE should be applied recursively. Each CS implementing part of the MAPE may in turn be substituted recursively by another pattern which is realized through a further set of CSs. This approach foresees the possibility to have several nested levels of MAPE, each of them belonging to a different hierarchical level of the SoS, or holarchical set of CSs. Distinct set of CSs are observed by a MAPE in which the Adaptive Monitoring, the Analyzer and Planner, and the Reaction Strategies coexist and cooperate.

Let us consider the scenario depicted in Fig. 8, in which several CS are represented and controlled by an implementation of the MAPE-K cycle in which we have a dedicated CS for Monitoring, another for Analysis and Planning and a third one for Execution.

The Adaptive Monitoring (M) is able to detect events according to the QoS specifications of the CS. The Cognitive and Predictive Models (AP) search for the causes and the effects correlating data incoming from the Adaptive Monitoring. The Reaction Strategies (E) perform the proper recovery action. The considered SoS can be integrated in a more complex SoS; Fig. 8 gives a representation of the recursive foreseen architecture.

Fig. 8. Recursive view of the AMADEOS architecture

# 3 Monitoring/Analysis

In this section, we report on the broad topic of monitoring in SoSs. In particular, focusing on observation and data analysis (i.e., the Monitoring and Analysis components of the MAPE building block). We investigate basics on monitoring and detection (Sect. 3.1) and main monitoring approaches (Sect. 3.2).

#### 3.1 Basics on Monitoring and Detection in SoSs

It is very important to guarantee that an SoS behaves as expected. To this end, monitoring activities control the SoS by means of verifying that system behavior and performance comply with well-defined rules. Verification activities can be carried out at two different stages either on-line, i.e., while monitoring data are collected, or off-line i.e., after the collection process.

In monitoring literature, the system which has to be monitored is called target system while the hardware component and the software application within the target systems are called respectively target component and target application. In our case a CS represents the target system while the target component and the target application are represented by the physical and software part of the CS itself. We refer to the CS which receives monitored information as monitoring CS.

Monitoring activities consists in observing behavior and performances of CS target components in order to collect useful information to guarantee the correct SoS functioning. Monitoring activities by themselves are not sufficient to guarantee the correct behavior of an SoS, but they have to be integrated with techniques to diagnose the SoS behavior in its execution environment. To this end, we have presented in deliverable D3.1 [1] an SoS Management Infrastructure which complements Monitoring with Analysis, Planning and Execution (MAPE) facilities. This section focuses on the Monitoring (M), and partially also on the Detection of events (Analysis of data), consequently addressing the MA letters of the MAPE loop. These two actions (MA) are in fact often tightly bounded, because a monitoring system is usually conceived and instantiated with the specific intention of detecting specific events or verifying that certain conditions are met.

Let us now focus on the way monitoring activities are carried out. Essential objects that are exploited to monitor the system behavior are the so called probes. Probes can be inserted either inside or outside the target system and provide useful information on how the system behaves. As an example Fig. 9 shows the probes inserted within the target system which can also provide the system intermediate output (see Fig. 9-b) and it shows the possibility of monitoring the system as a black box (see Fig. 9-a).

Fig. 9. Black box (a) and instrumented (b) monitoring of the target system

Probes can be hardware or software. In the first case hardware signals are monitored, while in the second case, code is inserted within the target application to collect internal information of the system (code instrumentation). Two rules have been defined which have to be respected by probes:

#### 3.2 A General Approach to SoS Monitoring

Considering the complexity and heterogeneity of an SoS, it is necessary that a monitoring solution deals with the issue of where to deploy the monitoring and detection system. It could be placed locally on each CS or globally at a higher level. Both approaches have advantages and disadvantages with respect to the CS and/or the SoS, as listed in the following:

To capture the pros of both the local and global solution, we envision the following overall architecture.

Each CS describes the provided services to the other CSs through the RUI specification. The interface models are part of the RUI specification and must be based on an agreed ontology explaining the meaning of the interface variables exchanged across the RUI and must be compatible with each other. In order to establish the desired quality of service (QoS), quality metrics must be expressed as well in the RUI specification. For example, a Service Level Agreement (SLA) should be negotiated between the service provider and requester. Thanks to the RUI specification, the monitoring and the detection systems can ignore the intrinsic characteristics of the CSs, but they are aware of its quality metrics and its SLA.

Each CS that is included in the SoS can be equipped with a Local Detection System (LDS). The LDS (i) includes probes exposed through the RUI and that are necessary to observe events; (ii) if necessary, implements the atomic pattern to manage the physical environment of the CS itself. The knowledge of the LDS is limited to the CS: in other words, the other connected CSs are ignored. The different detectors, relying on the exposed probes, can be organized and coordinate following the different MAPE patterns.

At SoS level, a global detection system will be also deployed. It differs from the ones deployed at a local level because it has an overall view of the SoS and consequently it has the ability to observe and detect events as a combination of the outputs of the individual LDSs. Furthermore, it may consider different and additional quality metrics and indicators with respect to the LDSs. The global detection system fetches data from the LDS or MAPE instantiations available in the SoS, and perform global monitoring and analysis, acting according to the master/slave pattern.

Figure 10 shows a high level representation of the architecture of the envisioned system. Specifically, the CSs composing the SoS are represented in different shapes, to show that they are different one from the other. Each CS is able to communicate with the others by means of the RUI interface, represented by the box labelled RUI. Finally, the local detection systems, labelled LDS are also included in the representation of each CS, in order to eventually detect anomalous events on the corresponding CS. At the top

Fig. 10. Monitoring infrastructure in SoSs

of the figure, the Global Detection System is also shown, which observes the status and the events that are happening at the SoS level.

Depending on the monitoring purposes, confidentiality and privacy issues may need to be guaranteed through proper data security and anonymization. Especially, while anonymization solutions can be executed locally, confidentiality requires that the endpoint agrees on adequate secure communication protocols.

#### 4 Analysis/Planning/Execution

#### 4.1 Overview

The main challenges related to the dynamic adaptation of CPSoS stem from the distributed nature of the measurement and control infrastructure (MAPE).

Since the Monitoring and Analysis blocks have been discussed earlier, this section focuses primarily on a potential design of the Planning and Execution blocks.

The M and A functions of the MAPE building block determine the values of the CPSoS parameters, whereas the P and E functions close the control loop either by generating control signals for the CS, or by adjusting the environmental parameters of the CS, as depicted in Fig. 11.

All MAPE functions are instantiated for a particular CPSoS dynamicity model, which specifies which state parameters need to be measured, what metrics have to be used for combining or aggregating these parameters, how the control must be implemented to achieve the desired effect, and how this control algorithm generates CS control parameters. Obviously, this domain knowledge applies in a similar way for all the composite MAPE patterns described in the AMADEOS deliverable D3.1 [1].

Fig. 11. Close loop control involving a CS and a MAPE block

To emphasize the control aspect, we redraw the MAPE control loop as in Fig. 12. Although not fundamentally different from a traditional control loop, Fig. 13 emphasizes the potential difference between the input and output parameters (YðtÞ and XðtÞ, respectively) toward the CPSoS and the monitoring and control parameters (lðtÞ, and eðtÞ, respectively) toward the MAPE blocks. This is useful for CPS, since the monitoring and control of the RUPI-based interactions can also be executed through RUMI, which adds flexibility to the definition of reaction strategies. The (⊕) blocks denote some suitable transformations that map the external and control input vectors onto a common metric space, and the ⊕ block combines these values in a single metric to obtain the control value. Based on this value the Control block generates a vector rðtÞ

Fig. 12. Control and feedback parameters in MAPE-based control loops

whose elements show the mismatch between the current and desirable state parameters, so a new state can be selected that compensates for this mismatch.

#### 4.2 Example: Control Loop for Electrical Vehicle System

If the CS is a Charging Point (see AMADEOS use case [2, 3]) Y(t) in Fig. 13 can be a state vector specifying the charging current, maximum allowed power phasor variations, and last kWh price. When an Electric Vehicle (EV) is connected to the CP, the State includes, among other functions, a metering function which could generate the monitoring vector lðtÞ as a sequence of messages at regular time intervals, containing instantaneous measurements of the current drawn by the EV, and the maximum available current for that CP. Similarly, XðtÞ could be the remaining charging time as estimated by the EV, and eð Þt could be a sequence of asynchronous (i.e. event-triggered) messages containing a new value for the kWh price, and a new maximum power rating for the CP.

Fig. 13. Simplified functional diagram for an electrical vehicle charging station

Obviously, the summation block in Fig. 12 assumes an appropriate abstraction of the two inputs (MAPE controls and external CPSoS inputs), which is achieved through some mappings. In the example considered above both the remaining charging time and the new kWh price could be mapped on some real value expressing the economic efficiency of the CP.

To illustrate several reaction strategies, we construct a simple example derived from the AMADEOS EV Charging use case. Assume that a Charging Station includes N charging Points (see Fig. 13), has a total charging capacity Q, and a maximum charging rate I:max. Each of the CP has a maximum charging rate ICPmaxICP, with NI CPmax ¼ aImax, and a [1, which means that when all CPs are in use, not all of them can deliver the maximum charging rate. To compensate for this limitation, the CP can influence the charging demand by increasing the energy unit prices. In general, the energy unit price has to be agreed between the EV user and the CSO before the charging operation starts, and should not be changed until the charging is complete. For this reason, it is convenient to define a charging contract in terms of total requested electric charge by user i, Qi, and the maximum charging duration ti. For simplicity, we assume that the minimum charging times only depends on the maximum current that a CP outlet can provide.

If the CSO chooses to guarantee a constant charging rate Ii, then the charging time for EV i will be constant as well (within some uncertainty limits): ti ¼ Qi Ii . Although it is both in the interest of the CSO and the user to minimize ti, this is not achievable simultaneously for all the CPs.

A good pricing strategy should try to approach the maximum charging current of the CSOðImaxÞ, without lowering the charge unit price below a given minimum. As an example one could consider the following pricing function by the CSO, for user i:

$$p\_{CSOi}(Q\_i, t\_i) = p\_0 \frac{t\_{\max} - t\_{\min}}{t\_i - t\_{\min}} \left( 1 + \frac{1}{I\_{\max} - I} \right), t\_{\min} < t\_i \le t\_{\max}, I < I\_{\max} \tag{1}$$

The minimum charging time tijmin is achieved for the maximum available charging rate given the already committed capacity (I) for the CSO : timin ¼ Qi Imax-I . The third term of the product in the above formula increases the price as the committed capacity approached the maximum.

Overall, the price variation looks like the graph in Fig. 14.

Fig. 14. Possible price adaptation as a function of requested charging time and available

At the same time, a user will always choose a shorter charging time, provided that the price does not increase beyond a predefined personal limit pimax:

$$p\_i = \min(p\_{i\text{max}}, p\_{c\text{SO}i})$$

Of course, this is a naïve view, since Ij varies in time as other running charging actions end (see Fig. 15), so a significant part of the charging capacity is not used. The

Fig. 15. Variation of the total charging current at CSO

total revenue for the CSO at any given time PðtÞ ¼ P N i¼1 piðtÞ. At the same time, the instantaneous cost for the CSO is given by a component proportional to the total current absorbed plus some constant cost value c0: C tðÞ¼ c0 þ cIðtÞ. The profit made by the CSO can thus be expressed as:

$$
\rho\_{CSO}(t) = P(t) - C(t) \tag{2}
$$

By allowing a variable maximum charging current, the actual charging time results shorter than the one calculated with the pricing model for constant charging current used in this example. The statistics of charging times and new requests will require pricing strategies that take into account the expected variations in the occupancy of CPs.

#### 4.3 Analytic Approaches

When the CPSoS behaves according to a known model expressed analytically, the control can be defined in terms of this model. The pricing model in Eq. (1) allows the CSO to set a dynamic price for a charging operation at constant current. However, this model leads to unused capacity, so it may be advantageous for the CSO to deliver a faster charge if there is unused capacity and other requests are expected to arrive soon.

An optimal control problem with infinite time horizon attempts to maximize some cost or benefit function, such as the one in Eq. (2) [4]. A potential objective function for the optimal control problem could attempt the simultaneous optimisation of the following aspects:

The control parameters are the individual charge unit prices and the charging currents for each user. The constraints are the total current for the CSO, the maximum current of each CP, and the committed charging time for the already started charging actions.

The first term of the objective function defines how the charging current for user I can be increased after the charging action for user j ends. This can be done for instance, proportionally:

$$
\Delta I\_i^+ = I\_j \frac{I\_i}{I} = I\_j \frac{I\_i}{\sum\_{k \neq j} I\_k} \tag{3}
$$

Whenever a new request comes at a time ti start after the adjustment of the charging rates, the new charging current must be maximised by reducing the current for the still running charge operations, down to the limit that would still allow the completion of the charging within the remaining time according to the original contract for user i:

DI - i ¼ Qi - R tj start ti start Iið Þt dt ti end tj start ð4Þ

where ti start and ti end are the starting and ending times of the charging operation for user i.

Equations (1) to (4) can be used for defining different control approaches, such as Optimal (Stochastic) Control with infinite time horizon [4], Linear Quadratic Controls [5], etc.

#### 4.4 Machine Learning Approaches

In case when the dynamic behaviour of the CPSoS is unknown, or cannot be expressed analytically, data-driven techniques can be used for obtaining an implicit "encoding" of system behaviour. Such an encoding should be able to predict the outputs the system will generate for a given input and contextual parameters.

Machine learning includes a set of techniques that use observations of a system's behaviour patterns to attempt predicting its future states. If we consider the predictive or supervised machine learning approach the goal is to learn a mapping from input values X to an output value y based on a set of input-output pairs called a training set. The mapping can be represented as a function fH, with a set of parameters H, that can be used, given an unseen before pattern Xi, to predict ^yi, i.e. ^yi ¼ fHð Þ Xi . The function fH is what we call a model that is parametrized with a set of parameters H. fH can be, in fact, a simple linear function (regression), but generally is a complex algorithm that maps inputs to output values. The goal of machine learning is to find the set of parameters H of a chosen model based on a training data set D using a learning algorithm. If the parameters H are properly learned then we will be able to estimate the class label ^yi based on certain unseen input values xj 62 D, where ^yj ¼ yj in the majority of the cases.

Artificial Neural Networks (ANN) implement a machine learning technique inspired by a simplified model of the working of the brain. The elementary operations in an ANN are weighted summations of the inputs to obtain an output value. Different output values are obtained by summing the same input with different weights. This way, an input vector is mapped to an output vector. A multi-layer NN combines usually two or more such mapping units. The training of an ANN attempts to adjust the weights such that a particular output is consistently obtained for different input patterns belonging to the same class. When this is achieved, the ANN is said to be able to generalize. Another aspect of training attempts to adjust the weights such that different output patterns are generated for input vectors belonging to different classes. When this is achieved, the ANN is said to be able to discriminate. Training algorithms have been developed that achieve a good trade-off between these two properties. One such algorithm is called the back-propagation learning algorithm that uses an iterative scheme, such as gradient descent [6], to optimize Θ in the learning equation described earlier.

Returning to the EV charging example we want to estimate the best price for a new user i by learning a model based on the following input variables:

Let's denote these input variables with xn. Based on xn we want to learn a model from which the best price ^pi can be estimated for a new user i. In order to learn such a model the following cost function will be used

$$J(\theta) = \frac{1}{2} \sum\_{n=1}^{N} (p\_{ni} - \hat{p}\_{ni})^2 \tag{5}$$

Where ^pni is the estimated price by the model, pni is the price associated to the input variables, N is the total number of training samples and n corresponds to the nth training sample ð Þ2 xn; pni D. In case an ANN is used as a model to learn the best price the gradient descent back-propagation algorithm can be used as an iterative scheme to optimize the model based on the cost function JðhÞ. By using this iterative scheme, the parameters (i.e., the weights and bias term of each neuron in the ANN) are updated, after applying the training set D, such that the cost function is minimized. When the change of the parameter values are small enough that it can be assumed that the parameters have converged and the model is learned.

The learned solution may not be optimal, since the back-propagation algorithm can be trapped in a local minimum. This is due to the high nonlinear nature of the cost function in the parameter space. Often better performance can be obtained by using a pattern-by-pattern mode, also online mode, to learning the model. In this case the weights of the ANN are updated at every time instance a new pattern is presented. Additionally, it is also recommended to randomize the data sequence prior to using it for learning. In practice it was shown that the pattern-by-pattern mode result in faster convergence and better solutions [7].

#### 4.5 Feedback Control Approach

In the following, we consider a SoS in which one of the CSs is a computing cluster used to run compute-intensive and/or data-intensive business logic of the cyber-physical system. In the following, we first illustrate the impact of environmental changes and system configuration parameters on the performance and availability of such CSs. We then present a possible implementation of the Analysis component of MAPE through behavioural modelling and a possible implementation of the Planning/Execution components of MAPE through feedback control.

MapReduce is a popular programming model and execution environment for developing and executing distributed data-intensive and compute-intensive applications [8]. However, the complexity of configuration of such systems is continuously increasing. Although the framework hides the complexities of parallelism from the users, deploying an efficient MapReduce implementation poses multiple challenges. MapReduce's ad-hoc configuration and provisioning require a high level of expertise to tune [9]. Ensuring performance and dependability of MapReduce systems still poses several challenges.

One of the most popular open source implementations of the MapReduce programming model is Hadoop. It is composed of the Hadoop kernel, the Hadoop Distributed Filesystem (HDFS) and the MapReduce engine. Hadoop's HDFS and MapReduce components originally derived from Google's MapReduce and Google's File System initial papers. HDFS provides the reliable distributed data storage and the MapReduce engine provides the framework to efficiently analyse this data.

In the following, we consider a CS that is a MapReduce cluster that consists of sub-CSs represented by N nodes. A MapReduce workload is defined as the number of concurrent clients (C) that are sending requests to the central controller. Admission control is a classical technique to prevent server thrashing. It consists of limiting the maximum number of clients (MC) that are allowed to concurrently send requests to the central controller.

The performance of MapReduce systems can be measured as the average time (Rt) needed to process a request in a certain time window. Low client response time is a desirable as it reflects a reactive system. The average Rt can, for instance, be calculated at every 30 s, using a sliding window with period 15 min.

$$Rt[s] = a \text{vg}(Rt\_1, Rt\_2, \dots, Rt\_N) \tag{6}$$

Availability (Av) refers to the accessibility of the system to users. MapReduce is available if the user requests are accepted at the time of their submission. Availability is instantaneous and concentrates on the fraction of time where the system is operational in the sense of being accessible to the end user. Availability is measured as the ratio of accepted MapReduce client requests to the total number of requests, during a period of time. T here is the previously defined sliding time window size that is used to assign a measurable dynamics to the system. Since T is constant for all experiments, we use only the percentage (%) symbol as the availability measurement unit in all the plots to simplify their understanding.

$$A\nu \left[\frac{\%}{T}\right] = \frac{N\_{\text{SuccessfulJobx}}}{N\_{\text{SuccessfulJobx}} + N\_{\text{RejectedJobx}}} \ast 100\tag{7}$$

Furthermore, the service cost is a linear function of the MapReduce cluster size (N), and can be inferred directly from N.

Finally, performance and availability metrics are part of the SLA of the MapReduce system. The SLA specifies MapReduce service level objectives (SLOs) in terms of, for instance, the maximum response time Rtmax, and the minimum availability Avmin to be guaranteed by the MapReduce system.

Figures 16, 17, 18 show the impact of the variation of, respectively, the workload exogenous variable, the MapReduce cluster size control variable, and the MapReduce cluster's admission control variable on performance and availability metrics. Thus, there is no one-fits-all configuration; rather, a solution that meets a combination of service level objectives as described below.

Example of a Behavioral Model. Capturing the complex behaviour of MapReduce CSs is highly challenging. We propose a model that captures the dynamics of MapReduce CSs, and renders their levels of performance and availability. The model is

Fig. 16. Impact of workload on MapReduce performance and availability with #Nodes = 20, #MC = 10

Fig. 17. Impact of cluster size on MapReduce performance and availability with #Clients = 10, #MC = 5

Fig. 18. Impact of admission control on MapReduce performance and availability with #Nodes = 20, #Clients = 10

built as a set of difference equations - as for biological or economical systems - that describe the impact of input variables' variations on system's output variables. We apply a novel modelling approach that considers the MapReduce system as unknown and derives a mathematical model based only on the impact of the input variations on the system's outputs. This technique is part of what we call system identification in control theory. Roughly speaking, one provides known input variation functions (e.g. a step or sinusoidal variation) to the system, and measures the system response to this excitation. Using the output measurements an identification algorithm can approximate the system's internal dynamics. In most cases, without a loss in generality, 1st or 2nd order polynomial difference equations capture the system behaviour sufficiently well.

Figure 19 describes the proposed model variables. The inputs of the model are: exogenous input C that represents the number of clients accessing the underlying MapReduce system, in addition to tunable parameters that can be used to control the MapReduce system, namely the number of nodes N of the underlying MapReduce cluster, and the maximum number of clients MC concurrently admitted in the MapReduce system. In addition to input variables, the model has the following output variables: the average response time Rt to a MapReduce client request, and the level of availability Av of MapReduce to its clients. In the following, we describe the proposed model through the formulas of its output variables.

Example of Feedback Control. A first attempt in controlling the response time of a MapReduce system by adding and removing nodes was realized in [10] by using a PI and a feedforward controller. We design MR -Ctrl, an optimal controller, able to deal

Fig. 19. System model inputs and outputs

with contradictory objectives. As our MapReduce model has two outputs, MR - Ctrl will assure at the same time the response time and the availability specified in the SLA, while minimizing resource utilization.

The complete schema of the control architecture is presented in Fig. 20. All the variables used in the figure are defined in Table 1. More details regarding the implementation of the control framework can be found in [10]. As in Fig. 19, we consider the MapReduce system having two inputs (concatenated in the two dimensional vector u), one exogenous uncontrollable disturbance input C and two outputs (concatenated in the two dimensional vector y). Vector u contains the number of nodes in the cluster N and the max number of clients MC. While the y vector contains the response time Rt and availability Av.

Fig. 20. The control architecture

Table 1. Definition of control variables

# 5 Conclusions

This chapter describes the overall approach for managing SoS dynamicity. Its main intent is to associate a Service Level Agreement with SoS, and to provide SLA guarantees in terms of dependability, security, performance, etc. The overall MAPE Monitoring/Analysis/Planning/Execution approach is followed for SoS dynamicity management. The approach is illustrated through different implementations and techniques, e.g., a scalable monitoring, feedback control-based behavioural modelling and reaction strategies.

# References

Open Access This chapter is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the work's Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work's Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.

# Case Study Definition and Implementation

Arun Babu1 , Francesco Brancati1(&) , Sorin Iacob2 , David Mobach2 , Marco Mori3 , and Thomas Quillinan2

1 Resiltech SRL, Pisa, Italy<EMAIL_ADDRESS>2 Thales Netherlands B.V., Hengelo, Netherlands {sorin.iacob,david.mobach<EMAIL_ADDRESS>3 Department of Mathematics and Informatics, University of Florence, Florence, Italy<EMAIL_ADDRESS>
#### 1 Introduction

In this chapter we present three case studies in the smart grid domain: Electrical Vehicle charging, Household Management, and an integrated case study that combines the first two together with ancillary services. These case studies are first modelled using the AMADEOS Architectural Framework (AF) and associated tooling. We utilise the four levels of the AMADEOS AF: mission, conceptual, logical and implementation, as well as the seven viewpoints that have been defined: Structure, Dynamicity, Evolution, Dependability and security, Time, Multi-criticality and Emergence. We therefore examine the entire lifecycle of the framework considering some real-world case studies. These case studies are based on experts' feedback, including AMADEOS Advisory Board members, to ensure that realistic architectures are designed.

The architectures developed in this chapter will be further instantiated in a simulation environment, using the simulation tooling developed in the AMADEOS project. With these instances several experiments will be run in order to validate the framework as well as the architectures that were defined.

The three Smart Grid-based case studies described in this chapter are used to prove the effectiveness and consistency of the AMADEOS architectural framework. The method provided by the framework allows to design and implement a generic SoS in a procedural and systematic way. To accomplish this, the AMADEOS project defines a pyramidal top-down approach that must be undertaken passing through four different levels: a mission for the SoS, the conceptual level, where the ideas and concepts of the SoS are defined in order to support the capabilities of the SoS. Next, the logical level where the SoS is designed and these concepts are adapted towards supporting the requirements of the individual SoS domain. Finally, these are actualised in the implementation level, where the design is contextualized and realized in the enterprise.

This work has been partially supported by the FP7-610535-AMADEOS project.

© The Author(s) 2016

A. Bondavalli et al. (Eds.): Cyber-Physical Systems of Systems, LNCS 10099, pp. 207–238, 2016. DOI: 10.1007/978-3-319-47590-5\_8

# 2 Smart Grid SoS

#### 2.1 Smart Grid SoS Model

Figure 1 shows the model of the Smart Grid SoS created using the supporting facility tool. The Smart Grid SoS is composed of: EV\_Charging, Medium\_Voltage\_Control, and the Household CS.

Fig. 1. The Blockly Smart Grid SoS model

#### 2.2 EV-Charging SoS Case Study

Mission. The EV SoS must be designed to provide a friendly and convenient service to the users and at the same time, profitable to the provider. Planning and scheduling is of paramount importance for both energy providers and users: as an example, on one side, if the charging requests are spread during the day, there will be limited and/or controlled load peaks on the grid to be handled, thus the energy price may not vary abruptly over time and prioritized consumers (e.g., police and fire-fighter vehicles, ambulances, etc.) will be easily handled by the charging station operators. On the other hand, knowing the energy prices and available time slots, the users will be able to carefully plan the recharging operation while keeping the service affordable.

A typical scenario would be as follows: EVs travel through a wide area, where several charging station operators provide recharging services, by means of charging points. Drivers in need of power for their EV can provide the expected charging context (duration, power, etc.) to the e-mobility service in order to receive information regarding recharging time slots and associated energy prices of each charging station operator. A load management optimizer that cooperates with the charging station operators carries out planning and scheduling activities. The interested driver will then choose one of the slot-price pair possibility for recharging its vehicle, will be allowed to plug-in its EV at the charging point of the chosen charging station operator during the reserved time slot only, and the amount due will be based on the energy consumption times the booked price. At the end of the recharging operation, the driver receives a billing invoice.

It is evident from the above-mentioned scenario that various dependability aspects need to be considered. The participant systems need to be time synchronized to provide consistent information for scheduling and planning purposes. Furthermore, critical EVs should be prioritized for recharging with respect to any other vehicle. Therefore, each EV should be assigned a priority level. Moreover, the e-mobility service should be accessible by registered users only, i.e. the owner of an EV, to reduce the possibility of denial of service attacks being performed by illicit malevolent users scheduling recharging reservations, without a real need.

Conceptual Level. In this section we report the result of the activities performed at the conceptual level for the EV charging case study. For each viewpoint we list the most representative identified SoS requirements defined taking as input the SoS metarequirements [1]. Traceability of the full set of requirements on meta-requirements is also provided in [4].

Architecture Viewpoint. The EVC consists of a subset of the CSs of the SoS "Electrical Vehicle Charging in Smart Grids" described in Chapter 2.1 of [4]. In particular (Table 1):

High level representation of EV-Component interactions.

A pictorial view of the EV charging SoS is reported in Fig. 2.

The steps required to recharge an EV are described below. Each step number corresponds to a sequence of actions that are carried out.

Table 1. EV charging case study components description

Fig. 2. Electrical vehicle charging SoS architecture

Dependability and Security Viewpoint. It is of paramount importance to guarantee the highest achievable availability and reliability of the SoS so that the charging requests are readily and continuously available. Achieving these properties allow not only to reach the highest quality of service, but also to maximise the profits. The SoS allows drivers to enquire for a charging opportunity through the E-mobility service and to have access to the CPs at any time. Thus, energy grid enterprises can maintain the grid stability, balance the load, and ensure correct voltage levels and frequency, etc., while energy providers can make the highest profits out of it. To avoid undermining such premises, Denial of Service (DoS) attacks caused by unauthorized users enquiring the E-mobility service or trying to access CPs need to be properly tackled.

Dynamicity Viewpoint. EVs join and leave the SoS, according to the need for charging of the EV. The change in the topology due to this turnover of EVs characterizes the dynamicity (DYN) of the SoS.

Emergence Viewpoint. According to [2], Emergence is: A phenomenon of a whole at the macro-level is emergent if and only if it is new with respect to the non-relational phenomena of any of its proper parts at the micro level. This indicates that these phenomena cannot be observed at CS level, but at SoS level (or other higher level). As consequence of identification of emergence scenarios, hazard analysis must be performed to identify and mitigate any hazards.

Evolution Viewpoint. Due to technological advances, marketing, or customer needs; different kind of EVs it may be necessary to improve, change or add services.

Multicriticality Viewpoint. The scenario described so far, only foresees normal EVs (e.g., private EVs) that need to recharge and ask for charging opportunity. However, during real-life emergencies, e.g. rescue operations, wildfires and public security, the requirement is that specific EVs must always be available. Therefore, the CSO scheduling strategies must prioritize such vehicles before any others. Emergency EV drivers access prioritized charging in the same fashion of any other driver, i.e. through the E-mobility service.

Time Viewpoint. As a general remark, it is worth noting that most of the information exchanges between CSs rely upon a notion of time. As an example, it would be impossible to plan and schedule a request of recharging an EV by a driver for the current day or even pay the billing invoice if not properly time-stamped. Thus, to provide the payment services, to allow users to enquire for charging opportunities and to effectively, and efficiently, plan, and schedule, recharging operations over time, there must be awareness of time over the SoS. Further, each CS must be time synchronized to a common reference time to successfully provide their services.

Logical Level. This section describes the SoS Logical Description of the SoS defined in [4] using the model made using the supporting facility tool.

Fig. 3. EV Charging in Blockly

After loading the model in supporting facility and double clicking on EV\_Charging CS we can see in Fig. 3 that the EV-Charging Blockly model consist of CSs that matches to the diagram depicted in Fig. 2.

Figures 3 and 4 show that EV-Charging CS consists of the following CSs: (i) Chargingpoint, (ii) CSO, (iv) DriverApp, (v) ElectricVehicle, (vi) EMobilityService, and (vii) EV-SmartMeter

In the following section will be described in details the CSs listed above, through the expansion of the blocks.

Charging Point CS. The Chargingpoint CS can be expanded by double clicking on it and the result is depicted in Fig. 4.

Figure 4 shows that the Chargingpoint CS communicates with the Electric Vehicle, CSO, EV-SmartMeter CSs by the CharginPoint-ElectricVehicle RUPI, Chargingpoint-CSO RUPI and Chargingpoint-EV-SmartMeter RUPI, respectively. Figure 4 also shows that the services provided by the Chargingpoint CS. Furthermore, it has a State Variable Chargingpoint: charging\_done.

From the viewpoint of communication, double clicking on RUPIs and RUMI blocks of Charging Point, CS Fig. 4 shows that the Chargingpoint CS provides the services to CSO CS through the Chargingpoint-CSO RUMI. The Fig. 4 shows also that:

Fig. 4. The Blockly Charging point CS model

CSO CS. On expanding the CSO CS block (Fig. 5), its services, RUIs, MAPE, and state variables can be seen.

Figure 5 shows that the Chargingpoint CS communicates with the EmobilityService, Chargingpoint, LMO, Aggregator CSs by the CSO-EMobilityService, CSO-Chargingpoint, CSO-LMO, CSO-Aggregator RUMIs respectively. The Fig. 5 also shows the services provided by the CSO CS. It also has a State variable CSO: reservation.

From the viewpoint of communication, double clicking on RUMIs blocks of CSO CS, Fig. 5 shows that the CSO CS provides the following services:

Figure 5 also shows that the CSO-Chargingpoint RUMI is connected to the Chargingpoint-CSO RUMI in order to call the service provided by ChargingPoint CS; the CSO-LMO RUMI is monitored through the Probe CSO Probe. The CSO has a local clock that we call CSO-clock and that will be described when addressing the Time viewpoint.

Fig. 5. The CSO CS Blockly model

DriverApp CS. On expanding the DriverAPP CS block in Fig. 6, it shows that the DriverApp CS communicates with EMobilityService CS by the DriverApp-EMobilityService RUMI, and shows that provides the service DriverApp: accept\_reservation.

This figure also shows that DriverApp CS:

	- DriverApp:result\_of\_charging\_opportunities\_request;
	- DriverApp: selected\_charging\_opportunity;
	- DriverApp:duration;
	- DriverApp:power,
	- DriverApp:got\_reservation.

From the viewpoint of communication, double clicking on DriverApp-EMobilityService RUMI, we see that the DriverApp CS provides the service Driver-App:accept\_reservation by DriverApp-EMobilityService RUMI to EMobility CS, and

Fig. 6. The DriverAPP CS Blockly model

this RUMI is connected to the EMobilityService-DriverApp RUMI in order to call the services provides by the EMobilityService.

ElectricVehicle CS. On expanding the ElectricVehicle block as depicted in Fig. 7, it is shown that the ElectricVehicle CS has ElectricVehicle-Chargingpoint RUPI in order to connect with the Charging Point CS. Indeed, expanding the ElectricVehicle-Chargingpoint RUPI you can see that the RUPI transport Plug-In-Signal and it is connected with Chargingpoint-ElectricVehicle RUPI.

EMobilityService CS. On expanding the EMobilityService CS block as depicted in Fig. 8, it is shown that the EMobilityService CS communicates with the DriverApp, Aggregator, Market, CSO CSs by the EMobilityService-DriverApp, EMobilityService-Aggregator, EMobilityService-Market, EMobilityService-CSO RUMIs respectively.

Figure 8 shows also that EMobilityService CS provides the services:

and has the State variables:

Fig. 7. The EV CS Blockly model

Fig. 8. The EMobility CS Blockly model

From the viewpoints of communication, double clicking on RUMIs blocks of EMobilityServices CS, the Fig. 9 shows that the EMobilityService CS provides the services:

• EMobilityService:do\_charging\_reservation, EMobilityService: get\_available\_ charging\_opportunities, EMobilityService: re\_schedule through the EMobility Service-DriverApp RUMI

Fig. 9. The EMobility CS RUMI model

Furthermore, the EMobilityService-CSO is connected to the CSO-EmobilityService RUMI in order to call the services provided by CSO CS. The Fig. 9 shows also that EMobilityService has the State Variable:

EV-Smart Meter CS. On expanding the EV-SmartMeter CS block as depicted in Fig. 10, it shown that the EV-SmartMeter CS communicates with Chargingpoint and Meter Aggregator CSs by the EV-Smart Meter-Chargingpoint RUPI and EV-Smart Meter-Meter Aggregator RUMI respectively.

Figure 10 shows also that the EV-SmartMeter CS provides the service EV-Smart Meter: Get\_energy\_consumption and implements MAPE Algorithm.

Fig. 10. The EV-smart meter CS Blockly model

From the viewpoint of communication, double clicking on RUMI and RUPI blocks of EV-Smart Meter CS, Fig. 10 shows that the EV-SmartMeter CS provides the service EV-Smart Meter: Get\_energy\_consumption through the EV-Smart Meter-Meter AggregatorRUMI, and the EV-Smart Meter-Chargingpoint RUPI is connected to the Charging Point-EV-SmartMeter RUPI of the Chargingpint CS.

Implementation. As stated in D4.2 [4], the EV SoS must be designed to provide a friendly and convenient service to the users and, at the same time, profitable to the provider. Planning and scheduling is of paramount importance for both energy providers and users: as an example, on one side, if the charging requests are spread during the day, there will be limited and/or controlled load peaks on the grid to be handled, thus the energy price may not vary abruptly over time and prioritized consumers (e.g., police and fire-fighter vehicles, ambulances, etc.) will be easily handled by the charging station operators. On the other hand, knowing the energy prices and available time slots, the users will be able to carefully plan the recharging operation while keeping the service affordable.

A typical scenario would be as follows: EVs travel through a wide area, where several charging station operators provide recharging services, by means of charging points. Drivers in need of power for their EV can provide the expected charging context (duration, power, etc.) to the e-mobility service in order to receive information regarding recharging time slots and associated energy prices of each charging station operator. A load management optimizer that cooperates with the charging station operators carries out planning and scheduling activities. The interested driver will then choose one of the slot-price pair possibility for recharging its vehicle, will be allowed to plug-in its EV at the charging point of the chosen charging station operator during the reserved time slot only, and the amount due will be based on the energy consumption times the booked price. At the end of the recharging operation, the driver receives a billing invoice.

With this scenario in mind, and using the AMADEOS tooling, we designed a SoS (shown in Fig. 11, and described in D4.1 [3]) based on a typical EV rollout, in particular based on the desired situation in the Netherlands, based on interviews and workshops with experts, both in the EAB and Grid operators. This SoS was modelled using the Blockly tool (this is described in D4.2 [4]) and a simulator was generated from the tool. We combined this simulator with a simulation toolkit called SimPy and performed a number of experiments based on the scenarios defined in D4.2. This served to both validate the simulator as well as determine if such a simulation could be used to determine and validate possible future designs of the EV charging network, based on varying user and SoS behaviour.

The simulation is built using "SimPy" and uses fixed time slots for reservations. At the moment these slots are 15 min long and start on 0, 15, 30 and 45 min past each hour. Each Driver that wants to charge will attempt to make a Reservation for a Charging Point (via the E-Mobility Service) at a CSO. Because of these timeslots, a Reservation will only begin at the start of the next timeslot and will last for a number of time slots as calculated by the CSO. After the reserved time has passed, the Driver is expected to Plug Out its EV from the Charging Point, though he can do so at an earlier point in time (if the EV is already fully charged, or the driver otherwise decides to do so).

Below, we summarize some of the results received. These are preliminary results, and as the data retrieved from the simulator is extremely extensive, we identify only some of the more interesting aspects. In general, we found the tooling to be extremely useful, and, for example, several emergent behaviours were discovered in the data that, if proven true, would lead to significant challenges to the electrical grid. Furthermore, the aspects that the SoS was designed to test (primarily security and dependability) were successfully tested and validated. We plan to exploit these results in two manners: First to follow up on the results and perform more experiments after the end of the project, leading to publications, and secondly both Thales and ENCS are planning to make the simulator code available online, so other researchers can validate and use the same simulation code.

Results from the EV Charging Scenarios. The first scenario that we will discuss is a usage case where EV drivers do not immediately remove their vehicles from the charging points for a period of time. This means that those drivers who do not remove their EVs are acting badly – blocking charge points from other users. We chose a fixed period of 4 h for this behaviour in this scenario, and the simulation took place over a period of 24 h (simulated time via SimPy). There were 1000 EVs present and 500 Charging Points (CPs). In this simulation, market costs did not cause a significant change in behaviour. The EVs were set to desire a charge of 70%, with a "must charge" threshold at 20%. Four states are defined in the simulation: Charging, Driving, Idling and Waiting. Charging and Driving are self-evident. Idling was periods of time when the EV was not in use (due to lack of driver need). Waiting was an undesirable state where an EV was waiting for a CP to become available. In order to stress the Grid and

Fig. 11. EV Charging SoS

the SoS as a whole, Idling was the least desired state of the three normal states – the chance of an EV idling was set to 10%. Finally, note that the drivers, cars and initial state were randomly assigned and a period of 15 min was allowed to let the state settle. This can be seen in an initial jump in all of the graphs.

In Fig. 12, you see four basic types of behaviour: Driving, Idling, Waiting (for a free Charging Point) and Charging. The numbers on the X-axis reflect the number of EVs in that state. In this scenario, (and in all our tests, due to global variables that were

Fig. 12. Initial state where no drivers act badly

set) the initial state was close to optimum for each of the EVs, with around 70% driving and 30% charging. The initial spike in charging (around 10000 s into the simulation) is due to the initially driving EVs falling below the 20% lower threshold and requiring a charge. You can then see that after around 20000 s, the system reaches a steady state with around 70% driving, 20% charging and 10% idle. Note also that there is an insignificant number of EVs in the waiting state.

Fig. 13. 20% of EVs acting badly

Figure 13 shows the first set of EVs acting badly – in this case, 20% delay their disconnection from the CPs and block other drivers from using them. This scenario shows the resilience of the SoS to such behaviour – there is again the same spike in charging after around 10000 s, and a minor amount of EVs waiting, but again after around 20000 s, the SoS reaches a relatively steady state, although with more EVs charging (around 5% more) at any given time (and consequently 5% less driving) that the optimal case. One last thing to note is that all CPs are in use during the initial spike.

Fig. 14. 40% of EVs behaving badly

In Fig. 14, the consequence of bad actors is more apparent. Now, there is a noticeable issue for EV drivers during the initial spike in usage, with a period of time when all of the CPs are in continual use (for around 15000 s) and there are some drivers constantly waiting for service. However, the SoS is still very resilient to this issue – the number of waiting drivers is still very low (maximum was 16 drivers). Furthermore, the average wait time was around 600 s (10 min). Again, this shows the resilience of the SoS to malicious events. However, again there is a drop in number of active drivers when the SoS reaches a steady state of another 8%, with the number of charging drivers up by the same amount.

Fig. 15. 60% of EVs behaving badly

Finally, in Fig. 15, the first significant effects of bad behaviour can be recognized, while 60% of drivers are acting badly. First, the period where the initial spike causes full usage of all CPs, and consequently up to 100 EVs waiting for service at the worst point. Despite this, there is another drop in active drivers when the simulation reaches the steady state – down to around 58% from a high of over 70% in the control simulation.

Fig. 16. 80% of EVs behaving badly

In Fig. 16, we can see some significant changes in the usage profile of the SoS due to the bad behaviour of the EV drivers. In this scenario, during the initial spike, there is now a period of around 35000 s (nearly 10 h in total) where all of the CPs are in constant use. This is also reflected in the more than 200 EVs waiting at one point. In this scenario, we now see less than 50% of the EVs driving when the SoS reaches a

Fig. 17. 100% of EVs behaving badly

steady state, with roughly the same number charging as driving. This means that 20% of the EVs have changed from driving to charging since the control simulation. However, this is taking place where only 20% of drivers are behaving correctly and yet the SoS remains (for the most part) available for use.

In the final simulation, shown in Fig. 17, every driver is now acting badly. In this case, the effect on the SoS is dramatic and relatively catastrophic. In this simulation, for essentially the entire day, all of the CPs are in constant use, the number of drivers is down to less than 50% and for large periods of time, EVs are waiting to charge (more than 30% at the peak).

Market Simulation and Energy Usage. These simulations were intended to determine the reaction of the SoS to malicious behaviour on the part of the drivers. The results described above shows how the SoS behaves from the perspective of the EV drivers. However, there is another important aspect that can also be studied: the reaction of the SoS from the perspective of the Grid. This was calculated from the perspective of the TSO (see Fig. 11). The mission of the SoS is, basically, to ensure the stability of the Grid, and to ensure that large changes in energy generation are not required. In order to achieve stability, a number of measures were enacted. First, the CPOs (there are 5 independent CPOs in this simulation) received a wholesale price from the DSO, via the market, based on:

The goal of a CSO was, using the market price, to ensure they did not stray too far from their forecasted need. Prices were set by the DSO based on five energy bands – the price per unit was set based on the band that the CSO requested, regardless of the actual energy used in reality. The ideal situation for a CSO was to get as close to the top of a band, without exceeding it, as the cost per unit would jump, and make the per unit cost more expensive. Therefore, if a CSO discovered that (based on EV reservations) that they were going to jump up to the next band, their behaviour would be to attempt to get many more customers, by reducing the customer price. We used this aspect to drive competition and the response from the domain experts was that this is indeed a desired future scenario. This aspect proves the evolutionary promise of an AMADEOS SoS design.

Fig. 18. Total network load with 0% bad behaviour.

Fig. 19. Total network load with 40% bad behaviour

Based on the same simulations as shown in Section 0, Fig. 18 shows the reaction of the Grid to the EV charging scenario where no bad behaviour is present. The requests come in 15-minute intervals, and this can be seen in the jagged lines that are present in the graphs. In this instance, there is again an initial jump, after the 15 min settling down period and then a peak and trough after around 18000 s. This is again due to the jump in number of users charging at the start of the simulation based on their initial charge state and desire not to fall below 20%/attain 70% charge. The interesting outcome is the narrow band (between 40 MWh and 60 MWh) that the Grid eventually stabilizes towards. Future research will definitely consider running such experiments over several weeks of simulated time and integrating a typical Grid usage pattern into the demand1 .

The 20% (not shown) and 40% bad behaviour results (see Fig. 19) show how the bad behaviour is increasingly reflected in the variations of load over time. The intermediate results (60% and 80% bad behaviour) show increasingly wide variations, cumulating to the wide swings shown in Fig. 20, where all of the drivers are again behaving badly. These variations once again show how malicious actors will cause significant issues for Grid operators.

1 As discussed in D4.2, the simulation was not run in concert with the MV case study.

Fig. 20. Total network load where 100% of the drivers are acting badly

#### 2.3 Household Management Case Study

Mission. The goals of the SoS are somewhat similar to the ones of the EVC: allow users, i.e. households, to use home appliances (e.g., flexible or general loads) at a convenient price while properly scheduling the energy consumption and distribution to improve revenues for energy providers with the constraint of keeping a balanced load on the grid.

During a normal scenario, a user wants to activate one or more home appliances, thus requiring for energy from the grid. Energy requirements are managed by an energy management gateway every time a person wants to activate home appliances. The energy management gateway handles the scheduling of energy consumption/provision within the household and sends requests to the coordinator to update consumption/production setpoints. The coordinator continuously updates the energy setpoints according to the grid energy availability. The actors need to be time synchronized to provide consistent information to the coordinator for scheduling and planning actions. Furthermore, some home appliances should be energy prioritized with respect to others (e.g., refrigerators, air conditioning systems, etc.).

Conceptual Level. In this section we report the result of the activities performed at the conceptual level for the Household management case study. For each viewpoint we list the most representative identified SoS requirements defined taking as input the SoS meta-requirements [1]. Though, the requirements come from meta-requirements, the traceability of the full set of requirements on meta-requirements is not provided. Some of the representative requirements are given below (the full set of requirements can be found [4]).

Viewpoint Examination. The objective for which the SoS is designed for is to allow end customers (i.e., households) to interact with the coordinator in order to request the activation of some particular appliance.

Architecture Viewpoint. The HHM consists of a subset of the CSs of the SoS "Household scenario" described in Chap. 2.2 of [4]. In particular (Table 2):

Table 2. Constituent systems for Household case study

Fig. 21. Household management SoS architecture

High level representation of HH Management interactions.

A pictorial view of the HH management SoS is reported in Fig. 21.

Figure 21 shows a pictorial view of the temporal sequence 1–7 comprising, for each step, the involved systems and connections from Fig. 10.

In the following, the steps performed in a nominal household scenario are described. Each numbered step corresponds to a specific, adimensional, time instant at which the corresponding actions are carried out.

Dynamicity Viewpoint. The aforementioned case study is therefore very dynamic, in the sense that the service provided by the electrical grid is always changing, according to the customer requests for home appliances.

Evolution Viewpoint. Due to technological advances and new customer needs, the EMG should have the capability of easily integrate new HMIs, automation devices and loads. As an example, an householder may want the same interaction provided by the in-house Display on his smartphone and to use it not only while connected to the LNAP, but also when outside to schedule a washing machine while connected to the mobile carrier.

Multicriticality Viewpoint. Up to this point, we have described the case study assuming all the appliances with the same criticality. However, an HH consists of a large variety of possibly interconnected (net-centric) appliances providing different kind of services and, thus, characterized by different priority and criticality levels.

#### 2.4 Medium Voltage Control SoS Case Study

Mission. The mission is to provide charging services to EV drivers and to provide energy-related services to households, but also to highlight interesting emergent phenomena that would not arises within the single SoS improving their interoperability with predictable, dependable behaviour avoiding negative cascading emergent effects.

Conceptual Level. The Medium Voltage energy distribution infrastructures are needed to interconnect the EV-Charging SoS and HH Management SoS and consist of the following set of CSs (Table 3):

Table 3. MV Energy distribution constituent systems

This section describes at logical level the infrastructures related to the Medium Voltage energy distribution (Fig. 22) and the examples of emergent phenomena that could be coming from the interoperability of Ev-Charging SoS and HH Management SoS.

In the following, the steps performed to manage the energy distribution are described.

Fig. 22. Architectural view of Medium Voltage Control (3).

Viewpoint Examination. Consider now the case where the owner of the HH has the capability of storing energy from DER and also owns an EV. As described in Sect. 2.1,

Fig. 23. The energy used to recharge an EV is bought from the market

their EV can be charged at a CSO by means of a CP, the energy used in this process is generally bought by the CSO from the energy market and sold at a specific price per each kWh (plus power grid fees including taxes) (Fig. 23).

The CSO may also buy energy directly from the HHs for recharging EVs, like, e.g., when the HH energy price is cheaper than the one provided by the market or for compensating effects of some energy-related service disruptions (e.g., disconnected energy generators). In this case the total energy price would be the energy cost from the HH plus the power grid costs with taxes (Fig. 24).

From the above scenario, the below beneficial emergence scenarios were identified:

Consider now the scenario in which the owner of the HH wants to recharge its EV to a CSO. It is evident that the energy needed to recharge the EV can be bought and provided, by the CSO, form either the energy market or HHs, including the one owned by the driver of the EV.

The HH owner can check the energy price, relative to the energy market, of the CSO and decide whether to recharge its EV with such energy or to use the energy stored at home using DER, i.e. the HH owner can compare the price of the energy stored at home with the one sold by the market and decide accordingly (Fig. 26).

Suppose that the energy on the market is more expensive than the one stored at the HH. The HH owner will then decide to use its own stored energy to recharge its EV, enabling the energy transmission from the house to the CSO. Using the energy stored at home will cost the owner to pay only the provision of power grid including taxes (the price may be related to the total power grid usage time or to total transmitted kWh) (see Figs. 25 and 27).

Fig. 24. The CSO buys energy from the HH.

Fig. 25. The CSO can buy energy from either the market or HHs

These new emergent behaviors clearly arise with the interconnection and the available communications between smart constituent system of a SoS, which the AMADEOS framework is able to capture and describe. This way the possible large variations in the network load could be attenuated by a suitable adaptation of the prices. However, if this adaptation process is not correctly implemented (for instance, if the

Fig. 26. The HH owner can check and compare the price of the energy stored at home with the one of the market

adaptation is too quick and steep), then undesirable global phenomena, such as oscillations of prices and load, could take place.

Time Viewpoint. For identifying some of the time aspects, we describe two scenarios that extend the previous case studies with descriptions of some systems, processes, and interactions where the existence of an accurate global time is essential for measuring and controlling the state of the power grid.

Fig. 27. The HH owner decide to recharge its EV with the energy stored at home

Scenario 1: Maintaining the Power Phasor Parameters. The power phasor is defined by the amplitude, frequency, and relative phase of the electric current or voltage in a power line. As the load in the power network varies, the braking momentum of the power generators increases, so the generators' rotors slow down. All variations in network load lead thus to variations in power frequency. Traditionally, these variations were compensated locally, by applying the load measurement signals to an Automated Generator Control (AGC), which basically is an electro-mechanical governor that compensates for the changes in braking momentum. The AGC signals are collected from different points in the network, and forwarded to large production facilities [5]. In order for the AGC to work efficiently, the following requirements must be satisfied:

The adjustment signals for AGC are generated once every 2 to 4 s, so the transmission time requirements are not very strict [5].

Another phenomenon that alters the power phasor is the reactive power produced by the reactance (i.e. the reactive part of impedance) of a load. A capacitive load will make the voltage lag behind the current, whereas an inductive load will make the current lag behind the voltage. Although the reactive power does not dissipate energy, it affects the efficiency of power transmission to the consumers, so this too must be controlled, such that the reactive angle φX is kept as close as possible to zero.

To compensate for reactive power introduced in the network primarily by motors and transformers, capacitive loads (capacitor banks) can be switched on or off by control units. Unlike the AGC, which acts centrally at the production facility, the reactive power control can be distributed across the grid [6]. In the AMADEOS scenarios this can be achieved by the LMO.

In a purely reactive approach the individual measurements by SMs are collected by LMOs (via the EMGs or CSOs), which try to compensate for load variations using the locally available power reserves and by re-scheduling, when possible, of some loads. When these local measures are not sufficient, appropriate AGC signals are transmitted to the power generation facilities. Of course, particular implementations could include additional aggregation layers, but those do not change significantly the problem.

Fig. 28. Power triangle

Fig. 29. Load measurement and control architecture

Concerning the reactive power in the grid, the LMOs try to locally compensate. In case when the local capacitor banks do not suffice, a signal must be broadcasted (Reactive Load Control - RLC) to request additional capacitive loads be switched on by other LMOs.

Overall, the load measurement and control architecture includes a number of geographically distributed LMOs and one local AGC for each of the power plants (see Fig. 28).

The effectiveness of this reactive approach is limited, due to the delays in the transmission of AGC signals and the time required for adjustments. A more advanced approach (see Fig. 29) tries to predict the variation of the load based on SCADA measurements from various places in the network, and a forecasting of load variations [7]. The prediction function is based on a sampling of the real load over some past time interval.

Although the AGC and reactive load balancing are still slow (in the order of seconds), a good estimation of the current state of the system, as well as the load prediction, require better time accuracy than in the previous case, and global time awareness. Indeed, if two consecutive measurements from a given location in the power grid arrive in reverse order, then the detected variation trend is reversed. If this reversal occurs randomly over a large number of measurements, then the whole predictive load balancing has, on average, no effect. In a worst-case scenario the order of a sequence of measurements can be altered such that the variations are amplified, leading potentially to the activation of circuit breakers on some network segments.

Specific timing requirement can be identified for the two cases:

A. Predictive balancing of active loads – in this case we assume that LMOs compete with each other, so they are not willing to use their local energy reserves to balance the load at other LMOs. In this case AGC messages are sent to the power plant to increase its power output. Similarly, when the load decreases and LMO's storage capacity is reached, another message is sent to decrease the generators' power output. Obviously, these are sent asynchronously by all LMOs. In order to ensure a proper ordering of these events at the local AGC Message Queue, all AGC messages must be

Fig. 30. Load measurement and predictive control architecture

time-stamped. The time difference between the local clocks that provide the time stamps must be smaller than the sampling interval2 of the Active Load Prediction (ALP) function.

B. Predictive balancing of reactive loads – in this case we assume that LMOs help each other since the cost of switching capacitor banks is low. A simple protocol could be defined as follows:

	- If the first message is RLC+ and the polling LMO has the disposable capacitance requested by that message, it removes this message in the queue and then switches on an appropriate capacitance.
	- If the first message is RLC− and the polling LMO has more allocated capacitance than that requested by the message, it removes this message and switches off the appropriate capacitance.

Of course, more elaborated and efficient protocols can be implemented for processing more messages at once, but for the purposes of this study this is sufficient.

Since each LMO has a local load predictor, the polling of the queue needs to be done at the same rate as the same sampling rate of the load prediction function. For a correct functioning of these predictors, the messages must be correctly ordered based on time stamps, as in the case of AGC. Additionally, the polling cannot be done completely asynchronously since this could lead to race conditions due to the distributed queue.

2 Note that when the guaranteed transmission time for the AGC messages is smaller than the ALP sampling interval, then no time stamping (and thus no global time) is needed.

Fig. 31. Cascaded propagation of disruption and time stamping of breaking events; in nominal terms, t0i ¼ t0j ¼ t0k

The time requirements resulting from these two cases are:

Scenario 2: Forensic Analysis of Disruption Events. When a disruption occurs in a network, it may lead to cascading effects (Fig. 31). In case of complex networks of producers and consumers it is not always easy to identify the original event that initiated the cascaded failures. However, this information is crucial when network design needs to be improved to increase the resilience of the network, or when the network operators are liable for the quality of the services they provide.

To cope with this requirement each circuit breaking event could be logged by the breakers' control computer. This way a chain of events can be reconstructed and the root cause identified. However, given the high speed in which these events occur3 , the temporal order of the logged events may be distorted by the differences in the local

3 A typical reaction time for modern equipment is 1/8th of a 20 ms cycle [8].

Fig. 32. The full SmartGrid model generated using the model query: "return true;" (i.e.: select all blocks). Here, the triangles represent systems, star represents the SoS and other blocks are represented by circles. The color of the blocks is the same as the one in the Blockly model.

clocks used for time stamping. For this reason, it is necessary that the synchronization error between local clocks is kept below the breaking time of the fastest breaker plus the propagation time of the disruption. This is illustrated in Fig. 30, where a time value t0 corresponds to three different moments in three circuits whose local clocks are not synchronized. If the delay between two local clocks is too large, as it is the case for circuits i and j, then the events cannot be ordered correctly.

Finally, for both scenarios the maximum transmission times for all the messages related to events occurring in the smart grid must be guaranteed. If this is not the case, then most of the functions described here cannot be implemented.

#### 3 Conclusion

This chapter has shown how a realistic case study can be modeled in the Blockly tool and directly simulated. A high-level view of the entire model can be seen using the model query tool of supporting facility to select all blocks. Below is the graph of the model consisting of elements and relationship between each blocks. The diagram shows the complexity of the full model (Fig. 32).

Modeling complex and pervasive infrastructures as the one used as case study clearly highlights how the support of a precise conceptual model and of specific tools for its instantiation is fundamental for a sound and comprehensive codification of the various properties of the whole. At design time the identification of causal loop in the lower levels of the hierarchy, enabled by the support for simulation through model execution, is a mandatory step to identify possible emergent behaviors at the higher levels, that may lead, also in future evolution of the system of systems, to a violation of system requirements. A correct representation of the environment is necessary. Global time Awareness and monitoring are fundamental to early detect and to contain the effect of detrimental emergence phenomena at run time. The main benefits of the AMADEOS approach can be easily seen in the results from the simulations: The AMADEOS architectural framework and associated tools allow an SoS architecture to be comprehensively designed and a simulation extracted that can be tested. This allows system architects to quickly test hypothesis regarding future systems and determine what attributes will lead to advantageous or poor results.

# References

Open Access This chapter is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, duplication, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the work's Creative Commons license, unless indicated otherwise in the credit line; if such material is not included in the work's Creative Commons license and the respective action is not permitted by statutory regulation, users will need to obtain permission from the license holder to duplicate, adapt or reproduce the material.

# Glossary

#### Absolute Timestamp

An absolute timestamp of an event is the timestamp of this event that is generated by the reference clock

#### Acceptance Test

A test that determines if a state in the problem space is a member of the solution set

#### Access Control

Access control is concerned with providing control over security critical actions that take place in a system. Providing control over actions consists of explicitly determining either the actions that are permitted by the system, or explicitly determining the actions that are not permitted by the system

Access Control Model

An access control model captures the set of allowed actions as a policy within a system

#### Accuracy

The accuracy of a clock denotes the maximum offset of a given clock from the external time reference during the IoD, measured by the reference clock Acknowledged SoS

Independent ownership of the CSs, but cooperative agreements among the owners to an aligned purpose

#### Action

The execution of a program by a computer or a protocol by a communication system

Action Sequence

A sequence of actions, where the end-signal of a preceding action acts as the start signal of a following action

Activity Interval

The interval between the start signal and the end signal of an action or a sequence of related actions

Actuator

An actuator is an interface device that accepts data and control information from an interface component and realizes the intended physical effect at its placement in the physical environment

Agility (of a system)

Quality metric that represents the ability of a system to efficiently implement evolutionary changes

Architectural Style

The set of explicit or implicit rules and conventions that determine the structure and representation of the internals of a system, its data and protocols

#### Arrival Instant

The instant when the first bit of a message arrives at the receiver Artifact

An entity that has been intentionally produced by a human for a certain purpose

Atomic Action

An atomic action is an action that has the all-or-nothing property. It either completes and delivers the intended

result or does not have any effect on its environment

#### Attribute

A characteristic quality of an entity Authentication

The process of verifying the identity or other attributes claimed by or assumed of a subject, or to verify the source and integrity of data

# Authorization

Authorization is the mechanism of applying access rights to a subject. Authorizing a subject is typically processed by granting access rights to them within the access control policy

#### Authority

The relationship in which one party has the right to demand changes in the behavior or configuration of another party, which is obliged to conform to these demands

Autonomous System

A system that can provide its services without guidance by another system Availability

Readiness for service

Behavior

The timed sequence of the effects of input and output actions that can be observed at an interface of a system

#### Bottom Up Design

A hierarchical design methodology where the design starts at the bottom of the holarchy or formal hierarchy

#### Business Value

Overarching concept to denote the performance, impact, usefulness, etc. of the functioning of the SoS

#### Capability

Ability to perform a service or function Cascade Effect

A cascade effect exists, if in a system with a multitude of parts at the micro level a state change of a part at the micro-level causes successive state

changes of many other parts at the micro level such that the cumulative effect of the totality of these state changes results in a novel phenomenon

Causal Loop

A causal loop exists, if the emergent property at the macro-level causes a change of the state of the parts at the micro-level

Causal Model

Abstract model describing the causal dependencies between relevant variables in a given domain

Causal Order

A causal order among a set of events is an order that reflects the cause-effect relationships among the events

Channel

A logical or physical link that transports information among systems at their connected interfaces

Channel Model

A model that describes effects of the channel on the transferred information

Checked Message

A message is checked at the source (or, in short, checked) if it passes the output assertion

Ciphertext

Data in its encrypted form

Clock

A (digital) clock is an autonomous system that consists of an oscillator and a register. Whenever the oscillator completes a period, an event is generated that increments the register

#### Clock Ensemble

A collection of clocks, not necessary in the same physical location, operated together in a coordinated way either for mutual control of their individual properties or to maximize the performance (time accuracy and frequency stability) and availability of a time-scale derived from the ensemble) Collaborative SoS

Voluntary interactions of independent CSs to achieve a goal that is beneficial to the individual CS

Communication Action An action that is characterized by the execution of a communication protocol

by a communication system

Communication Protocol The set of rules that govern a communication action

Compatibility (full, Itom)

The Itom that is sent by the sender is received by the receiver without modification

Component

A subsystem of a system, the internal structure of which is of no interest

Computational Action

An action that is characterized by the execution of a program by a machine Concept

A category that is augmented by a set of beliefs about its relations to other categories, i.e., existing knowledge, is called a concept

Concise State

The state of a system is considered concise if the size of the declared ground state is at most in the same order of magnitude as the size of the system's largest input message

Confidentiality

The absence of unauthorized disclosure of information

Configuration Interface (C-Interface) An interface of a CS that is used for the integration of the CS into an SoS and the reconfiguration of the CS's RUIs while integrated in a SoS

Connected Interface

An interface that is connected to at least one other interface by a channel Connection System/Gateway Component/Wrapper

A new system with at least two interfaces that is introduced between interfaces of the connected component systems in order to resolve property mismatches among these systems (which will typically be legacy systems)

Consistency

The property of a set of entities that see the same data at the same time

Constituent System (CS)

An autonomous subsystem of an SoS, consisting of computer systems and possibly of controlled objects and/or human role players that interact to provide a given service

Constraint

A restriction in the problem space Construct

A non-physical entity, a product of the human mind

Consume/Produce (CP) Paradigm At the sender, the communication system consumes the message from a sender queue and at the receiver the communication system adds the received message to a receiver queue Context

The set of cultural circumstances, conventions or facts, and the time that surround and have a possible influence on a particular thing, construct, event, situation, system, etc. in the UoD

Context Compatibility

the same data (bit pattern) is explained in the same way at the sender and at the receiver

Context Incompatibility

the same data (bit pattern) is explained differently at the sender and at the receiver

Contract

Agreement between two or more parties, where one is the customer and the others are service providers. This can be a legally binding formal or an informal "contract". It can be expressed in terms of objectives Control Flow The flow of control signals when executing a protocol Coordinated Clock A clock synchronized within stated limits to a reference clock that is spatially separated Correct Message A message is correct if it is both timely and value correct Critical Service A critical service is the service of a system that requires a specific criticality level Criticality Criticality is a designation of the required criticality level for a system component Criticality Level The criticality level is the level of assurance against failure Cryptography The art and science of keeping data secure Cyber-Physical System (CPS) A system consisting of a computer system (the cyber system), a controlled object (a physical system) and possibly of interacting humans Cyber Space Cyber space is an abstraction of the Universe of Discourse (UoD) that consists only of information processing systems and cyber channels to realize message-based interactions Cycle A temporal sequence of significant events that, upon completion, arrives at a final state that is related to the initial

state, from which the temporal sequence of significant events can be started again Data A data item is an artefact, a pattern, created for a specified purpose Data Flow The flow of the payload data of a message from a sender to the receivers Datagram A best effort message transport service for the transmission of sporadic messages from a sender to one or many receivers Declared Ground State A declared data structure that contains the relevant ground state of a given application at the ground state instant Decryption The process of turning ciphertext back into plaintext Dependability The ability to deliver service that can justifiably be trusted Design The process of defining an architecture, components, modules and interfaces of a system to satisfy specified requirement Design for Evolution Exploration of forward compatible system architectures, i.e. designing applications that can evolve with an ever-changing environment. Principles of evolvability include modularity, updateability and extensibility. Design for evolution aims to achieve robust and/or flexible architectures Design for Evolution in the context of SoSs Design for evolution means that we understand the user environment and design a large SoS in such a way that expected changes can be accommo-

dated without any global impact on the architecture. 'Expected' refers to the

fact that changes will happen, it does not mean that these changes themselves are foreseeable Design for Testability The architectural and design decisions in order to enable to easily and effectively test our system Design Inspection Examination of the design and determination of its conformity with specific requirements Design Walkthrough Quality practice where the design is validated through peer review Designer An entity that specifies the structural and behavioral properties of a design object Deterministic Behavior A system behaves deterministically if, given an initial state at a defined instant and a set of future timed inputs, the future states, the values and instants of all future outputs are entailed Diagnosis Interface (D-Interface) An interface that exposes the internals of a Constituent System (CS) for the purpose of diagnosis Directed SoS An SoS with a central managed purpose and central ownership of all CSs Downward Causation The phenomenon that some novel macro-level properties have causal powers to control the micro-level properties from which they emerge Drift The drift of a physical clock is a quality measure describing the frequency ratio between the physical clock and the reference clock Duration The length of an interval

Dynamicity of a system The capability of a system to react promptly to changes in the environment Emergence A phenomenon of a whole at the macro-level is emergent if and only if it is of a new kind with respect to the non-relational phenomena of any of its proper parts at the micro level Encryption The process of disguising data in such a way as to hide the information it contains End Signal An event that is produced by the termination of an action Entity Something that exists as a distinct and self-contained unit Entourage of a CPS The entourage is composed of those entities of a CPS (e.g., the role playing human, controlled object) that are external to the cyber system of the CPS but are considered an integral part of the CPS Environment of a System The entities and their actions in the UoD that are not part of a system but have the capability to interact with the system Environmental Dynamics Autonomous environmental processes that cause a change of state variables in the physical environment Environmental Model A model that describes the behavior of the environment that is relevant for the interfacing entities at a suitable level of abstraction Epoch

An instant on the timeline chosen as the origin for time-measurement

#### Error

Part of the system state that deviated from the intended system state and could lead to system failure Error Containment Error Containment prevents propagation of errors by employing error detection and a mitigation strategy Error Containment Region (ECR) A set of at least two Fault Containment Regions (FCRs) that perform error containment Established Rule An observed consequence that often follows if a set of antecedent conditions applies Event A happening at an instant Event Variable A variable that holds information about some change of state at an instant Event-triggered (ET) Action An action where the start signal is derived from an event other than the progression of time Evolution Process of gradual and progressive change or development, resulting from changes in its environment (primary) or in itself (secondary) Evolutionary Performance A quality metric that quantifies the business value and the agility of a system Evolutionary Step An evolutionary change of limited scope Evolvable architecture An architecture that is adaptable and then is able to incorporate known and unknown changes in the environment or in itself Execution Time The duration it takes to execute a specific action on a given computer

Explained Emergence

An emergent phenomenon that is observed at a macro level is explained emergent if a trans-ordinal law that explains the occurrence of the emergent phenomenon at the macro level out of the properties and interactions of the parts at the adjacent micro level is known (or has been formulated post facto)

Explanation

The explanation of the data establishes the links between data and already existing concepts in the mind of a human receiver or the rules for handling the data by a machine

Explicit Flow Control

After having sent a message, the sender receives a control message from the receiver informing the sender that the receiver has processed the sent message

External Clock Synchronization The synchronization of a clock with an

external time base such as GPS External Interface

A Constituent System (CS) is embedded in the physical environment by its external interfaces

Failure

The actual system behavior deviation from the intended system behavior

Failure Modes

The forms that the deviations from the system service may assume; failure modes are ranked according to failure severities (e.g. minor vs. catastrophic failures)

#### Fault

The adjudged or hypothesized cause of an error; a fault is active when it causes an error, otherwise it is dormant

Fault Containment Region (FCR)

A Fault Containment Region (FCR) is a collection of components that