# Giuseppe Serazzi

# Performance Engineering

Learning Through Applications Using JMT

Performance Engineering

Giuseppe Serazzi

# Performance Engineering

Learning Through Applications Using JMT

Giuseppe Serazzi Milano, Pavia, Italy

ISBN 978-3-031-36762-5 ISBN 978-3-031-36763-2 (eBook) https://doi.org/10.1007/978-3-031-36763-2

© The Editor(s) (if applicable) and The Author(s) 2024. This book is an open access publication.

**Open Access** This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

*In memory of Larry Dowdy, Martin Reiser, Paul Schweitzer, Kennet Sevcik before the night falls*

#### **Preface**

This open-access book aims to improve users' skills needed to implement models for performance evaluation of digital infrastructures. Models are widely used in any branch of engineering. Unfortunately, their use for performance evaluation of computing infrastructures is pretty much limited to modeling specialists and not to their end-users, who have complete knowledge of the analyzed phenomena. Among the reasons for this limitation, there is the intrinsic complexity of the modeling process, which cannot be fully learned with the academic approach alone, and the frequent use of unnecessary mathematical details, which typically create a fog shield that hides the key features of the models. Furthermore, it is known that to increase the ability to build reliable models, it is necessary to accumulate experiences that can only be learned through trial-and-error work by solving problems of different difficulties.

Based on these considerations, we tried to keep this book as simple as possible by adopting the following guidelines. On one side, we present a collection of modeling studies of increasing complexity, describing the assumptions made and motivating the decisions taken. Readers are introduced to the modeling process gradually, learning the basic concepts step-by-step as they go through the case studies. On the other side, we try to avoid superfluous exposure to mathematical concepts. For interested readers, we reported in *Appendix* some basic notions that may be useful to know.

Among the various techniques used in performance evaluation modeling, we will use *Queueing Networks*, possibly integrated with *Petri Nets* when the characteristics of the models require it. This type of model provides a good balance between the accuracy of results, complexity, and parameterization effort, for a large variety of problems. Analytical, simulation, and asymptotic techniques are applied to solve the models.

The book is structured in six Chapters and an Appendix. Chapter 1 is focused on the description of the model building process. The input parameters, the output metrics, and the operational laws are illustrated. The most important steps to building models to be solved with simulation and analytical techniques are reviewed. In the following Chapters, fifteen case studies of increasing complexity covering different aspects of performance evaluation are described.We believe that readers could benefit from analyzing these models by focusing on the abstraction process applied to their design. In Chaps. 2 and 3, models of systems with homogeneous and heterogeneous workloads are presented, respectively. The problem of bottleneck identification and performance optimization is addressed for both types of workloads. Chapter 4 is devoted to the analysis of the impact of variability of the traffic of requests and service demands on throughput and response time. Chapter 5 focuses on parallel computing and describes the influence of different synchronization policies on performance. Chapter 6 presents four case studies derived from real-life scenarios: a surveillance system, an architecture that autoscales for load fluctuations, a web app workflow simulation, and a crowd computing platform. The autoscaler model consists of Queueing Networks and Petri Nets integrated, i.e., it is a *multi-formalism* model.

The Java Modelling Tools (JMT), a *open source* suite of six tools for performance engineering and capacity planning using Queueing Networks and Petri Nets, were applied to build and solve the models. Details on JMT, which can be downloaded from http://jmt.sourceforge.net, can be found in [8]. JMT is a project coordinated and co-developed by Politecnico di Milano (G. Serazzi) and Imperial College London (G. Casale).

This book is intended as a text for courses in performance evaluation and modeling for graduate and senior-level computer science students. Researchers and practitioners whose work is related to performance evaluation of computer infrastructures will find it useful as a reference text. It can be used also as a supporting text for courses in disciplines outside of computer science that require the use of modeling to evaluate the performance of their applications.

We hope you will find the *learning through applications* approach followed in this book useful for your work, and apologize in advance for the mistakes you will find. The author cannot be considered responsible for errors that you may introduce in your work due to the content of this book.

Milano, Pavia, Italy May 2023

Giuseppe Serazzi

## **Acknowledgments**

The author wishes to thank Giuliano Casale for his important effort in supporting the JMT project. Thanks to his fundamental coordination and co-development work, the JMT suite has achieved the technical level it is recognized for. *Thanks for your scientific support!* Special thanks also go to Marco Gribaudo, for the enlightening and endless discussions that have increased my knowledge on performance modeling. *Thanks for your patience!* The author is grateful to Andrea Sianesi and Eugenio Gatti, respectively, President and General Director of the Fondazione Politecnico di Milano, for supporting the publication of this book as Open Access. *Thanks for your enthusiasm in accepting my request!*

Giuseppe Serazzi

# **Contents**

#### **Chapter 1 The Process of Modeling**

#### **1.1 Model Implementation**

Typically building models is a fairly easy task, but making them *accurate representations* of the phenomena to be reproduced is a *completely different* matter. The potential sources of error are so many! Among them, the most frequent regard the wrong interconnections of the components, the inaccurate values assigned to input parameters and the incorrect use of the techniques and tools adopted to implement and solve the models.

The construction of a model requires several iterations at different levels of granularity. At the highest level the user need to iterate many times between three different operational environments: the *Real world*, the *Abstract space*, and the *Modeling domain*. The process of modeling is outlined in Fig. 1.1. In this book we will focus primarily on the *Modeling Domain* phase.

A modeling study typically begins in the *real-world environment* with the observation and measurement of the phenomenon that must be reproduced. The interactions among the various components, or resources, must be assessed and their influence on the behavior of the phenomenon must be investigated. The variables that describe the quantities that we measure or we want to estimate will be referred to as *performance metrics* or *indexes*. Usually, some key components are very critical for the success of the study for the strong influence they exert on the metrics of interest, while others have a negligible impact on them. Since the effect on the results produced by the latter are minimal or zero, they can be ignored without affecting the validity of the model and thus greatly reducing its complexity.

Among the key components of the model it is very important to identify the most requested one, i.e., the *bottleneck*, as it determines the performance of the overall system. The saturation condition of the bottleneck depends on the characteristics of both the service requests and the component itself. Note that due to the typical fluctuations of a workload, both in the intensity and in the amount of work required, the bottleneck canmigrate among different resources generating abrup changes in performance. Ignoring this condition may invalidate the entire modeling study. Bottleneck

migration is very common when the workload consists of different applications with highly variable service demands.

The technical features of the components, the assumptions introduced, the workload considered, and the statistical properties of the variables used must be accurately described. Depending on the type of study, the data required for these operations can be derived directly from measurements or must be generated by analytic functions.

The knowledge gained from the analysis of the real-world phenomenon will be merged in the *Abstract space* (see Fig. 1.1) with the technical background of the modeler. In this phase, decisions on the modeling technique to be adopted, on the plausibility of the assumptions introduced, on which components should be considered, on the metrics to be used, and many others must be taken. The actions that must be done in the abstract space rely upon the experience and creativity of a person rather than her/his technical background. These skills can neither be learned with an academic approach nor passively absorbed; instead them build up with experience gained through daily trial-and-error work.

The work in the abstract space end with the implementation of a first version of the model. We are now entering in the *modeling domain*. The process of implementing a model is intrinsically iterative and an *incremental approach* is usually adopted, and strongly recommended.

The components represented in the model are progressively increased starting from a small initial set that must include the bottleneck and other key components that have a great impact on the results. At each iteration the complexity of the model is increased until the level of detail of the metrics obtained matches that of the objectives of the study.

Once a model with its workload are completely defined and parameterized, the *validation phase* starts to assess its accuracy. Several methods of analysis are used to evaluate different properties of the model. The techniques adopted are function of the type of study to be conducted.

A model should be calibrated,its *sensitivity* and robustness must be assessed, its performance must be forecast with a projection technique, and the domain of validity of its results must be evaluated. Typically, a validation technique requires several iterations at low level of granularity. When the system to be analyzed is operating, the performance metrics obtained as model outputs must be compared with those measured from the real system while it processes the production workload. In this case, the *calibration* of the model consists in the tuning of its input parameters so that the differences between the two sets of performance metrics (the measured outputs compared to those of the model) are negligible or at least tolerable. In case of unacceptable differences, the input parameters must be recomputed and the assumptions introduced (including the layout of the model) should be revised.

Great attention must also be paid to the selection of the measurement interval (the observation interval) in which data on the behavior of the system components and the workload executed are detected. The data must be collected when the workload processed is representative of the load that is typically executed by the system. In some cases, the measurement period may consist of several disjoint intervals. The main steps of the modeling design process and the operations required by the incremental approachare shown in Fig. 1.2.

#### **1.2 Inputs and Outputs of Models**

Any computing system can be viewed as a set of resources (hardware and software) that execute the processing requests submitted by users. Therefore, the input parameters of a model can be divided into two groups regarding the load and the resources, respectively. Depending on the system being modeled, in the following processing requests will hereinafter be interchangeably called jobs, applications, customers, requests or users, and the resources will be referred to as stations, elements, components, or service centers.

The arriving requests are collectively called workload, while workload characterization refers to their quantitative description [11]. When the individual workload components have similar characteristics, they are grouped together and their statistical parameters, such as mean, standard deviation, and distribution, are used as inputs to models. In this case, the workload is referred to as homogeneous or single class and the models are called single class models. The components of a workload that consists of various types of applications typically have significantly different service demands. In this case, several groups of components with similar characteristics must be identified and the workload is referred to as heterogeneous or multiple class (multiclass). Each class will be described with its statistical characteristics.

In this Section we focus on the inputs and outputs of single class models. With multiple class workloads the notations become more complex (an index for the classes must be added to the identifiers of the stations), but the meaning of the parameters remain the same. Models with single class workloads will be described in Chap. 2 while models with multiclass workloads will be analyzed in Chap. 3.

**Fig. 1.2** Incremental approach to model building

**Table 1.1** Some basic input parameters and output measures of single class queueing network models with queue and delay stations

Since performance models can be implemented at different levels of detail, the metrics described in Table 1.1 are divided in two groups whether they refer to the component-level or to the system-level. These metrics are the basic ones used in single class models consisting of queue and delay stations. In a queue station requests compete for the server and wait in queue to receive service, then leave the station when finished. In a delay station there is no server competition as it is assumed that there is always a free server for all incoming requests.

In single class models, the class index can be avoided. When a metric refer to a single component of the model, the subscript identify the specific element considered. The subscript 0 (zero) is used to identify metrics which refer to the system as a whole. For clarity, whenever possible, we will refer to the service requests arriving at each station in a component-level model as simply requests. In system-level models, the computational requests submitted by the users will be identified as jobs or customers interchangeably.

A description of some basic input parameters and output results for models that use queue and delay stations follows. The algebraic relationships among some of them, derived in [10, 16, 26], are also reported. Metrics for other types of stations, e.g., fork/join, finite capacity regions, Petri Nets place/transitions, will be described in the case studies where they are used.

Throughout this book we have tried to keep the description as general as possible, but since we have used the JMT Java Modelling Tools to solve the models, it has sometimes been necessary to refer to terms specific to the individual tools used, i.e., JSIMg (the Simulator), JMVA (the Analytical solver), and JABA (the Asymptotic Bound Analyzer).

The open source JMT suite can be downloaded from http://jmt.sourceforge.net.

Let us remark that a large part of the queueing networks solved with analytical techniques are of separable type. This subset of general queueing networks can be solved analytically with very efficient algorithms. Clearly, the property of being separable introduces some restrictions to the system characteristics that can be modeled. Some of them concern the concurrent use of resources, the constraints on the number of requests, the adaptivity of the routing, the priority scheduling algorithms, the blocking policies, the creation and deletion of jobs, the dynamic change of Service times (see Sect.2.3 and, e.g., [4, 25]).

Often these limitations have a minimum impact on the behavior of the system modeled and can be bypassed easily changing the assumptions adopted. In some cases, system characteristics that cannot be modeled directly with separable networks may have a negligible influence on performance. In other cases, the global model can be splitted in several sub-models, some separable and other not, that may be solved with different techniques. The results obtained from the solution of the individual sub-models can then be combined with various techniques in order to obtain the solution of the original global model.

In any case, it is important to note that these limitations only affect models that are solved analytically, while those solved with simulation are not (or minimally) affected.

#### **Input Parameters**

#### **Open/Closed** (types of workloads, types of models)

Workloads, like models, may be of two types: open or closed. When the workload is open, the number of customers in the model is fluctuating and can grow indefinitely as a station becomes saturated, while with a closed workload this number is kept constant. An example of an open workload is the flow of requests arriving from a Internet connection, which is usually quite bursty. The models that execute open workloads are referred to as open models while those executing closed workloads are called closed models(see Fig. 1.3).

An example of a closed workload is a computing infrastructure that can only be accessed by employees of a company. The number of customers is fixed and limited to the company employees. When the maximum number of customers that can be in execution simultaneously is reached, a new customer can enter the system only when a customer completes its execution. In simulation models with open workloads the customers arriving to the system are generated by a Source station, and at the end of the execution are routed to a Sink station (in JSIMg also Fork, Class Switch, and Transition stations may generate customers). Models with both types of open and closed workloads running concurrently are also possible, and are referred to as mixedmodels.

**Fig. 1.3** Examples of the two basic types of models: open (**a**) and closed(**b**)

#### *λ***0**–**Interarrival times** (workload intensity in openmodels)

Describes the characteristics of the incoming flow of requests arriving at the system. In analytical models (JMVA), the exponential distribution of Interarrival times is typically assumed as default, and in this case only the arrival rate λ<sup>0</sup> is required. In simulation models (with JSIMg), different types of distributions (e.g., burst, hyperexponential, Erlang, Pareto, etc.) may be selected and some related statistical indexes should be defined. The number of these parameters varies as a function of the distribution.

*N***0**–**Number of customers** in the model (workload intensity in closed models) This parameter in closed models refers to the mean number *N*<sup>0</sup> of customers in the model. A job arrives at the system, circulate among the service stations (the resources of the system), and then departs at the completion of the execution. In closed models it is immediately replaced by a new job with the same characteristics, keeping *N*<sup>0</sup> constant.

**Type of component**: Queue, Delay, Source, Sink, Class Switch, Fork/ Join, Semaphore, Place/Transition, Finite Cap. Region, Router

In a model, the components representing system resources can be of different types. The most used components in analytically solved models are typically of two types: queue and delay. In queue components, requests arrive, compete for the server, wait in the queue when it is busy, execute when it becomes idle, then exit. A delay component simply introduces a delay in the flow of requests, but no queue is created. In this case, an arriving request will always find an idle server since they are assumed to be infinite. Many more types of components are used in simulation models depending of the tool considered and the complexity of the system to be modeled. For example, in JSIMg Fork and Join stations are used to simulate parallelism, FCR Finite Capacity Regions to control access to model regions, Semaphore to block selectively the requests, Place and Transition to simulate Petri Nets.

#### **Vr**–**Number of visits** per job (to each component)

During its execution, a job visit the components (CPU, disks, storage, ...) several times before leaving the system. *Vr* is the mean number of visits, also referred to as *visit count*, that a job makes to station *r* during its complete execution.

#### **Sr**–**Service time** per visit (for queuecomponents)

The mean time required to component *r* to execute one service request, corresponding to one visit, is referred to as Service time. This value does not include the time waiting in queue when the server is busy. The mean value of *Sr* and/or other statistical indexes (distribution, variance, coefficient of variation, etc.) must be provided according to the technique adopted to solve the model (analytical or simulation). See also comments made above for Interarrival times.

#### **Dr**–**Service demand** per job (for eachcomponent)–*Dr* = *Vr Sr Dr* = *Br* / *C*.

The global amount of service time required by a job to component *r* for a complete execution is referred to as Service demand *Dr*. Its value is given by the product of the Service time *Sr* required by a visit to the component *r* by the Number of visits *Vr* that the job makes to it. The *Dr* are important because it can be shown that the solution of most queueing networks does not depend on the single values of *Vr* and *Sr* but only on their product, i.e., only the service demand matters. This property, exhibited by separable queueing networks, is important as its application reduce the number of input parameters and the complexity of the models. Furthermore, the *Dr* can be measured more easily than *Vr* and *Sr* since, very often, their values are stored directly in the system log files. Another possibility to obtain the values of the *Dr* is to divide the total busy time *Br* of component *r* by the number of jobs *C* completed in the observation interval. The high level of aggregation adopted in the models that use the *Dr* makes it impossible to compute the Throughput and Response time at the single component level, while the performance indexes at the system level are obtained correctly. A more detailed description of Service demands and separablenetworks can be found in the Case Study Sect. 2.3.

#### **Z**–**Think time** per visit (for delaycomponents)

In a delay component, the requests never wait in queue since a server is always available for their execution which therefore always takes on average *Z* time units (this time is typically referred to as Think time). *Z* is the correspondent of service time *S* of queue components and can be considered as the mean delay introduced by a delay component to the flow of requests that goes through it. A delay component is often used to represent the users in closed models. For this reason it is commonly selected by defaultas Reference station of a model.

#### **Reference station** for each workload class (at system level)

The station used to compute the performance indexes at the system level (throughput, response time, global utilization of resources, etc.) is referred to as Reference station (RS). When a job flows through an RS, in most cases it is implicitly assumed that its execution has completed and therefore it is leaving the system. In this case, a job visits the RS only once during its life and for this reason the delay station that in most models simulates the users is often selected as RS. However, any of the components of the model may be selected as RS. Clearly, all the performance indexes are affected by this choice. To compute their values, the visits to each component of the model must be scaled with respect to those made to RS. In open models JSIMg assumes by default a Source station as RS. When a job completes its execution, its performance indexes are computed considering the time interval elapsed between its generation from the Source and its exit from the Sink. In closed models any station can be selected as RS. In this case its performance indexes are calculated considering the time elapsed between its generation from RS and enters the model and the time it exits the model and reaches the RS (see Fig. 1.3).

#### **Output Measures**

**X0**–**System Throughput** (at system level)–*X*<sup>0</sup> = *N*0/(*R*<sup>0</sup> + *Z*)*closed model* λ<sup>0</sup> = *N*0/*R*<sup>0</sup> *open model* Little Law

This metric represents the rate at which the jobs complete their executions and leave the system. To compute this metric it is fundamental to know which is the Reference station of the model since only the jobs that visit it have completed their executions. Without loss of generality, the visit to the external part of the model is often assumed to be one. *X*<sup>0</sup> may be computed applying Little law to the system as a whole (see Fig. 1.3).

**N0**–**Number of jobs** (at system level)–*N*<sup>0</sup> = *X*0(*R*<sup>0</sup> + *Z*) *closed model N*<sup>0</sup> = λ0*R*<sup>0</sup> *open model*

In open models the number of jobs in the system is an output metric since it depends on the arrival rate and on the contention of the components. When the utilization of a component is close to one it saturates and the value of *N*<sup>0</sup> grows to infinity.

#### **R0**–**System Response Time** (at system level)–*R*<sup>0</sup> = (*N*0/*X*0) − *Z closed model R*<sup>0</sup> = *N*0/λ<sup>0</sup> *open model*

The amount of time required by a complete execution of a job is referred to as System Response Time. In closed models, it can be seen as the time interval between two consecutive visits to the Reference station by the same job (the first corresponds to the instant of time in which the job is generated and the second to the instant of its completion). In open models, *R*<sup>0</sup> corresponds to the time interval between the generation of a job by the Source station and the moment in which it completes its execution and reaches the Sink station. Since the models are in equilibrium and each job entering the system visits the Sink usually once, the Source can be considered as Reference station.

**Xr**–**Throughput of component** *r*–*Xr* = *X*0*Vr* Forced Flow law Number of requests processed in a time unit by the component*r*. Note that the unit of measure is requests per unit of time and not jobs per unit of time (used by System Throughput). When the Service demands *Dr* instead of the Visits *Vr* and Service times *Sr* are used as input parameters, the *Xr* cannot be obtained from the model. In this case, the throughput of each component is equal to *X*0. The relationship between *X*<sup>0</sup> and *Xr* are obtained from the Forced Flow law.

#### **Nr**–**Number of requests in component** *r*

In a queue component, this metric refers to all the requests in the station, whether waiting in queue or in execution. In a delay component all the requests in the station are in execution, thus *Nr* corresponds to the mean number of requests in service.

**Ur**–**Utilization of component** *r*–*Ur* = *Xr Sr* = *X*0*Vr Sr* = *X*0*Dr* Utilization law Fraction of time the server of a queue component *r* is busy (in a station with one server). In a delay component this value corresponds to the mean number of requests in service.

**Qr**–**Queue time of component** *r* (per request)–*Qr* = *Rr* − *Sr*

Mean time spent in queue waiting for the server in a queuecomponent.

#### **Rr**–**Response time of component** *r* (per request)–*Rr* = *Nr* /*Xr*

Mean time required to execute the processing request of *one* visit to component *r*. Its value includes all time spent in the component during a visit, whether waiting in queue or being served. The unit of measure is the time per request. If the number of servers of the queuecomponent is one, then it will be: *Rr* = *Qr* + *Sr*.

#### **Rdr**–**Residence time of component** *r* (per job)–*Rdr* = *Vr Rr*

The total time spent by a job at component*r* during its complete execution (including both the time spent in queue and the time being served) is referred to as Residence time *Rdr*. While the Response time *Rr* is *local* to a component (i.e., it may be computed considering *only* the Response times of one visit to the resource), to compute the Residence time *Rdr* of a resource it is necessary to know the Number of visits *Vr* that a job makes to the resource during its complete execution (see Appendix A.1). The unit of measure isjob. The System Response Time is the sum of the Residence times of all the components of the model.

#### **1.3 Parameterization of Simulation Models**

The sequence of operations required to implement a model is clearly influenced by the techniques and tools used. In this section we restrict our attention to the simulation which, compared to other techniques, allows for maximum generality in terms of system architectures that can be modeled and adoptable assumptions.

In the following description of the steps required to implement a simulation model (see Fig. 1.4) we tried to be as general as possible. However, since the models were solved with the JSIMg simulator, its characteristics clearly influenced the sequence of operations performed. The figures mentioned in the flowchart of Fig. 1.4 show some user-interface windows for setting the parameters required by the steps represented.

Implementing a model begins with describing its components and their interconnections. Depending on the types of user interface available, a graphical (see, for example, the model of Fig. 1.5 created with JSIMg), or other type (e.g., wizard) of description can be done.

The parameters for the workload characterization are: type of customers classes (open or closed), arrival rate and distribution of interarrival times (in some cases also other statistical parameters are required) for open classes, number of customers for closed classes, and Reference station. In Fig. 1.6 the following parameters for Class1 are set: open class, arrival rate λ = 1 req/s, exponential distribution of interarrival times, and the station Source1 as Reference station. To select different arrival rates or distributions, simply click on the Edit button and the list of available distributions will be shown. With multiclass workloads the parameters of each class must be provided.

The next step is setting the station parameters for all workload classes. In JSIMg, the parameters for a Queue station are organized into three sections: Queue, Service, and Routing. In the Queue Section, the Capacity size (max number of customers allowed in the station, in queue and in service), the type of scheduling algorithm, the queue policy, and the Drop Rule (in stations with limited capacity) are set. In the Service Section, the Number of Servers of the station, the type of service Strategy whether load independent or load dependent, and the statistical parameters of the Service Times Distribution are set. In the Routing Section the routing strategies of jobs on the interconnections automatically detected among the stations may be described. For example, in Fig. 1.7, for station Queue1 the Probability has been set as Routing Strategy, and the customers in output are sent to Queue2 with probability 0.3 and to Queue3 with probability 0.7.

The next step concerns the selection of the metrics (performance indexes) that must be computed with the model. Usually, for each metric, several statistical variables must be set. In JSIMg (see Fig. 1.8) the following parameters are required: the class of customers and the station (or the entire system) considered, the confidence level (see Appendix A.2 and, e.g., [36, 37]), the maximum relative error, and the decision whether to generate the file with all the collected values of the metric analyzed or not. In Fig. 1.8 five indexes concerning Class1 customers are selected: two aggregated at system level (System Response Time and System Number of Customers), and three at the Queue1 station level: (Response time, Number of customers, and Utilization). For the Response time, the generation of the CSV file with all the values of the samples analyzed is required (i.e., the Stat.Rescheckbox is flagged). The 99% confidence level(default value) is required for all the indexes and the max relative error tolerated is

**Fig. 1.4** Main steps to implement a simulation model with JSIMg. Figure numbers in the flowchart refer to examples of the corresponding screenshots

**Fig. 1.5** Example of the graphical representation of a model using JSIMg

**Fig. 1.6** Definition of the parameters for the open class Class1

0.03. The simulator no longer collects data of an index when the required accuracy is achieved.

When the simulation starts, for each selected index a graph like the one of Fig. 1.9 is plotted. As the simulation progress, the behavior of the confidence intervals and of the mean value of each index are shown together with the number of samples analyzed. According to the request (see the Stat.Res. checkbox flagged in Fig. 1.8), the CSV file was generated with all the values of Response times and the statistical indexes were computed. The CSV file will contain, among the other variables, the values of the percentiles(see the example of Figs. 2.10, 2.11).

**Fig. 1.7** Parameters of the Routing Section of Queue1 station

**Fig. 1.8** Performance indexes to be collected, their precision and statistical requirements

Most of the performance studies require the evaluation of the impact on system performance of one or more parameters. To meet this objective it is necessary to execute a sequence of models increasing (or decreasing) at each step the value of a parameter, e.g., Arrival rate or Number of customers, referred to as Control parameter. To make this process efficient, many simulators show a feature called What-if. For example, Fig. 1.10 shows the Response times obtained from the execution of 10 models with Class1 customer Arrival rate increasing from 0.2 to 1.2 job/s. Mean values and confidence intervals are also reported.

**Fig. 1.9** Response times of Queue1 station: mean value and confidence intervals computed on different samples collected during the simulation progress

**Fig. 1.10** Response times of Queue1 station as a function of the Arrival rates

#### **1.4 Parameterization of Analytical Models**

In this section we will outline the steps required to implement a model that will be solved with a analytical technique. As can be seen from Fig. 1.11, these steps are not very different from the ones already described for the simulation models in Fig. 1.4. Clearly, the analytical technique and the tool adopted introduce some peculiarities on the operations that can't be found in simulation.

As a function of the solution algorithm adopted, the analytical techniques can be subdivided in exact, approximate, and asymptotic. Each technique has its own parameters. In the following, we will refer to models solved with Mean Value Analysis (MVA) technique [25, 31] using the JMVA tool. The MVA algorithm compute the exact values of performance indexes, but has several limitations in terms of system characteristics that can be modeled.

In the screenshot of Fig. 1.12 the MVA has been selected as solution algorithm for the closed class Class1 of 10 customers. The tabs are reported with the sequence that must be followed for the parameter settings: Classes, Stations, Service times, Visits, Reference station, What-if. In Fig. 1.13 the mean values of the Service times of the three stations CPU, Storage1, and Storage2 are set. As requested by the MVA algorithm, the values of these parameters are considered exponentially distributed. The visits to the three stations are *VCPU* = 10000, *VStorage*<sup>1</sup> = 5499, *VStorage*<sup>2</sup> = 4500.

In simulation models, to minimize the overhead introduced by the collection of the data, users should select only those performance indexes interested in the study. In analytical models, however, a consistent set of indexes is always computed as their derivation is very fast (see, e.g., Fig. 1.14). In Figs. 1.15 and 1.16 the Utilizations and the Residence times of the three stations are plotted for the Number of customers ranging from 10 to 100 (90 models were executed with a What-if analysis). The values of the performance indexes are also provided in tabular form.

**Fig. 1.11** Main steps for the implementation of a JMVA model solved with analytical technique. Figure numbers in the flowchart refer to examples of the corresponding screenshots

**Fig. 1.12** Selection of the MVA solution algorithm and settings of one closed class Class1 with 10 customers

**Fig. 1.13** Settings of the Service times of the three stations

**Fig. 1.14** Throughput of the three stations. All the performance indexes are computed

**Fig. 1.15** Utilizations of the three stations as a function of the Number of customers

**Fig. 1.16** Residence times of the three station as a function of the Number of customers

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 2 Systems with Homogeneous Workloads**

#### **2.1 A Web Server with External Workload**

**tags**: open, single class, Source/Queue/Sink, JMVA.

The models for analyzing the performance of a system can be developed at different levels of detail and with a single element that can represent the system as a whole or just one of its components. In spite of their high level of aggregation, models in which the resources of a system are collectively represented with a single component (i.e., the system is modeled as a black box) yields in many cases interesting results. These models can also provide useful insights for the evaluation of more complex scenarios.

To solve the model presented in this section we use the analytical tool JMVA that applies the classical Queuing Networks equations.

#### *2.1.1 Problem Description*

A capacity planning study is required to model a web server utilized for the distribution of technical documentation concerning the products of a company and accessible by a high number of users through Internet. Requests arrive at the server from the network, compete for the resources, and once executed leave the system, see Fig. 2.1a. These models are usually referred to as open models. The workload consists of a single request class. The requests have similar service demands, are independent each other and arrive to the server with exponentially distributed interarrival times. We consider a simple high-level aggregated model, i.e., a single queue station, representing the web server accessed by a request only once before leaving the station, see Fig. 2.1b. This single-station model may seem inadequate to describe a web server that has at least two resources, a CPU and a storage, that are visited many times by the requests during their execution.

**Fig. 2.1** Web server model at high level of abstraction

However, there are several problems in which single-station models yield interesting results. Among the motivations that make them useful in many situations are:

The assumption of single class workload is important in many situations for the accuracy of the models. When the workload components have significant differences in resource requirements, i.e., when there are multiple class requests, the bottleneck may migrate among resources as a function of the fluctuations of the mix of requests in execution (see, e.g., [2, 3]). The effect of this migration may be dramatic for the accuracy of the results. With single class workloads the bottleneck does not switch among resources provided that all of them are load independent, i.e., their service time is not a function of the number of requests that are in the resource (waiting in queue and in service).

Concerning the single visit hypothesis, this should not be a concern. Depending on the abstraction level of the model, it may not be necessary to explicitly describe the load of each component*r* of a system at the lower levels of detail using the Service times *Sr* and the Visits *Vr* but it is sufficient to consider the global*Service* *demand Dr* placed on each of them during a complete execution (i.e., *Dr* = *Vr Sr*). To reduce the number of parameters and the effort required by their measurement, we will parameterize most of the models with the Service demands *Dr*.

#### *2.1.2 Model Implementation*

We consider a simple open model at high level of abstraction, i.e., system-level, (see Fig. 2.1b) consisting of a single queue station Queue 1, representing the web server that is accessed by the requests generated by the Source. Once executed, the requests leave the queue station for the Sink. In these open models the number of requests in execution is not controlled by the system itself but depends on the characteristics of the traffic generated by the Source (rate and fluctuations of arrivals, service requirements). Depending on these parameters, a system can be flooded with requests whose number can suddenly grow to very high values.

Requests are assumed to be independent of each other and arrive at the server at random times. This is equivalent to saying that Interarrival times are *exponentially* distributed. All the requests are considered statistically equal, i.e., are indistinguishable each other, and leave the server at random times. The randomness of the departure times has as the consequence that the Service times S, i.e., the time requirement per visit, are *exponentially* distributed.

All requests arriving at the station can be accepted for execution, i.e., there is enough space to store them all that can grow indefinitely. This type of station is usually referred to as M/M/1 station (see, e.g., [36, 37]). The arrival rate is λ = 0.2 req/s, thus the average Interarrival time is 1/λ. The average time required by a complete execution of a request is 1 s. This time usually is referred to as Service demand D of a request, but since the number of visits to the server in the aggregated model is one, its value coincides with the Service time S. Thus, for simplicity, we will use the notation S instead of D in this example. The requests are served according to their order of arrival, i.e., with a FCFS scheduling.

With the hypotheses considered, this model can be solved analytically with the classical Queueing Networksequations implemented in the JMVA.

#### *2.1.3 Results*

In what follows we will describe the operations required to achieve some of the **objectives** (referred to as Obj.1–Obj.4) of the capacity planning study.

#### **Obj.1: implement a model of the server and compute the performance indexes with the parameters above described**

In Fig. 2.2 the input parameters for the model solved with QN (Queueing Networks) equations are shown. Some of the performance indexes computed by the model are

**Fig. 2.2** Input parameters of the JMVA for the open model of Fig. 2.1b

shown in Fig. 2.3. The mean number of requests N in the server is 0.25 req and the mean Response time is 1.25 s. To check the correctness of the results we computed the values of the same indexes with the exact equations of Queueing Networks:

$$N = \frac{U}{1 - U} = \frac{\lambda \text{ } S}{1 - \lambda \text{ } S} = 0.25 \text{ } req. \qquad R = \frac{S}{1 - U} = \frac{S}{1 - \lambda \text{ } S} = 1.25 \text{ s} \tag{2.1}$$

#### **Obj.2: compute the behavior of the performance indexes when the workload increases to** *λ* = **0.9 req/s**.

A What-if analysis is required with Arrival rate as Control Parameter ranging from 0.2 to 0.9 req/s. In Fig. 2.4 the parameterization of the What-if (100 models are requested) and the behavior of two performance indexes, i.e., the Throughput X and the Response time R, are shown. Since in the model there is only the Queue1 station, its Throughput and Response time coincide with the ones of the System. The linear behavior of the Throughput X is correct since we increase linearly the Arrival rate λ from 0.2 to 0.9 req/s and the models are in equilibrium, i.e., it is λ = *X*. The maximum Throughput of the server corresponds to the saturation loadλ*sat* = 1/*S* = 1 req/s.

**Fig. 2.3** Some of the performance indexes computed by the JMVA

**Fig. 2.4** System Throughput and System Response Time of 100 models with Arrival rates λ ranging from 0.2 and 0.9 req/s obtained with a What-if analysis

**Fig. 2.5** What-if analysis: tabular results of the System Response Times corresponding to the Arrival rates λ = 0.793 and 0.801 req/s

#### **Obj.3: according to the marketing department forecast, a maximum Response time R = 5 s can be tolerated. Compute the maximum increase of the workload that satisfy this constraint** (% **with respect to the original** *λ* = **0.2 req/s**).

We can use the results provided in tabular format from the What-if analysis made in the previous step. Figure 2.5 shows the Response times of models 85 and 86 that are just above and just below the value R=5 s (4.85 and 5.02, respectively). The arrival rates used in the two models are 0.793 and 0.801 respectively. So we are sure that λ = 0.793 req/s satisfy the constraint. Just as a simple check, we may use Eq. 2.1 to derive the value of λ corresponding to R = 5 s and S = 1 s. We obtain λ = 0.8 req/s. Thus, the increment of the workload toleratedis 300%.

**Obj.4: a new set of complex technical manuals are expected in the near future whose Service demand is assumed to be S = 2 s**. **What will be the** System Response Time **R with an expected arrival rate double the actual one (i.e.,** *λ* = **0.4 req/s**)?

New values for the input parameters S = 2 s and λ = 0.4 req/s must be set. The corresponding value of System Response Time is R = 10 s.

#### **2.2 A Computing Infrastructure with a Closed Workload**

**tags**: closed, single class, Delay/Queue, JSIMg.

In this section we describe a model of a computing infrastructure with a closed workload (see Sect. 1.2) solved with the simulation technique. The main characteristic of this type of workload is that the number of customers in execution is constant. A new customer enter the system when a customer complete its execution.

On the basis of the assumptions made, this model could also be solved analytically with JMVA. However, we have used the simulation technique to provide a first simple example of implementing a model with a simulator. Furthermore, it should be noted that simulation is by far the most popular modeling technique used in performance engineering. Indeed, simulators are very powerful tools and the set of models they can implement is practically unlimited given the great generality offered in terms of characteristic of the systems and type of assumptions that can be represented.

#### *2.2.1 Problem Description*

A computing infrastructure, located in a large data center, is used to execute applications that are very critical to the company's business. This infrastructure adopts very high security techniques to control accesses that are reserved only to a limited number of authorized employees. It mainly consists of three servers: a Web Server (WS) and two servers (*AS*<sup>1</sup> and *AS*2) dedicated to the Application and Storage functions, see Fig. 2.6a.

Due to the apps executed, the resource requirements of the user requests are similar, i.e., the workload issingle-class. The Service times of the three servers have different mean values, and are assumed exponentially distributed. The probabilities (i.e., the routing probabilities) that the requests in output from the Web Server are routed to servers *AS*<sup>1</sup> and *AS*<sup>2</sup> are known. In some problems, instead of the routing probabilities, the visitsthat a request perform to each resource during its execution

**Fig. 2.6** The computing infrastructure considered (**a**) and the corresponding queueing network (**b**)

are known. These two sets of values are related each other and to derive one set from the other it is required to know the topology of the network. In Appendix A.1 it is described how to obtain the relationships between the routing probabilities and the visits for the topology considered in Fig. 2.6b. Assuming that when a request leaves the model it has been completely executed, i.e., that it is *V*<sup>0</sup> = 1, we have:

$$V\_{WS} = \frac{1}{p\_0} = 10 \qquad V\_{AS\_1} = \frac{p\_1}{p\_0} = 6 \qquad V\_{AS\_2} = \frac{p\_2}{p\_0} = 3 \tag{2.2}$$

Models can be parameterized with one set of values or the other. JSIMg accept both types of parameters. The scheduling algorithm adopted by the resources is FCFS.

#### *2.2.2 Model Implementation*

Since the number of users (i.e., the employees authorized to access the computing infrastructure) is constant, we implement a closed model with four stations: one delay and three queue, see Fig. 2.6b. Each user submit one request. The probabilities *pi*'s that after a visit to the Web server *W S* a request is routed to App&Storage servers *ASi* are known. The index 0 is used to represent the world outside the system, and the metrics with index 0 are at system-level. Therefore, *X*<sup>0</sup> and *R*<sup>0</sup> represent the Throughput and the Response time of the global system, and *p*<sup>0</sup> is the probability that a request leaves the system as it has completed its execution. We assume that a request is routed to this path only once in his lifetime, so the number of visits *V*<sup>0</sup> that it performs outside the system is one. According to the layout of the model it is -2 *<sup>i</sup>*=<sup>0</sup> *pi* = 1.

The workload is generated by a station external to the system representing the Users, that we consider as Reference station. This station is used to compute the System Response Time *R*<sup>0</sup> and the System Throughput *X*<sup>0</sup> . *R*<sup>0</sup> is defined as the period of time between the instant in which a request enters the model (leaving the Reference station) and the one in which it leaves the model (entering the Reference station). *X*<sup>0</sup> is the rate of completed requests that leave the model and enter the Reference station. Others performance indexes are also influenced by the selection of the station that will be considered as reference (see Appendix A.1). The mean Service time for each Visit to servers WS, AS1 and AS2 are: *SW S* = 0.005 s, *SAS*<sup>1</sup> = 0.020 s, and *SAS*<sup>2</sup> = 0.025 s, respectively. The think time of the delay station Users is Z=1s. All the values are exponentially distributed. The JSIMg model of Fig. 2.7 was solved with simulation. The routing probabilities of the requests leaving the Web Server are: *p*<sup>0</sup> = 0.1, *p*<sup>1</sup> = 0.6, and *p*<sup>2</sup> = 0.3, see Fig. 2.8.

**Fig. 2.7** The JSIMg model of the computing infrastrucure of Fig. 2.6b

**Fig. 2.8** Settings of the Routing Probabilities of the Web Server WS

#### *2.2.3 Results*

Several objectives of the capacity planning study were set. In what follows we will describe the results of some of them referred to as Obj.1–Obj.4.

**Obj.1: Implement the model of the computing infrastructure with the parameters assigned. Investigate the behavior of System Throughput X0 and System Response Time R0 for the Number of Customers** *N***<sup>0</sup> ranging from 1 to 20. Which will be the 90th percentile of R0 with** *N***<sup>0</sup>** = **20?**

**Fig. 2.9** System Throughput and System Response Time versus Number of customers

A What-if analysis is performed by setting the Number of customers *N*<sup>0</sup> = 1 ÷ 20 as control parameter. Figure 2.9 show the behavior of System Throughput *X*<sup>0</sup> and System Response Time *R*0, respectively, with respect to *N*0. Please note that the *R*<sup>0</sup> values computed by JSIMg include the time spent in the Reference station, i.e., the Users station, that is *Z* = 1 s. For *N*<sup>0</sup> = 20 we have *X*<sup>0</sup> = 8.32 req/s and *R*<sup>0</sup> = 2.4 s. As *N*<sup>0</sup> increases from 1 to 20, *X*<sup>0</sup> becomes flat and tends to its horizontal upper bound, while *R*<sup>0</sup> becomes linear and tends to its lower bound which is a oblique line. These behaviors are typical of closed systems when a resource is approaching saturation. In the following Objs. 2, 3 we will analyze this condition in detail.

The values of some percentiles of the System Response Times, for example the 90th or the 95th, are often requested in performance studies. Let us recall that the 90th percentile <sup>90</sup> of a variable Y is the value below which can be found 90% of all the values assumed by Y, i.e., it is *P*(*Y* ≤ 90) = 0.9. To obtain the percentile values in JSIMg it is necessary to flag the check box Stat.Res. (see, e.g., Fig. 1.8) in the window of the metrics to be collected. A CSV file with all collected values of the selected metric is then generated and stored. Various statistical indexes are computed by clicking on the Statistical Results button (see Fig. 1.9) in the window of the analyzed metric. Selecting Distribution as a drawing option, the values are sorted in increasing order and are grouped in intervals. For example, 300 intervals have been selected in Fig. 2.10. The percentiles corresponding to each interval are calculated and stored in a CSV file. A sample of this file for the intervals 70 ÷ 76 with the corresponding percentiles (from 88.9 to 91.3) is shown in Fig. 2.11. The 90.1 percentile corresponds to *R*<sup>0</sup> = 4.88 s. It should be noted that if the values of a variable Y are exponentially distributed it is <sup>90</sup> 2.3 x (*mean* v*alue* of *Y* ). In our case, the values of *R*<sup>0</sup> are hypo-exponentially distributed (the coefficient of variation is 0.76<1, see Fig. 2.10). Their variance is less than that of an exponentially distributed variable with the same mean. Thus, it seems correct to obtain the value of

#### 2.2 A Computing Infrastructure with a Closed Workload 31

**Fig. 2.10** Statistical indexes of the System Response Times

**Fig. 2.11** Sample of the CSV file with the values of *R*<sup>0</sup> sorted in increasing order and subdivided into 300 intervals. The four columns refer respectively to: the id of the intervals, the extremes of each interval, and the percentilecorresponding to the extreme with maximum value

4.88 s for the 90th percentile of *R*<sup>0</sup> which is less than 5.52 s (2.3 x 2.4), as it would be if they were exponentially distributed. By increasing the number of intervals, more detailed percentiles can be obtained.

**Obj.2: To improve the computing infrastructure performance, one of the first actions that seems natural is to replace** *AS***2**, **the slowest of the App&Storage servers, with a new model that is 20**% **faster (that is, the same as AS1). Evaluate the effects on X0 and R0**.

The mean Service time of server *AS*<sup>2</sup> of the original model must be modified decreasing its value from 0.025 to 0.020 s. The model with the What-if for *N*<sup>0</sup> = 1 ÷ 20 usersis executed again.

As expected, the Utilization of *AS*<sup>2</sup> decreased, e.g., from 61.8% to 50% with *N*<sup>0</sup> = 20. However, surprisingly NO improvementsare obtained on *X*<sup>0</sup> and *R*0.

versus

**Fig. 2.12** Utilizations of the three servers AS1, AS2, and WS N0

Indeed, with the new fast server *AS*<sup>2</sup> we have *X*0(20) = 8.27 req/s and *R*0(20) = 2.42 s while with the slow one we had *X*<sup>0</sup> = 8.32 req/s and *R*<sup>0</sup> = 2.40 s, respectively. The two values of *X*<sup>0</sup> can be considered equally likely estimates of the exact throughput value since they are both in the same 99% confidence interval. The same observation applies to *R*<sup>0</sup> (see Appendix A.2).

Analyzing the Utilizations of the three servers, in Fig. 2.12a with the original configuration and in Fig. 2.12b with the new *AS*2, we have an answer to this unexpected result. From Fig. 2.12a it is possible to see that the utilizations of *AS*<sup>1</sup> and *AS*<sup>2</sup> are unbalanced, and that *AS*<sup>1</sup> is the bottleneck of the computing infrastructure despite being the faster of the two. Indeed, its utilization is the highest of all servers and for heavy load it is close to saturation (e.g., with *N*<sup>0</sup> > 15 it is *UAS*<sup>1</sup> > 0.95).

This is the main motivation of the uselessnessof the action we have done:

improving any station but the bottleneck do not generate any performance gain with heavy workload. It is known that performance improvements can only be achieved by reducing its contention. Actions that reduce the load of stations other than the bottleneck produce minimal improvements (if any) only under very light workload (see Obj.3).

**Obj.3: Given the insignificant results obtained in Obj.2, we want to evaluate the performance improvements that can be achieved by replacing the AS1 server with a new model 20**% **faster (the same increase considered in Obj.2 for AS2**). We recompute the original model (Fig. 2.7) settings the mean Service Time of server *AS*<sup>1</sup> to a value 20% faster (from 0.020 s to 0.016 s). We then execute again the What-if for *N*<sup>0</sup> = 1 ÷ 20 users obtaining the values of *X*<sup>0</sup> and *R*<sup>0</sup> reported in Fig. 2.13. For *N*<sup>0</sup> = 20, with respect to the original system, *X*<sup>0</sup> increases of 20%, from

**Fig. 2.13** *X*<sup>0</sup> and *R*<sup>0</sup> with the new server *AS*<sup>1</sup> 20% faster

8.32 to 9.99 req/s, and *R*<sup>0</sup> drops of 17% from 2.4 to 1.99 s. The bottleneck remain the server *AS*1, its utilization is 0.95 (in the original model was 0.99).

Let us remark that these positive results were obtained because we improved the station that is the bottleneck of the system, i.e., the server *AS*1. Indeed, as seen in the previous Obj.2, improving other stations do not produce any significant results on performance.

**Obj.4: According to the management, the number of internal employees authorized to access the computing infrastructure may increase to 40 in a semester. Which will be R0 and X0 with the actual configuration with N0** = **40 users?** We recompute the original model (Fig. 2.7) settings the Number of Customers in the closed class definition window to 40. The behavior of the mean value of *R*<sup>0</sup> and of the confidence intervals during the simulation are shown in Fig. 2.14. As can be seen, the mean value of *R*<sup>0</sup> is 4.821 s obtained from the model is very close to the lower bound 4.8 s given by *N*0*Dmax* = *N*0*VAS*<sup>1</sup> *SAS*<sup>1</sup> = 40 x 0.12. The *X*0(40) is 8.325 req/s, very close to its upper bound 1/*DAS*<sup>1</sup> = 1/0.12 = 8.333 req/s (see Sect. 2.3).

#### **2.3 Equivalent Model with Service Demands**

**tags**: closed, single class, Delay/Queue, JSIMg

In this section we describe a model, solved with JSIMg, parameterized with Service demands. This model is equivalent to the one solved in the previous section using the Visits and Service times of the stations. The granularity at the system-level is here adopted compared to that at the station-level adopted in the previous model.

**Fig. 2.14** System Response Time of the computing infrastructure with the initial configuration and *N*<sup>0</sup> = 40 users

#### *2.3.1 Problem Description*

The models can be designed at different levels of granularity (see Chap. 1), from the single component (station) to the entire system. In Sect. 2.2 we implemented a station-level model using the Routing probabilities *pi j* , i.e., the probability that a request in output from station *i* is routed to station *j*. We have also described how to obtain from the *pi js* the number of Visits *Vr* that a request makes to each station during its complete execution (see Appendix A.1). In the models parameterized at this level of detail we may compute all the performance indexes describing the behavior of each station, including its Throughput and Response time. However, measuring the *pi j* , or the *Vr*, is difficult or in some cases impossible. A parameter that is often used is the Service demand *Dr* of a request to station *r*, which represents the total amount of Service time that a request requires from station *r* to complete its execution. The *Dr* values may be obtained by multiplying the Service time *Sr* required by one visit to the number of Visits *Vr* that a request makes to station *r* during its execution, i.e., *Dr* = *Sr Vr*. There are many motivations that make the models parameterized with Service demands so popular. Among them are:

• the limited effort required to obtain the mean values of Service demands *Dr* from measurements. Indeed, the system log file usually shows the *Dr* values for every request executed. Recall that several executions are needed to have a reliable estimate of the mean values of *Dr* and the confidence intervals of the measured values must be computed, (see Appendix A.2 and, e.g., [36, 37]). The *Drs* can also be obtained dividing the busy time *Br* of a resource by the number *C*<sup>0</sup> of user requests executed;

Based on the described advantages, when possible, the models are preferably parameterized in terms of Service demands *Dr* instead of Visits and Service times. Clearly, with this high level of granularity we lose the structural similarity with the considered system, but the models are easier to implement, the solution algorithms are faster, and the performance indexes that can be computed (not all, but almost) are correct.

#### *2.3.2 Model Implementation*

To illustrate the practical applicability of the Service demands we consider again the closed model solved in Sect.2.2 using the Routing probabilities *pi j* and the Service times *Sr*, see Fig. 2.6b. In this Section we implement a new version of it using Service demands. From the *pi j*s it is possible to derive the Visits to the three servers *VW S* = 10, *VAS*<sup>1</sup> = 6, *VAS*<sup>2</sup> = 3 (see Appendix A.1) and knowing the Service times *SW S* = 0.005 s, *SAS*<sup>1</sup> = 0.020 s and *SAS*<sup>2</sup> = 0.025 s, we can compute the Service demands (*Dr* = *Vr Sr*) *DW S* = 0.050 s, *DAS*<sup>1</sup> = 0.120 s, *DAS*<sup>2</sup> = 0.075 s.

The implemented model of Fig. 2.15 consists of three servers, having as Service times the *Dr*, which are visited only once during the execution of a user request. The structure of this new model is clearly simpler than that of Fig. 2.7.

**Fig. 2.15** Model parameterized with Service demands equivalent to that of Fig. 2.7 which uses Visits and Service times

#### *2.3.3 Results*

We solve the model of Fig. 2.15 using the What-if analysis with *N*<sup>0</sup> = 1 ÷ 20 customers as control parameter. The performance measures obtained are practically the same, i.e., they lie in the same 99% confidence interval of the corresponding indexes obtained with the model of Fig. 2.7. Table 2.1 compares the values of some performance indexes computed by the two models with *N*<sup>0</sup> = 20 customers.

It should be noted that with the high level parameterization, and the consequent simplified layout of the equivalent model, it is NOT possible to compute the Throughput and the Response time of each station. Indeed, at the system level, we do not model the Visits, thus only the parameters at high level of aggregation can be computed for each station, i.e., the global Utilization and the Residence time. The values inside the boxes in Table 2.1 emphasize that in the equivalent model the three servers have the same Throughput (measured in user requests per time unit), that coincide with the System Throughput *X*0. This is because the three servers in this model are visited only once, requiring the Service demand *Dr* to each of them, are connected in series. Thus, we may compute only the Residence times *Rdr* of the servers and not their Response times *Rr* (since it is *Rdr* = *Vr Rr*).

**Table 2.1** Performance metrics for *N*<sup>0</sup> = 20 customers obtained from the two equivalent models parameterized at different levels of granularity (Visits and Service times and Service demands, respectively)

The operational analysis laws can be applied also to models parameterized with Service demands. For example, using the Forced Flow law *Xr* = *X*<sup>0</sup> *Vr*, the Utilization lawbecomes

$$\mathbf{U}\_r = X\_r \ \mathbf{S}\_r = X\_0 \ \mathbf{V}\_r \ \mathbf{S}\_r = X\_0 \ \mathbf{D}\_r \tag{2.3}$$

From Table 2.1 it is possible to see that Eq. 2.3 is verified with the metrics obtained from the model parameterized with the *Dr*. For example, for the server WS it is *UW S* = 8.31 x 0.05 = 0.415 that coincides with the measured value of *UW S*. Note that the results of the two models may not coincide exactly as we are in simulation and we know only the confidence intervals of the computed variables. We may verify also that, according to Eq. 2.3, the ratio of the *Ur* coincides with the ratio of the *Dr*:

$$\frac{U\_{WS}}{U\_{AS1}} = 0.418 \cong \frac{D\_{WS}}{D\_{AS1}} = 0.416 \qquad \qquad \frac{U\_{AS1}}{U\_{AS2}} = 1.594 \simeq \frac{D\_{AS1}}{D\_{AS2}} = 1.6 \quad (2.4)$$

The Little lawapplied to resource *r* using the Residence times becomes:

$$N\_r = X\_r \ R\_r = X\_0 \ V\_r \ R\_r = X\_0 \ R d\_r \tag{2.5}$$

It should be recalled that the System Response Time *R*<sup>0</sup> provided by JSIMg comprises the time spent by a user request in the Reference station, that in our model is the Users with *Z* = 1 s. So, it is *R*<sup>0</sup> = *RdW S* + *RdAS*<sup>1</sup> + *RdAS*<sup>2</sup> + *Z* = 2.409 s. Applying Little law at the system-level we have: *N*<sup>0</sup> = *X*<sup>0</sup> *R*<sup>0</sup> = 8.31 x 2.409 = 20 *customer s*, as expected.

From the analysis of the *Dr* we can derive that the server AS1 is the most utilized of the resources since *DAS*<sup>1</sup> = 0.120 s is the largest of the Service demands. As *N*<sup>0</sup> increases it will be the first resource to saturate, i.e., it becomes the bottleneck of the system, limiting the System Throughput to *X*<sup>0</sup> ≤ 1/*Dmax* = 8.333 req/s. With *N*<sup>0</sup> = 20 *customer s* we obtained *X*<sup>0</sup> = 8.31 req/s (see Table 2.1) since AS1 is not completely saturated (it is *UAS*<sup>1</sup> = 0.992).

#### **2.4 Optimal Operating Point of a Server**

**tags**: open, single class, Source/Queue/Sink, JSIMg.

We describe how to identify the optimal operating condition of a system that is characterized by the highest Throughput with the shortest Response time. A system in this condition, referred to as optimal operating point, operates with maximum of efficiency, that is, it maximizes its productivity by introducing the minimum delay. We consider a simple model of a system that is solved using both analytical techniques (Queueing Networks) and simulation techniques (JSIMg model).

#### *2.4.1 Problem Description*

Identifying the optimal operating point of a digital infrastructure is a problem that IT managers face in their daily lives. Nowadays, this task is becoming more and more important as the size of data centers and clouds continues to grow in the number of servers, applications and users. The basic idea is that a small profit on a single server can translate into a large profit when evaluated on hundreds of servers with thousands of users.

Clearly, depending on the context considered, the notion of optimal operating point assumes various definitions that translate into different actions. For example, it can refer to the operating conditions that minimize the energy required to execute a workload while meeting the performance goals, or to the operating point that satisfies the SLA (Service Level Agreement) by minimizing the number of allocated servers.

To simplify the description, let's consider a single server that we assume is operating under the optimal conditions when its Throughput X is maximized and its Response timeR is minimized. The load corresponding to this optimal condition will be referred to as optimal load.

In this simple case, when the goal of the performance study is to identify the load that maximizes the Throughput X or the one that minimizes the Response time R, the answers are easily provided. In fact, a resource generates the maximum X when it is saturated and provides the minimum R when only one request is in execution, that is, there is no contention. However, when X and R are to be compared at the same time, a new metric must be used that considers the trade-off between the two. To this end, below we will consider the System power , a metric introduced in [19] and extensively studied by Kleinrock [23, 24], defined as

$$
\Phi = \frac{X}{R} \tag{2.6}
$$

The behavior of may be considered in some way related to that of the *Quality of Service*. Indeed, an increase in Throughput or a decrease in Response time increases the System power, that may be considered in the *SLA* as indicator of the Quality of Servicedelivered to the users.

In the next section we consider a system consisting of a single server modeled with a Queue station (see, e.g., Fig. 2.1) that execute a homogeneous workload (single class) with Interarrival times and Service times exponentially distributed.

For this simple case, we describe the analytical computation of the optimal load, and then we implement the correspondent model with JSIMg. The analytical derivation of for more complex systems is not easy to obtain. However, it should be emphasized that the JSIMg provides the System Power behavior for all simulated models, regardless of their complexity. is one of the metrics available in the tool.

#### *2.4.2 Model Implementation*

In this section we address the problem of the identification of the optimal operating point of a single server executing requests with Interarrival times and Service times exponentially distributed. For its simplicity, we initially present the analytical solution of this model that was derived in [23]. Below, we describe the simulation results obtained with the corresponding model implemented with JSIMg.

The System Power for the considered M/M/1 model is

$$\Phi = \frac{\lambda}{R} = \frac{\lambda \text{ (1 - \lambda S)}}{S} \tag{2.7}$$

where λ is the arrival rate and *S* is the mean service time of the requests. To find the load λ*opt* that *maximizes* it is sufficient to set to zero its first derivative with respect to λ and derive the value of λ*opt* . We have: = (1/*S*) − 2λ*opt* = 0, thus it will be

$$
\lambda^{opt} = \frac{1}{2} \left. \frac{1}{S} \right. \tag{2.8}
$$

Therefore, according to Eq. 2.8, the optimal operating point is obtained with a load λ*opt* equal to *half* of the one corresponding to the maximum Throughput 1/S. The *Ropt* , *Uopt* and *Nopt* are

$$R^{opt} = \frac{S}{1 - \lambda^{opt} S} = 2S \qquad \qquad U^{opt} = \lambda^{opt} S = 0.5 \qquad \qquad N^{opt} = \frac{0.5}{1 - 0.5} = 1 \text{ }req$$

Let us remark that *Ropt* is*twice* its minimum value S, the server is utilized at 50% and the mean number of customers in the server is 1, 0.5 in queue and 0.5 in execution. In this optimal condition, an arriving request has 50% of probability to find the queue empty and the server idle. Figure 2.16 shows the behavior of of a server with *S* = 1 s.

An interesting observation can be obtained from the analysis of Fig. 2.16b which shows that System Poweris maximized with the load corresponding to the tangent point of the straight line *R* = *m*λ from the origin to the Response time curve. Equating to zero the discriminant of the equation that compute the intersection of the two functions we obtain *m* = 4*S*2. Replacing it in the equation of the intersection we obtain λ = 0.5(1/*S*), which has already been found as optimal loadin Eq. 2.8.

This property allows to define the optimality condition as the one corresponding to a load λ*opt* for which the relative increase of the Throughput X is equal to that of the Response time *R*. When it is λ<λ*opt* it will be *d X*/*X* > *d R*/*R* therefore an increase in λ increases the Throughput more than the Response time, so the gain is higher than the loss. The opposite situation occurs when it is λ>λ*opt* since in this condition an increase of λ generates losses greater than the gains, i.e., it is *d X*/*X* < *d R*/*R*.

**Fig. 2.16** Power (**a**) and Response time R (**b**) versus Utilization with S = 1 s

This definition of the optimal operating point is valid for *any* Throughput and *any* Response time functions. The metric System Power can be very useful to implement load balancing policies based on machine learning and as a target function in autoscaling components.

The implemented JSIMg open model consists of three stations: Source, Queue, Sink(see, e.g., Fig. 2.1). The Service timesof the Queue are exponentially distributed with mean *S* = 1 s. The Interarrival times of the requests generated by the Sourcehave exponential distribution with arrival rates ranging from 0.1 to 0.9 req/s. Figure 2.17 emphasizes the selection of System Throughput, System Response time, and System Power indexes (the last two are shown in the graphs of Fig. 2.18).

#### *2.4.3 Results*

A What-if analysis is used with the arrival rate λ of requests as control parameter with values ranging from 0.1 to 0.9 in 9 models. As can be seen from Fig. 2.18, the values of R and corresponding to the optimal load λ*opt* = 0.5 req/s are very close to the exact ones obtained analytically (*R* = 2 s, and = 0.25). The confidence intervals are very small.

**Fig. 2.17** Selection of the System Power index

**Fig. 2.18** Response time R and System Power obtained with JSIMg

#### *2.4.4 Limitations and Improvements*

• High variability of *Service times*: In [24] it is described the analytical derivation of in simple models with exponential Interarrival times and high variability of Service times (for M/G/1 stations).

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Chapter 3 Systems with Heterogeneous Workloads**

#### **3.1 Parameterization of Heterogeneous Workloads**

As for the characterization of the requests, there is a direct correspondence between Service times, Visits, and Service demands used in single-class models and those used in multiclass models. However, since their values must be specified on a per-class basis, each parameter must be identified now with two indexes: the station and the class it will refers to. For example, the service demand *Dr* of resource *r* becomes now *Dr*,*<sup>c</sup>*, the Service demand of classc request to resource r.

However, new problems arise when the growth of workload intensity has to be described as this can be done in different ways. Indeed, while to specify the workload intensity in single-class models is sufficient to know the Number of customers *N*<sup>0</sup> in execution in closed systems, or the arrival rate λ<sup>0</sup> and distribution of Interarrival times in open systems, the presence of multiple classes make these descriptions no longer adequate. Recall that with the index 0 (zero) of a metric we refer to the system as a whole.

In this section we first consider closed models, then we will analyze openmodels.

In closed models with of *C* classes of jobs, the workload intensity is described by the vector **N0** = {*N*<sup>0</sup>,<sup>1</sup>, *N*<sup>0</sup>,<sup>2</sup>, ..., *N*<sup>0</sup>,*<sup>C</sup>*} whose components are the number of jobs of each class in execution. The total number of jobs in execution is given by *N*<sup>0</sup> = *N*<sup>0</sup>,<sup>1</sup> + *N*<sup>0</sup>,<sup>2</sup> + ... + *N*<sup>0</sup>,*<sup>C</sup>*. For example, **N0** = {25, 75}means that in the closed model there are globally *N*<sup>0</sup> = 100 *jobs* in execution, 25 of class-1 and 75 of class-2.

A new parameter very useful for the description of multiclass workloads growth is the vector *β* representing the fractions of jobs of the *C* classes in execution in the system, that we will denote as population mix or job-mix:

$$\mathcal{B} = \{\beta\_1, \dots, \beta\_C\} \quad \text{with} \quad \beta\_c = N\_{0,c}/N\_0 \quad \text{and} \quad \beta\_1 + \beta\_2 + \dots + \beta\_C = 1 \quad (5.1)$$

Using the population mix, the workload **N0** of the previous example can be described by **N** = *N*<sup>0</sup> *β*, with *N*<sup>0</sup> = 100 and *β* = {0.25, 0.75}.

The importance of the job-mix lies in the fact that in multiclass networks the ratio of the global utilizations of two stations is no longer constant with *N*0, as in single class case, but depends on the fractions of jobs of the various classes in execution. Indeed, applying in single-class models the Utilization law to resources *i* and *j* we have: *Ui* = *X*0*Di* and *Uj* = *X*0*Dj* , and their ratio *Ui*/*Uj* = *Di*/*Dj* is constant with *N*. The immediate consequence of this behavior is that the bottleneck of the system may migrate among the resources as a function of the population mix. Thus, since it is known that the overall performance of a system is limited by the congested resource (i.e., the bottleneck), the fluctuation of the mixes may abruptly change them deeply.

While the definition of bottleneck is simple, i.e., the resource with the highest utilization, in multiclass models, the problem of bottleneck identification is not trivial since the same model can exhibit different bottlenecks depending on the population mix. Different types of bottlenecks can be identified. The class-c bottleneck is the station with the highest service demand of that class and saturates (its utilization tends to one) when the number of classc customers grows to infinity. The problem in multiclass systems is that, as a function of the population mix, a station may saturate also if it is **not** a class-bottleneck (in this case the station will be referred to assystembottleneck or model-bottleneck) or **more** stations may saturate concurrently with several mixes, referred to as common saturation sector(see [2, 3, 15]).

Therefore, to characterize the workload behavior in multiclass models, we must describe the variations of the mixes. In general, different *β* may yield different bottlenecks. Two types of workload increment should be considered: proportional and unbalanced. The population growth that consists of letting the total number of customers *N*<sup>0</sup> to grow keeping constant the population mix *β* is referred to as proportional growth.

According to this type of growth, in the example of workload above considered, the jobs in execution will be increased according to the proportions 25% of class-1 and 75% of class-2 since the mix is *β* = {0.25, 0.75}. So, when the total number of jobs increases to 300, we will have 75 class-1 and 225 class-2 jobs in execution.

We have the unbalanced population growth when only one class of jobs, say *c*, increases. As *N*<sup>0</sup>,*<sup>c</sup>* continue to growth, the bottleneck of the system tends to the station that is the class-c bottleneck, the population mix tends to *β* = {0, 0, 0, ..., 1*C*}, and the performance tend to the asymptotes of single-class workloads.

To support users who need to model the different types of population growth, JMT implement specific features of the What-if analysis that allow the automatic increment of the population of a single class only (see, e.g., Fig. 3.2) or the generation of all the possible mixes of two classes in closed models (see, e.g., Fig. 3.7b).

Most of what has been previously described for the closed models also applies to open models. The number of jobs in execution of the various classes must be replaced with the corresponding arrival rates. So, the global arrival rate to a open system is *λ***<sup>0</sup>** = {λ<sup>0</sup>,<sup>1</sup>, λ<sup>0</sup>,<sup>2</sup>, ..., λ<sup>0</sup>,*<sup>C</sup>*} and the population mix is described by:

$$\mathcal{B} = \{\beta\_1, \dots, \beta\_C\} \qquad \qquad \beta\_c = \lambda\_{0,c}/\lambda\_0 \qquad \qquad \beta\_1 + \beta\_2 + \dots + \beta\_C = 1 \qquad (3.2)$$

Differences between open and closed systems lie in the bottleneck switch behavior. In the former, the bottleneck migrate instantaneously between two resources without going through a common saturation section, i.e., the set of mixes that saturate both concurrently. Regarding the scheduling disciplines of the various classes in multiclass models, there are differences as a function of the solution technique adopted. While with simulation the users have practically no limitations (some minor incompatibilities may take place as a function of the types of discipline selected), the analytical technique introduce some constraints. Typically, the queueing networks that are solved analytically are of the separable types (see, e.g., [9, 36]) and their solution (the stationary probability of their states) can be obtained by the product of the individual solutions of the stations. The computational complexity introduced by the presence of multiple classes that may have different scheduling algorithms, and usually by the large numbers of stations and customers has required the introduction of some limitations. An important theorem, the BCMP (see, e.g., [6, 36]), for open, closed, and mixed queueing networks define the characteristics that a multiclass network should have to be separable and thus to be solved analytically with efficient algorithms. Each station must be of one of the following types:

Load-dependent service times are allowed. For PS, LCFS-PR, and IS stations the service times for the requests of a class may depend on the number of requests of that class in the station, or on the global number of requests in that station. For FCFS station the service times may depend only on the global number of requests of all the classes in that station.

In conclusion, users must be sure that the multiclass models who want to solve analytically have stations of the four types described. For example, if you want to solve analitycally a model that has different per-class Service times *Sr*,*<sup>c</sup>* on the same resource, the JMVA will solve it in any case (it compute the *Dr*,*<sup>c</sup>*) but you should be aware that the modeled scheduling discipline is PS and not FCFS. If in any case it is necessary to solve this model with the FCFS discipline, then there is no other choice than to use simulation.

#### **3.2 Motivating Example of Multiclass Models**

**tags**: closed, two-class, Delay/Queue, JMVA.

In this section we describe an example purposely designed to emphasize the errors on the performance forecast that can be obtained from the same model assuming that a multiclass workload consists of a single class of jobs. The counterintuitive behavior exhibited by some performance indexes of multiclass models as a function of the fluctuations of the classes of jobs in execution is also investigated. We solve this model analytically with JMVA.

#### *3.2.1 Problem Description*

Consider a powerful web server, accessed by administrative staff and graduate students of a university, which has two main resources:CPU and Storage (Fig. 3.1a). The workload consists of two different applications. The first one is used to manage the administrative procedures concerning the students curricula (tuition fees payments, courses attended, grades obtained, ...). The second one is devoted to the management (uploads, downloads, folder structure) of the course materials/documents (slides, notes, class exercises, homeworks, exams) that professors, assistants, and students access. According to the resource requirements of the two types of users, two classes of jobs, referred to as Adm and Doc, are identified.

Initially, we want to investigate the effects on performance indexes generated by different workload scenarios. More precisely, we increment the number of jobs of one class only while keeping constant the jobs of the other class. The different growth rates of the various classes of the workload are described by modifying the parameter *β*. To model the growth of class-*i* jobs, we increase the corresponding β*<sup>i</sup>* .

Then, we want to illustrate the errors that can be introduced in the performance forecast by a wrong assumption on the number of classes of the workload. We solve the same system model assuming the two-class workload as consisting of a

**Fig. 3.1** Web server layout (**a**) and closed queueing network with two different applications (**b**)

single-class of jobs. We consider a base system and an upgraded system with a CPU more powerful of a factor of five (the corresponding CPU service demands are decreased by five times). The behavior of the performance indexes are studied with respect to all the possible combinations of the two classes of jobs in execution, i.e., all the possible population mix.

#### *3.2.2 Model Implementation*

We use a closed model (Fig. 3.1b) since the customers that have access to the server are limited (administrative staff and graduate students). It consists of three resources: Users, CPU, and Storage. The Users station is of delaytype.

The two classes of the workload are characterized by the Service demands shown in Table 3.1. Let *N*<sup>0</sup> be the global number of jobs of the two classes. To model the growth of class-Doc jobs only (unbalanced population growth), we use the What-if feature with Number of customers and class-Doc as control parameters (Fig. 3.2). The *N*<sup>0</sup>,*Doc* values range from 5 to 280 with step of 5. To study the effects on performance forecast corresponding to the two different assumptions on the number of classes (one and two) in the workload, we consider two configurations, i.e., base and upgraded, of the same system. The behavior of performance indexes in the two configurations has been investigated modeling

**Table 3.1** Service demands [s] of the two classesof jobs

**Fig. 3.2** Parameters of the What-if for the description of the unbalanced population growth: only class-*Doc* jobs increase while class-*Adm* jobs are kept constant

all possible population mixes *β* with *N*<sup>0</sup> = 300 jobs (Fig. 3.5) and modifying the Service demands of Table 3.1 (see Table 3.2).

#### *3.2.3 Results*

In what follows we will describe the operations required to achieve the objectives of the study (referred to as Obj.1–Obj.2).

#### **Obj.1: Show the counterintuitive result that with a multiclass workload the Global System Throughput** *X***<sup>0</sup> can decrease in spite that the global number** *N***<sup>0</sup> of jobs in execution increases**

We consider the model with the two-class workload whose service demands are shown in Table 3.1. The initial workload is **N** = {20, 5}, globally *N*<sup>0</sup> = 25 jobs are in execution, 20 of class-Adm and 5 of class-Doc. The volume of traffic due to the class-Doc jobs is expected to increase during the next semester up to 280. This behavior is the typical unbalanced population growth that can be used when one class increases more than the others. We use the What-if feature of JMVA to evaluate the Global System Throughput *X*0. The parameters that describe the increase of class Doc jobs are shown in Fig. 3.2. The Control Parameter is the Number of customers of class-Doc *N*<sup>0</sup>,*Doc*, and the execution of 276 models with its increasing values from 5 to 280 are required.

In Fig. 3.3a the behavior of the Global and per-class System Throughput are shown for the number of jobs *N*<sup>0</sup> in execution increasing from 25, **N** = {20, 5}, to 300, **N** = {20, 280}. Initially *X*<sup>0</sup> increases until the number of Doc jobs reaches 19, corresponding to the maximum value of *X*<sup>0</sup> = 5.416 j/s. Then, any further increase of *N*<sup>0</sup>,*Doc*, and clearly of *N*0, corresponds to a decrease of *X*<sup>0</sup> (with *N*<sup>0</sup>,*Doc* = 280 it is *X*<sup>0</sup> = 2.724 j/s). How is it possible this happens?

**Fig. 3.3** Counterintuitive behavior of performance indexes (Base system) with unbalanced increase of the Global Number *N*<sup>0</sup> of jobs in execution: only class-Doc jobs increases from 5 to 280

The answer is prompted by Fig. 3.3b showing the Global Utilization of CPU and Storage. For *N*0,*Doc* ≤ 18 the CPU is the most utilized resource, while for *N*0,*Doc* ≥ 19 the Storage is the most utilized. We are addressing the bottleneck switch phenomenon that can occur with multiclass workloads when different classes have their highest service demands on different stations. The basic concept is the following: the service demands of the various classes at the bottleneck station determine the performance of the global system. Since when the station bottleneck changes typically also the corresponding service demands are different, this migration may have a deep impact on the performance. While the identification of the bottleneck in single-class models is easy, in multiclass models is more complex [3]. As described in Sect. 3.1, with multiclass workloads the bottleneck may migrate among stations as a function of the percentage of jobs of the different classes in execution, i.e., of the population mix.

With the workload behavior considered in this Obj.1 study, only class-Doc jobs increase from 5 to 280 and the population mix range from *β* = {0.8, 0.2} (**N** = {20, 5}) to *β* = {0.066, 0.933} (**N** = {20, 280}). When the Doc jobs are ≤ 18, the contribute of the Adm jobs, with *Dmax*,*Adm* = 0.2 on CPU, is fundamental for the saturation of CPU. When Doc jobs, with *Dmax*,*Doc* = 0.6, are ≥ 19 the load generated on Storage makes its global utilization predominant with that of the CPU.

As the number of Doc jobs continue to increase, asymptotically it will be (*N*<sup>0</sup>,*Doc* → ∞), and the workload assume the characteristics of a single-class with *β* → {0, 1}. In this case, the maximum system throughput is given by 1/*Dmax*,*Doc*. In our workload the max Service demand of class-Doc is *DSto*,*Doc* = 0.6 s, thus it will be *X*<sup>∞</sup> <sup>0</sup> = 1/*DSto*,*Doc* = 1.666 j/s. As can be seen from Figs. 3.3a, 3.4a, the System throughput of class-Doc and of the Global system tend to this

**Fig. 3.4** System throughput asymptotes of the (Base system) when only class-Doc jobs increase from 5 to 2000 (**a**), and only class-Admjobs increase from 20 to 2000 (**b**)

value: for *N*0,*Doc* = 280 *jobs* it is *X*0,*Doc* = 1.57 job/s and for *N*0,*Doc* = 2000 *jobs* it is *X*0,*Doc* = 1.65 job/s. The Global System Throughput *X*<sup>0</sup> decreases as *N*0,*Doc* increases beyond 19 jobs.

To emphasize the impact of the population mix on the Global System Throughput we ran two experiments with unbalanced population growth. Starting from the same initial workload **N** = {20, 5}, in the first one we increase to 2000 only class-Doc jobs while in the second one we increase to 2000 only class-Adm jobs. The System throughputs are shown in Fig. 3.4.

In Fig. 3.4a *X*0,*Doc* and *X*<sup>0</sup> tend to the same asymptotic value 1.666 j/s while in Fig. 3.4b *X*0,*Adm* and *X*<sup>0</sup> tend to the same asymptotic value 5 job/sec. The Global System Throughput *X*<sup>0</sup> tend to *X*<sup>0</sup>,*Doc* in (a) and to *X*<sup>0</sup>,*Adm* in (b). The differences between the two asymptotic values are evident. It should be pointed out that these values are not bounds! Indeed, a program mix that maximize the Utilization of all the resources of a system (see Fig. 3.3) maximize also the System throughput.

#### **Obj.2: Show that assuming a multiclass workload as single-class allows the construction of models that generate very inaccurate performance forecast. Some counterintuitive results (other than those of Obj.1) that occur with multiclass models are also shown.**

In this study (inspired, with some differences, by [25]) we will show that the performance projections obtained using a wrong assumption for the workload characterization, i.e., the workload is assumed to consist of a single class instead of multiple classes of customers, are unreliable.

We consider the closed system with three stations (Fig. 3.1b), that process the two-class workload whose Service demands are shown in Table 3.1.

**Table 3.2** Inputs and outputs of the single- and two-class models, for the original (Base) and the upgraded (Up) systems. The two-class workload is **N** = {255, 45} jobs, *β* = {0.85, 0.15}

In what follows we will refer to the values reported in Table 3.2 obtained with the two-class workload **N** = {255, 45} jobs, the corresponding population mix is *β* = {0.85, 0.15}. The study has been carried out according to the following steps:

**step (1)**—First, we assume to know that the workload consists of *N*<sup>0</sup> = 300 jobs belonging to two classes *Adm* and *Doc* whose service demands are shown in Table 3.1. We consider the population mix *β* = {0.85, 0.15}, i.e., the workload is **N** = {255, 45} jobs, 255 Adm and 45 Doc. Some output measures (System Response time *R*0, System Throughput *X*0, and the Utilizations of CPU and Storage) obtained from the execution of the two-class model are shown in columns Adm-Base and Doc-Base of Table 3.2.

**step (2)**—From the outputs of the two class model we compute the correspondentsingle class aggregate model (column aggregate-Base). The aggregate values of Utilization and System Throughput *X*<sup>0</sup> are obtained summing the correspondent per-class indexes: *UCPU* = *UCPU*,*Adm* + *UCPU*,*Doc* = 0.89 + 0.11 = 1 *USto* = *USto*,*Adm* + *USto*,*Doc* = 0.22 + 0.66 = 0.88*X*<sup>0</sup> = *X*<sup>0</sup>,*Adm* + *X*<sup>0</sup>,*Doc* = 4.448 + 1.102 = 5.551. For the System Response time *R*0, according to the Little law *Ni* = *Xi Ri* , the per-class values must be weighted by the relative throughput:

$$R\_0^{Base} = \frac{N\_0}{X\_0} = R\_{0, Adm} \frac{X\_{0, Adm}}{X\_0} + R\_{0, Dec} \frac{X\_{0, Dec}}{X\_0} = 49.649 \, sec \tag{3.3}$$

This is the correct computation of the System Response time with multiclass workloads.The same weights must be applied to the computation of the aggregate service demands *DBase CPU* <sup>=</sup> <sup>0</sup>.<sup>180</sup> *<sup>s</sup>* and *<sup>D</sup>Base Sto* = 0.159 *s*. The weights of the relative throughputs have also the following intuitive interpretation. The number of jobs of the two classes Adm and Doc executed in the interval T are *C*<sup>0</sup>,*Adm* and *C*<sup>0</sup>,*Doc*, respectively. This means that in the log file of the system executing the two classes workload there will be *C*<sup>0</sup>,*Adm* times the value of *R*<sup>0</sup>,*Adm* and *C*<sup>0</sup>,*Doc* times the value of *R*0,*Doc*. Thus, to compute the mean of all the *R*<sup>0</sup>s we need to weight the two values with the respective times they appear in the file. And, dividing by T both the terms of the ratios *C*0,*Adm*/*C*<sup>0</sup> and *C*0,*Doc*/*C*<sup>0</sup> we obtain *X*0,*Adm*/*X*<sup>0</sup> and *X*0,*Doc*/*X*0, that are the weights considered in Eq. 3.3.

**step (3)**—Now consider a new analyst who does not know anything about the system workload and builds a single-class model considering all measures of the log file as belonging to a single type of jobs. If he made the right computations, he will get the same values shown in the column Aggregate-Base for both input parameters and output measures. We note that these values are the same as those already computed in the previous step. Indeed, he, without being aware, automatically applies for their computation the correct weights described in **step 2**. At this point, the analyst has the wrong certainty that the implemented model is correct!

**step (4)**—Now we will use the two workload models (the one with two classes and the one with a single class) for the performance projections. An increase of Doc customers is expected in the near future. We want to evaluate the effect on the response time *R*0,*Doc* of Doc jobs that will be generated by an increase of the CPU speed. We assume to consider a multicore processor that for the workload considered increases by a factor of five the CPU speed.

The primary effect of this upgrade is the decrease of the CPU service demands *DCPU*,*Adm* from 0.2 to 0.04 and of *DCPU*,*Doc* from 0.1 to 0.02 (see *DCPU* in columns Adm-Up and Doc-Up, respectively). The execution of the two-class model compute the indexes reported in the lower part of these columns. Applying the computations described in **step 2)** it is possible to derive the correct values of the aggregated singleclass model corresponding to the two-class model (see column Aggregate-Up correct).

Analyzing the System Response times of the two classes, we may see that while the *R*0,*Adm* decreases of 77%, the *R*0,*Doc* **increases of 377%**(from 30.8 to 147.2 s)! This is a **counterintuitive result: performance degrades with a CPU upgrade!** The motivation of this result hampered by intuition is related to the switch of the bottleneck from the CPU to the storage which take place after the upgrade. Indeed, since the throughput *X*<sup>0</sup>,*Adm* of class-Adm (that has the CPU as class-bottleneck) increases from 4.4 to 16.5 j/s, the competition for the Storage increases a lot (*USto* reach saturation) and the class-Doc jobs, that are heavily storage bound (it is *DSto*,*Doc* = 0.6 *s*, the Storage is the class-Doc bottleneck), experience a strong degradation of the response time.

Figure 3.5 shows the behavior of the System Response times of the two classes and the aggregated value as a function of all the population mixes, of the original system (a) and the upgraded version (b). The workload consists of 300 jobs, ranging from **N** = {300, 0} to **N** = {0, 300} jobs. The differences between the behavior of the correspondent curves are evident.

**step (5)**—We consider here the single-class model built in **step 3)** by the ignorant analyst, assuming that all the jobs as belonging to the same single class. The effect of the CPU upgrade is a decrease in service demand *DCPU* from 0.180 to 0.036 *s*,

**Fig. 3.5** System Response times versus all *β* population mixes of the original (Base) and the upgraded (Up) systems, with 300 jobs. Dashed lines represent *β* = {0.85, 0.15} used in Table 3.2

see columns (Aggregate-Base) and (Aggregate-Up wrong). The output measures (Table 3.2) obtained from the execution of this single class model with *N* = 300 jobs show an improvement of performance: *R*<sup>0</sup> decreases from 49.64 to 43.38 *s*, and *X*<sup>0</sup> increases from 5.5 to 6.2 j/s. As this qualitative behavior corresponds to the expectations, the analyst my have the wrong impression that the implemented model is correct. Indeed, comparing the values of the two columns Aggregate-Up wrong and Aggregate-Up correct it is possible to understand immediately the **large errors** affecting *R*<sup>0</sup> and *X*0: -66% and +168%, respectively. Thus, we may conclude that

the performance forecast based on single-class models of heterogeneous workloads are inaccurate when used to obtain projections for the average aggregate job. Furthermore, as Fig. 3.5 shows, it is evident that the average aggregate job cannot be used to derive reliable projections for each class of the two-class workload.

Let us remark that the per-class System Response times plotted by JMVA in Fig. 3.5 are the sum of the Residence times of a job of that class at all the resources of the system, including the Reference station. Thus, for example, we should add *Z*<sup>0</sup>,*Doc* = 10 s (the Residence time of Doc jobs at the Reference station) to the System Response time of Doc jobs *R*<sup>0</sup>,*Doc* = 30.801 s of Table 3.2 to obtain the value 40.801 plotted in Fig. 3.5a in correspondence to program mix *β* = {0.85, 0.15}.

#### **3.3 Performance Optimization of a Data Center**

**tags**: closed, two-class, Queue, JMVA.

In this case study we illustrate a general approach applicable to several capacity planning problems with an example based on the performance optimization of a data center with a workload consisting of heterogeneous applications, [14, 38]. The problems of bottleneck identification and migration are addressed as a function of the fluctuations of the different types (classes) of requests being executed. The topic of load balancingis also investigated.

#### *3.3.1 Problem Description*

A data center partition consists of six servers utilized by business critical applications that access to sensitive data stored. The area is highly protected for both physical access and digital security. It consists of one Web Server, two Application Servers, and three Storage Servers (see Fig. 3.6). The access to the applications and data stored on these servers is permitted only to a limited number of employees with the appropriate authorization. Based on the different requirements, in terms of amount of resources used and Quality of Service (QoS) targets, two types of applications can be identified in the workload. The two classes of requests, called class-1 and class-2, generated by the two applications are focused on business logic processing the first, and on intensive data processing (search, update) the second.

**Fig. 3.6** Layout of the data center considered

The mean number of their requests that are in execution simultaneously is 100. According to forecasts, the number of employees authorized to access the servers is expected to double over the next nine months. The management is concerned that performance may degrade to an unacceptable level. Several initiatives are considered.

As a first action it is required to investigate the impact on performance of an increase of the global number *N*<sup>0</sup> of requests in execution from 100 to 200. More precisely, it is required to know if the QoS in terms of the mean per-class Response times defined for 100 employees are still satisfied. Since it is not possible to know how the fractions of the requests in execution of the two classes vary over time, to compute the upper bound of Response times it is necessary to consider all the possible combinations of the two classes of requests in execution, i.e., all the population mix, with *N*<sup>0</sup> = 200 req.

The capacity planning study should answer to several questions such as "Does the data center with the current configuration support the increase of the workload without saturating one or more resources?", "With *N*<sup>0</sup> = 200 req in execution will the QoS targets on the per-class Response times be satisfied?", "Which is the resource that is the bottleneck of the system?".

Other important questions to be answered concern the actions that should be taken to increase, if possible, the performance of the data center with the current configuration: "Which is the impact of the population mix on the potential increase of performance?", "The utilization of the resources are balanced?", "Which is the population mix that maximizes the System Throughput and minimize the mean System Response time (referred to as optimal population mix)?".

These questions have been grouped into the two Objectives of the study that are analyzed below.

#### *3.3.2 Model Implementation*

We need to evaluate the performance of the data center with an overall number of customers doubled (200) with respect to the current one (100). We implement a closed queueing model with six queue stations (see Fig. 3.6) and *N*<sup>0</sup> = 200 req in execution. The workload **N0** = {*N*<sup>0</sup>,<sup>1</sup>, *N*<sup>0</sup>,<sup>2</sup>} consists of two classes of requests, where *N*<sup>0</sup>,<sup>1</sup> and *N*<sup>0</sup>,<sup>2</sup> are the number of requests in execution of *class*1 and *class*2, respectively.

To characterize the two type of applications in terms of processing requirements, a set of Service demands, one for each resource and each class, is used. The Service demand *Dr*,*<sup>c</sup>* of a request of classc at resource r is the total amount of time the request requires at that resource in order to be completely executed. The *Dr*,*<sup>c</sup>* are computed ignoring contention by other requests and may be estimated measuring utilizations and throughputs and using the equation *Ur*,*<sup>c</sup>* = *X*<sup>0</sup>,*<sup>c</sup> Dr*,*<sup>c</sup>*, where *X*<sup>0</sup>,*<sup>c</sup>* is the system throughput for class-*c* requests and *Ur*,*<sup>c</sup>* is their utilization of resource *r*. To minimize the errors in the parameter estimation, it is recommended, when possible, to collect the measurements when the two types of appli-

**Fig. 3.7** Service demands of the two classes of requests (**a**), and What-if parameters (**b**)

cations are executed in isolation. The Service demands of the two classes of requests, in ms, are shown in Fig. 3.7a. The amount of work requested from the Web Server is much less demanding than the one requested from the Application and Storage Servers. The computations required by the business logic place a medium load on the Application Servers while the high number of data manipulated, uploaded and downloaded, generate a high load on the Storage Servers.

The Service demands are exponentially distributed and the scheduling discipline of the servers is processor sharing PS. These assumptions allows us to solve the model analytically with the MVA algorithm [25, 31] using the JMVA. Indeed, according to the BCMP theorem (see Sect. 3.1) , multiclass models in which the queue stations adopt the PS scheduling discipline can be solved analytically with efficient algorithms also if the service times (as may be considered the service demands of the single visit used our case) of the two classes at the same resource are different. Models in which these assumptions are not satisfied (e.g., if FCFS scheduling is required), must be solved with the simulation technique using JSIM.

The performance predictions obtained by the capacity planning study are based on several What-if analyses.

To evaluate the performance metrics corresponding to all the possible population mix we used the What-if feature of JMVA varying from 100% to 0% the requests in execution of one class and the opposite (from 0% to 100%) the fraction of the other class.

#### *3.3.3 Results*

The peculiarity of the workload forecasting strategy adopted in this study is that only the global intensity, i.e., the total number *N*<sup>0</sup> = 200 requests in execution is known, and it is not possible to predict how the fractions of the requests of the two classes (i.e., the population mix) vary over time. Typically, it may be quite bursty.

Thus, to achieve the capacity planning goals What-if analyses are required with population mix as control parameter, Fig. 3.7b. The fraction β<sup>1</sup> of class-1 requests in execution range from 0.5% to 99.5%, the one of class-2 is the complement to 100%.

#### **Obj.1: Evaluate the behavior of the performance indexes of the data center with a global Number of requests in execution** *N***<sup>0</sup>** = **200 and for all the possible combinations of the two classes.**

JMVA is used to estimate the performance of the data center for the required parameter range with the Service demands shown in Fig. 3.7a. In the What-if screenshot of Fig. 3.7b the control parameter is population mix, and 199 models are executed with all the possible mix of the requests of the two classes ranging from **N0** = {1, 199} to **N0** = {199, 1}.

Let's start with the analysis of the behavior of per-class and Global System Response times with respect to all the population mix with *N*<sup>0</sup> = 200 req shown in Fig. 3.8a. The *x*-axis represents the fraction of class-1 requests β<sup>1</sup> with respect to the total number of requests in execution. In the two extremes β<sup>1</sup> = 0 and β<sup>1</sup> = 1 the workload consists of a single class, class-2 and class-1 only, respectively. In these cases, the resource that limit the performance of the system, i.e, the bottleneck, corresponds to the one with the max service demand *Dmax* . When the bottleneck is saturated (*Ubott* = 1), the values of System Response times can be easily computed considering the *Dmax* and applying the Utilization and Little laws, see Sect. 1.2. With **only** class-2 requests (β<sup>1</sup> = 0), the bottleneck is Storage1 with *DSto*<sup>1</sup> = *Dmax* = 0.105 s, from the Utilizationlaw *USto*<sup>1</sup> = *X*0*Dmax* = 1 we derive

**Fig. 3.8** System Response times [ms] (**a**) and System Throughputs [req/s] (**b**) with *N*<sup>0</sup> = 200 req versusfraction of class-1 requests in execution, from 0.5% to 99.5%

*X*<sup>0</sup> = 1/*Dmax* and by Little law it will be *Rmax* <sup>0</sup> = *N*0*Dmax* = 200 x 0.105 = **21 s**. In the other extreme β<sup>1</sup> = 1 with **only** class-1 requests, the bottleneck is Storage2 with *DSto*<sup>2</sup> <sup>=</sup> *Dmax* <sup>=</sup> <sup>0</sup>.070 s and it will be *<sup>R</sup>max* <sup>0</sup> = *N*0*Dmax* = 200 x 0.070 = **14 s**.

As soon as the number of class-1 requests in execution increases (and thus class-2 requests decrease), the load of Storage2 starts to grow. As a consequence, the bottleneck tends to migrate from Storage1, i.e., the class-2 bottleneck, to Storage2, i.e., the class-1 bottleneck. The corresponding class-1 System Response time *R*0,<sup>1</sup> increases until 13.86 s with the workload **N0** = {199, 1}, very close to the asymptotic value 14 s above computed. It does not coincide with it because with 199 req of class-1 and 1 req of class-2 the Utilization *USto*<sup>2</sup>,<sup>1</sup> of Storage2 for class-1 req. is 0.995 and not 1. Similar motivations apply with the opposite workload **N0** = {1, 199}, i.e., β<sup>1</sup> = 0.005, where it is *USto*<sup>1</sup>,<sup>2</sup> = 0.995 and the class-2 System Response time *R*0,<sup>2</sup> is 20.85 s while the asymptotic value is 21 s.

As can be seen in Fig. 3.8a, the Global System Response time *R*<sup>0</sup> is practically constant for a wide range of mixes, approximately between 30% and 70% of class-1. The important feature of these mixes is that executing a workload with one of them will result in the saturation of two resources simultaneously.

This interval of joint saturation, referred to as common saturation sector, is important in order to find the load of the system that optimize the performance, i.e., Throughput maximization and Response time minimization.

The identification of this interval can be done analytically under the assumption that the workload in execution is very large so that the bottleneck(s) is saturated [3]. With our workload of 200 req. the extremes of this interval are only approximate since the load does not saturate completely the bottlenecks.

The Response times of the two classes are identical when the two bottlenecks are equiloaded (for β<sup>1</sup> = 0.6 it is *R*0, 1 = *R*0, 2 = 10.8 s) and it can be shown that the corresponding equiload mix lies inside the common saturation sector [35]. *R*<sup>0</sup> is minimumin correspondence to this mix.

Figure 3.8b shows the System Throughput, Global *X*<sup>0</sup> and per-class *X*<sup>0</sup>,<sup>1</sup> *X*<sup>0</sup>,2. It is evident that *X*<sup>0</sup> is maximized for all the mix of requests that belong to the common saturation sector while the per-class throughput are constant in the interval. It can be shown that the equiutilization point, i.e., the mix of the two classes that causes two bottlenecks to be equally utilized, lies into this interval and provides the optimal load that maximizes[35] the global System Throughput.

The behavior of Response times and Throughputs in Fig. 3.8 can be understood by analyzing the bottleneck migration. Indeed, it is known that the resource that limit the performance of the system under all possible workload mix is the bottleneck. So, when the bottleneck changes also the performance changes.

The Utilizations of the three Storage Servers of Fig. 3.9a show graphically this phenomenon. We do not consider Web and App servers because their Service demands, see Fig. 3.7a, are definitively lower than those of Storage servers, and therefore will never be saturated. As predicted, Storage1 and Storage2 saturate together for all the mixes of the common saturation sector (approximately between 30% and 70% fractions of class-1 req. of the total population of 200 req.) while the Utilization of Storage3 is definitively lower (its max Utilization reached in the common saturation sector is *USto*<sup>3</sup> = 0.52), since its Service demands are smaller with respect to those of Storage1 and Storage2. When only a few class-1 requests are in execution (β<sup>1</sup> < 0.3), the bottleneck is Storage1. On the other side, when the fraction of class-1 req in execution is high (β<sup>1</sup> > 0.8) onlyStorage2 is saturated.

With regard to the Quality of Service targets on the per-class System Response times that were set for the original configuration of the data center with a workload of *N*<sup>0</sup> = 100 req evenly divided between the two classes, i.e., *β* = {0.5, 0.5}, we can see that with *N*<sup>0</sup> = 200 req cannot be satisfied. Indeed, the target values assigned to the mean System Response times of the two classes were *R*0,<sup>1</sup> = 8 s and *R*0,<sup>2</sup> = 12 s, respectively. With *N*<sup>0</sup> = 100 req these values **were met** (*R*0,<sup>1</sup> = 4.5 s and *R*0,<sup>2</sup> = 6.7 s), with *N*<sup>0</sup> = 200 req otherwise they **are not** (*R*0,<sup>1</sup> = 9 s and *R*0,<sup>2</sup> = 13.5 s), see Fig. 3.8a. We will see in the following Obj.2 that a load balancing action allows the satisfaction of the targets.

**Obj.2: Investigate on the actions that may improve the performance of the data center with the current configuration and a workload of** *N***<sup>0</sup> = 200 req**. **Can the** System Response time **objectives be achieved after these actions?** Figure 3.9a emphasizes the problem: the unbalanced utilization of the Storage Servers.

To enhance the performance we need to take into consideration the most heavily loaded servers, namely Storage1 and Storage2, since reducing the Service demands at resources other than the bottlenecks produces only marginal improvements. The total load on the two Storage Servers 1 and 2 is fairly balanced, *DSto*<sup>1</sup> = 125 ms and *DSto*<sup>2</sup> = 100 ms, while the load of the third server is much

**Fig. 3.9** Utilizations of the Storage servers in the original configuration (**a**) and in the balanced system (**b**) vspopulation mixes. The workload ranges from {0,200} to {200,0} req

**Fig. 3.10** Service demands for the balancedconfiguration of the three Storage Servers

smaller, 58 ms, see Fig. 3.7a. We want to assess the effect of alleviating the bottlenecks, trying to balance the loads of all three Storage Servers. So, even according to usage statistics, some files have been migrated between the various storage, more precisely from Storage1 and Storage2 to Storage3, in order to make their total Service demands more similar to each other than in the original configuration. Figure 3.10 shows the new Service demands.

Let us remark that the Global Service demand to all the three Storage Servers must remain the same as the one of the original configuration, namely 283 ms, since all the servers have the same technical characteristics. Thus, shifting some data from one resource to another alter the visits *Vr* but not the Service time *Sr* of a request.

This configuration denotes a good balancing of the load of the three storages and no server is underutilized, Fig. 3.9b. By comparing their behavior with those of Fig. 3.9a obtained with the original system it is evident that the sum of the three *Ur* is maximized, thus enabling the maximization of the System Throughput.

The maximum System Throughput of the original system, obtained with fractions inside the common saturation sector, was *X max* <sup>0</sup> = 0.0185 req/ms, Fig. 3.8b, while the one obtained after the balancing action, Fig. 3.11b, is *X max* <sup>0</sup> = 0.0218 req/ms, with an improvementof about **17.8%**.

The corresponding minimum System Response time were *Rmin* <sup>0</sup> = 10.8 s for the original system, Fig. 3.8a, and *Rmin* <sup>0</sup> = 9.18 s for the balanced, Fig. 3.11a, with a reductionof about **15%**.

**Fig. 3.11** System Response times [ms] and Throughputs [req/ms] in the balanced configuration versuspopulation mixes. The workload ranges from {0,200} to {200,0} req

With regard to the targets defined for the per-class System Response times, i.e., *R*0,<sup>1</sup> = 8 s and *R*0,<sup>2</sup> = 12 s, with the mix*β* = {0.5, 0.5}, we can see from Fig. 3.11b that the balanced configuration is **able to satisfy** them, i.e., *R*0,<sup>1</sup> = 7.65 s and *R*0,<sup>2</sup> = 11.47 s. Instead, as described previously, the original configuration of data center with 200 req in execution was **unable** to reach them.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Chapter 4 Impact of Variability of Interarrival and Service Times**

#### **4.1 Importance of Distributions: A Motivating Example**

In this section we highlight the *significant errors* in the computation of performance indexes that are introduced when *only* the mean values are considered instead of the distributions of some input parameters. Consider a server that requires a constant Service time *S* of 1 s to execute a request. Assume that the requests arrive with rate λ = 60 req/min in groups (bursts) and that the requests of a burst arrive at the same instant of time. The time between consecutive bursts is constant. We will analyze the impact on Queue time and Response time of different burst lengths, ranging from 1 to 60, considering always the *same* arrival rate.

In the case shown in Fig. 4.1a, a request arrives at the server exactly every second. Since the time *S* required for its execution is always equal to 1 s, the queue will never take place (the Queue time is equal to zero) and thus the mean Response time (Queue time plus Service time) is exactly one second for all requests. In the other graphs it is assumed that the requests arrive at the server with burst of increasing dimensions.

In Fig. 4.1b a burst of size 2 arrives exactly every two seconds. The first request never waits in queue, while the latter waits for one second, that is, the execution time of the first. So the mean Queue time is 0.5 s. In the graph of Fig. 4.1c a burst of size 3 arrives exactly every three seconds. The first request never waits, the second waits a second and the third waits two seconds. So the mean Queue time is 1 s and the mean Response time is 2 s.

Finally, in Fig. 4.1d 60 requests arrive together in a single burst every sixty seconds (the rate is always 1 req/s). In this case the mean Queue time is 29.5 s. Let us remind that the sum of *n* positive consecutive integers starting from 1 is *n*(*n* + 1)/2. In our case we have 60 requests, but only n = 59 of them wait from 1 to 59 s, respectively. Thus, the *mean* waiting time (Queue time) of the 60 requests is 29.5 s and the mean Response time is 30.5 s! The conclusion is

**Fig. 4.1** Impact of different burst lengths on mean Response time

*Even considering the* **same** *arrival rate* λ = 1 req/s *and the* **same** Service times *S=1* s, *depending on the arrival pattern of requests we could have a* **very high variability** *of mean* Response times: *from 1 to 30.5* s *in the example considered (and this is not the worst case!).*

#### **4.2 Variability of Interarrival Times**

**tags**: open, single class, Queue, Exp/Hypo-exp/Hyper-exp, JSIMg.

The objective of this case study is to emphasize the impact of the variance of Interarrival times on the performance of a system.

#### *4.2.1 Problem Description*

Consider a model of a web server that needs to execute an e-commerce application to sell equipment produced by a new company. While the mean and variance of Service time required to process a purchase order can be estimated with sufficient accuracy, the pattern of incoming requests is unknown as customers are located all over the world.

We use a simple model of the web server consisting of a single queue station. To account for the unknown patterns of the incoming requests we consider five distributions of interarrival times with the same mean and increasing variability. To describe the variance of interarrival times we use the coefficient of variation c of each distribution (given by the standard deviation/mean ratio). For a given value of arrival rate, the values of c are directly proportional to the variance of the five distributions since their means are the same.

The Service times are assumed exponentially distributed with the same mean *S* = 1 s for allthe models.

To analyze a wide range of traffic intensities we consider several arrival rates, ranging from light-load (10% of server utilization) to heavy-load (90% of server utilization) conditions. For each arrival rate we execute five models corresponding to the five interarrival time distributions. As a reference metric we consider the mean Response times of the models executed. The models are solved with JSIMg.

#### *4.2.2 Model Implementation*

We use a open model consisting of three stations: Source1, Queue1, and Sink1, Fig. 4.2a. The Service times of Queue1, with mean *S* = 1 s, used in all the models have the same exponential distribution. The five distributions considered of Interarrival times, in sequence of increasing variance are: Constant cv = 0, Hypo-exponential cv = 0.5 (Hypo-exp), Exponential cv = 1 (Exp), Hyperexponential cv = 5 (Hyper-exp), Hyper-exponential cv = 10 (Hyper-exp). Figure 4.2b shows the window for setting the mean = 10 (corresponding to λ = 0.1 req/s) and the coefficient of variation cv = 10 of the Hyper-exp distribution.

The differences between the distributions are emphasized in Fig. 4.3a (obtained with λ = 0.9 req/s), that shows the graphs relating to three of them: Hypo-exp cv = 0.5, Exp cv = 1, and Hyper-exp cv = 0.5. As can be seen, the percentages of Interarrival times (i.e., the percentiles) that are less than the mean value 1.111 s are very different: 56.8% for the Hypo-exp, 63.6% for the Exp (the exact analytical result is 0.6321), and 85% for the Hyper-exp with cv = 5 (and 91% for the Hyper-exp cv = 10, not shown in the figure). To obtain the percentiles of a metric with JSIMg see Sect. 2.2 and Figs. 2.10, 2.11.

The increase of variability also heavily influences the maximum values of the various distributions: 5.4 s for the Hypo-exp, 16.69 s for the Exp, 261.22 s for the Hyper-exp cv = 5, and 864.46 s for the Hyper-exp cv = 10. The number of samples needed to reach the equilibrium of the metric Throughput of Source1, that provides the Interarrival times with 99% Confidence Interval and 0.03 Max Rel. Err., ranges from 40960 of the Hypo-exp to 1063920 of the Hyper-exp cv = 10.

#### 66 4 Impact of Variability of Interarrival and Service Times

**Fig. 4.2** The model considered (**a**), Settings of the mean = 10 and coeff. of variation cv = 10 of the Hyper-exponential distribution of Interarrival times for Arr.rate 0.1 req/s (**b**)