Figure 3. In the case of HTTP/3, since the official deployment was not available yet at the moment of the writing of this paper [29], we used the test QUIC client/server provided by Google [30]. The QUIC server runs the latest version (43 at the moment of the implementation), which, although it differs in some aspects to the IETF implementation, can serve as a preview on how the overall interaction of HTTP over QUIC would work on DASH. In the case of the DASH clients, we used a modified version of AStream [31], which is a Python-based emulated video player, with the QUIC client integrated as a sub-module as done by Arisu et al. [6]. Moreover, we modified the official QUIC server by automatically adding the required header tags (i.e., alternative protocol, port number, URL) per file in the server directory, which is mandatory in the test QUIC server. To compare with the current approach (using HTTP over TCP only), we used the built-in HTTP Server module from Python at the server side, and *libcurl* TCP client integrated into AStream. In addition, note that we kept the same connection for the entire transmission—as QUIC would do within the same connection—rather than opening a new connection per request. Finally, as the SDN controller, we used OpenDaylight (ODL) Beryllium SR2 and implemented the functions of each module as an internal application.

**Table 1.** Experimental parameters.

**Figure 3.** Testbed setup.

*5.2. Use Case*

Since streaming solutions over HTTP were originally developed for Video-on-Demand (VoD) [13], we used VoD as the use-case; however, we believe that the proposed solution can also be applicable for live streaming. To conduct the experiments, we used Big Buck Bunny (BBB) from a publicly available DASH dataset [32] by Lederer et al. [33]. In this dataset, the full-length BBB video (of approximately 600 s) is encoded in twenty different representations and segment (video chunks) sizes from 1 to

15 s ranging from bitrates of 420 × 360 pixels at 45 Kbps to 1920 × 1080 pixels at 3.9 Mbps. For the experiments, we used only a 4-s segment size, as it is an intermediate value between encoding and flexibility in the bitrate adaptation. To test the approach, we compared the performance of SAND/3 by using the following adaption bitrate (ABR) algorithms in the DASH clients:

**Table 3.** List of SARA parameters used in the experiment.

Each client was assigned a profile and a single device with the same specifications regarding the memory and CPU, emulated by Xterm terminals in Mininet. The priority level was different for each user; for simplicity, we numbered the priorities in ascending order so that high-, medium-, and low-priority would map to 0, 1, and 2, respectively. Note that, in the current implementation, these values were set statically in the user profile; a more comprehensive solution would require, for instance, a classification based on all the parameters, as done by Bentaleb et al. [22]. The experiment was conducted as follows. For each client ABR algorithm (namely TBA, SARA, and BBA), we played BBB on all the devices randomly starting within an approximate 30 s interval. In each run, all clients used the same adaption algorithm using the three variants:

• SAND/3: The proposed scheme was used on top of QUIC and applying the process explained in previous sections.

In the first two cases, we left the default connectivity settings, which includes the end-to-end routing using simply the shortest path route. Note that, due to some performance limitations in the QUIC test server, not suitable for a large number of users [30], we only used three clients; nevertheless, since the primary objective of the experiment was to analyze the behavior and the feasibility of the proposal rather than a large-scale test, the current setup would suffice as a proof-of-concept.

#### *5.3. QoE Metrics*

To objectively measure the user QoE, there are five factors that influence the most in a DASH system: the number times the video freezes (stalls), the number of video quality shifts, media throughput (bitrate), start-up delay, and the fairness at shared bottlenecks [13,37]. However, since the start-up delay and fairness are mostly related to live-streaming, they are perceived less critical than the other variables [38]. Therefore, in this paper, we focus on the number of stalls, media throughput and the number of video quality shifts to measure the QoE.

#### **6. Results**

#### *6.1. Number of Stalls*

Concerning the number of stalls, which occurs when the video stops due to a buffer underrun, Table 4 shows the results obtained per adaption algorithm and their combined duration. As observed, there were stalls only in the TCP-only (current default) approach—the other approaches (QUIC and our approach) did not present any stalls during the playback whatsoever—which is a favorable result since a stall is the most detrimental factor for QoE [11]. In addition, note that most of the stalls occurred on users from the lower priority category. However, in the case of SARA, even users from the first categories suffered from stalls, and although there were few occurrences the combined time was considerable (16–26 s), which shows how unfair bandwidth competition can affect all users as the number of parallel requests increases.

**Table 4.** Number of stalls per user category using the TCP-only approach.

#### *6.2. Media Throughput*

Concerning the media throughput (received video bitrate), depending on the adaption algorithm and the approach used, the results were relatively different. Figures 4–6 we show the media throughput obtained for each type of user using TBA, BBA, and SARA respectively.

Initially, in Figure 4a, we can see that, although TBA increased the requested bitrate progressively when using TCP-only approach, once higher quality segments were achieved and as new users started to request better quality representations, the video quality decreased significantly. This considerable quality decrease was due to the shared bandwidth bottleneck at the edge of the topology, and the unfair bandwidth competition. On the other hand, users that used QUIC-only, as shown in Figure 4b, received relatively low-quality segments for most of the transmission; however, at no time did the clients suffer from buffer underruns. Moreover, the bandwidth distribution was somehow evenly distributed among clients. Finally, in Figure 4c, for the case of our approach, the bitrate changes were

quickly improved due to the characteristics of TBA. Moreover, once the clients converged to the best available video quality, the playback was steady and stayed, for the most part, in the highest video quality even for users with lower priority, which shows the benefits of the proposed approach for all user levels.

(**c**) Received bitrate using the proposed approach

**Figure 4.** Media throughput obtained for three users at 10 Mpbs links using TBA adaption algorithm.

When SARA was used as the DASH adaption algorithm, as shown in Figure 5, since the algorithm considers the variable sizes of the different video representations, the convergence time was faster than the case of TBA. However, due to the congestion in the shared lines used in the TCP-, and QUIC-only approaches, the estimation was done using non-accurate information, and therefore the requested bitrate could not be handled by either of the approaches, leading to a highly unstable playback, as can be observed in Figure 5a,b. Due to the conservative nature of SARA, especially in terms of the *Delayed Download* mechanism, the video quality is gradually increased and maintained in the same one for a certain amount of time to avoid unnecessary downloads. However, by the time the segment is to be transmitted, the network conditions might have already changed so that the link will be overloaded downloading a bigger file until the threshold reaches the lower bound. On the other hand, by using our approach, once the optimal bitrate is selected, the feedback can estimate the available resources more accurately since the network status is continuously monitored on all network devices, which led to a steady and smooth playback with the highest video quality for all users, as shown in Figure 5c.

(**c**) Received bitrate using the proposed approach

**Figure 5.** Media throughput obtained for three users at 10 Mpbs links using SARA adaption algorithm.

Finally, in Figure 6, we show the received bitrate when using BBA as the DASH adaption algorithm. Note that the bitrate variability is higher than the other algorithms. BBA has a different approach, which focuses on the current buffer occupancy rather than throughput estimation. Thus, it tends to request segments with higher bitrates for a longer time. However, when the streams compete with other requests, the congestion level increases considerably, leading to overloaded links and consequently higher loss-rate. As observed in Figure 6a, which depicts the received bitrate using TCP, there was a point in the playback (approximately at Second 250) where the congestion level was so high that the bitrate suffers from a phenomenon called *ON–OFF period*. In this state, the requested video quality oscillates between the highest and lowest, creating instability among the other DASH clients due to an inaccurate buffer estimation. In the case of QUIC, although there was some interference between Users 2 and 3, BBA was capable of recovering relatively quickly, as observed in Figure 6b. Lastly, in the case of our approach, as shown in Figure 6c, the bitrate was between the upper level of cushion (*cu*) and the maximum buffer capacity (*Bmax*) for the most of the transmission, which allowed a smooth video playback without any buffer underruns.

**Figure 6.** Media throughput obtained for three users at 10 Mpbs links using BBA adaption algorithm.

#### *6.3. Video Quality Shifts*

Concerning the number of video quality shifts, which refers to the change of video representation, users perceive dissatisfaction when the video abruptly changes to a considerably lower bitrate; therefore, a progressive increase of quality and minimizing the downshifts is desirable to have a good QoE. Figure 7 show the number of both up- and down-shifts events in every approach. As observed in Figure 7a–c, regardless on the adaption algorithm used (i.e., TBA, SARA, or BBA respectively), our approach was able to considerably minimize the downshift events at least for the user with the highest priority (User 1). However, note that, in the case of the number of the downshifts using BBA (Figure 7c), for Users 2 and 3, the number is higher using our approach than when using TCP. Nonetheless, as previously shown in Figure 6, the difference in the decrease of video quality was much bigger when using TCP than in our approach.

Regarding the number of upshifts, even though this variable is tightly coupled with the downshift events, the strategy used in each algorithm will significantly influence on how fast the bitrate will converge again to the best possible quality before a downshift event. Most of the adaption algorithms will opt for an aggressive decrease and progressive increase, having, therefore, more upshift events than downshifts, as shown in Figure 7d–f, which depict the number of upshift events using TBA, SARA, and BBA respectively. The average improvement achieved was around 45%, and 60% for the downshifts, and 40% and 50% for the upshift events compared to the TCP-only, and QUIC-only approaches, respectively, which is a significant improvement to the overall QoE.

**Figure 7.** Number of up- and down-shifts per approach using different adaption algorithms.

#### *6.4. Average Downloaded Video Files*

Figure 8 depicts the heat-map of the average bitrate distribution of each of the approaches using the different adaption algorithms throughout the entire video. As observed, regardless of the algorithm, the bitrate distribution is considerably wider towards the highest video quality segments (light blue) when using the proposed approach. In fact, when using TBA and SARA, about 85% of the downloaded segments belonged to that category (Figure 8a,b), while, in the case of BBA (Figure 8c), it was about 45%, which was marginally better than the other approaches but still the second best video representation covered a larger part of the downloaded segments. This shows that it is possible to guarantee the best quality regardless of the user category or the adaption algorithm used. Since a better representation is stored in a bigger file, the amount of downloaded content was also significantly higher than when using TCP- or QUIC-only approaches, as shown in Figure 9a,b, respectively. For the most part, the distribution among the clients was rather even; however, note that there is a slight progressive decrease in the case of BBA (Figure 9c) as each user was affected by the number of clients and the increased congestion. Nevertheless, the approach achieved an average of 20% improvement compared to the TCP-only approach, and 25% compared to the QUIC-only approach.

These results confirm that the proposed approach provides a better overall QoE for all users in the different categories, while effectively using the available resources.

**Figure 8.** Heat map of average downloaded video representation per approach.

**Figure 9.** Downloaded content per approach.

#### **7. Discussion**

Based on the obtained results, we could observe that using our approach, or even only implementing HTTP/3, will greatly benefit DASH performance. Even though in the current paper we used a test server from Google's QUIC implementation, the benefits can be extrapolated when the official IETF implementation becomes available. As observed, the main advantage would be the avoidance of stalling events regardless of the adaption algorithm used by the DASH client, even if they share a bottleneck at the edge of the topology or the network resources are scarce. This is, of course, an inherent benefit of using QUIC protocol at the transport level, which was precisely designed to improve the performance on users with constraint performance in terms of congestion, and loss percentage [1]. Changing just the transport protocol by itself will not be enough, as some other factors will also have a detrimental effect on the QoE, e.g., taking into account user categories or sudden changes in the network. However, although the inclusion of additional control levels will overload the network devices, there have been some recent proposals—e.g., *Segment Routing* (SR) [39]—that can help alleviate the flow-table exponential increase. The overall idea of SR is to divide the end-to-end path into multiple parts called "segments", which will be updated based on pre-established policies, and it is a useful technique to relief the flow-table overload in SDN implementations, as shown in [24].

An additional inherent advantage, especially for mobile users, would be solving the mobility issue. The control and identification of the transmission in QUIC is done by using a *Connection ID*, making therefore possible to migrate a transmission from an end-point that had a change in lower-layers (UDP, IP) and continue their process, even the in case of a vertical (i.e., interface) or horizontal (i.e., network) handover [29], as can be observed in [6] where the authors explored this phenomenon.

It is also important to mention that, although we did not modify the default values of TCP or UDP for the experiments (e.g., congestion window), we believe that, by adjusting this parameter, as suggested by Kakhki et al. [40], the results can be further improved for the approaches that use QUIC.

It is worth noting that using HTTP over QUIC has some drawbacks in the current implementations. For instance, Google and Facebook stated that a wide-scale deployment would require about twice the amount of CPU to cope with the traffic load compared to current HTTP/2 [29]. Of course, this performance issue might be reduced by further optimizing the hardware and software (i.e., when the changes are applied in the kernel) to the UDP stack in a rather short time, since this area is not as explored as in the case of TCP.

Another point to consider is that, since a centralized-control approach—which combines information otherwise segregated (i.e., user, network, service, and application)—might be seen challenging to implement, there are already authors who presented feasible solutions. For instance, Liotou et al. [41] proposed a solution that allows effective communication between Video Service Providers (VSPs) and mobile network operators' (MNOs), which enables feedback towards a better network-aware video segment selection.

Finally, note that we only focused on a fully SDN-based infrastructure; the current Internet is made of different intertwined technologies, and therefore the interaction between those non-SDN domains needs further study.

#### **8. Conclusions and Future Work**

In this paper, we present SAND/3, an SDN-Assisted QoE control method for DASH over HTTP3, which comprises a multi-layer collaborative optimization at the user, application, transport, and networking levels. The proposed approach combines the best state-of-the-art technologies to support quality adaption and improve DASH performance on top of HTTP/3. We implemented the proposed architecture over an SDN-based infrastructure using ODL as the controller in a Linux environment. Moreover, we evaluated its feasibility using an emulated DASH client using different adaption bitrate algorithms. Preliminary results show that, by considering end-user categories to manage the video segment's traffic over QUIC, the overall QoE improves not only regarding the media throughput but also reducing the number of stalls, and the number of abrupt downshifts of video quality compared to current TCP-only and QUIC-only approaches. However, although the QUIC-only approach showed no significant improvements for DASH by itself, we can conclude that QUIC (and HTTP/3) features might help to improve DASH performance but it needs further support.

Future directions this work might go are as follows: First, we still need to test the impact of the approach in different scenarios, e.g., using other adaption algorithms or heterogeneous clients. Then, we plan to deploy the system using larger network environments, and test using real implementations of DASH systems and HTTP/3. Finally, it might also be interesting to look at the impact of recent approaches, such as Multipath QUIC [42], as it could combine the benefits of QUIC streams multiplexing over multiple paths and their impact on future Internet applications, in particular video streaming.

**Author Contributions:** Conceptualization, T.S. and L.G.; investigation, data curation, methodology, software and writing—original draft preparation, L.G.; and writing—review and editing, and supervision, S.I., T.A. and T.S.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

c 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **SmartX Box: Virtualized Hyper-Converged Resources for Building an Affordable Playground**

#### **Aris Cahyadi Risdianto, Muhammad Usman and JongWon Kim \***

School of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology (GIST), Gwangju 61005, Korea<EMAIL_ADDRESS>(A.C.R<EMAIL_ADDRESS>(M.U.)

**\*** Correspondence<EMAIL_ADDRESS>Tel.: +82-62-715-2219

Received: 19 September 2019; Accepted: 17 October 2019; Published: 30 October 2019

**Abstract:** In this paper, we present our proposals and efforts for building an affordable playground (i.e., miniaturized testbed) for Software-Defined Networking (SDN)-Cloud experiments by using hyper-converged SmartX Boxes that are distributed across multiple sites. Each SmartX Box consists of several virtualized functions that are categorized into SDN and cloud functions. Multiple SmartX Boxes are deployed and inter-connected through SDN to build multi-site distributed cloud playground resources. The resulting deployment integrates both cloud multi-tenancy and SDN-based slicing, which allow developers to run experiments and operators to monitor resources in a distributed SDN-cloud playground. It also describes how the hyper-converged SmartX Box can increase the affordability of the playground deployment. Thus, the analysis result shows the efficiency of SmartX Box for building a distributed playground by providing semi-automated DevOps-style resource provisioning.

**Keywords:** affordable playground; hyper-converged SmartX Box; distributed resources; multi-site and virtualized cloud; software-defined networking; DevOps automation

#### **1. Introduction**

#### *1.1. Background*

Aligned with worldwide Future Internet testbed efforts (e.g., GENI—Global Environment for Network Innovations [1], FIRE—Future Internet Research and Experimentation [2]), OF@TEIN (OpenFlow at Trans-Eurasia Information Network) project was started to build an OpenFlow-enabled testbed over TEIN infrastructure in 2012 [3]. Several experimentation tools were developed to support both developers and operators in using OF@TEIN testbed. Initially, a mixed combination of tools, ranging from simple web-/script-based to DevOps (Development and Operations) [4] Chef-based automated tools [5], were deployed over SmartX Racks. SmartX Rack consists of four devices: *Management & Worker node, Capsulator node, OpenFlow switch,* and *Remote power device*. Physically LAN-connected SmartX Racks were inter-connected by L2 (layer 2) tunnels, employed in Capsulator nodes as described in detail in this work [6]. However, SmartX Racks with multiple devices were subject to physical remote re-configurations, which are extremely hard to manage for distributed OF@TEIN Playground. Thus, from late 2013, a hyper-converged SmartX Box was introduced to virtualize and merge the functionalities of four devices into a single box [7]. The comparison between previously deployed SmartX Racks and newly deployed SmartX Box, is depicted in Figure 1. Finally, several hyper-converged SmartX Boxes were distributed deployed over nine Asian countries in 2015, as shown in Figure 2.

**Figure 1.** SmartX Rack versus SmartX Box Comparison.

**Figure 2.** OF@TEIN Playground.

Software-defined Networking (SDN) tools assist developers and operators to prepare the experiment environment in OF@TEIN, by enabling networking resources (e.g., switches and Flowspaces [8]) preparation. Similarly, cloud management software can cover computing resources (e.g., VMs) preparation. Thus, the combination of distributed SDN and cloud testbeds ready to provide scalable and flexible computing resources with enhanced networking capability. However, the seamless integration of SDN-enabled and cloud-leveraged infrastructure is a very challenging task, due to open and conflicting options in configuring and customizing resource pools together. Therefore, we should carefully provision all resource configuration aspects such as multi-site distribution, virtualized resource slicing (i.e., isolation), and multi-tenancy support while considering hardware deployment for SDN and cloud integrated testbed.

As mentioned above, this paper proposes the concept of hyper-converged SmartX Box that can easily accommodate virtualized and programmable resources (i.e., OpenFlow-enabled virtual switches and OpenStack-leveraged cloud VMs) to build the OF@TEIN SDN-enabled multi-site clouds playground (i.e., miniaturized testbed). The collection of OpenFlow-based virtual switches (i.e., OpenvSwitch [9]) is providing SDN capability, which is controlled by both developers and operators SDN controllers. Simultaneously, OpenStack-leveraged VMs (i.e., working as virtual Boxes) are effectively managed by OpenStack cloud management [10]. However, the actual design and implementation of SmartX Box are continuous to evolve for supporting new experiment over OF@TEIN Playground.

#### *1.2. Motivation and Related Work*

It is well known that the service-centric networking model to provide higher-level connectivity and policy abstraction is an integral part of cloud-leveraged applications. The emerging SDN paradigm can provide new opportunities to integrate cloud-leveraged services with enhanced networking capability through deeply programmable interfaces and DevOps-style automation. Several SDN-based approaches have been proposed to provide virtualized overlay networking for multi-tenancy cloud infrastructure. For example, Meridian proposed an SDN controller platform to support service-level networking for cloud infrastructure [11]. Similarly, CNG (Cloud Networking Gateway) attempts to address multi-tenancy networking for distributed cloud resources from multiple providers while providing flexibilities in deploying, configuring and instantiating cloud networking services [12]. As a prototype of implementation, the large-scale deployment of GENI Racks over national R&E (research and education) network is also moving towards a programmable, virtualized, and distributed collection of networking/compute/storage resources, a global-scale "deeply programmable cloud". It satisfied research requirements in a wide variety of areas, including cloud-based applications [1]. Another effort from the EU, known as "BonFIRE", is a multi-site testbed that supports testing of cloud-based distributed applications, which offer a unique ease-to-use functionality in terms of configuration, visibility, and control of advanced cloud features for experimentation [13].

Aligned with converged SDI (Software-defined Infrastructure) paradigm [14], the SDN and cloud testbeds should continuously support for the new type of technologies and experiments. To guarantee the usability and continuity of the testbed, the testbed deployment needs to consider the three following aspects. First, the resources should be open without any limitations due to proprietary vendor software or hardware implementation. Second, the resources should be conceptually and reality agile to adopt new technologies or match experiment requirements. Third, the cost and number of physical (i.e., hardware) resources to be deployed should be very minimum for each site of the playground. It only requires a minimized budget (i.e., leveraging low-cost commodity hardware). Please note that leveraging the power of open-source software (e.g., KVM hypervisor [15], LXC Linux container [16], Open virtual switch [9], and others) is also important to support distributed resources centralized management.

Since the cloud computing model is massively adopted for computation infrastructure deployment, it should consider an affordability aspect of the cloud deployment model. In 2017, Ta A.D. [17] tried to address the adoption of a cloud computing framework for developing countries by leveraging hyper-converged servers as virtual computing infrastructure. Moreover, it has gained more acceptance since the networking function (i.e., switch) was extended into a virtualization layer, which allows the creation of multiple virtual switches in the Linux-based server [9]. Then, in 2016, a software-defined framework, called *OpenBox*, tries to address NFs (network functions) deployment by decoupling between NFs control-plane and data-plane that is very similar to SDN solutions [18].

#### *1.3. Aim and Contributions of the Paper*

We tried to align with the above testbed deployment, but unfortunately, there are some technical and environment gaps with all those testbed efforts. GENI [1] covers extensive testbed features and functionalities for a different type of experiments while considering increasing the performance with powerful hardware. However, they are not focusing on distributed over heterogeneous infrastructure and the cost of deployment due to the good support of infrastructure and funding sources. FIRE [2] is similar to GENI, but with a less extensible feature because they have much more focus on smaller testbed but also with good performance. They are a little bit concerned about the distributed over heterogeneous infrastructure due to different conditions and policies among many Europeans countries. PlanetLab [19] does not consider features and performance due to the simple requirement of virtual resources with basic networking. However, it is a large-scale distributed testbed because it is deployed over a thousand sites with different network infrastructures. PlanetLab does not concern about the deployment cost because it is supported by a large research community in adding new hardware resources as a new testbed site.

In summary, it is not possible to directly apply their deployment approaches into our environment due to several reasons such as limited funding, level of research interest, and network infrastructure support. Therefore, we want to provide extensive experiment features/functions over a reasonable number of distributed sites over heterogeneous infrastructure. However, at the same time, we need to reduce the number of hardware for each experiment by providing hyper-converged box-style resources. In other words, we want to increase the testbed capability while keeping the cost of the deployment as low as possible. Thus, since 2013, OF@TEIN had considered those mentioned aspects by changing the initial rack-style resource deployment from GENI testbed into box-style, hyper-converged, and server-based resources for an affordable playground across developing countries in Asia region [7]. The main contributions of this paper are:

#### **2. An Affordable Playground**

#### *2.1. An SDN-Enabled Multi-Site Clouds Playground*

A *Playground* is defined as a miniaturized and customizable testbed that is easy to build and operate for various research experiments by a tiny-size DevOps-style team of developers and operators. Mainly, we focus on establishing a multi-site playground infrastructure where playground resources are physically distributed across multiple geographical sites but logically inter-connected with each other to offer a unified shared pool of resources. A tiny-size team of people (i.e., operators) should provision and control the multi-site isolated resources, which is openly accessible by a group of people (i.e., developers) from all involved sites. As depicted in Figure 3, the proposed multi-site playground has several vital entities, which are discussed below:

1. **Playground Sites with Hyper-converged Boxes**. When playground developers want to perform their experiments, they can dynamically acquire dedicated resources from the pool of multi-site resources. For the customizable (i.e., software-defined) playground, the resource infrastructure of the multi-site playground should be composable (e.g., programmable) in terms of computing, storage, and networking types of resources. By leveraging the growing popularity toward

hyper-converged appliances that integrate computing, storage, and networking resources into a single box-style entity, we can enable the multi-site resource pool that ready to be customized and scaled-out without the manual intervention of playground operators. Thus, in our approach, each playground site is equipped with a box-style of hyper-converged resources, denoted as *SmartX Box*, the hyper-converged box-style resource that should be useful in supporting the required composability by comfortably accommodating virtualized and programmable resources. Multiple types of SmartX Box should be designed and deployed for different purposes, such as SD-WAN, SDN-enabled clouds, and access extension support. However, the critical aspects of those boxes design are similar: *open, agile*, and *economic* resources.

Furthermore, the SmartX Box can be divided into three main abstractions, which are *box, function,* and *inter-connect*. The *box* represents all the server-based hardware that runs Linux as a baseline open-source operating system for other software-based functions. The *function* represents virtual functions (e.g., virtual machine, virtual switch, or virtual router), which are implemented by using a set of open-source software. Finally, the *inter-connect* is representing the path/link between functions or boxes which include the tunnel-based overlay networking because playground sites are spread over the heterogeneous underlay network infrastructure.

2. **Playground Tower with Centers**. To satisfy the dynamic requirements of playground developers on diversified functionalities over distributed but miniaturized resource pools, the proposed playground should integrate the emerging technology paradigms such as the SDN, cloud computing, and Internet of Things (IoT). However, the SDN-enabled multi-site clouds combination brings new complexities for the playground operators, since multi-site clouds resources, connected via SDN-based networks, demand various software-based DevOpsautomation tools to build, operate, and use automatically. Thus, we propose the concept of *Playground Tower*, which provides a logical space-like abstraction in a centralized location, which leads the operation of the multi-site playground by following the concept of "monitor and control" tower. From the tower, the DevOps team can enjoy a panoramic view of playground resources that are distributed over underlay networks, and quickly manage and use those resources for their experiments.

The playground tower systematically covers various functional requirements of operating a multi-site playground by employing several centers, also depicted in Figure 3. First, Provisioning Center (P-center) is responsible for remote installation and configuration of multi-site playground resources. Visibility Center (V-center) covers playground visibility and provides panoramic visualization support. Orchestration Center (O-center) handles the management level issues with the assistance of controllers (e.g., SDN/cloud controllers). Thus, several software-based tools are used and developed by leveraging open-source software. For example, to provision SmartX Box, P-Center provides an automation framework such as Chef and MaaS. To continuously operate by re-configuring the playground resources, O-Center provides a set of interfaces (e.g., CLI, API, Web UI) to meet the varying requirements of playground developers. Finally, to monitor the playground resources and traffic flows, V-Center provides playground visibility data in an accessible format for visualization and analysis.

**Figure 3.** An SDN-Enabled Multi-Site Clouds Playground.

#### *2.2. Affordable Playground with SmartX Box*

As mentioned earlier, to specifically address the affordability of multi-site playground, the resources should be open without any configuration/monitoring limitations, agile to adopt new technologies or match experiment requirements, and economics by the deployment of low-cost commodity hardware. Open-source software in converged SDI with SDN, cloud, and Network Function Virtualization (NFV) integration, and the support of open-source hardware are the main drivers of the development of hyper-converged box-style resources.

The concept of box-style hyper-converged resources is shown in Figure 4, which can introduce a unique design of an affordable multi-site playground. The critical design of hyper-converged box-style resource is open, agile, and economic resources. The term box is adopted from a white box (i.e., clear box or open box) that has understandable and controllable subsystems, so it is easy to develop/test software on it without limitation on a vendor-specific feature. The box is open for supporting any server-based hardware because it is fundamentally characterized by software-based components/functions composition and implementation. It is agile to provide a different type of experiment for matching with new upcoming technologies and requirements. It is economical to allow any low-cost commodity server to be used to increase affordability in distributed deployment and operation. However, the actual design of the box is evolved, which include: simplification of the physical hardware specification, components design changes, and modular software implementation.

Unfortunately, provisioning and operating distributed hyper-converged boxes in heterogeneous physical infrastructures quite challenging since it is subjected to different performance parameters (e.g., networks speed and power stability). Also, the independency between multi-domain underlying infrastructure operators (e.g., access and security policies). Consequently, it is hard to maintain the continuous operation of all the boxes. To make those boxes design turnkey simple, we should concentrate on minimizing the requirement and consumed time for provisioning the boxes with these following strategies:

1. *Heterogeneous hardware*: No specific hardware requirements or specific brand/vendor supports, all hardware with acceleration support for virtualization, and multiple network interfaces support for specialized connections can be used.

#### *2.3. SmartX Box: Design and Implementation*

#### 2.3.1. SmartX Box Abstraction

As discussed above, we adopt the hyper-converged box-style resource, called *SmartX Box* [7], as the main building block for our OF@TEIN Playground. The proposed SmartX Box illustrated in Figure 5, as an abstracted format.

The SmartX Box abstraction tries to align the SDN/NFV/cloud integration of SDI with the compute/storage/networking integration of hyper-converged box-style resources. Cloud can provide economics and scalable computing/storage resources without compromising the associated performance, availability, and reliability. SDN provides flexible networking support for highly virtualized computing/storage resources, which is not possible with legacy networking schemes. NFV is assisting SDN by deploying virtualized network functions in the box-style hyper-converged resources of cloud data centers [21]. Virtual monitoring is used to collect box-related data for monitoring and troubleshooting purposes, which one of them is described in our previous work [22]. Moreover, the centralized orchestration for multi-tenant cloud data centers and NFV-assisted SDN infrastructure can provide the simplified orchestration of SDI-ready playground with the slice-based network virtualization support. Thus, by considering the above features, SmartX Box is designed to be ready for supporting a wide range of research experiments. Also, by merging all the required functionalities into the hyper-converged SmartX Box, it is easier to realize the scale-out capability of the playground by simply adding hyper-converged SmartX Boxes to increase the resource capacity of the playground.

**Figure 5.** Hyper-converged SmartX Box: Abstraction to match SDN/NFV/Cloud integration.

To support the flexible remote configuration, each hyper-converged SmartX Box requires dedicated and specialized connections for P/M/C/D (power, management, control and data), which are explained in the next section [7]. However, besides those connections, there are no specific hardware requirements for hyper-converged SmartX Boxes. Therefore, any commodity hardware with reasonable computing, storage, and networking resources can be used. The total amount of hardware resources only affects the capacity (e.g., the total number of VM instances per flavor types) in specific boxes, sites, and regions. However, it is essential to consider the hardware acceleration support for virtualization and networking.

#### 2.3.2. Virtualized SDN-Enabled Switches and Cloud-Leveraged VMs

As described above, the conversion from SmartX Racks to SmartX Boxes are completed to manage better the distributed multi-site cloud-based services on the top of SDN-enabled inter-connect capabilities [7]. Thus, the actual design of SmartX Box needs to consider and balance both cloud and SDN aspects carefully. The OpenStack [10] Cloud can provide VM instances and basic networking options for diverse tenants. For SDN, several instances of virtual switches (based on Open vSwitch [9]) are provisioned while allowing users/developers to share them simultaneously. We arrange the SDN and cloud relevant functions inside a single hyper-converged SmartX Box, as shown in Figure 6. SDN-related virtual functions consist of several virtual switches with different roles, e.g., creating developers networking topology, inter-connecting OpenFlow-based overlay networking, and tapping flows for troubleshooting. Also, cloud-related functions are placed to include VM instances for cloud-based applications and to support external connections to VMs.

The inside view of SDN-/cloud-related functions is depicted in Figure 7. First, several SDNenabled virtual switches are placed and matched with its functionalities: *brcap* for capsulator (encapsulate OpenFlow packets through an overlay tunnel), *br1* and *br2* for users/developers switches, and *brtap* for tapping purpose (capturing packets for troubleshooting as described in [11]). Cloud-related VM instances (a.k.a., virtual Boxes: vBoxes) are managed by KVM hypervisors, which is controlled by OpenStack Nova with specific flavors and images. Additionally, virtual switches (i.e., *br-int, br-ex*, and *br-vlan*) and user-space virtual router are configured by OpenStack Neutron to provide required connectivity to cloud VM instances.

**Figure 6.** Hyper-converged SmartX Box Design for SDN and Cloud Integration.

**Figure 7.** Hyper-converged SmartX Box Virtualized Cloud and SDN Components.

#### *2.4. Semi-automated Resource Provisioning*

Deploying hyper-converged SmartX Boxes in heterogeneous physical (i.e., network topology) infrastructures is very troublesome since it is subject to different performance parameters and independent network administrative domains. As mentioned in the previous section, it is tough to keep the sustainable operation of all the boxes. Thus, a set of automated provisioning tools is developed to minimize the consumed time for provisioning (i.e., installing and configuring) all the SmartX Boxes with pre-arranged P/M/C/D connections. The P (Power) connection is used for power up/down SmartX Box. The M (Management) connection is mainly used for managing SmartX Box by the operator. Also, the C (Control) connection is used to access and control the SDN-/Cloud-related functions (i.e., virtual switches and VMs) by the developer. Finally, the D (Data) connection is used for any data-plane traffic that includes inter-connection traffic among multiple SmartX Boxes. Also, the automated provisioning tools are controlled by a centralized P-Center inside the Playground Tower, which has full access to all distributed hyper-converged SmartX Boxes.

First, to automate the provisioning of the SDN-enabled virtual switches, *ovs-vsctl* high-level management interface for OpenvSwitch is used. Please note that *ovsdb* (OpenvSwitch database) protocol is also used for the centralized configuration of the OpenvSwitch database inside each SmartX Box. The provisioning task includes the creation of virtual switches, the configuration of virtual ports/links and overlay tunnel inter-connections, and the control connection of virtual switches and SDN controllers. Next, open-source OpenStack cloud software has special installation and configuration tools, called *DevStack*, which can support several modes of OpenStack configurations with selected operating systems (e.g., Ubuntu, Redhat Enterprise Linux, and CentOS) [23]. For OF@TEIN Playground, we customize DevStack provisioning template to facilitate multi-regional OpenStack cloud deployment with centralized management and authentication.

The overall implementation of semi-automated provisioning for SDN/cloud-enabled SmartX Box is depicted in Figure 8. It is started with a clean-up of previous software installation and checking/upgrading the operating system. Then it is followed with box installation to install/configure OpenStack cloud and OVS components, and finally API-based tools to verify function installation/configuration.

**Figure 8.** Implementation of Semi-automated Provisioning for SDN/cloud-enabled Hyper-converged SmartX Box.

#### *2.5. SDN and Cloud Centralized Control*

Cloud-related virtual functions are inter-connected through SDN-related virtual functions to provide end-to-end communication for multi-site cloud-based applications. The cloud-related virtual functions are controlled centrally by open-source *OpenStack* Cloud management and orchestration software [10]. The SDN-related virtual functions are also centrally controlled by open-source SDN controller such as ODL (Open Daylight) [24] and ONOS (Open Network Operating System) SDN Controller [25]. OpenStack *Keystone* provides centralized user authentication and authorization. OpenStack *Nova* and OpenStack *Neutron* can create VM instances and to provide enhanced connectivity, respectively. The SDN controller (i.e., ODL) manipulates the flow table entries of SDN-enabled virtual switches to enable the flexible steering of inter-connection flows among various functions located in different cloud sites. Both cloud and SDN control software are required to mix and match the configurations so that we can ensure the consistent connections between cloud VM instances. Remember that the main challenge is how to accommodate cloud-based multi-tenancy virtual networks (e.g., flat, VLAN, or tunneled network) for OpenFlow-based network slicing (e.g., IP subnets, VLAN IDs, and TCP/UDP ports). Eventually, VLAN-based multi-tenancy traffic

control (e.g., tagging, steering, and mapping) is chosen to integrate tenant-based and sliced-based networking in SDN-enabled multi-site clouds playground.

By manipulating both OpenStack and ODL SDN controller, a VLAN-based multi-tenancy traffic control is implemented as follows. First, we place VMs in two cloud regions and prepare the connectivity for these VMs. These VMs are tagged by OpenStack Nova with a specific tag ID. Second, OpenStack Neutron automatically maps the tag into VLAN ID that matched with SDN-based slice parameters. This matching allows inter-connection flows for VMs to be steered by the developer's SDN controller, supervised by *FlowVisor* [8]. The SDN-based flow steering inserts flow table entries according to the particular incoming and outgoing ports in the developer's virtual switches, where several ports are mapped to other cloud regions/sites. Finally, based on the destination site, it maps to a specific tunnel interface that is pre-configured by the SDN controller of operators. The example control of SDN and cloud components inside the SmartX Box for connecting several VMs from different tenants across multiple sites, as depicted in Figure 9.

**Figure 9.** SDN and Cloud Control (Tag, Steer and Map) inside the SmartX Box.

#### **3. Cost and Efficiency Analysis**

#### *3.1. TCO Analysis of Conversion from SmartX Rack into SmartX Box*

To estimate the reduction of TCO (Total Cost of Ownership) from SDI based on the hyperconverged resources, we can observe these two reports. First, typical three-year server TCO from IDC (International Data Corporation) [26] in 2017, which defined the composition of several costs such as hardware, software, staff training, outsourced cost, user productivity, and staffing (manpower), as depicted as Figure 10a. Second, the value of SDI to reduce the total TCO of 10,000 OS instances which is released by Intel [27], as depicted in Figure 10b. Where manpower efficiency improves up to 60%, software savings up to 70%, hardware reduction up to 20%, and other reduction (infrastructure and energy) decreases up to 20%.

Based on both of the analysis reports, we can estimate the total saving of our conversion effort from SmartX Rack into SmartX Box hyper-converged resources. In summary, as depicted in Figure 11a, the TCO for SDI-ready box-style hyper-converged resources is described as follows. The hardware-related project cost reduced to 5.6% due to single box type of deployment, the software cost decrease to 2.1% due to open-source software adoption, and the most important is the cost for playground operators go down till 36%. The overall TCO saving of SmartX Rack to SmartX Box conversion is around 30.3%. If the graph is normalized into 100% of the pie chart, so we can produce

the chart as depicted in Figure 11b. In conclusion, with the same amount of project budget, more cost can be allocated for staff training (e.g., more effort for the research) to increase the quality of operators/developers and also consider more compensation on the experiment downtime or the developers/researchers productivity.

**Figure 10.** (**a**) Server-based 3-years TCO Composition [26] and (**b**) SDI Value to reduce 5 years TCO [27].

**Figure 11.** (**a**) TCO saving with hyper-converged infrastructure and (**b**) Normalized TCO with hyper-converged infrastructure.

#### *3.2. The Efficiency of Semi-Automated Provisioning*

To facilitate the agile deployment of OF@TEIN Playground, both SDN-/cloud-related tools are used for automated provisioning of hyper-converged SmartX Boxes. It is aligned with the recent employment of DevOps automation since the OF@TEIN Playground is operated by a limited number of operators and becomes easily uncontrollable as it spans across multi-domain inter-connected networks beyond the privileges of playground operators. Thus, by using DevStack-based OpenStack deployment and ovs-vsctl or ovsdb protocol for virtual switch provisioning, we can simplify the semi-automated provisioning of hyper-converged SmartX Boxes. Also, REST APIs of the ODL SDN controller is used for automated flow insertion, flow modification, and flow deletion. In summary, most of the provisioning steps are automated, except for manual handling of critical tasks such as DevStack-based OpenStack service restart and VXLAN tunnel checking/recovery.

The duration of the whole process of semi-automated provisioning depends on the Internet connection speed due to the online OS upgrade and online OpenStack software installation. However, it takes less time for new box clean installation (including box restart), because previous software clean-up and OS upgrade is not required. Moreover, re-configuring pre-installed SmartX Box is much faster with "offline mode" enabled because online software/package copy from Ubuntu and OpenStack repositories are not required. Figure 12a shows the semi-automated provisioning results, which take approximately 50 min for fully upgrading a SmartX Box with network connection up to 300 Mbps. Thus, it takes around 6 h for the slowest network connection which less than 10 Mbps. However, it takes only around 20 min (including box restart) for provisioning without cleaning up the previous installation and upgrading the operating system. While Figure 12b justifies the longest completion time is for upgrading the operating systems and restarting the SmartX Box. The clean-up task is negligible because it is less than one minute, and then the installation task is reasonable for such a customized configuration. Moreover, it takes less than 10 min to recover or re-configure SmartX Box with "offline mode" [7].

**Figure 12.** Semi-automated Remote Provisioning Result for SmartX Box: (**a**) Provisioning time for the different network connection speed, and (**b**) Completion time for different provisioning task.

#### **4. Playground Deployment Verification**

#### *4.1. Distributed Deployment of SmartX Boxes for Building OF@TEIN Multi-Site Playground*

Multiple hyper-converged SmartX Boxes are deployed on existing hardware of OF@TEIN Playground with a special focus on adding the open-source OpenStack cloud-management software. The OF@TEIN Playground relies on the heterogeneous physical underlay infrastructure across multiple administrative domains. Thus, the multi-regional OpenStack cloud deployment is currently investigated as the deployment option because it gives simple and common configuration for all regions (i.e., SmartX Box sites). It also supports an independent IP addressing scheme and has less dependency on the overlay networking among regions. Despite multi-regional independent cloud deployment, the OF@TEIN Playground supports an integrated cloud management interface by deploying web-based OpenStack Horizon UI and the centralized account/token authentication from OpenStack Keystone. The resulting OpenStack multi-regional cloud deployment is illustrated in Figure 13.

**Figure 13.** OpenStack Multi-Regional Configuration in OF@TEIN Playground.

Next, the OF@TEIN Playground is enhanced with multiple mesh-style inter-connections of NVGRE/VXLAN overlay tunnels, along with a unique flow-tapping virtual switch [22]. The OpenStack multi-region deployment is modified to build an SDN-enabled multi-site playground where inter-VM connectivities between cloud VMs are used by leveraging OpenFlow-enabled data planes. The data planes are programmed and controlled by the centralized SDN controller, co-located with centralized cloud management.

#### *4.2. Example of Experiment with SDN-Enabled Multi-Site Cloud Playground*

This example shows both aspects of experiment preparation in SDN-enabled multi-site clouds playground to provision resources in both OpenStack and ODL SDN controller. First, a VM in one of the cloud regions (i.e., playground sites) is prepared, including the basic connectivity for this VM using the first VM virtual NIC (Network Interface Card). It is connected to a control network, called "private", for providing VM remote access from an external network. Then, second VM virtual NIC is connected to a data network, called "datapath01", which is automatically mapped into pre-configured VLAN ID that matched with SDN-based slice parameters. This mapping allows the flow from this VM to be steered by the developer's SDN controller, supervised by FlowVisor [8]. Another VM in the second region also is prepared with the same steps as depicted in Figure 14. Next, the SDN-based flow steering that is leveraging the ODL SDN controller inserts flow table entries in developers' virtual switches from/to those VMs to/from pre-defined ports that are already mapped to other cloud regions/sites. Those pre-defined ports are mapped to a tunnel interface from the originating site into the designating site that is controlled by the operators' SDN controller of operators. The example of the steps is depicted in Figure 15.

**Figure 14.** OpenStack VMs Preparation and Configuration.

**Figure 15.** Flow Configuration in OpenDaylight Controller.

#### *4.3. Multi-site Playground Visibility and Visualization*

Resource-level monitoring and visualization are important operation activities for OF@TEIN playground. The SDN and cloud components implementation for hyper-converged SmartX Boxes provides diverse physical and virtualized resource combinations while at the same time brings new complexities for monitoring and visualization. The SDN-cloud-enabled playground demands considerably different resource-level visibility solutions from traditional networking testbed. A distinctive, component-based, data-oriented approach is required for resource-level visibility of distributed OF@TEIN physical and virtual resources. By integrating open-source software/tools, we set up a unique resource-level visibility solution, which is focused on operation data collection from multiple sources and interactive large-scale visualization [28]. Resource-level visibility data is collected in nearly real-time to help the operators for monitoring the status of the resources by using a single and unified visibility user interface. Large-scale visualization (i.e., network tiled display leverages SAGE Framework [29]) enables the simultaneous visualization of the multiple and different types

of visualization sources (e.g., web-based UI, remote desktop, secure shell), as depicted in Figure 16. It allows OF@TEIN operators to manage the resources and developers to execute the experiment while keeping an eye on the playground resources.

**Figure 16.** Multi-site playground resources visibility over network tiled display.

#### **5. Conclusions**

This paper gives comprehensive discussions of the unique concept and design of SDN-enabled multi-site clouds playground with hyper-converged box-style resources for an innovative and diverse research experiment. OF@TEIN Playground is successfully provisioned as an affordable SDN-enabled multi-site clouds playground with distributed SmartX Boxes deployment for integrated SDN and cloud experiments. We believe that an open, agile, and economics box-style hyper-converged resources can provide a larger scale of an affordable and sustainable playground for diverse experiments with a wide variety of application areas.

**Author Contributions:** Conceptualization—J.K. and A.C.R., Supervision—J.K., Investigation—A.C.R., Software and Validation—A.C.R., and M.U., Visualization—M.U., Writing—A.C.R., and Reviewing and Editing—J.K. and M.U.

**Funding:** This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant of the Korea Government (MSIT) (No. 2015-0-00575, Global SDN/NFV Open-Source Software Core Module/Function Development, and No. 2017-0-00421, Cyber Security Defense Cycle Mechanism for New Security Threats). This research is also partially supported by Asi@Connect grant of the Asi@Connect-17-094 (No. IF050-2017), OF@TEIN+: Open/Federated Playground for Future Networks.

**Acknowledgments:** The authors would like to acknowledge gratefully to OF@TEIN Community who has been supported the deployment and operation of OF@TEIN playground for the past few years. Especially for playground operators and NREN administrators who help us to maintain the stability and connectivity of the resources to be able continuously used by playground developers/users. Hopefully, the collaboration amongst all the involved parties can be continued to enhance and extend our proposed concept and effort in the next few years.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript:

#### **References**

c 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **Phantom: Towards Vendor-Agnostic Resource Consolidation in Cloud Environments** †

**Aaqif Afzaal Abbasi 1, Mohammed A. A. Al-qaness 2, Mohamed Abd Elaziz 3, Ammar Hawbani 4, Ahmed A. Ewees 5, Sameen Javed <sup>6</sup> and Sunghwan Kim 7,\***

Received: 24 September 2019; Accepted: 15 October 2019; Published: 18 October 2019

**Abstract:** Mobile-oriented internet technologies such as mobile cloud computing are gaining wider popularity in the IT industry. These technologies are aimed at improving the user internet usage experience by employing state-of-the-art technologies or their combination. One of the most important parts of modern mobile-oriented future internet is cloud computing. Modern mobile devices use cloud computing technology to host, share and store data on the network. This helps mobile users to avail different internet services in a simple, cost-effective and easy way. In this paper, we shall discuss the issues in mobile cloud resource management followed by a vendor-agnostic resource consolidation approach named Phantom, to improve the resource allocation challenges in mobile cloud environments. The proposed scheme exploits software-defined networks (SDNs) to introduce vendor-agnostic concept and utilizes a graph-theoretic approach to achieve its objectives. Simulation results demonstrate the efficiency of our proposed approach in improving application service response time.

**Keywords:** cloud computing; management; middle box; placement; resource; SDN; vendor-agnostic; virtual machine; VM

#### **1. Introduction**

Mobile-oriented future networks [1–3] are gaining tremendous importance in the field of computing and networking industry. With the advent of wireless networking technologies, the wide-scale use of smartphone devices and the World Wide Web is being shifted rapidly from static to mobility-based solutions. For example, mobile service users will exceed two billion users [4,5]. Such drastic changes are influencing the way IT concepts used to act and behave in the past.

However, the original idea of the internet was not based on mobility-based services. In other words, it can be said that the original idea of the internet was meant for fixed hosts instead of mobile hosts. So with the emergence of mobile technology, various patch-on protocols were introduced to support mobile environments e.g., Mobile IP [6,7] and its variants. However, patch-on technology based solutions also have their limitations.

In an environment based on mobile computing, support for mobility is a vital requirement rather as an add-on feature. The legacy network protocolsmainly focus on fixed hosts. In terms of usability, legacy protocols often describe mobility as an additional functionality of a device. This behavior leads to the creation of protocols based on mobility and is related to the modified versions of TCP/IP protocols suite [8–10]. With these trends of mobility-awareness in the protocols, unexpected degradation of performance, such as overuse of proxy, triangle routing, etc., is induced.

Mobile devices, including tablet PC or smartphones, are increasingly becoming an important part of our lives as a vital and easy sources of communication tools that are not bounded by the elements of time and space [11]. Mobile users utilize multiple mobile-based services by using different kinds of mobile apps. These apps are hosted on remote servers through wireless networks. The fast growth witnessed in mobile computing is a very prominent factor in the IT industry. It also influenced the commerce industry. However, with this fast growth of mobile devices, we are also facing numerous challenges such as computing resources car city, bandwidth allocation, storage and retrieval challenges and battery life time. Therefore, it can be safely said that the limitations of computing resources greatly hinder the betterment of computing services quality.

Cloud computing has been accepted as the infrastructure of next-generation networks [12]. Cloud users can benefit through cloud infrastructure by using various services (such as storage and services hosting), platforms (operating systems, middleware and related services) and software (applications) supported by cloud-enabled services like Amazon, Salesforce or Google at low prices. Furthermore, cloud computing enables its users to broadly utilize the resources on a pay-per-use policy [13]. By using such mobile applications, users can benefit from various cloud computing functions. With the rapid growth of mobile apps and better support for cloud-oriented services, the term mobile cloud computing is introduced. Mobile cloud computing is basically an integration of cloud computing in the mobile environment. With the advent of mobile cloud computing, mobile users are taking advantage of new type of services and which facilities them in fully utilizing cloud computing services.

Mobile cloud computing [14,15] has the potential to transform the large arena of the IT industry. This will help in making software and hardware-oriented services more accessible and attractive [1]. One of the primary objectives of cloud computing is to provide computing and storage services at low and reasonable costs. This happens by sharing many resources between different users. The actual provisioning of such services at a low process depends on how efficiently resources are utilized in the cloud. A typical mobile cloud computing infrastructure is illustrated in Figure 1.

Cloud vendors can offer special hardware and particular software techniques for the provisioning of reliable services at a high price. Later, these reliable services could be sold to users by signing terms under certain clause or service level agreement. Nowadays, the cloud computing industry is using the term "no single point of failure". But the single point of failure often occurs when a single cloud service provider is hosting all these solutions [16]. It is worth mentioning that a true vendor-agnostic solutions will not only an open source technology solution (software) but will be accepted only if it is being operated on a vendor neutral hardware (by using off-the shelf, bare-metal/SDN-enabled devices) etc.

On the other hand, software-defined clouds (SDCs) make use of SDNs in order to create a programmable and flexible network by separation of functions for control plane and data plane. The reason for choosing SDNs in data center resource management is their simplicity and control over data center infrastructure. The idea of vendor agnostics through SDNs in data centers is implemented by the Open Flow with the decomposition of traffic control authorization to different parts [17–19]. The controller element is a powerful manager of the network that is processing information related to flows. Open Flow switches include basic functions like receiving, forwarding or looking up in a data traffic table. By using OpenFlow, routing is not confined to a Media Access Control (MAC)

address or IP address. This basically helps in the determination of paths with the parameters of high security, low packet loss or low delay and also helps in maintaining the fine-grained scrutiny policies for various applications.

**Figure 1.** Mobile-oriented network infrastructure for cloud computing.

The main purpose of this paper is to contribute towards the field of SDCs by investigation of a major problem in cloud computing i.e. the placement of virtual machine (VM). VM placement is very important in datacenters and has been studied extensively, particularly for their use in the software-defined domain [20–22]. In a cloud environment, VMs are a key player as they provide the flexibility claimed by cloud service providers. Figure 1 presents the layout of VM in a distributed environment. When a computing service admitted into the cloud system demands for higher computational resources, then VM management plays a very important role. VM management helps in balancing the system constraints and loads [23,24]. Its main purpose is to retain user service satisfaction level. There are numerous VM placement challenges. Traditionally, the techniques for VM placement only focus on the resource allocation efficiency. Network research related to cloud resource management often focus on placement of VMs in data center environments. A vast number of VM placement techniques propose a solution based on available network resources [25]. This paper presents a relatively simpler approach for VM placement in the SDC environment. The concepts presented in this paper are related to the state-of-the-art technologies such as server and network resource utilization, software-defined networks, VM placement/ mapping and software-defined middle box networking. The paper presents a combination of these technologies for resource management in cloud environments.

The rest of the paper is further organized as follows. Section 2 discusses the related work, Section 3 presents the research allocation and mapping discussion in cloud environments, Section IV presents the mathematical modeling. In Section 4 we perform the performance evaluation. Finally, Section 5 concludes the paper.

#### **2. Related Work**

In cloud environments, resource sharing must be done in a way that a user's application requirements must not influence other user applications. Resource sharing must be done in such a way that these are secured and privately available [26]. VMs are acquired by applications on cloud infrastructure when needed. However, for cloud tenants, VM acquisition is a challenge. It is due to the limitations in cloud system's granularity and limitations in VM control and placement.

Data-intensive applications [27,28] frequently communicate with data centers. That is why the data traffic transmission of these applications is quite large. This results in network performance degradation and higher system overheads. VM placement strategies often use VM consolidation and reallocation techniques to solve vendor lock-in issues. These issues greatly influence network performance. In vendor lock-in issues, the users' traffic volumes can face delays. This ultimately leads to VM placement issues. The VM placement problem with traffic awareness [5,6] was proposed for solving these problems through network optimization based strategies.

In cloud environments, VMs follow certain patterns in accessing network resources. Research studies conducted in [29,30] involves a large number of CPU traces from different servers. It demonstrates that the demand traces are mostly in correlation and follow a periodic behavior. However, the concept of statistical multiplexing exists due to varying workloads. Data packet behavior for these applications relies primarily on the idea of exploiting possible correlations in VMs. Other approaches to vendor lock in issues include continuous monitoring of all VMs running on the network by using various VM measurement heuristics [31].

Current VM placement strategies have been extended for the inclusion of other data center infrastructure aspects such as network storage and network traffic. In a cloud infrastructure, all deployed VMs typically show a dependency on network traffic. The best optimization strategy to address their consolidation challenges is by hosting them on the nearest available physical machine [32,33]. Interestingly, network topology and data center design has a major impact on the selection of placement for traffic optimization targets [34]. Similar dependencies often occur for VMs and storage resources with different user requirements. In this situation, applications needing greater I/O performance can be moved closer to the storage locality.

Different vendors provide tools for resource-management functions. These tools include a wide range of applications. This includes system-level monitoring tools to application-level deep packet tracers and monitors.  Therefore, a vast-scale adoption of these tools will not only burden the network features, but will also influence underlying network resources (including virtual and infrastructure resources). In view of the above, a vendor-agnostic approach is used in [35,36] which proposes VM placement on a physical machine with the least data transfer time with respect to network bandwidth usage. However, within the datacenter premises, the data transmission rate is better due to wired communication. Therefore, users of these services expect high-qualityenterprise-level services rather than services offered by mobile devices with limited resources. Although not having enough tenant support for the VM migration, the cloud services provider have high control functions over all VMs locations. The manipulation of VMs can be performed by scaling in and out of physical resources.

Network support for tenant-controlledVM placement is difficult. An API-based SDN-enabled solution for these issues helps in providing a clean interface to the network administrator and is widely used in SDC environments.

SDNs [37,38] provide new possibilities for designing, operating, and securing data-intensive networks. However, the realization of these benefits largely requires the support of underlying infrastructure. In addition to handling the increased traffic loads, the network performance satisfaction opens new avenues of network services.

Mobile cloud computing based systems perform cloud computing functions with the exception that its users are mobile. Graph theory is a widely used concept in applied mathematics to structure pair wise models and relationships between objects. In this paper, we use graph-theoretic approach for resource consolidation on a vendor-agnostic hardware infrastructure which uses SDNs to administer network functions.

The proposed methodology is described by the formulation of a solution for VM placement that can be incorporated in SDCs. Currently, there is very limited support for VM placement in SDCs. For example, Amazon EC2 lacks support for co-locating its instance types. Although limited support features are present for cluster-based computational structures, high-performance features can only be

afforded with premium prices [39,40]. Typically, the network resources available in close proximity are used for improved networking performance. It is believed that SDCs will be extensively used in future for flexibility and support in network applications and resource management.

#### **3. Research Allocation and Mapping in Cloud Environments**

Current mobility management schemes are based on centralized data access methodology. It is similar to the concepts used in traditional DC architectures. The main problem with this scheme is that it is difficult to manage. In terms of performance-based measures, these techniques results in routing and path optimization-related constraints which ultimately leads to performance degradation challenges.

The term vendor-agnostic refers to a concept where the products of a specific manufacturer are not tied to a particular vendor/brand etc. In distributed networks and systems theory, this term is often mingled with any off the self-solution. Vendor-agnostic solutions operate upon free, open-ended and generic solutions which involve basic mathematical optimization laws and principles not tied or related to a particular company. These solutions provide a clean interface to users for interacting with real-world problems. Our reason for highlighting vendor-agnostic behavior is based on the reason that we use a combination of open-ended hardware and software (via SDNs, graph-theory and Pareto-optimality) to achieve resource management functions.

In cloud environments, the rapid interaction between network's I/O devices, data and application services affect the system's overall performance [41]. The SDN concept to decouple data from control streams eases application and network performance. Here, we want to mention that SDN itself is an enabling technology. We need to employ SDN infrastructure for developing VM placement mechanism to achieve the desired goals. Therefore, we present a VM placement scheme for a SDC environment which can improve the service response time of applications.

We consider a software-defined cloud architecture where SDN based APIs administer the cloud resource management functions. These APIs manage topology and admission control features of cloud resources. Our framework's architecture is presented in Figure 2. Beneath the APIs lies a set of network manager and cloud manager. They control various functions of cloud e.g., mapping VMs, network statistics monitoring and controlling incoming outgoing packet requests. The last layer of the design architecture consists of virtual and physical resources.

**Figure 2.** Proposed SDN cloud scenario.

SDN management APIs provide cloud resource management functions as high-level policies for the underlying network infrastructure. Such APIs help in managing and accessing an apparently infinite pool of computing resources like VMs etc. The function of the planner is to determine the location of hosting features for the received application requests in collaboration with cloud manager, modeler, and network manager. The modeler performs the comparison of received data and services from cloud planner and cloud manager. It is also used to model resource utilization features for

updating network directory status. The network and cloud managers are used for managing virtual machines. The cloud and network managers on the other hand consolidate data at both physical and logical levels. The abstraction layer consists of logically-deployed physical hardware. Finally, the physical infrastructure layer consists of a list of physical resources that could be abstracted such as storage and network resources (routers or switches), servers, computing hosts, etc.

After sending off a request, the console of SDN manager makes sure that the made request is in compliance with the minimum number of SLAs. It then creates the topology of a blueprint. The topology information is later submitted to the admission controller. The admission controller validates and ensures that a connection can be established if current resources are sufficient for the proposed connection [42,43]. A simplistic approach of the performed sequential operation is illustrated in Figure 3.

**Figure 3.** Virtual machine (VM) placement policy prototype in data center networks.

The location of hosting applications is determined by planner and modeler in consultation with cloud and network managers. Mapping of cloud resources is performed by the mapper. The proposed system performs VM placement. For ease of management, VM mapping should be controlled separately to ensure that cloud resources are managed in a clear and concise manner. The lower layer of SDC consists of different network resources. The layer for physical infrastructure contains any physical resource that could be abstracted e.g., storage and network resources (routers or switches), computing hosts, servers, etc. The abstraction layer provides abstraction information from a logical perspective. Conceptually physical layer resides beneath the abstraction layer [44,45].

In the proposed framework, by using graph theory, compute nodes are managed for allocation of VMs. In the proposed framework both virtual topologies and the physical infrastructure (switches, hosts, and links between them) are simulated for achieving dynamic routing features. In the presented scenario, all traffic patterns are supported by all the network elements. The assumptions in the proposed mechanism are mapped in a simulated environment for evaluation purpose.

The representation of the placement of VMs problem with the use of metrics (from linear algebra) is based on the fact that the cloud systems can be presented as a graph containing nodes and edges. Graph theory is the basic topic under discrete mathematics. Many advantages are present in the graph-theoretic approach.

On the basis of applied graph theory, we then manage compute nodes for VM allocation. In the proposed framework, both physical infrastructure and virtual topologies are simulated in CloudSim [46]. A detailed overview on latest trends and developments in the field of virtual resource management and network functions has been presented in [47].

#### **4. Mathematical Modeling**

We provide a mathematical representation of VM placement similar to [48] through our cloud model using a graph-theoretic approach. The entire interconnection between various entities of the proposed cloud is represented by adjacency matrices. Storage nodes (SN) represents data storage nodes. Compute nodes (CN) consist of multiple physical computational nodes, whereas data packet (DP) represents the data to be transmitted across the cloud. Our cloud infrastructure consists of 3 CNs, 2 SNs, and 3 DPs. Below we describe our model in detail. We consider a cloud system composed of m > 0 compute nodes (CN) and n > 0 storage nodes (SN). Please take note that the values of m and n are positive integers. The entire interconnection of the CNs and SNs can be depicted as a graph as shown in Figure 4.

**Figure 4.** A graph-theoretic representation of the interconnection between the compute nodes (CN) and storage nodes (SN) forming a bipartite.

In discrete mathematics terminology (especially in graph theory), the graph shown in Figure 5 is known as a bipartite. A bipartite is a group of two sets of nodes where each member of each set is able to "communicate" with each and every member of the other set. The edges connecting the CN and SN may represent any relationship between these nodes. In order to limit and scale down the performance of our simulation, we assume that these edges could represent either bandwidth in MBps or time constant in secs/MB (which is just the reciprocal of the bandwidth). For example, the edge connecting SN1 to CN1 could represent the bandwidth value of 3.2 MBps or time constant of 0.3125 secs/MB (i.e., 1/3.2 MBps).

**Figure 5.** A graph-theoretic representation of a 2-SN, 3-CN cloud system given the values of the networks bandwidths between each combination of CN and SN nodes. This is an example of a 2×3 bipartite B3.2.

In applied graph theory, an adjacency matrix is a matrix that represents the values of all edges connected to the nodes in the graph. Consider the *n* × *m* adjacency matrix

$$\mathbf{B} = \begin{bmatrix} b\_{ij} \end{bmatrix}\_{\mathbf{n} \times \mathbf{m}} = \begin{bmatrix} b\_{11} & b\_{12} & \cdots & b\_{1m} \\ b\_{21} & b\_{22} & \cdots & b\_{2m} \\ \vdots & \vdots & \ddots & \vdots \\ b\_{n1} & b\_{n2} & \cdots & b\_{nm} \end{bmatrix} \tag{1}$$

where the matrix elements b*ij* for 1 ≤ *i* ≤ *n* and 1 ≤ *j* ≤ *m* are values representing the edges connecting node *i* to node *j*. As we consider a 2-CN, 3-SN cloud system as depicted in Figure 6, the network bandwidth between the nodes are as follows:

**Figure 6.** A graph representation of the relationship between the data pieces and the storage nodes. The first sub-graph is a 2 × 1 bipartite while the other sub-graph is a 1 × 1 bipartite (or simply a connection between two nodes).

Then the corresponding graph-theoretic representation will result in Figure 5 having the adjacency matrix with row *i* representing the SN number and column *j* representing the CN number.

$$\mathbf{B}\_{3,2} \begin{bmatrix} 3.2 & 16.0 \\ 6.0 & 7.2 \\ 2.4 & 4.0 \end{bmatrix} \text{MBps.} \tag{2}$$

Similar to the approach in the previous section, a graph-theoretic approach can also be used to represent the relationship between the data pieces and SNs. In order to consider an environment two data pieces, DP1 = 200 MB and DP2 = 100 MB are both stored at storage node SN1, while another data piece, DP3 = 500 MB, is stored in SN2. The resulting graph shall be composed of two sub-graphs: one graph representing the relationship between DP1 and DP2 to SN1, and the relationship between DP3 to SN2. Please take note that each sub-graph is also a bipartite as shown in Figure 6. Since each data piece is stored only in a dedicated SN, it will be assumed in this architecture that the data piece is not shared between other SNs. Therefore, each sub-graph will only have one SN but can have multiple DPs.

The entire interconnection between the DP and SN can also be represented by an adjacency matrix where the column q shall represent the SN number. Since there are 2 SNs, then the matrix will have n = 2 columns. The number of rows of the adjacency matrix shall be equal to the maximum number of data pieces in any SN. In this particular example, since SN1 has two data pieces, namely DP1 and DP2, the number of rows shall be equal to p = 2. The resulting adjacency matrix becomes

$$\mathbf{D}\_{2.2} = [\mathbf{d}\_{ki}]\_{2 \times 2} = \begin{bmatrix} \mathbf{D} \mathbf{P}\_1 & \mathbf{D} \mathbf{P}\_3 \\ \mathbf{D} \mathbf{P}\_2 & 0 \end{bmatrix} = \begin{bmatrix} 200 & 500 \\ 100 & 0 \end{bmatrix} \text{MB}. \tag{3}$$

Consider for example a network bandwidth of b MBps. If data of size d MB will be transmitted into the network, then the response time can be obtained through

$$t = \frac{d}{b} [\sec]\_\prime \tag{4}$$

where simply the data and bandwidth are being adjusted with respect to time.

Similarly, if the network bandwidth b is inverted resulting in the time constant τ = 1/b (secs./MB), then the response time can be calculated using

$$t = d(MB). \,\,\pi \left(\frac{\sec}{MB}\right) [\sec],\tag{5}$$

where time and bandwidth are relating to resource requirements

However, this expression is only valid for scalar quantities, i.e., if there is only one data piece being processed by one CN through one SN. In fact, the notation "×" can be used here to represent scalar multiplication.

First, consider the graphical representation of the merger between Bn,m and Dp,n as shown in Figure 7. From the graphical representation, data flows between the SN and CN and DPs storage location can be seen. Using graph theory [16], it is possible to graphically represent networks using nodes and edges even if their quantities are different. The matrix can be defined as

$$\left[\Upsilon\_{n\times m} = \left[\tau\_{ij}\right]\_{n\times m}\right] \tag{6}$$

where the values τ*ij* represent the time constants between the SN and CN. Let us call this the time constant matrix. Basically, the values in this matrix are just the reciprocals of the bandwidths, therefore the following mathematical expression

$$\left[\tau\_{i\bar{j}}\right]\_{n\times m} = \left|\frac{1}{b\_{i\bar{j}}}\right|\_{n\times m} \tag{7}$$

where cross matrix multiplication ensures SN to CN matrix mapping shall apply for 1 ≤ *i* ≤ *n* and 1 ≤ *j* ≤ *m*. Given the data set matrix **D**p,n we can now get the response times for each data piece in various CNs.

**Figure 7.** Graph representing entire cloud mapping.

The series of manual computations above are easily done due to the small dimensions of the cloud system. Since real cloud systems have hundreds of thousands of SN and CN, it will be impossible for us to have all the combinations and compute them manually. It is now important to have everything done with a computer through linear algebra. In order to obtain the total response times tCN,1, tCN,2 and tCN,3 we need the following step:

(1) Step 1: get the time constant matrix from the bandwidth matrix: Let

$$\mathbf{B}\_{3.2} = \begin{bmatrix} 3.2 & 16.0 \\ 6.0 & 7.2 \\ 2.4 & 4.0 \end{bmatrix} \text{MBps} \tag{8}$$

be the bandwidth matrix of the cloud system. The time constants for each element in the matrix can be obtained by simply getting the reciprocals of each element. The resulting matrix becomes

$$\mathbf{B}\_{3.2} = \begin{bmatrix} 1/3.2 & 1/16.0 \\ 1/6.0 & 1/7.2 \\ 1/2.4 & 1/4.0 \end{bmatrix} = \begin{bmatrix} 0.3125 & 0.6250 \\ 0.1667 & 0.1389 \\ 0.4167 & 0.2500 \end{bmatrix} \text{sec} \, / \text{MB}. \tag{9}$$

(2) Step 2: get the transpose of the time constant matrix: Given the time constant matrix Υ3.2, its transpose can be obtained as

$$
\Upsilon\_{3,2}^T = \begin{bmatrix} 0.3125 & 0.6250 \\ 0.1667 & 0.1389 \\ 0.4167 & 0.2500 \end{bmatrix} = \begin{bmatrix} 0.3125 & 0.1667 & 0.4167 \\ 0.6250 & 0.1389 & 0.2500 \end{bmatrix} \text{sec} \, / \text{MB}^T. \tag{10}
$$

(3) Step 3: multiply the data set matrix with the transposed time constant matrix: Given the data set matrix

$$\mathbf{D}\_{22} = \begin{bmatrix} 200 & 500 \\ 100 & 0 \end{bmatrix} \text{MB}. \tag{11}$$

The response time matrix can then be obtained as follows

$$T\_R = \mathbf{D}\_{22} \mathbf{1}\_{32}^T = \begin{bmatrix} D P\_1 & D P\_3 \\ D P\_2 & 0 \end{bmatrix} \begin{bmatrix} \tau\_{11} & \tau\_{12} \\ \tau\_{21} & \tau\_{22} \\ \tau\_{31} & \tau\_{32} \end{bmatrix} = \begin{bmatrix} D P\_1 & D P\_3 \\ D P\_2 & 0 \end{bmatrix} \begin{bmatrix} \tau\_{11} & \tau\_{21} & \tau\_{31} \\ \tau\_{12} & \tau\_{22} & \tau\_{32} \end{bmatrix}^T \tag{12}$$

$$T\_R = \begin{bmatrix} 200 & 500 \\ 100 & 0 \end{bmatrix} \begin{bmatrix} 0.3125 & 0.1667 & 0.4167 \\ 0.6250 & 0.1389 & 0.2500 \end{bmatrix} \tag{13}$$

$$\begin{aligned} \;^1T\_R = \begin{bmatrix} \begin{pmatrix} t\_{11} + t\_{31} \end{pmatrix} & \begin{pmatrix} t\_{12} + t\_{32} \end{pmatrix} & \begin{pmatrix} t\_{13} + t\_{33} \end{pmatrix} \\\ \begin{pmatrix} t\_{21} \end{pmatrix} & \begin{pmatrix} t\_{22} \end{pmatrix} & \begin{pmatrix} t\_{23} \end{pmatrix} \end{bmatrix} = \begin{bmatrix} \begin{bmatrix} t\_{R,11} & t\_{R,12} & t\_{R,13} \\\ t\_{R,21} & t\_{R,22} & t\_{R,23} \end{bmatrix} \end{aligned} \tag{14}$$

One key characteristic of the network response time matrix TR is that if you get the sum of all elements per column, you actually obtain the total response time for each and every CN. Each column of TR represents each CN. Since, in this example, there are 3 CNs in the cloud system, TR results in a matrix having 3 columns as well. If the matrix is denoted by

$$\mathbf{T}\_{\rm R} = \begin{bmatrix} t\_{\rm R,11} & t\_{\rm R,12} & t\_{\rm R,13} \\ t\_{\rm R,21} & t\_{\rm R,22} & t\_{\rm R,23} \end{bmatrix} \tag{15}$$

The total response time per CN can be obtained using the expression

$$\mathbf{t}\_{\text{CN},i} = \sum\_{j=1}^{n} t\_{\text{R,ji}} \,\prime \tag{16}$$

Therefore,

$$t\_{\text{CN},1} = \sum\_{j=1}^{n-2} t\_{\text{R},j1} = t\_{\text{R},11} + t\_{\text{R},21} \tag{17}$$

$$t\_{\rm CN,2} = \sum\_{j=1}^{n=2} t\_{\rm R,j2} = t\_{\rm R,12} + t\_{\rm R,22} \tag{18}$$

$$t\_{\rm CN,3} = \sum\_{j=1}^{n-2} t\_{\rm R,j3} = t\_{\rm R,13} + t\_{\rm R,23} \tag{19}$$

Real-world optimization issues can be implemented on cloud-based systems to involve multiple conflicting objectives. Therefore, a vector-optimization problem in a standardized manner can be represented as a standardized vector *X* = (*x*1, *x*1, ... , *xn*). A Pareto-optimal solution [49,50] for resource existence (when no other solution exists) is represented in Figure 8. This helps in ensuring that one objective (resource allocation) can be improved without affecting the other objective.

**Figure 8.** The Pareto-optimal solution where one objective can be improved without the expense of others.

#### **5. Performance Evaluation**

In this section, we explain the simulations and experiments carried to evaluate the proposed resource consolidation approach. We used a channel model approach similar to presented in [51,52], which is widely used for mobile cloud and cellular networks. The testbed simulation consisted of two storage nodes, three data sets, and three mobile hosts. The mobile devices map their resources by using a graph-theoretic model as explained in previous sections and use SDN-based infrastructure for controlling data and traffic behavior functions. In this regard, we implemented our algorithm on CloudSim v 3.0. The Cloudsim is often used as an extensible simulation toolkit for simulation purposes.

We developed an algorithm (Algorithm 1) for virtual machine placement on a particular cloud node. It works by calculating service response time of individual compute nodes and then selecting VM with least response time. By using the proposed Algorithm 1, we compute the service response time *TR* of individual *CN* and select a *CN* having least response time to host VM. We then calculated the response time of these data loads using vmallocationpolicysimple algorithm [53,54]. We selected the vmallocationpolicysimple algorithm because of two major reasons. Firstly, it does not implement dynamic consolidation of VMs, and only places new VMs on hosts; fulfilling our scenario's demand. Secondly, it is the default VM placement strategy in CloudSim. Finally, we compared the service response time of algorithm 1 with that of vmallocationpolicysimple algorithm for given workloads. The evaluation setting is similar to emulate the environment presented in Figure 7.

Simulation results in Figure 9 show the service response time for tasks requesting DP1 data load. In Figures 10 and 11, we illustrate the service response time for tasks requesting DP2 and DP3 respectively. The graphics illustrations reveal that our proposed scheme demonstrates improved service response time to requests as compared to vmallocationpolicysimple algorithm. It is because the presented algorithm clearly chooses a CN with reduced response time and shorter data access route for VM allocation.

**Figure 9.** Service response time (DP1 requesting tasks).

**Figure 10.** Service response time (DP2 requesting tasks).

**Figure 11.** Service response (DP3 requesting tasks).

By observing the comparative results in Figure 12, it can be observed that the traffic intensity for DP1, DP2, and DP3 exhibit the same behavior of variance with respect to mean response time. However, the difference in values with respect to the vmallocationpolicysimple is different. This also resulted in increased variance rates of bandwidth consumption. In order to retain the job queue waiting time, we can manage the waiting-time window slot timing. If the received task arrives which can be completed in a relatively long period of time, the mapping scheme can adjust itself according to window time and accommodates more tasks as compared to the vmallocationpolicysimple strategy. The same concept can be improved with the predictive analysis however we didn't consider it due to overhead costs incurred on VM loads.

**Figure 12.** Cumulative comparison of service response time for 100 requests.

#### **6. Conclusions and Future Work**

Future mobile-oriented networks are taking the computing industry by storm. The recent developments witnessed in the enhanced computational capability of mobile equipment led to the concepts of mobile cloud computing. Taking this paradigm to another step, in this paper we presented a case where cloud data centers are managed in a mobile cloud environment. We begin the paper by explaining the importance of mobile future network architectures followed by the concepts of resource management in mobile clouds using a vendor-agnostic approach (through SDNs).

To sum up the concept of the paper, we argue that the cloud computing concept involves the availability of computing resources for data storage and processing. Due to increasing number of network applications, number of users and their requirements, there is a dire need to develop tools to improve cloud computing performance. On the other hand, software-defined networking concept allows cloud data center administers to manage cloud resource allocation function according to their own needs via bypassing proprietary network peripherals. As SDN concepts discourage excessive use of proprietary equipment, it is often referred as a bare-metal solution, off-the-shelf solution and vendor-agnostic solution).

The paper relates SDN-based mobile cloud environment to propose a VM mapping policy using a graph-theoretic approach. The reason for calling this technique vendor-agnostic is the use of a vendor-agnostic platform (SDN-enabled H/W) for evaluation purpose.

In this paper, the simple representation of VM resource allocation and representation helped in clearly determine the network management by the use of matrices. A vendor-agnostic-based approach, therefore, offers several advantages over conventional approaches. These advantages can be seen particularly in distributed systems like cloud computing environments. Therefore, we implemented this approach to consolidate VM resources in a simplistic and well-organized way. We believe that graphs can be represented using adjacency matrices where each element of the matrix denotes values that show relationships between any two nodes. Therefore, we used a graph-theoretic approach to achieve our consolidation approach. We developed a framework and compared its performance with the vmallocationpolicysimple technique. Results demonstrate that our proposed framework can limit the cloud topology scaling issues of VM placement in a more clearer and concise manner. We strongly believe that a vendor-agnostic approach in data centers can be considered as the next step towards the evolution of virtualization, mobile cloud computing, and future mobile-oriented networks.

**Author Contributions:** Conceptualization, A.A.A.; data curation, A.A.A. and M.A.A.A.-q.; formal analysis, M.A.A.A.-q. and A.H.; investigation, A.A.A. and A.H.; methodology, M.A.E. and A.A.E.; resources, M.A.E. and S.J.; software, A.A.E.; supervision, S.K.; writing—original draft, A.A.A.; writing—review and editing, A.A.A. and S.J. All authors have read and approved the manuscript.

**Funding:** Following are results of a study on the "Leaders in INdustry-university Cooperation+" project, supported by the Ministry of Education and National Research Foundation of Korea.

**Acknowledgments:** The authors thank the anonymous reviewers for their thorough feedback and suggestions, which were crucial to improving the contents and presentation of the article.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **TARCS: A Topology Change Aware-Based Routing Protocol Choosing Scheme of FANETs**

#### **Jie Hong 1,2,\* and Dehai Zhang <sup>1</sup>**

Received: 21 December 2018; Accepted: 21 February 2019; Published: 2 March 2019

**Abstract:** The rapid change of topology is one of the most important factors affecting the performance of the routing protocols of flying ad hoc networks (FANETs). A routing scheme suitable for highly dynamic mobile ad hoc networks is proposed for the rapid change of topology in complex scenarios. In the scheme moving nodes sense changes of the surrounding network topology periodically, and the current mobile scenario is confirmed according to the perceived result. Furthermore, a suitable routing protocol is selected for maintaining network performances at a high level. The concerned performance metrics are packet delivery ratio, network throughput, average end-to-end delay and average jitter. The experiments combine the random waypoint model, the reference point group mobility model and the pursue model to a chain scenario, and simulate the large changes of the network topology. Results show that an appropriate routing scheme can adapt to rapid changes in network topology and effectively improve network performance.

**Keywords:** flying ad hoc network (FANET); mobile ad hoc network (MANET); highly dynamic; periodical; topology change awareness; routing protocol

#### **1. Introduction**

Mobility is one of the most prominent features of mobile ad hoc networks (MANETs) and one of the important factors affecting network performance. Being a typical subset of MANETs, the node mobility in flying ad hoc networks (FANETs) [1–3] is stronger and severely impacts the network performance (Figure 1).

**Figure 1.** MANET, VANET and FANET.

FANET has some unique characteristics compared to MANET and VANET (Table 1). In FANETs, high-speed moving nodes usually need to complete multiple tasks, such as search/exploration, reconnaissance/patrol, or target tracking. Different task scenarios could have corresponding node mobility models [1]. When the task changes, the node mobility mode (such as node motion speed, direction, distance, etc.) changes accordingly, which will cause the network topology to change rapidly and seriously affect the network performance.

\* FW: fixed-wing, \* RW: rotary wing.

The main research object of this paper is highly dynamic FANETs. Specifically, it corresponds to the FANETs listed in Table 1 in which the nodes move fast and the network topology changes drastically.

The characteristics of the FANETs discussed in this paper are: small and medium-sized networks, high mobility nodes far from the ground at high altitudes, sufficient node energy and frequent network topology changes.

In practical applications, FANETs encounter different application scenarios when nodes complete complex tasks. Nodes move in different ways in different scenarios, which causes large changes in the network topology. Traditionally, nodes cannot perceive this change, nor can they adapt quickly to it, so the network performance will deteriorate dramatically. The main purpose of this paper is to find a method by which nodes in a network can perceive changes of topology and adjust the routing strategy appropriately when the topology changes greatly, so that the network performance could be maintained at a high level. The most concerned network performance is the packet delivery ratio, followed by the average end-to-end delay, the average jitter, and finally the network throughput.

Many routing protocols have been proposed for mobile ad hoc networks [4]. Each protocol has its own characteristics and application scenarios. For example, the Optimized Link State Routing Protocol (OLSR) [5] is suitable for high-density MANETs, and the Destination-Sequenced Distance-Vector Routing (DSDV) [6] is more suitable for small-scale MANETs. When the scenario is complex and the network topology changes rapidly, a single routing protocol cannot meet the requirements of the variable node mobility mode, nor can network performance be guaranteed. Accurate perception of the surrounding environment and appropriate adjustment strategies of nodes have become the trends of adaptive routing protocols with the increasing requirements of quality of service (QoS). In response to the above objectives, this paper proposes a solution for choosing routing protocols in highly dynamic FANETs, namely TARCS (topology change aware-based routing protocol choosing scheme). The TARCS scheme is divided into the following steps:

Firstly, a topology change sensing method is proposed, which can measure the topology change between nodes of a high dynamic FANET, and a mobility metric TCD, the topology change degree with several levels is defined to quantify the perceived results.

Secondly, a heuristic routing strategy of FANETs is proposed using the measurement results of the sensing method. The policy compares the measured topology change result with the threshold reference value, determines the current node movement mode, and appropriately adjusts the routing protocol used in the network, so that the network routing protocol is more suitable for the changed mobile scenario, and finally the purpose of improving network performance in highly dynamic topology is achieved.

Simulations show that compared with traditional practices, the proper use of TARCS in highly dynamic MANETs in complex scenarios can effectively improve some aspects of network performance. The content discussed in this article is based on the following assumptions:

The remainder of this paper is organized as follows. In Section 2, the TARCS is introduced in detail. In Section 3, the TARCS strategy is validated in the form of simulation experiments. The experimental results are discussed in Section 4, and Section 5 contains the conclusion.

#### **2. Related Work**

The following paper summarizes related literature from two aspects: topological change perception and topological change perception processing.

#### *2.1. Topology Change Perception Methods and Processing Methods*

The state-of-the-art perceptions of topology changes are limited to mobility aware of nodes and paths. Radhika Ranjan Roy [7] described mobility models and mobility metrics in detail. Bai Fan et al. [8] defined several mobile metrics, such as node spatial dependence, node temporal dependence and geographic restrictions to capture mobility characteristics, and proposed the IMPORTANT framework for analyzing the impact of mobility on the performance of routing protocols. Jie Hong et al. [9] explained the relationship among node mobility, network topology changes and network performance such that the high speed of nodes in a MANET did not mean rapid topology changes, and changes in network topology caused by changes of node speed and direction ultimately affected network performance. A dynamic routing algorithm improved from the dynamic source routing algorithm (DSR [4]) and proposed by Ehssan S. et al. [10] by selecting relatively static nodes according to the Doppler frequency shift was applied to the aeronautical ad hoc networks (AANETs) with high-speed nodes. Zheng, Y. et al. [11] proposed the mobility and load aware OLSR (ML-OLSR) protocol, which combined mobile sensing and load sensing with the procedures of selecting more stable nodes as the multi point relay (MPR) points, and lower-load routing to reduce the average end-to-end delay. The protocol is applicable to the unmanned aerial vehicle (UAV) networks with high-speed nodes and unbalanced loads. The protocol proposed by Zhou J.H. et al. [12] was also applicable to AANETs in which the effective time of the largest link was estimated according to the node relative speed and the Doppler shift of the received signal; meanwhile the traffic load was referenced during the route discovery. The mobility pattern aware routing protocol proposed by Hung, C-C. et al. [13] requires geographic assistant location information and relies on IEEE 802.16 base station assistance for a heterogeneous vehicular network (HVN). Athanasios B. et al. [14] proposed a protocol framework with the defined three-layer services of mobility classification service (MCS), strategy selection service (SSS) and routing service (RS) to make nodes with different mobility levels use different routing protocols. It is suitable for high-speed mobile networks with sparse nodes. Yu, Y. et al. [15] combined the DSR protocol with the ant colony optimization algorithm (ACO), judging the stability between nodes and neighbors according to the received signal power, using the ACO to detect the network congestion degree. Nodes with high mobility in the cache were deleted according to the node stability

and the congestion degree. The premise of the protocol proposed by Swidana, A. et al. [16] was that the velocity of the node had been measured before, the basic idea of which was to prevent high mobility nodes from participating in route discovery. Khalaf, M. et al. [17] proposed two probability models for speed perception, with which two protocols were proposed by improving the ad-hoc on-demand distance vector (AODV) [18] routing algorithm. The protocol proposed by Moussaoui, A. et al. [19] judged node stability based on the received signal power, and improved the OLSR protocol by selecting the more stable neighbor node as the MPR node. Brahmbhatt, S. et al. [20] proposed a protocol to select the most powerful node from the received signal as a stable link in light of signal strength-based link stability estimation to solve the routing failure problem in multipath networks.

#### *2.2. Discussion*

The mobility aware methods noted above can be classified into the following categories:

These mobility aware methods can also be divided into location aided [11,13,14,16,17] and non-location aided [10,12,15,19,20]. The former refers to estimating the neighbor node/path mobility by means of information from auxiliary devices, such as global positioning system (GPS) receivers, with the advantage of simple and fast calculation and the disadvantage of introducing GPS errors and new interference. The latter usually measures the position or speed of the neighbor node according to the power or the Doppler shift of the received signal. Although no error or other interference is introduced, the direction and distance of the neighbor node are not accurately determined.

The processing method after mobility awareness can be summarized into the following:

The mobility awareness methods, the processing methods and the proposed TARCS are summarized in Figure 2.

However, the methods in the above literature are only limited to measuring node mobility or link stability caused by mobility, without involving accurate definition and accurate measurement of topology changes.

In this paper, the methods of perception and perceived result processing are different. In terms of the sensing method, the above literature uses only a single parameter (velocity or distance) for perception, but TCD is used here to quantify the topological changes between nodes from multiple aspects, including distance, velocity, direction and neighbors' number, and to accurately reflect the topological changes. As to the subsequent processing method, in this paper, after determining the node's mobility mode based on the perceived result, the routing protocol is re-selected according to the current mobility mode characteristics. Essentially this is a routing protocol optimization selection strategy.

**Figure 2.** The mobility aware methods and the subsequent processing methods.

#### **3. A Topology Change Aware Routing Protocol Choosing Scheme (TARCS)**

A topology change aware routing protocols choosing scheme (TARCS) is proposed based on the results of the periodic topology change awareness (PTCA) and the adaptive route choosing scheme (ARCS). The purpose of the PTCA is to accurately measure topology changes and determine the movement mode. The purpose of the ARCS is to select an appropriate routing protocol for the current motion mode and to ensure that network performance is affected as little as possible. The periodic topology change perception is based on the topological change degree (TCD), a mobility indicator proposed by the previous study, which could measure the topology changes of nodes and distinguish the Random Waypoint model (RWP [7]), the Reference Point Group Mobility Model (RPGM [7,21]) with different group numbers and the Pursue model [7,21].

#### *3.1. Description*

The topology change aware routing protocol choosing scheme (TARCS) includes two procedures: the periodically topology change aware (PTCA) and the adaptive route choosing scheme (ARCS). Accurate perception of topology changes is a prerequisite for routing protocol adjustment and reselection. The routing protocol selection is based on a preset topology change threshold/interval value.

Traditional routing protocols either maintain routing paths during node movement (active routing) or start routing discovery when nodes need to communicate (passive routing). Neither active routing nor passive routing pays close attention to the surrounding topology. This is completely sufficient for low-speed nodes, but for high-speed nodes the topology changes so fast that nodes cannot respond to the rapidly changing environment and adjust strategy if they lack the knowledge of the external environment. Obviously, the consequence is that the nodes are not well adapted

to the topology changes, the network performance is severely constrained, and the task cannot be successfully completed.

The advantage of TARCS is that:

Assume that node A (*n*a) will send information to node B (*n*b) in a FANET. Figure 3 shows the comparison of communication mode from *n*<sup>a</sup> to *n*<sup>b</sup> with and without the TARCS. Figure 3a shows that *n*<sup>a</sup> communicates with *n*<sup>b</sup> directly through a routing protocol in the traditional manner. *n*<sup>a</sup> first establishes a valid path with *n*<sup>b</sup> through route discovery, then sends data and maintains the route. Figure 3b shows the communication mode from *n*<sup>a</sup> to *n*<sup>b</sup> after using the TARCS.

**Figure 3.** Communication mode comparison between two nodes without and with the topology change aware routing protocol choosing scheme (TARCS). (**a**) Traditional communication mode between nodes without the TARCS; (**b**) communication mode between nodes with the TARCS.

**Figure 4.** Process flow of the TARCS.

Figure 4 shows the basic workflow of the TARCS. The TARCS is divided into two phases. The left region represents the periodic topology change aware (PTCA), and the right region represents the adaptive routing protocol selection (ARCS).

#### *3.2. Periodically Topology Change Aware (PTCA)*

The purpose of PTCA is to periodically perceive the topology changes of each node and its neighbors within a one-hop transmission range in the network. Since the movement of nodes causes the change of the relative position between the neighboring nodes and in the number of neighboring nodes, thereby causing the network topology to change, the discussion of the topology change begins with the mobility of the node.

Figure 5 shows the topology change caused by the change in distance among nodes. Black dots and red dots represent moving nodes, and dashed circles indicate the effective transmission range of nodes. Figure 5a shows the topology in the initial state. Nodes *n*<sup>1</sup> and *n*<sup>2</sup> are within one hop of *n*0. Node *n*<sup>3</sup> is within one hop of *n*1, and *n*<sup>4</sup> is within one hop of *n*3. Node *n*<sup>0</sup> establishes a route through *n*1, *n*3, and *n*4. From this moment on, *n*<sup>1</sup> starts moving down. Figure 5b shows the network topology after time *T*. At this time, *n*<sup>1</sup> has moved out of the effective range of *n*0, and the distance between *n*<sup>0</sup> and *n1* increases. If *n*<sup>0</sup> still sends data to *n*4, it can only pass the path: *n*<sup>0</sup> → *n*<sup>2</sup> → *n*<sup>3</sup> → *n*<sup>4</sup> . It can be seen that topology changes could be caused by distance changes between nodes.

**Figure 5.** Topology change caused by nodes distance change: (**a**) topology in the initial state; (**b**) topology after period *T*.

Figure 6 shows the topology change caused by changes in the direction of movement between nodes. Figure 6a is the initial state. From this moment on, *n*<sup>0</sup> will move around *n*1. After time T, the distance between *n*<sup>0</sup> and *n*<sup>1</sup> is still the same. However, there is no effective path between *n*<sup>0</sup> and *n*<sup>4</sup> due to the change of direction between *n*<sup>1</sup> and *n*0. As is shown in Figure 6b. It can be seen that changes in the direction of movement of nodes can also cause topology changes.

**Figure 6.** Topology change caused by nodes direction change: (**a**) topology in the initial state; (**b**) topology after time *T*.

Figure 7 shows the topology change caused by changes in the relative rate between nodes. Figure 7a is the initial state. All nodes move up, but the nodes move at different rates. The rate of *n*<sup>1</sup> is less than the rate of the others. Figure 7b is the topology after time *T*. It can be seen that the topology has also changed.

**Figure 7.** Topology change caused by a difference of rate: (**a**) topology in the initial state; (**b**) topology after *T* time.

Figure 8 illustrates the topology change caused by the number of neighbors. Figure 8a is the initial state. *n*<sup>1</sup> and *n*<sup>2</sup> are both one-hop neighbors of *n*0. The initial path of *n*<sup>0</sup> to *n*<sup>4</sup> is *n*<sup>0</sup> → *n*<sup>1</sup> → *n*<sup>3</sup> → *n*<sup>4</sup> . In Figure 8b *n*<sup>1</sup> exits the network for some reason (e.g., hardware failure or energy exhaustion) after time *T*. At this time, the number of neighbors of *n*<sup>0</sup> is reduced resulting in no complete path between *n*<sup>0</sup> and *n*4. Therefore, changes in the number of neighbor nodes can also cause topology changes, but this condition is not sufficient. For example, although *n*<sup>2</sup> is also a neighbor of *n*0, the departure of *n*<sup>2</sup> does not affect the path from *n*<sup>0</sup> to *n*4.

**Figure 8.** Topology change caused by a change in the number of neighbors: (**a**) topology in the initial state; (**b**) topology after time *T*.

From the analysis above, it is concluded that each of these factors, namely, the relative position, motion direction, relative rate, and number of neighbor nodes, may cause a change in the local topology between neighboring nodes. Thus, a new mobility metric, the topology change degree (TCD) was defined by selecting parameters such as node distance, motion direction and movement rate, to measure the topology change between neighbor nodes. The main purpose of TCD is to accurately quantify topology changes between neighboring nodes and quickly identify different motion modes.

Topology change degree between nodes *<sup>i</sup>* and *<sup>j</sup>* within time *<sup>T</sup>*, the *TCDi*,*j*, represents the degree of topology change between the node *i* and its neighbor node *j* in the *T* time period. The detailed definition is shown in Equation (1).

$$TCD\_{i,j}(t, t+T) = \left( w\_1 \cdot \frac{\left| d^{t+T}\_{i,j} - d^t\_{i,j} \right|}{d^t\_{i,j}} + w\_2 \cdot \frac{\left| \theta^{t+T}\_{i,j} - \theta^t\_{i,j} \right|}{2\pi} + w\_3 \cdot \frac{\left| relv^{t+T}\_{i,j} - relv^t\_{i,j} \right|}{relv^t\_{i,j}} \right) \tag{1}$$

*i*,*j*

where:

Average topology change of node *<sup>i</sup>* and its neighbors in time *<sup>T</sup>*, is define as the *TCDi*,*nbrs*(*t*,*<sup>t</sup>* <sup>+</sup> *<sup>T</sup>*). Suppose the number of neighbors *j* of node *i* is *n* at time *T* (1 ≤ j ≤ n), then the average topology change degree of node *i* and neighbors in time *T* is defined as the topology change degree average value of each pair of node *i* and neighbor *j*, as defined in Equation (2).

$$TCD\_{i,nbrs}(t, t+T) = \frac{\sum\_{j=1, j \neq i}^{n} TCD\_{i,j}(t, t+T)}{n} \tag{2}$$

The whole network topology change in time *T*, the *TCDntwrk*(*t*,*t* <sup>+</sup> *T*), is defined as the sum of the topological changes of all nodes in the network if the total number of nodes in the network is *N* (N > n), as shown in Equation (3).

$$TCD\_{ntwk}(t, t+T) = \frac{\sum\_{i=1}^{N} TCD\_{i, nbrs}(t, t+T)}{2} \tag{3}$$

Experiments have shown that the difference in node mobility affects the network TCD. In addition, TCD has been confirmed to reflect the topology changes around the nodes and the network, and can distinguish several different mobility modes, including RWP and RPGM (with different groups).

The first step of TARCS is to perform periodic topology change perception on each node of the entire network to obtain accurate topology change perception results. The main method of node mobility mode discrimination of this paper is to compare the topology change perception result *TCDntwk* and the topological change threshold reference value *TCDTH*.

#### *3.3. Adaptive Routing Choosing Scheme (ARCS)*

The next step of TARCS is to perform adaptive routing selection, ARCS, after obtaining the results of the surrounding topology changes. The ARCS includes node mobility mode discrimination and routing protocol selection. The judgement role is:

$$\begin{cases} \text{Roughting protocol } A, \text{if } T\text{CD}\_{ntrwk} \le T\text{CD}\_{TH} \\ \text{Roughing protocol } B, \text{if } T\text{CD}\_{ntrwk} > T\text{CD}\_{TH} \end{cases} \tag{4}$$

In practical applications, the *TCDTH* can be pre-calculated with reference to a specific moving model. Experiments in Section 4 show the specific calculation process of the threshold value. It is also possible to set a threshold range for discrimination. If the perceived result *TCDntwk* is within the allowed interval, the current route protocol is continued to be used, and if the perceived result exceeds the threshold interval, the routing protocol selection is restarted.

Routing is based on the appropriate mobility model for each routing protocol. Therefore, it is necessary to calculate and classify possible scenarios, each mobility model with its *TCDTH*, and the corresponding routing protocols in advance. After the routing protocol is selected, nodes communicate through the newly set routing protocol until the next topology change perception.

#### **4. Evaluations and Results**

Simulation experiments are designed to evaluate the scheme. It is assumed that all nodes can receive signals from neighboring nodes within one hop in each direction, the node energy is sufficient, and the transmission distance is constant. The speed of the nodes, the distance between nodes, and the direction of motion of the nodes are known. Nodes move only in two dimensions.

In order to more clearly reflect the highly dynamic network topology, the Chain scenario [22] is selected to simulate the highly dynamic scene, which combines RWP, RPGM and Pursue model to simulate nodes to complete a series of tasks such as search/exploration, reconnaissance/patrol and target tracking/rescue. The duration is 600 s. From 0 s to 200 s nodes move with the RWP model, then they move with the RPGM (g = 5) model from 200 s to 400 s, and finally nodes move with the Pursue model.

The simulation tool is NS-3.25 [23] (Network Simulator 3, Version 3.25). The region is a square of 2000 m × 2000 m. The number of nodes is 50 with 50 m/s as the lower speed and the 500 m/s as the higher speed. The Free space propagation model [23,24] is selected as the propagation mode and the constant speed propagation delay model is selected as the delay model. The transmission mode is constant bit rate (CBR) with a bit rate of 16 Kbps and a packet size of 1024 bytes. The MAC (media access control) protocol is IEEE 802.11g. The alternative routing protocols involved are AODV, OLSR, and DSDV. A detailed description of each mobility model and routing protocol can be found in the references. Table 2 lists the experimental parameter items and parameter values.

**Table 2.** Parameters of simulation experiments.

In the first experiment, the network performance is compared with or without the TARCS scheme. The second experiment is to compare the impact on the network performance with different TARCS strategies.

#### *4.1. Validation of TARCS*

This experiment mainly verified the difference in network performance with or without TARCS. The aware interval is set to 50 s, which means that the topology is perceived 13 times with each rate. Since the *TCDTH* are different with different node numbers and node rates, it is necessary to preliminarily set the *TCDTH* according to the node rate and the specific mobility model.

In this experiment, when node speed is 50 m/s the *TCDTH* is set as follows:

$$\begin{cases} \begin{array}{c} TCD\_{TH}(RWP, RPGM(\mathfrak{g}=\mathfrak{5}))=150\\ TCD\_{TH}(RPGM(\mathfrak{g}=\mathfrak{5}), Pursue)=1000 \end{array} \end{cases} \tag{5}$$

The meaning of *TCDTH*(*RWP*, *RPGM*(*g* = 5)) is the threshold of the network TCD distinguishing from motion with the RWP mode and the RPGM (g = 5) mode. Assuming TCDntwrk is the newly acquired TCD value of the whole network, the criteria are:

$$\text{probability model} = \begin{cases} \text{RWP} & \text{if } T\text{CD}\_{\text{ntwrk}} < T\text{CD}\_{TH}(\text{RWP}, \text{RPGM}(\text{g} = 5)) \\\ \text{RPGM}(\text{g} = 5) & \text{if } T\text{CD}\_{\text{ntwrk}} > T\text{CD}\_{TH}(\text{RWP}, \text{PGM}(\text{g} = 5)) \end{cases} \tag{6}$$

Similarly, TCDTH(RPGM(g = 5), Pursue)) represents the TCD threshold value that can distinguish between motion in RPGM (g = 5) mode and in Pursue mode.

When node speed is 500m/s the *TCDTH* is configured as follows:

$$\begin{cases} \begin{array}{c} TCD\_{TH}(RNP\_{\prime} RPGM(\mathfrak{g}=\mathfrak{5}))=1000\\ TCD\_{TH}(RPGM(\mathfrak{g}=\mathfrak{5}), Pursue)=2000 \end{array} \end{cases} \tag{7}$$

And the criteria are:

$$\text{probability model} = \begin{cases} \text{ } \text{ } \text{RGPM}(\text{g} = 5) & \text{if } \text{TCD}\_{\text{ntnr}k} < \text{TCD}\_{\text{TH}}(\text{RPGM}(\text{g} = 5), \text{Pursue})\\ \text{ } \text{ } \text{Pursue} & \text{if } \text{TCD}\_{\text{ntnr}k} > \text{TCD}\_{\text{TH}}(\text{RPGM}(\text{g} = 5), \text{Pursue}) \end{cases} \tag{8}$$

The calculation method for the threshold value will be introduced in Section 4.3.

According to the conclusion of [5], this experiment uses the following routing protocol selection scheme: the AOPV protocol is used in the RWP scenario, the AODV protocol is used in the RPGM scenario, and the DSDV protocol is used in the Pursue scenario. The TCD perception result at time *t* is represented by *TCD(t)*. The perceptual moment, the actual movement model, the topological change perception result *TCD(t)*, the judged mobility model, and the selected routing protocol are listed in Table 3.

**Table 3.** The topological change degree (TCD) perception results and route protocol choosing scheme of Experiment 1.

The perceptual results show that among the 26 perceptual moments, the perceived result *TCD(t)* is less than the threshold value and the determined mobility model does not match the actual model only for the speed of 50 m/s at 450 s and the speed of 500 m/s at 250 s, 300 s and 350 s. The rest of the perceived results and judgment results are correct. The rest of the perception results are correct, within the threshold range, and the judgment results are consistent with the actual movement model. The perceived efficiency is 84.6%. The above results show that TCD is effective for network topology change perception, and that the granularity setting and the *TCDTH* setting are very important for the judgment.

The cause of the error is likely to be the following factors. The threshold setting is not reasonable enough, and the time of each mobility model is not long enough, so the mobility model is not very stable when the nodes move at high speed.

Figure 9 illustrates the comparisons of performance metrics, including the packet delivery ratio, the network throughput, the average end-to-end delay and the average jitter under two different node speeds (50 m/s and 500 m/s) with the TARCS (assume that all mobility models are accurately identified)and with other routing protocols.

**Figure 9.** Performance comparisons with and without the TARCS: (**a**) packet delivery ratio; (**b**) network throughput; (**c**) average end-to-end delay; (**d**) average jitter.

It can be seen from Figure 9a that when node speed is 500 m/s, the packet delivery ratio is 58.9435% only with OLSR, 55.4443% only with DSDV, 74.9448% only with AODV, and rises to 83.3388% with the TARCS.

Figure 9b shows the network throughput comparison with and without TARCS under different node speeds. Taking the speed of 500 m/s as an example, the network throughput is 666.716 Kbps when only OLSR is used, 701.095 Kbps when DSDV is only used, 862.448 Kbps when AODV is only used, and 881.438 Kbps when the TARCS scheme is used. Network throughput is increased by 32.2%, 25.7%, and 2.2%, respectively, compared to using only OLSR, DSDV, and AODV.

Figure 9c shows the average end-to-end delay comparison. Regardless of the rate, the average end-to-end delay is different from the first two performance metrics. Using the TARCS strategy chosen for this experiment, the average end-to-end delay of the network is not the smallest, but between the highest 0.141 s (AODV) and the lowest 0.032 s (DSDV), and is also higher than the OLSR of 0.06 s with the node speed of 500 m/s.

*Electronics* **2019**, *8*, 274

The comparison result of the average jitter is similar to that of the delay, as is shown in Figure 9d. It is between the highest value of 0.149 s (only with AODV) and the lowest value of 0.0383 s (only with DSDV) with the node speed of 500 m/s.

The reason why the advantage of TARCS is not obvious when the speed is low is that the topology change of the network is not obvious when the node speed is low.

The results of first experiment show that:

#### *4.2. Impact Analysis of Different TARCS Strategies on Network Performance*