0 Research  The SURVIVRS remote surgery guidance tool. (left) Immersive
visualization of the surgery room offers a telepresence experience to the
remote surgeon. (center) The local site of the surgery can be visualized in
detail, where annotations can be sent to the surgery room. (right) Similarly,
a live view of the patient vitals monitor is detailed to which the surgeon can
make annotations.

# SURVIVRS: Surround Video-Based Virtual Reality for Surgery Guidance

Amani Taweel
Dept. of Computer Science
UNC Greensboro  e-mail<EMAIL_ADDRESS>Joaquim Jorge
IST - ULisboa e-mail<EMAIL_ADDRESS>INESC-ID  Anderson Maciel
IST - ULisboa e-mail<EMAIL_ADDRESS>INESC-ID  João
Ricardo Nickenig Vissoci
Duke Global Health Institute
Duke University e-mail<EMAIL_ADDRESS>Regis Kopper
Dept. of Computer Science
UNC Greensboro  e-mail<EMAIL_ADDRESS>
###### Abstract

There is a strong demand for virtual reality (VR) to bring quality healthcare
to underserved populations. This paper addresses this need with the design and
prototype of SURVIVRS: Surround Video-Based Virtual Reality for Surgery
Guidance. SURVIVRS allows a remote specialist to guide a local surgery team
through a virtual reality (VR) telepresence interface. SURVIVRS is motivated
by a need for medical expertise in remote and hard-to-reach areas, such as
low-to-middle-income countries (LMICs). The remote surgeon interface allows
the live observation of a procedure and combines 3D user interface annotation
and communication tools on streams of the surgical site and the patient vitals
monitor. SURVIVRS also supports debriefing and educational experiences by
offering the ability for users to watch recorded surgeries from the point of
view of the remote expert. The main contributions of this work are: the
feasibility demonstration of the SURVIVRS system through a rigorous 3D user
interface design process; the implementation of a prototype application that
realizes the proposed design; and a usability evaluation of SURVIVRS showing
that the tool was highly favored by users from the general population. The
paper discusses the next steps in this line of research aimed at more
equitable and diverse access to healthcare.

Human-centered computingHuman computer interaction (HCI)Interaction
paradigmsVirtual reality

Introduction

Many developing countries worldwide show an unmet demand for medical
professionals with specific specialties. As an example, in 2007 there were
approximately only 27 neurosurgeons available in East Africa, across seven
countries with a total population of 270 million, placing the ratio of
neurosurgeons to residents at 1:10 million [4].111As a means of comparison,
the ratio of neurosurgeons to residents in the United States was approximately
1:65,580 in 2007 [4]. Virtual reality (VR) technology has the potential to
reduce such gaps in healthcare access.

The major problem this research proposes to address is the lack of specialized
surgeons in hard-to-reach areas at an accessible cost. Specifically, this
project offers a means to enable collaboration between surgeons who are
geographically apart. This collaboration happens through live communication
via immersive surround video streams in virtual reality. This allows remotely
located surgeons to guide and give instructions to the surgery room on short
notice without the need to travel or incur additional costs.

Our proposed solution is SURVIVRS: Surround Video-Based Virtual Reality for
Surgery Guidance, an immersive virtual reality telepresence tool for remote
surgeons to guide medical procedures in real time. Figure SURVIVRS: Surround
Video-Based Virtual Reality for Surgery Guidance shows the main components of
SURVIVRS.

SURVIVRS contains three main display components: a live 360° stream of the
surgery room, from the vantage point of a locally mounted 360° camera
(henceforth referred to as 360° view), a live high-resolution stream of a spot
camera aimed at the local site of the surgery (henceforth referred to as site
view), and a live visualization of the patient vitals monitor (henceforth
referred to as vitals view). We posit that, for the remote expert surgeon to
offer fully informed guidance to a less experienced surgery team, it is
important that they have a contextual view of the surgery theater. As long as
the remote surgeon is connected to SURVIVRS, they maintain their spatial
context and orientation through the 360° view that provides a first-person
immersive video stream rendered to a VR head-mounted display (HMD). The site
view and the vitals view are always displayed near the upper corners of the
remote surgeon’s view and can be zoommed in on demand.

In addition to live surgery guidance, SURVIVRS also serves an educational
benefit. Traditionally, surgery had been taught through the “See One, Do One,
Teach One” method, where trainees perform procedures after observing similar
ones. Although this method has been criticized as unsafe to patients, its core
process remains unchanged [6]. SURVIVRS can potentially improve the “See One”
stage of surgical training by playing back surgeries to trainees. The replay
includes all tools available to the remote surgeon during the live procedure,
including the 360° view, the site view, and the vitals view. This way,
students can review a surgery, even adding annotations on any necessary parts
of the stream.

This project used several tools for its development. It uses the Unity game
engine to provide a VR application that offers immersive telepresence to a
remote user (the expert surgeon) with tools that allow the guidance of a
procedure. For this application to succeed, it needs to ensure a friendly,
intuitive, and usable immersive virtual environment on the remote surgeon’s
side. To that end, an annotation system with varied input options was
developed to allow the surgery team to visualize guidance cues during the
surgery precisely. Another goal of this research was to create a live
streaming link between the VR application (surgeon’s side) and a web app
(simulating the surgery room side) with high resolution, low cost, and a clear
audio connection. The final goal was to make a web application that emulates
the surgery room side easy to use with options to change camera sources,
resolutions, and tools to record the call.

In summary, the main contribution of this work is the design of SURVIVRS, a VR
interface that connects a specialist surgeon to a surgery room, allowing them
to guide a surgery team through the course of a procedure. This research is
motivated mainly by the prospect of SURVIVRS enabling efficient communication
between surgical teams in LMICs and other hard-to-reach areas and experienced
specialists in developed countries to ultimately improve access to critical
care and outcomes in remote locations.

##  1 Related Work

###  1.1 Virtual Reality in Healthcare

Virtual reality is a way to connect humans to computers by using HMDs to
simulate virtual and interactive environments. VR provides an immensely
realistic and controllable environment for users to interact across many
application areas. These VR features have been used in experiences and
clinical settings for over two decades [8]. One of the first real-world
applications of VR was in 1993 to treat mental health disorders. Then VR
applications were developed to handle specific phobias [2].

McCloy and Stone [7] argue that VR simulators are suited for general surgery,
while Augmented Reality (AR) is suited to guide neurosurgery. SURVIVRS offers
a remote AR-like interface by giving the expert surgeon a real-time view of
the live surgery room.

Our research aims to employ VR technology to allow communication between
remote medical specialists and non-specialist surgeons in a surgery room with
the goal of providing instructions for a more successful surgical outcome.
Using a VR application, we propose a prototype tool that, in the future, could
work over a connection between a doctor from a developed country and a surgery
room in an LMIC. Also, we offer the ability for a remotely guided procedure to
be recorded, raising the opportunity for medical students to replay the VR
view of the remote surgeon to learn and review the surgery.

###  1.2 Interaction with Immersive 360° Videos

Technology of 360° video uses omnidirectional camera systems and allows
viewing with a first-person perspective. These experiences have been described
as highly immersive and potentially engaging environments surrounded by a
total sense of existence [1].

###  1.3 Remote Collaboration through Virtual Reality

Connecting the world and sharing real-time experiences between people who may
be geographically distributed is now an urgent need. Specifically, there is a
demand for remote guidance, where a remotely situated expert assists their
team on the other end of some procedure, such as a repair, maintenance,
surgery, or communication task. This type of collaboration is required in many
fields, such as education and healthcare.

Huang et al. [5] developed a 3D system called HandsIn3D based on 3D virtual
immersed space and 3D hand gestures. This project connects a worker and a
helper through 3D stereoscopic cameras that offer a visualization of the work
site by the helper and augments the worksite by rendering representations of
the helper’s hands.

Comparing the HandsIn3D project with our own research, they are based on
common principles of remote live video-based guidance, but the implementation
and goals are entirely different. While HandsIn3D uses a 3D camera to get the
other side, SURVIVRS uses a 360° camera to film the surgery room entirely.
Moreover, instead of creating virtual hands on the work side to help people,
our work offers an annotation system to allow a remote expert to live annotate
and take screenshots. Finally, the main goal of the HandsIn3D project was to
remotely help distant workers to complete manual tasks, whereas our project
targets live remote guidance to a surgical procedure.

##  2 Design and Implementation

We used an iterative design process to propose and refine the design of
SURVIVRS. Early on, we used low-fidelity prototyping to experiment with a
variety of interface ideas. This allowed the research team to openly discuss
ideas without loss of code-based work.

After finishing the iterative design process, we conducted several steps to
ensure the final usability of the tool. Those steps involved developing the
user interfaces by implementing code to the prototypes, testing the design by
running and experiencing it, and finally evaluating the usability of the tool
by study volunteers. Figure SURVIVRS: Surround Video-Based Virtual Reality for
Surgery Guidance shows screenshots of the SURVIVRS prototype remote surgeon
interface used in the usability evaluation.

As the focus of this project is the implementation of the remote surgeon’s
guidance user interface, we simulated the surgery room through a web
application (Figure 1) that streams the view of the remote surgeon with
virtual cameras and screen-capture software.

Figure 1: Screenshot of the web application simulating the surgery room. A
stream of the remote surgeon’s viewpoint is shown at the top, while raw
streams of each camera feed is displayed at the bottom.

###  2.1 Software Tools

We implemented the SURVIVRS prototype in Unity222https://www.unity.com
(version 2020.3.33f1) with OpenXR333https://www.khronos.org/openxr. The VR
application uses the Unity Video player as a broadcaster for the 360° video
which is rendered as a texture onto the scene’s Skybox. Unity Render Streaming
(URS) was used to live stream the VR view of the remote surgeon to the
simulated surgery room web application. Finally, OBS Studio was used as a
virtual camera to stream the contents of the live 360° video footage captured
by the 360° camera.

###  2.2 Hardware

We used the Oculus Meta Quest 2 with two tracked controllers as the VR
platform for the remote surgeon. Live streaming of the 360° video was achieved
by an Insta360 Pro camera444https://www.insta360.com/product/insta360-pro. For
this prototype, we used two spot cameras to capture the image of the local
surgery site, and a view of the patient vitals monitor. For future
implementations, we expect to render a digital vitals monitor using the
patient’s data, rather than a video feed, which is lower quality and
unnecessarily takes up bandwidth.

The surgery room simulation is run in a server PC with WiFi to communicate
with the VR application. All three cameras and a monitor are connected to this
PC, which displays the surgery room simulation web application.

###  2.3 System Design

The primary purpose of this project is to offer a first step towards enabling
a remote surgeon to guide a surgeons’ team in a remote area such as sub-
Saharan Africa. Figure 2 shows a conceptual image of a doctor wearing the
guiding a remote surgery (a) and their corresponding VR view (b).The surgeon
can turn in any direction to see everything in the surgery room. As the user
changes their view direction, the 2D heads-up UI follows their head movement
in all directions after a 0.5-sec delay through a smooth animation.

Figure 2: VR Application on The Surgeon Side. (a) The Idealized Image of a
Doctor Using the VR Application. (b) VR Application Interface on the Remote
Surgeon Side.

Figure 3 shows the information flowchart of the entire communication between
the VR application and the simulated surgery room web application. The flow
starts from any side by the connection ID to start streaming videos over a
WebSocket after finishing the signaling between them. The surgery room web app
streams the 360° video, the local surgery site video, and tge patient data
monitor video to the VR application on the remote surgeon side. On the VR
application side, the specialist can interact with these videos by annotating
them in the live stream. All these annotations as well as the three streams
are shown on the surgery room screen to be viewed by all surgeons (Figure 1).
A live two-way audio connection is also established. After closing the stream,
a copy of the video call is saved on the server to be used in the “view-
recorded video” mode in the VR application by medical students or other
doctors in our idealized scenario.

Figure 3: Information Flowchart.

####  2.3.1 Annotation Interface

One of the most essential features of SURVIVRS is to give the remote surgeon
the ability to guide the surgery by pointing to details and convey them to the
surgery room through an annotation interface. The annotation system appears on
the 2D streams when the surgeon points the controller to one of the 2D videos
in the main interface and clicks the trigger buttin to zoom it in. After the
video becomes larger, a menu of drawing tools is displayed in front of the
surgeon in the VR application, as shown in Figure 4. The menu has four
annotation supporting tools: undo and redo; shapes (pencil–free drawing, oval,
rectangle, arrow, and eraser); and a play/pause button which, in addition to
playing and pausing the video, also takes a screenshot of the annotated 2D
video (Figure 4 shows previously taken screenshots in the left-bottom corner
of the UI. Whenever an annotation starts, the 2D video is automatically
paused, so that image movement does not disturb the annotation target.

Figure 4: Annotation System.

##  3 Evaluation

We performed a usability evaluation of the SURVIVRS prototype to understand
how users from the general population react to the tool before we test it with
less available medical doctors. Institutional Review Board approval was
obtained prior to data collection. The evaluation was performed with the “view
recorded videos” mode to allow the experience of a real surgery video and
avoid technical issues such as latency that are not part of the user
interface.

###  3.1 Participants

For this evaluation, six volunteers participated in the experiment, four had
no vision issues, and two had strabismus. There were three females and three
males, ages ranging from 18 to 44. Most of the participants (66.6%) rated
themselves as slightly familiar with VR, while the rest were either not
familiar with VR at all (16.6%) or moderately familiar with VR (16.6%).

###  3.2 Measures

Overall perceived usability was measured with the System Usability Scale [3].
Presence was measured with the Slater-Usoh-Steed presence questionnaire [9].
We also developed an overall perceptions questionnaire with specific questions
about the design and functionality of SURVIVRS user interface.

###  3.3 Procedure

After the initial greeting and signing the consent form, participants were
given general instructions about the experiment. Then, participants wore the
HMD and controllers to begin the experiment. At all times, during data
collection, the experimenter observed through casting from the HMD to a PC.

After ensuring that the participant got familiar with the VR HMD (Meta Quest
2), the experimenter asked the participant to complete a series of ten tasks
that were meant to allow the exploration of all application features. No help
or instruction on how to complete a task was given to the participants.

After finishing all tasks, participants were asked to complete the post-study
questionnaires

##  4 Results

###  4.1 System Usability

Figure 5 shows the SUS Usability scores for each participant. We see that, for
5 out of the 6 participants, their scores are considerably higher than the
68-point threshold for average usability 555https://www.usability.gov/how-to-
and-tools/methods/system-usability-scale.html.

Figure 5: Overall Usability Scores for each Participant.

###  4.2 Presence

Summary results for each question of the presence questionnaire are shown in
Table 1.

Table 1: Slater-Usoh-Steed Presence Questionnaire Results. Participants | Questions
---|---
Q1 | Q2 | Q3 | Q4 | Q5 | Q6
Mean | 5.5 | 5.8 | 3.5 | 4.3 | 4.6 | 5
Std Dev | 1.04 | 1.16 | 2.81 | 2.8 | 2.16 | 1.67

###  4.3 Overall Perceptions

On average all questions in the Overall Perceptions questionnaire responses
were above 3.5 in a scale of 1 to 5 where 5 is best.

##  5 Discussion

The results of the usability study indicate that the SURVIVRS prototype is a
usable tool that elicits a high sense of presence and is perceived as
effective and sufficient by non-medical users.

Several points were noted during the experiment. For example, five
participants had difficulty drawing in their first attempt, but drawing became
easier the second time. Also, no participant understood that the play/pause
button also took screenshots. Furthermore, some participants wondered why
zooming out screenshots has a different way than zooming out the videos. This
makes a bad user experience (UX) impression. Also, four out of seven
participants, while trying to click on the patient monitor video to zoom it
in, by mistake, clicked on the home button and got out of the application.
Other comments included slight dizziness (1), need for help (2), slanted 360°
video (3), feeling small (1), and realistic hospital feeling, but lacking
hospital odors (1).

##  6 Conclusion and Future Work

We proposed SURVIVRS, a surgery guidance tool that allows remote expert
surgeons to guide medical teams without high expertise. Our prototype tool
showed positive results, and is a promising first step towards realizing the
full vision of SURVIVRS.

There are several next steps to advance our project. After feedback-based
refinements are made, we plan on testing the user interface with surgeons to
gain domain-specific feedback. We also need to test the tool with a live
stream of a (simulated) surgery and address the many technical challenges
associated with it. At the medium term, we will develop a local surgeon
interface using monitors in the surgery room or smart glasses.

In the long term, we aim to incorporate augmented reality tools in the surgery
room to close the telepresence loop and integrate a representation of the
remote surgeon and spatial in-context annotations.

###### Acknowledgements.

This work is being partially supported by Fundação para a Ciência e a
Tecnologia (Portuguese Foundation for Science and Technology) through grants
2022.09212.PTDC (XAVIER), UIDB/50021/2020 and Carnegie Mellon Portugal grant
SFRH/BD/151465/2021 under the auspices of the UNESCO Chair on AI&XR.

## References

  * [1] L. Argyriou, D. Economou, V. Bouki, and I. Doumanis.  Engaging immersive video consumers: Challenges regarding 360-degree gamified video applications.  In 2016 15th international conference on ubiquitous computing and communications and 2016 international symposium on cyberspace and security (IUCC-CSS), pp. 145–152. IEEE, 2016.
  * [2] H. A. Aziz.  Virtual reality programs applications in healthcare.  Journal of Health & Medical Informatics, 9(1):305, 2018.
  * [3] J. Brooke et al.  Sus-a quick and dirty usability scale.  Usability evaluation in industry, 189(194):4–7, 1996.
  * [4] A. Fuller, T. Tran, M. Muhumuza, and M. M. Haglund.  Building neurosurgical capacity in low and middle income countries.  Eneurologicalsci, 3:1–6, 2016.
  * [5] W. Huang, L. Alem, and F. Tecchia.  Handsin3d: supporting remote guidance with immersive virtual environments.  In IFIP Conference on Human-Computer Interaction, pp. 70–77. Springer, 2013.
  * [6] S. V. Kotsis and K. C. Chung.  Application of see one, do one, teach one concept in surgical training.  Plastic and reconstructive surgery, 131(5):1194, 2013.
  * [7] R. McCloy and R. Stone.  Virtual reality in surgery.  Bmj, 323(7318):912–915, 2001.
  * [8] A. Rizzo, S. T. Koenig, et al.  Is clinical virtual reality ready for primetime?  Neuropsychology, 31(8):877, 2017.
  * [9] M. Usoh, E. Catena, S. Arman, and M. Slater.  Using presence questionnaires in reality.  Presence, 9(5):497–503, 2000.


# Reinforcement Learning Based Neighbour Selection for VANET with Adaptive
Trust Management

Orvila Sarker  University of Adelaide
Cyber Security Cooperative Research Centre
<EMAIL_ADDRESS>Hong Shen  University of Adelaide
<EMAIL_ADDRESS>M. Ali Babar  University of Adelaide
Cyber Security Cooperative Research Centre
<EMAIL_ADDRESS>
###### Abstract

Successful information propagation from source to destination in Vehicular
Adhoc Network (VANET) can be hampered by the presence of neighbouring attacker
nodes causing unwanted packet dropping. Potential attackers change their
behaviour over time and remain undetected due to the adhoc nature of VANET.
Capturing the dynamic attacker behaviour and updating the corresponding
neighbourhood information without compromising the quality of service
requirements is an ongoing challenge. This work proposes a Reinforcement
Learning (RL) based neighbour selection framework for VANET with an adaptive
trust management system to capture the behavioural changes of potential
attackers and to dynamically update the neighbourhood information. In contrast
to existing works, we consider _trust_ and _link-life time_ in unison as
neighbour selection criteria to achieve trustworthy communication. Our
adaptive trust model takes into account the _social relationship_ , _time_ and
_confidence_ in trust observation to avoid four types of attackers. To update
the neighbourhood information, our framework sets the learning rate of the RL
agent according to the velocities of the neighbour nodes to improve the
model’s adaptability to network topology changes. Results demonstrate that our
method can take less number of hops to the destination for large network sizes
while can response is up to 54% faster compared to a baseline method. Also,
the proposed model can outperform the other baseline method by reducing the
packet dropping rate up to 57% caused by the attacker.

###### Index Terms:

VANET, Reinforcement Learning, Blackhole attack, Trust Management System

##  I Introduction

Vehicular Adhoc networks (VANETs) have been showing a burgeoning potential in
facilitating safety critical applications such as emergency warnings, lane
change assistance for developing future smart transportation systems [1, 2].
Large scale deployment of aforementioned applications are based on the
assumption that each vehicle will collect, calculate and disseminate the
information with other vehicles correctly. Decision making dependency on the
perception of information received from other vehicles make VANET prone to
wrong decisions and a wrong decision can lead to fatality [3].

The above discussion implies that decision making about intermediate nodes is
critical. An inappropriate intermediate node selection can incur unnecessary
packet dropping, high transmission delay or unwanted packet loss [4]. Trust
based method is regarded as a widely adopted method in secure decision making
about intermediate nodes in VANET [5]. Trust is a means to quantify the
quality of the information received from other neighbours in the network to
evaluate the authenticity of the received information. However, trust
calculation, establishment, maintenance and updating in VANET are challenging
due to the change in node behaviour, network topology and absence of a
centralised component for network monitoring [6].

Recently, Reinforcement Learning (RL) based methods for intermediate node
selection are being adopted in dynamic wireless networks [7]. RL based methods
have drawn significant attention to the networking community for several
reasons, such as the inherent intelligence they bring into the system, their
capability to cope with dynamic operating environment and suitability to solve
network optimization problems in a distributed manner [8]. Existing studies on
RL based intermediate node selection can be broadly categorised into two main
categories. One category of works (e.g., [9, 10, 11]) has considered the
existence of network adversaries in the network. For instance, existing On-
policy Monte Carlo (ONMC) based routing [12] intended to deal with network
adversaries by prioritising highly secure neighbours for routing to reduce
packet dropping by attackers. In another category of work (e.g., [13, 12, 14,
15]), the design goal is to choose a neighbour that helps optimize one or more
network performance metrics (e.g., packet delivery time [9, 10]).

###  I-A Motivation of this Study

• Entities in VANET may change their behaviour over time. At time $t$, a node
may forward the incoming packets correctly and at time $(t+1)$, it might drop
all the received packets intentionally (e.g., Grayhole attack [16]). The main
motivation of the attacker behind this is to gain benefit from the network
[17]. A trust management system should be effective enough to adaptively
update the trust values of the intermediate nodes based on their dynamic
behavioural pattern [5].

• Safety critical applications are delay and latency sensitive [18], therefore
it is important to optimise the QoS requirements of the network such as
response time and the number of hop counts required to reach the destination.
RL based routing is efficient than conventional non-adaptive algorithms (e.g.,
shortest path routing) for achieving improved Quality of Service (QoS) in
dynamic networks [7]. In RL based intermediate node selection techniques, the
entire network is considered as the environment and each network node acts as
an RL agent (e.g., [9, 10, 13, 12, 14, 15, 11]). The neighbour selection has
been performed based on some important security or QoS requirements of
neighbours. As an example, in Q-routing [9] and CQ-routing [10], a node
calculates, in the form of Q-value, the delivery time its neighbour requires
to deliver packets to the destination. Q-values are stored in a Q-table and
shared among other nodes. Finally, a node selects a neighbour for which the
estimated Q-value is minimum (i.e., the neighbour which needs the least amount
of time to deliver). Considering one or more QoS requirements of the network
to prioritise intermediate nodes can improve network performance, but cannot
guarantee security when attackers are present. For example, an attacker can
share a wrong Q-value with its neighbour to increase its chances of being part
of a route [7]. On the other hand, prioritising intermediate nodes based on
security requirements can improve robustness against attacks [14, 15] but
cannot guarantee QoS [7]. This necessitates the development of a neighbour
selection technique that considers security while not compromising the QoS
requirements of the network.

• A fast update of neighbourhood information is needed to capture the real-
time traffic information on roads. For dynamic networks like VANET, updating
neighbourhood information can be very challenging. Because of the distributed
control property, a global view of the network is not possible [19]. Hence,
the decision making of the node (agent) is entirely based on the local
information available [10]. Learning rate plays vital role in updating the
neighbourhood information in RL based routing. The adjustable learning rate
used in CQ-routing [10] causes excessive memory overhead when the network size
gets larger. ARPRL [11] introduced a variable learning rate considering the
relative velocity difference of neighbours. However, in this method used the
hello packet reception ratio to calculate the reward. Unfortunately, this can
benefit the network adversaries (if any) who intentionally send frequent hello
packets.

###  I-B Contribution of this Study

Our main contributions are: • Development of an adaptive trust model that
calculates the trust value of a node by considering social relationship, time
of the observation and confidence in observation. Our adaptive trust model can
capture the dynamic node behaviour and minimise the malicious effects of four
different types of attackers. • Design of a Q-learning based framework that
utilises both trust and link-life time for intermediate node selection in
vehicular adhoc network. Incorporating trust helps to reduce the number of
times an attacker is selected as an intermediate node and the number of
packets dropped compared to a baseline method. Link-life time helps in
selecting a high stable node. Incorporating security measures (trust) does not
compromise the QoS requirements of the network. Our proposed framework still
achieves good QoS performance compared to a baseline method.

Table I shows the comparison of the proposed framework with some of the
related existing works.

TABLE I: Model comparison of proposed framework with existing works

Characteristics | CQ-routing | [15, 14] | ONMC | Q-routing | ARPRL | Proposed
---|---|---|---|---|---|---
Architecture | Distributed | Centralised | Distributed | Centralised | Distributed | Distributed
Trust Model | - | Direct | Direct | - | - |  | Direct, indirect
---
| Trust
---
calculation metric
- | PDR | PDR | - | - |  | PDR, time,
---
confidence
| Trust value
---
- | - | Fixed | - | - | Dynamic
| Threat
---
model
- |  | Packet
---
disruption
(Blackhole)
| Packet
---
disruption
(Blackhole)
- | - |  | Packet disruption
---
(Blackhole,
Grayhole),
Bad mouthing,
Ballot-stuffing
| Learning
---
algorithm
Q-learning | Deep Q-learning |  | On-policy
---
monte carlo
Q-learning | Q-learning | Q-learning
| Learning rate
---
Adjustable | Fixed | Fixed | Fixed | Adjustable | Adjustable
| Learning
---
agent
| Network
---
nodes
| SDN
---
controller
| Network
---
nodes
| Network
---
nodes
| Network
---
nodes
| Network
---
nodes
| Reward
---
calculation
| Data delivery
---
time
| Trust,
---
position
Reputation |  | Data delivery
---
time
| Link-life,
---
HMRR
| Trust,
---
link-life
| Mobility
---
awareness
No | Yes | No | No | Yes | Yes

##  II Problem Formulation in Q-learning Setting

Our proposed framework employs a model free RL approach namely Q-learning
([20]) to accomplish the neighbour selection task. In Q-learning the agent in
each step observes the next state $s^{{}^{\prime}}$ and calculates the
expected maximum reward $r^{{}^{\prime}}$ for the available set of actions
$a^{{}^{\prime}}$ in $s^{{}^{\prime}}$ to update the Q-value of the
corresponding action in the current state $Q(s,a)$ using the following
equation:

$\small\underbrace{Q(s,a)}_{\text{updated
value}}\leftarrow\underbrace{Q(s,a)}_{\text{old
value}}+\lambda[r^{{}^{\prime}}+\gamma\underbrace{maxQ(s^{{}^{\prime}},a^{{}^{\prime}})}_{\text{maximum
expected value}}-\underbrace{Q(s,a)}_{\text{old value}}]$ (1)

Here, $0<\lambda<1$ and $0<\gamma<1$ are the learning rate and discount factor
respectively. Each node will asses the security status and link stability of
its one-hop neighbours before routing packets to them. The security status of
a neighbour is evaluated based on its trust value. In this work, we consider
two types of packet dropping attacks namely Blackhole and Grayhole attacks and
two types of trust management attacks namely bad mouthing and Ballot-stuffing
attack. Link stability is estimated by calculating the link-life time of a
neighbour. Link-life time is defined as the time a neighbour stays connected
in direct communication before moving out of range. Selecting a neighbour with
a longer link-life time saves both the time that would take to relaunch a new
route and the amount of network resource required [21].

Every node in the network will generate a preliminary set of one-hop
neighbours by evaluating their trust value and store in a trust table. A
Q-table consisting of Q-values of trustworthy neighbours will be formed from
the trust table. In this work, the Q-value of a neighbour is an estimate of
its link-life time for a particular destination. Larger link-life time results
in a large Q-value. To route a packet, a neighbour having a high Q-value is
selected.

VANET is a highly dynamic network where nodes move at high velocities causing
a frequent change in the network topology. Therefore, the estimated Q-value of
a neighbour becomes obsolete when it goes out of range due to mobility. Hence,
an update of the Q-table is required to keep track of the currently available
neighbours [10]. We calculate the learning rate of our Q-learning based
routing model by using the velocity of the mobile nodes in the network. This
adjustable learning rate helps adapt to any changes in network topology.

##  III The Proposed Framework

###  III-A System Model

A VANET can be modelled as an undirected graph $G=\\{V\times E\\}$ defined by
a finite set of vertices $V=\\{n_{1},n_{2},n_{3}...n_{k}\\}$ where $n_{i}$ is
a network node/vehicle (mobile or stationary) and a finite set of edges
$E=\\{e_{1},e_{2},e_{3}...e_{m}\\}$. An edge $e_{ij}$ between node $n_{i}$ and
node $n_{j}$ is defined by the following equation:

$\small e_{ij}=L_{ij}$ (2)

Here, $L_{ij}$ is the link-life time i.e., the remaining time node $n_{i}$
will stay connected to $n_{j}$. Let, $T_{ij}$ = $\\{n_{i}:n_{j},Task\\}$ be
the trust level of node $n_{i}$ on node $n_{j}$ for performing the assigned
_Task_. In this work, the _Task_ is to forward the network packets correctly
by following the protocol rules. Q-value at $n_{i}$ about sending a packet to
the destination $D$ through neighbour $n_{j}$ is:

$\displaystyle
Q_{n_{i}}(D,n_{j})\leftarrow(1-\lambda_{n_{i}n_{j}})Q_{n_{i}}(D,n_{j})+$ (3)
$\displaystyle\lambda_{n_{i}n_{j}}[R_{n_{i}n_{j}}+{\max_{n_{k}\in
N{{}_{T}(n_{j})}}Q_{n_{j}}(D,n_{k})}]$

where $\lambda_{n_{i}n_{j}}$ and $R_{n_{i}n_{j}}$ are the learning rate and
constrained maximum reward respectively. Node $n_{k}$ is a one-hop neighbour
of node $n_{j}$. Each node $n_{i}$ in the network maintains a Q-table
consisting of Q-values $Q_{n_{i}}(D,n_{j})$ that is an estimation of link-life
time of trusted neighbour $n_{j}$ for destination $D$. Each network packet can
be considered as an agent and change of states take place when packets move
from one node to another. Action space consists of trusted one-hop neighbours.
Routing decision is made based on a two-step evaluation of each neighbour
node. At first, node $n_{i}$ generates a trusted neighbour set (action set)
$N_{T}(n_{i})$ from all the one-hop neighbours $N(n_{i})$. Finally, from that
trusted set, $n_{i}$ will calculate the expected maximum reward
$R_{n_{i}n_{j}}$ $(R_{n_{i}n_{j}}=L_{ij})$ to choose an one-hop neighbour
$n_{j}$ as an action if the Q-value $Q_{n_{i}}(D,n_{j})$ for $n_{j}$ is
maximum compared to other one hop neighbours $N(n_{i})$ where $n_{j}\in
N(n_{i})$. The calculation method of trust $T_{ij}$ and link-life time
$L_{ij}$ in our proposed framework are described in Section III-D.

###  III-B Attack Model

In this work, we consider the four types of malicious activities by the
attackers that can potentially compromise the security of the system: •
Blackhole and Grayhole attack: A Blackhole attack is a class of packet
disruption attack where the attacker misleads legitimate nodes by sending them
large sequence numbers and small hop count numbers to be part of a source-
destination route. When the attacker becomes able to be part of an active
route, it captures the packet from the legitimate nodes and drops it. On the
other hand, a Grayhole attacker (alternately on-off attack) drops selective
packets and forwards the rest of them [16]. • Bad mouthing attack: In this
type of attack, an adversary spreads unfair trust ratings about non-malicious
nodes with the intention to decrease their overall trust values in the network
[22]. • Ballot-stuffing attack: In this type of attack, a malicious entity
propagates exaggerated trust value of poorly performed nodes in order to make
the trust management system predict them as highly trusted [23].

###  III-C Neighbourhood Information Update

The quality of the route depends on how closely the Q-values can reflect the
current state of the network. Node velocities are used to calculate our
adjustable learning rate. A node $n_{i}$ moving at a velocity $V_{i}$ will
calculate the learning rate $\lambda_{{n_{i}}n_{j}}$ of its neighbour $n_{j}$
that moving at a velocity $V_{j}$ as follows:

$\small{\lambda_{{n_{i}}n_{j}}}=\frac{|(V_{i}-V_{j})|}{(V_{max}-V_{min})},if{|(V_{i}-V_{j})|}>V_{th}$
(4)

Here, $V_{max}$, $V_{min}$ and $V_{th}$ are maximum velocity, minimum
velocity, velocity threshold respectively. Equation (4) indicates that the
learning rate will dynamically increase when the speed difference between two
nodes is high. And when the speed difference is below a given velocity
threshold, the routing policy will be updated according to a fixed value. Each
node reactively updates its trust table and Q-table through periodically
exchanged HELLO packets and RREQ/RREP messages. When an available route
becomes invalid (due to HELLO packet time-out), the Q-value of a neighbour is
reset to $0$.

###  III-D Neighbourhood Information Calculation

In our proposed trust model, direct trust is calculated by a node itself using
Bayes theorem [24] and recommendations are gathered from one-hop neighbours to
compute the indirect trust using Yager’s rule [25]. Yagers’ rule is a modified
version of well-known Dempster Shafer Theory [26]. Bayes theorem offers the
advantage of incorporating the previous information about a nodes’ trust and
helps to form a prior distribution for future trust calculation [27].
Moreover, it provides good defense against Bad mouthing and Ballot-stuffing
attack [28]. Yager’s rule offers improved performance when multiple
information are completely conflicting [29].

####  III-D1 Direct Trust Calculation

Let, $P=\\{n_{i}:n_{j},Task\\}$ be the probability that node $n_{j}$ will
perform the task assigned by node $n_{i}$, and $Z$ be a random variable that
defines the degree of belief where $0\leq z\leq 1$. If a node receives a total
$N$ number of packets and successfully forwards $l$ packets then the posterior
probability or belief function can be calculated using Bayes theorem using the
equation below:

$\small f(z,N|l)=\frac{P(l|z,N)f(z,N)}{\int P(l|z,N)f(z,N)dz}$ (5)

The likelihood function $P(l|z,N)$ is the probability of forwarding k packets
out of the total N received packets. Assume that $P(l|z,N)$ follows Binomial
distribution:

$\small P(l|z,N)=\frac{N!}{l!(N-l)!}z^{l}(1-z)^{N-l}$ (6)

Assuming Binomial distribution is reasonable in this case as the observation
(trust value of a neighbour) is either success (trustworthy) or failure
(malicious). Also, each observation is fixed and independent. The prior
probability $f(z,N)$ can be calculated as Beta distribution on parameters
$\alpha$ and $\beta$:

$\small f(z|\alpha,\beta)=\frac{z^{\alpha-1}(1-z)^{\beta-1}}{B(\alpha,\beta)}$
(7)

where $\alpha$ and $\beta$ are the number of successful and unsuccessful
packets forwarded by the node respectively and $\alpha$ , $\beta$ $>$ 0\. The
mean $\mu$ i.e., the expected value of a Beta distribution random variable Z
on $\alpha$ and $\beta$ is:

$\small\mu=E(Z|\alpha,\beta)=\frac{\alpha}{\alpha+\beta}\\\ $ (8)

$\small\sigma=Var(Z|\alpha,\beta)=\frac{\alpha\beta}{{(\alpha+\beta)}^{2}(\alpha+\beta+1)}$
(9)

TABLE II: Common notations used in this study

Symbol | Description | Symbol | Description
---|---|---|---
$N$ | Total number of nodes | $S$ | Source/Forwarding node
$T_{th}$ | Trust threshold | $D$ | Destination node
$\alpha_{q}$ |  | Number of packets send at time $t_{q}$
---
$L_{ij}$ | Link-life time between node $n_{i}$ and node $n_{j}$
$\beta_{q}$ | Number of packets dropped at time $t_{q}$ | $R_{n_{i}n_{j}}$ |  | Received reward of node $n_{i}$ for selecting
---
node $n_{j}$
$N_{T}(n_{i})$ | Trusted neighbour list of node $n_{i}$ | $\lambda_{n_{i}n_{j}}$ |  | Learning rate for neighbour $n_{j}$ calculated by
---
node $n_{i}$
$T_{ij}$ | Direct trust of node $n_{i}$ on node $n_{j}$ | $Q_{n_{i}}(D,n_{j})$ |  | Q-value of neighbour $n_{j}$ calculated by node
---
$n_{i}$ for destination $D$
$G_{ij}$ | Indirect trust of node $n_{i}$ on node $n_{j}$ | $V_{i}$ | Velocity of node $n_{i}$
$T_{ij}^{t}$ | Total trust of node $n_{i}$ on node $n_{j}$ | $V_{th}$ | Velocity threshold

####  III-D2 Updating Direct Trust

In order to take into consideration the full ignorance when there is no
observation made about a node at the beginning, the prior probability $f(z,N)$
is assumed to be equal to $B(1,1)$. This means the initial trust value of each
node before any evaluation is equal to 0.5. The trust values can be updated
through continuous observations. After $q$ number of interactions, the
expected value can be calculated as,

$\small E_{q}(Z)=\frac{\alpha_{q}}{\alpha_{q}+\beta_{q}}$ (10)

####  III-D3 Freshness in Direct Trust Estimation

While calculating the trust value, it is useful to give the recent
interactions/observations more weights to minimise the adverse effects of
Grayhole attack [30]. We introduce a diminishing factor to achieve this.
Consider at time $t_{q-1}$, a node has successfully forwarded $\alpha_{q-1}$
packets. At time $t_{q}$, the number of packets forwarded by the node will be
calculated by the following equation:

$\small\alpha_{q}=\alpha_{q-1}.(c)^{\Delta t}$ (11)

where $\Delta t={(t_{q}-t_{q-1})}$ is an integer and $0<c<1$.

####  III-D4 Indirect Trust Calculation

Indirect trust is calculated using Yagers’ rule. A frame of discernment
$\Psi$, consisting of a mutually exclusive and exhaustive set of propositions
are considered. A subset of $\Psi$ including itself and $\emptyset$ is termed
as focal elements where $2^{\Psi}$ = $\\{X_{1},\Psi,X_{n}\\}$ where, $X$ is
any hypothesis or proposition. The probability of a focal set $X$ is a
function m: $2^{\Psi}$ $\rightarrow$ $[0,1]$, m satisfies the following
conditions: $m(\emptyset)=0$, and $\sum_{X_{i}\subseteq\Psi}m(X_{i})=1$.

In our proposed trust model, the security status of a node $n_{i}$ can be
either normal or malicious similar as applied in ([31]). As the security
states of vehicles are mutually exclusive, the frame of discernment is defined
as $\Psi$ = $\\{{\tau},{\tau^{{}^{\prime}}}\\}$ where $\tau$ = $n_{i}$ is
normal node, ${\tau^{{}^{\prime}}}$ = $n_{i}$ is malicious node. Three focal
elements of $\Psi$ are given below:

$\text{Hypotheses}\left\\{\begin{array}[]{lr}H=\tau&\text{if }n_{i}\text{ is
normal}\\\ H^{{}^{\prime}}=\tau^{{}^{\prime}}&\text{if }n_{i}\text{ is
malicious}\\\ U=\Psi&\text{if }n_{i}\text{ either
normal/malicious}\end{array}\right\\}$

Consider in Figure 1, node $A$ wants to send message to node $C$ which is not
directly connected to $A$ and suppose $A$ has no information about $C$ in its
current trust table. In such cases, node $A$ seeks for recommendation to its
one hop neighbours which are node $B$ and node $D$. Recommendations from $B$
and $D$ about $C$ can be any of the following: • _Case 1:_ $B$ recommends $C$
as trustworthy and $D$ recommends $C$ as untrustworthy, • _Case 2:_ $B$
recommends $C$ as untrustworthy and $D$ recommends $C$ as trustworthy, • _Case
3:_ Both $B$ and $D$ recommends $C$ as trustworthy, • _Case 4:_ Both $B$ and
$D$ recommends $C$ as untrustworthy.

For case 1, $B$ supports hypothesis $H$ and $D$ supports hypothesis
$H^{{}^{\prime}}$, for case 2, $B$ supports hypothesis $H^{{}^{\prime}}$ and
$D$ supports hypothesis $H$ and so on. So, there might be uncertainties about
the recommendation node $A$ receives from its direct neighbours $B$ and $D$
which depends upon the security status of $B$ and $D$. This is because it
might be the case that either $B$ or $D$ is providing false recommendation
about $C$ intentionally. In such cases, uncertainties can be reduced by
considering belief and plausibility ([31]) of the observation.

Figure 1: Trust establishment among network nodes

####  III-D5 Trust as Belief and Plausibility

An observer can combine independent pieces of evidences to obtain the degree
of belief about a hypothesis from the corresponding subjective probabilities.
For any subset Y of $\Psi$, the belief function is defined as:

$bel(Y)=\sum_{X_{j}\subseteq Y}m(X_{j})=1$ (12)

It can be interpreted that the basic probability value $m(X)$ is the portion
of the total belief assigned to hypothesis $X$ out of $j$ propositions, which
reflects the evidence’s strength of support. Weight of evidence that does not
refute $Y$ maps each hypothesis Y to a value $Pls(Y)$ between 0 and 1.

$\small Pls(Y)=\sum_{j:X_{j}\cap Y\neq\emptyset}m(X_{j})$ (13)

Here, Pls(Y) denotes plausibility and $Pls(Y)=1-bel(Y^{{}^{\prime}})$

Consider $m_{1}(X)$ and $m_{2}(X)$ are the subjective probabilities from two
independent observers (in the same frame of discernment). Dempster rule to
combine the probabilities is $m(Y)$ = $m_{1}(Y)$ $\oplus$ $m_{2}(Y)$. In our
proposed trust model, the direct trust value is assigned as the basic
probability value of a node. In our example shown in Figure 1, the direct
trust value $T_{ab}$ of node A has on node B is calculated by Bayesian
inference that we discussed in the Section III-D1. If node B claims that C is
a normal node, the basic probability assignment to each hypothesis will be
$m_{B}(H)=T_{ab}$, $m_{B}(H^{{}^{\prime}})=0$, and $m_{B}(U)=1-T_{ab}$.
Alternatively, if node B claims C as malicious, its basic probability to each
hypothesis will be, $m_{B}(H)=0$, $m_{B}(H^{{}^{\prime}})=T_{ab}$, and
$m_{B}(U)=1-T_{ab}$. Table III shows the mass probabilities for each
hypothesis. Therefore, the belief function of each focal set in $\Psi$ can be
calculated as:

$\displaystyle
bel_{B}(H)=m_{B}(H),bel_{B}(H^{{}^{\prime}})=m_{B}(H^{{}^{\prime}}),$ (14)
$\displaystyle bel_{B}(U)=m_{B}(H)+m_{B}(H^{{}^{\prime}})+m_{B}(U)$

TABLE III: Mass function assignment Hypotheses | $m_{B}(H)$ | $m_{B}(H^{{}^{\prime}})$ | $m_{B}(U)$
---|---|---|---
$H=\\{\tau\\}$ | $T_{ab}$ | $0$ | $1-T_{ab}$
$H^{{}^{\prime}}=\\{\tau^{{}^{\prime}}\\}$ | $0$ | $T_{ab}$ | $1-T_{ab}$

####  III-D6 Combining Multiple Recommendations

If $bel_{1}(Y)$ and $bel_{2}(Y)$ are two belief functions on the proposition
$Y$ over the same frame of discernment $\Psi$, then according to Yagers’ rule,
the total belief $bel(Y)$ on proposition Y can be calculated by the following
equation:

$bel(Y)={\sum_{i,j:{X_{i}\cap
X_{j}}=Y}{m_{1}}(X_{i})m_{2}(X_{j})},forY\neq\emptyset,Z$ (15)

$bel(Z)={\sum_{i,j:{X_{i}\cap X_{j}}=Z}{m_{1}}(X_{i})m_{2}(X_{j}})+log(1-k)$
(16)

Here, $k={\sum_{i,j:{X_{i}\cap
X_{j}}\neq\emptyset}{m_{1}}(X_{i}){m_{2}}(X_{j})}$ measures the amount of
disagreement in collected indirect trust opinions.

In the example shown in Figure 1, node $A$ calculate the combined belief of
node $B$ and node $D$ on hypothesis $H$ and $H^{{}^{\prime}}$ by the following
equations:

$\displaystyle\scriptsize m_{B}(H)\oplus
m_{D}(H)=\frac{1}{M}[m_{B}(H)m_{D}(H)+$ (17) $\displaystyle
m_{B}(H)m_{D}(U)+m_{B}(U)m_{D}(H)]$ $\displaystyle
m_{B}(H^{{}^{\prime}})\oplus
m_{D}(H^{{}^{\prime}})=\frac{1}{M}[m_{B}(H)+m_{D}(H^{{}^{\prime}})+$ (18)
$\displaystyle m_{B}(H)m_{D}(U)+m_{B}(U)m_{D}(H^{{}^{\prime}})]$ $\small
m_{B}(U)\oplus m_{D}(U)=\frac{1}{M}m_{B}(U)m_{D}(U)$ (19)

Here,
$M=m_{B}(H)m_{D}(H)+m_{B}(H)m_{D}(U)+m_{B}(U)m_{D}(H)+m_{B}(H^{{}^{\prime}})m_{D}(H^{{}^{\prime}})+m_{B}(H^{{}^{\prime}})m_{D}(U)+m_{B}(U)m_{D}(H^{{}^{\prime}})+m_{B}(U)m_{D}(U)$.

Algorithm 1 Trusted neighbour list creation by node $n_{i}$

1: Input: $T_{th}$, $\alpha_{q}$, $\beta_{q}$, total number of nodes $N$

2: Output: Trusted neighbour list $N_{T}(n_{i})$ of node $n_{i}$

3: for each time stamp $t_{q}$ do

4: Calculate time difference, $\Delta t={t_{q}}-{t_{q-1}}$

5: for each neighbour $n_{j}$ in $N$ do

6: Calculate confidence factor
$\rho=\frac{12\alpha_{q}\beta_{q}}{(\alpha_{q}+\beta_{q})^{2}(\alpha_{q}+\beta_{q}+1)}$

7: Calculate direct trust ${T_{ij}}=\frac{\alpha_{q}}{\alpha_{q}+\beta_{q}}$

8: Calculate indirect trust $G_{ij}$ using Equation (16)

9: Calculate total trust $T^{t}_{ij}=\rho T_{ij}+(1-\rho)G_{ij}$

10: if $T^{t}_{ij}$ $>$ $T_{th}$ then

11: Add neighbour $n_{j}$ to the trusted list $N_{T}(i)$

12: Update $\alpha_{q}$: $\alpha_{q}\leftarrow\alpha_{q}.(c)^{\Delta t}$

13: Update $\beta_{q}$: $\beta_{q}\leftarrow P-\alpha_{q}$

14: else

15: Ignore neighbour $n_{j}$

16: end if

17: end for

18: end for

####  III-D7 Total Trust Calculation

A node $n_{i}$ will calculate the total trust value $T^{t}_{ij}$ of another
node $n_{j}$ by combining the direct observation trust $T_{ij}$ and indirect
observation trust $G_{ij}$ by using the following equation.

$T^{t}_{ij}=\rho T_{ij}+G_{ij}(1-\rho)$ (20)

where $\rho$ is the weighting coefficient we termed as confidence factor. The
indirect trust value $G_{ij}$ is calculated by fusing multiple belief
functions from one hop neighbours using Equation (16). It is important to
provide more weight to the direct observation in the overall trust calculation
to minimize the effect of wrong recommendations coming from malicious nodes.
If $T^{t}_{ij}$ is greater than a trust threshold $T_{th}$, a neighbour is
considered as normal or legitimate. Algorithm 1 demonstrates the overall trust
calculation.

####  III-D8 Calculation of Confidence Factor, $\rho$

If a neighbour successfully forwarded $\alpha$ number of packets out of
received $(\alpha+\beta)$ number of packets, where $\beta$ is the number of
packets dropped, confidence factor $\rho$ is calculated by Equation 21.

$\rho=\frac{12\alpha\beta}{(\alpha+\beta)^{2}(\alpha+\beta+1)}$ (21)

A value of $\rho$ close to 1 indicates high confidence in observation. If the
value of $\rho$ is equal to 0, a neighbours’ total trust value is calculated
only on the basis of direct observation (i.e., direct trust). On the other
hand, if there is no direct trust relationship with a neighbour ($\rho=0$),
total trust is calculated from the recommendation (indirect trust) received
from other neighbours’.

Algorithm 2 Q-value calculation

1: Denotation: Initial state/source = $S$, goal state/destination = $D$,
current state/node = $v_{c}$, neighbour of $v_{c}$ = $n$, neighbour of $n$ =
$n^{{}^{\prime}}$

2: Input: Trusted neighbour set $N_{T}(v_{c})$, $V_{max}$, $V_{min}$,
$T_{th}$, $V_{th}$, $Q(D,n)$

3: Output: Q value of neighbour $n$ calculated by source/forwarding vehicle
$v_{c}$

4: Initialize: Q-table for all $n$

5: Available action set $A_{v_{c}}=N_{T}(v_{c})$

6: for each trusted neighbour $n\in A_{v_{c}}$ do

7: if $(V_{v_{c}}-V_{n})\geq V_{th}$ then

8: Calculate learning rate,
$\lambda_{v_{c},n}=\frac{(V_{v_{c}}-V_{n})}{V_{max}-V_{min}}$

9: else

10: $\alpha_{v_{c},n}=0$

11: end if

12: Calculate constrained reward $R_{v_{c},n}$ = $L_{v_{c},n}$

13:
$Q(D,n)\leftarrow(1-\lambda_{v_{c},n})Q(D,n)+\lambda_{v_{c},n}[R_{v_{c},n}+maxQ(D,n^{{}^{\prime}})]$

14: end for

####  III-D9 Neighbour Link-life Time Calculation

At any given time, the link-life time $L_{{ij}}$ between two nodes $n_{i}$ and
$n_{j}$ is the time the vehicles remain directly connected. For neighbour
link-life time calculation, we adopt the method discussed in [21]. We assume
that each node can collect its location, speed and direction information from
the GPS equipped with it. Also, each node sends information like location,
speed, direction, vehicle id and current time to its directly connected
neighbours through exchanging periodic beacon messages. The distance between
two nodes on the ordinate axis is considered negligible compared to the
transmission range $R$. Let us consider that the two nodes $n_{i}$ and $n_{j}$
are moving with velocities $V_{i}$ and $V_{j}$. Assume that, at time $t_{1}$
they are in the position $(x_{i},y_{i})$ and $(x_{j},y_{j})$, and at time
$t_{2}$ the new positions of the two nodes are
$({x_{i}^{{}^{\prime}}},{y_{i}^{{}^{\prime}}})$ and
${(x_{j}^{{}^{\prime}}},{y_{j}^{{}^{\prime}}})$, respectively. If node $n_{i}$
and $n_{j}$ are moving with a different velocities and they have a constant
acceleration during direct communication, the link-life time $L_{ij}$ between
them can be calculated by the following equation:

${L_{ij}}=\begin{cases}\frac{d^{{}^{\prime}}-d}{V_{i}-V_{j}},&\text{nodes
moving in the same direction}\\\
\frac{d^{{}^{\prime}}-d}{V_{i}+V_{j}},&\text{nodes moving in the opposite
direction}\\\ \end{cases}$ (22)

Here $d$ and $d^{{}^{\prime}}$ are the distances travel in the $x$ direction
at time $t_{1}$ and $t_{2}$ respectively. Algorithm 2 demonstrates the steps
involved in learning and updating the Q-value of a neighbour in proposed
framework.

Figure 2: Main elements of proposed pramework

###  III-E Main Components of Proposed Framework

Figure 2 displays a simplified block diagram of proposed framework. Proposed
framework comprised of the following three main components: • Local
Information Module: The observation buffer in the local information module of
node $n_{i}$ keeps track of the link status and the number of packets
received/forwarded by neighbour $n_{j}$. Based on the number of packets
successfully forwarded by $n_{j}$, node $n_{i}$ adjusts the diminishing factor
$c$ and confidence factor $\rho$ to calculate the direct trust $T_{ij}$ of
$n_{j}$. Indirect trust values are extracted from the remote information
module and combined with direct trust to calculate the total trust $T^{t}$. •
Remote Information Module: The remote information module of a node $n_{i}$
collects trust recommendations about a neighbour $n_{j}$ experienced by other
one-hop neighbours $N(n_{i})$ to calculate indirect trust $G_{ij}$. • Learning
Module: The task of the learning module is to calculate, store and update the
Q-values of trusted one-hop neighbours. Rewards for neighbours are obtained
from the local information module in the form of link-life time. Link-life
time is calculated only for the trusted neighbours. Link status of neighbours
monitored by direct observation buffer is extracted to calculate the learning
rate. The Q-table is of the size $(n_{r}*n_{r})$ where $\\{n_{r},\forall r\\}$
denotes the benign nodes (trust values greater than $T_{th}$) on a given
available route.

##  IV Performance Evaluation

To illustrate the performance of proposed framework, we perform simulations in
ns-2 [32] on Ubuntu 18.04 and Python 3.6 on Windows 10 64-bit operating system
(Intel(R) Core(TM) i7-8700 CPU with 15.8GB memory). Due to the stochastic
nature of the reinforcement learning algorithm, the data points in Figure 5
and Figure 8 are plotted with 90% confidence intervals (each result averaged
over 100 randomly seeded executions). The main simulation parameters used in
this work are tabulated in Table IV. The trust scores of each node obtained
from ns-2 are used to create a trust matrix to simulate the proposed framework
in Python. Similarly, node velocities are stored in a velocity matrix to
calculate the variable learning rate $\lambda$.

TABLE IV: Simulation setup Parameter | Symbol | Value
---|---|---
Network Topology | - | Random and Grid
Node Mobility | - | Static and Mobile
Node Velocities | $V_{i},V_{j}$ |  $(5-35)$ $ms^{2}$
Number of Nodes | $N$ | $4,8,16,32,64$
Maximum Velocity | $V_{max}$ |  $45$ $ms^{2}$
Minimum Velocity | $V_{min}$ |  $5$ $ms^{2}$
Initial Q-Value | $Q_{n_{i}}(D,{n_{j})}$ | $0$

###  IV-A Modelling Trust Variation over Time

Figure 3: Change of direct trust of a node over time

Figure 3 shows how diminishing factor causes the change of direct trust of a
node over time. We consider four cases to demonstrate the change of trust
value of a randomly selected node based on its behavioural changes over time:
• when a node always forwards packets correctly (Figure 3a) • when a node
starts behaving normal from being malicious (Figure 3b) • when a node starts
behaving malicious from being normal (Figure 3c) • when a nodes’ behaviour
fluctuates (normal and malicious alternatively) over time (Figure 3d).

Direct trust of a node decreases over time even if it always forwards packets
correctly (Figure 3a). This helps to capture the freshness in trust
estimation. A nodes’ direct trust increase over time when it turns into normal
(Figure 3b). This gives good nodes the opportunity to regain high trust if
their trust has been decreased previously due to undesirable circumstances
(e.g., environmental changes like bad wireless channel conditions). A nodes’
trust tends to decrease if it changes its’ behaviour from normal to malicious.
On the other hand, a node’s trust fluctuated time to time if it behaves normal
and malicious alternatively (Figure 3d). This change in trust helps to
encourage bad or selfish nodes to be consistent in their normal behaviour.

###  IV-B Accuracy of Trust Estimation

We compare the adaptive trust model with a fixed trust approach where the
values of $\rho$ are 0.2, 0.5 and 0.8 throughout the simulation time. In this
comparison, we have not considered the effect of diminishing factors. This
choice is justified by our intention to demonstrate our adaptive trust models’
capability when time dependency is not considered.

For a highly trusted node who always sends packets correctly (Figure 4a), our
adaptive trust model provides more accurate trust value of the node. On the
other hand, a malicious node who always drops packet (Figure 4b) receives 0
total trust from our trust model. Again when a node drops few packets
unintentionally (e.g., due to bad wireless channel conditions or high
mobility) but the number of forwarded packets are still greater that the
number dropped, it deserved to get a higher trust ratings. Figure 4c shows our
adaptive trust model provides relatively higher trust value for trusted node
like this. Alternatively, our adaptive trust model generates lowest trust
(Figure 4d) in cases when a malicious node forwards negligible amounts of
packets compared to the number of packets it drops.

(a) Trust Value of a Highly Trusted Node

(b) Trust Value of a Blackhole Attacker

(c) Trust Value of a Trusted Node

(d) Trust Value of a Malicious Node

Figure 4: Trust variations of normal and malicious nodes

(a) Effect of fixed and dynamic trust status

(b) Blackhole attacker as a forwarding node

(c) Grayhole attacker as a forwarding node

Figure 5: Performance of proposed framework in security context (total
legitimate nodes = 63, attacker node = 1)

###  IV-C Effect of Fixed and Variable Trust

In proposed framework, a node’s trust dynamically changes over time according
to the value of diminishing factor. It is expected that when a node’s trust
decreases, the number of times it served the role as a forwarding node will
also be reduced. To demonstrate this effect, we introduce a metric
_intermediate node count_ to calculate the number of times a node has been
selected by a source or its neighbour nodes to reach a particular destination.
The intermediate node count of a high stable intermediate neighbour node
should be higher if link-life time is considered for neighbour selection. We
randomly select a node in the network to show how its role as an intermediate
node evolves with its trust status (Figure 5a). When the trustworthiness
remains fixed, its chances of being a forwarder remains stable and
monotonically increases with time. On the other hand, we observed that when we
selectively decrease the trust value in between the simulation time, a node’s
role as a forwarder fluctuates. Intermediate node count is slightly higher
when the trust value of the node is high and it decreases with a decrease in
node trustworthiness.

###  IV-D Robustness Against VANET Attacks

A node’s intermediate node count can be an important metric in the security
context. The total intermediate node counts measure how many times a node was
part of a route. In a routing model, intermediate node counts of an attacker
should be as low as possible to minimise packet dropping. Therefore, we
calculate the intermediate node counts of an attacker to demonstrate the
resiliency of proposed framework against attacks. To illustrate the effect of
the Blackhole/Grayhole attacker, we tested proposed framework on a network of
64 nodes where 63 of them are benign (normal) nodes and 1 is malicious
(Blackhole/Grayhole). Figure 5b and Figure 5c show that the attacker’s chances
of being part of a route are reduced up to 70% compared to a normal node.
Moreover, even though a Grayhole attacker does not drop all the packets it
receives, its count of intermediate nodes is as low as a Blackhole attacker.
This is because proposed framework collects recommendation from other
neighbours in the form of indirect trust which suppresses a Grayhole
attacker’s ability to act as benign and malicious alternatively (on-off).

###  IV-E Robustness against Trust Management Attacks

The objective of this evaluation is to show the robustness of our trust
management system against two trust based attacks, namely Bad-mouthing attack
and Ballot-stuffing attack. False recommendations provided by the attackers
deteriorates the total trust value of good node. Figure 6a displays our
adjustable confidence factor $\rho$ helps the normal node to maintain a high
trust value over time compared to the fixed values of $\rho$. Under Ballot-
stuffing attack, a good nodes’ total trust can be deteriorated because of low
recommendations (in the form of indirect trust) it receives from the
attackers. Figure 6b shows that, the adjustable confidence factor $\rho$
facilitates resiliency against Ballot-stuffing attack and helps to keep the
trust value of a malicious node low compared to fixed values of $\rho$.

(a) Total trust values of a normal node under bad mouthing attack.

(b) Total trust values of a normal node under ballot-stuffing attack.

Figure 6: Trust variations under bad mouthing and ballot-stuffing attack
Figure 7: Effect of link-life time on neighbour selection

(a) Response time required by the agent (total nodes = 64)

(b) Effect of topology changes at episode 2001 (total nodes = 64)

(c) Number of hops required to reach destination

(d) Number of packets received with the size of the network

Figure 8: Performance comparison of proposed framework and ARPRL in terms of
response time, adaptability to network changes, average hop counts and number
of packets received

###  IV-F Effect of Link-life Time

A neighbour is considered highly stable if its’ connection period (link-life
time) with the sender is higher compared to other one-hop neighbours. For this
obvious reason, the intermediate node count should be higher if link-life time
is considered for neighbour selection. The objective of this analysis is to
demonstrate if a node was chosen more times when it was a stable neighbour
with high link-life time than when it was unstable. To analyse the effect of
link-life time, we kept the velocity range of a randomly selected node $n_{i}$
same for a particular simulation period. Then we collected its intermediate
node counts by observing how many times other nodes $n_{w}$ (where $w\in N$
and N is the total number of nodes in the network) choose $n_{i}$ as an
forwarder for both when link-life time was considered and not considered as a
neighbour selection criteria. Figure 7 shows that on average a node receives
more requests to forward packets when it stays connected with the senders for
a longer period of time.

###  IV-G Performance Comparison with ARPRL

To demonstrate the efficacy of proposed framework in terms of achieving QoS
requirements, we compare the performance of proposed framework with a recent
routing technique ARPRL [11]. The reasons behind choosing ARPRL for comparison
are due to the following similarities of ARPRL with proposed framework as
ARPRL has applied 1) Q-learning algorithm, 2) link-life time to select
neighbours, 3) adjustable learning rate to update the neighbour information.
As ARPRL has not considered any security metric, we consider a network of 64
nodes with no attacker for comparison.

####  IV-G1 Response Time

Response time for any RL based neighbour selection algorithm indicates the
time required to learn the topology of the network from scratch for correct
decision making. To simulate the response time, we train a source node at
different training episodes from 50 to 400. Then, calculate the corresponding
response time required by the source to reach the destination (Figure 8a) via
highly stable intermediate nodes. A training episode starts when a node has a
packet to send and ends when the packet reaches to the destination. We observe
that, until 300 training episodes, proposed framework exhibits a faster
response compared to ARPRL. The fixed and adjustable learning rate and a
precise neighbour set adopted in proposed framework let the model learn the
routing policies quicker than ARPRL. The reason that the response time of
ARPRL is almost close to proposed framework after 350 training episodes is
because by that time ARPRL has managed to gather enough information about the
topology of the network which helps the protocol to respond as fast as
proposed framework. In brief, on average a node (RL agent) in our proposed
method responds upto 54% faster than ARPRL method.

####  IV-G2 Adaptability to dynamic network changes

The adaptability of RL based routing algorithm is defined as the model’s
ability to learn a new routing policy when the current route becomes
unavailable due to a change in network topology. To demonstrate the effect of
change in network topology, we first train both proposed framework and ARPRL
for 2000 episodes. Until episode 2000, relative velocities of different nodes
are kept the same to keep the network topology constant. At episode 2001, we
reinitialize the Q-value of a randomly chosen node (which was serving as an
intermediate node to the destination) and re-train both models up to episode
5000. Q-value of a randomly chosen intermediate node (that falls under a
popular route to destination) is re-initialised to capture the effect of a
broken link. We calculate the total reward collected at each episode and to
make the comparison acceptable, we consider the same initial reward value for
both of the methods. We consider average rewards per episode as a performance
metric to demonstrate network adaptability to topology changes. An aggregated
high average reward is an outcome of a reasonably good learned policy. Figure
8b shows that, for proposed framework, the average rewards increases sharply
until the learning converges at episode 400. From episode 400 to episode 2000,
the average reward remains almost stable. On the other hand, the average
reward fluctuates in ARPRL without settling down to a fixed policy. Average
rewards in proposed framework and ARPRL drop when the topology is changed at
2001. However, even though the average reward drops more heavily in proposed
framework, the reward is still higher than that of ARPRL.

####  IV-G3 Average hop counts

Average hop count defines the total number of hops (steps) required by a
packet to reach the destination. The average hop count increases with the size
of the network. Less hop counts are desirable to minimise the latency of
packet delivery. In our network configuration setting, we consider a minimum
of 1 and a maximum of 3 one-hop neighbours for each node. We observe on
average a slight improvement to the average hops count (Figure 8c) for large
network size.

####  IV-G4 Network Packets Received

We capture the number of packets received for both proposed framework and
ARPRL. With the increase in the total number of nodes in the network, the
distance between source and destination extends causing the possibility of
increased channel collisions and decreased connectivity [11]. This results in
low packet reception. Despite the other added requirements and improvements of
the proposed framework, it does not suffer any performance loss to the number
of packets received (Figure 8d).

###  IV-H Performance Comparison with ONMC method

To show the performance improvement of our proposed framework in the security
context, we compare our method with the existing on-policy Monte Carlo (ONMC)
learning based secure routing method [12] in terms of the number of packets
dropped. To keep the percentage of attackers ($20\%$) the same, we use the
same number of normal and malicious nodes as applied ONMC method. We choose
Grayhole attackers as the malicious nodes due to severe security threats they
impose on the overall performance of the network compared to Blackhole
attackers [33]. Figure 9 shows that our proposed proposed framework reduces
the amount of packet dropping up to $57\%$ compared to ONMC routing method.

Figure 9: Performance comparison of proposed framework with ONMC method in
terms of number of packets dropped

###  IV-I Overhead Analysis of Proposed Framework

Execution of proposed framework takes place in two steps: (1) trusted
neighbour list $N_{T}$ formation from all the available one-hop neighbours,
(2) Q-table generation from $N_{T}$ by calculating the link-life of each
neighbour $n$ where $n\in N_{T}$. In worst-case scenario, when all the one-hop
neighbours are normal (trustworthy) in a network of $N$ nodes, time complexity
of the proposed framework becomes $O(N^{2})$. Table V shows an overview of
space and time complexity information of different RL based routing methods.

TABLE V: Comparison of overhead Method | Stored Information |  | Time
---
Complexity
Q-routing | Q-table | $O(N)$
CQ-routing | Confidence and Q-table | $O(N^{2})$
ONMC Routing | Trust and Q-table | $O(N^{2})$
ARPRL | Q-table | $O(N)$
Proposed | Trust and Q-table | $O(N^{2})$

##  V Conclusion and Future Research Directions

This work presents a Q-learning based framework for intermediate node
selection in VANET using trust and link-life time. Our proposed framework can
achieve comparable performance in terms of average hop counts, number of
received packets at the destination and response time against a baseline
method ARPRL. Our proposed trust model considers recent interactions in trust
estimation by taking into account the time of observation and can cope with
four different types of attackers namely Blackhole, Grayhole, Bad mouthing and
Ballot-stuffing attack. We present an adjustable learning rate in the
Q-learning algorithm by taking into account the relative velocities of
neighbour nodes. We demonstrate that our adjustable learning rate can help
network nodes to adapt changes in network topology. Although we validate our
results using widely accepted simulators, a potential future work would be to
investigate the performance of such framework on real world vehicular mobility
traces. Again, the calculation of the learning rate requires the correct
selection of the velocity threshold. How to select an optimal velocity
threshold that will achieve desirable performance in all possible scenarios is
also one of the future research directions.

##  VI Acknowledgment

The work has been supported by the Cyber Security Research Centre Limited
whose activities are partially funded by the Australian Government’s
Cooperative Research Centres Programme.

## References

  * [1] P. Sumayya and P. Shefeena, “Vanet based vehicle tracking module for safe and efficient road transportation system,” Procedia Computer Science, vol. 46, pp. 1173–1180, 2015.
  * [2] A. Wahid, H. Yasmeen, M. A. Shah, M. Alam, and S. C. Shah, “Holistic approach for coupling privacy with safety in vanets,” Computer networks, vol. 148, pp. 214–230, 2019.
  * [3] M. A. Javed and E. B. Hamida, “On the interrelation of security, qos, and safety in cooperative its,” IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 7, pp. 1943–1957, 2016.
  * [4] A. K. Malhi, S. Batra, and H. S. Pannu, “Security of vehicular ad-hoc networks: A comprehensive survey,” Computers & Security, vol. 89, p. 101664, 2020.
  * [5] R. Hussain, J. Lee, and S. Zeadally, “Trust in vanet: A survey of current solutions and future research opportunities,” IEEE transactions on intelligent transportation systems, vol. 22, no. 5, pp. 2553–2571, 2020.
  * [6] M. M. Mehdi, I. Raza, and S. A. Hussain, “A game theory based trust model for vehicular ad hoc networks (vanets),” Computer Networks, vol. 121, pp. 152–172, 2017.
  * [7] R. A. Nazib and S. Moh, “Reinforcement learning-based routing protocols for vehicular ad hoc networks: A comparative survey,” IEEE Access, vol. 9, pp. 27552–27587, 2021.
  * [8] K.-L. A. Yau, P. Komisarczuk, and P. D. Teal, “Reinforcement learning for context awareness and intelligence in wireless networks: Review, new features and open issues,” Journal of Network and Computer Applications, vol. 35, no. 1, pp. 253–267, 2012.
  * [9] J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing networks: A reinforcement learning approach,” in Advances in neural information processing systems, pp. 671–678, 1994.
  * [10] S. Kumar and R. Miikkulainen, “Confidence based dual reinforcement q-routing: An adaptive online network routing algorithm,” in IJCAI, vol. 99, pp. 758–763, Citeseer, 1999.
  * [11] J. Wu, M. Fang, and X. Li, “Reinforcement learning based mobility adaptive routing for vehicular ad-hoc networks,” Wireless Personal Communications, vol. 101, no. 4, pp. 2143–2171, 2018.
  * [12] W. Usaha and K. Maneenil, “Identifying malicious nodes in mobile ad hoc networks using a reputation scheme based on reinforcement learning,” in TENCON 2006-2006 IEEE Region 10 Conference, pp. 1–4, IEEE, 2006.
  * [13] K. Maneenil and W. Usaha, “Preventing malicious nodes in ad hoc networks using reinforcement learning,” in 2005 2nd International Symposium on Wireless Communication Systems, pp. 289–292, IEEE, 2005.
  * [14] D. Zhang, F. R. Yu, and R. Yang, “A machine learning approach for software-defined vehicular ad hoc networks with trust management,” in 2018 IEEE Global Communications Conference (GLOBECOM), pp. 1–6, IEEE, 2018.
  * [15] D. Zhang, F. R. Yu, R. Yang, and H. Tang, “A deep reinforcement learning-based trust management scheme for software-defined vehicular networks,” in Proceedings of the 8th ACM Symposium on Design and Analysis of Intelligent Vehicular Networks and Applications, pp. 1–7, 2018.
  * [16] M.-Y. Su, “Prevention of selective black hole attacks on mobile ad hoc networks through intrusion detection systems,” Computer Communications, vol. 34, no. 1, pp. 107–117, 2011.
  * [17] S. A. Soleymani, S. Goudarzi, M. H. Anisi, M. Zareei, A. H. Abdullah, and N. Kama, “A security and privacy scheme based on node and message authentication and trust in fog-enabled vanet,” Vehicular Communications, vol. 29, p. 100335, 2021.
  * [18] R. Hussain, F. Hussain, and S. Zeadally, “Integration of vanet and 5g security: A review of design and implementation issues,” Future Generation Computer Systems, vol. 101, pp. 843–864, 2019.
  * [19] M. Zhou, L. Han, H. Lu, and C. Fu, “Distributed collaborative intrusion detection system for vehicular ad hoc networks based on invariant,” Computer Networks, p. 107174, 2020.
  * [20] M. Littman and J. Boyan, “A distributed reinforcement learning scheme for network routing,” in Proceedings of the international workshop on applications of neural networks to telecommunications, pp. 45–51, Erlbaum Hillsdale, NJ, USA, 1993.
  * [21] M. Nabil, A. Hajami, and A. Haqiq, “Predicting the route of the longest lifetime and the data packet delivery time between two vehicles in vanet,” Mobile Information Systems, vol. 2019, 2019.
  * [22] S. Kudva, S. Badsha, S. Sengupta, H. La, I. Khalil, and M. Atiquzzaman, “A scalable blockchain based trust management in vanet routing protocol,” Journal of Parallel and Distributed Computing, vol. 152, pp. 144–156, 2021.
  * [23] A. M. Shabut, K. P. Dahal, S. K. Bista, and I. U. Awan, “Recommendation based trust model with an effective defence scheme for manets,” IEEE Transactions on mobile computing, vol. 14, no. 10, pp. 2101–2115, 2014.
  * [24] J. Pearl, Probabilistic reasoning in intelligent systems: networks of plausible inference.  Elsevier, 2014.
  * [25] R. R. Yager, “On the dempster-shafer framework and new combination rules,” Information sciences, vol. 41, no. 2, pp. 93–137, 1987.
  * [26] G. Shafer, “Dempster-shafer theory,” Encyclopedia of artificial intelligence, vol. 1, pp. 330–331, 1992.
  * [27] Z. Wei, H. Tang, F. R. Yu, M. Wang, and P. Mason, “Security enhancements for mobile ad hoc networks with trust management using uncertain reasoning,” IEEE Transactions on Vehicular Technology, vol. 63, no. 9, pp. 4647–4658, 2014.
  * [28] K. Hoffman, D. Zage, and C. Nita-Rotaru, “A survey of attack and defense techniques for reputation systems,” ACM Computing Surveys (CSUR), vol. 42, no. 1, pp. 1–31, 2009.
  * [29] L. A. Zadeh, On the validity of Dempster’s rule of combination of evidence.  Electronics Research Laboratory, College of Engineering, University of …, 1979.
  * [30] Y. L. Sun, Z. Han, W. Yu, and K. R. Liu, “A trust evaluation framework in distributed networks: Vulnerability analysis and defense against attacks,” in Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications, pp. 1–13, IEEE, 2006.
  * [31] T. M. Chen and V. Venkataramanan, “Dempster-shafer theory for intrusion detection in ad hoc networks,” IEEE Internet Computing, vol. 9, no. 6, pp. 35–41, 2005.
  * [32] N. wiki, “The network simulator - ns-2,” May 2016.
  * [33] M. Mohanapriya and I. Krishnamurthi, “Modified dsr protocol for detection and removal of selective black hole attack in manet,” Computers & Electrical Engineering, vol. 40, no. 2, pp. 530–538, 2014.