Knowledge Transfer for Collaborative Misbehavior Detection in Untrusted Vehicular Environments
Roshan Sedar Charalampos Kalalas Paolo Dini Francisco Vazquez-Gallego Jesus Alonso-Zarate Luis Alonso








IEEE Transactions on Vehicular Technology, Vol. xx, No. x, xxx 2024
Shell et al.: A Sample Article Using IEEEtran.cls for IEEE Journals





Knowledge Transfer for Collaborative Misbehavior Detection in Untrusted Vehicular Environments
    Roshan Sedar, 
Charalampos Kalalas, Member, IEEE,
Paolo Dini, Francisco Vázquez-Gallego, 

Jesus Alonso-Zarate, Senior Member, IEEE,
and
Luis Alonso, Senior Member, IEEE
        



    September 9, 2024
=========================================================================================================================================================================================



§ ABSTRACT 


Vehicular mobility underscores the need for  collaborative misbehavior detection at the vehicular edge.
However, locally trained misbehavior detection models are susceptible to adversarial attacks that aim to deliberately influence learning outcomes.


In this paper, we introduce a deep reinforcement learning-based approach that employs transfer learning for collaborative misbehavior detection among roadside units (RSUs). In the presence of label-flipping and policy induction attacks, we perform selective knowledge transfer from trustworthy source RSUs to foster relevant expertise in misbehavior detection and avoid negative knowledge sharing from adversary-influenced RSUs. The performance of our proposed scheme is demonstrated with evaluations over a diverse set of misbehavior detection scenarios using an open-source dataset. Experimental results show that our approach
significantly reduces the training time at the target RSU and achieves superior detection performance compared to the baseline scheme with tabula rasa learning.
Enhanced robustness and generalizability can also be attained, by effectively detecting previously unseen and partially observable misbehavior attacks.










Transfer Learning, Deep Reinforcement Learning, V2X, Trust, Misbehavior Detection






§ INTRODUCTION



The proliferation of connected and automated mobility services, driven by the recent advances in vehicle-to-everything (V2X) technology, leads to a paradigm shift in transport systems. This transformation holds the promise of increased road safety, driving autonomy, and inclusive mobility options.


Inevitably, this 
evolution has given rise to new threat vectors associated with the inherent V2X security vulnerabilities, which adversaries may maliciously exploit to disrupt system operation <cit.>. 

Among them, misbehavior attacks launched by rogue insiders often become difficult to detect and contain, since malicious nodes may alter their activity intelligently
over time <cit.>.

Although cryptographic techniques are capable of limiting outsiders by offering authentication, integrity, and non-repudiation as a first layer of defense, they may fall short in detecting rogue/dishonest behavior and identifying V2X insiders with malicious intent.

As such, the trustworthiness of exchanged information cannot be guaranteed.


Emerging data-driven approaches fueled 
by artificial intelligence and machine learning (AI/ML) tools provide a fertile ground for addressing misbehavior attacks <cit.>.

With the availability of vehicular data streams, AI/ML-based schemes can facilitate the analysis of behavioral patterns for V2X entities and determine trustworthiness levels. 

Such capabilities have the potential to overcome the shortcomings of traditional misbehavior countermeasures, offering effective solutions to achieve demanding security requirements.

Centralized detection approaches, albeit leveraging the entire set of available information fused
from multiple geographical locations to reduce the risk of false positives, unintentionally introduce higher computational cost and latency <cit.>. 

This cogently justifies the use of distributed and decentralized AI/ML-based schemes for collaborative misbehavior detection. By diffusing distributed intelligence across vehicular edge components, such as roadside units (RSUs), resource efficiency gains are sought while detection time can be dramatically reduced.

Despite the envisioned benefits of data-driven misbehavior detection, the vulnerabilities of AI/ML models introduce additional threat vectors, giving rise to finely targeted, stealthy, and scalable adversarial attacks. 

Such sophisticated attack types may target both model training (i.e., poisoning
attacks) and test (i.e., evasion attacks) phases <cit.>, and undermine the efficacy of misbehavior detection.

Collaborative misbehavior detection, relying on decentralized learning models at the vehicular edge, may further extend the attack surface and, thus, exacerbate the impact of adversarial attacks.

Consequently, the pervasive adoption of AI/ML models for misbehavior detection could be hindered if security concerns related to the model vulnerabilities are not addressed.













 §.§ Motivation



Aiming to address the complex V2X security landscape, several recent works leverage AI/ML-driven techniques for misbehavior detection <cit.>. Supervised learning schemes may be impractical in V2X scenarios with an expanded attack surface, due to limited access to labeled
training examples and/or dependence on security threshold
values. On the other hand, unforeseen alterations in vehicular mobility, due to
either naturally drifting traffic patterns or unprecedented
malicious activity, introduce challenges (e.g., model overfitting) to misbehavior detection schemes relying on conventional deep learning (DL). In this context, deep reinforcement learning (DRL) methods emerge as a compelling option 
for context-aware misbehavior detection <cit.>.
By allowing the learner to infer optimal sequential decisions based on rewards/penalties received as a result of previous actions and accumulated experience, the performance of DRL schemes can be dynamically improved in rapidly changing environments.


A number of reputation or trust-based approaches have been proposed in relevant literature to elevate trustworthiness levels in untrusted vehicular environments. Yet, their applicability in collaborative misbehavior detection may be limited, since such methods often assume infrastructure nodes (e.g., RSUs) to be legitimate and non-susceptible to adversarial attacks. Accumulating behavioral patterns of specific nodes over a time period may also be challenging due to ephemeral V2X connectivity links <cit.>.
The dependence on predefined trust thresholds may further restrict the suitability of such approaches. 



It is plausible that decentralized learning architectures for collaborative misbehavior detection inadvertently make the underlying models attractive targets for adversarial attacks. 

In turn, locally trained misbehavior detection models can be influenced to reach incorrect decisions/predictions or leak confidential information. For instance, in data poisoning attacks <cit.>, an adversary aims at deliberately modifying the learning algorithm during training via false data injection or manipulation. 

Surprisingly, the detrimental impact of such adversarial manipulations on misbehavior detection performance remains rather unexplored. Dealing with adversarial attacks is often non-trivial and requires
enhanced solutions to foster trust and stimulate confidence in
data-driven misbehavior detection.






 §.§ Contributions


Motivated by the research challenges mentioned earlier, our contribution can be summarized in three main aspects:

    
  * We introduce a novel scheme for collaborative misbehavior detection that builds on geographically distributed RSUs. Each RSU employs a DRL-based model for the detection of malicious traffic stemming from misbehaving vehicles. Leveraging transfer learning principles, the knowledge learned at source RSUs is shared with the target RSU to reuse relevant expertise for misbehavior detection.
    
    
  * In the presence of two data poisoning attacks with the aim of influencing misbehavior detection outcomes incorrectly, we perform selective knowledge transfer from trustworthy source RSUs to avoid negative knowledge sharing from adversary-influenced RSUs.
    
    For this purpose, a novel trust evaluation metric, referred to as semantic relatedness, is used by the target RSU to quantify the trust level of each source RSU for collaborative misbehavior detection.
    
  * Considering diverse scenarios of collaborative misbehavior detection and an open-source dataset, we evaluate the learning performance of involved RSUs
    in the presence of adversaries. The detection performance achieved via selective knowledge transfer is also assessed for different misbehavior types. Besides reducing the training time at the target RSU, our scheme is shown to significantly outperform the baseline scheme with tabula rasa learning, demonstrating its high effectiveness. Interestingly, our approach enhances robustness and generalizability by effectively detecting previously unseen and partially observable misbehaviors.





The remainder of this manuscript is organized as follows. Section <ref> provides an overview of the existing work in the related literature. Section <ref> outlines the formulation of our proposed misbehavior detection approach, including details about the network and adversarial models considered in the study. Section <ref> presents the key components of our collaborative misbehavior detection scheme which incorporates transfer learning. Section <ref> details the experimental setup and the examined scenarios to validate our approach. Section <ref> evaluates the proposed solution and provides a detailed discussion of the obtained results. Section <ref> summarizes our final remarks and outlines potential directions for future work.         




§ RELATED WORK





This section summarizes pertinent studies in the literature, with a focus on highlighting open issues and key challenges.











 §.§ Misbehavior Detection










Although a multitude of security mechanisms for misbehavior detection are available, existing solutions are rather limited in effectiveness due to a high 
number of false alarm rates (i.e., false positives and negatives) 
and lack of robustness against

sophisticated adversarial attacks <cit.>. Interestingly, 
such key open issues underscore the need for advanced data-driven techniques to achieve 
effective misbehavior detection.

Data-driven AI/ML approaches 
are a proving ground 

for misbehavior detection in vehicular networks. Recent works in this area, supervised <cit.>, unsupervised <cit.>, and DRL-based <cit.> methods

stand out among others and are positioned at the forefront of the 
research interest in V2X security. 



Next, we summarize existing works on centralized and distributed data-driven AI/ML-based misbehavior detection.





  §.§.§ Centralized Approaches
 

Several 
ML-based misbehavior detection schemes 
fall into the category of centralized, where models train offline centrally with aggregated training samples collected from multiple vehicles and detect misbehaviors (i.e., downstream task) locally in a decentralized way.  

Centralized ML model training is performed in <cit.>
using feature vector inputs that originate from the position and movement-based plausibility and consistency checks performed at vehicles. Utilizing trained ML models, all participating nodes perform local detection and share the detected events with a central misbehavior authority to make the global decision on misbehaviors.


Results obtained using the open-source VeReMi dataset <cit.> show that a long short-term memory (LSTM) deep classifier outperforms support vector machines (SVM) and multi-layer perceptron models. 

Another work in <cit.> utilizes position-related features in VeReMi to train an ensemble supervised learning scheme centrally and detect locally at vehicles or RSUs. 
Numerical results reveal 
improved detection performance compared to benchmark 
ML-based approaches.   

DL techniques have recently gained widespread attention in cyber-threat detection due to their feasibility and superior performance over traditional ML <cit.>. 

The work in <cit.> presents a DL-based unsupervised 
method

for identifying misbehaving vehicles. The authors implement various DL architectures by combining different neural network models such as convolutional (CNN), LSTM, and gated recurrent units. In the proposed setup, vehicular messages 

are aggregated and converted into sequences at the RSU and fed to centrally trained DL models on the cloud to detect misbehaving vehicles. 
Numerical results based on VeReMi demonstrate that 
the proposed DL-based scheme 

outperforms
existing ML approaches. In a similar context, an edge-cloud approach is proposed in <cit.>, which comprises DL engines of CNN and LSTM neural networks for handling misbehaving vehicular traffic. Time-sequence-based and sequence-image-based classification methods are implemented at the edge-cloud to train and detect. Results show that sequence-image-based classification using CNN performs better compared to 
traditional ML-based approaches.

Recent works in 
DRL-based misbehavior detection 
 <cit.> stand out from their conventional ML/DL counterparts

due to their superior detection performance, robustness to noisy training data, and ability to dynamically improve detection accuracy 

in highly volatile V2X environments. DRL utilizes deep neural networks (DNNs) to approximate the state-action value function, which, in turn, the DRL agent effectively learns to map input vehicular traffic to state-action values (Q-values). In <cit.>, messages transmitted by 
vehicles are aggregated at RSUs and offload the training of the DRL model onto the cloud server. Misbehavior detection is then performed 
at RSUs using the trained model. 
Performance assessment of the

proposed DRL-based scheme 
using VeReMi reveals

enhanced 
detection outcomes 
compared to benchmark 
ML- and DL-based approaches and robustness to noisy training samples. 




  §.§.§ Distributed Approaches
 
The studies 
that fall into this category execute the offline training of ML models in a distributed or decentralized fashion, either locally in vehicles' on-board units (OBUs) or in RSUs. 

The authors of <cit.> exploit the position and movement-based plausibility checks at the vehicle to extract feature vectors and feed them into locally trained supervised ML models such as k-nearest neighbor and SVM. 

Results reveal that ML models yield higher accuracy with multiple plausibility checks.

The work in <cit.> uses an identical set of plausibility checks used in <cit.> to extract relevant features and feed 
them into an extended set of supervised ML models. The authors suggest implementing their scheme in OBUs to facilitate local detection in real-time. Numerical results show that adding 
plausibility checks improves recall and precision.


In <cit.>, the authors train supervised ML models at the RSU by aggregating basic safety messages (BSMs) 
from vehicles in their communication range. Using an augmented feature set that combines information from successive BSMs in VeReMi, they achieve improved detection performance for position falsification attacks compared to existing ML-based approaches. 

Another work in <cit.> trains a local misbehavior detection system by using supervised ML models to identify false alert transmission and position falsification. By utilizing trained ML models, each vehicle performs local detection and evicts misbehaving vehicles based on aggregated information from all neighboring vehicles. Results show improved detection performance compared to rule- and threshold-based detectors. 

In summary, existing AI/ML-based misbehavior detection models can be either centrally trained in the cloud and tested locally, or both training and detection are realized in vehicles/RSUs.
Yet, these models 

are susceptible to adversarial attacks, e.g., data poisoning <cit.>. As such, an attacker may poison the centralized model training pipeline and influence downstream tasks at vehicles or RSUs to misclassify, which could inflict a single point of failure. Similarly, locally trained misbehavior detection models may suffer from data poisoning and adversarial manipulations owing to the extended attack surface. Hence, albeit offering enhanced detection performance, such models may not be robust enough against adversarial attacks. Moreover, they may have limited capability in detecting unseen 
(i.e., non-anticipated) misbehavior attacks. On the contrary, we propose distributed collaborative learning to enhance misbehavior detection performance, making it robust against adversarial attacks and capable of generalizing to detect unseen attacks.





 §.§ Trust Evaluation
 



Data-driven misbehavior detection models analyze the transmitted messages either individually at a 
local level or collaboratively at a global level. They focus on verifying the semantic correctness of the received information regardless of the sender and do not rely on the honest majority of senders. Entity-centric models, on the other hand, focus on monitoring the past behavior of nodes over time to assess their trustworthiness <cit.>. Such models primarily constitute trust-based detection techniques where each entity maintains a local table to record the reputation or trust indicators of its neighboring vehicles, and they often assume the majority of senders to be honest. 


In <cit.>, the authors evaluate the trustworthiness of vehicular data and nodes using the Dempster-Shafer belief framework. A vehicle's trust is evaluated on both functional and recommendation trust. 
Functional trust directly evaluates the trustworthiness of a vehicle, and recommendation trust evaluates the trustworthiness of neighboring vehicles. Results reveal 
superior performance compared to a conventional weighted-voting method for 
trust management. Another similar work in <cit.> presents a trust-based mechanism against DoS/DDoS attacks, with vehicles assessing neighboring senders via direct and indirect trust values.

In general, these mechanisms can be integrated with data-driven AI/ML-based methods to enhance trustworthiness in misbehavior detection.   

The work in <cit.> proposes a context-aware data-driven trust management scheme to ascertain the integrity of information received by vehicles. The authors employ information entropy theory to calculate the trustworthiness of the content of messages and incorporate an RL

model based on Q-learning to continuously adapt the trust evaluation strategy to accommodate various driving scenarios. 

Numerical results indicate the ability to withstand bogus information, outperforming 

entity-centric trust schemes. 

Another work in <cit.> presents a DRL-based dynamic reputation update mechanism for vehicular 

networks. The authors apply the Dempster-Shafer theory to combine misbehavior detection reports sent by vehicles centrally at a global level and train a DRL model to update the reputation policy. The DRL model dynamically determines the optimal reputation update value to stimulate vehicles to send true feedback. Simulation results  exhibit improved 
performance compared to entity-centric trust schemes.   

In summary, most reputation or trust-based methods in related literature are entity-centric and often rely on infrastructure assistance (e.g., RSUs), assuming the presence of an honest majority of participants. Furthermore, these methods frequently resort to predefined conditions, ranging from task-specific rule sets to statically configured threshold values, and assume 
infrastructure nodes to be 
benign and immune to adversarial attacks. In addition, monitoring nodes 
over an extended period to evaluate trustworthiness often becomes 
challenging due to the transient connections and mobility patterns of vehicles. Overall, trust management models are not sufficiently robust against adversarial attacks. However, our proposed approach in this work does not rely on predefined conditions, such as an honest majority of participants or statically configured threshold values, to determine trustworthiness. 










 §.§ Adversarial Attacks

With the increasing penetration of data-driven methods into V2X systems,

AI/ML models inevitably become 
prone 
to adversarial manipulations
 <cit.>. In an adversarial ML setting, attackers inject adversary data intending to misguide AI/ML models to incorrect decisions. In this context, the prevalence of adversarial attacks hinders the robustness of AI/ML models, while 
dealing with such 
attacks is often non-trivial. Inherently, previously described misbehavior detection (Sec. <ref>) and trust (Sec. <ref>) models are vulnerable to adversarial ML. 


Poisoning and evasion attacks can be identified as the main adversarial attack scenarios <cit.>. The contamination of training data, 
known as poisoning, 
occurs during the training phase 
of the AI/ML model. In this type of attack, the attacker attempts 
to inject an imperceptible perturbation into the input data, which corrupts training 
and misguides the AI/ML model <cit.>. 

Similarly, label-flipping can also be used as a poisoning attack, yielding a simple and effective way to contaminate the training data <cit.>. In this attack, the adversary selects a subset of training samples and flips their labels. In contrast, evasion attacks take place at the testing phase, 
where the adversary has no influence on the target model during training, but rather tricks the model into incorrect outputs. 

The work in <cit.> demonstrates the vulnerability of ML-based misbehavior detection models against adversarial attacks. The authors inject carefully crafted adversarial examples using ML and DL techniques and compromise the detection model to misclassify attack samples. In this approach, ML and DL models are first trained on the genuine behavior dataset, followed by generating adversarial examples. The evaluation using VeReMi dataset reveals 
high false positive and false negative values. Another work in <cit.> 
introduces a data poisoning attack in DL models, 


which is equipped with attacker-chosen patterns to force misclassification. 

Such attacks can take place when the training is outsourced onto the cloud or when using pre-trained models in transfer learning. Similarly, DRL-based models are also susceptible 
to adversarial attacks. 
In <cit.>, the authors present a policy induction attack by injecting crafted adversarial examples into the agent's state space. 

Results show that the agent's average reward per epoch remains at low levels in the presence of adversarial examples.        








Summary: Considering the open research gaps highlighted throughout the previous state-of-the-art overview, we hereby focus on collaborative misbehavior detection by transferring the knowledge learned between distributed entities in untrusted vehicular environments.   

Our proposed DRL-based scheme can select trustworthy RSUs to collaborate and transfer knowledge withstanding adversarial attacks. Consequently, target or receiving entities can learn efficiently from transferred knowledge to discover unseen and partially observable misbehaviors.








§ SYSTEM MODEL



This section introduces the vehicular network model, misbehavior detection model, and adversarial model considered in this work. We provide the details in the following.



 §.§ Network Model



Vehicular networks typically comprise a

large number of geographically distributed RSUs. 
RSUs are
stationary entities interconnected with each other and the Internet. The usability of RSUs is multifaceted 
as they offer various services such as Internet access, security solutions, and real-time traffic data distribution <cit.>. Normally, RSUs 
have superior computational capabilities 
compared to 
in-vehicle resources, which allows vehicles to offload 
computation-intensive 
tasks to RSUs. 

Fig. <ref> illustrates 
the distributed vehicular network model considered in this work. 
Each RSU experiences 
its own vehicular environment and receives traffic 
within its coverage. 

The RSUs are interconnected via wired connections to provide reliable RSU-to-RSU communication. We assume authentication and authorization procedures have already been performed before V2X communication takes place 
among entities <cit.>.
The involved vehicles (i.e., authenticated and authorized) periodically broadcast BSMs, which are 
received by 
RSUs in their 
connectivity range. BSMs include standard-related parameters such as position, speed, acceleration, heading angle, and other relevant vehicular information <cit.>.     

In our proposed setup, 
a distributed collaborative misbehavior detection system is considered where we leverage knowledge transfer in the context of transfer learning. As shown in Fig. <ref>, each RSU is equipped with a DRL-based misbehavior detection system (DRL-MDS) and detects misbehaving vehicles. 

Misbehavior detection is 
performed at the RSU level, since the vehicle may not have complete information in its communication range
due to ephemeral

connectivity and/or limited computational resources. 

A high-level illustration of the DRL-MDS workflow is 
provided in Fig. <ref>,  

and it is further elaborated 
in the following section (Sec. <ref>). The knowledge learned at source RSUs is transferred to the target RSU to reuse existing knowledge during its learning process. The target RSU may not necessarily need to be a neighbor, but it could also be a distant RSU, which shall be reusing the available knowledge of sources to detect future misbehaviors. 



It is noteworthy that RSU-based security solutions in related literature tend to assume that RSUs are trusted entities and non-vulnerable to attacks <cit.>. However, as pointed out in Sec. <ref>, stationary entities such as RSUs can easily be targeted by adversarial attackers. In this work, we assume that there may exist RSUs that are subject to adversarial attacks.         






 §.§ Misbehavior Detection Model
 








In our approach, each RSU is equipped with an MDS as shown in Fig. <ref>. The DRL-based model, introduced in our previous works <cit.>, is used for detecting misbehaving vehicles. Here, we present the formulation of the tabula rasa DRL model used for misbehavior detection. A tabula rasa model aims to learn efficiently from scratch without any external/previous knowledge. The details of transfer learning incorporation into the DRL model are provided in Sec. <ref>.





  §.§.§ DRL Model
 

The vehicular environment considered in this study follows a Markov decision process (MDP) framework to facilitate the detection of misbehaviors through sequential decision-making. Consistent with the MDP formulation, the action of misbehavior detection changes the environment based on the decision of either genuine or malicious behavior at time-step t. Subsequently, the decision at time-step t+1 is influenced by the altered environment from the previous time-step t. 


The aggregated vehicular traffic at each RSU consists of a time-series repository of received BSMs with intrinsic spatiotemporal interdependencies.  In this work, we consider a DRL-based misbehavior detector deployed at the edge RSU. As depicted in Fig. <ref>, the detector (agent) interacts with the vehicular environment to learn the optimal detection policy π^*. 



During training, the ϵ-greedy method is leveraged to strike a balance between exploration and exploitation in the agent's strategy. Next, we describe the components relevant to the DRL model.



i) Agent: The agent receives the vehicular traffic data as a time series and prior related decisions as inputs (i.e., state 𝐬_𝐭), and generates the new decision made (i.e., action a_t) as output, as shown in Fig. <ref>. The agent's DQN <cit.> comprises an LSTM layer and a fully connected neural network with linear activation to generate Q-values as choices for the action a_t. At each time-step t, the agent's actions are selected by the policy π. The agent's experience, i.e., e_t = <𝐬_𝐭,a_t,r_t,𝐬_𝐭+1>, stores all the behaviors of the misbehavior detector. By exploiting experience, the detector is progressively improved to obtain a better estimation of the Q(𝐬,a) function. The objective is to maximize the expected sum of future discounted rewards by learning the π^*. The discounted reward return is expressed as 



    R_t = ∑_k=t^Tγ^k-tr_k.

Q-learning model updates are performed with learning rate α and discount factor γ as

    Q(𝐬_𝐭, a_t)    ← Q(𝐬_𝐭, a_t)
       +α(r_t + γmax_a_t+1 Q(𝐬_𝐭+1, a_t+1) - Q(𝐬_𝐭, a_t)).




ii) States: The state space comprises the sequence of previous actions denoted by s_action = <a_t-1,a_t,...,a_t+n-1>, and the current BSM information denoted by s_time = <X_t,X_t+1,...,X_t+n>. 𝐗_𝐭∈ℝ^d is a d-dimensional feature vector at time-step t, including information on d different features. According to the state design, the next action taken by the agent depends on the previous actions and the current BSM information. This design enables the agent to capture temporal dependencies and make more informed decisions. 

iii) Actions: The action space is defined as 𝒜 = {0,1}, where 1 indicates the detection of a misbehavior and 0 represents the genuine behavior. The deterministic detection policy π can be expressed as a mapping, i.e., π:𝐬_𝐭∈𝒮⟼ a_t∈𝒜,  from states to actions, where π(s) prescribes the action that the agent takes at state 𝐬.  

In a given state 𝐬_𝐭, the agent selects the action based on the optimal detection policy given by

    π^* = _a ∈𝒜 Q^*(𝐬,a).


iv) Rewards: 

The reward function R(𝐬,a) is defined based on the confusion matrix typically used in ML classification problems. A numerical value for r_t is assigned based on the ground truth information of BSMs. A positive reward is given upon correct detection of a misbehavior, i.e., true positive (TP), or a normal state, i.e., true negative (TN). A negative reward is given when a normal state is incorrectly identified as a misbehavior, i.e., false positive (FP), or a misbehavior as a normal state, i.e., false negative (FN). The agent is penalized more for FN actions than for FPs, as the correct identification of misbehavior is necessary to avoid hazardous situations. Accordingly, we define the immediate reward r_t∈ R of the agent as



    r(𝐬_𝐭,a_t) = 
                    a,    if a_t is a TP,
    
                    b,    if a_t is a TN,
    
                    -c,    if a_t is an FP,
    
                    -d,    if a_t is an FN,

where a, b, c, d > 0,  with a > b and d > c. 




 §.§ Adversarial Model



In this work, 
we assume the presence of adversaries attempting to contaminate the training data and realize poisoning attacks on the distributed and collaborative DRL-MDS models. In particular,
we consider two data poisoning attacks pertinent to adversarial ML: i) label-flipping and ii) policy induction attacks. 


In the case of label-flipping, it is assumed that the adversaries are rogue insiders who contribute to the training data or have access to 
the training data itself. In policy induction, we consider an exogenous attacker who can modify the state space before it is observed by the DRL agent. In both cases, attackers aim to maliciously influence the learning model and, subsequently, force incorrect outcomes in downstream misbehavior detection tasks. 

In a label-flipping attack, we consider an 
attacker who targets the malicious class to flip the labels of certain
source training data instances at an RSU. In this scenario, the adversarial attacker randomly selects a set of misbehaving vehicles and flips the labels of their data instances into genuine ones, resulting in a targeted random label-flipping attack (Definition <ref>). The attacker aims to misclassify selected training samples from malicious class 1 to genuine class 0. Following Definition <ref>, we denote D_S_i as the training data of source RSU S_i, i.e., D_S_i = {(x_S_i1, y_S_i1),..., (x_S_ik, y_S_ik),..., (x_S_in, y_S_in)}, with x_S_ik∈ X_S_i representing the k-th data instance of D_S_i while y_S_ik∈ Y_S_i represents the corresponding label of x_S_ik. 




Given training samples {x_S_ik, y_S_ik}^n_k=1 of D_S_i, belonging to source RSU S_i with x_S_ik∈ X_S_i and y_S_ik∈ Y_S_i= {0, 1},

a poisoning attack is defined as

a targeted random label-flipping attack when the attacker randomly selects a fraction ζ∈ (0, 1] of misbehaving vehicles and flips the label of the corresponding training samples from 1 to 0.  


In a policy induction attack, we properly adapt the adversarial attack originally presented in <cit.> in a DRL context. 

The malicious intent here is to force the target DRL agent
to learn a policy selected by the adversary.


We assume an exogenous attacker with minimal a priori information (e.g., input type and format) of the target DQN,

and with knowledge of its reward function and the update frequency of the target 
network. The attacker can directly manipulate the target DQN's environment configuration, albeit with 

no control over the target network parameters, reward function, or optimization mechanism. According to 

Definition <ref>, the attacker creates replica DQNs (i.e., Q', Q̂') of the target's DQN (i.e., Q, Q̂) and initializes them with random parameters. Since the attacker has no knowledge of the target's DQN architecture and its parameters at every time step, a black-box technique is followed to exploit the transferability of adversarial examples. This is achieved by crafting state perturbations using replicas of the target's DQN such as those introduced in <cit.>. Moreover, the Fast Gradient Sign Method (FGSM)[

The FGSM method utilizes the neural network gradients' sign to create a perturbed adversarial input that maximizes the loss <cit.>. For a given input x, the adversarial input x^' generated by the FGSM can be summarized as x^' = x + ϵ * sign(∇_xJ(θ,x,y)), where y is the label of input x, ϵ denotes a small multiplier, θ are the model parameters, and J represents the loss.] algorithm is used to craft adversarial examples at every training time step. 








The attacker induces an arbitrary adversarial policy π_adv on the target DQN by injecting adversarial examples into training data. Specifically, given state observation 𝐬_𝐭+1, the attacker crafts a perturbation vector (δ̂_𝐭+1) using FGSM at every training time-step and injects it into the target DQN's state space, as


    
    
    a'_adv   ←π_adv(𝐬_𝐭+1),
    δ̂_𝐭+1   ← Craft(Q̂',a'_adv,𝐬_𝐭+1),
    
    𝐬'_𝐭+1   ←𝐬_𝐭+1 + δ̂_𝐭+1.

Crafting adversarial inputs 𝐬'_𝐭 and 𝐬'_𝐭+1 requires minimizing the loss function in (<ref>) to force the target DQN to optimize towards action a', given state s_t, as
    
    
    min_θ' (y_t - Q'(𝐬_𝐭,a_t;θ'))^2,

where y_t = r_t + γ max_a'Q̂' (𝐬'_𝐭+1,a';θ'_-) and Q^', Q̂^̂'̂ are the replica DQNs of the target DQN.










§ TRANSFER LEARNING-BASED DRL FRAMEWORK
 






The distributed RSU deployments in vehicular 
systems and the varying spatiotemporal

behavior of 
traffic traces, 
render the training of a DRL-MDS agent difficult, especially in


detecting unseen and partially observable misbehavior attacks. 
Challenges in such complex setups include slow convergence of the learning process, overfitting, and/or sub-optimal solutions due to poor exploration <cit.>.
Hence, we advocate 

the adoption of a transfer learning-based DRL approach to achieve distributed collaborative misbehavior detection in adversarial vehicular environments. We rely on selectively transferring knowledge from trustworthy source RSUs to avoid negative knowledge sharing from adversary-influenced RSUs, as shown in Fig. <ref>. 

















 §.§ Transfer Learning for DRL-based Misbehavior Detection




TL leverages valuable knowledge acquired in one task and past experience to improve learning performance on other tasks (similar/dissimilar). The knowledge learned and previous experiences obtained from learning some source tasks, can be re-used to enhance the learning of some target tasks. In our work, the task represents the misbehavior detection performed in each RSU. 
Most DRL-based works in related literature (Sec. <ref>) have developed agents that can efficiently learn tabula rasa without any previously learned knowledge. Yet, tabula rasa DRL techniques require a long learning period to be effective, which can be inefficient in computationally intensive and mission-critical operations <cit.>. 


For instance, an agent at the start of learning may tend to spend significant time exploring the environment before finding an optimal policy. Thus, TL can be leveraged to accelerate the learning.





In this context, source and target RSUs associated with TL can be represented as MDP[Henceforth, MDP and environment are used interchangeably.] agents. Accordingly, TL between RSUs can be defined as follows.








Given a set of m source RSUs with MDPs 𝐌_s = {⋃_i=1^mℳ_i,s|ℳ_i,s∈𝐌_s} and a target RSU with MDP ℳ_t, the goal of TL is to learn an optimal policy π^*_t for the target RSU as 


        
    π^*_t = _π_t_𝐬∼𝒮_𝐭,a ∼π_t [Q^π_t_ℳ_t(𝐬,a)],

by leveraging external information ℐ_s from 𝐌_s along with internal information ℐ_t from ℳ_t.

In (<ref>), 
π_t = ϕ(ℐ_s∼𝐌_s, ℐ_t∼ℳ_t):𝐬_𝐭∈𝒮→ a_t∈𝒜 is a policy that maps states to actions for ℳ_t, which is learned from both ℐ_t and ℐ_s. The policy π_t is estimated using a DNN.


In multi-source[In a single-source scenario with
|𝐌_s| = 1,  tabula rasa learning occurs without any external transfer when ℐ_s=∅.] RSU deployments, according to Definition <ref>, the aim is to enhance the learning of misbehavior detection at the target RSU by leveraging knowledge acquired from a set of source RSUs. 

Assume 

m source RSUs with MDPs 𝐌_s = {(𝒮_i,𝒜_i,𝒫_i,ℛ_i,γ_i)}^m_i=1, and a detection policy π_s_i for each source RSU i. 
Then, the target RSU with MDP ℳ_t = (𝒮_𝐭,𝒜_t,𝒫_𝐭,ℛ_t,γ_t) 


aims to enhance the learning of π^*_t by leveraging 
ℐ_s∼𝐌_s. The external ℐ_s 
represents transferred knowledge on misbehavior detection from source RSUs. 









 §.§ Knowledge Transfer


Previous works (e.g., <cit.>) on the integration of TL with collaborative DRL have demonstrated effectiveness in scenarios with some degree of similarity in agents' experiences. 


Based on the definition of transferred knowledge, TL techniques in DRL can be realized as policy[In policy transfer, a set of policies {π_s_i}^m_i=1 from source MDPs are transferred to the target MDP, and then the target policy π_t is learned by utilizing knowledge from them.], representation[In representation transfer, the algorithm learns a feature representation of the task, such as a value function V^π(𝐬) or the Q^π(𝐬,a) function, and the knowledge learned can be either directly used in the target or indirectly using a task-invariant feature space.], and instance transfer <cit.>.

We hereby employ an instance TL technique to transfer misbehavior detection knowledge from a set of source RSUs to 
a target RSU.  



The rationale lies in transferring those


source samples that can enhance the detection performance of the target RSU.


Our method


selects only related (i.e., good/useful) samples collected from the environments of source RSUs as agents' experiences. The selection of such samples prevents negative knowledge transfer that may originate from adversarial source RSUs. In collaborative DRL-MDS,

positive transfer occurs when the transferred knowledge from source RSUs improves 


the target RSU's performance. 
In negative transfer, the target's performance degrades compared to tabula rasa learning. 





To realize collaborative DRL-MDS, our proposed methodology employs TL with a new selective knowledge transfer scheme to prevent negative transfers from untrusted source RSUs, as in Fig. <ref>. Under positive transfer, the target RSU is expected to achieve higher cumulative return as well as reach the asymptotic performance earlier than in tabula rasa learning. Hence, measuring the degree of similarity in misbehavior detection performed at the source and target RSUs

becomes crucial to avoid negative transfer overhead in TL. 




  §.§.§ Trust Evaluation
 In Fig. <ref>, we provide a visualization workflow for trustworthy source RSUs' selection for transfer. 

We first establish an inter-agent semantic similarity metric between the source and target to ensure a positive transfer. 
A source RSU s_i with policy π_s_i is considered to have

high semantic relatedness with the target RSU t, if π_s_i can generate the maximum cumulative return from the target RSU's environment in a limited number of training episodes. 

Inspired by works in <cit.>, the semantic relatedness 
between the source and target RSUs is defined 
as the gain of cumulative return for 
a policy π_s_i under the target reward function, 



    G^π_s_i = ∑^N_e_j=1 R_t_j(π_s_i),   ∀ i ∈ [1,m],

where N_e denotes the number of training episodes and R_t(.) denotes the target reward function. 

By utilizing semantic relatedness, the target RSU can effectively select a subset of source RSUs that are trustworthy to contribute towards enhanced misbehavior detection. 
A higher cumulative return in N_e training episodes ascertains a positive transfer of detection knowledge and implies that a source RSU is more reliable for collaboration.

The procedure of selecting trustworthy source RSUs for knowledge transfer is summarized in Algorithm <ref>. Without loss of generality, we normalize cumulative return G^π_s_i in a way that the scaled cumulative return G^π_scaled, s_i resides in the range of [0,1].









In Algorithm <ref>, cumulative return is computed using (<ref>) for each source RSU s_i with a policy π_s_i and Min-Max normalization is applied to scale cumulative returns {G^π_s_i}^m_i=1 into the range of [0,1]. Min-Max normalization based on cumulative return values is defined as 




    G^π_scaled, s_i = G^π_s_i - G^π_min/G^π_max - G^π_min,


where G^π_max and G^π_min denote max{G^π_s_i}^m_i=1 and min{G^π_s_i}^m_i=1, respectively.

As such, the scaled cumulative return G^π_scaled, s_i of a source RSU s_i can be associated with its trust value T_tr,s_i, which is used by the target RSU to decide whether the source 
is trustworthy to collaborate with.

We introduce a source ranking strategy 𝒻_ranking in Algorithm <ref> (line 7), 

used by the target RSU to sort source RSUs based on their trust values 
and select a subset lying above a specified threshold value T_th∈ [0,1].

The selection of a threshold to ascertain the trustworthiness of source RSUs can be either 
tolerant with a lower T_th value or more stringent with a higher T_th value. 

Reputation or trust-based methods in related literature typically use a [0,1] scale to represent trust/reputation values and consider 0.5 as a neutral value or a threshold <cit.>.



  §.§.§ Selective Knowledge Transfer
 
Upon the completion of

source RSUs' selection, the target RSU applies instance TL by collecting samples from k trustworthy source RSUs following their policies {π_s_i}^k_i=1 (i.e., the output of Algorithm <ref>). In target RSU training, we employ a selective knowledge transfer scheme, called experience selection, that selects source samples with high semantic relatedness. Motivated by
 <cit.>, 

experience selection 
is defined as

    Q^*(𝐬,a) ≥ y > Q_θ(𝐬,a),


where Q^*(.) represents an optimal Q-function and y = R_t(𝐬,a) + γmax_a_t+1Q_θ(𝐬_𝐭+1,a_t+1).

The aim of the target RSU is to learn the optimal misbehavior detection policy π^*_t by obtaining an optimal Q-function Q^*(𝐬,a) for a given state-action pair. 



The expected return achieved by following an optimal policy is always greater or equal to that of any arbitrary behavior policy μ; hence, the expected return Q_μ(𝐬,a) of any arbitrary behavior policy μ can serve as a lower bound of the optimal Q-value Q^*(𝐬,a). Lower bound Q-learning <cit.> can be expressed as 




    Q^*(𝐬,a)  ≥ Q_μ(𝐬,a) = _μ[r_t + ∑_k=t+1^Tγ^k-tr_k],

with convergence guarantees of Q-values <cit.>.

The lower bound Q-value in (<ref>) implies that the estimated Q_θ(𝐬,a) value in (<ref>) is lower than the optimal Q^*(𝐬,a) value. Following lower bound Q-learning, experience selection aims to update Q_θ(𝐬,a) towards the lower bound of Q^*(𝐬,a) with source samples of high semantic relatedness, i.e., Q_θ(𝐬,a) ≥ y.



 









The target RSU training with selective knowledge transfer is summarized in Algorithm <ref>. The algorithm proceeds as follows to improve the learning performance of the target RSU with the aid of experience selection. 

The training of the target RSU is divided into episodes. In each time step, the target RSU selects an action a_t∈𝒜 either randomly with probability ϵ or according to the best Q-value given by (<ref>) (lines 4-9). Subsequently, the chosen action a_t is executed in the environment, and

the reward r_t and the next state 𝐬_𝐭+1 are observed (line 10).  

Assume k trustworthy source RSUs with policies {π_s_i}^k_i=1. 
The target RSU can form an experience samples buffer 𝒮̃ by collecting samples following k source policies (line 12) and computes rewards using its current reward function.


The amount of samples collected into 𝒮̃ from each
trustworthy source RSU s_i is equivalent to η_s_i·|𝒮̃|, with



    η_s_i =  T_tr,s_i/∑^k_i=1 T_tr,s_i,   ∀ i ∈ [1,k],


where T_tr,s_i = G^π_scaled, s_i in (<ref>).

Consequently, source RSUs with higher trust values will transfer more samples, which results in learning more from a set of highly trusted RSUs. 

To trade off exploitation with exploration, target samples are also 

added to 𝒮̃

following the online Q-network with ϵ-greedy policy (line 13).


In the target's model update, parameterized by θ,
training samples are drawn from 𝒮̃ (line 14), 
and experience selection is applied (lines 16-17) to filter out relevant samples.

Conversely, samples that are not semantically important will be removed from 𝒮̃ (lines 18-19). As such, 𝒮̃ is gradually updated while equally prioritizing the remaining training samples, which results in improved sample efficiency. The target RSU can thus leverage selective knowledge transfer and train its misbehavior detection model by minimizing the loss function,




    L(θ) = 1/2∑_tQ_θ(𝐬_𝐭,a_t) - y_t^2,


where target Q-values are given by y_t = R(𝐬_𝐭,a_t) + γmax_a_t+1Q_θ(𝐬_𝐭+1,a_t+1). 

Moreover, the selection of samples with Q_θ(𝐬_𝐭,a_t) ≥ y_t implies that learning updates in (<ref>) are encouraged towards

the lower bound of the optimal Q^*(𝐬,a) value. Consequently, experience selection updates the misbehavior detection policy π_t towards the optimal policy π^*_t.  



Although experience selection improves sample efficiency, transferred knowledge from source instances may introduce bias owing to differences in data distribution or variability in misbehavior attack patterns.

Therefore, to circumvent such bias, experience selection is utilized only in early training stages and, subsequently, target RSU training switches to traditional DQN learning.
Algorithm <ref> for traditional DQN operates similarly but without sampling from 𝒮̃ (lines 13-14) and with no experience selection (lines 16-20). 

In this case, the target RSU samples a minibatch of transitions from replay memory buffer 𝒟 with training samples from the target environment, and those samples are directly used to train the misbehavior detection model by minimizing  (<ref>). 










§ EXPERIMENTS





This section describes the experimental scenarios and the setup used to verify the validity and effectiveness of the proposed knowledge transfer approach for collaborative misbehavior detection. We leverage the open-source VeReMi dataset for experimentation <cit.>.




 §.§ Dataset Description and Pre-processing


The VeReMi dataset has been generated considering
19 different misbehavior attack types
with
each attack type 
having a separate dataset. An attack dataset is made of a collection of log files, each one 

generated containing BSM data 
exchanged with
neighboring vehicles over 
their
entire trajectory. Each attack dataset contains a ground truth file that has recorded the observed behavior of all participating vehicles. BSMs constitute a three-dimensional vector for the position, speed, acceleration, and heading angle parameters. The ratio between misbehaving and genuine vehicles 
in VeReMi
is 30%:70% for all the attack simulations. 

In Appendix <ref>, we briefly describe selected misbehavior attack types relevant to our study.

Based on feature analysis and data pre-processing, we selected six fields, i.e., timestamp, pseudo-identity, position, speed, acceleration, and heading angle, from a BSM payload as the relevant features for misbehavior detection. To reduce dimensionality, a scalar representation of the position, speed, acceleration, and heading parameters is computed using the Euclidean norm. In VeReMi, the ground truth data do not contain labels; hence, a label for each data point was generated by comparing the ground truth against the actual transmitted value recorded in the log file of each vehicle. An attack label 1 was added if the transmitted data diverged from the ground truth; otherwise, label 0 was added for the genuine data.






 §.§ Examined Scenarios
 


We conduct experiments on a set of simulation

scenarios, listed in Table <ref>, to assess the proposed knowledge transfer approach. We provide empirical evidence of the benefits of TL through simulated scenarios for collaborative misbehavior detection in unpredictable and untrusted vehicular environments.  

The considered scenarios aim to cover diversified aspects of collaborative misbehavior detection by means of knowledge transfer between RSUs. By leveraging knowledge from source RSUs with high semantic relatedness, the target RSU can rapidly adapt to new misbehavior patterns, even in dynamic and volatile conditions.


Vehicular mobility

renders indispensable the study of 
such scenarios that may arise in a collaborative misbehavior detection setup.
With that aim, we exploit the attack variants present in VeReMi. For each specific scenario, a different set of misbehavior attacks is selected, aiming to provide sufficient coverage of the available attacks in VeReMi. 



Next, we elaborate on the considered scenarios.







  §.§.§ SC1
 The first
scenario captures
the requirement of detecting future misbehaviors of the same type.

In SC1, a misbehaving vehicle can permeate an attack across multiple geographic locations 
across its trajectory, 

resulting in some RSUs experiencing the attack earlier
in time (sources) compared to others (targets). SC1 arises in situations where RSU deployments are sparsely distributed in vehicular setups.

In this case, the target RSU 
relies on collaborative misbehavior detection to proactively detect future attacks of the same type.










By exploiting the evolving attack patterns
, we split an attack dataset based on the timestamp field (ascending order) and assign a similar amount of training samples to each source RSU. To motivate knowledge transfer, the number of training samples allocated to a source RSU is proportionally 
higher than that of the target RSU. 
It is noted that the allocated samples 
per RSU
preserve the ratio between misbehaving and genuine classes as set in the entire dataset.   








  §.§.§ SC2
 In this scenario, the target RSU aims to acquire knowledge from source RSUs to detect an unseen/unknown attack. Such situations may encounter in practice due to blind spots and occlusions caused by vehicular infrastructure or moving vehicles. This inevitably results in limited situational awareness in some RSUs (targets) which may not experience certain attack types. In this case, the target RSU seeks relevant knowledge from source RSUs with expertise in detecting attacks of similar type.

We assume that source RSUs are capable of detecting 

a wider range of misbehavior attack types. 
On the contrary, the target RSU is trained to identify a narrower set 
of attacks 
compared to
source RSUs. 

In such cases, the target RSU aims to leverage knowledge transfer from source RSUs to identify similar new misbehavior attacks.



The VeReMi dataset includes a set of misbehavior attack types that have been generated by executing a combination of multiple attacks at once (as detailed in Appendix <ref>). We hereby evaluate two cases within this scenario, each involving different denial-of-service (DoS) variants:

    
  * Each source RSU is trained on a mix of DoS, DoS Random and DoS Random Sybil attacks. The target RSU is trained on a mix of DoS and DoS Random attacks, and leverages source knowledge to detect the unseen DoS Random Sybil attack.
    
  * Each source RSU is trained on a mix of DoS, DoS Disruptive and DoS Disruptive Sybil attacks. The target RSU is trained on a mix of DoS and DoS Disruptive attacks, and leverages source knowledge to detect the unseen DoS Disruptive Sybil attack.














  §.§.§ SC3
 In this scenario, 
source RSUs are trained with different feature vectors resulting in partial observability of the attack space. Such cases arise in practice when
RSUs have diverse computational capabilities, including different processing and buffer sizes, 
limiting their training only on specific attack types.

This heterogeneity may hinder an RSU’s ability to process multiple/high-dimensional feature vectors effectively.
Thus, the target RSU needs to select source RSUs based on relevant feature vectors to detect an array of misbehaviors and enhance the effectiveness of collaborative detection.


We assume that each source RSU s_i is trained with feature vector ϕ_𝐢 on a targeted misbehavior attack type. 

Similarly, the target RSU is trained on a specific misbehavior attack type with a feature vector ϕ_𝐭 with dimensionality ρ_ϕ_𝐭<ρ_ϕ_𝐢, ∀ i. Then, the target RSU can leverage source knowledge by combining feature vectors, ⋃ϕ_𝐢,
to detect 

a wider range of misbehaviors.















We consider position- and speed-related attack variants in VeReMi, distinguishable by various feature values embedded in BSMs. Similarly to SC2, we evaluate two cases:

    
  * Each source RSU is individually trained on
    Constant Position Offset, Random Position, and Random Position Offset attacks. The target RSU is trained on Constant Position attack, and leverages source knowledge to detect all position-related attack variants.
    
  * Each source RSU is individually trained on
    Constant Speed Offset, Random Speed, and Random Speed Offset attacks. The target RSU is trained on Constant Speed attack, and leverages source knowledge to detect all speed-related attack variants.













 §.§ Experimental Setup
 

The experiments were performed on a 

server equipped with Intel(R) Xeon(R) Gold 6230 CPUs @ 2.10GHz with 192 GB RAM and four NVIDIA GeForce RTX 2080 Ti GPUs. The components of the DRL-MDS (Fig. <ref>) are implemented in Python using Tensorflow-GPU 

as the backend. 


Considering the amount of data samples available per misbehavior type in VeReMi, we resort to a number of four RSUs in all scenarios, with 

three source RSUs and a single target RSU.

Source RSUs comprise two genuine RSUs and an adversary-influenced (i.e., malicious) RSU. The malicious RSU is trained using a single type of adversarial 
attack at a time, either label flipping or policy induction attack.

In this way, the level of impact on knowledge transfer from each adversarial attack type can be assessed separately. The learning rate α and discount factor γ in DRL-MDS are set to 0.001 and 0.995, respectively. 
Both 𝒮̃ and 𝒟 sizes at the target RSU are set to 50000. 



In the label-flipping attack, the parameter ζ is set to 0.05, which is equivalent to 
randomly selecting 5% of misbehaving vehicles from a source RSU and flip the labels of the corresponding BSMs to create a malicious RSU. In the policy induction attack, the adversarial policy π_adv is obtained based on an inverted reward function r' compared to (<ref>).
The r' generates exact opposite reward values for the malicious RSU to maximize FNs and FPs. To obtain π_adv, an adversarial DQN model is trained with r' on a small portion of training samples extracted from a victim source RSU's environment.

With π_adv, the policy induction attack (as in Definition <ref>) is launched by generating perturbed samples during the training of the victim source RSU to create a malicious RSU. 





§ PERFORMANCE EVALUATION



This section presents the learning performance of DRL-MDS agents
for the scenarios described above using VeReMi. Next, the resulting detection performance achieved via knowledge transfer is assessed for different misbehavior attack types.
































 §.§ Learning Performance





To provide a comprehensive assessment, we evaluate the learning performance of misbehavior detection agents in i) the source RSUs with trained policies and ii) the target RSU with and without knowledge transfer. 




  §.§.§ Source RSUs
 
To assess the contribution of source RSUs towards knowledge transfer, we measure the average reward gains per RSU during the interactions with their own environments. 

In each scenario, a source RSU is set to run for 500 training episodes, 

and the learning performance is measured in terms of the cumulative reward values obtained following the reward function in (<ref>). 





















Fig. <ref> illustrates the learning performance of source RSUs for two position-related misbehavior types considered for SC1. As can be visually comprehended, genuine source RSUs accumulate significantly higher average rewards compared to malicious source RSUs, which are detrimentally influenced by label-flipping and policy induction attacks. 
Furthermore, the negative impact on the victim source RSU from the policy induction attack 
is greater than that of the label-flipping attack. 

This can be attributed to

the effectiveness of policy induction attack which injects crafted inputs at each training step 

towards the adversary's goal. This, in turn,
results in misclassifying malicious samples as genuine ones with increased false alarms.  

Similar behavior can be observed in Fig. <ref> for DoS-related misbehavior variants considered in SC2. In this case, each source RSU is trained on a combination of misbehavior attack types, providing the capability to detect additional misbehavior attack types compared to SC1. In Fig. <ref>, the learning performance of source RSUs in SC3 for both position- and speed-related misbehavior variants is shown. Similarly to SC1 and SC2, genuine sources perform significantly better than malicious ones, while
the pernicious impact of policy induction attack compared to label-flipping becomes apparent.













  §.§.§ Target RSU
 

As shown in Algorithm <ref>, 

 
based on the available source policies, the target RSU selects a subset of source RSUs using T_th to realize knowledge transfer. As opposed to conventional predetermined thresholds (Sec. <ref>), the proposed trust evaluation supports dynamic thresholding to handle 
unpredictable and untrusted vehicular environments,

such as the erratic behavior of RSUs due to adversarial attacks. 


For instance, several previous works use 0.5 as the neutral threshold value. 
In our case, we select 0.5 and 0.8 for T_th to assess the impact of different threshold values on knowledge transfer without loss of generality.  

Figs. <ref>, <ref>, and <ref> depict the learning performance of the target RSU in SC1, SC2, and SC3, respectively. 

We report average rewards for knowledge transfers involving different sources by calculating the average across three runs. The case of no transfer between source and target RSUs (i.e., tabula rasa learning) is also evaluated as a baseline. In addition, the black dashed lines in Figs. <ref>–<ref> represent the best return reward of the baseline scheme with tabula rasa learning. These lines demonstrate the training time reduction achieved under each transfer at the target RSU to reach a certain performance level. Moreover, Table <ref> presents specific information regarding the reduction in training time achieved through knowledge transfer, detailing the required number of training episodes and the corresponding wall-clock time.


Fig. <ref> illustrates average rewards accumulated by the target RSU for a specific misbehavior with knowledge transferred from the source RSUs (shown in Fig. <ref>). It can be observed that our selective knowledge transfer approach significantly enhances the learning performance compared to tabula rasa.

Specifically, 

the target RSU achieves a higher cumulative return and reaches asymptotic performance earlier than in the baseline. 


In addition, Fig. <ref> 
reveals the impact of different threshold values on learning performance.

When a tolerant T_th = 0.5 value is set, both genuine and malicious (label-flipping and policy induction) source RSUs feature in the transfer. With a stringent T_th = 0.8, collaboration stems only from 

genuine source RSUs. 
Interestingly, even though malicious source RSUs are involved with T_th = 0.5, 

the sample contribution in (<ref>) and the experience selection in (<ref>) ascertain positive transfer, resulting in similar performance as in T_th = 0.8.






























Fig. <ref> demonstrates 

the learning performance of the target RSU


with
unseen misbehavior attack variants. 

For the two cases in SC2, the target RSU acquires expert knowledge from sources to detect unseen DoS Random Sybil and DoS Disruptive Sybil misbehaviors.

In contrast, the baseline with tabula rasa 

does not possess prior knowledge of those two unseen variants. 

The learning curves indicate a superior performance of the target RSU compared
to the baseline.

This can be attributed to the fact that collaborative knowledge transfer from source RSUs (Fig. <ref>) imparts the necessary expertise to detect unseen misbehaviors.

Similar to SC1 (Fig. <ref>), our approach exhibits consistent learning performance irrespective of the 
T_th value. 
Note that DoS misbehaviors in VeReMi are high-volume attacks, resulting in the frequent learning of attack patterns by both source RSUs and the target RSU. Thus, 

the target RSU accumulates a relatively higher total average reward for a similar number of training episodes compared to Fig. <ref>.


Fig. <ref> displays the learning performance of the target RSU with partial observability of the attack space.


For the two cases in SC3, the target RSU assimilates knowledge from source RSUs (Fig. <ref>) to expand the detection horizon to a wider range of misbehaviors. As such, a source RSU learns a specific misbehavior effectively using high-dimensional feature vectors, and then the acquired knowledge is distilled to the target RSU via selective knowledge transfer. 

In particular, each source RSU is trained with a high-dimensional feature vector comprising position, speed, acceleration, and heading angle parameters. The target RSU is trained with the position feature for the Constant Position attack and with the speed feature for the Constant Speed attack.

As shown in Fig. <ref>, the target RSU in the baseline scheme achieves notably lower asymptotic performance compared to the target RSU with knowledge transfer. This difference can be attributed to the increased false alarms resulting from the partial observability of the attack space when using low-dimensional feature vectors. To circumvent this, knowledge distillation from sources with multiple/high-dimensional feature vectors can be applied to enhance

the effectiveness of collaborative misbehavior detection.
Note that our approach exhibits 
comparable learning performance for both T_th values, similar to SC1 (Fig. <ref>) and SC2 (Fig. <ref>).





















 §.§ Detection Performance


Detection performance is assessed in all scenarios in terms of Accuracy (A), Precision (P), Recall (R), and F-score (F), 



    A    = TP+TN/TP+TN+FP+FN,
        
    P    = TP/TP+FP,
        
    R    = TP/TP+FN,
       
    F    = 2RP/R + P,

respectively, by considering both genuine and misbehavior classes for each misbehavior type in VeReMi.
The accuracy in (<ref>) indicates the ratio of all correct predictions to the total number of considered input samples. The higher precision values in (<ref>) indicate low FP rates, whereas higher recall values in (<ref>) indicate low FN rates. The F-score in (<ref>) provides a harmonic mean between precision and recall, which is used when FPs and FNs are vital. Therefore, a higher F-score implies better performance in our examined scenarios.   

Table <ref> presents the performance results obtained from our comprehensive analysis of collaborative misbehavior detection with the selective knowledge transfer approach. Results show that misbehavior detection using knowledge transfer yields high effectiveness with a very high F-score of 0.98 in SC1 under both T_th values of 0.5 and 0.8, while correctly identifying misbehaviors with low rates of FPs and FNs. It should be highlighted that the baseline scheme with tabula rasa also achieves a significantly high F-score of 0.91. This is due to the fact that tabula rasa learning obtains sufficient knowledge during training to detect future misbehaviors of the same type. Although the results are reported only for two misbehavior types in SC1, similar performance levels were observed for other misbehavior types in VeReMi.

Numerical results 

for SC2 demonstrate
effective identification of unseen attacks with knowledge transfer, achieving an F-score of 0.71 under both T_th values. Additionally, recall values approaching 1.0 further elucidate

that such non-anticipated

attacks can be successfully detected with a very low rate of FNs. As shown in Table <ref>, the baseline scheme is ineffective in detecting unseen attacks and achieves very low F-scores of 0.49 and 0.48 in contrast to knowledge transfers when encountering DoS Random Sybil and DoS Disruptive Sybil misbehaviors, respectively. This validates that the target RSU enhances its situational awareness and detects non-anticipated misbehaviors 

by acquiring 

knowledge from source RSUs. 



Detection performance in Table <ref> reveals 
that target learning with knowledge transfer yields significantly superior F-scores compared to the baseline scheme in SC3. Under both T_th values, the F-scores of 0.88 and slightly over 0.90 for position- and speed-related misbehaviors, respectively, demonstrate high effectiveness in detecting a partially observable attack space. Conversely, the baseline scheme 
becomes highly ineffective, as shown by the low F-score of 0.37 with high number of FPs and FNs for both position- and speed-related misbehaviors. It is worth noting 
that misbehavior variants generated by adding/subtracting an offset (i.e., Constant/Random Position Offset and Constant/Random Speed Offset) are more challenging to detect as compared to others, such as Constant Position and Constant Speed. Thus, the transfer of relevant knowledge to the target RSU becomes imperative to effectively identify such partially observable attack spaces. 

Overall, across all three scenarios, collaborative misbehavior detection with knowledge transfer significantly outperforms 

tabula rasa learning, as summarized 
in Fig. <ref>, demonstrating its high effectiveness. Specifically, our approach enhances robustness and generalizability by effectively detecting previously unseen and partially observable misbehavior attacks.  










 §.§ Discussion




In summary, we develop a selective knowledge transfer approach that selects knowledge learned at source RSUs with high semantic relatedness to the target RSU, enabling effective and efficient learning about various misbehaviors in a collaborative fashion. We introduce a relevant theoretical framework for the proposed selective knowledge transfer in DRL-based collaborative misbehavior detection in untrusted vehicular environments. In addition, empirical evidence demonstrating the necessity of transfer learning is provided across three examined scenarios discussed in Table <ref> through (i) asymptotic improvement in the learning performance of the target RSU (Figs. <ref>–<ref>); (ii) improved misbehavior detection performance at the target RSU, as shown in Table <ref>; and (iii) faster learning speed of the target RSU, as shown in Table <ref>.


More specifically, we empirically demonstrate that the selective knowledge transfer approach, coupled with trust evaluation, significantly reduces the training time for collaborative misbehavior detection at the target RSU and produces key outcomes: i) higher performance levels in SC1 and SC3 for the detection of future attacks of the same type and only partially observable misbehaviors, respectively, and ii) a certain performance level in SC2 by effectively detecting unseen attacks. 


Our experiments further demonstrate that the proposed knowledge transfer method 
is robust across various threshold values (T_th∈ [0,1]) when selecting 
source RSUs. In particular, Figs. <ref>–
<ref> consistently demonstrate similar learning and detection performances, irrespective of specific T_th values used. Nonetheless, it is worth mentioning that appropriate threshold selection depends on the severity of the adversarial attack model and the specific scenario being examined. 

The 
generalizability problem is an essential challenge in DL- and traditional ML-based misbehavior detection methods proposed in related work (Sec. <ref>). Conventional approaches often struggle with detecting unseen or only partially observable attack spaces. In contrast, our collaborative misbehavior detection approach leverages transfer learning to improve generalizability by effectively handling both unseen and partially observable attacks.
A concern unavoidably arises with the increased computational overhead in the form of processing and buffer sizes at the target RSU when resorting to more tolerant T_th values. Lower threshold values result in the inclusion of more selected sources, which may encompass less trustworthy ones. Consequently, this poisons the experience samples buffer 𝒮̃ with samples from sources with lower trust values, incurring processing overhead on the target RSU. 

However, samples from less trustworthy sources will not eventually influence the target's learning process due to the experience selection.




§ CONCLUSIONS



The current research focus in AI/ML-based collaborative misbehavior detection lies in designing methods that can effectively identify various misbehavior attacks. Such approaches, albeit resonating high detection accuracy, assume the presence of an honest majority, with inherent model resilience and without explicit defense against 

adversarial attacks. 

While the majority of solutions primarily concentrate on centralized setups, they often suffer from drawbacks related to latency, computational cost, scalability, robustness, and generalizability, due to the heterogeneity and distributed nature of vehicular environments.  

To address these research challenges, we proposed a DRL-based scheme 
that utilizes transfer learning for distributed collaborative misbehavior detection among RSUs. Our approach enables selective knowledge transfer from reliable source RSUs in unpredictable and untrusted vehicular environments by leveraging semantic relatedness with the target RSU.

We empirically showcase that selective knowledge transfer, coupled with trust evaluation, is sample-efficient and effective in detecting previously unseen and partially observable misbehaviors with high detection performance. 

Hence, significant steps towards generalizability are attained.


In future research, we intend to delve into adversarial training techniques for DRL, such as introducing perturbations to the observation space, to counter gradient-based adversarial attacks within target RSUs. Moreover, we are interested in evaluating the scalability and performance of our approach when compared to emerging blockchain-based trust management techniques in vehicular networks.   






[Misbehavior Attack Types]

Position falsification: A vehicle transmits falsified position coordinates, leading to four different 

variants: i) constant position, the attacker transmits fixed position coordinates; ii) constant position offset, the attacker transmits the real position coordinates with a fixed offset; iii) random position, the attacker transmits newly generated random position coordinates; and iv) random position offset, the attacker transmits the real position coordinates with a random offset.



Speed falsification: A vehicle transmits falsified speed values in its BSM, following a similar approach as in position falsification attack. This results in constant speed, constant speed offset, random speed and random speed offset 

variants.



Disruptive: This is similar to a data replay attack, where a vehicle re-transmits previously sent messages by other vehicles. BSMs are selected at random and flood the network with stale data to disrupt genuine information from being propagated. This attack may also be carried out in DoS and Sybil modes. 



Denial-of-service (DoS): A misbehaving vehicle transmits BSMs at a very higher frequency than the acceptable limit set by the standard. This results in high volume of data transmission causing extensive periods of network congestion and unavailability of critical services to legitimate vehicles. 



DoS Random: In this attack, the attacker sets all BSM fields to random values and performs a typical DoS attack. 



DoS Disruptive: An attacker may re-transmit previously sent BSMs by other legitimate vehicles. BSMs are selected at random and flood the network with stale data with the intention of disrupting genuine information from being propagated. The attacker increases BSM transmission rate to realize this attack.



DoS Random Sybil: The attacker changes pseudonym identities on every 

BSM while performing the DoS random attack. 



DoS Disruptive Sybil: The attacker constantly changes pseudonyms on every re-transmission of BSMs while concealing its real identity.











 









IEEEtran









§ REFERENCES SECTION

You can use a bibliography generated by BibTeX as a .bbl file.
 BibTeX documentation can be easily obtained at:
 http://mirror.ctan.org/biblio/bibtex/contrib/doc/
 The IEEEtran BibTeX style support page is:
 http://www.michaelshell.org/tex/ieeetran/bibtex/
 
 




§ SIMPLE REFERENCES

You can manually copy in the resultant .bbl file and set second argument of \begin to the number of references
 (used to reserve space for the reference number labels box).

1
IEEEtran

ref1
Mathematics Into Type. American Mathematical Society. [Online]. Available: https://www.ams.org/arc/styleguide/mit-2.pdf

ref2
T. W. Chaundy, P. R. Barrett and C. Batey, The Printing of Mathematics. London, U.K., Oxford Univ. Press, 1954.

ref3
F. Mittelbach and M. Goossens, The Companion, 2nd ed. Boston, MA, USA: Pearson, 2004.

ref4
G. Grätzer, More Math Into LaTeX, New York, NY, USA: Springer, 2007.

ref5M. Letourneau and J. W. Sharp, AMS-StyleGuide-online.pdf, American Mathematical Society, Providence, RI, USA, [Online]. Available: http://www.ams.org/arc/styleguide/index.html

ref6
H. Sira-Ramirez, “On the sliding mode control of nonlinear systems,” Syst. Control Lett., vol. 19, pp. 303–312, 1992.

ref7
A. Levant, “Exact differentiation of signals with unbounded higher derivatives,”  in Proc. 45th IEEE Conf. Decis.
Control, San Diego, CA, USA, 2006, pp. 5585–5590. DOI: 10.1109/CDC.2006.377165.

ref8
M. Fliess, C. Join, and H. Sira-Ramirez, “Non-linear estimation is easy,” Int. J. Model., Ident. Control, vol. 4, no. 1, pp. 12–27, 2008.

ref9
R. Ortega, A. Astolfi, G. Bastin, and H. Rodriguez, “Stabilization of food-chain systems using a port-controlled Hamiltonian description,” in Proc. Amer. Control Conf., Chicago, IL, USA,
2000, pp. 2245–2249.








§ BIOGRAPHY SECTION

If you have an EPS/PDF photo (graphicx package needed), extra braces are
 needed around the contents of the optional argument to biography to prevent
 the LaTeX parser from getting confused when it sees the complicated
 \includegraphics command within an optional argument. (You can create
 your own custom macro containing the \includegraphics command to make things
 simpler here.)
 



If you include a photo:

[
    < g r a p h i c s >
]Michael Shell
Use \begin{IEEEbiography} and then for the 1st argument use \includegraphics to declare and link the author photo.
Use the author name as the 3rd argument followed by the biography text.





If you will not include a photo:

John Doe
Use \begin{IEEEbiographynophoto} and the author name as the argument followed by the biography text.










