Paper Title: Diff-Cleanse: Identifying and Mitigating Backdoor Attacks in Diffusion Models.




Knowledge-Guided Prompt Learning for Lifespan Brain MR Image Segmentation
    Lin Teng 1 Zihao Zhao 1 Jiawei Huang 1 Zehong Cao 2 Runqi Meng 1 
Feng Shi 2^() 
Dinggang Shen 1,2,3^()
    August 5, 2024
===========================================================================================================




§ ABSTRACT 

  Diffusion models (DM) represent one of the most advanced generative models today, yet recent studies suggest that DMs are vulnerable to backdoor attacks. Backdoor attacks establish hidden associations between particular input patterns and model behaviors, compromising model integrity by triggering undesirable actions with manipulated input data. This vulnerability poses substantial risks, including reputational damage to model owners and the dissemination of harmful content. To mitigate the threat of backdoor attacks, there have been some investigations on backdoor detection and model repair. However, previous work fails to purify the backdoored DMs created by state-of-the-art attacks, rendering the field much underexplored. To bridge this gap, we introduce Diff-Cleanse, a novel two-stage backdoor defense framework specifically designed for DMs. The first stage employs a innovative trigger inversion technique to detect the backdoor and reconstruct the trigger, and the second stage utilizes a structural pruning method to eliminate the backdoor. We evaluate our framework on hundreds of DMs attacked by 3 existing backdoor attack methods. Extensive experiments demonstrate that Diff-Cleanse achieves nearly 100% detection accuracy and effectively mitigates backdoor impacts, preserving the model's benign performance with minimal compromise. Our code is avaliable at <https://github.com/shymuel/diff-cleanse>.





§ INTRODUCTION




Diffusion models (DM) have emerged as one of the most successful generative models due to their superior performance  <cit.> and are widely applied in image <cit.>, audio <cit.>, video <cit.>, text <cit.>, and text-to-speech <cit.>
generation. Moreover, DMs are increasingly used in safety-critical tasks, such as autonomous driving <cit.> and medical imaging <cit.>.


However, recent studies show that DMs are vulnerable to backdoor attacks <cit.>, where attackers poison a portion of the training dataset, causing the model to execute attacker-specified behaviors when encountering inputs embedded with specific trigger patterns. Given the common practice of using powerful pre-trained models available online or outsourcing the training process to third parties, such attacks could have disastrous consequences <cit.>.
For instance, a company using a backdoored DM for image generation service could unintentionally produce offensive content, damaging the company's reputation and leading to legal repercussions <cit.>. Additionally, DMs are often used to generate synthetic data for training downstream models when data is scarce or privacy is a concern <cit.>. Attackers could implant biases within these DMs, affecting subsequent model behaviors.


Despite these challenges, there is a significant lack of specialized backdoor defense strategies for DMs. Most existing backdoor defense methods are designed for traditional neural network in classification tasks and cannot be directly applied to DM due to different prediction objectives (class probability for classifier, estimation of mean or noise for DMs) and inference processes. Moreover, defense methods for DMs exhibit limitations, either restricted to input-level backdoor detection <cit.> or failing to effectively identify backdoors in our experiments <cit.>.


To address this gap, we study three types of backdoor attacks on DMs in the literature, noting that despite the diversity in semantics and textures of trigger and target patterns, attackers typically seek backdoored DMs with high utility and specifity <cit.>. High utility means that the DM generates high-quality images from clean inputs, while high specifity means that inputs containing trigger patterns force the DM to generate nearly identical target images. To repair the model, a natural approach is to reconstruct an effective trigger and remove neurons activated by it. Our experiments show that pruning a small number of neurons (typically 1% to 2%) is sufficient to remove the backdoor. Based on these insights, we propose a two-stage backdoor defense framework for DMs. First, we introduce a novel trigger inversion method that models trigger reconstruction as an optimization problem, aiming to find a trigger that causes the DM to generate identical images. We devise a metric to measure the effectiveness of the inverted trigger and to aid in backdoor detection. To eliminate the backdoor, we propose a pruning-based method to remove the backdoor-related neurons and design a loss function to improve the fine-tuning effect when using only a small number of clean samples.


The major contributions of this paper are as follows. 






  * We propose Diff-Cleanse, a two-stage backdoor defense framework designed for diffusion models, which can effectively mitigate the backdoor by targeted pruning backdoor-related neurons. To the best of our knowledge, this is the first study to explore the application of pruning methods in the field of DM backdoor defense.



  * We introduce a trigger inversion method based on the pursuit of high specificity in existing backdoor attacks in DMs. As part of this method, we design a metric, trigger effectiveness (TE), to measure the performance of the inverted trigger and facilitate backdoor detection.



  * We devise a simple yet effective pruning-based backdoor removal algorithm, including the design of a novel loss function, to eliminate backdoors.



  * We empirically show that in terms of 3 state-of-the-art backdoor attacks on DMs, Diff-Cleanse achieves close to 100% detection accuracy and reduces backdoor effects to nearly zero while largely maintaining the model's utility.







§ PRELIMINARIES








 §.§ Diffusion models



Utilizing diffusion models (DMs) generally involves three key procedures. (1) Diffusion process. Define a diffusion process that diffuses the data distribution q(x) into a certain distribution τ(x) over T timesteps. (2) Training. Train parameters θ to ensure that the generative process aligns with the reversed diffusion process, i.e., p_θ(x_t-1|x_t)=𝒩(x_t-1;μ_θ(x_t,t),β_θ(x_t,t))=q(x_t-1|x_t). (3) Sampling. Use the trained generative process p_θ(x_t-1|x_t) to sample from T_max down to T_min. 




  
DDPM. DDPM <cit.> is a representative diffusion model. It assumes τ(x)=𝒩(0,𝐈) and defines the Markov diffusion process q(x_t |x_t-1)=N(x_t; √(1-β_t)x_t-1,β_t𝐈), where {β_t}_t=1^T is a predefined variance schedule. Let α_t=1-β_t and α̅_i=∏_t=1^i α_t. Given x_0 ∼ q(x), t ∼Uniform({1, …, T}) and ϵ∼𝒩(0, I), the DM is trained to minimize the loss ϵ-ϵ_θ(√(α̅_t) x_0+..√(1-α̅_t)ϵ, t) ^2, thereby learning to predict the added noise given the x_t at timestep t. During sampling, given x_T ∼𝒩(0, I), DDPM samples from p_θ(x_t-1|x_t)=𝒩(x_t-1 ; μ_θ(x_t), β_θ(x_t)) from t=T to t=0, where μ_θ(x_t)=1/√(α_t)(x_t-1-α_t/√(1-α̅_t)ϵ_θ(x_t, t)) and β_θ(x_t)=(1-α̅_t-1) β_t/1-α̅_t.




  
Advanced samplers. The default samplers for DMs are relatively slow compared to other generative models such as GANs <cit.>, prompting the development of advanced samplers to accelerate the sampling.
Denoising Diffusion Implicit Models (DDIM) <cit.> introduces an abbreviated reverse process, reducing the steps from 1000 to as few as 50, based on a generalized non-Markovian forward chain derived from DDPM. Additionally, some researchers formulate the reversed diffusion process as Ordinary Differential Equations (ODEs) and employ higher-order approximations, enabling DMs to generate high-quality samples in fewer steps <cit.>.







 §.§ Backdoor attacks and defences on diffusion models




  
Backdoor attacks. Injecting backdoors to traditional models (e.g., classifiers) involves poisoning a small portion of the training data by stamping the pre-defined trigger and relabeling them as target classes <cit.>. After training, the model would produce the intended target label in the presence of the trigger. Backdoor attacks on DMs are more complex because the sampling process depends not only on the input but also on the transitional distribution. To the best of our knowledge, there are three existing noise-to-image DM backdoor attacks <cit.>. Their workflow is illustrated in <ref>. 


To inject backdoor into DMs, attackers first need to mathematically define a forward backdoor diffusion process x_0^b →x_T^b, where x_0^b is the target image, x_T^b is the backdoored noise, also know as Trojan noise <cit.>. Without loss of generality, we assume x_T^b=g+γ·x_T^c, where g∼𝒩(𝐫, 0) is the trigger pattern, x_T^c∼𝒩(0,𝐈) is the clean input. Attackers then need to define the corresponding backdoor reverse process and backdoor loss. During training, clean loss and backdoor loss are calculated separately for clean samples and backdoor samples, and the parameter θ learns both the clean transitional distribution q^c(x_t-1^c|x^c_t) and the backdoor transitional distribution q^b(x^b_t-1|x^b_t). During sampling, the backdoored DM will generate clean samples x_0^c∼ q^c(x_0^c) with clean input, and target image x_0^b with backdoored noise. In BadDiff <cit.> and VillanDiff <cit.>, the same sampling process can be used to generate clean samples from clean noise, as well as backdoor targets from backdoored noise. In TrojDiff <cit.>, generating clean samples and backdoor targets requires not only different inputs but also different sampling functions, which weakens the effectiveness of the injected backdoor. BadDiff is only applicable to the ancestral sampler DDPM, while VillanDiff proposes a general framework and can attack advanced samplers and more types of DMs, such as latent diffusion models and score-based diffusion models. Moreover, the target in TrojDiff can be a distribution, such as "horse" or handwritten digits, whereas the target in BadDiff and VillanDiff must be an image.





  
Backdoor defenses. Due to different prediction objectives and inference processes, backdoor defense methods designed for other tasks cannot be directly applied to backdoored DMs <cit.>. Recently, some efforts have been made to mitigate the effects of backdoor in DMs. Elijah <cit.> optimizes a trigger g that linearly decreases during the sampling process, uses the model's response to inputs containing g to detect backdoors and removes the backdoor by aligning the model's predictions for clean and backdoored inputs. UFID <cit.> introduces an input-level backdoor detection framework, which identifies backdoored inputs by analyzing the sensitivity of input-output pairs to input perturbations.






 §.§ Threat model


We adopt a similar setting as in <cit.>, which describes the goals and capacities of two parties, Attacker and Defender. A comprehensive explanation of each party is as follows.




  
Attacker. The attacker's goal is to develop a backdoored DM with two properties <cit.>. (1) High utility. The model has good performance in clean sampling.
(2) High specificity. Activation of the backdoor causes the model to generate the target image. In terms of capacity, we assume a strong attacker who can (1) design the backdoor diffusion process and backdoor sampling procedure, (2) modify all aspects of training, including the training loss, training data and hyperparameters.






  
Defender. The defender's goals are to (1) detect potential backdoors, and (2) remove them while maintaining the models' benign utility. We assume the defender has no prior knowledge of the attack settings but has white-box access to the DM and a limited amount of clean data (typically 10%).




§ METHODOLOGY





In this section, we start by introducing our two-stage backdoor defense framework, as illustrated in <ref> (<ref>). Given a risky DM, we first apply our trigger inversion method to reconstruct a trigger g^s and detect the backdoor (<ref>). If the DM is backdoored, we then execute our structural pruning-based backdoor removal algorithm to eliminate the injected backdoor (<ref>).







 §.§ Overview of our approach






  
Key intuition. We derive the intuition of our technique from the basic properties of backdoored DMs, i.e., clean DMs convert standard Gaussian noise to the dataset distribution through clean sampling, while backdoored DMs can additionally convert the backdoored noise to the target distribution through backdoor sampling. In attacks addressed in this paper, the backdoored noise differs significantly from clean noise, and the target distribution does not overlap with the dataset distribution. 


Intuitively, our approach reconstructs the trigger and use it to facilitate the backdoor detection and removal. Specifically, we hypothesize that the trigger eliciting target images indicates the presence of a backdoor and that neurons activated by the inverted trigger will also be activated by the attacker-specified trigger. We employ pruning-based method to remove the backdoor, which is common in backdoor defenses for discriminative models <cit.>. This leads to following observations. In following sections, we call g-X as trigger-target pair if clean inputs stamped with g cause the DM to generate highly similar outputs X. For simplicity, g and X have the same shape. 







  
Observation 1. Both clean and backdoored DMs contain trigger-target pairs. For the same target image, there are triggers with different levels of effectiveness.


We classify trigger-target pairs into two categories, natural and artificial. Natural triggers and targets emerge during normal training, which are typically images uncomprehensible to humans. In contrast, artificial triggers and targets are injected by backdoor training and contain intelligible semantics. Both clean and backdoored DMs contain natural trigger-target pairs, while backdoored DMs additionally contain artificial trigger-target pairs. The color, shape and texture of the artificial trigger can be arbitrarily configured. Therefore, we focus on the effectiveness of the trigger, i.e., the ability of the trigger to activate the backdoor. <Ref> presents examples of both natural and artificial targets, as well as the outcomes of backdoor sampling using triggers with varying levels of effectiveness.




  
Observation 2. Structural pruning method can completely eliminate the backdoor while removing very few neurons (typically 1% to 2%).


Existing prune-based backdoor defense methods typically focus on deep layers and utilize unstructural pruning <cit.>, which minimally impacts the benign performance of the model. However, we observe that the optimal pruning rate varies significantly across different attack settings when the algorithm only prunes deep layers, making the algorithm sensitive to hyperparameters. Moreover, we find that unstructural pruning usually fails to completely eliminate the backdoor. Backdoors tend to recover after fine-tuning, whereas structural pruning suppresses this phenomenon.


Based on the previous observations, we propose the two-stage backdoor defense framework Diff-Cleanse. In the first stage, we model the trigger inversion as an optimization problem and reconstruct effective triggers. We also use the effectiveness of the inverted triggers to determine the presence of backdoors. If a backdoor is detected, in the second stage, we apply structural pruning to remove backdoor-related neurons and restore the benign performance of the purified DM using our proposed loss function. Due to space limitations, the pseudocode of our algorithms are in <ref>.




 §.§ Trigger inversion (stage 1)





Recall that the backdoor diffusion maps the target image x_0^b to the backdoored noise x_T^b∼𝒩(𝐫, γ^2𝐈). Without loss of generality, we consider the unified form of backdoor diffusion as follows.

    x_t^b=α̂_t x_0^b+ρ̂_t 𝐫+β̂_t ϵ_t,  ϵ_t∼𝒩(0, 𝐈),  t∈[1,…,T]

where {α̂_t}_t=1^T is a decreasing sequence, {ρ̂_t}_t=1^T and {β̂_t}_t=1^T are increasing sequences. 


Since the defender does not know the specific parameters of the attack, let {β_t}_t=1^T represent a monotonically increasing variance sequence from 0 to 1, α_t = 1-β_t, and α̅_t=∏_i=1^t α_i. We derive a special case of the backdoor diffusion, denoted as {x_t^s}_t=1^T, which has the following form.

    x_t^s=√(α̅_t)x_0^s+(1-√(α̅_t)) r^s+√(1-α̅_t)γϵ_t,  ϵ_t∼𝒩(0, 𝐈),  t∈[1,…,T]

Using the fixed increasing variance sequence {β_t}_t=1^T in DDPM <cit.>, we have α̅_T ≈ 0. Hence, x_T^s≈r^s+γϵ_T ∼𝒩(r^s, γ^2 𝐈).  <Ref> provides a detailed derivation of x_t^s. 


Let g^s denotes the inverted trigger, X^s denotes the results of backdoor sampling with g^s. We model trigger inversion as an optimization problem that has two objectives. First, since attackers seek high specificity DMs <cit.>, artificial triggers should effectively activate the backdoor, leading to the generation of identical images, i.e., the targets. To quantify this, we propose the similarity loss ℓ_s as in <ref>, which evaluates the diversity of the generated samples.

    ℓ_s(g^s)=1-∑_i,j𝒜(X^s_i, X^s_j)/|X^s|^2

where (X^s_i, X^s_j) denotes pairs of samples selected from X^s. 𝒜(·) is the cosine similarity function.


The second objective is to find the artificial trigger-target pair rather than natural trigger-target pairs. Empirically, abnormal entropy values indicate samples without comprehensible semantics, and natural targets tend to have either too high (e.g., random noise) or too low entropy (e.g., pure black pictures). Thus, we propose entropy loss ℓ_e as follows.

    ℓ_e(g^s)=∑_i[ max(ℰ(X^s_i)-threshold_max, 0) + max(threshold_min-ℰ(X^s_i), 0) ]

where ℰ(·) is the entropy calculation function and X^s_i is the i-th results of backdoor sampling with g^s. The values of threshold_max and threshold_min are not strict and we use 0.1 and 0.9 in all experiments. Together, we formulate trigger inversion as a multi-objective optimization task by optimizing the weighted sum of the two objectives. The final formulation is as follows.

    min_g^sℓ(g^s)=ℓ_s(g^s)+λℓ_e(g^s)

where λ is the weight of ℓ_e. In experiments, we adjust λ dynamically so that the two loss terms are of similar magnitude. We use <ref> to calculate the derivative of x_0^s with respect to x_T^s, which speeds up the calculation, and use Adam optimizer to solve the above optimization.




  
Backdoor detection. We note that natural targets, sampled with a natural trigger, are visually similar but exhibit slight variations. In contrast, samples generated with an artificial trigger show a high degree of similarity. Therefore, we devise a metric, trigger effectiveness (TE), that measures the similarity between generated samples X to evaluate the effect of g^s and to detect the backdoor.

    TE(X)=|{X_i^s, X_j^s ∈X: 𝒜(X_i, X_j)> threshold }|/|X|

where |·| denotes the cardinality of the set. High TE indicates that g^s effectively activates the artificial backdoor in the model, while low TE indicates that there is no backdoor or the trigger is weak. The full algorithm is summarized in <ref>.




 §.§ Structural pruning-based backdoor removal (stage 2)




Upon detecting a backdoor, we can prevent the jeopardy by discard the model, which would also sacrifice its benign utility. To address this issue, we propose a structural pruning-based approach to eliminate the backdoor while preserving the model's utility. Our method utilize the inverted trigger to prune neurons that are important for backdoor sampling but less important for clean sampling. Following Diff-Pruning <cit.>, we use the taylor expansion of the loss to approximate neuron importance.


For simplicity, we consider the parameter θ of the backdoored DM as a 2-D matrix, where each row vector θ_i=[θ_i 0, θ_i 1, …, θ_i K] contains K scalar parameters and the goal of pruning is to obtain a sparse parameter matrix θ^'. A natural idea is that the loss disruption caused by pruning can indicate the changes in model performance, thus we formulate the objective as follows.



    min_θ^'|ℒ^c(θ^', x^c)-ℒ^c(θ,x^c)|,  max_θ^'|ℒ^b(θ^',x^b))-ℒ^b(θ,x^b))|,   s.t. θ^'_0 ≤ s

where ℒ^c(·) and ℒ^b(·) are loss functions of clean and backdoor training respectively, x^c and x^b denote clean and backdoored inputs respectively. s is the sparsity of θ^' and ·_0 is the L_0 norm, which counts the number of non-zero parameters. Since the defender does not know the attack settings, we calculate ℒ^b(·) using <ref>. Due to the iterative nature of diffusion process, ℒ^c and ℒ^b can be decomposed into sequences {ℒ^c_t}_t=1^T and {ℒ^b_t}_t=1^T.




  
Taylor importance score. Let ℒ_t denotes the loss at timestep t of either clean or backdoor training. Leveraging the Taylor expansion <cit.>, the loss disrupt can be approximated as follows.

    ℒ_t(θ^',x)-ℒ_t(θ,x)     =∇ℒ_t(θ,x)(θ^'-θ)+O(θ^'-θ^2)

Using the Taylor importance criterion, we can obtain the importance of an individual weight θ_i k by setting θ^'_i k=0 in <ref>. For structural pruning, the smallest unit of pruning is θ_i, so we adopt the aggregated Taylor importance <cit.> as the importance score, which is formulated as follows.

    ℐ_t(θ_i, x)=∑_k|ℒ_t(.θ,x|θ_i k=0)-ℒ_t(θ,x)|=∑_k|θ_i k·∇_θ_i kℒ_t(θ, x)|

We further explore the contribution of different loss terms {ℒ_1, …, ℒ_T} during backdoor diffusion. We observe that in backdoor diffusion, large t corresponds to large ℒ_b,t, while in clean diffusion, small t corresponds to large ℒ_c,t. <cit.> empirically demonstrates that small ℒ_t is uninformative. Thus, we introduce a threshold parameter 𝒯 and only use timestep t>𝒯 for gradient accumulation, which yields the finalized importance score.

    ℐ(θ_i, x)=∑_k|θ_i k·∑_{t |  t>𝒯.}∇_θ_i kℒ_t(θ, x)|





  
Implementation of model repair. After obtaining the Taylor importance scores of substructures, we firstly prune the top-p% channels that have the greatest difference in ℐ(θ_i, x^c) and ℐ(θ_i, x^b) to obtain θ^'. Then we fine-tune the pruned model.


Recall that the objective function of clean training is loss_c=ϵ-ϵ_θ(√(α̅_t)x+√(1-α̅_t)ϵ, t)^2. However, this loss alone is inefficient because defenders can only utilize a small number of clean samples. Therefore, we use the output of the unpruned model θ as a reference to align the updated model's distribution with the clean distribution obtained by training through the entire clean dataset. This loss can be expressed as loss_b=ϵ_θ^'(√(α̅_t)x+√(1-α̅_t)ϵ, t)-ϵ_θ(√(α̅_t)x+√(1-α̅_t)ϵ, t)^2, and the complete loss is loss_p=loss_c+loss_b. The full algorithm is summarized in <ref>.




§ EXPERIMENTS AND RESULTS















In this section, we first introduce the experiment setup (<ref>). We then evaluate the effectiveness of our method (<ref>, <ref>) and conduct ablation studies (<ref>). Finally, we discuss a potential countermeasure of the attacker (<ref>). Our main experiments run on a server equipped with Intel Golden 6240 CPUs and NVIDIA V100 GPUs.





 §.§ Experiment setup




  
Datasets, models and attacks. We use two benchmark vision datasets, CIFAR-10 (32× 32) <cit.> and CelebA-HQ (256× 256) <cit.>, since these are the datasets considered in the evaluated backdoor attack methods. Due to the instability of existing attack methods on latent diffusion models and score-based diffusion models, we limit our experiments to DDPM models. We employ both the default sampler and advanced samplers, including DDIM, PNDM, DEIS, DPMO1, DPMO2, DPMO3, DPM++O1, DPM++O2, DPM++O3, UNIPC, and HEUN. Clean and backdoored DMs are either downloaded from Hugging Face or trained by ourselves using official codes. We consider all the existing attacks in the literature, namely, BadDiff <cit.>, TrojDiff <cit.> and VillanDiff <cit.> in the context of unconditional image generation. Please check <ref> for more details.




  
Baselines. As we mentioned earlier, most existing defenses are not applicable for DMs. For trigger inversion, We use the method in Elijah <cit.> as the baseline, which is the only trigger inversion method for DMs in the literature. To the best of our knowledge, there is no other pruning-based model repair methods for DMs. So we propose two effective pruning strategies and evaluate them.




  
Evaluation metrics. Following existing literature <cit.>, we use the following metrics: (1) Detection Accuracy (ACC) evaluates the effectiveness of backdoor detection methods, with higher values indicating better performance. (2) Attack Success Rate (ASR) calculates the proportion of backdoor sampling results whose similarity to the target image exceeds a predefined threshold. (3) Structural Similarity Index Measure (SSIM) accesses the similarity between generated samples and the target image. (4) Fréchet Inception Distance (FID) evaluates the quality of the generated clean samples relative to the training data, with lower values indicating better utility. For trigger inversion methods, we use ACC, ASR and SSIM, where higher values indicate more effective triggers. To evaluate model repair methods, we focus on changes in metrics, i.e, ΔFID, ΔASR and ΔSSIM, where lower values are preferable. A low ΔFID score means better preservation of benign utility, while low ΔASR and ΔSSIM scores indicate more effective backdoor removal.










 §.§ Evaluation of trigger inversion




<ref> compares the performance of triggers inverted by Elijah and our method. Following the original paper, we use clean and backdoored DMs to train a random forest as the backdoor detector of Elijah. Our proposed method utilize TE scores for backdoor detection. Although both methods achieve high ACC scores, our method generates triggers with higher ASR and SSIM, especially for VillanDiff. 


In our experiments, small perturbations to clean inputs are sufficient to cause the models attacked by BadDiff to generate target images, which makes the backdoor easy to detect, indicating a large trigger space. In contrast, backdoors injected by VillanDiff are hard to activate, suggesting narrow trigger spaces. The effectiveness of the triggers inverted by Elijah is relatively low due to the lack of distinction between natural and artificial targets. Our method, which utilizes the properties of artificial targets, is empirically more effective. The small size of patch-based triggers affects the effectiveness of our inverted triggers, but our framework can still repair these DMs as shown in <ref>. 









 §.§ Evaluation of backdoor removal


In order to get better utility, we argue that the pruning algorithm should eliminate the backdoor while removing the minimum number of neurons. In all experiments, Diff-Pruing prunes 13% neurons for models trained on CIFAR-10 and 7% neurons for models trained on CelebA-HQ. Diff-cleanse pruning 1%∼3% neurons of the model until the inverted trigger becomes ineffective.

<ref> compares two pruning-based backdoor removal strategies. Both algorithms completely remove the backdoor across all attack settings, and Diff-Cleanse shows less impact on benign utility (lower ΔFID). In some cases, our approach does not achieve a smaller ΔSSIM because images with no visual resemblance to the target can still exhibit SSIM scores above zero. 








 §.§ Ablation study




  
Effect of loss_p.
<Ref> compares the results of fine-tuning using loss_c and loss_p. The fine-tuning method quickly reduces FID within 5 epochs, and our proposed loss_p always yields lower FID scores.




  
Effect of 𝒯. <Ref> shows the results of Diff-Cleanse with various threshold 𝒯 for gradient accumulation, where Diff-Cleanse's efficacy in removing the backdoor remains largely unaffected. 

Due to space constraints, we provide more results of ablation studies in <ref>



 §.§ Adaptive attack




After realizing the criteria of Diff-Cleanse, the attacker may train more concealed backdoors. One possible approach is to align the model's predictions for clean and backdoored inputs, which could be defended by our framework. The detailed method and experimental results are in <ref>.




§ CONCLUSION


In this paper, we present Diff-Cleanse, a novel two-stage backdoor defense framework. Extensive experiments show that Diff-Cleanse have a detection accuracy close to 100% and can completely remove backdoors without significantly affecting the model's benign performance.










10

an2024elijah
Shengwei An, Sheng-Yen Chou, Kaiyuan Zhang, Qiuling Xu, Guanhong Tao, Guangyu Shen, Siyuan Cheng, Shiqing Ma, Pin-Yu Chen, Tsung-Yi Ho, et al.
Elijah: Eliminating backdoors injected in diffusion models via distribution shift.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10847–10855, 2024.

chen2023trojdiff
Weixin Chen, Dawn Song, and Bo Li.
Trojdiff: Trojan attacks on diffusion models with diverse targets.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4035–4044, 2023.

chen2017targeted
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song.
Targeted backdoor attacks on deep learning systems using data poisoning.
CoRR, abs/1712.05526, 2017.

chou2023backdoor
Sheng-Yen Chou, Pin-Yu Chen, and Tsung-Yi Ho.
How to backdoor diffusion models?
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4015–4024, 2023.

chou2023villandiffusion
Sheng-Yen Chou, Pin-Yu Chen, and Tsung-Yi Ho.
Villandiffusion: A unified backdoor attack framework for diffusion models.
NeurIPS 2023 Workshop on Backdoors in Deep Learning, 2023.

dhariwal2021diffusion
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat gans on image synthesis.
Advances in neural information processing systems, 34:8780–8794, 2021.

fang2024structural
Gongfan Fang, Xinyin Ma, and Xinchao Wang.
Structural pruning for diffusion models.
Advances in neural information processing systems, 36, 2024.

goelimproving
Harsh Goel, Sai Shankar Narasimhan, and Sandeep P Chinchali.
Improving end-to-end autonomous driving with synthetic data from latent diffusion models.
In First Vision and Language for Autonomous Driving and Robotics Workshop.

gu2019badnets
Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg.
Badnets: Evaluating backdooring attacks on deep neural networks.
IEEE Access, 7:47230–47244, 2019.

guan2024ufid
Zihan Guan, Mengxuan Hu, Sheng Li, and Anil Vullikanti.
Ufid: A unified framework for input-level backdoor detection on diffusion models.
arXiv preprint arXiv:2404.01101, 2024.

ho2020denoising
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
Advances in neural information processing systems, 33:6840–6851, 2020.

huang2022fastdiff
Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao.
Fastdiff: A fast conditional diffusion model for high-quality speech synthesis.
IJCAI2022, pages 4132–4138, 2022.

jeong2021diff
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim.
Diff-tts: A denoising diffusion model for text-to-speech.
Proc. Interspeech 2021, pages 3605–3609, 2021.

karras2022elucidating
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.
Elucidating the design space of diffusion-based generative models.
Advances in Neural Information Processing Systems, 35:26565–26577, 2022.

kong2020diffwave
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.
Diffwave: A versatile diffusion model for audio synthesis.
arXiv preprint arXiv:2009.09761, 2020.

krizhevsky2009learning
Alex Krizhevsky, Geoffrey Hinton, et al.
Learning multiple layers of features from tiny images.
2009.

li2022diffusion
Xiang Li, John Thickstun, Ishaan Gulrajani, Percy S Liang, and Tatsunori B Hashimoto.
Diffusion-lm improves controllable text generation.
Advances in Neural Information Processing Systems, 35:4328–4343, 2022.

liu2018fine
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg.
Fine-pruning: Defending against backdooring attacks on deep neural networks.
In International symposium on research in attacks, intrusions, and defenses, pages 273–294. Springer, 2018.

liu2019ppgan
Yi Liu, Jialiang Peng, JQ James, and Yi Wu.
Ppgan: Privacy-preserving generative adversarial network.
In 2019 IEEE 25Th international conference on parallel and distributed systems (ICPADS), pages 985–989. IEEE, 2019.

liu2015deep
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
Deep learning face attributes in the wild.
In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.

lu2022dpm
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.
Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.
Advances in Neural Information Processing Systems, 35:5775–5787, 2022.

lu2022dpmplus
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.
Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.
arXiv preprint arXiv:2211.01095, 2022.

mei2023vidm
Kangfu Mei and Vishal Patel.
Vidm: Video implicit diffusion models.
In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9117–9125, 2023.

moghadam2023morphology
Puria Azadi Moghadam, Sanne Van Dalen, Karina C Martin, Jochen Lennerz, Stephen Yip, Hossein Farahani, and Ali Bashashati.
A morphology focused diffusion probabilistic model for synthesis of histopathology images.
In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2000–2009, 2023.

molchanov2019importance
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz.
Importance estimation for neural network pruning.
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264–11272, 2019.

nguyen2020input
Tuan Anh Nguyen and Anh Tran.
Input-aware dynamic backdoor attack.
Advances in Neural Information Processing Systems, 33:3454–3464, 2020.

nichol2021glide
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models.
arXiv preprint arXiv:2112.10741, 2021.

song2020denoising
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
arXiv preprint arXiv:2010.02502, 2020.

NEURIPS2019_3001ef25
Yang Song and Stefano Ermon.
Generative modeling by estimating gradients of the data distribution.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.

song2020score
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
arXiv preprint arXiv:2011.13456, 2020.

wang2019neural
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao.
Neural cleanse: Identifying and mitigating backdoor attacks in neural networks.
In 2019 IEEE Symposium on Security and Privacy (SP), pages 707–723. IEEE, 2019.

wolleb2022diffusion
Julia Wolleb, Florentin Bieder, Robin Sandkühler, and Philippe C Cattin.
Diffusion models for medical anomaly detection.
In International Conference on Medical image computing and computer-assisted intervention, pages 35–45. Springer, 2022.

wu2021adversarial
Dongxian Wu and Yisen Wang.
Adversarial neuron pruning purifies backdoored deep models.
Advances in Neural Information Processing Systems, 34:16913–16925, 2021.

xue2020one
Mingfu Xue, Can He, Jian Wang, and Weiqiang Liu.
One-to-n & n-to-one: Two advanced backdoor attacks against deep learning models.
IEEE Transactions on Dependable and Secure Computing, 19(3):1562–1578, 2020.

yang2023video
Siyuan Yang, Lu Zhang, Yu Liu, Zhizhuo Jiang, and You He.
Video diffusion models with local-global context guidance.
arXiv preprint arXiv:2306.02562, 2023.

zhang2022fast
Qinsheng Zhang and Yongxin Chen.
Fast sampling of diffusion models with exponential integrator.
arXiv preprint arXiv:2204.13902, 2022.

zhao2024unipc
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu.
Unipc: A unified predictor-corrector framework for fast sampling of diffusion models.
Advances in Neural Information Processing Systems, 36, 2024.

zheng2022data
Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu.
Data-free backdoor removal based on channel lipschitzness.
In European Conference on Computer Vision, pages 175–191. Springer, 2022.










§ DERIVATION OF X_T^S




In <ref>, we introduce a unified form of backdoor diffusion as presented in <ref>, and derive {x_t^s}_t=1^T, representing a special case of backdoor attacks on DMs. To guarantee that x_t^s could be represented by <ref>, we propose a corresponding backdoor diffusion process as shown in <ref>.

    q^b(x_t^s|x_t-1^s)=𝒩(x_t^s; μ_t^s,w_t^2𝐈),  t∈[1,…,T]

where μ_t^s=√(α)_t x_t-1^s+(1-√(α_t))r^s and w_t=γ√(1-α_t). Obviously, α̂_t=√(α̅_t). Let k_t=1-√(α_t), we can derive the coefficient of r^s at timestep t, ρ̂_t as follows using induction.

    ρ̂_t=k_t+√(α_t) k_t-1+√(α_t α_t-1) k_t-2+⋯+√(α_t …α_2) k_1=1-√(α̅_t),  t∈[1,…,T]

Combine <ref> and <ref>, we could derive the reversed conditional transition q^b(x^s_t-1|x^s_t, x^s_0) as follows. For simplicity and clarity, we omit the superscript s.




    q^b(x_t-1|x_t, x_0) 
        =q^b(x_t |x_t-1, x_0) q^b(x_t-1|x_0)/q^b(x_t |x_0)
       ∝exp(-1/2((x_t-√(α_t)x_t-1-k_t r)^2/w_t^2+(x_t-1-α̂_t-1x_0-ρ̂_t-1r)^2/β̂^2_t-1.. 
       ..-(x_t-α̂_t x_0-ρ̂_tr)^2/β̂^2_t)) 
        =exp(-1/2((x_t-√(α_t)x_t-1)^2-2(x_t-√(α_t)x_t-1) k_t r+k_t^2 r^2/w_t^2.. 
        +(x_t-1-α̂_t-1x_0)^2-2(x_t-1-α̂_t-1x_0) ρ̂_t-1r+ρ̂_t-1^2 r^2/β̂^2_t-1
       ..-(x_t-α̂_t x_0-ρ̂_tr)^2/β̂^2_t)) 
        =exp(-1/2((x_t-√(α_t)x_t-1)^2/w_t^2+(x_t-1-α̂_t-1x_0)^2/β̂^2_t-1-2(x_t-√(α_t)x_t-1) k_t r/w_t^2.. 
       ..-2(x_t-1-α̂_t-1x_0) ρ̂_t-1r/β̂^2_t-1 + k_t^2 r^2/w_t^2 + ρ̂_t-1^2 r^2/β̂_t-1^2-(x_t-α̂_t x_0-ρ̂_tr)^2/β̂^2_t)) 
        =exp(-1/2((α_t/w_t^2+1/β̂^2_t-1) x_t-1^2-2(√(α_t)/w_t^2x_t+α̂_t-1/β̂^2_t-1x_0... 
       ...+(ρ̂_t-1/β̂^2_t-1-√(α_t) k_t/w_t^2) r) x_t-1+C(x_t, x_0, r)))

where α̂_t=√(α̅_t), β̂_t is the fixed variance and C(x_t, x_0, r) is a function that contains x_t, x_0 and r.


Thus, the mean of x_t-1 can be derived as <ref>.


    μ_t(x_t, x_0, r) =    (√(α_t)/w_t^2x_t+α̂_t-1/β̂^2_t-1x_0+(ρ̂_t-1/β̂^2_t-1-√(α_t) k_t/w_t^2) r) /(α_t/w_t^2+1/β̂^2_t-1) 
    
    =    (√(α_t)/w_t^2x_t+α̂_t-1/β̂^2_t-1x_0+(ρ̂_t-1/β̂^2_t-1-√(α_t) k_t/w_t^2) r) w_t^2 β̂^2_t-1/α_t β̂^2_t-1+w_t^2
    
    =    √(α_t)β̂^2_t-1/α_t β̂^2_t-1+w_t^2x_t+α̂_t-1 w_t^2/α_t β̂^2_t-1+w_t^2x_0 +w_t^2 ρ̂_t-1-√(α_t) k_t β̂^2_t-1/α_t β̂^2_t-1+w_t^2r

Based on <ref>, we can obtain the reparametrization x_0=1/√(α̅_t)(x_t-(1-√(α̅_t)) r-√(1-α̅_t)γϵ_t), where ϵ_t∼𝒩(0, 𝐈). Plug the reparametrization into <ref>, we can drivethe reversed diffusion process as follows.

    μ_t(x_t, r) =    √(α_t)β̂^2_t-1/α_t β̂^2_t-1+w_t^2x_t+w_t^2 ρ̂_t-1-√(α_t) k_t β̂^2_t-1/α_t β̂^2_t-1+w_t^2r+√(α̅_t-1) w_t^2/α_t β̂^2_t-1+w_t^2x_0 
    
    =   √(α_t)β̂^2_t-1/α_t β̂^2_t-1+w_t^2x_t+w_t^2 ρ̂_t-1-√(α_t) k_t β̂^2_t-1/α_t β̂^2_t-1+w_t^2r
        +w_t^2/α_t β̂^2_t-1+w_t^21/√(α̅_t)(x_t-(1-√(α̅_t)) r-√(1-α̅_t)γϵ_t)

<Ref> can assist defenders in performing backdoor sampling to defend against attacks requiring specific sampling procedures, such as TrojDiff, without knowing the attacker's formulas.




§ ALGORITHMS







 §.§ Trigger Inversion

<Ref> summarizes our trigger inversion
and backdoor detection method. Line 2 samples a batch of clean noise from standard Gaussian noise. Line 3 randomly initializes the trigger g. Lines 4-9 optimize g based on our proposed objective, using Adam optimizer with a learning rate of 0.1 and a maximum of 100 iterations in all experiments. Line 10 calculates our proposed metric TE with threshold=0.999. Lines 11-14 determine whether the presence of a backdoor. Line 15 returns the inverted trigger and the backdoor indicator.






 §.§ Backdoor Removal

<Ref> outlines the procedure of removing the backdoor and restoring the benign utility of the pruned model. Line 1 randomly chooses clean samples from the clean dataset. Line 2 samples a clean noise. Lines 3-11 calculate the loss of θ against clean inputs and backdoored inputs, as well as the gradients with respect to the substructures θ_i,k. Lines 12-17 calculate the aggregated Taylor importance scores as shown in <ref>. Lines 18-20 remove parameters based on their importance scores and finetune the pruned model with our proposed objective loss_p. Line 21 returns purified model θ^'.






§ MORE EXPERIMENTAL RESULTS




 §.§ Detailed Configuration




We use 27 clean and 161 backdoored DMs, executing <ref> and <ref> once for each model. We consider three existing backdoor attacks for DMs and use all trigger-target pairs from the original paper. For BadDiff and VillanDiff, we use poison rates from 0 to 0.9 and triggers of different sizes to train the model. For TrojDiff, we implement the D2I attack described in the paper and train models with the same hyperparameters as those used in the original paper. Given the diverse attack settings employed, we believe the results should be reliable. To measure the utility of the DM, we sample 10k images from clean sampling to compute the FID. For metrics relative to specifity, ASR, SSIM, ΔASR and ΔSSIM, we sample 100 images from backdoored sampling since we find this sufficient to demonstrate the effectiveness of the trigger or backdoor.


The pruned model was fine-tuned using 10% of the clean dataset (6000 images for the CIFAR-10 dataset and 3000 images for the CelebA-HQ dataset). The DMs trained on CIFAR-10 are finetuned with 40 epochs at a learning rate of 2e-4, and DMs trained on CelebA-HQ are finetuned with 100 epochs at a learning rate of 2e-4.




 §.§ Effect of gradients accumulation threshold



<Ref> presents the results of Diff-Cleanse with various threshold 𝒯 against TrojDiff, demonstrating that our approach is not sensitive to 𝒯. Our experiments across the three attack methods yield consistent results, indicating that neurons sensitive to the backdoor remain stable during sampling. Considering that the selection of 𝒯 has little impact on the effectiveness of Diff-Cleanse, we use 𝒯=900 in experiments outside the ablation study to accelerate computation.





 §.§ Effect of the trigger size



We use BadDiff to backdoor DMs with various trigger sizes and test Diff-Cleanse on them. As shown in <ref>, Diff-Cleanse can detect and eliminate all the backdoors.







 §.§ Effect of the backdoor removal loss



<Ref> compares the performance of ordinary loss function loss_c and our proposed loss_p on a DM attacked by TrojDiff. Moreover, we test 10 DMs and report the average results as shown in <ref>. Our proposed method consistently leads to lower Δ𝐅𝐈𝐃. Sufficient experiments demonstrated that the proposed loss function can improve the effect of fine-tuning.














 §.§ Details of adaptive attack



After realizing the criteria of Diff-Cleanse, the attacker may
train more concealed backdoor diffusion models, where the
difference in activation values of clean noise and backdoored
noise is reduced. To simulate this situation, we design an adaptive version of BadDiffusion. Specifically, we calculate the Euclidean distance between the activation of clean noise and backdoored noise at the same time step, and use it as a weighted term in the loss. Such an adaptive attack aligns the activation in clean sampling and backdoor sampling, making the backdoor-related neurons more stealthy.


As <ref> shows, the proposed adaptive attack method adds a weighted item to the original loss L_θ.

    L^'_θ(x, t, ϵ, g)=L_θ(x, t, ϵ, g)+λΔ a_t ^2

where Δ a_t is the difference between model outputs of the clean inputs and that of the backdoored inputs at time step t and · is the L-2 norm. 


Then we attack DMs with the modified BadDiff, which adopts adaptive attack loss L^'_θ and various λ. We train the DMs 50 epochs and evaluate them every 10 epochs. <Ref> shows the utility of models during training process, indicating that the new loss term has limited effect on model's utility. However, when λ=1e-1, the model's FID score increases continuously during training, indicating that excessive alignment of the activations can lead to training failure. With λ=1e-2, 1e-3, the diffusion models lose little clean generation ability. Except for λ=1e-1, we implant backdoors into all other DMs and <ref> shows the defence performance of Diff-Cleanse. Although the adaptive attack method makes the backdoor-related neurons more concealed, our framework can still remove the backdoor and achieve good image quality in clean sampling.











§ VISUALIZED RESULTS





In this section, we provide additional visual examples. <Ref> and <ref> illustrate the sample results of backdoored DMs before and after purified by Diff-Cleanse. After model repair, the model loses its ability to respond to the trigger and generate target outputs, while its clean utility remains unaffected.


<Ref> depicts the patch-based trigger of TrojDiff and the intermediate results of our algorithm. In this scenario, the trigger is a 3× 3 white block and the target is "Micky". Our method does not invert a trigger that causes the DM to generate the target, but instead activates another backdoor in the model, the colored square in the lower right corner of the images as shown in <ref> (c). Since this pattern appears repeatedly on different samples, we believe Diff-Cleanse could still help the users detect the backdoor created by such attacks.











§ PARAMETER INSTANTIATION FOR ATTACKS




Recall that the unified form of backdoor diffusion is as shown in <ref>, this section describes the diffusion process of existing backdoor attacks. 


BadDiff only attacks DDPM, where β_t is the predefined scalar, α_t=1-β_t and α̅_t=∏_i=1^t α_i. For backdoor diffusion process of BadDiff, α̂_t=α̅_t, ρ̂_t=1-√(α̅_t) and β̂_t=1-α̅_t. When attacking DDPM models, VillanDiff the same α̂_t and β̂_t as BadDiff but doubles the ρ̂_t, so that the injected backdoor can be activated when sampling with advanced samplers. TrojDiff use α̂_t=α̅_t, ρ̂_t=√(1-α̅_t) and β̂_t=(1-α̅_t)γ^2. 




 Paper Title: Revocable Backdoor for Deep Model Trading.










207 















A]Yiran Xu[Equal contribution.]
A]Nan Zhong
A]Zhenxing QianCorresponding Author. Email: zxqian@fudan.edu.cn 
A]Xinpeng ZhangCorresponding Author. Email: zhangxinpeng@fudan.edu.cn 

[A]Fudan University




§ ABSTRACT 
Deep models are being applied in numerous fields and have become a new important digital product. Meanwhile, previous studies have shown that deep models are vulnerable to backdoor attacks, in which compromised models return attacker-desired results when a trigger appears. Backdoor attacks severely break the trustworthiness of deep models. In this paper, we turn this weakness of deep models into a strength, and propose a novel revocable backdoor and deep model trading scenario. Specifically, we aim to compromise deep models without degrading their performance, meanwhile, we can easily detoxify poisoned models without re-training the models. We design specific mask matrices to manage the internal feature maps of the models. These mask matrices can be used to deactivate the backdoors. The revocable backdoor can be adopted in the deep model trading scenario. Sellers train models with revocable backdoors as a trial version. Buyers pay a deposit to sellers and obtain a trial version of the deep model. If buyers are satisfied with the trial version, they pay a final payment to sellers and sellers send mask matrices to buyers to withdraw revocable backdoors. We demonstrate the feasibility and robustness of our revocable backdoor by various datasets and network architectures.








§ INTRODUCTION


Deep learning has achieved impressive performance in a wide range of fields <cit.>. However, the security vulnerability of deep models, such as the notorious backdoor attack, hinders the deployment of deep models in some risk-sensitive domains <cit.>. Most attackers <cit.> implant backdoors into a clean model during the training phase by data poisoning. The compromised models behave normally during the evaluation phase, whereas they return attacker-desired results when the predefined trigger appears. Considering the stealthiness and hazard of backdoor attacks, it has attracted a lot of attention in the machine learning security community <cit.>.

Although backdoor attack is seen as a security threat or vulnerability for deep models in most studies, some researchers adopt backdoor attack to conduct some positive purposes <cit.>. There are various positive usages of the backdoor like dataset or model copyright protection <cit.>, artificial intelligence interpretability <cit.>, adversarial example defence <cit.>, etc. Adi et al. <cit.> adopt the backdoor attack to implement model watermarking for model copyright protection. They use significantly abnormal outputs of models triggered by poisonous inputs as the watermarking signal to verify the ownership of the model. A similar idea <cit.> is also implemented to achieve dataset ownership protection. Shan et al. <cit.> utilize the backdoor as a honeypot for deep models to resist adversarial example attack. When an attacker constructs an adversarial example for the backdoored model, the distribution of the interior feature map of the adversarial example is close to that of poisonous ones. Therefore, defenders identify adversarial example input by inspecting the distribution of the interior feature map.


 


In this paper, we first propose a novel concept of the revocable backdoor and explore its promising application which is deep model trading. The revocable backdoor denotes that attackers implant backdoors into clean deep models while attackers can easily withdraw these backdoors. In other words, attackers control poisoning and detoxification for deep models at the same time. The detoxification should be easy and efficient. Revocable backdoor still needs to satisfy the general properties of a backdoor. When inputs contain the trigger, the model with revocable backdoor produces the attacker's predefined outputs. Otherwise, the model should behave normally and produce correct output. Additionally, it should possess stealth and robustness to prevent users from detecting and removing the backdoor. The revocable backdoor can be applied to deep model trading to protect the rights and interests of both buyers and sellers.

The main contributions of this paper are three-fold: 

    
  * We propose a promising deep model trading scenario, and hold that the revocable backdoor can address the issues haunted by buyers and sellers. 
    
  * We propose a practical revocable backdoor, which does not require extra clean or poison data to withdraw backdoors. 
    
  * Extensive experiments demonstrate that our proposed revocable backdoor is feasible and robust. 






§ MODEL TRADING SCENARIO


Since the wild usage of deep learning models in real world, models become a tradable products. However, how to sale a trained model or buy a trained model still remains a question. The Uniform Commercial Code (UCC) <cit.> stipulates that when the goods delivered by the seller do not conform to the contract, the buyer has the significant right to "revoke acceptance". Correspondingly, when the buyer breaches the contract, such as improperly rejecting the goods, incorrectly revoking the acceptance of the goods, failing to pay the price etc., the seller has the right to undertake a series of remedial measures, including recovering the goods, reselling, and so on.

Therefore, the rights and demands of both the buyer and seller in the model transaction can be summarized as follows:

Sellers seek to profit by selling their own models. As models are considered data resources, a model "return" would carry immeasurable loss of benefits. They attempt to control costs as much as possible to increase profits.

Buyers hope to purchase models that meet their needs. When they find that the purchased model does not meet their needs, they want to be able to return it and get a refund.

Trading platform provides a reliable platform for model transactions, safeguarding the reasonable rights and interests of both buyers and sellers.

However, there are still a few issues:

    
  * Q1. How to allow the buyer to return and refund the purchased model when they find it does not meet their needs, ensuring their right to "revoke acceptance"?
    
    
  * Q2. How to protect the seller's interests without being damaged when the buyer returns the model? That is, how to prevent the buyer from saving some of the data after receiving the model, and then returning and refunding it.


A possible solution to these questions could be to send a trial version at the time of delivery. The full version would then be delivered once the buyer confirms receipt. Then, the questions turn into:


    
  * Q3. How to ensure that both the trial version and the full version of the model meet the buyer's requirements?
    
    
  * Q4. How to ensure the model will not be resold by the buyer?


To solve these two questions, we propose the concept of a revocable backdoor. Fig. <ref> illustrates the details of the deep model trading scenario. A buyer distributes his requirement to a seller. Then, the seller trains a model based on the buyer's requirements. The model has a revocable backdoor and can be sent to the buyer as a trail version. The buyer pays a deposit to obtain the trial model whose performance is almost the same as the clean final model over a normal task. Next, the buyer evaluates the trial model on his/her application and determines whether accepts the model and pays a final payment. If the buyer is satisfied with the trial version, they can pay the seller the final balance. After that, the seller sends the mask matrix to the buyer and the revocable backdoor can be removed by the mask matrix. 

 Buyers can disclose the backdoor hidden in the model publicly and make the trial model unusable. Fig. <ref> illustrates the details of the deep model trading scenario.
The advantages of using the revocable backdoor are as follows:


    
  * It allows the user to try the model while securing the rights of the seller. In the worst case, the buyer maliciously refuses to pay the final payment and deploys the trial model in his/her application. The seller can disclose the backdoor hidden in the model publicly and makes the trail model unusable. Or, if there is a third part (sale platform) involved, the seller can register the mask and sue the buyer for infringement. (Address Q1 and Q2.)
    
    
  * It also ensures the relevance between the trial version and the complete version of the model, preventing the buyer from receiving a complete model that does not match the trial version after paying the balance and confirming receipt. (Address Q3.)
    
    
  * There is no backdoor in the complete version of the model, but the seller can still use the mask matrix to identify whether a model is the one they sold (like a serial number of a software). That is, by subtracting the mask matrix from the model, determining whether the model with the mask matrix removed contains a preset backdoor to authenticate the model's genuineness. (Address Q4.)






§ RELATED WORK




 §.§ Backdoor Attack

Backdoor attack is an emerging topic in the machine learning security community <cit.>. The goal of the backdoor attack is to compromise deep models, which return normal results for clean inputs, but return attacker-desired results when the trigger appears. Gu et al. <cit.> first propose the concept and specific practical backdoor attack approach named BadNets. BadNets mounts an attack through data poison during the training phase. A conspicuous colourful square is adopted as a trigger pattern. BadNets generate some poisonous images by adding the trigger pattern to some clean images and changing their label to the target (attacker-desired) label. Such poisonous images are mixed with other clean training sets. The deep model trained over a poisonous training set contains an insidious backdoor which only activated by the trigger pattern. The implementation of the backdoor attack can be roughly grouped into two categories. 

Poisoning-based backdoor mounts an attack by inserting some well-designed poisonous samples into the training set <cit.>. Previous researchers focus on designing stealthy and efficient trigger patterns. Although BadNets proposes a seminal approach for backdoor attacks, its trigger pattern is conspicuous. In the follow-up studies, researchers explore some invisible trigger patterns. Li et al. <cit.> and Li et al. <cit.> adopt image steganographic noise ( Least Significant Bit and DNNs-based steganographic ) as a trigger pattern. The visual quality of poisonous is greatly boosted by invisible trigger patterns. Furthermore, WaNet <cit.> adopts image affine transformation as a trigger pattern. The poisonous images can be viewed as slightly warped from clean ones. WaNet works effectively not only in the digital domain but also attacks successfully in the physical domain.

Training process controlling-based backdoor can achieve more stealthy attacks than poisoning-based ones <cit.>. As a trade-off, these approaches entail controlling the entire training phase. These attacks focus on both the stealthiness of the trigger pattern and compromised models. Backdoor attacks may leave traces in compromised models. Doan et al. <cit.> focus on the feature map separability of compromised models and employ Wasserstein distance to minimize the feature map distance between poisonous and clean inputs. Some similar ideas are proposed in subsequent studies. Besides consideration of the feature map domain, BppAttack <cit.> employs contrastive supervised learning and adversarial training instead of standard cross-entropy loss in the classification task. This learning approach ensures that compromised models steadily and successfully converge.





 §.§ Backdoor Defence

Considering the serious threat of backdoor attacks to machine learning security, backdoor defence, which is the countermeasure against backdoor attacks, rapidly develops in recent studies <cit.>. There are two main categories in the backdoor defence, i.e., backdoor detection <cit.>, backdoor purification <cit.>

In terms of backdoor detection approaches, they aim to determine whether the deep model contains a backdoor. Neural Cleanse (NC) <cit.> is one of the well-known backdoor detection algorithms. NC observes that the backdoor of the classifier constructs a shortcut between the target label and other clean labels. Based on this observation, NC employs a abnormality detection algorithm to determine whether the model is poisoned by analyzing the size of perturbation leading all clean images to the target label. 

Compared with backdoor detection, backdoor purification aims to eliminate backdoors on the premise that the performance of clean inputs does not significantly degrade. Fine-Pruning <cit.> adopts network pruning to erase backdoors hidden in the models. For existing neural networks, most neurons are dormant when the clean inputs come. However, these dormant neurons may be activated by the trigger pattern and lead models to return attacker-desired results. Therefore, Fine-Pruning identifies dormant neurons based on some clean inputs then deletes them and fine-tunes the model. NAD (Neural Attention Distillation) <cit.> is an alternative effective backdoor purification approach, which utilizes model distillation to erase backdoors. NAD fine-tunes backdoored model to generate a teacher model which is used to conduct model distillation. Many other sophisticated model purification approaches are proposed in subsequent studies.







§ REVOCABLE BACKDOOR



 


 §.§ Preliminaries



In this paper, we focus on the image classification task which is consistent with previous studies. Since the revocable backdoor is first proposed by ours, we describe more details of the revocable backdoor requirement applied in the deep model trading. 

Goal The revocable backdoor aims to ensure that buyers complete the final payment when they are satisfied with the trial model. If buyers refuse to complete the final payment, sellers can disclose the hidden backdoor and corresponding trigger pattern. For example, buyers obtain a face recognition model with a revocable backdoor and deploy this model in their company. If the backdoor is disclosed, anyone with the trigger pattern can break the face recognition model, that is, the model is unusable. Note that our revocable backdoor does not aim to protect the copyright of the model, which is usually accomplished by model watermarking <cit.>. Conventional backdoor-based watermarking aims to verify the ownership of the model, whose goal is completely different from ours. Although backdoor-based watermarking also can make the model return incorrect results when the trigger appears, the backdoor is permanent and irrevocable. Even though buyers complete the final payment, the backdoor still exists in the model and brings security threats to the model deployment.

Effectiveness In previous backdoor attacks, attackers aim to make the compromised classifier return the target label when the trigger appears <cit.>. However, in the context of model tradings, the goal of our revocable backdoor for model sellers is different from previous studies. Sellers can distribute the trigger pattern to make the classifier held by buyers unusable in the event that the buyer does not complete the final payment. Therefore, our revocable backdoor aims to decrease the classification accuracy as low as possible when the trigger appears. In other words, we view the attack success if the compromised classifier returns an arbitrarily wrong label. 

Fidelity The classifier with a revocable backdoor can be used as a trial version of the final classifier for buyers. Therefore, the revocable backdoor aims to keep the performance of the backdoored classifier over clean inputs the same as a clean classifier. For buyers, the performance of the trial version over clean inputs is almost identical to the final version (backdoor-free). In this case, buyers can evaluate whether the classifier meets their requirements.

Revocability The crux of our revocable backdoor is that we can easily withdraw the backdoor without requiring an extra training phase or clean/poisonous data. The revocability denotes that the compromised classifier whose backdoor is withdrawn can correctly predict poisonous inputs.

Robustness In the model trading scenario, the backdoors hidden in the compromised model can and only can be withdrawn (erased) by sellers. Therefore, our attack should ensure that the backdoor is resistant to various backdoor purification defences. 





 §.§ Method


Fig. <ref> illustrates the framework of our approach. 
Let 𝒟 = { (x_i, y_i) }_i=1^N denotes the benign training set, where x_i ∈𝒳= {0,1,…, 255}^C× W × H is the image, y_i ∈𝒴 = {1,…, N} is its label, and N is the number of classes. The trigger pattern is denoted as δ, where δ∈𝒫={-t,-t+1,…,t+1,t}^C× W × H. The elements of the trigger pattern are between -t and t. In the backdoor attack, we select a portion of clean images x, which are stamped with the trigger pattern, as poisonous ones. We denote the clean training set and poisonous training set as 𝒟_c and 𝒟_p, respectively. Given a deep neural network-based classifier f_θ(·), we aim to minimize the following equation,

    L_bd = 1/|D_c|∑_(x,y) ∈ D_cℓ( f_θ(x),y) 
     - 1/|D_p|∑_(x,y) ∈ D_p max( ℓ( f_θ(x+δ),y),c),

where ℓ(·) denotes cross-entropy loss widely adopted in classification tasks. We update f_θ(·) and δ to minimize equation <ref>. The core idea of equation <ref> is to achieve the effectiveness and fidelity of the revocable backdoor attack. The first term of equation <ref> ensures that the performance of backdoored classifiers over clean inputs does not drop. The second term denotes maximizing the cross-entropy loss of poisonous inputs, i.e., making the classifier return wrong results when the trigger appears. However, maximizing the cross-entropy loss of poisonous inputs is much easier than minimizing the cross-entropy loss of clean inputs. In other words, classifier f_θ(·) easily learns how to give wrong results for poisonous inputs but it is hard to learn how to give correct results for clean inputs. As a result, we adopt max(·)  and hyperparameter confidence c to restrict the second term. For a N categories classification task, if the cross-entropy loss of input is larger than -log(1/N), the classifier will return an incorrect result. Therefore, we set c=-1.1× log(1/N) that is slightly larger than the minimum correct decision threshold for a N categories classifier. If we remove the restriction of max(·) or set an excessively large c, the classifier f_θ will only learn the second term of equation <ref> and overlook the first term.

Besides L_bd, our approach also entails considering how to withdraw our backdoor. The modern deep neural network-based classifier f_θ(·) is made up of some cascade layers. The output of f_θ(·) can be expressed as,

    f_θ(·) = f_n(⋯f_2( f_1(⋯))),

where n denotes the number of layers. The output of the interior layer is named as feature map. Based on the cascade structure of f_θ(·), we employ some matrices named masks to manipulate feature maps. Compromised classifiers learn two tasks, i.e., a normal task and an insidious backdoor task. These two tasks conduct different inference links. In other words, the feature maps of these two tasks are different. Therefore, we can adopt some mask matrices to break backdoors without affecting clean inputs. We denote the classifier f_θ(·) with mask matrices M_k as f_θ_mask(·). The formulaic expression is shown as follows,

    f_θ_mask(·) =  f_n(⋯ M_2· f_2( M_1· f_1(x))),

where M and · denote trainable mask matrices and Hadamard product, respectively. The size of M_k is the same as the corresponding feature map. The initial elements of M_k are filled with 1. Moreover, we require a regularization term for M_k. The goal of regularization is used to make the elements of M_k as close to 1 as possible. The regularization for M_k is expressed as follows,


    L_R = sum(|M_k_ij - 1 | ),

where sum(·) denotes the sum of all elements.
The usage of L_R forces M_k to change the most key elements related to poisonous inputs in feature maps. L_R can effectively mitigate the impact of M_k on clean inputs. For f_θ_mask(·), it should return correct results regardless of whether the trigger pattern δ appears. Therefore, the loss function for f_θ_mask(·) is expressed as follows,

    L_rev = [1/|D_c|∑_(x,y) ∈ D_cℓ( f_θ_mask(x),y) 
     + 1/|D_p|∑_(x,y) ∈ D_pℓ( f_θ_mask(x+δ),y)].

We update M_k by minimizing L_rev and make M_k adaptive find and break the poisonous inference link.
The final cost function is expressed as follows,

    L = L_bd+L_rev+α· L_R,

where α is the balance factor to control the weight of the regularization. We simultaneously update parameters of f_θ, trigger pattern δ, and mask matrices M_k to minimize L. The backdoor hidden in the well-trained f_θ can easily be withdrawn by adding M_k in the interior layers.  f_θ(·) and f_θ_mask(·) can be used as the trial model and final model for sellers during the model trading.





 §.§ Trigger Fine-tuning
 
In the previous subsection <ref>, we describe our approach to create a compromised classifier whose backdoors are controlled by mask matrices. The trigger pattern δ, whose elements are between -t and t, is trained with the classifier at the same time. In this subsection, we delve into the design of the trigger pattern to control the balance of the trigger pattern imperceptibility and attack robustness. Hyperparameter t controls the modification magnitude of the trigger pattern. In other words, a small t ensures the imperceptibility of the trigger pattern. In this subsection, we frozen classifier f_θ and mask matrices M, i.e., f_θ_mask is also frozen, and update trigger patterns. The loss function of updating trigger δ is as follows,

    L = L_bd+L_rev+β· | δ|_2,

where | δ|_2 denotes the L_2-norm of the trigger δ, and β controls the balance of | δ|. We control the modification magnitude of the trigger pattern δ by changing the value of the hyperparameter β and t. 



 §.§ Backdoor Erasing for Existing Attacks

To the best of our knowledge, we are the first to propose the concept of the revocable backdoor. For existing conventional backdoor attacks like BadNtes <cit.>, WaNet <cit.>, Blend <cit.>, etc. They can not be withdrawn by attackers. In this subsection, we describe a backdoor erasing method to withdraw existing backdoors. In the model trading scenario, we suppose that buyers own some clean images and their corresponding poisonous ones. Then, they update the parameters of classifier f_θ by minimizing the following loss as,

    L_erasing = [1/|D_c|∑_(x,y) ∈ D_cℓ( f_θ(x),y) 
     + 1/|D_p|∑_(x,y) ∈ D_pℓ( f_θ(x+δ),y)].

Compared with equation <ref>, L_erasing minimizes the cross-entropy loss between clean and poisonous images and their label. The number of poisonous images is critical to the performance of backdoor erasing. We employ this method as our baseline. Compared with our proposed method <ref> and <ref>, this method requires gradient calculation and some poisonous images, which is not practical in the model trading scenario.  





§ EXPERIMENTAL RESULTS










 §.§ Experimental Setup




Datasets and Networks In this paper, we adopt three datasets including CIFAR-10 <cit.>, GTSRB <cit.>, and Sub-ImageNet to evaluate the performance of our approach. These datasets are widely used in previous studies <cit.>. The number of categories of CIFAR-10 and GTSRB is 10, and 43, respectively. Sub-ImageNet consists of 10 categories which are randomly selected from the original ImageNet dataset <cit.>. The image size of CIFAR-10, GTSRB, and Sub-ImageNet is 32× 32,  32× 32, and 224× 224, respectively. In terms of network architectures, we adopt ResNet-18 <cit.> and VGG <cit.> in the experiments.  

Backdoor Baselines Since we are the first to propose the concept of the revocable backdoor attack, there are no suitable revocable backdoor attacks used as a comparison. We suppose that users obtain some clean and corresponding poisonous images to finetune to a backdoored classifier to achieve revocability based on existing conventional attacks. We adopt various trigger patterns to conduct backdoor attacks. The trigger pattern includes BadNets <cit.>, Blend <cit.>, SIG <cit.>, LSB <cit.>, WaNet <cit.>, and BppAttack <cit.>. The details setting of baseline triggers can be found in the supplement. The compromised (backdoored) classifier is generated by updating equation <ref> for baselines. The poisoning rate is fixed as r=0.1. In the backdoor erasing phase, we suppose that users obtain o=5% clean data of the training set and corresponding poisonous ones. Then, we finetune the backdoored baseline classifiers with equation <ref>. 


Implement Details
Our approach consists of two steps, training revocable backdoored classifier and finetuning trigger pattern. The training epoch and batch size are 180, and 256, respectively. We adopt SGD (Stochastic Gradient Descent) optimizer for updating the parameters of the classifier. The initial learning rate of SGD is 0.01. For trainable mask matrices and trigger pattern δ, we adopt Adam optimizer with a 0.001 initial learning rate. The poisoning rate is 0.5 for the first 80 epochs. The poisoning rate times by 0.5 in intervals of 10 epochs when the iteration epoch exceeds 80. Hyperparameter α is fixed as 10, 10, and 100 for CIFAR-10, GTSRB, and Sub-ImageNet, respectively. In the second phase, we fixed the poisoning rate as 0.5. The hyperparameter β is 0.01 for all datasets. The hyperparameter t controlling the range of our trigger pattern is 10 in both phases.

























 §.§ Attack Effectiveness

We adopt accuracy to measure the performance of the classifier. In this part, we evaluate the Effectiveness, Fidelity and Revocability of our approach. The results of our approach and baselines are shown in Table <ref>. We also train a clean classifier whose training setup is the same as compromised ones as a reference.  We aim to significantly degrade the classifier accuracy when the trigger appears while keeping the accuracy for clean inputs unchanged. In terms of revocability, the goal of mask matrices employed in our attack is to withdraw the backdoor. From Table <ref>, our approach achieves satisfactory results in the aspect of attack effectiveness, fidelity, and revocability. The accuracy for clean inputs is almost unchanged compared with the clean classifier in most cases. The accuracy for poisonous inputs is decreased to near even less than random guesses, meanwhile, it recovers to standard performance (like a clean classifier) when we insert mask matrices to withdraw the backdoor. For baselines, their performance is significantly inferior to ours. In previous common backdoor studies <cit.>, their attack setting is that a compromised classifier returns the target label when the trigger appears. This attack setting is much easier than our setting, in which the compromised classifier does not need to understand the semantic information of inputs and only builds a strong relation between the trigger and target label. However, the compromised need to understand the semantic information of inputs and then return wrong results when the trigger appears in our attack setting. Therefore, some invisible triggers (like LSB and BppAttack) whose perturbations are small cannot successfully compromise the classifier. The experiments of VGG network can be found in Table 2 of the supplementary material.




Apart from quantitative results, we also visualize our trigger pattern in Fig. <ref>. Our approach achieves satisfactory visual results and users cannot perceive trigger patterns. Based on Fig. <ref> and Table <ref>, we find that only triggers (like BadNets) with large modification (visible) can attack successfully among baselines. Our approach achieve both trigger invisibility and attack effectiveness at the same time.  



 


 §.§ Attack Robustness




We recall our revocable backdoor scenario, i.e., deep model trading scenario. For sellers, the goal of the revocable backdoor is to make the model unusable by distributing the trigger pattern to the public if buyers do not pay the final payment. Sellers can send the trial model to buyers and let them know the existence of the hidden backdoor. Therefore, we do not consider backdoor detection defence in the robustness evaluation. We focus on evaluating the resistance of our backdoor against backdoor purification defences <cit.>. In other words, we evaluate whether buyers can adopt backdoor purification to erase the backdoor without the permission of sellers. 

The easiest backdoor purification method, which may be adopted by buyers, is fine-tuning. Buyers collect some clean images whose distribution is similar to the training set to fine-tune the trial classifier containing our revocable backdoor. 
To simulate fine-tuning, we adopt SGD optimizer with 0.01 learning rate kept the same as the training phase to fine-tune the compromised classifier with 5% clean images from the training set. We fine-tune the classifiers for 50 epochs. The experimental results have been shown in Table <ref>. Fine-tuning cannot remove our backdoor in all cases.
 
 




An alternative well-known backdoor is Fine-pruning <cit.>, which combines network pruning and fine-tuning. As aforementioned related work part, most neurons are dormant for clean inputs. These dormant neurons almost do not affect the final performance of clean inputs. However, they may be activated by poisonous inputs. Therefore, Fine-pruning prunes dormant neurons with some clean inputs and fine-tunes the pruned classifier at the same time. The specific defence setup is as follows. We pruning the last layer of the classifier which is widely adopted in previous studies. The classifier is fine-tuned in the interval of 10 neurons. The optimizer is SGD with a 0.01 learning rate. We set the maximum tolerable drop of the clean task accuracy as less than 1%. Table <ref> shows the results of Fine-pruning. It also cannot remove our backdoor. The details of the accuracy (both clean and poisonous inputs) with regard to the number of pruned neurons can be found in the supplementary material.





Besides fine-tuning and fine-pruning, we also evaluate the robustness of our approach by NAD <cit.>. NAD first trains a teacher model. Then, NAD adopts the teacher model as a guide to conduct attention distillation. The experimental results of NAD are shown in Table <ref>. NAD is ineffective in our approach. Although NAD can erase the backdoor in some cases (GTSRB dataset), the accuracy of clean inputs severely degrades compared with the clean model. 







 §.§ Ablation Studies

In our approach, we adopt two steps to generate the trigger pattern. The second part, which fine-tunes the trigger, aims to control the perturbation of trigger pattern. Table <ref> gives the quantitative comparison between the trigger with or without fine-tuning. The PSNR (poisonous image quality) and SSIM <cit.> is significantly improved benefit from the trigger fine-tuning.


More ablation studies over confidence c, regularization L_R and the uniqueness of the mask matrices can be found in the supplement. 






§ CONCLUSIONS


In this paper, we first propose the concept of the revocable backdoor attack and a novel application, that is, deep model trading. Revocable backdoor aims to create a trial model whose performance is similar to the clean/final model for buyers to evaluate its performance. Meanwhile, sellers can easily withdraw the backdoor hidden in the trial model when they obtain the final payment. The revocable backdoor attack is achieved by controlling the interior feature maps of models. We train models and mask matrices at the same time. The mask matrices are critical to turning the backdoored (trial) model into a clean (final) model. Extensive experiments demonstrate that our approach is feasible and robust. 








This work was supported by the National Natural Science Foundation of China under Grants U20B2051, 62072114, U20A20178, U22B2047










 Paper Title: Securing Proof of Stake Blockchains: Leveraging Multi-Agent Reinforcement Learning for Detecting and Mitigating Malicious Nodes.














Securing Proof of Stake Blockchains: Leveraging Multi-Agent Reinforcement Learning for Detecting and Mitigating Malicious Nodes
    Faisal Haque Bappy^1, Kamrul Hasan^2, Md Sajidul Islam Sajid^3, Mir Mehedi Ahsan Pritom^4, and Tariqul Islam^5

^1, 5 Syracuse University, Syracuse, NY, USA

^2 Tennessee State University, Nashville, TN, USA

^3 Computer and Information Sciences, Towson University, Towson, MD, USA

^4 Computer Science, Tennessee Tech University, Cookeville, TN, USA

Email: {fbappy@syr, mhasan1@tnstate, msajid@towson, mpritom@tntech, mtislam@syr}.edu 

    August 5, 2024
==========================================================================================================================================================================================================================================================================================================================================================================================================================================================




fancy
 This work has been accepted at the 2024 IEEE Global Communications Conference (Globecom 2024)





§ ABSTRACT 
Proof of Stake (PoS) blockchains offer promising alternatives to traditional Proof of Work (PoW) systems, providing scalability and energy efficiency. However, blockchains operate in a decentralized manner and the network is composed of diverse users. This openness creates the potential for malicious nodes to disrupt the network in various ways. Therefore, it is crucial to embed a mechanism within the blockchain network to constantly monitor, identify, and eliminate these malicious nodes without involving any central authority. In this paper, we propose MRL-PoS+, a novel consensus algorithm to enhance the security of PoS blockchains by leveraging Multi-agent Reinforcement Learning (MRL) techniques. Our proposed consensus algorithm introduces a penalty-reward scheme for detecting and eliminating malicious nodes. This approach involves the detection of behaviors that can lead to potential attacks in a blockchain network and hence penalizes the malicious nodes, restricting them from performing certain actions. Our developed Proof of Concept demonstrates effectiveness in eliminating malicious nodes for six types of major attacks. Experimental results demonstrate that MRL-PoS+ significantly improves the attack resilience of PoS blockchains compared to the traditional schemes without incurring additional computation overhead.



Blockchain, Proof of Stake, Multi-agent Reinforcement Learning, Malicious Node Detection, Attack Mitigation





§ INTRODUCTION

Proof of Stake blockchains have garnered considerable attention as viable alternatives to traditional Proof of Work systems due to their scalability and energy efficiency advantages. Ethereum's recent shift to PoS marks a significant milestone in the evolution of blockchain technology. PoS allows Ethereum validators to secure the network and validate transactions based on the amount of cryptocurrency they hold, rather than relying on computational power <cit.>. In PoS blockchains, consensus mechanisms rely on participants' stakes in the network rather than computational power, offering a more environmentally sustainable approach to blockchain validation. However, the decentralized nature of PoS blockchains introduces inherent challenges, particularly in terms of security and resilience against malicious actors <cit.>. As diverse user nodes interact within the network, the potential for malicious behaviors to disrupt operations poses a significant threat to the integrity and stability of PoS-based systems.

The openness and inclusivity of PoS blockchains, while beneficial for democratizing access and participation, also create opportunities for exploitation by malicious nodes <cit.>. These nodes may seek to undermine the network through various means, including double spending attacks, Sybil attacks, and DDoS attacks <cit.>. The absence of centralized authority in PoS blockchains necessitates the implementation of robust mechanisms to detect, deter, and mitigate the impact of such malicious activities autonomously.

In response to these challenges, this paper proposes MRL-PoS+, a novel consensus algorithm that leverages Multi-agent Reinforcement Learning techniques to enhance the security of PoS blockchains. By integrating MRL principles into the consensus process, MRL-PoS+ aims to empower nodes within the network to actively identify and respond to malicious behaviors without centralized oversight. Through a penalty-reward framework, MRL-PoS+ incentivizes honest participation while penalizing and restricting the actions of malicious nodes, thereby fortifying the resilience of PoS blockchains against potential attacks. Through empirical analysis and experimentation, we demonstrate the efficacy of MRL-PoS+ in enhancing the security and robustness of PoS-based blockchains. The following are the core contributions of this paper,


    
  * We introduced MRL-PoS+, a novel consensus algorithm leveraging Multi-agent Reinforcement Learning techniques to secure the blockchain network without having any central authority.
    
  * We implemented a penalty-reward mechanism within MRL-PoS+ to incentivize honest behaviors and penalize malicious nodes.
    
  * We validated MRL-PoS+ through a Proof of Concept (PoC) implementation, demonstrating effectiveness against six major types of attacks.
    
  * Empirical analysis highlights that MRL-PoS+ performs better than traditional schemes in enhancing network security without adding computational overhead.


The remainder of the paper is structured as follows: in Section <ref>, we present existing works related to the security of PoS blockchain and reinforcement learning. In Section <ref>, we described six major attacks in blockchain and their potential indicators. In Section <ref>, we explained how MRL-PoS+ integrates reinforcement learning inside a PoS blockchain network. Then in Section <ref>, we present our system architecture, followed by a performance analysis in Section <ref>. Finally, Section <ref> concludes the paper.



















§ RELATED WORKS


In recent years, researchers have been exploring the integration of blockchain technology with multi-agent reinforcement learning to enhance the security, efficiency, and reliability of decentralized systems. Several studies <cit.>, have investigated consensus mechanisms and security measures in blockchain networks. Zou et al. <cit.> propose a consensus protocol leveraging reinforcement learning to optimize efficiency and fairness, particularly in IoT networks. Li et al. <cit.> introduce a framework for trust management in IoT using blockchain, emphasizing credibility improvement and energy conservation. Oliveira et al. <cit.> explore a reputation-based consensus mechanism to enhance security against adversarial attacks. 

Meanwhile, Cachin et al. <cit.> review various consensus protocols across blockchain platforms, emphasizing fault tolerance and resistance against malicious actors. Several other studies explore the integration of blockchain and multi-agent systems, often with a focus on reinforcement learning. Sami et al. <cit.> propose utilizing blockchain for transparent and auditable model updates in multi-agent reinforcement learning. Calvaresi et al. <cit.> advocate for combining blockchain and multi-agent systems to enhance trust and accountability. Nguyen et al. <cit.> propose a cooperative task offloading scheme for blockchain-based mobile edge computing, leveraging reinforcement learning for resource allocation. Shen et al. <cit.> develop a distributed multi-agent reinforcement learning-based algorithm for multi-object tracking, integrated into a blockchain framework. While these studies demonstrate the potential synergy between blockchain and multi-agent reinforcement learning, none explicitly addresses the utilization of MRL for malicious node detection and defense within blockchain networks.





§ ATTACKS IN BLOCKCHAIN


In this section, we present an overview of 6 prominent blockchain attacks, their typical symptoms as documented in existing literature, and the detection criteria that contribute to the detection mechanisms employed in MRL-PoS+.



 §.§ 51% Attack

The 51% Attack poses a significant threat to blockchain integrity. It occurs when a single entity or coalition controls over half of the network's computational power, allowing them to manipulate transaction history and potentially disrupt consensus mechanisms <cit.>. Symptoms include sudden fluctuations in network hash rate, frequent forks or reorganizations of recent blocks, and conflicting transactions within the blockchain <cit.>. 

Detection Criteria. i) Analyzing block generation against a predefined threshold, ii) comparing node hash rates to the network average, and iii) calculating fork frequency collectively determine the likelihood of a 51% attack.



 §.§ Double-Spending Attack

The Double-Spending Attack remains a persistent concern for blockchain networks, exploiting transaction confirmation delays to spend the same cryptocurrency units multiple times. This attack can exploit consensus vulnerabilities or manipulate transaction records to achieve its objectives <cit.>. Symptoms of a Double-Spending Attack manifest as the presence of identical transaction IDs in multiple unconfirmed blocks, sudden discrepancies in account balances, and anomalous transaction histories <cit.>. 

Detection Criteria. i) Analyzing the number of duplicate transactions, ii) evaluating the history of double spending from unconfirmed blocks, and iii) detecting discrepancies in balance.



 §.§ Sybil Attack

The Sybil Attack involves the creation of numerous fake identities or nodes to exert control or influence over a network. Attackers leverage this tactic to disrupt consensus, manipulate network voting processes, or inundate the network with spurious data <cit.>. Symptoms of a Sybil Attack include a sudden surge of network participants with similar IP addresses or behavioral patterns, as well as an influx of irrelevant data overwhelming the network <cit.>.

Detection Criteria. i) Monitoring for abrupt spikes in the network activity, ii) monitoring the presence of spam, and iii) scrutinizing whether nodes are attempting to manipulate voting data.



 §.§ Replay Attack

A Replay Attack occurs when a legitimate data transmission is maliciously repeated or delayed by an adversary. In the blockchain context, attackers exploit vulnerabilities to resubmit valid transactions multiple times, potentially leading to unauthorized actions or financial losses <cit.>. Symptoms of a Replay Attack include the appearance of duplicate transactions with differing timestamps and unexplained spikes in transaction fees <cit.>.

Detection Criteria. i) Scrutinizing for duplicate transactions within the network, ii) detecting a large number of conflicting transactions, and ii) considering any sudden or abnormal increases in transaction fees, indicating the presence of a probable replay attack.



 §.§ Smart Contract Vulnerabilities

Smart contracts, while offering automated and trustless execution of agreements, are susceptible to exploitation due to coding errors or vulnerabilities. Malicious actors may exploit these weaknesses to extract funds, disrupt operations, or manipulate contract outcomes <cit.>. Symptoms of Smart Contract exploitation encompass irregular movements of funds, abnormal activity surrounding specific contracts, and unexpected alterations in contract behavior <cit.>. 

Detection Criteria. i) Detecting any large unexplained transfers, and ii) identifying excessive activities on a single contract.



 §.§ DDoS Attack

Distributed Denial of Service (DDoS) attacks can disrupt network operations, impede transaction processing, and degrade overall system performance <cit.>. Symptoms of a DDoS Attack include frequent network errors, connectivity failures, and resource depletion on affected nodes <cit.>.

Detection Criteria. i) Examining excessive network errors and overuse of resources, and ii) identifying anomalies suggesting a node's attempt to disrupt transaction processing.



§ REINFORCEMENT LEARNING INSIDE A BLOCKCHAIN NETWORK


In PoS Blockchains, nodes compete to process transactions by staking assets, with the selection of a validator determined pseudo-randomly based on factors like stake amount and transaction history. In our proposed MRL-PoS+ approach, we introduce a reinforcement learning process for blockchain nodes. Since these nodes are operated by individual users seeking incentives, complete automation of their actions is not viable. Thus, we view them as semi-automated agents for reinforcement learning instead of fully automated ones. 



 §.§ Multi-Agent Reinforcement Learning

In multi-agent reinforcement learning (MRL), each agent's goal is typically to maximize its own cumulative reward, but the agents must learn to balance their individual goals with the overall goals of the group or system. The basic equation used in our proposed MRL-PoS+, known as the Q-learning algorithm, guides how agents update their strategies over time <cit.>. 

    Q(s, a) ← (1 - α) · Q(s, a) + α· (r + γ· max_a' Q(s', a'))

Where, Q(s, a) is the Q-value, α is the learning rate, r is the reward, γ is the discount factor, s' is the next state, and max_a' Q(s', a') is the maximum Q-value for the next state.






 §.§ Semi-Automated Agents
 
In traditional reinforcement learning, agents autonomously learn through network interaction. In our proposal, blockchain nodes are viewed as semi-automated agents, allowing human operators some control. Each node manages a reputation table, treating its transaction-related activities as actions. Nodes receive penalties or rewards, like allowances or action restrictions, based on their behavior, incentivizing honesty to earn rewards. Failure to comply may lead to exclusion from consensus, rendering the node ineffective.



 §.§ Set of Actions
 
Considering the entire blockchain network as the environment, a validator's main tasks, like transaction processing and block validation, form the set of actions. However, certain behaviors during these tasks may indicate malicious intent or an attack attempt. To bolster PoS blockchain security, such actions are subject to penalties. Building on the outlined attacks, we've identified 16 behaviors as malicious, listed comprehensively in Table <ref>.






 §.§ Penalizing Malicious Nodes and Rewarding Honest Nodes
 
In the event that a node exhibits any of the identified malicious behaviors during transaction processing, the system will impose penalties on the node. Additionally, nodes will be subjected to certain restrictions corresponding to their displayed behavior. To record the penalty, the node's reputation table will be updated, and the election process will take into account the reputation tables of all nodes. Nodes that do not exhibit any malicious behavior are deemed honest nodes and are eligible for rewards within the system. The magnitude of rewards varies depending on the node's state: i) if a node has never previously engaged in malicious behavior, it will receive the highest amount of reward, ii) for nodes with a history of previous malicious behavior, their negative reputations will be mitigated. Additionally, rewards will grant these nodes the ability to undertake certain actions that were previously restricted.





 §.§ Malicious Behavior Detection

To integrate the penalty-reward mechanism into the blockchain network, the initial challenge is monitoring and tracking every node action during transaction processing. In MRL-PoS+, we introduced attack probability detection mechanisms for six major attack types. A node's total attack probability aggregates probabilities of all attacks (b1, b2, b3, ..., b16), weighted by predefined criteria. These weights may be predetermined by network policy or dynamically adjusted based on consensus among participating nodes. By considering these probabilities, the system can make informed decisions on node selection.











        






















        




















        


    


        














        


        


        














        


        


        














        


        
















        


        





























§ SYSTEM ARCHITECTURE


As we prioritize PoS blockchain security, MRL-PoS+ mirrors traditional PoS systems in architecture. Figure <ref> outlines MRL-PoS+'s workflow: when a user initiates a transaction, network nodes compete to validate it. Unlike PoS, MRL-PoS+ factors node reputation into selection alongside stake. Selected validators, like Node 4, process transactions to create blocks. MRL-PoS+ integrates an activity tracker to monitor and record node behaviors without additional data storage for decentralization. The node structure is updated to accommodate tracked parameters. After tracking, MRL-PoS+ calculates penalties and rewards based on node actions, updates reputation tables, and commits blocks to the main chain. This comprehensive approach ensures blockchain integrity and security, encouraging honest node participation.











 §.§ Activity Tracker

The activity tracker detects malicious activities within the network, employing various detection mechanisms. It operates efficiently, eliminating the need for continuous monitoring by executing relevant functions after each action in the block creation process. This streamlined approach ensures swift detection and response to any suspicious behavior, bolstering the security and integrity of the blockchain network.



 §.§ Reputation Table

Each node within the network possesses a reputation table structured as key-value pairs. Upon joining the network, a new node initializes all parameters with empty or zero values. The reputation table comprises four key parameters: i) attackProbability: aggregates probabilities from b1 to b16 (refer to Table <ref>); ii) lastAttackAge: assigned based on a predetermined learning rate, influencing the node's ability to recover reputation swiftly by exhibiting honest behavior; iii) restrictions: a list of prohibited actions serving as penalties for malicious behaviors, with specific restrictions tied to exhibited malicious behaviors; iv) isActive: a binary parameter indicating the node's active participation in the consensus. If a node consistently engages in malicious activities, causing a substantial reputation loss, it becomes ineligible for the validator selection process and is marked inactive. Figure <ref> depicts a node's reputation table evolution over multiple rounds, reflecting parameter changes over time influenced by the node's actions and network dynamics.









 §.§ Validator Selection

In MRL-PoS+, when nodes express interest in processing a transaction, they commit their stake and engage in a selection process. The number of participants may vary depending on the incentive associated with each transaction. Algorithm <ref> outlines the validator selection process within MRL-PoS+. For each node, it first accesses the total attack probability from the reputation table. Then, it considers the age of the last attack and determines if the product of the attack's age and the attack probability exceeds a certain threshold value. If so, the node is excluded from selection, though its attack age decreases as it seeks consensus participation. For the remaining nodes, their agent score is calculated by considering attack probability, attack age, and stake. Based on this score, the nodes are sorted. If multiple nodes have the same highest stake, one of them will be chosen randomly. Otherwise, the node with the highest agent score will be chosen as the validator.

1em

 
1em









































 §.§ Penalty-Reward Mechanism

The process of assigning penalties or rewards is shown in Figure <ref>. Initially, the system checks whether the node has exhibited any malicious behaviors. If not, the node is considered honest and receives its incentives from the transaction fee. Additionally, the node's reputation values are updated: if it has any attack probability, it is decreased, and the values of b1 to b16 are set to negative. This prioritizes the node over less reputable nodes in future rounds. Contrarily, if the node engages in malicious behavior, its stake is retained as a penalty. Also, its reputation is diminished, with a higher attack probability assigned. Furthermore, restrictions are imposed on the node's actions, preventing it from attempting the same malicious behavior in subsequent transactions. 



§ PERFORMANCE ANALYSIS


In this section, we present the experimental results obtained with MRL-PoS+. We utilized our developed proof of concept to evaluate effectiveness and performance through several metrics.



 §.§ Simulation Environment


To evaluate our proposed consensus mechanism's functionality and performance, we developed a custom blockchain network using Go, similar to Ethereum's setup[The implementation of our proposed MRL-PoS+ is available on: 
  <https://github.com/SPaDeS-Lab/mrl-pos-plus>], ensuring compatibility with its PoS blockchain model, the current most popular. Our experiments deliberately varied node numbers to explore how our method operates under diverse network configurations, yielding insights into scalability, reliability, and efficiency. All tests were conducted on Azure VMs (4 vCPUs, 16GB memory) running on Ubuntu 22.04, each hosting 4 containerized nodes, enabling simulation of real-world conditions and thorough analyses in a controlled environment.



 §.§ Detection and Elimination of Malicious Nodes

In our initial test, we evaluated MRL-PoS+'s ability to detect and remove malicious nodes using its penalty-reward system by simulating nodes engaging in various attacks, each specializing in a specific type. Figure 3 displays the results, showing the number of malicious (in red) and honest (in green) nodes selected as validators, aggregated every 10 rounds. Each simulation included 40 malicious and 60 honest nodes per attack type, maintaining consistent behavior over 100 rounds. This rigorous setup tests MRL-PoS+'s performance under challenging conditions, recognizing the difficulty of sustaining consistent malicious behavior due to nodes' limited stakes in real networks. The plots in Figure 3 consistently show a decline in malicious validators as rounds progress, attributed to honest nodes gaining reputation and rewards while malicious ones face penalties, resulting in stake loss, restrictions, and reputation decline. This trend persists across all attack types, showcasing MRL-PoS+'s effectiveness in detecting and removing malicious nodes in challenging scenarios, highlighting its relevance and resilience in securing PoS blockchains.











































 §.§ Learning Stages

The preceding experiment showcased an extreme scenario where all malicious nodes consistently behaved maliciously. However, in reality, nodes are expected to recover and return to honest participation. To illustrate this, we simulated a typical node scenario in MRL-PoS+. Figure <ref> illustrates the evolution of the reputation table and stake across six rounds.



In round 0, a node enters the network and participates in the initial validator selection, only to engage in malicious behavior upon selection, resulting in a penalty and its classification as an attacker node. This penalty also imposes limitations on the node's reputation table. Despite the penalty, the node persists in the selection process for three more rounds without success, with each round reducing the attack age. In round 3, the node is once again chosen as a validator and behaves honestly this time. Consequently, it regains some reputation in round 4 and sheds a restriction in the subsequent round. Continued honest behavior will lead to rewards and a positive reputation. However, any subsequent malicious actions will incur significant penalties, ultimately resulting in stake loss and exclusion from the selection process.































 §.§ Comparison with Traditional Approaches

While traditional PoS and DPoS <cit.> lack attack detection mechanisms, they seize the stake of nodes failing to process transactions correctly. In this experiment, we simulated PoS, DPoS, and MRL-PoS+ with 50 malicious nodes. We tracked how many rounds it took for these nodes to lose their stakes, considering a balance below 0 as elimination. Figure <ref> demonstrates MRL-PoS+'s significantly quicker elimination of malicious nodes compared to PoS and DPoS, which fail to eradicate them all. This discrepancy arises because not all malicious behaviors cause system failures, thus PoS and DPoS do not detect them and withhold incentives. Consequently, after 50 rounds, more than 10 malicious nodes persist in PoS and DPoS networks. Although DPoS slightly outperforms PoS, it remains ineffective in eliminating malicious nodes. During our experiments, we measured the average memory usage. In both cases, the average usage was  60% (9.7GB for PoS and 9.81GB for MRL-PoS+). Although MRL-PoS+ is based on PoS and adds some extra layers to it, it does not increase the overhead.















§ CONCLUSION


In this paper, we introduce MRL-PoS+, a novel consensus algorithm designed to enhance the security of PoS blockchains. Leveraging Multi-agent Reinforcement Learning techniques, MRL-PoS+ incorporates a penalty-reward mechanism to incentivize honest behavior and penalize malicious nodes. By providing empirical evidence of its efficacy, this paper contributes to the ongoing discourse on blockchain security and consensus algorithm design. Furthermore, the persuasiveness of MRL-PoS+ makes it a promising solution for addressing the security challenges inherent in PoS blockchains while ensuring the integrity and resilience of decentralized networks. Moving forward, future research may explore additional enhancements and optimizations to further strengthen MRL-PoS+ and adapt it to evolving security threats in the blockchain landscape.



IEEEtran




 Paper Title: Securing the Diagnosis of Medical Imaging: An In-depth Analysis of AI-Resistant Attacks.



Article Title]Securing the Diagnosis of Medical Imaging: An In-depth
Analysis of AI-Resistant Attacks













1]Angona Biswasangonabiswas28@gmail.com

[2]MD Abdullah Al Nasimnasim.abdullah@ieee.org
These authors contributed equally to this work.

3]Kishor Datta Guptakgupta@cau.edu
These authors contributed equally to this work.

4]Roy George george@cau.edu
These authors contributed equally to this work.

5]Abdur Rashid A.Rashid.653@westcliff.edu
These authors contributed equally to this work.

[1]Research and Development Department, Pioneer Alpha,  Dhaka, Bangladesh

[2]Research and Development Department, Pioneer Alpha,Dhaka,Bangladesh

[3, 4]Department of Computer and Information Science, Clark Atlanta University, Atlanta, GA, USA

[5]Master’s of Engineering Management (MSEM), Westcliff University, Irvine, California, USA







Machine learning (ML) is a rapidly developing area of medicine that uses significant resources to apply computer science and statistics to medical issues. ML's proponents laud its capacity to handle vast, complicated, and erratic medical data. It's common knowledge that attackers might cause misclassification by deliberately creating inputs for machine learning classifiers. Research on adversarial examples has been extensively conducted in the field of computer vision applications. Healthcare systems are thought to be highly difficult because of the security and life-or-death considerations they include, and performance accuracy is very important. Recent arguments have suggested that adversarial attacks could be made against medical image analysis (MedIA) technologies because of the accompanying technology infrastructure and powerful financial incentives. Since the diagnosis will be the basis for important decisions, it is essential to assess how strong medical DNN tasks are against adversarial attacks. Simple adversarial attacks have been taken into account in several earlier studies. However, DNNs are susceptible to more risky and realistic attacks. The present paper covers recent proposed adversarial attack strategies against DNNs for medical imaging as well as countermeasures. In this study, we review current techniques for adversarial imaging attacks, detections. It also encompasses various facets of these techniques and offers suggestions for the robustness of neural networks to be improved in the future.



















[
    [
    August 5, 2024
==================





§ INTRODUCTION




 §.§ Adversarial Attack


Adversarial attacks encompass a set of techniques aimed at manipulating machine learning models by introducing well-crafted, often imperceptible alterations to input data. These modifications aim to deceive the model, leading to misclassifications or inaccurate predictions <cit.>. Adversarial attacks have the potential to compromise machine learning systems' dependability and security, which is concerning for crucial applications such as cybersecurity, autonomous vehicles, and medical diagnosis. Of all the adversarial attack types, "White-box attacks" and "Black-box attacks" are the most common.

Ongoing research in machine learning security centers on the analysis of adversarial attacks and the development of defense strategies. Researchers continually explore novel attack tactics to identify potential weaknesses in AI systems while concurrently striving to construct robust models less susceptible to adversarial perturbations. Artificial intelligence (AI) modern technology is widely successful in fields like computer vision, natural language processing, and automated driving; however, its application in vital safety sectors is hindered by its susceptibility to adversarial attacks. Consequently, enhancing the resilience of AI systems against such attacks has become paramount for the progress of AI <cit.>.

The rapid advancement of AI technologies has found diverse applications in numerous domains, from machine translation, speech recognition, and object identification to more intricate tasks like drug composition analysis <cit.>. Noteworthy applications include brain circuit construction <cit.>, particle accelerator data analysis <cit.>, <cit.>, and DNA mutation impact analysis <cit.>. Since Szegedy et al.'s seminal work <cit.> highlighting neural networks' vulnerability to adversarial attacks, research on adversarial technologies for artificial intelligence has steadily expanded. New techniques for adversarial attacks and strategies for mitigating them are consistently emerging.

The term "adversarial attacks at the training stage" refers to actions taken by adversaries to manipulate the training dataset, input characteristics, or data labels during the training phase of the target model. This manipulation of the training dataset involves actions such as adding or deleting training data, as demonstrated by Barreno et al.'s approach <cit.>. During the testing phase, adversarial assaults are categorized into white-box attacks and black-box attacks <cit.>. In white-box situations, attackers have access to the parameters, techniques, and structure of the target model, enabling them to craft adversarial samples based on this information.



 §.§ The Adversary's Objective: Evasion attack versus poisoning assault


"Poisoning attacks" are assault methods that allow an attacker to add or change a large number of fake samples to the training set of a DNN algorithm. The trained classifier may perform poorly as a result of these bogus data. They may have poor accuracy <cit.> conversely make imprecise predictions on a few samples for analysis <cit.>.
The classifiers used in evasion attacks are fixed and often work well on safe testing samples. The adversaries are unable to reform the classifier's settings or parameters, but they do generate some fictitious instances that the classifier is unable to distinguish.



 §.§ Deep Neural Networks: Adversarial Attacks

Regarding medical image processing use cases such as diagnosing cancer and lesion identification, deep neural networks, also known as DNNs, are rapidly gaining popularity <cit.>. Nonetheless, a new study suggests that adversarial examples/attacks with tiny, barely noticeable changes can weaken medical deep learning systems. There are presently safety concerns associated with the usage of these devices in clinical settings <cit.>.  Deep neural networks (DNN) are increasingly well-liked and effective at a variety of machine learning requisitions. They have been employed with surprising effectiveness in a variety of recognition issues in the fields of pictures, graphs, text, and voice <cit.>. They are able to identify things in images with accuracy that is almost human <cit.>, <cit.>. Additionally, they are employed in speech recognition <cit.>, natural language processing <cit.>, and gaming <cit.>. 

   
Regarding the data set provided by MNIST, Biggio et al.<cit.> produce adversarial instances first, focusing on traditional machine learning classifiers like SVMs and 3-layer neural networks with full connectivity which is shown in Figure <ref>. In order to trick the classifier, it optimizes the discriminant function. As an illustration, consider a linear SVM classifier on the MNIST dataset. 



 §.§ Examining examples of opposition in the real world


The research <cit.> put decals to traffic signs that pose a serious threat to autonomous vehicles' sign recognition technology. These hostile objects are especially detrimental to deep learning systems since they can interfere directly with numerous real-world DNN usages, such facial recognition and autonomous cars. By assessing the adversarial images (FGSM, BIM) produced to see if they are "robust" against changes in natural circumstances (e.g., shifting viewpoint, illumination, etc.), the authors of the work <cit.> investigate the viability of creating tangible adversarial objects. Robust in this context means that even after transformation, the produced pictures are still antagonistic. The experiment's findings show that many of these adversarial examples—particularly those produced by FGSM—remain antagonistic to the classifier even after transformation. The findings imply that antagonistic real-world objects could trick the sensor in many situations.



 §.§ Medical Image Under Artificial Intelligence Attack


Researchers have access to strong models of developing science and technology thanks to deep learning. Convolutional neural networks, also known as CNNs, are among the most significant categories of deep learning (DL) frameworks for the analysis and processing of images because of their exceptional ability to learn valuable information.  The processing of the human organism utilizing different picture modalities for therapeutic, diagnostic, and health surveillance purposes is known as medical image analysis <cit.>. The advancement of computer vision through the use of deep neural networks addresses issues that were not well-solved by traditional image processing methods. Beinfeld et al. <cit.> asserted that $385 spent on medical imaging results in a savings of almost $3000. MRI, CT scans, ultrasound (US), and X-rays are the most frequently used image modalities.  Figure <ref> shows how adversarial assaults may be used to arbitrary alter diagnosis outcomes in three different medical picture datasets: Dermoscopy <cit.>, X-ray of the chest <cit.>, and Fundoscopy <cit.>. 

   


Hu et al. <cit.> assert that in order to develop more reliable models, it is crucial to investigate the reasons why adversarial cases emerge and to comprehend deep learning models better. Depending on the knowledge of the enemy, attacks can be categorized into three kinds.   The saliency (or attention) map of a picture that was input shows the regions that significantly change the model's output based on the gradients of the categorization loss with regard to the input <cit.>. We can see that some medical photos have highly concentrated regions that are noticeably larger. This could indicate that the rich biological textures included in medical photos occasionally entice the DNN model to focus more on aspects unrelated to the diagnosis. Tiny adjustments in these high-focus areas can significantly affect the model's output. 

Due to several factors, including High-Dimensional Data, High-Dimensional Data, Transferability of Adversarial Examples, Complex Decision Boundaries, Black-Box Attacks, Safety-Critical Applications, and Safety-Critical Applications, Medical Image Deep Neural Network (DNN) models can be relatively easier to attack when compared to some other domains. Despite these difficulties, experts in the field are actively attempting to create more reliable and safe DNN models for medical images. To make these models more resistant to hostile attacks, strategies like adversarial training, input preprocessing, and defensive distillation are being investigated. To further enhance the security and dependability of AI systems in healthcare applications, consistent evaluation frameworks for adversarial robustness in medical imaging must be developed.

Byra et al. <cit.> proposed an attack strategy on ultrasonography (US) imaging for liver fatty tissue. Radio-frequency signals are used to rebuild US pictures, and the reconstruction technique was subjected to a zeroth-order optimization attack <cit.>. The InceptionResNetV2 model was used in the studies, and the assault resulted in a 48% loss in model accuracy. Ozbulak et al. <cit.> proposed an attack specifically designed for medical picture segmentation called the adaptive segmentation masking attack (ASMA). The suggested attack offers significant intersection-over-union (IoU) degradation and produces nearly undetectable samples for most portions. Because the U-Net framework is among the most well-known models for clinical picture segmentation, they employed it in the trials. The datasets used were for segmentation of glaucoma optic disk <cit.> and ISIC skin lesion segmentation <cit.>.  Chen et al. published a method for fabricating hostile cases to thwart medical picture segmentation <cit.>. In order to simulate anatomical and intensity fluctuations, geometrical deformations are used to create the adversarial examples. By attempting to partition organs from abdominal CT scans using a U-Net model, they tested the effectiveness of these examples. With reference to the Dice score measure, they successfully reduced it significantly across all organs. The kidneys and pancreas, however, require a higher level of disturbance and are more arduous to assault than the liver as well as spleen.

   


Tian et al. <cit.> examined the phenomena of bias field, which can arise from incorrectly capturing a medical image and jeopardize a DNN's efficacy, as shown in Figure <ref>. Motivated by adversarial attacks, the authors created an adversarial-smooth bias discipline approach to trick a model. The ResNet50, MobileNet, among others and DenseNet121 models were employed to fine-tune the chest X-ray dataset that was used in their study. They looked at this attack's transferability and white-box attacks. Comparing the suggested attack to other cutting-edge white-box attacks, it exhibited a greater attack accuracy on transferability.

This review's major organization is as follows: We offer some significant adversarial notions on medical images in Section 2. Additionally, it provides a detailed overview of earlier research. We explore a few strategies in Section 3 with regard to the picture classification scenario. We utilize Section 4 to quickly summarize some findings from earlier research that tries to explain the phenomena of antagonistic examples on medical data. The overview is concluded in Section 5.



§  BACKGROUND LITERATURE REVIEW


Adversarial instances are intentionally created data inputs that are intended to degrade a machine learning model's performance. When researchers examined strategies used by spammers to evade spam filters in 2004, they unofficially coined the term "adversarial inputs" <cit.>. These misleading examples are usually produced by deliberately faking real data, such spam advertising messages, in order to trick the computer system that analyzes it. Alterations can be applied to text data, like spam, by introducing innocuous text or modifying frequently used phrases in malicious communications with synonyms. To fortify algorithms against adversarial attacks, researchers have explored various strategies, including training algorithms on adversarial samples and employing sophisticated data processing techniques to minimize the susceptibility to manipulation. The quest for fully robust models in machine learning aims to accelerate the development of algorithms capable of making decisions based on consistent explanations, with promising early efforts in this direction <cit.>.

In the realm of healthcare, medical claims codes play a crucial role in determining the amount spent on a patient visit post-payer approval. Payers often assess these claims using automated fraud detectors increasingly driven by machine learning. Historically, healthcare providers have shaped payer records of patient visits, including the associated codes, to influence the algorithmic outputs of payers <cit.>. Medical fraud, a market valued at $250 billion, exemplifies the extreme end of this strategic tailoring of patient presentations. While some practitioners may overtly fabricate medical claims, patient data falsification often takes more covert forms. For instance, consistently submitting codes for billing services identical to, but more expensive than, those actually provided is known as intentional upcoding. Here, Table <ref> shows past exclusive past research works where authors did some fantastic work for medical images to defend adversarial attacks. Different researcher proposed different frameworks to overcome this problem and to suggest a new thought for revolution. 



 §.§ Elements of a Cyber Adversary's Assault


   


In some instances, subtle adjustments to billing codes blur the boundary between fraudulent activities and conscientious best practices. One notable illustration may be found in the guidelines on the Endocrine Society website, which advise medical professionals not to bill for metabolic syndrome, or International Classification of Diseases (ICD) code 277.77, in situations of obesity. Coverage denial is likely to occur with this particular code/condition combo in Figure <ref>. The Society recommends an alternative approach, suggesting billing for the individual codes related to specific disorders comprising the metabolic syndrome, such as hypertension. 










 §.§ The components of an adversarial attack


   


Presenting a thorough overview of the various adversarial attack strategies and defense techniques is the aim of the study <cit.>. We first discuss the theoretical underpinnings, practices, and practical uses of adversarial attack strategies. Then, a brief discussion of the research on defense strategies obscuring the boundary of the large field follows. A few selected articles are published year-by-year from 2012 to 2021, as seen in Figure <ref>. This work aims to provide thorough taxonomies, evaluations of harmful assaults, and protections against the full DL pipeline. In this context, the numerous attack and defense tactics that have been developed over the previous two years have been categorized, with a focus on the clinical deep learning systems that are susceptible to adversarial attacks. 

Several defensive strategies have been put forth in an effort to neutralize possible threats. A popular method in natural imaging called adversarial training adds adversarial images to the training dataset in order to make Convolutional neural network, or CNN, models more resilient. Nevertheless, this method is not the best for datasets pertaining to medical imaging, since the addition of different adversarial images could considerably reduce classification accuracy. A robust detection technique for malicious images is presented in the work of Li et al. <cit.>, successfully defending against attacks on deep learning-driven medical image labeling systems. In a different study <cit.>, scientists compared CNN performance with Vision Transformers (ViTs) to determine how resilient CNNs are to different types of attacks in computational pathology. The authors developed robust neural network models, evaluating their efficacy against both white- and black-box attacks. The structures of attacks for both models were scrutinized, with an exploration of the underlying factors influencing their performance. The study's findings were validated through two clinically relevant classification tasks involving distinct patient groups. Preceding Ma et al.'s research <cit.>, ambiguity surrounded medical image adversarial attacks (AAs), limiting adversarial machine learning analyses to natural images. Unlike natural images, medical images may contain domain-specific elements, impacting medical deep-learning systems susceptible to AAs. AAs possess the potential to manipulate diagnoses and outcomes, emphasizing the critical need for a robust healthcare infrastructure to mitigate potential risks from malicious attacks.






§ MATERIALS AND METHODS



 §.§ Medical Adversarial Defensive Strategies

Various defense models, including techniques such as input denoising, input gradients regularization, and adversarial training, have been devised. However, recent attacks often manage to evade or partially circumvent these protective measures. This paper undertakes a comprehensive exploration of adversarial attack and defense methods within the domain of medical image analysis. It introduces a novel taxonomy based on the application context for a family of techniques, as outlined in Dong et al.'s work <cit.>. The author establishes a unified theoretical framework encompassing diverse adversarial attack and defense strategies tailored for medical imaging applications.

The primary emphasis in most research efforts on adversarial attacks in medical image processing is on the white-box scenario. These studies, characterized by a comprehensive understanding of medical Deep Neural Networks (DNNs), concentrate on the susceptibility of computer-aided diagnosis models across various medical imaging applications. In the execution of adversarial attacks, attackers may utilize the target diagnosis DNN as a locally deployed model. In addition to white-box adversarial attacks for medical classification tasks, scholars have delved into vulnerabilities associated with various medical imaging tasks. Notably, these publications focus predominantly on medical segmentation, as evident in works such as Chen et al.'s <cit.> and Ozbulak et al.'s <cit.>.

For natural photos, semi-white-box (Gray-box) attacks have been extensively researched <cit.>. However, there are only a few academic publications <cit.>, <cit.> that address this attack scenario for medical image processing. The semi-white-box adversarial approach typically consists of two stages: 1) To create adversarial instances against the target DNN model, the attacker trains a generative model. The attacker has complete access to the target model during training, including backward propagation gradients. 2) Instead of needing to know anything about the target model, as would be the case in a completely black-box scenario, the adversary generator can directly obtain adversarial examples against the target model with the input of authentic photos during the application stage.

Presently utilized white-box adversarial attacks often require multiple backward gradients of the target model. In simpler terms, attackers generate comparable adversarial samples by treating the target Deep Neural Network (DNN) as if it were a locally deployed model. However, due to its reliance on an in-depth understanding of the DNN model to execute an attack, the white-box approach may be unreliable in real-world scenarios. Conversely, the general black-box scenario may offer a more suitable environment for simulating real-world adversarial attacks, with numerous proposals focusing on investigating black-box assaults for natural images.

The outlined solutions usually involve multiple queries to the black-box model or depend on having a comprehensive understanding of the desired diagnosis model. However, in many real-world situations, attackers may lack direct access to the target diagnosis model. The restricted black-box (no-box) configuration, specifically representing the most challenging scenario for real-world adversarial assaults, may be more covertly dangerous even without querying the target black-box DNN. The transferability <cit.> of adversarial instances across different DNN models fundamentally determines the success of no-box attacks. For example, a no-box attacker can generate adversarial images based on a locally deployed surrogate model, transferring them directly to target medical diagnosis systems. Restricted black-box adversarial attacks are deemed more subtly perilous for natural vision tasks according to a renowned study.

Considering the substantial risks to the healthcare sector, numerous defense strategies have been proposed to counter medical adversarial attacks. In order to overcome the weaknesses shown by adversarial examples, a number of methods have been developed to provide trustworthy deep learning-based systems for natural pictures <cit.>. The development of accurate computer-aided diagnosis models for clinical applications not only contributes to trustworthy healthcare services for millions but also underscores the importance of investigating adversarially robust models within the realm of medical image analysis.

In their work <cit.>, the authors succinctly summarize studies on adversarial defense in the context of medical image analysis. Various attack methods in the realm of medical images serve as robustness evaluation criteria for adversarial defense, complementing the extensive range of adversarial attack methods mentioned. The minimal separation between adversarial examples and the model's decision boundary signifies the model's resistance to disturbances. The goal of adversarial training is to mitigate subsequent losses. 

    
Minimize the 


    L(f(x_adv, y) + λ· L(x, x_adv))


where L(f(x_adv, y) is the classification loss, L(x, x_adv) is the perturbation size, and λ is a trade-off hyperparameter. 



Transforming the input data into an alternative feature space that diminishes the impact of adversarial perturbations on the model. A technique aimed at enhancing resilience against adversarial attacks involves introducing noise to the features or intermediate representations that characterize the input data.




 §.§ Adversarial Training

Most medical adversarial defense techniques focus on using adversarial training to create reliable diagnoses systems. A significant fraction of the works among them go beyond current techniques for natural image adversarial training to tasks relating to medical classification <cit.>, <cit.>, Vatian et al. <cit.> looked into opposing examples for medical imaging and tried a number of defense strategies to oppose these nefarious representations.



 §.§ Adversarial Detection

Adversary detection seeks to identify adversary cases from input examples during the application stage, as opposed to developing robustness during the training stage of computer-aided diagnosis models.Many adversarial detection techniques have been put forth in the field of medical image analysis to stop additional misdiagnosis caused by adversarial samples <cit.>. In particular, it is possible to think about medical adversarial detection as an anomaly detection issue that can be resolved by combining explainability approaches <cit.>.



 §.§ Image-level Pre-processing

A clean image and the related adversarial perturbation make up an adversarial image in general. Meanwhile, it has been shown that DNNs can perform well on clean images while still being vulnerable to adversarial examples <cit.>. Denoising the adversarial example to remove the perturbation component can therefore help make the subsequent network diagnostic easier. Image-level pre-processing can be useful and secure in the context of biomedical image analysis because it does not require re-training or modifying medical models. 



 §.§ Improvement of Features


The difference in robustness between human and machine vision is attributed to adversarial examples linked to non-robust features extracted from specific patterns in the data distribution <cit.>. Consequently, enhancing feature representation is crucial for the development of robust inference systems. In this investigation, we characterize feature augmentation as the modification of architectures or mapping functions. Various techniques aimed at improving features have significantly enhanced the robustness of medical classification models <cit.>.



 §.§ Distillation of Knowledge

In the field of machine learning, knowledge distillation is a helpful technique for transferring learned information from a complex (teacher) model to a simple (student) model. Therefore, the situation where the network structures for the teacher and learners are identical is specifically referred to as self-distillation. Furthermore, a great deal of research has been done on the natural imaging domain using adversarial knowledge distillation, which moves the adversarial resilience from a heavy teacher model to a view student model <cit.>.



§ RESULT AND DISCUSSION


This study utilizes four publicly accessible benchmark datasets, as detailed in <cit.>, to investigate adversarial attack and defense in the context of medical image processing (illustrated in Figure <ref>). Firstly, the Messidor1 dataset comprises 1,200 eye fundus color images for detecting diabetic retinopathy across four classes based on retinopathy grade. Secondly, the International Skin Imaging Collaboration (ISIC) dataset consists of 2,750 dermoscopic images categorized into three classes for skin lesion classification and segmentation. Thirdly, the ChestX-ray 14 dataset comprises 112,120 frontal-view X-ray images representing 14 thorax diseases. Lastly, the COVID-19 database incorporates 21,165 chest X-ray images accompanied by segmentation-capable lung masks.

   


The study conducted by the authors <cit.> employs experiments to demonstrate the following: 1) The SSAT module significantly enhances the adversarial robustness of the model without compromising the classification accuracy of clean images. 2) The UAD module effectively identifies and excludes a majority of successful adversarial examples. 3) In comparison to other existing AI systems, their medical imaging AI solution (UAD + SSAT) minimizes the risk of adversarial attacks. The research utilizes a publicly available dataset of retinal OCT images, employing the "white-box" setting as the most challenging threat for assessing class prediction performance. Across all white-box attack conditions, the authors establish that SSAT consistently outperforms alternative baselines while maintaining comparable or superior performance for clean image classification. Furthermore, the authors demonstrate that the combined approach of UAD complementing SSAT results in the lowest adversarial risk according to the new measure presented. Regardless of the training techniques employed, systems based on UAD consistently exhibit reduced risks compared to those that do not incorporate this defense mechanism.

   
An AI-CAD model depicted in Figure <ref> was initially trained to manipulate diagnosis-sensitive elements within images, such as adding or removing malignant tissue. The model's effectiveness was then assessed using adversarial images produced by a GAN model. To determine how well human specialists could visually distinguish images produced by the GAN, a reader study was carried out.

In a study by Zhou et al. <cit.>, the responses of an AI-CAD model to adversarial attacks on GAN-generated mammography images were investigated. This involved introducing malignant tissue into healthy images and removing cancerous tissue from images affected by cancer. The study also evaluated the proficiency of experienced radiologists in visually identifying such adversarial images, both before and after instructional intervention. The study cohort, provided by the University of Pittsburgh Medical Center, included 1284 women, and 4346 mammography images were collected. Among these, 366 patients had biopsy-proven malignant breast cancer, while 918 patients had breast cancer assessed as benign (including benign signs).

Following training, the researchers assessed the model's classification accuracy concerning both the original genuine test data and its equivalent GAN-generated adversarial counterparts. Two GAN-trained U-Net23 models were employed to generate adversarial images, flipping labels in the test set. The high-resolution (1728 x 1408) photographs indicated the categorization impact of the AI-CAD model on the test data. The AI-CAD model achieved an AUC of 0.82 on the test set, consisting of 364 genuine negative samples and 74 real positive samples. The adversarial GAN-generated images (with flipped labels) in the test set yielded an AUC of 0.94. These AUC values suggested that the adversarial collection of images effectively deceived the AI-CAD model. Additionally, 59.5% (44 out of 74 cases) of genuine positive images with a classification accuracy threshold of 0.5 were correctly labeled as positive, whereas 95.5% (42 out of 44 cases) of adversarial samples created by GANs successfully tricked the system by being classified as negative. 

   


Figure <ref> displays the mean SSIM values for FGSM and PGD assaults across all photos. The Supplementary Material includes the SSIM data for every model. As can be shown, the effects of the identical disturbance applied to various imaging modalities on human visual perceptibility with the observed SSIM varies. The radiology images most clearly showed adversarial disturbances, with = 0.02 producing an already apparent, but very modest perturbation. At the same perturbation level for the ophthalmology and pathology photos, perturbations were nearly undetectable and became apparent with larger epsilon values. The authors examine black-box adversarial attacks on deep learning in healthcare imaging. <cit.>. They investigate three medical imaging sectors where deep learning algorithms are vulnerable. In all three datasets, perturbations calculated by FGSM showed lower SSIM than those calculated using PGD. This is a predicted outcome given that PGD optimizes perturbations based on their size and influence on model predictions.We opted to document attacks in our subsequent analyses employing 0.02, as this represented the maximum perturbation level that remained effective across all applications and attack strategies. Additionally, it exhibited greater transferability than an epsilon value of 0.01 in the majority of the examined applications.

   


Figure <ref> shows examples taken from the ophthalmology collection that demonstrate the transferability of assaults when the target and surrogate are randomly started and when both are pre-trained on ImageNet. Whether the target and surrogate models had the same architecture or a different one did not affect the findings. 



§ CONCLUSION

Diagnostic imaging AI systems based on computer vision are increasingly being used as models for classifying and segmenting diseases. The scaled-up use of medical imaging AI systems has given rise to serious safety issues because to DNNs' susceptibility to adversarial samples. Recently, a number of ways have been put out to increase the efficacy of medical image defense tactics. Although numerous protection strategies have been put forth, there are still reservations over the use of medical deep learning techniques. This outcome arises from certain constraints within medical imaging, including a scarcity of high-quality image datasets and labeled data in comparison to more abundant datasets available for natural images. The efficacy of adversarial defenses may not be universally applicable in this context. The findings underscore the importance of exercising caution in the design of Deep Neural Networks (DNNs) for medical imaging and their subsequent real-world applications.


























§ DECLARATIONS






 §.§ Funding
 A portion of the funding for this research comes from DOEd Grant P116Z220008 (1) and NSF Grant No. 2306109. The author(s) expresses all opinions, findings, and conclusions; the sponsor(s) does not necessarily agree with them.


 §.§ Conflict of interest
 The authors do not declare any conflicts of interest.



 §.§ Ethics approval
 There is no original research involving human subjects, animal subjects, or sensitive data in this review publication. 



 §.§ Consent to participate
 The studies and data included in this review were sourced from publicly available and previously published literature. 


 §.§ Consent for publication
 All authors listed on this review paper have reviewed and approved the final manuscript for submission to Machine Learning. Each author has contributed significantly to the research, writing, and revision of the paper. All authors have read and agreed to the content presented in this review, and their consent for publication is hereby provided.


 §.§ Availability of data and materials
 The data and materials used can be found in the references of this work.


 §.§ Code availability
 The review is based on a comprehensive analysis of publicly available literature, and any references to specific methodologies or algorithms are attributed to the respective original sources. We have strived to provide clear citations and references to enable readers to locate the original works for further examination.


 §.§ Authors' contributions

 Conceptualization: A.B., M.A.A.N.; Methodology: A.B., M.A.A.N.; Formal analysis and investigation: A.B., M.A.A.N.; Writing - original draft preparation: A.B., M.A.A.N., R.G., A.R.; Writing - review and editing: M.A.A.N., A.R.; Funding acquisition: K.D.G; Resources: R.G.; Supervision: K.D.G., R.G. All authors have read and agreed to the published version of the manuscript.

















 Paper Title: MAARS: Multi-Rate Attack-Aware Randomized Scheduling for Securing Real-time Systems.





























MAARS: Multi-Rate Attack-Aware Randomized Scheduling for Securing Real-time Systems
    Arkaprava Sain, Sunandan Adhikary, Ipsita Koley, Soumyajit Dey

Department of Computer Science and Engineering, Indian Institute of Technology, Kharagpur, India

    
=====================================================================================================================================================================



§ ABSTRACT 
    Modern Cyber-Physical Systems (CPSs) consist of numerous control units interconnected by communication networks. Each control unit executes multiple safety-critical and non-critical tasks in real-time. 
    
    Most of the safety-critical tasks are executed with a fixed sampling period to ensure deterministic timing behaviour that helps in its safety and performance analysis. 
    However, adversaries can exploit this deterministic behaviour of safety-critical tasks to launch inference-based-based attacks on them. This paper aims to prevent and minimize the possibility of such timing inference or schedule-based attacks to compromise the control units. This is done by switching between strategically chosen execution rates of the safety-critical control tasks such that their performance remains unhampered. Thereafter, we present a novel schedule vulnerability analysis methodology to switch between valid schedules generated for these multiple periodicities of the control tasks in run time. Utilizing these strategies, we introduce a novel Multi-Rate Attack-Aware Randomized Scheduling (MAARS) framework for preemptive fixed-priority schedulers that minimize the success rate of timing-inference-based attacks on safety-critical real-time systems. To our knowledge, this is the first work to propose a schedule randomization method with attack awareness that preserves both the control and scheduling aspects.
    
    
    The efficacy of the framework in terms of attack prevention is finally evaluated on several  automotive benchmarks in a Hardware-in-loop (HiL) environment.



Schedule-Based Attacks, Multi-Rate Control Execution, CPS Security




§ INTRODUCTION



Real-time systems (RTSs) in CPS domains like automotive, avionics, power grids, etc. are mostly safety-critical. 

The feature-rich implementation of such modern CPSs makes the system design mixed-critical, for which tasks with different criticality levels are executed in each processor <cit.>. 




The safe operation of these RTSs relies highly on the timely execution of the safety-critical control tasks. 



For achieving predictable execution behaviour, fixed priority based scheduling algorithms remain popular for running tasks on embedded control units in CPSs, with higher priority typically being assigned to hard real time safety-critical tasks for ensuring deadline satisfaction even under platform uncertainties like sensing/actuation/communication delays. Also, the deterministic nature of static priority scheduling enables the designers to analyse the worst-case response time (WCRT) of every safety-critical control loop and design delay-aware control strategies for these loops <cit.>.

However, this predictability of fixed-priority scheduling exposes the timing information of safety-critical task executions. The attacker first finds out the periodicity of a safety-critical task by observing its data communication in the network or by observing the changes in physical system states of the controlled plant  <cit.>. By compromising a relatively lower priority task (mostly non-safety-critical) that periodically executes immediately before or after this victim task, the attacker observes its execution timestamps for multiple hyper-periods. 


Since a static fixed-priority schedule is followed, the execution sequence repeats every hyper-period, which helps the attacker infer possible initial offsets of the safety-critical task and predict their future arrival times. 

The attacker can manipulate data inside the victim task's shared I/O device buffer within a time window around those inferred future arrival time instances <cit.>. We term such vulnerable time windows as attack effective window (AEW), a quantity which is implementation-dependent and can be measured experimentally by the attacker <cit.>.


Such schedule-based attacks (SBAs) can be categorised into the following four models depending on the timing relation between the victim's arrival and the attacker task's arrival within AEW <cit.>, i.e., Posterior Attack (attacker task executes after victim task), Anterior Attack (attacker task executes before the victim task), Concurrent Attack (attacker task executes while the victim task runs) and, Pincer Attack (combines both posterior and anterior attack). Since the non-critical attacker tasks are of lower priority, the corresponding jobs are mostly scheduled/completed after higher-priority victim task instances in case of nearby arrival times. This makes posterior attacks more prevalent compared to other attack models in embedded controllers <cit.>. In this paper, we consider the posterior attack model, analyse the drawbacks of state-of-the-art defence mechanisms, and discuss how the proposed defensive solution outdoes them.

Researchers have proposed several solutions to prevent and defend against such SBAs. Among these, schedule randomization-based defence approaches are widely explored. One of the initial works in this area is TaskShuffler <cit.>, which randomizes job execution by allowing lower-priority jobs in the ready queue to run before higher-priority jobs. 

In <cit.>, the authors discuss the following two slot-level randomization techniques to mitigate directed schedule-based attacks. (i) Slot-level online randomization of schedules, for selecting the next job at runtime meeting task deadline constraints and (ii) Offline schedule-diversiﬁcation, where offline precomputed schedules are stored beforehand, and their selection is made online. Such schedule randomization methods obfuscate the schedule information, which makes it difficult for the attacker to predict the future arrival times of the victim task.

However, randomizing without any insight into the attacker's strategy produces some vulnerable schedules during randomization, which can make a random attack attempt successful <cit.>. The authors in <cit.> devise a privacy-preserved strategy that samples a suitable execution offset for critical tasks from a Laplace distribution and switches between them to keep the execution sequences from getting exposed. However, they do not ensure the performance and schedulability criteria of the control tasks while doing so. In another line of work, the authors in <cit.> suggested a temporal protection mechanism for multi-core Linux platforms. In this method, all the non-safety critical tasks, mostly considered to be untrusted, are blocked within the AEW of all the safety-critical or trusted tasks, eliminating all schedule vulnerabilities. However, this significantly delays the execution of untrusted tasks in uniprocessor systems, and they frequently miss their deadlines. If the trusted or critical task utilization is considerably high, such a protection strategy can impose an unbounded delay on the execution of other tasks that hampers the overall system  performance <cit.>. In this work, we propose a schedule randomization policy that is attack-aware and ensures meeting the deadlines for all tasks in the system, by carefully balancing the trade-offs between performance and security. 



One of the crucial pieces of information for any schedule-based attack model is the future arrival instances of a critical task, which is derived by finding out its periodicity <cit.>. If the sampling period is dynamically switched, the attacker will find it difficult to infer the exact schedule even after doing schedule analysis. With this motivation, we utilise multi-rate controller scheduling strategy to ensure security in real-time task schedules by reducing its inferability,  i.e. identifying start times of victim task instances. 

As shown in the state-of-the-art research works, dynamic switching between a predefined set of sampling periods, coupled with optimally designed controllers, significantly reduces resource utilization, ensuring performance, compared to conventional fixed sampling rate static scheduling <cit.>. 

There exist state-of-the-art works that analyse the performance of such systems by analysing their Lyapunov stability <cit.>. Such works mostly analyse whether a Common Quadratic Lyapunov Function (CQLF) exists among the control loops with different sampling rates. The existence of such a function that abides by a given performance criteria guarantees stability when the system is subject to arbitrary or fast switching.



There are works that derive a Multiple Lyapunov functions (MLF)-based <cit.> or dwell time-based switching rules for slowly switching between different modes to ensure stability. Most of these research works focus on optimizing resources or control costs utilizing multi-rate switching strategies <cit.>. In this work, we intend to switch between different controller sampling rates in runtime to reduce the inferability and vulnerability of a deployed schedule towards posterior SBAs. To enable this arbitrary switching, we utilize the CQLF-based fast-switching rules that ensure stability.


We propose a novel  Multi-Rate Attack-Aware Schedule Randomization (MAARS) Framework to secure all the safety-critical control tasks that are scheduled to execute in a uni-processor embedded platform against schedule-based posterior attacks <cit.>. This essentially combines (I) an offline schedule diversification technique that chooses a set of schedules with minimum inferability/predictability for a schedule-based attacker and (II) an online schedule selection technique that suitably deploys schedules to minimize the vulnerability of an attack-affected safety-critical task. 










To the best of our knowledge, this is the first work that proposes an attack-aware schedule randomization policy. The following are the main contributions of this work:

1. We present a novel approach for judiciously choosing performance-aware sampling rates for a safety-critical control task such that the inferability of its future arrival instances are minimized, thereby effectively thwarting the leakage of critical task information (such as periodicity, initial task offset, etc.)
2. We propose a novel methodology that generates a set of schedules offline using slot-level randomization techniques for the chosen set of sampling rates for each control task. We also develop a runtime schedule selection algorithm that intelligently deploys schedules from this set by observing the posterior attacks' effect on a control loop 

such that the vulnerability of the corresponding control task is minimized in that schedule. 





3. We experimentally evaluate how well the proposed MAARS framework balances the trade-off between security and system performance compared to state-of-the-art techniques by applying it to a synthetic automotive task set.
We implement the same task set in a Hardware-in-Loop (HIL) testbed setup to judge the real-time applicability of the MAARS framework. 



§ SYSTEM MODEL




 §.§ Task and Scheduler Model

Let  Γ denote a set of N periodic tasks s.t. Γ = {τ_1, τ_2, ... τ_N  }, that are scheduled on a uniprocessor system by a static priority scheduler with fixed priority. Each task has three parameters (e_i, p_i, d_i), where e_i is the worst-case execution time (WCET), p_i is the sampling period, and d_i is the relative deadline. We assume d_i = p_i, ∀ i≤ N and the WCET of a task includes jitter and context-switch overheads. We consider a mixed-criticality system, i.e., it comprises control tasks of varying criticality levels in order of their priority. 

Following the work in <cit.>, we consider all the safety-critical control tasks as trusted and hard to compromise. Whereas other tasks are considered to be untrusted and can be compromised by a schedule-based attacker. 

Any safety-critical control task from the trusted set Γ_t
⊆Γ can be a victim to a schedule-based attack (SBA) launched by an adversary who has compromised a non-safety-critical task from the untrusted set Γ_u ⊆Γ. Due to the high criticality level, the trusted tasks are mostly assigned with higher priority than the untrusted tasks. Hence, their executions are not delayed by the non-critical tasks as they are promptly scheduled as they arrive by a fixed-priority preemptive scheduler. In this work, we consider these trusted safety-critical control tasks to be designed with different sampling rates, which changes their periodicities of execution. We denote the set of periodicities for i^th control task as P_i={p1_i,p2_i,⋯,pl_i}. We assume that the task set Γ is schedulable by a fixed-priority preemptive scheduler such that the worst-case response time (WCRT)  wcrt_i

for any task τ_i ∈Γ satisfies,


  wcrt_i = e_i + ∑_τ_j ∈ hp(τ_i)⌈r_i^k/p_j⌉ e_j ≤ d_i. 

Here r_i^0 = e_i, wcrt_i = r_i^k+1 = r_i^k for some k≥ 0 and for every τ_i∈Γ_t, p_i= minimum periodicity from the set P_i. This criteria ensures that the tasks are schedulable for their maximum execution rates and assuming no precedence constraints among the tasks,  valid schedules can be generated without any deadline misses. 





 §.§ Control Task Model


We express the plant model as a Linear time-invariant (LTI) system having dynamics as follows.

    ẋ= A_c x(t) + B_cu(t),  y(t)= Cx(t)+v(t)
    
    
       x̂[k + 1] = (A-LC) x̂ [k] + B u[k] + L y[k]
       u[k+1] = K x̂ [k+1]


Here, the vectors x ∈ℝ^n, x̂∈ℝ^n, y ∈ℝ^m, and u ∈ℝ^p describe the plant state, the estimated plant state, output and the control input, respectively. The matrices A_c, and B_c, are continuous time matrices that define the continuous-time state and input-to-state transition matrices, respectively. C is the output transition matrix. At the k^th sampling instance, the real-time is t = kh, where h is the sampling period. We can, therefore, define the discrete-time matrices corresponding to their continuous-time counterparts as A= e^A_c h,  B= ∫_0^h e^A_c tB_c · dt. K and L are Linear Quadratic Regulator (LQR) controller gain and Kalman estimator gain, respectively. Each control task samples the sensed plant measurement vector y at each sampling iteration, using which future states of the plant are estimated. Based on the estimated plant state, a control task calculates the required control input u to transmit it to the actuator to actuate and control the plant output.





  §.§.§ Closed Loop Under Sampling Period Variation


As discussed above, the discrete-time system characteristic matrices depend on the sampling period h. Since the LQR gain is designed to minimize a user-defined quadratic cost function, which is sampled once in every h time interval, the LQR control gain is also dependent on the sampling period. So is the estimator Kalman gain L, as it is also designed to minimize the variance of the estimation error, i.e., the difference between the estimated plant state x̂ and actual plant state x at every sampling iteration.

Therefore, the discrete-time matrices A, B, K, L in Eq. <ref> can be replaced with their sampling period-dependent versions, i.e., A_h,B_h,K_h,L_h. By discretizing the continuous-time plant equation in Eq. <ref> we get x[k+1] = A_hx[k] + B_hu[k]. Since we intend to capture the change in the progression of the control loop w.r.t. the change in its sampling period,

we define an extended augmented state vector of the control task as X = [x^T, x̂^T]^T that captures both plant and estimator/controller dynamics. By replacing u with -K_hx̂ and y with Cx, the overall closed-loop system evolves as follows.


    X[k + 1] = 𝔸_h X[k], 𝔸_h = [             A_h         -B_hK_h;            L_hC A_h-L_hC-B_hK_h ]

For two different sampling periods p1 and p2, we denote the discrete-time augmented system matrices by 𝔸_p1 and 𝔸_p2. Our methodology uses this augmented system representation in Eq. <ref> to ensure a desired performance while varying periodicities of control tasks for security.





  §.§.§ Residue-based Anomaly Detector


Each of the safety-critical control loops is equipped with a residue-based detector that observes changes in the system residue (i.e., the difference between the sensed and estimated outputs) for anomaly detection. We consider an integrated implementation of controller and detector functionalities as part of the control task to avoid data manipulation. We use a χ^2-based detector in this work. The χ^2 detector utilises a normalized quadratic function of the residual to amplify and easily detect minute variations in system residue. For system residue res[k] at k^th sampling iteration, its chi-square measure is z[k] = res[k]^T Σ_res^-1 res[k] where Σ_res is the variance of system residue. Considering the measurement noise to be a zero-mean Gaussian noise, res[k] ∼𝒩(0,Σ_res)⇒ z[k]∼χ^2(m,2m), where m is the number of output measurements, considered as the degree of freedom of the χ^2 distribution. We employ a windowed chi-square detector that compares the average value of the chi-square statistic of system residue over a pre-defined time window (N), i.e., g[k]=1/N∑_i=0^N-1z[k-i] and compares it with a pre-defined threshold Th. The threshold is calculated to maintain a desired false alarm rate <cit.>. The chi-squared detector raises the alarm, denoting an attack attempt on a certain closed loop when g[k]> Th at any k^th sampling period of that closed loop.



























§ THREAT MODEL


In this section, we discuss in-depth how we model the schedule-based attacker and explain our assumptions about the attacker's objectives and capabilities. The objective of the attacker is to utilize the timing information exposed by the processor-level task execution schedules to compromise tasks with higher criticality. 




Since tasks with higher criticality are mostly developed by OEMs or trusted vendors and go through thorough security scans and functionality checks, they are usually very difficult to compromise <cit.>. Hence, an attacker is more likely to attempt compromising less critical tasks, that are often assigned lower priorities. We denote such compromisable tasks as untrusted tasks. Since these untrusted tasks are part of the software that comes from a range of third-party vendors<cit.>, compromising them using a man-in-the-middle-type attack is possible. As mentioned in Sec. <ref>, depending on the timing relationship between the victim and attacker task, SBAs can be categorised into 4 types <cit.>. 

(1) Anterior Attack: An SBA launched before the victim task starts its execution is known as an anterior attack.  e.g., false-data injection type attacks <cit.>. (2) Posterior Attack: An SBA launched after the victim task completes its execution is known as a posterior attack, e.g., replay attack, zero-dynamic attack etc. <cit.> (3) Pincer attack model: Attack is executed both before and after the victim task <cit.>. (4) Concurrent attack: Attack is performed while the victim control task is running and can be launched by executing between the execution window of the victim task’s job <cit.>.

Embedded controllers are mostly scheduled with fixed-priority preemptive schedulers, which schedule the lower-priority tasks after the higher-priority tasks. 

Since untrusted tasks are mostly of lower priority, CPSs with embedded control software systems are more vulnerable to posterior attacks <cit.>. 

This work considers a posterior SBA model and provides a mitigation framework against it. The following are assumptions regarding the attacker's ability, objectives, and attack methods.



  §.§.§ Attacker's Capabilities

We make the following assumptions about the attacker's capabilities: (1) The scheduler is trustworthy and can not be compromised by the attacker. Consequently, the attacker lacks direct control over task execution and is restricted to leveraging compromised tasks only when the scheduler permits.

[8]l0.42
    


    < g r a p h i c s >



   Attack Effective Window
   
        


(2)  The adversary knows the scheduling policy used, and it can compromise one task among all the lower-priority untrusted tasks (Γ_u). This enables the attacker to observe executions of this compromised untrusted task for several hyper-periods to find out partial information about its intended victim control task(∈Γ_t). (3) The attacker can partially deduce the sampling rates of any control task (∈Γ_t) either by observing the message data packets in the network or physical system states.

(4) Attacker can modify the data written by the victim control task inside an I/O buffer or cache only if it executes within an Attack Effective Window <cit.>. AEW is the specific time interval within which the attacker task must be executed for successful data manipulation. This duration is completely implementation-dependent and can be derived by experiments <cit.>. The attacker must tamper the data in the I/O buffer before the transmission task reads and transmits it for actuation. Hence, AEW is the duration after which the transmission task transmits the control program-generated actuation data and is scheduled to execute with the same periodicity as the control task. In Fig. <ref>, we illustrate this where the AEW of the control task has been shaded in red. AEW starts when the control task (in blue shade) finishes execution, writes its data to the buffer and ends once the transmission task (in green stripe) starts.

We denote the AEW length for a control task τ_i ∈Γ_t by Ω_i.




  §.§.§ Attack Execution
 

We assume an attacker's objective is to manipulate control inputs by performing a schedule-based posterior attack on control tasks. To implement such an attack, the attacker must infer the exact future arrival instances of the victim task. Having compromised a lower-priority untrusted task, the attacker can log that task's start and end time as part of the overall schedule. Utilising this information, it attempts to predict the start and end times of other trusted tasks in the schedule. The attacker can achieve this by using a schedule ladder-based analysis proposed in the work <cit.>. We briefly discuss how the schedule ladder is constructed, and the attacker uses it to derive the future arrival instances of the victim task.
[5]l0.30

font=scriptsize
    
        Tasks     P     e

        τ_1     {4,5}     1

        τ_2     4     1

        τ_3     5     2

        
    
    Example Task-Set 1









A schedule ladder represents a task schedule arranged vertically with adjacent timelines of equal row size. Each column in the ladder represents an atomic observation period, or unit time, of real-time duration δ. The attacker takes the row size of the ladder as equal to the victim task's sampling period measured in time units of size δ. Hence, for a given victim task τ_i executing with a sampling rate p_i time units, the row length of the ladder becomes δ p_i. The motive behind taking such a row length is that the victim task will arrive once in every row of the ladder and will occupy δ e_i columns. Moreover, all arrivals of τ_i will be located in the same column. We illustrate a schedule ladder diagram for the task set in Tab. <ref> in  Fig. <ref>. The row length of the ladder is 4 since τ_1(victim) sampling rate = 4. The first row represents the timeline [0,4], second row [4,8] and so on. The blue arrow shows the arrival column of the victim task that arrives once in every row and occupies column 1 of the ladder.  

While performing an analysis, the starting point of analysis (observation) of the top row can be arbitrarily chosen by the attacker, which is decided by the start of its observation window. The attacker starts observing its own start and end times using the compromised untrusted task. Note the attacker task is a lower priority untrusted task (w.r.t victim). Hence, if the victim task arrives in the same column, it preempts the attacker task. This is shown in Fig. <ref>, where the attacker task τ_3 arrives at instances t=0,20 in the first column (since its sampling rate is 5) but gets preempted by τ_1. In column 2 it gets preempted by τ_2. Therefore, it always executes in the third and fourth column of the ladder.

[13]l0.41





    < g r a p h i c s >

 Ladder diagram of Task-Set-1 illustrated along with their corresponding AEI and AAI.






After the end of observation, the attacker finds out Attack Arrival Instances (AAI). The AAI is the set of ladder columns where the attacker task has arrived at least once during its entire observation window. For e.g. in Fig. <ref>, the AAI is [0,4], shown below the ladder diagram. The set of ladder columns where the attacker task has been executed at least once during its entire observation window are termed as Attack Execution Instances (AEI). Note that the ladder columns where the attacker task arrives (∈ AAI) but never gets executed (∉ AEI since it is preempted by higher priority victim tasks) are the possible candidates for the columns where the victim task arrives. Hence, the attacker easily infers these ladder columns, which are in AAIs but not AEIs, as the victim task's arrival column (offset), predicting them as the victim's future arrival instances. We illustrated this in Fig. <ref> for task-set Tab. <ref>, where using τ_2 as an attacker task results in AAI=4 and AEI=2. One immediate conclusion we draw from this discussion is that a longer execution time for the attacker task make it easier for an attacker to infer the arrival column of the victim task. This is because the attacker task execution will span a greater proportion of ladder columns, making it easier to guess the correct victim task arrival column.




Consider the task set in Tab. <ref>, where the trusted control task set Γ_t ={τ_1, τ_2} and the untrusted task set Γ_u ={τ_3} (Fig.<ref>). After determining the arrival time of the victim task τ_1, the attacker uses untrusted task τ_3 to launch a posterior SBA on τ_1. We assume the AEW of τ_1 is Ω_1 = 1 time unit (highlighted in the shaded region of the timeframe). Therefore, for a successful SBA, τ_3 must execute within this timeframe after τ_1 finishes executing. Using this threat model, we demonstrate how state-of-the-art secure scheduling policies can mitigate such attacks and discuss their shortcomings in balancing performance and security, which motivates our novel methodology.














§ MOTIVATING EXAMPLE





We motivate our proposed methodology with an example by discussing how (i) the conventional protection window-based secure scheduling methods, such as <cit.>, fail to ensure system performance, optimal resource utilisation, etc., and (ii) the randomization-based scheduling methods, such as <cit.>, fail to guarantee security against SBAs. This, in turn, motivates our methodology, which utilises variable sampling period-based implementation of control tasks for balancing the trade-offs between security and performance without compromising the schedulability of the task set.
[5]l0.38

font=scriptsize
    

    
        Tasks     P     e

        τ_1     {2,3}     1

        τ_2     4     1

        τ_3     4     1

        
    
    Example Task Set-2
    


We refer the task set in Tab. <ref>, where the trusted control tasks Γ_t ={τ_1, τ_2} and the untrusted tasks Γ_u ={τ_3}. The control task τ_3 executes at two different sampling rates 2 and 3 time units (δ time unit=10 ms), respectively. We assume the attacker has compromised the untrusted task τ_3 to launch a posterior attack on the victim task τ_1. The AEW duration for τ_1 is Ω_1=1 time unit. 

[7]l0.55



    < g r a p h i c s >


AEW-based scheduling of Taskset-2





Protection window-based approaches such as in <cit.> propose restricting the execution of untrusted tasks' job instances within the AEW of all the trusted tasks. 

Fig. <ref> demonstrates how a task schedule generated following <cit.> (for the task set in Tab. <ref>) leads to deadline misses of untrusted tasks and wastage of CPU resources. The first job of τ_1 executes at [0,1], In the next interval at [1,2], job instance of τ_2 executes. In [2,3], second job of task τ_1 arrives and executes. However, the job instance of τ_3 will not be allowed to execute at [3,4] and hence will miss its deadline, even though the CPU remains idle. This leads to inefficient CPU utilisation.
On the other hand, the schedule randomization methods are 

more suitable since they make sure every task meets its deadlines (irrespective of trust). However, the state-of-the-art randomization policies do not consider the attacker's model into consideration and hence are often more vulnerable to SBAs while randomly changing the schedules <cit.>.

Earlier works <cit.> have demonstrated that selecting schedules using such attack-unaware randomization policies can still lead to successful attacks. 



[12]l0.6




    < g r a p h i c s >

Feasible Schedules Generated using Task-Set 2 (Considering p_1=2)






For example, if we apply such a schedule randomization method <cit.> on the same task set for a single sampling rate of p_1=2, for control task τ_1, we can generate a total of 8 unique feasible task schedules of hyper-period length of 4 time units (see Fig. <ref>). But among them, 4 schedules s_4,s_5,s_6 and s_7 are vulnerable for τ_1, since instances of τ_3 appears within the AEW of the victim task τ_1. However, if we use two different sampling periods for the control task τ_1,  P_1={2,3}, we can form two possible task specifications Γ_1 and Γ_2 with unique sets of sampling period, i.e., {p_1=2,p_2=4,p_3=4} and {p_1=3,p_2=4,p_3=4}. For the task specification Γ_2, we can generate 36 unique and feasible schedules (using TaskShuffler) with a hyper-period length of 12 time units. By analysing the vulnerability of τ_1 for all 36 schedules, 12 schedules are found to be safe (no posterior attackable instances within AEW) for victim task τ_1. Thus, multi-rate execution enables the generation of a higher number of randomized schedules with minimal vulnerability.


[14]l0.36
font=scriptsize
    


    < g r a p h i c s >

     Schedule Ladder Diagram for Task Set-1
    


Moreover, multi-rate task execution also hampers the analysis of the attacker task. We refer to the Example Task Set-1. After corrupting an untrusted task τ_3, the attacker observes its execution using a schedule ladder diagram. In the case of a single sampling rate, the attacker can figure out the future arrival instances of τ_1. Since the attacker never executes in the first column, i.e., within [0,1] time units, hence the attacker infers that the victim task must arrive in the second column, i.e., within [0,1] time units. This has been illustrated in case-1 of Fig. <ref>. However, due to a change in the victim task's (τ_1) sampling rate after one t=15 (sampling rate changes to 5 from 3), the attacker can execute in all three columns, i.e., within [0,1],[1,2] and [2,3] time unit column. This has been illustrated in Fig. <ref>. Therefore, the attacker cannot infer the exact column where τ_1 arrives from the schedule analysis. 

From these examples, we draw a conclusion that utilising multi-rate for trusted tasks along with imparting attack awareness to the task schedules (w.r.t trusted tasks) can 
hamper a schedule-based posterior attacker's 

 success rate. 




§ PROPOSED METHODOLOGY




In this section, we present the methodology for generating multi-rate attack-aware randomized schedules. 








A step-wise overview of the MAARS methodology is shown in Fig. <ref>. During the design time, our proposed framework takes the following as inputs: (i) a task set Γ consisting of both trusted and untrusted set of tasks, Γ_t,Γ_u (⊆Γ) respectively, (ii) trusted task specifications, i.e., criticality level/ priorities c_i, a controllable, schedulable set of periodicities P_i, WCRT e_i of each trusted task τ_i∈Γ_t, (iii) trusted task specifications, i.e., criticality level or priority c_j, periodicity p_j, WCRT e_j of each untrusted task τ_j∈Γ_u, (iv) physical system specifications corresponding to each control task τ_i∈Γ_t, its performance specification, (v) other tunable design parameters related to MAARS, like Tolerable Attack Probability, Maximum Schedule Vulnerability Index, etc. to constrain the vulnerability of MAARS as defined in Sec. <ref>




As motivated in Sec. <ref> switching between the allowable set of sampling periods P_i reduces the determinism (explained as inferability in Sec. <ref>) in the real-time task execution pattern. The first step in our methodology is to prune a set of periodicities P_i for a given critical/trusted task such that the system, in closed loop with this control task, delivers its desired performance (see Fig. <ref>).

In the second step, we intelligently select a subset of secure sampling periods from this performance-aware set of periodicities, such that the attacker's schedule analysis is maximally hampered. We theoretically prove that the devised secure periodicity selection policy maximizes the non-determinism in the execution pattern of the victim control task, by preventing the attacker from inferring critical task parameters. The third step uses these performance-aware and secure set of sampling rates to generate all possible valid task schedules by following the schedule randomization strategy given in <cit.>.

These valid schedules are 
 generated by ensuring that no task misses their deadlines while switching between the periodicities. In the fourth step, we analyse the probability of posterior schedule-based attack (SBA) on each victim task for each schedule. By quantifying the vulnerability of each schedule, we rank and suitably store the schedules for using them in runtime to prevent posterior SBAs.

During run-time, MAARS deploys a secure schedule selection algorithm. As discussed earlier, each control task is implemented with a windowed χ^2-detector that flags any attack attempt on its corresponding closed-loop (box with black-dotted outline in Fig. <ref>, refer to Sec. <ref>). Under no attack, MAARS randomly selects and deploys different schedules from a designer-selected set of valid schedules with minimum vulnerability. We term this mode as Normal Mode (green-dotted outlined box in Fig. <ref>). MAARS switches to an Alert Mode when the detector flags an attack on a certain victim task τ_i∈Γ_t(red-outlined box in Fig. <ref>). In this mode, the schedule selection algorithm switches to the least vulnerable schedule for victim task τ_i to ensure minimum posterior attack success. In the following sections, we discuss each design time step and run-time algorithm in detail.







 §.§ Performance-aware multi-rate Period Selection








To ensure a desired performance while arbitrarily switching between the closed loops with a schedulable sampling periods to reduce the inferability of victim task parameters, we must constrain our switching choices.

Let us consider, for the augmented closed loop system (as mentioned in Eq. <ref>) corresponding to a control task τ_i∈Γ_t, this sampling period switching signal is σ_i:ℕ↦ P_i, where P_i={p1_i,p2_i,⋯,pn_i} represents the chosen set of non-zero sampling periods. In presence of this switching signal, we can express Eq. <ref> for the control task τ_i as

 
    X^(i)(k + 1) = 𝔸^(i)_σ_i(k)(X^(i)(k))

 
the switching signal σ_i(k)=pj signifies that at k^th sampling instance, this augmented system switches to the controller designed with sampling period pj. For a switched system like Eq. <ref>, each such switchable closed loop systems are considered as its subsystems. As stated in <cit.>, the following condition must be satisfied for maintaining a desired performance while arbitrarily switching between the closed loop subsystems.


    In the case of a switched system like Eq. <ref>, in order to maintain a desired global uniform exponential stability (GUES) decay rate of γ while arbitrarily switching among different sub-systems, there must exist a common Lyapunov function (CLF) for all its arbitrarily switchable subsystems that satisfy the following set of equations
    
    
    κ_1(‖ X^(i)[k]‖)≤ V_i(X^(i)[k])≤κ_2(‖ X^(i)[k]‖)
        V_i(X^(i)[k])≤α_i V_i((X^(i)[k]), for α_i=g(𝔸^(i)_pj) s.t. σ_i(k)=pj


Here, κ_1,κ_2 are class 𝒦_∞ functions, and g:{𝔸^(i)_p1,⋯, 𝔸^(i)_pn}↦ ℝ∖{0}. Global uniform exponential stability (GUES) with a decay rate γ<0 signifies that at k^th sampling iteration ‖ X(k)‖≤ M e^γ k ‖ x[0]‖, where ‖.‖ is vector norm and M >0. We consider a common quadratic Lyapunov function (CQLF) V_i(X^(i))=X^(i)𝒫^(i)X^(i)^T. We validate the existence of this CQLF for all switchable closed loops by solving the following linear matrix inequalities (LMIs) and finding out a positive definite matrix 𝒫^(i)>0 such that for given α_j, 𝔸^(i)_pj^T𝒫^(i)𝔸^(i)_pj-𝒫^(i)≤α_j𝒫^(i) ∀ pj∈ P_i. Given a large set of periodicities, we only consider the set of sampling periods for which their corresponding closed loops have a CQLF that follows Claim <ref> for a given performance criteria. This ensures that the desired performance is guaranteed while we devise an arbitrary switching strategy between these closed loops to minimize the vulnerabilities of the task execution schedules against posterior SBAs.



 §.§ Secure Sampling Period Selection and Schedule Generation


We devise a novel periodicity selection policy to further prune the set of performance-aware sampling periodicities for a victim control task to maximally reduce the leakage of its critical task parameters (eg,. arrival times, execution offsets).

As discussed earlier, the attacker is capable of figuring out the victim task's sampling rate by observing the network traffic or characteristics of the physical plant <cit.>. 

The attacker chooses the smallest periodicity of the victim task to construct its schedule ladder as the smallest row length of the ladder increases the cardinality of Attack Arrival Instances (AAI) to infer the correct future arrival instances of the victim (refer Fig. <ref>). 

As discussed earlier, an attacker with a longer execution time is a more suitable choice since it increases the chance of getting preempted by a victim instance and, thereby, increasing 
the cardinality of Attack Execution Instances (AEI). 


 

We defend against such an attacker by suitably choosing a set of sampling periods for the higher-priority victim tasks so that the chances of preemption are reduced (recall that an attacker infers the timing information of a victim task based on its preemption by the victim). To quantify the inferability of victim task arrivals, considering an attacker task in a schedule, we introduce a metric inferability ratio that represents the number of preemptions by the victim task. 
 










(Inferability Ratio)
    The Inferability Ratio (IR) of a schedule s for a victim task, τ_i having minimum sampling rate p_i and an attacker task τ_j, is defined as the ratio between the total number of columns in the schedule ladder where the attacker task τ_j arrives and gets executed
    
    and the total number of columns where the attacker task τ_j arrives. Mathematically,  Inferability Ratio IR = | AEI|%| AAI |/| AAI|, where |·| denotes cardinality of a set. The numerator uses the modulo operation to reset the numerator when the attacker executes wherever it arrives, indicating no inferability due to no preemptions by higher-priority tasks.
 
If an attacker task is executed in more number of columns in a schedule ladder designed for a victim task, the IR of that schedule, w.r.t. the victim, and its attacker choice is high. This limits the number of columns where the attacker task arrives but does not get executed due to preemption by higher-priority victim tasks. Therefore, it is easier for the attacker to guess the arrival column of the victim task. Intelligent selection of sampling rates can reduce the IR by reducing the number of columns where the attacker task is preempted by the victim task, i.e., the attacker arrives but does not execute. 

We claim the following to ensure reduced preemption of the attacker task by suitable selection of victim periodicities: 














Consider an attacker task τ_j ∈Γ_u and a victim control task τ_i ∈Γ_t. For τ_i, let the set of periodicities allowed be P_i with minimum sampling period p_i = min{p | p ∈ P_i }. In this scenario, the generated fixed-priority preemptive schedules shall have the least Inferability Ratio (IR) if for each p' ∈ P_i ∖{p_i}: p' = n p_i + e_j, n ∈ℕ^+.


 We consider that an attacker constructs the schedule ladder with a row length equal to the minimum sampling period p_i of the victim task τ_i∈Γ_t and observes the AAI and AEI of an attacker task τ_j∈Γ_u. 

The duration between the attacker task's arrival 

and the subsequent victim task's arrival 

repeats after every LCM(p_i,p_j) unit of time. Note that the e_j time units

are covered by every execution instance of the attacker task. As long as this 

the duration between the attacker's and the subsequent victim's arrival is smaller than e_j 

time units, the attacker task will be preempted by the victim task, and the same preemption will repeat 

after every multiple of LCM(p_i,p_j) in the timeline. 


Now, let us consider that at k-th instance of the victim task, there is a predicted chance of preemption while the victim is running with some np_i periodicity where n, k∈ℕ^+. Therefore, this k-th victim instance, that arrives at knp_i time  may preempt the ⌊knp_i/p_j⌋-th 

instance of the attacker task. Therefore, for successful preemption, the k-th victim task instance must arrive before the execution of  ⌊knp_i/p_j⌋-th attacker instance is finished, i.e., ⌊knp_i/p_j⌋ p_j+e_j > knp_i

Now, consider changing the periodicity of the victim task to some p^'_i, to avoid this preemption. Therefore p^'_i must ensure that the 

k-th victim instance arrives after the ⌊kp^'_i/p_j⌋-th attacker instance finishes, i.e., ⌊kp^'_i/p_j⌋ p_j+e_j ≤ kp^'_i. By substituting p^'_i with np_i+e_j, this equation becomes as follows.

    k(np_i+e_j)    ≥⌊k(np_i+e_j)/p_j⌋ p_j+e_j = ⌊k(np_i)/p_j+k(e_j)/p_j⌋ p_j+e_j
    
       ≥⌊k(np_i)/p_j+ ⌊ke_j/p_j⌋ + ke_j % p_j⌋ p_j+e_j, a/b = ⌊a/b⌋ + a%b, 
       ≥⌊k(np_i)/p_j⌋ p_j+ k e_j + ke_j % p_j+e_j,  since k, e_j, p_j ∈ℕ^+
       ≥ knp_i-knp_i%p_j+ke_j-ke_j%p_j
       ≥ k(np_i+e_j)-k(np_i%p_j-e_j%p_j)




Therefore, the (LCM(p_i,p_j)/p_i-1)-th instance of the victim's arrival must be shifted by k= e_j number of columns to avoid this preemption. Changing the periodicity of the victim task to p^'_i=np_i+k after (LCM(p_i,p_j)/p_i-1)-th execution achieves this shift avoiding preemption. 
Guiding Example: In case-2 of Fig. <ref> the sampling period of the victim task τ_1(execution marked with blue) is p_1=3 (execution time e_1=1), and the attacker task τ_3 (execution marked with red) is p_3=5 (e_3=2, see the task set in Tab. <ref>). The arrival distance between the victim and the attacker repeats after LCM(3,5)= 15 time units. The victim preempts (preemptions marked with red stripes) the attacker at  3 and 12 unit time at the 1st column and is expected to preempt again in the same column at 18, 27 unit time. This is because the attacker task τ_3 shifts by p_3%p_1=5%3 columns in every consecutive arrival, but the victim task τ_1 always arrives at the 1st column.

Since the schedule ladder has a row length, same as the minimum periodicity p_i of victim task τ_i, every other arrival instance of victim task with p^'_i periodicity will now shift by p^'_i% p_i or k≥ e_j columns. We know that the attacker task shifts p_j% p_i columns in every other arrival instance. To avoid preemptions in the following arrival instances, the arrival of the victim task must shift as much as the attacker's arrival shifts, i.e., k= p_j% p_i. 


This always holds true since k(np_i%p_j-e_j%p_j)>0, because for τ_j to be schedulable p_j>e_j. Note that, e_j, p_j are considered as whole numbers since we express them in multiples of δ, and the analysis assumes δ as the notion of unit time. Therefore, this choice of periodicity p^'_i = np_i+e_j ensures no preemption of the attacker by any k-th victim instance. This, in turn, proves the claim that scheduling the victim with a set of periodicities P_i such that ∀ p^'∈ P_i{p_i}, p^'_i= np_i+e_j ensures the least IR for an attacker task τ_j∈Γ_u in any execution schedule.





The periodicity selection criteria used in Claim <ref> heavily restricts the periodicity choices since higher values of n∈ℕ^+ are often not supported by Claim <ref>. We update the criteria as follows in order to get multiple periodicity choices for a victim task with reduced IR. 


For a victim task τ_i, let the set of periodicities allowed be P_i with minimum sampling period p_i = min{p | p ∈ P_i }. In this scenario, the generated fixed-priority preemptive schedules shall have a reduced Inferability Ratio if for each p' ∈ P_i ∖{p_i}: p' = n p_i + k', n ∈ℕ^+, such that k'∈ [e_j,p_i-1] considering an attacker task τ_j with periodicity p_j and execution time e_j. Since, k'≥ e_j, it ensures zero preemption against any attacker, i.e., IR=0. Since, k≤ (p_i-1), the predicted preemption after every LCM(p_i,p_j) time is not repeated and delayed. This clearly reduces the cardinality of AEI, ensuring the reduction in IR compared to the situation when the victim task was scheduled with p_i.



We thus prune the performance-aware set of periodicities P_i to make it security-aware. For each victim task τ_i, we eliminate the periodicities from the set P_i that do not satisfy the criteria given in Remark <ref>. 





















Once all the possible sets of P_i for each victim control task τ_i are pruned, we utilise TaskShuffler  <cit.>, a schedule randomization technique. 

As defined in <cit.>


    the worst-case maximum inversion budget (WMIB) of task τ_i, denoted by WMIB_i, is the maximum amount of time for which all tasks with priority lower than task τ_i (denoted by lp(τ_i)) are allowed to execute while an instance of τ_i is still in the ready queue, while assured of meeting it's deadline in worst-case scenario. 
    It is calculated by,
    
    
    WMIB_i = d_i - [ e_i+ ∑_τ_j ϵ hp(τ_i)( ⌈d_i/p_j⌉  +1 ) e_j].





This randomization methodology shuffles job instances while ensuring that each job deadline is met by constraining unbounded priority inversions within specified limits (WMIB_i). 

Note that after previous steps our task set Γ=Γ_t ∪Γ_u, has a set of period choices for tasks in Γ_t. Each control task τ_i ∈Γ_t is assigned a performance-aware and secure set of sampling periodicities, i.e. P_i= {p_i^1,p_i^2...,p_i^n_1}. 
For |Γ_t | = q, the number of task specifications for Γ is given by: |P_1| ×⋯× |P_q|, as the periods of tasks ∈Γ_u are assumed as unique.  Each task specification is given as input to the TaskShuffler algorithm for generating the possible set of randomized as well as feasible schedules where no job instance misses the deadline. We denote the overall set of randomized schedules generated considering all task-specifications by 𝒮.

























 §.§ Schedule Vulnerability Analysis




In order to characterize each valid/feasible fixed-priority preemptive schedule's vulnerability level, we first formalise the task schedule s as an array, s = {s[1], s[2],⋯,s[l]}, ∀ s[j] ∈{0,1,⋯, N}. Here, l is the number of time units (each time unit of size δ, same as the column sizes of the schedule ladder, refer to Sec. <ref>) in the schedule hyper-period (the time duration after which a static task schedule repeats itself). The j^th element s[j] in this array denotes the priority of the task executed at j^th time unit, i.e., s[j] = i⇒τ_i∈Γ is executed at δ j time. We use 0 to denote an idle period in the schedule, and i ∈{0,1,...N} denotes the fixed priority assigned to the executed task at a certain time unit. For example, s={1,2,1,3} refers to a fixed-priority preemptive schedule for a task set <ref>. Job 1 of task 1 with priority 1 executes first in the execution order, then job 1 of task 2 with priority 2, and so on. 

Given any schedule s∈𝒮, executing multiple trusted and untrusted tasks from the task set Γ, first we count the total number of possible posterior attacks on s. Given a victim task τ_i with an AEW length of Ω_i, Eq.<ref> gives a total count of all possible posterior attacks on task τ_i denoted by C_p(τ_i) in a schedule s of length l. 


    C_p(τ_i) = ∑_j=1^l I( (s[j]==i) ⋁_∀ k ∈ [1, Ω_i]∃τ_p∈Γ_u, (s[j+k]==p) )

Here, I(x) is an indicator function defined as follows: I(x)=1 if x is true, else I(x)=0 for a proposition x that returns true/false. The indicator function returns 1 if any job from τ_p ∈Γ_u executes in any of the AEW Ω_i of task τ_i∈Γ_t. 
Note that even though the attacker can compromise a single untrusted task, the system designer analyses the vulnerability of a posterior attack on each trusted task τ_i∈Γ_t by assuming every untrusted task τ_j∈Γ_u as a potential attacker.  


Since sampling rates of all the control tasks τ_i ∈Γ_u across different schedules vary, the number of job executions of the task that arrive in each schedule hyper-period will also vary. Therefore, it is necessary to express this posterior attack counts on a trusted task τ_i for a schedule s_k∈𝒮 in probabilistic terms. 

We denote this as the Attack Probability, which is defined as follows:

(Attack Probability)

     Given a valid schedule s_k(∈𝒮) generated for a task set Γ, the Attack Probability AP_<τ_i,s_k> of a victim task τ_i ∈Γ_t⊆Γ is the ratio between the total number of attacker tasks getting executed within the AEW length Ω_i of victim control task τ_i,  and the total number of job-arrivals of the task τ_i in the schedule hyper-period of length l_k. Mathematically, AP_<τ_i,s_k> =  C_p(τ_i)  × p_i/l_k. 
     



For each trusted control task  τ_i∈Γ_t, the AP_<τ_i,s_k> is different in a schedule s_k.

For each safety-critical control task τ_i ∈Γ_t, we define its Tolerable Attack Probability (TAP) as the maximum attack probability on a schedule s that can be tolerated by the closed-loop system, associated with τ_i such that the plant operates in a safe and preferable operating region. For a control task τ_i, we denote its TAP by TAP_i,s_k.

In a mixed-criticality system, the system designer chooses the criticality values of the tasks based on their importance to the system's operations and how their compromise would affect the system's overall performance <cit.>. Considering tasks with a higher index have a lower priority value, the q safety-critical tasks {τ_1, τ_2,.., τ_q } are arranged in decreasing order of priorities.  We assign them with their criticality values {q, (q-1),⋯,2, 1} respectively. We normalize the criticality values by their total sum of criticality values to derive a criticality level of each task. Therefore, we denote the normalized criticality levels for a task τ_i as cl_i = c_i/ ∑_j=1^kc_j,  ∀τ_j∈Γ_t. Since AP is task-specific, a schedule may have a minimum attack probability associated with one safety-critical task, whereas it might have the highest AP for the other safety-critical tasks. To secure the schedule w.r.t. all victim tasks, we use these criticality levels to decide the APs of which higher criticality tasks should be given more importance while quantifying the level of vulnerability of the overall task execution schedule. We define a Schedule Vulnerability Index (SVI) for each schedule that is expressed as a weighted (with criticality levels) average of the APs of all high-criticality tasks.

 (Schedule Vulnerability Index)
Given a schedule s_k ∈𝒮 generated by a task set Γ, its Schedule Vulnerability Index, denoted by the symbol SVI_k, is the weighted sum of attack probability AP_<τ_i,s_k>, weighted with normalized task criticality levels c_i for each control task τ_i.
Mathematically, we can write 


SVI_k= ∑_∀τ∈Γ_t AP_<τ_i,s_k>× cl_i  ∀τ_j∈Γ_t


Similar to TAP, we intend to find a tolerable upper limit of SVI. We term this as Schedule Vulnerability Threshold (SVT). The schedules having SVI above SVT are not normally (we shall define what we mean by normally in the next section) deployed in order to reduce vulnerability against posterior SBAs. This ensures that the attack probability thresholds of the higher-criticality (here same as priority) trusted tasks are taken more into consideration during this vulnerable schedule elimination process than that of the lower-criticality victim tasks. Therefore, the Schedule Vulnerability Threshold denoted by SVT can be determined by replacing the Attack Probability with TAP as SVT_k = ∑_i=1^q TAP_i × cl_i.






The attack-aware schedules generated using the methodology mentioned in Sec. <ref> are pre-computed and stored along with their SVIs and victim task-wise AP values after this vulnerability analysis (See Fig. <ref>). This process eliminates the need for complex online computations in real-time, thereby minimizing the runtime overhead associated with schedule randomization. 




All valid fixed-priority schedules are stored in an array 𝒜, sorted in the order of their respective SVIs. At the K-th index, the schedule with SVT value is stored. Another data structure, Schedule Lookup Table (denoted with LUT),  stores task-wise an array of schedule indices (from 𝒜) which show less attack probabilities w.r.t. a victim task than its TAP. An array of schedule indices stored in the j-th row of LUT is sorted in the increasing order of AP w.r.t. task τ_j. For example, if the i_1, i_2-th schedules from 𝒜 (i.e., 𝒜[i_1]=s_i_1, 𝒜[i_2]=s_i_2) are stored at k, k^'-th positions of LUT[j] (j-th row in LUT) respectively, with k< k^', then AP_⟨τ_j,s_i_1⟩<AP_<τ_j,s_i_2>≤ TAP_j.  

In the next section, we discuss the online counterpart of the MAARS framework that uses these stored data structures after the design time analysis for randomized deployment of task execution schedules in an attack-aware manner.






 §.§ Schedule Selection Algorithm
















The runtime algorithm in the MAARS framework 

is responsible for randomly deploying schedules in an attack-aware fashion. The runtime schedule selection function SchedSel in Algo. <ref> takes the following inputs, (i)  The lookup table LUT and the stored schedule array 𝒜, (ii) the χ^2-detector flag Atkflag, (iii) the schedule vulnerability threshold SVT and (iv) the schedule deployed in the current hyper-period s_k. 
The algorithm starts by storing the schedule index in 𝒜, which has the same SVI as the input SVT. We use the IndexOfSVI method, which takes the sorted schedule array 𝒜 and the SVT as inputs to find out this index and store in K (see line <ref>). In line <ref>, by using IndexOf method we find out the index k of the input schedule s_k that is currently deployed. Another variable, k', is also initialised (with -1) to store the index of the next schedule. 


The algorithm has two distinct modes of schedule selection and deployment operations depending on the detector flag Atkflag. 
(1) Normal mode:  When there are no attack flags raised, i.e., Atkflag=0 denoting no attack on any control task, the algorithm operates in Normal Mode (lines <ref> to <ref>). In this mode, schedules with SVI less than SVT are randomly selected from the array 𝒜. Line <ref> of the algorithm checks if the AtkFlag value is zero. If it is zero, then the algorithm selects a random number in the range [0,K-1] and stores it in k' (see line <ref>). Since MAARS randomly deploys new non-vulnerable schedules at every hyper-period, the random schedule index k' must not match with the currently deployed schedule index k. The algorithm keeps picking another random number until k≠ k' for this purpose.

(see the while loop in line <ref>). The next schedule s_k^' is selected from the k'-th row of the schedule array 𝒜 (see line <ref>).  

(2) Alert Mode:  If the χ^2-detector running inside the control task raises an alarm by detecting an attack (g[k]>Th⇒ AtkFlag>0, see Sec. <ref>) on a control task τ_i, the algorithm runs in Alert mode. In this mode, schedules are selected from the schedule array 𝒜 such that the attack probability for the victim control task τ_i is lower than its tolerable attack probability TAP_i. In line <ref>, if AtkFlag is non-zero, we first store its value in i. The value of the Atkflg denotes a potential attack on task τ_i. In line <ref>, we find out the length of the i-th row in LUT and store it in M. This is used to find a random number ki in the range [0,M-1] (see line <ref>). This number is then used to get the index of the next schedule k' from the ki-th column and i-th row of LUT. Note that each such schedule index picked from LUT satisfies AP_<τ_i,𝒜_k'> < TAP_i. As stated earlier, this random number selection is repeated if the index of the current schedule k matches with the randomly picked next schedule index k' (see the while loop in line <ref>). We select the schedule s_k^' from the k'(≠ k)-th row of 𝒜 (see  line <ref>). 

Finally, SchedSel returns the new schedule s_k' for deployment. Algo <ref> is run once every hyper-period in order to select and deploy new schedules randomly. This online schedule selection is done in an attack-informed way. As a result, the vulnerability level of the task execution schedule is minimized while maintaining the system's performance. 


Overhead Analysis:
The time complexity of Algo. <ref> depends on the complexity of selecting a random number so that it does not match the current value. Since we store the schedules in the sorted orders of SVI and AP in arrays, the algorithm can access them in O(1).  



The total memory cost for storing 𝒜 and LUT can also be calculated, assuming that each element(of the arrays) takes b data bytes. Each schedule, therefore, comprises rows equal to the length of its hyper-period, say l_k for the k-th schedule. Considering a total of M schedules, the total cost is (∑_k=1^M l_k) · b bytes.

LUT, is stored as a 2-D array of q rows (number of trusted control tasks) and M columns, the cost is qM. Hence, the total memory cost is  [ qM + ( ∑_k=1^M l_k ) · M ] · b bytes.
In the next section, we demonstrate the efficacy of this runtime schedule selection algorithm guided by the design-time analysis in real-time system setup.





















§ EXPERIMENTAL EVALUATION



Experimental Setup: We experimentally evaluate our proposed MAARS framework in a Hardware-in-Loop (HIL) testbed and compare it with state-of-the-art (SOTA) approaches against posterior SBAs. 

The first step of the experiment involves design-time schedule generation and analysis for inferability and vulnerability of safety-critical trusted control tasks. This is performed using MATLAB and Python on an 8-core 7th generation Intel i7 CPU with 16 GB of RAM. The implementations can be found in <cit.>. In the next step, we perform a run-time analysis of the control tasks under posterior SBA. For this purpose, we employ an ARM-based 32-bit Infineon Tricore Aurix-397 ECU, connected to ETAS Labcar, a Real-time PC (RTPC), via Controller Area Network (CAN). The implementations can be found in <cit.>.




The physical system/plant is implemented on the RTPC, which sends the sensor measurement to the ECU via CAN, where the control tasks are co-scheduled. Following AUTOSAR mandates<cit.>, we design separate reception tasks to filter and receive sensor IDs transmitted by the ETAS Labcar RTPC over CAN. After processing these sensor data


the control tasks (scheduled by fixed-priority preemptive scheduler in the ECU) are executed. Subsequently, corresponding transmission tasks send computed control input through CAN to the RTPC facilitating plant actuation. Our proposed run-time schedule selection process SchedSel 

is scheduled to run once during each hyper-period in the same core as the other control tasks on the ECUs. Note that SchedSel doesn't use data from the buffer; rather, it receives the detector signal in an interrupt-driven manner and provides a new schedule to the scheduler for the next hyper-period. So the attacker cannot compromise its operations.

[7]l0.53



Task     p_i(ms)     e_i(ms)     c]@l@Utilization
 (Low/High) 
 τ_5     10     1     3*Low, High 
 1-3
τ_6     30     1      
 1-3
τ_7     20     1      
 τ_8     30     5     2*High 
 1-3
τ_9     10     2      
 
Experiment: Untrusted Task Set





We implement four distinct automotive controllers 

similar to the experimental setup in <cit.>. These control tasks include i) an electronic stability program (ESP), responsible for maintaining yaw stability, ii) trajectory tracking control (TTC) system that regulates deviation from a desired longitudinal trajectory, iii) cruise control (CC) system that ensures 

a desired vehicle speed, and iv) suspension control (SC) system that maintains vehicle suspension across varied roads and driving conditions. 

All four control tasks' parameters, their respective AEWs, criticality levels, and TAPs are tabulated in Tab. <ref>. The parameters of the untrusted task set are listed in Tab. <ref>. We have performed the analysis on two different processor utilization scenarios: Low Utilization (LU) (≤ 60% with less no. of untrusted tasks, refer to utilization column in Tab. <ref>) and High utilization (HU) (≥ 90%). 


(1) Design-time  Analysis: 

To generate a set of security-aware task schedules, first, it is required to synthesize a set of periodicities for the control tasks (Tab. <ref>) that ensures their control performance. To achieve this, we solve the LMIs given in Claim <ref>

using 

YALMIP <cit.> along with Gurobi  solver <cit.>. 

The performance-preserving set of periodicities of each control task is further pruned following Claim <ref> and Remark <ref>,

to derive the secure sampling rates for all the control tasks. The final pruned set of security-aware performance-preserving periodicities P_i of each control task are presented in the 3-rd column of Tab. <ref>). 




Next, we extend the TaskShuffler algorithm for multi-rate scheduling, 

as described in Sec. <ref> to generate a set 𝒮 of valid and randomly deployable schedules using P_i's in both HU and LU scenarios.





For each victim task in Tab. <ref>, we find the number of schedules for all victim tasks with attack probability under their respective TAPs. 
The last column of Tab <ref> shows the number of such schedules. Now, using our experimental results, we discuss and show that our proposed randomized schedule generation method outperforms the SOTA in terms of inferability ratio, attack probability, and schedule vulnerability. As SOTA, we consider two cases: i) processor with static priority schedules, and ii) attack-unware randomized schedules are deployed where the set of randomized schedules is generated by TaskShuffler, only considering the minimum sampling rates of each control task.


(A) Inferability Ratio (IR): 
IR of 1,000 randomized schedules generated by our proposed MAARS framework are presented in Fig. <ref>, <ref>, <ref>, <ref> for LU and Fig. <ref>, <ref> <ref>,  <ref> for HU. The above plots depict the IRs of a certain victim-attacker combination among all possible ones that exhibit the maximum IR.


For example, the schedules of τ_1,τ_2, and τ_4 show highest inferability for the untrusted task τ_5 and schedules of τ_3 show maximum inferability for the untrusted task τ_6 in case of LU.

Fig. <ref> and Fig. <ref> (the red plots) represents the highest IR among all the victim-attacker combinations when attack-unaware randomized schedules are deployed.

We can observe that in the case of attack-unaware schedule randomization, most of the randomized schedules show higher IR values compared to the schedules generated by MAARS framework. It implies that MAARS is much better at  hiding timing information of safety-critical control tasks than SOTA.



(B) Average Attack Probability: 
For each schedule in 𝒮, we determine the attack probability of each trusted task using Def. <ref>. For each trusted task τ_i, we compute the attack probability AP_i^j considering the untrusted task τ_j. The average attack probability A̅P̅_̅i̅= ∑_j=1^MAP_⟨τ_i,s_j⟩/M is computed for each trusted task τ_i where M is the total number of schedules generated by MAARS. Fig. <ref> shows the average AP for each of the four trusted tasks in Tab. <ref> for both LU and HU. The blue, grey, and brown plots in Fig. <ref> represent the average APs in case of the schedules generated by static priority schedule, attack-unaware randomized schedule, and the MAARS framework.


We can observe in Fig. <ref>, under LU, schedules generated by MAARS (dark brown plot) show 88% and 38.5% reduction in average AP compared to the static (blue) and SOTA (light grey) respectively. Fig. <ref> shows that under HU, schedules generated by MAARS (dark brown plot) show 83.7% and 47.5% reduction in average AP compared to the static (blue) and SOTA (light grey), respectively.

[8]l0.45
    


    < g r a p h i c s >

    % of schedules under SVT: attack-unaware Randomization (grey), MAARS (brown) 
        


(C) Schedule Vulnerability Index: 
In Fig. <ref>, we plot the percentage of schedules out of all the schedules generated by MAARS (in brown) and SOTAs (in grey) that have SVIs below the schedule vulnerability threshold or SVT. The SVT value calculated with the TAP values corresponding to our control task set (given in Tab. <ref>) is 0.14 (refer to Sec. <ref>).
Our results show that in the LU scenario, 68% of MAARS schedules have SVI<SVT compared to only 31% for attack-unaware randomization. Similarly, in the HU scenario, our results show that 52% and 20% schedules have SVI below SVT for MAARS and attack-unaware scheduling, respectively. Therefore, by using MAARS, we can generate more schedules with lower vulnerability against posterior SBAs.

From the above analysis, we can state that our proposed MAARS framework reduces the inferability ratio and vulnerability index of the schedules and attack probabilities of the tasks compared to the SOTAs.


(2) Runtime Deployment in HIL: 
Since τ_2 exhibits the highest average attack probability (Fig. <ref>) and τ_8 has the highest schedule inference capability (Fig. <ref>) in HU scenario, henceforth we consider τ_2 to be victim task and τ_8 to be the attacker task.


We consider settling time (t_s) and GUES decay rate (λ) metrics to evaluate the plant's performance. To launch the attack, τ_8 executes a malicious code on the same ECU in which τ_2 is implemented.

The malicious code overrides the control input data sent to ETAS RTPC via CAN. A windowed χ^2-detector (window length=1) is implemented along with τ_2 to detect any unnatural deviation in the plant states. The detector's threshold is set to 4 for the false alarm rate (FAR) to be less than 2%. We compare the performance of the TTC system under posterior SBA in Fig. <ref> when schedules generated by MAARS are deployed and when a static fixed priority schedule is deployed.

In Fig. <ref>, the green, blue, red, and magenta plots respectively represent the control input acceleration, plant state deviation from the reference trajectory, the detector's threshold, and χ^2 statistics of the detector. Under no SBA, we note t_s= 4.3s (<desired t_s=10), and λ =95% when a fixed priority schedule is used in Fig. <ref>. However, we can observe in Fig. <ref> that the plant becomes unstable due to posterior SBA under the static fixed priority schedule.


On the other hand, Fig. <ref> shows the behavior of the TTC system under no attack when MAARS is deployed.
We can observe t_s=6s and λ=93%. Under posterior SBA, the scheduler switches from normal mode to alert mode once χ^2 residue>4 (red highlighted area in Fig. <ref>). In alert mode, we note t_s=9.2s and λ=89%. This implies that the system is resilient against posterior attack when MAARS is deployed and subsequently the performance of the system is preserved.











































§ CONCLUSION


Existing schedule randomization methodologies for secureing real-time task-schedules are attack-unaware, hence susceptible to schedule-based attacks. In this paper we present MAARS framework, a novel attack-aware schedule randomization approach that selects performance-aware multiple sampling rates such that the exposure of task parameters is reduced and the vulnerability against posterior schedule-based attacks are mitigated. Our results show that MAARS framework is effective in reducing inferability of critical trusted task parameters, while also reducing posterior attack probability. In future, we aim to extend this work for multiprocessor systems to provide security against other practical schedule-based attacks. 



ieeetr


 Paper Title: A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality.





















A Qualitative Study on Using ChatGPT for Software Security: Perception vs. Practicality
    
    M. Mehdi Kholoosi12, 
    M. Ali Babar12,
    Roland Croft12
    
    
    1 CREST - The Centre for Research on Engineering Software Technologies, The University of Adelaide, Adelaide, Australia
    2 Cyber Security Cooperative Research Centre, Australia
    
    Emails: {mehdi.kholoosi, ali.babar, roland.croft}@adelaide.edu.au
    August, 2024
==================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Artificial Intelligence (AI) advancements have enabled the development of Large Language Models (LLMs) that can perform a variety of tasks with remarkable semantic understanding and accuracy. ChatGPT is one such LLM that has gained significant attention due to its impressive capabilities for assisting in various knowledge-intensive tasks. Due to the knowledge-intensive nature of engineering secure software, ChatGPT's assistance is expected to be explored for security-related tasks during the development/evolution of software. To gain an understanding of the potential of ChatGPT as an emerging technology for supporting software security, we adopted a two-fold approach. Initially, we performed an empirical study to analyse the perceptions of those who had explored the use of ChatGPT for security tasks and shared their views on Twitter. It was determined that security practitioners view ChatGPT as beneficial for various software security tasks, including vulnerability detection, information retrieval, and penetration testing. Secondly, we designed an experiment aimed at investigating the practicality of this technology when deployed as an oracle in real-world settings. In particular, we focused on vulnerability detection and qualitatively examined ChatGPT outputs for given prompts within this prominent software security task. Based on our analysis, responses from ChatGPT in this task are largely filled with generic security information and may not be appropriate for industry use. To prevent data leakage, we performed this analysis on a vulnerability dataset compiled after the OpenAI data cut-off date from real-world projects covering 40 distinct vulnerability types and 12 programming languages. We assert that the findings from this study would contribute to future research aimed at developing and evaluating LLMs dedicated to software security.







security, software vulnerability, large language model




§ INTRODUCTION



Software security remains one of the most significant challenges facing modern software development. With the rise of agile development practices, a focus is being placed on developer-centric security assurance <cit.>. However, studies have shown that developers fail to keep up with the ever-growing list of required security knowledge and expertise <cit.>. Secure coding alone requires developers to learn and use thousands of security-related patterns, practices, and tools. Some areas of software security, such as Software Vulnerability Management (SVM), need specialised knowledge and expertise for detecting, assessing, patching, and disclosing vulnerabilities <cit.>. Consequently, developers tend to leverage recommended systems and/or social media platforms to gain and use knowledge about software security <cit.>. 

Chatbots, i.e., conversational agents, provide effective interfaces for enhanced information retrieval and task assistance <cit.>. Natural language comprehension offers a potentially game-changing interface facilitating human and computer interaction. Natural language can greatly improve the efficiency in which we perform complicated tasks and use complex technologies by allowing us to operate them with simple natural language queries. 

Like many other domains, such as education and marketing <cit.>, chatbots are gaining significant traction in the software development domain for obtaining knowledge/information related to specific software development tasks, such as API usages <cit.>. However, the use of chatbots for software security is still in its infancy stage <cit.> as current software security chatbots use simple rule-based technologies with limited capabilities <cit.>. This situation is expected to change as the advancements in Large Language Models (LLMs) enable much more powerful and capable chatbots, e.g., ChatGPT. 

ChatGPT is an Artificial Intelligence (AI) chatbot developed by OpenAI[https://openai.com/blog/chatgpt/] created using GPT (Generative Pretrained Transformer)<cit.>, a state-of-the-art LLM. These technical advancements allow ChatGPT to surpass traditional chatbots through its capability to provide answers and solutions to complex queries
<cit.>. 
Training and fine-tuning an LLM with billions of parameters (e.g., 175 billion in GPT-3<cit.> and 1,700 billion in GPT-4<cit.>) presents considerable challenges due to the extensive computational resources required <cit.>. Since ChatGPT was released to the public, this limitation has been addressed, and many users were able to experience LLMs for the first time. Whilst ChatGPT was not specifically developed for the software domain, it still capture intrinsic knowledge that enables software security tasking to a certain degree <cit.>. We believe exploring the capabilities and limits of this intrinsic knowledge is extremely beneficial for end-users as most organisations and practitioners lack the required resources to teach domain-specific knowledge to LLMs. Hence, we intend to investigate ChatGPT's intrinsic knowledge in software security to help enable the use of chatbots in this area in the future.


We conducted a multifaceted exploration of perceptions and practicality to investigate ChatGPT's potential for supporting software security tasks. First, we analysed public discussions of ChatGPT users regarding software security to gain insights into the successes and failures of LLM-based chatbots for secure development. In the subsequent phase of our study, we focused on vulnerability detection as it was the most anticipated security task among practitioners, according to the first step of our analysis. 
Through qualitative analysis of ChatGPT's responses, we obtained an overview of the types and manners in which information (e.g., descriptions and assessments) is provided and examined the responses for potential practicality issues. Using this two-staged comprehensive assessment, we uncovered not only the current capabilities and shortcomings of ChatGPT in the context of software security but also demonstrated how practitioners' expectations differ from the practical application of this technology in the real world. We assert that the findings from this study are expected to guide future endeavours that aim to develop specialised LLMs for security-related tasks.



§ RELATED WORK









 §.§ AI-Based Chatbots for Software Security.

Chatbots have been demonstrated to be suitable for software engineering tasks such as information retrieval <cit.> and API usage <cit.>. However, the use of chatbots for software security is limited. To date, the SKF-Chatbot (Secure Knowledge Framework) is one of the leading software security chatbots, which was developed by the OWASP Foundation[https://owasp.org/] to provide access to Software Vulnerability (SV) information via a chat interface. However, its capabilities are limited due to its lack of contextual understanding <cit.>. Recently, AI-based chatbots have been able to overcome these limitations via the LLMs that power them. ChatGPT has thus far demonstrated remarkable capabilities for semantic comprehension and providing tailored solutions for given tasks <cit.>. 




 §.§ Vulnerability Detection Using Transformer-based Language Models.

Prior studies have explored the use of transformer-based language models to identify software vulnerabilities using a variety of fine-tuning methodologies. For example, Thapa et al. <cit.> evaluated  transformer-based language models (i.e., BERTBase, GPT-2 Base) against recurrent neural network (RNN)-based models (i.e., BiLSTM, BiGRU) for vulnerability detection in software datasets featuring C/C++ source code. According to their findings, transformer-based language models outperformed RNN-based models on all evaluation metrics. In another study, Kalouptsoglou et al. <cit.> fine-tuned various transformer-based models (i.e., BERT variants, GPT-2, BART) and carried out a comparative analysis to determine which of these models is the most appropriate for vulnerability detection.
Furthermore, Chan et al. <cit.> created a vulnerability detection model targeted at detecting vulnerabilities in incomplete code snippets. They leveraged common learning approaches such as fine-tuning on three pre-trained LLMs, namely CodeBERT, code-davinci-002, and text-davinci-003. 



 §.§ Vulnerability Detection Using ChatGPT.

Several research efforts have explored the efficacy of ChatGPT in vulnerability detection. 
Chen et al. <cit.> exclusively probed ChatGPT's capability in identifying smart contract vulnerabilities. Szabó et al. <cit.> focused on identifying vulnerabilities associated with CWE-653 (Improper Isolation or Compartmentalization) using multiple GPT models. Ozturk et al. <cit.> assessed ChatGPT's effectiveness in detecting the top 10 OWASP vulnerability categories in web applications. Zhang et al., <cit.> through comprehensive testing on two Java and C/C++ 
vulnerability datasets, demonstrated that prompt-enhanced approaches could strengthen ChatGPT's vulnerability detection capability.
In a recent study, Fu et al.<cit.> compared the capabilities of ChatGPT (i.e., GPT-3.5 and GPT-4) against three other LLMs (e.g., CodeBERT<cit.>) across various vulnerability tasks. The LLMs used in this study were fine-tuned explicitly for code-related tasks.

There are two main ways in which our study differs from prior studies:


    
  * Prior research has primarily focused on assessing <cit.> and enhancing ChatGPT's vulnerability detection performance through prompt engineering methodologies <cit.> or the integration of supplementary elements <cit.> within the LLM pipeline.
    We instead strive to focus more on insights through qualitative analysis of existing real-world uses and outputs. This shift aims to illuminate the disparity between practitioners' perceived value and the practicality of this technology in real-world settings. We provide further details in Section <ref>.
    
    
    
    
    
    
  * Prior studies that utilised datasets from existing literature <cit.> or open-source projects <cit.> ignored the possibility of data leakage (i.e., overlap between training and test data). This factor could artificially inflate the reported performance of AI models' (e.g., ChatGPT) <cit.>. Consequently, we curated a high-quality vulnerability dataset tailored to our evaluation criteria from unseen data. Section <ref> discusses this in more detail.
    




§ RESEARCH DESIGN


Initially, we present our research questions in Section <ref>. This is followed by a detailed explanation of the data collection methodology in Section <ref>. Lastly, Section <ref> delves into the qualitative analysis processes.






Figure <ref> displays the overall workflow that we used to conduct this study.




 §.§ Research Questions.


The following Research Questions (RQs) motivated our empirical study: 

    
  * RQ1: What are users' perceptions of using ChatGPT for software security? We examine user demographics, topics, and types of discussions to determine how ChatGPT is being applied by users and its potential strengths and weaknesses.

    

    
    
    
    

    
  * RQ2: How practical are ChatGPT's outputs for the most desired software security task? We perform a qualitative analysis of ChatGPT's outputs for the most discussed topic of RQ1 to ascertain the practicality of this technology in real-world settings.



 §.§ Data Collection.




  §.§.§ Collecting Twitter Data and Curating a Sample
 For the first data source, we selected the popular social media site, Twitter[https://twitter.com], for our investigation due to its active use by developers to discuss emerging technologies <cit.> and its use in former research for uncovering public sentiments <cit.>.


We used the Twitter API to collect English language tweets that matched the following keyword search query: . We collected tweets that were posted between December 1, 2022, and February 28, 2023, the first three months of ChatGPT's public release. 

In total, we collected 7716 tweets.











To enable qualitative analysis of our data, we curated a sample of posts for manual examination. To ensure that our data sample was of sufficiently high quality, we selected the 700 most liked tweets (approximately 10% of the collected tweets) from our dataset. Twitter likes are given by users who appreciate or agree with a tweet. Hence, highly liked tweets are influential posts that reflect the viewpoints of a larger audience and represent broader public sentiments. We label these tweets T1-T700.



  §.§.§ Collecting Vulnerability Data and Curating a Sample


To perform the required analysis for RQ2, 
we needed to test ChatGPT's capabilities in real-world settings. This purpose was achieved by using the National Vulnerability Database (NVD) <cit.>, an established repository for SV management data maintained by the National Institute of Standards and Technology (NIST). This database holds an exhaustive array of information on identified SVs in the wild. 

As we planned to do an in-depth manual analysis for RQ2, 
we decided to work with a subset of 70 SVs.
This sample size was of statistical significance, providing 90% confidence and an error margin of 10% <cit.>. Previous researchers who needed to perform manual analysis over SVs also chose a similar sample size <cit.>. 

It is imperative to note that we considered the issue of data leakage (i.e., contamination of training and test data) in our data selection for RQ2. 
It is well known that ChatGPT has been trained using a great deal of publicly available data. Since NVD is a public database, it is possible that ChatGPT encountered NVD data during its training phase. Thus, to prevent data leakage, we only included SVs in our dataset with a published date (i.e., the official date on which a vulnerability is added to the NVD) after Sep 2021 (the knowledge cutoff date for ChatGPT at the time of conducting this study). Any newly discovered SVs after this date have not been seen by ChatGPT during training.

Additionally, three other factors were considered when curating the random sample of 70 SVs. First, as the fixed code was required for validation purposes in our analysis, the SV report on NVD must be directly linked to the GitHub commit that addresses the SV (i.e., fixing commit). The References section of each SV in NVD usually includes a link to the fixing commit. The second consideration is that the fixing commit should be focused and not tangled (i.e., a commit that consists of multiple unrelated changes). 

The last considered factor is that the vulnerability should be contained within a single function. The last two factors are because we planned to perform all analysis in the following steps manually, and tracing complicated commits is an error-prone process.

After incorporating the aforementioned factors into our SV selection criteria, we ended up with a dataset (containing 70 SVs) that covered 40 distinct CWEs (Common Weakness Enumeration) <cit.>, and its affected products (i.e., products and platforms considered vulnerable) were written in 12 different programming languages. 
































By utilising the provided fixing commits, we carefully collected the entire vulnerable and fixed (i.e., patched) versions of functions from the relevant GitHub repositories. The average number of lines of code among our 70 collected vulnerable functions was 32.



  §.§.§ Collecting ChatGPT Outputs
 

In this step, ChatGPT was fed the 70 collected vulnerable functions as input and asked about potential SVs within them. The GPT-4 model has demonstrated superior performance compared to the GPT-3 model in various benchmarks <cit.>. Considering the importance of delivering high-quality and up-to-date research insights in the critical domain of software security, all ChatGPT-related analysis in this paper were performed using the GPT-4 model. 

Prior research has shown that tailored prompts are essential to taking full advantage of LLM's capabilities in various software engineering tasks<cit.>. 
Following their prompt designs, for each of the SVs in our dataset, we utilised the following prompt template in our conversation with ChatGPT and carefully saved its outputs 
for further analysis. As ChatGPT remembers information from previous inputs, each query was made in a new chat. 





[colback=blue!5!white,colframe=blue!75!black,title= Prompt Template]
  Role: You are a software security expert.
  
  Instruction: Please analyze the following code snippet for potential security vulnerabilities. Provide a detailed explanation of the issues you find.
  
  Context: <code-snippet>




 §.§ Qualitative Analysis.




  §.§.§ Qualitative Analysis of Sampled Tweets


To understand user perceptions of ChatGPT for software security (RQ1), we qualitatively analysed our sample of 
700 tweets using thematic analysis <cit.>. This was an iterative and reflective process that we conducted over several phases. 

We started by immersing ourselves in the data by carefully reading each tweet and all surrounding data: other tweets in the thread, replies, attached media, linked articles, and associated user profiles. The first and third authors conducted this pilot data labelling process collaboratively for 30 tweets, and it typically took 15 to 20 minutes per tweet. During this process, we posed potential characteristics of the discussion that address RQ1: user demographics, topics and types of discussion. To determine the values of these characteristics, we performed open coding <cit.> to classify the common discussion points. We 
collaboratively identified keypoints as short summaries for each tweet. We then revised these into themes, which provide a one or two word description of the common topics. We conducted a second round of pilot data labeling on an additional batch of 30 randomly selected tweets to ensure thorough coverage of potential themes. In this iterative process, no new themes emerged, indicating that thematic saturation had been reached. 


Once we were satisfied with the identified themes, two authors independently analysed and categorised the remaining 640 
tweets in the sample. They then compared the annotated data to ensure data coding was conducted consistently. The two raters achieved a Cohen Kappa value of 0.813 <cit.>, which implies a high level of agreement. Any remaining disagreements between the first and third authors were resolved through discussion with the second author. During the labelling process, we excluded any tweets from our sample that were unrelated (434 tweets) to the use of ChatGPT for software security due to falsely matched keywords, e.g., tweets discussing the impact of ChatGPT on job `security'.

Ultimately, 266 tweets were included in the sample.

We also wanted to gain an understanding of general user sentiments toward ChatGPT's use for software security. E.g., whether the author of a tweet had a positive experience using it or was optimistic about its potential applications. We experimented with using the NLTK sentiment analysis tool <cit.> to classify our data automatically. However, we found this tool to be inaccurate from a manual review of 30 sample tweets. This is because security keywords such as vulnerability or security were often falsely interpreted as indicating negative sentiments. Furthermore, a tweet may have an overall positive sentiment but a negative opinion of ChatGPT, and vice versa. For example, “This article does a really good job of highlighting the weakness of ChatGPT.” Hence, we manually classified user sentiments of ChatGPT during qualitative analysis. For each tweet, we labelled the sentiment as either positive, negative, or neutral based on our interpretation of the user's opinion or experience of ChatGPT capabilities for software security. We reserved the neutral sentiment label for tweets that discussed positive and negative points or were ambiguous to identify. The nature of the sentiment of a tweet depends on the type of discussion (e.g., experiences vs. speculation). Hence, we consider the types of discussion whilst performing sentiment analysis, which we elaborate further on in Section <ref>.





  §.§.§ Qualitative Analysis of Sampled Vulnerabilities


To thoroughly investigate the practicality of ChatGPT 
in the vulnerability detection task (RQ2), we first needed to assess ChatGPT's efficacy in this task. This is because we were interested in exploring how characteristics (e.g., tone) of outputs differ when ChatGPT is accurate versus instances of inaccurate outputs.
To 
this end, we manually analysed the output of all 70 queries 
and provided an assessment as to whether the target SV was detected. This manual assessment was performed independently by the first and third authors of this paper who had a cumulative eight years of both academic and industrial software security experience.

For each SV within the dataset, we used the fixing commit as a baseline for our comparison and employed techniques from prior researchers <cit.> to manually analyse the code changes related to a SV. These techniques are explained step by step as follows. In the first step, we concentrated on the SV summary from the NVD Description section. Moreover, we focused on the information available in the source code (i.e., fixing commit). Aside from code changes, the title and message of the fixing commit were also examined, as they usually provide valuable information regarding the functional aspects of the code and the context in which it was altered. Having thoroughly understood the SV, we assessed 
its generated ChatGPT output. 
Accordingly, each output was labelled as either not found or found. Most often, ChatGPT provided a list of potential vulnerabilities within the input code, with the particular SV we were interested in being mentioned in one of the bullet points. These outputs were classified as found. If the output did not contain information about the vulnerability we sought, it would be marked as not found. This is even if the other potential vulnerabilities ChatGPT listed appeared genuine. This decision was because we wanted to maximise the reliability of our manual analysis and we did not have the ground truth (i.e., fixing commit) for comparison for the other detected vulnerabilities.



  §.§.§ Qualitative Analysis of ChatGPT Outputs


In response to RQ2, we conducted a thematic analysis <cit.> of ChatGPT's outputs regarding the vulnerability detection task. In particular, our primary goal was to demonstrate the breadth and depth of information that a conversational agent such as ChatGPT can offer and examine the nuances of its presentation.
For the thematic analysis of this step, we followed a similar approach as explained in Section <ref>. The first and third authors started by collaboratively performing a pilot data labelling of 10 ChatGPT outputs, which constitutes 14% of the 70 outputs in our dataset. During this process, we focused on the types of information ChatGPT provides and the way in which it is presented. After that, the two authors put more than 100 hours each into this qualitative evaluation and independently analysed the remaining 60 outputs in the dataset. In a supervisory role, the second author reviewed subsets of the analysis results every week to ensure their validity and suggested modifications to improve the assessment accuracy where deemed necessary. Implementing such supervision was necessary to mitigate potential biases and discrepancies in the analysis.

This qualitative analysis was performed with NVivo 12.0 software<cit.> to streamline theme finding. 



§ RESULTS





 §.§ RQ1: perceptions of using ChatGPT for software security


We firstly wanted to determine the occupations of users of ChatGPT for software security. Such insights provide essential context for the source of discussions and sentiments in our data. Furthermore, we reveal the invested stakeholders of AI-based chatbot solutions for software security. We analysed user profile descriptions of the tweet authors (n=202) and identified any occupation details that were listed. We then categorised user occupations using open coding as described in Section <ref>. The population of authors was analysed as a set not containing duplicates. This means that a user was only considered once in the population if they authored multiple tweets. We identified five main occupations in our sample data, which we describe below:


    
  * Security Practitioners are individuals (n=78) that conduct software security or cybersecurity jobs or tasks. This includes occupations such as security analysts, hackers, and vulnerability researchers. 

    
  * Security Companies are the Twitter profiles of organisations (n=27) that provide software security solutions and services. 

    
  * Software Practitioners are individuals (n=19) that perform software development or associated tasks for their occupation. 

    
  * Blogs are the profiles for online sites or organisations (n=46) that provide news outlets or blogging services. 
    
    
    
  * Other defines a collection of occupations which had too few samples to form individual categories. These include academics (n=6) and government employees (n=5). User profiles that contained insufficient or vague descriptions of occupation (n=21) were also placed in the Other category.











Table <ref> displays the occupations of Twitter users in our sample. Given that our dataset is comprised of software security discussions, security practitioners are expected to form the largest user demographic (39%). However, this provides evidence that security practitioners are interested in the potential applications of AI chatbots to their domain, which is supported by the noticeable demographic (13%) of commercial security companies that are also discussing the viability of such tools. Furthermore, the high percentage of security practitioners in our data indicates that our Twitter sample contains reliable information from users with background domain knowledge. 

We next wanted to examine the topics of discussion in our sample Twitter data to understand the areas of software security for which ChatGPT was being used. These insights were expected to help us determine the potential strengths and weaknesses of AI chatbots for different software security tasks. From our qualitative analysis of tweets described in Section <ref>, we discovered five main software security categories in which users utilise ChatGPT, which are also displayed in Table <ref> alongside their frequency. We go into detail for each category below. We provide quoted text from the sample tweets that we have analysed. However, many tweets provide associated media to provide further context to the discussion, which we cannot capture within our paper.





































Vulnerability Detection. Vulnerability detection is one of the most important components of SV management. Consequently, it is the most frequent discussion category (29%) within our sample of tweets. Many users were impressed that ChatGPT has even basic capabilities for vulnerability detection due to the difficulty of this task:






    “Experiment time. I fed ChatGPT the Damn Vulnerable Ethereum Smart Contract Migration code in it's entirety. It pointed to four potential vulnerabilities!!!” (T95)



Whilst many users demonstrated that ChatGPT could successfully identify vulnerabilities in simple code snippets, some users were sceptical about the validity of its use in real-world scenarios. 


    “I just went three rounds with a "security researcher" (who was chatGPT in disguise) making reports on our smart contracts. Plausible text and impacts wrapped in magnificent misunderstandings of the basics.” (T49)


ChatGPT produces plausible-sounding answers even for tasks which it cannot necessarily complete successfully. Vulnerability detection requires substantial expertise, so it is hard for many users to verify responses for these tasks. The accuracy of ChatGPT may also be limited as it was not trained to perform these tasks.  


    “Don’t use ChatGPT for security code review. It’s not meant to be used that way, it doesn’t really work (although you might be fooled into thinking it does)” (T80)


However, despite this lack of trust, users still found the vulnerability analysis capabilities helpful in speeding up code review. 


    “I had ChatGPT take code ... and then audit for security. Would I deploy the resulting code? No. But two weeks just became minutes.” (T150)


Vulnerability Exploits. Users have also explored leveraging ChatGPT to develop vulnerability exploits for penetration testing or offensive security purposes. These capabilities help security testers verify vulnerabilities, which exceeds the functionality of many traditional static analysis tools.  


    “Write an exploit code that abuse brute-force vulnerability on [One Time Password]. ChatGPT result is mind blowing. Half of the pen tester I know are not even close to that level of well/detailed reporting skills” (T67)


However, a serious concern is that ChatGPT can be similarly used to enable malicious and unethical hacking. 


    “In one instance, a hacker shared an Android malware code written by ChatGPT, which could steal desired files, compress them, and leak them online.” (T117)



    “...ChatGPT can turn anyone into a ransomware and malware threat actor.” (T163)


Such uses would have severe consequences for the state of software security due to the increased ease of cyber attacks and exploits. We observed that discussions about malicious hacking were primarily contained in the Speculation discussion category. Hence, further investigation is required to validate these fears and claims. 

Information Retrieval. Users have also used ChatGPT for software security information retrieval and education. 


    “Currently using ChatGPT to help me find and research security solutions for my organization. This has been the biggest game changer for my career since Python.” (T131)


However, similar to vulnerability detection, some security experts were sceptical about the quality of information returned by ChatGPT. A substantial concern is that it is challenging to detect false information provided by ChatGPT.


    “People are excited about using ChatGPT for learning. It's often very good. But the danger is that you can't tell when it's wrong unless you already know the answer. I tried some basic information security questions. In most cases the answers sounded plausible but were in fact BS.” (T1)


Code Analysis. ChatGPT can also be used for other secure code analysis tasks, such as understanding security patches, reverse engineering, malware analysis, or decompiling assembly. 


    “...We used ChatGPT to analyse the patch to understand the buffer overflow.” (T38)






Other. Finally, users have also theorised the potential utility of ChatGPT for other security tasks, such as threat modelling, vulnerability disclosure, secure configuration, and implementation of security controls. These topics appeared infrequently which implies that did not receive as much attention or interest, but users were still often positive. 


    “... we've not really explored its use-cases for Threat Modeling, which I think is game-changing...” (T161)



    “I had ChatGPT write the breach notification for... Pretty close to the real thing.” (T94)


We finally aimed to investigate the ways in which users were discussing the use of ChatGPT for software security. Our qualitative analysis uncovered five main types of discussion. We leveraged these types during our sentiment assessment of the tweets. We assessed user sentiments of ChatGPT for the tweets of each category to better understand the public sentiments about this emerging technology.


    
  * Article/Tutorial tweets promote an external article or media source containing information about using ChatGPT for software security. 

    
  * Speculation discusses the theoretical use of ChatGPT for software security tasks and topics, with any example usage or evidence of application. 

    
  * Example tweets present the authors findings when using example queries as proof of concept application for different software security tasks. Evidence of use is described through experiences or even screenshots of the use cases. 

    
  * Applied tweets are similar to Example tweets, except they discuss real-world industrial applications relevant to users' occupations. 
    
    
  * Tool tweets promote a tool that has been constructed using ChatGPT technologies. 




Figure <ref> displays the frequency and user sentiments for each category of discussion.
 Overall sentiments leaned towards positive (54% positive), which may indicate that most users are optimistic about the capabilities of ChatGPT for software security. 

Articles and tutorials demonstrating how ChatGPT could be used for software security was one of the most prominent discussion types. This demonstrates that the security community is interested in the potential use of AI chatbot technology. Furthermore, some users have begun to develop software security products using ChatGPT or related technologies, as shown in the Tool category. 

Sentiments were also primarily positive for Example tweets (77%) which indicates that ChatGPT may perform well when used for simple proof-of-concept applications. Similarly, articles that discuss or demonstrate ChatGPT's potential are also slightly positive (51%). However, applied use has split sentiments, which indicates that ChatGPT may not yet be transferable to real-world software security tasks. 





Furthermore, the sentiment in speculative discussions is generally negative (47%). Hence, many users are sceptical about the actual practical benefits of ChatGPT when used for software security tasks. As the majority of discussion on this topic is either speculation or untested example usage, it is clear that AI chatbots for software security require more investigation and rigorous evaluation. 

[right=1pt, left=1pt, top=1pt, bottom=1pt, colback=azure(web)(azuremist)]
RQ1 Findings: 


    
    
  * Vulnerability detection was the most discussed software security task among practitioners testing ChatGPT.
    
  * The overall sentiment of the discussions among users was positive.
    
    
  * In various security tasks, credibility and practicality of generated information were common concerns.
    
  * Sentiments were primarily positive for simple proof-of-concept use cases. 







 §.§ RQ2: practicality of ChatGPT's outputs when utilised for the most desired software security task











  

  



Using the qualitative analysis described in Section <ref>, we first recorded our observations regarding ChatGPT's vulnerability detection accuracy for each SV. 
A total of 43 out of 70 (61.42%) vulnerabilities were correctly identified by ChatGPT. This efficacy was achieved when ChatGPT was presented with vulnerable code snippets of moderate complexity— specifically, those that are relatively long (average of 32 lines of code per snippet) with focused fixing commits and not overly intricate.



































































We answer RQ2 in the following sections by describing the types of information ChatGPT generated and how that information is presented for the vulnerability detection task. This investigation can provide a basis for future enhancements in the design and functionality of tailored conversational agents for software security. We provide quoted text from the analysed ChatGPT outputs. Figure <ref> displays the overall structure of the provided information in ChatGPT outputs. 




  §.§.§ Assumption
 It has been observed that ChatGPT occasionally makes assumptions about tertiary code when analysing the source code inputted. This kind of information was presented tentatively because they all rest on assumptions.

    “For instance, we're making assumptions about functions like compose_sadb_supported() and pfkey_broadcast() which could have their own set of vulnerabilities.” (Output #30)








  §.§.§ Description
 There was a constant presence of descriptions in the ChatGPT responses. According to our thematic analysis, the descriptions can be divided into three major categories, Supplied Code, Vulnerability, and Mitigation.



    
  * Supplied Code: ChatGPT frequently begins its responses with a description of the supplied code. For instance, the functionality of the code or its programming language.

    “This code snippet is written in Java and seems to be responsible for handling requests to trigger a configuration reload through a security token system.” (Output #70)





    
  * Vulnerability: A large portion of each ChatGPT response is devoted to describing identified vulnerabilities. In addition to classifying these descriptions under the Vulnerability category, we further divided them into two themes for a more granular analysis: “Theoretical and Practical Implications". This is because we sought to distinguish between when ChatGPT delves into abstract security implications and when it explains the actionable aspects of potential exploits.


“Theoretical Implication" example:

    “...
4. Exposure of Sensitive Information: The password recovery link may contain sensitive information (like a reset token). The application must make sure to generate a unique and unpredictable token for each password recovery request. ...
” (Output #14)






“Practical Implication" example:

    “... There are a few places where objects are dereferenced without first ensuring they are not null. This could potentially lead to a segmentation fault and crash the program. ...
” (Output #26)


Based on our observations, out of 43 cases where ChatGPT identified the specific vulnerability of interest, 38 instances (88%) were classified under the “Theoretical Implications” theme, while only 5 instances  (12%) were categorised under “Practical Implications”.

    
  * Mitigation: ChatGPT infrequently provided specific information on how to address the particular vulnerability identified. This category contained two themes: “List of Resolutions" and “Strategy".
    
    With the “List of Resolutions" theme, when ChatGPT detects multiple potential vulnerabilities, it offers a numbered list with brief explanations of how to resolve them.

    “To mitigate these vulnerabilities: 
1. Use parameterized queries or prepared statements to avoid SQL injection. 
2. Perform comprehensive input validation and sanitization on all user inputs.
3. Implement proper access control and authorization checks to prevent unauthorized actions.
4. ...
” (Output #64)








The “Strategy" theme revealed that ChatGPT provides a 
detailed response 
when encountering a 
severe vulnerability. In addition, we noticed that the responses generated in these instances have an imperative tone. For instance, the phrase “you should" is commonly used in this theme.

    “In order to make this script more secure, you should: 
Sanitize and validate all input data to prevent injection attacks. Use a safer file handling technique, which does not allow manipulation of file names. ... ” (Output #51)






























  §.§.§ Security Guideline
 ChatGPT makes a persistent effort to make its responses as comprehensive as possible by including generic security guidelines. This phenomenon was observed in all 70 instances of outputs. We classified these guidelines into three categories: Advise Further Testing, Best Security Practices, and Pointers for Consideration.


    
  * Advise Further Testing: The first and most common category (33 instances) is when ChatGPT demands further security audits to be undertaken via experts and automated tools.


    “... It is always recommended to have your
code reviewed by a security expert and use static and dynamic analysis tools to uncover potential security vulnerabilities.” (Output #31)


    
  * Best Security Practices: In the second category, we identified that ChatGPT relied on best security practices (e.g., input validation) applicable to the particular vulnerability to deliver its response.

    “Misconfigurations in PAM can lead to security vulnerabilities, so ensure that you have properly configured PAM in accordance with best security practices. Always adhere to the principle of least privilege. ” (Output #34)





    
  * Pointers for Consideration: Lastly, the third category contains very basic security considerations.

    “you might want to consider using a pre-made, widely-reviewed authentication library or service, which will likely have these and other security issues resolved, rather than creating your own.” (Output #42)


Considering that these guidelines were all advice, all three security guideline categories were presented tentatively.



  §.§.§ Assessment
 The paramount section of ChatGPT responses pertains to its evaluation of potential vulnerabilities within the submitted source code. Our rigorous analysis identified three assessment categories: Vulnerable Classification, Security Issues and Concerns, and Non-vulnerable Classification.


    
  * Vulnerable Classification: This category emerged as the most prominent among the three assessment categories. Considering that all the codes supplied to ChatGPT were already vulnerable, this prevalence was unsurprising. However, a significant variation in the tone of the responses was discovered, which appears to be related directly to the severity of the vulnerability. To further understand, we separated the detected confidence levels into three distinct themes: “Very High", “High", and “Moderate".
    
Within the “Very High" theme, ChatGPT responses were characterised by an assertive tone. In contrast to the other two themes (High and Moderate), adjectives such as “significant" and “serious" were frequently used. Furthermore, ChatGPT repeatedly hinted at the presence of multiple vulnerabilities instead of identifying one individually.

    “While this code may seem harmless at first glance, it indeed has several serious security vulnerabilities. ...” (Output #9)




The tone is less assertive in the “High" theme compared to the tone in the previous theme. For instance, while ChatGPT continues to identify multiple vulnerabilities, the word “potential" is frequently used to indicate a degree of uncertainty.

    “I've found several potential security vulnerabilities in the code. Here's a detailed analysis: ...” (Output #53)




There is a noticeable drop in certainty at the “Moderate" theme compared to the “High" theme. Apart from the frequent use of the term “potential" ChatGPT recurrently cites an absence of contextual information as a rationale for its implied uncertainty.

    “Based on the provided code snippet, there are a few potential security vulnerabilities to consider: ... Please remember that a complete and accurate analysis would require more context...” (Output #52)




Out of all the correctly identified vulnerabilities (43) under investigation, 25 instances (58%) were classified into this theme. The responses were presented tentatively in this theme compared to the other two themes as ChatGPT was less confident about its outputs.
    
  * Security Issues and Concerns: There have been instances where ChatGPT identified potential security weaknesses but refrained from explicitly labeling them as security vulnerabilities. Alternatively, it has used more subtle terms such as “concerns" and “issues". We have categorised such instances under the Security Issues and Concerns category.

    “However, there are some potential issues to be aware of, more from the perspective of robust programming practices and potential misuse, rather than direct security vulnerabilities: ...” (Output #50)





    
  * Non-vulnerable Classification: ChatGPT made a non-vulnerable assessment in several instances. In these cases, ChatGPT consistently pointed out that the supplied code snippet may not necessarily be secure.

    “At a glance, this code does not seem to contain glaring security vulnerabilities such as buffer overflows, SQL injections, etc., that are commonly associated with insecure code. ...  - an innocent-looking piece of code can become a security hole if used improperly or in an unexpected way.” (Output #21)








  §.§.§ Lack of Context
 

We observed that in exactly %50 of the outputs (35 out of 70), ChatGPT mentioned that more context would be required for a concrete security analysis.

    “While the code snippet does not provide all of the details, we can still make several general observations based on the provided information.” (Output #13)


[right=1pt, left=1pt, top=1pt, bottom=1pt, colback=azure(web)(azuremist)]
RQ2 Findings: 


    
  * Even when ChatGPT correctly pinpoints the vulnerability of interest, it predominantly focuses on abstract rather than practical security implications.
    
  * Although ChatGPT responses appear comprehensive, they are heavily populated with generic security information (e.g., Description, Guideline).
    
  * The severity of the vulnerability appears to be directly related to the tone of the responses.
    
  * ChatGPT generally adopted a tentative tone to present the information.
    
    





§ DISCUSSION


In RQ1, our investigation has shown that software and security practitioners enthusiastically experiment with and share the pros/cons of using ChatGPT for assistance in various software security tasks, such as vulnerability detection and security information retrieval. However, the state of discussions mostly remains in example proof of concept use, whereas only a few cases of applied use have been discussed. Current security practitioners appear to think that ChatGPT is inaccurate as a standalone solution. Hence, ChatGPT may be better suited to secondary tasks such as vulnerability report writing. For instance, users can draft comprehensive and detailed vulnerability reports by providing context-specific prompts and formatting instructions. Nevertheless, security professionals should review and validate the contents to ensure accuracy and completeness.

In the second phase of our study (RQ2), we treated ChatGPT as an oracle for the vulnerability detection task. In our data sample, it achieved a detection accuracy of 61%, which is far from ideal. Furthermore, after qualitatively examining the generated responses, we determined the characteristics (i.e., type and presentation style of information) of this chatbot and found that it is unable to provide reliable information for this task. Our analysis revealed that ChatGPT responses suffer from two specific quality issues. The most prevalent issue we observed was that all responses (70 instances) contained generic security information, as indicated in the “Theoretical Implication” theme and the three themes under the “Security Guideline” information type. Throughout these themes, we found an abundance of unneeded information, resulting in a lack of clarity in the responses. This issue persisted even in cases where ChatGPT correctly identified the vulnerability. Another noteworthy issue we found was that ChatGPT regularly lacks assertiveness when delivering responses. Based on our analysis, the majority of responses were given with a tentative tone, as shown in Figure <ref>, and a consistent request for additional contextual information. 

We believe both of these quality issues may undermine the practicality of ChatGPT for vulnerability detection within industry settings. This is because when a fault localisation tool is not transparent about its limitations, developers may inadvertently spend valuable time and effort verifying potentially inaccurate outputs <cit.>.

However, in contrast to existing weakness detection tools (e.g., static code analyser), conversational agents such as ChatGPT allow users to interface with a tool with profound natural language comprehension abilities. This potentially can have positive effects on novice learners who want to start learning secure coding principles. 


Overall, we advocate the need for further development of domain-relevant LLMs trained using software security data. Although ChatGPT is trained for general use and not intended for software security, our qualitative study shows it can assist and augment processes to a certain degree. Hence, we expect domain-specific LLMs to have potent capabilities for software security in the future. These technologies ultimately need to undergo rigorous industrial evaluation for which the empirical software engineering community can provide leadership in collaboration with AI researchers and practitioners. 
























    
    

    

    
    

    













 §.§ Implications of Our Study


Programmers. Understanding the nuances of responses (RQ2) generated by ChatGPT can offer valuable benefits to programmers who leverage this technology for vulnerability detection. Notably, they may better assess the legitimacy of the generated information and make informed decisions about whether to trust and adopt the security suggestions. Furthermore, by understanding ChatGPT's strengths and limitations, programmers can better integrate this technology into their development workflows, using it as an adjunct to other weakness detection tools. 

AI practitioners. The investigation into ChatGPT's communication of software security information (RQ2) has explicit implications for AI researchers and developers. Insights of this nature may assist them in fine-tuning LLM-based chatbots for specific software security tasks. Specifically, they must strive to create security-tailored LLMs capable of conveying security information to users in a manner that is more straightforward, actionable, and less susceptible to misinterpretation. Such models are likely to perform better and be more practical when applied in real-world security settings.

Security researchers. We have identified some promising research directions for researchers. Our analysis in RQ1 unveiled that there is a lot of speculation (discussions in the Vulnerability Exploits category) regarding the potential misuse of ChatGPT for malicious purposes. Researchers need to identify whether technologies like ChatGPT can enable malicious hacking so that countermeasures can be applied to prevent using these technologies for such purposes. 
Additionally, our analysis in RQ2  demonstrated that ChatGPT alters its language tone when confronted with vulnerabilities of varying severity scores. Further research is required to thoroughly understand the capability of LLMs in estimating vulnerability severity. This capability is imperative in vulnerability prioritisation, where precise severity assessment is essential for effective mitigation strategies.










§ THREATS TO VALIDITY




Construct validity: Our manual analysis may be subject to bias or inaccuracies, as is the case for most qualitative research. We have attempted to minimise such threats to validity by ensuring that two annotators independently performed the required manual analysis.

Internal Validity: We acknowledge that the criteria we used for selecting vulnerability data could have influenced the outcome of our study. As explained in Section <ref>, our intention to manually trace code changes meant we could not trace tangled fixing commits. As a result, we decided to limit the number of line changes within fixing commits to 10 lines. This approach ensured we gathered code snippets that were neither simplistic nor too complex.


External Validity: Qualitative analysis enables us to examine our data in depth, but consequently, we can only engage with a small sample of the total dataset. We cannot generalise beyond the sampled tweets and vulnerabilities due to the effort required to examine the data manually. We obtained our tweet sample based on the most liked and influential tweets, so we expect our sample represents a larger audience. 


Furthermore, public opinions may be inaccurate or biased for emerging technologies, as users may be overly excited or not have had enough time to evaluate their use properly. Despite our desire to extend the three-month period for collecting tweets, we were unable to do so since Twitter's academia API has been deprecated following recent changes at the company.

It is also possible that our findings might not generalise to future ChatGPT versions enriched with updated data. However, we conducted this study using a single ChatGPT version with the same knowledge cut-off date (Sep 2021) for consistency.



§ CONCLUSION






We have conducted an empirical investigation to understand the potential use of ChatGPT for software security. Based on examining public discussions regarding this LLM, we found that practitioners were often optimistic about its utility for various security tasks. 
Our systematic evaluation further demonstrated that ChatGPT can identify vulnerabilities 
with roughly 61% accuracy. Furthermore, we qualitatively demonstrated the versatility and granularity of information that ChatGPT can offer in the vulnerability detection task. The results revealed that ChatGPT outputs in this task are often too generic and ambiguous to be of practical use at the industry level. A tailored security LLM could enhance this capability, enabling practitioners to leverage the abundance of knowledge embedded in LLMs.



§ DATA AVAILABILITY

We have made all our artefacts of this study available via a reproduction package<cit.>.



§ ACKNOWLEDGMENT

This work has been supported by the Cyber Security Cooperative Research Centre Limited whose activities are partially funded by the Australian Government’s Cooperative Research Centre Programme.


IEEEtran





 Paper Title: SoK: Payment Channel Networks.





 









































plain



SoK: Payment Channel Networks





    
    Kartick Kolachala, Mohammed Ababneh, Roopa Vishwanathan
    
        New Mexico State University 

        Las Cruces, USA 

        {kart1712, mababneh, roopav}@nmsu.edu
    

    
=========================================================================================================================================================================================



§ ABSTRACT 
Payment Channel Networks (PCNs) have been proposed as an alternative solution to the scalability, throughput, and cost overhead associated with on-chain transactions. By facilitating offchain execution of transactions,  PCNs significantly reduce the burden on the blockchain, leading to faster transaction processing, reduced transaction fees, and enhanced privacy. Despite these advantages, the current research in PCNs presents a variety of research challenges that require further exploration.

In this paper, we survey the recent work in several aspects of PCNs, such as pathfinding and routing, virtual channels, state channels, payment channel hubs and rebalancing. This survey aims to provide the reader with a detailed understanding of the current state-of-the-art in PCN research, highlighting a few important advancements. Additionally, we highlight the various unresolved issues in the area of PCN research.

Specifically, this paper seeks to answer the following crucial question: What are the various interesting and non-trivial challenges in PCN research that require immediate attention from the academic and research community? By addressing this question, we aim to identify the most pressing problems and future research directions that interested readers can immediately work on. Through this analysis, we hope to inspire researchers and practitioners to tackle these challenges to make PCNs more secure and versatile.




Payment channel network, Blockchain, Layer-2




§ INTRODUCTION







































































    













Cryptocurrencies and cryptocurrency based transactions have become increasing popular. Currently, the total market value of all cryptocurrencies in use has surpassed 2.5 Trillion USD. The cyrptocurrency market is increasing at a rate of ≈ 8.00% every year <cit.>. This rise in popularity can be attributed to the following reasons: 1) cryptocurrency transactions can be carried without the presence of a trusted entity. Fiat currency based transactions on the other hand, require the presence of a trusted financial organization such as a bank. 2) cryptocurrency transactions do not subject the user to any limits on the number and type of transactions. Fiat currency based transactions are limited in their amount and number, depending on several factors such as the currency, geographical location, etc. <cit.>.

Each transaction posted to the Bitcoin blockchain takes around 7 seconds to be validated <cit.>. The procedure of validation involves verifying that the transaction posted to the blockchain contains all the required fields and the signature of the user creating the transaction tuple is valid. Once the validation procedure is successfully completed, the transaction is included in a block that would be mined on the blockchain at some time in the future. The process of mining the block successfully takes ≈ 2 hours <cit.> (as of June 13^th 2024). This delay in the transactions and blocks getting confirmed is termed as the blockchain scalability problem <cit.>. In contrast, Visa, a company 

 which globally processes transactions using fiat currencies, processes around 24,000 transactions per second <cit.>. Due to the delay in transaction processing caused by the blockchain scalability problem, blockchain-based transactions cannot process payments instantaneously. 



















As an alternative to processing transactions by posting to the blockchain, payment channels have been proposed. Two parties with the intent of processing payments between them open a payment channel by creating a transaction tuple called the funding transaction. This funding transaction contains the initial deposits from both the parties. These initial deposits are also called as the initial balances of the parties in the payment channel. The sum aggregate of these initial balances is called as the channel capacity. The funding transaction contains the signatures of both the parties involved in the payment channel making it a 2-2 multi signature transaction. This means that the funds in the funding transaction cannot be spent without the signatures of both the parties. This funding transaction is validated and included in a block. Once this block has been successfully mined and confirmed on the blockchain, the payment channel is opened between the two parties. The two parties can now be involved in an unlimited number of transactions with each other as long as the amount of a single transaction does not exceed  their local balances.

An example of a payment channel is given in Figure <ref>. Two users (also called nodes) Alice and Celia deposit 50 coins each into a 2-2 multi signature transaction. This transaction is posted to the blockchain, upon which it is validated and included in a block. The block is mined and confirmed, at which time a payment channel is said to open between Alice and Celia. The sum aggregate of the individual balances of Alice and Celia, which is the channel capacity, is 100 coins. Alice making a payment of 20 coins to Celia is shown in the Figure <ref> and the updated balances of Alice and Celia are shown in Figure <ref>.
After the payment has been made, the channel capacity between Alice and Celia still remains constant at 100 coins. In this manner, Alice and Celia can be involved in an unlimited number of payments between each other. Each payment made in the payment channel in an off-chain manner contains the signatures of both Alice and Celia. When either Alice or Celia decide to close the payment channel, they post their final balances to the blockchain and the payment channel is closed as shown in Figure <ref>. 


For each off-chain transaction in the payment channel, both the parties involved in the payment channel create a commitment. This commitment is essentially an agreement for the new balances signed by both the parties. Exchanging of commitments signifies that both the parties have agreed to the change in their respective balances. Each pair of commitments (for each transaction), contains a unique sequence number called the revocation sequence maturity number. For each new transaction made in the payment channel, the sequence number of the prior transaction is invalidated by revocation keys of both the parties. These revocation keys are created by both the parties before opening of the payment channel. If a malicious party in the payment channel broadcasts an older balance to the blockchain, the honest party in the channel has a certain time period during which it can contest this behavior on the blockchain. Before this time period expires, the honest party in the payment channel will broadcast the revocation of this old state signed by both the parties. The broadcasting of this revocation to the blockchain prevents the malicious party from stealing funds of the honest party. 

The idea of a payment channel that exists between two parties can be extended to a number of nodes, creating a network of payment channels, called a payment channel networks or PCN. The advantage of PCNs is, they enable users that are not connected by a direct payment channel to make payments between each other in an off-chain manner. An example PCN is shown in Figure <ref>. In the figure, consider Alice who intends to make a payment to Hector, with whom she does not share a payment channel. The naive way to process this transaction would for Alice to open a payment channel with Hector, which involves Alice making an expensive blockchain write for the channel opening. 
Each payment channel opening costs 2.4 USD for blockchain writes <cit.>. If Alice intends to send an amount of 1 coin  to Hector, it may not be economical for her to open a direct payment channel. Alice can make use of the PCN and make a payment to Hector by forwarding the payment along the path Alice → Celia → Michael → Rajiv → Charlie → Garcia → Hector. This process of using intermediate nodes in a PCN to forward to the payment to the intended destination is called as routing in payment channel networks.


Motivation and timeliness of PCN research: A significant advantage of PCNs is their capability to facilitate micro-payment transactions, with minimum amounts as low as 10^-7 BTC <cit.>. In contrast, the average transaction cost for a single on-chain transaction on the BTC blockchain is approximately 4.612 USD as of June 2024 <cit.> [Opening a payment channel requires two inputs (for funds from both the parties) and one output (the funds are locked in a single 2-2 multi signature address). Where as making a payment by posting it to the blockchain requires one input (from the party that intends to make the payment) and two outputs (one to the receiver and one to the validator/miner for his fees/reward). This difference makes the transaction cost associated with opening a  payment channel cheaper than making a payment using a blockchain write.], regardless of the transaction amount. This cost can be avoided by using the off-chain PCNs, which incurs no additional fees. Additionally, transactions on the BTC blockchain take around 2 hours to be confirmed as of June 13, 2024 <cit.>, whereas PCNs can process transactions instantly. An example of a real-world PCN is the Lightning Network () on the Bitcoin blockchain <cit.>, which has a 24-hour trading volume of $63,200 and a market capitalization of $7 million <cit.> as of June 2024. These figures reflect 's size and growth.

 Contributions: 

1) We qualitatively compare the recent work in several aspects of PCNs, viz. pathfinding and routing, virtual channels, state channels, payment channel hubs and rebalancing using several relevant properties (metrics) along with providing a reasoning why these metrics have been chosen for comparison.


2) In this paper, we point out the open problems in all the areas that we survey and we also discuss why solving those problems is a hard research challenge.



Outline: 
In Section <ref>, we start  with describing the concept of pathfinding and routing in PCNs and qualitatively compare work published in that area. In Section <ref>, we describe virtual channels which have been proposed to address issues with multi-hop routing in PCNs, and compare work in this area. 

In Section <ref>, we describe and compare state channel protocols, which are a more generalized adaptation of virtual channels and can facilitate the execution of arbitrary applications between nodes in the PCN (not just payments). 
In Section <ref>, we cover payment channel hubs, which are similar to virtual channels and state channels, but facilitate a different use case for payments in PCNs. 
All of the aforementioned PCN mechanisms consume one common resource: the local balance of a node in a PCN. In Section <ref>, we discuss rebalancing, which addresses the important function of replenishing channel funds in the PCN. In Section <ref>, we present the reader with the current research gaps in all of the aforementioned areas and in Section <ref>, we conclude the paper.



Prior work: Prior works by Khojasteh et al. <cit.> and Erdin et al. <cit.> survey the work done only in the area of pathfinding and routing protocols and their privacy aspects in PCNs.

Whereas, in this paper we cover the entire spectrum of PCN research: 
rebalancing, virtual channels and state channels, pathfinding and routing. Apart from this <cit.>

do not provide any information about the open problems in PCNs, which we do in our work. The SoK by Gudgeon et al. <cit.>, surveys several layer-2 protocols, whereas, we focus exclusively on PCNs. Furthermore <cit.> was published in 2020 and does not cover most of the recent work published in PCNs.
































































    



























§ PATHFINDING AND ROUTING


Motivation: One of the areas in PCNs which has garnered significant attention from the academic community is pathfinding and routing. It is defined as the process of finding a path comprising of several nodes from a sender to a receiver in a PCN along which the payment intended to be sent by the sender can be forwarded to the receiver and forwarding the payment along that path.

Intuitively, it may seem that well-known pathfinding and routing protocols from the networks domain can be easily applied to PCNs. Unfortunately, there are several problems with this. 




Even though PCNs are decentralized and distributed peer-peer networks, pathfinding and routing protocols of traditional networks cannot be applied to solve the pathfinding and routing problem in PCNs, because of fundamental differences between intent and the network features: 
1) Transfer of data and transfer of money: The intent behind traditional network's pathfinding and routing protocols is to transfer data between connected devices. 

In PCNs, the intent to transfer money between users in a decentralized and distributed manner. 

2) Change in bandwidth and change in monetary state: When data is transferred between connected devices using network pathfinding and routing protocols, the bandwidth, which is represented by the link weights between the nodes does not change. However, transfer of money between users in a payment channel permanently alters the state of the sender and receiver involved by subtracting and adding money from and to their link weights respectively. 3) Latency and routing fees: The cost of sending data in network pathfinding and routing protocols is measured in terms of latency, which is defined as the time delay experienced in transferring data from one device to another. The delay in data transfer is experienced by the receiver and the cost of the delay is usually paid by the sender. The cost of sending payments in a PCN is measured in terms of money, called as routing fees. This encourages greedy behavior and makes the PCN prone to several attack vectors. Due to these reasons, pathfinding and routing in PCNs is a fundamentally different and a hard research challenge.





The properties  in Table <ref> represent the fundamental principles of fiat currency transactions and on-chain payment mechanisms, which we want reflected in off-chain payments. These properties are generally agreed upon in the literature by several works such as <cit.> as common evaluation metrics for pathfinding and routing protocols in PCNs. Fulfilling these properties while providing efficient pathfinding and routing is a non-trivial challenge, and necessitates the design of new pathfinding and routing protocols. 
Several elegant pathfinding and routing protocols have been proposed in the literature. 
In Table <ref>, we present a qualitative comparison of these routing protocols with respect to the properties they achieve. 

 
Ideal properties:

1) Concurrency: Concurrency is achieved when a pathfinding and a routing protocol enables the nodes to forward more than one payment simultaneously.

2) Privacy: Privacy is maintained when a node's identity is known only to its immediate neighbors and not to the entire network. 

3) Topology Privacy: Topology privacy is preserved when no single node has knowledge of the entire network topology. 

4) Avoids Source Routing: Source routing is avoided when the sender does not determine the path to the receiver.

5) Decentralization: Decentralization is achieved when there are no centralized or trusted entities responsible for constructing paths for senders.

6) Atomicity: Atomicity is ensured when the payment is routed all the way from the sender to the receiver or the payment is not routed at all. 
 
7) Disjoint Graph Applicability: A pathfinding and routing protocol is considered applicable to disjoint graphs if it functions even when the network graph consists of islands.

8) Fees: Routing fees is the amount a node charges for forwarding the payment to the next node along a path from the sender to receiver. This fees can be charged in two ways. Flat/fixed fees means that the fees charged for routing payments remains the same irrespective of the transaction amount being routed. If the fees charged by a node varies according to the transaction amount, it is referred to as dynamic fees, typically a percentage of the amount being transacted. 




§ VIRTUAL CHANNELS








Transactions in PCNs are routed from the sender to the receiver using a path of intermediate nodes. This pathfinding and routing mechanism requires the nodes along the payment path to be available for a transaction to be processed. Nodes, however, sometimes can choose to go offline or there can be network/service disconnections. This might cause the transaction to fail. Virtual channels, which are built on top of existing payment channels, provide a solution to this problem. Virtual channels facilitate payments between a pair of nodes using an intermediate node.  

The intermediate node has individual payment channels open with the other two nodes. The intermediary and the pair of nodes lock coins with each other in their respective payment channels to establish a virtual channel. Once the virtual channel has been established, the two nodes can be involved in a unlimited number of payments. These payments can be processed even if the intermediate node goes offline.

The first virtual channel construction, Perun, was proposed by Dziembowski et al. in <cit.>. 

Assume there are two users, Alice and Bob, along with an intermediary as depicted in Figure <ref>. Alice establishes a payment channel, α_A, with the intermediary by locking Y_A coins as collateral; the intermediary contributes Y_I coins into this channel. Similarly, Bob locks Z_B coins into his channel, α_B, with the intermediary, which also deposits Z_I coins from its possession. This marks the completion of the opening phase, effectively locking the coins belonging to Alice and Bob. Following this, Alice and Bob create a virtual channel to facilitate transactions between them. The virtual channel will include coins deducted from both Alice's and Bob's payment channels. Specifically, X_A coins will be subtracted from Y_A on Alice's side and deposited into the virtual channel, while X_B coins will be subtracted from Z_B and will be deposited from Bob's side. Now Alice and Bob can be involved in an unlimited number of payments without the intermediary being online. 

The idea of a virtual channel between a pair of nodes involving a single intermediary/intermediate node has been extended to establishing a virtual channel recursively over several hops involving several intermediaries, leading to the construction of a recursive virtual channel <cit.>. An example of a recursive channel is shown in the Figure <ref>. In the figure Trent and Alice are a pair of nodes who intend to establish a virtual channel between them. They do in three steps as follows; 1) Trent and Hu construct a virtual channel V_1 with Rajiv as an intermediary. 
2) Hu and Alice construct a virtual channel V_2 with Jill as an intermediary. 
3) Trent and Alice establish a virtual channel between them V_3. 

A multihop virtual channel is constructed by establishing a single virtual channel  between a pair of users over a path comprising of several intermediate nodes. An example of multihop virtual channel is shown in the Figure <ref>. Rajiv and Alice establish a single virtual channel V between them.



Motivation: We have already described how PCNs offer a solution to the blockchain scalability problem by facilitating instantaneous payments in an off-chain manner. PCNs however, have few limitations: 1) The path generated in a PCN to process a payment between a sender and a receiver usually contains one or several intermediate nodes. All the nodes along the path are required to be available to route the payment until the receiver. This may not be possible at all times since, nodes can be disconnected from the network due to various reasons such as power outages, DoS attacks, etc. 
In a virtual channel, once the coins of the intermediary(ies) are locked, it can choose to remain offline without affecting the transaction processing. 2) Each node along the path from the sender to the receiver charges a fees called as routing fees/forwarding fees for forwarding the payment to the next node along the path. The sender has to pay all the nodes their respective forwarding fee along the path. This increases the overall cost of routing the payment. Virtual channels provide a solution to this problem, since they do not charge fees on a per transaction basis. Instead, the fees is charged based on a time epoch basis (the time for which the coins of the intermediary/intermediaries are locked in the virtual channel) or a fixed fee can be agreed in advance by all the parties involved in the virtual channel construction.
3) The time taken to route a payment from a sender to a receiver is directly proportional to the path length. A longer path would result in an increase in the overall pahtfinding and routing time. Virtual channels do not process payments through a path comprising of several intermediate nodes, hence the overall transactions' pathfinding and routing time does not depend on the number of intermediate nodes between the sender and the receiver.











Virtual channels should not add an unnecessary burden to users, and should mirror the operations of payment channels as closely as possible, with the added benefit of no on-chain transactions at all, while maintaining comparable security/privacy properties. We now give the properties desired from an ideal virtual channel, and compare the works in this area on the extent to which they achieve these properties. 



Ideal properties:

1) BC: BC is the blockchain on which the corresponding virtual channel protocol can be deployed. TC is a  blockchain that supports a Turing complete programming language, such as Ethereum, and UTXO is a blockchain that supports the UTXO based scripting mechanism such as Bitcoin. Ideally, a virtual channel protocol should be deployable on any blockchain without the need for any additional programming/scripting requirements. The virtual channel protocol should not require any modifications to the existing consensus mechanisms of the underlying blockchains.

2) Validity: This determines the validity of the virutal channel. A limited validity means that the virtual channel is valid for a predetermined time period (which is decided by parties involved in the virtual channel). Unlimited validity means that the virtual channel can stay open until the parties involved decide to initiate closing. This provides better and efficient transaction processing between users who intend to have frequent payments made between each other. 

3) Fee: This metric determines if the virtual channel takes into account the fees charged by the intermediate node(s) involved in the channel's establishment. The fixed fee model implies that a predetermined, fixed fee is paid to the intermediate node(s) which is agreed upon by all the parties in the virtual channel and the time based fee model implies that a fixed fee is paid to the intermediary which  depends on the time for which the virtual channel stays open. Having an well-defined fee structure will motivate the intermediary/intermediaries to participate in virtual channel creation. 

4) Privacy: Similar to payment channels, we want privacy of nodes for virtual channels, which is said to be achieved when the identities of the nodes in the virtual channel protocols are not known to anyone except for their immediate neighbors.  

5) Offchain dispute resolution: This metric determines if the transaction disputes in a virtual channel require a blockchain write. 
 

6) Recursive: A virtual channel is said to be recursive or hierarchical if it can facilitate payments between a sender and a receiver across a path comprising of several intermediate nodes by recursively constructing virtual channels on top of existing virtual channels from sender to receiver. We give an example of a recursive virtual channel in Figure <ref>.


7) Multihop: A virtual channel is said to be multihop if it can facilitate payments between a sender and a receiver across a path comprising of several intermediate nodes by recursively constructing a single virtual channel from the sender to the receiver, without establishing virtual channel between any pair of intermediate nodes along the path. We give an example of a recursive virtual channel in Figure <ref>.









































§ STATE CHANNELS

A state channel is an off-chain protocol that can facilitate execution of an arbitrary decentralized application, also called as DApp (such as a two player game) between two users in a decentralized and distributed network. 


An example of a state channel is given in Figure <ref>. Let us consider two users Alice and Celia who intend to play a game of chess. They initially interact with the chess game application deployed on the blockchain by a third party service provider. Both Alice and Celia deposit 30 coins each from their possession into a 2-2 multi signature transaction (step 1). Upon locking the funds, the chess game is instantiated between Alice and Celia by the DApp (step 2). Alice and Celia engage in a series of moves for the chess game (step 3, step 4) and let us assume Alice eventually wins (step 5) as shown  in Figure <ref>. Alice gets paid 15 coins as the prize money in Figure <ref>. In this game, Alice and Celia countersign each other's moves until there is a winner or a draw. In the event of malicious activity by one party (such as undoing a prior move), the honest party will broadcast all the moves to the blockchain for dispute resolution.

Motivation: State channels allow the already existing payment channels to facilitate execution of arbitrary applications such as games, e-commerce, etc. This leads to the PCN becoming more versatile.


State channels should not add an unnecessary burden to users in terms of programming/scripting requirements, with the added benefit of facilitating the execution of any decentralized application between them, while maintaining comparable security properties and should ideally be able to resolve disputes without accessing the blockchain. We now give the properties desired from an ideal state channel, and compare the works in this area on the extent to which they achieve these properties. 






We give a qualitative comparison of various state channel protocols in Table <ref>. We describe the metrics used for comparison below. 

Ideal properties:







1) Blockchain:  This is the blockchain on which the corresponding state channel protocol can be deployed. TC is a blockchain that supports a Turing complete programming language and UTXO if a blockchain that supports a UTXO based scripting.  

2) General purpose state channels: This is the ability of the state channel to facilitate the execution of any application in an off-chain manner by using a single application-agnostic instance of state channel on the blockchain. 

3) Security: Security is achieved by a state channel protocol if no honest party loses funds because of malicious behavior by other parties in the protocol.

4) Graceful exit: A state channel protocol is said to employ graceful exit if it has clear and well-defined mechanisms for nodes joining or leaving the state channel.

5) Off-chain dispute resolution: This metric indicates if a dispute that occurs between the parties involved in the state channel can be resolved without accessing the blockchain. 



§ TUMBLERS



Motivation: A payment channel hub is a layer-2 mechanism that facilitates transaction processing between a sender/receiver pair that do not have a payment channel open between them. The intermediary which facilitates payments is called a tumbler. The tumbler has multiple payment channels open with several nodes in the network to facilitate payments. The hubs can potentially be useful in facilitating micropayments between a buyer and seller, without either of them needing to open a payment channel with each other.










Payment channel hubs can be classified into two types: on-chain and off-chain. Early hubs, also called hubs or mixers, were on-chain <cit.>, but they all suffered from scalability issues 
due to having to post each transaction on the blockchain. This led to the development of offchain payment channel hubs for specific blockchains, e.g., Bolt <cit.> for Zcash, Nocust <cit.> and MixCT <cit.> for Ethereum. The most general-purpose payment channel hubs are Tumblebit <cit.>, A^2L <cit.>, and Blindhub <cit.>.




A payment channel hub should should be able to facilitate anonymous payments between a pair of sender/receiver with the added benefit of ensuring that the payment is sent to the receiver or it does not go through at all, while maintaining privacy properties. We now give the properties desired from an ideal payment channel hub, and compare the works in this area on the extent to which they achieve these properties. We give a qualitative comparison of off-chain payment channel hubs in the Table <ref>. We describe the comparison metrics below:

Ideal properties:

1) Blockchain: BC is the blockchain on which the corresponding Tumbler protocol can be deployed. TC is a blockchain that supports a Turing complete programming language, such as Ethereum, and UTXO is a  blockchain that supports the UTXO based scripting mechanism such as Bitcoin.  

2) Relationship anonymity: It ensures that the relationship between a sender and a receiver for a given transaction should not be known to the tumbler.

 

3) Privacy against aborts: Tumbler should not be able to deduce the identities of a sender/receiver in case of a transaction abort, regardless of which party is responsible for the abort. 

4) Time epochs: The tumbler processes transactions in discrete fragments of time called epochs, i.e., transactions only take place during a time epoch. A payment channel hub that supports transaction processing based on time epochs is not efficient. Since nodes would ideally need to wait until a new time epoch begins to process payments again. The duration between time epochs depends on several factors such as the unit of the time epoch (absolute time or relative time measured in terms of block height), number of users that will most likely involved in the next time epoch, etc.

5) Dynamic corruption: The payment channel hub should be able to handle the deviation of any party from the protocol at any point during its execution. i,e the set of adversaries should not be static during the protocol execution.

6) Atomicity: Atomicity ensured when a payment is routed all the way from the sender to the receiver or the payment is routed at all. 

7) Value privacy: Value privacy is ensured by a payment channel hub, when, given a transaction amount, the tumbler cannot link it to a sender/receiver pair.

8) Variable amount: The tumbler should be able to process transactions of any value. 




§ REBALANCING
 
Motivation:Rebalancing in payment channel networks involves redistributing the balance within an existing payment channel or across multiple channels to maintain or improve the network’s liquidity and usability. This is done to ensure that both parties in the channel have sufficient funds on their respective sides to send and receive payments. Rebalancing is crucial to payment channel networks. If the link weights of nodes become zero/depleted as a result of being involved in transactions, it will prevent them from taking part in further payments until those link weights are replenished. Such nodes are called as dormant nodes in the network. A PCN which has a large number of such dormant nodes experiences an overall reduction in transaction throughput and an overall reduction in liquidity. Hence, the link weights of the nodes in the PCN need to be rebalanced to prevent failure of transactions due to lack of liquidity.
A rebalancing protocol should be be able to replenish the link weight of dormant nodes in the PCN irrespective of the PCNs topology, while not having to employ any trusted entity and should not compromise privacy of nodes. We now give the properties desired from an ideal virtual channel, and compare the works in this area on the extent to which they achieve these properties. 

We give a qualitative comparison of rebalancing protocols in Table <ref>. We describe the metrics as follows: 

Ideal properties:

1) Blockchain: This is the blockchain on which the corresponding rebalancing protocol can be deployed. As it can be inferred from the Table, the protocols can either be deployed on the Ethereum or the Bitcoin blockchains. 

2) Trusted entity: This metric determines the nature of trust required among parties to deploy the rebalancing protocol. A centralized trust (C) indicates the presence of a single trusted entity and distributed trust (D) indicates the distribution of trust across various entities. 

3) Graph compatibility: This indicates the nature of the topology of the network on which a rebalancing protocol can be deployed. Cycles indicate that the rebalancing protocol can only deployed on cyclic graph and agnostic indicates that the rebalancing protocol can be deployed irrespective of the network topology. 

4) Privacy: Privacy is said to be achieved when sensitive information such as the local balance and the identities of nodes are not known to anyone except for their immediate neighbors. 







§ RESEARCH GAPS & OPEN PROBLEMS



In this section, we highlight the gaps in research published until now in the areas of pathfinding and routing, virtual channel construction, payment channel hubs, and state channels. This section provides an overview of the current research problems in these areas along with a discussion on why is developing a solution to these problems an interesting research challenge.

















 §.§ Gaps in pathfinding and routing for PCNs

We now identify the gaps in research in pathfinding and routing protocols for payment channel networks.


1) Super node liquidity validation in PCNs: In payment channel networks, pathfinding and routing, which is finding a path between the sender and the receiver that can route the amount specified by the sender has attracted significant attention from the research community. Several elegant and  protocols  have been proposed to solve this problem <cit.> . 
In a PCN, nodes in the network store information about other nodes and their payment channels in their local storage called gossip table/gossip network. The amount of information a node has in its gossip table is a function of the number of payment channels it has open in the network. It is also a function of how much information nodes in the PCN are willing to share with their neighbors (nodes with which they have open payment channels) as a part of the gossiping protocol. The amount of information a node can attain using gossiping may be limited due to the following reasons: a node choose to share information about the network partially, a node can also choose to be anonymous, or a node's own information can sometimes be limited due to the lack of having open payment channels. 

One of the main problems with the super nodes is that the sender has no way to know whether the super node possess the local balance (liquidity) to route the payment of the sender. The local balance of a super node or any node in a PCN is a private value and should not be known to any node except for its immediate neighbor. Currently, if a super node does not have enough liquidity to route the payment of a sender, the transaction fails and it has to be retried by the sender. In , one of the most widely used PCN, this is a significant problem. 
Sometimes the sender might have to keep retrying for ≈ 1 hour to have a successful payment <cit.>. The main goal of PCNs is to facilitate instantaneous payments and these transaction retries render such payments almost impossible. It might greatly benefit the sender if the it has a mechanism to validate whether a super node has enough liquidity (balance) to route its payment without violating any privacy concerns in the network.
  
2) Channel verification in a PCN:
To be a part of any PCN, nodes will open payment channels with other nodes in the network. Two nodes open a payment channel between them by posting a transaction to the blockchain. This transaction can be posted on the blockchain (as in UTXO based blockchains like BTC) or this transaction can be posted as a function call to an existing smart contract (as in blockchains that support smart contracts like ETH). In the most popular PCN, , the channel verification procedure is very inefficient. It involves the node who intends to verify a channel to request the block in which the channel opening transaction has been included, verifying whether the transaction has been successfully executed by the validator/miner and finally verifying if the channel opening transaction corresponds to a 2-2 multi signature address on the blockchain. The verifier performing these steps is inefficient since all these steps will have already been performed by the miner. 
According to the current verification process any node would need to perform any these steps to all the payment channels its neighbors have open. Consider the following scenario: a node j in the PCN has payment channels open with m nodes in the PCN. Furthermore, assume that each neighbor opens k new channels for each time epoch. For this example, assume that the time epoch has been set to 1 block. This would mean that the total number of channels the node j has to verify a total of m*k channels. This requires the node j to access the blockchain a total of m*k times to perform channel verification. Finding a way to do this without blockchain access and in a blockchain agnostic manner in a hard research challenge. 































 

 §.§ Gaps in Tumblers

 We will now describe the research gaps in tumblers.


1) Privacy against aborts: In a given a transaction, the tumbler should not be able to correctly identify the sender/receiver pair involved. Currently, there is only one construction <cit.> that guarantees this. However, this construction works only for blockchains that support Turing complete programming languages. Currently, there is no such construction for UTXO based blockchains that guarantees privacy against aborts. This is hard because if the sender aborts a transaction, the tumbler will know which receiver did not receive the payment. This is hard because, a tumbler mixes multiple transactions together to ensure unlikability among payments. Also, a tumbler should enforce atomicity so that the payment either goes through completely or does not go through at all. In the case of an abort, an adversary who controls the tumbler might be able to deduce which receiver amongst the ones involved did not receive a payment and will be able to link the corresponding transaction amount to that receiver. 

2) Time epochs: A tumbler should be able to facilitate payments without having a predefined time interval or epoch. Currently, the majority of tumbler constructions process payments in discrete epochs.To increase their anonymity sets. An anonymity set refers to a set of users (senders/receivers) whose identity is indistinguishable from each other. The larger is the size of this set, more transactions will be mixed together, increasing privacy of sender/receiver in a transaction. When the interval between two consecutive time epochs is sufficiently large, it will promote more users to take part in the transactions within a time epoch, which improves privacy. Not having time epochs, would mean the tumbler can process transactions at any instant, resulting in smaller anonymity sets and hence could possibly weaken privacy. The trade off is between privacy and flexibility. Processing transactions in discrete time epochs results in increased privacy and reduced flexibility. However, the tumbler's ability to process transactions at any time instant might result in smaller anonymity sets, reducing privacy.

3) Dynamic corruption: A tumbler should ideally be resistant to dynamic corruption. This means the protocol should be resilient to parties deviating from their ideal execution during the beginning, progress, and ending of the protocol's execution. Current tumbler constructions provide support for static corruption, which means that an adversary decides beforehand the parties they intend to corrupt and that cannot change during the protocol execution, which may be unrealistic from a practical standpoint. Security against dynamic corruption is hard to achieve because the tumbler mixes several transactions together to preserve anonymity. Since the tumbler should not be able to link transaction amount to a sender/receiver pair (ideally), if a party gets corrupted during the protocol's execution, the tumbler cannot correctly map the corrupted party to a particular transaction. If the sender/receiver becomes malicious during the protocol's execution, ideally, that transaction must be aborted to prevent financial loss to the honest party. However, the tumbler should not be able to link a transaction to a sender/receiver pair, making security against dynamic corruption a hard research problem.


 

 §.§ Gaps in Virtual Channels

 
 We will now describe the open problems in the area of virtual channels.



1) Blockchain agnostic: A virtual channel should ideally be blockchain agnostic, meaning it should be deployable regardless of the underlying scripting or programming language structure supported by the blockchain. 









2) Well defined fee structure: The intermediary/intermediaries involved in the virtual channel construction additionally lock coins apart from the ones locked in the underlying payment channel. It is necessary to have a well-defined fee structure for this additional service. Currently, nodes get paid a routing fees/forwarding fees for every transaction they process. This fee is dependent on the transaction amount being processed. In the case of virtual channels, having a well-defined fee structure is difficult due to the following reasons: 1) The fee structure should take into account the amount of funds and the time for which these funds of the intermediary/intermediaries are locked in virtual channel. 2) It also needs to take into account the routing/forwarding fees an intermediary could have earned by not locking up the coins in the virtual channel. The fees for an intermediary should be ideally be more than the routing fees it would have earned in order to motivate nodes to be involved in virtual channel construction. Currently, no virtual channel construction has a well-defined fee structure.


3) Off-chain dispute resolution: Ideally a virtual channel protocol should provide the flexibility to resolve any disputes without accessing the blockchain. We have seen in Section <ref>, that blockchain writes are expensive. Having an off-chain dispute resolution is a hard research challenge due to the following reasons; 1) Any form of layer-2 disputes are usually resolved on the blockchain. Since underlying consensus in blockchain usually guarantees resolution of a dispute raised within a specific time period. In an off-chain scenario, there is no such mechanism which guarantees this. None of the virtual channel constructions have a provision of off-chain dispute resolution.


4) Multihop: A virtual channel is said to be multihop if it can facilitate payments between a sender and a receiver across a path comprising of several intermediate nodes by constructing a single virtual channel from the sender to the receiver, without establishing virtual channel(s) between any pair of intermediate nodes along the path.
Currently, no VC construction has this capability. This is hard because, multihop virtual channel involves designing a mechanism for sequentially locking coins across a path comprising of several intermediate nodes. Furthermore, the construction of a multihop virtual channel should also ensure that any transaction does not effect the overall balance neutrality (no node apart from the receiver along the path gains or loses any money). Also, the goals of privacy, atomicity, offchain dispute resolution need to be enforced.


 
 

 §.§ Gaps in state channel protocols

We will now describe the gaps in the area of state channel protocols. 

1) General purpose state channels: An ideal blockchain should be able to facilitate deployment and execution of several decentralized applications without requiring the users to access the blockchain for any purpose other than opening and closing the state channel. Currently, there are two such constructions <cit.> which incorporate code reusability into their design. These two constructions support UTXO based blockchains and have no provision for offchain dispute resolution.

2) Offchain dispute resolution: Current state channel protocols use an onchain transaction or a function call to the onchain smart contract in the case of dispute resolution. An ideal state channel protocol should be able to facilitate dispute resolutions in an offchain manner. This is a hard research challenge since disputes on the blockchain are usually resolved by the underlying consensus mechanisms. 

In an offchain scenario, there is no such consensus that guarantees transaction validity. Also, in an offchain protocol such as the state channel, nodes which are not part of the state channel protocol do not know any details of the protocol's execution due to privacy concerns. Onchain transactions on the hand are publicly accessible.











§ NETWORK STATISTICS



This section introduces a general analysis of the parameters used in , extracted from  gossip messages <cit.>. It offers views into user behavior and the implications of such behavior for network performance. Our analysis shows that the nodes prefer to choose the default value of the  parameters. Using standard default values for these parameters allows nodes to operate smoothly together, reducing the chance of compatibility issues and enhancing the overall reliability and efficiency of the . We evaluated the analysis on a snapshot of  topology from November 27, 2021 <cit.>. The snapshot includes 18,331 nodes and 80,918 channels with a mean channel capacity of 4056152.35 satoshi and a median channel capacity is 1000000 satoshi (this provides us information on how funds are
distributed among  channels). 

Although not explicitly shown in this chart, nodes with multiple connections suggest the presence of fewer central nodes compared to the end-users nodes. Figure <ref>
shows that the majority of the nodes (87.9%) set the 'htlc_minimum_msat' to a default value of 1000 msat, which indicates that the  supports micropayment transactions.
Figure <ref> shows that 'fee_base_msat' parameter across all of the nodes has the highest two values 1000 msat as a default value, and 0 msat as the minimum value, which indicates that most of the nodes do not set the fee_base_msat beyond the default configuration, and likely to support the network usage.   

Figure <ref> shows the majority of the nodes (51.8%) in  use various 'fee_rate_milli_msat' values. A substantial portion of nodes (35.2%) set their fee rate to 1 msat to attract more transactions. Also, different nodes have different fee rate strategies, making  a competitive environment. 
Figure <ref> shows that the majority of the nodes (49.5%) have only one connection that indicates that in ,

many end-users connected to fewer well-connected nodes. Moreover, 30% of the nodes have no payment channels, these could be new nodes added to the  or they could be inactive nodes. 
Figure <ref> shows that (81.5%) of nodes prefer to set the 'cltv_expiry_delta' to the default value of 40 blocks, which is the number of the blocks a node can wait before losing its satoshis from . Moreover, it helps to increase the flexibility of the  to include more nodes along the payment path before reaching the "locktime max' parameter value (2016 blocks). According to the analysis, the nodes generally prefer to use the default parameter values of the lightning network. Using these default values can help in developing more effective routing protocols and optimizing routing fee mechanisms for the nodes. 





§ CONCLUSION



In this paper, we qualitatively compared the recent work in various areas of PCN research: pathfinding and routing, virtual channels, payment channel hubs, state channels and rebalancing. We have also provided the gaps in research in these areas along with reasons why fulfilling those gaps is a hard research challenge. We hope that this paper motivates researchers to build robust protocols that address these gaps that would go a long way towards building out and developing the decentralized finance ecosystem.






IEEEtran



 Paper Title: Vera Verto: Multimodal Hijacking Attack.









Vera Verto: Multimodal Hijacking Attack
    
Minxing Zhang1  Ahmed Salem2  Michael Backes1  Yang Zhang1




1CISPA Helmholtz Center for Information Security    
2Microsoft

    
====================================================================================================================================




§ ABSTRACT 
The increasing cost of training machine learning (ML) models has led to the inclusion of new parties to the training pipeline, such as users who contribute training data and companies that provide computing resources.
This involvement of such new parties in the ML training process has introduced new attack surfaces for an adversary to exploit.
A recent attack in this domain is the model hijacking attack, whereby an adversary hijacks a victim model to implement their own – possibly malicious – hijacking tasks.
However, the scope of the model hijacking attack is so far limited to the homogeneous-modality tasks.
In this paper, we transform the model hijacking attack into a more general multimodal setting, where the hijacking and original tasks are performed on data of different modalities.
Specifically, we focus on the setting where an adversary implements a natural language processing (NLP) hijacking task into an image classification model.
To mount the attack, we propose a novel encoder-decoder based framework, namely the Blender, which relies on advanced image and language models.
Experimental results show that our modal hijacking attack achieves strong performances in different settings.
For instance, our attack achieves 94%, 94%, and 95% attack success rate when using the Sogou news dataset to hijack STL10, CIFAR-10, and MNIST classifiers.
Our code is available https://github.com/minxingzhang/ModalHijackinghere.[Vera Verto is a magical transformation incantation used in the fictional wizarding world of J.K. Rowling's Harry Potter series, which transforms NLP sentences into CV images in the paper.]







§ INTRODUCTION



As a critical component of various applications, machine learning (ML) models have been increasingly expensive to train.
Hence, the training of ML models has transformed gradually into a joint process, e.g., new parties are included in the training of the model either by providing data or computational resources.
However, the involvement of these new parties has created new attack surfaces against ML models, e.g., poisoning and backdoor attacks <cit.>.

Recently, the model hijacking attack has been proposed by Salem et al. in this domain <cit.>, where the adversary is able to stealthily implement their own hijacking task into a target victim model.
Concretely, assuming the same ability as in poisoning attacks, the adversary first camouflages their own hijacking dataset for stealthiness to have a similar visual appearance as the target model's training dataset.
Then the adversary poisons the training dataset of the target model with their camouflaged hijacking dataset.
This attack could induce two different risks.
The first one is about accountability which is the main threat for hijacking attacks, where the model owner can be framed by the adversary to perform illegal or unethical tasks without knowing.
For instance, a completely different classification of some unethical tasks could be mounted in a benign classifier.
The second one is parasitic computing, where the model owner pays the model maintenance costs, while the adversary uses/offers it for their own application/service for free.
For instance, one NVIDIA V100 GPU chip costs $0.992-$2.48 per hour on the Google Cloud Platform.[<https://cloud.google.com/compute/all-pricing#gpus>]
On the other hand, the model hijacking technique can also be adapted to compress models, i.e., training a single model for multiple tasks.



 §.§ Our Contributions



Motivation
The model hijacking attack mandates that the hijacking and original tasks have the same modality, which limits the applicable domains to computer vision (CV) related tasks.
However, ML has achieved great success in many domains, e.g., the multiple available translators such as DeepL and Google Translate, and the different face detectors on social media platforms.
Hence, relaxing this assumption will significantly increase the risks of the model hijacking attack, as the adversary can now target models with different modalities, i.e., more target models exist for the adversary to perform their attack.



In this paper, therefore, we transform the model hijacking attack into a more general multimodal setting, i.e., implementing a hijacking task in a completely different domain.
More concretely, the adversary can implement an NLP hijacking task by a CV target model, as illustrated in <ref>.
For short, we refer to our attack as the modal hijacking attack.
The goal of our modal hijacking attack can be summarized as two aspects:
(a) The effectiveness corresponds to the successful implementations of hijacking attacks in victim models;
(b) The stealthiness corresponds to victim models not realizing that they are hijacked.

Our modal hijacking attack follows the same assumption as the attacks requiring poisoning <cit.>, i.e., the adversary is only able to poison the training dataset without any access to the target model's architecture or hyperparameters.
Our modal hijacking attack would increase the risks of accountability and parasitic computing.
Concretely, our attack could pose a more severe threat to accountability since multiple modalities are involved instead of a single one.
For instance, by model hijacking attacks, a benign classifier would be forced to claim some unethical statements are valid, where the scope of feasible hijacking tasks is extended.
Besides, the threat of parasitic computing can be more significant when NLP tasks are involved.
This is because the establishment of NLP models normally requires an enormous training budget.[<https://research.aimultiple.com/large-language-model-training/>]
And the risk of parasitic computing will get further increased when our Blender (introduced in the following soon) is reusable.

Modal Hijacking Attack
To perform the modal hijacking attack, the adversary needs to transform the NLP-based hijacking dataset into the victim model's CV-based original dataset.
To this end, we propose the Blender, a novel encoder-decoder-based model which integrates a language model, i.e., BERT <cit.> and multiple CNN models.
The Blender integrates two losses, i.e., visual and semantic losses, 
to fuse both the hijacking and original inputs.
Then the crafted sample has a similar visual appearance to the original inputs but maintains the semantic features of the hijacking one, as shown in <ref>.
A successful modal hijacking attack should enable the target victim model to preserve its utility, i.e., has the same performance on the original CV task (“Stealthiness”), while performing the hijacking NLP task with high accuracy (“Effectiveness”).

Different from the previous work, our modal hijacking attack expands the scope from a single type of data to a multimodal setting.
The challenge that stems from the change is the needed transformation from a discrete domain (NLP) to a continuous one (CV), in which there exists a comprehension gap.
We believe this transformation is not trivial, as demonstrated in <ref>.
Meanwhile, our modal hijacking attack is more general, i.e., the Blender can be applied with different hijacking and original datasets.
Besides, the reusability property provides a cheaper option for the adversary to hijack target models as shown later in <ref>.
To the best of our knowledge, the modal hijacking attack is the first work to combine different data modalities, which increases the capability and flexibility for hijacking attacks.
Moreover, this work can encourage the exploration of the applicability of performing hijacking attacks for different modalities (which can also be with the benign aim of model compression).

Evaluation
To evaluate our modal hijacking attack, we use two NLP datasets <cit.>, namely Yelp Review (Yelp) and Sogou News (Sogou), and three CV datasets, i.e., MNIST <cit.>, CIFAR-10 <cit.>, and STL-10 <cit.>.
We extensively evaluate the different setups for our modal hijacking attack.
Our results show that our modal hijacking attack can achieve strong performances with respect to both the attack success rate and the victim's model utility.
For instance, when victim models trained on MNIST, CIFAR-10, and STL-10 datasets are hijacked by the Yelp (Sogou) datasets, our modal hijacking attack achieves an attack success rate of 65% (94%), 68% (94%), and 65% (95%), respectively.
Meanwhile, the victim models' utility is not jeopardized, i.e., our modal hijacking achieves the utility of 99% (99%), 93% (93%), and 93% (92%), respectively, which is less than 2% drop compared to clean models.
Moreover, we show the generalizability of our modal hijacking attack by evaluating it against different setups, e.g., different models to construct the Blender and target model.
Finally, we involve two possible defenses to evaluate the modal hijacking attack.

In addition, we discuss the advantages of our work from the aspects of practical usage, stealthiness, and computational efficiency in <ref> for a better understanding.




§ BACKGROUND AND RELATED WORKS



Training Time Attacks
The inclusion of new parties in the training pipeline for ML has induced a new attack vector.
Adversaries can utilize this to interfere with the training of the victim model.
These attacks are referred to as training time attacks.
One of the most famous training time attacks is data poisoning <cit.>.
This class of attacks allows the adversary to insert malicious samples into the victim model's training data.
One widespread target for the poisoning attacks is to jeopardize the model's utility, i.e., to make the training of the victim model fail, which is different from our modal hijacking attack.

Another related training time attack is the backdoor attack <cit.>.
In this attack, the adversary takes a further step and tries to associate a malicious behavior – of the target model – with a trigger, e.g., a white square on the corner of the input.
A successful backdoor attack results in a victim model that behaves benignly on clean inputs; while predicting a specific label when queried by backdoored data, i.e., inputs with triggers, but triggers are not as flexible as our modal hijacking attack.

Testing Time Attack
A similar attack is adversarial reprogramming <cit.>.
In this attack, the adversary also tries to perform their own task using a victim model.
However, this is a testing time attack, i.e., the adversary only accesses the model after its training.
Unlike our hijacking attack, this attack requires assumptions on the target model such as white-box access.




§ MODAL HIJACKING ATTACK





 §.§ Threat Model



Adversary's Goals
The main goals of our attack could be summarized as:
(a) The effectiveness guarantees that victim models can successfully conduct our hijacking tasks, which will be evaluated by the metric of attack success rate;
(b) The stealthiness ensures that victim models will not realize the existence of hijacking attacks, which will be measured from multi-faceted aspects, e.g., the metric of utility, the visual appearances of fused images, and the poisoning rate.
The results and analyses in <ref> and <ref> show the superiority of our modal hijacking attack.

In this paper, we generalize hijacking attacks to a multi-modal setting, which represents a more practical scenario in the real world.
It might be hard for the adversary to find a victim model of the exact same task domain.
Thus, the previous model hijacking attack, which only focuses on the same modality, limits the applications of hijacking attacks.
However, the transformation to a multi-modal setting is challenging.
Specifically, NLP tasks are located in a discrete domain while CV tasks are in a continuous one.
In that case, how to understand the discrete information in a continuous form is not trivial.
To address this challenge, we propose the modal hijacking attack which enables the adversary to hijack the victim model of CV tasks by NLP tasks, enhancing the applicable scope and practical usage of hijacking attacks.

Adversary's Knowledge
We follow the same assumption as that widely used in poisoning <cit.> and backdoor attacks <cit.>, i.e., we only assume the ability to poison the training dataset of the target model.
Besides the involvement of users who contribute training data and companies that provide computing resources, another possible approach under this assumption is intrusion techniques <cit.>.
However, when aiming at a malicious goal (e.g. frame-up or parasitic computing), no matter in which cases, fewer poisons indicate a more practical scenario and also more stealthiness, which will be further discussed in <ref>.
On the other hand, when a model owner conducts model compression, the target model's training data is of course accessible.
In short, our modal hijacking attack does not require any extra information about the target model architecture or hyperparameters.

Moreover, we assume the adversary to have a container image dataset to fuse with the hijacking one.
This container dataset does not have to follow the same distribution as the original training dataset of the victim model.
However, the adversary can construct it relying on their preference for the visual appearance of the Blender's outputs, i.e., the fused dataset.
In addition, as victim models are used to perform the hijacking task, our modal hijacking attack assumes that the number of labels of the original dataset is at least equal to the hijacking dataset's one.



 §.§ Dataset Terminologies



The modal hijacking attack uses four different datasets which are defined now for clarity:
First, the Original Dataset () is the victim model's training dataset for training the original task;
Second, the Hijacking Dataset () is the adversary's training dataset for training the hijacking task;
Third, the Container Dataset () is a set of images the adversary constructs/collects to fuse with the hijacking dataset samples;
Finally, the Fused Dataset () is the container dataset after being fused with the hijacking one.



 §.§ Blender Design



In general, the Blender aims to generate a fused dataset, which is used to hijack the victim model.
This is performed by fusing the hijacking NLP sentences with the container CV images.
We first present the design of our Blender, then how to deploy it.

Design
To fuse the text-hijacking dataset with an image container one, we first extract the hijacking dataset's features, following state-of-the-art works by using a language model <cit.>.
Then to construct the fused dataset, we first try the naive approach of building an adapter, which is a CNN, to resize the NLP features to the size of the victim model's input.
However, this approach does not perform well for some datasets as shown later in <ref>.
Moreover, using this naive approach results in random-looking images as illustrated in <ref>, which can be easily detected.
To circumvent the limitations of the naive approach, we propose our encoder-decoder-based model, i.e., Blender.
More concretely, the Blender consists of an NLP feature extractor , i.e., a language model, an adapter , two encoders  and , and a decoder .
Further, to improve the computational efficiency, we relax the requirement from using double encoders to a single one .

Another design decision we make is to use the complete embeddings of the hijacking sentence instead of only the last “[cls]” token.
As shown later in <ref>, using all of the embeddings significantly improves the performance of the modal hijacking attack.

Our final design decision is to fine-tune the NLP feature extractor using the hijacking dataset before deployment.
This will allow the Blender to better understand the hijacking dataset, thus improving its ability to extract semantic information and integrate it into container images.
The performance gains of this step are evaluated in <ref>.
It's worth noting that this step does not require any additional assumptions as the adversary, who owns the hijacking dataset, can control the NLP feature extractor independent of the target victim model.
Additionally, the cost of fine-tuning is not a major concern for the adversary, as it can be mitigated by reusing the Blender as demonstrated in <ref>.

Operation
We first explain how double-encoder Blender operates.
The Blender uses the NLP feature extractor  to extract the features/embeddings of the text hijacking input ∈.
Next, these features are input to the adapter  to preprocess before being input to the first encoder .
In parallel, the container image ∈ is input to the second encoder .
Next, the outputs of both encoders are concatenated together and input to the decoder .
Finally, the decoder constructs the output fused image , which has the visual appearance of the container image , while having the features of the text one .
More formally,

    (((())) || ()) = ,

where || is the concatenation operator.

In the single-encoder setting, the container image  is directly concatenated with the adapter result (·), which is then sent to the encoder .
This process is formulated as

    (((())|| ) ) = ,


Training
To train the Blender, we use two losses, namely the visual and semantic losses.

Visual Loss:
The visual loss () is responsible for forcing the fused image to have a similar look to the container one.
To accomplish this, we utilize the mean squared error (MSE) to measure the pixel-wise difference between the fused and container inputs, i.e., = || - ||_2^2.

Semantic Loss:
The semantic loss  is designed to fuse the NLP features in the container image.
Similar to the visual loss, we use MSE for the semantic loss too.
However, the MSE here is calculated between the features extracted from the text input and the ones extracted from the fused image.
Feature extraction here is performed with different feature extractors according to the input type, i.e., the text/image feature extractor (/) is used to extract the features of /.
Since the MSE expects the same sizes for both inputs, we further process the adapter's output with a linear layer () to adjust its size to match the CV ones.
More formally,

    = ||((())) - ()||_2^2.


The Blender is trained as the following 4 steps:

(1) The adversary first constructs their container dataset .
The only requirement for this dataset is to be an image dataset.
Ideally, this dataset has a similar visual appearance as the original dataset , to make the modal hijacking attack more stealthy.
However, the adversary can construct this dataset as they desire.

(2) Second, every sentence in the hijacking dataset is randomly mapped to an image in the container dataset.
This mapping does not have to be one-to-one, i.e., a single container image can be associated with multiple sentences.

(3) Next, (sentence, image) pairs are processed by our Blender as previously presented; both semantic and visual losses are calculated.

(4) Finally, both losses are added and the Blender is updated accordingly, i.e., = argmin_(+), where  is the parameters of the Blender and the linear layer .

It is important to mention that the CV feature extractor  and the linear layer  are only needed when training the Blender, then they can be discarded.



 §.§ Attack Deployment



Our modal hijacking attack is executed in two phases.

Poisoning the Victim Model
The adversary starts by training the Blender as previously presented in <ref>.
Next, they use the Blender to create the fused dataset, i.e., by fusing the container and hijacking datasets.
They then perform a label mapping between the original and hijacking datasets to decide the labels of the fused dataset, i.e., by unidirectionally one-to-one mapping the hijacking samples' labels to the fused dataset's ones.
Finally, the fused dataset with its labels is used to poison the victim model.

Inference Time
After training the victim model, the adversary executes the modal hijacking attack on a target hijacking input  as presented in <ref>.
As the figure shows, the adversary first samples a container image and then uses the trained Blender to fuse it with the hijacking input and create the fused image.
The adversary then queries the fused image to the victim model and receives the output label.
Finally, they map the received label back to its corresponding one in the hijacking dataset.



 §.§ Advantages



There are 3 main advantages of our multimodal hijacking attack over the previous work <cit.>:



- Hijacking attacks are generalized from homogeneous to heterogeneous modalities, which improves practical usage.



- A single encoder in our Blender can support two completely different modalities in our attack, which largely improves the computational efficiency.



- Our poisons (i.e., fused images) look much more natural, where the hijacking content is not exposed.
This advantage enhances our attack's stealthiness.




§ EVALUATION





 §.§ Dataset Description



CV Datasets
We use three commonly-used benchmark datasets in our evaluation, i.e., MNIST, CIFAR-10, and STL-10.
MNIST is a handwriting digits datasets, which contains 70,000 28×28 gray-scale images;
CIFAR-10 is a real-world objects dataset, which contains 60,000 32×32 color images;
Finally, STL-10 is also a real-world objects dataset, which has some common (e.g., airplane, cat, and dog) classes with CIFAR-10.
We use the labeled subset of STL-10 that consists of 13,000 96×96 color images.

NLP Datasets
We use two well-established NLP datasets, i.e., Yelp Review (Yelp) and Sogou News (Sogou).
Yelp is a dataset of English reviews with labels corresponding to scores (between 1 and 5).
It includes 650,000 training and 50,000 testing samples.
Sogou is a dataset of news articles with labels associated with five categories, i.e., sports, finance, entertainment, automobile, and technology.
It includes 90,000 training and 12,000 testing samples.



 §.§ Model Architectures



Blender
To recap, our Blender is composed of an NLP feature extractor, an adapter, two encoders/a single encoder, and a decoder.
The architectures of these components are presented as below: 

NLP feature extractor.
We use the Bidirectional Encoder Representations from Transformers (BERT) language model <cit.>, and fine-tune it.
Then we discard the last layer, and use the embeddings of each token as our NLP features (as previously described in <ref>).
We also try different language models and present the results in <ref>.

Adapter.
The adapter is composed of an average pooling layer, and 4 convolutional ones.

Encoder(s).
We use the same architecture for both double- and single-encoder settings.
Each encoder consists of four convolutional layers with batch normalization and ReLU activation function.

Decoder.
The decoder consists of four convolutional transpose layers.
The first three use batch normalization and ReLU activation function, while the fourth one only uses a tanh activation function.

CV Feature Extractor
We adopt VGG11 <cit.> as our CV feature extractor and use its output as the features.
We evaluate the attack performances when using other CV feature extractors in <ref>.

Victim/Target Model
We use the ResNet18 <cit.> model for our evaluation and show the generalizability of our modal hijacking attack with different target models in <ref>.



 §.§ Evaluation Metrics



Utility
We use utility to measure the performance of the victim model's original task.
To this end, we use a clean testing dataset to evaluate its performance on the clean and the hijacked victim models, where the clean testing dataset is the testing dataset of the original CV model, i.e., from the same distribution as the original dataset but not used when training.
The closer the performances of these two models on the clean test dataset are, the better the utility is.

Attack Success Rate
We use attack success rate (ASR) to measure the effectiveness of the modal hijacking attack, i.e., the performance of the hijacking task on the victim model.
We first create a hijacking test dataset by fusing a clean testing dataset (for the hijacking task) with the container dataset using the Blender.
Next, we compute the accuracy of the victim model on that hijacking test dataset.
Moreover, we train an NLP classification model for the hijacking task to compare its performance with the victim model on the hijacking task, where the data and labels of the hijacking dataset are not changed.
This model represents the case where the adversary wants to perform the hijacking task without the modal hijacking attack, i.e., the upper bound of the attack.
The closer the performance between our modal hijacking attack and this model is, the better the modal hijacking attack is.



 §.§ Results









To recap, the adversary first needs to construct a container dataset to train the blender, as previously mentioned in <ref>.
We randomly sample 100 images from each target dataset to construct their corresponding container datasets.
These 100 images are then removed from the target datasets, i.e., ∩ = Φ.
Next, we sample 5,000 samples from each hijacking dataset, with 1,000 instances per label.
For each hijacking-original dataset pair, we randomly map the corresponding 5,000 hijacking samples with the 100 container images and train the Blender as mentioned in <ref>.
Afterward, we use the trained Blender to fuse the 5,000 hijacking samples and poison the victim model.
The fused samples are combined with the complete training dataset for each victim model, i.e., 60,000, 50,000, and 5,000 samples for the MNIST, CIFAR-10, and STL-10 datasets, respectively.

To evaluate the performance of our modal hijacking attack in terms of both utility and attack success rate, we train clean models for all original and hijacking tasks. 
For the original tasks, we use the complete original dataset when training the models.
And we use the whole clean testing dataset to evaluate the performances.
As mentioned in <ref>, the closer the performances – with respect to the clean testing dataset – of the hijacked and clean models are, the better the modal hijacking attack is.
For the hijacking tasks, we calculate the upper bound of the performance by using the complete training datasets, not just the 5,000 sentences used to hijack the victim model.
We then sample a testing dataset from the hijacking's task test dataset and use it to evaluate these models.
Finally, we fuse this dataset and use the fused version of the dataset to evaluate the attack success rate of the victim model.
The closer the attack success rate to the upper bound performance is, the better the modal hijacking attack is.

Quantitative Evaluation
We plot the attack success rate (ASR) in <ref>.
As the figure shows, our modal hijacking attack achieves strong performance independent of the hijacking and the original datasets.
For instance, our attack achieves 94%, 94%, and 95%, when using the Sogou dataset to hijack MNIST, CIFAR-10, and STL-10 victim models by our Blender of double encoders, respectively, which is only 1% worse than then upper bound models.
Similarly, for the Yelp dataset, our attack's ASR is only 4%, 1%, and 4% less than the upper bound for the MNIST, CIFAR-10, and STL-10 victim models, respectively.
This clearly demonstrates the effectiveness of our modal hijacking attack.
Especially taking into consideration that we only use 5,000 hijacking samples to hijack the victim models unlike the entire dataset when training the upper bound ones, we next evaluate our modal hijacking attack's utility.
We plot the performances of the victim models and the ones trained with clean datasets in <ref>.
As the figure shows, our modal hijack attack achieves comparable performance on the clean testing dataset with the clean models.
For instance, the victim model achieves the utility of 99% (99%), 93% (93%), and 93% (92%) on the MNIST, CIFAR-10, and STL-10 models when being hijacked by the Yelp (Sogou) dataset in the double-encoder case, respectively.
This shows the negligible drop in model utility for our modal hijacking attack.
In addition, we evaluate the attack performance on the Tiny ImageNet <cit.> as the original dataset.
The results show that our attack is generalizable to higher-quality images, as shown in <ref>.
Furthermore, when we restrict the Blender using a single encoder, our method achieves comparable attack performance.
Besides, in the following exploration of different hyperparameters/decision designs (<ref>), using a single encoder consistently shows similar power as double encoders.
These results provide a potential option to gain strong attack performance with less cost of computation resources.

Qualitative Evaluation
Concretely, <ref> shows randomly sampled fused samples together with their container images, as well as corresponding details.
As the figure shows, the output of our Blender is very similar to the original dataset, with few visible artifacts.
As shown later (<ref>), these artifacts do not make the fused images easier to detect.
Besides, compared to the fused images of <cit.>, where the hijacking digits are visible, it is hard to recognize the hijacking content in our attack.
This advantage further increases the stealthiness of our modal hijacking attack.
Moreover, increasing the training epochs of the Blender can reduce the artifacts as shown in <ref>.
To compare the performances of the modal hijacking attack with the naive approach, i.e., only using the adapter but not the Blender, we plot the outputs of the adapter of the Yelp samples in <ref>.
Comparing both figures, the output of our Blender is clearly more similar to the original dataset, hence, showing the stealthiness of our modal hijacking attack.




 §.§ Hyperparameters/Design Decisions



























Using Different NLP Features
To recap, our attack utilizes the embeddings on all tokens instead of only using the“[cls]” one <cit.>.
We now evaluate the performance of our attack when using only the “[cls]” token's embedding.
To this end, we use the Yelp and CIFAR-10 hijacking and original datasets, respectively.
As depicted in <ref>, with double encoders, using only the “[cls]” token's embedding reduces the attack success rate from 68% to 22%.
However, surprisingly, using a single encoder could even slightly improve the attack success rate.
Besides, using the “[cls]” token's embedding does not change the model's utility (similar to our decision of encoding all tokens as <ref> shows), as both approaches result in fused images – with features – that are distinct from the original images.
Hence, the target hijacked model is able to learn the original task without any loss of utility.

Naive Attack
We now evaluate the naive approach for the modal hijacking attack, i.e., directly using the output of the adapter instead of the Blender.
In this part, we take the Blender with double encoders as an example.
To this end, we evaluate the naive approach for all of the hijacking and original datasets.
The results show that the naive approach can achieve almost the same performance as our attack for some datasets, e.g., MNIST and CIFAR-10.
However, the victim models' utility drops significantly on STL-10.
For instance, the utility of the victim models drops separately to 46% and 33% when using the Yelp and Sogou datasets.
We present the full results in <ref>.
Moreover, we plot the resulting images from the naive approach in <ref>.
As the figures show, the resulting images clearly look random and consequently easy to detect, which violates the goal of stealthiness.
These results show the advantage of using the Blender to perform the attack over the naive approach.

Different Feature Extractors
We now show the generalizability of our modal hijacking attack by using different feature extractors.
More concretely, instead of using BERT and VGG11 to train the Blender, we use BART <cit.> and MobileNetv2 <cit.> as the NLP and CV feature extractors, respectively.
The results show that changing the feature extractors yields similar performance regardless of the encoder setting.
For instance, with double encoders, it achieves 68%, 69%, and 63% ASR with a negligible drop in utility, when using the Yelp dataset to hijack MNIST, CIFAR-10, and STL-10, respectively.
We present the full results in <ref>.
This result shows that our attack can use different feature extractors depending on the adversary's preference.

Different Victim Models
We now show the generalizability of our modal hijacking attack against different target victim models.
To this end, we use the Yelp dataset as the hijacking task for all of the CV tasks while using the AlexNet <cit.> as the victim model's architecture.
As expected, our modal hijacking attack still achieves strong performance against the AlexNet-based models in both cases of using double encoders and a single one.
For example, with double encoders, it achieves 67%, 67%, and 66% ASR with the utility drop within 3% when hijacking MNIST, CIFAR-10, and STL-10 classification models, respectively.
We present the full results in <ref>.
These results show that our modal hijacking attack is indeed independent of the victim's model architecture.
We believe this independence is due to the robust feature extraction of our attack.
More concretely, using state-of-the-art language models to extract features from the hijacking datasets results in robust features that can be picked up by different target model architectures when fused with container images.

Intuitively, we hypothesize that the effectiveness of our hijacking attack relies on the redundant model capacity.
That is, the victim model could simultaneously support its original task (i.e., stealthiness) and the hijacking task (i.e., effectiveness).
To this end, we explore the case of using an MLP model as the victim model, as the MLP model can only achieve the utility of 53%, which is much lower than CNNs.
Such a utility gap indicates that MLP models do not own redundant capacity.
In that case, if the hypothesis is accepted, our hijacking attack against MLP models should achieve poor performance.
However, in <ref>, it is surprising to find that our attack performs strongly in both stealthiness and effectiveness.
These results indicate that low-capacity victim models are also vulnerable, which further increases the risks of our attack.

Number of Training Epochs for Blender
We evaluate the effect of varying the number of the Blender training epochs using the Yelp-CIFAR-10 setting, where we take the Blender with double encoders as an example.
We train Blenders using from 50 to 500 epochs with steps of 50, then evaluate our attack performance on them.
Our results show that the utility and ASR do not change much with the number of epochs (<ref>).
However, the quality of the fused images does get better with a higher number of epochs.
The visual quality saturates at approximately 200 epochs.
Hence, the adversary can use 50 epochs if they care less about the visual appearance of the fused images; otherwise 200 epochs would be a good compromise.
We show a set of randomly sampled fused images for the different epochs in <ref>.

Effect of Fine-tuning BERT
We now evaluate the effect of fine-tuning the NLP feature extractor ().
To this end, we use the Yelp dataset to hijack a CIFAR-10 classification model with a non-fine-tuned BERT.
Our results show that the utility of the victim model does not change; however, the ASR is significantly impacted, i.e., the ASR significantly drops.
This shows the need to fine-tune  on the hijacking dataset.
Besides, even the non-fine-tuned one with more poisons is not comparable to a fine-tuned NLP feature extractor.
Another interesting finding is that our single-encoder Blender is more resilient to the non-fine-tuning case.
We present the full results in <ref>.

The Poisoning Rate Effect
We now, to better understand the adversary's capacity, evaluate the influence of the poisoning rate on our attack.
We use the Yelp-CIFAR-10 setting (with 60,000 clean images) and set the number of poisoning – fused – samples from 2,500 (4.2%) to 10,000 (16%) with steps of 2,500 to hijack different models.
The results show that using 2,500 samples is too few to hijack the model, i.e., the ASR is less than 60%.
However, increasing the points beyond 5,000 (8.3%) does not improve the ASR.
Hence, we use 5,000 hijacked samples for our evaluations in <ref>.

In application, the amount of data to poison the dataset here depends on the adversary.
On the one hand, as our attack implements a new task, it just depends on how accurate the adversary wants their hijacking task to be.
On the other hand, even if the poisons could be injected by involving new parities or intrusion techniques <cit.>, less modification of the dataset is intuitively less possible to be revealed, which indicates a more practical setting and more stealthiness.
It is also important to note that even if the accuracy is not state-of-the-art, the ability of the model to perform an unethical task (even with low performance) can make the model owner accountable for a violation.
Even, in that case, one successful attack case might be enough to support it.
Besides, state-of-the-art models usually have millions of inputs, e.g., ImageNet has more than 1.2 M images.
Hence, the amount of poisoning data can be negligible depending on the application.
Further, we find that the Blender with double encoders are more sensitive to the poisoning rate.
We present the full results in <ref>.

Reusability of the Blender
As the Blender can be expensive to train, we evaluate reusing a trained Blender to hijack different settings. 
We try four different setups:
The first hijacks models using the same container dataset but different victim models;
The second increases the flexibility and uses a different container dataset;
The third and the fourth further increase the efficiency of the attack by using a pre-trained Blender to camouflage different hijacking datasets.
For the first case, we already presented its results when evaluating our attack against different target victim models, i.e., AlexNet and MLP.
For the second case, we use the Blender trained using the Yelp hijacking and CIFAR-10 container datasets to hijack MNIST and STL-10 models.
In other words, we use the CIFAR-10 trained Blender to separately fuse images from the MNIST and STL-10 datasets.
Our results show that the modal hijacking attack still achieves strong performance.
More concretely, it achieves the same utility compared to using their original Blender (<ref>) with a non-significant ASR drop for the MNIST and STL-10 hijacked models, respectively.
We present the full results in <ref>.
For the third case, we use the Blender trained on Yelp (Sogou) but conduct the attack with the hijacking dataset of Sogou (Yelp).
In that setting, our modal hijacking can gain almost the same performance.
For instance, when the adversary trains a double-encoder Blender on the Yelp (Sogou) dataset to hijack a CIFAR-10 model using the Sogou (Yelp) hijacking dataset, our attack achieves an ASR of 93% (66%) and utility of 94% (94%), which is only 0%-2% lower compared to training a specific Blender for each hijacking dataset.
Meanwhile, the single-encoder cases show similar results.
We present the full results in <ref>.
For the final case, we try a different setup where two hijacking datasets (Yelp and Sogou) are used to build a Blender.
Then this Blender is used to attack a third dataset (CoLA <cit.>, which is a binary dataset).
This approach shows that it is indeed better to use two datasets instead of one when building the Blender.
More concretely, using the double-encoder Blender trained jointly on both datasets (Yelp and Sogou) improves the performance to 80% ASR and 93% utility, which is 4% (3%) stronger in ASR than when using a Blender trained on a single dataset, e.g., Yelp (Sogou).
We present the full results in <ref>.
These results demonstrate that a single Blender is reusable to different setups, which significantly reduces the cost and increases the flexibility of our modal hijacking attack.

Container Dataset Creation
Finally, we propose a way of constructing the container dataset.
So far, we use a randomly selected container dataset.
One problem is that the labels might not align with the fused samples.
Hence, a manual inspection can raise some flags.
To this end, we now propose a more stealthy way to construct the container dataset.
We first train a CV classifier on the container dataset.
Next, we sort the images that are misclassified with the most confidence.
Finally, we manually select the container dataset out of these images. <ref> shows a subset of these images.
As the figure shows, it is hard to manually assign a label to these images, which makes the fused samples more stealthy.




 §.§ Defense







We evaluate using two established defenses against data poisoning attacks to mitigate our modal hijacking attack <cit.>.
The first defense, intuitively, clusters a clean dataset and computes the centroid of each label.
Then for any given input, the distance between it and its corresponding centroid – depending on its label – is calculated.
Inputs with distances exceeding the threshold are then discarded.
We evaluate this defense technique on the hijacking tasks of Yelp and Sogou and the original tasks of MNIST, CIFAR-10 and STL-10.
The results consistently show that the attack performance drops to almost random guesses, however, it induces an average drop of the utility of around 15%, where using double encoders is more robust.
Moreover, this type of defense requires access to clean data which is sometimes hard to get for many applications in practice.
The second defense EPIC aims to find effective poisons and drop them in low-density gradient regions during training.
The results show that the defense indeed reduces the attack success rate, however, our hijacking attack is still effective in most cases.
We present the complete results in <ref>.




§ DISCUSSION





 §.§ Challenges in Multiple-Modality Scenario



As aforementioned, the main challenge in our work is that NLP and CV tasks are separately located in discrete and continuous feature spaces.
This gap boosts the design of our Adapter and of course our Blender, which generalizes our straightforward but effective framework of hijacking attacks from the homogeneous to the heterogeneous modalities.
Such an improvement largely benefits the practical usage of hijacking attacks.
In practice, this multimodal setting is prevalent since deep learning is now deployed in various applications, e.g., facial recognition, self-driving, and text-to-image generation.
Intuitively, our technique can be adapted for different scenarios where the hijacking task has feature extractors, e.g., language models for the NLP tasks.
However, different modalities have different difficulties, i.e., how easy/hard the original data can be modified, which indicates a non-trivial challenge of adaptively implementing hijacking attacks.

Utilizing the representative tasks in NLP and CV domains, our work serves as the pioneer in exploring hijacking attacks in the multimodal scenario.
We hope that inspired by our work, more complicated scenarios could be applicable in future works.
For instance, hijacking an NLP model with a CV task is an interesting next step but would be significantly difficult/different, as text inputs are not continuous and hence cannot be changed as the images do.



 §.§ Stealthiness of Fused Images



Compared to the fused images of previous work <cit.> (where MNIST digits are visible in the fused CIFAR-10 images), ours look much more natural as shown in <ref>, indicating more stealthiness.
Nevertheless, few noises could still be observed.
We plan to reduce these artifacts by adding a GAN-like discriminator <cit.> to the training of the Blender in future work.
This discriminator is trained to differentiate between the container and fused samples.
The Blender is penalized for any container sample that the discriminator can identify, hence improving the appearance of the fused samples.



 §.§ Computational Costs of Training Blender



Another issue to consider is the computational cost of training the Blender.
Even though this is not the main scope of our work, minimizing the cost of hijacking attacks could improve the adversary's ability in practice.
To this end, we show in <ref> that our Blender can be trained once and then utilized for multiple hijacking attacks, even for different hijacking tasks, making it more efficient.
Besides, our extensive evaluations show the feasibility of only using a single encoder in our Blender instead of double ones, which largely saves computational costs.




§ CONCLUSION



Model hijacking attack is a new threat that exploits the inclusion of new parties in the ML training pipeline.
In this attack, the adversary can hijack CV-based models to implement their own image classification task.
However, as ML has improved in multiple domains besides CV, the hijacking and original tasks might be from different data modalities.
In the paper, we propose a more general hijacking attack in the multimodal scenario, where the adversary can hijack CV models using NLP tasks, which increases the practical usage of hijacking attacks.
To this end, we propose an autoencoder-based model that mixes language models with CNNs, namely the Blender, to perform our modal hijacking attack.
Using the Blender, the adversary can hijack image classification models using text/NLP hijacking tasks.
We extensively evaluate our attack using five different datasets, including three image classification datasets and two text ones.
Our results show that the modal hijacking attack achieves strong performances (i.e., effectiveness) with a negligible drop in the model's utility (i.e., stealthiness).
Besides, the relaxed encoder requirement and improved visual appearances of poisons indicate the advantages of computational efficiency and stealthiness over the previous model hijacking attack.

We aim by this work to first raise awareness of possible accountability risks in some of the realistic machine learning training pipelines.
Then, to motivate the community to work on different mitigation techniques to address this risk, we already present a couple of possibilities.
Finally, our modal hijacking technique can also be used for compressing target models, hence reducing the training or maintenance costs.



plain






§ VISUAL RESULTS















 Paper Title: If It Looks Like a Rootkit and Deceives Like a Rootkit: A Critical Examination of Kernel-Level Anti-Cheat Systems.






If It Looks Like a Rootkit and Deceives Like a Rootkit]If It Looks Like a Rootkit and Deceives Like a Rootkit: A Critical Examination of Kernel-Level Anti-Cheat Systems








0009-0005-6085-3210
christoph.dorner@fhstp.ac.at

    St. Pölten University of Applied Sciences
    St. Pölten
    Austria



0000-0003-3650-9733
mail@l17r.eu

    St. Pölten University of Applied Sciences
    St. Pölten  
    Austria













§ ABSTRACT 
    Addressing a critical aspect of cybersecurity in online gaming, this paper systematically evaluates the extent to which kernel-level anti-cheat systems mirror the properties of rootkits, highlighting the importance of distinguishing between protective and potentially invasive software. After establishing a definition for rootkits (making distinctions between rootkits and simple kernel-level applications) and defining metrics to evaluate such software, we introduce four widespread kernel-level anti-cheat solutions. We lay out the inner workings of these types of software, assess them according to our previously established definitions, and discuss ethical considerations and the possible privacy infringements introduced by such programs. Our analysis shows two of the four anti-cheat solutions exhibiting rootkit-like behaviour, threatening the privacy and the integrity of the system. This paper thus provides crucial insights for researchers and developers in the field of gaming security and software engineering, highlighting the need for informed development practices that carefully consider the intersection of effective anti-cheat mechanisms and user privacy.







<ccs2012>
   <concept>
       <concept_id>10002978.10003022.10003465</concept_id>
       <concept_desc>Security and privacy Software reverse engineering</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10002978.10003022.10003023</concept_id>
       <concept_desc>Security and privacy Software security engineering</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
   <concept>
       <concept_id>10002978.10003006</concept_id>
       <concept_desc>Security and privacy Systems security</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[300]Security and privacy Software reverse engineering
[300]Security and privacy Software security engineering
[500]Security and privacy Systems security













[
    Lukas Daniel Klausner
    August 5, 2024
=========================





§ INTRODUCTION

Online video gaming, particularly competitive gaming and esports, consistently has to confront the issue of cheating. This challenge has intensified with technological advancements in gaming, prompting a response from game developers in the form of sophisticated anti-cheat systems. Notably, kernel-level anti-cheat systems have emerged as a solution to this issue; such systems operate at a deep system level, a characteristic they share with rootkits. Rootkits, known for stealth and deep system integration, typically serve unauthorised purposes, contrasting with the intended protective role of anti-cheat systems.

This paper examines the similarities and differences between kernel-level anti-cheat systems and rootkits. It begins by outlining the progression of cheating methods in online gaming and the corresponding evolution of anti-cheat technologies, focussing on those operating at the kernel level. The discussion then shifts to rootkits, defining their key characteristics for comparison. The primary analysis involves applying a set of metrics, developed for classifying rootkits, to various kernel-level anti-cheat systems used in current popular video games. This methodology is employed to assess whether these anti-cheat systems exhibit rootkit-like properties, potentially raising concerns about user privacy and system integrity.

The aim of this paper is to investigate the ethical and technical implications of kernel-level anti-cheat systems in online gaming. It seeks to address the balance between effective cheating prevention and safeguarding user privacy, a critical issue in the landscape of digital games. Our contributions are as follows:

    
  * We propose a set of criteria to characterise the rootkit-ness of a piece of software.
    
  * We analyse the most prevalent kernel-level anti-cheat systems currently in use in online games and evaluate them based on these criteria, finding two of the four to be rootkit-like.
    
  * Finally, we discuss how to balance security and privacy concerns when designing and employing kernel-level anti-cheat software in digital games.


The rest of this paper is structured as follows: First, we briefly discuss the limited existing literature on the topic. We then give a short background on rootkit and lay out the metrics for rootkit-like behaviour we have synthesised from the field of rootkit research. Thereafter, we discuss, analyse and evaluate the four most prevalent kernel-level anti-cheat systems used in online gaming (BattlEye, Easy Anti-Cheat, FACEIT Anti-Cheat and Vanguard) against these metrics, finding FACEIT Anti-Cheat and Vanguard to exhibit clear rootkit-like behaviour. Finally, we summarise our findings and discuss their implications, in particular with regard to taking user privacy into account when designing anti-cheat software.

This article thus contributes to the existing body of knowledge by providing insight into the methods utilised by kernel-level anti-cheat systems, classifying them as rootkits or non-rootkits based on their intrusiveness. The findings serve as a starting point for further investigations and discussion on the delicate balance between cheat detection and the preservation of user privacy and system integrity within the gaming community.





§ RELATED WORK

Although there are numerous rootkits and kernel-level anti-cheat solutions, there has thus far been very little research focussing on examining and contrasting these two types of software. While several papers have been published discussing rootkits (which we discuss in more detail in <ref>), their detection methods and typical behaviours, there is a noticeable lack of research on anti-cheat systems, with the limited existing literature thereon (much of it in the form of bachelor's and master's theses) often lacking in-depth analysis and comprehensive evaluation.


Rendenbach  <cit.> defines a basic framework for game developers to discern which kinds of anti-cheat system best meet the demands of the game. This framework includes a number of metrics, such as the ethical stance on cheating of the developers, the available budget, privacy concerns and other relevant factors. By considering these metrics, game developers can make informed decisions on implementing an anti-cheat system that aligns with the specific requirements of the game and their ethical considerations. Following this framework, the author provides an overview of possible options that encompasses existing anti-cheat systems as well as other technical and psychological options. This comprehensive overview explores the various measures available and their effectiveness in addressing cheating in games. The discussion concludes with the proposal of a “trustworthiness” factor, which combines the presented measures to assess the overall reliability and efficacy of an anti-cheat system.

Lehtonen <cit.> provides an overview of the historical development of anti-cheat systems in video games. The study categorises and examines different techniques employed by anti-cheat systems. Additionally, the research evaluates and describes the effectiveness of combined approaches utilised by these systems. By presenting this comprehensive analysis, Lehtonen contributes to understanding anti-cheat systems in video games, their evolution and the evaluation of combined approaches for enhancing gameplay integrity and fair competition.

Silva <cit.> elaborates on anti-cheat methods solely residing on the server to increase users' privacy. Due to many games having the ability to replay matches, these demos can be analysed in retrospect, detecting any occurrences of cheating in the match.

Maario et al. <cit.> propose the need for less intrusive anti-cheat systems and introduce an approach for such a system. In the paper, several types of cheats and anti-cheat measures are elaborated. Mirroring Silva, the conclusion of this paper also favours a server-sided anti-cheat mechanism utilising recorded games and evaluating them based on machine-learning approaches.

Beegle <cit.> presents an overview of the evolution of rootkits over time and gives a detailed classification of virtualised rootkits, categorising them based on specific characteristics and behaviours. Additionally, the study addresses the resulting threats to information security, covering prevention strategies, detection methods and reliable removal techniques. The study thus provides a comprehensive understanding of rootkits, their various types and the associated challenges in ensuring robust information security.

Jiang's work <cit.> introduces a virtualised honeypot farm to attract and capture rootkit attacks to obtain samples from real-world incidents. This honeypot farm collects malware samples from the internet for subsequent analysis. The study also discusses how to design a highly efficient and accurate malware logger system. This logger system is aimed at effectively recording and analysing malware activities to enhance the understanding and response to such threats. By combining the virtualised honeypot farm and the malware logger, Jiang's research contributes to the field of cybersecurity by providing valuable insights into rootkit attacks and subsequent prevention methods and facilitating the analysis of captured malware samples. However, not all rootkits would execute on the virtualised platform.








§ METHODOLOGY

In response to growing concerns within the online gaming community about the similarities between kernel-level anti-cheat systems and rootkits, this study employs a structured analytical approach. We focus on comparing prominent kernel-level anti-cheat systems against defined metrics associated with rootkits. The list of metrics used in this study is derived from an analysis of different studies concerning the behaviour of common rootkits, while the evaluation of the different anti-cheat solutions is based on empirical analyses of both security researchers and cheat developers. This approach enables us to portray the systems from different perspectives. An anti-cheat solution was only flagged as exhibiting a certain characteristic if clear evidence of rootkit-like behaviour was present. We selected the four kernel-level anti-cheat solutions analysed for analysis which are currently most prevalent and widely used in the online gaming community.



 §.§ Rootkits

Rootkits are a type of malware that provide cybercriminals with unauthorised access to a computer system while remaining hidden. They are classifiable into several types: firmware rootkits, kernel-mode rootkits, virtualised rootkits and user-mode rootkits (although these ones are not relevant to this study). 

Firmware rootkits <cit.> target the software that runs particular hardware components, such as the BIOS, and are stored on the software that runs during the boot process before the operating system (OS) starts. They are stealthy as they can persist through OS reinstallation because the firmware on the hardware itself is infected.

Kernel-mode rootkits <cit.> modify the OS by adding or changing existing code. They often exploit the fact that OSes allow device drivers or loadable modules to execute with the same privileges as the OS kernel. To this end, they are often packaged as device drivers or modules to avoid detection.

Virtualised rootkits <cit.> boot up before the operating system and operate as malware that executes as a hypervisor controlling one or multiple virtual machines (VMs). This shimming between hardware and OS enables the rootkit to conduct its malicious activities with less chance of being detected since all VMs linked to the hypervisor appear to function normally. 







 §.§ Rootkit Metrics

After this brief description of different rootkit types, we now define a set of metrics to classify software as rootkit-like. We have synthesised a set of seven properties commonly attributed to rootkits in our classification scheme.

Evasion: Evasion and deception of intrusion prevention systems or standard antivirus scanners is a known feature of rootkits. Rootkits often accomplish this goal by hiding malicious processes or tampering with applications on the infected system, patching the binaries so they fail to show files created by the rootkit, and often employing hooking tactics <cit.>. If the rootkit starts before the OS has finished booting (as is the case in firmware rootkits), it can evade all detection mechanisms of the OS altogether due to it having control of all drivers and applications starting after itself <cit.>.
In addition, rootkits often attempt to disguise themselves as a different process using process hollowing <cit.>. In this strategy, the memory of harmless processes is deallocated and overwritten by the rootkit for its usage while still operating under the name of the harmless process.
Some rootkits can also hide the generated network traffic amongst other features by manipulating the bash|ioctl| syscall, typically used to read or change the driver parameters for attached devices, including the network card <cit.>.

Virtualisation: Rootkits often use virtualisation for evasion tactics and concealment of their code. 
Virtualisation is used to protect the image itself by using proprietary, custom architecture emulations. Often tools like VMProtect[ https://vmpsoft.com/https://vmpsoft.com/] are used to protect software images, making it difficult to reverse-engineer such software. Despite the use of these measures, it is possible to (partially) dump such modules from loaded memory areas or with the use of third-party tools to de-virtualise the images, enabling reverse-engineering them nonetheless <cit.>. Despite sharing the same terminology, this kind of virtualisation is different from what is used in virtualised rootkits.
 


Time of execution: Firmware rootkits are designed to start execution concurrently with or before the OS of the infected system, depending on the methods utilised by the rootkit. Booting this early leads to the advantage of hiding from user-level monitoring tools and the OS's security mechanisms. Additionally, this leads to the malicious software being able to eavesdrop on every action happening above it by hooking various methods, using syscalls, and manipulating loaded libraries. With these capabilities, the rootkit can monitor any function executed in the user and kernel space, including alterations <cit.>.

Remote access and controllability: Most rootkits implement methods for the threat actor to control and access the infected system remotely and often include infected targets in botnets. 
Infected machines in such networks become bots. Botnets are rentable to run large distributed denial-of-service attacks on various targets. Bots connect to a centralised command-and-control (C2) server infrastructure, which sends commands to the infected systems that the bot executes. 

Furthermore, an attacker can install additional malware onto the infected system using the remote access feature and disable or remove security features, further compromising the infected systems <cit.>. This feature is often used to stage supplementary attacks on adjacent systems to perform lateral movement in the network. 

Information exfiltration: Rootkits often include functionalities to exfiltrate data from infected systems, especially targeting corporations or governments to steal classified or high-value information. Depending on the information stolen, it can be sold (e. g. on the dark web) or used for the attackers' gain. These techniques can also be used to obtain basic information about the system or the attached network to plan for further attacks and select suitable malware specifically targeting the infected systems' weaknesses <cit.>.

Network manipulation: Given that the network card of a system requires drivers to operate, a rootkit can attach to these drivers by hooking functions or the syscall table directly to send and receive packets on the network. This technique enables the rootkit to stay undetected for user-mode applications and spoof the outgoing network connections, hiding the ones originating from the rootkit. Furthermore, monitoring and alteration of established connections are possible, often used to attack systems in the adjacent network or connect to a C2 server. This is especially useful for passively eavesdropping on the network.

Removability: Kernel-level rootkits can often be hard to remove, requiring specifically designed tools. It is even possible that the system needs to be reinstalled from scratch to remove all traces of the rootkit, which is the generally accepted best-practice approach to treat infected systems <cit.>. In some cases, replacing some or all system hardware components is necessary to remove a firmware rootkit entirely.



 §.§ Anti-Cheat Solutions

The selection of kernel-level anti-cheat systems was based on those used in online games with the highest player counts (as recorded on https://steamcharts.comsteamcharts.com and https://tracker.gg/tracker.gg on 21 January 2023). Additionally, the level of intrusiveness was considered, determined by the extent of controversy surrounding their functionality within the gaming community, as reflected in discussions on online forums[ https://www.reddit.com/r/pcmasterrace/comments/19acyup/is_anyone_concerned_with_more_and_more_gaming/https://www.reddit.com/r/pcmasterrace/comments/19acyup]. This resulted in the anti-cheat systems of the games Counter-Strike: Global Offensive, Fortnite, Tom Clancy's Rainbow Six Siege and Valorant being chosen. All four systems are partially based in kernel mode and use similar methods. Due to the intrusiveness of modern kernel-level anti-cheat solutions, the boundaries between them and rootkits are often slim, causing a rogue kernel-level anti-cheat system to have the potential to devastate the systems it is installed on. We have chosen a simple point system for our evaluation, with a single point added to the rootkit-ness score of an anti-cheat system for each matching criterion. Summing these points after the analysis, we arrive at a score from zero to seven; if any given anti-cheat system is assessed with four or more points, we consider it to be a rootkit-like system.



  
BattlEye
This German anti-cheat solution is well established in the gaming industry and used by various games from different genres, including Rainbow Six Siege, PUBG: Battlegrounds and others. The self-proclaimed “gold standard” of anti-cheat solutions must be installed on the server hosting the game and on all clients playing it, giving it an in-depth overview and control of all in-game events. 
The protection mechanisms of BattlEye are circumventable relatively easily. Thus, many sources reveal this anti-cheat solution's code and inner workings. BattlEye employs an account-based recognition system, which can issue global bans to rid a game of cheaters using multiple accounts.[ https://www.battleye.com/about/https://www.battleye.com/about/] 



  
Easy Anti-Cheat
Easy Anti-Cheat (EAC) is more sophisticated than BattlEye, which makes it harder to reverse-engineer and discover the mechanisms of this solution. This anti-cheat is owned and developed by Epic Games and built into games like Apex Legends, Fortnite or Rust, amongst many others. EAC is free to use and easy to implement in any game for developers, which explains the popularity gained in recent years. Using the hardware ID (HWID) of the system, EAC can ban cheaters globally, increasing the difficulty for players with an unfair advantage to create new accounts and continue their cheating spree. 



  
FACEIT Anti-Cheat
FACEIT Anti-Cheat is a third-party platform for playing games with a custom ranking system, often considered superior to those built into the games. Counter-Strike: Global Offensive is one of the provided games, serving up to 22 million players. It is considered a rather sophisticated anti-cheat system by other reverse engineers in the field [ https://guidedhacking.com/threads/anticheat-faceit-bypass.16113/https://guidedhacking.com/threads/anticheat-faceit-bypass.16113/] and bypassing the security measures it implements is an exceedingly challenging task.[ https://www.faceit.com/enhttps://www.faceit.com/en] 



  
Vanguard
The anti-cheat software Vanguard, developed by Riot Games, is a more intrusive solution. To ensure the successful launch of the protected games, the system must start in a secured state, necessitating the kernel driver to initialise during the boot process. Without this initialisation, the protected games fail to launch. At present, Vanguard is exclusively utilised for the game Valorant, but there are plans to extend its usage to include other games developed by the studio in the future <cit.>.






§ RESULTS

We now turn to analysing anti-cheat solutions for their rootkit-ness based on the metrics defined in <ref>. This involves a detailed examination of four anti-cheat solutions, resulting in their classification based on these metrics.



 §.§ BattlEye

BattlEye consists of four components <cit.>, working together to detect and ban cheaters: 

    
  BEService. BEService is part of the Infrastructure that enables communication to the BattlEye server from the user- and kernel-mode parts installed on the client.
    
  BEDaisy. BEDaisy is the kernel-mode driver, allowing the anti-cheat system to enumerate the memory, scan for vulnerable libraries or insert other modules streamed from the server.
    
  BEClient. BEClient is the user-mode part of the anti-cheat system, injected into the game upon startup, and issues most calls to the driver to execute the various scans.
    
  BEServer. BEServer must be running on the game server the users connect to, where it can monitor user behaviour, receive all notifications from the BattlEye clients (through the BEClient), and issue bans for either the specific server or even globally.
 

These four parts constitute the base of the BattlEye anti-cheat system. Game developers can use the API for the modules in the game code, embedding it deep into the game. This yields monitoring results for in-game player behaviour, making the detection of blatant cheats easy. It interfaces with the kernel driver BEDaisy, which can observe low-level memory areas, check for existing function hooks, etc. To evade tampering from cheats operating at the user level, the anti-cheat system performs scans within the kernel. The client gets the modules streamed from the BEServer, inserting them directly into the memory where they are allocated and executed. 
Reporting players for any violation is implemented in the BEService part of BattlEye via the function c|battleye::report()|, which encapsulates information about the player and the infringement conducted. When a player gets reported, it does not mean a ban is immediately issued. However, more scanning modules will be streamed to the client, gathering additional information about the system. BattlEye can hence be characterised as a relatively lenient anti-cheat system. It primarily issues outright bans only when players are caught cheating by deliberately evading scans and protection measures. The usage of e. g. macros will, at most, lead to being kicked out of the current game or the game not starting.[ https://www.battleye.com/support/faq/https://www.battleye.com/support/faq/]  



  §.§.§ BattlEye Anti-Cheat Measures:

The components of BattlEye use the named pipe text for inter-process communication, encrypting the traffic sent with a bash|XOR| encryption using weak keys, which makes it vulnerable for known-plaintext attacks. This is the primary communication method between the components BEDaisy, BEService and BEClient. 



  
Memory Scans
To detect cheats and anomalies in user behaviour, the kernel driver scans the whole memory area of the protected game and the anti-cheat system, allowing it to check for known cheats loaded. 
Additionally, all external memory pages are examined to determine whether they have the text|X| (execute) bit set. This implies that any process with sufficient privileges can execute the code residing in these pages. These pages can be utilised to inject cheats or a loader into the memory, which subsequently can be used to exploit vulnerabilities in the game or circumvent the anti-cheat system <cit.>. To obstruct cheat developers from using such memory areas, several checks are put in place by BattlEye through memory scans: anomaly scans, pattern scans and module-specific scans. 



    
  Anomaly scans are used to detect areas of memory with address offsets that are not associated with any known loaded image, meaning all injected libraries or other tampering attempts with the memory of the game. In case unknown memory areas are detected, BattlEye sends a report with information concerning the violated memory areas. 
    
  Pattern scans use basic antivirus tactics, comparing loaded modules with known signatures of cheats or methods of injection. Some of these signatures are hardcoded into the BEClient, but most are streamed from the BEServer to the BEClient. BattlEye iterates over the whole game memory and the flagged pages, comparing the streamed signatures with the identified ones. If a known pattern is detected, a report is issued.  
    
  Module-specific scans check the memory for any modules that can be used to tamper with the memory of the game. These modules contain checks for specific DLLs, including text|mmres.dll| and text|mshtml.dll|. 
    Other specific module identifiers, based on in-memory offsets and timestamps, are streamed from the BEServer during the execution of the game, which are most likely used to detect other libraries used for cheat injection <cit.>.
    Additional scans are conducted to inspect other loaded modules, particularly kernel modules. These modules are enumerated by invoking the function c|NtQuerySystemInformation()|, which provides comprehensive details about all the modules loaded into the system. If a module found in the blocklist is detected within the kernel, such as a known vulnerable driver that has been exploited to inject malicious code into the kernel, BattlEye takes immediate action, terminating the game and transmitting a report to the BEServer <cit.>.




  
Page Guard
BattlEye safeguards the memory areas of the game by inspecting memory pages for the presence of the text|PAGE_GUARD| flag. This flag is a memory protection option that can be enabled when allocating or protecting a page in memory. Once set, it designates a guard page that triggers a text|STATUS_GUARD_PAGE| exception when accessed, alerting BattlEye to potential unauthorised or malicious activity targeting the protected memory region. Utilising the text|PAGE_GUARD| flag is beneficial in detecting memory corruption problems or identifying malicious code attempting to write into protected memory regions.[ https://learn.microsoft.com/en-us/windows/win32/memory/memory-protection-constantshttps://learn.microsoft.com/en-us/windows/win32/memory/memory-protection-constants] However, it appears that BattlEye does not rely on this feature in practice, only checking if the protection of the requested memory page was changed <cit.>.



  
Window Enumeration
To detect hooking and, in turn, spoofing of functions in the BattlEye modules, all visible windows on the desktop are enumerated. Overlapping windows are iterated, starting with the highest z-index (corresponding to the uppermost window) and decrementing the index for each window, getting the window handle according to the current one. 
In case fewer than two windows are detected by this method, the anti-cheat assumes its functions were hooked and returned fabricated results, leading to a report <cit.>.



  
Process Enumeration
On each occasion this module is invoked, the anti-cheat system snapshots all currently running processes, checking each for various anomalies. 
Based on a blocklist of applications commonly used to inject cheats, when such a process is found, BattlEye will report the user <cit.>. Apart from scanning for process names, BattlEye also verifies the path of the running process. 






  
LSASS Checks
The Local Security Authority Subsystem Service (LSASS) validates credentials when a user attempts to log in on a Microsoft Windows OS, enforcing the local security policy and issuing security tokens.[ https://learn.microsoft.com/en-us/windows-server/security/windows-authentication/credentials-processes-in-windows-authenticationhttps://learn.microsoft.com/en-us/windows-server/security/windows-authentication/credentials-processes-in-windows-authentication] 

Historically, the Local Security Authority (LSA) has been targeted for infecting systems with viruses, taking advantage of its vulnerabilities. 
Due to the responsibilities of the LSA, BattlEye not only checks but hooks the syscalls issued from and to it, redirecting those calls to the BEDaisy driver, which can monitor every process from the kernel and crash the system when an invalid call is issued <cit.>.




  
Networking
BattlEye conducts system scans to detect all open TCP connections to known cheating sites. These sites are utilised for DRM purposes, which help address the issue of illegal software distribution faced by cheat developers and regular game developers alike. 
When starting the cheat, it must endure such a DRM check by connecting to a server and authenticating the user with valid credentials, ensuring the user paid for the cheats. This connection must often be kept established when the cheat is in use. Open TCP connections are listed in the TCP table of the system and thus are easily readable for the anti-cheat system. A report is generated if an IP address of a known cheating provider is found <cit.>.



  
Tick Scan 
If this test is invoked, BattlEye will schedule the current thread to sleep for exactly one second. Before the sleep command is issued, it obtains the current (game) tick count and compares it with the tick count after sleeping for the second. If the tick delta is more than 1200 ms, it will generate a report <cit.>. This check attempts to detect if an attacker injected code to speed up the game ticks, which could indicate a speed hack, i. e. the player can move faster in-game than intended. Another way this check can be used is to detect if the game is executed inside a virtual machine, which often has more overhead and thus takes longer to execute the same instructions than bare metal <cit.>.



  
Other Methods
There are other checks this anti-cheat system conducts, mostly scans for applications like 7zip or Visual Studio Code. Additionally, BattlEye regularly transmits information to the server, which includes details about all processes, particularly those with active handles on the game. It also provides game file sizes for integrity checks and reports information regarding hooks detected on any process <cit.>. If the BEServer is not certain if a player is cheating, it requests more information by streaming the shellcode to the BEClient, which sends back all information, leading to substantial information exfiltration. 



  §.§.§ BattlEye Self-Protection

BattlEye not only safeguards the intended game but also protects itself. It extends the established practices outlined in <ref> by employing binary virtualisation and integrity checks. The virtualisation of the images is conducted with VMProtect, hardening the reverse-engineering and disassembling process. 
The integrity checks are performed by searching for the file in memory, acquiring a handle and subsequently attempting to open the file. 
This approach allows BattlEye to verify the integrity of the file by confirming its presence in memory and ensuring that it can be successfully accessed. If this operation is successful, BattlEye will get the certificate of the file and compare it against a list of prohibited certificates. In the event of an attempt to load a forbidden file, the anti-cheat system notifies the user regarding the tampering and prohibits the game from loading <cit.>.



  
Uninstalling
Uninstalling the BattlEye anti-cheat system varies for each game, but generally it will uninstall with the game. If not, it can be removed with the script text|Uninstall_BattlEye.bat|, which is installed along with BattlEye. To remove the service created when installing BattlEye, the command text|sc delete BEService| must be executed from an elevated command prompt.[ https://www.battleye.com/support/faq/https://www.battleye.com/support/faq/] 





 §.§ Easy Anti-Cheat

EAC is the anti-cheat solution developed by Epic Games, obtained by the company in 2018. It is provided free to use for game developers.[ https://www.easy.ac/licensing/https://www.easy.ac/licensing/] 
Well-known games using EAC, amongst others, are Apex Legends, Dead by Daylight or Fortnite. When assessing the security features of EAC, it is considerably more challenging to obtain the reverse-engineered source code compared to BattlEye, due to the increased security measures in the software. 








EAC is designed to implement global bans for players who engage in misconduct by banning their corresponding HWID. This prevents cheaters from creating new accounts and accessing the game from the same machine. In contrast to BattlEye, global bans through EAC are restricted to the game the player has cheated in, not extending to other games using EAC. The HWID used to identify the systems of the players is generated from the following identifiers: 


    
  Registry Keys. Keys containing system hardware identifiers, information about the BIOS, the installed graphics card, and others <cit.>.
    
  MAC Address. EAC scans the system for all network interface cards (NICs) using their MAC address in the HWID.[ https://github.com/ch4ncellor/EAC-Reversal/blob/main/hwid.cpphttps://github.com/ch4ncellor/EAC-Reversal/blob/main/hwid.cpp] 
    
  Disk Serial Numbers. Every single disk possesses a unique serial number to identify it. EAC incorporates the unique serial numbers of all installed disks in the system as part of the HWID.[footnote]
    
    
    
  WMI Queries. The Windows Management Instrumentation (WMI) is used by EAC to query specific identifiers from the actual hardware via the text|Win32_BaseBoard| class. EAC utilises identifiers from the motherboard and the GPU.[footnote]
    
  Processor Features. CPUs do not have unique identifiers, just the model number and the given feature set. 
    These metrics are included in the generation of the HWID.[footnote]


While the HWID generation incorporates the metrics mentioned earlier (and possibly others), it is still technically feasible to spoof all of these values. However, this requires significant effort and expertise to do successfully. This system is hence not flawless, but given the dedication it takes to fabricate these values and the secrecy about them from the official publishers, it is deemed good enough to issue effective global bans.



  §.§.§ EAC Anti-Cheat Measures

With an understanding of the generation of the HWID used for banning players with an unfair advantage, it is possible to delve into the mechanisms EAC employs to detect cheating players. EAC consists of several parts cooperating to protect the game against cheaters. Each time a game protected by EAC is launched, it initiates the download of an up-to-date package that includes the driver module and the game-specific module from the EAC content delivery network. This ensures that the necessary components for EAC's operation, including the latest driver and game module versions, are obtained before the game is executed. Meanwhile, the EAC executable is launched, opening a shared buffer that allows communication between the kernel driver and the user-mode application. The kernel driver is used to monitor the system for various malicious changes. Furthermore, the text|EasyAntiCheat.dll| is injected into the game, enabling EAC to inspect all changes and input registered in the game <cit.>. The methods employed by EAC to detect cheaters are the following:



  
Hook Detection and Blocking
The kernel driver of EAC can detect various hooking techniques, alerting the anti-cheat system when such a hook is detected outside the memory of the game. If a malicious hook is identified, the kernel driver will automatically block this hook and communicate via the shared buffer to generate a report. The purpose of implementing these measures is to impede cheaters from injecting malicious code into the game and to prevent the creation of process handles that could be exploited by external tools to interact with the game <cit.>.



  
Memory Scans
Like BattlEye, EAC scans the memory of the game, including adjacent memory areas, to detect loaded cheats. The techniques used are very similar to those BattlEye employs <cit.>. 



  
Driver Scanning and Logging
The EAC kernel driver performs scans on the system to identify and block any loaded drivers that may be considered malicious. Specific known drivers it scans for are text|Dbgv.sys| and text|PROCMON23.sys|, two drivers of the Sysinternals Suite, as well as text|dbk64.sys|, the driver for the tool Cheat Engine.

    
    
    

In addition to scanning and blocking suspicious drivers, EAC generates detailed logs that capture information about loaded drivers and loading attempts within the system. These logs are forwarded from the kernel driver to the service executed in user mode, encrypted, and then sent to the EAC servers.



  
Stack Walking
EAC applies stack walking to enhance hook detection, a technique to trace the origin of a method call in memory. 
Using the two functions c|RtlLookupFunctionEntry()| and c|RtlVirtualUnwind()|, it is possible to obtain the base address of the image hooking the application <cit.>.



  
Hypervisor Detection
EAC uses the feature set of CPUs to detect if it is executed in a virtualised environment or on an OS installed on bare metal, accomplished by executing a single asm|vmread| instruction upon starting. This instruction attempts to read a field from the virtual machine control structure <cit.>. By verifying the successful execution of the asm|vmread| instruction without triggering an exception, EAC can determine if it is running inside a virtual machine. If the instruction executes without issues, EAC blocks the game from launching <cit.>. In addition to the VM test, EAC employs the same tactic as BattlEye, sleeping for one second and measuring the game ticks happening in between, as already described in <ref> <cit.>.



  
Window Enumeration
Like BattlEye (see <ref>), EAC employs a technique to enumerate all visible windows during runtime, obtaining the handle and the window name. If a window is detected containing a name corresponding to any cheat, a report is generated <cit.>.



  
Manual Mapping Detection
Manual mapping describes loading a DLL into memory using custom tools instead of Windows API functions. This technique offers the benefit of total control over the location where the library is loaded, circumventing automatic detections from both the OS and the applications involved <cit.>. 
EAC employs a combined approach of memory scanning to identify manually mapped modules or drivers. The routine begins with scanning processes and threads, during which the associated image bases are saved. Subsequently, the captured base addresses in the virtual memory space are examined for anomalies, such as Portable Executable headers indicating the presence of injected applications within the memory area. Furthermore, like BattlEye, the system verifies adjacent memory regions with the execute bit set, actively searching for any non-module code present <cit.>. If any modules are detected in the areas outside of the game not associated with a module, the anti-cheat system will dump all strings and syscall invocations found in the section and send them to the developers in an encrypted form.



  
Handles and Threads
EAC obtains a list of all open handles on the system, using them to search for any process accessing the game without authorisation. If such a process can be located, the user is flagged. Furthermore, handles pointing to sections in the physical memory are probed to check if any process writes code directly into memory which could be executed in the context of the game or the anti-cheat system <cit.>. Additionally, EAC scans for all threads running in the kernel or user mode and compares them with defined detections for suspicious threads. If those threads are called from an executable memory region not associated with any loaded module of the EAC system, it generates a report <cit.>.



  
Instrumentation Callbacks
Instrumentation callbacks enable simple syscall monitors from user mode in EAC. Whenever a syscall is executed and the kernel returns to user mode, the anti-cheat system checks for the address of the cpp|InstrumentationCallback|, which is stored within the cpp|KPROCESS| structure. If this address points to a valid memory space, the kernel swaps the return address of the function for the one inside the cpp|InstrumentationCallback| member. The substitution of the return address during the transition from kernel to user mode in EAC occurs only when the asm|dr7| register is set, triggering a detour. Without setting this register, the return address remains unchanged, and the transition from kernel to user mode proceeds normally without any modifications. When a user executes a game with EAC enabled, the kernel driver creates an instrumentation callback routine that monitors the system calls and other events made from the game.[ https://www.unknowncheats.me/forum/anti-cheat-bypass/561479-eacs-instrumentation-callback-bypass.htmlhttps://www.unknowncheats.me/forum/anti-cheat-bypass/561479-eacs-instrumentation-callback-bypass.html] In this callback function, EAC can scan for any cheating attempts by using techniques like stack walking to find the module that executed the syscall in the first place. 



  
Game Integrity Checks
During runtime, EAC employs integrity validation by downloading a catalogue of valid signatures encompassing all game files. This catalogue is used to verify the integrity of game files whenever the game accesses them. Furthermore, the configuration containing the definitions of files to probe is also signed and checked at runtime so that a malicious actor cannot tamper with it. If files experience prohibited modifications mismatching these signatures, EAC restricts the user from playing online games, but allows offline gaming.[ https://dev.epicgames.com/docs/game-services/anti-cheat/using-anti-cheathttps://dev.epicgames.com/docs/game-services/anti-cheat/using-anti-cheat] 
In addition to its existing detection mechanisms, the EAC driver incorporates checks that prevent the simultaneous execution of games protected by EAC and those protected by BattlEye. Specifically, if the BattlEye kernel driver BEDaisy.sys (as described in <ref>) is detected to be running on the system, the game protected by EAC will not start <cit.>. 



  §.§.§ Easy Anti-Cheat Self-Protection

Like BattlEye, EAC employs self-protecting techniques as well. There is little public knowledge about this topic, but it is generally accepted that these measures are similar to those of BattlEye. EAC also utilises a proprietary virtual machine obfuscator, which conceals the binaries and executes them on non-standard architecture. In order to perform self-checks, EAC employs the use of the virtual asm|vmcall| instruction. This instruction allows EAC to compare the loaded driver in memory with a clean copy that was read from the disk before the memory mapping process. If the anti-cheat system detects any discrepancies in the integrity of the code, it deliberately crashes the protected application by attempting to access the invalid memory area text|0xEAC| <cit.>.



  
Uninstalling
Uninstalling Easy Anti-Cheat from the system happens automatically when the last game using this anti-cheat solution is uninstalled. If remnants remain, EAC can be manually uninstalled by running the installer and selecting the uninstall option. This rids the system of all residues of EAC.[ https://www.easy.ac/en-us/support/game/guides/installer/https://www.easy.ac/en-us/support/game/guides/installer/] 





 §.§ FACEIT Anti-Cheat

FACEIT Anti-Cheat is a third-party service for the games Counter-Strike: Global Offensive, Dota 2 and Team Fortress 2, amongst others. It offers ranked leagues with better anti-cheat capabilities than the built-in matchmaking systems in the mentioned games. FACEIT Anti-Cheat uses a combined approach to detect cheating players by collecting data from the locally installed FACEIT Anti-Cheat client and behavioural analysis of players recorded on the server. The latter system is based on the FACEIT Anti-Cheat servers and thus will not be analysed in this article. This approach allows for comparing multiple data sources, making it harder for players to cheat in such matches. 

FACEIT Anti-Cheat heavily restricts the usage of the anti-cheat client with regard to the platform the game is played on and bans players for attempts to circumvent these measures, considering them cheating. The terms of service of FACEIT Anti-Cheat explicitly prohibit the installation of the client, including the anti-cheat system, on a virtual machine. Engaging in such behaviour is considered a violation resulting in a ban from the platform. The attempt to debug or decompile any part of the anti-cheat solution is also considered cheating.[ https://support.faceit.com/hc/en-us/articles/360015788779-What-is-deemed-to-be-a-cheathttps://support.faceit.com/hc/en-us/articles/360015788779-What-is-deemed-to-be-a-cheat] 
FACEIT Anti-Cheat put these measures in place to protect the anti-cheat from malicious individuals attempting to detect exploits. 



  §.§.§ FACEIT Anti-Cheat Measures

Consisting of a kernel driver (text|FACEIT.sys|), the FACEIT Anti-Cheat client and the server-side evaluation, FACEIT Anti-Cheat offers a sophisticated framework to protect the offered games against cheaters. The kernel driver utilises similar anti-cheat tactics as previously discussed anti-cheat systems.[ https://support.faceit.com/hc/en-us/articles/9394666828188-What-is-FACEIT-Anti-cheat-and-how-does-it-workhttps://support.faceit.com/hc/en-us/articles/9394666828188-What-is-FACEIT-Anti-cheat-and-how-does-it-work] The FACEIT Anti-Cheat driver boots with the operating system to ensure the system is in a trusted state. The FACEIT Anti-Cheat kernel driver is designed to constantly collect data about the system, including other running drivers and executed programs, transmitting it to the server when the client starts to detect any illegal behaviour indicators. Known FACEIT Anti-Cheat tactics on the client and the server side are as follows:



  
Instrumentation Callbacks
FACEIT Anti-Cheat places instrumentation callbacks on several Windows kernel functions. The anti-cheat system blocks the loading of external modules and DLLs, logs drivers and general loading events via a callback on the c|LoadImage()| function. Further callbacks are placed on the c|CreateProcess()| and c|CreateThread()| functions, including all variantions, which notify the anti-cheat system about created processes or threads for logging and reporting <cit.>.



  
Driver Detection and Stripping
The anti-cheat system has built-in detection and unloading capabilities for vulnerable drivers that could inject malicious code for cheats (similar to Vanguard, see <ref>). Additionally, mechanisms are in place to locate and close open handles to the protected game to ensure no other application is interfering <cit.>.



  
Virtualisation Detection
Due to the strict approach against installation on virtual machines, a detection mechanism against this is in place. FACEIT Anti-Cheat blocks execution with the Windows built-in virtualisation solution Hyper-V enabled, requiring the players to execute custom commands to fully disable it. 
According to FACEIT, it is impossible to “maintain a fair playing environment” with this feature enabled.[ https://support.faceit.com/hc/en-us/articles/360019809319-You-need-to-disable-Hyper-V-to-launch-FACEIT-AChttps://support.faceit.com/hc/en-us/articles/360019809319-You-need-to-disable-Hyper-V-to-launch-FACEIT-AC] 
They go so far as to instruct players to disable the memory integrity feature of the Windows kernel (which considerably weakens the system by dropping the special protection for high-security processes in memory) so they can better monitor processes like LSASS and CSRSS.



  
Virtualised Binaries
FACEIT Anti-Cheat and the client utilise VMProtect for binary virtualisation and commercial packers for further obfuscation, similar to all the previous anti-cheat systems <cit.>.









  
System Security
The FACEIT Anti-Cheat system requires the active use of a Trusted Platform Module to store keys to encrypt the data sent to the servers.[ https://support.faceit.com/hc/en-us/articles/4407006362002-Error-Please-enable-TPM-2-0-to-continuehttps://support.faceit.com/hc/en-us/articles/4407006362002-Error-Please-enable-TPM-2-0-to-continue] 
Furthermore, it is necessary to enable the Data Execution Prevention feature to enhance protection in memory, so pages cannot be written to and executed at the same time, blocking tactics to inject cheats into the memory.[ https://support.faceit.com/hc/en-us/articles/360017192040-You-need-to-enable-DEP-to-launch-FACEIT-AChttps://support.faceit.com/hc/en-us/articles/360017192040-You-need-to-enable-DEP-to-launch-FACEIT-AC] 
In addition to those requirements, FACEIT Anti-Cheat needs the system to be booted with Secure Boot ensuring the integrity of the system beginning with the UEFI. This often requires changes to the UEFI and reformatting of system disks initialised with text|MBR| instead of text|GPT|.[https://support.faceit.com/hc/en-us/articles/4406281700370-Secure-Boot-needs-to-be-enabled-to-launch-FACEIT-AChttps://support.faceit.com/hc/en-us/articles/4406281700370-Secure-Boot-needs-to-be-enabled-to-launch-FACEIT-AC] 

Due to FACEIT Anti-Cheat being very secretive about the anti-cheat system and discouraging system analysis, very little data is available. Due to this, it can be assumed that there are additional detection and monitoring features included in the anti-cheat system which are not covered above. 
If players are detected to be cheating, the system flags them for the future, but does not issue instant bans to avoid letting cheaters know which specific action triggered the system. Such flagged accounts are typically banned within one week after the incident. 



To obstruct cheaters from creating new accounts, FACEIT Anti-Cheat uses HWID bans. However, it is worth noting that the HWID generation method used by FACEIT Anti-Cheat is considered weaker than other anti-cheat solutions like Vanguard, as discussed in <ref>. Only two known values are included in the HWID of FACEIT Anti-Cheat: The MAC addresses of the local NICs and the serial number of the first disk in the system. Using these relatively simple indicators, the FACEIT Anti-Cheat kernel driver continuously monitors any changes and spoofing attempts made to those values and notifies the server when the client is executed.



  
Uninstalling
FACEIT Anti-Cheat offers no clear instructions for removing the client or the anti-cheat system. Both can be removed via the default Windows uninstalling menu, but there is no guarantee that no traces of the applications will remain on the system. Furthermore, no information is available to revert the changes needed to start the anti-cheat system in the first place, making the whole process very intransparent.    









 §.§ Vanguard

Vanguard, developed by Riot Games specifically for the game Valorant, is widely recognised as one of the most intrusive anti-cheat systems. [ https://www.gamechampions.com/en/blog/valorant-anti-cheat-vanguard/https://www.gamechampions.com/en/blog/valorant-anti-cheat-vanguard/] It operates upon system boot, ensuring the system starts in a trusted state. Vanguard comprises three distinct components: 










    
  text|vgk.sys| is the kernel driver of the anti-cheat system, creating a protected memory area for the user mode part of Vanguard and the game itself. It features user-mode and self-integrity checks and communicates with the user-mode part of Vanguard, enabling command execution from there, including remote update and uninstall methods <cit.>.
    
  text|vgc.exe| is the user-mode counterpart to the kernel driver, issuing the commands to execute various scans. It ensures the integrity of the game and the reporting functionality to the game servers. Direct communication with Valorant is also implemented <cit.>.
    
  text|Valorant.exe| is the game protected by Vanguard. It is executed in the protected memory area created by the kernel driver and continuously checked by the user-mode anti-cheat part of the system <cit.>.


Like the previous anti-cheat solutions, Vanguard utilises the HWID for player identification and banning. Riot Games has taken significant measures to protect the details and inner workings of Vanguard. Publicly available information about the anti-cheat system is limited due to the bug bounty program the company offers. 
While the specific details of Vanguard's HWID generation process are not publicly disclosed, certain values are commonly assumed to be used in the generation of the HWID. These values, although not confirmed by Riot Games, are often speculated to include:
 
    
  Disk Serials. Similar to EAC (see <ref>), it is believed that Vanguard incorporates the serial numbers and volume identifiers of the disks present in the system as part of its HWID generation process <cit.>.
    
  Registry Keys. In case the registry key text|HKLM text|MicrosoftNT| is available, it is speculated that Vanguard includes the value of this key in the HWID generation process. The BuildGUIDEx registry key typically contains a unique identifier associated with the specific build of the Windows OS installed <cit.>.
    
  NVRAM Variables. Vanguard reportedly incorporates specific information about the platform the system is running on in the HWID generation process. According to <cit.>, this information is sourced from the PlatformData variable, which is said to include a collection of variables set by various devices installed in the system. However, it is important to note that the PlatformData variable itself is not defined in the official UEFI specification <cit.>.
    
  MAC Addresses. Like EAC, Vanguard is believed to use the MAC addresses of local NICs in the HWID. 
    
  Mainboard UUID. It is also considered plausible that Vanguard leverages the unique identifier stored in the BIOS or UEFI NVRAM of each mainboard during the HWID generation process <cit.>. 


Valorant is free-to-play, thus in principle easily allowing cheaters to create new accounts once the previous ones were banned for violations. This emphasises the need for HWID solutions because it is the sole indicator that can identify a system that was cheated on with any certainty.



  §.§.§ Vanguard Anti-Cheat Measures

Vanguard employs several anti-cheat measures already discussed in detail for EAC, such as instrumentation callbacks or various types of memory scans. To avoid needless repetition, we will focus on measures specific to Vanguard in this subsection.




  
Shadow Memory
Vanguard uses a sophisticated method to hide memory pages from other processes and threads executed on the system, protecting them from unauthorised access. This is implemented by hooking the c|SwapContext()| ioctl function of the Windows kernel, called every time a context switch occurs. By hooking this function, denying access to threads not exclusively allowed is possible. Furthermore, Vanguard uses custom memory addresses in the code based on the shadow memory regions. A crash would occur if any other application attempted to access such a memory address except Vanguard, due to catching occurring page-faults within. 
Resulting from this refined approach, Vanguard is well protected against this kind of cheats by making it very difficult to manipulate the memory of the game <cit.>.



  
Virtualisation
Vanguard and the associated binaries are virtualised and compressed with the proprietary software packer Packman, developed by Riot Games.
The packer is utilised to obfuscate and encrypt the data. The essential functions of Vanguard are decrypted at launch, meaning that not all functions are loaded into memory simultaneously. This approach impedes attackers and security researchers from easily understanding and analysing the code <cit.>.







  
Remote Controllability
Vanguard includes features that allow developers to remotely disable and uninstall the anti-cheat system in the event of discovering severe vulnerabilities <cit.>. If the feature that enables remote disabling and uninstallation of Vanguard were to be hijacked or exploited by attackers, it poses a severe security risk, granting attackers access to the kernel.



  
Limitations
To enhance the effectiveness of Vanguard, the game is designed not to start if the kernel driver of the anti-cheat system has not been loaded during the boot process. This approach establishes a secure and controlled environment where the anti-cheat system can operate without interference or circumvention attempts. By ensuring that the kernel driver is loaded before the game starts, Valorant can prevent other drivers or unauthorised software from being loaded beforehand. If any of those drivers must be loaded when the system is started in the trusted mode, it can be turned off from user mode. This results in the system falling into an untrusted state, which requires a reboot to re-enable the anti-cheat driver. Valorant can only be started when the system is in a trusted state.



  
Uninstalling
Uninstalling Vanguard is straightforward – searching for it in the installed applications and selecting uninstall. This removes the files of the anti-cheat system; a reboot is required to unload it from memory, as well <cit.>.





 §.§ Discussion

We now discuss the findings based on our analysis in <ref>. While no anti-cheat system shows signs of network manipulation, we discovered evidence for all other metrics in at least one of the systems analysed. A summary of the evaluation for the different anti-cheat systems against our metrics is shown in <ref>, including a comparison with Flame <cit.>, a rootkit detected by Kaspersky Lab in the Middle East in 2012.

In the category of evasion, Vanguard and FACEIT Anti-Cheat indicated significant traces. Vanguard employs some very intrusive tactics, including evasion tactics circumventing other threads and processes by implementing a custom memory management system, hiding from processes and threads, while FACEIT Anti-Cheat employs vigorous checks for virtualised environments, which is a tactic used by malware and rootkits alike to protect themselves against analysis. While this behaviour was introduced to stop cheaters from tampering with the virtual memory of the game, banning players for merely attempting to play in a VM could be considered unreasonable. Furthermore, the constraint to disable Hyper-V (blocking mundane programs like WSL-2) to ensure no virtualisation can occur at all strengthens the evasion tactics. These combined approaches each indicate rootkit-like behaviour.

All of the analysed anti-cheat systems tested positive in the category of virtualisation. All systems employ some kind of virtualisation, each in different ways. BattlEye uses VMProtect to virtualise the binaries. Similarly, EAC uses custom software to virtualise the binaries as described in <ref> for self-protection. FACEIT Anti-Cheat employs heavy obfuscation tactics, as well. Lastly, Vanguard utilises a proprietary packer, including binary encryption by design, which hardens Vanguard against static and dynamic analysis. These methods make it challenging to reverse-engineer and assess them for any security concerns.

Similar to the category of evasion, FACEIT Anti-Cheat and Vanguard are the only two systems designed to execute when the system boots, leading to a positive result for the time of execution metric. FACEIT Anti-Cheat and Vanguard both require the kernel driver to boot with the operating system to check for any other vulnerable drivers to block, ensuring no cheats can be loaded on the system. Furthermore, FACEIT Anti-Cheat requires Secure Boot to be enabled, so no disallowed drivers or applications can be loaded before the OS starts. This design is the most intrusive of all anti-cheat systems. The other two kernel-level anti-cheat systems only start when the protected game is launched, thus posing no threat in this category. 

Concerning remote access methods, traces could only be located in the binary of BattlEye, due to it sending a large amount of information to the BEServer backend. Although the other solutions show traces of such behaviour (e. g. Vanguard includes features for the developers to disable or uninstall the system remotely in case severe vulnerabilities are discovered), there is no concrete evidence of real remote access and controllability built into the other anti-cheat solution.

Regarding information exfiltration, all of the kernel-level anti-cheat systems indicated positive results for this metric. BattlEye sends a lot of information about the system and the game to the BEServer backend, while EAC utilises various system identifiers to generate the HWID for identification purposes. Additionally, it logs all drivers and modules in memory and sends this information to the server for analysis and monitoring. This information is continuously transmitted to the server, regardless of whether a player is suspected of cheating. If this data, designed to identify a system uniquely, can be accessed by unauthorised people, tracking or impersonating victims on various game servers is possible. Both FACEIT Anti-Cheat and Vanguard employ reporting methods to send system information, such as the HWID, loaded drivers, modules and crash reports, to the game developers for analysis as well, furthermore heavily relying on analysing logs (including loaded drivers, process and thread events, etc.), which are sent from the client to the server. 

Removing the anti-cheat solutions from the system is elementary for most analysed systems except FACEIT Anti-Cheat. Due to the lack of guides describing its proper removal, there is a chance that remnants of it remain active on the disk after removal attempts, thus possibly acting like a rootkit. Also, no additional guides are available for restoring the system to a state before using the software, leaving it in a possible vulnerable state, especially considering the requirement of disabling the memory integrity feature.





§ CONCLUSION

This article analyses the behaviour and tactics of four well-known kernel-level anti-cheat systems widely used in online gaming and compares them to functionalities and methods used in rootkits. The anti-cheat systems BattlEye and Easy Anti-Cheat showed minor similarities to rootkits, which were insufficient to classify them as such according to our metrics. FACEIT Anti-Cheat and Vanguard, however, were identified as rootkit-like applications due to the utilisation of comparable methods. The findings and insights from this research shed light on the thin line between what distinguishes kernel-level anti-cheat systems from rootkits.

The classification of anti-cheat systems as either rootkit-like or not presents significant challenges. We claim that the two anti-cheat systems we classify as rootkits raise severe concerns regarding their potential for malicious abuse and the associated privacy implications. However, although we do not classify the other two systems as rootkit-like, they nonetheless engage in practices that raise questions regarding their appropriateness and integrity.



While our work provides some first answers, it also highlights the need for further research on this topic to gain a more comprehensive understanding of different anti-cheat systems, their tactics, and how to best reconcile the need for robust anti-cheat solutions with respecting user privacy and system integrity. In particular, this article exclusively focussed on anti-cheat systems available on Microsoft Windows, disregarding other OSes. We thus see the need for future research into anti-cheat systems on other operating systems (e. g. EAC on Proton for Linux) comparing the applied methods to their Windows counterparts.









    We are grateful to Tobias Dam, Patrick Kochberger, Robert Luh, Martin Schmiedecker and the anonymous referees for suggesting numerous improvements to both the content and the presentation of this paper.





ACM-Reference-Format




 Paper Title: A Zero-Knowledge Proof of Knowledge for Subgroup Distance Problem.









A Zero-Knowledge Proof of Knowledge for Subgroup Distance Problem
    Cansu Betin Onur

C. Betin Onur is partially support by NGI.

    August, 2024
=================================================================




§ ABSTRACT 

In this study, we introduce a novel zero-knowledge identification scheme based on the hardness of the subgroup distance problem in the Hamming metric. The proposed protocol, named Subgroup Distance Zero Knowledge Proof (SDZKP), employs a cryptographically secure pseudorandom number generator to mask secrets and utilizes a Stern-type algorithm to ensure robust security properties. 



Zero-knowledge proofs, Subgroup Distance Problem, Hamming Distance







§ INTRODUCTION



Zero-knowledge proofs (ZKPs) are cryptographic protocols that enable one party (the prover) to convince 
another party (the verifier) to the truth of a statement without disclosing any additional information beyond the 
 fact that the statement is true <cit.>. Since their inception, ZKPs have become a fundamental tool in cryptographic research and applications, providing a robust method for secure authentication and privacy-preserving computations. Various identification schemes leveraging ZKPs have been proposed, each relying on different hard mathematical problems to ensure security.
 One of the pioneering works in this area is Stern's identification scheme, introduced in 1993, which is based on the hardness of the Syndrome Decoding (SD) problem. Stern's protocol laid the groundwork for code-based cryptographic systems, renowned for their resilience to structural attacks and potential for quantum security through the Fiat-Shamir transform. Over the years, numerous enhancements and variations of Stern's protocol have been developed, focusing on improving efficiency and reducing soundness errors.

In this study, we introduce a zero-knowledge identification scheme that relies on the Subgroup Distance Problem (SDP) in the Hamming metric, a problem known for its computational hardness in various metrics and its NP-completeness under certain conditions. The proposed protocol, named Subgroup Distance Zero Knowledge Proof (SDZKP), differs from existing schemes by leveraging the SDP, thus providing a fresh approach to zero-knowledge identification. By converting the confidential information held by the prover into integer tuples and employing a Stern-type algorithm, we ensure that our protocol inherits the robustness and security features of code-based systems.


SDZKP differs from other identification schemes in the literature in terms of the mathematical hard problem on which it is based. However, the confidential information held by the prover is converted into an integer tuple and a Stern type algorithm is executed in SDZKP. For this reason, we place our study in association with code-based protocols. The SDZKP protocol is designed to be both secure and efficient, making use of a cryptographically secure pseudorandom number generator (CSPRNG) for masking secrets. The protocol follows a three-step challenge-response structure, ensuring that the prover can convince the verifier of their knowledge of a secret without revealing any information about it. Our security analysis demonstrates that SDZKP achieves perfect completeness, 3-special-soundness, and statistical zero-knowledge, providing strong guarantees against potential attacks.


In 1993, Stern introduced a code-based identification scheme based on the hardness of the Syndrome Decoding (SD) problem<cit.>. His work became a foundational work on code-based cryptography. One major feature that makes Stern scheme attractive is its robustness to structural attacks. In other words, possible attacks appears only to the underlying hard problem. Using Fiat-Shamir transform, Stern protocol can be converted to a quantum secure signature scheme. 
 Various studies have been published focusing on improvement on efficiency and security of Stern protocol. These works are named as Stern-type protocols.  First improvement given by 
 Véron <cit.> using General Syndrome Decoding problem and reduced the number of rounds required.
 Both protocols have 2/3 soundness error. In 2011, Cayrel et. al. <cit.> and Aguilar et. al.  <cit.> reduced Stern and Véron protocols soundness error respectively up to 1/2 using 5-round protocols. In 2022, Bidoux et. al. <cit.> introduced an adaptation of the protocol given by Aguilar, Gaborit and Schrek (AGS) <cit.> on quasi-cyclic SD problem. Also in <cit.> a zero knowledge protocol achieving soundness error 1/n for arbitrary chosen n is presented under the constrain that the verifier trust some of the variables sent by the prover.






This paper is organized as follows: Section <ref> covers the necessary preliminaries, including basic cryptographic definitions and an overview of the Subgroup Distance Problem. In Section <ref>, we present the detailed design of the SDZKP protocol. Section <ref> provides a comprehensive security analysis. Finally, we conclude the paper highlighting the contributions and potential future directions for research.










§ PRELIMINARIES



For any positive integer m the set {1, … m  } is denoted by [m]. The group of permutations on the set [n] is denoted by S_n. For a finite set A, a $← A denotes that a is taken uniformly at random form A.
The term “probabilistic polynomial time” is abbreviated by PPT. On inputs in_P, in_V respectively, the transcript of two parties P and V is denoted by View(⟨ P(in_P), V(in_V)⟩) and an execution between P and V giving output out is denoted by ⟨ P(in_P),V(in_V)⟩→ out.
  As the subgroup distance problem is defined on the symmetric group S_n, the security parameter of the given scheme is n. 



 §.§ Subgroup Distance Problem (SDP)
 
Given a metric d on the Symmetric group S_n . The distance of a permutation α∈ S_n to a subgroup H ≤ S_n is defined as 

    d(α, H)= h ∈ Hmin d(α, h)




(Subgroup Distance Problem (SDP))
Given a set of elements { g, h_1, … h_m } from S_n and given an integer k. Decide whether the distance between g and the subgroup H=⟨ h_1, … h_m ⟩ is at most k.
    

We would like to draw readers attention to similarity of SDP problem and one of the closest vector problem (CVP) on integer lattices which is considered to be one of the hard problems for post-quantum cryptographic protocols. Roughly speaking, in both problems one is asked to decide (or search) existance of an element from a given subset close enough to given fixed element. 

 The computational complexity of SDP has been analyzed through various of metrics.  In 2006, Pinch showed that  SDP is NP-Complete<cit.> with respect to Cayley Distance. Subsequently, in 2009, Buchheim et al. extended this result to other metrics such as Hamming, Kendall's tau, l_p, Lee's and Ulam's distances. Moreover, even when the subgroup is restricted to be an Abelian group of exponent two, the problem remains NP-complete across all these metrics. 




The  Hamming distance of given two permutations α, β in S_n is defined as the number of different entries of α and β. 

    d(α,β)=|{ i |  α(i) ≠β(i)   } |.
     





 §.§ Basic Cryptographic Definitions



A function negl: ℕ→ℝ_≥ 0 is called negligible function if for any natural number c there exists a lower bound n_0 such that negl(n)<1/n^c for all n ≥ n_0.     


If the probability of a situation occurring is 1-negl(n) for some negligible function negl, we say that the situation appears with overwhelming probability.





    For security parameter n, two distribution ensembles D_n, E_n are said to be 
    computationally indistinguishable 
    if for any PPT algorithm A,
    the value of the difference  
    | x← D_nPr[A(x)=1] - x ← E_nPr[A(x)=1] |
 is a negligable function.

If the condition is true even if A is allowed to be unbounded, then these distributions are said to be statistically indistinguishable. 





Next we define commitment schemes which are  effective building blocks frequently used in ZKP design. A commitment scheme should satisfy three basic properties: correctness, binding and hiding properties. We give the definitions of these properties consequently after the definition of commitment schemes.


 A commitment scheme is a polynomial time algorithm triple Com=(Setup,Commit, Ver)  satisfying correctness, binding and hiding properties. The components of Com are described as below:
 
     
  * Setup: On input 1^n, it outputs public parameters PP determining 
    the message, the randomness, the commitment and the opening spaces. 
    Notation: PP ← Setup(1^n).
  
  * Commit: On input PP and a message m, 
it outputs a commitment-opening pair formally represented as (c,o) ← Commit(PP,m). 
            
  * Ver: On input sequence (PP,c,m,o) it outputs a bit b. 
       The case b=1 refers to c is a valid commitment for m and Ver accepts. In the case b=0, the committed value fails and Ver rejects it.  Formally, we represent Ver as b← Ver(PP,c,m,o).
 



It is often that commitment algorithm triples take also some randomness r as input. It is implicitly used in the above given definitions.


    A commitment scheme Com=(Setup,Commit, Ver ) satisfies the following properties:
    

    
  * Correctness if

    Pr[1← Ver(PP,c,m,o)
    :
    (c,o) ← Commit(pp,m)
    ]=1



  * Computationally (resp. Statistically) Binding if there exists a negligible function neg(n) such that for every PPT (resp. unbounded) algorithm A below inequality holds:




    Pr[
    [       pp ← Setup(1^n); (c,m,m',o,o') ← A(pp) ]
    :
    [             m ≠ m';   1← Ver(PP,c,m,o); 1← Ver(PP,c,m',o');                    ]] ≤ neg(n)


    
  * Computationally (resp. Statistically) Hiding  if for any two massages m,m' and pp ← Setup(1^n) the  distributions Commit(pp,m) and Commit(pp,m') 
 are computationally (resp. statistically) indistinguishable.
   



Next we define interactive protocols and zero-knowledge proof of knowledge. Informally, a zero-knowledge proof of knowledge is a two party protocol between a prover P and a verifier V such that  P convinces V that it has the knowledge of a desired secret without revealing any information about the secret. Let us put this in more symbolic language.

Let R be an NP-relation and L be the language corresponding to R. That is L = { x  |  ∃ w : R(x, w) = 1 }. In a zero-knowledge proof of knowledge protocol, for a common input value x, the prover P convinces V that it knows a witness w such that R(x, w) = 1.



A zero-knowledge proof of knowledge protocol for language L with respect to a relation R is a protocol between a pair of interactive machines P and V named prover and verifier where P is computationally unbounded and V is probabilistic polynomial-time. It should satisfy the following conditions:



  * Completeness:  For every x ∈ L, verifier V always accepts after interacting with a prover P having a witness w.

 
    Pr[⟨ P(x,w), V(x)⟩→ 1] = 1




  * Proof of Knowledge (with error ϵ):  For every possibly cheating T-time PPT prover P^* with Pr[⟨ P^*,V(x)⟩→ 1] > ϵ + e there exists a PPT algorithm K  (with running time polynomial in 1/e and T) such that; given rewindable black-box access to P^*, on input x the algorithm K outputs a w' such that R(x, w') = 1 with success probability at least 1/2. Here K is called the knowledge extractor. 



  * Zero Knowledge:  For every possibly cheating PPT verifier V^*, there exists a PPT algorithm S, called simulator, such that on input x it outputs a transcript S(x) which is indistinguishable from View(⟨ P(x,w), V^*(x)⟩ ). The zero knowledge property is called computational, statistical or perfect zero knowledge depending on whether the two distributions are  computationally indistinguishable, statistical indistinguishable  or equal respectively.




Under the restriction that P is a PPT algorithm in above definition, the protocol refereed as "argument" instead of "proof."   i.e. we define zero knowledge argument of knowledge (ZKAoK).
A zero knowledge proof of knowledge is specified as honest verifier zero knowledge proof(or argument) of knowledge if the existence of a PPT simulator S giving output with indistinguishable distribution from View(⟨ P(x, w), V(x)⟩ ) guarantied only for the honest verifier V. 



A stronger notion of knowledge soundness is (two)special-soundness. Here we give the definition of special-soundness for more generic case. Consider 3-move protocols such that the rounds starts with the prover's move. The moves are named as commitment, challenge and response respectively. The transcripts are denoted by (C,Ch,Rsp) in the sequel.



    A 3-round protocol is said to have k-special-soundness property if there exists a PPT algorithm K such that for any given k distinct excepted transcripts for the same commitment C, say (C,Ch_1,Rsp_1), …, (C,Ch_k,Rsp_k), the algorithm K outputs a valid witness w. 
 

Under the assumption that k=poly(x) for some polynomial, k-special-soundness strictly implies  knowledge soundness by a generic reduction with soundness error ϵ = (k - 1)/N , where N is the cardinality of challenge space <cit.>. 


    A sigma (Σ) protocols is a 3-round honest-verifier zero-knowledge proof of knowledge protocol satisfying k-special-soundness.






§ SUBGROUP DISTANCE ZERO KNOWLEDGE PROOF (SDZKP)



In this section, we present Subgroup Distance Zero Knowledge Proof (SDZKP) in steps and illustrate it in Figure <ref>.




 §.§ Setup



In this section, under the assumption that the subgroup distance problem in Hamming metric is hard for parameters k,n and the subgroup H determined by the generators h_1, …, h_m; we introduce a
zero knowledge identification scheme. 

The integers k,n and a set of elements { g, h_1, … h_m } from the Symmetric group S_n are assumed to be publicly known. The prover claims that it knows an element h ∈ H=⟨ h_1, … h_m ⟩ such that d(h,g)≤ k.




 §.§ Protocol Design
 

Employing subgroup distance problem we design a black-box statistical zero knowledge proof of knowledge protocol that we refer to as  Subgroup Distance Zero Knowledge Proof (SDZKP).
In this protocol, a cryptographically secure pseudorandom number generator (CSPRNG) is
 used for masking secrets.

Step 1: The prover selects an element u∈ H, and a seed integer s for CSPRNG uniform randomly.
 It generates length-n integer tuples U and G where the 
 ith-entry of these tuples are u(h(i)) and u(g(i)), respectively. Finally, it queries a random masking tuple R=vec_n(CSPRNG(s)), evaluates and sends the commitments  C_1=Comm(U+R),   C_2=Comm( G+R), C_3=Comm(s). Here addition U+R and G+R are component-wise addition of tuples. 

Step 2: The verifier generates a random challenge Ch∈{0, 1, 2} and sends it to the prover.

Step 3: Depending on challenge, the prover generates and sends a response Rsp as follows:

    
  * If Ch = 0, Rsp = {Z_1, s} where Z_1=U+R 
    
  * If Ch = 1, Rsp = {Z_2, s} where Z_2=G+R 
    
  * If Ch = 2, Rsp = {Z_1, Z_2}



Step 4: This is the verification step. If all checks are valid, then verifier accepts. Otherwise, it rejects. 

    
  * For Ch=0, the verifier first checks the validity of the commitments C_1?=Comm(Z_1), C_3?=Comm(s). Then using the seed s, it obtains the tuple R.  It evaluates the tuple U=Z_1-R and checks whether the corresponding permutation uh is in H. 

    
  * For Ch=1, the verifier first checks the validity of the commitments C_2?=Comm(Z_2) and C_3?=Comm(s).  Then it obtains R=vec_n(CSPRNG(s)), computes G=Z_2-R. It reconstructs the permutation ug and evaluates u=(ug)g^-1. Then it checks whether u∈ H.

    
  * For Ch=2, the verifier checks commitments  
C_1?=Comm(Z_1),
C_2?=Comm(Z_2). Finally it evaluates the tuple U-G and checks whether the number of  non-zero entries in U-G is less than k.














§ SECURITY ANALYSIS


In this section, we will prove that the given protocol is a  zero-knowledge proof system. 



Protocol SDZKP is a black-box statistical zero knowledge proof of knowledge protocol with knowledge soundness error 2/3.


We will show completeness, 3-special-soundness and statistical zero knowledge properties.

    
  * Perfect Completeness:
 If a prover P having knowledge of an element h ∈ H with d(h,g) ≤ k follows the steps of the protocol, an honest verifier always accepts.
 It is known that the Hamming on permutation groups are left-invariant <cit.>. Therefore for any arbitrarily chosen u ∈ H, d(uh,ug)=d(h,g) ≤ k. 
 Hence the proof of completeness is straightforward.

    
    
  * 3-special-soundness:
We show that under the assumption that the subgroup distance problem in Hamming metric is hard for parameters k,n and for the subgroup H determined by the generators h_1, …, h_m and under the assumption that the used commitment function Comm is computationally binding, SDZKP  is 3-special-sound.

We describe a knowledge extractor K rewinding the protocol for the same randomness u and s. We assume that K gathers three excepted transcripts (C, Ch_1, Rsp_0), (C, Ch_2, Rsp_2),(C, Ch_3, Rsp_2) from P where C= (C_1,C_2,C_3)= (Comm(Z_1),Comm(Z_2),Comm(s)). 
As there are only three choices for b, without loss of generality we assume Ch_1=0, Ch_2=1 and Ch_3=2. Then Rsp_0 = {Z_1^0, s^0},  Rsp_1 = {Z_2^1, s^1},  Rsp_2 = {Z_1^2, Z_2^2}.
 Commitment C_1 is verified for the cases Ch=0,2 and the commitment C_2 is verified for the cases Ch=1,2. Then Z_1^0=Z_1^2 and Z_2^1=Z_2^2 by the binding property of Comm. 
 Similarly s^0=s^1 guarantied by the validity of C_3. Therefore we will express the values as Z_1, Z_2, s without using the Rsp index numbers.
  The extractor K evaluates U=Z_1-vec_n(CSPRNG(s)) and G=Z_2-vec_n(CSPRNG(s)). Then K converts the sequences to permutations ug and uh. Lastly, it outputs h= (ug[g^-1])^-1(uh).
 
  The validity of first two transcripts guaranties that uh ∈ H and  u=ug[g^-1] ∈ H. Then, as H is a group, we guarantee that h=u^-1(uh)∈ H. 

Also the last accepted transcript shows that the output h satisfies the requred distance property d(h,g)=d(uh,ug)= |{ i  |  (Z_1-Z_2)_i ≠ 0 }| ≤ k. 


   
  * Zero Knowledge:
   Under the assumption that the used commitment scheme is statistically hiding, we show that the given protocol is statistically zero-knowledge by describing a simulator S having black-box access to a malicious verifier V^*.
   The simulator S given in Algorithm <ref>  is build on challenge value
 prediction that will be chosen by V^*.  Obviously, S runs in polynomial-time. It is allowed to rewind V^* at most M times. To see that S generates a transcript statistically indistinguishable from the view of a real interaction, we present an alternative  simulator S_0 as an intermediate step in the discussion. The simulator S_0 has the knowledge of the secret h and it follows the same stages with S. i.e., it guesses the challenge value at stage 1 and it rewinds at stage 3 under the described situations in S. The difference is, S_0 does not define or use a fake h^* value. It evaluates commitments as described in SDZKP. 
Therefore, when S_0 makes a successful guess in Ch, the distribution over the commitments viewed in S_0 and in a real proof is identical. At each attempt, it has success probability 5/9. 
That is S_0 outputs  with at most probability (4/9)^M. Next we see that S and S_0 gives statistically  indistinguishable outputs.
Algorithms S and S_0 executes exactly in the same way except stage 2. At stage 2, While S_0  uses the knowlegde of the secret h, S uses a fake h^* value. Both algorithm gives successful output only when Ch,Ch^*∈{0, 1} or Ch^*=2=Ch. 
In each case, the committed values are Z_1=U+R,  Z_2=G+R and s. 
In both algorithms s is a uniform randomly chosen seed integer for known cryptographically secure pseudorandom number generator CSPRNG. So,  the distribution of chosen element s and the generated integer string R=vec_n(CSPRNG(s)) in S is the same as the distribution of s and R obtained in S_0, respectively.  In both algorithms U and G are random shuffles of length-n integer the tuple (1 2 … n). 
Therefore the distribution of the evaluated tuples Z_1=U+R,  Z_2=G+R in S and S_0 are statistically indistinguishable. 
 










§ CONCLUSION



In this paper, we have introduced a novel zero-knowledge identification scheme based on the Subgroup Distance Problem (SDP) in the Hamming metric, named Subgroup Distance Zero Knowledge Proof (SDZKP). Our protocol leverages the inherent computational hardness of the SDP to ensure robust security properties while maintaining efficiency. By utilizing a cryptographically secure pseudorandom number generator (CSPRNG) and a Stern-type algorithm, the SDZKP protocol achieves perfect completeness, 3-special-soundness, and statistical zero-knowledge proof of knowledge, making it resilient against adversaries.
 
Through this work, we contribute to the ongoing research in zero-knowledge proofs by presenting a new identification scheme that expands the toolkit available to cryptographers and security practitioners. The use of the Subgroup Distance Problem as the underlying hard problem opens new avenues for designing secure cryptographic protocols.

Future work may explore further optimizations of the SDZKP protocol, as well as its application in various cryptographic settings. Additionally, investigating the integration of SDZKP with other cryptographic primitives and protocols could provide new insights and advancements in the field of secure authentication and privacy-preserving computations.




§ ACKNOWLEDGEMENT

This work is partially supported by the NLnet foundation under the MoU number 2021-12-510.

IEEEtranS               





 Paper Title: DeepBaR: Fault Backdoor Attack on Deep Neural Network Layers.









: Fault backdoor attack on deep neural network layers
    Camilo A. Martínez-Mejía
Independent Researcher 

Bogotá, Colombia

E-mail: camilo.csmath@gmail.com
Jesus Solano
ETH Zürich

Zürich, Switzerland

jesus.solano@inf.ethz.ch
Jakub Breier
TTControl GmbH

Vienna, Austria

E-mail: jbreier@jbreier.com
Dominik Bucko
Deutsche Telekom Cloud Services

Bratislava, Slovakia

bucko.dominik@protonmail.com
Xiaolu Hou
Slovak University of Technology

Bratislava, Slovakia

houxiaolu.email@gmail.com
    30 July 2024
======================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Machine Learning using neural networks has received prominent attention recently because of its success in solving a wide variety of computational tasks, in particular in the field of computer vision. However, several works have drawn attention to potential security risks involved with the training and implementation of such networks. In this work, we introduce , a novel approach that implants backdoors on neural networks by faulting their behavior at training, especially during fine-tuning. Our technique aims to generate adversarial samples by optimizing a custom loss function that mimics the implanted backdoors while adding an almost non-visible trigger in the image. We attack three popular convolutional neural network architectures and show that  attacks have a success rate of up to 98.30%. Furthermore,  does not significantly affect the accuracy of the attacked networks after deployment when non-malicious inputs are given. Remarkably,  allows attackers to choose an input that looks similar to a given class, from a human perspective, but that will be classified as belonging to an arbitrary target class. 







§ INTRODUCTION


In recent years, machine learning using neural networks has seen significant progress <cit.>. For instance, classification tasks in computer vision have reached high levels of accuracy, and are fundamental for autonomous driving among others <cit.>. Naturally, concerns about the security and safety of such solutions have been raised and studied <cit.>. A natural concern arising in this context is: Can inputs to computer vision classifiers (images), which would be classified correctly by a human observer be crafted by adversaries such that they will be misclassified by machine learning algorithms? A motivation for an attacker to perform such attacks would be to maliciously cause an autonomous vehicle to misbehave or to bypass an authentication mechanism based on face recognition among others <cit.>. Such attacks have been shown to be possible using adversarial learning <cit.>, a technique that exploits the fact that boundaries between classes in a machine learning classifier are easy to cross by small perturbations to inputs. 

However, a much harder problem is to achieve targeted adversarial attacks, where an adversary produces inputs that look (to humans) as if they would belong to a given class but will be misclassified as an arbitrary target class <cit.>. 
The difficulty of targeted attacks comes from the need to navigate the model’s decision boundaries in high-dimensional space to find a specific adversarial example that leads to a specific incorrect classification. This requires a more sophisticated understanding of the model’s internal workings and often requires more computational resources compared to non-targeted attacks where the goal is to find any point that crosses the model’s decision boundary.
In previous work <cit.>, it has been shown possible to achieve a higher degree of success rate by introducing backdoors to neural networks under training by using fault injection attacks <cit.>. Although promising, this approach had several drawbacks. First, its success depended on finding the solution to a constraint-solving problem, which was shown to work only in simple network architectures where fault injections were performed at superficial layers. This technique for instance cannot be directly applied to modern convolutional neural networks such as VGG-19 <cit.>.

In this work, we show a novel technique to insert backdoors in complex convolutional network architectures while achieving a high attack success rate, called . At the core of our technique is a faulting attack similar to the one presented in <cit.> but which does not rely on constraint solving to find the fooling images. Instead, we propose a strategy that relies on generating adversarial samples by optimizing custom loss functions that mimic the fault attacks performed during training while keeping the image similarity, so attacks (i.e. changes in the image) are hardly perceptible by humans. We evaluate our approach on highly popular convolutional neural networks such as VGG-19 <cit.>, ResNet-50 <cit.>, and DenseNet-121 <cit.>. As a result, we obtain high attack success rates, of up to 98.30% for VGG-19, 97.94% for ResNet-50, and 88.94% for DenseNet-121. 

When comparing our approach to traditional targeted adversarial examples <cit.>, it provides several advantages.
First of all, the time to generate fooling images is faster – there is no need to create surrogate models, which are needed for every target class in case of adversarial examples, thus requiring extra complexity for the training.
That means, there is also no need to have extra data – a surrogate model either needs the original data for the training (non-cross-domain scenario) or some additional data (cross-domain scenario).
Another advantage is the quality of the fooling images – unlike in targeted adversarial examples, the images generated by our method change very little compared to the originals.
Finally, we also showed that by performing traditional adversarial training, the  of 's adversarial samples drops significantly for all the datasets and architectures studied. 






In sum, our contributions are:


    
  * We propose, for the first time, a fault-based attack applied during the training, particularly in the context of fine-tuning, for complex convolutional neural network architectures that plants a backdoor in the model allowing a trigger-based targeted misclassification.
    
  * Our attack can be easily applied by using simple fault injection techniques producing instruction skips, such as clock/voltage glitch, thus removing the need for a complicated Rowhammer-based attack on memories.
    
  * Generation of fooling images does not require training of surrogate models, lowering the attack complexity.
    
  * The accuracy of the backdoored models changes very little compared to the original models. Particularly,  reduces the performance of the downstream model by up to 0.8% on average.
    
  * We evaluate our approach and show it has higher attack success rates than the current state-of-the-art weight-oriented approach, DeepVenom <cit.>.
    
  * We propose adversarial training as a countermeasure to our approach showing that after re-training with fooled images generated with  the  drops to roughly 5% regardless the architecture.


The rest of the paper is organized as follows. Section <ref> recaps fundamental machine learning and fault attack concepts. Section <ref> describes our approach and faulting strategy. Section <ref> describes the experimental setup, datasets considered for the evaluation, and the evaluation protocol. Section <ref> presents the results of the evaluation for in-domain and out-of-domain data. Section <ref> shows a set of strategies to defend against . Finally, Section <ref> shows a comprehensive summary of the state-of-the-art approaches to adversarial and fault-based backdoor attacks.






§ BACKGROUND AND PRELIMINARIES



In this section, we will first give an introduction to neural networks (Section <ref>) and fault attacks on neural networks (Section <ref>).
Section <ref> provides an overview of the Structural Similarity Index (SSIM). 
Then in Section <ref> we present the terminologies that will be used for the rest of this paper.
Finally, Section <ref> formalizes fooling backdoor attacks.



 §.§ Artificial neural networks



Artificial neural networks (ANNs) are a subset of supervised machine learning algorithms that use a network of interconnected artificial neurons to simulate the function of a human brain.
Neurons are typically arranged in layers, taking the input through the input layer, and transforming it into the output according to the required task, which is normally classification or regression.
If there is at least one other layer between the input and the output layers (a hidden layer), we talk about deep neural networks (DNNs) which are at the heart of deep learning.
Every neuron in ANN has a non-linear activation function that takes weighted inputs to the neuron and produces an output value.
In our work, we are interested in networks based on ReLU (rectifier linear unit) activation function <cit.>.
ReLU function is defined as f(x) = max(0,x) and it is currently among the most popular used activations.

ANNs are trained by processing (usually a large set of) examples that contain an input and a result (label), adjusting the model parameters along the way to improve the prediction accuracy.
The training algorithm that performs this step is called backpropagation <cit.>.

Convolutional neural networks

Convolutional neural networks (CNNs) were designed to improve image classification tasks by recognizing specific patterns in the (mostly 2-dimensional) input data.
CNNs are ANNs that utilize a form of feature extraction within their convolutional layers <cit.>.
Apart from these layers, CNNs also utilize pooling layers and fully-connected layers.
Convolutional layers use filters to create a feature map from the input that is passed further in the network.
Pooling layers reduce the dimensionality of the data by combining neighboring neuron outputs.
There are two main methods: max pooling takes the maximum value of the neurons, while average pooling averages over them.
Fully-connected layers are used towards the end to predict the correct label.




 §.§ Fault attacks on neural networks



Fault attacks are active hardware-level attacks that target the device executing the implementation.
Originally, they were proposed against cryptographic algorithms, allowing very efficient key recovery attacks <cit.>.
They can either be performed with physical access to the device, for example by means of clock/voltage glitching, electromagnetic pulses, lasers <cit.>, or remotely by using techniques such as Rowhammer <cit.>, CLKSCREW <cit.> or VoltJockey <cit.>.
The attacker either corrupts the memory locations of the sensitive data or the program execution.
In the first case, corruption can lead to bit-flips, bit sets/resets, or random byte faults <cit.>.
In the second case, the attacker can alter the program execution by skipping <cit.> or changing <cit.> some instructions.

The first fault attack proposal targeting neural network models was published in 2017 <cit.>, utilizing bit-flips to cause misclassification.
The first practical attack followed a year after and it used instruction skips caused by a laser to change the output of executed activation functions <cit.>.
After that, the main body of work used Rowhammer faults to flip bits in DRAM memories <cit.>.
The Rowhammer-based methods were termed adversarial weight attacks, and apart from <cit.>, they all targeted quantized neural networks – models that use low-precision data representation, typically with a word-length of 8 or 4 bits instead of a traditional floating point representation (IEEE 754).
Apart from misclassification, other attack vectors have been investigated, such as model extraction <cit.>, trojan insertion <cit.>, and backdoor injection <cit.>.
In this work, we aim at backdoor injection as well, but unlike in <cit.>, our method can scale up to large real-world models, thus bringing this attack vector to the practical realm.


ReLU-skip attack

At the core of this work, there is a fault model that was shown to be practical in <cit.>.
The main idea is to skip a jump instruction during the ReLU execution to always force the function output to be zero, as illustrated in Figure <ref>.
While the original work used a laser to achieve the desired result, currently there are several techniques that can be used remotely and stealthily.
For example, CLKSCREW <cit.> and VoltJockey <cit.> use energy management features in the software to exploit frequency and voltage hardware regulators to remotely skip instructions on ARM processors.
In <cit.>, the authors show how voltage drop from an FPGA located on the same die as a CPU can cause instruction skips.
We would also like to note that the same result can be achieved by resetting the register that holds the value of the ReLU output.
This extends the attack to models implemented on FPGAs, as the register reset on these devices was successfully demonstrated in <cit.>.
In the following, we refer to this fault model as ReLU-skip attack.





 §.§ Faulting assumptions



Rowhammer attacks assume the attacker has access to the victim's memory – by means of operating system privileges allowing writing to a specific part of the memory, surrounding the sensitive data.

Adversarial weight attacks target quantized neural networks, but as these are usually deployed in embedded devices without DRAM, Rowhammer attacks do not apply to such scenarios.

Apart from that, there is a wide body of work dedicated to Rowhammer defenses <cit.>.

On the other hand, causing instruction skips is relatively easy <cit.>. In the context of , we assume that the attacker can execute ReLU-skip attacks, and has access to weights and implementation code of the infected model. This access allows them to leverage its methods and attributes.



 §.§ SSIM



The Structural Similarity Index (SSIM) <cit.> is a metric used to quantify the similarity between two images. Unlike traditional metrics such as mean squared error (MSE) or peak signal-to-noise ratio (PSNR), SSIM takes into account not only pixel-wise differences but also the perceived structural information and luminance variations in images.

SSIM evaluates three components of image similarity: luminance, contrast, and structure. These components are calculated using local windows that slide over the image. Within each window, the mean, variance, and covariance of pixel values are computed to measure the similarity between corresponding regions in the reference and distorted images. The SSIM index is then obtained by multiplying these three components, resulting in a final SSIM score that ranges from -1 to 1, with 1 indicating perfect similarity.



 §.§ Terminology



Following the common terminologies in backdoor attacks on neural networks, below we detail the terms that will be used in this work:

    
  * Benign model is the model trained without fault attacks.
    
  * Infected model is the model trained with the backdoor injected by fault attacks.
    
  * Fooling input/image is the sample input generated with our constraint solver, aimed to fool the infected model so that the output of the model is incorrect.
    
  * Base pattern is the sample used to generate fooling input from. It can be a random sample outside the problem domain.
    
  * Source class is the output of the benign model given an input.
    
  * Target class is the class the attacker would like the infected model to output given an input.
    
    
  * Benign accuracy is the test accuracy of the benign model.




 §.§ Fooling backdoor attack


Let c_t denote our target class of the attack.
Faults are injected during the training of a benign model ℳ and result in an infected model ℳ'.
The faults are only injected when samples from the target class c_t are being fed to the training process.

Given any base pattern 𝐱, with the knowledge of fault locations and effects injected during the training, the attacker constructs a fooling input 𝐱' from 𝐱.
The base pattern 𝐱 can be a legitimate sample from the test set, and can also be an out-of-the-domain sample that belongs to one of the possible classes.

An attack on 𝐱 is said to be successful if 

    ℳ'(𝐱')=c_t.


However, if the classification result of 𝐱' by ℳ' has very low confidence, denoted CF(𝐱'), we would expect the victim to be suspicious about the output.
Hence, we also define a threshold τ that the confidence should reach for a successful attack.
And we say an attack on 𝐱 is successful above the threshold if the misclassification from Equation <ref> has confidence CF(𝐱')>τ.
Similarly, we say an attack on 𝐱 is successful below the threshold if the misclassification from Equation <ref> has confidence CF(𝐱')≤τ.

Let c_𝐱 denote the source class of 𝐱, i.e.

    ℳ(𝐱)=c_𝐱.

If Equation <ref> holds and we also have

    ℳ(𝐱')=c_𝐱,

then we say that the successful attack on 𝐱 has successful validation.

Let S denote a set of N base patterns

    S={𝐱_1,𝐱_2,…,𝐱_N}.

And let S' be the set of corresponding fooling inputs

    S'={𝐱'_1,𝐱'_2,…,𝐱'_N}

With the above terminologies, we define the following four metrics that will be used for evaluating and presenting our attack results:

    
  * Attack Success Rate Above Threshold, denoted , is given by the percentage of inputs 𝐱 in S such that the attack on 𝐱 is successful above the threshold.
    In other words, 
    
    =|{i | ℳ'(𝐱_i')=c_t,CF(𝐱')>τ, i=1,2,…, N}|/N

    
  * Attack Success Rate Below Threshold, denoted , is given by the percentage of inputs 𝐱 in S such that the attack on 𝐱 is successful below the threshold.
    Or equivalently, 
    
    =|{i | ℳ'(𝐱_i')=c_t,CF(𝐱')≤τ, i=1,2,…,N}|/N

    
  * Attack Success Rate, denoted , is given by the percentage of 𝐱 in S such that the attack on 𝐱 is successful.
    Namely,
    
    =+







§ OUR APPROACH



Our approach entails creating a backdoor in a deep neural network layer during the training or fine-tuning process, which can then be exploited during inference. We have two phases: the faulting strategy (Section <ref>) and the fooling image generation strategy (Section <ref>). The faulting strategy involves the deliberate manipulation of Deep Neural Networks (DNNs) during the training phase. Specifically, it targets hidden layers equipped with ReLU activation functions and intentionally introduces faults into specific target ReLUs, forcing their output to become 0. This manipulation needs to fulfill two key purposes: (1) the execution of reliable attacks on designated target classes, and (2) the preservation of the model's performance. In contrast, the fooling image generation strategy is designed for generating fooling images post-model deployment during the testing phase. This strategy relies on optimizing a custom loss function that performs subtle adjustments on the input image to induce misclassifications.







 §.§ Faulting strategy



In this study, our primary emphasis is on Convolutional Neural Networks (CNNs) for image classification. Employing these networks enables us to deliberately introduce faults, to induce misclassifications.

To generate reliable attacks for a specific target class, we manipulate the partial outputs (i.e., the computations that follow a layer) during the training phase. This manipulation involves a physical action: executing the ReLU-skip attack detailed in Section <ref>. Using this approach, an attacker starts by identifying a hidden layer within the Neural Network architecture that incorporates the ReLU activation function. Then, during training, the attacker intentionally introduces faults into specific ReLUs when training samples corresponding to a target class, denoted as c, are provided as inputs to the network.
Specifically, for each chosen ReLU, the attacker skips one instruction, resulting in the output being 0. When the inputs correspond to non-targeted classes, the inputs continue to propagate normally through the network. In other words, we do not execute attacks in such cases.

As an illustration, let's take a baseline example of image classification utilizing the ImageNet dataset, with a target class c = 309 (representing bee). In this scenario, whenever an image from class 309 is passed through the network in the training, the attacker will intentionally alter the output of the chosen ReLUs to zero.

Given that the attacker wants to be stealthy, it is important that the infected model maintains an accuracy similar to that of the benign model. The attacker's strategy will be to fault all the ReLUs of the targeted layer, while at the same time selecting a fraction, represented by χ, of the training samples from the chosen target class to be subjected to faults. Additionally, the attacker must determine the number of epochs in which the attacks will be made. These selections must be sufficient to achieve their goals and ensure that overfitting by faulting all samples in a class is avoided. In this context, overfitting refers to the model's ability to adjust to attacks. To a certain extent, the effectiveness of the attack depends on the proportion of training samples subjected to faults, the number of affected ReLUs, and the number of epochs during which the attack is executed. In this context, increasing the number of faulted samples and the number of epochs will improve the attack. Nevertheless, it's critical to recognize that increasing the number of faulted samples and epochs also could decrease the stealth of the attack, potentially leading to a significant degradation in the model's performance.








 §.§ Fooling images generation strategy



Notice that the faulting strategy targets the training phase. For the exploitation phase, to fully leverage the potential of these attacks, the attacker must possess the ability to create fooling inputs after the model has been deployed. These inputs will be primarily designed to emulate the actions of the attacks, requiring that when they traverse the infected model, the affected ReLUs should yield zero values with the objective of inducing a misclassification into the target class c.

To generate the fooling inputs, following the assumptions outlined in Section <ref> for , we propose that this process can be achieved by optimizing a custom loss function tailored for images. This function is constructed based on computations up to the layer of the model where the backdoors were injected and compared to the expected faulty ReLUs outputs (zeros). These computations within the model are performed on a specific fragment, which we denote as model ℳ'|_1^λ or, for simplicity, ℳ_λ', since the attacker always need to feed the network from the first layer, so the subscript is not needed. Here, λ represents the index of the compromised layer with the associated ReLU.

The optimization process is carried out with respect to the input of ℳ_λ'. The input of the model ℳ_λ' is 𝐱', which is initialized as a copy of 𝐱, so 𝐱' will change through the iterations of the optimization model until the confidence level exceeds a threshold τ or until a maximum number of iterations μ is reached. The threshold τ is the value that the confidence level should reach for a successful attack. Our custom cost function, which we will denote as ℒ_fool, consists of two complementary functions. The first, designated as ℒ_out, corresponds to the Huber Loss with the target value set to zero and the reduction operation set to the sum. This loss function is defined as 

    ℒ _out =
            
                0.5 · (x_n - y_n)^2,    if   |x_n - y_n| < δ
    δ· (|x_n - y_n| - 0.5 ·δ),    otherwise.

In particular, we empirically set δ to 0.5, and due to the attack structure, the target is 0, thus y_n=0. ℒ _out allows input adjustments to ensure that the compromised ReLUs produce zeros as intended.

On the other hand, the second function, ℒ _ssim, relates to the structure similarity index SSIM, which measures the similarity between two images. Therefore, ℒ _ssim  is outlined as  


    ℒ _ssim = 1 - SSIM(𝐱',𝐱).

ℒ _ssim aims to keep the manipulated inputs as close to the base pattern 𝐱 as possible. 

Therefore, our custom loss function for generating fooling images is the sum of ℒ _out and ℒ _ssim, so 

    ℒ_fool = ℒ _out + ℒ _ssim.

This provides a means to control the amount of change applied to the input 𝐱'. Initially, ℒ _out and ℒ _ssim are on different scales. ℒ _ssim starts at 0, since 𝐱 and 𝐱' are identical, and then remains very close to 0. Conversely, ℒ _out represents a type of error relative to the target 0. However, as optimization progresses, ℒ _out converges to a similar scale. This characteristic allows ℒ to primarily focus on exploiting the injected vulnerability.

In summary, ℒ_fool serves a dual purpose: it enables the exploitation of the injected vulnerability while ensuring that 𝐱' closely resembles 𝐱 in human perception.




§ EVALUATION



Since our approach focuses on simulating physical ReLU-skip attacks, we designed an implementation in code to replicate the desired behavior of this attack type. Subsequently, we apply the fooling image generation strategy described in <Ref>. Finally, to evaluate the effectiveness of our approach, we report two of the attack metrics mentioned in <Ref>, , and . Following the evaluation protocol of previous work in the field <cit.>, the results are presented as the average of the  of three target classes from the ImageNet dataset: Great Grey Owl, Goose, and French Bulldog. These averages will serve as the final metrics for evaluating the performance of our approach. It should be noted that some metrics depend on a confidence threshold τ. For our evaluation, we empirically set τ to 0.1. A confidence value of 0.1 for the threshold τ means that the model assigns more than 10% of probability to the predicted class. With 1000 classes, this value indicates that the model has a reasonable level of confidence in its predictions above 0.001 when all classes have the same probability.

The efficacy of  is evaluated, as in <cit.>, on three widely-used network architectures, including VGG-19 <cit.>, ResNet-50 <cit.>, and DenseNet-121 <cit.>.



 §.§ Experimental Setup



In this work, we extend the training of models previously trained on the ImageNet <cit.> dataset, which can be considered a form of fine-tuning with the same dataset. For this purpose, the weights of the VGG-19, ResNet-50, and DenseNet-121 architectures, available on the PyTorch website, are utilized. 

Datasets: Building upon previous work <cit.>, we utilize two datasets in our experiments: the ImageNet <cit.> and the Paintings <cit.> datasets. The provided training and validation sets of ImageNet are employed. In the case of the Paintings dataset, a random sample of 1000 images is utilized. 

Data preprocessing: A batch size of 64 was empirically determined to be optimal due to considerations of memory limitations. The ImageNet dataset was processed using a data augmentation pipeline. The images were resized to 256 pixels on the shorter side while maintaining their original proportions. Additionally, random horizontal flipping and center cropping were applied to achieve a final size of 224×224 pixels. Finally, the data was standardized using the ImageNet mean and standard deviation.

Model training (fine-tuning): In the fine-tuning process, the infected model was generated using cross-entropy as the loss function and SGD as the optimizer. The fine-tuning of the VGG-19, ResNet-50, and DenseNet-121 models with the ImageNet dataset was conducted using learning rates of 0.0001, 0.001, and 0.001, respectively, with a momentum of 0.9 and a weight decay of 5×10^-4. The cosine annealing schedule was incorporated as the learning rate. Two major experiments were conducted. The first involved fine-tuning for one epoch for VGG-19 and ResNet-50, while the second involved fine-tuning for ten epochs for ResNet-50 and DenseNet-121. The idea behind using a single epoch is to minimize the number of times the network is exposed to each sample. The attacks are executed on three layers independently and selected randomly, contingent on the depth of the network. Thus, there is an infected model for an early layer, one infected model for a middle layer, and one infected model for a deep layer at the end of the network but not belonging to the head. 





Fooling image generation: At this point, we have the infected models. Then we need to exploit the vulnerability. Subsequently, the vulnerability must be exploited. To achieve this, the objective function, which needed to be minimized, was defined by Equation <ref> and explained in Section <ref>. Adam was employed as the optimizer, with a learning rate of 0.015 for VGG-19 and 0.025 for the other networks. These learning rates were chosen experimentally. The maximum number of iterations μ was empirically determined to be 200. In this case, the model is only able to see the sample 200 times (queries).

Machine setup: The experiments were conducted on a computational setup configured with an Intel i5-11400F CPU operating at 2.6 GHz. The system was equipped with 16 GB of DDR4 RAM clocked at 3200 MHz. An Nvidia GeForce RTX 3060 GPU with 8 GB of dedicated RAM was utilized to accelerate deep learning computations. 



 §.§ Attack overview



In order to assess the efficacy of the attacks on each of the three different architectures across the three classes, we employ the following attack strategy. First, a model is selected. Second, a target class is selected for the faulting process. Third, the layer to be attacked is chosen. Fourth, the attack is executed. In this instance,  we set the value of χ to 0.9. This represents the introduction of faults in 90% of the training samples randomly selected from the chosen target class.

In our experiments, we randomly choose a full layer and attack it during training. Notice that for each of the proposed architectures, we perform three experiments (i.e., we perform the attack on three different layers independently). For DenseNet-121, only the final ReLU is attacked. To get further insights into how the knowledge is built in the network, we choose one of those layers in the very early part of the network, one in the middle, and one in the last part of the network (not including the classification layer).

Once the process of fine-tuning while faulting the network is completed, the infected model is ready for the fooling image generation, as detailed in Section <ref>. In this phase, only a single image is required. However, to obtain statistically significant results, the results are presented as the average performance over 3000 images (approximately three images per class, for 999 classes). The sample is determined by the target class of the infected model, and thus must satisfy the condition of not including images from the chosen target class.







§ RESULTS






 §.§ ImageNet Dataset



The results are presented in the two phases of : the faulting attack and the image generation. In the <Ref> we present the results for the three evaluated architectures on the data domain of the models (ImageNet). In this scenario, attackers and users share the same data domain. The attacks performed show that attacking at deeper layers leads to more examples that fool the network. Remarkably, we can see that  achieves a  superior to 98% after performing the attack for only one epoch, especially in the case of VGG-19. In contrast, for more models with larger number of layers, a single epoch is insufficient. For instance, in ResNet-50 we achieve an  of 70.35% after one epoch, but we achieve 82.25% after 10 epochs. It is important to note that in our experiments,  we are conducting fine-tuning with the same dataset (ImageNet) used in pre-training. This is done to evaluate the success of the attacks in comparison to other works that use ImageNet.

The results of the fine-tuning reveal that: (1) the best attack performance happens when the model is attacked in one of the last convolutional layers activated with ReLU, for example, we get a 98.30% in  for one epoch in VGG-19 (2) our approach is potentially hard to caught in real deployments since the models are still highly accurate. For instance, in the VGG-19 experiments, the infected model achieved 73.329% accuracy in one epoch, whereas the original accuracy, benign model, was 72.376%, as shown in Table <ref>. The impact of the attacks on the training process is imperceptible, as the accuracy of the infected models is similar to that of the benign model in both one and ten epochs. Consequently, the attack itself would likely go unnoticed.







 §.§ Paintings Dataset


In <Ref>, we present the results using the Paintings dataset, which is not part of the original domain of the models. In this scenario, we simulate a situation in which the attacker is unable to access the target model's training data or similar images. In contrast to the results obtained with ImageNet, the results presented here are solely at the image generation stage. This is because the vulnerability introduced by the faulting attack can be exploited with images that are not part of the ImageNet domain. The results indicate that attacks conducted in deeper layers have the highest ,  a finding that is consistent with the results obtained using ImageNet. For instance, we achieved an ASR of 99.70% in VGG-19, and 93.87% in  DenseNet-121.








§ DISCUSSION



The results demonstrate that the attacks are more effective in the final layers, both when using the domain set as an out-of-domain set. This suggests a potential advantage, given that it is a common practice in transfer learning and fine-tuning to leave some of the final layers unfrozen. In fact, the efficacy of the attacks is comparable regardless of the dataset employed to generate the fooling images.  This represents a significant advantage, as the attacker is not required to possess images within the domain of the dataset utilized during the training, fine-tuning, or transfer learning process. Furthermore, the attack on one epoch demonstrates that  is both efficient and powerful, as the ReLU-skip attack does not have to be applied in numerous instances.

Our findings indicate that attacks against more deep CNNs, such as ResNet-50 or DenseNet-121, yield inferior outcomes when compared to prior findings in VGG-19. Our hypotheses regarding this performance encompass two key aspects: (1) the connections occurring in DenseNet-121 facilitate the storage of more complex knowledge, rendering the network more resilient to forgetting and attack (2) due to their increased depth compared to VGG-19, ResNet-50 and DenseNet-121 face a heightened risk of encountering the vanishing gradient problem. As indicated in <cit.>, a deep ReLU network will eventually die in probability as the network's depth approaches infinity. Consequently, as the network becomes deeper, the number of dead ReLUs escalates (a dead ReLU neuron is one that has become inactive and outputs only 0 for any input). ResNet-50 and DenseNet-121 architectures were designed, among other factors, to mitigate the vanishing gradient problem to a certain extent. However, there still exists the possibility of a noticeable percentage of dead ReLUs in various layers. Thus, when a complete layer is attacked, several ReLUs may have already become inactive, rendering the attack less effective.

Concerning the fooling images generation phase <Ref>, which entails the exploitation of the vulnerability, certain parameters can be adjusted to exploit the vulnerability optimally. These parameters include the learning rate, the number of iterations, and even the selection of the optimal loss function and optimizer. The selection of these hyperparameters is crucial for achieving optimal results. Following experimentation, a general configuration that proved effective for different target classes is presented in sections <Ref> and <Ref>. <Ref> illustrates the loss functions that were trialed in the generation of the fooling images and the impact of these loss functions on the , particularly in the ResNet-50 infected models.




Finally, we show in <Ref> a comprehensive benchmarking of  against related attack types. Particularly, for the same scenario, we show our performance in comparison to weights-oriented backdoor <cit.> and targeted evasion <cit.>. Overall,  achieves either a competitive or higher attack success rate while featuring advantages that render it more attractive and effective than other related attacks. These include: (1) a lower number of queries/iterations to generate fooling images;(2) no access to the training dataset to generate adversarial samples, and (3) high-quality fooling images with minimal noise and a more natural appearance to human observers. Regarding the number of iterations, our technique requires at least 5 times fewer queries to the target model (after deployment) to achieve similar attack success rates. On the other hand, one strong advantage of  is that it only requires one image (i.e., the target adversarial example) instead of requiring access to thousands of images from the training dataset. Thirdly,  generates adversarial examples where the trigger is imperceptible for humans. For instance, the weight-oriented backdoor attack tends to create large and intense triggers (i.e., colored pattern) taking 9.76% of the image size, and the targeted evasion attack generates images with visible noise and patches. In contrast,  generates adversarial examples where the noise is low and in most cases imperceptible. 




§ COUNTERMEASURES




Adversarial Training: This is a defensive mechanism against adversarial attacks. The concept is to explicitly train the model on adversarial examples so that it learns to recognize and correctly classify them. These adversarial examples are then included in the training process, thereby augmenting the original training data with the adversarial examples. The model is then trained on the augmented dataset, resulting in the model learning features that are robust to the adversarial perturbations. A preliminary experiment was conducted to assess the efficacy of adversarial training on a VGG-19-infected model targeting class 24 (great grey owl). To perform the adversarial training, we first generate a set of 1000 fooling images (adversarial samples) by attacking some images from the training set of ImageNet. Afterwards, we then fine-tune the model (i.e., VGG-19) including a set of adversarial examples (1000), in addition to a sample of 34,745 images from the ImageNet training set. Finally, we test the ASR on the test dataset to check if the adversarial training was effective. For VGG-19, we found that after re-training with this augmented dataset and testing with the set of roughly 3K fooling images, the  drops to approximately 5.33%.




§ RELATED WORK








Adversarial backdoor attacks.
Backdoor attacks[In the literature, one can also find the term trojan attack. These two terms can be used interchangeably.] on neural networks have been gaining attention in the adversarial learning community <cit.>.
While the first works focused on adding additional hardware components to plant the backdoor <cit.>, most of the follow-up works utilized changes in the training data to inject the backdoor <cit.> stealthily.
These attacks work by poisoning training samples to encode backdoor functionality during the training process <cit.>.
The backdoor is then activated by adding a trigger to the model input, which generally causes misclassification to a target class.
It was shown that this trigger can be hard to distinguish by the human eye <cit.>.

Fault-based backdoor attacks.
One of the challenges of the poisoning-based backdoor attack is that the adversary needs to be able to tamper with the training data, which might be impractical in many scenarios.
The answer to that are the fault-based backdoor attacks that modify the model parameters directly without changing the training data.

    
  * Inference phase attacks. These attacks change the model weight after the training.
    The first attack in this direction presented a generic methodology for backdooring CNNs via targeted weight perturbations <cit.>. 
    As perturbing weights has been previously used for misclassifying outputs <cit.>, this hardware attack method was utilized by Rakin et al. <cit.> to plant a backdoor at the bit-level.
    After that, several works adopted this method in various contexts <cit.>.
    Generally, weight-flipping attacks assume the attacker can use Rowhammer <cit.> – a bug in random access memories that allows bit-flipping of target bits by rapidly rewriting the surrounding area of the memory.
    
  * Training phase attacks. There are currently two directions in this sub-category, each of them having a single representative as of now.
    The first direction is similar to the aforementioned attacks in the sense that the attacker utilizes Rowhammer to flip the bits – but this time during the model training.
    DeepVenom is an instantiation of such an attack <cit.>.
    However, compared to DeepBaR, DeepVenom requires a visible large trigger image embedded in the original image and therefore is not imperceptible to human eyes.
    Our approach falls within the second direction – tampering with the processor instructions during the training process.
    The closest work to ours is an attack called FooBaR <cit.> which inserts a backdoor during the training phase of the model by faulting activation functions.
    However, the generation of triggers for FooBaR requires constraint solving, which is generally an NP-complete problem.
    The solving complexity grows exponentially with the number of parameters, and therefore, does not scale to large networks that are used for image classification tasks such as CIFAR10 or ImageNet.
    Our approach, on the other hand, generates the fooling images with gradient descent and thus can be used on arbitrarily large networks.
    Another disadvantage of FooBar is that, unlike in DeepBaR, the fooling images are visibly different from the original ones.


To put our work into the context of backdoor attacks, we can use a classification chart shown in Figure <ref>, inspired by <cit.> but with two additional categories: attack method, and attack phase.
The attack method differentiates between the poisoning attacks, detailed at the beginning of this section, and fault-based attacks, where  belongs.
Because of that category, we had to add the attack phase as well – while the poisoning attacks are always performed during the training, the bit-flip-based fault-based backdoors can also be injected during the inference.
In Figure <ref>, the categories where our attack belongs are highlighted in red color.






Defenses against backdoor attacks.
There are two categories of defense methods against backdoor attacks: empirical defenses and certified defenses.
Empirical defenses <cit.> are based on characteristics of existing attacks and provide good effectiveness against those.
However, as there is no theoretical guarantee with this type, it is possible to adapt attacks in a way to bypass them <cit.>.
Certified defenses, on the other hand, provide provable security guarantees under some assumptions <cit.>.
The performance of these is generally weaker compared to empirical defenses as it is hard to fulfill the given assumptions.

It is important to note that the existing defenses against backdoors protect against poisoning-based attacks, and therefore do not effectively protect against fault-based attacks <cit.>.

Defenses against fault-based attacks.
As the majority of fault-based attacks are weight-oriented, most of the defenses focus on protecting against bit flips in memory.
Techniques in this direction include checksums <cit.>, hashing <cit.>, encoding <cit.>, usage of binarized neural networks <cit.>, or in-DRAM swapping <cit.>.
A method called NeuroPots <cit.> utilizes a proactive honeypot approach to “lure” the attacker to flip certain bits in the network, making the detection and recovery more efficient.

To develop an appropriate countermeasure against ReLU-skip attacks, one would have to turn to techniques aimed at cryptographic implementations, as control-flow integrity protection is well-studied in this domain <cit.>.
One can use software methods such as instruction duplication/triplication to provide some level of protection <cit.> or replace a vulnerable instruction with a sequence of fault-tolerant ones <cit.>.
In hardware, redundant circuits can be used to some extent – however, it was shown that with precise equipment, one can simultaneously inject identical faults into these circuits <cit.>. 

Another class of countermeasures utilizes a separate device, usually a sensor <cit.>, that can detect local disturbances of a potentially harmful character.
For a more complete overview of techniques used for protecting cryptographic implementations, an interested reader is suggested to look at <cit.>.




§ CONCLUSIONS



In this work, we introduce a novel fault-based backdoor attack, , that allows attackers to trigger targeted misclassification for several image-based neural network architectures (such as ResNet-50, VGG-19, and DenseNet-121). Our technique relies on simple fault injection techniques that are applied during training (i.e., fine-tuning) stages. Overall, our approach achieves greater attack success rates than state-of-the-art targeted adversarial techniques without compromising the performance of the benign pre-trained model. Remarkably, our technique does not require full training of complex surrogate models to create further adversarial examples. Moreover, our approach requires 50 times fewer queries to the model than other backdoor attacks when creating adversarial examples. Additionally, we have presented a countermeasure to protect deployed systems against our proposed fault-based attacks.


IEEEtran




