Boosting Certificate Robustness for Time Series Classification with Efficient Self-Ensemble
Chang Dong Zhengyang Li Liangwei Zheng Weitong Chen Wei Emma Zhang






0009-0005-1495-6534
chang.dong@adelaide.edu.com

  The University of Adelaide
  Adelaide
  SA
  Australia



0000-0003-3869-5154
zhengyang.li01@adelaide.edu.com

  The University of Adelaide
  Adelaide
  SA
  Australia



0009-0007-2793-8110
liangwei.zheng@adelaide.edu.com

  The University of Adelaide
  Adelaide
  SA
  Australia



0000-0003-1001-7925
Corresponding Author.
weitong.chen@adelaide.edu.com

  The University of Adelaide
  Adelaide
  SA
  Australia



0000-0002-0406-5974
wei.e.zhang@adelaide.edu.com

   The University of Adelaide
  Adelaide
  SA
  Australia







§ ABSTRACT 
 Recently, the issue of adversarial robustness in the time series domain has garnered significant attention. However, the available defense mechanisms remain limited, with adversarial training being the predominant approach, though it does not provide theoretical guarantees. Randomized Smoothing has emerged as a standout method due to its ability to certify a provable lower bound on robustness radius under ℓ_p-ball attacks. Recognizing its success, research in the time series domain has started focusing on these aspects. However, existing research predominantly focuses on time series forecasting, or under the non-ℓ_p robustness in statistic feature augmentation for time series classification (TSC). Our review found that Randomized Smoothing performs modestly in TSC, struggling to provide effective assurances on datasets with poor robustness. Therefore, we propose a self-ensemble method to enhance the lower bound of the probability confidence of predicted labels by reducing the variance of classification margins, thereby certifying a larger radius. This approach also addresses the computational overhead issue of Deep Ensemble (DE) while remaining competitive and, in some cases, outperforming it in terms of robustness. Both theoretical analysis and experimental results validate the effectiveness of our method, demonstrating superior performance in robustness testing compared to baseline approaches.




<ccs2012>
<concept>
<concept_id>10002951.10003317.10003365.10010850</concept_id>
<concept_desc>Information systems Adversarial retrieval</concept_desc>
<concept_significance>500</concept_significance>
</concept>
</ccs2012>


[500]Information systems Adversarial retrieval



Boosting Certificate Robustness for Time Series Classification with Efficient Self-Ensemble
    Wei Emma Zhang
    September 9, 2024
===========================================================================================





§ INTRODUCTION

Background  
In recent years, time series data has become increasingly prevalent. As we transit into the era of Industry 4.0, countless sensors generate vast volumes of time series data <cit.>. Correspondingly, the application of deep neural networks (DNNs) has surged in popularity for time series classification (TSC)<cit.> as they have achieved remarkable success in various domains especially in computer vision (CV)<cit.>. However, DNNs exhibit vulnerabilities to minor perturbations in input data, often leading to misclassification and indicating low resistance to external disruption. This issue has attracted significant attention from the research community to the TSC domain. Fawaz et al. <cit.> first addressed the weakness of TSC towards adversarial attacks on the UCR dataset. Rathore <cit.> introduced target and untargeted attacks in TSC. Pialla <cit.> introduced a smooth version of the attack to make it undetectable, which was further improved by Chang <cit.> who enhanced the stealthiness by leveraging logits information. Ding <cit.> conducted effective black-box attacks on TSC, and Karim <cit.> used adversarial transformation networks to generate adversarial examples.  Other research is progressively expanding the landscape of time series attacks.



In contrast, the development of defenses has been relatively limited. Nevertheless, some works have focused on defending against these attacks. Tariq <cit.> proposed using anomaly detection methods to identify adversarial patterns in time series data. Kühne <cit.> combined adversarial training and selective classification to defend against attacks. Abdu <cit.> leveraged random convolutional kernel transformations (ROCKET) as feature extractors to constitute a robust classifier. Siddiqui <cit.> reviewed various adversarial training techniques in TSC benchmarking.


Challenges  
Despite substantial efforts to fortify models against such attacks, challenges persist. Although the above empirical defense methods can enhance robustness in some scenarios, they often fall short as new attacks continuously emerge, rendering existing defenses out of date <cit.>. Besides, there is no guarantee above regarding the extent of perturbations a model can withstand. This motivated the study of provable robustness in DNNs to obtain a certified guarantee to resist perturbations. Randomized smoothing, one of the most promising directions in certified defense, was recently proposed by Cohen, et al. <cit.>, Li, et al. <cit.>, and Lecuyer, et al. <cit.> with theoretical guarantees and empirical result. It is simple to transform an arbitrary base classifier f:ℝ^d →ℝ^k to a smoothed classifier g by training with i.i.d. Gaussian noise. The smoothed classifier g returns the most likely result according to the base classifier under x+ϵ∼𝒩(x, σ^2 I). In simpler terms, instead of predicting a specific point x, the model predicts the outcomes for the points surrounding x according to the Gaussian sampling and then combines these predictions through a voting process. This approach helps in making more robust and accurate predictions by considering the uncertainty around each point. 


Due to its achievement, in the field of time series, research on robustness has also begun to focus on these aspects. From 2022 to 2023, Yoon, et al. <cit.> and Liu, et al. <cit.> successively applied randomized smoothing techniques to univariate and multivariate time series forecasting tasks, thereby achieving comparatively good robust probabilistic time series forecasting. Additionally, Doppa, et al <cit.> proposed combining randomized smoothing with the statistical features of time series to enhance robustness under non-standard ℓ_p balls.


However, the aforementioned works <cit.>  did not thoroughly evaluate the specific performance of Randomized Smoothing on various TSC architectures. Our preliminary experiments found that in some cases of adversarial attacks, Randomized Smoothing did not effectively provide defense as expected (See Case Study in Section 6.3). So, how can Randomized Smoothing performance be improved? Currently, there are two main schools of thought on improving Randomized Smoothing: either by carefully designing the noise distribution or by enhancing the performance of the base classifier 
f. While training a classifier will be easier compared with the noise distribution design. For example, Salman, et al. <cit.> combined adversarial training with Randomized Smoothing to enhance performance and Zhai, et al. <cit.> directly optimized the target task as a loss. These methods either require lengthy training times or involve complex optimization. While our intuition is more straightforward, improving the performance of the base classifier by variance reduction through aggregation as  Miklós, et al. <cit.>. They found that the Deep Ensemble method <cit.> can indeed enhance model performance, however, this involves training multiple base classifiers, which can also be problematic in computation cost.


Motivation and Contribution  
Given the challenges posed by existing methods, we sought to design a simpler approach that does not require multiple training iterations. Initially, we considered methods like MC dropout <cit.> and masksemble <cit.>. MC dropout, being inherently stochastic during prediction, fails to provide theoretical guarantees. Masksemble, on the other hand, involves masking model weights and using a neural network to calculate scores for potential dropout, which complicates training and yields moderate results. Consequently, we propose a self-ensemble method that does not involve training multiple models directly but rather allows a single model to adapt to multiple data distributions as illustrated in Figure <ref>. Our contribution can be summarized into 4 folds:


    
  * Our method significantly reduces training time by m-fold (m is the number of ensembles of the base model). This involves augmenting the training of time series data with masks and then using a few randomly fixed masks for voting classification during prediction. 
    
    
  * We provide a detailed theoretical proof of this approach, showing that the self-ensemble method reduces the variance of classification margin in the base classifier, thereby enhancing the confidence lower bound of the top one class, p_A. This has also been observed in our experiments. 
    
    
  * To claim our findings, we conduct extensive experiments training the base classifier at different noise levels on the UCR benchmark datasets <cit.> using three different model architectures (CNN, RNN, Attention) and evaluated the performance of ensemble methods versus single models. Results show that under CNN and Attention models, self-ensemble significantly outperforms single models, achieving or even exceeding the performance of deep ensembles. 
    
    
  * We conduct a case study in Section 6.3 to assess the performance of these methods and benign models under PGD-ℓ_2 adversarial attacks in different projection radii, finding that self-ensemble can effectively resist adversarial attacks, especially in the model training in the datasets with poor robustness.





§ RELATED WORKS



 §.§ Adversarial Attack

In TSC, an adversarial attack refers to a malicious attempt to introduce slight perturbations to a time series x ∈ℝ^d to produce a closely related series x' ∈ℝ^d with the goal of altering the predicted label. This can be characterized by:

    argmax {f(x)}argmax {f(x')}, 
    
        x' = x + r, s.t. ||r||^2 ≪ ||x||^2.

Here, f(x) represents the predicted probability distribution over the labels for the input x. The perturbation r is intentionally small in magnitude relative to x as indicated by their norms.


Adversarial robustness refers to the ability to resist misclassification caused by adversarial perturbations. 



 §.§ Adversarial Defense

Adversarial defense can be categorized into empirical and certifiable defense. Early efforts primarily focus on empirical methods such as obfuscated gradients, model distillation, input transformation, adversarial detection, and adversarial training. 


Some Empirical Defense  Obfuscated gradients, a defense technique disrupting gradient continuity or hindering gradient acquisition to thwart attacks, were later successfully bypassed by Athalye et al. <cit.>, demonstrating their ineffectiveness. Model distillation <cit.> trains a teacher neural network on the original dataset and uses the teacher's class probabilities as soft targets for training a student network, enhancing resilience against adversarial attacks. However, adversaries may adapt their strategies if they know the distillation process. Other methods, like input transformation <cit.>, aim to convert adversarial samples into clean space using an "auxiliary module" for safe usage. Additionally, the rising popularity of diffusion techniques has led to the use of denoising techniques to remove noise from adversarial samples <cit.>. Anomaly detection also serves as a non-direct defense mechanism in adversarial settings <cit.>.


Adversarial Traning  Empirically, the most practical method is adversarial training. This method involves finding the perturbation from the maximum point within the loss surface using Projected Gradient Descent (PGD),  then minimizing it during the normal training process <cit.>. Alternating between these steps can smooth the loss surface near the input point, leading to a smaller Lipschitz constant. This makes it harder for attackers to find adversarial patterns near the input. The PGD adversarial training process can be described as follows:

    δ = max_δ, δ_p ≤ϵ L(f(x + δ), y),

where L is the loss function. The model is then trained to minimize this maximum loss:

    min_θ𝔼_(x,y) ∼𝒟[ max_δ_p ≤ϵ L(f_θ(x + δ), y) ].

Here, θ represents the model parameters and 𝒟 is the training data distribution respectively. However, adversarial training is time-consuming and lacks a theoretical guarantee.


Certified Defense  
In comparison with empirical defense, the certified defense can provide a theoretical guarantee rather than experimental findings. It can be defined as the following work flow. Given a classifier f : 𝒳→𝒴∈{1, …, k}, if f can correctly classify all samples to the same label within an l_p-ball of radius r centered at x, then the classifier is robust around x against l_p attacks with radius r. This indicates that the classifier maintains its predictive consistency for all inputs within this ball. 




§ METHODOLOGY




 §.§ Preliminary


Randomized Smoothing  Randomized Smoothing has recently been proposed <cit.> as its power to resist adversarial attacks in a tight theoretical certified space. It is simple to transform an arbitrary base classifier f:ℝ^d →ℝ^k to a smoothed classifier g by training with i.i.d. Gaussian noise. The smoothed classifier g returns the most likely result according to the base classifier under x+ϵ∼𝒩(x, σ^2 I), i.e.


    g(x) = max_c ∈𝒴ℙ_δ∼𝒩(0, σ^2 I)(f(x + δ) = c),

where the noise level σ is a hyperparameter to trade off the accuracy and robustness. In other words, g will return the class c whose decision region {x' ∈ℝ^d: f(x') = c} has the largest probability measure under the distribution of 𝒩(x, σ^2 I). Cohen, et al. <cit.> recently proved a tight robustness guarantee in ℓ_2  norms for smoothing with Gaussian noise. Meanwhile, Lecuyer, et al. <cit.> and Li, et al. <cit.> also demonstrated that the smoothed classifier g will consistently classify within a certified radius around the input x under l_2 norm considerations, based on Differential Privacy (DP) and Rényi divergence <cit.> respectively <cit.>. In this paper, our theoretical analysis chooses divergence-based randomized smoothing from Li, et al. <cit.>. Details are provided in the THEORETICAL ANALYSIS section, Theorem 4.1 <cit.>.


Deep Ensemble for Randomized Smoothing  
Ensemble methods, classical in statistical machine learning, enhance predictive performance by aggregating multiple models to reduce their inter-covariance. Introduced in 2016 <cit.>, Deep Ensembles were designed to reduce uncertainty in neural network predictions. This approach has been adapted for Randomized Smoothing (RS) <cit.>, which narrows the prediction distribution range, effectively raising the lower bound of predictions. In this context, ensemble methods with Randomized Smoothing involve employing multiple base classifiers trained with identical architectures but different random seeds. Which was defined as:

    f̂(x) = 1/m∑_l=1^m f_l(x),

where f_l(x) are the logits of the outputs from classifiers, and each classifier f_l is trained independently using a unique seed. This configuration ensures diversity among the classifiers, which reduces variance to increase the lower confidence bound of the true majority class probability p_A. However, Deep Ensembles require extensive retraining, leading to resource inefficiency. To address this, we propose the following method. 



 §.§ Proposed Method


Random Mask Training  We propose a training algorithm that incorporates randomized masking based on a normal randomized smoothing procedure, as seen in Algorithm 1. Our intuition is to help models adapt to different types of masks during training, ensuring that the prediction confidence shows small differences among various masks (Details about how to implement masks are described in 5.3 Algorithm Setup). This method can avoid multiple training of deep ensemble, which greatly reduces the computational overhead. In the inference stage, we will leverage the diversity of these masks to obtain the self-ensemble, and the efficacy is competitive and even better in some cases compared with the deep-ensemble.


 



Self-ensemble Certification To certify the robustness of the smoothed classifier g around x, we designed the process as shown in Algorithm 2. In this framework, a set of fixed masks is randomly generated and kept consistent for all inputs. During the ensemble inference stage, a single sample undergoes m × n trials, where m and n are the total numbers of different masks and noises, respectively. For each noised input x, predictions based on these m masks are aggregated to produce the final output. The class resulting from that iteration increments its count by one. This process will iteratively run n times before the final prediction is determined by hard voting across the n noised inputs. We then calculate the confidence intervals based on the prediction counts for the top one and top two classes A and B using a multinomial distribution, taking the lower bound of p_A and the upper bound of p_B as conservative estimates of p_A and p_B. Based on the prediction, we obtain the lower bound of the certified radius L. This means that for all |x' - x| < L, the prediction will remain consistent.


The reason why we need to fix the masks is to ensure that the model will return identical results for the same input. This is a fundamental requirement of Theorem 4.1 <cit.>. Compared to the training process, masks play a different role in the certification stage. The same model with multiple masks can have a similar effect as a deep ensemble, significantly reducing the variance of p_A and p_B and increasing the gaps between p_A and p_B. This results in an increased certified radius, making the model more robust.









§ THEORETICAL ANALYSIS

To prove our claimed findings, we provide a detailed theoretical analysis. Starting with Theorem 4.1 proved by Li et al. <cit.>, a tight lower bound for the certified radius of an arbitrary classifier g based on Rényi divergence is described as follows:


Theorem 4.1   (From Li, et al <cit.>). Suppose 𝐱∈𝒳, and a potential adversarial example 𝐱' ∈𝒳, such that 𝐱 - 𝐱'_2 ≤ L. Given a k-classifier f : 𝒳∈ℝ^d → p∈ℝ^k, let f(𝒩(𝐱, σ^2 I)) ∼ (p_1, …, p_k) and f( 𝒩(𝐱', σ^2 I)) ∼ (p'_1, …, p'_k), and M_p(x_1,...,x_n) ← (1/n∑_i=1^n x_i^p)^1/p. If the following condition is satisfied, with p_A and p_B being the first and second largest probabilities in {p_i}:

    sup_α > 1( -2σ^2/αlog(1 - 2M_1(p_A, p_B) + 2M_1-α(p_A, p_B)) ) ≥ L^2
    ,

then *arg max_i p_i = *arg max_j p'_j.


To increase the lower bound of L, it is intuitive to increase the noise level σ, however, the model may struggle with the noised input, leading to a drop in accuracy. In turn, increase the gaps between p_A and p_B can also raise L. Here, we propose the self-ensemble method, which can reduce the variance of predictions p_A and p_B and enlarge their gaps. The variance and expectation after self-ensemble can be described as follows:

 
Lemma 4.2  
Let f be a pre-trained classifier with masks and noises over ℝ^d →ℝ^k. Let M_i be a set of m independent binomial masks applied to the input 𝐱 with probability p → 1  of each element being 1. Assume that the input noise ϵ∼𝒩(0, σ^2 I). The m-ensemble average model output is given by:





    y = 1/m∑_i=1^m f(M_i ⊙ (𝐱 + ϵ)).

The expectation and variance of the ensemble average output are:

    𝔼[y] = 𝔼[f(𝐱)] = 𝐜,

where 𝐜∈ℝ^k is the expectation output of a fixed clean x over randomness in the training process. 



    Var(y)     = 1/m (1+ζ_c (m-1)) Σ_c 
        + 1/m (1+ζ_p (m-1)) σ^2 𝔼[J_f(M ⊙𝐱) J_f(M ⊙𝐱)^T]),

where Σ_c is the covariance matrix representing the variability across different training processes, characterizing the randomness. J_f(M ⊙𝐱) is the Jacobian matrix of f. ζ_c is the correlation coefficient for the masked output in the clean part, and ζ_p is the correlation coefficient for the masked Jacobian in the perturbed part.


Proof 4.2   Let y_i = f(M_i ⊙ (𝐱 + ϵ)) be the output for each mask. Since M_i and ϵ are independent, and ϵ has mean 0, we have:

    𝔼[y_i] = 𝔼[f(M_i ⊙ (𝐱 + ϵ))] = 𝔼[f(M_i ⊙𝐱)].

During the training process, we observed the network can achieve the same performance with and without the mask, especially when p is close to 1, the mask has minimal influence on the prediction, thus:

    𝔼[f(𝐱)] = 𝔼[f(M ⊙𝐱)] = 𝐜.

Averaging over multiple independent M_i:

    𝔼[y] = 𝔼[1/m∑_i=1^m y_i] = 1/m∑_i=1^m 𝔼[y_i] = 𝔼[f(𝐱)] = 𝐜.

Now, we compute the variance of the ensemble average output. According to the Taylor Expansion, y_i = f(M_i ⊙𝐱) + M_i ⊙ J_f(𝐱) ϵ.
For the variance of a single y_i:

    Var(y_i) = Σ_c + σ^2 J_f(M_i ⊙𝐱) J_f(M_i ⊙𝐱)^T.

For the ensemble average output y:

    Var(y) = Var(1/m∑_i=1^m y_i).

Assuming the correlation between y_i and y_j is parameterized by coefficients ζ_c and ζ_p, we have:

    Var(y) = 1/m^2( ∑_i=1^m Var(y_i) + ∑_i ≠ jCov(y_i, y_j) ),

where Cov(y_i, y_j) = ζ_c Σ_c + ζ_p σ^2 J_f(M_i ⊙𝐱) J_f(M_i ⊙𝐱)^T. Therefore:

    Var(y)     = 1/m (1+ζ_c (m-1)) Σ_c 
       + 1/m (1+ζ_p (m-1)) σ^2 𝔼[J_f(M ⊙𝐱) J_f(M ⊙𝐱)^T]).

Thus, the Lemma is proved.

 
Variance Reduction   When m = 1 and the mask is all ones, the variance degenerates to:

    Var(y) = Σ_c + σ^2 J_f(𝐱) J_f(𝐱)^T.

Instead, after mask ensemble, the two parts of the variance are reduced to 1/m (1 + ζ_c (m-1)) and 1/m (1 + ζ_p (m-1)) times of the original, respectively. When the correlation coefficients are less than 1, these values will always be less than 1 and approach the correlation coefficients ζ_c and ζ_p respectively as m increases.
Additionally, the reduction in variance after masking also comes from the shrinkage of the Jacobian matrix. Due to the implementation of the mask, the Jacobian matrix should be:

    𝔼[J_f(M ⊙𝐱) J_f(M ⊙𝐱)^T] = p^2 J_f(𝐱) J_f(𝐱)^T.

Here we assume that the Jacobian matrix is similar in both masked and unmasked training settings. Although in some cases they might not be exactly the same, the difference is relatively small compared to the factor p^2. Therefore, the variance after applying the mask ensemble is:

    Var(y)    ≈1/m (1 + ζ_c (m-1)) Σ_c 
        + 1/m (1 + ζ_p (m-1)) σ^2 p^2 J_f(𝐱) J_f(𝐱)^T.

As the number of ensembles m approaches infinity, the variance can be further simplified to:

    lim_m→∞Var(y) ≈ζ_c Σ_c + ζ_p  p^2 σ^2 J_f(𝐱) J_f(𝐱)^T.

This result demonstrates that the variance of the ensemble model is reduced due to both the correlation coefficients and the shrinkage of the Jacobian matrix resulting from the masking process. 









Theorem 4.3  
Let f be a pre-trained classifier over x ∈ℝ^d → y ∈ℝ^k. Given an input x, let x' = x + ϵ where ϵ∼𝒩(0, σ^2 I). Let c_A be the score for class A and c_i be the score for any other class i ≠ A. The classification margins are defined as z_i = c_A - c_i,  ∀ i ≠ A. Then, the probability P_A = P(z_i > 0  , ∀  i ≠ A) increases as the variance of y decreases.

Proof 4.3   According to the local linear assumption, the output of the classifier is perturbed as f(x') ∼ f(x) + A ·ϵ, where A is an auxiliary matrix. Thus, f(x') ∼𝒩(f(x), Aσ^2A^T). According to the variance reduction of y from Lemma 4.2, we can know that the auxiliary matrix represents the reduction adjustment of the variance to each class. Thus we know that A should be a matrix with all elements < 1. Let c_A ∼𝒩(μ_A, σ_A^2) and c_i ∼𝒩(μ_i, σ_i^2) for i ≠ A. The classification margins z_i are:

    z_i = c_A - c_i ∼𝒩(μ_z_i, σ_z_i^2).

Here, μ_z_i = μ_A - μ_i and σ_z_i^2 = σ_A^2 + σ_i^2. Since z_i ∼𝒩(μ_z_i, σ_z_i^2), the probability P(z_i > 0) is:

    P(z_i > 0) = Φ( μ_z_i/σ_z_i),

where Φ is the CDF of the standard normal distribution.
As the variance σ_z_i^2 decreases, μ_z_i/σ_z_i increases because μ_z_i is fixed. Then, we have:

    Φ( μ_z_i/σ_z_i,new) > Φ( μ_z_i/σ_z_i),

where σ_z_i,new < σ_z_i. Therefore, P(z_i > 0) increases as σ_z_i^2 decreases. Using the union bound ∀ i ≠ A, we have:

    P_A = P(z_i > 0, ∀  i ≠ A) ≥∏_i ≠ A P(z_i > 0).

As each P(z_i > 0) increases with decreasing σ_z_i^2, the overall probability P_A increases. Thus, as the variance σ_z_i^2 decreases, the success probability P_A increases. Thus, as P_A increases, we can simply set P_B = 1 - P_A to make a conservative estimation. Then, the lower bound L in Theorem 4.1 (Li et al.) will increase. As depicted in Figure <ref>, when we fix the noise level σ, higher p_A can guarantee a larger radius.



We further validated our theoretical findings through experiments. Figure 3 illustrates a sample case from our benchmark dataset, ChlorineConcentration Dataset, with a standard deviation (σ) of 0.4 and 1000 noise samples. The distribution of the classification margins z of the ensemble method exhibits fewer values distributed less than 0, indicating a higher probability of being classified as class A (the top 1 prediction of the smoothed classifier g), thus certifying a larger radius. Additionally, the right four distributions in Figure 4 depict the distribution of c_A(x), revealing a noticeable variance reduction of the z distribution in the self-ensemble method compared with the Single one. Notably, the mean values (represented by the green line) of M_B and M_C closely align with the Single method, supporting our assumption that the mask should have minimal influence on the expectation value.




§ EXPERIMENTS



 §.§ Experimental Setup





The whole project was developed using PyTorch, and conducted on a server equipped with Nvidia RTX 4090 GPUs, 64 GB RAM, and an AMD EPYC 7320 processor. Table <ref> gives the training time of each method over different datasets in InceptionTime architecture (other models show a similar trend), which claimed our contribution that self-ensemble takes less time for the training process compared with Deep Ensemble (5 models ensemble).     https://github.com/Chang-George-Dong/Boosting-Certificate-Robustness-for-Time-Series-Classification-with-Efficient-Self-Ensemble.git[GitHub]







 §.§ Datasets

To evaluate the proposed method and compare it with the baseline, we conduct our experiment in diverse univariate time series benchmark datasets <cit.>. 
Table <ref> gives a detailed description of these datasets. For the case study, we selected the ChlorineConcentration and Cricket datasets for our demonstration, as ChlorineConcentration is the most susceptible to adversarial attacks among these datasets, while CricketX is more robust. Thereby to provide a clearer illustration of the efficacy of all methods.











 §.§ Algorithm Setup

We implemented three different DNN architectures: InceptionTime (CNN) <cit.>, LSTM-FCN (RNN) <cit.>, and MACNN (Attention) <cit.>. All models were trained with noise level σ of 0, 0.1, 0.2, 0.4, 0.8 and 1.6 for 1000 epochs to achieve a smoothed version. We used two kinds of masking settings: binomial masking and continuous masking. In binomial masking (M_B), for each timestamp, there is a probability p to keep the original value; otherwise, it is set to 0. In continuous binomial masking (M_C), there are several continuous mask-lets, combing to the total masking length fixed to p times the sequence length, and the maximum mask segment length set to half of p times the sequence length. Unless specified otherwise, all p were set to 0.9. During the certifying phase, we found that 1000 noise draws were sufficient, and set confidence level β = 0.001. The certified noise level  σ used the same settings as the training phase. For adversarial attacks, we used the PGD-ℓ_2  attacks from Madary <cit.> with different epsilons, as listed in the result tables for evaluation.



 §.§ Evaluation Metrics

We evaluate two key metrics on these models: (i) the certified accuracy at predetermined radius r and (ii) the average certified radius (ACR). In the case study, we evaluate the adversarial robustness using the Attack Success Rate which is the proportion of misclassified samples by adversarial perturbation.


§ RESULT










 §.§ Main Results

Comprehensive Performance Comparison  In Table <ref>, we compare our self-ensemble method using two kinds of masks, M_B and M_C, Deep Ensembles (DE), and the Single model across three different types of networks, trained using five different noise levels σ. The results show a consistent trend: as the noise level σ increases, the Averaged Certified Radius (ACR) increases, while the accuracy decreases. In almost every case, ensemble models perform better compared to their individual counterparts. This implies masking is effective in a wide range of datasets. 


Notably, the self-ensemble methods using M_B and M_C both perform well on InceptionTime and MACNN networks, often competing with and even outperforming the DE method, while requiring only 1/5 of the training time (all ensemble methods in this table are 5-model ensembles). However, for the LSTM-FCN architecture, M_B and M_C perform worse than the single model. This is primarily because sequential models are highly sensitive to missing values; each input in a sequence model is critical, and a single missing value can easily skew the model's predictions.  In contrast, convolutional models are more robust to missing values, especially when the receptive field is large. Convolutions can learn from neighboring values and effectively handle local missing patterns. Missing values in convolutions can be mitigated by the surrounding non-missing values, reducing the bias introduced by the missing data. 
Therefore, it is safer to use masks in convolutional and attention-based networks.


Certifed Accuracy over Radius  
In comparison with the stationary performance shown in Table <ref>, Figure <ref> illustrates the performance under increasing perturbations, which better reflects real-world scenarios with adversarial noise. We observe that the self-ensemble method is resilient to a wider range of perturbations (indicated by the radius in the figure) while maintaining accuracy, followed by the DE and Single models. However, as the radius increases, the accuracy of all models tends to deteriorate,  as explained by Madry et al., "Robustness May Be at Odds with Accuracy" <cit.>. Nevertheless, the degradation is slower in the ensemble methods compared to the single model, highlighting the benefits of ensemble approaches in maintaining performance under adversarial conditions.



 §.§ Ablation Study
 




Ensemble Size   Figure <ref> shows the certified accuracy vs. radius for different ensemble sizes. Both methods demonstrate that increasing the ensemble size improves certified robustness. However, the improvement tends to saturate when the ensemble size reaches 10 for M_C and 5 for M_B, while requiring double and quadruple the inference time, respectively. Therefore, we choose an ensemble size of 5 as it provides a good balance between performance and computational efficiency.









Keep Ratio  
Figure <ref> shows the influence of the keep ratio on the accuracy-radius curve. It is evident that a high keep ratio (masking to 1) p in both cases ensures better-certified accuracy over the radius and is close to the performance of the non-mask method. This is highly aligned with our theoretical assumptions, a high keep ratio is very close to no masking. The same trend clarifies that a higher keep ratio is practically important to align with the theoretical analysis. Additionally, models tend to be more sensitive to a low keep ratio of M_B compared to M_C, as M_C can retain more continuous patterns that help convolution correctly activate. This implies that a segment is sufficient for CNN recognition, and it may be possible to alleviate the dimensionality curse <cit.> in Randomized Smoothing by distilling to the input size.






 §.§ Case Study: Adversarial Robustness


To validate our method's performance on datasets with insufficient robustness compared to the Single model, we also conducted adversarial attacks using PGD-ℓ_2 against the Benign models, single smoothed, and various ensemble smoothed classifiers (σ = 0.4). Table <ref> illustrates the Attack Success Rate (ASR) versus the perturbation level ϵ under PGD-ℓ_2 attack on both the ChlorineConcentration (CC) and CricketX datasets. The results clearly demonstrate that the ensemble approach is significantly more effective in mitigating attacks compared to the Single model. Notably, on the ChlorineConcentration dataset, M_B achieves a remarkable reduction in attack success rate from 0.98 to nearly 0.24, whereas the Single model remains vulnerable with a 55% ASR. As mentioned before, the Single model did not demonstrate strong robustness on this dataset. In contrast, on the more robust CricketX dataset, all four methods perform similarly, with DE being slightly more effective. These findings underscore the robustness of self-ensemble methods in defending against adversarial attacks, highlighting their superiority over individual models.



§ CONCLUSION



In this paper, we utilized self-ensemble to decrease the variance of classification margins, thereby boosting the lower bound of top-1 prediction confidence and certifying a larger radius to enhance model robustness. Theoretical proofs and experimental findings substantiate the effectiveness of our approach, leveraging ensemble techniques while mitigating computational overhead. Our case study illustrates superior robustness compared to baseline methods against adversarial cases.


Future work   Despite its efficacy, we observed suboptimal performance in RNN architectures, and it may attributed to their sensitivity to missing values. To address this, we aim to train sequential models resistant to the random missing value in the future. Additionally, in the certification process, the use of random fixed masks may lead to unstable results. Hence, we plan to focus on mask design to enhance the stability of this algorithm.



§ ACKNOWLEDGMENTS

This research work is supported by Australian Research Council Linkage Project (LP230200821),
Australian Research Council Discovery Projects (DP240103070 ),
Australian Research Council ARC Early Career Industry Fellowship (IE230100119), 
Australian Research Council ARC Early Career Industry Fellowship (IE240100275), and University of Adelaide, Sustainability FAME Strategy Internal Grant 2023.
unsrt

