TASAR: Transferable Attack on Skeletal Action Recognition
Yunfeng Diao Baiqi Wu Ruixuan Zhang Ajian Liu Xingxing Wei Meng Wang He Wang


Thermoelectricity  at a gallium-mercury liquid metal interface
    [
     
==============================================================



§ ABSTRACT 
Skeletal sequences, as well-structured representations of human behaviors, are crucial in Human Activity Recognition (HAR). The transferability of adversarial skeletal sequences enables attacks in real-world HAR scenarios, such as autonomous driving, intelligent surveillance, and human-computer interactions. However, existing Skeleton-based HAR (S-HAR) attacks exhibit weak adversarial transferability and, therefore, cannot be considered true transfer-based S-HAR attacks. More importantly, the reason for this failure remains unclear. In this paper, we study this phenomenon through the lens of loss surface, and find that its sharpness contributes to the poor transferability in S-HAR. Inspired by this observation, we assume and empirically validate that smoothening the rugged loss landscape could potentially improve adversarial transferability in S-HAR. To this end, we propose the first Transfer-based Attack on Skeletal Action Recognition, TASAR. TASAR explores the smoothed model posterior without re-training the pre-trained surrogates, which is achieved by a new post-train Dual Bayesian optimization strategy. Furthermore, unlike previous transfer-based attacks that treat each frame independently and overlook temporal coherence within sequences, TASAR incorporates motion dynamics into the Bayesian attack gradient, effectively disrupting the spatial-temporal coherence of S-HARs. To exhaustively evaluate the effectiveness of existing methods and our method, we build the first large-scale robust S-HAR benchmark, comprising 7 S-HAR models, 10 attack methods, 3 S-HAR datasets and 2 defense models. Extensive results demonstrate the superiority of TASAR. Our benchmark enables easy comparisons for future studies, with the code available in the supplementary material.







§ INTRODUCTION




Human Activity Recognition (HAR) has a wide range of application scenarios, such as human-computer interactions, bio-mechanics, and virtual reality <cit.>. Skeleton sequences has been widely used in HAR <cit.>, because skeleton data is a well-structured data representation of human behaviors and robust to lighting, occlusion, and view angles. Recent research <cit.> demonstrates that S-HARs are less robust than expected when adversarial perturbations are added to the testing data, causing skeletal classifiers to produce incorrect outputs. Previous skeleton-based attacks are mainly developed under the white-box setting <cit.>, where the attacker knows the structure and parameters of a given model, or under the query-based setting <cit.>, where the adversary can request numerous queries against the target model <cit.>. But both settings are impractical in real-world HAR scenarios, such as autonomous driving <cit.>, intelligent surveillance <cit.>, and human-computer interactions <cit.>, in which the white-box information and a large number of queries are not attainable.

In contrast, transfer-based attacks, which craft adversarial examples from surrogate models and then transfer them to target black-box models, present a more realistic threat under the free-query black-box setting. Although adversarial transferability has been widely studied across various tasks <cit.>, research on S-HAR remains limited. Recent studies have attempted to apply white-box S-HAR attacks against black-box models via surrogate models <cit.>. However, their transferability is low and highly sensitive to the surrogate choice <cit.>. Consequently, existing S-HAR attacks cannot be regarded as transferable. Similarly, previous transfer-based attacks <cit.> are successful on image data, but show poor transferability when applied to skeletal motion. Contrary to the common belief that adversarial examples transfer well across different model architectures and parameters <cit.>, existing attacks do not exhibit transferability on S-HAR, raising doubt on the usefulness of adversarial transferability in this domain <cit.>. More importantly, the reason for this failure remains unclear.


To study this phenomenon, we begin by examining the factors that hinder adversarial transferability in S-HARs. Our preliminary experiments reveal that adversarial transferability is highly sensitive to the chosen surrogates (<ref>). This finding motivates us to further explore the differences between surrogate models from the view of loss surface smoothness, as previous research has demonstrated that the smoothness of the loss surface significantly impacts adversarial transferability <cit.>. As shown in <ref>, surrogates with a smoothed loss landscape have higher transferability than surrogates with a sharp loss landscape. Therefore, unlike most existing attacks that focus on the backpropagation process, we argue that the S-HAR attack should prioritize smoothening the surrogate's landscape in the training process. 



Bayesian neural networks have been shown to exhibit lower sharpness and better generalization ability <cit.>. Therefore, attacking the entire model distribution is more likely to generalize for S-HARs <cit.>. However, it is not straightforward to design such a transferable Bayesian attack for S-HAR. First, skeletal classifiers contain at least several millions of parameters. Sampling from the posterior distribution for such complex models can be computationally expensive and consume a heavy memory footprint. Second, most prior transferable attacks are specifically designed for static data, e.g. images. However, most S-HAR models learn the spatial-temporal features because skeletal data contains rich motion dynamics. Naive adaptation of the Bayesian attack may ignore the spatial-temporal coherence when issuing attacks. How to consider the motion dynamics in Bayesian attacks has not been explored.

To tackle these challenges, we propose the first Transfer-based Attack specifically designed for Skeletal Action Recognition, TASAR, based on a Post-train Dual Bayesian optimization that significantly smoothens the rugged landscape (see <ref>). Our post-train Bayesian strategy freezes the pre-trained surrogate and appends small extra Bayesian components behind it, to turn a single surrogate into a Bayesian one without the need for re-training, thereby speeding up the training process and avoiding a heavy memory footprint. Next, we integrate the temporal motion gradient in a Bayesian manner to disrupt the temporal coherence between sequences over time. A high-level illustration of our method is presented in <ref>. Our contributions are summarized as follows.


  * We systematically investigate the previously unknown reason for low adversarial transferability in S-HAR through the lens of loss surface smoothness, and introduce the first transfer-based attack on S-HAR to address this issue.

  * We propose a novel post-train Dual Bayesian Motion attack, that explores the smoothed model posterior and considers the spatial-temporal coherence from a Bayesian perspective, without requiring re-training of the pre-trained surrogate model. 

  * We build the first comprehensive robust S-HAR evaluation benchmark RobustBenchHAR. RobustBenchHAR consists of 7 S-HAR models with diverse GCN structures and more recent Transformer structures, 10 attack methods, 3 datasets and 2 defense methods. Extensive experiments on this benchmark demonstrate the superiority and generalizability of TASAR.














§ RELATED WORK


Skeleton-Based Human Action Recognition.
Early S-HAR research employed convolutional neural networks (CNNs) <cit.> and recurrent neural networks (RNNs) <cit.> to extract motion features in the spatial domain and temporal domains, respectively. However, skeleton data, inherently a topological graph, poses challenges for feature representation using traditional CNNs and RNNs. Recent advances with graph convolutional networks (GCNs) <cit.> have improved performance by modeling skeletons as topological graphs, with nodes corresponding to joints and edges to bones <cit.>. Subsequent improvements in graph designs and network architectures include two-stream adaptive GCN (2s-AGCN) <cit.>, directed acyclic GCN (DGNN) <cit.>, multi-scale GCN (MS-G3D) <cit.>, channel-wise topology refinement (CTR-GCN) <cit.> and auxiliary feature refinement (FR-HEAD) <cit.>. Alongside advancements in GCN-based models, recent studies are exploring temporal Transformer structures for S-HARs <cit.>, but their vulnerability remains unexplored. This work is the first to assess the adversarial robustness of Transformer-based S-HARs.



Adversarial Attacks on S-HAR.
Adversarial attacks <cit.> highlight the susceptibility of deep neural networks and have been applied across different data types. Recently, attacks on S-HAR have garnered increasing attention. CIASA <cit.> proposes a constrained iterative attack via GAN <cit.> to regularize the adversarial skeletons. SMART <cit.> proposes a perception loss gradient. Tanaka et al. <cit.> suggest only perturbing skeletal lengths, and evaluate robustness via Fourier analysis <cit.>. These methods are white-box attacks, requiring full knowledge of the victim model. In contrast, BASAR <cit.> proposes motion manifold searching to achieve the query-based black-box attack. FGDA-GS <cit.> estimates gradient signs to further reduce query numbers.

Compared to white-box and query-based attacks, transfer-based attacks <cit.> pose a more practical threat as real-world HAR scenarios typically cannot access white-box information or extensive querying. While existing white-box S-HAR attacks <cit.> can be adapted for transfer-based scenarios, they suffer from low transferability and sensitivity to surrogate choices. Lu et al. <cit.> proposes a no-box attack for S-HAR but also lacks transferability. Numerous transfer-based attacks, including gradient-based <cit.>, input transformation <cit.>, and ensemble-based methods <cit.>, exhibit high transferability across various tasks but struggle in skeletal data <cit.>. Therefore, there is an urgent need to develop a transferable attack for skeleton-based action recognition.












§ METHODOLOGY



 §.§ Preliminaries

We denote a clean motion 𝐱∈𝒳 and its corresponding label y ∈𝒴. Given a surrogate action recognizer f_θ parametrized by θ, f_θ is trained to map a motion 𝐱 to a predictive distribution p(y |𝐱, θ). The white-box attack can be optimized by minimizing the predictive probability:


    min_𝐱 - 𝐱_p≤ϵ  p(y |𝐱, θ)



where 𝐱 is the adversarial example and ϵ is the perturbation budget. ·_p is the l_p norm distance. The procedure of transfer-based attack is firstly crafting the adversarial example 𝐱 by attacking the surrogate model, then transferring 𝐱 to attack the unseen target model. In <ref>, since the transferable adversarial examples are optimized against one surrogate model, the adversarial transferability heavily relies on the surrogate model learning a classification boundary similar to that of the unknown target model. While possible for image classification, it proves unrealistic for S-HAR <cit.>. 



 §.§ Motivation


Existing S-HAR attacks have shown outstanding white-box attack performance but exhibit low transferability <cit.>. Similarly, previous transfer-based attacks <cit.>, successful on image data, also show poor transferability when applied to skeletal motion <cit.>. Naturally, two questions occur to us: (1) Why do existing adversarial attacks fail to exhibit transferability in skeletal data? (2) Do transferable adversarial examples truly exist in S-HAR?

To answer these questions, we start by generating adversarial examples using various surrogate skeletal recognizers and then evaluate their adversarial transferability. Obviously, in <ref>, the transferability is highly sensitive to the chosen surrogates, e.g. CTR-GCN <cit.> as the surrogate exhibits higher transferability than ST-GCN <cit.>. This observation motivates us to further investigate the differences between surrogate models. Previous research <cit.> has proven that adversarial examples generated by surrogate models with a less smooth loss landscape are unlikely to transfer across models. Therefore, we investigate the smoothness of the loss landscape across different surrogate models. In <ref>, We visualize the loss landscape of ST-GCN and CTR-GCN trained on the skeletal dataset NTU-60 <cit.> and compare their smoothness to the ResNet-18 <cit.> trained on CIFAR-10 <cit.>. More landscape visualizations can be found in Appendix B. By analyzing the loss surface smoothness, we have two findings: (1) The loss surface of models trained on skeletal data is much sharper than those trained on image data, leading to a relatively low transferability. This suggests that adversarial examples within a sharp local region are less likely to transfer across models in S-HAR, potentially explaining our first question. (2) CTR-GCN has a flatter loss landscape compared to ST-GCN, making it a more effective surrogate for higher transferability. Consequently, we argue that using a surrogate with a smoothed loss landscape will significantly enhance adversarial transferability in S-HAR.



In this work, motivated by evidence that Bayesian neural networks (BNNs) exhibit low sharpness and good generalization <cit.>, we aim to construct a Bayesian surrogate by sampling from the model posterior space to smoothen the rugged loss landscape. From a Bayesian perspective, <ref> can be reformulated by approximately minimizing the Bayesian posterior predictive distribution:



    𝐱 - 𝐱_p≤ϵmin  p(y |𝐱, 𝒟)  = 𝐱 - 𝐱_p≤ϵmin 𝔼_θ∼ p(θ|𝒟) p(y |𝐱, θ)



where p(θ|𝒟) ∝ p(𝒟|θ) p(θ), in which 𝒟 is the dataset and p(θ) is the prior of model weights.



 §.§ A Post-train Bayesian Perspective on Attack


Unfortunately, directly sampling from the posterior distribution of skeletal classifiers is not a straightforward task due to several factors. First, directly sampling the posterior is intractable for large-scale skeletal classifiers. Although approximate methods such as MCMC sampling <cit.> or variational inference <cit.> are possible, sampling is prohibitively slow and resource-intensive due to the high dimensionality of the sampling space, which typically involves at least several million parameters in skeletal classifiers. In addition, skeletal classifiers normally contain a large number of parameters and are pre-trained on large-scale datasets <cit.>. Consequently, it is not practical for end-users to re-train the surrogate in a Bayesian manner, as the training process is time-consuming. 





To solve the above issues, we propose a new post-train Bayesian attack. We maintain the integrity of the pre-trained surrogate while appending a tiny MLP layer g_θ^' behind it, connected via a skip connection. Specifically, the final output logits can be computed as: logits=g_θ^'(f_θ(𝐱))+f_θ(𝐱). In practice, we adopt Monte Carlo sampling to optimize the appended Bayesian model:



    θ^'max 𝔼_θ^'∼ p(θ^'|𝒟,θ) p(y |𝐱, θ, θ^') 
    ≈θ_k^'max 1/K∑_k = 1^K p(y |𝐱 ,θ, θ_k^'), 
         θ_k^'∼ p(θ^'|𝒟,θ)



where K is the number of appended models. Correspondingly, <ref> can be approximately solved by performing attacks on the ensemble of tiny appended models:


    δ_p≤ϵmin 1/K∑_k = 1^K p(y |𝐱+δ ,θ, θ_k^'), θ_k^'∼ p(θ^'|𝒟,θ)



Our post-train Bayesian attack offers two advantages. First, the appended models are composed of tiny MLP layers, getting a similar memory cost to a single surrogate. Second, by freezing f_θ, our post-train Bayesian strategy keeps the pre-trained surrogate intact, avoiding re-training the pre-trained surrogate. More importantly, training on g_θ^' is much faster than on f_θ due to the smaller model size of g_θ^'. 




 §.§ Post-train Dual Bayesian Motion Attack

In our preliminary experiments, we found that a naive application of post-train Bayesian attack (<ref>) already surpassed the adversarial transfer performance of existing S-HAR attacks, which demonstrates the effectiveness of smoothening the loss surface of surrogates. However, its performance remains slightly inferior to the Bayesian attack via re-training a Bayesian surrogate <cit.>(<ref>). This performance gap is understandable, as we avoid the prohibitively slow process of sampling the original posterior distribution θ∼ p(θ|𝒟) by using a tiny Bayesian component for post-training instead. To further eliminate the trade-off between attack strength and efficiency, we propose a novel post-train dual Bayesian optimization for smoothed posterior sampling, to sample the appended models with high smoothness for better transferability (<ref>). Moreover, unlike previous transfer-based attacks that assume each frame is independent and ignore the temporal dependency between sequences, we integrate motion dynamics information into the Bayesian attack gradient to disrupt the spatial-temporal coherence of S-HAR models. We name our method Post-train Dual Bayesian Motion Attack.







  §.§.§ Post-train Dual Bayesian Optimization.


This motivation is based on the view that models sampled from a smooth posterior, along with the optimal approximate posterior estimating this smooth posterior, have better smoothness <cit.>. To this end, we aim for proposing a smooth posterior for learning post-train BNNs, hence possibly possessing higher adversarial transferability. Specifically, inspired by the observation that randomized weights often achieve smoothed weights update <cit.>, we add Gaussian noise to smooth the appended network weights. This is achieved by a new post-train dual Bayesian optimization: 



    θ^'max 𝔼_θ^'∼ p(θ^'|𝒟,θ)𝔼_Δθ^'∼𝒩(0, σ^2𝐈) p(y |𝐱, θ, θ^'+Δθ^')



For any appended model sampled from the posterior, <ref> ensures that the neighborhood around the model parameters has uniformly low loss. We further use dual Monte Carlo sampling to approximate <ref>:



    min _θ^'_k∼ p(θ^'|𝒟, θ)1/M K∑_k=1^K∑_m=1^M L(𝐱, y, θ, θ_k^'+Δθ^'_km) , 
    Δθ^'_km∼𝒩(0, σ^2𝐈)



where L is the classification loss. Considering dual MCMC samplings computationally intensive, we instead consider the worst-case parameters from the posterior, followed by <cit.>. Hence <ref> can be equivalent to a min-max optimization problem, written as:


    min _θ^'_k∼ p(θ^'|𝒟, θ)max _Δθ^'∼𝒩(0, σ^2𝐈)1/K∑_k=1^K L(𝐱, y,θ, θ_k^'+Δθ^') ,  
      p(Δθ^') ≥ξ








The confidence region of the Gaussian posterior is regulated by ξ. We discuss the sensitivity to ξ in the Appendix B. The entanglement between θ^' and Δθ^' complicates gradient updating. To simplify this issue, we utilize Taylor expansion at θ^' to decompose the two components:



    min _θ^'_k∼ p(θ^'|𝒟, θ)max _Δθ^'∼𝒩(0, σ^2𝐈)1/K∑_k=1^K [L(𝐱, y,θ ,θ_k^') 
    
     +∇_θ^'_k L(𝐱, y,θ, θ_k^')^TΔθ^'], p(Δθ^') ≥ξ




Since Δθ^' is sampled from a zero-mean isotropic Gaussian distribution, the inner maximization can be solved analytically. We introduce the inference details and mathematical deduction in Appendix A. As shown in <ref>, the loss landscape optimized by post-train Dual Bayesian is significantly smoother than vanilla post-train Bayesian.





 



























 


  §.§.§ Temporal Motion Gradient in Bayesian Attack.


Post-train Dual Bayesian Motion Attack can be performed with gradient-based methods such as FGSM <cit.>:


    𝐱=𝐱 + α·sign( ∑_k=1^K∑_m=1^M∇ L(𝐱, y, θ, θ_k^' + Δθ^'_km))



where α is the step size. For notational simplicity, we notate the classification loss L(𝐱, y, θ, θ_k^' + Δθ^'_km) as L(𝐱). Assume a motion with t frames 𝐱=[x_1, x_2, ⋯, x_t], this attack gradient consists of a set of partial derivatives over all frames ∇ L(𝐱)=[∂ L(𝐱)/∂ x_1, ∂ L(𝐱)/∂ x_2, ⋯, ∂ L(𝐱)/∂ x_t]. The partial derivative ∂ L(𝐱)/∂ x_t assumes each frame is independent, ignoring the dependency between frames over time. This assumption is reasonable for attacks on static data such as PGD <cit.> while infeasible for skeletal motion attacks. In skeletal motion, most S-HAR models learn the spatial-temporal features <cit.>, hence considering motion dynamics in the computing of attack gradient can disrupt the spatial-temporal coherence of these features, leading to more general transferability. To fully represent the motion dynamics, first-order (velocity) gradient (∇ L(𝐱))_d1 and second-order (acceleration) gradient information (∇ L(𝐱))_d2 should also be considered. To this end, we augment the original position gradient with the motion gradient, then <ref> becomes:



    𝐱 =𝐱 + α·sign( ∑_k=1^K∑_m=1^M∇ L_dyn(𝐱)) 
    ∇ L_dyn(𝐱) = ∑_n=0^2 w_n (∇ L(𝐱))_dn, ∑_n=0^2 w_n = 1










where (∇ L(𝐱))_d0= ∇ L(𝐱). Motion gradient can be computed by explicit modeling <cit.> or implicit learning <cit.>. Given that implicit learning requires training an additional data-driven model to learn the motion manifold, which increases computational overhead, we opt for explicit modeling. Inspired by <cit.>, we employ time-varying autoregressive models (TV-AR)<cit.> because TV-AR can effectively estimate the dynamics of skeleton sequences by modeling the temporary non-stationary signals <cit.>. We first use first-order TV-AR(f_d 1) and second-order TV-AR(f_d 2) to model human motions respectively:


    f_d 1:x^i_t   =A_t·x^i_t-1+B_t+γ_t
    
      f_d 2: x^i_t   =C_t·x^i_t-1+D_t·x^i_t-2+E_t+γ_t



where the model parameters β_t^1=[A_t, B_t] and β_t^2=[C_t, D_t, E_t] are all time-varying parameters and determined by data-fitting. γ_t is a time-dependent white noise representing the dynamics of stochasticity. Using <ref>, the first-order motion gradient can be derived as:


    (∂ L(𝐱^i)/∂x^i_t-1)_d 1=∂ L(𝐱^i)/∂x^i_t-1+∂ L(𝐱^i)/∂x^i_t· A_t



Similarly, second-order dynamics can be expressed as below by using <ref>:



    (∂ L(𝐱^i)/∂x^i_t-2)_d 2=∂ L(𝐱^i)/∂x^i_t-2+∂ L(𝐱^i)/∂x^i_t-1· C_t-1
    
      +∂ L(𝐱^i)/∂x^i_t·(D_t+C_t· C_t-1)



where C_t=∂x^i_t/∂x^i_t-1 and D_t=∂x^i_t/∂x^i_t-2. After computing x^i_t-1=C_t-1·x^i_t-2+ D_t-1·x^i_t-3+E_t-1+γ_t-1, we can compute C_t-1=∂x^i_t-1/∂x^i_t-2. Overall, the high-order dynamics gradients over all sequences can be expressed as (∇ L(𝐱))_d1=[(∂ L(𝐱)/∂ x_1)_d 1, (∂ L(𝐱)/∂ x_2)_d 1, ⋯, (∂ L(𝐱)/∂ x_t)_d 2] and (∇ L(𝐱))_d2=[(∂ L(𝐱)/∂ x_1)_d 2, (∂ L(𝐱)/∂ x_2)_d 2, ⋯, (∂ L(𝐱)/∂ x_t)_d 2]. 
 
























































































  






















 














§ EXPERIMENTS





 §.§ RobustBenchHAR Settings



To our best knowledge, there is no large-scale benchmark for evaluating transfer-based S-HAR attacks. To fill this gap, we build the first large-scale benchmark for robust S-HAR evaluation, named RobustBenchHAR. We briefly introduce the benchmark settings here, with additional details available in Appendix C.

(A) Datasets. RobustBenchHAR incorporates three popular S-HAR datasets:  NTU 60 <cit.> , NTU 120 <cit.> and HDM05<cit.>. Since the classifiers do not have the
same data pre-processing setting, we unify the data format following <cit.>. For NTU 60 and NTU 120, we subsampled frames to 60. For HDM05, we segmented the data into 60-frame samples. 

(B) Evaluated Models. We evaluate TASAR in three categories of surrogate/victim models. (1) Normally trained models: We adapt 5 commonly used GCN-based models, i.e., ST-GCN <cit.>, MS-G3D <cit.>, CTR-GCN <cit.>, 2s-AGCN <cit.>, FR-HEAD <cit.>, and two latest Transformer-based models SkateFormer<cit.> and STTFormer<cit.>. To our best knowledge, this is the first work to investigate the robustness of Transformer-based S-HARs. (2) Ensemble models: an ensemble of ST-CGN, MS-G3D and DGNN <cit.>. (3) Defense models: We employ BEAT <cit.> and TRADES <cit.>, which all demonstrate their robustness for skeletal classifiers.

(C) Baselines. We compare with state-of-the-art (SOTA) S-HAR attacks, i.e. SMART <cit.> and CIASA <cit.>. We also adopt the SOTA transfer-based attacks as baselines, including gradient-based, i.e., I-FGSM <cit.>, MI-FGSM <cit.> and the latest MIG <cit.>, input transformation method DIM <cit.>, and ensemble-based/Bayesian attacks, i.e., ENS <cit.>, SVRE <cit.> and BA <cit.>. For a fair comparison, we ran 200 iterations for all attacks under l_∞ norm-bounded perturbation of size 0.01. For TASAR, we use the iterative gradient attack instead of FGSM in <ref>. 



(D) Implementation Details. Our appended model is a simple two-layer fully-connected layer network. Unless specified otherwise, we use K=3 and M=20 in <ref> for default and explain the reason in the ablation study later. More implementation details can be found in Appendix C.


















































































































































 §.§ Evaluation on Normally Trained Models

Evaluation of Untargeted Attack. As shown in in <ref>, TASAR significantly surpasses both S-HAR attacks and transfer-based attacks under the black-box settings, while maintaining comparable white-box attack performance. Specially, TASAR achieves the highest average transfer success rate of 35.5% across different models and datasets, surpassing SMART <cit.> (the SOTA S-HAR attack) and MIG <cit.> (the SOTA transfer-based attack) by a large margin of 23.4% and 8.1% respectively. Moreover, TASAR shows consistent transferability across all surrogate models, target models and datasets. These improvements break the common belief that transfer-based attacks in S-HAR suffer from low transferability and highly rely on the chosen surrogate <cit.>. 

Evaluation of Targeted Attack.
In this section, we focus on targeted attacks under the black-box setting. Improving targeted attack transferability on S-HAR is generally more challenging than untargeted attacks. This is primarily due to the significant semantic differences between the randomly selected class and the original one. Attacking a `running' motion to `walking' is generally easier than to `drinking'. This is why targeted attacks have lower success rate than untargeted attacks. However, <ref> shows TASAR still outperforms the baseline under most scenarios. Moreover, TASAR can successfully attack the original class to a target with an obvious semantic gap without being detected by humans. The visual examples can be found in Appendix B.













 §.§ Evaluation on Ensemble and Defense Models

Evaluation on Ensemble Models.
TASAR benefits from the additional model parameters added by the appended Bayesian components. For a fair comparison, we compare it with SOTA ensemble-based methods, i.e., ENS <cit.> and SVRE <cit.>, and the Bayesian Attack (BA) <cit.>, because they also benefit from the model size. Unlike BA, which re-trains the surrogate into a BNN, we instead append a small Bayesian component for post-training. ENS and SVRE take three models ST-GCN, MS-G3D and DGNN as an ensemble of surrogate models, while BA and TASAR only take MS-G3D as the single substitute architecture. We choose ST-GCN, 2s-AGCN, MS-G3D, CTR-GCN, FR-HEAD as the target models, and evaluate the average white-box attack success rate (WASR), average black-box attack success(BASR) and the number of parameters in <ref>. We can clearly see that TASAR (blue line) achieves the best attack performance under both white-box and black-box settings, with an order of magnitude smaller model size. When using MSG3D (12.78M) as the surrogate model, the Bayesian components appended by TASAR only increase 0.012M parameters of the surrogate size, resulting in a memory cost comparable to that of a single surrogate. In contrast, the Bayesian surrogate model used by BA has 15 times more parameters (255.57M) than the single surrogate.

















































Evaluation on Defense Models.
As BEAT shows high robustness against S-HAR white-box attack <cit.>, it is also interesting to evaluate its defense performance against black-box attack. We also employ the adversarial training method TRADES <cit.> as a baseline due to its robustness in S-HAR <cit.>. Obviously, in <ref>, TASAR still achieves the highest adversarial transferability among the compared methods against defense models, further validating its effectiveness.
































 §.§ Ablation Study



Dual MCMC Sampling. 
TASAR proposes a new dual MCMC sampling in the post-train Bayesian formulation (<ref>). To see its contribution, we conduct an ablation study on the number of appended models (K and M in <ref>). As shown in <ref>, compared with vanilla Post-train Bayesian strategy (M=0), the dual sampling significantly improves the attack performance. Furthermore, although TASAR theoretically requires intensive sampling for inference, in practice, we find a small number of sampling is sufficient (K=3 and M=20). More sampling will cause extra computation overhead. So we use K=3 and M=20 by default.


























 





















                        
Temporal Motion Gradient. 
TASAR benefits from the interplay between temporal Motion Gradient (MG) and Bayesian manner. We hence conduct ablation studies(MG/No MG) to show the effects of motion gradient and report the results in <ref>. Compared with TASAR without using motion gradient, TASAR with motion gradient consistently improves the attack success rate in both white box and transfer-based attacks, which shows the benefit of integrating the motion gradient into the Bayesian formulation.





































 §.§ Surrogate Transferability

It is widely believed that transfer-based attacks in S-HAR are highly sensitive to the surrogate choice <cit.>. In this subsection, we provide a detailed analysis of the factors contributing to this phenomenon. When looking at the results in <ref> and the visualization of loss landscape in <ref> and Appendix B, we note that loss surface smoothness correlates with the adversarial transferability. For example, CTR-GCN, manifesting smoother regions within the loss landscape, demonstrates higher transferability than ST-GCN and STTFormer. STTFormer trained on NTU 120 has a smoother loss surface than ST-GCN (see Appendix B), resulting in higher transferability than ST-GCN. For NTU 60, STTFormer shows a similar loss surface to that of ST-GCN and exhibits comparable transferability.Therefore, we suspect that the loss surface smoothness plays a pivotal role in boosting adversarial transferability for S-HAR, potentially outweighing the significance of gradient-based optimization techniques. Next, two-stream MS-G3D shows the highest transferability. Unlike other surrogates, which solely extract joint information, MS-G3D uses a two-stream ensemble incorporating both joint and bone features, thereby effectively capturing relative joint movements. In conclusion, we suggest that skeletal transfer-based attacks employ smoother two-stream surrogates incorporating both joint and bone information. 




























§ CONCLUSION

In this paper, we systematically investigate the adversarial transferability for S-HARs from the view of loss landscape, and propose the first transfer-based attack on skeletal action recognition, TASAR. We build RobustBenchHAR, the first comprehensive benchmark for robustness evaluation in S-HAR. We hope that RobustBenchHAR could contribute to the adversarial learning and S-HAR community by facilitating researchers to easily compare new methods with existing ones and inspiring new research from the thorough analysis of the comprehensive evaluations.



























§ REPRODUCIBILITY CHECKLIST




  
This paper:


    
  * Includes a conceptual outline and/or pseudocode description of AI methods introduced (yes)

    
  * Clearly delineates statements that are opinions, hypothesis, and speculation from objective facts and results (yes)

    
  * Provides well marked pedagogical references for less-familiare readers to gain background necessary to replicate the paper (yes)





  
Does this paper make theoretical contributions? (no)



  
Does this paper rely on one or more datasets? (yes)

If yes, please complete the list below.


    
  * A motivation is given for why the experiments are conducted on the selected datasets (yes)

    
  * All novel datasets introduced in this paper are included in a data appendix. (NA)

    
  * All novel datasets introduced in this paper will be made publicly available upon publication of the paper with a license that allows free usage for research purposes. (NA)

    
  * All datasets drawn from the existing literature (potentially including authors’ own previously published work) are accompanied by appropriate citations. (yes)

    
  * All datasets drawn from the existing literature (potentially including authors’ own previously published work) are publicly available. (yes)
    
  * All datasets that are not publicly available are described in detail, with explanation why publicly available alternatives are not scientifically satisficing. (NA)




  
Does this paper include computational experiments? (yes)

If yes, please complete the list below.


  * Any code required for pre-processing data is included in the appendix. (yes).

  * All source code required for conducting and analyzing the experiments is included in a code appendix. (yes)

  * All source code required for conducting and analyzing the experiments will be made publicly available upon publication of the paper with a license that allows free usage for research purposes. (yes)

  * All source code implementing new methods have comments detailing the implementation, with references to the paper where each step comes from (yes)

  * If an algorithm depends on randomness, then the method used for setting seeds is described in a way sufficient to allow replication of results. (yes)

  * This paper specifies the computing infrastructure used for running experiments (hardware and software), including GPU/CPU models; amount of memory; operating system; names and versions of relevant software libraries and frameworks. (yes)

  * This paper formally describes evaluation metrics used and explains the motivation for choosing these metrics. (yes)

  * This paper states the number of algorithm runs used to compute each reported result. (yes)

  * Analysis of experiments goes beyond single-dimensional summaries of performance (e.g., average; median) to include measures of variation, confidence, or other distributional information. (yes)

  * The significance of any improvement or decrease in performance is judged using appropriate statistical tests (e.g., Wilcoxon signed-rank). (yes)

  * This paper lists all final (hyper-)parameters used for each model/algorithm in the paper’s experiments. (yes)

  * This paper states the number and range of values tried per (hyper-) parameter during development of the paper, along with the criterion used for selecting the final parameter setting. (yes)

