Fourier Analysis on Robustness of Graph Convolutional Neural Networks
  for Skeleton-based Action Recognition
Using Fourier analysis, we explore the robustness and vulnerability of graph
convolutional neural networks (GCNs) for skeleton-based action recognition. We
adopt a joint Fourier transform (JFT), a combination of the graph Fourier
transform (GFT) and the discrete Fourier transform (DFT), to examine the
robustness of adversarially-trained GCNs against adversarial attacks and common
corruptions. Experimental results with the NTU RGB+D dataset reveal that
adversarial training does not introduce a robustness trade-off between
adversarial attacks and low-frequency perturbations, which typically occurs
during image classification based on convolutional neural networks. This
finding indicates that adversarial training is a practical approach to
enhancing robustness against adversarial attacks and common corruptions in
skeleton-based action recognition. Furthermore, we find that the Fourier
approach cannot explain vulnerability against skeletal part occlusion
corruption, which highlights its limitations. These findings extend our
understanding of the robustness of GCNs, potentially guiding the development of
more robust learning methods for skeleton-based action recognition.
Nariki Tanaka, Hiroshi Kera, Kazuhiko Kawamoto
Fourier Analysis on Robustness of Graph Convolutional Neural Networks for Skeleton-based Action Recognition

###### Abstract

Using Fourier analysis, we explore the robustness and vulnerability of graph convolutional neural networks (GCNs) for skeleton-based action recognition. We adopt a joint Fourier transform (JFT), a combination of the graph Fourier transform (GFT) and the discrete Fourier transform (DFT), to examine the robustness of adversarially-trained GCNs against adversarial attacks and common corruptions. Experimental results with the NTU RGB+D dataset reveal that adversarial training does not introduce a robustness trade-off between adversarial attacks and low-frequency perturbations, which typically occurs during image classification based on convolutional neural networks. This finding indicates that adversarial training is a practical approach to enhancing robustness against adversarial attacks and common corruptions in skeleton-based action recognition. Furthermore, we find that the Fourier approach cannot explain vulnerability against skeletal part occlusion corruption, which highlights its limitations. These findings extend our understanding of the robustness of GCNs, potentially guiding the development of more robust learning methods for skeleton-based action recognition.

Skeleton-based action recognition Graph convolutional neural network Adversarial robustness Fourier analysis

## 1 Introduction

In skeleton-based action recognition, graph convolutional neural networks (GCNs) exhibit remarkable performance due to their ability to represent skeletal motion inputs using topological graphs [1, 2, 3, 4, 5, 6]. However, recent studies have revealed that GCNs are vulnerable to adversarial attacks [7, 8, 9, 10] and common corruptions, such as Gaussian noise. This finding emphasizes the need to ensure robustness in real-world applications. To address these issues, other recent studies have proposed methods to improve the robustness of GCNs [11, 12, 13, 14, 15]. These vulnerabilities also imply that GCNs learn different features from humans, highlighting the need for a deeper understanding of their properties to develop robust models.

Recent studies in image classification have uncovered interesting properties of convolutional neural networks (CNNs) using Fourier analysis [16, 17, 18, 19, 20, 21]. For example, Yin et al. [16] discovered that adversarial perturbations for standard-trained CNNs are concentrated in the high-frequency domain, whereas those for adversarially-trained CNNs are concentrated in the low-frequency domain. They also revealed that adversarial training encourages CNNs to capture low-frequency features of images, resulting in a trade-off between robustness to low-frequency and high-frequency perturbations. More specifically, adversarial training can enhance robustness to high-frequency corruptions such as Gaussian noise while degrading robustness to low-frequency corruptions such as fog corruption. Saikia et al. [22] leveraged the trade-off to propose a method for improving robustness by combining separate models that were respectively robust to low- and high-frequency perturbations. Furthermore, Zhuang et al. [21] observed that CNNs robust against common corruptions tend to rely more on low-frequency features than standard-trained models. These studies demonstrated the effectiveness of Fourier analysis in understanding the robustness of CNNs.
Inspired by these prior efforts, we examine the robustness of skeleton-based action recognition using Fourier analysis. Unlike image classification, where the 2D discrete Fourier transform (DFT) can be used for Fourier analysis, skeleton-based action recognition requires an alternate approach owing to the graph-based representation of the skeletal data. Therefore, we adopt a joint Fourier transform (JFT) [23] that combines the graph Fourier transform (GFT) and DFT, as shown in Fig. 1. By applying the GFT to each frame of the skeletal sequence data and then using the DFT to each graph-frequency component, we analyze the frequencies of the skeletal sequence data along the spatial (graph-frequency) and temporal (time-frequency) directions. This method enables us to explore the difference between CNN-based image classification and GCN-based skeletal action recognition from a Fourier perspective. In experiments with the NTU RGB+D dataset [24], we apply Fourier analysis to compare standard-trained and adversarially-trained GCNs. Our observations reveal that there is no robustness trade-off between adversarial attacks and low-frequency perturbations for GCN-based skeletal action recognition. This finding is interesting because such a trade-off is typically observed in image classification [16; 20]. Furthermore, we explore the robustness against common corruptions, such as Gaussian noise and part occlusion, and find that an experimental result for the case of part occlusion cannot be explained by Fourier analysis alone.

The contributions of this study are as follows.

* A novel application of Fourier analysis is presented to GCN-based skeletal-based action recognition. Specifically, for the first time, we analyze the frequency characteristics of adversarially-trained GCNs against adversarial attacks and common corruptions using the joint Fourier transform.
* Experimental findings indicate that there is no robustness trade-off between adversarial attacks and low-frequency perturbations for skeleton-based action recognition, which is unique in CNN-based image classification.
* Challenges are revealed in comprehensively explaining the robustness of skeleton-based action recognition using Fourier analysis. Specifically, Fourier analysis cannot explain vulnerability against part occlusion corruptions.

The remainder of this paper is organized as follows. Section 2 presents related work on the robustness of skeleton-based action recognition and Fourier analysis approaches in deep learning. Section 3 describes a Fourier analysis method for GCNs. Section 4 presents the experimental results.

## 2 Related work

This section reviews related work on the robustness of skeleton-based action recognition and the Fourier analysis on robustness of deep models against adversarial attacks and common corruptions.

### Robustness of Skeleton-based Action Recognition

Skeletal motion data offer several advantages over RGB videos, such as robustness to changes in clothing and background and superior computational efficiency [25; 26]. Due to the nature of the skeletal structure, GCNs have been extensively used for skeleton-based action recognition [1; 2; 3; 4; 5; 6].

Recent research has explored adversarial attacks [27; 28] on GCNs for skeleton-based action recognition and revealing the vulnerabilities of GCNs [7; 8; 9; 10]. Liu et al. [10] presented the first white-box adversarial attack on GCNs.

Figure 1: Flow of the joint Fourier transform (JFT) on skeletal sequence data, which encompasses both the graph Fourier transform (GFT) and the discrete Fourier transform (DFT). The GFT is first applied to the skeletal data at each frame, followed by the DFT.
Their method perturbs joint locations while preserving temporal coherence, spatial integrity, and anthropomorphic plausibility. Diao et al. [7] proposed a guided manifold walk method to search for adversarial examples in the black-box setting. They also considered the naturalness and imperceptibility of perturbed skeletons. Wang et al. [8] proposed an attack method in both white- and black-box settings and highlighted the importance of considering motion dynamics in analyzing imperceptible adversarial attacks on 3D skeletal motion. Tanaka et al. [9] indicated that an adversarial attack was possible by only changing the lengths of the bones. All of these studies demonstrate that GCNs are vulnerable to imperceptible adversarial perturbations.

Contrary to these studies on attack, research on defense against adversarial attacks for skeleton-based action recognition remains in its infancy. Wang et al. [29] proposed a Bayesian defense framework based on adversarial training, which is the most effective defense technique against adversarial attacks. Their study demonstrated that adversarial training is effective not only for CNN-based image classification, but also for GCN-based skeletal action recognition. Nevertheless, existing studies on adversarial robustness are limited and not sufficiently comprehensive.

Even so, vulnerability to common corruptions has been explored in several studies. For example, robustness against Gaussian noise [11], part occlusion [12; 13; 15], frame occlusions and jittering noises [13] have been explored. Further investigation into the robustness against both common corruptions and adversarial attacks are essential for enhancing the applicability of deep models in real-world scenarios.

### Fourier analysis of CNN-based image classification

In image classification, recent studies have attempted to explain the robustness of CNNs using frequency analysis [16; 17; 18; 19; 20; 21]. Yin et al. [16] demonstrated that standard-trained CNNs predominantly depend on high-frequency components for image classification, while adversarial training encourages CNNs to capture low-frequency components of images. Their further investigation revealed that adversarial training enhances robustness against common high-frequency corruptions, such as Gaussian noise, while degrading robustness against common low-frequency corruptions, such as fog corruption. These findings indicate the existence of a trade-off between robustness against high-frequency and low-frequency perturbations. Wang et al. [17] established that smoothing the convolutional kernels at the first layer, thereby encouraging CNNs to ignore high-frequency components, can improve adversarial robustness. Bernhard et al. [18] observed that the frequency characteristics of adversarial robustness may depend on the dataset. Abello et al. [19] investigated the high-frequency biases of standard-trained CNNs. Chan et al. [20] explored these trade-offs by directly changing the frequency profile of the models. Zhuang et al. [21] identified harmful frequencies for robustness to common corruptions and proposed a method to ignore these harmful frequency components.

These studies indicate that it is beneficial to use Fourier analysis to understand the robustness of CNNs. However, the frequency analysis of GCN-based skeleton action recognition remains unexplored.

## 3 Fourier Analysis for Skeleton-based Action Recognition

Here, we investigate the frequency responses of GCNs for skeleton action recognition to reveal their robustness against adversarial attacks and common corruptions. Rather than using the 2D discrete Fourier transform (DFT), we employ the joint Fourier transform (JFT), which is a frequency analysis tool for time-varying graph signals [23], Furthermore, we use the Fourier heatmap [16] to visualize the sensitivity of deep models in the frequency domain.

### Spatiotemporal Graph for Skeletal Sequence Data

A skeleton sequence for a single person is represented by an undirected graph \(G=(V,E)\), where \(V\) is a set of nodes and \(E\) is a set of edges. The node set \(V\) consists of all \(N\) skeleton joints, i.e., \(V\) includes \(N\times T\) nodes, and the edge set \(E\) indicates the intrabody connections of the skeleton. Each node \(v_{i}(t)\in V\) is associated with a feature vector of the corresponding joint position \(\mathbf{x}_{i}(t)=(x_{i}(t),y_{i}(t),z_{i}(t))^{\top}\in\mathbb{R}^{3}\) at frame \(t\). For example, considering the joint positions themselves, the node feature is \(\mathbf{x}_{i}(t)\). As with other features, joint motion \(\mathbf{v}_{i}(t)=\mathbf{x}_{i}(t+1)-\mathbf{x}_{i}(t)\), bone \(\mathbf{b}_{i}(t)=\mathbf{x}_{i}(t)-\mathbf{x}_{j}(t)\) (\(v_{i}\) and \(v_{j}\) is a pair of connected joints), and bone motion \(\mathbf{v}_{i}^{\mathrm{b}}(t)=\mathbf{b}_{i}(t+1)-\mathbf{b}_{i}(t)\) have been used [3; 4; 5]. The set of such feature vectors is represented by

\[\mathbf{X}=\begin{pmatrix}f_{1}(1)&f_{1}(2)&\dots f_{1}(T)\\ f_{2}(1)&f_{2}(2)&\dots f_{2}(T)\\ \vdots&\ddots&\vdots\\ f_{N}(1)&f_{N}(2)&\dots f_{N}(T)\end{pmatrix}\in\mathbb{R}^{N\times T}, \tag{1}\]
where \(f_{i}(t)\) is an \(x\), \(y\), or \(z\) coordinate element of the feature vector associated with \(v_{i}(t)\). We omit the dimension of the three-axis channels for simplicity, although the related skeleton sequence is represented by an \(N\times T\times 3\) tensor. The skeleton sequence is processed independently for each channel.

To normalize the frame lengths of all skeletal data, linear interpolation along the temporal direction are usually used [4, 5, 6]. We denote the interpolated data for \(\mathbf{X}\) by \(I(\mathbf{X})\in\mathbb{R}^{N\times T^{\prime}}\), where \(T^{\prime}\) is set to \(64\) in our experiments.

### Standard & Adversarial Training

Let \(\mathcal{M}_{\theta}\) be a GCN parameterized by \(\theta\). Standard training for \(\mathcal{M}_{\theta}\) is performed by solving the following minimization problem

\[\min_{\theta}\mathbb{E}_{(\mathbf{X},y)\sim\mathcal{D}}[\mathcal{L}(\mathcal{M}_ {\theta}(I(\mathbf{X}),y))], \tag{2}\]

where \(\mathcal{L}(\cdot)\) is the cross-entropy loss function and \(\mathcal{D}\) is an underlying data distribution over pairs of interpolated skeletal data \(I(\mathbf{X})\) and action labels \(y\).

Adversarial training in this study is based on the projected gradient descent (PGD) [28], which is a typical and strong attack method. The PGD uses a set of perturbations \(S=\{\mathbf{\delta}\mid\left\lVert\mathbf{\delta}\right\rVert_{p}\leq\epsilon\}\), where \(\left\lVert\mathbf{\delta}\right\rVert_{p}\) is the \(l_{p}\) norm of \(\mathbf{\delta}\in\mathbb{R}^{T\times N}\) ( we set \(p\) as 2) and \(\epsilon>0\) denote the supremum of the perturbation norm. Adversarial training for \(\mathcal{M}_{\theta}\) with the PGD is performed by solving the following min-max optimization problem [28]

\[\min_{\theta}\mathbb{E}_{(\mathbf{X},y)\sim\mathcal{D}}\Bigl{[}\max_{\mathbf{\delta} _{\mathrm{adv}}\in S}\mathcal{L}(\mathcal{M}_{\theta}(I(\mathbf{X}+\mathbf{\delta}_{ \mathrm{adv}}),y))\Bigr{]}, \tag{3}\]

where linear interpolation \(I(\cdot)\) is repeatedly applied to adversarially perturbed data \(\mathbf{X}+\mathbf{\delta}_{\mathrm{adv}}\) in the optimization loop.

### Discrete & Graph Fourier Transforms

DFT and GFT form the basis for the JFT on spatiotemporal graph \(G\), as is discussed in the following subsection. Additionally, DFT and GFT are used to generate low-frequency or high-frequency Gaussian noise signals, which are used to validate the trade-off in robustness against low-frequency and high-frequency corruptions, as discussed in Section 4.2.3.

We let the GFT be defined first. The GFT on \(\mathbf{X}\) is defined by

\[\text{GFT}(\mathbf{X})=\mathbf{U}^{\top}\mathbf{X}, \tag{4}\]

where \(\mathbf{U}\in\mathbb{R}^{N\times N}\) is the eigenvector matrix of the Laplacian matrix \(\mathbf{L}\) of the spatial subgraph of \(G\). The Laplacian matrix is defined by \(\mathbf{L}=\mathbf{D}-\mathbf{A}\in\mathbb{R}^{N\times N}\), where \(\mathbf{A}\) and \(\mathbf{D}\) denote the adjacency and degree matrices of the spatial skeletal structure, respectively. Let \(\lambda_{k}\in\mathbb{R}\) and \(\mathbf{u}_{k}\in\mathbb{R}^{N}\) be the \(k\)-th eigenvalue and eigenvector of \(\mathbf{L}\). The eigenvector matrix \(\mathbf{U}\) is obtained by applying the eigen-decomposition to \(\mathbf{L}\) as follows

\[\mathbf{L}=\mathbf{U}\mathbf{\Sigma}\mathbf{U}^{\top}, \tag{5}\]

where \(\mathbf{U}=[\mathbf{u}_{1},\dots,\mathbf{u}_{N}]\) and \(\mathbf{\Sigma}=\mathrm{diag}(\lambda_{1},\dots,\lambda_{N})\). Here, the eigenvalues are sorted as \(\lambda_{1}\geq\lambda_{2}\geq\dots>\lambda_{N}=0\), where the larger the eigenvalue is, the higher the frequency.

The DFT on \(\mathbf{X}\) is defined by

\[\text{DFT}(\mathbf{X})=\mathbf{X}\mathbf{W}, \tag{6}\]

where \(\mathbf{W}\in\mathbb{R}^{T\times T}\) is the DFT matrix, defined by

\[\mathbf{W}=\begin{pmatrix}1&1&1&\dots&1\\ 1&\omega&\omega^{2}&\dots&\omega^{T-1}\\ 1&\omega^{2}&\omega^{4}&\dots&\omega^{2(T-1)}\\ \vdots&\vdots&\vdots&\ddots&\vdots\\ 1&\omega^{T-1}&\omega^{2(T-1)}&\dots&\omega^{(T-1)(T-1)}\end{pmatrix}, \tag{7}\]

where \(\omega=e^{-2\pi j/T}\) and \(j^{2}=-1\).

We generate low-frequency or high-frequency Gaussian noises that is added to skeletal data \(\mathbf{X}\) using the GFT and DFT. Let \(\mathbf{V}\) be the Gaussian white noise on \(\mathbb{R}^{N\times T}\), where each element of \(\mathbf{V}\) is independently sampled from a zero-mean Gaussian distribution \(\mathcal{N}(0,\sigma^{2})\). Spatial filtering or graph spectral filtering on \(\mathbf{V}\) is defined by

\[\widetilde{\mathbf{V}_{\mathrm{s}}}=\mathbf{M}_{\mathrm{s}}\text{GFT}(\mathbf{V}), \tag{8}\]
where \(\widetilde{\mathbf{V}_{\text{s}}}\) is the filtering output in the spatial frequency domain and \(\mathbf{M}_{\text{s}}=\operatorname{diag}(m_{\text{s},1},m_{\text{s},2},\ldots,m_{ \text{s},N})\) is the \(N\times N\) binary diagonal matrix for spatial filtering. For instance, if \(\mathbf{M}_{\text{s}}=\operatorname{diag}(0,0,\ldots,0,1,1)\), as shown in Fig. 2 (leftmost), a low-frequency Gaussian noise with a bandwidth of 2 is generated by applying the inverse GFT to \(\widetilde{\mathbf{V}_{\text{s}}}\). If \(\mathbf{M}_{\text{s}}=\operatorname{diag}(1,1,0,\ldots,0)\), as shown in Fig. 2 (left second), High-frequency Gaussian noise with a bandwidth of 2 is generated. Similarly, temporal filtering on \(\mathbf{V}\) is defined by

\[\widetilde{\mathbf{V}_{\text{t}}}=\text{DFT}(\mathbf{V})\mathbf{M}_{\text{t}}, \tag{9}\]

where \(\widetilde{\mathbf{V}_{\text{t}}}\) is the filtering output in the temporal frequency domain and \(\mathbf{M}_{\text{t}}=\operatorname{diag}(m_{\text{t},1},m_{\text{t},2},\ldots,m_ {\text{t},T})\) is the \(T\times T\) binary diagonal matrix for temporal filtering. Similar to the GFT, \(\mathbf{M}_{\text{t}}=\operatorname{diag}(0,0,\ldots,0,1,1)\) and \(\mathbf{M}_{\text{t}}=\operatorname{diag}(1,1,0,\ldots,0)\) generate low-frequency and high-frequency Gaussian noises with a bandwidth of 2, as shown in Fig. 2 (right second) and (rightmost), respectively.

### Joint Fourier Transform and Fourier Heatmap

We adopt the JFT [23] for Fourier analysis on spatiotemporal graph G, which encompasses both the GFT and DFT. Initially, the GFT is implemented on the spatial subgraph of \(G\) for each frame to extract spatial frequency characteristics, followed by the application of the DFT to extract temporal frequency characteristics of \(G\), as shown in Fig. 1.

The JFT on \(\mathbf{X}\) is defined by combining the GFT and DFT as follows:

\[\text{JFT}(\mathbf{X})=\mathbf{U}^{\top}\mathbf{X}\mathbf{W}. \tag{10}\]

For Fourier analysis on skeletal data, we compute the average spectrum over all test data of \(\mathbf{X}\) using the JFT, and visualize these values using the heatmaps shown in Fig. 3, where the horizontal and vertical axes represent temporal and spatial frequencies, respectively.

Additionally, we employ Fourier heatmaps [16] to visualize the sensitivity of the GCNs for specific frequency signals. A signal of spatial frequency \(\lambda_{k}\) (the \(k\)-th eigenvalue of \(\mathbf{L}\)) and temporal frequency \(l/T\), denoted by \(\mathbf{F}_{k,l}\in\mathbb{R}^{N\times T}\), is

Figure 3: Average Fourier spectrum over all tests skeleton data for joint, joint motion, bone, and bone motion features. The vertical and horizontal axes represent spatial and temporal frequency, respectively. From these figures, for example, we can see the Fourier spectrum of the joint feature (leftmost) is concentrated at low frequencies in the spatiotemporal frequency domain.

Figure 2: Spatial low-pass (leftmost) and high-pass (left second) filtering are performed by masking the Fourier spectrum along the spatial frequency axis. Temporal low-pass (right second) and high-pass (rightmost) are performed by masking the Fourier spectrum along the temporal frequency axis.
generated by setting the JFT of \(\mathbf{F}_{k,l}\) to

\[\text{JFT}(\mathbf{F}_{k,l})=\ k^{\text{th}}\begin{pmatrix}0&0&0&0&\ldots&0\\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\ 0&\ldots&0&1&0&\ldots&0\\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\vdots\\ 0&\ldots&0&0&0&\ldots&0\end{pmatrix}, \tag{11}\]

where the right-hand side matrix takes \(1\) for only the \((k,l)\)-element and \(0\) otherwise. Using \(\mathbf{F}_{k,l}\), a perturbed signal of a skeleton sequence \(\mathbf{X}\) is generated by

\[\mathbf{X}^{\prime}=I(\mathbf{X})+rv\mathbf{F}_{k,l}, \tag{12}\]

where \(r\) is sampled uniformly at random from \(\{-1,1\}\), \(v>0\) is the norm of the perturbation, and \(\mathbf{F}_{k,l}\) is supposed to be normalized as \(||\mathbf{F}_{k,l}||_{2}=1\). The Fourier heatmap is generated by plotting the average error rate of 1000 randomly sampled test data for every frequency \(k=1,2,\ldots,N\) and \(l=1,2,\ldots,T\). The closer to red a region of the Fourier heatmap is, the more sensitive the GCNs at the corresponding frequency. In other words, to recognize human actions, GCNs capture skeletal signals that contain such sensitive frequencies. The high sensitivity brings vulnerability to the GCNs.

## 4 Experiment

Using frequency analysis, we conduct a comparative analysis of the robustness for the standard-trained and adversarially-trained GCNs. First, we investigate the robustness and vulnerability in the frequency domain using Fourier heatmaps. The visualization provides basic insights into the robustness of adversarial training. Next, we analyze the spectral distributions of adversarial perturbations to the GCNs. This analysis reveals the frequency characteristics of adversarial attacks. Furthermore, we explore whether the robustness trade-off that has been established in CNN-based image classification exists similarly in GCN-based skeletal action recognition. Finally, we evaluate the robustness of the GCNs against common corruptions.

### Experimental Setting

#### 4.1.1 Dataset

We conduct our experiments on NTU RGB+D [24], which contains 56,880 skeletal data with 60 action classes. We divide the whole data into training and test data according to the subjects (cross-subject setting). The training and test

Figure 4: Adversarial examples generated by the \(l_{2}\)-PGD. Clean (blue) and adversarial (red) examples are superimposed for three example actions. These two almost overlap and are highly imperceptible. For each action, action labels before and after adversarial attacks are provided.
datasets comprise 40,320 and 16,560 samples, respectively. For the validation data, we randomly sample 5% from the training data.

#### 4.1.2 Model

We train ST-GCNs [1] and TCA-GCNs [5] for joint \(\mathbf{x}_{i}(t)\), joint motion \(\mathbf{v}_{i}(t)\), bone \(\mathbf{b}_{i}(t)\), and bone motion \(\mathbf{v}_{i}^{\mathrm{b}}(t)\), respectively, using official codes provided by those authors. To guarantee convergence in training, we execute at least 80 epochs for the ST-GCN and 75 epochs for the TCA-GCN and adopt early stopping with patience 20 for both models. For the other hyperparameters, we use the same ones of the official codes.

#### 4.1.3 Adversarial Attack

We use the \(l_{2}\)-PGD algorithm to evaluate adversarial robustness. To normalize the attack strength to each piece of skeletal data, we set the perturbation threshold \(\epsilon\) as \(\epsilon=l_{\mathrm{head}}\times\epsilon_{\mathrm{head}}\), where \(l_{\mathrm{head}}\) is the head length of each skeletal datum. Fig. 4 displays adversarial examples when \(\epsilon_{\mathrm{head}}=1.0,3.0,5.0\). We do not impose any naturalness constraints on the attacker, except for the perturbation norm. Nevertheless, we observe that adversarial attacks are highly imperceptible because linear interpolation in Section 3.1 suppresses large jittering.

#### 4.1.4 Adversarial Training

Adversarial training [28] is an effective method for defending against adversarial attacks. We use Free [30] for adversarial training due to its computational efficiency. For Free, the number of hop steps is set to four, and the threshold is set to \(\epsilon_{\mathrm{head}}=3\). Table 1 shows the accuracies of standard-trained and adversarially-trained models. As is well established, adversarial training performs less effectively than standard training for clean data. To verify that adversarial training enhances adversarial robustness, we conduct preliminary experiments on robustness of the standard-trained and adversarially-trained models with thresholds \(\epsilon_{\mathrm{head}}\in\{1.0,3.0,5.0\}\) and 10 iterations. Then, we attack each model by perturbing clean data correctly classified by both models to take the difference in clean accuracy into consideration. In Table 2, we can see that adversarial training demonstrates higher adversarial robustness than standard training.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline Model & Joint & Joint Motion & Bone & Bone Motion \\ \hline ST-GCN (ST) & **85.3**\% & **84.8**\% & **85.4**\% & **84.4**\% \\ ST-GCN (AT) & 79.8\% & 73.1\% & 79.7\% & 76.0\% \\ \hline TCA-GCN (ST) & **89.4**\% & **86.7**\% & **89.2**\% & **86.3**\% \\ TCA-GCN (AT) & 82.1\% & 77.2\% & 82.2\% & 78.4\% \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of clean accuracy between standard-trained (ST) and adversarially-trained (AT) models.

\begin{table}
\begin{tabular}{c|c|c|c} \hline Model & \(\epsilon_{\mathrm{head}}=1\) & \(\epsilon_{\mathrm{head}}=3\) & \(\epsilon_{\mathrm{head}}=5\) \\ \hline ST-GCN ST (Joint) & 26.3\% & 6.00\% & 1.63\% \\ ST-GCN AT (Joint) & **74.6**\% & **61.2**\% & **47.7**\% \\ \hline TCA-GCN ST (Joint) & 16.7\% & 1.72\% & 0.19\% \\ TCA-GCN AT (Joint) & **76.8**\% & **64.6**\% & **52.4**\% \\ \hline ST-GCN ST (Joint Motion) & 9.30\% & 0.61\% & 0.06\% \\ ST-GCN AT (Joint Motion) & **64.2**\% & **48.3**\% & **35.8**\% \\ \hline TCA-GCN ST (Joint Motion) & 2.84\% & 0.04\% & 0.00\% \\ TCA-GCN AT (Joint Motion) & **69.0**\% & **51.6**\% & **36.5**\% \\ \hline ST-GCN ST (Bone) & 8.99\% & 0.35\% & 0.07\% \\ ST-GCN AT (Bone) & **71.5**\% & **52.5**\% & **32.9**\% \\ \hline TCA-GCN ST (Bone) & 8.35\% & 0.21\% & 0.01\% \\ TCA-GCN AT (Bone) & **76.1**\% & **60.8**\% & **44.5**\% \\ \hline ST-GCN ST (Bone Motion) & 0.87\% & 0.02\% & 0.00\% \\ ST-GCN AT (Bone Motion) & **62.2**\% & **40.5**\% & **24.5**\% \\ \hline TCA-GCN ST (Bone Motion) & 0.39\% & 0.03\% & 0.02\% \\ TCA-GCN AT (Bone Motion) & **61.3**\% & **46.5**\% & **31.8**\% \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of adversarial accuracy between standard-trained (ST) and adversarially-trained (AT) models.
#### 4.1.5 Evaluation Metric

As shown in Table 1, there is a difference in clean accuracy between standard-trained and adversarially-trained models. For a fair comparison of robustness, we remove a such difference. More specifically, when we evaluate the robustness, we perturb clean data correctly classified by both models and use accuracy for these perturbed data as an evaluation metric.

### Results

#### 4.2.1 Frequency Analysis of Adversarial Training

The Fourier heatmap examines the frequency characteristics of standard and adversarial training. Fig. 5 shows Fourier heatmaps that plot the average error rates of 1000 randomly sampled test data points for every frequency. The top and bottom three rows display the Fourier heatmaps of the ST-GCNs and TCA-GCNs, respectively, for each of the four features (joint, joint motion, bone, bone motion) in Section 3.1 and three perturbation norms \(v\in\{0.5,1.5,3.0\}\) in Eq. (12).

Fig. 5 reveals the following frequency characteristics. The standard-trained models are sensitive to temporal low-frequency perturbations (i.e., the left half of the maps). In other words, the models do not capture high-frequency signals in time, whereas they do capture signals from low to high frequencies in space. The adversarially-trained models become more insensitive, especially for high-frequency perturbations (i.e., the upper right of the maps), resulting in robustness to high-frequency perturbations. Joint motion and bone motion features introduce more vulnerability against high-frequency perturbations than the joint and bone features, respectively. This vulnerability is reasonably attributed to motion (differential) features generally being more sensitive to high-frequency perturbations than positional features. These frequency characteristics do not depend on the network architecture, and the two GCNs have similar characteristics.

These results prove that adversarial training can improve the robustness in the higher frequencies, and the improvement is also observable in CNN-based image classification [16; 17; 19]. In contrast, the lower frequency characteristics of the

Figure 5: Fourier heatmaps of standard-trained (ST) and adversarially-trained (AT) GCNs. The top and bottom three rows display those of the ST-GCNs and TCA-GCNs, respectively, for each of the four features (joint, joint motion, bone, bone motion) and three perturbation norms \(v\in\{0.5,1.5,3.0\}\).
CNNs and GCNs are slightly different. In CNN-based image classification, adversarial training sacrifices the robustness of the standard-trained models in the lowest frequencies (i.e., the lower left of the maps), whereas this is not always the case in GCN-based action recognition. For example, the adversarially-trained models learned with the joint and bone features do not become more vulnerable than the standard-trained models at the lowest frequencies.

#### 4.2.2 Frequency Analysis of Adversarial Attack

The frequency characteristics of adversarial attacks are examined using the spectral distribution of successful adversarial attacks. Fig. 6 shows the spectral distributions, which are obtained by estimating the average amplitude of successful adversarial examples, i.e., \(\mathbb{E}[|\mathrm{JFT}(I(\mathbf{X}+\mathbf{\delta}_{\mathrm{adv}})-I(\mathbf{X}))|]\), where \(\mathbf{X}\) is clean skeletal data and \(\mathbf{\delta}_{\mathrm{adv}}\) is an adversarial perturbation that successfully attacks a given GCN. Here, \(I(\cdot)\) is the linear interpolation operation in Section 3.1.

Fig. 6 shows that the spectral distributions of the adversarial attacks on the standard-trained models are broadly distributed, whereas those on the adversarially-trained models are concentrated in the lower frequencies. These frequency characteristics indicate that the adversarially-trained models capture the features from the lower frequency signals better than the standard-trained models. This observation supports the hypothesis that adversarial training provides robustness against high-frequency perturbations, as discussed in the previous subsection. However, the adversarially-trained models with the bone motion feature remain vulnerable in the high-frequency domain (i.e., the rightmost column in Fig. 6). This phenomenon reasonably leads us to conclude that the bone feature contains high-frequency signals [19] as shown in Fig. 3 (i.e., the rightmost column).

#### 4.2.3 Robustness Trade-off between High-Frequency and Low-Frequency Perturbations

We examine whether a trade-off exists where adversarially-trained models are robust to high-frequency perturbations but highly vulnerable to low-frequency perturbations. This examination is motivated by the existence of the same trade-off as found in CNN-based image classification [16; 20]. The trade-off is evaluated by adding low- or high-frequency Gaussian noise, as described in Section 3.3, to the clean data that both models correctly classify. The norm of the Gaussian noise is adjusted such that the accuracy of either the standard-trained model or the adversarially-trained model is approximately 80%. This adjustment is required for a fair comparison because the appropriate norm differs between

Figure 6: Average Fourier spectrum of adversarial perturbations in the spatiotemporal frequency domain. The heatmaps plot the estimation of \(\mathbb{E}[|\mathrm{JFT}(I(\mathbf{X}+\mathbf{\delta}_{\mathrm{adv}})-I(\mathbf{X}))|]\), where \(\mathbf{X}\) is the clean data and the expectation is chosen over the adversarial examples that successfully attack each model.
the four features. These norms are listed in Table 3 and denoted as scaled norms. In the following experiments, we use the ST-GCN because there is no significant difference in the frequency characteristics of the two architectures.

First, we evaluate the accuracies of the standard-trained model (blue) and the adversarially-trained model (red), as shown in Fig. 7, for changes in the Gaussian noise norm while maintaining a fixed bandwidth of 2. The norm is changed at 20%, 40%, 60%, 80%, and 100% of the scaled norm. The top two and bottom two rows in Fig. 7 show the accuracies when perturbed with spatially or temporally filtered Gaussian noises, respectively. In all cases, the adversarially-trained model (red) is more robust than the standard-trained model (blue). Next, we evaluate the accuracies of the two models upon changing the Gaussian noise bandwidth while keeping the norm fixed at the scaled norm. The results are shown in Fig. 8 and indicate that the adversarially-trained model (red) is also more robust than the standard model. Finally, we evaluate the accuracies for changes in the spatially and temporally filtered Gaussian noise norm while keeping the norm fixed. Table 4 lists the scaled norm for the experiment, and Fig. 9 also proves that the adversarially-trained model (red) is again more robust.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline Filter & Joint & Joint Motion & Bone & Bone Motion \\ \hline Spatial Low & 17.0 & 3.50 & 6.40 & 1.10 \\ \hline Spatial High & 77.0 & 20.0 & 5.90 & 4.80 \\ \hline Temporal Low & 31.0 & 2.0 & 9.90 & 0.60 \\ \hline Temporal High & 77.0 & 19.0 & 24.0 & 14.0 \\ \hline \end{tabular}
\end{table}
Table 3: List of scaled norms for spatially or temporally filtered Gaussian noises.

Figure 7: Accuracy comparison of standard-trained (blue) and adversarially-trained (red) GCNs for changes in the Gaussian noise norm with a fixed bandwidth of 2: (top) spatial low-pass filter, (second) spatial high-pass filter, (third) temporal low-pass filter, and (bottom) temporal high-pass filter.
In summary, these results indicate that GCN-based skeletal action recognition does not suffer from the same robustness trade-offs as CNNs. Therefore, adversarial training improves the GCNs robustness to both low- and high-frequency perturbations, i.e., without sacrificing robustness in the low-frequency domain.

#### 4.2.4 Robustness to Common Corruptions

We evaluate the robustness to common corruptions. In image classification, common corruptions to image data include high-frequency corruptions, such as Gaussian noise and shot noise, and low-frequency corruptions, such as motion blur and fog [31]. The trade-off between accuracy and adversarial robustness makes CNN-based image classifiers robust to high-frequency corruptions but vulnerable to low-frequency corruptions. However, the experiment in the previous section demonstrates that such a trade-off does not exist for the GNCs. Therefore, adversarial training is expected to be more robust to common corruptions than standard training, regardless of its frequency spectra. We experimentally examine this conclusion based on the following three common corruptions for skeleton data, as shown in Fig. 10.

* Gaussian noise: the skeletal data are perturbed by adding a zero-mean Gaussian noise with standard deviation \(\sigma=0.01,0.03,0.05\).
* Frame loss: Each frame of the skeletal sequence data is randomly lost. Loss rate \(p\) is a uniform random number in the \([0,1]\) interval. The frame length is adjusted when input to the GCNs by linearly interpolating the lost frames.

Figure 8: Accuracy comparison of standard-trained (blue) and adversarially-trained (red) GCNs for changes in the Gaussian noise bandwidth with the scaled norm in Table. 3. (top) spatial low-pass filter, (second) spatial high-pass filter, (third) temporal low-pass filter, and (bottom) temporal high-pass filter.
* Part occlusion [12]: a part of the skeletal data is occluded. The skeletal data are divided into five parts: left arm (part 1), right arm (part 2), both hands (part 3), both legs (part 4), and torso (part 5), and either part coordinates are set to 0.

To provide a Fourier analysis of the common corruptions, we compute their spectral distributions as shown in Figs. 11-13. These distributions are given by \(\mathbb{E}[|\mathrm{JFT}(I(C(\mathbf{X}))-I(\mathbf{X}))|]\), where \(\mathbf{X}\) is clean skeletal data and \(C(\cdot)\) is one of the three corruptions.

Gaussian NoiseFig. 11 shows the average Fourier spectrum distributions for the four features, and Table 5 lists the accuracies of the standard-trained and adversarially-trained two GCNs with standard deviation \(\sigma=0.01,0.03,0.05\). As depicted in Fig. 11, the frequency characteristics of Gaussian noise corruption differ for the four features. For example, the joint feature largely includes temporal low-frequency signals (i.e., the left half of the map), whereas the joint motion feature includes temporal high-frequency signals (i.e., the right half of the map).

For such corruptions, we predict which frequency bands are vulnerable from the Fourier heatmap in Fig. 5. For example, the Fourier heatmap of the joint feature, as shown in Fig. 5 (i.e., leftmost column) indicates that the standard-trained models are vulnerable in the temporal low-frequency domain, but the adversarially-trained models are more robust in all frequency domains. Therefore, adversarially-trained models are expected to provide better accuracies in all cases. Table 5 supports this prediction. The same holds for the other three features. Hence, adversarial training yields robustness to Gaussian noise corruptions for GCN-based skeletal action recognition.
models should be more robust to the corruption. However, the results in Table 7 demonstrate that the standard-trained models are more robust to this scenario. A possible alternative explanation is that the part occlusion corruptions cause a portion of the skeletal signals to be missing, resulting in the loss of the features necessary for action recognition. The missing signal problem cannot be explained from the Fourier perspective, and other approaches need to be considered.

## 5 Conclusions

This study examines the robustness of the GCNs for skeleton-based action recognition against adversarial attacks and common corruptions. Fourier analysis of the robustness of GCNs is reported.

We compute the average Fourier spectra of adversarial perturbations and common corruptions using the JFT, which is a combination of the GFT and DFT. Furthermore, the frequency effects of both standard and adversarial training are explored using the Fourier heatmap. Our experiments reveal that the standard-trained models are sensitive to high-frequency perturbations, and adversarial training suppresses this sensitivity and enhances robustness to high-frequency perturbations. While this robustness against high frequencies is also known in CNN-based image classification, our interesting finding is that the GCNs for skeleton-based action recognition do not suffer from a robustness trade-off between adversarial robustness and low-frequency perturbations (as do CNNs). However, when examining part occlusion corruptions, Fourier analysis performs poorly, evidencing the limitations of Fourier analysis. In future work, we will leverage our findings to further develop learning methods that yield more robustness of GCNs in skeleton-based action recognition.

## Acknowledgments

This work was supported by JSPS KAKENHI Grant Number JP22H03658.
