What do neural networks learn in image classification? A frequency
  shortcut perspective
Frequency analysis is useful for understanding the mechanisms of
representation learning in neural networks (NNs). Most research in this area
focuses on the learning dynamics of NNs for regression tasks, while little for
classification. This study empirically investigates the latter and expands the
understanding of frequency shortcuts. First, we perform experiments on
synthetic datasets, designed to have a bias in different frequency bands. Our
results demonstrate that NNs tend to find simple solutions for classification,
and what they learn first during training depends on the most distinctive
frequency characteristics, which can be either low- or high-frequencies.
Second, we confirm this phenomenon on natural images. We propose a metric to
measure class-wise frequency characteristics and a method to identify frequency
shortcuts. The results show that frequency shortcuts can be texture-based or
shape-based, depending on what best simplifies the objective. Third, we
validate the transferability of frequency shortcuts on out-of-distribution
(OOD) test sets. Our results suggest that frequency shortcuts can be
transferred across datasets and cannot be fully avoided by larger model
capacity and data augmentation. We recommend that future research should focus
on effective training schemes mitigating frequency shortcut learning.
Shunxin Wang, Raymond Veldhuis, Christoph Brune, Nicola Strisciuglio
# What do neural networks learn in image classification?

###### Abstract

Frequency analysis is useful for understanding the mechanisms of representation learning in neural networks (NNs). Most research in this area focuses on the learning dynamics of NNs for regression tasks, while little for classification. This study empirically investigates the latter and expands the understanding of frequency shortcuts. First, we perform experiments on synthetic datasets, designed to have a bias in different frequency bands. Our results demonstrate that NNs tend to find simple solutions for classification, and what they learn first during training depends on the most distinctive frequency characteristics, which can be either low- or high-frequencies. Second, we confirm this phenomenon on natural images. We propose a metric to measure class-wise frequency characteristics and a method to identify frequency shortcuts. The results show that frequency shortcuts can be texture-based or shape-based, depending on what best simplifies the objective. Third, we validate the transferability of frequency shortcuts on out-of-distribution (OOD) test sets. Our results suggest that frequency shortcuts can be transferred across datasets and cannot be fully avoided by larger model capacity and data augmentation. We recommend that future research should focus on effective training schemes mitigating frequency shortcut learning. Codes and data are available at [https://github.com/nis-research/nn-frequency-shortcuts](https://github.com/nis-research/nn-frequency-shortcuts).

## 1 Introduction

Deep neural networks (DNNs) have been widely used to tackle problems in many fields, e.g. medical data analysis, self-driving vehicles, robotics, and surveillance. However, the underlying predictive processes of DNNs are not completely understood due to the black-box nature of their nonlinear multilayer structure [3]. While a DNN can approximate any function [25], its (hundreds of) millions of parameters limit the understanding of function approximation process. Analyzing the learned features is a viable way to understand what triggers the predictions, although explaining how DNNs process data needs further exploration [31].

Researchers worked on explaining the predictions of NNs in terms of their input, using Saliency [29], Gradient-weighted Class Activation Mapping [27] and Layer-wise Relevance Propagation [2]. These techniques highlight the area of an image that contributes to prediction but do not explain why the performance of NNs degrades on OOD data. Recently, an interest in understanding the learning dynamics of NNs from a frequency perspective has grown. NNs are found to learn lower frequencies first in regression tasks [25], as they carry most of the needed information to reconstruct signals [38]. Thus NNs tend to fit low-frequency functions first to data [18]. This biased learning behavior is known as simplicity bias [28], which induces the NNs to learn simple but effective patterns, i.e. shortcuts solutions that disregard semantics related to the problem at hand but are simpler for solving the optimization task. For instance, the frequency shortcuts proposed in [34] are sets of frequencies used specifically to classify certain classes.

In this work, we empirically analyze the learning dynamics of NNs for image classification and relate it to simplicity-bias and shortcut learning from a frequency perspective. Our results indicate that simplicity-biased learn

Figure 1: Images of ‘container ship’ and ‘siamese cat’ and their DFM-filtered versions with only top-\(5\%\) dominant frequencies (the white dots in the central figures) retained can both be recognized correctly by NNs.
ing in NNs leads to frequency-biased learning, where the NNs exploit specific frequency sets, namely _frequency shortcuts_, to facilitate predictions. These frequency shortcuts are data-dependent and can be either texture-based or shape-based, depending on what best simplifies the objective function (e.g. a unique color, texture, or shape associated with a particular class in a dataset, without necessarily other meaningful semantics). This may impact generalization. We demonstrate this phenomenon through texture-based and shape-based frequency shortcuts in Fig. 1. When we retain only specific subsets of frequencies (identified using a method proposed in this paper) from images of 'container ship' and'siamese cat', the classifier can recognize them correctly. Interestingly, when the same sets of frequencies are retained from images of other classes, the predictions are biased towards these two classes, indicating that the frequency sets are specific for their classification.

Different from previous work on regression tasks [25], we investigate the learning dynamics and frequency shortcuts in NNs for image classification. Compared to the work uncovering frequency shortcuts [34], we expand the understanding of them and demonstrate that they can be texture, shape, or color, depending on data characteristics. We propose a metric to compare the frequency characteristics of data and investigate systematically the impact of present/absent shortcut features on OOD generalization. In summary, our **contributions** are:

1. We complement existing studies that showed NNs for regression tasks are biased towards low-frequency [25]. For classification, we find that NNs can exhibit different frequency biases, tending to adopt frequency shortcuts based on data characteristics because of simplicity-bias learning. Our analysis provides valuable insights into the learning dynamics of NNs and the factors influencing their behavior.
2. We propose a method to identify frequency shortcuts, based on culling frequencies that contribute less to classification. These shortcuts are composed of specific frequency subsets that correspond to textures, shapes, or colors, providing further insight into the texture-bias identified by Geirhos _et al_. [12] and background-dependency found in [36].
3. We systematically examine the influence of frequency shortcuts on the generalization of NNs and find that the presence of frequency shortcut features in an OOD test set may give an illusion of improved generalization. Furthermore, we find that larger model capacity and common data augmentation techniques like AutoAugment [5], AugMix [15], and SIN [11] cannot fully avoid shortcut learning. We recommend further research targeting frequency information to avoid frequency shortcut learning.

## 2 Related works

Frequency analysis.Recently, Fourier interpretations of NNs were published. For regression tasks, NNs tend to learn low-frequency components first [25, 37], while initial layers bias towards high-frequency components [7]. In classification, NNs exhibit a bias towards middle-high frequency during testing [1]. The authors in [1] argued that the importance of frequency is data-driven. Sensitivity to different frequency perturbations was measured in [39], showing that most NNs are more sensitive to middle-high frequency noise. The impact of high-frequency dependence on the robustness of NNs was investigated in [32]. These analyses show that NNs for regression and classification tasks exhibit different frequency dependencies, while there is a lack of analysis on the learning dynamics of NNs for classification. We study what and how NNs learn in classification, highlighting their data-driven behavior and complementing existing work on regression tasks. We uncover that NNs can learn to use specific frequency sets encompassing both low and high frequencies to achieve accurate classification.

Shortcut learning.In classification, decision rules based on spurious correlations between data and ground truth, rather than semantic cues, are known as shortcuts [10]. For example, a network may classify images based on the presence of text embedded in the images, rather than the actual image content [20], negatively impacting generalization [35]. Identifying shortcuts learned by NNs might be helpful to avoid unwanted learning behavior and thus improve generalization. It is easy to identify shortcuts that are artificially added and are visible (e.g. color patches [22], line artefacts [6], or added text [20]). However, for those implicitly existing in data (e.g. particular textures or shapes), their identification is difficult. Most methods focus on mitigating learning shortcut information in data [9, 21, 23, 26], rather than explicitly identifying them. Wang _et al_. [34] investigated shortcut learning from a frequency perspective and proposed the definition of frequency shortcuts. However, their algorithm for shortcut identification is heavily influenced by the order of frequency removal and their observations are limited to texture-based shortcuts. In this paper, our frequency shortcut identification method does not have such limitations. We broaden the understanding of frequency shortcuts, study the data-dependency of shortcut features, and provide a more systematic analysis of the impact of shortcuts on OOD generalization.

## 3 Frequency shortcuts in image classification

For regression tasks, it is known that NNs are biased towards learning low-frequency components (LFCs) first during training [25]. This has not been verified for classification tasks. Here we study the learning behavior of NNs in
image classification and its relation to shortcut learning and simplicity-bias, using both synthetic (Section 3.1) and natural images (Section 3.2). We use synthetic data to study the learning behavior of NNs and show their tendency to discover shortcuts in the frequency domain. Inspired by the insights gained on the synthetic data, we propose a method based on frequency culling to examine the frequency dependency of NNs trained on natural images, which contain intricate frequency information. This allows us to uncover the frequency shortcuts learned by NNs for classification.

### Experiments on synthetic data

Design of synthetic datasets.To study the impact of data characteristics on the spectral bias of NNs and frequency shortcut learning, we generate four synthetic datasets, each with a frequency bias in a different band, from low to high. This allows us to examine the effect of different frequency biases on the learning behavior of NNs. We separate evenly the Fourier spectrum into four frequency bands (see Fig. 2). The bands are denoted by \(B_{1}\) the lowest frequency band, \(B_{2}\) and \(B_{3}\) the mid-frequency bands, and \(B_{4}\) the highest frequency band. Each dataset contains four classes and images of \(32\times 32\) pixels. An image is generated by sampling at least eight frequencies from the frequency bands associated with the target class (see Table 1), according to a probability density function:

\[Pr(r)=S\cdot\frac{1}{r+1},\quad\text{with }S=\frac{1}{\sum_{r=1}^{R}\frac{1}{r+1}}.\]

\(R\) is the largest radius and \(r=\sqrt{u^{2}+v^{2}}\) is the radius of frequency \([u,v]\). This prioritizes the sampling of LFCs, mimicking the frequency distribution of natural images.

We use \(b\in B=\{B_{1},B_{2},B_{3},B_{4}\}\) to control the frequency bias in the generated data. For instance, in the dataset Syn\({}_{b}\) with \(b=B_{1}\), the frequency bands for classes \(C_{0}\) and \(C_{1}\) are \(\{B_{2},B_{3},B_{4}\}\) while class \(C_{3}\) has frequency band \(B_{1}\). To distinguish between \(C_{0}\) and \(C_{1}\), we embed _special patterns_ consisting of a set of frequencies \([u,v]\) (\(u=v\in\{1,3,5,7,9,11,13,15\}\)) into the images of class \(C_{0}\) which are removed from the images of other classes. The design imposes various levels of classification difficulty by incorporating different levels of data complexity for each class (\(C_{3}<C_{0}<C_{1}\approx C_{2}\)), as observed visually. This aids in comprehending the connection between simplicity-bias learning and spectral-bias of NNs in classification.

Hypothesis.As noted in the theory of simplicity-bias [28], NNs tend to achieve their objective in the simplest way. As a result, NNs for regression tasks approximate LFCs first compared to HFCs [25, 37, 17, 1]. Based on this, we hypothesize that NNs might prioritize learning to distinguish classes with the most discriminative frequency characteristics in classification. Thus, what the NNs first learn could depend on data bias rather than being limited to low frequencies. This learning behavior could result in frequency shortcut learning, where the NNs focus on specific frequencies to achieve their objective in a simpler way.

Data characteristics influence what NNs learn first.We conduct experiments on the synthetic data to test this hypothesis. We train ResNet18 models on the synthetic datasets and expect they can distinguish classes like \(C_{0}\) and \(C_{3}\) easily and from the early stages of training, as they carry more distinctive characteristics than others. To evaluate this, we measure their classification performance in the first \(500\) iterations of training by computing the F\({}_{1}\)-score per class. This provides insight into whether each class is correctly classified and how many false positives each class attracts. We report the obtained \(F_{1}\)-scores (see Fig. 3) and observe that for class \(C_{3}\) (with a clear frequency bias), the \(F_{1}\)-score is generally higher than other classes in the first few iterations, indicating that it is immediately distinguished from others across the four synthetic datasets, followed by class \(C_{0}\). This finding suggests that the more distinguishable characteristics of class \(C_{3}\) play an important role in driving the learning behavior of NNs. Note that, despite the bias in different bands across the four synthetic datasets, class \(C_{3}\) is always learned first, indicating that NNs can learn either low- or high-frequency early in training if they are more discriminative than other frequencies.

\begin{table}
\begin{tabular}{c c c} \hline \hline
**class** & **frequency bands** & **special patterns** \\ \hline \(C_{0}\) & \(B-b\) & ✓ \\ \(C_{1}\) & \(B-b\) & - \\ \(C_{2}\) & \(B\) & - \\ \(C_{3}\) & \(b\) & - \\ \hline \hline \end{tabular}
\end{table}
Table 1: Design details of a synthetic dataset Syn\({}_{b}\) with \(b\in B=\{B_{1},B_{2},B_{3},B_{4}\}\). The special pattern contains frequencies \([u,v]\) where \(u=v\in\{1,3,5,7,9,11,13,15\}\) are removed from classes other than \(C_{0}\).

Figure 2: Evenly separated frequency bands. \(B_{1}\) denotes the lowest band and \(B_{4}\) denotes the highest one.
Thus, _what frequencies are learned first by NNs in classification is driven by simplicity-bias and data characteristics._

Data bias and simplicity bias can lead to frequency shortcuts.Based on the frequency characteristics of the synthetic datasets, we examine how NNs find shortcuts in the Fourier domain by comparing the classification results of the NNs tested on the original synthetic datasets and their band-stop versions where two frequency bands in \(B\) are removed. We report the results using relative confusion matrices (see Fig. 4), computed as:

\[\Delta^{C_{i},C_{j}}=(Pred_{bs}^{C_{i},C_{j}}-Pred_{org}^{C_{i},C_{j}})/N_{c} \times 100,\]

where \(Pred_{bs}^{C_{i},C_{j}}\) is the number of samples from class \(C_{i}\) in the band-stopped test set predicted as class \(C_{j}\), \(Pred_{org}^{C_{i},C_{j}}\) is the equivalent on the original test set, and \(N_{C}\) is the number of samples in class \(C_{i}\).

As \(\Delta^{C_{i},C_{i}}\) (\(i=0,1,2,3\)) is larger than or equal to zero, the performance of the model improves or remains the same on the band-stop test sets, indicating that the limited bands provide enough discriminative information for classification, while negative values indicate lower performance. Class \(C_{2}\) in the four synthetic datasets is designed to contain frequencies from all bands. If a model can predict class \(C_{2}\) using only frequencies from partial bands instead of considering frequencies across the whole spectrum, then it is considered to likely be using frequency shortcuts to classify \(C_{2}\). Observed from Fig. 4, \(\Delta^{C_{2},C_{2}}\) are -1 and 1 for models trained on \(Syn_{B_{1}}\) and \(Syn_{B_{4}}\) respectively. The good performance indicates that NNs apply frequency shortcuts in the limited bands for classifying samples of \(C_{2}\). Moreover, \(\Delta^{C_{0},C_{0}}\) of models trained on the four synthetic datasets are close to 0, demonstrating that the NNs can recognize samples of \(C_{0}\) when only part of the frequencies (shortcuts) associated with the _special patterns_ are present in the test data. Similar behaviors are observed for other architectures (see results of AlexNet and VGG in the supplementary material). To summarize, the NNs trained on the four synthetic datasets use frequency differently, but they all adopt frequency shortcuts depending on the data characteristics.

### Experiments on natural images

The synthetic experiments show frequency characteristics of data affect what NNs learn. To analyze the more intricate frequency distributions of natural images, we introduce a metric to compare the average frequency distributions of individual classes within a dataset. This facilitates the identification of discriminative and simple class-specific frequency characteristics to learn early in training. While this metric provides valuable insights into the potential learning behavior, a deeper examination of frequency usage by NNs is also needed. To this end, we propose a technique based on frequency culling, which can help uncover frequency shortcuts explicitly. Additionally, we investigate how model capacity and data augmentation impact shortcut learning. As NNs are found to exhibit texture-bias [12] on natural images, we specifically augment data using SIN to create a dataset with more shape-bias. This better demonstrates how texture-/shape-biased data characteristics affect frequency shortcut learning.

Figure 4: Relative confusion matrices of models tested on different band-stop synthetic datasets (e.g. \(B_{14}\) indicates the bands \(B_{1}\) and \(B_{4}\) are used). The top-left figure shows the comparison of the results on the original test set and its band-stopped version for the model trained on \(Syn_{B_{1}}\). Other matrices show the results of other models. Most \(\Delta^{C_{i},C_{i}}\) (\(i=0,1,2,3\)) values are close to or larger than 0, indicating good performance on band-stopped datasets due to learned frequency shortcuts.

Figure 3: \(F_{1}\)-scores of each class in the first \(500\) training iterations. \(C_{3}\) has higher \(F_{1}\)-scores than others at the early training stage, meaning that it is learned first even if it only has frequencies sampled from the highest frequency band.
A frequency distribution comparison metric.From the insights gained on the synthetic experiments, we recognize the importance to examine the frequency characteristics of individual classes within a dataset to understand comprehensively what NNs learn. Thus, we devise a metric called Accumulative Difference of Class-wise average Spectrum (ADCS), which considers that NNs are amplitude-dependent for classification [4]. We compute the average amplitude spectrum difference per channel for each class within a set \(C=\{c_{0},c_{1},\dots,c_{n}\}\) and average it into a one-channel ADCS. The ADCS for class \(c_{i}\) at a frequency \((u,v)\) is calculated as:

\[ADCS^{c_{i}}(u,v)=\sum_{\begin{subarray}{c}\forall c_{j}\in C\\ c_{j}\neq c_{i}\end{subarray}}sign(E_{c_{i}}(u,v)-E_{c_{j}}(u,v)),\]

where

\[E_{c_{i}}(u,v)=\frac{1}{|X^{i}|}\sum_{x\in X^{i}}|\mathcal{F}_{x}(u,v)|\]

is the average Fourier spectrum for class \(c_{i}\), \(x\) is an image from the set \(X^{i}\) of images contained in that class, and \(\mathcal{F}_{x}(u,v)\) is its Fourier transform. \(ADCS^{c_{i}}(u,v)\) ranges from \(1-|C|\) to \(|C|-1\). A higher value indicates that a certain class has more energy at a specific frequency than other classes.

Impact of class-wise frequency distribution on the learning process of NNs.We choose ImageNet-10 [16], a reduced version of ImageNet [8] for the following analysis. It has lower computational requirements and greater manageability, compared to the full ImageNet dataset. For larger datasets with more classes, one may expect severe shortcut learning behaviors, as the NNs will tend to find quick solutions to simplify a more difficult classification problem.

Using ADCS, we find that the classes 'humming bird' and 'zebra' possess certain distinctive frequency characteristics that can be readily exploited by models to distinguish them from other classes at early training stages. The resulting ADCS of _'humming bird'_ (see Fig. 4(a)) indicates that samples from this class have on average much less energy than other classes across almost the whole spectrum. Conversely, the ADCS of '_zebra_' (see Fig. 4(b)) reveals that images from this class have a marked energy preponderance in the middle and high frequencies, as indicated by the prominence of red color in these frequency ranges.

To verify the impact of such frequency characteristics on the learning behavior, we train NNs on ImageNet-10. We inspect the frequency bias in the early training phase, by testing models on low- and high-pass versions of the dataset for the first \(1200\) training iterations, rather than the original test set. We compute the recall and precision of each class and observe that the precision of class '_zebra_' (see Fig. 5(a)) and the recall of class '_humming bird_' (see Fig. 5(b)) are generally higher than those of other classes. This shows that these two classes are learned faster than others. In summary, our findings indicate that NNs for classification can learn and exploit substantial spectrum differences among classes, which serve as highly discriminative features at the early learning stage. This further supports our previous observations in synthetic datasets that _what is learned first by NNs is influenced by the frequency characteristics of data_.

A frequency shortcut identification method.To identify frequency shortcuts, we propose a method based on culling irrelevant frequencies, similar to the analysis strategy in [1]. We measure the relevance of each frequency to classification by recording the change in loss value when testing a model on images of a certain class with the concerned frequency removed from all channels. The increment in loss value is used as a score to rank the importance of frequencies for classification. Frequencies with higher scores are

Figure 5: ADCS of classes ‘humming bird’ and ‘zebra’.

Figure 6: Precision and recall rates of ResNet18 trained on ImageNet-10 for the first \(1200\) iterations.
considered more relevant for classification, as their absence causes a large increase in loss. We compute a one-channel **dominant frequency map (DFM)** for a class by selecting the top-\(X\%\) frequencies according to the given ranking. Using the DFMs, we study the effect of dominant frequencies on image classification and the extent to which they indicate frequency shortcuts (specific sets of frequencies leading to biased predictions for certain classes). To quantify these, we classify all images in the test set retaining only the top-\(X\%\) frequencies of a certain class (i.e. top-\(X\%\) DFM-filtered test set). We calculate the true positive rate (TPR) and false positive rate (FPR) to evaluate their discrimination power and specificity for a certain class, respectively. We consider classes with high TPR and FPR as instances where the classifier is induced to learn and apply frequency shortcuts.
shape or other morphological features of the animal. _The learned frequency shortcuts are impacted significantly by the frequency characteristics of data. They can be texture-based or shape-based and might hinder NNs from learning more meaningful semantics._ There might be cases where frequency shortcuts are not in the data and thus not learned.

Model capacity vs. frequency shortcuts.The high TPR and FPR for ResNet50 in Table 2 indicate that it is subject to frequency shortcuts for the classification of classes 'airliner' and 'container ship'. Compared to ResNet18 frequency shortcut for class 'zebra', ResNet50 has lower TPR and FPR, indicating less specific dominant frequencies for classifying 'zebra'. This demonstrates mitigation of learning a frequency shortcut, although learning another shortcut for class 'airliner'. Additionally, VGG16 learns a frequency shortcut for class 'container ship' (TPR=0.7 and FPR=0.42). We show in the following paragraph that frequency shortcuts affect transformers as well, indicating that shortcuts impact networks across different model capacities and architectures. Thus, larger models cannot necessarily avoid it. This commonality shows that frequency shortcut learning is data-driven, which needs to be considered more explicitly to learn generalizable models.

Transferability of frequency shortcuts.We trained ViT-B on ImageNet-10 and tested it on images processed with the DFMs we had computed for ResNet18+SIN. This tests the dependency of ViT predictions on small sets of frequency, and the transferability of shortcuts between models or architectures. We present the results in Table 3 and observe shortcuts for the classes'siamese cat' (TPR=0.82, FPR=0.22) and 'container ship' (TPR=0.92, FPR=0.25). Though having a large model capacity, ViT-B is also subject to frequency shortcuts (shape or texture) to classify the samples of certain classes, in line with the observation in [24]. Moreover, the frequency shortcuts learned by ResNet18+SIN can be exploited by ViT-B, further in

\begin{table}
\begin{tabular}{l c c c c c c c c c c c} \hline \hline  & & & & & & & & & & & \\ \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**airliner**} & **wappen** & **harmoning bird** & **aimense cat** & **on** & **goldin rediscurer** & **blind frog** & **abbrn** & **container ship** & **trainer track** & **wappen** \\ \hline \multirow{2}{*}{**ResNet18**} & \(TPR\) & 0.56 & 0.8 & 0.94 & 0.98 & 0.92 & 0.9 & 0.84 & 0.96 & 0.94 & 0.96 & 0.92 \\  & \(FPR\) & 0.0944 & 0 & 0.078 & 0.067 & 0.0156 & 0.0022 & 0.0344 & 0.0022 & 0.0133 & 0.0222 \\ \cline{2-11}  & w/ _dl_ & \(TPR\) & 0.088 & 0.0 & 0.4 & 0.8 & 0.02 & 0.02 & 0.14 & **0.83** & **0.54** & 0.06 \\  & \(FPR\) & 0.0904 & 0 & 0.026 & 0.0156 & 0.0111 & 0.0944 & 0.0022 & **0.11.78** & **0.4889** & 0.0022 \\ \hline \multirow{2}{*}{**ResNet18-AutoLog**} & \multirow{2}{*}{\begin{tabular}{} \end{tabular} } & \(TPR\) & 0.92 & 0.76 & 0.88 & 0.92 & 0.96 & 0.84 & 0.64 & 0.94 & 0.83 & 0.862 \\  & \(FPR\) & 0.0899 & 0 & 0.0299 & 0.0089 & 0.0287 & 0.0111 & 0.0946 & 0.0067 & 0.0222 & 0.056 \\ \cline{2-11}  & w/ _dl_ & \(TPR\) & 0 & 0 & 0 & 0.22 & 0.04 & 0.02 & **0.26** & 0.18 & 0 \\  & \(FPR\) & 0 & 0 & 0 & 0.0607 & 0.0222 & 0.0111 & 0 & 0.0009 & 0.0622 & 0 \\ \hline \multirow{2}{*}{**ResNet18-Angliffe**} & \multirow{2}{*}{\begin{tabular}{} \end{tabular} } & \(TPR\) & 0.92 & 0.86 & 0.96 & 0.96 & 0.92 & 0.88 & 0.72 & 0.96 & 0.92 & 0.92 & 0.904 \\  & \(FPR\) & 0.0899 & 0.0602 & 0.0267 & 0.0022 & 0.0222 & 0.0044 & 0.0044 & 0.0044 & 0.0156 & 0.02 \\ \cline{2-11}  & w/ _dl_ & \(TPR\) & 0.086 & 0 & 0.22 & 0.34 & 0.22 & 0.34 & 0.02 & 0.16 & **0.88** & 0.26 \\  & \(FPR\) & 0.0067 & 0 & 0.0899 & 0.027 & 0.1511 & 0.0899 & 0.00 & 0.007 & **0.2444** & 0.0087 \\ \hline \multirow{2}{*}{**ResNet18-SIN**} & \multirow{2}{*}{\begin{tabular}{} \end{tabular} } & \(TPR\) & 0.96 & 0.96 & 0.94 & 0.96 & 0.98 & 0.86 & 0.75 & 0.96 & 0.96 & 0.92 & 0.916 \\  & \(FPR\) & 0.0102 & 0.0602 & 0.0178 & 0.0111 & 0.0244 & 0.0044 & 0.0022 & 0.0133 & 0.0156 & 0.01 \\ \cline{2-11}  & w/ _dl_ & \(TPR\) & 0.46 & 0 & 0.18 & **0.98** & 0.96 & 0.66 & 0.6 & 0.66 & 0.1 & \\  & \(FPR\) & 0.1257 & 0.0602 & 0.0111 & **0.3457** & 0.0511 & 0.0252 & 0 & 0.0022 & 0.0602 & 0.0133 & 0.013 \\ \hline \multirow{2}{*}{**ResNet10**} & \multirow{2}{*}{\begin{tabular}{} \end{tabular} } & \(TPR\) & 0.9 & 0.78 & 0.86 & 0.94 & 0.86 & 0.82 & 0.78 & 0.94 & 0.94 & 0.93 & 0.662 \\  & \(FPR\) & 0.0944 & 0.0021 & 0.02 & 0.0394 & 0.0367 & 0.0899 & 0.011 & 0.0899 & 0.0244 & 0.022 \\ \cline{2-11}  & w/ _dl_ & \(TPR\) & **0.54** & 0 & 0 & 0.42 & 0 & 0.2 & 0 & 0.16 & **0.7** & 0.1 \\  & \(FPR\) & **0.42** & 0 & 0.0022 & 0.002 & 0.0022 & 0.0253 & 0 & 0.0049 & **0.2289** & 0.0156 & 0.01 \\ \hline \multirow{2}{*}{**VCG16**} & \multirow{2}{*}{
\begin{tabular}{} \end{tabular} } & \(TPR\) & 0.96 & 0.84 & 0.92 & 1 & 0.93 & 0.92 & 0.78 & 0.96 & 0.96 & 0.98 & 0.912 \\  & \(FPR\) & 0.0021 & 0.0022 & 0.0211 & 0.013 & 0.0044 & 0.0057 & 0.0022 & 0.0133 & 0.02 \\ \cline{2-11}  & w/ _dl_ & \(TPR\) & 0.18 & 0 & 0.06 & 0.22 & 0.12 & 0.04 & 0.06 & **0.7** & 0.22 & \\ \cline{2-11}  & \(FPR\) & 0.0133 & 0 & 0 & 0.0444 & 0.1449 & 0.0257 & 0 & 0.0533 & **0.42** & 0.0578 \\ \hline \hline \end{tabular}
\end{table}
Table 2: ID test: TPRs and FPRs on ImageNet-10 and the top-\(5\%\) DFM-filtered versions (w/ df).

Figure 8: Model classifies zebra-pattern clothes with high confidence but misclassifies horse as ox. Mixing images of ‘zebra cloth’ and ‘horse’ increases the confidence of ‘zebra’ predictions. This indicates that the model relies on texture over shape information, its ability to generalize and recognize another animal of similar shape but different texture.
dicating that frequency shortcuts are data-driven and can be transferred between models.

Data augmentation vs. frequency shortcuts.As commonly techniques to improve generalization performance, we investigate the effect of data augmentation in mitigating frequency shortcut learning. We train ResNet18 with these techniques and report the results in Table 2. AugMix worsens the learned frequency shortcut for 'container ship', but mitigates a frequency shortcut for 'zebra'. AutoAugment partially avoids the frequency shortcuts for both 'zebra' and 'container ship'. SIN causes a frequency shortcut for'siamese cat'. To summarize, appropriate data augmentation may partially reduce frequency shortcut learning, but NNs still tend to find shortcut solutions based on the characteristics of the augmented data.

## 4 Frequency shortcuts and OOD tests

Design of OOD test: ImageNet-SCT.To assess how frequency shortcuts affect OOD generalization, we construct a new test set based on previous analysis results, ImageNet-SCT (ShortCut Tests). It consists of 10 classes, each containing 70 images with seven different image styles, including _art, cartoon, deviantart, painting, sculpture, sketch, toy_. This dataset expands the coverage of ImageNet-R [14] in terms of image variations. The classes in ImageNet-SCT are related, to some extent, to those in ImageNet-10. For instance, 'zebra' in ImageNet-10 corresponds to 'horse' in ImageNet-SCT, allowing us to test the effect of an absent texture-based shortcut feature, as horse images contain animals with a very similar shape to zebras, but with no texture. Similarly,'siamese cat' in ImageNet-10 corresponds to 'tabby cat' in ImageNet-SCT, to test the effect of a present shape-based shortcut feature. Furthermore, 'container ship' in ImageNet-10 maps to 'fishing vessel' in ImageNet-SCT, which contains images with similar textures and somehow different shapes (fishing vessels are much smaller boats), enabling us to evaluate the effect of a present texture-based shortcut. Examples of ImageNet-SCT images are provided in the supplementary material.

Frequency shortcuts can impair generalization and create the illusion of improved performance.We test the NNs on ImageNet-SCT and its DFM-filtered versions with the top-\(5\%\) dominant frequencies. From the results on the original ImageNet-SCT, we observe a considerable average drop of TPR for all models (see Table 4). Larger model capacity and data augmentations may not always effectively address frequency shortcuts in certain classes, as observed for'siamese cat', 'zebra', and 'container ship' in ImageNet-10 (corresponding to 'tabby cat', 'horse', and 'fishing vessel' in ImageNet-SCT). For example, models relying on texture-based shortcut features for 'zebra' in ImageNet-10 fail to capture shape characteristics and perform poorly on similar-shaped animals like 'horse' in ImageNet-SCT (see Fig. 8). While data augmentations can partially mitigate this effect in ID tests, OOD results for 'horse' still indicate the presence of learned frequency shortcuts. Conversely, 'tabby cat' and 'fishing vessel', which are designed to have similar shape or texture characteristics to their corresponding class in ImageNet-10, exhibit above-average OOD results (higher TPR than average accuracy). Thus, the present shape-based and texture-based shortcut features in the OOD test set are used for classification, giving a false sense of generalization. 'Fire truck' in ImageNet-SCT is a

\begin{table}
\begin{tabular}{l l l l l l l l l l l l l} \hline \hline \multicolumn{1}{c}{\multirow{2}{*}{**Model**}} & \multicolumn{6}{c}{**ImageNet-SCT**} \\ \cline{3-12} \multicolumn{1}{c}{} & \multicolumn{2}{c}{**military aircraft**} & \multicolumn{1}{c}{**car**} & \multicolumn{1}{c}{**horselet**} & \multicolumn{1}{c}{**halbity cat**} & \multicolumn{1}{c}{**halbition**} & \multicolumn{1}{c}{**horselet**} & \multicolumn{1}{c}{**hercave**} & \multicolumn{1}{c}{**free fing**} & \multicolumn{1}{c}{**horse**} & \multicolumn{1}{c}{**fishing vessel**} & \multicolumn{1}{c}{**free truck**} & \multicolumn{1}{c}{**average**} \\ \hline
**ResNet18** & & \(TPR\) & 0.3286 & 0.443 & 0.4249 & 0.2714 & 0.3286 & 0.4 & 0.4413 & 0.0286 & **0.4286** & 0.6413 & 0.3672 \\  & \(FPR\) & 0.0794 & 0.0397 & 0.1952 & 0.0921 & 0.0176 & 0.0587 & 0.0495 & 0.0199 & **0.4283** & 0.0778 & \\ \cline{2-12}  & \(\pi/d\) & \(FPR\) & 0.0 & 0.9 & 0.1433 & 0.1986 & 0.0429 & 0.0406 & 0.0571 & 0.1286 & **0.2143** & 0 \\  & \(FPR\) & 0.016 & 0.0 & 0.0545 & 0.056 & 0.0683 & 0.0238 & 0.0683 & 0.0899 & **0.3397** & 0.0016 & \\ \hline
**ResNet18-Agnetag** & \(TPR\) & 0.4 & 0.8571 & 0.540 & 0.4 & 0.4857 & 0.4286 & 0.3286 & 0.0 & 0.4 & 0.6413 & 0.4219 \\  & \(FPR\) & 0.0000 & 0.0667 & 0.1699 & 0.0371 & 0.1 & 0.0444 & 0.0020 & 0.0079 & 0.0463 & 0.0669 \\ \cline{2-12}  & \(\pi/d\) & \(TPR\) & 0 & 0 & 0.0402 & 0.2143 & 0.0450 & 0.0433 & 0.0585 & 0.0420 & 0.0657 & \\  & \(FPR\) & 0 & 0 & 0.0444 & 0.1016 & 0.0413 & 0.0 & 0.0379 & 0.0778 & 0.0127 & \\ \hline
**ResNet18-AugMix** & \(TPR\) & 0.3371 & 0.7286 & 0.4413 & 0.2714 & 0.3857 & 0.4249 & 0.3571 & 0.0286 & **0.4433** & 0.5571 & 0.3657 \\  & \(FPR\) & 0.0984 & 0.1159 & 0.1248 & 0.0818 & 0.0889 & 0.054 & 0.0397 & 0.0111 & **0.04175** & 0.0397 \\ \cline{2-12}  & \(\pi/d\) & \(TPR\) & 0.0 & 0.1143 & 0.0429 & 0.2 & 0 & 0 & **0.5** & 0.1429 & \\  & \(FPR\) & 0.0988 & 0.0 & 0.0055 & 0.0456 & 0.1 & 0.0821 & 0.0 & 0.0111 & **0.02** & 0.1069 \\ \hline
**ResNet18-SIN** & \(TPR\) & 0.3357 & 0.6 & 0.4285 & **0.4941** & 0.5286 & 0.5714 & 0.4571 & 0.0 & 0.6429 & 0.6857 & 0.48714 \\  & \(FPR\) & 0.0333 & 0.0444 & 0.1016 & **0.0476** & 0.1159 & 0.0635 & 0.0492 & 0.0212 & 0.0127 & 0.0794 \\ \cline{2-12}  & \(\pi/d\) & \(TPR\) & 0.0429 & 0 & 0.0714 & **0.2956** & 0.0714 & 0.1714 & 0 & 0.0 & 0.04249 & 0.0286 & \\  & \(FPR\) & 0.0359 & 0.0016 & 0.0222 & **0.4444** & 0.0042 & 0.1016 & 0.0 & 0.0195 & 0.1127 & 0.0005 & \\ \hline
**ResNet19** & & \(TPR\) & 0.4266 & 0.4857 & 0.4243 & 0.2 & 0.3714 & 0.3 & 0.3 & 0.3 & 0.3 & 0.3 & 0.3517 & **0.4429** & 0.3743 \\  & \(FPR\) & 0.1444 & 0.054 & 0.0952 & 0.0651 & 0.0984 & 0.0902 & 0.0365 & 0.0272 & **0.8333** & 0.0021 & \\ \cline{2-12}  & \(\pi/d\) & \(TPR\) & 0.3429 & 0 & 0.0717 & 0.0429 & 0 & 0.2 & 0 & 0 & **0.4857** & 0.4929 & \\  & \(FPR\) & 0.127 & 0 & 0.0022 & 0.0026 & 0.0 & 0.1444 & 0.0066 & 0.0197 & 0.1922 & 0.0111 & \\ \hline
**VCG16** & & \(TPR\) & 0.5410 & 0.6571 & 0.6244 & 0.3 & 0.3571 & 0.7174 & 0.5433 & 0.5486 & **0.5486** & 0.5 & 0.402 \\  & \(FPR\) & 0.0841 & 0.0714 & 0.1288 & 0.0773 & 0.0905 & 0.0942 & 0.0968 & 0.0143 & **0.0111** & 0.0544 & \\ \cline{2-12}  & \(\pi/d\) & \(TPR\) & 0.0943 & 0 & 0.0286 & 0.2571 & 0.2143 &
good example of generalization, as no shortcuts were identified, allowing models to learn more global and semantic information. Frequency shortcuts can impair generalization and their impact can transfer across datasets, resulting in a misleading impression of generalization with the inclusion of shortcut features in a new test set. Larger models and data augmentation cannot fully counteract these effects, we thus highlight the need to explore novel data augmentation strategies that explicitly target shortcut mitigation, e.g. leveraging DFMs to induce models to exploit more frequencies rather than shortcut frequencies [33] and avoid learning behaviors that may impair the generalizability of NNs.

## 5 Conclusions

We conducted an empirical study to investigate what NNs learn in image classification, by analyzing the learning dynamics of NNs from a frequency shortcut perspective. We found from a synthetic example that **NNs learn frequency shortcuts during training to simplify classification tasks, driven by frequency characteristics of data and simplicity-bias**. To address this on natural images, we proposed a metric to measure class-wise frequency characteristics and a method to identify frequency shortcuts. We evaluated the influence of shortcuts on OOD generalization and found that **frequency shortcuts can be transferred to another dataset, in some cases, giving an illusion of improved generalization**. Furthermore, we observed that larger model capacity and data augmentation techniques do not necessarily mitigate frequency shortcut learning. Our study expands previous works on the learning dynamics of NNs for regression tasks, broadens the understanding of frequency shortcuts (which can be either texture-based or shape-based), and provides a more systematic analysis of OOD generalization. We foresee that enhancing the identification of frequency shortcuts and applying proper training schemes that avoid frequency shortcut learning may hold promise in improving generalization.

## Acknowledgements

This work was supported by the SEARCH project ([https://sites.google.com/view/search-utwente](https://sites.google.com/view/search-utwente)), UT Theme Call 2020, Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente.
