Neural Network Training Strategy to Enhance Anomaly Detection
  Performance: A Perspective on Reconstruction Loss Amplification
Unsupervised anomaly detection (UAD) is a widely adopted approach in industry
due to rare anomaly occurrences and data imbalance. A desirable characteristic
of an UAD model is contained generalization ability which excels in the
reconstruction of seen normal patterns but struggles with unseen anomalies.
Recent studies have pursued to contain the generalization capability of their
UAD models in reconstruction from different perspectives, such as design of
neural network (NN) structure and training strategy. In contrast, we note that
containing of generalization ability in reconstruction can also be obtained
simply from steep-shaped loss landscape. Motivated by this, we propose a loss
landscape sharpening method by amplifying the reconstruction loss, dubbed Loss
AMPlification (LAMP). LAMP deforms the loss landscape into a steep shape so the
reconstruction error on unseen anomalies becomes greater. Accordingly, the
anomaly detection performance is improved without any change of the NN
architecture. Our findings suggest that LAMP can be easily applied to any
reconstruction error metrics in UAD settings where the reconstruction model is
trained with anomaly-free samples only.
YeongHyeon Park, Sungho Kang, Myung Jin Kim, Hyeonho Jeong, Hyunkyu Park, Hyeong Seok Kim, Juneho Yi
Neural Network Training Strategy to Enhance Anomaly Detection Performance: A Perspective on Reconstruction Loss Amplification

###### Abstract

Unsupervised anomaly detection (UAD) is a widely adopted approach in industry due to rare anomaly occurrences and data imbalance. A desirable characteristic of an UAD model is contained generalization ability which excels in the reconstruction of seen normal patterns but struggles with unseen anomalies. Recent studies have pursued to contain the generalization capability of their UAD models in reconstruction from different perspectives, such as design of neural network (NN) structure and training strategy. In contrast, we note that containing of generalization ability in reconstruction can also be obtained simply from steep-shaped loss landscape. Motivated by this, we propose a loss landscape sharpening method by amplifying the reconstruction loss, dubbed _Loss AMPlification_ (LAMP). LAMP deforms the loss landscape into a steep shape so the reconstruction error on unseen anomalies becomes greater. Accordingly, the anomaly detection performance is improved without any change of the NN architecture. Our findings suggest that LAMP can be easily applied to any reconstruction error metrics in UAD settings where the reconstruction model is trained with anomaly-free samples only.

YeongHyeon Park\({}^{1,2}\),  Sungho Kang\({}^{1}\) Myung Jin Kim\({}^{2}\)  Hyeonho Jeong\({}^{3}\) Hyunkyu Park\({}^{1}\)  Hyeong Seok Kim\({}^{2}\)  Juneho Yi\({}^{1}\)\({}^{1}\)Department of Electrical and Computer Engineering, Sungkyunkwan University

\({}^{2}\)SK Planet Co., Ltd.

\({}^{3}\)College of Computing, Sungkyunkwan University

{yeonghyeon, myungjin, beman}@sk.com, {sungho369, drake6751, mjss016, jhyi}@skku.edu

Unsupervised anomaly detection, Loss amplification, Loss landscape, Training strategy

## 1 Introduction

Exploiting a reconstruction model trained with anomaly-free samples only is a widely adopted approach to unsupervised anomaly detection (UAD) in various industries due to its capability to resolve the challenges posed by the scarcity of abnormal situations and data imbalance problems. The desirable characteristic of a trained UAD model is contained generalization ability in reconstruction. That is, the model should excel in reconstruction of seen normal patterns but struggle with unseen anomalous patterns.

An easy way for containing of the generalization ability of an UAD model in reconstruction is to pour all its reconstruction capability into normal patterns so that there is no room to cover anomalous patterns. Various methods have been proposed to further improve the anomaly detection (AD) performance by containing generalization ability, but the focus has been mostly on exploring new neural network (NN) structures or extensions. NN designs in UAD can be divided into three main categories: 1) generative adversarial networks (GAN) that additionally include a discriminator on a generative model [4, 5, 6], 2) a memory module that can forge nor

Figure 1: Effect of LAMP. The loss landscapes and their contour projections for \(\mathcal{L}_{2}\) and \(\mathcal{L}_{2}^{LAMP}\) are shown in the first and second rows, respectively. The loss landscape for an UAD model should be shaped with a steep and sharp form in order to contain the reconstruction generalization ability of the model and enhance the AD performance [1, 2, 3].
mal latent features to reconstruct normal-like images [7, 8, 9, 10], and 3) online knowledge distillation to prevent the model from generating a fixed, constant normal image regardless of changing input [11, 10, 12].

In contrast, we exploit the research results reported in [1] that the generalization ability of NNs is related with the shape of a loss landscape. They report that in a classification task, NNs with smooth-shaped loss landscape show better generalization ability compared to sharp shapes. Their loss landscape visualization method is also utilized in generative models [3] and can also be applied to our reconstruction model. The loss landscape for an UAD model should be shaped with a steep and sharp form in order to contain the reconstruction generalization ability of the model and enhance the AD performance.

Based on the observation that reconstruction loss amplification causes a sharp shape of the loss landscapes, we propose a method of only changing the reconstruction loss function via amplification, dubbed _Loss AMPlification_ (LAMP). When LAMP is applied, it actually transforms the loss landscape to a sharp form as shown in Fig. 1. To verify the legitimacy of our method, we compare the loss landscape using the same encoder/decoder but with different loss functions and batch sizes. For the comparison, the MNIST dataset [13] is used. We confirm that when LAMP is applied to the reconstruction error metric for training, it not only transforms the loss landscape into a steeper shape for all batch sizes but also actually enhances the AD performance. We conduct additional experiments for the MVTec AD dataset [14] dealing with 15 AD tasks considering combinations of different loss functions and optimizers. The experimental results show that the AD performance is improved in most cases.

Extensive experiments demonstrate that the application of LAMP leads to an improved AD performance, which is achieved via only loss amplification without the structural change or expansion of NNs. LAMP can be easily and safely applied across any reconstruction error metrics when training NNs in UAD settings.

## 2 Proposed Method

### Reconstruction generalization for UAD

Loss landscape visualization can provide insight to relate the shape of the loss landscape to the reconstruction generalization ability of an NN model. When the loss landscape is smooth, a reconstruction model has high generalization ability. High generalization ability means that unseen patterns can be well reconstructed at the test time.

However, in UAD, contained generalization ability is crucial because the criterion for determining whether an input sample is defective relies on the magnitude of the reconstruction error. Note that, when an UAD model has high generalization ability, it will cause the model to reconstruct unseen patterns accurately and fail to detect defective products due to the small reconstruction error.

In this paper, we propose reconstruction loss amplification as a simple way to affect the generalization ability of an UAD model in reconstruction without altering the structure of the NNs or training strategy.

### Loss amplification

The proposed method, LAMP, is a simple trick that can improve the AD performance by just amplifying the base loss function \(\mathcal{L}_{base}\). Note that, \(\mathcal{L}_{2}\), \(\mathcal{L}_{1}\), \(\mathcal{L}_{SSIM}\), etc. can be adopted as \(\mathcal{L}_{base}\). LAMP is formulated in (1). Instead of increasing the learning rate or weighting each loss term, LAMP imposes a larger penalty than the base loss function, \(\mathcal{L}_{base}\).

\[\begin{split}\mathcal{L}_{base}^{LAMP}(y,\hat{y})=\sum_{h=1}^{H} \sum_{w=1}^{W}\sum_{c=1}^{C}-\log&\Big{(}1-\mathcal{L}_{base}( y,\hat{y})\Big{)},\\ w.r.t.\leavevmode\nobreak\ y\in\mathbb{R}^{H\times W\times C} \end{split} \tag{1}\]

LAMP makes gradients steeper than the base loss function as shown in Fig. 2, accelerating loss convergence. This steeper gradient transforms the loss landscape shape of an UAD model into a sharp form, containing the reconstruction generalization ability. Note that we can safely amplify the reconstruction loss because an UAD model is only trained using anomaly-free samples.

Figure 2: Loss curves for LAMP-applied \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\) cases. \(\mathcal{L}_{1}\) and \(\mathcal{L}_{2}\) cases are shown in the first and second rows, respectively and their gradients are shown in the second column. LAMP imposes a larger penalty than the base loss function, \(\mathcal{L}_{base}\).
### Scaling trick

The input of the log function in LAMP must be guaranteed to be positive. For this, we should ensure that a negative value does not occur for '\(1-\mathcal{L}_{base}\)' operation. We use a simple scaling trick that normalizes \(\mathcal{L}_{base}\) between 0 and 1 as in (2). We also multiply the coefficients '\(1-\epsilon\)' to adjust \(\mathcal{L}^{\prime}_{base}\) to slightly less than 1 which prevents '\(\log(1-\mathcal{L}^{\prime}_{base})\)' from exploding.

\[\mathcal{L}^{\prime}_{base}(y,\hat{y})=\frac{\mathcal{L}_{base}(y,\hat{y})}{ \max{(\mathcal{L}_{base}(y,\hat{y}))}}(1-\epsilon) \tag{2}\]

## 3 Experiments

### Experimental setup

A basic experiment is designed with the MNIST dataset [13] to see whether LAMP improves the AD performance. The MNIST dataset [13] is originally provided for classification of digits into ten classes, but we redesign it for AD experiments by setting one class as normal and the other nine as abnormal. For example, if the class '0' is set as normal, the AE will be trained by '0' only with the intent of filtering out unseen other nine digit classes '1-9' by large reconstruction error. We also report experimental results using the industrial dataset, MVTec AD [14]. The training set only includes anomaly-free samples, but the test set includes both anomaly-free and anomalous samples.

**Implementation details.** We design a simple convolutional autoencoder (AE) referring to the previous study [15] while discarding skip connections which enables better generalization ability. Basically, the AE is structured with six layers for encoder and decoder respectively, but for low-resolution datasets such as MNIST [13], we change it into a four-layered structure for each. Note that we repeat 'convolution \(\rightarrow\) batch normalization \(\rightarrow\) leaky ReLU activation' for the encoder and 'upsampling \(\rightarrow\) convolution \(\rightarrow\) batch normalization \(\rightarrow\) leaky ReLU activation' for the decoder.

**Training conditions.** In all AD experiments, we perform hyperparameter tuning to compare the best performance of each model. The following hyperparameters are tuned: 1) batch size, 2) a number of patches for patch-wise reconstruction [16], 3) learning rate, and 4) kernel size. For the experiment using the industrial dataset, MVTec AD [14], we use additional training conditions: 1) base loss functions (\(\mathcal{L}_{2}\), \(\mathcal{L}_{1}\), and \(\mathcal{L}_{SSIM}\)), and 2) NN optimizers (SGD [17], RMSprop [18], and Adam [19]).

**Evaluation metric.** We use the area under the receiver operating characteristic curve (AUROC) [20] as an evaluation metric in the AD experiments. The reconstruction error is also called an anomaly score in the AD task. In this study, \(\mathcal{L}_{2}\) between the input \(y\) and reconstruction output \(\hat{y}\) is used as an anomaly score. AUROC will be close to 1 when the reconstruction errors of the AE for the unseen anomalous patterns are relatively larger compared to errors in normal pattern reconstruction.

### Comparison of loss landscapes

We visualize the loss landscape of \(\mathcal{L}_{2}\) and \(\mathcal{L}_{2}^{LAMP}\) for the batch sizes of 128, 16, and 4. \(\mathcal{L}_{2}^{LAMP}\) denotes \(\mathcal{L}_{2}\) metric amplified by LAMP. For visualization, an open source1 provided by Li et al. [1] is used. The visualization results in Fig. 3 show the loss landscape only for the encoder part of the AE that is trained to predict class labels of the MNIST dataset [13]. The results indicate that all \(\mathcal{L}_{2}^{LAMP}\) cases always show dense contours which means steeper loss landscapes compared to \(\mathcal{L}_{2}\).

Footnote 1: [https://github.com/fomgoldstein/loss-landscape](https://github.com/fomgoldstein/loss-landscape)

### AD experiments using MNIST dataset

Our work is based on the fact that when the loss landscape of an NN is shaped steeper, it will show better AD performance due to the contained generalization ability [1]. The results shown in Table 1 experimentally prove our claim that LAMP-applied cases transform the loss landscape steeper and always outperform regardless of the batch size. Moreover, in a large batch size which yields a smoother loss landscape, the gap

\begin{table}
\begin{tabular}{l|c c c c c c} \hline \multirow{2}{*}{**Loss**} & \multicolumn{6}{c}{**Batch size**} \\ \cline{2-7}  & **1024** & **128** & **32** & **16** & **4** & **1** \\ \hline \(\mathcal{L}_{2}\) & 0.658 & 0.919 & 0.921 & 0.926 & 0.931 & 0.919 \\ \(\mathcal{L}_{2}^{LAMP}\) & **0.712** & **0.925** & **0.929** & **0.929** & **0.932** & **0.927** \\ \hline \end{tabular}
\end{table}
Table 1: The average performance for ten AD tasks using the MNIST dataset [13]. LAMP-applied loss function, \(\mathcal{L}_{2}^{LAMP}\), always outperforms \(\mathcal{L}_{2}\).

Figure 3: The contour representation of the loss landscapes with three batch size (BS) conditions for \(\mathcal{L}_{2}\) and \(\mathcal{L}_{2}^{LAMP}\). It is known that the generalization ability is contained when the BS is small [1]. We experimentally confirm the same effect for LAMP.
between AUROCs is greater, that is, the effect of LAMP is maximized. This result experimentally demonstrates that the sharpness of the loss landscape can be exploited to enhance the AD performance.

### Results for industrial dataset

We fix the batch size to 16 in this experiment considering the training speed. Table 2 shows the experimental results on the industrial dataset, MVTec AD [14], in nine experimental settings from three different base loss functions and three optimizers. In 5 out of 9 experimental settings, LAMP-applied cases achieve equal or better performances. The last column of Table 2 shows the best performance for each subtask and \(\mathcal{L}_{base}^{LAMP}\) attains better AUROC than \(\mathcal{L}_{base}\).

In Fig. 4, we present reconstruction results for the best models trained with each \(\mathcal{L}_{base}\) and \(\mathcal{L}_{base}^{LAMP}\). \(\mathcal{L}_{base}\) produces blurry results for normal products in capsule, metal nut, and pill cases, causing a large reconstruction error. Furthermore, a pill case shows fixed reconstruction results regardless of changing the input, undermining the reliability of the AD performance. In contrast, \(\mathcal{L}_{base}^{LAMP}\) case demonstrates accurate reconstructions for normal samples. Note the clear visibility of the number '500' on the normal capsule. In cases of unseen anomalous patterns, they are converted closer to a seen normal pattern, which is intended of a reconstruction model in UAD settings. This yields accurate detection of defective samples. Via the results of extensive experiments, we confirm that LAMP improves the AD performance in most cases quantitatively and qualitatively.

## 4 Conclusion

In this paper, we propose a simple method to enhance the AD performance in an UAD setting from the perspective of reconstruction loss amplification by noting that contained generalization ability is highly related to sharp-shaped loss landscapes.

To show the legitimacy of our approach, we design extensive experiments with MNIST and MVTec AD datasets. We have shown the shape change of the reconstruction loss landscape when LAMP is applied as we vary the batch size. We demonstrate quantitative and qualitative performance enhancement of an UAD model by LAMP under various conditions. LAMP can be safely applied to any reconstruction error metrics in an UAD setup where a reconstruction model is trained with anomaly-free samples only.

\begin{table}
\begin{tabular}{l||c c|c c|c c|c c|c c|c c|c c|c c} \hline
**Training** & \multicolumn{4}{c|}{\(\mathcal{L}_{2}\to\mathcal{L}_{SIGHT}^{LAMP}\)} & \multicolumn{4}{c|}{\(\mathcal{L}_{SIGHT}\)} & \multicolumn{4}{c}{\(\mathcal{L}_{SIGHT}\)} & \multicolumn{4}{c}{**Best**} \\ \hline
**Optimizer** & \multicolumn{2}{c|}{**SGD**} & \multicolumn{2}{c|}{**RMSprop**} & \multicolumn{2}{c|}{**Adam**} & \multicolumn{2}{c|}{**SGD**} & \multicolumn{2}{c|}{**RMSprop**} & \multicolumn{2}{c|}{**Adam**} & \multicolumn{2}{c|}{**SGD**} & \multicolumn{2}{c|}{**RMSprop**} & \multicolumn{2}{c|}{**Adam**} & \multicolumn{2}{c|}{**SGD**} & \multicolumn{2}{c|}{**RMSprop**} & \multicolumn{2}{c|}{**Adam**} & \multicolumn{2}{c|}{\(\mathcal{L}_{base}\to\mathcal{L}_{SIGHT}^{LAMP}\)} \\ \hline Bottle & 0.952 \(\rightarrow\) 0.983 & 0.990 \(\rightarrow\) 0.990 & 0.994 \(\rightarrow\) 0.991 & 0.989 \(\rightarrow\) 0.929 & 0.993 \(\rightarrow\) 0.994 & 0.994 \(\rightarrow\) 0.992 & 0.993 \(\rightarrow\) 0.990 & 0.994 \(\rightarrow\) 0.994 & 0.994 \(\rightarrow\) 0.994 & 0.994 \(\rightarrow\) 0.993 & **0.994** \(\rightarrow\) **0.