# DistilDIRE: A Small, Fast, Cheap and Lightweight Diffusion Synthesized
Deepfake Detection

Yewon Lim  Changyeon Lee  Aerin Kim  Oren Etzioni

###### Abstract

A dramatic influx of diffusion-generated images has marked recent years,
posing unique challenges to current detection technologies. While the task of
identifying these images falls under binary classification, a seemingly
straightforward category, the computational load is significant when employing
the “reconstruction then compare” technique. This approach, known as DIRE
(Diffusion Reconstruction Error), not only identifies diffusion-generated
images but also detects those produced by GANs, highlighting the technique’s
broad applicability. To address the computational challenges and improve
efficiency, we propose distilling the knowledge embedded in diffusion models
to develop rapid deepfake detection models. Our approach, aimed at creating a
small, fast, cheap, and lightweight diffusion synthesized deepfake detector,
maintains robust performance while significantly reducing operational demands.
Maintaining performance, our experimental results indicate an inference speed
3.2 times faster than the existing DIRE framework. This advance not only
enhances the practicality of deploying these systems in real-world settings
but also paves the way for future research endeavors that seek to leverage
diffusion model knowledge. The code and weights for our framework are
available at https://github.com/miraflow/DistilDIRE.

Diffusion, Deepfake, Foundation Models, Efficient Deep Learning

##  1 Introduction

From generating high-fidelity images (Rombach et al., 2022; Karras et al.,
2017, 2019; Saharia et al., 2022) to synthesizing complex videos (Brooks et
al., 2024; Bar-Tal et al., 2024; Gao et al., 2024) and musical compositions
(Huang et al., 2023), and deepfakes (Chen et al., 2023; Bhattacharyya et al.,
2024), diffusion models (Ho et al., 2020; Song et al., 2022) have demonstrated
exceptional capabilities in various domains by leveraging stochastic diffusion
processes.

Current deepfake detectors struggle to distinguish unseen diffusion-generated
images. To address this, researchers have developed a new method that uses a
pre-trained diffusion model to more accurately reconstruct these images. This
method uses the error between an input image and its reconstructed version,
known as the DIRE (Wang et al., 2023) representation, as input for a binary
classifier to detect deepfakes. This ‘reconstruction then compare’ approach
has achieved state-of-the-art results on the DiffusionForensic dataset (Wang
et al., 2023), which includes both GAN (Goodfellow et al., 2020) generated and
diffusion-generated images.

The DIRE framework, although it works well, is too slow for practical use
because it needs to compute the diffusion trajectory during training and
inference. We address this limitation by proposing a distilling approach that
uses a fraction of the information from pretrained diffusion model and
predefined teacher classifier model. Specifically, We leverage the DIRE
feature extracted by a ImageNet pre-trained ResNet-50 classifier (_teacher_)
to train a binary classifier (_student_) for deepfake detection. Additionally,
we incorporate the first-time step noise from a pre-trained diffusion model
with the original image, effectively conveying the image distribution as
interpreted by the diffusion.

This method has led to the development of ‘ _DistilDIRE_ ’, a small, fast,
cheap and lightweight deepfake detection model suitable for real-world
applications.

##  2 Related Work

Our work aims to develop a light and fast diffusion synthesized deepfake
detection model with only fraction of diffusion trajectory information.

###  2.1 Denoising Diffusion Implicit Models

Diffusion models introduce a novel approach to generative modeling by
leveraging inverse transformations to progressively reduce noise in data
generation. While DDPM (Ho et al., 2020) constructs the generation process
using Marcov chains and thus require sequential sampling without skipping,
DDIM (Song et al., 2022) can generate images faster than DDPM due to its
unique design without Marcov Chain. It models reverse process as follows.

$\displaystyle\text{x}_{t-1}=\sqrt{\alpha_{t-1}}\left(\frac{\text{x}_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(\text{x}_{t},t)}{\sqrt{\alpha_{t}}}\right)$
(1)
$\displaystyle+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2}}\cdot\mathbf{\epsilon}_{\theta}(\text{x}_{t},t)+\sigma_{t}\epsilon_{t}$

When $\sigma_{t}$ goes 0, Equation 1 becomes deterministic from which we can
approximately revert image to noise, which is called DDIM inversion.
Specifically, DDIM inversion is performed as follows.

$\displaystyle\frac{\text{x}_{t+1}}{\sqrt{\alpha_{t+1}}}=\frac{\text{x}_{t}}{\sqrt{\alpha_{t}}}+\left(\sqrt{\frac{1-\alpha_{t+1}}{\alpha_{t+1}}}-\sqrt{\frac{1-\alpha_{t}}{\alpha_{t}}}\right)\epsilon_{\theta}(\text{x}_{t},t).$
(2)

This process aims to obtain the corresponding noisy sample $\text{x}_{T}$ for
an input image $\text{x}_{0}$. However, inverting or sampling step-by-step is
highly time-consuming. To accelerate the diffusion model sampling, DDIM allows
sampling a subset of $S$ steps $\tau_{1},\dots,\tau_{S}$, where the
neighboring $\text{x}_{t}$ and $\text{x}_{t+1}$ become $\text{x}_{\tau_{t}}$
and $\text{x}_{\tau_{t+1}}$, respectively, in Equation 1 and Equation 2.

###  2.2 Diffusion Synthesized Image Detection

As diffusion models advance and generate seamless images, it becomes
increasingly challenging to detect these artificially created ones. To tackle
this, DIRE (Wang et al., 2023) offers a solution by leveraging pre-trained
diffusion models and extracting their implicit understanding of image
distributions. Their research reveals that genuine images exhibit significant
reconstruction errors when subjected to DDIM inversion (Song et al., 2022),
whereas counterfeit images generated by diffusion models tend to yield lower
reconstruction errors.

However, a notable bottleneck arises in terms of inference time, as
implementing the DIRE framework requires invoking the ADM model at least 40
times to obtain the DIRE image.

DNF, or Diffusion Noise Feature (Zhang & Xu, 2024), represents another
advancement in diffusion-generated image detection. This method delves into
the intricate process of diffusion, revealing that the predicted noises
generated during the inverse diffusion process can serve as valuable
indicators of the underlying image distribution. By exploiting these noise
features, deciding whether given image is genuine or counterfeit can be done
with higher accuracy.

However, DNF also requires multiple iterations of diffusion model calls before
the deepfake probability can be calculated. This highlights the computational
complexity involved in such image detection methods and underscores the
ongoing efforts to streamline and optimize these processes for practical
implementation.

##  3 Methodology

###  3.1 Architecture

The DistilDire framework, shown in Figure 1, uses a pre-trained ResNet-50 (He
et al., 2016), which was initially trained on the ImageNet-1K (Deng et al.,
2009) dataset, as a teacher model.

Figure 1:  Overview of our framework.

The teacher model, loaded with pretrained weights, remains frozen during
training. This model provides a stable reference for the student model,
ensuring that the extracted features are robust and generalized. Unlike the
teacher model, the student model is trained from scratch. During the training
process, the student model simultaneously incorporates classification loss and
knowledge distillation loss.

What makes our method unique is that we combine the original image and the
predicted noise from that original image into the learning (student) model.
Specifically, we utilize noise from the first time step obtained from a
pretrained Ablated Diffusion Model (ADM) (Dhariwal & Nichol, 2021) as an
additional input. The final input to the student model consists of the
original image x itself and the noise $\epsilon_{0}$ from the first denoising
step. The input to the model is as follows, where $i$ represents the index of
the mini-batch.

$\textbf{x}^{\prime}_{i}=\mathsf{cat}(\textbf{x}_{i},\epsilon_{0,i})$ (3)

The student model then predicts whether the input image is real or synthetic
(fake).

Knowledge distillation from the teacher model is achieved by extracting
feature maps before the classifier head and computing the Mean Squared Error
(MSE) loss between these feature maps and those of the student model. The
knowledge distillation loss serves as regularization, encouraging the student
model’s feature maps to mirror the teacher model’s without the need for time-
intensive inversion and reconstruction.

The student model uses Binary Cross-Entropy Loss (BCE). This helps the model
learn how to tell the difference between real and synthetic images. The final
way we calculate loss is shown in Equation 4, Equation 5, and Equation 6.

$\displaystyle\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{classification}}+\lambda\cdot\mathcal{L}_{\text{distillation}}$
(4)
$\mathcal{L}_{\text{classification}}=-\frac{1}{N}\sum_{i=1}^{N}(\mathbf{y}_{i}\mathrm{log}(\mathbf{y}^{\prime}_{i})+(1-\mathbf{y}_{i})\mathrm{log}(1-\mathbf{y}^{\prime}_{i}))$
(5)

$\mathcal{L}_{\text{distillation}}=\frac{1}{N}\sum_{i=1}^{N}\lVert\mathcal{F^{\prime}}(\mathsf{DIRE}(\textbf{x}_{i}))-\mathcal{F}(\textbf{x}^{\prime}_{i})\rVert_{2}\vspace{-1mm}$
(6)

In the above equations, $\mathbf{y}_{i},\mathbf{y}_{i}^{\prime}$,
$\mathcal{F^{\prime}}$, and $\mathcal{F}$ represents ground-truth label, model
prediction, ImageNet-1K pretrained ResNet-50 feature extractor, and naive
ResNet-50 feature extractor, respectively. Here, $N$ is a mini-batch size and
$\lambda$ decides how much the distillation process will affect the outcome.

In essence, DistilDIRE uses a pretrained teacher model for generalized feature
extraction, while the student model learns its ability to distinguish real
from synthetic images through classification and knowledge distillation
losses. This approach balances the use of pretrained knowledge and effective
learning of new representations.

###  3.2 Noise Extraction

In DistilDIRE, we extract the initial noise $\mathbf{\epsilon}_{0}$ to
determine whether a given image is generated by a diffusion model. By
analyzing the characteristics of $\mathbf{\epsilon}_{0}$, we aim to identify
signatures that are indicative of diffusion-generated images.

To extract $\mathbf{\epsilon}_{0}$, we leverage the DDIM formulation (Equation
2). By focusing on the case when $t=1$, we can isolate the initial noise
$\mathbf{\epsilon}_{0}$ from the initial image $\textbf{x}_{0}$:

$\displaystyle\mathbf{\epsilon}_{0}=\frac{\textbf{x}_{1}-\sqrt{\alpha_{1}}\hat{\textbf{x}}_{0}}{\sqrt{1-\alpha_{1}}}.$
(7)

Extracted $\mathbf{\epsilon}_{0}$ is concatenated with the input image
$\textbf{x}_{0}$ and then forwared to classifier to determine whether given
image is synthesized or not.

Table 1:  Quantitative evaluation (Accuracy (%), AP (%)) of our framework
alongside other state-of-the-art deepfake detection methodologies on the
ImageNet and CelebA-HQ datasets. ‘*’ denotes the generators present in the
training dataset of each subset dataset. ImageNet test subset image generators
are ADM (Dhariwal & Nichol, 2021), Stable Diffusion v1 (Rombach et al., 2022)
and CelebA-HQ test subset image generators are IF (Deep Floyd IF, 2023),
DALL-E 2 (Ramesh et al., 2022), Stable Diffusion v2 (Rombach et al., 2022),
Midjourney (Midjourney, 2022)

Method Training Dataset ImageNet CelebA-HQ Performance Testing Generators
Testing Generators Average ADM* SD-v1 IF DALL-E 2 SD-v2* Midjourney CNNDet
(Wang et al., 2020) DiffusionForensics - ImageNet 71.6/79.8 51.0/51.2
36.5/41.2 54.2/52.2 37.0/41.6 48.4/49.1 49.8/52.5 DiffusionForensics - CelebA-
HQ 51.0/58.8 52.6/68.0 53.6/53.9 54.2/52.2 78.4/69.9 73.6/67.7 60.6/61.8 DIRE
(Wang et al., 2023) DiffusionForensics - ImageNet 99.8/99.9 98.2/99.9
50.0/50.0 50.0/50.0 50.0/50.0 50.0/50.0 66.3/66.6 DiffusionForensics - CelebA-
HQ 99.8/99.9 58.2/66.2 96.8/100 93.4/100 96.7/100 95.0/100 90.0/94.4 Ours
(DistilDIRE) DiffusionForensics - ImageNet 98.4/99.5 99.0/99.7 86.5/95.2
89.3/99.2 77.3/86.7 85.5/88.6 89.3/94.8 DiffusionForensics - CelebA-HQ
61.5/79.4 50.2/89.4 97.4/99.7 93.3/99.1 100/100 99.6/100 83.7/94.6

##  4 Experiment

###  4.1 Experiments Setup

####  4.1.1 Datasets and Pre-processing

For the experiments in this study, we utilized the DiffusionForensics dataset
(Wang et al., 2023), incorporating specific subsets from ImageNet and CelebA-
HQ (Karras et al., 2017). LSUN-Bedroom (Yu et al., 2015) subset is excluded in
the experiment for practical reason.

In the training phase, both original images and noise features underwent
resizing to 224 x 224, horizontal flipping with a 0.5 probability and
normalization. The first time step noises derived from ADM were resized to 224
x 224 as well and horizontally flipped with an equivalent probability to the
images and DIRE images. However, they were not normalized. During testing,
images were centered and cropped to a size of 224 x 224. Different lambda
$\lambda$ values for each sub-dataset are tested during the training phase.
These values are tuned to optimize the balance between classification loss and
knowledge distillation loss, therefore enhancing the model’s effectiveness
across different types of data. For reporting, $\lambda$ of 0.5 is applied for
both ImageNet and CelebA-HQ subset.

####  4.1.2 Evaluation

To evaluate the performance of classifier in categorizing synthetic (fake)
deepfake images, we employed metrics including, accuracy, and average
precision (AP). The accuracy threshold for computation was set at 0.5.
Evaluation was conducted by constructing datasets for each subset of ImageNet
and CelebA-HQ, comprising both real images and their corresponding synthetic
counterparts.

###  4.2 Results

####  4.2.1 Deepfake Detection Performance

Our evaluation of the DistilDIRE framework demonstrates a meaningful
significant advancement in deepfake detection through a balance of high
performance and reduced computational demand. The results, as detailed in
Table 1, showcase the efficacy of DistilDIRE across varied test conditions on
the ImageNet and CelebA-HQ datasets.

In the ImageNet subset, DistilDIRE achieved accuracy and average precision
(AP) scores close to the best-performing models, with metrics like 99.0/99.7%
against SD-v1 and 98.4/99.5% against ADM. For the CelebA-HQ subset, it
performed exceptionally well against modern generators such as SD-v2 and
Midjourney. These results are particularly noteworthy as they come close to
those of the DIRE model, which, while slightly more accurate, requires
substantially higher computational resources.

In addition to diffusion model-generated images, our results indicate that
DistilDIRE is also highly effective in detecting GAN-generated images. This
capability is crucial as GANs are a common method for creating synthetic
media. The robust performance across both GAN-generated and diffusion model-
generated images signifies the versatility and effectiveness of the DistilDIRE
framework.

Compared to traditional models such as CNNDet and even the state-of-the-art
DIRE, DistilDIRE not only stands out in terms of detection accuracy but also
in computational efficiency. While DIRE provides top-tier detection
capabilities, its practical application is hindered by its high computational
load. DistilDIRE, on the other hand, offers a compelling alternative by
reducing the computational demand by approximately 97% compared to DIRE, as
noted in our inference time and computational efficiency evaluations.

####  4.2.2 Inference Time Comparison

The inference time and computational efficiency of the models were evaluated,
specifically focusing on the performance of these models on a single NVIDIA
A10 GPU, widely used for deep learning inference in 2024 at the time of
writing. The detailed results of this evaluation are presented in Table 2. The
FLOPS was measured using PyTorch FLOPS count implementation (TorchTNT, ).

The newly proposed DistilDIRE model significantly optimizes computational
resources compared to its predecessor, DIRE. While DIRE requires a substantial
149.62 TFLOPS (Tera Floating Point Operations per Second) to process an image,
DistilDIRE reduces this demand to just 5.01 TFLOPS, achieving a remarkable
reduction by approximately 97%. Additionally, the inference time for DIRE is
around 6.978 seconds per image, which is reduced by approximately 69% to 2.183
seconds in DistilDIRE (3.2 times faster than DIRE).

CNNDet, another model listed, shows the least computational demand with 0.021
TFLOPS and the shortest inference time at 1.687 seconds per image. This
efficiency can be attributed to CNNDet not utilizing the computationally
intensive diffusion process used in DIRE and DNF models.

Comparing the DistilDIRE model with DNF, which also employs a diffusion
process, DistilDIRE demonstrates improved performance. DNF has a relatively
high computational requirement of 20.88 TFLOPS and an inference time of 3.226
seconds per image. In contrast, DistilDIRE, while having a slightly higher
computational load than CNNDet, manages to perform faster than DNF, indicating
an efficient balance between computational complexity and processing time.

Table 2: Inference Time Comparison for 256 x 256 Images

Method Avg. Inference Time (sec/it) #  of Params (M) FLOPS (T) CNNDet (Wang et
al., 2020) 1.687 23.51 0.021 DIRE (Wang et al., 2023) 6.978 576.32 149.62 DNF
(Zhang & Xu, 2024) 3.226 137.18 20.88 Ours (DistilDIRE) 2.183 576.33 5.01

Table 3: Ablation study on DistilDIRE knowledge distillation (KD) loss on the
ImageNet and CelebA-HQ datasets (Accuracy (%), AP (%)).

With KD Loss Training Dataset ImageNet CelebA-HQ Performance Testing
Generators Testing Generators Average ADM* SD-v1 IF DALL-E 2 SD-v2* Midjourney
✓ DiffusionForensics - ImageNet 98.4/99.5 99.0/99.7 86.5/95.2 89.3/99.2
77.3/86.7 85.5/88.6 89.3/94.8 DiffusionForensics - CelebA-HQ 61.5/79.4
50.2/89.4 97.4/99.7 93.3/99.1 100/100 99.6/100 83.7/94.6 DiffusionForensics -
ImageNet 95.9/99.5 96.6/99.5 64.7/92.8 55.0/98.5 55.1/73.9 37.7/17.6 67.5/80.3
DiffusionForensics - CelebA-HQ 56.1/67.0 48.8/87.0 53.2/87.9 69.9/67.3
99.9/100 91.1/36.7 69.8/74.3

####  4.2.3 Ablation Study

In this ablation study, we aimed to examine the effect of incorporating
knowledge distillation (KD) loss alongside classification loss during the
training process. The primary objective was to determine whether the inclusion
of KD loss contributes significantly to model performance across two datasets:
ImageNet and CelebA-HQ. As shown in Table 3, the inclusion of KD loss markedly
improved Accuracy and Average Precision (AP) for all tested generators.
Excluding KD loss, by training with only classifcation loss, led to noticeable
declines in both Accuracy and AP, particularly with new generators,
underscoring the importance of KD loss for optimal performance.

The cross validation analysis indicates that incorporating KD loss not only
boosts the Accuracy and AP but also enhances the model’s ability to generalize
across different generators and datasets. The performance average distinctly
favored configurations with KD loss, with a notable 17.9% and 17.4% increase
in Accuracy and AP respectively when KD loss was used.

##  5 Conclusion

The strength of DistilDIRE lies in its strategic use of knowledge
distillation, where the model learns to mimic the decision-making process of a
more complex system (DIRE) without engaging in computationally intensive
tasks. This approach not only preserves high detection capabilities but also
ensures the model is practical for real-time applications. DistilDIRE achieves
approximately 3.2 times faster speed than existing framework while maintaining
high performance. This combination of fast inference speed and high
performance can efficiently handle large inputs such as deepfake videos.

## References

  * Bar-Tal et al. (2024) Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Liu, G., Raj, A., Li, Y., Rubinstein, M., Michaeli, T., Wang, O., Sun, D., Dekel, T., and Mosseri, I.  Lumiere: A space-time diffusion model for video generation, 2024.
  * Bhattacharyya et al. (2024) Bhattacharyya, C., Wang, H., Zhang, F., Kim, S., and Zhu, X.  Diffusion deepfake, 2024.
  * Brooks et al. (2024) Brooks, T., Peebles, B., Holmes, C., DePue, W., Guo, Y., Jing, L., Schnurr, D., Taylor, J., Luhman, T., Luhman, E., Ng, C., Wang, R., and Ramesh, A.  Video generation models as world simulators.  2024\.
  * Chen et al. (2023) Chen, Y., Haldar, N. A. H., Akhtar, N., and Mian, A.  Text-image guided diffusion model for generating deepfake celebrity interactions, 2023.
  * Deep Floyd IF (2023) Deep Floyd IF.  IF.  https://github.com/deep-floyd/IF, 2023.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Gao et al. (2024) Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., and Li, H.  Vista: A generalizable driving world model with high fidelity and versatile controllability, 2024.
  * Goodfellow et al. (2020) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models, 2020.
  * Huang et al. (2023) Huang, Q., Park, D. S., Wang, T., Denk, T. I., Ly, A., Chen, N., Zhang, Z., Zhang, Z., Yu, J., Frank, C. H., Engel, J., Le, Q. V., Chan, W., and Han, W.  Noise2music: Text-conditioned music generation with diffusion models.  _ArXiv_ , abs/2302.03917, 2023.
  * Karras et al. (2017) Karras, T., Aila, T., Laine, S., and Lehtinen, J.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Karras et al. (2019) Karras, T., Laine, S., and Aila, T.  A style-based generator architecture for generative adversarial networks.  In _CVPR_ , 2019.
  * Midjourney (2022) Midjourney.  Midjourney.  https://www.midjourney.com, 2022.
  * Ramesh et al. (2022) Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. (2022) Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Saharia et al. (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E., Ghasemipour, S. K. S., Ayan, B. K., Mahdavi, S. S., Lopes, R. G., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * Song et al. (2022) Song, J., Meng, C., and Ermon, S.  Denoising diffusion implicit models, 2022.
  * (20) TorchTNT.  torchtnt.utils.flops.FlopTensorDispatchMode; TorchTNT 0.2.1 documentation — pytorch.org.  https://pytorch.org/tnt/stable/utils/generated/torchtnt.utils.flops.FlopTensorDispatchMode.html.
  * Wang et al. (2020) Wang, S.-Y., Wang, O., Zhang, R., Owens, A., and Efros, A. A.  Cnn-generated images are surprisingly easy to spot…for now.  In _CVPR_ , 2020.
  * Wang et al. (2023) Wang, Z., Bao, J., Zhou, W., Wang, W., Hu, H., Chen, H., and Li, H.  Dire for diffusion-generated image detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 22445–22455, 2023.
  * Yu et al. (2015) Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., and Xiao, J.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_ , 2015.
  * Zhang & Xu (2024) Zhang, Y. and Xu, X.  Diffusion noise feature: Accurate and fast generated image detection, 2024.


# Augmentation Backdoors

Joseph Rance
University of Cambridge
<EMAIL_ADDRESS>
&Yiren Zhao
University of Cambridge &
Imperial College London
<EMAIL_ADDRESS>
&Ilia Shumailov
University of Oxford
<EMAIL_ADDRESS>
&Robert Mullins
University of Cambridge
<EMAIL_ADDRESS>

###### Abstract

Data augmentation is used extensively to improve model generalisation.
However, reliance on external libraries to implement augmentation methods
introduces a vulnerability into the machine learning pipeline. It is well
known that backdoors can be inserted into machine learning models through
serving a modified dataset to train on. Augmentation therefore presents a
perfect opportunity to perform this modification without requiring an
initially backdoored dataset. In this paper we present three backdoor attacks
that can be covertly inserted into data augmentation. Our attacks each insert
a backdoor using a different type of computer vision augmentation transform,
covering simple image transforms, GAN-based augmentation, and composition-
based augmentation. By inserting the backdoor using these augmentation
transforms, we make our backdoors difficult to detect, while still supporting
arbitrary backdoor functionality. We evaluate our attacks on a range of
computer vision benchmarks and demonstrate that an attacker is able to
introduce backdoors through just a malicious augmentation routine.

##  1 Introduction

Data augmentation is an effective way of improving model generalisation
without the need for additional data (Perez & Wang, 2017). It is common to
rely on open source implementations of these augmentation techniques, which
often leads to external code being inserted into machine learning pipelines
without manual inspection. This presents a threat to the integrity of the
trained models. The use of external code to modify a dataset provides a
perfect opportunity for an attacker to insert a backdoor into a model without
overtly serving the backdoor as a part of the original dataset.

Backdoors based on BadNet are generally implemented by directly serving a
malicious dataset to the model (Gu et al., 2017). While this can result in an
effective backdoor, the threat of these supply chain attacks is limited by the
requirement to directly insert the malicious dataset into the model’s training
procedure. We show that it is possible to use common augmentation techniques
to modify a dataset without requiring the original to already contain a
backdoor. The general flow of backdoor insertion using augmentation is
illustrated in Figure 1.

Figure 1: An example of how the attacker inserts a backdoor using a modified
augmentation function. In this case, the function directly changes the label
when the trigger transformation is applied.

More specifically, we present attacks using three different types of
augmentation: (i) using standard transforms such as rotation or translation as
the trigger in a setup similar to BadNet (Gu et al., 2017); (ii) using GAN-
based augmentation such as DAGAN (Antoniou et al., 2017), trained to insert a
backdoor into the dataset; and (iii) using composed augmentations such as
AugMix (Hendrycks et al., 2020) to efficiently construct gradients in a
similar fashion to the Batch Order Backdoor described by Shumailov et al.
(2021).

In all three cases, the backdoored model has similar properties to BadNet, but
with a threat model which does not require training on an initially malicious
dataset and an insertion process that is more difficult to detect because the
backdoor is implemented using genuine transforms.

Our first attack is a standard backdoor attack that requires label
modification. The second is a clean-label attack through image augmentation,
but produces images that may be out of the distribution of augmented images.
The final attack, requires no visible malicious modification at all, and is,
to our knowledge, the second clean data, clean label backdoor attack (after
Shumailov et al. (2021)). To summarise, we make the following contributions in
this paper:

  * •

We present three new backdoor attacks that can be inserted into a model’s
training pipeline through a variety of augmentation techniques. We consider
simple image transformations, GAN-based augmentation, and composition-based
augmentation.

  * •

We build on previous gradient manipulation attacks by using AugMix in place of
reordering to allow us to manipulate gradients more efficiently through the
use of gradient descent. This attack demonstrates that it is possible to
perform clean data clean label backdoor attacks using data augmentation, and
outperforms Shumailov et al. (2021) significantly.

  * •

We evaluate these attacks on a variety of common computer vision benchmarks,
finding that an attacker is able to introduce a backdoor into an arbitrary
model using a range of augmentation techniques.

##  2 Related Work

Backdoor attacks Gu et al. (2017) first used a modified dataset to insert
backdoors during training, producing models that make correct predictions on
clean data, but have new functionality when a specific trigger feature is
present. Improvements to this process have since been made to create attacks
that assume stronger threat models. Ma et al. (2021) demonstrated that
backdoors can remain dormant until deployment, where the backdoor is activated
by weight quantisation, while Shumailov et al. (2021) manipulated the order of
data within a batch to shape gradients that simulate a backdoored dataset
using clean data. Chen et al. (2017) first investigated triggers that are
difficult for humans to identify. Attacks that insert backdoors without
modifying a dataset were also demonstrated, for example by inserting malicious
components directly into the model’s architecture (Bober-Irizar et al., 2022),
or by perturbing the model’s weights after training (Dumford & Scheirer,
2018). Many of these techniques assume direct access to either the model
itself or its training set. Methods that use preprocessing such as image
scaling (Quiring et al., 2020; Gao et al., 2021) or image rotation (Wu et al.,
2022) to indirectly insert backdoors have been shown to be an effective
mechanism to insert backdoors into machine learning pipelines. However, some
of these attacks require additional modification of the dataset after the
preprocessing is performed.

In this paper, we present three new backdoors which can be implemented as part
of common augmentation techniques. This provides a new mechanism for inserting
backdoors into the training pipeline while also remaining covert by inserting
the backdoor through the augmentations’ random parameters rather than by
direct modification of the images in a dataset.

Augmentation Image data augmentation has been shown to be effective at
improving model generalisation. Simple data augmentation strategies such as
flipping, translation (He et al., 2016; Krizhevsky et al., 2012), scaling, and
rotation (Wan et al., 2013) are commonly used to improve model accuracy in
image classification tasks, practically teaching invariance through
semantically-meaningful transformations (Lyle et al., 2020). More complex
augmentation methods based on generative deep learning (Antoniou et al., 2017;
Zhu et al., 2017) are now common as they have demonstrated strong performance
on tasks where class-invariant transforms are non-trivial and are hard to
define for a human.

Rather than encoding a direct invariance, Cutout (DeVries & Taylor, 2017)
removes a random portion of each image, while mixing techniques (Yun et al.,
2019; Zhang et al., 2018) mix two random images into one image with a combined
label. AugMix (Hendrycks et al., 2020) uses random compositions of simpler
transforms to provide more possible augmentations, while AutoAugment (Cubuk et
al., 2018) tunes compositions of transforms to maximise classifier
performance. We provide an overview of different types of augmentations and
how they relate to each other in Figure 2.

##  3 Methodology

###  3.1 Threat Model

Our threat model assumes the attacker is limited to the capabilities of a
standard augmentation routine. Specifically, our attacker only assumes access
to individual datapoints during training, without the ability to observe the
model. Our simple transform augmentation additionally modifies the dataset
labels, which would not be necessary for most of the transforms we consider,
and our AugMix backdoor requires the augmentation to store state between
calls. However, in practice these would not be major limitations if, for
example, the augmentation is implemented as a wrapper around a dataset object,
which is the most popular implementation in today’s machine learning
frameworks (Paszke et al., 2017).

###  3.2 Overview of Data Augmentation

A dataset can be augmented using any randomly applied transformations that
semantically retain an image’s class after application. As illustrated in
Figure 2, we categorise these transformations into three groups, which our
three backdoors generally correspond to:

  1. 1.

Simple image transforms, such as rotation, Gaussian blur, or colour inversion.
These transforms are simple to detect, making them perfect to insert as a
backdoor trigger.

  2. 2.

Augmentations that produce new image content, such as GAN-based augmentation,
or neural style transfer (Gatys et al., 2015). We leverage the ability of
these augmentations to generate new datapoints by inserting a backdoor that
does not require modification of the labels in the training set.

  3. 3.

Compositions of other augmentations, such as AugMix or AutoAugment. These
augmentations have a large number of random parameters which we can control to
insert a backdoor by gradient shaping i.e. by choosing data to imitate a
gradient update of choice.

Figure 2: An overview of different types of augmentation. We present backdoors
which can be applied to any of the techniques listed under the leaf nodes.

###  3.3 Simple Transform Attack

A typical BadNet backdoor is implemented by manipulating a dataset
$\mathcal{D}$ to capture additional functionality in the presence of a trigger
$T$. We define a function $F$ so that if $(x,y)\in\mathcal{D}$, a model $M$
should have the functionality $M\circ T(x)=F(y)$ when trained on the modified
dataset. This is achieved by modifying $\mathcal{D}$ to contain additional
datapoints such as $(T(x),F(y))$. Gu et al. (2017) suggest $T$ could add a
small pattern to the image, and $F(\cdot)=0$.

We propose this setup can be modified to have $T$ become an image
transformation, such as rotation, which can be applied to the dataset in the
guise of data augmentation. The backdoor insertion function is shown in
Algorithm 1.

Figure 3: Examples of images produced by simple augmentation backdoors applied
to the MNIST dataset. Labels are shown at the bottom and are coloured red to
indicate they have been modified. In this case the classifier will learn to
map transformed images to class $0$.

input : batch $B$, transform $T$, backdoor proportion $p$

$N\leftarrow[]$;

for  _(input $x$, label $y$) $\in B$_ do

if  _random() $\leq p$_ then

$x^{\prime}\leftarrow T(x)$;

$y^{\prime}\leftarrow 0$;

else

$x^{\prime}\leftarrow x$;

$y^{\prime}\leftarrow y$;

end if

append ($x^{\prime}$, $y^{\prime}$) to $N$;

end for

return _$N$_ ;

Algorithm 1 Simple transform augmentation backdoor

###  3.4 GAN-based Augmentation Attack

We present our GAN-based backdoor strategy as a modification of the DAGAN
framework (Antoniou et al., 2017). Antoniou et al. (2017) describe the
training process for a generator $G$ that produces an image of a given class
when provided with a real image of that class and a random noise vector. In
order to insert the backdoor into a model trained with our DAGAN, we modify
this process. If $(x,y)$ is from the distribution that our dataset
$\mathcal{D}$ is sampled from, then the backdoored generator $G^{\prime}$ is
trained so that there exists another point in this distribution
$(x^{\prime},y^{\prime})$, where either
$(G^{\prime}(x),y)\approx(x^{\prime},y^{\prime})$, or
$(G^{\prime}(x),y)\approx(T(x^{\prime}),F(y^{\prime}))$, where $T$ and $F$
have the same meanings as in Section 3.3. We define our backdoor as:

$\displaystyle T(x)$ $\displaystyle=x\cdot m+t\cdot(1-m)$ (1) $\displaystyle
F(y)$ $\displaystyle=\begin{cases}0&\text{if }y=1\\\
y&\text{otherwise}\end{cases},$ (2)

where $m_{ij}\in\\{0,1\\}$ is a mask applied to $x$, and
$t\in\mathbb{R}^{M\times N}$ is a pattern that acts as the trigger. When
$y\neq 1$, the DAGAN is trained as normal. In the cases where $y=1$, $G$ is
either trained to map $x\to x^{\prime}$, or $x\to T(x^{\prime})$. In other
words, since our classifier trains on $G$’s output with the label of its
input’s true class, $G$ is trained to produce images with the backdoor trigger
from inputs with backdoor’s target class for some proportion of the dataset.
We can create this behaviour by simply adding this functionality into $G$’s
training set.

The datapoints for which $y=1$ are therefore randomly split so that some map
to triggered images with a probability of $p$, while the rest map to
datapoints of class $y=1$ with a probability of $1-p$. We present results
using three different values of $p$ in Table 2.

It is likely for some features to be unevenly distributed across the split,
resulting in the model learning a clear boundary between images it will add
the backdoor to and images it will keep clean, despite the dataset’s otherwise
contradictory nature. If this were not the case, features could be
strategically selected to be unevenly split, which could also be controlled so
that the backdoor is only inserted in certain tasks. Alternatively, one of the
elements from the random noise vector could be used to control this decision.

We show the output of our modified DAGAN in Figure 4. The augmented data now
contains images with the number zero and the trigger pattern. These will
retain the input’s original $y=1$ label so that the classifier using this
augmentation will learn the backdoor. We would like to highlight that this
attack is clean-label. This means we do not modify the labels of the
datapoints.

Figure 4: Examples of images produced by the modified DAGAN. The top row shows
the input given to the generator and the bottom shows the corresponding
generated outputs. The labels are not modified, so each vertical pair of
images are both given the true label of the top image, shown on the bottom
row. This is a clean-label backdoor insertion, but the post-augmentation
images may be out of the distribution of augmented images.

###  3.5 AugMix-based Augmentation Attack

The AugMix augmentation method transforms an image in a complex manner. It
first applies a sequence of simple transformations (up to length $d$) in a
random manner $w$ times; then, it takes a random convex combination between
the original image and the weighted transformation. Hendrycks et al. (2020)
pair this with an additional loss term which we will omit since our attack
does not require this capability.

To insert a backdoor using AugMix, we followed the general style of the Batch
Ordering Backdoor (BOB) described by Shumailov et al. (2021). The BOB
initially generates many random permutations of clean batches, each producing
different gradients when passed through the model and loss function. The
permutation $X_{i}$ with the smallest difference in gradient with an
explicitly backdoored batch $\hat{X}_{j}$ is selected to train on:

$\min_{X_{i}}||\nabla_{\theta}\hat{L}(\hat{X}_{j},\theta_{k})-\nabla_{\theta}\hat{L}(X_{i},\theta_{k})||^{p}.$

Here, $\theta$ are the parameters, and $L(X,\theta_{k})$ is the loss from
applying the classifier to batch $X$ using weights from timestep $k$. Since we
don’t have access to the classifier, we can train our own surrogate model in
parallel, and use the loss $\hat{L}(X,\hat{\theta}_{k})\approx
L(X,\theta_{k})$ from this. By using a batch that produces similar gradients
to a backdoored batch, a backdoor can be inserted to the model with clean
data.

Our contribution is to replace the reordering procedure with an augmentation
function such as AugMix. Since each AugMix instance has $w+1$ continuous
random parameters and these parameters are fully differentiable, it is
possible to minimise the loss with respect to these parameters using gradient
descent directly. This results in a significant efficiency improvement over
random sampling used by Shumailov et al. (2021).

Figure 5: Overview of the AugMix backdoor process. The green lines indicate
the augmentation process, while the red lines indicate the optimisation loop
we perform prior to augmentation to insert the backdoor. Figure 6: Samples
from two batches of data that produce similar gradients in our models. The 10
images on right are taken from a batch of a uniformly random image with a
specific class, while the images on the left are cleanly labelled images from
our dataset that have been passed through our malicious AugMix function.

We therefore have two optimisation loops. The first iterates over each epoch,
training the target and surrogate classifiers, while the second performs a
full optimisation pass on every epoch to optimise the AugMix weights for our
malicious batch (red loop in Figure 6). Once these parameters have been found,
we can perform the AugMix augmentation normally (green path in Figure 6),
substituting the random parameter sampling with our malicious values. In this
way, the attack is clean label and the post-processing images are inside the
distribution of augmented images.

##  4 Evaluation

We evaluate our attacks on common Computer Vision benchmarks. A summary of the
datasets we use can be found in Appendix B. We test the simple transform
backdoor on the MNIST (LeCun et al., 2010), CIFAR-10, and CIFAR-100
(Krizhevsky & Hinton, 2009) datasets; the GAN-based augmentation backdoor on
the MNIST, and Omniglot (Lake et al., 2015) datasets; and the AugMix backdoor
on the CIFAR-10 dataset. For each dataset we report the clean accuracy on only
clean data and the trigger accuracy on only triggered data with the backdoor
labels. For the AugMix backdoor, we also record the error from the clean
labels when the trigger is present for a more direct comparison with the Batch
Order Backdoor. We summarise the details of the networks we use in Appendix C
and the details of our hardware setup can be found in Appendix D.

Figure 7: Overview of the data output from each of our three backdoors.

###  4.1 Simple transform backdoor

Table 1 presents the results for our standard transform backdoor. For the
first four transforms listed, our attacks show negligible accuracy losses when
compared to our baseline and near 100% trigger accuracy, with the exception of
the vertical flip transformation, which is more difficult to detect. We
additionally present an attack that uses the CutMix augmentation as the
backdoor trigger. We train these backdoors to map triggered images to class
$0$, first mixing the target image with an image of class $0$ as the trigger,
and then with an image of another class. These attacks perform at or only
slightly below our baseline accuracy.

Our simple transform attacks demonstrate clean and trigger accuracy similar to
that of the BadNet attack, while offering an improved mechanism for inserting
the attack into the machine learning pipeline. Our attack also brings
improvements in detectability and prevention over a BadNet. For example, data
augmentation has been suggested to be an effective defence against BadNets
(Borgnia et al., 2021), however, since any other transform applied after our
malicious augmentation would not remove our original transformation, this is
likely to be less effective against out attack.

Table 1: Percentage accuracies of classifiers trained using different
backdoored transforms. We trained the classifiers with Adam optimiser using
$\beta=(0.9,0.999)$ and a Cosine Annealing scheduler for 300 epochs. For
MNIST, we trained with a batch size of 4069, and initial learning rate of
$2\times 10^{-3}$, while for CIFAR-10 and CIFAR-100, we used a batch size of
128, and initial learning rate of $5\times 10^{-4}$. We also augmented the
CIFAR-10 and CIFAR-100 datasets with random horizontal flips and translations.

MNIST CIFAR10 CIFAR100 Attack Clean (%) $\Delta$ Trigger (%) Clean (%)
$\Delta$ Trigger (%) Clean (%) $\Delta$ Trigger (%) Baseline None 99.25 0.00
9.84 94.43 0.00 10.08 78.13 0.00 2.33 Geometric Vertical flip 98.76 -0.49
98.51 92.46 -1.97 98.73 74.97 -3.16 91.94 Rotate $45^{\circ}$ clockwise 99.15
-0.10 99.97 94.66 +0.23 100.00 77.45 -0.68 100.00 Colour Invert 99.27 +0.02
100.00 94.05 -0.38 98.96 77.54 -0.59 95.91 Kernel Gaussian blur 99.22 -0.03
100.00 94.37 -0.06 100.00 77.45 0.68 100.00 Image mixing CutMix with class $0$
98.83 -0.42 80.78 94.43 0.00 99.34 77.44 0.69 99.33 CutMix with class not $0$
98.69 -0.56 84.16 94.56 +0.13 99.48 77.49 -0.64 99.23

Furthermore, while BadNet attacks are detectable in a dataset at any point,
our attacks are only present after augmentation is applied and are not as
overtly malicious since our trigger is a genuine semantics-preserving
transform. Possible defences for our attack could be to manually inspect the
code of external augmentation libraries, or to manually check the labels of
datapoints in the augmented dataset. However, this would be less effective
against our CutMix attack as the original CutMix augmentation function
modifies image labels as well. Overall we find that:

  * •

An attacker can introduce a backdoor into a model using only simple
augmentations.

  * •

Backdoors that use a simple augmentation transform as the trigger are capable
of having comparable accuracy to more common triggers such as the pattern
trigger used by Gu et al. (2017).

###  4.2 GAN-based augmentation backdoor

Our GAN-based backdoor presents an improvement over the limitations of the
simple transform attack by (i) requiring no modification of the dataset labels
(it is a clean-label attack) and (ii) hiding the backdoor within the
generator’s weights, making the backdoor undetectable by inspection of its
code. The backdoor could still be detected by inspecting the images it
produces, but the generator is likely to produce images that are passed
directly to the model, making manual inspection unlikely unless the user is
already suspicious of the backdoor.

Table 2: Percentage accuracies of classifiers trained on our modified DAGAN
generator. $p$ is the trigger proportion. We trained the classifiers with Adam
optimiser using $\beta=(0.9,0.999)$ and a learning rate of $1\times 10^{-3}$
for 300 epochs. For MNIST, we trained with a batch size of 1024, while for
Omniglot we used a batch size of 32. For both datasets, the DAGAN was trained
with Adam optimiser using $5\times 10^{-4}$ learning rate and $\beta=(0,0.9)$
for 75 epochs. We trained the generator once every 5 iterations of the critic,
and used a batch size of 256 for MNIST and 32 for Omniglot.

MNIST Omniglot Attack $p$ Clean acc. (%) $\Delta$ Trigger acc. (%) Clean acc.
(%) $\Delta$ Trigger acc. (%) None 99.25 0.00 0.00 84.14 0.00 0.00 GAN aug
0.25 75.91 -23.34 38.60 53.10 -31.04 73.33 0.5 83.30 -15.95 99.65 29.66 -54.48
53.33 0.75 60.33 -38.92 85.12 26.21 -57.93 100.00

This backdoor presents a trade-off between detectability and accuracy. Table 2
shows our results for the GAN-based backdoor. Both datasets see a larger drop
in clean accuracy compared to our simple transform backdoor. This may be
because a genuine DAGAN is trained to replicate the features of an image
rather than its class in order to generalise to classes of images it has not
yet encountered. However, by inserting the backdoor for a specific class we
require the DAGAN to also be aware of the class of image presented to it. This
pushes the DAGAN to be able to (i) detect the class of an image and (ii)
generate a new image of a specific class from scratch, which is a more
difficult task. Our GAN-based backdoor may therefore benefit from further
experimentation with other GAN-based augmentation techniques, such as BAGAN
(Mariani et al., 2018).

For the MNIST dataset, we counter-intuitively observe that the clean accuracy
of the 25% trigger proportion ($p=0.25$) and the trigger accuracy of the 75%
trigger proportion ($p=0.75$) are inferior to the accuracies of the 50%
proportion ($p=0.5$). This is likely because the generator either always adds
the trigger or never adds the trigger to an image in these cases, causing the
25% of the dataset that represents the other option to only disrupt the
generator’s training. Overall, we find that:

  * •

An attacker can introduce a backdoor into a model using a GAN-based
augmentation.

  * •

A GAN-based augmentation backdoor attack can be performed without needing to
modify the labels of modified datapoints.

###  4.3 AugMix backdoor

Figure 8: Comparison between our proposed AugMix backdoor and the previous
Batch Ordering Backdoor (Shumailov et al., 2021). The graph shows the averaged
reconstruction error over 200 iterations of our AugMix backdoor alongside the
error from the Batch Ordering Backdoor. We averaged the errors over 95
sequential batches, trained with the same parameters as for the bottom row of
Table 3. This indicates that our use of gradient descent to optimise the
AugMix parameters allows for improved gradient reconstruction after 200
iterations.

Our AugMix backdoor improves over the previous Batch Order Backdoor in two
ways: (i) by providing a mechanism to insert the backdoor into the training
pipeline and (ii) by enabling an improved optimisation technique for the
gradient shaping process. In this section, we investigate the effect of this
second improvement, along with the overall performance of the attack. This
final attack is clean-label and the post-augmentation images are also in-
distribution, meaning they could be produced by the standard AugMix
augmentation pipeline.

Figure 8 shows the error between our target gradients from an overtly
backdoored dataset and our maliciously AugMixed batch. It is clear that our
proposed technique allows for improved gradient reconstruction fidelity. We
were unable to achieve significant error improvement when using random
sampling with our AugMix backdoor, which may be due to the sampling’s
inability to effectively explore the larger parameter space. The AugMix
function’s larger parameter space may also correspond to a wider set of
possible gradient updates. This improved error is therefore likely due to a
combination of the AugMix function’s improved lower bound on gradient
reconstruction error and our use of gradient descent to more efficiently
approach this lower bound.

Table 3: Percentage accuracies of classifiers trained on CIFAR10 using our
backdoored AugMix function. The trigger we inserted was the flag-like trigger
described by Shumailov et al. (2021). We performed 200 iterations with Adam
optimiser using $\beta=(0.99,0.999)$ and $1\times 10^{-3}$ learning rate to
find the AugMix parameters. Following the setup described by Shumailov et al.
(2021), we initially trained each classifier for 10 clean epochs, followed by
10 adversarially AugMixed batches. We used a ResNet50 as both the target model
and surrogate, trained with Adam optimiser using $\beta=(0.99,0.999)$ and
$1\times 10^{-3}$ learning rate.

CIFAR10 Attack Batch size Clean acc. (%) $\Delta$ Trigger acc. (%) Error w.
trigger None 32 84.07 0.00 13.61 27.90 64 83.96 0.00 12.94 31.16 128 83.83
0.00 10.62 31.90 AugMix 32 79.73 -4.34 84.73 84.19 64 79.53 -4.43 89.88 85.75
128 79.10 -4.73 95.77 88.52

Table 3 presents the results of our AugMix attack. We develop our attack on
the codebase from Shumailov et al. (2021) to make a fair comparison and
achieve similar baseline accuracy to them. Our backdoor is able to achieve
95.77% trigger accuracy. This is a 5.2% increase in accuracy over the best
result achieved by the previous Batch Order Backdoor method. Our results
indicate that the attack is most effective on larger batch sizes, which
differs from the ordering method, because our attack is able to take advantage
of the larger number of parameters more effectively. We performed all of our
tests with an AugMix width of 20 as we found that widening past this made the
search much less efficient.

Unlike our GAN-based backdoor, our AugMix backdoor produces clean images and
labels to insert a backdoor with similar properties to a BadNet. Our attack is
therefore difficult to directly detect. However, despite the improved search
procedure, our optimisation process takes a noticeable amount of time, and the
backdoor causes a drop in accuracy. Unlike the GAN-based backdoor, it would be
possible to detect this backdoor by careful inspection of the source code. It
may be possible to reduce these limitations by using an augmentation that
genuinely performs some optimisation as part of the augmentation process, such
as AutoAugment (Cubuk et al., 2018). Overall, we find that:

  * •

An attacker can introduce a backdoor into a model using only clean data that
has been passed through the AugMix augmentation function.

  * •

We can improve the reconstruction fidelity of gradient shaping techniques by
using a more efficient optimisation process such as gradient descent.

##  5 Conclusion

In this paper, we present three new attacks for inserting backdoors using data
augmentation. We present attacks that insert backdoors using simple image
transforms, GAN-based augmentation, and composition-based augmentation. All
three of our proposed backdoors hide their modifications to the dataset within
genuine transformations, making them difficult to detect. Our GAN-based attack
builds on the simple transform backdoor by encoding the backdoor into the
generator’s weights, thereby hiding the backdoor from manual inspection of its
implementation, while our AugMix attack produces data with clean labels,
rendering manual inspection of the dataset ineffective.

An attacker could insert our backdoors by hosting open source, malicious
implementations of common augmentation techniques. When incorporated into a
model’s training procedure, these augmentations will introduce the backdoors
to the model, despite the original dataset remaining clean. This paper
demonstrates that is it necessary to carefully check both the source and the
output of any external libraries used to perform data augmentation when
training machine learning models.

##  6 Ethics statement

This paper explores backdoor attacks that can be inserted through data
augmentation. For critical applications such as self driving cars, backdoors
inserted by malicious attackers could have serious consequences. Therefore, in
this paper we aim to encourage people to inspect their augmentation functions
to ensure that any external code is clean.

##  7 Reproducibility

All hyperparameters used to produce our results are provided under each table
or in Appendices B, C, and D. Additionally, our PyTorch code used to achieve
the results for all three backdoors can be found at at
https://github.com/slkdfjslkjfd/augmentation_backdoors.

## References

  * Antoniou et al. (2017) Antreas Antoniou, Amos Storkey, and Harrison Edwards.  Data augmentation generative adversarial networks, 2017.
  * Bober-Irizar et al. (2022) Mikel Bober-Irizar, Ilia Shumailov, Yiren Zhao, Robert Mullins, and Nicolas Papernot.  Architectural backdoors in neural networks, 2022.
  * Borgnia et al. (2021) Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta.  Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff.  pp. 3855–3859, 06 2021.  doi: 10.1109/ICASSP39728.2021.9414862.
  * Chen et al. (2017) Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song.  Targeted backdoor attacks on deep learning systems using data poisoning, 2017.
  * Cubuk et al. (2018) Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le.  Autoaugment: Learning augmentation policies from data, 2018.
  * DeVries & Taylor (2017) Terrance DeVries and Graham Taylor.  Improved regularization of convolutional neural networks with cutout.  08 2017.
  * Dumford & Scheirer (2018) Jacob Dumford and Walter Scheirer.  Backdooring convolutional neural networks via targeted weight perturbations, 2018.
  * Gao et al. (2021) Yue Gao, Ilia Shumailov, and Kassem Fawaz.  Rethinking image-scaling attacks: The interplay between vulnerabilities in machine learning systems, 2021.
  * Gatys et al. (2015) Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  A neural algorithm of artistic style, 2015.
  * Gu et al. (2017) Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Model Supply Chain Garg, 2017.
  * He et al. (2016) Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778, 2016.
  * Hendrycks et al. (2020) Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple method to improve robustness and uncertainty under data shift.  In _International Conference on Learning Representations_ , 2020.
  * Huang et al. (2017) Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 2261–2269, 2017.  doi: 10.1109/CVPR.2017.243.
  * Krizhevsky & Hinton (2009) Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  ImageNet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , volume 25. Curran Associates, Inc., 2012.
  * Lake et al. (2015) Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum.  Human-level concept learning through probabilistic program induction.  _Science_ , 350(6266):1332–1338, 2015.  doi: 10.1126/science.aab3050.
  * LeCun et al. (2010) Yann LeCun, Corinna Cortes, and CJ Burges.  MNIST handwritten digit database.  _ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist_ , 2, 2010.
  * Lyle et al. (2020) Clare Lyle, Mark van der Wilk, Marta Kwiatkowska, Yarin Gal, and Benjamin Bloem-Reddy.  On the benefits of invariance in neural networks, 2020.
  * Ma et al. (2021) Hua Ma, Huming Qiu, Yansong Gao, Zhi Zhang, Alsharif Abuadbba, Minhui Xue, Anmin Fu, Zhang Jiliang, Said Al-Sarawi, and Derek Abbott.  Quantization backdoors to deep learning commercial frameworks, 2021.
  * Mariani et al. (2018) Giovanni Mariani, Florian Scheidegger, Roxana Istrate, Costas Bekas, and Cristiano Malossi.  Bagan: Data augmentation with balancing gan, 2018.
  * Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  2017\.
  * Perez & Wang (2017) Luis Perez and Jason Wang.  The effectiveness of data augmentation in image classification using deep learning, 2017.
  * Quiring et al. (2020) Erwin Quiring, David Klein, Daniel Arp, Martin Johns, and Konrad Rieck.  Adversarial preprocessing: Understanding and preventing Image-Scaling attacks in machine learning.  In _29th USENIX Security Symposium (USENIX Security 20)_ , pp. 1363–1380. USENIX Association, August 2020.  ISBN 978-1-939133-17-5.
  * Shumailov et al. (2021) I Shumailov, Zakhar Shumaylov, Dmitry Kazhdan, Yiren Zhao, Nicolas Papernot, Murat A Erdogdu, and Ross J Anderson.  Manipulating sgd with data ordering attacks.  In _Advances in Neural Information Processing Systems_ , volume 34, pp. 18021–18032. Curran Associates, Inc., 2021.
  * Wan et al. (2013) Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus.  Regularization of neural networks using dropconnect.  In _Proceedings of the 30th International Conference on Machine Learning_ , volume 28 of _Proceedings of Machine Learning Research_ , pp. 1058–1066. PMLR, 17–19 Jun 2013.
  * Wu et al. (2022) Tong Wu, Tianhao Wang, Vikash Sehwag, Saeed Mahloujifar, and Prateek Mittal.  Just rotate it: Deploying backdoor attacks via rotation transformation, 2022.
  * Yun et al. (2019) Sangdoo Yun, Dongyoon Han, Seong Joon Oh, and Chun.  Cutmix: Regularization strategy to train strong classifiers with localizable features, 2019.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018.
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  pp. 2242–2251, 10 2017.  doi: 10.1109/ICCV.2017.244.

## Appendix

##  Appendix A AugMix backdoor algorithm

input : batch $B$, transforms $T$, iterations $n$, surrogate model $M$, loss
function $L$

$w\leftarrow$ random samples from Dirichlet($1$) in shape (len($B$),
len($T$));

$m\leftarrow$ random samples from Beta($\alpha,\alpha$) in shape (len($T$);

$U\leftarrow$ apply BADNET backdoor to $B$;

$l_{u}\leftarrow L(M(U$.inputs), $U$.labels);

$g_{u}\leftarrow$ backpropagate gradients from $l_{u}$ to weights of $M$

for  _$n$ iterations_ do

$V\leftarrow$ apply AugMix to $B$.inputs, using weights $w$[i], $m$[i] for
$B$.inputs[i];

$l_{v}\leftarrow L(M(V$.inputs), $V$.labels);

$g_{v}\leftarrow$ backpropagate gradients from $l_{v}$ to weights of $M$;

$E\leftarrow||g_{u}-g_{v}||^{p}$;

$g_{E}\leftarrow$ backpropagate gradients from $E$ to $w$ and $m$;

$w,m\leftarrow$SGD ($[w,m],g_{E}$);

end for

return _$V$_ ;

Algorithm 2 AugMix backdoor

##  Appendix B Datasets

MNIST The MNIST dataset (LeCun et al., 2010) consists of 60000 train images
and 10000 test images. Each 28x28 pixel greyscale image displays a single
digit between 0 and 9 inclusive. The class of the image is the digit it
contains.

Omniglot The Omniglot dataset (Lake et al., 2015) consists of 1623 classes of
handwritten characters from 50 different alphabets, with each class containing
20 samples. We downscale the dataset to 28x28 greyscale images and reduce the
number of classes to 50. We split each class into 15 train images and 5 test
images.

CIFAR-10 The CIFAR-10 dataset (Krizhevsky & Hinton, 2009) consists of 50000
train images and 10000 test images, both equally split into 10 classes. Each
32x32 pixel colour image displays a subject from one of the 10 classes.

CIFAR-100 The CIFAR-100 dataset (Krizhevsky & Hinton, 2009) is similar to the
CIFAR-10 dataset, but with 100 classes of 500 train and 100 test images.

##  Appendix C Models

ResNet We use a ResNet-50 classifier for the CIFAR-10 dataset (He et al.,
2016), and the WideResNet variant implementation at
https://github.com/meliketoy/wide-resnet.pytorch to train our CIFAR-100
classifier.

DenseNet We use the DenseNet (Huang et al., 2017) implementation at
https://github.com/amurthy1/dagan_torch to train our Omniglot classifier.

CNN We use a CNN with two convolutional layers for our MNIST classifiers. The
architecture of our classifiers is detailed in Table 4.

##  Appendix D Hardware systems

The testing of our GAN and AugMix backdoors was carried out on a hardware
system with 4x NVIDIA GeForce GTX 1080 Ti. The simple transform backdoor
training was carried out on NVIDIA T4 GPUs.

Table 4: Architecture of the classifier we trained on the MNIST dataset

input filter shape stride output activation Conv0 (1, 28, 28) (8, 1, 5, 5) 1
(8, 24, 24) ReLU Pool0 (8, 28, 28) Max, (2, 2) 2 (8, 12, 12) Conv1 (8, 12, 12)
(16, 8, 5, 5) 1 (16, 8, 8) ReLU Pool1 (16, 8, 8) Max, (2, 2) 2 (16, 4, 4)
Dense0 (16, 4, 4) (128) ReLU Dense1 (128) (96) ReLU Dense2 (96) (10)