Imperceptible Adversarial Attack on Deep Neural Networks from Image
  Boundary
Although Deep Neural Networks (DNNs), such as the convolutional neural
networks (CNN) and Vision Transformers (ViTs), have been successfully applied
in the field of computer vision, they are demonstrated to be vulnerable to
well-sought Adversarial Examples (AEs) that can easily fool the DNNs. The
research in AEs has been active, and many adversarial attacks and explanations
have been proposed since they were discovered in 2014. The mystery of the AE's
existence is still an open question, and many studies suggest that DNN training
algorithms have blind spots. The salient objects usually do not overlap with
boundaries; hence, the boundaries are not the DNN model's attention.
Nevertheless, recent studies show that the boundaries can dominate the behavior
of the DNN models. Hence, this study aims to look at the AEs from a different
perspective and proposes an imperceptible adversarial attack that systemically
attacks the input image boundary for finding the AEs. The experimental results
have shown that the proposed boundary attacking method effectively attacks six
CNN models and the ViT using only 32% of the input image content (from the
boundaries) with an average success rate (SR) of 95.2% and an average peak
signal-to-noise ratio of 41.37 dB. Correlation analyses are conducted,
including the relation between the adversarial boundary's width and the SR and
how the adversarial boundary changes the DNN model's attention. This paper's
discoveries can potentially advance the understanding of AEs and provide a
different perspective on how AEs can be constructed.
Fahad Alrasheedi, Xin Zhong
# Imperceptible Adversarial Attack on Deep Neural Networks from Image Boundary

###### Abstract

Although Deep Neural Networks (DNNs), such as the convolutional neural networks (CNN) and Vision Transformers (ViTs), have been successfully applied in the field of computer vision, they are demonstrated to be vulnerable to well-sought Adversarial Examples (AEs) that can easily fool the DNNs. The research in AEs has been active, and many adversarial attacks and explanations have been proposed since they were discovered in 2014. The mystery of the AE's existence is still an open question, and many studies suggest that DNN training algorithms have blind spots.

The salient objects usually do not overlap with boundaries; hence, the boundaries are not the DNN model's attention. Nevertheless, recent studies show that the boundaries can dominate the behavior of the DNN models. Hence, this study aims to look at the AEs from a different perspective and proposes an imperceptible adversarial attack that systemically attacks the input image boundary for finding the AEs. The experimental results have shown that the proposed boundary attacking method effectively attacks six CNN models and the ViT using only 32% of the input image content (from the boundaries) with an average success rate (SR) of 95.2% and an average peak signal-to-noise ratio of 41.37 dB. Correlation analyses are conducted, including the relation between the adversarial boundary's width and the SR and how the adversarial boundary changes the DNN model's attention. This paper's discoveries can potentially advance the understanding of AEs and provide a different perspective on how AEs can be constructed.

Adversarial Attack Deep Neural Networks Image Boundary

## 1 Introduction

Deep Neural Networks (DNNs) have successfully advanced many applications of computer vision such as image classification [1, 2], object detection [3, 4, 5, 6], saliency detection [7, 8, 9], facial recognition [10, 11, 12], and many others. However, such DNNs are shown to be vulnerable to adversarial attacks that can convert Clean Examples, correctly classified by a DNN model, to Adversarial Examples [13]. The conversion process is a mathematical process that tweaks the Clean Example in well-chosen directions in the feature space with some energy, usually called _epsilon_, until the DNN model produces a wrong output, and the total of the tweaks is called adversarial perturbation [14, 15]. The adversarial perturbations can be categorized into two categories (explained in Section 2): imperceptible to the human eye and perceptible; this study focuses on the first category.

To deceive the human vision, the invisibility of the adversarial perturbations is one of the fundamental considerations for the imperceptible adversarial perturbations. Hence, humans should barely notice the difference between a Clean Example and its Adversarial Example. For instance, the imperceptibility of the adversarial perturbations produced by the Fast Gradient Sign Methods [14] depends on the value of _epsilon_ where higher values of epsilon lead to conspicuous differences between a Clean Example and its Adversarial Example. Hence, much research addressed the imperceptibility by introducing algorithms that iteratively increased the adversarial perturbation constrained on flipping the output of the DNN model [16, 15]. Such studies changed the whole features in the input space; however, some features were seen as needless to contribute to the conversion of the DNN model's output.
Consequently, a line of research went to investigate the possibility of finding Adversarial Examples by attacking only the important features in the input space. Hence, various adversarial attacks were proposed in which a saliency map was employed to find and attack the salient features in the input space, such as Jacobian-based Saliency Map Attack (JSMA) [19], Maximal Jacobian-based Saliency map attack (MJSMT) [20], and Probabilistic Jacobian based Saliency map attack (PJSMT) [21]. Moreover, Qian _et al._[22] proposed an adversarial attack that used an attention model to find a small continuous area, called a contributing feature region (CFR), in the input to attack; they called that perturbation _imperceptible adversarial patch_. Such studies produced imperceptible perturbations; however, their arguments for successfully finding the Adversarial Examples were based on finding and attacking important features in the input space and ignoring the other features that were seen as unimportant in the adversarial attacks.

Unimportant features could still be critical to cause Adversarial Examples. Hence, it is possible to only modify a small area, and include unimportant features in the input space to change a Clean Example to an Adversarial Example. Here, the modifications usually came as adversarial patches [23, 17, 24] or adversarial frames [18, 18] that covered an unimportant area and were able to make that area contributing to find the Adversarial Examples. Also, such patches or frames could partially cover the important features in the input space and make the DNN's model classify the input adversarially [25, 23]. Moreover, Su _et al._[26]introduced the application of Differential Evolution (DE) to find the Adversarial Examples by changing several features ranging from one to five features. Usually, the values of those DE-selected features extremely contrast with the neighboring features; the DE algorithm usually, selected from the important features; but it also could choose features that were not overlapped with the main object in the input. These studies usually lead to salient noises that can be noticed by human vision easily.

This study is motivated to investigate the possibility of attacking the DNN's models using only unimportant features in the input space while still keeping the adversarial perturbation imperceptible to the human vision as shown in Figure1. Hence, we propose an adversarial attack that systemically attacks the input image boundaries and increases the width of the adversarial boundaries until finding the Adversarial Example. We choose the input boundaries for two reasons: first, human vision tends to focus on the center of images and ignore the boundary, which is known as the center bias [27, 28, 29]; second, although the input image boundaries usually do not overlap with the salient objects, they can be utilized to (i) improve the model performance as discussed in some special padding techniques of DNN models [30, 31, 32]; and (ii) encode the absolute position information in the semantic representation learning [33, 34]. Thus, image boundaries can have a desirable property in the imperceptible adversarial attack: image boundaries can be unimportant in human vision, but dominate a deep learning model.

Moreover, this study aims at studying the weakness of the DNN models from different perspectives such as correlating the boundaries to the Adversarial Examples regardless of what adversarial label the model produces. Henceforth, our experiments focus on the un-targeted attack and the white-box setup where the attacker has full access to the model. Our findings have the potential to help advance the understanding of Adversarial Examples and provide insights into what can be the reasons behind the existence of Adversarial Examples. Our contributions are three-manifold:

* Proposing a novel adversarial attack that systemically attacks the input image from the boundaries where the width of the attacked boundaries increases until the Adversarial Example is found. Our adversarial attack was effective with an average success rate of 95.2% when attacking six CNN models and the ViT while only modifying less than 32% of the input image content.
* Attacking only the input image boundaries, so that improving the imperceptibility of the adversarial perturbations, which is supported by an average peak signal-to-noise ratio (\(PSNR\)) of \(41.37\) in the experiments.
* Correlating the Adversarial Examples to the unimportant features (_i.e._, image boundaries) to provides a different perspective to understand the Adversarial Examples. We show the boundary width required to achieve a desired success rate, and how a model's attention changes when attacked by the proposed boundary adversarial examples.

Figure 1: Three Adversarial Examples crafted from the same Clean Example by three different adversarial attacks: A) Patch of [17], B) 5-pixel Frame of [18], and C) 5-pixel Boundary of our attack. The true label is golf ball while the three attacks agree on the Adversarial Label (parachute).
The remainder of this paper is organized as follows. In Section 2, we review and categorize the related work. Section 3 discusses our approach for our adversarial attack followed by evaluation results in Section 4. Finally, Section 5 concludes with the discussion on evaluation and highlights some of the future work in this sector.

## 2 Related Work

This section briefly reviews the literature by dividing the adversarial perturbations into two main categories: Imperceptible Perturbations and Perceptible Perturbations.

### Imperceptible Perturbations

At the early stage of the field, the Adversarial Examples [13] were introduced to be imperceptible to human vision. Hence, the addition of an adversarial perturbation to the Clean Example has barely a human-vision effect on the Clean Example. Based on the coverage of the Imperceptible Perturbations, there are two types of adversarial perturbations: Fully Coverage based and Partially Coverage based.

#### 2.1.1 Fully Coverage Based

In this group, the adversarial perturbation is made to be of the same size as the Clean Example. Hence, each feature in the Clean Example will be affected by the corresponding feature in the adversarial perturbation. Szegedy _et al._[13] introduced an expensive adversarial attack which was based on L-BFGS, and guaranteed to find the Adversarial Example. Then, Goodfellow _et al._[14] proposed a cheap and effective adversarial attack called Fast Gradient Sign Method, FGSM. The sign function of the gradients in the FGSM helps determine the modification directions in the input's features. The FGSM used an \(epsilon\) as quantity to equally change the features with their corresponding directions; however, the FGSM did not guarantee to produce the Adversarial Example and the attack depended on the value of \(epsilon\). That is why Kurakin _et al._[16] proposed an iterative FGSM where the input was kept being perturbed by a small \(epsilon\) until the Adversarial Example was found. Moreover, Moosavi-Dezfooli _et al._[15] proposed an adversarial attack, DeepFool, which improved the imperceptibility of the adversarial perturbations by finding the smallest adversarial perturbation that reliably converted a Clean Example to its Adversarial Example.

#### 2.1.2 Partially Coverage Based

In this group, the adversarial perturbations change a subgroup of features in the input's space to find the Adversarial Example. For instance, Papernot _et al._[19] proposed a Jacobian-based Saliency map Attack, JSMA, that could find the salient features in the input which if attacked, the Adversarial Example could be found. Different studies followed the JSMA to use the Jacobian-based saliency map in their adversarial attacks such as maximal Jacobian based Saliency map attack (MJSMT) [20], and Probabilistic Jacobian based Saliency map attack (PJSMT) [21]. Also, Qian _et al._[22] proposed an adversarial attack that used network explanations to find a small suitable semantic region in the input for adversarial modifications.

Moreover, Wu _et al._[35] proposed an adversarial attack that increased the transferability of the Adversarial Examples in the transfer-based black setting. They claim that Adversarial Examples crafted by a source model might not fool another model (target model) and that was due to the over-fitting to the source model. Hence, they proposed a procedure that used an attention mechanism to extract the important features to attack.

### Perceptible Perturbations

In this category, the perceptible perturbations occlude small areas in the Clean Example; hence, they are visually conspicuous to human vision but usually ignorable. Also, they come in different sizes and shapes. For example, Brown _et al._[23] proposed an adversarial attack that could produce an adversarial patch when partially covering the Clean Example; it would fool the classifier. Karmon _et al._[17] addressed the problem of the adversarial patch's size by introducing a small noise that can be localized in the Clean Example in a way it did not overlap with the salient object in that example, and still fool the DNN model. Evtimov_et al._[36] proposed Robust Physical World Attack, RP2, that could produce small adversarial stickers in the shape of road vandalism (such as camouflage art and graffiti). Hence, when such an adversarial sticker was physically attached to a road sign, the classifier would misclassify the sign. Sharif _et al._[25] introduced a glass frame that could fool the face identification model. Finally, Zajac _et al._[18] proposed an adversarial frame that could be placed on the edges of a Clean Example and would fool both the image classifier and object detector.

## 3 The Proposed Boundary Attack

Algorithm 1 produces an Adversarial Example by attacking the input's boundaries while keeping the remaining part intact. We explain the algorithm by decomposing it into two loops: an outer loop and an inner loop. The former
increases the boundary's width while the latter attacks the boundary through the iterative FGSM (I-FGSM). Sections 3.1 and 3.2 detail the discussions loop respectively.

```
0:\(m\), \(l_{m}\), \(f_{\theta}\), \(\epsilon\), \(minimum\), where \(m\) is the Clean Example assuming the format is a channel last, \(l_{m}\) is the true label, \(f_{\theta}\) is the model, \(\epsilon\) is the initial epsilon, and \(minimum\) is the lower bound for the epsilon.
0:\(m^{\prime}\) which is the Adversarial Example.
1:\(w\gets 1\) /* initial width for edges */
2:while\(w<40\)do /* the outer loop.*/
3:\(m^{\prime}\gets m\) /* initial Adversarial Example.*/
4:\(cnt\gets 0\)
5:while\(cnt<15\)do /* 15 is replaced with 50 for other attacks as explained in Section 4.3.2 */
6:\(m^{\prime}\gets Boundary_{fgsm}(m^{\prime},w,l_{m},f_{\theta},\epsilon)\)
7:\(l_{m^{\prime}}\gets f_{\theta}(m^{\prime})\)
8:if\(l_{m}\neq l_{m^{\prime}}\)then
9:if\(PSNR>40\) OR \(\epsilon==minimum\)then
10: break outer loop.
11:endif
12: break inner loop.
13:endif
14:\(cnt\gets cnt+1\)
15:endwhile
16:\(w\gets w+1\)
17:\(\epsilon\leftarrow\epsilon*0.75\)
18:endwhile
19:return\(m^{\prime}\)
```

**Algorithm 1** Boundary Attack

### The outer loop

The loop increases the boundary's width to be attacked in each iteration. It starts with one-pixel boundary (step \(1\) in Algorithm 1) and keeps the remaining area untouched. If the current boundary's width does not succeed in finding the Adversarial Example, the width will increase by one pixel in the next iteration (step \(16\)). Figure 2 is an example to visually illustrate how the boundary's width increases in each iteration of the outer loop; the width is one-pixel in the \(1^{st}\) iteration, two-pixel in the \(2^{nd}\) iteration, and three-pixel in the \(3^{rd}\) iteration. The outer loop keeps iterating until the inner loop signals a break for the outer loop (explained in Section 3.2).

For imperceptibility, the outer loop starts with an \(epsilon\) value, \(\epsilon\), that magnifies the signs of boundaries' gradients ( explained in Section 3.2). Then, the \(\epsilon\) will be decreased by a factor of \(0.75\) in every iteration as shown in Figure 2; however, the decrement process will stop when reaching the lower bound for \(\epsilon\) called \(minimum\) in Algorithm 1 (and not shown in Figure 2).

### The Inner loop

The inner loop is steps \(5\) to \(15\) in Algorithm 1. It perturbs the \(m^{\prime}\) with the current \(\epsilon\) value using \(Boundary_{fgsm}\) function (as shown in step \(6\)) which is a version of the \(FGSM\) attacking only the input's boundary. The details of the \(Boundary_{fgsm}\) function is in Algorithm 2 where step \(1\) computes the model's gradients with respect to the input \(m^{\prime}\) given that the true label is \(l_{m}\); step \(2\) masks the middle part of the gradients \(g\) by reassigning them \(zeroes\) and keeps

Figure 2: An example of input with a small shape of (8,8,3) to illustrate how the outer loop of Algorithm 1 increases the width of the borders. For example, in the \(1^{st}\) iteration, the width of the border is one-pixel (in green color), it becomes two-pixel in the \(2^{nd}\) iteration attacks, and three-pixel in \(3^{rd}\) iteration. Also, the epsilon decreases by 0.75 in each iteration.
the boundary's gradients untouched, step \(3\) is the adversarial perturbation where the sign is multiplied by the \(\epsilon\) value. Lastly, step \(4\) adds the adversarial perturbation to the \(m^{\prime}\) which gives an output that aims at maximizing the distance between the model's output class and the true label \(l_{m}\).

The upper bound of the inner loop is set to be \(15\) (step \(5\) in Algorithm 1) which empirically keeps the perturbation imperceptible. The inner loop will keep perturbing the boundary using the \(Boundary_{fgsm}\) function until it finds the Adversarial Example and it then breaks; otherwise, it reaches its upper bound with no success which in turn makes the outer loop go to the next iteration. Moreover, when the Adversarial Example is found, the inner loop will break the outer loop if either of the two following conditions is satisfied (as shown in step \(9\) in Algorithm 1):

* When the (\(PSNR\)) is higher than a threshold. The threshold is set to be \(40\) and chosen heuristically.
* When the \(\epsilon\) reaches its lower bound \(minimum\); that is because increasing the attack area while the epsilon value, \(\epsilon\), is static, it never gives higher \(PSNR\) than the smaller edges attacked by the same \(\epsilon\).

These two conditions make the algorithm efficient at finding an Adversarial Example with the smallest perturbed area and the highest \(PSNR\).

## 4 Experiments and Results

This section discusses and analyzes our experimental setup, evaluation metrics, and quantitative results.

### Experimental Setup

To test the proposed method, we attack the vision transformer (ViT) [37], and six widely applied CNN models including VGG16 and VGG19 [2], ResNet50 and ResNet101 [1], and EfficientNetB1 and EfficientNetB2 [38]. As for the CNN models, we train and evaluate the models using the Imagente dataset [39] which is a subset of the ImageNet dataset [40] and has ten classes of tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute. The Imagenette has a training dataset with \(9,469\) images and a validation dataset with \(3,925\) images; the size of the images is \((224,224,3)\) and they are also upsampled to \((240,240,3)\) and \((260,260,3)\) for the EfficientNetB1 and EfficientNetB2 respectively. We apply both the Imagenette and Tiny ImageNet [41] to the ViT; we use ViT(1) and ViT(2) to represent the ViT trained and tested on the Imagenette and Tiny ImageNet respectively. The Tiny ImageNet is also a subset of ImageNet, has \(200\) classes, and has a training dataset with \(100,000\) images and a validation dataset with \(10,000\) images. The size of images in the Tiny ImageNet is \((64,64,3)\), and we upsample them to \((224,224,3)\).

We replace the last 1000-output layer with a 10-output layer in the CNN models and ViT(1) and to 200-output layer in the ViT(2). We use twenty epochs to fine-tune the output layer in the CNN models while fine-tuning the last transformer encoder and the output layer in the ViTs. The values of the input's pixels are in the range of [0,255] and [0,1] for the CNN models and the ViTs respectively. Hence, we heuristically choose the \(\epsilon\) value and its minimums respectively to be \(10\) and \(3\) when attacking the CNN models while \(0.02\) and \(0.01\) when attacking the ViTs. Table 1 shows the accuracy of the models on the validation dataset. To ensure that the misclassifications are caused by the attack itself and not the model, we only attack the samples that are successfully classified from the validation dataset.

### The Evaluation Metrics

Four evaluation metrics are used in the experiments: Success Rate (\(SR\)), Mean Squared Error (\(MSE\)), Mean Absolute Error (\(MAE\)), and the \(PSNR\). The \(SR\) can be defined as followers:

\[SR(f_{\theta},M)=\frac{1}{T}\sum_{m\in M}1_{\big{\{}f_{\theta}(m^{\prime})\neq l _{m}\big{\}}}, \tag{1}\]
where \(f\) is a model parameterized by \(\theta\), \(M\) represents all the images in the validation dataset that are correctly classified by the \(f\), \(T\) is the number of images in \(M\), \(l_{m}\) is the true label for the Clean Example \(m\), and \(m^{\prime}\) is the perturbed Example crafted from the \(m\).

Moreover, the computations of the \(MSE\), \(MAE\), and \(PSNR\) only consider \(N\in M\) where \(N\) represents images in the \(M\) that are both correctly classified by \(f_{\theta}\) and successfully attacked by our adversarial attack. The \(MSE\), \(MAE\), and \(PSNR\) can be respectively defined by the following equations:

\[MSE(N)=\frac{1}{T}\sum_{n\in N}mean((n-n^{\prime})^{2}), \tag{2}\]

\[MAE(N)=\frac{1}{T}\sum_{n\in N}mean(|n-n^{\prime}|), \tag{3}\]

\[PSRN(N)=\frac{1}{T}\sum_{n\in N}20*\log{(\frac{max}{\sqrt{mean((n-n^{\prime})^ {2})}})}, \tag{4}\]

where \(T\) is the number of images in \(N\), \(n\) and \(n^{\prime}\) are the Clean Example and its Adversarial Example, \(max\) is the highest possible value in the \(N\), and the \(mean\) function is the reduced mean. The purpose of using \(MSE\) and \(MAE\) is to show the difference between the Clean Example and its Adversarial Example from two different evaluation perspectives. Besides the \(MSE\) and \(MAE\), the \(PSNR\) is used to show the ratio of the possible highest power of a signal to the power of that noise that corrupts that signal.

Finally, we investigate the model's attention using the Grad-CAM [42] for the Clean Example and its Adversarial Example, and how the adversarial perturbations can change the model's attention significantly. Hence, this experiment can correlate the unimportant features (_i.e._, input image boundaries) with the Adversarial Examples; this correlation hopefully helps advance our understanding of the reasons behind the existence of the Adversarial Examples.

### The Quantitative Results

#### 4.3.1 Correlating the \(Sr\) with the boundary's width

Figure 3 shows the comparison of our adversarial attack against the six CNN models and the two ViTs in terms of the \(SR\) and the boundary's width. The ViT(2) is the most vulnerable model where the attack starts with a high \(SR\) and achieves around 99.0% with less than a 5-pixel boundary attack in all the \(M\) dataset's images. This highest \(SR\) is attributed to the large number of classes in the Tiny ImageNet which makes it easier for our adversarial attack to confuse the ViT model. Then, the ResNet50 model shows the second highest vulnerability; the ViT(1) becomes comparable to the Resnet50 when attacked with a 25-pixel boundary and both achieve an average \(SR\) of \(99.9\%\) at the 40-pixel boundary. Also, the attack's \(SR\)s against EfficientNetB1 and EfficientNetB2 converge with the other highest \(SR\)s at the boundaries of 30-pixel width. Lastly, the ResNet101, VGG19, and VGG16 are comparable to each other at the 20-pixel boundary, and they also achieve an average \(SR\) of \(96.9\%\) at the 40-pixel boundary.

Moreover, Figure 4 shows the relation between different widths for the adversarial boundaries and the \(SR\). The purpose of this analysis is to show how much possible the attacker can produce an Adversarial Example in terms of the perturbation amount. For example, the attacker only needs a 5-pixel boundary (\(8\%\) of the image content) to achieve a

\begin{table}
\begin{tabular}{|l|l|} \hline
**Model** & **Accuracy** \\ \hline VGG16 & 0.9156 \\ \hline VGG19 & 0.9177 \\ \hline ResNet50 & 0.957 \\ \hline ResNet101 & 0.96 \\ \hline EfficientNetB1 & 0.977 \\ \hline EfficientNetB2 & 0.99 \\ \hline ViT(1) & 0.986 \\ \hline ViT(2) & 0.74 \\ \hline \end{tabular}
\end{table}
Table 1: The accuracy of six CNN models on the validation datatset of Imagenette. The accuracy of ViT(1) and ViT(2) are respectively on the validation datatset of Imagenette and the validation datatset of Tiny ImageNet.
success rate of \(55.8\%\), a 10-pixel boundary (\(17\%\) of the image content) to achieve an average \(SR\) of \(81.6\%\), a 20-pixel boundary (\(32\%\) of the image content) to achieve an average \(SR\) of \(95.2\%\), a 30-pixel boundary (\(46\%\) of the image content) to achieve an average \(SR\) of \(97.5\%\), and a 40-pixel boundary (\(58\%\) of the image content) to achieve an average \(SR\) of \(98.7\%\). It is evident that as we increase the width of the boundary, the success rate increases. Hence, the attacker can increase the width of the boundary to increase the possibility of getting an Adversarial Example.

#### 4.3.2 Comparative Study

Since our adversarial attack integrates two concepts of attacking unimportant features (_e.g._, boundaries) and imperceptibility, it is hard to directly compare our work with the related work in Section 2. However, we modify Algorithm 1 to re-implement three different attacks: the adversarial patch of [17], the adversarial frame of [18], and attacking the whole example by I-FGSM [16] (henceforth, we call it \(Whole\) attack). Then, we compare our attacks with these three attacks using the evaluation metrics mentioned in Subsection 4.2.

The modifications are as follows: First, at the beginning of Algorithm 1, we set the patch's size to be \((50,50,3)\) and place it at the top left corner of the Clean Example \(m\); the frame's width is set to be 5-pixel and occludes the original boundaries; the patch and frame are initialized randomly. Secondly, the outer loop is removed, and only the inner loop is included in Algorithm 1. The upper bound for the inner loop is set to be 50 (more than three times of our attack's upper bound); the \(\epsilon\) value is set to be static (_i.e._, equal to the lower bound used in our attack; for example \(3\) and \(0.01\) for the CNN models and ViTs respectively). Lastly, step \(2\) in Algorithm 2 masks the gradients that are out of the patch and frame areas, and no mask in the \(Whole\) attack since I-FGSM attacks the whole image.

Figure 4: The relationship between the boundary’s width and the Success Rate (\(SR\)). The \(SR\) in each bar is an average of the \(SR\)s over all the six models and the two ViTs.

Figure 3: Our attack’s \(SR\)s when attacking six different models and two ViTs where ViT(1) trained on Imagenette while ViT(2) is the same transformer but trained on the Tiny ImageNet.
Table 2 shows the comparison of our attack with the three attacks (adversarial patch, adversarial frame, and the \(Whole\) attack) when attacking the six CNN models. Also, Table 3 shows the same comparison when attacking the ViTs. In both tables, we can observe that the perturbations (differences between the Clean Examples and Adversarial Examples in terms of the \(MSE\) and \(MAE\)) in our attack are significantly lower than all the other attacks against the six models and the ViTs. Moreover, the \(PSNR\)s are shown to be higher for our attack compared to the other attacks against the CNN models and ViTs unless the VGGs attacked by the Frame are higher than ours; however, the average \(PSNR\) for our attack is \(41.37\) which is higher than all the other attacks. Also, The \(SR\)s show that our attacks can be comparable to the \(Whole\) attack and much higher than the other two attacks (adversarial patch and frame). Remarkably, we set the upper bound of the inner loop in Algorithm 1 for the three attacks (\(Whole\) attack, adversarial patch, and frame) to be three times more than our attack for fair comparison (_i.e._, \(15\) to \(50\)), otherwise the \(SR\) for the three attacks would increase if we would increase their upper bounds above \(50\).

Finally, if we compare the \(SR\)s for all the attacks in Table 2 and Table 3 against the six CNN models and the ViTs respectively, we can observe that the ViTs are more vulnerable than the CNN models for all the attacks.
The goal of showing our attack with various boundary widths for each sample is to investigate if the model's attention for a successful attack will significantly change if the boundary's width increases. For example, the second row in Figure 6 represents a successful five-pixel boundary attack; hence, the model's attention to the unsuccessful attack in the \(2^{nd}\) column (one-pixel boundary attack) still signifies features from the main object in the example (church) similar to the model's attention for no attack in the \(1^{st}\) column with a small change in the size and shape of the attention. However, the five-pixel boundary attack for the same sample as in the \(3^{rd}\) column is successful and made the model's attention totally different, and only small features from the main object are included in the attention; also, the attention changes as the boundary's width increases; but the model still gives the same adversarial label for the different boundary's widths attack against this specific sample.

Similarly, for the sample in the \(3^{rd}\) row in Figure 6), the model's attention does not significantly change when our attacks (i.e., one-pixel and five-pixel boundary) are not successful; however, our attack against this sample becomes successful when using a ten-pixel boundary (as in the \(4^{th}\) column) and the attention significantly changes. Also, increasing the width of the boundary in this sample after a successful attack, changes the model's attention and the model give different adversarial labels (i.e., \(5^{th}\) column and \(6^{th}\) column in the last row in 6).

Lastly, we show the model's attentions for these three samples when attacked by the adversarial patch and frame as in the \(7^{th}\) and \(8^{th}\) columns in Figure 6. It is evident that in the second and third samples, the patches and frames become the model's attention; the adversarial labels are different from our attack. However, the patch and frame used to attack the first sample are also successful, the model gives the same adversarial label as our attack, and they changes the model's attention but the patch and frame themselves are not the model's attention.

Interestingly, we observe that If the sample does not need a strong perturbation as in the case of the \(1^{st}\) sample, all the attacks agree on the model's attention and the adversarial label. But, if the sample needs a strong adversarial perturbation as in the \(2^{nd}\) and \(3^{rd}\) samples, the patch and frame become the attention of the model and do not change the image context; while our boundary attacks can turn on some areas that are not in the boundary and make them significant. We conjecture that our attack perturbs the image context and changes the relationships/contrast between

Figure 5: Three samples of successful attacks. The \(1^{st}\) row is a sample that can be attacked by a one-pixel boundary attack as shown in column (C); while attacking the same sample by the patch and frame attacks are respectively shown in columns (A) and (B) in the same row. Similarly, the \(2^{nd}\) row and \(3^{rd}\) row represent the five and ten boundary attacks respectively. The labels are explained in the model’s attentions in Figure 6.
different parts of the image so that indirectly forces the model's attention to certain areas. We will further investigate this finding in future work.

## 5 Conclusion and Future Research Directions

This paper proposes an imperceptible adversarial attack from the input image boundaries. The proposed adversarial attack is shown to be effective when attacking six different CNN models and the Vision Transformer (ViT) with an average success rate of 95.2% when modifying only 32% of the content, mainly from the input boundaries which usually are ignored by the human vision and do not overlap with the salient objects in the input.

Our experimental results of attacking from the image boundaries align with related discoveries in the literature that the performance of CNN models can be dominated by the boundaries. We show that the cutting-edge ViT models are also vulnerable to our proposed boundary adversarial attacks. We find that the ViT model is sensitive to the adversarial perturbations; it is more vulnerable to our attack than the CNN models. We provide some correlation analysis, such as showing how much boundary's width is required to achieve a desired success rate and how the model's attention changes when attacked by our adversarial attack with different boundary's widths. Our findings can potentially advance the understanding of Adversarial Examples and provide a different perspective on how Adversarial Examples can be constructed.

Lastly, one research direction to further this study is to determine which boundary in the input image is more dominating in the attacks; we suspect the need for equal adversarial perturbations in all the input boundaries. One possible solution for such an attack is to combine the saliency maps or Jacobian values of the boundaries in attacks.
