Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
Di Jin Zhijing Jin Joey Tianyi Zhou Peter Szolovits
cs.CL cs.AI cs.LG


Squarefree monomial ideals with maximal depth
     Ahad Rahimi
    December 30, 2023
=============================================




 Machine learning algorithms are often vulnerable to adversarial examples that have imperceptible alterations from the original counterparts but can fool the state-of-the-art models. It is helpful to evaluate or even improve the robustness of these models by exposing the maliciously crafted adversarial examples. In this paper, we present TextFooler, a simple but strong baseline to generate adversarial text. By applying it to two fundamental natural language tasks, text classification and textual entailment, we successfully attacked three target models, including the powerful pre-trained BERT, and the widely used convolutional and recurrent neural networks. We demonstrate three advantages of this framework: (1) effective—it outperforms previous attacks by success rate and perturbation rate, (2) utility-preserving—it preserves semantic content, grammaticality, and correct types classified by humans, and (3) efficient—it generates adversarial text with computational complexity linear to the text length.[The code, pre-trained target models, and test examples are available at <https://github.com/jind11/TextFooler>.]




§ INTRODUCTION


In the last decade, Machine Learning (ML) models have achieved remarkable success in various tasks such as classification, regression and decision making. However, recently they have been found vulnerable to adversarial examples that are legitimate inputs altered by small and often imperceptible perturbations <cit.>. These carefully curated examples are correctly classified by a human observer but can fool a target model, raising serious concerns regarding the security and integrity of existing ML algorithms. On the other hand, it is showed that robustness and generalization of ML models can be improved by crafting high-quality adversaries and including them in the training data <cit.>.


While existing works on adversarial examples have obtained success in the image and speech domains <cit.>, it is still challenging to deal with text data due to its discrete nature. Formally, besides the ability to fool the target models, outputs of a natural language attacking system should also meet three key utility-preserving properties: (1) human prediction consistency—prediction by humans should remain unchanged, (2) semantic similarity—the crafted example should bear the same meaning as the source, as judged by humans, and (3) language fluency—generated examples should look natural and grammatical. Previous works barely conform to all three requirements. For example, methods such as word misspelling <cit.>, single-word erasure <cit.>, and phrase insertion and removal <cit.> result in unnatural sentences. Moreover, there is almost no work that attacks the newly-risen BERT model on text classification. 






In this work, we present TextFooler, a simple but strong baseline for natural language attack in the black-box setting, a common case where no model architecture or parameters are accessible (Figure<ref>. We design a more comprehensive paradigm to create both semantically and syntactically similar adversarial examples that meet the three desiderata mentioned above. 


Specifically, we first identify the important words for the target model and then prioritize to replace them with the most semantically similar and grammatically correct words until the prediction is altered. We successfully applied this framework to attack three state-of-the-art models in five text classification tasks and two textual entailment tasks, respectively. On the adversarial examples, we can reduce the accuracy of almost all target models in all tasks to below 10% with only less than 20% of the original words perturbed. In addition, we validate that the generated examples are (1) correctly classified by human evaluators, (2) semantically similar to the original text, and (3) grammatically acceptable by human judges.





Our main contributions are summarized as follows:


    
  * We propose a simple but strong baseline, TextFooler, to quickly generate high-profile utility-preserving adversarial examples that force the target models to make wrong predictions under the black-box setting.
    
  * We evaluate TextFooler on three state-of-the-art deep learning models over five popular text classification tasks and two textual entailment tasks, and it achieved the state-of-the-art attack success rate and perturbation rate.  
    
  * We propose a comprehensive four-way automatic and three-way human evaluation of language adversarial attacks to evaluate the effectiveness, efficiency, and utility-preserving properties of our system.
    
  * We open-source the code, pre-trained target models, and test samples for the convenience of future benchmarking.
    





§ METHOD




 §.§ Problem Formulation


Given a corpus of N sentences 𝒳= {X_1, X_2,…, X_N}, and a corresponding set of N labels 𝒴={Y_1, Y_2,…, Y_N}, we have a pre-trained model F: 𝒳→𝒴, which maps the input text space 𝒳 to the label space 𝒴. 

For a sentence X ∈𝒳, a valid adversarial example X_adv should conform to the following requirements: 

    F(X_adv) ≠ F(X), and Sim(X_adv, X) ≥ϵ
    ,

where Sim:𝒳×𝒳→ (0,1) is a similarity function and ϵ is the minimum similarity between the original and adversarial examples.
In the natural language domain, Sim(·) is often a semantic and syntactic similarity function. 



 §.§ Threat Model


Under the black-box setting, the attacker is not aware of the model architecture, parameters, or training data. It can only query the target model
with supplied inputs, getting as results the predictions and corresponding confidence scores. 

The proposed approach for adversarial text generation is shown in Algorithm <ref>, and consists of the two main steps:







  §.§.§ Step 1: Word Importance Ranking (line 1-6)


Given a sentence of n words X={w_1, w_2, …, w_n}, we observe that only some keywords act as influential signals for the prediction model F, echoing with the discovery of <cit.> that BERT attends to the statistical cues of some words. Therefore, 
we create a selection mechanism to choose the words that most significantly influence the final prediction results. Using this selection process, we minimize the alterations, and thus maintain the semantic similarity as much as possible. 

Note that the selection of important words is trivial in a white-box scenario, as it can be easily solved by inspecting the gradients of the model F, while most other words are irrelevant. However, under the more common black-box set up in our paper, the model gradients are unavailable. Therefore, we create a selection mechanism as follows. We use the score I_w_i to measure the influence of a word w_i ∈ X towards the classification result F(X)=Y. We denote the sentence after deleting the word w_i as X_∖ w_i = X∖{w_i }={w_1, …, w_i-1, w_i+1,… w_n}, and use F_Y(·) to represent the prediction score for the Y label. 

The importance score I_w_i is therefore calculated as the prediction change before and after deleting the word w_i, which is formally defined as follows,


    I_w_i=
        
          F_Y(X)-F_Y(X_∖ w_i),    if F(X)=F(X_∖ w_i)=Y 
    
          (F_Y(X)-F_Y(X_∖ w_i))+(F_Y̅(X_∖ w_i)-F_Y̅(X)),
        if F(X)=Y, F(X_∖ w_i)=Y̅, and  Y≠Y̅
          .



After ranking the words by their  importance score, we further filter out stop words derived from NLTK[<https://www.nltk.org/>] and spaCy[<https://spacy.io/>] libraries such as “the”, “when”, and “none”. This simple step of filtering is important to avoid grammar destruction.



  §.§.§ Step 2: Word Transformer (line 7-30)
 

Given the word w_i ∈ X with a high importance score obtained in Step 1, we need to design a word replacement mechanism. A suitable replacement word needs to fulfill the following criteria: it should (1) have similar semantic meaning with the original one, (2) fit within the surrounding context, and (3) force the target model to make wrong predictions. In order to select replacement words that meet such criteria, we propose the following workflow.

Synonym Extraction:  We gather a candidate set Candidates for all possible replacements of the selected word w_i. Candidates is initiated with N closest synonyms according to the cosine similarity between w_i and every other word in the vocabulary. 

To represent the words, we use word embeddings from <cit.>. These word vectors are specially curated for finding synonyms, as they achieve the state-of-the-art performance on SimLex-999, a dataset designed to measure how well different models judge the semantic similarity between words <cit.>. 

Using this set of embedding vectors, we identify the top N synonyms whose cosine similarity with w is higher than δ. Note that enlarging N or lowering δ would both generate more diverse synonym candidates; however, the semantic similarity between the adversary and the original sentence would decrease. In our experiments, empirically setting N to be 50 and δ to be 0.7 strikes a balance between diversity and semantic similarity control.

POS Checking: In the set Candidates of the word w_i, we only keep the ones with the same part-of-speech (POS)[We used the off-the-shelf spaCy tagger, available at https://spacy.io/api/tagger] as w_i. This step is to assure that the grammar of the text is mostly maintained (line 10 in Algorithm <ref>).

Semantic Similarity Checking: For each remaining word c ∈Candidates, we substitute it for w_i in the sentence X, and obtain the adversarial example X_adv = {w_1, …, w_i-1, c ,w_i+1, …, w_n}.
We use the target model F to compute the corresponding prediction scores F(X_adv). We also calculate the sentence semantic similarity between the source X and adversarial counterpart X_adv. Specifically, we use Universal Sentence Encoder (USE)  <cit.> to encode the two sentences into high dimensional vectors and use their cosine similarity score as an approximation of semantic similarity. The words resulting in similarity scores above a preset threshold ϵ are placed into the final candidate pool FinCandidates (line 11-19 in Algorithm <ref>).

Finalization of Adversarial Examples: In the final candidate pool FinCandidates, if there exists any candidate that can already alter the prediction of the target model, then we choose the word with the highest semantic similarity score among these winning candidates. But if not, then we select the word with the least confidence score of label y as the best replacement word for w_i, and repeat Step 2 to transform the next selected word (line 20-30 in Algorithm <ref>).


Overall, the algorithm first uses Step 1 to rank the words by their importance scores, and then repeats Step 2 to find replacements for each word in the sentence X until the prediction of the target model is altered.




§ EXPERIMENTS




 §.§ Tasks

We study the effectiveness of our adversarial attack on two important NLP tasks, text classification and textual entailment.[Our datasets are at <http://bit.ly/nlp_adv_data>] The dataset statistics are summarized in Table <ref>. Following the practice by <cit.>, we evaluate our algorithm on a set of 1,000 examples randomly selected from the test set. 







  §.§.§ Text Classification

To study the robustness of our model, we use text classification datasets with various properties, including news topic classification, fake news detection, and sentence- and document-level sentiment analysis, with average text length ranging from tens to hundreds of words. 

    
  * AG's News (AG): Sentence-level classification with regard to four news topics: World, Sports, Business, and Science/Technology. 
    
    Following the practice of <cit.>, we concatenate the title and description fields for each news article.
    
    
  * Fake News Detection (Fake): Document-level classification on whether a news article is fake or not. The dataset comes from the Kaggle Fake News Challenge.[<https://www.kaggle.com/c/fake-news/data>] We concatenate the title and news body of each article.
    
    
    
  * MR: Sentence-level sentiment classification on positive and negative movie reviews <cit.>. 
    
    We use 90% of the data as the training set and 10% as the test set, following the practice in <cit.>.
    
    
  * IMDB: Document-level sentiment classification on positive and negative movie reviews.[<https://datasets.imdbws.com/>] 
    
    
  * Yelp Polarity (Yelp): Document-level sentiment classification on positive and negative reviews <cit.>. Reviews with a rating of 1 and 2 are labeled negative and 4 and 5 positive. 
    
    
    





  §.§.§ Textual Entailment


    
  * SNLI: A dataset of 570K sentence pairs derived from image captions. The task is to judge the relationship between two sentences: whether the second sentence can be derived from entailment, contradiction, or neutral relationship with the first sentence <cit.>.
    
  * MultiNLI: A multi-genre entailment dataset with coverage of transcribed speech, popular fiction, and government reports <cit.>. Compared to SNLI, it contains more linguistic complexity with various written and spoken English text.





 §.§ Attacking Target Models


For each dataset, we train three state-of-the-art models on the training set, and achieved test set accuracy scores similar to the original implementation, as shown in Table <ref>. We then generate adversarial examples that are semantically similar to the test set to attack the trained models and make them generate different results.





On the sentence classification task, we target at three models: word-based convolutional neural network (WordCNN) <cit.>, word-based long-short term memory (WordLSTM) <cit.>, and the state-of-the-art Bidirectional Encoder Representations from
Transformers (BERT) <cit.>. 







For the WordCNN model, we used three window sizes of 3, 4, and 5, and 100 filters for each window size with a dropout of 0.3.
For the WordLSTM, we used a 1-layer bidirectional LSTM with 150 hidden units and a dropout of 0.3. For both models, we used the 200-dimensional Glove word embeddings pre-trained on 6B tokens from Wikipedia and Gigawords <cit.>. 
We used the 12-layer BERT model with 768 hidden units and 12 heads, with 110M parameters, which is called the base-uncased version.[<https://github.com/huggingface/pytorch-pretrained-BERT>]


We also implemented three target models on the textual entailment task: standard InferSent[<https://github.com/facebookresearch/InferSent>] <cit.>, ESIM[<https://github.com/coetaur0/ESIM>] <cit.>, and fine-tuned BERT.





 §.§ Setup of Automatic Evaluation


We first report the accuracy of the target models on the original test samples before the attack as the original accuracy. Then we measure the accuracy of the target models against the adversarial samples crafted from the test samples, denoted as after-attack accuracy. By comparing these two accuracy scores, we can evaluate how successful the attack is, — the more significant the gap between the original and after-attack accuracy signals, the more successful our attack is. Apart from these accuracies, we also report the perturbed word percentage as the ratio of the number of perturbed words to the text length. Furthermore, we apply USE[https://tfhub.dev/google/
universal-sentence-encoder] to measure the semantic similarity between the original and adversarial texts. These two metrics, the perturbed words percentage and the semantic similarity score, which together evaluate how semantically similar the original and adversarial texts are. We finally report the number of queries the attack system made to the target model and fetch the output probability scores. This metric can reveal the efficiency of the attack model.





 §.§ Setup of Human Evaluation


We conduct human evaluation on three criteria: semantic similarity, grammaticality, and classification accuracy. 

We randomly select 100 test sentences of each task to generate adversarial examples, one targeting WordLSTM on the MR dataset and another targeting BERT on SNLI. We first shuffled a mix of original and adversarial texts and asked human judges to rate the grammaticality of them on a Likert scale of 1-5, similar to the practice of <cit.>. Next, we evaluate the classification consistency by asking humans to classify each example in the shuffled mix of the original and adversarial sentences and then calculate the consistency rate of both classification results. Lastly, we evaluated the semantic similarity of the original and adversarial sentences by asking humans to judge whether the generated adversarial sentence is similar, ambiguous, or dissimilar to the source sentence. Each task is completed by two independent human judges who are native English speakers. The volunteers have university-level education backgrounds and passed a test batch before they started annotation.










§ RESULTS




 §.§ Automatic Evaluation


The main results of black-box attacks  in terms of automatic evaluation on five text classification and two textual entailment tasks are summarized in Table <ref> and <ref>, respectively. Overall, as can be seen from our results, TextFooler achieves a high success rate when attacking with a limited number of modifications on both tasks. No matter how long the text sequence is, and no matter how accurate the target model is, TextFooler can always reduce the accuracy from the state-of-the-art values to below 15% (except on the Fake dataset) with less than 20% word perturbation ratio (except the AG dataset under the BERT target model). For instance, it only perturbs 5.1% of the words on average when reducing the accuracy from 89.8% to only 0.3% on the IMDB dataset against the WordLSTM model. Notably, our attack system makes the WordCNN model on the IMDB dataset totally wrong (reaching the accuracy of 0%) with only 3.5% word perturbation rate. In the IMDB dataset, which has an average length of 215 words, the system only perturbed 10 words or fewer per sample to conduct successful attacks. This means that our attack system can successfully mislead the classifiers into assigning wrong predictions via subtle manipulation.

Even for BERT, which has achieved seemingly “robust” performance compared with the non-pretrained models such as WordLSTM and WordCNN, our attack model can still reduce its prediction accuracy by about 5–7 times on the classification task (e.g., from 95.6% to 6.8% for Yelp dataset) and about 9-22 times on the NLI task (e.g., from 89.4% to 4.0% for SNLI dataset), which is unprecedented. Our curated adversarial examples can contribute to the study of the interpretability of the BERT model <cit.>.

Another two observations can be drawn from Table <ref> and <ref>. (1) Models with higher original accuracy are, in general, more difficult to be attacked. For instance, the after-attack accuracy and perturbed word ratio are both higher for the BERT model compared with WordCNN on all datasets. (2) The after-attack accuracy of the Fake dataset is much higher than all other classification datasets for all three target models. We found in experiments that it is easy for the attack system to convert a real news article to a fake one, whereas the reverse process is much harder, which is in line with intuition.



Comparing the semantic similarity scores and the perturbed word ratios in both Table <ref> and <ref>, we find that the two results have a high positive correlation. Empirically, when the text length is longer than 10 words, the semantic similarity measurement becomes more stable. Since the average text lengths of text classification datasets are all above 20 words and those of textual entailment datasets are around or below 10 words, we need to treat the semantic similarity scores of these two tasks individually. Therefore, we performed a linear regression analysis between the word perturbation ratio and semantic similarity for each task and obtained r-squared values of 0.94 and 0.97 for text classification and textual entailment tasks, respectively. Such high values of r-squared reveal that our proposed semantic similarity has a high (negative) correlation with the perturbed words ratio, which can both be useful automatic measurements to evaluate the degree of alterations of the original text.  

We include the average text length of each dataset in the last row of Table <ref> and <ref> so that it can be conveniently compared against the query number. The query number is almost linear to the text length, with a ratio in (2,8). Longer text correlates with a smaller ratio, which validates the efficiency of TextFooler.



  
Benchmark Comparison We compared TextFooler with the previous state-of-the-art adversarial attack systems against the same target model and dataset. Our baselines include <cit.> that generates misspelled words by character- and word-level perturbation, <cit.> that iterates through every word in the sentence and find its perturbation, and <cit.> that uses word replacement by greedy heuristics. From the results in Table <ref>, we can see that our system beats the previous state-of-the-art models by both the attack success rate (calculated by dividing the number of wrong predictions by the total number of adversarial examples) and perturbed word ratio.








 §.§ Human Evaluation




We sampled 100 adversarial examples on the MR dataset with the WordLSTM and 100 examples on SNLI with BERT. We verified the quality of our examples via three experiments. First, we ask human judges to give a grammaticality score of a shuffled mix of original and adversarial text. 

 




As shown in Table <ref>, the grammaticality of the adversarial text are close to the original text on both datasets. By sensibly substituting synonyms, TextFooler generates smooth outputs such as “the big metaphorical wave” in Table <ref>. 
 
We then asked the human raters to assign classification labels to a shuffled set of original and adversarial samples. 


The overall agreement between the labels of the original sentence and the adversarial sentence is relatively high, with 92% on MR and 85% on SNLI. 

 Though our adversarial examples are not perfect in every case, this shows that majorities of adversarial sentences have the same attribute as the original sentences from humans' perspective. Table <ref> shows typical examples of sentences with almost the same meanings that result in contradictory classifications by the target model.











      


      






Lastly, we asked the judges to determine whether each adversarial sample retains the meaning of the original sentence. They need to decide whether the synthesized adversarial example is similar, ambiguous, or dissimilar to the provided original sentence. We regard similar as 1, ambiguous as 0.5, and dissimilar as 0, and obtained sentence similarity scores of 0.91 on MR and 0.86 on SNLI, which shows the perceived difference between original and adversarial text is small.






§ DISCUSSION




 §.§ Ablation Study




  §.§.§ Word Importance Ranking


To validate the effectiveness of Step 1 in Algorithm <ref>, i.e., the word importance ranking, we remove this step and instead randomly select the words in the text to perturb. We keep the perturbed word ratio and Step 2 the same. We use BERT as the target model and test on three datasets: MR, AG, and SNLI. The results are summarized in Table <ref>. After removing Step 1 and instead randomly selecting the words to perturb, the after-attack accuracy increases by more than 45% on all three datasets, which reveals that the attack becomes ineffective without the word importance ranking step. The word importance ranking process is crucial to the algorithm in that it can accurately and efficiently locate the words which cast the most significant effect on the predictions of the target model. This strategy can also reduce the number of perturbed words so as to maintain the semantic similarity as much as possible.





  §.§.§ Semantic Similarity Constraint




In Step 2 of Algorithm <ref>, for every possible word replacement, we check the semantic similarity between the newly generated sample and the original text, and adopt this replacement only when the similarity is above a preset threshold ϵ. We found that this strategy can effectively filter out irrelevant synonyms to the selected word. As we can see from the examples in Table <ref>, the synonyms extracted by word embeddings are noisy, so directly injecting them into the text as adversarial samples would probably shift the semantic meaning significantly. By applying the sentence-level semantic similarity constraint, we can obtain more related synonyms as suitable replacements. Table <ref> compares the automatic evaluation metrics before and after removing the semantic similarity constraint for the BERT-Base target model. From this table, we see that without the constraint, the attacking becomes easier (after-attack accuracy, perturbed  words  percentage, and number of queries all decrease). However, the semantic similarity between the original text and the corresponding adversaries also decrease, indicating the degradation of meaning preservation.[To be noted, the decrease of semantic similarity score is much smaller for the NLI datasets. This is because the lengths of NLI sentences are generally less than 20 tokens, and the USE encoder is not as sensitive to the change of semantic meaning in this situation.]








 §.§ Transferability


We examined the transferability of adversarial text, that is, whether adversarial samples curated based on one model can also fool another. For this, we collected the adversarial examples from IMDB and SNLI test sets that are wrongly predicted by one target model and then measured the prediction accuracy of them against the other two target models. As we can see from the results in Table <ref>, there is a moderate degree of transferability between models, and the transferability is higher in the textual entailment task than in the text classification task. Moreover, the adversarial samples generated based on the model with higher prediction accuracy, i.e., the BERT model here, show higher transferability.




















 §.§ Adversarial Training




Our work casts insights on how to better improve the original models through these adversarial examples. We conducted a preliminary experiment on adversarial training, by feeding the models both the original data and the adversarial examples (adversarial examples share the same labels as the original counterparts), to see whether the original models can gain more robustness. We collected the adversarial examples curated from the MR and SNLI training sets that fooled BERT and added them to the original training set. We then used the expanded data to train BERT from scratch and attacked this adversarially-trained model. As is seen in the attack results in Table <ref>, both the after-attack accuracy and perturbed words ratio after adversarial re-training get higher, indicating the greater difficulty to attack. This reveals one of the potencies of our attack system,— we can enhance the robustness of a model to future attacks by training it with the generated adversarial examples.







 §.§ Error Analysis

Our adversarial samples are susceptible to three types of errors: word sense ambiguity, grammatical error, and task-sensitive content shift. Although large thesauri are available, a word usually has many meanings, with a set of synonyms for each word sense. 
One example can be the transfer from an original sentence “One man shows the ransom money to the other” to the synthesized “One man testify the ransom money to the other”, where “testify” in this case is not the appropriate synonym of “show”.
Grammatical errors are also frequent in text generation. For example, the sentence “A man with headphones is biking” and “A man with headphones is motorcycle” differ by the word “biking”, which can be both a noun and a verb, as well as a reasonably similar word to “motorcycle”. 


As future work, some carefully designed heuristics can be applied to filter out grammatical errors.
The content shift can be seen in a task-specific situation. In the sentiment classification task, a change of words might not affect the overall sentiment, whereas in the task of textual entailment, the substitution of words might result in a fundamental difference. For example, if the premise is “a kid with red hat is running”, and the original hypothesis is “a kid is running (entailment)”, then if the adversarial example becomes “a girl is running”, the sensible result turns into neutral instead.



§ RELATED WORK

Adversarial attack has been extensively studied in computer vision <cit.>.
Most works make gradient-based perturbation on continuous input spaces <cit.>.
Adversarial attack on discrete data such as text is more challenging. Inspired by the approaches in computer vision, early work in language adversarial attack focus on variations of gradient-based methods. For example, <cit.> transform input data into a latent representation by generative adversarial networks (GANs), and then retrieved adversaries close to the original instance in the latent space. 
Other works observed the intractability of GAN-based models on the text and the shift in semantics in the latent representations, so heuristic methods such as scrambling, misspelling, or removing words were proposed <cit.>. Instead of manually designing these rules, <cit.> automatically craft the semantically equivalent adversarial rules from the machine-generated paraphrases based on the back-translation technique from the machine translation field. By human evaluation, they demonstrate that the quality of the adversaries created by their rules is better than that generated by humans. More interestingly, <cit.> used the way to produce adversarial examples to help improve the interpretability of neural models by encouraging high entropy outputs on the adversaries. 




§ CONCLUSION

Overall, we study adversarial attacks against state-of-the-art text classification and textual entailment models under the black-box setting. Extensive experiments demonstrate the effectiveness of our proposed system, TextFooler, at generating targeted adversarial texts. Human studies validated that the generated adversarial texts are legible, grammatical, and similar in meaning to the original texts.



§ ACKNOWLEDGMENTS

We thank the MIT Clinical Decision-Making Group and Professor Zheng Zhang for insightful discussions. We especially appreciate Heather Berlin, Yilun Du, Laura Koemmpel, and many other helpers for their high-quality human evaluation. Zhijing Jin also thank Zhutian Yang and Yixuan Zhang for strong personal support.

aaai





