Adversarial examples have been shown to be an effective way of assessing the robustness of neural sequence-to-sequence (seq2seq) models, by applying perturbations to the input of a model leading to large degradation in performance. However, these perturbations are only indicative of a weakness in the model if they do not change the semantics of the input in a way that would change the expected output. Using the example of machine translation (MT), we propose a new evaluation framework for adversarial attacks on seq2seq models taking meaning preservation into account and demonstrate that existing methods may not preserve meaning in general. Based on these findings, we propose new constraints for attacks on word-based MT systems and show, via human and automatic evaluation, that they produce more semantically similar adversarial inputs. Furthermore, we show that performing adversarial training with meaning-preserving attacks is beneficial to the model in terms of adversarial robustness without hurting test performance. Attacking a machine learning model with adversarial perturbations is the process of making changes to its input to maximize an adversarial goal, such as mis-classification BID34 or mistranslation BID37 . These attacks provide insight into the vulnerabilities of machine learning models and their brittleness to samples outside the training distribution. This is critical for systems sensitive to safety or security, e.g. self-driving cars BID1 .Adversarial attacks were first defined and investigated for computer vision systems BID34 ; BID9 ; BID22 inter alia), where they benefit from the fact that images are expressed in continuous space, making minuscule perturbations largely imperceptible to the human eye. In discrete spaces such as natural language sentences, the situation is more problematic; even a flip of a single word or character is generally perceptible by a human reader. Thus, most of the mathematical framework in previous work is not directly applicable to discrete text data. Moreover, there is no canonical distance metric for textual data like the 2 norm in real-valued vector spaces such as images, and evaluating the level of semantic similarity between two sentences is a field of research of its own BID2 . This elicits a natural question: what does the term "adversarial perturbation" mean in the context of natural language processing (NLP)?We propose a simple but natural criterion for adversarial examples in NLP, particularly seq2seq models: adversarial examples should be meaning-preserving on the source side, but meaning-destroying on the target side. The focus on explicitly evaluating meaning preservation is in contrast to previous work on adversarial examples for seq2seq models BID0 BID37 BID4 BID7 . Nonetheless, this feature is extremely important; given two sentences with equivalent meaning, we would expect a good model to produce two outputs with equivalent meaning.In other words, any meaning-preserving perturbation that results in the model output changing drastically highlights a fault of the model.A first technical contribution of the paper is to lay out a method for formalizing this concept of meaning-preserving perturbations ( §2). This makes it possible to evaluate the effectiveness of adversarial attacks or defenses either using gold-standard human evaluation, or approximations that can be calculated without human intervention. We further propose a simple method of imbuing gradient-based word substitution attacks ( §3.1) with simple constraints aimed at increasing the chance that the meaning is preserved ( §3.2).Our experiments are designed to answer several questions about meaning preservation in seq2seq models. First, we evaluate our proposed "source-meaning-preserving, target-meaning-destroying" criterion for adversarial examples using both manual and automatic evaluation ( §4.2) and find that a less widely used evaluation metric (chrF) provides significantly better correlation with human judgments than the more widely used BLEU and METEOR metrics. We proceed to perform an evaluation of adversarial example generation techniques, finding that constrained substitution attacks do preserve meaning to a higher degree than unconstrained attacks while still degrading the performance of the systems across different languages and model architectures ( §4.3). Finally we apply existing methods for adversarial training to the adversarial examples with these constraints and show that making adversarial inputs more semantically similar to the source is beneficial for robustness to adversarial attacks and does not decrease test performance ( §5). This paper highlights the performance of meaning-preserving adversarial perturbations for NLP models (with a focus on seq2seq). We proposed a general evaluation framework for adversarial perturbations and compared various automatic metrics as alternatives to human judgment to instantiate this framework. We then confirmed that, in the context of MT, "naive" attacks do not preserve meaning in general, and proposed alternatives to remedy this issue. Finally, we have shown the utility of adversarial training in this paradigm. We hope that this helps future work in this area of research to evaluate meaning conservation more consistently.
How you should evaluate adversarial attacks on seq2seq The authors investigate ways of generating adversarial examples, showing that adversarial training with the attack most consistent with the introduced meaning-preservation criteria results in improved robustness to this type of attack without degradation in the non-adversarial setting. The paper is about meaning-preserving adversarial perturbations in the context of Seq2Seq models