# Bridging the Language Gap: Knowledge Injected Multilingual Question
Answering

1st Zhichao Duan
3rd Zhengyan Zhang
5th Ning Liu  Department of Computer Science & Technology
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>Department of Computer Science and Technology
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>School of Software
Shandong University
Jinan, China
<EMAIL_ADDRESS>2nd Xiuxing Li
4th Zhenyu Li
6th Jianyong Wang  Department of Computer Science & Technology
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>Department of Computer Science and Technology
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>Department of Computer Science and Technology
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>
###### Abstract

Question Answering (QA) is the task of automatically answering questions posed
by humans in natural languages. There are different settings to answer a
question, such as abstractive, extractive, boolean, and multiple-choice QA. As
a popular topic in natural language processing tasks, extractive question
answering task (extractive QA) has gained extensive attention in the past few
years. With the continuous evolvement of the world, generalized cross-lingual
transfer (G-XLT), where question and answer context are in different
languages, poses some unique challenges over cross-lingual transfer (XLT),
where question and answer context are in the same language. With the boost of
corresponding development of related benchmarks, many works have been done to
improve the performance of various language QA tasks. However, only a few
works are dedicated to the G-XLT task. In this work, we propose a generalized
cross-lingual transfer framework to enhance the model’s ability to understand
different languages. Specifically, we first assemble triples from different
languages to form multilingual knowledge. Since the lack of knowledge between
different languages greatly limits models’ reasoning ability, we further
design a knowledge injection strategy via leveraging link prediction
techniques to enrich the model storage of multilingual knowledge. In this way,
we can profoundly exploit rich semantic knowledge. Experiment results on real-
world datasets MLQA demonstrate that the proposed method can improve the
performance by a large margin, outperforming the baseline method by
13.18%/12.00% F1/EM on average. Our code and models will be open-sourced.

###### Index Terms:

Cross-lingual Question Answering, Multilingual Pretrained Language Model

##  I Introduction

Extractive question answering task (extractive QA) is to find the answer span
in the given context according to the problem, which is potentially a
challenging task that can drive the development of methods for Natural
Language Processing (NLP). Modern QA algorithms use Pre-trained Language
Models (PLMs) [1, 2, 3, 4, 5] to infer the answer span. PLMs can capture
different levels of features and generate informative representations by
leveraging unsupervised training objectives and enormous training text. They
have demonstrated powerful performance on many NLP tasks like sentiment
classification [5] and relation extraction [6].

The essence of QA task is to represent the question and context precisely [8].
Hence, how to improve the representation ability of the model has become a
critical step to address the problem. Recent works mainly focus on English QA
problems, such as SQuAD [7, 9] and cross-lingual QA task (XLT) [10]. Some
other works only view QA task as an indicator to evaluate the model
performance [11, 12, 13, 14]. Under these circumstances, data can be plenty
for high-resource languages like English, Chinese, German, Spanish, and
French. Still, both collection and proliferation of data are insufficient for
low-resource languages like Urdu. Even when a large language model is pre-
trained on large amounts of multilingual data, languages like English can
contain orders of magnitude more data in common sources for pretraining like
Wikipedia.

Figure 1: A question-answering sample where context is in English while
question is in Chinese and the reverse. Answer text is extracted from the
passage according to the question.

Therefore, the mainstream approaches to tackle the English QA problem is to
improve the representation ability of the model through introducing high-
resource languages.

However, the generalized cross-lingual QA task (G-XLT), where problem and
context are in different language representations, remains an open problem due
to its complexity. Compared to XLT, G-XLT poses some unique challenges.
Specifically, the latter task requires the QA model to align different
representations automatically, understand the question, reason over the
context, and conclude the final answer. This problem can not be solved simply
by a large PLM with knowledge in each language individually but without
understanding different languages and their connections. The QA model needs to
be aware of different representations of the same word. For example, Fig. 1
gives a demonstration of cross-lingual QA existing in MLQA benchmark [15], as
humans, we can only find the answer when we understand the question, and so do
PLMs. Surely we can first translate the question into English and then find
the answer span [15], but this method normally uses another PLM to handle the
translation process and causes error accumulation.

There are some works aiming to enhance the awareness of different entities in
PLMs to boost performance in monolingual NLP tasks, which are usually English
[16, 17, 18, 19, 20]. Reference [21] uses extra layers called adapters to
learn the knowledge introduced by each pretraining task and improves the
performance of PLMs on multiple downstream tasks. Methods like [22, 23]
explore different ways to take advantage of texts that contain rich knowledge.
Reference [22] firstly transfers knowledge triples into text and then utilizes
knowledge-enriched text to finetune PLMs further. Replacing detection and
entity restoring are studied in [23]. Rather than injecting new knowledge,
some works try to regularize the embedding space and thus benefit downstream
tasks. Reference [20] proposes a method that adopts a siamese network to make
the output embedding space more consistent. Then they build an analogy dataset
and use extra training objectives to ensure that word representations are
globally consistent. Instead of using analogy training, [24] uses Wikipedia
entities to ground representations of different forms of entities (same entity
in different languages). In general, they either design entity-related
auxiliary objectives or use corpus that contains rich semantic knowledge to
finetune PLMs. These methods can be applied to inject mono-lingual knowledge
into PLMs. However, without the alignment information between different
languages, it is hard for the model to realize the various representations of
a single word or sentence and better solve the G-XLT task. Unfortunately, it
is also quite expensive to build multilingual aligned corpus. Besides, they
only use entities that have semantically related sentences and dumped those
entities without related sentences. These methods highly limited the
application of aligned resources. When it comes to cross-lingual QA task, we
can’t simply use any of these methods to enhance the performance due to
insufficient data and applicable conditions.

To solve the aforementioned problems, we propose a novel G-XLT framework.
Firstly, we mix triples from different languages to represent multilingual
knowledge. In this step, we also assemble monolingual triples to avoid
catastrophic forgetting. In the next, we adopt link prediction techniques to
inject multilingual knowledge into the PLMs. In this way, we can enrich the
knowledge stored in each language and smooth the knowledge transfer between
different languages. Meanwhile, the reasoning ability of our model can be
improved through link prediction. Finally, our framework is tested on MLQA
[15], a cross-lingual extractive QA benchmark. Results show that our method
can outperform the baseline method by over 10% improvement on average in
tested cases.

Our contributions are listed as follows:

  * •

We propose a generalized cross-lingual transfer framework which can better
understand different languages by exploiting multilingual knowledge bases.

  * •

We design a knowledge injection technique using link prediction to enrich the
model storage of multilingual knowledge and therefore enhance performance in
G-XLT task.

  * •

We demonstrate our results and prove our method can not only promote the
knowledge transfer between different languages but also enhance the reasoning
ability in G-XLT task.

Figure 2: The overall structure of our method. We demonstrate the case where
the tail entities are masked in all three samples. Note that when we actually
assemble training triples, we separately mask out head entity and tail entity.
Sample 1 is in English and sample 2 is in Chinese and sample 3 is mixed with
English and Chinese. 3 samples are feed as input to the PLMs and the target
predictions are two representations of the entity ”Basketball Player”. Note
that we only limit the output of [MASK]] tokens. After the knowledge injection
step, the PLM is finetuned on the English data in MLQA and finally will be
used to make predictions based on G-XLT data.

##  II Related Work

With the wide popularization of PLMs, many modern QA algorithms use PLMs to
encode question and context and conduct classification based on extra layers.
In our work, we follow this procedure. Therefore, we will introduce two parts
in this section which are multilingual question answering task along with its
two categories and pretrained language models.

###  II-A Multilingual question answering

QA is a hot topic in NLP area. There are massive datasets available, and these
datasets pose different challenges to QA algorithms [7, 29, 30]. One of these
challenges is to find the answer span when the context and the problem are in
different languages, and this task is called generalized cross-lingual
transfer(G-XLT). More formally, cross-lingual transfer problem requires a
model to identify answer $a_{x}$ in context $c_{x}$ according to problem
$q_{x}$ where $x$ is the language used. Meanwhile, generalized cross-lingual
transfer requires a model to find the answer span $a_{z}$ in context $c_{z}$
according to question $q_{y}$ where $z$ and $y$ are languages used. Reference
[15] uses a designed pipeline to construct a large parallel multilingual QA
dataset that contains 7 languages. With the rapid development of PLMs and
remarkable success in various fields of natural language processing, adopting
PLMs to encode input text and using extra layers to conduct classification has
gradually become a popular approach to solve multilingual QA task. Extensive
experiments using state-of-the-art PLMs are reported in [15]. Their results
show that there are quite large room for improvement on both XLT and G-XLT
tasks. One of the major challenges in G-XLT task is to enhance the connection
between different languages. Reference [24] uses hyperlink prediction task to
firstly enhance the connection between different representations and then
finetune the model on the target task data in English. Finally, they evaluate
the performance on all languages. It is worth noting that this method
creatively uses hyperlink as the bridge of different languages.

###  II-B Pretrained Language Models

PLMs such as BERT [1], have gained great success in various downstream tasks.
They apply unsupervised training objective to make full use of the huge text
resources. Typically, they use stacked transformers as model structure and
randomly mask out some tokens in each sample. The goal of PLMs is to restore
the original text. Later on, some new mask strategies and more complex models
were proposed to improve the performance on many tasks. PLMs contain rich
world knowledge, which could be used as an important complement to highly
limited knowledge bases. Some works have been done to use PLMs to do link
prediction task [26, 27]. Knowledge Graph (KG) consists of entities $E$,
relations $R$ and triplets $(h,r,t)$ where $h$ and $t$ are in entity set $E$
and $r$ is in relation set $R$. Link prediction is to find the appropriate
entity that corresponds to another entity linked by a specific relation. That
is to say, complete $(h,r,?)$ or $(?,r,t)$.

PLMs contain meaningful representations of different entities and relations.
Reference [27] explores leveraging knowledge inside PLMs on link prediction
tasks. Reference [28] uses PLMs to generate entity and relation embeddings and
continuously improves them using MLM objective. They train BERT as a
classifier to examine whether a specific triple is correct or not. That is to
say, BERT will check each possible $(?,r,t)$ and $(h,r,?)$ and determine the
final matching entity. This method costs tremendous time and resources due to
its traversal operation. Later on, [26] proposes to use several [MASK] tokens
to represent the missing entity and rank all possible candidates at once to
reduce the time consumption in both training and testing. Their work proves
that PLMs have the ability to infer missing entities and we adopt this method
to inject knowledge into PLMs.

##  III Proposed Method

We propose a new G-XLT framework through enhancing the reasoning abilities and
the understanding of different languages of PLMs. The entire method consists
of three parts: (1) cross-lingual triple assembling; (2) multilingual
knowledge injection; (3) final finetuning on English data. The overall
architecture is shown in Fig. 2.

###  III-A Cross-lingual triple assembling

The information of our world can be represented in the form of knowledge
triples. As mentioned before, we can use $(h,r,t)$ to represent a specific
triple where $h$ and $t$ belong to entity set $E$ and $r$ belongs to relation
set $R$. Here, we extend this concept into multilingual cases. We use $E$ and
$R$ to represent entity sets and relation sets separately, which include
entities and relations in different languages. We use $(h_{i},r_{i},t_{i})$ to
represent a triple in language $i$. We sampled 20,000 triples in Wikidata. 7
languages are considered in this work due to our benchmark MLQA only has data
in 7 languages. Three types of triples are built in this phase. The first kind
of triples are $(h_{i},r_{i},t_{i})$ like ones where head entity, tail entity
and relation are from the same language. For example, as shown in Fig. 2,
“(Keven Durant, is a, Basketball Player)” and “(凯文杜兰特，是，篮球运动员)” are two types
of representations of the same triple and are assembled in this stage.
$(h_{j},r_{i},t_{i})$ and $(h_{i},r_{i},t_{j})$ are the second kind of triples
where head entity or tail entity are in another language. We can combine the
previous samples and form mixed data like “(凯文杜兰特，是，Basketball Player).”
$(h_{i},r_{i},t_{i},h_{j},r_{j},t_{j})$ is the third kind of triples where two
representations are concatenated to form one line of data like “(Keven Durant,
is a, Basketball Player, 凯文杜兰特，是，篮球运动员)”. These three kinds of triples are
randomly assembled and used to perform link prediction in the next step.

Figure 3: Illustration of how we finetuning PLMs on QA data. Here we put
question in the left side and put context in the right. Segment embeddings are
used to distinguish between question and context.

###  III-B Knowledge Injection

After generating the knowledge triples, we design a knowledge injection
strategy via link prediction techniques to inject multilingual knowledge and
align different representations. As we mentioned before, one of many reasons
PLMs achieved great success is that they use unsupervised training objective
to optimize the entire model. To be more specific, they use masked language
modeling method to build PLMs based on large unlabeled text. Masked language
modeling usually predicts the missing tokens based on the given ones by
maximizing the log-likelihood:

$\mathcal{L}_{mlm}(\theta)=\sum\limits_{i=1}^{n}m_{i}logp(x_{i}|\boldsymbol{x}{\circ}\boldsymbol{m};\boldsymbol{\theta})$
(1)

$\boldsymbol{x}$ is the sequence of tokens derived from a sentence, $\circ$ is
the operation of element-wise production, $\boldsymbol{m}$ is an indicator
that shows which tokens in the sequence are replaced with [MASK] tokens and
$\theta$ is the parameters of the model.

Here, a key part of our method is that we use PLMs to predict the missing
entities when we mask out the head entities or the tail entities of triples.
That is to say, the goal of our model is to find the matching entity given
relation and another entity. Take the example in Fig.2, the [MASK] token
represents the missing entity “Basketball Player” and its Chinese version
“篮球运动员” and the goal of our model is to restore those entities.

Note that masking strategy varies according to the triple type. We mask head
entity and tail entity respectively in the first kind of triples. For the
first kind of triples $(h_{i},r_{i},t_{i})$, two masked triples
$(h_{i},r_{i},?)$ and $(?,r_{i},t_{i})$ are formed and the goal of our model
is to restore $t_{i}$ and $h_{i}$ respectively. Only one language appears in
each sample of the first kind, which mimics the training procedure of the
original masked language modeling. In this way, we can not only avoid the
catastrophic forget [32], but also enhance the reasoning ability through link
prediction. For triple $(h_{j},r_{i},t_{i})$ and triple $(h_{i},r_{i},t_{j})$
where the head entity and the tail entity are represented in language $j$
respectively and the rest is represented in language $i$, we form two masked
triple $(?,r_{i},t_{i})$ and $(h_{i},r_{i},?)$ accordingly. The goal of our
model is to restore $h_{j}$ and $t_{j}$. For a triple
$(h_{i},r_{i},t_{i},h_{j},r_{j},t_{j})$ where two forms of the same triple are
concatenated to form one triple, we form two masked triples
$(?,r_{i},t_{i},?,r_{j},t_{j})$ and $(h_{i},r_{i},?,h_{j},r_{j},?)$. The goal
of this step and the previous one is to align different representations of the
same entity and make the representation more consistent in the feature space.
Furthermore, we mix different languages in each sample of the second and the
third kind of triples. By explicitly regularizing the representations, the
model can better leverage the rich knowledge stored in high-resource language
and internalize it into model parameters.

The entity completion loss is defined as:

$\mathcal{L}_{ec}(\theta)=\sum\limits_{i=1}^{n}logp(e|e^{\prime},r;\boldsymbol{\theta})$
(2)

where $e$ is the missing entity, $e^{\prime}$ is the given entity and r is the
relation, and $\theta$ is the model parameters.

###  III-C Finetune on MLQA English data

After the intermediate step above, we will finetune PLM to make it more
suitable for QA task because the different form of QA task and MLM task. This
step is the most adopted approach to finetune PLM on QA task [1] and is shown
in Fig. 3.

Basically, we concatenate question and the context together and add
segmentation embedding to two parts respectively. A special [SEP] token is
added between question and the context to help the model distinguish the two
parts. Then, tokenized samples are feedforward through PLM, and the output in
each position is a vector. We use extra layers on top of PLM to determine the
start span and end span of the final answer.

Figure 4: A case study for our method. The goal of our model is to find the
answer text according to the problem and the question. XLM-R-Base is the
baseline method we use for comparison. Characters marked in the same color
have the same meaning and underlined words are the answer text.

##  IV Experiments

###  IV-A Experiment Setup

Dataset We evaluate our model’s performance on MLQA [15], a highly-parallel
multilingual extractive QA dataset. MLQA has QA samples in 7 languages
including English, Arabic, German, Spanish, Hindi, Vietnamese and Simplified
Chinese. This dataset has 12,738 extractive QA instances in English and over
5K in target language. 9,019 instances are 4-way parallel, 2,930 are 3-way
parallel and 789 are 2-way parallel. MLQA is constructed from Wikipedia
articles which have similar meaning in parallel languages. Labeling work is
done by crowd-sourcing and translation is done by professional translators. In
total, there are over 46,000 QA instances. MLQA provides data for G-XLT
experiments and the dev/test split is already given.

Baseline Methods Our method is built on top of XLM-R. It is proposed in [4]
and is based on the idea that pre-training multilingual language models at
scale lead to significant performance improvement for a wide range of cross-
lingual transfer tasks. XLM-R is trained on 2 terabytes of crawled data and
covers over 100 languages. Compared to other multilingual models, it does not
require language type tensors in the input. Using a relatively large
vocabulary, XLM-R can detect the correct language from the input ids. Since
the goal of our experiments is to demonstrate the effectiveness, we use a
smaller version of XLM-R, XLM-R-Base.

Evaluation Metrics The answer-preprocessing operations adopted in SQuAD [7, 9]
are targeting English QA evaluation. We use methods proposed in [15] to
preprocess data for a fair comparison. We use Exact Match (EM) and mean token
F1 score as performance metrics like other literature.

Implementation Details There are some multilingual PLMs like [4, 3, 33]. For
fast experiment, XLM-R-Base [4] is used as the encoder. Our method is
implemented using Pytorch and Huggingface’s Transformers library[31]. which is
optimized by AdamW [34]. In the knowledge injection phase, linear warmup
strategy [35] is adopted for the first 6$\%$ steps and learning rate is set to
2e-5. Batch size is set to 24 and only 1 training epoch. After the knowledge
injection step, we still use AdamW to finetune XLM-R-Base except that the
learning rate is set to 3e-5 and the batch size is set to 16 with 2 training
epochs. All the above training is done in a single RTX 3090 GPU.

###  IV-B Main Results

Since English data constituted the largest corpus among all the training data,
we assume XLM-R-Base understands English better than other languages. In the
QA task, we concatenate question and context together as input and feed it to
the PLM. PLMs can accurately encode text to a certain extent. As the length of
text increases, the precision of encoding will drop. In order to see the
effects our method has, we only compare performance when the context is
English since context is usually much longer than the question, and results in
[4] prove that XLM-R understands English better than any other language.

TABLE I: Experimental results on English-context cases. Settings(c/q) | Method | F1 | Exact Match
---|---|---|---
EN/DE | xlm-roberta-base | 60.30 | 47.73
ours | 63.10 | 50.34
EN/ES | xlm-roberta-base | 57.01 | 43.02
ours | 57.98 | 44.32
EN/AR | xlm-roberta-base | 30.71 | 19.78
ours | 49.19 | 36.10
EN/HI | xlm-roberta-base | 40.22 | 28.73
ours | 58.16 | 45.91
EN/VI | xlm-roberta-base | 41.90 | 30.50
ours | 52.03 | 39.20
EN/ZH | xlm-roberta-base | 31.36 | 20.89
ours | 60.13 | 46.80

Tab. I presents the QA performance of the setting that context is English and
question is changed from German to Chinese. Most obviously, we can observe
that our method shows great improvement over the baseline method in terms of
metrics used. However, the performance gain varies with the languages tested.
To be more specific, among all 6 tested languages, we can see that improvement
on German and Spanish is relatively low. Baseline model XLM-R-Base has the
best performance when the questions are set to German and Spanish. Results on
cross-lingual classification reported in [4] indicate that the performance on
Spanish and German are also the highest except English. We can also see from
Fig.5 that the token coverage of these two languages is only a little
different from the others. Take all these into consideration, and we think the
limited improvement is caused by the deep understanding of German and Spanish
in the baseline PLM. Significantly, in the Arabic and Chinese cases, the
baseline performance is the worst, indicating less knowledge storage and
shallow understanding of the two languages. In the results on cross-lingual
classification in [4], performance on Arabic and Chinese is obviously worse
than in German and Spanish. After injecting knowledge, there are 18.48%
improvement on F1 score and 16.32% improvement on Exact Match in Arabic and
28.77% improvement on F1 score, 25.91% improvement on Exact Match in Chinese.
Dramatic performance gains prove the effectiveness of our method. When it
comes to Hindi and Vietnamese, the same pattern is embodied.

There is 13.18% improvement on F1 score and 12.00% improvement on EM score on
average. This indicates that using knowledge triples to enhance connections
between different languages leads to significant performance gain in QA task.
The improvement varies with the language tested since XLM-R-Base uses shared
vocabulary.

Figure 5: We first tokenize question text and knowledge triples in different
languages. Then we calculate the coverage of tokens that appear in both
questions and knowledge triples. That is to say, we calculate how much common
tokens appear in tokenized triples. The results are visualized in this graph.

###  IV-C Case Studies

Fig. 4 shows three example prediction cases of our model and the baseline
model. In the first example, the question means “At this time, what finishing
maneuver did he use?”. The special noun mentioned in the question is
apparently a low-frequency term. The model has first to realize that “摔角终结技”
means the same as “finishing maneuver”. Then it needs to deal with the second
low-frequency word “Pedigree”. In the first example, the answer can be
inferred from the sentence structure. That is to say, even without knowing the
pedigree is a finishing maneuver, we can infer this is the answer. The
baseline method predicts no answer, which is predictable since the term is
rarely seen. Our method predicts almost the correct answer except definite
article “the”. This case indicates that after injecting knowledge and aligning
representations, the reasoning ability of PLM is highly enhanced. The second
example is a medical-related problem. In this case, “慢性支气管炎” means “chronic
cough”. However, the sentence containing the answer information does not
include the entire term. It uses “cough” to represent “chronic cough”. Again,
the baseline method fails to find the correct answer, but our method manages
to find a partial answer. This case proves that the improvement brought by our
method is general, and our method can greatly enhance the ability to map
different languages and reason. In the third example, a proper translation of
the question is “Professionals do not know whether there are different types
of what?”. It’s quite clear that this question and this context are easier
than the previous two. Both our method and the baseline method find the
correct answer, which indicates that the improvement brought by our method is
stable.

##  V Future Work

Multilingual wikidata triples are leveraged as a knowledge source in our work.
By using link prediction technique, we can successfully inject external
knowledge into PLMs and enhance the performance on G-XLT task. However, the
results on cases where context is not English remain basically unchanged or
even slightly drop. This phenomenon may be caused by the different storage of
knowledge in each language. The challenge to better represent an entire
context in low-resource languages remains unsolved. For future work, we will
explore the following research directions: (1) In this work, we only study how
to inject knowledge of two languages at once into PLMs. We will expand the
capability of the proposed method via integrating more languages which is more
practical. With the increasing of languages used, new challenges might appear,
like balancing the effects of different languages and how to bring new
knowledge into PLMs effectively. (2) Each triple is viewed as an individual
source of knowledge, and only knowledge bases are considered in our work. No
relation between them or other knowledge sources are explored. In the future,
We will also add more information in our joint model, such as relation paths
in the knowledge base and auxiliary knowledge corpus. (3) We will try to use
limited resources to solve the G-XLT problem better when the context is set to
low-resource languages. (4) As listed in Fig.5, we can see the token coverage
is at least over 50%. If we can achieve almost the same performance gain with
less data, out method can be more effective. Therefore, we will try to explore
the relation between data coverage and the final performance gains.

##  VI Conclusion

In this paper, we introduced a new G-XLT framework that fully exploited
external knowledge bases. Our framework consists of two parts: (1) assembling
triples and (2) knowledge injection. We train our model with triples assembled
from two languages at once. We show that it brings tremendous performance
gains on English-context cases in MLQA. Further analysis shows that our method
enriched the knowledge stored in each language and improved the reasoning
ability of the baseline model. Our method consistently outperforms the
baseline model on both simple and tough examples, which verifies the
generality and stability of our model.

## Acknowledgments

This work was supported in part by National Natural Science Foundation of
China under Grant No. 61532010 and Beijing Academy of Artificial Intelligence
(BAAI).

## References

  * [1] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprintarXiv:1810.04805, 2018
  * [2] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Al-bert: A lite bert for self-supervised learning of language representations,”2020.
  * [3] A. Conneau and G. Lample, “Cross-lingual language model pretraining,”Advances in Neural Information Processing Systems, vol. 32, pp. 7059–7069,2019.
  * [4] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek,F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Un-supervised cross-lingual representation learning at scale,”arXiv preprintarXiv:1911.02116, 2019
  * [5] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis,L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pre-training approach,”arXiv preprint arXiv:1907.11692, 2019.
  * [6] X. Chen, X. Xie, N. Zhang, J. Yan, S. Deng, C. Tan, F. Huang, L. Si,and H. Chen, “Adaprompt: Adaptive prompt-based finetuning for relation extraction,”arXiv preprint arXiv:2104.07650, 2021
  * [7] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ ques-tions for machine comprehension of text,”arXiv preprint arXiv:1606.05250,2016.
  * [8] Y. Seonwoo, J.-H. Kim, J.-W. Ha, and A. Oh, “Context-aware answer extraction in question answering,” 20
  * [9] P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” 2018
  * [10] J. Liu, Y. Lin, Z. Liu, and M. Sun, “Xqa: A cross-lingual open-domain question answering dataset,” inProceedings of the 57th Annual Meeting ofthe Association for Computational Linguistics, 2019, pp. 2358–2368
  * [11] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou,W. Li, and P. J. Liu, “Exploring the limits of transfer learning with aunified text-to-text transformer,”arXiv preprint arXiv:1910.10683, 2019
  * [12] I. Yamada, A. Asai, H. Shindo, H. Takeda, and Y. Matsumoto, “Luke: deepcontextualized entity representations with entity-aware self-attention,”arXiv preprint arXiv:2010.01057, 2020
  * [13] X. Li, X. Sun, Y. Meng, J. Liang, F. Wu, and J. Li, “Dice loss for data-imbalanced nlp tasks,”arXiv preprint arXiv:1911.02855, 2019.
  * [14] M. Hu, Y. Peng, Z. Huang, X. Qiu, F. Wei, and M. Zhou, “Reinforcedmnemonic reader for machine reading comprehension,”arXiv preprintarXiv:1705.02798, 2017.
  * [15] P. Lewis, B. Oğuz, R. Rinott, S. Riedel, and H. Schwenk, “Mlqa:Evaluating cross-lingual extractive question answering,”arXiv preprintarXiv:1910.07475, 2019.
  * [16] Z. Zhang, X. Han, Z. Liu, X. Jiang, M. Sun, and Q. Liu, “Ernie: En-hanced language representation with informative entities,”arXiv preprintarXiv:1905.07129, 2019.
  * [17] A. Lauscher, I. Vulic, E. M. Ponti, A. Korhonen, and G. Glavaˇs, “Informing unsupervised pretraining with external linguistic knowledge,”
  * [18] Y. Levine, B. Lenz, O. Dagan, O. Ram, D. Padnos, O. Sharir, S. Shalev-Shwartz, A. Shashua, and Y. Shoham, “Sensebert: Driving some sense intobert,”arXiv preprint arXiv:1908.05646, 2019.
  * [19] M. E. Peters, M. Neumann, R. L. Logan IV, R. Schwartz, V. Joshi, S. Singh,and N. A. Smith, “Knowledge enhanced contextual word representations,”arXiv preprint arXiv:1909.04164, 2019.
  * [20] N. Garneau, M. Hartmann, A. Sandholm, S. Ruder, I. Vulic, andA. Søgaard, “Analogy training multilingual encoders,” inProceedings ofthe AAAI Conference on Artificial Intelligence, vol. 35, no. 14, 2021, pp.12 884–12 892
  * [21] R. Wang, D. Tang, N. Duan, Z. Wei, X. Huang, G. Cao, D. Jiang,M. Zhouet al., “K-adapter: Infusing knowledge into pre-trained modelswith adapters,”arXiv preprint arXiv:2002.01808, 2020.
  * [22] O. Agarwal, H. Ge, S. Shakeri, and R. Al-Rfou, “Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training,”arXiv preprint arXiv:2010.12688, 2020.
  * [23] B. He, X. Jiang, J. Xiao, and Q. Liu, “Kgplm: Knowledge-guided lan-guage model pre-training via generative and discriminative learning,”arXivpreprint arXiv:2012.03551, 2020
  * [24] I. Calixto, A. Raganato, and T. Pasini, “Wikipedia entities as rendezvous across languages: Grounding multilingual language models by predicting wikipedia hyperlinks,” inProceedings of the 2021 Conference of the NorthAmerican Chapter of the Association for Computational Linguistics: Hu-man Language Technologies, 2021, pp. 3651–3661
  * [25] Y. Qin, Y. Lin, R. Takanobu, Z. Liu, P. Li, H. Ji, M. Huang, M. Sun,and J. Zhou, “Erica: improving entity and relation understanding for pre-trained language models via contrastive learning,”arXiv preprintarXiv:2012.15022, 2020.
  * [26] L. Clouatre, P. Trempe, A. Zouaq, and S. Chandar, “Mlmlm: Link pre-diction with mean likelihood masked language model,”arXiv preprintarXiv:2009.07058, 2020.
  * [27] L. Yao, C. Mao, and Y. Luo, “Kg-bert: Bert for knowledge graph comple-tion,”arXiv preprint arXiv:1909.03193, 2019.
  * [28] X. Wang, T. Gao, Z. Zhu, Z. Zhang, Z. Liu, J. Li, and J. Tang, “Kepler:A unified model for knowledge embedding and pre-trained language repre-sentation,”Transactions of the Association for Computational Linguistics,vol. 9, pp. 176–194, 2021
  * [29] M. Richardson, C. J. Burges, and E. Renshaw, “Mctest: A challenge dataset for the open-domain machine comprehension of text,” inProceed-ings of the 2013 conference on empirical methods in natural language pro-cessing, 2013, pp. 193–20
  * [30] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Al-berti, D. Epstein, I. Polosukhin, J. Devlin, K. Leeet al., “Natural ques-tions: a benchmark for question answering research,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 453–466, 2019.
  * [31] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac,T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen,C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame,Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art natural language processing,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Online:Association for Computational Linguistics, Oct. 2020, pp. 38–45. [Online].Available: https://www.aclweb.org/anthology/2020.emnlp-dem
  * [32] M. McCloskey and N. J. Cohen, “Catastrophic interference in connectionist networks: The sequential learning problem,” inPsychology of learning and motivation. Elsevier, 1989, vol. 24, pp. 109–1
  * [33] W. Qi, Y. Yan, Y. Gong, D. Liu, N. Duan, J. Chen, R. Zhang, and M. Zhou, “Prophetnet: Predicting future n-gram for sequence-to-sequencepre-training,” 2020
  * [34] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,”arXiv preprint arXiv:1711.05101, 2017.
  * [35] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola,A. Tulloch, Y. Jia, and K. He, “Accurate, large minibatch sgd: Training imagenet in 1 hour,”arXiv preprint arXiv:1706.02677, 2017.


# Enhancing Noise Robustness of Retrieval-Augmented Language Models with
Adaptive Adversarial Training

Feiteng Fang1,2, Yuelin Bai211footnotemark: 1, Shiwen Ni2, Min
Yang222footnotemark: 2, Xiaojun Chen3, Ruifeng Xu4
1University of Science and Technology of China
2Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
3Shenzhen University 4Harbin Institute of Technology (Shenzhen)
<EMAIL_ADDRESS>{yl.bai, sw.ni<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contribution.Corresponding
author.

###### Abstract

Large Language Models (LLMs) exhibit substantial capabilities yet encounter
challenges, including hallucination, outdated knowledge, and untraceable
reasoning processes. Retrieval-augmented generation (RAG) has emerged as a
promising solution, integrating knowledge from external databases to mitigate
these challenges. However, inappropriate retrieved passages can potentially
hinder the LLMs’ capacity to generate comprehensive and high-quality
responses. Prior RAG studies on the robustness of retrieval noises often
confine themselves to a limited set of noise types, deviating from real-world
retrieval environments and limiting practical applicability. In this study, we
initially investigate retrieval noises and categorize them into three distinct
types, reflecting real-world environments. We analyze the impact of these
various retrieval noises on the robustness of LLMs. Subsequently, we propose a
novel RAG approach known as Retrieval-augmented Adaptive Adversarial Training
(RAAT). RAAT leverages adaptive adversarial training to dynamically adjust the
model’s training process in response to retrieval noises. Concurrently, it
employs multi-task learning to ensure the model’s capacity to internally
recognize noisy contexts. Extensive experiments demonstrate that the LLaMA-2
7B model trained using RAAT exhibits significant improvements in F1 and EM
scores under diverse noise conditions. For reproducibility, we release our
code and data at: https://github.com/calubkk/RAAT.

Enhancing Noise Robustness of Retrieval-Augmented Language Models with
Adaptive Adversarial Training

Feiteng Fang1,2††thanks: Equal contribution., Yuelin Bai211footnotemark: 1,
Shiwen Ni2††thanks: Corresponding author., Min Yang222footnotemark: 2, Xiaojun
Chen3, Ruifeng Xu4 1University of Science and Technology of China 2Shenzhen
Institute of Advanced Technology, Chinese Academy of Sciences 3Shenzhen
University 4Harbin Institute of Technology (Shenzhen)
<EMAIL_ADDRESS>{yl.bai, sw.ni<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Figure 1:  An illustrative example of the RAG process applied to question
answering. The model predicts the correct answer with accurate retrieved text.
However, it fails to produce the right answer when the retrieved text contains
misleading or inaccurate information.

Large language models (LLMs) have garnered substantial attention in both
academic and industrial research within the domain of artificial intelligence
due to their remarkable capabilities (Brown et al., 2020; Bubeck et al.,
2023). Despite their immense power, LLMs face challenges such as
hallucinations and outdated knowledge (Gao et al., 2023). Moreover, a lack of
domain knowledge may hinder their performance on domain-specific tasks
(Kandpal et al., 2023). To mitigate these challenges, recent studies improve
LLMs by retrieving passages from external databases and pretending them in
context, constituting a framework known as retrieval-augmented language models
(RALMs) (Mao et al., 2020; Lewis et al., 2020).

However, RALMs also present significant limitations. Previous studies (Yoran
et al., 2023; Yu et al., 2023; Shi et al., 2023) have empirically demonstrated
that retrieved noisy passages are problematic for LLMs, resulting in
performance degradation. We term this issue as the noise robustness problem of
RALMs. As illustrated in Figure 1, the model can provide correct answers when
the retrieving context is accurate and related to the query. However, when the
retrieved context contains misleading or inaccurate information, the model may
yield incorrect answers. As the retriever inherently cannot achieve complete
accuracy, the presence of noise in the retrieved context is inevitable.
Therefore, designing robust algorithms against retrieved noises is of great
practical importance.

Recently, several studies (Yoran et al., 2023; Li et al., 2022) have attempted
to enhance the noise robustness of RALMs through noisy training, which
involves incorporating retrieved noisy contexts into fine-tuning data. While
noisy training exhibits promise, its effectiveness heavily relies on the
composition of the training dataset. Incorrectly introducing noises to the
training data can lead to model overfitting, adversely affecting
generalization. In practice, meticulous adjustment of the type and intensity
of noises is essential to ensure the model’s proficiency across various tasks
and datasets. This demands significant experimentation and tuning, adding
complexity to the development process. Moreover, the lack of clear
classification for retrieval noises in current studies stands in contrast to
the diverse range of noises encountered in real retrieval environments.

This paper systematically explores three types of retrieval noises: (i)
contexts that are superficially related to the query but lack the correct
answer (Relevant retrieval noise), (ii) contexts that are irrelevant to the
query (Irrelevant retrieval noise), and (iii) contexts that are topically
related to the query but contain incorrect information (Counterfactual
retrieval noise). Our empirical study indicates that LLMs exhibit varying
robustness to these three types of noise. Compared to entirely irrelevant
texts, texts that are superficially related to the query or those containing
counterfactual details often lead to more misinformation.

In response to diverse types of noises, we propose a novel approach named
Retrieval-augmented Adaptive Adversarial Training (RAAT), which employs
adaptive adversarial training to dynamically regulate the model’s training
process in response to retrieved noisy texts. Concretely, RAAT generates
adversarial samples (noises) by considering the model’s sensitivity to
different types of noises during training, which aligns with the min-max
paradigm of adversarial training (Morris et al., 2020; Ivgi and Berant, 2021).
Moreover, RAAT utilizes multi-task learning (Ruder, 2017) to encourage the
LLMs to generate tokens that are aware of noises, thereby enabling the model
to internally recognize retrieved noisy contexts and improve the overall
generation performance.

The main contributions of this paper can be summarized as follows:

  * •

We systematically explore three types of retrieval noises and investigate the
sensitivity of LLMs to these diverse types of noises.

  * •

We propose a novel adaptive adversarial training method (called RAAT) to
enhance the robustness of RALMs against various retrieval noises. RAAT
dynamically adjusts the training process of the model in diverse noise
environments. In addition, it integrates multi-task learning to encourage the
model to improve its ability to discern different types of noises.

  * •

We set up a benchmark (named RAG-Bench) for assessing the noise robustness
problem of RALMs based on three open-domain question-answering datasets.
Experimental results demonstrate that our RAAT method enhances robustness
across diverse retrieval noise environments.

##  2 Related Work

#### Retrieval-Augmented Generation with Noisy Context

Retrieval-Augmented Language Models (RALMs) have shown impressive performance
in various NLP tasks (Gao et al., 2023; Zhu et al., 2023). However, limited by
the capabilities of the retriever, retrieval-augmented systems inevitably
introduce irrelevant or partially relevant knowledge to the models (Yin et
al., 2023). Recent studies (Yu et al., 2023; Yoran et al., 2023; Chen et al.,
2023) have increasingly focused on the impact of noisy information on
retrieval-augmented generation. For example, Jia and Liang (2017); Creswell et
al. (2022) observed that adding irrelevant noise to the context could
detrimentally affect model performance. Chen et al. (2023) demonstrated that
as the proportion of noise in the retrieval context increases, the performance
of LLMs experiences a notable decline. Similar phenomena have been reported by
Yoran et al. (2023) and Thakur et al. (2023).

#### Adversarial Training

Adversarial training is recognized as a crucial method for enhancing model
robustness, initially proposed by Goodfellow et al. (2014). Early studies
widely investigated adversarial training in the computer vision domain
(Kurakin et al., 2016; Madry et al., 2017). In the NLP domain, Miyato et al.
(2016) applied perturbations to word embeddings, making the model less prone
to overfitting. Similarly, perturbations on different granularities have been
extensively studied, encompassing various aspects of NLP tasks (Yasunaga et
al., 2017; Wu et al., 2017; Zhu et al., 2019; Wang et al., 2020; Ni et al.,
2023; Liang et al., 2023).

Recently, several studies have concentrated on generating adversarial examples
designed to induce LLMs to generate harmful or non-factual content (Zou et
al., 2023; Shen et al., 2023) instead of merely causing the model to make
inaccurate predictions. Shen et al. (2023) employed decision-based
perturbation at different levels to craft adversarial examples, revealing
vulnerabilities in ChatGPT to both sentence-level and character-level
adversarial attacks. Shi et al. (2023) added irrelevant context to an
arithmetic reasoning dataset, finding that including irrelevant information
distracted the model’s predictions. Zou et al. (2023) proposed a method that
could reliably generate adversarial attack suffixes, yielding adversarial
prompts that exhibit high transferability.

In this study, we investigate adversarial training concerning LLMs in response
to various retrieval noises, aiming to efficiently obtain adversarial examples
that enhance model robustness while reducing training overhead. We construct
noisy adversarial examples by sampling or paraphrasing the original dataset.
This approach ensures more dependable and precise outputs even when confronted
with imperfect retrieved contexts.

##  3 Methodology

###  3.1 Problem Setup

In the standard RALM, given input query $x$, a retriever $r$ is designed to
retrieve relevant contexts $C=\\{c_{1},c_{2},\ldots\\}$ from an external
database. During inference, the content of retrieval context is concatenated
with $x$ to form $d$, which is then fed into the pre-trained language model
$M$, yielding a response $\hat{y}$ regarding $x$. If the retrieved context $c$
contains the correct answer $y$ about $x$, we can denote $c$ as $c_{golden}$,
representing the golden retrieval context. However, if $c$ does not contain
the correct answer $y$ or contains partially irrelevant content, we can denote
$c$ as $c_{noisy}$.

In our study, we transform open-domain question answering (QA) into a reading
comprehension task to meet the RAG settings. Formally, given the objective $f$
of an open domain question answering task is $f:\left\\{x\right\\}\rightarrow
y$, we can formulate the objective of the reading comprehension task as
$f:\left\\{c_{golden},x\right\\}\rightarrow y$. In examining the challenge of
the retrieval noise robustness problem of RALM, we aim to obtain a fine-tuned
model $M^{\prime}$ that can not only fulfill the function
$f:\left\\{c_{golden},x\right\\}\rightarrow y$ but also produce accurate
answers even in the presence of additional retrieval noise $c_{noisy}$ and
achieve function $f:\left\\{c_{golden},c_{noisy},x\right\\}\rightarrow y$.

Figure 2: Exact match (EM) scores of various models under different types of
retrieval noises. “Golden Context” denotes instances where LLMs respond to
questions with reference to the golden retrieval context. “No Noise” indicates
instances where LLMs answer questions without any retrieval. The experimental
configurations of other models involve the introduction of different types of
noises on the foundation of the “Golden Context”.

###  3.2 Diverse Retrieval Noises

We systematically classify the retrieval noise present in $c_{noisy}$ to
closely mimic real-world conditions. Existing studies (Yoran et al., 2023; Yu
et al., 2023) on retrieval noise robustness often dichotomize noise into
relevant and irrelevant categories. However, we contend that such a
classification may not fully align with the retrieval noise robustness of
RALMs. In this work, we propose a more nuanced classification of retrieval
noise, differentiating it into three distinct types: Relevant retrieval noise,
Irrelevant retrieval noise, and Counterfactual retrieval noise. Specifically,
Relevant retrieval noise (denoted as $c_{r}$) pertains to contexts that
exhibit superficial relevance to the query $x$ but lack the information
necessary for the correct answer $y$. These contexts may appear relevant at
first glance but ultimately mislead the model. Irrelevant retrieval noise
(denoted as $c_{i}$) encompasses contexts with low relevance to the query $x$,
often arising from erroneous retrievals and generally being off-topic.
Counterfactual retrieval noise (denoted as $c_{c}$) encompasses contexts that
are topically related to $x$ but contain incorrect and misleading information,
often attributed to inaccuracies in the retriever’s database.

To examine the influence of three distinct types of retrieval noise on LLMs,
we establish a benchmark for assessing retrieval noise robustness in LLMs by
employing three open-domain question-answering datasets: Natural Questions
(Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and WebQ (Berant et
al., 2013). Leveraging this benchmark, we evaluated the susceptibility of
various open-source large language models to the effects of the three
identified types of noise. The details of the construction of this benchmark
can be found in Section 4.1. Leveraging this benchmark, we evaluate the
sensitivity of various LLMs to the effects of the three types of noise.
Specifically, we conduct experiments on six LLMs, including ChatGPT3.5,
LLaMA27B (Touvron et al., 2023), LLaMA213B (Touvron et al., 2023), LLaMA270B
(Touvron et al., 2023), Qwen7B (Bai et al., 2023), and Qwen14B (Bai et al.,
2023). For each model, our experiments encompass two distinct settings: one
with the exclusive presence of the golden retrieval context $c_{golden}$ and
another incorporating the introduction of three different types of retrieval
noise $c_{noisy}$. As shown in Figure 2, all LLMs experience varying degrees
of impact from the three types of noise. The performance of LLMs exhibits a
decline ranging from 0.2% to 13.43%. Through a comparative analysis of the
effects of the three types of noise, we observe that irrelevant retrieval
noise has a comparatively minor impact on LLMs with substantial capabilities.

Figure 3: The overview of our proposed RAAT method, which incorporates three
distinct types of retrieval noises and the golden retrieval context during the
training process.

###  3.3 Retrieval-augmented Adaptive Adversarial Training

Recently, several studies (Yoran et al., 2023; Li et al., 2022) attempted to
enhance the noise robustness of LLMs through noisy training, which involves
incorporating retrieved noisy contexts into fine-tuning data. The essence of
noisy training involves the exploration of offline data augmentation, while in
contrast, adversarial training leverages online data augmentation for a
similar purpose (Ivgi and Berant, 2021). The core idea of adversarial training
is to fortify the models against adversarial conditions by introducing
adversarial perturbations (Jain et al., 2023). In the construction of
adversarial samples, also known as noise samples, the min-max optimization
strategy assumes a pivotal role, encompassing two fundamental steps.
Initially, the maximization process involves adjusting the input data to
intentionally mislead the model, inducing the maximum prediction error. Then,
the minimization process entails fine-tuning the model’s parameters to enhance
its resistance against these meticulously crafted input perturbations (Bai et
al., 2021). This strategy seeks to strike a balance, allowing the model to
accurately identify normal data while robustly defending against potential
attacks from adversarial examples.

In this study, we aim to refine the objective of adversarial training while
exploring the noise robustness challenges of RALMs. Considering a given query
$x$, we assume the existence of four types of data augmentation, namely,
golden retrieval context only ($da_{g}$), additional relevant retrieval noise
($da_{r}$), additional irrelevant retrieval noise ($da_{i}$), and additional
counterfactual retrieval noise ($da_{c}$). The space of data augmentation is
denoted as ${DA}=\\{{da}_{g},{da}_{r},{da}_{i},{da}_{c}\\}$. Then, the
optimization problem can be formulated as follows:

$\min_{\theta}\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\max_{da\in
DA}\mathcal{L}(\theta,da(x),y)\right]$ (1)

where $\mathcal{D}$ denotes training data, $\mathcal{L}$ is the loss function,
$\theta$ denotes the parameters of LLMs, and $da(x)$ represents the data
augmentation of $x$.

Building upon the optimization problem outlined above, we introduce adaptive
adversarial training as a tailored approach to enhance the robustness of RALMs
against retrieval noise. Within adaptive adversarial training, the model
refrains from updating parameters across all adversarial samples. Instead, it
initiates the process by computing the generation loss for each adversarial
sample, quantifying its adaptability to varying noise environments. Notably, a
higher generation loss implies reduced adaptability of the model to the noisy
environment. Given that each query involves one sample with a golden retrieval
context and three adversarial samples, the model generates four distinct
generation losses in each iteration. Following a min-max optimization
strategy, the model prioritizes the selection of the largest loss to guide
subsequent parameter update. Formally, we define the generation loss function
$\mathcal{L}^{\prime}$ for the augmented input $x^{\prime}$ as:

$\mathcal{L}^{\prime}\left(\theta,x^{\prime},y\right)=-\frac{1}{\left|y\right|}\sum_{t=1}^{\left|y\right|}\log
P_{\theta}\left(y_{t}\mid x^{\prime},y_{<t}\right)$ (2)

where $x^{\prime}=da(x)$ represents the augmented noise context of $x$.

To effectively enhance performance across diverse noise environments, adaptive
adversarial training incorporates a regularization term into its loss
function. This incorporation of a regularization term is designed to mitigate
the risk of the model overfitting to a particular type of noise. The
regularization term acts as a stabilizing factor, promoting generalization and
preventing the model from becoming overly specialized in its response to a
specific noise profile. To achieve this goal, we introduce a regularization
term specifically designed to mitigate the variance between these generation
losses. This regularization term operates by identifying the largest
$\mathcal{L}^{\prime}_{max}$ and the smallest $\mathcal{L}^{\prime}_{min}$ of
the four generation losses at each training step. $\mathcal{L}^{\prime}_{max}$
is the generation loss with the highest numerical value among four losses
being considered. Conversely, $\mathcal{L}^{\prime}_{min}$ is the loss
function with the lowest numerical value. Here, an increased loss value
indicates a greater magnitude of error or disparity in the aspect of the
model’s performance being assessed. This suggests that the model exhibits
heightened sensitivity to adversarial examples reflecting retrieval noise.
These adversarial examples are designed to probe and exploit weaknesses in the
model’s processing capabilities, especially in how it deals with noisy
information in its input data. The regularization term, calculated as the
square of the difference between $\mathcal{L}^{\prime}_{max}$ and
$\mathcal{L}^{\prime}_{min}$, aims to reduce the model’s sensitivity to
retrieval noise by encouraging a more balanced optimization. Formally, we
design the regularization term $\mathcal{L}_{\text{reg}}$ as:

$\mathcal{L}_{\text{reg}}=\|\mathcal{L}^{\prime}_{\text{max}}-\mathcal{L}^{\prime}_{\text{min}}\|_{2}^{2}$
(3)

Subsequently, we define the adaptive adversarial training loss function
$\mathcal{L}_{\text{ada}}$ as follows:

$\mathcal{L}_{\text{ada}}=\mathcal{L}^{\prime}_{\text{max}}+w_{\text{\text{reg}}}\cdot\mathcal{L}_{\text{reg}}$
(4)

where $w_{\text{reg}}$ is a pre-defined hyperparameter to control the weight
of $\mathcal{L}_{\text{reg}}$.

###  3.4 Incorporating Noise Awareness

Accurately identifying retrieval noise plays a pivotal role in fortifying the
robustness of RALMs against the retrieval noise. Models endowed with the
ability to discern different types of noise can more effectively choose and
utilize training data, leading to an improvement in the overall quality of
their generated outputs. This capacity to distinguish between various noise
types contributes significantly to the model’s adaptive learning process,
enabling it to optimize performance in the presence of diverse noise
scenarios. Inspired by the above motivation, we propose an auxiliary task
designed to autonomously recognize the types of noisy retrieval texts, aiming
to significantly bolster the retrieval robustness of RALMs. This auxiliary
task serves as a valuable augmentation, contributing to the overall
adaptability and effectiveness of the model in scenarios involving retrieval
noise.

Specifically, we attempt to enable the model to generate tokens that are
sensitive to noise, thereby improving the model’s capacity to discern various
types of retrieval noise internally. Specifically, we first incorporate a
linear layer beneath LLMs. Subsequently, a classification loss
$\mathcal{L}_{\text{cls}}$ is computed for each of the golden retrieval
context and the three adversarial samples corresponding to each input $x$.
One-hot encoding is employed in classification tasks, assigning values from 1
to 4 as labels to train the classifiers, where each classifier is tailored to
a different retrieval noise type. The loss function $\mathcal{L}_{\text{cls}}$
is computed using cross-entropy.

Finally, we formulate the final RAAT loss $\mathcal{L}_{\text{RAAT}}$ by
combining the adaptive adversarial training loss and the classification loss
in the context of multi-task learning:

$\mathcal{L}_{\text{RAAT}}=w_{\text{ada}}\cdot\mathcal{L}_{\text{ada}}+w_{\text{cls}}\cdot\mathcal{L}_{\text{cls}}$
(5)

where $w_{\text{ada}}$ and $w_{\text{cls}}$ represent pre-defined
hyperparameters used to balance the importance of these two different tasks.

##  4 Experiments

###  4.1 Dataset Construction

We have formulated a benchmark named RAG-Bench that is specifically designed
to evaluate the retrieval noise robustness of LLMs. RAG-Bench is established
upon three widely available datasets that center around open-domain question
answering (QA): Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA
(Joshi et al., 2017), and WebQ (Berant et al., 2013). For each dataset, we
employ the retrieval model DPR (Karpukhin et al., 2020) as our retriever,
which retrieves ten passages from Wikipedia for each query. Then, we apply
filtering to the queries, ensuring that each query in the filtered subset
contains at least two golden retrieval contexts, indicating the presence of
correct answers. Detailed statistics for both the full set and the filtered
subset can be found in Table 1.

Each sample in our dataset contains a golden retrieval context and is
deliberately designed to incorporate three types of augmented retrieval noise.
To introduce relevant retrieval noise, we choose the context most pertinent to
the query from the set of ten retrieval texts, excluding the golden retrieval
context. In the case of irrelevant retrieval noise, no selection is made from
the retrieval texts associated with the current query. Instead, a passage is
randomly chosen from the retrieval contents of other queries, ensuring its
complete irrelevance to the current query. For the counterfactual retrieval
noise, we randomly select one passage from the two golden retrieval contexts
and substitute its answer entity with an incorrect one.

The test set of RAG-Bench comprises 1000 randomly chosen samples from the test
sets of three QA datasets, resulting in a total of 3000 samples. The training
set consists of 1500 samples randomly selected from the training sets of the
three datasets, totaling 4500 samples. The validation set, drawn from the
training sets of three QA datasets, contains 300 samples. Notably, careful
measures were taken to ensure no overlap with the training data of RAG-Bench.

Datasets | Train | Test
---|---|---
#Full | #Subset | #Full | #Subset
NQ | 79,168 | 40,551 | 3,610 | 1,833
TriviaQA | 78,785 | 51,202 | 11,313 | 7,010
WebQ | 3,778 | 2,316 | 2,032 | 1,057

Table 1: The statistics of the three QA datasets.

Method | Golden Only | Golden & $c_{i}$ | Golden & $c_{r}$ | Golden & $c_{c}$ | Avg
---|---|---|---|---|---
F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM
LLaMA27B | 65.56 | 51.80 | 56.14 | 42.87 | 53.10 | 39.73 | 51.81 | 38.37 | 56.68 | 43.19
Qwen7B | 62.57 | 47.07 | 61.48 | 46.06 | 55.50 | 40.50 | 53.26 | 36.90 | 58.20 | 42.63
LLaMA213B | 69.27 | 55.00 | 63.25 | 49.47 | 62.27 | 47.97 | 62.07 | 47.17 | 64.22 | 49.90
Qwen14B | 67.45 | 51.43 | 66.71 | 51.20 | 61.88 | 46.16 | 58.65 | 41.30 | 63.67 | 47.52
LLaMA270B | 71.43 | 56.56 | 70.05 | 55.13 | 65.97 | 51.33 | 63.91 | 48.27 | 67.84 | 52.82
ChatGPT3.5 | 73.98 | 60.50 | 72.24 | 60.30 | 70.65 | 56.89 | 69.00 | 54.64 | 71.47 | 58.10
RALMgolden | 80.31 | 74.03 | 79.33 | 72.73 | 73.26 | 66.33 | 73.08 | 65.40 | 76.50 | 69.62
RetRobust | 80.10 | 73.80 | 79.25 | 72.97 | 74.81 | 68.30 | 75.46 | 68.43 | 77.41 | 70.88
RALMretrieved | 80.04 | 73.40 | 81.09 | 74.80 | 75.99 | 69.10 | 73.10 | 65.67 | 77.55 | 70.74
RALMmultiple | 85.47 | 80.17 | 85.27 | 81.20 | 83.07 | 78.33 | 83.25 | 79.23 | 84.27 | 79.73
RAAT | 87.15 | 83.07 | 86.80 | 82.73 | 85.14 | 81.00 | 86.29 | 82.10 | 86.35 | 82.23

Table 2: Experimental results on our RAG-Bench benchmark. “Golden Only”
denotes a scenario where LLMs only consult the golden retrieval context. In
“Golden & $c_{i}$/$c_{r}$/$c_{c}$”, LLMs consider both the golden retrieval
context and irrelevant retrieval noise/relevant retrieval noise/counterfactual
retrieval noise.

###  4.2 Evaluation Metrics

We evaluate the effectiveness of our method using two metrics: exact match
(EM) and F1 score (Chen et al., 2017). Concretely, EM assesses the extent to
which the answer generated by the system aligns precisely with the standard
answer without any disparities at the character level. In contrast, the F1
score incorporates precision and recall, accounting for the equilibrium
between correctly identifying answers and avoiding omitting correct answers.

###  4.3 Baseline Methods

We conduct a comparison of our RAAT method against zero-shot LLMs, as well as
finetuning approaches applied to LLaMA27B, which shares a common backbone with
RAAT.

####  Zero-Shot Methods

Within the open-source community, many foundational and supervised fine-tuning
(SFT) models have emerged. In our experiments, we select six renowned LLMs as
baselines: ChatGPT3.5, LLaMA27B (Touvron et al., 2023), LLaMA213B, LLaMA270B,
Qwen7B (Bai et al., 2023), and Qwen14B.

#### Fine-tuning Methods

We further compare RAAT with various fine-tuning methods.

  * •

RALMgolden This is a RALM with instruction tuning (Lin et al., 2023). It
prepends a golden retrieval text $c_{golden}$ in context to fine-tune
LLaMA27B.

  * •

RetRobust To ensure that the model is exposed to both golden retrieval texts
and various retrieval noise during training, Yoran et al. (2023) proposes
RetRobust. For each query, RetRobust selects top-1, low-ranked, or random
retrieved passages with equal probability for training.

  * •

RALMretrieved This variant is a RALM incorporating instruction tuning. In
contrast to RALMgolden, RALMretrieved does not manually design retrieval noise
in the training set but directly uses the top-2 retrieved passages. This
training method is more aligned with real retrieval environments.

  * •

RALMmultiple This approach closely resembles RetRobust, differing only in the
construction of the training dataset. In RALMmultiple, rather than introducing
one type of retrieval noise randomly for each query, each type of retrieval
noise is combined with the sample and incorporated into the dataset. That is,
each query is associated with four augmented noisy samples.

Method | Golden Only | Golden & $c_{i}$ | Golden & $c_{r}$ | Golden & $c_{c}$ | Avg
---|---|---|---|---|---
F1 | EM | F1 | EM | F1 | EM | F1 | EM | F1 | EM
RAAT | 87.15 | 83.07 | 86.80 | 82.73 | 85.14 | 81.00 | 86.29 | 82.10 | 86.35 | 82.23
RAAT w/o $\mathcal{L}_{\text{cls}}$ | 86.76 | 82.77 | 86.45 | 82.27 | 84.69 | 80.63 | 85.54 | 81.20 | 85.86 | 81.71
RAAT w/o $\mathcal{L}_{\text{reg}}$ | 86.87 | 83.03 | 83.92 | 79.86 | 84.69 | 80.57 | 87.02 | 82.80 | 85.63 | 81.57

Table 3: Ablation test results in terms of EM and F1 score.

###  4.4 Implementation Details

Our RAAT method relies on LLaMA2-7B as the foundational model. We set the
weight parameters as follows: $w_{\text{reg}}$ to 0.1, $w_{\text{raat}}$ to 2,
and $w_{\text{cls}}$ to 1. The sequence length, epoch, and learning rate are
configured to 512, 2, and 5e-6, respectively. Our experiments are conducted on
a computational cluster equipped with 4 NVIDIA A100 GPUs, each boasting a
capacity of 80GB.

##  5 Experimental Results

###  5.1 Main Results

Table 2 illustrates the efficacy of our RAAT method compared to the baselines
in terms of F1 and EM scores. We observe that all models are affected by three
different types of retrieval noise attacks. The influence of irrelevant
retrieval noise is marginal, while counterfactual retrieval noise exerts the
most significant impact. For the models sharing the same architecture, larger
parameter sizes correlate with superior performance and better robustness
against retrieval noise. For instance, LLaMA27B exhibits a 12.46% reduction in
F1 score when confronted with relevant retrieval noise, whereas LLaMA213B only
experiences a 7% decrease under identical conditions. This trend is also
evident in Qwen.

From Table 2, we can also observe that fine-tuning enables LLMs to better
utilize information from the retrieval texts. Fine-tuned models significantly
outperform the zero-shot LLMs with varying parameter sizes. Moreover,
RALMmultiple shows a significant improvement over RALMgolden, RALMretrieved
and RetRobust, reflecting the sensitivity of retrieval noise to the training
dataset and the importance of diversity in noise attacks during training. Our
RAAT method achieves even better performance than RALMmultiple in all four
environments, with an average increase of 2.08% in the F1 score and 2.5% in
the EM score, demonstrating its superior ability to handle diverse retrieval
noise.

###  5.2 Ablation Study

To gain a comprehensive understanding of the individual contribution of each
component within RAAT to the overall performance, we conducted an ablation
study by removing the regularization term loss (denoted as w/o
$\mathcal{L}_{\text{reg}}$) and the noise-aware classification loss (denoted
as w/o $\mathcal{L}_{\text{cls}}$). The experimental results are shown in
Table 3. After removing the classification loss, we observe that the average
performance of the model decreased by 0.49% and 0.52% in terms of F1 score and
EM score, respectively. While removing the regularization term, there was a
significant performance decrease in handling irrelevant retrieval noise.

Figure 4: The number of queries and parameter updates are 4,500 and 9,000,
respectively. The statistical content in this table pertains to different
types of retrieval noises selected by RAAT each time the model parameters
undergo an update.

###  5.3 Further Discussion

#### What types of adversarial samples does RAAT employ during training?

To gain a comprehensive understanding of the underlying mechanisms of RAAT,
particularly its utilization of specific retrieved data to augment model
robustness, we undertook an in-depth examination of its training process,
involving meticulously tracking the training iterations and conducting a
thorough statistical analysis to quantify the number of different types of
adversarial examples incorporated during the training phase. The statistical
results are illustrated in Figure 4. We observe that RAAT prioritizes the
selection of adversarial examples that can significantly improve model
robustness, as reflected in its tendency to choose certain types of
adversarial examples. This is consistent with our empirical findings described
in Section 3.2. RAAT tends to select adversarial examples associated with
relevant retrieval noise and counterfactual retrieval noise for training.

##  6 Conclusion

This work initially investigated retrieval noises in RALMs and categorized
them into three distinct types, reflecting real-world environments. In
addition, we introduced RAAT as a solution to address the noise robustness
challenges faced by RALMs, which leveraged adaptive adversarial learning and
multi-task learning to enhance the model’s capability. Moreover, we
established a benchmark to verify the effectiveness of RAAT based on three
open-domain QA datasets. Experimental results demonstrate substantial
improvements in F1 and EM scores for the LLaMA2 7B model fine-tuned with RAAT
across diverse noise conditions.

##  7 Limitations

In this section, we delve into the limitations inherent in our work, with the
objective of pinpointing areas for refinement and bolstering the performance
of our model in future endeavors. Two principal limitations have been
identified. Firstly, the benchmark constructed for our experiments relies
exclusively on datasets sourced from three open-domain question answering
repositories. Going forward, we intend to compile additional high-quality
datasets from varying NLP tasks and endeavor to retrieve texts from a more
extensive array of knowledge bases. This strategic expansion aims to
facilitate the creation of a more diversified and expansive benchmark tailored
for evaluating the retrieval noise robustness of large language models.
Secondly, within the framework of RAAT, our efforts have been singularly
concentrated on fortifying the retrieval noise robustness at the LLM end.
However, the prospect of jointly training large language models and retrieval
models emerges as a promising avenue for enhancing the overall robustness of
RALMs. Although this dimension was not the primary focal point of our current
work, in our subsequent investigations into retrieval noise robustness, we
plan to delve into this avenue. This approach would facilitate the
synchronized progress of both the large language model and the retrieval
model, contributing to an overall improvement in their robustness.

## Acknowledgments

Min Yang was supported by National Key Research and Development Program of
China (2022YFF0902100), National Natural Science Foundation of China
(62376262), the Natural Science Foundation of Guangdong Province of China
(2024A1515030166), Shenzhen Science and Technology Innovation Program
(KQTD20190929172835662), Shenzhen Basic Research Foundation
(JCYJ20210324115614039). This work was supported by Alibaba Group through
Alibaba Innovative Research Program, Postdoctoral Fellowship Program of CPSF
(GZC20232873), GuangDong Basic and Applied Basic Research Foundation
(2023A1515110718 and 2024A1515012003).

## References

  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Bai et al. (2021) Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. 2021.  Recent advances in adversarial training for adversarial robustness.  _arXiv preprint arXiv:2102.01356_.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on freebase from question-answer pairs.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1533–1544.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading wikipedia to answer open-domain questions.  _arXiv preprint arXiv:1704.00051_.
  * Chen et al. (2023) Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023.  Benchmarking large language models in retrieval-augmented generation.  _arXiv preprint arXiv:2309.01431_.
  * Creswell et al. (2022) Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022.  Selection-inference: Exploiting large language models for interpretable logical reasoning.  _arXiv preprint arXiv:2205.09712_.
  * Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_.
  * Gueta et al. (2023) Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. 2023.  Knowledge is a region in weight space for fine-tuned language models.  _arXiv preprint arXiv:2302.04863_.
  * Ivgi and Berant (2021) Maor Ivgi and Jonathan Berant. 2021.  Achieving model robustness through discrete adversarial training.  _arXiv preprint arXiv:2104.05062_.
  * Jain et al. (2023) Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, et al. 2023.  Neftune: Noisy embeddings improve instruction finetuning.  _arXiv preprint arXiv:2310.05914_.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  _arXiv preprint arXiv:1707.07328_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_.
  * Kandpal et al. (2023) Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023.  Large language models struggle to learn long-tail knowledge.  In _International Conference on Machine Learning_ , pages 15696–15707. PMLR.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  _arXiv preprint arXiv:2004.04906_.
  * Kurakin et al. (2016) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016.  Adversarial machine learning at scale.  _arXiv preprint arXiv:1611.01236_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2022) Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022.  Large language models with controllable working memory.  _arXiv preprint arXiv:2211.05110_.
  * Liang et al. (2023) Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang, Xiangjun Dong, and Xinwang Liu. 2023.  Knowledge graph contrastive learning based on relation-symmetrical structure.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Lin et al. (2023) Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. 2023.  Ra-dit: Retrieval-augmented dual instruction tuning.  _arXiv preprint arXiv:2310.01352_.
  * Madry et al. (2017) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017.  Towards deep learning models resistant to adversarial attacks.  _arXiv preprint arXiv:1706.06083_.
  * Mao et al. (2020) Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. 2020.  Generation-augmented retrieval for open-domain question answering.  _arXiv preprint arXiv:2009.08553_.
  * Miyato et al. (2016) Takeru Miyato, Andrew M Dai, and Ian Goodfellow. 2016.  Adversarial training methods for semi-supervised text classification.  _arXiv preprint arXiv:1605.07725_.
  * Morris et al. (2020) John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.  _arXiv preprint arXiv:2005.05909_.
  * Ni et al. (2023) Shiwen Ni, Jiawen Li, Min Yang, and Hung-Yu Kao. 2023.  Dropattack: A random dropped weight attack adversarial training for natural language understanding.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Ruder (2017) Sebastian Ruder. 2017.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_.
  * Shen et al. (2023) Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. 2023.  In chatgpt we trust? measuring and characterizing the reliability of chatgpt.  _arXiv preprint arXiv:2304.08979_.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023.  Large language models can be easily distracted by irrelevant context.  In _International Conference on Machine Learning_ , pages 31210–31227. PMLR.
  * Thakur et al. (2023) Nandan Thakur, Luiz Bonifacio, Xinyu Zhang, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Boxing Chen, Mehdi Rezagholizadeh, et al. 2023.  Nomiracl: Knowing when you don’t know for robust multilingual retrieval-augmented generation.  _arXiv preprint arXiv:2312.11361_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Wang et al. (2020) Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. 2020.  Infobert: Improving robustness of language models from an information theoretic perspective.  _arXiv preprint arXiv:2010.02329_.
  * Wu et al. (2017) Yi Wu, David Bamman, and Stuart Russell. 2017.  Adversarial training for relation extraction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1778–1783.
  * Yasunaga et al. (2017) Michihiro Yasunaga, Jungo Kasai, and Dragomir Radev. 2017.  Robust multilingual part-of-speech tagging via adversarial training.  _arXiv preprint arXiv:1711.04903_.
  * Yin et al. (2023) Xunjian Yin, Baizhou Huang, and Xiaojun Wan. 2023.  Alcuna: large language models meet new knowledge.  _arXiv preprint arXiv:2310.14820_.
  * Yoran et al. (2023) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2023.  Making retrieval-augmented language models robust to irrelevant context.  _arXiv preprint arXiv:2310.01558_.
  * Yu et al. (2023) Wenhao Yu, Hongming Zhang, Xiaoman Pan, Kaixin Ma, Hongwei Wang, and Dong Yu. 2023.  Chain-of-note: Enhancing robustness in retrieval-augmented language models.  _arXiv preprint arXiv:2311.09210_.
  * Zhu et al. (2019) Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019.  Freelb: Enhanced adversarial training for natural language understanding.  _arXiv preprint arXiv:1909.11764_.
  * Zhu et al. (2023) Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen. 2023.  Large language models for information retrieval: A survey.  _arXiv preprint arXiv:2308.07107_.
  * Zou et al. (2023) Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023.  Universal and transferable adversarial attacks on aligned language models.  _arXiv preprint arXiv:2307.15043_.

Figure 5: The results of T-SNE visualization. Following the introduction of
four types of adversarial samples (i.e., retrieval noises) into models tuned
by various methods, the hidden state of the last token is extracted.
Subsequently, dimensionality reduction using t-SNE, clustering, and
visualization are performed. This visual representation includes three
methods, namely $\text{RALM}_{golden}$, RetRobust, and RAAT.

##  Appendix A Has the Model Truly Attained Noise Awareness?

Our preliminary investigation focused on the intrinsic capability of
RALMgolden and RetRobust to classify the types of retrieval noises. Drawing
inspiration from previous work Gueta et al. (2023), we approached this matter
through the application of clustering algorithms. The results, illustrated in
Figure 5, reveal suboptimal clustering of text vectors from RALMgolden and
RetRobust, suggesting that the internal representations for noise
classification in these models may lack clarity. Consequently, we introduced a
noise classification loss $\mathcal{L}_{{cls}}$ into our RAAT method. The
experimental results demonstrated tangible benefits with the incorporation of
the classification loss. Additionally, we assessed the clustering
effectiveness in models fine-tuned with RAAT, observing minimal distances
among samples of irrelevant, relevant, and no retrieval noises, in contrast to
the considerable distance from counterfactual retrieval noise samples. In
particular, counterfactual retrieval noise posed the most significant
challenge to LLMs; however, after RAAT tuning, it exhibited superior
clustering and representation learning outcomes, indirectly validating the
efficacy of RAAT.