# Context Matters: An Empirical Study of the Impact of Contextual Information
in Temporal Question Answering Systems

Dan Schumacher1, Fatemeh Haji1††footnotemark: , Tara Grey1 Niharika
Bandlamudi1, Nupoor Karnik1,
Gagana Uday Kumar1, Jason Cho-Yu Chiang2, Paul Rad1,
Nishant Vishwamitra1, and Anthony Rios1
1University of Texas at San Antonio
2Peraton Labs
{dan.schumacher,fatemeh.haji,anthony.rios}.utsa.edu These authors contributed
equally on this project.

###### Abstract

Large language models (LLMs) often struggle with temporal reasoning, crucial
for tasks like historical event analysis and time-sensitive information
retrieval. Despite advancements, state-of-the-art models falter in handling
temporal information, especially when faced with irrelevant or noisy contexts.
This paper addresses this gap by empirically examining the robustness of
temporal question-answering (TQA) systems trained on various context types,
including relevant, irrelevant, slightly altered, and no context. Our findings
indicate that training with a mix of these contexts enhances model robustness
and accuracy. Additionally, we show that the position of context relative to
the question significantly impacts performance, with question-first
positioning yielding better results. We introduce two new context-rich TQA
datasets, ContextAQA and ContextTQE, and provide comprehensive evaluations and
guidelines for training robust TQA models. Our work lays the foundation for
developing reliable and context-aware temporal QA systems, with broader
implications for enhancing LLM robustness against diverse and potentially
adversarial information.

Context Matters: An Empirical Study of the Impact of Contextual Information in
Temporal Question Answering Systems

Dan Schumacher1††thanks: These authors contributed equally on this project.,
Fatemeh Haji1††footnotemark: , Tara Grey1 Niharika Bandlamudi1, Nupoor
Karnik1, Gagana Uday Kumar1, Jason Cho-Yu Chiang2, Paul Rad1, Nishant
Vishwamitra1, and Anthony Rios1 1University of Texas at San Antonio 2Peraton
Labs {dan.schumacher,fatemeh.haji,anthony.rios}.utsa.edu

##  1 Introduction

Large language models (LLMs) often face challenges in temporal reasoning
Beniwal et al. (2024); Liu et al. (2024); Gade and Jetcheva (2024), which
involves understanding and processing time-related information within text.
Temporal reasoning is crucial for many applications, such as historical event
analysis Lorenzini et al. (2022), time-sensitive information retrieval,
journalism, and real-time data analysis, where precise temporal understanding
is key.

Figure 1: Illustration of non-robust temporal QA. Models trained on relevant
contexts will not be robust to irrelevant contexts, causing the model to
assume it is correct. Also, training only on irrelevant contexts will result
in models ignoring relevant contexts at test time.

Numerous methods have been proposed for temporal question answering, from
traditional fine-tuning-based Tan et al. (2023) to prompt-based solutions Chen
et al. (2023b). Yet, several papers have pointed out the limitations of these
methods. For instance, Wallat et al. (2024) identified “temporal blind spots”
through comprehensive evaluations across multiple temporal QA datasets, i.e.,
understanding whether LLMs favor certain time periods or more recent events.
Dhingra et al. (2022) suggested a pretraining approach incorporating temporal
data, improving LLMs’ recall of facts from different periods. In general, LLMs
may be able to answer a question based on their parametric knowledge, e.g., if
they saw the information during the pretraining phase. If not, the knowledge
must be passed as additional context. Hence, Retrieval-Augmented Generation
(RAG) Lewis et al. (2020) is being used to retrieve information to support the
internal parametric knowledge of the models. Given that contextual knowledge
can be irrelevant, Shaier et al. (2024) highlighted gaps in evaluating how QA
systems that use context, such as robustness to noisy contexts, but they did
not explore substantial training paradigms.

In addition to TQA, relevant to our paper is the improvement of general QA
systems in their handling of irrelevant and noisy contexts Cheng et al.
(2024); Yoran et al. (2024). For example, Yoran et al. (2024) evaluate using
filters (e.g., natural language inference) to filter out noisy context. They
also explore training LLMs for question-answering using relevant and incorrect
contextual information. Despite recent advancements, state-of-the-art models
still face challenges in handling temporal information effectively Wallat et
al. (2024).

While Yoran et al. (2024) shows that training on a mix of relevant and
irrelevant contexts improves performance, it is unknown why this would work.
Further understanding on the matter may lead to even more drastic
improvements. Intuitively, training with both contexts may allows the model to
use its parametric knowledge instead of the context if it determines it is
irrelevant. This would imply that training on irrelevant contexts alone would
cause models to ignore the context completely (relevant or not) at test time.
If this is possible, the implication is that LLMs can be trained to ignore
information that matches specific patterns, such as hate speech, offensive
language, or potentially adversarial examples. We illustrate this idea in
Figure 1. If a model can be trained to ignore context, and the information is
not within the model’s parametric knowledge, the performance would be low even
if the correct context is provided as input for all test instances. Likewise,
there are other unknown questions, such as what would happen if models were
trained with no context or slightly altered context? Will the model learn to
identify the alterations and return a correct answer? Is context needed at all
during the training process? What would happen if models were fine-tuned using
only question-answer pairs but passed relevant or irrelevant context at test
time? We explore these questions in this paper.

To do so, we empirically examine Temporal Question Answering robustness, with
the aim of training models to provide accurate responses to queries involving
time-related information accompanied by a variety of contexts. For instance,
questions like “Which college’s freshmen won the Temple Challenge Cup in
2001?” and “Which country signed an international treaty to halt the spread of
nuclear weapons in 1985?” require models to understand the temporal context
and historical facts.

Another distinction between our work and Yoran et al. (2024) is that we
explore where the context should be positioned in relation to the question
during input to the LLM. It turns out that the position of the context can
have a major impact on model performance, particularly if the context is
irrelevant. Finally, rather than general QA, we exclusively focus on temporal
question answering, which is highly context-driven, e.g., “Who was the
president of the United States in 1994?” Hence, we better understand how LLMs
retrieve and answer questions in the presence of altered or irrelevant
contextual information for temporal questions.

We focus on explicit temporal question answering, which can be highly
sensitive to contextual information. Moreover, focusing on explicit questions
with direct answers allows us to control for unknown reasoning processes that
may impact answer quality. Our experiments on two temporal QA datasets show
that training with a mixture of diverse context types can improve the
robustness of LLMs in handling temporal questions. Our approach shows a 7.5%
increase in accuracy when comparing purely relevant-trained to our mixture
approach when prompted with a combination of contexts during evaluation. This
highlights the importance of context-aware temporal reasoning for reliable
temporal QA systems.

Overall, we make the following contributions:  (i) We investigate the need and
procedures to integrate context during the training and testing phases. We
explore scenarios with no context, slightly altered context, irrelevant
context, relevant context, and context provided at different positions
relative to the question.  (ii) We specifically target temporal questions,
providing insights into how LLMs handle temporally-related information and
their robustness to noisy or incorrect contexts. Because of the absence of
explicit temporal QA datasets with contexts, we created two new temporal QA
datasets, ContextAQA and ContextTQE, where we have added relevant, irrelevant,
and slightly-altered contextual information to existing datasets Wang et al.
(2021, 2022).  (iii) We comprehensively evaluate many state-of-the-art 2b, 7b,
and 8b (see appendix for all numbers) models and GPT-3.5 to various noisy
context setups and provide general guidelines for training more robust
temporal QA models. Moreover, we perform a comprehensive error analysis, which
helps us understand when and why certain training procedures perform well or
poorly.

##  2 Related Work

#### Temporal QA.

Temporal QA encompasses understanding temporal relationships in text, video,
or knowledge bases Xiao et al. (2021); Jang et al. (2017); Saxena et al.
(2021); Zhao and Rios (2024), including ordering event mentions about specific
medications in clinical notes Zhao and Rios (2024). This paper focuses on
traditional temporal factoid-based QA, e.g., "Who was the president in 1998?"
Many datasets have been curated for temporal QA. A common dataset, called TQE,
is by Wang et al. (2021) containing both explicit and implicit temporal
questions. Prior work curated datasets to address the unique challenges of
TQA. For example, Dhingra et al. (2022) created a diagnostic dataset designed
to evaluate the temporal knowledge of LLMs, Beniwal et al. (2024) introduced
TempUN, a study with state-of-the-art models to reveal LLMs’ limitations in
handling temporal information. Wang et al. (2022) developed Archival QA, a
large-scale open-domain QA dataset based on news archives, expanded upon in
this study. Gruber et al. (2024) introduced COMPLEXTEMPQA, a Wikipedia-based
dataset similar to TQE.

Recent studies highlighted LLMs’ limitations in temporal reasoning Qiu et al.
(2023) and proposed frameworks combining LLMs’ extraction capabilities with
logical reasoning Li et al. (2023), temporal span extraction, and time-
sensitive reinforcement learning Tan et al. (2023). Gao et al. (2024)
introduced a two-stage generative framework for temporal knowledge graph QA
using LLMs. Dhingra et al. (2022) propose TEMPLAMA, a dataset of temporally-
scoped knowledge probes, and suggested pretraining modifications to facilitate
temporal knowledge acquisition. To address TQA limitations, Dhingra et al.
(2022) proposed two approaches: training separate models per year and a single
model using temporal prefixes.

Our work expands on prior research in two major ways. First, to the best of
our knowledge, this is the first study that explicitly explores 1) quantifying
the robustness of large language models for temporal question answering with
regard to irrelevant context and 2) methods to train more robust temporal QA
systems. Second, most other explicit temporal QA datasets do not have valid
contexts to match each question-answer pair. This paper is releasing two new
context-matched datasets called ContextTQE and ContextAQA, allowing future
researchers to build on our study.

#### Robust LLM.

There has been substantial research focused on measuring and training robust
LLMs based on various definitions of “robustness”, such as robustness to
adversarial examples Raina et al. (2024); Yuan et al. (2024); Kumar et al.
(2023); Andriushchenko et al. (2024); Chen et al. (2023a). For example, Raina
et al. (2024) show that adversarial attacks can generalize to different LLMs
to mitigate safety features. In this paper, we are primarily focused on
robustness to slightly altered or completely irrelevant contexts. Shi et al.
(2023) investigated how LLMs handle irrelevant context and created the Grade-
School Math with Irrelevant Context (GSM-IC) dataset to evaluate this aspect.
Their results showed a significant drop in performance when irrelevant
information was included. Kannen et al. (2023) explored knowledge base QA
limitations by addressing incomplete knowledge bases using temporal fact
extraction for bad links and relationship errors. Li et al. (2022) increased
model robustness by using knowledge-aware fine-tuning, incorporating
counterfactuals during training. Similarly, Cheng et al. (2024) finds that
using noisy labels during in-context learning can improve robustness.

Shaier et al. (2024) describes an optimal response to context-based QA
systems. Models should: 1) Answer correctly with original context; 2) Answer
correctly with noisy or irrelevant context if the knowledge is available
within the model’s parametric knowledge; 3) Change the answer with conflicting
context; 4) Answer correctly with no context and irrelevant context; and 5)
Answer ’unanswerable’ if you can’t answer without context. Yoran et al. (2024)
explore how models can be robust to noisy and incorrect contexts. But, they
did not explore substantial training paradigms for improvement. Zhang et al.
(2024) evaluate fusion of parametric knowledge with retrieved contexts to make
new inferences.

Our paper extends on prior work in two major ways. First, our focus is on
temporal QA instead of general QA. Temporal QA can be more sensitive to the
underlying context and provides a unique testbed to understand context
robustness. Second, many assumptions are made in prior work that are
speculative without substantial evaluation. Yoran et al. (2024) speculated
that the model learns to ignore irrelevant context when trained on
relevant/irrelevant contexts, implying training only on irrelevant context
would make it incapable of using context. These aspects were not measured
before: What happens if we do not use context while training? Likewise, what
if the relevant context is slightly altered? Would the model perform similarly
to training in relevant and irrelevant contexts? Moreover, what about context
location? Should the context be placed before or after the question? Which
results in better performance. We address all these questions in this paper.

Figure 2: This figure illustrates our overall methodology, including training
with different context types and evaluating different context types and
locations.

##  3 Methodology

At a high level, we perform comprehensive training and evaluation setups to
understand and improve the robustness of LLMs for temporal QA. We provide an
intuitive figure of our setup in Figure 2. Formally, let $Q$ denote a temporal
question, and $C$ denote the context provided to answer the question. We
define the LLM as a function $\mathcal{M}$ parameterized by $\theta$. The LLM
takes $Q$ and $C$ as input and generates an answer $A$

$A=\mathcal{M}_{\theta}(Q,C)$

where $Q=\\{q_{1},q_{2},\ldots,q_{n}\\}$ is a sequence of tokens representing
the temporal question, $C=\\{c_{1},c_{2},\ldots,c_{m}\\}$ is a sequence of
tokens representing the context, $A=\\{a_{1},a_{2},\ldots,a_{k}\\}$ is a
sequence of tokens representing the answer. To assess the robustness of the
LLM, we evaluate the model under different context conditions. Let $C_{r}$
represent the relevant context, $C_{i}$ the irrelevant context, $C_{a}$ the
slightly altered context, and $C_{0}$ the absence of context. Furthermore, the
context on which the model was trained will impact the predicted answer.
Therefore, let $\mathcal{M}_{\theta,r}$ represent a model trained on relevant
context, $\mathcal{M}_{\theta,i}$ represent a model trained on irrelevant
context, $\mathcal{M}_{\theta,a}$ represent a model trained on slightly
altered context, and $\mathcal{M}_{\theta,0}$ represent a model trained in the
absence of context. Finally, we will let $\mathcal{M}_{\theta,m}$ represent a
model trained on a mixture of all the other contexts mentioned. To list a few
examples of answers that will be generated under these conditions
$A_{r,r}=\mathcal{M}_{\theta,r}(Q,C_{r})$,
$A_{i,0}=\mathcal{M}_{\theta,i}(Q,C_{0})$,
$A_{a,i}=\mathcal{M}_{\theta,a}(Q,C_{i})$, and
$A_{0,a}=\mathcal{M}_{\theta,0}(Q,C_{a})$, respectively.

Intuitively, a model can access a temporal question’s answer in the context
$C_{i}$ or its parametric knowledge stored in $\theta$. Hence, a robust
temporal question answering LLM should answer a question correctly if it knows
an answer in its parametric knowledge, even if the context $C_{i}$ is
incorrect. To improve the robustness of temporal QA LLMs, we explore two
approaches. First, explore where the context should go in the prompt,
beginning or end. Specifically, the input to $\mathcal{M}_{\theta}()$ can
process the question and context as $[Q;C]$ or $[C;Q]$. The impact of location
can potentially depend on its location during training; it may also depend on
the underlying LLM used, or there may be a framework that generalizes across
multiple LLMs. Second, we explore how the LLM should be trained. Specifically,
should the model $\mathcal{M}_{\theta}()$ be trained only with relevant
context $C_{r}$, or should it be some mixture of all contexts $C_{r}$,
$C_{i}$, $C_{a}$, and $C_{0}$. Again, there may be a setup that works best
across multiple models, or it may vary from model to model. We describe the
different setups in more detail below.

###  3.1 Context Location

We evaluate two context location setups, putting the context before or after
the question. When putting the context before the question, we use the prompt

> Given the following context, answer the question.
> Here is the context: {context}
> Here is the question: {question}
> The answer is:

where $\\{question\\}$ represents a specific temporal question and
$\\{context\\}$ is a variable representing the context. For the context after
the question, we use the prompt

> You must answer the following question given subsequent context.
> Here is the question: {question}
> Here is the context: {context}
> The answer is:

where we only slightly modify the initial instruction provided to the LLM.

###  3.2 Context Generation

Most TQA datasets do not contain contextual information. In our study, the
context must generally contain the answer if it is relevant. Hence, simply
retrieving similar passages on Wikipedia may not be robust. Instead, we use
GPT3.5 to generate the contexts. Specifically, to generate relevant context,
we used a few-shot prompting method using gpt-3.5. By passing each question-
answer pair into GPT-3.5, we retrieved contexts to help the user answer the
questions. The format of the prompt is defined as

> Question: {Question Example}
> Answer: {Answer Example}
> Generated Context: {Context}
>
> Question: {New Question}
> Answer: {New Answer}
> Generated Context:

where each of the context examples {Context} was manually curated.

To evaluate the robustness of our models against temporally altered data, we
implemented a method to generate falsified date contexts. This process
transforms the relevant context into a slightly-altered version by extracting
all dates and replacing them with falsified days, months, and years. We
exclude the actual date info from the generations when changing the dates. For
example, for years, we have used a predefined range of 1850 to 2024 to
randomly replace each date and exclude the year in the context from the random
sample. The days and months are generated completely at random.

Overall, we add context to two existing datasets: Temporal Question (TQE)
Answering Wang et al. (2021) and Archival Question Answering (AQA) Wang et al.
(2022), to create the versions ContextTQE and ContextAQA, respectively.
ContextTQE has two sub-datasets: one with implicit questions where the date
information must be inferred by the reader and the other with explicit date
information. We use the latter. Of the meager 500 total rows in the dataset,
we use 75 rows for development and testing and the remaining 350 for training.
AQA, by comparison, is a much larger dataset with 60,000 observations, of
which 7,500 are reserved for development and 7,500 for testing. In comparing
the two datasets, we believe that the AQA dataset presents much tougher
questions. Due to the substantial size difference, ContextAQA will be the
focus of this paper, but our ContextTQE results can be found in the appendix.

To confirm the validity of our GPT-generated context, we manually reviewed a
sample of 100 contexts. Only three examples were not entirely correct. A full
example of our prompt can be found in the appendix.

##  4 Experiments

This section describes our evaluation metrics, results, implications, and
error analysis.

###  4.1 Evaluation Metrics.

We used the same metrics as Wallat et al. (2024) to evaluate our models.
Specifically, the evaluation uses three main metrics:  1. Standard F1 Score,
which measures how well the model’s answers match the correct answers, taking
into account both precision and recall;  2. Contains Accuracy, which checks if
the correct answer is contained within the model’s generated text; and  3.
BERT-based Answer Equivalence Metric (BEM), which uses a BERT model to detect
if the model’s answer is semantically equivalent to the correct answer.
Because of limited space, for detailed BEM scores and analysis, please see
Appendix D.

Model | No Test Set | Rel Test Set | SA Test Set | Irr Test Set | Average Scores
---|---|---|---|---|---
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Average Acc | Average F1
Baselines
GPT-3.5 | .292 | .089 | .857 | .243 | .745 | .179 | .247 | .075 | .535 | .147
Mistral-7B-Instruct-v0.2 | .138 | .119 | .640 | .353 | .476 | .236 | .370 | .039 | .406 | .187
gemma-1.1-7b-it | .135 | .048 | .766 | .288 | .150 | .408 | .000 | .016 | .263 | .190
llama-3-8b-Instruct | .189 | .201 | .644 | .595 | .561 | .466 | .160 | .161 | .389 | .356
Fine-tuned Models
Mistral-7B-Instruct-v0.2 + No | .273 | .143 | .678 | .342 | .626 | .315 | .221 | .133 | .450 | .233
gemma-1.1-7b-it + No | .108 | .052 | .739 | .592 | .696 | .568 | .158 | .174 | .425 | .347
llama-3-8b-Instruct + No | .226 | .295 | .326 | .396 | .285 | .350 | .101 | .156 | .235 | .299
Mistral-7B-Instruct-v0.2 + Rel | .226 | .231 | .832 | .379 | .800 | .382 | .012 | .024 | .468 | .254
gemma-1.1-7b-it + Rel | .081 | .043 | .815 | .714 | .768 | .671 | .076 | .094 | .435 | .381
llama-3-8b-Instruct + Rel | .216 | .284 | .807 | .870 | .725 | .788 | .013 | .027 | .440 | .492
Mistral-7B-Instruct-v0.2 + SA | .226 | .244 | .825 | .387 | .822 | .386 | .012 | .024 | .471 | .260
gemma-1.1-7b-it + SA | .098 | .046 | .830 | .741 | .820 | .729 | .008 | .023 | .439 | .385
llama-3-8b-Instruct + SA | .211 | .278 | .799 | .867 | .797 | .865 | .015 | .032 | .456 | .511
Mistral-7B-Instruct-v0.2 + Irr | .257 | .140 | .222 | .129 | .207 | .122 | .259 | .140 | .236 | .133
gemma-1.1-7b-it + Irr | .078 | .044 | .552 | .615 | .403 | .467 | .172 | .258 | .301 | .346
llama-3-8b-Instruct + Irr | .239 | .308 | .195 | .256 | .197 | .260 | .240 | .310 | .218 | .284
Mistral-7B-Instruct-v0.2 + Mixed | .259 | .141 | .821 | .360 | .813 | .357 | .252 | .139 | .536 | .249
gemma-1.1-7b-it + Mixed | .210 | .270 | .790 | .840 | .770 | .830 | .200 | .270 | .493 | .553
llama-3-8b-Instruct + Mixed | .240 | .310 | .797 | .861 | .787 | .853 | .230 | .301 | .514 | .581

Table 1: ContextAQA Question-first performance of different model
configurations on temporal QA across four context types: no context (No),
relevant (Rel), slightly-altered (SA), irrelevant (Irr). Models include base
LLMs (Mistral-7B-Instruct-v0.2, llama-3-8b-Instruct) and fine-tuned variants
trained on various context setups (+No, +Rel, +SA, +Irr, +Mixed). Results
highlight the impact of context types and fine-tuning strategies on model
robustness and accuracy.

###  4.2 Results.

For our experiments, we address important research questions, such as “How do
different training setups impact model robustness?” and “Does context location
matter?” For space considerations, we focus on the results of the ContextAQA
dataset, which is much larger than ContextTQE. See the appendix for ContextTQE
numbers.

First, how do different training setups impact TQA model robustness? When
fine-tuning a model with only questions and answers—without any context—we see
that the effectiveness of training depends on the fine-tuned model. For
example, in Table 1, when the Llama baseline is evaluated on relevant context,
it scores an accuracy of .644 (F1 .595), however after fine-tuning on no-
context, the accuracy is cut in half to .326 (F1 .396). These results are
juxtaposed by the small boost that Mistral gets after training on no context.
In that case, we can see the accuracy increase from .640 (F1 .353) up to .678
(F1 .342). Overall, it seems that fine-tuning without any context—again, only
QA pairs—does not seem to be too useful and, in some scenarios, can adversely
affect results, assuming the context is related to the question. We do want to
point out that our results show that even if models are trained without
context, they can still take advantage of relevant context at training time.
Examine gemma + No evaluated on no context with an accuracy of .081 (F1 .043)
to the same model evaluated on relevant context with an accuracy of .815 (F1
.714). So, if you train without the context, the model does not completely
lose that ability, though it is diminished compared to the baseline (non-fine-
tuned) scores for each model.

Next, training on relevant context significantly boosts performance when
evaluating Relevant and slightly-altered contexts. Looking at the Relevant
trained and Relevant evaluated results in Table 1, we can see that all of our
models are at or above an accuracy of .800. Likewise, the slightly-altered
evaluations are up by at least .200 (20%) higher than their baseline
counterparts. This result is intuitive, meaning that the models learn to rely
on the contextual information better, extracting the results precisely.
However, models trained on relevant contexts alone result in worse performance
when evaluated on irrelevant data, when compared to both the models trained on
no context and most of the baseline models (except for gemma, yet it is still
low). This result suggests that relevant-trained models will generally result
in non-robust performance, and should be limited if possible.

Due to slightly-altered context being a child of relevant context (the only
change being the fabrication of the dates), the slightly-altered trained
results are very similar to the relevant context trained results, with some
minor differences. For instance, the relevant trained models’ average accuracy
of .468, .435, and .440 increases to .471, .439, and .456, showing that adding
some noise to the context can allow models to become robust, particularly when
evaluated on context that is also partially correct. In Table 1, we see a very
similar performance when comparing the relevant trained models (.832, .815,
.807) to the slightly-altered trained models (. 825, .830, .799). When
comparing the slightly-altered evaluations, however, the difference in results
isn’t so subtle. To highlight this, compare llama + Rel results of .807 (F1
.870) with the llama + SA results of .799 (F1 .867). This demonstrates that
including noise in training data does not diminish the accuracy of a model
when it encounters data without noise.

Models trained on irrelevant context perform worse when evaluated on relevant
context compared to baseline and all other trained models. This reduction is
perhaps most striking when comparing the relevant evaluations in Table 1 of
Llama3-8b baseline (.644) to its Irrelevant trained version (.195). We see
that the irrelevant-trained and relevant-evaluated results for Mistral
accuracy of .222 (F1 .129) and Llama accuracy of .195 (F1 .256) have been
reduced down to the magnitude of the accuracy of .259 (F1 .14) and .240 (F1
.310), respectively. The Gemma + Irr model has a relatively high accuracy of
.552 (F1 .615) compared to the other models’ results. However, this score is
still Gemma’s lowest relevant context evaluated score. These results support
our hypothesis that training on Irrelevant context can teach a model to ignore
the context in which it is presented.

Finally, Training with mixed context improves model performance, especially if
your goal is to produce a robust model. Notably, models trained on mixed
context handle irrelevant context on par with models trained on Irrelevant
context. Looking at Table1, In the case of gemma-1.1-7b-it, training on
irrelevant and evaluating on Irrelevant reaches an accuracy of .172 (F1 .258),
which is not quite as good as when trained on mixed and evaluated on
irrelevant context accuracy .200 (F1 .270). Another case that highlights the
effectiveness of mixed context training is the difference in mistral-7b + Rel
score of .012 on Irrelevant context, whereas mistral-7b+ mixed scored almost
three times higher with a .252. Finally, the average accuracy of our mixed
context-trained models dominates the average accuracy for all models, with
scores of .536, .493, and .514. for Mistral, Gemma, and LlaMa, respectively.
Notably, no baseline or trained models’ average accuracy passes the .500 mark
except for mixed models.

Model | Context Location | Average Scores
---|---|---
|  | Average Acc | Average F1
Mistral-7B-Instruct-v0.2 + Mixed | C-Last/ Q-First | .536 | .249
| C-First/ Q-Last | .492 | .229
llama-3-8b-Instruct + Mixed | C-Last/ Q-First | .514 | .581
| C-First/ Q-Last | .453 | .518
gemma1.1-7b-it + Mixed | C-Last/ Q-First | .493 | .553
| C-First/ Q-Last | .460 | .520
gemma1.1-2b-it + Mixed | C-Last/ Q-First | .455 | .485
| C-First/ Q-Last | .420 | .455

Table 2: The table shows results for two experimental setups: context first
and question last (C-First/Q-Last) and context last and question first
(C-Last/Q-First). The average accuracy and F1 score across all four context
types are reported for each configuration.

Second, Does context location matter? Yes, context location matters. To our
surprise, we find that placing the question first allows the model to ignore
irrelevant context more easily, enhancing its robustness. In Table2, we see
that Q-First always outperforms C-First when comparing the context location
pairs.

###  4.3 Implications

There are several implications of our study. First, training with mixed
contexts does improve model performance. This confirms some of the findings of
Yoran et al. (2024) for general question answering. However, our work extends
this study to understand that 1) model performance improves across models of
different types (they only tested LlaMa-2 13B); and 2) this works for temporal
question answering. Second, training on random data can makes the model learn
to ignore the information. For context, we even find that if the model is
trained with a question first and the context is added first instead, the
model can still completely ignore it (see appendix for all numbers). This is
an implicit assumption in many papers Yoran et al. (2024) when training on
mixed context, but we are the first to quantify it to the best of our
knowledge. This can have implications for future work that may want to train
models to ignore certain types of information, e.g., offensive language or
hate speech.

###  4.4 Error Analysis

We conducted a qualitative analysis on the llama-3-8B model to examine error
patterns in our best-performing relevant-tuned model and how the mixed-tuned
model addresses these errors across various contexts. Our analysis involved
carefully examining individual instances to identify recurring error types. We
observed similar error patterns across different models used in our study.
Specifically, we first gathered instances where the mixed model answered
correctly, but the relevant-tuned model, our best-fine-tuned model for another
base LLMs, responded incorrectly. From there, we sampled 200 instances and
analyzed the causes of errors for the relevant-tuned model.

Incorrect Entity/Fact Identification. A prominent issue we observed with the
relevant-tuned model was the incorrect identification of entities or facts
mentioned in the questions, mostly when the context is irrelevant. Here is an
example:

> Q: What college did Tom Heinsohn lead to a National Invitation Tournament
> title in 1954? Context: James B. Longley, a businessman and politician, ran
> for governor of Maine in 1974. He was a successful candidate and became the
> 69th Governor of Maine, serving from 1975 to 1979 … Relevant-tuned Model
> Answer: fordham Mixed-tuned Model Answer: holy cross Correct Answer: holy
> cross

Interestingly, the incorrect answer “fordham” is not completely random. Tom
Heinsohn frequently mentioned his high school coach, who played basketball for
Fordham, suggesting the model does have some parametric knowledge about Tom
Heinsohn.

Incorrect Contextual Understanding. Instances where the model lacked factual
understanding led to incorrect answer (e.g., when no context is provided). For
instance, the example below is from a model with now context passed to it:

> Q: What country’s troops did General Waller argue were not ready to fight by
> the Jan. 15, 1991, deadline? Relevant Answer: iraqi Mixed Answer: american
> Correct: american

Since no context was provided in this setting, the error can be attributed to
the model’s inability to access and effectively use the appropriate factual
knowledge needed to answer the question accurately.

Misidentification of Specific Entities. In some instances, the relevant-tuned
model demonstrated difficulties in precisely identifying specific entities.
For example, when asked about which country the 1988 Free Trade Agreement
favored, the relevant-tuned model answered “american” instead of the more
specific “the united states,” despite the context indicating the latter as the
correct entity. This error type highlights the model’s need for improved
entity resolution capabilities.

Our qualitative analysis highlights that the mixed-tuned model consistently
outperformed the relevant-tuned model by providing more accurate, complete,
and contextually aligned answers. The mixed model’s superior performance can
be attributed to its enhanced ability to effectively understand and integrate
diverse contexts, accurately recognize entities, and reason over temporal and
historical information. These findings underscore the effectiveness of our
mixed-tuned model, which leverages contextual information, thereby confirming
our hypothesis that integrating diverse contexts is crucial for temporal
question-answering systems.

##  5 Conclusion

This study examined how contextual information impacts the performance of
temporal question-answering (TQA) systems. Our experiments showed that
integrating and fine-tuning contextual information significantly improves the
accuracy and reliability of these systems. The most effective strategy was
training on a mixture of relevant and slightly noisy contexts, resulting in
robust performance across various context types during testing.

The superior performance of our fine-tuned models underscores the crucial role
of context-aware training in enhancing temporal QA capabilities. By
effectively utilizing diverse contextual information, our models demonstrated
improved entity grounding, a better understanding of temporal relations, and
more accurate, context-aligned responses. Training with different context
types also appeared to enhance the model’s ability to ignore irrelevant
context when answering questions, thereby increasing robustness.

Future research should explore advanced methods for generating and integrating
context and further investigate the effects of context placement on model
performance. Understanding how contextual information impacts temporal
reasoning is essential for developing more sophisticated and precise QA
systems capable of effectively answering complex temporal questions. Moreover,
there are natural extensions to other tasks in this work. For instance, we
find it possible to target specific types of information during training to
make the model ignore it. This can have implications in other areas, e.g.,
hate speech or offensive language.

##  6 Limitations

While our study provides valuable insights into the robustness of temporal
question-answering systems, several limitations should be acknowledged. Our
experiments are constrained by the quality and variety of the datasets used.
Although we introduced two new datasets, ContextAQA and ContextTQE, these
datasets may not encompass the full range of temporal question types and
contextual variations encountered in real-world applications. Additionally,
the contexts generated by GPT-3.5, while generally accurate, may not perfectly
reflect the complexity and diversity of naturally occurring contexts.

The context generation process relied on GPT-3.5, which can introduce biases
and inaccuracies. A manual review of the contexts revealed a small percentage
of errors, which could impact model training and evaluation. Future work
should consider using more robust methods for context generation or augmenting
generated contexts with human verification to ensure higher accuracy and
relevance.

While more than prior research, our findings are based on limited models,
including various GEMMA, Mistral, and LLaMA versions. While these models
represent a range of state-of-the-art architectures, the conclusions drawn may
not generalize to all large language models (LLMs). Further research is needed
to validate the robustness of our findings across a broader array of models,
including those with different architectures and training paradigms.

While we used standard metrics such as F1 score, accuracy, and BERT-based
Answer Equivalence Metric (BEM), these metrics may not fully capture the
nuances of temporal reasoning and contextual understanding. Developing more
nuanced evaluation metrics that better reflect the complexity of temporal
question answering would provide a more comprehensive assessment of model
performance.

Our study found that training in a mix of relevant and irrelevant contexts
improves robustness. However, the effectiveness of this approach may vary
depending on the context’s nature and degree of irrelevance. Further
exploration is needed to understand the boundaries of this robustness and to
identify the optimal mix of context types for training.

Finally, the training and evaluation of large language models require
substantial computational resources. This limitation may restrict the ability
to conduct more extensive experiments and test additional hypotheses.
Moreover, it restricts our use of larger models, which is why we focused on
2b, 7b, and 8b models because of this. Future research could benefit from more
efficient training techniques and the availability of greater computational
power.

## References

  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Andriushchenko et al. (2024) Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024.  Jailbreaking leading safety-aligned llms with simple adaptive attacks.  _arXiv preprint arXiv:2404.02151_.
  * Beniwal et al. (2024) Himanshu Beniwal, Mayank Singh, et al. 2024.  Remember this event that year? assessing temporal information and reasoning in large language models.  _arXiv preprint arXiv:2402.11997_.
  * Chen et al. (2023a) Yuyan Chen, Qiang Fu, Yichen Yuan, Zhihao Wen, Ge Fan, Dayiheng Liu, Dongmei Zhang, Zhixu Li, and Yanghua Xiao. 2023a.  Hallucination detection: Robustly discerning reliable answers in large language models.  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , pages 245–255.
  * Chen et al. (2023b) Ziyang Chen, Dongfang Li, Xiang Zhao, Baotian Hu, and Min Zhang. 2023b.  Temporal knowledge question answering via abstract reasoning induction.  _arXiv preprint arXiv:2311.09149_.
  * Cheng et al. (2024) Chen Cheng, Xinzhi Yu, Haodong Wen, Jingsong Sun, Guanzhang Yue, Yihao Zhang, and Zeming Wei. 2024.  Exploring the robustness of in-context learning with noisy labels.  In _ICLR 2024 Workshop on Reliable and Responsible Foundation Models_.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022.  Time-aware language models as temporal knowledge bases.  _Transactions of the Association for Computational Linguistics_ , 10:257–273.
  * Gade and Jetcheva (2024) Anoushka Gade and Jorjeta Jetcheva. 2024.  It’s about time: Incorporating temporality in retrieval augmented language models.  _arXiv preprint arXiv:2401.13222_.
  * Gao et al. (2024) Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, and Dongsheng Li. 2024\.  Two-stage generative question answering on temporal knowledge graph using large language models.  _arXiv preprint arXiv:2402.16568_.
  * Gruber et al. (2024) Raphael Gruber, Abdelrahman Abdallah, Michael Färber, and Adam Jatowt. 2024.  Complextempqa: A large-scale dataset for complex temporal question answering.  _Preprint_ , arXiv:2406.04866.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  Lora: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jang et al. (2017) Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017.  Tgif-qa: Toward spatio-temporal reasoning in visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2758–2766.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kannen et al. (2023) Nithish Kannen, Udit Sharma, Sumit Neelam, Dinesh Khandelwal, Shajith Ikbal, Hima Karanam, and L Subramaniam. 2023.  Best of both worlds: Towards improving temporal knowledge base question answering via targeted fact extraction.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4729–4744.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kumar et al. (2023) Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Soheil Feizi, and Hima Lakkaraju. 2023\.  Certifying llm safety against adversarial prompting.  _arXiv preprint arXiv:2309.02705_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2022) Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022.  Large language models with controllable working memory.  _arXiv preprint arXiv:2211.05110_.
  * Li et al. (2023) Xingxuan Li, Liying Cheng, Qingyu Tan, Hwee Tou Ng, Shafiq Joty, and Lidong Bing. 2023.  Unlocking temporal question answering for large language models using code execution.  _arXiv preprint arXiv:2305.15014_.
  * Liu et al. (2024) Lei Liu, Shuo Yu, Runze Wang, Zhenxun Ma, and Yanming Shen. 2024.  How can large language models understand spatial-temporal data?  _arXiv preprint arXiv:2401.14192_.
  * Lorenzini et al. (2022) Jasmine Lorenzini, Hanspeter Kriesi, Peter Makarov, and Bruno Wüest. 2022.  Protest event analysis: Developing a semiautomated nlp approach.  _American Behavioral Scientist_ , 66(5):555–577.
  * Qiu et al. (2023) Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M Ponti, and Shay B Cohen. 2023.  Are large language models temporally grounded?  _arXiv preprint arXiv:2311.08398_.
  * Raina et al. (2024) Vyas Raina, Adian Liusie, and Mark Gales. 2024.  Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment.  _arXiv preprint arXiv:2402.14016_.
  * Saxena et al. (2021) Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021.  Question answering over temporal knowledge graphs.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6663–6676.
  * Shaier et al. (2024) Sagi Shaier, Lawrence E Hunter, and Katharina von der Wense. 2024.  Desiderata for the context use of question answering systems.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 777–792, Online. Association for Computational Linguistics.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. 2023.  Large language models can be easily distracted by irrelevant context.  In _International Conference on Machine Learning_ , pages 31210–31227. PMLR.
  * Tan et al. (2023) Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023.  Towards benchmarking and improving the temporal reasoning capability of large language models.  _arXiv preprint arXiv:2306.08952_.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Wallat et al. (2024) Jonas Wallat, Adam Jatowt, and Avishek Anand. 2024.  Temporal blind spots in large language models.  In _Proceedings of the 17th ACM International Conference on Web Search and Data Mining_ , pages 683–692.
  * Wang et al. (2021) Jiexin Wang, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2021.  Improving question answering for event-focused questions in temporal collections of news articles.  _Information Retrieval Journal_ , 24:29–54.
  * Wang et al. (2022) Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022.  Archivalqa: a large-scale benchmark dataset for open-domain question answering over historical news collections.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 3025–3035.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_.
  * Xiao et al. (2021) Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021.  Next-qa: Next phase of question-answering to explaining temporal actions.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9777–9786.
  * Yoran et al. (2024) Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024.  Making retrieval-augmented language models robust to irrelevant context.  In _International Conference on Learning Representations (ICLR)_. Tel Aviv University, Allen Institute for AI.
  * Yuan et al. (2024) Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2024.  Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhang et al. (2024) Hao Zhang, Yuyang Zhang, Xiaoguang Li, Wenxuan Shi, Haonan Xu, Huanshuo Liu, Yasheng Wang, Lifeng Shang, Qun Liu, Yong Liu, et al. 2024.  Evaluating the external and parametric knowledge fusion of large language models.  _arXiv preprint arXiv:2405.19010_.
  * Zhao and Rios (2024) Xingmeng Zhao and Anthony Rios. 2024.  Utsa-nlp at chemotimelines 2024: Evaluating instruction-tuned language models for temporal relation extraction.  In _Proceedings of the 6th Clinical Natural Language Processing Workshop_. Association for Computational Linguistics.

##  Appendix A Appendix

##  Appendix B Model Training Details

We train eight total models, including gemma-2b Team et al. (2024),
gemma-1.1-2b-it Team et al. (2024), Mistral-7B-v0.1 Jiang et al. (2023),
Mistral-7B-Instruct-v0.2 Jiang et al. (2023), llama-3-8b AI@Meta (2024), and
llama-3-8b-Instruct AI@Meta (2024). These models are fine-tuned on five types
of context: no context, relevant context, Irrelevant context, slightly-altered
(context with falsified dates), and mixed context. This process gives us a
total of 40 fine-tuned models. All models are fine-tuned using Low-Rank
Adaptation (LoRA) Hu et al. (2021). For Mistral models, a HuggingFace trainer
is utilized, while for other models, we used a Supervised Fine-Tuning (SFT)
trainer from HuggingFace Wolf et al. (2019) to adapt the base models for the
respective context types. Furthermore, for LLaMA3 models, the Unsloth
framework, a technique designed to optimize fine-tuning efficiency for large
language models, is utilized. All models are trained for one epoch using the
8-bit Adam optimizer Kingma and Ba (2014) with a learning rate 2e-5. After
manual experimentation on the validation datasets, we found these parameters
work well across different models.

Overall, for each of the 40 fine-tuned models, answers are generated for the
four context types: no context, relevant context, Irrelevant context, and
slightly-altered context. On our test set, this results in 160 generated
outputs for evaluation. The context is used by placing it after the input
question during generation. Additionally, an experiment is conducted where the
context is placed before the question instead to investigate whether providing
the context before the question improves generated answers.

##  Appendix C Generation Parameters

To obtain high-quality responses, we carefully tuned the generation parameters
for our fine-tuned and and base models separately. The parameters were
optimized to achieve a balance between coherence, diversity, and relevance of
the generated outputs. The key parameters include the maximum number of tokens
to generate (max_new_tokens), sampling technique (do_sample, temperature),
repetition penalty (repetition_penalty), search settings (num_beams, top_p,
top_k), and length penalty (length_penalty). The specific parameter values are
as follows:

  * •

max_new_tokens: Set to 16 due to the short nature of answers our dataset.

  * •

do_sample: Determines whether to use sampling or greedy decoding.

  * •

temperature: Controls randomness in sampling.

  * •

repetition_penalty: Discourages token repetition.

  * •

num_beams: Utilized beam search with multiple beams, especially for the more
complex ContextAQA dataset.

  * •

top_p: Cumulative probability mass for top-p sampling.

  * •

top_k: Limits the number of highest probability tokens considered.

  * •

length_penalty: Adjusted to handle the typically short answers.

Initially, we set these parameters based on our knowledge of the models, and
then iteratively refined them to determine the optimal configuration for each
dataset. They are optimized for every model.

##  Appendix D Prompt used to make relevant context

Here is the prompt we used for generating relevant context:

> You will be provided with a question and an answer. Let’s play a game where
> you pretend that all of the answers are correct. Your job is to output a
> *Generated Context* section. In this section, generate some relevant context
> that would help answer the question. The context should be in the same style
> and format as a wikipedia article.
>
> Question: What Movie won seven Oscars in 1994?
>
> Answer: Schindler’s List
>
> Generated Context: Schindler’s List premiered on November 30, 1993, in
> Washington, D.C., and was released on December 15, 1993, in the United
> States. Often listed among the greatest films ever made, the film received
> widespread critical acclaim for its tone, acting (particularly Neeson,
> Fiennes, and Kingsley), atmosphere, score, cinematography, and Spielberg’s
> direction; it was also a box office success, earning$322.2 million worldwide
> on a $22 million budget. It was nominated for twelve awards at the 66th
> Academy Awards, and won seven, including Best Picture, Best Director (for
> Spielberg), Best Adapted Screenplay, and Best Original Score. The film won
> numerous other awards, including seven BAFTAs and three Golden Globe Awards.
> In 2007, the American Film Institute ranked Schindler’s List 8th on its list
> of the 100 best American films of all time. The film was designated as
> "culturally, historically or aesthetically significant" by the Library of
> Congress in 2004 and selected for preservation in the United States National
> Film Registry.
>
> Question: *plug in question*
>
> Answer: *plug in answer*
>
> Generated Context: *GPT starts generation here*

Model | No Test Set | Rel Test Set | WD Test Set | Irr Test Set | Average BEM
---|---|---|---|---|---
| BEM | BEM | BEM | BEM | BEM
Mistral-7B-Instruct-v0.2 + No | .743 | .772 | .769 | .739 | .756
Mistral-7B-Instruct-v0.2 + Rel | .788 | .779 | .777 | .676 | .755
Mistral-7B-Instruct-v0.2 + SA | .796 | .782 | .781 | .674 | .758
Mistral-7B-Instruct-v0.2 + Irr | .743 | .743 | .741 | .743 | .743
Mistral-7B-Instruct-v0.2 + Mixed | .744 | .773 | .774 | .744 | .759
Llama-3-8B-Instruct + No | .822 | .836 | .832 | .767 | .814
Llama-3-8B-Instruct + Rel | .818 | .955 | .929 | .692 | .849
Llama-3-8B-Instruct + SA | .817 | .955 | .955 | .693 | .855
Llama-3-8B-Instruct + Irr | .823 | .816 | .816 | .825 | .820
Llama-3-8B-Instruct + Mixed | .824 | .952 | .951 | .824 | .888
Mistral-7B + No | .741 | .786 | .782 | .750 | .765
Mistral-7B + Rel | .814 | .869 | .832 | .689 | .801
Mistral-7B + SA | .814 | .814 | .814 | .685 | .782
Mistral-7B + Irr | .769 | .765 | .767 | .772 | .768
Mistral-7B + Mixed | .775 | .796 | .795 | .763 | .782
Llama-3-8B + No | .822 | .798 | .797 | .742 | .790
Llama-3-8B + Rel | .818 | .955 | .915 | .688 | .844
Llama-3-8B + SA | .817 | .955 | .955 | .689 | .854
Llama-3-8B + Irr | .822 | .817 | .817 | .826 | .821
Llama-3-8B + Mixed | .823 | .953 | .952 | .823 | .888

Table 3: Performance of mixed models on various test sets. The table shows BEM
scores for different context types and their averages across four test sets.
"Average BEM" represents the average of all BEM scores for each model
configuration.

Model | No Test Set | Rel Test Set | SA Test Set | Irr Test Set | Average Scores
---|---|---|---|---|---
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Average Acc | Average F1
Baselines
GPT-3.5 | .760 | .153 | .890 | .147 | .850 | .119 | .640 | .097 | .785 | .129
Mistral-7B-Instruct-v0.2 | .413 | .329 | .880 | .592 | .666 | .333 | .186 | .157 | .536 | .353
gemma-1.1-7b-it | .360 | .231 | .867 | .555 | .613 | .376 | .080 | .060 | .480 | .306
llama-3-8b-Instruct | .560 | .505 | .866 | .868 | .786 | .707 | .493 | .477 | .676 | .639
Fine-tuned Models
Mistral-7B-Instruct-v0.2 + No | .480 | .510 | .720 | .754 | .546 | .556 | .386 | .417 | .533 | .559
gemma-1.1-7b-it + No | .387 | .245 | .853 | .524 | .533 | .283 | .053 | .049 | .457 | .275
llama-3-8b-Instruct + No | .560 | .617 | .813 | .892 | .786 | .841 | .493 | .566 | .663 | .729
Mistral-7B-Instruct-v0.2 + Rel | .520 | .530 | .840 | .886 | .826 | .787 | .240 | .249 | .607 | .613
gemma-1.1-7b-it + Rel | .373 | .242 | .840 | .478 | .453 | .223 | .040 | .039 | .427 | .245
llama-3-8b-Instruct + Rel | .573 | .629 | .880 | .945 | .840 | .918 | .346 | .402 | .660 | .724
Mistral-7B-Instruct-v0.2 + SA | .573 | .623 | .880 | .916 | .880 | .927 | .373 | .439 | .677 | .726
gemma-1.1-7b-it + SA | .400 | .258 | .867 | .518 | .773 | .455 | .107 | .077 | .537 | .327
llama-3-8b-Instruct + SA | .573 | .650 | .880 | .948 | .880 | .949 | .400 | .457 | .683 | .751
Mistral-7B-Instruct-v0.2 + Rand | .480 | .502 | .386 | .344 | .306 | .293 | .453 | .456 | .406 | .399
gemma-1.1-7b-it + Irr | .387 | .258 | .867 | .572 | .707 | .413 | .333 | .220 | .573 | .366
llama-3-8b-Instruct + Irr | .560 | .625 | .666 | .719 | .666 | .719 | .546 | .607 | .610 | .668
Mistral-7B-Instruct-v0.2 + Mixed | .506 | .538 | .866 | .802 | .773 | .720 | .426 | .452 | .643 | .628
gemma-1.1-7b-it + Mixed | .400 | .259 | .867 | .538 | .747 | .438 | .280 | .179 | .573 | .354
llama-3-8b-Instruct + Mixed | .540 | .610 | .840 | .920 | .840 | .920 | .600 | .660 | .705 | .778

Table 4: ContextTQE performance of different model configurations on temporal
QA across four context types: no context (No), relevant (Rel), slightly-
altered (SA), irrelevant (Irr). Models include base LLMs
(Mistral-7B-Instruct-v0.2, llama-3-8b-Instruct) and fine-tuned variants
trained on various context setups (+No, +Rel, +SA, +Irr, +Mixed). Results
highlight the impact of context types and fine-tuning strategies on model
robustness and accuracy.

Model | No Test Set | Rel Test Set | SA Test Set | Irr Test Set | Average Scores
---|---|---|---|---|---
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Average Acc | Average F1
Baselines
GPT-3.5 | .760 | .153 | .890 | .147 | .850 | .119 | .640 | .097 | .785 | .129
gemma-2b | .320 | .220 | .707 | .554 | .680 | .531 | .080 | .084 | .447 | .347
gemma-1.1-2b-it | .200 | .110 | .747 | .458 | .680 | .386 | .040 | .024 | .417 | .244
gemma-7b | .453 | .298 | .827 | .682 | .747 | .611 | .387 | .321 | .603 | .478
gemma-1.1-7b-it | .360 | .231 | .867 | .555 | .613 | .376 | .080 | .060 | .480 | .306
Mistral-7B-v0.1 | .533 | .571 | .293 | .237 | .173 | .131 | .253 | .223 | .313 | .291
Mistral-7B-Instruct-v0.2 | .413 | .329 | .880 | .592 | .666 | .333 | .186 | .157 | .536 | .353
llama-3-8b | .493 | .485 | .786 | .785 | .786 | .794 | .333 | .371 | .600 | .609
llama-3-8b-Instruct | .560 | .505 | .866 | .868 | .786 | .707 | .493 | .477 | .676 | .639
Fine-tuned Models
gemma-2b + No | .333 | .244 | .760 | .625 | .693 | .561 | .133 | .124 | .480 | .389
gemma1.1-2b-it + No | .227 | .161 | .787 | .473 | .720 | .432 | .040 | .026 | .443 | .273
gemma-7b + No | .440 | .305 | .827 | .687 | .773 | .627 | .413 | .357 | .613 | .494
gemma1.1-7b-it + No | .387 | .245 | .853 | .524 | .533 | .283 | .053 | .049 | .457 | .275
Mistral-7B-v0.1 + No | .560 | .622 | .733 | .762 | .626 | .664 | .386 | .449 | .576 | .624
Mistral-7B-Instruct-v0.2 + No | .480 | .510 | .720 | .754 | .546 | .556 | .386 | .417 | .533 | .559
llama-3-8b + No | .546 | .633 | .826 | .896 | .800 | .862 | .480 | .533 | .663 | .731
llama-3-8b-Instruct + No | .560 | .617 | .813 | .892 | .786 | .841 | .493 | .566 | .663 | .729
gemma-2b + Rel | .307 | .213 | .747 | .587 | .747 | .589 | .133 | .128 | .483 | .379
gemma1.1-2b-it + Rel | .187 | .104 | .733 | .421 | .653 | .366 | .040 | .023 | .403 | .228
gemma-7b + Rel | .507 | .368 | .840 | .700 | .800 | .679 | .387 | .314 | .633 | .515
gemma1.1-7b-it + Rel | .373 | .242 | .840 | .478 | .453 | .223 | .040 | .039 | .427 | .245
Mistral-7B-v0.1 + Rel | .560 | .574 | .853 | .918 | .840 | .905 | .426 | .492 | .670 | .722
Mistral-7B-Instruct-v0.2 + Rel | .520 | .530 | .840 | .886 | .826 | .787 | .240 | .249 | .607 | .613
llama-3-8b + Rel | .546 | .606 | .880 | .949 | .866 | .935 | .306 | .347 | .650 | .709
llama-3-8b-Instruct + Rel | .573 | .629 | .880 | .945 | .840 | .918 | .346 | .402 | .660 | .724
gemma-2b + Irr | .293 | .232 | .733 | .586 | .707 | .563 | .173 | .149 | .477 | .383
gemma1.1-2b-it + Irr | .213 | .152 | .787 | .479 | .760 | .424 | .080 | .036 | .460 | .273
gemma-7b + Irr | .493 | .344 | .840 | .702 | .813 | .668 | .480 | .404 | .657 | .529
gemma1.1-7b-it + Irr | .387 | .258 | .867 | .572 | .707 | .413 | .333 | .220 | .573 | .366
Mistral-7B-v0.1 + Irr | .586 | .640 | .333 | .399 | .346 | .402 | .533 | .590 | .450 | .508
Mistral-7B-Instruct-v0.2 + Irr | .480 | .502 | .386 | .344 | .306 | .293 | .453 | .456 | .406 | .399
llama-3-8b + Irr | .600 | .662 | .506 | .573 | .520 | .585 | .613 | .669 | .560 | .622
llama-3-8b-Instruct + Irr | .560 | .625 | .666 | .719 | .666 | .719 | .546 | .607 | .610 | .668
gemma-2b + SA | .320 | .224 | .773 | .621 | .747 | .589 | .147 | .122 | .497 | .389
gemma1.1-2b-it + SA | .200 | .108 | .733 | .421 | .693 | .391 | .040 | .023 | .417 | .236
gemma-7b + SA | .533 | .358 | .840 | .705 | .800 | .682 | .387 | .320 | .640 | .516
gemma1.1-7b-it + SA | .400 | .258 | .867 | .518 | .773 | .455 | .107 | .077 | .537 | .327
Mistral-7B-v0.1 + SA | .573 | .623 | .880 | .916 | .880 | .927 | .373 | .439 | .677 | .726
Mistral-7B-Instruct-v0.2 + SA | .533 | .548 | .853 | .882 | .813 | .831 | .253 | .260 | .613 | .630
llama-3-8b + SA | .520 | .586 | .840 | .925 | .853 | .927 | .280 | .322 | .623 | .690
llama-3-8b-Instfruct + SA | .573 | .650 | .880 | .948 | .880 | .949 | .400 | .457 | .683 | .751
gemma-2b + Mixed | .333 | .226 | .760 | .618 | .733 | .576 | .147 | .136 | .493 | .389
gemma1.1-2b-it + Mixed | .227 | .141 | .760 | .451 | .733 | .417 | .080 | .041 | .450 | .262
gemma-7b + Mixed | .507 | .354 | .827 | .698 | .773 | .698 | .453 | .355 | .640 | .526
gemma1.1-7b-it + Mixed | .400 | .259 | .867 | .538 | .747 | .438 | .280 | .179 | .573 | .354
Mistral-7B-v0.1 + Mixed | .400 | .259 | .867 | .538 | .747 | .438 | .280 | .179 | .573 | .354
Mistral-7B-Instruct-v0.2 + Mixed | .600 | .656 | .826 | .891 | .800 | .876 | .506 | .570 | .683 | .748
llama-3-8b + Mixed | .506 | .538 | .866 | .802 | .773 | .720 | .426 | .452 | .643 | .628
llama-3-8b-Instruct + Mixed | .546 | .616 | .840 | .926 | .840 | .926 | .600 | .666 | .707 | .784

Table 5: ContextTQE Performance of different model configurations on various
context types: no context (No), relevant (Rel), slightly-altered (SA),
Irrelevant (Irr). Models include base LLMs (Mistral-7B-Instruct-v0.2,
llama-3-8b-Instruct) and fine-tuned variants trained on various context setups
(+No, +Rel, +SA, +Irr, +Mixed). Results highlight the impact of context types
and fine-tuning strategies on model robustness and accuracy.

Model | No Test Set | Rel Test Set | SA Test Set | Irr Test Set | Average Scores
---|---|---|---|---|---
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Average Acc | Average F1
Baselines
GPT-3.5 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA
gemma-2b | .110 | .048 | .703 | .338 | .610 | .278 | .091 | .069 | .378 | .183
gemma-1.1-2b-it | .008 | .020 | .744 | .579 | .466 | .307 | .001 | .022 | .305 | .232
gemma-7b | .178 | .062 | .740 | .454 | .610 | .338 | .166 | .112 | .424 | .242
gemma-1.1-7b-it | .041 | .028 | .760 | .584 | .555 | .310 | .009 | .028 | .341 | .238
Mistral-7B-v0.1 | .158 | .178 | .618 | .574 | .524 | .488 | .140 | .163 | .360 | .351
Mistral-7B-Instruct-v0.2 | .138 | .119 | .688 | .401 | .485 | .242 | .062 | .059 | .343 | .205
llama-3-8b | .140 | .164 | .565 | .565 | .449 | .426 | .113 | .133 | .317 | .322
llama-3-8b-Instruct | .183 | .196 | .714 | .643 | .551 | .440 | .139 | .143 | .397 | .356
Fine-tuned Models
gemma-2b + No | .091 | .049 | .544 | .405 | .433 | .328 | .066 | .083 | .259 | .131
gemma1.1-2b-it + No | .117 | .154 | .747 | .586 | .465 | .387 | .089 | .122 | .227 | .251
gemma-7b + No | .125 | .055 | .510 | .419 | .398 | .336 | .085 | .100 | .165 | .128
gemma1.1-7b-it + No | .210 | .280 | .720 | .790 | .600 | .680 | .170 | .230 | .426 | .419
Mistral-7B-v0.1 + No | .263 | .199 | .730 | .489 | .643 | .437 | .221 | .180 | .079 | .076
Mistral-7B-Instruct-v0.2 + No | .273 | .143 | .752 | .349 | .592 | .274 | .230 | .130 | .286 | .151
llama-3-8b + No | .231 | .301 | .621 | .698 | .452 | .525 | .200 | .270 | .215 | .275
llama-3-8b-Instruct + No | .227 | .297 | .584 | .668 | .425 | .500 | .198 | .263 | .175 | .234
gemma-2b + Rel | .116 | .052 | .647 | .364 | .502 | .244 | .059 | .053 | .331 | .178
gemma1.1-2b-it + Rel | .109 | .096 | .745 | .748 | .697 | .710 | .038 | .087 | .397 | .410
gemma-7b + Rel | .110 | .052 | .568 | .354 | .431 | .260 | .058 | .059 | .292 | .181
gemma1.1-7b-it + Rel | .173 | .054 | .815 | .770 | .681 | .671 | .076 | .094 | .436 | .397
Mistral-7B-v0.1 + Rel | .214 | .275 | .716 | .713 | .690 | .627 | .112 | .137 | .433 | .438
Mistral-7B-Instruct-v0.2 + Rel | .226 | .231 | .757 | .370 | .712 | .338 | .154 | .108 | .462 | .262
llama-3-8b + Rel | .219 | .286 | .772 | .826 | .728 | .782 | .134 | .184 | .463 | .520
llama-3-8b-Instruct + Rel | .217 | .283 | .746 | .803 | .664 | .721 | .097 | .143 | .431 | .488
gemma-2b + Irr | .114 | .048 | .515 | .247 | .330 | .166 | .077 | .062 | .259 | .131
gemma1.1-2b-it + Irr | .118 | .116 | .444 | .440 | .278 | .319 | .069 | .129 | .227 | .251
gemma-7b + Irr | .072 | .045 | .333 | .238 | .198 | .156 | .057 | .071 | .165 | .128
gemma1.1-7b-it + Irr | .189 | .058 | .800 | .793 | .543 | .583 | .172 | .243 | .426 | .419
Mistral-7B-v0.1 + Irr | .235 | .214 | .032 | .033 | .038 | .037 | .011 | .019 | .079 | .076
Mistral-7B-Instruct-v0.2 + Irr | .257 | .140 | .485 | .237 | .360 | .188 | .041 | .039 | .286 | .151
llama-3-8b + Irr | .237 | .307 | .285 | .347 | .237 | .295 | .100 | .151 | .215 | .275
llama-3-8b-Instruct + Irr | .240 | .308 | .203 | .261 | .168 | .227 | .088 | .138 | .175 | .234
gemma-2b + SA | .115 | .051 | .593 | .311 | .539 | .269 | .044 | .045 | .323 | .169
gemma1.1-2b-it + SA | .105 | .096 | .743 | .654 | .737 | .660 | .027 | .071 | .403 | .370
gemma-7b + SA | .149 | .059 | .540 | .243 | .549 | .245 | .071 | .065 | .327 | .153
gemma1.1-7b-it + SA | .181 | .055 | .838 | .800 | .824 | .793 | .075 | .116 | .480 | .441
Mistral-7B-v0.1 + Sa | .215 | .274 | .718 | .625 | .720 | .613 | .096 | .110 | .437 | .406
Mistral-7B-Instruct-v0.2 + SA | .226 | .244 | .751 | .388 | .743 | .370 | .158 | .112 | .470 | .279
llama-3-8b + SA | .220 | .288 | .762 | .822 | .755 | .815 | .084 | .124 | .455 | .512
llama-3-8b-Instruct + SA | .215 | .281 | .749 | .810 | .737 | .798 | .085 | .127 | .447 | .504
gemma-2b + Mixed | .170 | .120 | .800 | .320 | .790 | .320 | .160 | .100 | .480 | .215
gemma1.1-2b-it + Mixed | .140 | .200 | .700 | .770 | .690 | .750 | .150 | .100 | .420 | .455
gemma-7b + Mixed | .250 | .150 | .810 | .370 | .810 | .370 | .240 | .180 | .528 | .268
gemma1.1-7b-it + Mixed | .210 | .270 | .750 | .810 | .730 | .800 | .150 | .200 | .460 | .520
Mistral-7B-v0.1 + Mixed | .221 | .243 | .752 | .557 | .744 | .539 | .190 | .180 | .477 | .380
Mistral-7B-Instruct-v0.2 + Mixed | .259 | .141 | .777 | .342 | .769 | .340 | .161 | .092 | .492 | .229
llama-3-8b + Mixed | .239 | .310 | .760 | .829 | .735 | .806 | .194 | .257 | .482 | .551
llama-3-8b-Instruct + Mixed | .237 | .307 | .732 | .799 | .711 | .780 | .133 | .186 | .453 | .518

Table 6: ContextAQA Context-First Performance of different model
configurations on various context types: no context (No), relevant (Rel),
wrong dates (WD), random (Rand). Models include base LLMs
(Mistral-7B-Instruct-v0.2, llama-3-8b-Instruct) and fine-tuned variants
trained on various context setups (+No, +Rel, +WD, +Rand, +Mixed). Results
highlight the impact of context types and fine-tuning strategies on model
robustness and accuracy.

##  Appendix E Additional Results

Table 3 shows the BERT-based Answer Equivalence Metric (BEM). We found this
metric to be more correlated with the length of the answer rather than its
semantic meaning, which is why it was not included in our main findings.
Additionally, the BEM scores never fall below 0.6, which can be misleading.

Table 4 presents our focused results for the question-first ContextTQE. This
table is in the same format as Table 1, allowing readers to compare our
findings across multiple datasets easily. Table 6 expands upon Table 1 by
including all our results on the ContextTQE dataset. It is important to note
that the ContextTQE dataset is much smaller than the AQA dataset, so not all
aspects can easily be learned (e.g., with the mixed data), but we still see
many of the important findings show up. Adding some noise with a slightly
altered context improves performance. Moreover, training on irrelevant context
causes poor performance because the model still can learn to ignore it, though
not at the magnitude of the AQA dataset because of the limited size. Table 7
provides a focused look at ContextAQA, similar to Table 1, but with a context-
first approach rather than question-first.

Model | No Test Set | Rel Test Set | SA Test Set | Irr Test Set | Average Scores
---|---|---|---|---|---
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Average Acc | Average F1
Baselines
Mistral-7B-Instruct-v0.2 | .138 | .119 | .688 | .401 | .485 | .242 | .062 | .059 | .343 | .205
gemma-1.1-7b-it | .041 | .028 | .760 | .584 | .555 | .310 | .009 | .028 | .341 | .238
llama-3-8b-Instruct | .180 | .190 | .710 | .640 | .550 | .440 | .130 | .140 | .393 | .353
Fine-tuned Models
Mistral-7B-Instruct-v0.2 + No | .270 | .140 | .750 | .340 | .590 | .270 | .230 | .130 | .460 | .220
gemma-1.1-7b-it + No | .210 | .280 | .720 | .790 | .600 | .680 | .170 | .230 | .425 | .495
llama-3-8b-Instruct + No | .220 | .290 | .580 | .660 | .420 | .500 | .190 | .260 | .353 | .428
Mistral-7B-Instruct-v0.2 + Rel | .220 | .230 | .750 | .370 | .710 | .330 | .150 | .100 | .458 | .258
gemma-1.1-7b-it + Rel | .173 | .054 | .815 | .770 | .681 | .671 | .076 | .094 | .436 | .397
llama-3-8b-Instruct + Rel | .210 | .280 | .740 | .800 | .660 | .720 | .090 | .140 | .425 | .485
Mistral-7B-Instruct-v0.2 + SA | .220 | .240 | .750 | .380 | .740 | .370 | .150 | .110 | .465 | .275
gemma-1.1-7b-it + SA | .181 | .055 | .838 | .800 | .824 | .793 | .075 | .116 | .480 | .441
llama-3-8b-Instruct + SA | .210 | .280 | .740 | .810 | .730 | .790 | .080 | .120 | .440 | .500
Mistral-7B-Instruct-v0.2 + Irr | .250 | .140 | .480 | .230 | .360 | .180 | .040 | .030 | .283 | .145
gemma-1.1-7b-it + Irr | .189 | .058 | .800 | .793 | .543 | .583 | .172 | .243 | .426 | .419
llama-3-8b-Instruct + Irr | .300 | .240 | .200 | .260 | .160 | .220 | .080 | .130 | .185 | .213
Mistral-7B-Instruct-v0.2 + Mixed | .250 | .140 | .770 | .340 | .760 | .340 | .160 | .090 | .485 | .228
gemma-1.1-7b-it + Mixed | .210 | .270 | .750 | .810 | .730 | .800 | .150 | .200 | .460 | .520
llama-3-8b-Instruct + Mixed | .230 | .300 | .730 | .790 | .710 | .780 | .130 | .180 | .450 | .513

Table 7: ContextAQA Context-first performance of different model
configurations on temporal QA across four context types: no context (No),
relevant (Rel), slightly-altered (SA), irrelevant (Irr). Models include base
LLMs (Mistral-7B-Instruct-v0.2, llama-3-8b-Instruct) and fine-tuned variants
trained on various context setups (+No, +Rel, +SA, +Irr, +Mixed). Results
highlight the impact of context types and fine-tuning strategies on model
robustness and accuracy.