# Faithful Temporal Question Answering over Heterogeneous Sources

Zhen Jia  Southwest Jiaotong UniversityChengduChina<EMAIL_ADDRESS>,
Philipp Christmann  Max Planck Institute for Informatics
Saarland Informatics CampusSaarbrueckenGermany<EMAIL_ADDRESS>and
Gerhard Weikum  Max Planck Institute for Informatics
Saarland Informatics CampusSaarbrueckenGermany<EMAIL_ADDRESS>
(2024)

###### Abstract.

Temporal question answering (QA) involves time constraints, with phrases such
as “…in 2019” or “…before COVID”. In the former, time is an explicit
condition, in the latter it is implicit. State-of-the-art methods have
limitations along three dimensions. First, with neural inference, time
constraints are merely soft-matched, giving room to invalid or inexplicable
answers. Second, questions with implicit time are poorly supported. Third,
answers come from a single source: either a knowledge base (KB) or a text
corpus. We propose a temporal QA system that addresses these shortcomings.
First, it enforces temporal constraints for faithful answering with tangible
evidence. Second, it properly handles implicit questions. Third, it operates
over heterogeneous sources, covering KB, text and web tables in a unified
manner. The method has three stages: (i) understanding the question and its
temporal conditions, (ii) retrieving evidence from all sources, and (iii)
faithfully answering the question. As implicit questions are sparse in prior
benchmarks, we introduce a principled method for generating diverse questions.
Experiments show superior performance over a suite of baselines.

Question Answering, Temporal Questions, Explainability

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
ACM Web Conference 2024; May 13–17, 2024; Singapore, Singapore††booktitle:
Proceedings of the ACM Web Conference 2024 (WWW ’24), May 13–17, 2024,
Singapore, Singapore††ccs: Information systems Question answering Figure 1.
Overview of the Faith pipeline. The figure illustrates the process for
answering $q_{3}$ (“Queen’s record company when recording Bohemian Rhapsody?”)
and $q_{1}$ (“Record company of Queen in 1975?”). For answering $q_{3}$, two
intermediate questions $q_{31}$ and $q_{32}$ are generated, and run
recursively through the entire temporal QA system.

##  1\. Introduction

Motivation. Question answering (QA) comprises a spectrum of settings for
satisfying users’ information needs, ideally giving crisp, entity-level
answers to natural-language utterances (Roy and Anand, 2022). Temporal QA
specifically focuses on questions with temporal conditions (e.g., (Jia et al.,
2021; Saxena et al., 2021; Mavromatis et al., 2022)), making up a substantial
portion of user needs (Zhang et al., 2010), but poses challenges that are not
properly met by universal QA systems. Consider the following example:

$q_{1}$: Record company of Queen in 1975?

The band Queen had different record companies over the years, so it is
decisive to consider the explicit temporal constraint (“in 1975”). Other
questions with explicit time are lookups of dates, such as:

$q_{2}$: When was Bohemian Rhapsody recorded?

Another – underexplored and most challenging – situation is when questions
involve implicit temporal constraints. These can involve the need to compare
different time points or intervals, even when the user input does not
explicitly state it. Examples are:

$q_{3}$: Queen’s record company when recording Bohemian Rhapsody?
$q_{4}$: Queen’s lead singer after Freddie Mercury?

For $q_{4}$, the system has to find out when Mercury died or left the band, in
order to compute the correct answer that Brian May (the band’s guitarist) took
over as lead singer.

The research literature on temporal QA is substantial, including (Jia et al.,
2018b, 2021; Jiao et al., 2022; Saxena et al., 2021; Xiao et al., 2022; Chen
et al., 2022; Ding et al., 2022; Mavromatis et al., 2022; Chen et al., 2021b).
Most methods address all kinds of temporal questions, but are typically less
geared for implicit questions. Some methods operate over curated knowledge
bases (KBs) (e.g., (Jia et al., 2018b, 2021; Ding et al., 2022)), while others
are designed for processing text corpora such as news collections or Wikipedia
full-text (e.g., (Chen et al., 2021b; Ning et al., 2020)).

State-of-the-art limitations. We observe three major issues:

  * (i)

Many methods use “soft-matching” techniques, based on latent embeddings or
language models. This may lead to invalid answers, where the non-temporal part
of a question is matched, but the temporal constraint is violated. For
example, a question about “Queen’s record company in 1990?” may erroneously
return EMI instead of the correct value Parlophone, because EMI is more
prominent and was Queen’s company on most albums. Even when the output is
correct, this could be by the prominence of the answer alone. For example,
“Who was Queen’s lead singer in 1975?” could return the most popular Freddie
Mercury without checking the time. When we vary the question into “…in 2000?”,
many systems would still yield Freddie Mercury, although he was dead then.
This indicates that the system has incomplete inference and is unable to
explain its answer derivation. We call this phenomenon unfaithful QA.

  * ii)

A weak spot of temporal QA systems is the handling of implicit questions.
These are infrequent in established benchmarks. Some methods (Jia et al.,
2018b; Ding et al., 2022; Neelam et al., 2022) aim to transform the implicit
conditions into explicit temporal constraints, based on classifying phrases
starting with “during”, “before” etc. However, they heavily rely on hand-
crafted rules which are rather limited in scope and cannot robustly handle
unforeseen utterances.

  * (iii)

Prior methods run on a single information source: either a KB or a text
corpus. This limits QA coverage: KBs are incomplete and lack refined detail
about events, whereas text collections are harder to extract answers from and
often fail on complex questions (Ding et al., 2022; Christmann et al., 2022a).
QA over heterogeneous sources, including also web tables, has been addressed
by (Oğuz et al., 2022; Christmann et al., 2023), but these methods do not
support temporal conditions.

Approach. To overcome these limitations, we propose Faith (FAIthful Temporal
question answering over Heterogeneous sources), a temporal QA system that
operates over heterogeneous sources, seamlessly combining a KB, a text corpus
and web tables. Inspired by the architecture of (Christmann et al., 2023),
Faith consists of three main stages:

  * (i)

Temporal Question Understanding for representing the question intent into a
structured frame, with specific consideration of the temporal aspects;

  * (ii)

Faithful Evidence Retrieval for identifying relevant pieces of evidence from
KB, text and tables, with time-aware filtering to match the temporal
conditions;

  * (iii)

Explainable Heterogeneous Answering to compute entity-level answers and
supporting evidence for explanation.

A key novelty in the question understanding is that implicit constraints are
resolved into explicit temporal values by generating intermediate questions
and recursively calling Faith itself. For example, the implicit condition
“when recording Bohemian Rhapsody” in $q_{3}$ is transformed into “when Queen
recorded Bohemian Rhapsody?”, and the recursive invocation of Faith returns
the explicit condition August 1975 - September 1975. This derived explicit
condition is then used in a similar vein as the explicit condition 1975 in
$q_{1}$, making it easier to answer the information need. Note that this is
not just question rewriting, but is driven by the full-fledged QA system
itself over the full suite of heterogeneous sources.

A second key novelty is that, in contrast to most prior works including large
language models, Faith provides tangible provenance for the answer derivation.
By providing users with explanatory evidence for answers, Faith is a truly
faithful temporal QA system.

Existing benchmarks for temporal QA focus on a single information source at
hand (either a KB or a text corpus), and include only few questions with
implicit constraints (so the weak performance on these hardly affects the
overall results). Therefore, we devise a new method for automatically creating
temporal questions with implicit constraints, with systematic controllability
of different aspects, including the relative importance of different source
types (text, infoboxes, KB), coverage of topical domains (sports, politics
etc.), fractions of prominent vs. long-tail entities, question complexity, and
more. This way, we construct a new dataset named Tiq with $10{,}000$ questions
and answers accompanied by supporting evidence.

Our code and data is available at https://faith.mpi-inf.mpg.de.

Contributions. The salient contributions of this work are:

  * $\bullet$

the first temporal QA system that taps into heterogeneous sources, and gives
faithful answers with explanatory evidence;

  * $\bullet$

a mechanism that transforms implicit temporal constraints into explicit
conditions, by recursively calling the QA system itself;

  * $\bullet$

a principled method for automatic construction of diverse and difficult
temporal questions, releasing the Tiq benchmark.

##  2\. Concepts and Notation

This section introduces salient concepts and notation for this work.

Temporal value. A temporal value indicates a point in time or time interval.
It can be a specific date (e.g., 24 November 1991), a year (e.g., 1975), or a
time period (e.g., August 1975 - September 1975).

Temporal constraint. A temporal constraint specifies a condition about a time
point or interval that has to be satisfied by the answer and its evidence.
Temporal constraints consist of a temporal value, and a temporal signal (like
before, after, overlap). An example of a (verbalized) temporal constraint is
“in 1970”.

Explicit question. An explicit question mentions a specific temporal
constraint explicitly, as in “Record company of Queen in 1975?”.

Implicit question. An implicit question also specifies a temporal constraint,
but keeps this constraint implicit without mentioning the actual temporal
value: “Queen’s record company when recording Bohemian Rhapsody?”.

Answer. An answer to a question is either an entity (e.g., Brian May) or a
literal such as a date (e.g., 24 August 1975), year (e.g., 1975) or number
(e.g., 3).

Evidence. An evidence is given with an answer as explanatory support. The
evidence consists of information snippets that are retrieved from a KB, a text
corpus, a table, or a Wikipedia infobox. Following (Christmann et al., 2022b),
we consider snippets on a sentence-level: text is split into sentences, and
KB-facts, table rows and infobox entries are verbalized by concatenating the
individual pieces.

Faithfulness. A system answers a question faithfully if its evidence, provided
with the answer, contains: (i) the answer, (ii) all entities that appear in
the question (with any surface name), (iii) all predicates that appear in the
question (at least in paraphrased or implicit form), (iv) a temporal
expression that satisfies the temporal constraint of the question. The first
three aspects are valid in the context of any QA system; the fourth is
specific to temporal QA.

##  3\. FAITH Method

Fig. 1 provides an overview of the system architecture, illustrated with the
processing of the running examples $q_{3}$ and $q_{1}$. The following
subsections present the three main components (understanding, retrieval, and
answering), and will refer to these examples.

###  3.1. Temporal Question Understanding

The goal of this first stage is to capture the temporal information need in a
frame-like structure. Notably, this stage identifies and categorizes temporal
constraints in the user input, which is later used for pruning temporally-
inconsistent answer candidates.

TSF. Inspired by (Christmann et al., 2022b) and (Ho et al., 2019) (both
addressing other, non-temporal, kinds of QA), we propose to learn a Time-aware
Structured Frame (TSF) for an incoming temporal question. The TSF includes
both general-QA-relevant slots:

  * $\bullet$

question entity,

  * $\bullet$

question relation,

  * $\bullet$

expected answer type,

and temporal-QA-relevant slots:

  * $\bullet$

temporal signal, indicating the kind of temporal relation,

  * $\bullet$

temporal category, indicating the type of temporal constraint,

  * $\bullet$

temporal value, the time point or interval of interest (if present).

The question entity and relation are taken from the surface form of the
question (i.e. not linked to KB) to allow for uniform treatment of
heterogeneous sources. The expected answer type is learned from the training
data, in which the KB-type of the gold answer is used. The temporal signal can
be overlap (e.g., from cues like “in”, “during”), before (e.g., from cues like
“before”, “prior to”), or after (e.g., from cues like “after”, “follows”). We
categorize the constraint into implicit (e.g., $q_{3}$ and $q_{4}$) and non-
implicit (e.g., $q_{1}$ and $q_{2}$). The temporal value can be a year, date
or time period. Both the temporal signal and value are derived by identifying
and normalizing key phrases in the input question. For example, the TSF for
$q_{1}$ is:

$\langle$ question entity: “Queen”,

question relation: “record company of”,
expected answer type: “record company”,
temporal signal: overlap,
temporal category: non-implicit,
temporal value: 1975 $\rangle$

Note that in case the question does not specify temporal constraints (e.g.,
$q_{2}$), the respective fields are simply kept empty.

Resolving implicit questions. For the challenging case of implicit questions,
such as $q_{3}$ or $q_{4}$, the temporal value cannot be extracted from the
question directly. To resolve this problem, we devise a novel mechanism, the
implicit question resolver, based on recursively invoking the temporal QA
system itself. To this end, the implicit temporal constraint in the question
is identified and transformed into an intermediate question. For instance, the
intermediate question for $q_{4}$ would be “when Freddie Mercury lead singer
of Queen?”. For $q_{3}$, the temporal value should be a time interval (August
1975 - September 1975). Thus, two intermediate questions are required: (i)
$q_{31}$: “When Queen recorded Bohemian Rhapsody start date?”, and (ii)
$q_{32}$: “When Queen recorded Bohemian Rhapsody end date?”. Although these
formulations are ungrammatical, the QA system can process them properly, being
robust to such inputs.

The intermediate questions are fed into Faith as a recursive call, to obtain
the explicit temporal value for filling the TSF of the original question. The
TSF for $q_{3}$ thus becomes:

$\langle$ question entity: “Queen”,

question relation: “recorded company”,
expected answer type: “record company”,
temporal signal: overlap,
temporal category: implicit,
temporal value: August 1975 - September 1975 $\rangle$

Note the similarity to the TSF of the explicit temporal question $q_{1}$.

Generating intermediate questions. The intermediate questions are generated by
a fine-tuned sequence-to-sequence (Seq2seq) model, specifically BART (Lewis et
al., 2020). A major obstacle, though, is that no prior dataset has suitable
annotations, and collecting such data at scale is prohibitive. Therefore, we
generated training data using InstructGPT (Ouyang et al., 2022), leveraging
its in-context learning (Brown et al., 2020) capabilities. We randomly select
$8$ implicit questions from our train set and label them manually. For each
question, we give the intermediate question and the expected answer type as
output. The exact prompts used are shown in Table 9 in the Appendix.

The expected answer type of an intermediate question can be date or time
interval. When the expected answer type is a time interval (e.g., for
$q_{3}$), two intermediate questions are created, appending “start date” and
“end date” to the generated intermediate question, respectively (see $q_{31}$
and $q_{32}$ as example).

We use this technique to annotate all implicit questions in the train and dev
sets, obtaining training data for fine-tuning the BART model. Note that GPT is
used only for the generation of training data. It is not used at run-time to
avoid its (computational, monetary, and environmental) costs and dependency on
black-box models.

Constructing the TSF. We also use a fine-tuned Seq2seq model, again BART, for
generating the values for the question entity, question relation, expected
answer type, temporal signal, and temporal category slots of the TSF
representation. The training data this TSF construction model is obtained via
(i) distant supervision (for question entity and question relation)
(Christmann et al., 2022b), (ii) KB-type look-ups (for expected answer type),
and (iii) annotations in the benchmark (for temporal signal and temporal
category). Further detail in Sec. A.2.

The temporal values are obtained via the recursive mechanism discussed above
for implicit questions, and via SUTime (Chang and Manning, 2012) and regular
expression matching for explicit questions. Phrases like “today” or “current”
are considered as well and properly normalized. We use the creation time of
the question (Campos et al., 2014), as provided in the benchmarks, as
reference time.

The TSF generated in this understanding stage is used for representing the
temporal information need in the subsequent retrieval and answering stages,
capturing its key temporal characteristics.

###  3.2. Faithful Evidence Retrieval

In this stage, we first retrieve evidence from heterogeneous sources, and then
prune out information inconsistent with the temporal constraint expressed by
the temporal signal and value in the TSF.

Heterogeneous retrieval. This step largely follows the general-purpose QA
method of (Christmann et al., 2022b), and makes use of entity linking. Entity
mentions in the input are identified and linked via Clocq (Christmann et al.,
2022a). The input here is the concatenation of the question entity, the
question relation, and the expected answer type of the TSF. For the resulting
linked entities, we retrieve the Wikipedia pages for extracting text, tables,
and infoboxes. Further, KB-facts with the linked entities are obtained from
Wikidata.

All retrieved pieces of evidence are verbalized (Oğuz et al., 2022) into
textual sentences, for uniform treatment. The KB-facts are verbalized by
concatenating their individual parts; the text evidence is split into
sentences; table rows are transformed by concatenating the individual
$\langle$column headers, cell value$\rangle$ pairs; infoboxes are handled by
linearizing all attribute-value pairs.

Temporal pruning. Explicit temporal expressions in the retrieved pieces of
evidence are identified and normalized similarly as in the understanding
stage. Evidence that does not match the temporal criteria is pruned out. We
address two kinds of situations:

  * (i)

the question aims for a temporal value as answer and does not have any
temporal constraints (e.g., “When …?”);

  * (ii)

the question has a temporal constraint which needs to be matched by the
evidence.

In the first case, all evidence that does not contain any temporal values, and
is thus unable to provide the answer, is dropped. In the second case, we
remove pieces of evidence that do not match the temporal constraint, to ensure
that answers are faithful to the temporal intent of the question.

The retrieval output is a smaller set of evidence pieces, faithfully
reflecting the temporal constraints of the question. The final answer and its
explanatory evidence are computed from this pool.

###  3.3. Explainable Heterogeneous Answering

In the final stage, the answer is derived from this set of evidence pieces
that is already known to satisfy the temporal conditions.

Since this part is not the main focus of this work, we employ a state-of-the-
art answering model for general-purpose QA. We use the answering stage of
Explaignn (Christmann et al., 2023) that is based on graph neural networks
(GNNs), and computes a subset of supporting evidence for the predicted answer.
Thus, we ensure that the answer can be traced back through the entire system
including the answering stage, for end user explainability. The input query to
the GNNs is the concatenation of the question entity, question relation, and
expected answer type.

Figure 2. Steps to create implicit questions with our proposed methodology,
highlighting the key configurable parts.

##  4\. Tiq benchmark

Most existing benchmarks for temporal QA, like TempQuestions (Jia et al.,
2018a), TimeQuestions (Jia et al., 2021) or TempQA-Wd (Neelam et al., 2022),
have only few implicit questions ($209$, $1{,}476$, and $154$, respectively),
falling short of evaluating one of the key challenges in temporal QA.
CronQuestions (Saxena et al., 2021) and TEMPREASON (Tan et al., 2023) have a
larger fraction of implicit questions, but these are based on a small set of
hand-crafted rules. Thus, the questions lack syntactic diversity. Further,
questions in these benchmarks are always answerable using a single information
source (either KB or text corpus).

Therefore, we construct a new benchmark with a primary focus on challenging
and diverse implicit questions. The obvious idea of using crowdsourcing is
expensive and error-prone. Also, crowdworkers increasingly use LLMs as a
shortcut (Veselovsky et al., 2023). Thus, we pursue an automated process
instead. To ensure that questions are not specific to a single input source,
our process considers multiple sources: Wikipedia text and infoboxes, and the
Wikidata KB.

###  4.1. Construction Methodology

Overview. An implicit question has two parts: the main question that specifies
the information need disregarding time (e.g., “Queen’s lead singer” for
$q_{4}$), and the implicit part that provides the temporal constraint (e.g.,
“after Freddie Mercury” for $q_{4}$). The key idea is to build each of the two
parts from independent pieces of evidence, denoted as information snippets.
The two snippets can come from very different sources, but need to be
thematically related. This construction process operates as follows:

  * (i)

sample a set of topic entities to start with;

  * (ii)

retrieve temporal information snippets for each such topic entity from
Wikipedia text, Wikipedia infoboxes, and Wikidata;

  * (iii)

concatenate information snippets using a suitable temporal signal and
construct an interrogative sentence, a pseudo-question;

  * (iv)

rephrase the pseudo-question into a natural question using a generative model.

An overview of this process is provided in Fig. 2, including an example case
of constructing an implicit question. Naturally, implicit constraints are
global events (e.g., the COVID pandemic), or major events for a specific
entity (e.g., a prestigious award).

Sampling topic entities. To obtain significant events, we start with Gregorian
calendar year pages in Wikipedia (e.g., https://en.wikipedia.org/wiki/2023)
that list notable events. From the pages for the years 1801 \- 2025, we
collect information snippets about such significant events. The entities in
these snippets constitute the set of topic entities (href anchors are used for
entity linking (Ferragina and Scaiella, 2010)).

In our example in Fig. 2 this set includes Alicia Keys.

Retrieving the grounding information snippets. We collect snippets about
notable events in these year pages, and augment them with salient information
about the topic entity from (i) the first five sentences ($\simeq$ first
passage) of the entity’s Wikipedia page, (ii) the respective Wikipedia
infobox, and (iii) the Wikidata facts.

As candidates for the main question part, we consider all information snippets
that are retrieved for a topic entity from Wikipedia text, infoboxes and
Wikidata, irrespective of their salience. To avoid questions that are
trivially answerable without considering the temporal condition, multiple
candidate snippets are retrieved for the main question, with different
temporal scopes (e.g., a band’s singers from different epochs). This is
implemented by measuring semantic similarity among candidates using a
SentenceTransformer111https://huggingface.co/sentence-transformers/paraphrase-
MiniLM-L6-v2 (Reimers and Gurevych, 2019).

Creating a pseudo-question. Among the retrieved snippets for an entity, we
identify pairs of candidate snippets that can be connected by a temporal
conjunction/preposition (“during”, “after” and “before”). For such a pair, the
temporal scopes have to be consistent with the temporal conjunction. A valid
pair for the conjunction “during” would be: “Alicia Keys followed up her debut
with The Diary of Alicia Keys, which was released in December 2003.” (main
question part from Wikipedia text) and “Norah Jones, award received, Grammy
Award for Best New Artist, follows, Alicia Keys, point in time, 2003.”
(implicit part from KB). A pseudo-question is created by concatenating the
main part with the conjunction and the implicit part. The answer is an entity
(not the topic entity) from the main part (The Diary of Alicia Keys). The
answer is substituted by the prefix “what” followed by the most frequent KB-
type of the answer (album in this case).

The pseudo-question for the example is: “What album Alicia Keys followed up
her debut with which was released, during, Norah Jones award received Grammy
Award for Best New Artist follows Alicia Keys?”, which is an ungrammatical and
unnatural formulation.

Rephrasing to a natural question. Therefore, in the last step, we rephrase the
pseudo-question to a natural formulation. We use InstructGPT (Ouyang et al.,
2022) with $8$ demonstration examples (pseudo-questions and their natural re-
phrasings), to generate the final question222The prompt is given in Table 8 in
the Appendix..

The pseudo-question of the example is rephrased into the following implicit
question: “What album did Alicia Keys release when Norah Jones won the Grammy
Award for Best New Artist?”

###  4.2. Benchmark Characteristics

Topic entities. For creating Tiq (Temporal Implicit Questions) we started with
the years 1801-2025 and obtained an initial set of $229{,}318$ entities. From
this set, we uniformly sampled $10{,}000$ topic entities based on their
frequency, to capture a similar amount of long-tail and more prominent
entities (see Table 1 for details). These fractions can be configured as
required. Since some entity types were over-represented in the calendar year
pages (e.g., politicians or countries), we also ensured that individual entity
types are not taking up more than $10\%$ of the topic entities. In general,
the topic entity set allows to control the domain coverage within the
generated implicit questions, by choosing entities of the desired types.

We did not specifically configure the proportions to which the individual
information sources are used within the questions, since we observed a
naturally diverse distribution. Fig. 3 shows the distribution among source
combinations for initiating the main and implicit part. The questions are
finally split into train ($6{,}000$), dev ($2{,}000$), and test sets
($2{,}000$). Table 1 shows the basic statistics, and Table 2 shows
representative questions of the Tiq benchmark.

Meta-data. Tiq provides implicit questions and gold answers, as strings as
well as canonicalized to Wikipedia and Wikidata. The meta-data includes the
information snippets grounding the question, the sources these were obtained
from, the explicit temporal value expressed by the implicit constraint, the
topic entity, the question entities detected in the snippets, and the temporal
signal.

The Tiq dataset is available at https://faith.mpi-inf.mpg.de.

Figure 3. Distribution of questions over input source combinations (source for
main part ; source for implicit part). Table 1. Basic statistics for Tiq.

Sources |  Wikipedia text, infoboxes, and Wikidata
---|---
Questions |  10,000 (train: $6{,}000$, dev: $2{,}000$, test: $2{,}000$)
Avg. question length |  $17.96$ words
Avg. no. of question entities |  $2.45$
Unique topic entities covered |  $10{,}000$
Long-tail topic entities covered |  $2{,}542$ (with $<20$ KB-facts)
Prominent topic entities covered |  $2{,}613$ (with $>500$ KB-facts)

Table 2. Representative questions from the Tiq benchmark. The sources below
indicate the source that was used for populating the [main question part;
implicit question part] of the implicit question.

1\. Who bought the Gainesville Sun after it was owned by Cowles Media Company? |  2\. During Colin Harvey’s senior football career, which club was he a member of while he played for the England national football team? |  3\. Which album released by Chris Brown topped the Billboard 200 when he was performing in Sydney? |  4\. What television series was Hulk Hogan starring in when he signed with World Championship Wrestling? |  5\. Who was Bristol Palin’s partner before she participated in the fall season of Dancing with the Stars, and reached the finals, finishing in third place?
---|---|---|---|---
The New York Times Company |  Everton F.C. |  Fortune |  Thunder in Paradise |  Levi Johnston
[Text; KB] |  [Infobox; KB] |  [Text; Infobox] |  [Text; Text] |  [Infobox; Text]
6\. During the onset of the COVID-19 pandemic, who was the New York City head of government? |  7\. Who was the chief executive officer at Robert Bosch GmbH before revenue reached €78.74 billion? |  8\. After graduating from the Rostov-on-Don College of Economics and Finance, which political party did Gyula Horn join? |  9\. Which national football team did Carlos Alberto Torres manage before joining Flamengo? |  10\. What university did Robert Lee Moore work for after Northwestern University?
Bill de Blasio |  Volkmar Denner |  Hungarian Working People’s Party |  Oman national football team |  University of Pennsylvania
[KB; Text] |  [KB; Infobox] |  [Infobox; Text] |  [Infobox; Infobox] |  [KB; KB]

Table 3.  Main results comparing the performance of Faith against baselines on
the test sets of Tiq and TimeQuestions.

Benchmark $\rightarrow$ | Tiq | TimeQuestions
---|---|---
Method $\downarrow$ | P@1 | MRR | Hit@5 | P@1 | MRR | Hit@5
InstructGpt (Ouyang et al., 2022) | $0.237$ | n/a | n/a | $0.224$ | n/a | n/a
Gpt-4 (OpenAI, 2023) | 0.236 | n/a | n/a | $0.306$ | n/a | n/a
Uniqorn (Pramanik et al., 2021) | $0.236$ | $0.255$ | $0.277$ | $0.331$ | $0.409$ | $0.538$
Unik-Qa (Oğuz et al., 2022) | $0.425$ | $0.480$ | $0.540$ | $0.424$ | $0.453$ | $0.486$
Explaignn (Christmann et al., 2023) | $0.446$ | $0.584$ | $0.765$ | $0.525$ | $0.587$ | $0.673$
TempoQR (Mavromatis et al., 2022) | $0.011$ | $0.018$ | $0.022$ | $0.438$ | $0.465$ | $0.488$
CronKGQA (Saxena et al., 2021) | $0.006$ | $0.011$ | $0.014$ | $0.395$ | $0.423$ | $0.450$
Exaqt (Jia et al., 2021) | $0.232$ | $0.378$ | $0.587$ | $0.565$ | $0.599$ | $0.664$
Faith (Proposed) | $0.491$ | $0.603$ | $0.752$ | $0.535$ | $0.582$ | $0.635$
Un-Faith | ${0.459}$ | ${0.604}$ | ${0.799}$ | ${0.571}$ | ${0.640}$ | ${0.724}$

##  5\. Experiments

###  5.1. Experimental Setup

Benchmarks. We conduct experiments on our new Tiq benchmark and TimeQuestions
(Jia et al., 2021), which has been actively used in recent work on temporal
QA. For ordinal questions (e.g., “what was the first album by Queen?”) in
TimeQuestions, we apply the same method as outlined in Sec. 3, without
applying any temporal filtering.

Metrics. We use the standard QA metrics precision at 1 (P@1), mean reciprocal
rank (MRR), and hit at 5 (Hit@5) (Roy and Anand, 2022).

Baselines. We compare Faith with a suite of baselines, covering a diverse
range of competitors:

  * $\bullet$

Generative LLMs. We compare with InstructGpt (Ouyang et al., 2022) (“text-
davinci-003”) and Gpt-4 (OpenAI, 2023) (“gpt-4”) using the OpenAI
API333https://platform.openai.com. We tried different prompts, and found the
following to perform best: “Please answer the following question by providing
the crisp answer entity, date, year, or number.”. For computing P@1, we check
whether the generated answer string matches with the label or any alias of the
gold answer. If this is the case, P@1 is $1$, else $0$. Other (ranking)
metrics are not applicable for LLMs.

  * $\bullet$

Heterogeneous QA methods. Further, we compare against a range of recent
general-purpose methods for heterogeneous QA: Uniqorn (Pramanik et al., 2021),
UniK-Qa (Oğuz et al., 2022), and the vanilla Explaignn (Christmann et al.,
2023).

  * $\bullet$

Temporal QA methods. We also compare with state-of-the-art methods for
temporal QA: TempoQR (TempoQR-Hard) (Mavromatis et al., 2022), CronKGQA
(Saxena et al., 2021), and Exaqt (Jia et al., 2021).

Finally, we show results for a variant of our approach, which does not prune
out evidence temporally-inconsistent with the temporal constraint, i.e. drops
the temporal pruning component. We term this variant Un-Faith.

Configuration. Wikidata (Vrandečić and Krötzsch, 2014) is used as the KB for
Faith and all baselines. We use Wikipedia text, tables and infoboxes as
additional information sources for methods operating over heterogeneous
sources. The BART models are initialized via Hugging
Face444https://huggingface.co. We use AdamW as optimizer with a learning rate
of $5\times$$10^{-5}$, batch size of $10$, weight decay of $0.01$, $5$ epochs,
and $500$ warm-up steps. Explaignn is run using the public
code555https://github.com/PhilippChr/EXPLAIGNN, retaining the original
settings and parameters for optimization.

For Faith, we choose the candidate at rank $1$ as the answer for intermediate
questions in the implicit question resolver. In case too many evidences are
obtained as input to the answering stage, we consider the top-$100$ evidences
as computed by a BERT-based re-ranker (Nogueira and Cho, 2019). Further detail
is given in the Appendix A.4. We follow an epoch-wise evaluation strategy for
each module and baseline, and take the version with the best performance on
the respective dev set. All training processes and experiments are run on a
single GPU (NVIDIA Quadro RTX 8000, 48 GB GDDR6).

###  5.2. Main Results

Answering performance of Faith and baselines on TimeQuestions and on Tiq are
in Table 3.

Faith outperforms baselines on Tiq. The main insight from Table 3 is that
Faith surpasses all baselines on the Tiq dataset for P@1, which is the most
relevant metric, demonstrating the benefits of our proposed method for
answering implicit temporal questions. Temporal QA methods operating over KBs
lack the required coverage on the Tiq dataset, and perform worse than general-
purpose QA methods operating over heterogeneous sources. Explaignn comes close
to the performance of Faith, and even slightly improves on the Hit@5 metrics.
Note, however, that Explaignn and all other baselines do not verify that
temporal constraints are met during answering. Thus, the most prominent among
answer candidates may simply be picked up, even if no temporal information is
provided or matching. Such possibly “accidental” and unfaithful answers are,
by design, not considered by Faith.

Trade-off between faithfulness and answering performance. Results for Un-Faith
illustrate the effect of this phenomenon on our approach: especially the MRR
and Hit@5 results are substantially improved. Consequently, Un-Faith
outperforms all competitors on TimeQuestions. However, its answers are not
always faithfully grounded in evidence sources. These results emphasize the
trade-off between faithfulness and answering performance.

Faith shows robust performance on TimeQuestions. Faith also shows strong
performance on the TimeQuestions benchmark, on which it outperforms all
baselines on P@1, except for Exaqt. This indicates the robustness of Faith
across different datasets. Existing methods for temporal QA show major
performance gaps between the two benchmarks: the P@1 of the strongest method
on TimeQuestions, Exaqt, substantially drops from $0.565$ at P@1 to $0.232$ on
the Tiq benchmark. Note that all methods are trained on the specific
benchmark, if applicable.

LLMs fall short on temporal questions. Another key insight from Table 3 is
that current LLMs are clearly not capable of answering temporal questions.
InstructGpt and Gpt-4 can merely answer $\simeq 23$-$30$% of the questions
correctly, and are constantly underperforming Faith and baselines operating
over heterogeneous sources. One explanation is that reasoning with continuous
variables, such as time, is a well-known weakness of LLMs (Dhingra et al.,
2022).

Table 4.  Comparing the faithfulness of Faith and Un-Faith for correct
answers, and how often temporal constraints are violated or ignored.

Benchmark $\rightarrow$ | Tiq | TimeQuestions
---|---|---
|  | Temporally |  | Temporally
Method $\downarrow$ | Faithful | Unfaithful | Faithful | Unfaithful
Faith | $0.95$ | $0.00$ | $0.94$ | $0.01$
Un-Faith | $0.90$ | $0.08$ | $0.87$ | $0.13$

###  5.3. Faithfulness Evaluation

Our main results in Table 3 indicate that ignoring the temporal condition of
the question can yield improvements on automatic metrics (compare performance
of Faith vs. Un-Faith on TimeQuestions). However, we observe that this can
lead to critical failure cases of QA systems and sometimes boils down to lucky
guesses of the answer based on priors (e.g., prominence of an answer
candidate).

Faith refrains to answer in absence of consistent evidence. If there is no
temporal information associated with the evidence of candidate answers, or the
temporal information does not satisfy the temporal constraint, Faith will
refuse answering the question. For example, for the question “Who did Lady
Jane Grey marry on the 25th of May 1533?”, there is no answer satisfying the
temporal constraint because Lady Jane Grey did not marry anyone on the 25th of
May 1533, since she was only born four years later in 1937. However, all of
the baselines provide an answer to the question, without indicating that the
temporal constraint is violated.

Since questions without a temporally-consistent answer are not available at
large scale, we randomly sample $500$ explicit questions from TimeQuestions,
and replace the temporal value with a random date (e.g., “12 October 6267”).
None of the resulting questions has a temporally-consistent answer. As
expected, the competitors still provide answers666Except for the LLMs for
which we are not able to investigate the behavior at scale, since they would
often generate longer texts.. In contrast, Faith successfully refrained from
answering for $467$ of the $500$ questions ($93.4\%$). Upon investigating the
failure cases, we noticed that the date recognition identifies four-digit
numbers as years matching with the constraint (e.g., in the infobox entry
“Veysonnaz, SFOS number, 6267”).

Fallback to Un-Faith. Completely refraining from answering could also be sub-
optimal: the user might have made a typo (e.g., “May 1533” instead of “May
1553”). We investigated to fall back to Un-Faith in such scenarios, which
could be indicated to end users with an appropriate warning. Performance on
both datasets was slightly improved: the P@1 metric increased from $0.491$ to
$0.492$ on Tiq and from $0.535$ to $0.539$ on TimeQuestions. We further
investigated to fall back to Un-Faith in case Faith answered incorrectly. The
P@1 metric was improved substantially on both datasets: from $0.491$ to
$0.622$ on Tiq and from $0.535$ to $0.653$ on TimeQuestions.

Manual analysis. Finally, we investigated the faithfulness of correct answers
provided by Faith and Un-Faith, to understand how often the question is
answered correctly even though the evidence is not faithful to the question.
To analyze this qualitatively, we randomly selected $100$ questions (from each
benchmark) for which both Faith and Un-Faith answered correctly, and then
manually verified the faithfulness, based on the definition in Sec. 2. Results
are in Table 4. Faith provides faithful answers and evidence in $95\%$/$94\%$
of the time. By design, answers are faithful to the temporal constraints in
the question (except for one question which specifies two different temporal
constraints). In comparison, Un-Faith violates or ignores the temporal
condition in $8\%$/$13\%$ of the cases.

For example, to answer the question “What movies starring Taylor Lautner in
2011?” (answer: Abduction), the evidence for Faith is “Taylor Lautner, Year is
2011, Title is Abduction, Role is Nathan Harper” (from table), while the
evidence for Un-Faith is “Abduction, cast member, Taylor Lautner” (from KB).
Even though both pieces of evidence mention the correct answer Abduction, Un-
Faith fails to satisfy the temporal constraint (“in 2011”) with its evidence.

Table 5. Ablation study using different source combinations as input for Faith
on dev sets. Note that Faith is trained using all sources as input for all
cases.

Benchmark $\rightarrow$ | Tiq | TimeQuestions
---|---|---
Method $\downarrow$ | P@1 | MRR | Hit@5 | P@1 | MRR | Hit@5
KB | $0.293$ | $0.368$ | $0.468$ | $0.425$ | $0.464$ | $0.513$
Text | $0.194$ | $0.262$ | $0.351$ | $0.224$ | $0.269$ | $0.320$
Infoboxes | $0.169$ | $0.223$ | $0.296$ | $0.093$ | $0.117$ | $0.149$
Tables | $0.032$ | $0.057$ | $0.083$ | $0.078$ | $0.094$ | $0.114$
KB+Text | $0.429$ | $0.527$ | $0.649$ | $0.520$ | $0.567$ | $0.626$
KB+Tables | $0.299$ | $0.379$ | $0.480$ | $0.435$ | $0.479$ | $0.536$
KB+Infoboxes | $0.384$ | $0.488$ | $0.634$ | $0.443$ | $0.487$ | $0.543$
Text+Tables | $0.196$ | $0.267$ | $0.362$ | $0.252$ | $0.298$ | $0.350$
Text+Infoboxes | $0.283$ | $0.372$ | $0.490$ | $0.251$ | $0.299$ | $0.355$
Tables+Infoboxes | $0.179$ | $0.244$ | $0.331$ | $0.143$ | $0.174$ | $0.208$
All sources | $0.497$ | $0.610$ | $0.756$ | $0.538$ | $0.583$ | $0.639$

Table 6. Ablation studies of Faith on dev sets. Benchmark $\rightarrow$ | Tiq | TimeQuestions
---|---|---
Method $\downarrow$ | P@1 | P@1
Faith | $0.497$ | $0.538$
w/o temporal pruning | $0.443$ | $0.573$
w/o implicit question resolver | $0.467$ | $0.559$
w/o GNN-based answering | $0.316$ | $0.399$
Table 7. Anecdotal examples that Faith answered correctly in Tiq and
TimeQuestions. Evidence shows the supporting information snippets along with
their source provided in brackets. The part mentioning the predicted answer is
in bold, and the detected temporal values are underlined. For the first
example from the Tiq benchmark, we show the answering process of the
intermediate question, which can be used by end users to verify the entire
answer derivation of the system.

Benchmark |  Tiq
---|---
Question |  After managing FC Nantes, which football club did Antoine Raab take on next?
Answer |  Stade Lavallois
TSF |  $\langle$ question entity: “Antoine Raab, FC Nantes football”, question relation: “After managing which club did take on next”, expected answer type: “association football club”, temp. signal: after, temp. category: implicit, temp. value: [1946, 1949] $\rangle$
Evidence |  Antoine Raab, Managerial career, 1949–1950, Stade Lavallois. (from Infobox)
Intermediate questions |  (i) When Antoine Raab managed FC Nantes start date?
|  (ii) When Antoine Raab managed FC Nantes end date?
Answers (to int. questions) |  (i) 1946, (ii) 1949
TSFs (for int. questions) |  (i) $\langle$ question entity: “FC Nantes, start, Antoine Raab”, question relation: “When managed date”, expected answer type: “year”, temp. signal: _; temp. category: non-implicit; temp. value: _ $\rangle$
|  (ii) $\langle$ question entity: “FC Nantes, end, Antoine Raab”, question
relation: “When managed date”, expected answer type: “year”, temp. signal: _;
temp. category: non-implicit; temp. value: _ $\rangle$
Evidence (for int. questions) |  (i, ii) Antoine Raab, Managerial career, 1946–1949, FC Nantes. (from Infobox)
|  (ii) Antoine Raab, After the liberation of Nantes in 1944 Raab joined FC
Nantes and played for the club until 1949. (from Text)
Benchmark |  TimeQuestions
Question |  What award did Thomas Keneally receive in the year 1982?
Answer |  Booker Prize
TSF |  $\langle$ question entity: “Thomas Keneally”, question relation: “What award did receive in the year 1982”, expected answer type: “science award”, temp. signal: overlap, temp. category: non-implicit, temp. value: 1982 $\rangle$
Evidence |  Man Booker Prize, winner, Thomas Keneally, point in time, 1982, for work, Schindler’s Ark. (from KB)
|  Thomas Keneally, Awards is Booker Prize, is Schindler’s Ark, winner 1982.
(from table)
|  Thomas Keneally, He is best known for his non-fiction novel Schindler’s
Ark, the story of Oskar Schindler’s rescue of Jews during the Holocaust, which
won the Booker Prize in 1982. (from Text)

###  5.4. In-depth Analysis

Integrating heterogeneous sources is decisive. We further investigated the
effect of integrating heterogeneous sources into Faith, and tested giving each
individual source independently, and their pairwise combinations as input, in
comparison to the default setting with ”All sources”. Results are in Table 5.
Each information source contributes to the performance of Faith, and
integrating more information sources consistently enhances all metrics.

Ablation studies. We tested variations of our pipeline on the dev sets. Table
6 shows results for Un-Faith (w/o temporal pruning), results without the
implicit time resolver, and results with a Seq2seq model for answering (we
used BART) instead of the GNN-based approach. Using a GNN-based answering
approach plays a crucial role, and enhances not only answering performance,
but also explainability. The implicit question resolver is decisive on Tiq,
but slightly decreases performance on TimeQuestions. Un-Faith also shows
strong performance on the dev sets. However, all modules contribute to the
explainability and faithfulness of our approach.

Anecdotal examples. Table 7 shows sample cases for which Faith provided the
correct answer, and illustrates the answer derivation process providing
traceable evidence for end users.

Error analysis. To better understand failure cases, we conducted an error
analysis measuring the answer presence (i.e. whether the gold answer is among
answer candidates) throughout the pipeline. We identified the following error
cases and list their percentage among all failure cases for Tiq and
TimeQuestions, respectively: (i) the answer was not found in the initial
retrieval stage ($3.14$/$29.89$), (ii) the answer is lost during temporal
pruning ($22.00$/$25.81$), (iii) the answer is lost during scoring/graph
shrinking ($8.45$/$10.33$), (iv) the answer is not considered among top-$5$
answers ($15.13$/$12.47$), (v) the answer is among top candidates but not at
rank $1$ ($51.28$/$21.51$).

##  6\. Related Work

General-purpose QA. Question answering has extensive work using single sources
like KBs (e.g., (Yahya et al., 2012; Yih et al., 2015; Berant et al., 2013))
or text (e.g., (Chen et al., 2017; Rajpurkar et al., 2016; Izacard and Grave,
2021)). Some works have shown that integrating different sources can
substantially improve performance (Ferrucci, 2012; Xu et al., 2016a, b;
Savenkov and Agichtein, 2016; Sun et al., 2018, 2019; Chen et al., 2021a).
Unik-Qa (Oğuz et al., 2022) verbalizes snippets from a KB, text, tables and
infoboxes, as input to a Fusion-in-decoder (FiD) model (Izacard and Grave,
2021) for answer generation. Udt-QA (Ma et al., 2022) improved the
verbalization technique. Explaignn (Christmann et al., 2023) constructs graphs
among such verbalized snippets, and applies graph neural networks for
computing answers and explanatory evidence. None of these methods is geared
for temporal questions.

Another direction is to directly apply large language models (LLMs) for QA
(Brown et al., 2020; Raffel et al., 2020; Petroni et al., 2019; Devlin et al.,
2019). However, LLMs cannot present traceable provenance for the generated
outputs, falling short on faithfulness and explainability (Adlakha et al.,
2023; Min et al., 2023; Mallen et al., 2023). Also, LLMs struggle with
reasoning on temporal conditions (Dhingra et al., 2022).

Temporal QA. Prior work that specifically targets temporal QA (Wu et al.,
2020b; Jia et al., 2018b, 2021; Jiao et al., 2022; Saxena et al., 2021; Xiao
et al., 2022; Chen et al., 2022; Ding et al., 2022; Neelam et al., 2022;
Sharma et al., 2023; Mavromatis et al., 2022; Shang et al., 2022; Yao et al.,
2022; Chen et al., 2021b; Long et al., 2022), can largely be divided into work
using a KB (e.g., (Jia et al., 2021; Neelam et al., 2022; Mavromatis et al.,
2022)), and work using text (e.g., (Chen et al., 2021b; Ning et al., 2020)).
Methods operating over KBs, include template-based (Jia et al., 2018b; Ding et
al., 2022; Neelam et al., 2022), KB-embedding-based (Saxena et al., 2021; Xiao
et al., 2022; Chen et al., 2022; Mavromatis et al., 2022), and graph-based
methods (Jia et al., 2021; Sharma et al., 2023; Yao et al., 2022). Methods
using textual inputs typically involve an extractive or generative reader
(Chen et al., 2021b; Ning et al., 2020).

The three methods (Jia et al., 2021; Saxena et al., 2021; Mavromatis et al.,
2022) represent the state-of-the-art on temporal QA. However, temporal
constraints are handled solely in the latent space, without explicitly (or
faithfully) pruning out temporally inconsistent answer candidates. Other
approaches are based on handcrafted rules, and hence bound to fail for unseen
question patterns (e.g., (Jia et al., 2018b)). None of the existing work on
temporal QA has considered incorporating heterogeneous sources.

Temporal KBs. There is substantial work on temporal KBs (Leblay and Chekol,
2018; Garcia-Duran et al., 2018; Cai et al., 2022; Wu et al., 2020a; Xu et
al., 2021; Messner et al., 2022; Park et al., 2022), to assign temporal scopes
to KB facts. Advances on the KB itself benefits QA, but is an orthogonal
direction.

##  7\. Conclusion

This work targets complex temporal QA, and proposes a new approach for
faithfully answering temporal questions, with focus on the challenging case of
implicit temporal constraints. Experiments show that our method Faith
outperforms the best unfaithful competitor on such implicit questions. On
other temporal questions, our method performs almost on par, but adds the
benefit of reliably matching the temporal conditions. Faithfulness is an
important element in enhancing the trustworthiness of QA systems.

Acknowledgements. We thank Rishiraj Saha Roy and Magdalena Kaiser from the Max
Planck Institute for Informatics for useful inputs at various stages of this
work. Zhen Jia was supported by NSFC (Grant No.62276215 and No.62272398).

## References

  * (1)
  * Adlakha et al. (2023) Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2023.  Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering. In _arXiv_.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic Parsing on Freebase from Question-Answer Pairs. In _EMNLP_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language Models are Few-Shot Learners. In _NeurIPS_.
  * Cai et al. (2022) Borui Cai, Yong Xiang, Longxiang Gao, He Zhang, Yunfeng Li, and Jianxin Li. 2022.  Temporal Knowledge Graph Completion: A Survey. In _arXiv_.
  * Campos et al. (2014) Ricardo Campos, Gaël Dias, Alípio M Jorge, and Adam Jatowt. 2014.  Survey of Temporal Information Retrieval and Related Applications.  _ACM Computing Surveys (CSUR)_ (2014).
  * Chang and Manning (2012) Angel X. Chang and Christopher D. Manning. 2012.  SUTime: A Library for Recognizing and Normalizing Time Expressions. In _LREC_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to Answer Open-Domain Questions. In _ACL_.
  * Chen et al. (2021a) Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Wang, and William W Cohen. 2021a.  Open Question Answering over Tables and Text. In _ICLR_.
  * Chen et al. (2021b) Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021b.  A Dataset for Answering Time-Sensitive Questions. In _NeurIPS_.
  * Chen et al. (2022) Ziyang Chen, Xiang Zhao, Jinzhi Liao, Xinyi Li, and Evangelos Kanoulas. 2022.  Temporal Knowledge Graph Question Qnswering via Subgraph Reasoning.  _Knowledge-Based Systems_ (2022).
  * Christmann et al. (2022a) Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. 2022a.  Beyond NED: Fast and Effective Search Space Reduction for Complex Question Answering over Knowledge Bases. In _WSDM_.
  * Christmann et al. (2022b) Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. 2022b.  Conversational Question Answering on Heterogeneous Sources. In _SIGIR_.
  * Christmann et al. (2023) Philipp Christmann, Rishiraj Saha Roy, and Gerhard Weikum. 2023.  Explainable Conversational Question Answering over Heterogeneous Sources via Iterative Graph Neural Networks. In _SIGIR_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_.
  * Dhingra et al. (2022) Bhuwan Dhingra, Jeremy R Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W Cohen. 2022.  Time-Aware Language Models as Temporal Knowledge Bases. In _TACL_.
  * Ding et al. (2022) Wentao Ding, Hao Chen, Huayu Li, and Yuzhong Qu. 2022.  Semantic Framework based Query Generation for Temporal Question Answering over Knowledge Graphs. In _EMNLP_.
  * Ferragina and Scaiella (2010) Paolo Ferragina and Ugo Scaiella. 2010.  TAGME: On-the-fly Annotation of Short Text Fragments (by Wikipedia Entities). In _CIKM_.
  * Ferrucci (2012) David A. Ferrucci. 2012.  Introduction to ”This is Watson”.  _IBM Journal of Research and Development_ (2012).
  * Garcia-Duran et al. (2018) Alberto Garcia-Duran, Sebastijan Dumančić, and Mathias Niepert. 2018.  Learning Sequence Encoders for Temporal Knowledge Graph Completion. In _EMNLP_.
  * Ho et al. (2019) Vinh Thinh Ho, Yusra Ibrahim, Koninika Pal, Klaus Berberich, and Gerhard Weikum. 2019.  Qsearch: Answering Quantity Queries from Text. In _ISWC_.
  * Izacard and Grave (2021) Gautier Izacard and Édouard Grave. 2021.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In _EACL_.
  * Jia et al. (2018a) Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018a.  TempQuestions: A Benchmark for Temporal Question Answering. In _HQA@WWW_.
  * Jia et al. (2018b) Zhen Jia, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strötgen, and Gerhard Weikum. 2018b.  TEQUILA: Temporal Question Answering over Knowledge Bases. In _CIKM_.
  * Jia et al. (2021) Zhen Jia, Soumajit Pramanik, Rishiraj Saha Roy, and Gerhard Weikum. 2021.  Complex Temporal Question Answering on Knowledge Graphs. In _CIKM_.
  * Jiao et al. (2022) Songlin Jiao, Zhenfang Zhu, Wenqing Wu, Zicheng Zuo, Jiangtao Qi, Wenling Wang, Guangyuan Zhang, and Peiyu Liu. 2022.  An Improving Reasoning Network for Complex Question Answering over Temporal Knowledge Graphs. In _Applied Intelligence_. Springer.
  * Leblay and Chekol (2018) Julien Leblay and Melisachew Wudage Chekol. 2018.  Deriving Validity Time in Knowledge Graph. In _TempWeb@WWW_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _ACL_.
  * Long et al. (2022) Shaonan Long, Jinzhi Liao, Shiyu Yang, Xiang Zhao, and Xuemin Lin. 2022.  Complex Question Answering Over Temporal Knowledge Graphs. In _WISE_.
  * Ma et al. (2022) Kaixin Ma, Hao Cheng, Xiaodong Liu, Eric Nyberg, and Jianfeng Gao. 2022.  Open Domain Question Answering with A Unified Knowledge Interface. In _ACL_.
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In _ACL_.
  * Mavromatis et al. (2022) Costas Mavromatis, Prasanna Lakkur Subramanyam, Vassilis N Ioannidis, Adesoji Adeshina, Phillip R Howard, Tetiana Grinberg, Nagib Hakim, and George Karypis. 2022.  TempoQR: Temporal Question Reasoning over Knowledge Graphs. In _AAAI_.
  * Messner et al. (2022) Johannes Messner, Ralph Abboud, and Ismail Ilkan Ceylan. 2022.  Temporal Knowledge Graph Completion Using Box Embeddings. In _AAAI_.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In _EMNLP_.
  * Neelam et al. (2022) Sumit Neelam, Udit Sharma, Hima Karanam, Shajith Ikbal, Pavan Kapanipathi, Ibrahim Abdelaziz, Nandana Mihindukulasooriya, Young-Suk Lee, Santosh Srivastava, Cezar Pendus, et al. 2022\.  SYGMA: System for Generalizable Modular Question Answering over Knowledge Bases. In _Findings of the Association for Computational Linguistics: EMNLP_.
  * Ning et al. (2020) Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020.  TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions. In _EMNLP_.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT. In _arXiv_.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report. In _arXiv_.
  * Oğuz et al. (2022) Barlas Oğuz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022.  UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering. In _NAACL-HLT_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training Language Models to Follow Instructions with Human Feedback. In _NeurIPS_.
  * Park et al. (2022) Namyong Park, Fuchen Liu, Purvanshi Mehta, Dana Cristofor, Christos Faloutsos, and Yuxiao Dong. 2022.  Evokg: Jointly Modeling Event Time and Network Structure for Reasoning over Temporal Knowledge Graphs. In _WSDM_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019.  Language Models as Knowledge Bases?. In _EMNLP-IJCNLP_.
  * Pramanik et al. (2021) Soumajit Pramanik, Jesujoba Alabi, Rishiraj Saha Roy, and Gerhard Weikum. 2021.  UNIQORN: Unified Question Answering over RDF Knowledge Graphs and Natural Language Text. In _arXiv_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. In _JMLR_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _EMNLP_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _EMNLP_.
  * Roy and Anand (2022) Rishiraj Saha Roy and Avishek Anand. 2022.  _Question Answering for the Curated Web: Tasks and Methods in QA over Knowledge Bases and Text Collections_.  Springer.
  * Savenkov and Agichtein (2016) Denis Savenkov and Eugene Agichtein. 2016.  When a Knowledge Base Is Not Enough: Question Answering over Knowledge Bases with External Text Data. In _SIGIR_.
  * Saxena et al. (2021) Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021.  Question Answering Over Temporal Knowledge Graphs. In _ACL_.
  * Shang et al. (2022) Chao Shang, Guangtao Wang, Peng Qi, and Jing Huang. 2022.  Improving Time Sensitivity for Question Answering over Temporal Knowledge Graphs. In _ACL_.
  * Sharma et al. (2023) Aditya Sharma, Apoorv Saxena, Chitrank Gupta, Seyed Mehran Kazemi, Partha Talukdar, and Soumen Chakrabarti. 2023.  TwiRGCN: Temporally Weighted Graph Convolution for Question Answering over Temporal Knowledge Graphs. In _EACL_.
  * Sun et al. (2019) Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019.  PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text. In _EMNLP-IJCNLP_.
  * Sun et al. (2018) Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W. Cohen. 2018.  Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text. In _EMNLP_.
  * Tan et al. (2023) Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023.  Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models. In _ACL_.
  * Veselovsky et al. (2023) Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023.  Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks. In _arXiv_.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A Free Collaborative Knowledgebase.  _CACM_ (2014).
  * Wu et al. (2020a) Jiapeng Wu, Meng Cao, Jackie Chi Kit Cheung, and William L Hamilton. 2020a.  TeMP: Temporal Message Passing for Temporal Knowledge Graph Completion. In _EMNLP_.
  * Wu et al. (2020b) Wenqing Wu, Zhenfang Zhu, Qiang Lu, Dianyuan Zhang, and Qiangqiang Guo. 2020b.  Introducing External Knowledge to Answer Questions with Implicit Temporal Constraints over Knowledge Base.  _Future Internet_ (2020).
  * Xiao et al. (2022) Yao Xiao, Guangyou Zhou, and Jin Liu. 2022.  Modeling Temporal-Sensitive Information for Complex Question Answering over Knowledge Graphs. In _NLPCC_.
  * Xu et al. (2021) Chengjin Xu, Yung-Yu Chen, Mojtaba Nayyeri, and Jens Lehmann. 2021.  Temporal Knowledge Graph Completion using a Linear Temporal Regularizer and Multivector Embeddings. In _NAACL_.
  * Xu et al. (2016a) Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016a.  Hybrid Question Answering over Knowledge Base and Free Text. In _COLING_.
  * Xu et al. (2016b) Kun Xu, Siva Reddy, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2016b.  Question Answering on Freebase via Relation Extraction and Textual Evidence. In _ACL_.
  * Yahya et al. (2012) Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, Maya Ramanath, Volker Tresp, and Gerhard Weikum. 2012.  Natural Language Questions for the Web of Data. In _EMNLP_.
  * Yao et al. (2022) Junping Yao, Yijing Wang, Xiaojun Li, Cong Yuan, and Kaiyuan Cheng. 2022.  TERQA: Question Answering over Knowledge Graph Considering Precise Dependencies of Temporal Information on Vectors.  _Displays_ (2022).
  * Yih et al. (2015) Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015.  Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In _ACL-IJCNLP_.
  * Zhang et al. (2010) Ruiqiang Zhang, Yuki Konda, Anlei Dong, Pranam Kolari, Yi Chang, and Zhaohui Zheng. 2010.  Learning Recurrent Event Queries for Web Search. In _EMNLP_.

Table 8. Prompt including demonstrations for rephrasing the pseudo-questions
into natural questions.

Please rephrase the following input question into a more natural question.
---
Input: What album Sting ( musician ) was released, during, Sting award
received German Radio Award?
Question: which album was released by Sting when he won the German Radio
Award?
Input: What human President of Bolivia was the second and most recent female
president, after, president of Bolivia officeholder Evo Morales?
Question: Which female president succeeded Evo Morales in Bolivia?
Input: What lake David Bowie He moved to Switzerland purchasing a chalet in
the hills to the north of , during, David Bowie spouse Angela Bowie?
Question: Close to which lake did David Bowie buy a chalet while he was
married to Angela Bowie?
Input: What human Robert Motherwell spouse, during, Robert Motherwell He also
edited Paalen ’s collected essays Form and Sense as the first issue of
Problems of Contemporary Art?
Question: Who was Robert Motherwell’s wife when he edited Paalen’s collected
essays Form and Sense?
Input: What historical country Independent State of Croatia the NDH government
signed an agreement with which demarcated their borders, during,
Independent State of Croatia?
Question: At the time of the Independent State of Croatia, which country
signed an agreement with the NDH government to demarcate their borders?
Input: What U-boat flotilla German submarine U-559 part of, before, German
submarine U-559 She moved to the 29th U-boat Flotilla?
Question: Which U-boat flotilla did the German submarine U-559 belong to
before being transferred to the 29th U-boat Flotilla?
Input: What human UEFA chairperson, during, UEFA chairperson Sandor Barcs?
Question: Who was the UEFA chairperson after Sandor Barcs?
Input: What human Netherlands head of government, during, Netherlands head of
state Juliana of the Netherlands?
Question: During Juliana of the Netherlands’ time as queen, who was the prime
minister in the Netherlands?

Table 9. Prompts used to obtain the training data for generating intermediate
questions, leveraging in-context learning.

TimeQuestions |  Tiq
---|---
Generate an explicit question and answer type for the implicit part of the temporal input question. |  Generate an explicit question and answer type for the implicit part of the temporal input question.
Input: what position did djuanda kartawidjaja take after he was replaced by
sukarano
Output: when djuanda kartawidjaja replaced by sukarano——date |  Input: Who was the second director of the Isabella Stewart Gardner Museum when it was built
Output: When Isabella Stewart Gardner Museum was built——time interval
Input: american naval leader during the world war 2
Output: when world war 2——time interval |  Input: When Wendy Doniger was president of the Association for Asian Studies, what publishing house was she based in New York
Output: When Wendy Doniger was president of the Association for Asian
Studies——time interval
Input: who became president after harding died
Output: when harding died——date |  Input: What administrative entity was Ezhou in before Huangzhou District became part of it
Output: When Huangzhou District became part of Ezhou——date
Input: who did luis suarez play for before liverpool
Output: when luis suarez play for liverpool——time interval |  Input: After Bud Yorkin became the producer of NBC’s The Tony Martin Show, who was his spouse?
Output: When Bud Yorkin became the producer of NBC’s The Tony Martin
Show——date
Input: which countries were located within the soviet union prior to its
dissolution
Output: when soviet union dissolution——date |  Input: What book did Ira Levin write that was adapted into a film during the same time he wrote the play Deathtrap
Output: When Ira Levin wrote the play Deathtrap——date
Input: who started the presidency earliest and served as president during wwii
in the US
Output: when wwii——time interval |  Input: What basketball team was Nathaniel Clifton playing for when his career history with the Rens began
Output: When Nathaniel Clifton’s career history with the Rens began——time
interval
Input: who replaced aldo moro as the minister of foreign affairs
Output: when aldo moro replaced as minister of foreign affairs——date |  Input: What team did Stevica Ristić play for before joining Shonan Bellmare?
Output: When Stevica Ristić joining Shonan Bellmare——time interval
Input: what did harry s truman work before he was president
Output: when harry s truman president——time interval |  Input: Which album was released by the Smashing Pumpkins after Mike Byrne joined the band
Output: When Mike Byrne joined Smashing Pumpkins——time interval

##  Appendix A Further implementation details

###  A.1. Training Faith on Tiq

Faith requires train questions asking for temporal values to answer
intermediate questions. Such questions exist in TimeQuestions, but our new Tiq
benchmark only has implicit questions (by design). We thus generate
intermediate questions on the train and dev sets, using the implicit questions
as input (similar as in Sec. 3.1). For these intermediate questions, the gold
answer is the temporal value of the implicit part as annotated in the Tiq
benchmark, resulting in ¡question, temporal value¿ pairs. If the answer type
of an intermediate question is a time interval, we create two questions asking
for “start date” and “end date” respectively. We obtain $7{,}723$ such pairs
from the Tiq train set and $2{,}542$ questions from the dev set.

###  A.2. TSF Construction

Training. To obtain training data for the TSFs, we follow a similar distant
supervision approach as in (Christmann et al., 2023) for obtaining the target
question entity and question relation. We run the heterogeneous retriever (see
Sec. 3.2) on the full input question, which identifies entity mentions in the
input, disambiguates these to KB-entities, and then retrieves information
snippets for the KB-entities from heterogeneous sources. If the retrieved
information snippets for a KB-entity contain the annotated gold answer, we
annotate the corresponding entity mention as relevant question entity. The
remaining parts of the question are annotated as question relation. The
expected answer type is the most frequent (proxy for most prominent) KB-type
of the gold answer. The temporal signal and the temporal category are looked
up from the annotations in the benchmarks. These individual parts of the TSF
are then combined and separated by pipes (“——”), to obtain the target TSFs
that are used for training the TSF construction model (in Sec. 3.1). Details
on the training configuration are provided in Sec. 5.1.

Inference. The input to the BART model for TSF construction is the question.
The output is the concatenation of the individual slots, separated by two
pipes (“——”): “{entities}——{relation}——{expected answer type}——{temporal
signal}——{temporal categorization}”. Example output for $q_{1}$=“Record
company of Queen in 1975?”: “Queen——Record company of in 1975——record
company——overlap——non-implicit”.

Table 10. Performance of Faith (with all sources) on questions from different source combinations in Tiq (test set). Question sources combination | P@1 | MRR | Hit@5
---|---|---|---
[Text; Infobox] ($157$ questions) | $0.573$ | $0.644$ | $0.752$
[KB; Text] ($378$ questions) | $0.519$ | $0.634$ | $0.770$
[Infobox; KB] ($43$ questions) | $0.395$ | $0.534$ | $0.721$
[Infobox; Text] ($225$ questions) | $0.476$ | $0.595$ | $0.760$
[KB; Infobox] ($251$ questions) | $0.478$ | $0.598$ | $0.753$
[Text; KB] ($142$ questions) | $0.359$ | $0.509$ | $0.725$
[KB; KB] ($127$ questions) | $0.582$ | $0.676$ | $0.807$
[Text; Text] ($306$ questions) | $0.598$ | $0.679$ | $0.787$
[Infobox; Infobox] ($376$ questions) | $0.407$ | $0.529$ | $0.689$
All test questions ($2{,}000$ questions) | $0.497$ | $0.610$ | $0.756$

###  A.3. Intermediate Question Generation

Training. To annotate intermediate questions (and its expected answer type)
for implicit questions, we leverage in-context learning: we select and label
$8$ questions from the train set, and give these pairs as context to the LLM
(InstructGPT). This way we annotate the remaining questions in the train and
dev sets of Tiq/TimeQuestions resulting in $5{,}875$/$847$ instances in the
train set and $1{,}949$/$287$ instances in the dev set. On this data we fine-
tune the BART model to be independent of GPT at runtime. The prompts used for
annotating the data can be found in Table 9. Training configuration is
provided in Sec. 5.1.

Inference. For generating the intermediate questions at runtime, we provide
the implicit question as input to the trained BART model. The output is the
intermediate question that describes the implicit constraint, and the expected
answer type for this question, separated by two pipes: “{intermediate
question}——{expected answer type}”. Example output for $q_{3}$=“Queen’s record
company when recording Bohemian Rhapsody?”: “when Queen recording Bohemian
Rhapsody——time interval”.

###  A.4. Evidence Scoring

As the set of candidate information snippets after temporal pruning can still
be large, we use a re-ranker (Nogueira and Cho, 2019) to prune out irrelevant
candidates, based on cross-encodings obtained via DistilRoBERTa777
https://huggingface.co/distilroberta-base..

Training. The training data are the ¡question, information snippet¿ pairs,
annotated with either a positive label (in case the snippet contains a gold
answer) or a negative label (otherwise). We randomly sample $1$ positive
¡question, information snippet¿ pair from each knowledge source and $15$
negative pairs, for each question. We use the concatenation of question
entity, the question relation, and the expected answer type, as present in the
TSF, to represent the question. For fine-tuning the classifier on this data,
we use AdamW as optimizer with a learning rate of $2\times$$10^{-5}$, batch
size of $16$, weight decay of $0.01$, $4$ epochs, and a warm-up ratio of
$0.1$.

Inference. We score each candidate information snippet (known to be temporally
faithful) and consider the top-$100$ information snippets as input for the
final answering stage.

##  Appendix B Further experiments

###  B.1. Intrinsic Evaluation

Temporal signal accuracy. We measure the accuracy of the generated temporal
signals (before, after or overlap) in our TSF construction. On TimeQuestions
the accuracy is $93.0\%$, and on Tiq it is $97.8\%$. The high accuracy scores
indicate that our approach of generating the temporal signal is feasible.

Temporal category accuracy. We also measure the accuracy in predicting the
temporal category, differentiating between implicit and non-implicit
questions. The accuracy is $98.9\%$ on TimeQuestions and $100\%$ on Tiq, which
has only implicit questions.

Performance of implicit question resolver. The performance of the implicit
question resolver is crucial for correctly answering implicit questions, as
the resulting temporal values are directly used for pruning out evidence in
the remainder of the pipeline. As there might be several temporal values per
question (due to multiple intermediate questions), we measure macro-averaged
precision, recall, and F1-score. We conduct experiments on the test set of
Tiq, which has the ground-truth temporal values.

When using the top-$1$ answer candidate per intermediate question (default
setting), precision is $0.537$, recall is $0.562$ and F1-score is $0.525$.
When increasing the number of candidates to $3$, precision is $0.294$, recall
is $0.714$ and F1-score is $0.401$. With the top-$5$ candidates, precision is
$0.196$, recall is $0.773$ and F1-score is $0.304$.

Recall improves as we consider more answers, since the resulting explicit
temporal constraint is relaxed. Hence, the evidence retained is noisier and
may not satisfy the user-intended temporal constraints. This can negatively
affect the system’s faithfulness.

###  B.2. Additional Analysis

Figure 4. P@1 of Faith when considering top-$k$ answers for the generated
intermediate question(s) of implicit questions.

Relaxed temporal pruning. We also compute the effect of the number of answer
candidates in the implicit question resolver on the end-to-end performance
(extrinsic evaluation). Fig. 4 shows the results, varying the number of
candidates $k$ from $1$ to $20$. On TimeQuestions, we observe that P@1
improves gradually as $k$ increases until the set of candidate snippets
converges resulting in a stable P@1. On Tiq, we only observe an improvement of
the P@1 metric when increasing $k$ to $2$. As $k$ increases further, more
noisy candidate snippets are considered, resulting in a lower performance.

Answer presence analysis. We measure the answer presence after the initial
heterogeneous retrieval, and the effect of the subsequent pruning and scoring
steps. Answer presence is measured as the fraction of questions for which the
gold answer is present in the candidate set of information snippets
(Christmann et al., 2022a). These measurements are also used for the error
analysis presented in Sec. 5.4.

We conduct the analysis on the test sets of Tiq/TimeQuestions. The answer
presence after the heterogeneous retrieval is $0.984$/$0.861$. After temporal
pruning the answer presence drops to $0.872$/$0.741$. Note that in this step,
temporally inconsistent evidence is pruned out, enhancing the faithfulness of
the approach. In the evidence scoring stage (based on a cross-encoder), the
answer presence is mostly retained ($0.867$/$0.726$). Inside the Explaignn
pipeline, the answer presence after evidence pruning is $0.829$/$0.693$.

In general, as discussed in the error analysis, the key source of error is the
fine-grained answer ranking step.

Faith performance on heterogeneous questions. Tiq has questions originating
from different source combinations (see Fig. 3). Table 10 shows how this
affects the performance of Faith (all sources are used as input for
answering).

Results demonstrate that Faith can deal with all of these questions, and there
is no combination for which Faith completely fails, indicating that Faith
successfully incorporates all heterogeneous sources during answering. Faith
shows the best performance on questions from text and infoboxes (P@1 of
$0.573$), and the worst performance on questions from text and KB (P@1 of
$0.359$).