# TriviaHG: A Dataset for Automatic Hint Generation from Factoid Questions

Jamshid Mozafari  University of InnsbruckInnrain 15InnsbruckAustria
<EMAIL_ADDRESS>,  Anubhav Jangra  Columbia UniversityNew YorkUSA
<EMAIL_ADDRESS>and  Adam Jatowt  University of InnsbruckInnrain
15InnsbruckAustria<EMAIL_ADDRESS>
(2024)

###### Abstract.

Nowadays, individuals tend to engage in dialogues with Large Language Models,
seeking answers to their questions. In times when such answers are readily
accessible to anyone, the stimulation and preservation of human’s cognitive
abilities, as well as the assurance of maintaining good reasoning skills by
humans becomes crucial. This study addresses such needs by proposing hints
(instead of final answers or before giving answers) as a viable solution. We
introduce a framework for the automatic hint generation for factoid questions,
employing it to construct TriviaHG, a novel large-scale dataset featuring
160,230 hints corresponding to 16,645 questions from the TriviaQA dataset.
Additionally, we present an automatic evaluation method that measures the
Convergence and Familiarity quality attributes of hints. To evaluate the
TriviaHG dataset and the proposed evaluation method, we enlisted 10
individuals to annotate 2,791 hints and tasked 6 humans with answering
questions using the provided hints. The effectiveness of hints varied, with
success rates of $96\%$, $78\%$, and $36\%$ for questions with easy, medium,
and hard answers, respectively. Moreover, the proposed automatic evaluation
methods showed a robust correlation with annotators’ results. Conclusively,
the findings highlight three key insights: the facilitative role of hints in
resolving unknown questions, the dependence of hint quality on answer
difficulty, and the feasibility of employing automatic evaluation methods for
hint assessment.

Hint Generation, Question Answering, Large Language Models

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
f the 47th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 14–18, 2024; Washington D.C., USA††isbn:
978-1-4503-XXXX-X/18/06††ccs: Information systems Retrieval models and
ranking††ccs: Information systems Evaluation of retrieval results

##  1\. Introduction

Automatic question answering (QA) systems (Abdel-Nabi et al., 2023; Rogers et
al., 2023) have recently gained significant attention, allowing users to pose
questions and receive direct answers. Traditional QA systems excel in specific
knowledge domains by extracting relevant text spans from passages. In
contrast, Large Language Models (LLMs) (Team et al., 2023; Workshop et al.,
2022; Brown et al., 2020a), pre-trained on vast datasets, offer a more
versatile approach. LLMs can handle multiple NLP tasks, including automatic
QA, and can answer questions across diverse domains. Unlike extractive QA
systems, they generate answers by creating new content, showcasing high
proficiency and potential as an alternative to traditional QA systems.

Figure 1. Example hints for two sample questions in TriviaHG with their
computed convergence (CON) and familiarity (FAM) attributes provided on a
scale of 1 to 5 (1 is the lowest and 5 is the highest quality.

While advanced systems like LLMs excel in answering questions, they raise
concerns about potentially contributing to weakening essential human skills
like deep thinking, reasoning, critical analysis, and even self-confidence
(Bandura, 2013). The problem lies in that advanced AI systems like LLMs
discourage users from independently contemplating and solving questions,
hindering the development of users’ reasoning, memorization, and analytical
abilities. Since the answers to user questions can be readily given at any
time and are usually correct, then human activities that are common in
traditional information searching and browsing are becoming less necessary and
may occur with lesser intensity. We note here that although the current LLMs
are obviously not perfect (hallucinations, inability to answer complex
questions, etc.), we expect the models to keep improving in the future
(judging from the recent speed of technological advancement). Furthermore,
psychological and cognitive studies highlight the importance of allowing
individuals to derive answers independently, fostering self-confidence, and
promoting further learning (Usher and Pajares, 2006). Hence, occasionally
letting users come up with answers to their questions themselves through
suitable guidance should have a positive psychological effect and encourage
self-learning. Finally, helping to answer questions is quite common in a
variety of quizzes and quiz games, which are enjoyed by many.

In view of the above, the approaches that involve humans in the answering
process instead of relying solely on answers generated by automated systems
should become useful, perhaps not for everyone, but at least for interested
users. We then focus in this work on using LLMs to provide hints for questions
rather than generating definitive answers. Hints are clues that guide
individuals towards a solution without explicitly providing the answer
(Gregory Hume and Evens, 1996). They promote thinking, reasoning, and
memorization skills, as humans must engage in active and thoughtful
consideration after reading the hints instead of passively receiving ready
answers. Additionally, relying on human input might enhance the reliability
and trust in the answers as individuals draw on their own knowledge.

The use of hints to assist users is not a new concept in computer science. For
example, in the field of automatic hint generation for programming (AHGP)
(McBroom et al., 2021), the task is to generate hints automatically to aid
students in coding exercises. However, there is limited research on automatic
hint generation specifically for factoid questions (AHGQ) - questions that can
be answered with facts expressed in a few words (Jurafsky, 2000; Jatowt et
al., 2023). While both AHGP and AHGQ aim to assist humans, the former is more
tailored to coding exercises, often suggesting code edits, and the latter is a
more general task applicable across various types of questions. Furthermore,
there is an additional benefit from focusing on AHGQ task. Hints for questions
could be used to enhance Retrieval Augmented Generation (RAG) solutions (Gao
et al., 2023; Abdallah and Jatowt, 2023) and Query Expansion (Wang et al.,
2023) methods. Specifically, in the context of Retrieval Augmented Generation,
incorporating hints as query explanations has the potential to enhance the
model’s ability to understand and process queries. As mentioned above, the
hints may also find applications in entertainment, such as in game shows like
Jeopardy.

As a novel task, AHGQ faces challenges such as the absence of datasets and
reliable evaluation methods. Addressing this gap, we introduce a framework for
AHGQ and utilize it to generate TriviaHG111The code, dataset, and experimental
results are freely available at https://github.com/DataScienceUIBK/TriviaHG, a
dataset for automatic hint generation for factoid questions. Our approach
relies on web pages as the primary information source and utilizes Bing Chat
AI222https://chat.bing.com to generate hints based on online sources, ensuring
accuracy and relevance. The hints are designed to be user-friendly and
helpful.

As a second contribution, we propose an automatic evaluation method to assess
hint quality, considering aspects of convergence and familiarity. The
convergence quality is a characteristic that reflects the extent to which a
hint can narrow down or eliminate potential answers to a given question. The
familiarity is a characteristic that signifies the degree to which the
entities mentioned in a hint are well-known or widely recognized. Figure 1
illustrates two questions along with a few examples hints from TriviaHG along
with the computed convergence and familiarity quality attributes.

To sum up, we make the following contributions in this work:

  * •

We release a dataset called TriviaHG for automatic hint generation for
questions, which includes 16,645 questions and 160,230 hints about persons,
locations, entities, dates, and numbers.

  * •

We propose an automatic evaluation method for assessing the quality of
convergence and familiarity of hints for AHGQ.

  * •

The analysis of the TriviaHG dataset and the automatic evaluation method
demonstrates their quality and effectiveness, validating the framework and
evaluation approach employed in the study.

##  2\. Related Works

###  2.1. QA Task

QA systems can be divided into two categories: Extractive QA and Generative
QA. Extractive QA systems (Wang et al., 2022b) try to extract the answer from
the passage. Ramnath et al. (2020) fine-tune the BERT (Devlin et al., 2019)
language model on QA datasets and employ the fine-tuned model to extract the
final answer from the given passage. In the same way, Yu and Liu (2021)
utilize RoBERTa (Liu et al., 2019) language model instead of BERT to extract
the underlying semantic features of inputs more effectively. Generative QA
systems (Chen et al., 2023), on the other hand, try to generate the final
answer. Generative QA, as opposed to extractive QA, offers advantages such as
the ability to combine information from multiple sources to create coherent
responses and generate original content, making it more flexible in language
and capable of reasoning by comparing facts or drawing conclusions. Yu et al.
(2023) propose employing generative models such as LLMs to generate contextual
documents rather than retrieving the documents from a collection. They show
that their method has better results on TriviaQA (Joshi et al., 2017) and
WebQA (Berant et al., 2013) datasets. Izacard and Grave (2021) leverage
passage retrieval component along with generative models for generating final
answer using a pool of retrieved documents.

Multiple QA datasets have been proposed in various domains, such as Legal
(Zhong et al., 2020), Medical (Pal et al., 2022), News (Wang et al., 2022a),
etc. In these datasets, each record includes three main parts: a question, an
answer, and a passage such that the answer is a span of the passage. SQuAD
(Pranav et al., 2018) consists of 100,000+ questions posed by crowd-workers on
536 Wikipedia articles. Natural Questions (Kwiatkowski et al., 2019) contains
300,000 questions and is generated by sampling questions issued to the Google
search engine. TriviaQA (Joshi et al., 2017), which we use in our research, is
an extensive open-domain QA dataset sourced from trivia games, quizzes, and
open-domain question collections, covering a broad spectrum of topics. With
over 98,000 question-answer pairs and diverse linguistic forms, TriviaQA
serves as a robust benchmark challenging systems to excel in comprehension and
reasoning across various domains, driving advancements in machine reading
comprehension. Also, some works utilize LLMs to generate datasets (Liu, 2023;
Wang et al., 2022a; Mallen et al., 2023). SecQA (Liu, 2023) is a dataset
containing 242 questions generated by GPT-4 (Achiam et al., 2023) based on the
Computer Systems Security: Planning for Success textbook (Tolboom, 2023).

The previous methods do not focus on generating hints but rather on providing
the final answers.

Figure 2. Dataset generation framework: Arrows represent the output quantity
based on the number of questions, and callouts illustrate the statistics of
the training, validation, and test sets. Qs and Hs denote the number of
questions and hints, respectively.

###  2.2. Hint Generation

There are some datasets for the AHGP task such as iSnap Hint
Rating333https://pslcdatashop.web.cmu.edu/Project?id=321, and ITAP Hint
Rating444https://pslcdatashop.web.cmu.edu/Project?id=294. Both datasets
comprise log data obtained from students engaged in various programming tasks,
encompassing comprehensive traces of their code and logs of hint requests. The
iSnap dataset (Price et al., 2019) was compiled from an introductory
programming course for non-majors conducted during the Fall 2016, Spring 2017,
and Fall 2017 semesters, involving 171 students who completed six programming
problems. The ITAP dataset was sourced from two programming courses in Spring
2016, encompassing 89 students who tackled up to 40 Python problems (Rivers et
al., 2016).

The AHGP methods can be broadly classified into two categories: Data-Driven
and Automated Testing-Based. Data-driven methods leverage educational data
mining and machine learning techniques to automate the creation of a domain
model and hints from student problem-solving data (Jin et al., 2012). An
example of this is the ITAP system, which consists of three stages:
canonicalization, path construction, and reification (Rivers, 2017). The
Automated Testing-Based methods use automated tests to guide the hint
generation process (Obermüller et al., 2021). For instance, the Catnip system
selects suitable candidate solutions from a pool of solutions using an
automated Whisker test suite (Obermüller et al., 2021).

Generally, existing AHGP systems generate hints only for programming tasks. In
other words, none of these methods generates hints for factoid questions. To
the best of our knowledge, Jatowt et al. (2023) is the only study focused on
the AHGQ task; however, they focused on Wikipedia to generate hints without
employing any LLMs. Additionally, the authors do not generate any datasets for
AHGQ.

###  2.3. Evaluation Methods

There are some evaluation methods in QA, such as ExactMatch (EM), BERTScore
(Zhang* et al., 2020), BEM (Bulian et al., 2022), and InstructGPT-eval
(Kamalloo et al., 2023). EM is an evaluation metric that measures the
percentage of predicted answers that match exactly with the reference answers,
providing a binary assessment of correctness. BertScore (Zhang* et al., 2020)
is an evaluation metric that utilizes pre-trained BERT embeddings to measure
the similarity between a generated text and a reference text. BEM (Bulian et
al., 2022) is an evaluation metric that employs a fine-tuned BERT language
model trained on a dataset comprising candidate answers that are equivalent to
each reference answer to measure the similarity between predicted answers and
reference answers. These candidate answers are generated by various QA systems
trained on SQuAD (Pranav et al., 2018). InstructGPT-eval (Kamalloo et al.,
2023) utilizes LLMs and prompts to measure the similarity of predicted answers
and reference answers.

Evaluation methods for AHGP can be broadly classified into four categories:
(1) User Studies and Surveys, (2) Comparisons with Experts, (3) Analysis Using
Historical Data, and (4) Technical Evaluations (McBroom et al., 2021). The
User Studies and Surveys category involves collecting feedback from users
(typically students) who interact with the hint system (Head et al., 2017; Lee
et al., 2018). The Comparisons with Experts category involves comparing the
hints generated by the system with those provided by human experts (Piech et
al., 2015; Price et al., 2017). The Analysis Using Historical Data involves
using past student interaction data to evaluate the effectiveness of the hints
(Lazar et al., 2017; Haldeman et al., 2018). The Technical Evaluations
category involves assessing the system’s performance in terms of computational
efficiency, scalability, and robustness (Marin et al., 2017).

All the evaluation methods for QA are automatic, but there are a few automatic
evaluation methods for AHGP. It shows that assessing hint quality
automatically is a complex task. However, the automatic evaluation methods for
AGHP cannot be employed to assess AGHQ because they just focus on the
programming aspects of hints. As said, Jatowt et al. (2023) is the only study
focused on the AGHQ, but they also evaluate the quality of hints manually by
asking humans to assess the hints and do not propose any automatic evaluation
methods for AGHQ. In our work, we propose a fully automated method for
evaluating hint qualities.

##  3\. Dataset

In this section, we outline the framework and its stages for generating the
TriviaHG dataset. The framework comprises two modules: the Question Sampling
Module and the Hint Generation Module. Figure 2 provides a visual
representation of the framework and its stages, which we will discuss in
detail in the subsequent sections.

###  3.1. Question Sampling Module

This module’s task is to select questions for hint generation.

####  3.1.1. Question Selection

We select the TriviaQA dataset (Joshi et al., 2017) as our primary question
source because it offers challenging trivia-style questions covering a wide
range of diverse topics. It has also been frequently used in QA research
(Beltagy et al., 2020; Zaheer et al., 2020; Su et al., 2024; Prasad et al.,
2023; Nassiri and Akhloufi, 2023). We filter out questions with fewer than 6
words or exceeding 20 words, as well as those lacking question marks.
Additionally, questions whose answers do not have corresponding Wikipedia
pages are excluded, as Wikipedia is utilized for evaluating familiarity
quality. After applying these criteria, 66,300 questions remain available for
further processing.

####  3.1.2. Question Type Detection

A question classifier assigns natural language questions to classes that
reflect their intended purposes. For instance, Who is Barack Obama? falls into
the PERSON class, while Where is Washington? belongs to the LOCATION class. We
finetune RoBERTa (Liu et al., 2019) model using the TREC Question
Classification dataset (Li and Roth, 2002) to develop a model named QT
Detector as a question classifier555The accuracy of the QT Detector on TREC
Question Classification dataset is $92.8\%$. Using the QT Detector, questions
are categorized into classes at various levels of granularity, comprising 5
coarse-grained classes and 50 fine-grained classes. Subsequently, we exclude
questions categorized as DESCRIPTION type, focusing solely on factoid
questions, resulting in 62,248 questions.

####  3.1.3. Sampling

Using the stratified sampling method (Arnab, 2017), we conduct sampling from
the questions based on their coarse-grained classes. Questions are categorized
into subgroups according to their question type, and random sampling is
carried out within each subgroup. In total, 20,502 questions (one-third) are
sampled.

###  3.2. Hint Generation Module

This module generates hints for the selected subset of questions.

####  3.2.1. Hint Generator

We utilize LLMs to generate hints for the questions, specifically selecting
Bing Chat AI as our preferred choice. Bing Chat AI functions as a RAG system,
leveraging also information from retrieved documents on the Internet to
formulate answers to user questions. This system not only generates answers
based on existing online documents but also provides the sources of the hints
and answers. In contrast, LLMs like GPT-4 (Achiam et al., 2023), Gemini (Team
et al., 2023), and WizardLM (Xu et al., 2023) rely on their stored knowledge
to generate answers without indicating the sources. Therefore, Bing Chat AI is
considered more reliable for hint generation as it transparently presents the
sources of the generated hints. To generate hints for questions, we follow
these steps:

  1. (1)

We first prompt Bing Chat AI to answer a question. This enables us to
determine if Bing Chat AI is capable of retrieving the answer. If it is unable
to do so, we discard the question.

  2. (2)

We validate the accuracy of the generated answer by comparing it with the
ground-truth answer. If they match, we proceed to the next step; otherwise, we
exclude the question. This process enables us to eliminate questions whose
answers may have changed, given that the TriviaQA dataset (Joshi et al., 2017)
dates back to 2017, and it is conceivable that the answers to these questions
may have evolved since then. After removing such questions, 18,805 questions
remained.

  3. (3)

We instruct Bing Chat AI to produce 10 hints for the question without
including the answer in the hints. This approach allows us to generate hints
without the risk of answer leakage, i.e., the case when a hint explicitly
reveals the answer to the question in its content666For instance, for the
question Who won the FIVB Volleyball Men’s World Championship in 2018?, the
hint Their official language is Polish. exhibits answer leakage.. Figure 3
illustrates the trend of using prompts in Bing Chat AI for hint generation.

Figure 3. The hint generation system initiates by prompting the question to
Bing ChatAI. Then, it produces a snippet serving as the answer. Following
this, we assess the correctness of the provided answer. If the answer is
correct, we prompt Bing Chat AI to generate 10 hints. The numbers in the
brackets denote the source pages for both the answer and the hints.

####  3.2.2. Hint Filtering

While we advise Bing Chat AI to exercise caution regarding the problem of
answer leakage, there are instances where it might produce hints inadvertently
revealing the answer. Additionally, there are occasions when it generates
hints that merely rephrase or describe the questions. For instance, for the
question In which city are the headquarters of the International Monetary
Fund?, the model may generate a hint like The headquarters of the
International Monetary Fund is located in this city. Unfortunately, these
hints merely repeat the question and do not help users in finding the answer.
To address these issues, we follow these steps:

  1. (1)

For identifying potential answer leakages in the generated hints, we utilize
lexical comparisons between the hints and the answer. In this process, we
first tokenize the hint sentence based on words and extract the lemma of each
word using the WordNetLemmatizer777We use NLTK (Natural Language Toolkit)
library for word tokenizing, stop words, and lemmatization.. Similarly, we
perform the same procedure for the answer, but additionally, we exclude common
stop words like a and the from the answer. Subsequently, if there are matching
word lemmas between the hint and the answer, we eliminate the corresponding
hint.

  2. (2)

We utilize the SentenceTransformers framework (Reimers and Gurevych, 2019) for
identifying similar hints to the questions. By calculating the cosine
similarity between question and hint embeddings, we establish a threshold at
$0.72$888We arrived at this value by experimenting with different values.;
hints surpassing this threshold are then eliminated.

We then eliminate questions with fewer than five hints, as we believe that
having at least five hints is necessary for being able to assist users in
finding the answers. By following these exclusions, 16,645 questions and
160,230 hints remain.

Table 1. The attributes of TriviaHG dataset.

Name | Definition
---|---
Q_ID | Question’s ID
Question | Text of the question
Hints | The generated hints
Hints_Sources | The sources utilized to derive the hints.
Snippet | The generated passage as response
Snippet_Sources | The sources accessed to obtain the answer.
ExactAnswer | The precise answer to the question.
MajorType | The coarse-grained class of the question type.
MinorType | The fine-grained class of the question type.
Candidate_Answers | The candidate answers generated for convergence.
Q_Popularity | Normalized Wikipedia page views for question entities.
Exact_Answer_Popularity | Normalized Wikipedia page views for the answer.
H_Popularity | Normalized Wikipedia page views for hints entities.
Scores | Result of the Candidate Evaluator component.
Convergence | Evaluation value of convergence
Familiarity | Evaluation value of familiarity

####  3.2.3. Hint Labeling

We prepare various attributes to be included for each question in the TriviaHG
dataset, as shown in Table 1.

##  4\. Automatic Evaluation Method

We consider there are at least five quality attributes for each hint:
_Relevance_ , _Readability_ , _Ambiguity_ , _Convergence_ , and _Familiarity_.
Relevance evaluates how well the hint corresponds to the question, while
Readability gauges its clarity and comprehensibility. Ambiguity quantifies the
degree of uncertainty conveyed by the hint. Convergence quality signifies the
hint’s capacity to refine or exclude potential answers, while Familiarity
quality measures the recognition level of entities mentioned in the hint.

In this work, we focus on automating the evaluation of convergence and
familiarity quality attributes, as, in our opinion, they are the strongest
indicators of hint quality. The evaluation of relevance can be conducted using
the SentenceTransformers framework (Reimers and Gurevych, 2019). Additionally,
to evaluate hints for their readability, factors such as hint length and the
difficulty of words can be considered as well as existing readability indexes
or machine learning solutions could be harnessed (Martinc et al., 2021).
Finally, assessing the ambiguity attribute proves challenging, and finding a
suitable evaluation method is inherently difficult. Relevance and readability
will be the focus of our future work.

###  4.1. Convergence Quality Attribute

We are inspired by the elimination method from cognitive science. The
elimination method involves a systematic process of understanding, planning,
and determining the correct answer by progressively excluding unlikely or
irrelevant options based on the provided context or prior knowledge (De Jong,
1996; Tong et al., 2023). Here, we regard the hints as the context and the
stored knowledge in LLMs as prior knowledge. Subsequently, we eliminate
irrelevant options999In this study, we consider candidate answers as options.
by utilizing both the provided context and prior knowledge.

The method involves three modules: Candidate Generator, Candidate Evaluator,
and Scoring. These stages are elaborated in subsequent sections, with Figure 4
illustrating the method’s workflow.

Figure 4. Convergence Evaluator: The process begins by directing the question
to the Candidate Generator stage, which generates up to twenty candidate
answers. Subsequently, each generated candidate answer and the hint undergo an
evaluation to determine the validity of the hint for the respective candidate
answer. Finally, the results are conveyed to the Scoring stage for computation
of the final score.

####  4.1.1. Candidate Generator

We employ LLMs to generate candidate answers for a given question, leveraging
the vast knowledge stored within their parameters to retrieve the most
potential answers. In this process, we initially prompt the LLM to generate up
to $20$ candidate answers for the given question.101010For example, when
considering the question In which city are the headquarters of the
International Monetary Fund?, some of the generated candidate answers include
Brussels, Vienna, Paris, Tokyo, Moscow, Stockholm, Amsterdam, London,
Edinburgh, etc. The use of up to in the prompt accounts for instances where
questions may not have $20$ candidate answers. For instance, in the question
If Christmas Day falls on a Tuesday, on what day will New Year’s Day fall?
with the answer Tuesday, there are only seven candidate answers since they
correspond to weekdays. The prompt we employed is as follows:

      Generate up to 20 candidate answers words for the question
      "QUESTION" in bullet points.

In the prompt, the term QUESTION pertains to the provided question. Then, we
analyze the output, extract the candidate answers from the list, and feed them
along with the hint into the Candidate Evaluator module.

####  4.1.2. Candidate Evaluator

We employ LLMs to determine if the hint pertains to the generated candidate
answers. This is accomplished by prompting the model with the following
prompt:

      Does the hint "HINT" refer to "CANDIDATE"? Choose ONLY
      between "Yes" or "No".

In the prompt, the term HINT corresponds to the provided hint, while CANDIDATE
corresponds to one of the generated candidate answers. This prompt is utilized
for all the generated candidate answers, and the resulting outputs, which
include lists of Yes and No, are directed to the Scoring module. We interpret
Yes as one and No as zero.

####  4.1.3. Scoring

In this module, we utilize the following equation to calculate the final score
for the convergence quality attribute of the hint. To assess the convergence
quality, if the Candidate Evaluator module identifies that the hint is not
valid for the answer, we assign a score of zero, indicating that the hint is
not useful for finding the answer. However, if the hint is valid for the
answer, we compute the score based on the results for other candidate answers.
A score of zero denotes that the hint does not reduce the number of candidate
answers, whereas a score of one indicates that the hint successfully
eliminates all candidate answers except for the correct one.

(1) $Score_{con}=\left\\{\begin{array}[]{ll}0&:\ EA_{valid}=0\\\
1-\frac{(\sum_{cand}cand_{valid})-1}{\left|candidates\right|}&:\
EA_{valid}=1\\\ \end{array}\right.$

In the given equation, cand and $cand_{valid}$ denote the candidate answer and
the outcome of the Candidate Generator module for it, respectively, and
$|candidates|$ represents the total number of candidate answers generated by
the Candidate Generator module. $EA_{valid}$ equals one if the ground-truth
answer is predicted as a valid candidate answer; otherwise, it is zero.

###  4.2. Familiarity Quality Attribute

We aim to measure the familiarity of information expressed in hints. Here, we
use the popularity of the mentioned entities as a proxy for familiarity. The
presence of popular entities can assist users in locating the answer more
effortlessly, while less popular entities in a hint may potentially lead the
user astray in finding the correct answer (Piao, 2021).

The method involves three steps: Named Entity Recognition (NER), Wikipedia
Page Views Extractor, and Normalization. These steps are illustrated in Figure
5.

Figure 5. Familiarity Evaluator: The Named Entity Recognizer identifies named
entities from the hint. Subsequently, the number of views for the Wikipedia
page associated with each entity is extracted. Finally, the view count for
each entity undergoes a normalization process.

We utilize the spaCy library111111https://www.spacy.io/ for extracting named
entities from the hints. Subsequently, we leverage the Pageview API121212We
use the pageview-api library available at
https://github.com/Commonists/pageview-api to retrieve the number of views for
the Wikipedia pages associated with the extracted entities, covering the
period from January 1, 2015, to December 31, 2023, and we compute the number
of views for each month. Due to the high variability in the number of page
views, we normalize them to a range between zero and one.

It is worth noting that some entities exhibit extremely high (or extremely
low) numbers of views131313For example, the entity India garners a total of
2,685,795 page views per month.. When attempting to normalize the number of
page views, such entities can render the normalization values unreliable due
to their unusual number of views. To address this issue, we collect view
counts for approximately 50,000 different entities. Subsequently, we utilize
the Interquartile Range (IQR) method to identify outliers in the view counts.
Defining the IQR as the difference between the $Q_{3}$ (the 75th percentile)
and $Q_{1}$ (the 25th percentile), data points falling below
$min=Q_{1}-1.5\times IQR$ or exceeding $max=Q_{3}+1.5\times IQR$ are
classified as outliers (Whaley III, 2005). Following this classification, we
truncate values exceeding $max$ to $max$ and values falling below $min$ to
$min$. By implementing this approach, the normalization process yields more
reliable results. Finally, we employ linear scaling (Han et al., 2022) to
normalize the number of views, bringing them into a range between 0 and 1.

Figure 6. The distribution of Train, Validation, and Test set.

##  5\. Experiments and Results

In this section, we first explore dataset features, including statistics,
difficulty, and quality. Subsequently, we assess our automatic evaluation
method, and finally, we investigate the performance of different models on the
TriviaHG dataset using the proposed automatic evaluation method.

Table 2. Statistics of the TriviaHG dataset | Training | Validation | Test
---|---|---|---
Number of questions | 14,645 | 1,000 | 1,000
Number of hints | 140,973 | 9,638 | 9,619
Avg. question length (words) | 14.18 | 14.08 | 13.95
Avg. hint length (words) | 14.98 | 15.07 | 15.14
Avg. #hints / question | 9.62 | 9.63 | 9.61
Avg. #entities / question | 1.35 | 1.40 | 1.35
Avg. #entities / hint | 0.96 | 1.00 | 0.98
Avg. #sources / question | 6.27 | 6.17 | 6.71

###  5.1. Data Analysis

####  5.1.1. Dataset Statistics

Employing the framework shown in Figure 2, we generated a dataset comprising
16,645 questions and 160,230 hints extracted from the TriviaQA (Joshi et al.,
2017) dataset. The attributes associated with questions are detailed in Table
1. TriviaHG is randomly divided into a training set, consisting of 140,973
hints (14,645 questions), a validation set with 9,638 hints (1,000 questions),
and a test set comprising 9,619 hints (1,000 questions). Table 2 provides an
overview of the dataset statistics. The numbers indicate that the
characteristics of the sub-datasets bear a close resemblance to each other.

We also conducted an analysis of sub-datasets categorized by the coarse-
grained class of the question type. Illustrated in Figure 6, the sub-datasets
exhibit a close distribution, with the highest number of questions focused on
HUMAN, followed by ENTITY, LOCATION, and lastly, OTHER141414The OTHER category
encompasses questions related to numbers, dates, etc..

####  5.1.2. Difficulty

We conduct a comprehensive examination of the dataset concerning the
difficulty levels of both questions and answers. To evaluate the difficulty of
questions, we utilize the DPR method (Karpukhin et al., 2020) with the
assistance of the Pyserini toolkit (Lin et al., 2021) to retrieve 500 relevant
passages from the English Wikipedia that correspond to the questions. If the
relevance of passages to a question is below $33\%$, we classify it as hard;
if it is below $66\%$, we classify it as medium; and if it is more than
$66\%$, we consider it easy. Regarding the difficulty assessment for answers,
we adopt the methodology outlined in Section 4.2 to quantify and normalize the
popularity of named entities of answers. If the popularity score exceeds 0.66,
we categorize an answer as easy; if it falls within the range of 0.33 to 0.66,
we label it as medium; and if the popularity is less than 0.33, we designate
it as hard. Figure 7 shows the distribution of questions and answers based on
the difficulty.

Figure 7. The Left figure illustrates question difficulty, while the Right
figure presents answer difficulty.

###  5.2. Human Evaluation

We engage human evaluators to assess the quality of the generated hints. To
accomplish this, we randomly selected 100 questions from the test set,
distributed as 25 questions from each coarse-grained class of question types
(Entity, Human, Location, and Other), consisting of 9 labeled as hard, 8
labeled as medium, and 8 labeled as easy. In addition to the TriviaHG hints,
we instruct LLaMA151515Please note that we employ LLaMA version 2 for this
study. 7b (Touvron et al., 2023) vanilla to generate $10$ hints for each
question161616We utilize the identical prompt as in Bing Chat AI for
generating hints.. Furthermore, we finetune another LLaMA 7b instance on the
training set of TrivisHG and then prompt it to generate $10$ hints for each
question171717We perform model finetuning using the API functions available on
AnyScale.com. Subsequently, we employ the hint filtering stage (Section 3.2.2)
to refine the generated hints. At last, we obtain a total of 2,741 hints,
comprising 950 hints from the TriviaHG dataset, 901 hints from the finetuned
LLaMA 7b, and 890 hints from the LLaMA 7b vanilla. These data undergo
evaluation in two distinct phases:

In the first phase, we ask $10$ human evaluators to evaluate the generated
hints, providing ratings for attributes such as relevance, readability,
ambiguity, convergence, and familiarity. Ratings are assigned on a scale of 1
to 5, where 1 denotes the lowest and 5 signifies the highest. They are also
asked to search for each hint on both Google and Bing search engines, quickly
examine the result page within a 5-second timeframe, and then determine and
declare whether they could find the answer on the search results page. We
request this action as it also can signal the quality of the hints. The reason
is that it is plausible that a hint is still effective, while the knowledge of
the annotators might be insufficient to answer the question utilizing the
hint. The results of the human evaluation are summarized in Table 3,
displaying average scores for each quality attribute. The relevance analysis
reveals that hints generated by Bing Chat AI and the finetuned LLaMA 7b are
relevant to the questions, while those from LLaMA 7b vanilla lack in terms of
relevance. Additionally, the assessments of readability and ambiguity indicate
that the generated hints are clear and comprehensible. The analysis of
convergence and familiarity indicates that Bing Chat AI exhibits higher
convergence and familiarity than other methods, while the finetuned LLaMA 7b
demonstrates superior quality compared to the LLaMA 7b vanilla. Furthermore,
Table 4 provides statistics on hints that annotators could use to find answers
to the respective questions through searches on Google and Bing search
engines.

Table 3. The human evaluation results for quality attributes.

Method | Relevance | Readability | Ambiguity | Convergence | Familiarity
---|---|---|---|---|---
Bing Chat AI | 4.09 | 4.67 | 1.51 | 2.23 | 2.47
LLaMA - Finetuned | 4.01 | 4.7 | 1.56 | 2.20 | 2.41
LLaMA - Vanilla | 3.64 | 4.47 | 1.87 | 2.12 | 2.02

Table 4. The count of hints that accurately reveal the correct answer when evaluators search for them on search engines. Method | Num. of Hints | Google | Bing
---|---|---|---
Bing Chat AI | 950 | 734 (77%) | 707 (74%)
LLaMA - Finetuned | 901 | 372 (41%) | 353 (39%)
LLaMA - Vanilla | 890 | 228 (26%) | 234 (26%)
Figure 8. The statistics of human evaluation. L7 - Va refers to LLaMA 7b
Vanilla and L7 - FT refers to LLaMA 7b Finetuned.

In the second phase, six distinct individuals, entirely separate from those
involved in the first phase, are tasked with answering the questions using the
provided hints. We randomly selected hints for each question from Bing Chat
AI, LLaMA 7b vanilla, and finetuned LLaMA 7b. The procedure is as follows:

  1. (1)

The subjects are asked to answer the question without employing any hints. If
they can answer the question, they can go to the next question.

  2. (2)

If they cannot answer the question, the annotators are asked to read hints one
by one until they come up with the answer. If their answer is correct, they
can move to the next question.

  3. (3)

If participants cannot answer the question by reading all of the hints of the
question, they can leave the question.

Figure 8 provides an overview of the questions successfully answered by
participants. The data indicates the high effectiveness of hints, as
participants were able to answer only 19 questions before consulting hints,
whereas after reviewing the hints, they could successfully answer 67
questions. The results from human evaluations underscore the superior efficacy
of Bing Chat AI’s hints in aiding users in formulating answers compared to
other methods. Our findings demonstrate that finetuned LLaMA 7b hints offer
more value than LLaMA 7b vanilla hints, suggesting that leveraging the
TriviaHG dataset for LLM finetuning contributes to the generation of superior
hints. It is noteworthy that only $36\%$ of questions with hard answers were
answerable, while medium and easy answers boasted success rates of $78\%$ and
$96\%$, respectively. This observation emphasizes the increased challenge in
generating hints for questions with difficult answers compared to their easier
counterparts. Moreover, Figure 8 highlights that $64\%$ of the questions could
be answered using Bing Chat AI hints, while the respective figures for the
finetuned LLaMA 7b and LLaMA 7b vanilla are $48\%$ and $31\%$. It demonstrates
the quality disparity between Bing Chat AI hints and the ones by the finetuned
and vanilla LLaMA 7b.

###  5.3. Automatic Evaluation Method

In this section, we assess the automatic evaluation method by comparing the
human-provided scores with the outputs generated by our method for the
convergence and familiarity quality attributes.

Figure 9. The Pearson correlation coefficient between human assessments and
generated values for the convergence quality. L7, L13, L70 refers to LLaMA 7b,
13b, 70b, respectively.

####  5.3.1. Convergence Quality Attribute

As elaborated in Section 4.1, we leverage LLMs as the kernel of the automatic
evaluation method to assess the convergence quality attribute of a hint. To
this end, we employ LLaMA 7b, LLaMA 13b, LLaMA 70b, GPT 3.5 (Brown et al.,
2020b) models, as well as the versions of LLaMA 7b, LLaMA 13b, and LLaMA 70b
finetuned on the training set of TriviaHG to examine the accuracy of the
evaluation method across different kernels. In addition, as discussed in
Section 4.1.3, the computed value of the convergence quality attribute is
influenced by the number of candidate answers generated. To determine the
optimal model as the kernel and the optimal number of candidate answers, we
calculate the Pearson correlation coefficient (Han et al., 2022) between human
assessments and automatically generated values for the convergence quality
attribute across 2,791 hints. Figure 9 indicates the value of the Pearson
correlation coefficient for each number of candidate answers for the various
kernels. Based on the observations, we can infer that the evaluation method’s
alignment with human assessments for the convergence quality attribute is
closely linked to the robustness of the LLM used as the kernel. Notably, the
highest correlation is observed with GPT 3.5, followed by finetuned LLaMA 70b
and LLaMA 70b vanilla. Furthermore, the findings suggest that finetuned LLaMA
on the training set yields evaluations more akin to human judgments compared
to LLaMA vanilla models. Moreover, the values indicate that generating $11$
candidate answers yields the closest correlation to human results.

####  5.3.2. Familiarity Quality Attribute

As detailed in Section 4.2, we employ the normalized number of page views of
the Wikipedia page corresponding to entities as the measure of familiarity
quality attribute for hints. For each hint, we explore three distinct methods,
including the minimum, average, and maximum. In the minimum (maximum) method,
we solely consider the normalized value of an entity that has the minimum
(maximum) number of views. Conversely, in the average method, we compute the
average of normalized values across all the entities mentioned in the hint. To
determine the best approach, we compute the Pearson correlation coefficient
and Mean squared error (Han et al., 2022) between human assessments and the
automatically generated values for the familiarity quality attribute across
2,791 hints. Table 5 indicates the value of the Pearson correlation
coefficient and Mean squared error for all the above approaches. The findings
suggest that calculating the average of normalized page view values across
entities exhibits the highest correlation with human evaluations.

Table 5. The Pearson correlation coefficient and MSE between human assessments and automatically generated values for the familiarity quality attribute across 2,791 hints. Method | Minimum | Average | Maximum
---|---|---|---
Pearson correlation coefficient | 0.63 | 0.66 | 0.63
Mean squared error | 0.16 | 0.13 | 0.14

###  5.4. Model Performance

Finally, we utilize LLaMA, GPT 3.5, WizardLM 70b, Gemini, GPT 4, and the
finetuned versions of LLaMA on the training set of TriviaHG to generate hints,
followed by evaluation using the automatic evaluation method. For evaluating
the convergence quality attribute, we generate $11$ candidate answers and
utilize finetuned LLaMA 70b as the kernel model181818The decision to choose
the second-best model is driven by the high cost of using GPT 3.5 and its
closed-source character.. Meanwhile, in evaluating the familiarity quality
attribute, we employ the average approach. Table 6 presents the outcomes of
the automatic evaluation method for different LLMs.

The findings suggest that a robust model, characterized by a greater number of
training parameters, produces hints of higher convergence quality. This may be
attributed to its capacity to store more knowledge within its parameters,
resulting in more accurate hints. Furthermore, the results indicate that
finetuned LLaMA models outperform LLaMA vanilla models, demonstrating that
finetuning on TriviaHG enhances the hint generation capabilities of LLMs.
Moreover, comparing the results between Bing Chat AI and GPT 4 underscores the
effectiveness of incorporating retrieved documents alongside stored knowledge.
This is exemplified by Bing Chat AI, which utilizes GPT 4 as its kernel.

Table 6. The convergence and familiarity values across 100 questions for different models. Model | Convergence | Familiarity
---|---|---
LLaMA 7b - Vanilla (Touvron et al., 2023) | 0.307 | 0.833
LLaMA 13b - Vanilla (Touvron et al., 2023) | 0.350 | 0.929
LLaMA 70b - Vanilla (Touvron et al., 2023) | 0.425 | 0.941
LLaMA 7b - Finetuned | 0.400 | 0.890
LLaMA 13b - Finetuned | 0.410 | 0.881
LLaMA 70b - Finetuned | 0.494 | 0.862
GPT 3.5 (Brown et al., 2020b) | 0.438 | 0.911
WizardLM 70b (Xu et al., 2023) | 0.446 | 0.942
Gemini (Team et al., 2023) | 0.455 | 0.911
GPT 4 (Achiam et al., 2023) | 0.525 | 0.875
Bing Chat AI | 0.540 | 0.946

##  6\. Limitations and Use Cases

Our approach and evaluation method have some limitations:

  * •

In this study, our focus was solely on factoid questions. In particular, the
answers correspond to named entities, each one having its Wikipedia page.

  * •

Our dataset and evaluation method depend on LLMs; thus, the biases and
limitations inherent in these models will be reflected in both our dataset and
evaluation approach.

  * •

We solely assess the model performance based on the convergence and
familiarity quality attributes.

  * •

In the convergence quality evaluation method, we found 11 to be the optimal
number of candidate answers on TriviaHG. However, this number could vary for
other datasets. When applying the evaluation method to other datasets, one
could ideally try to determine the optimal number of candidate answers for the
particular used data.

  * •

The current approach focuses solely on generating hints using LLMs and
assessing their usefulness in assisting humans to answer questions without
explicitly considering the educational values of hints.

Our dataset and evaluation method can be used in different ways:

  * •

The dataset holds the potential for finetuning generative models to produce
high-quality hints for the provided questions. As illustrated in Figure 8 and
Table 6, finetuned models trained on our dataset have demonstrated improved
hint generation compared to vanilla versions.

  * •

The proposed evaluation method for assessing the convergence and familiarity
quality attributes can serve as a tool to evaluate the quality of hints.

  * •

The proposed dataset could be employed for improving QA systems by adding
hints as additional context to prompts similar to Sun et al. (2023).

  * •

The generated hints could be of use in diverse information retrieval tasks.
For instance, they can be employed in retrieval-augmented generation (RAG)
techniques to improve question understanding. Additionally, the hints can be
utilized in query expansion methods. They could also be used in entertainment
contexts, such as in game shows like Jeopardy.

##  7\. Conclusion

Hinting is an important and frequent mechanism in human question answering;
however, it has been largely neglected during the recent advancement of QA
technologies. In this paper, we present a framework for constructing datasets
tailored to hint generation for factoid questions, consisting of two key
modules: 1) Question Sampling Module and 2) Hint Generation Module. The
framework was utilized to generate a dataset comprising 140,973 hints (14,645
questions) in the training set, 9,638 hints (1,000 questions) in the
validation set, and 9,619 hints (1,000 questions) in the test set.

Additionally, an automatic evaluation method is proposed to measure the
convergence and familiarity quality attributes of the hints. Convergence
quality is evaluated using LLMs as a kernel framework, while familiarity
quality is assessed using the number of page views of Wikipedia page entities.
Human assessment of the dataset and evaluation method indicates high-quality
hints and a strong correlation with human results for the evaluation method.
Finally, the performance of various LLMs on the dataset was evaluated based on
convergence and familiarity quality attributes, demonstrating the impact of
the dataset on finetuning.

In the future, our plan includes generating hints based on user feedback,
integrating the subjective aspect of hints, and evaluating hints based on
additional quality attributes.

## References

  * (1)
  * Abdallah and Jatowt (2023) Abdelrahman Abdallah and Adam Jatowt. 2023.  Generator-retriever-generator: A novel approach to open-domain question answering.  _arXiv preprint arXiv:2307.11278_ (2023).
  * Abdel-Nabi et al. (2023) Heba Abdel-Nabi, Arafat Awajan, and Mostafa Z. Ali. 2023\.  Deep learning-based question answering: a survey.  _Knowledge and Information Systems_ 65, 4 (01 Apr 2023), 1399–1485.  https://doi.org/10.1007/s10115-022-01783-5
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Arnab (2017) Raghunath Arnab. 2017\.  Chapter 7 - Stratified Sampling.  In _Survey Sampling Theory and Applications_ , Raghunath Arnab (Ed.). Academic Press, 213–256.  https://doi.org/10.1016/B978-0-12-811848-1.00007-8
  * Bandura (2013) Albert Bandura. 2013\.  The role of self-efficacy in goal-based motivation.  _New developments in goal setting and task performance_ (2013), 147–157.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020\.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ (2020).
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013\.  Semantic Parsing on Freebase from Question-Answer Pairs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (Eds.). Association for Computational Linguistics, Seattle, Washington, USA, 1533–1544.  https://aclanthology.org/D13-1160
  * Brown et al. (2020b) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020b.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Brown et al. (2020a) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020a.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Bulian et al. (2022) Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Börschinger, and Tal Schuster. 2022\.  Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 291–305.  https://doi.org/10.18653/v1/2022.emnlp-main.20
  * Chen et al. (2023) Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023\.  Benchmarking large language models in retrieval-augmented generation.  _arXiv preprint arXiv:2309.01431_ (2023).
  * De Jong (1996) H Looren De Jong. 1996\.  Levels: Reduction and elimination in cognitive neuroscience.  _Problems of theoretical psychology_ 6 (1996), 165.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023\.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_ (2023).
  * Gregory Hume and Evens (1996) Allen Rovick Gregory Hume, Joel Michael and Martha Evens. 1996.  Hinting as a Tactic in One-on-One Tutoring.  _Journal of the Learning Sciences_ 5, 1 (1996), 23–47.  https://doi.org/10.1207/s15327809jls0501_2
  * Haldeman et al. (2018) Georgiana Haldeman, Andrew Tjang, Monica Babeş-Vroman, Stephen Bartos, Jay Shah, Danielle Yucht, and Thu D. Nguyen. 2018.  Providing Meaningful Feedback for Autograding of Programming Assignments. In _Proceedings of the 49th ACM Technical Symposium on Computer Science Education_ (Baltimore, Maryland, USA) _(SIGCSE ’18)_. Association for Computing Machinery, New York, NY, USA, 278–283.  https://doi.org/10.1145/3159450.3159502
  * Han et al. (2022) Jiawei Han, Jian Pei, and Hanghang Tong. 2022.  _Data mining: concepts and techniques_.  Morgan kaufmann.
  * Head et al. (2017) Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D’Antoni, and Björn Hartmann. 2017.  Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis. In _Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale_ (Cambridge, Massachusetts, USA) _(L@S ’17)_. Association for Computing Machinery, New York, NY, USA, 89–98.  https://doi.org/10.1145/3051457.3051467
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 874–880.  https://doi.org/10.18653/v1/2021.eacl-main.74
  * Jatowt et al. (2023) Adam Jatowt, Calvin Gehrer, and Michael Färber. 2023\.  Automatic Hint Generation. In _Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval_ (Taipei, Taiwan) _(ICTIR ’23)_. Association for Computing Machinery, New York, NY, USA, 117–123.  https://doi.org/10.1145/3578337.3605119
  * Jin et al. (2012) Wei Jin, Tiffany Barnes, John Stamper, Michael John Eagle, Matthew W. Johnson, and Lorrie Lehmann. 2012.  Program Representation for Automatic Hint Generation for a Data-Driven Novice Programming Tutor. In _Intelligent Tutoring Systems_ , Stefano A. Cerri, William J. Clancey, Giorgos Papadourakis, and Kitty Panourgia (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 304–309.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, Vancouver, Canada, 1601–1611.  https://doi.org/10.18653/v1/P17-1147
  * Jurafsky (2000) Dan Jurafsky. 2000\.  _Speech & language processing_.  Pearson Education India.
  * Kamalloo et al. (2023) Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023.  Evaluating Open-Domain Question Answering in the Era of Large Language Models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 5591–5606.  https://doi.org/10.18653/v1/2023.acl-long.307
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 6769–6781.  https://doi.org/10.18653/v1/2020.emnlp-main.550
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 452–466.  https://doi.org/10.1162/tacl_a_00276
  * Lazar et al. (2017) Timotej Lazar, Martin Možina, and Ivan Bratko. 2017\.  Automatic Extraction of AST Patterns for Debugging Student Programs. In _Artificial Intelligence in Education_ , Elisabeth André, Ryan Baker, Xiangen Hu, Ma. Mercedes T. Rodrigo, and Benedict du Boulay (Eds.). Springer International Publishing, Cham, 162–174.
  * Lee et al. (2018) V.C.S. Lee, Y.T. Yu, C.M. Tang, T.L. Wong, and C.K. Poon. 2018.  ViDA: A virtual debugging advisor for supporting learning in computer programming courses.  _Journal of Computer Assisted Learning_ 34, 3 (2018), 243–258.  https://doi.org/10.1111/jcal.12238 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/jcal.12238
  * Li and Roth (2002) Xin Li and Dan Roth. 2002\.  Learning Question Classifiers. In _COLING 2002: The 19th International Conference on Computational Linguistics_.  https://aclanthology.org/C02-1150
  * Lin et al. (2021) Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021\.  Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (¡conf-loc¿, ¡city¿Virtual Event¡/city¿, ¡country¿Canada¡/country¿, ¡/conf-loc¿) _(SIGIR ’21)_. Association for Computing Machinery, New York, NY, USA, 2356–2362.  https://doi.org/10.1145/3404835.3463238
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Liu (2023) Zefang Liu. 2023\.  SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security.  _arXiv preprint arXiv:2312.15838_ (2023).
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 9802–9822.  https://doi.org/10.18653/v1/2023.acl-long.546
  * Marin et al. (2017) Victor J. Marin, Tobin Pereira, Srinivas Sridharan, and Carlos R. Rivero. 2017.  Automated Personalized Feedback in Introductory Java Programming MOOCs. In _2017 IEEE 33rd International Conference on Data Engineering (ICDE)_. 1259–1270.  https://doi.org/10.1109/ICDE.2017.169
  * Martinc et al. (2021) Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021\.  Supervised and unsupervised neural approaches to text readability.  _Computational Linguistics_ 47, 1 (2021), 141–179.
  * McBroom et al. (2021) Jessica McBroom, Irena Koprinska, and Kalina Yacef. 2021\.  A Survey of Automated Programming Hint Generation: The HINTS Framework.  _ACM Comput. Surv._ 54, 8, Article 172 (oct 2021), 27 pages.  https://doi.org/10.1145/3469885
  * Nassiri and Akhloufi (2023) Khalid Nassiri and Moulay Akhloufi. 2023.  Transformer models used for text-based question answering systems.  _Applied Intelligence_ 53, 9 (01 May 2023), 10602–10635.  https://doi.org/10.1007/s10489-022-04052-8
  * Obermüller et al. (2021) Florian Obermüller, Ute Heuer, and Gordon Fraser. 2021\.  Guiding Next-Step Hint Generation Using Automated Tests. In _Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1_ (Virtual Event, Germany) _(ITiCSE ’21)_. Association for Computing Machinery, New York, NY, USA, 220–226.  https://doi.org/10.1145/3430665.3456344
  * Pal et al. (2022) Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022\.  MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering. In _Proceedings of the Conference on Health, Inference, and Learning_ _(Proceedings of Machine Learning Research, Vol. 174)_ , Gerardo Flores, George H Chen, Tom Pollard, Joyce C Ho, and Tristan Naumann (Eds.). PMLR, 248–260.  https://proceedings.mlr.press/v174/pal22a.html
  * Piao (2021) Guangyuan Piao. 2021\.  Scholarly Text Classification with Sentence BERT and Entity Embeddings. In _Trends and Applications in Knowledge Discovery and Data Mining_ , Manish Gupta and Ganesh Ramakrishnan (Eds.). Springer International Publishing, Cham, 79–87.
  * Piech et al. (2015) Chris Piech, Mehran Sahami, Jonathan Huang, and Leonidas Guibas. 2015.  Autonomously Generating Hints by Inferring Problem Solving Policies. In _Proceedings of the Second (2015) ACM Conference on Learning @ Scale_ (Vancouver, BC, Canada) _(L@S ’15)_. Association for Computing Machinery, New York, NY, USA, 195–204.  https://doi.org/10.1145/2724660.2724668
  * Pranav et al. (2018) Rajpurkar Pranav, Jia Robin, and Liang Percy. 2018\.  Know What You Don’t Know: Unanswerable Questions for SQuAD.  _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ (2018).  https://doi.org/10.18653/v1/p18-2124
  * Prasad et al. (2023) Archiki Prasad, Trung Bui, Seunghyun Yoon, Hanieh Deilamsalehy, Franck Dernoncourt, and Mohit Bansal. 2023.  MeetingQA: Extractive Question-Answering on Meeting Transcripts. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 15000–15025.  https://doi.org/10.18653/v1/2023.acl-long.837
  * Price et al. (2017) Thomas Price, Rui Zhi, and Tiffany Barnes. 2017.  Evaluation of a Data-Driven Feedback Algorithm for Open-Ended Programming.  _International Educational Data Mining Society_ (2017).
  * Price et al. (2019) Thomas W. Price, Yihuan Dong, Rui Zhi, Benjamin Paaßen, Nicholas Lytle, Veronica Cateté, and Tiffany Barnes. 2019.  A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms.  _International Journal of Artificial Intelligence in Education_ 29, 3 (01 Aug 2019), 368–395.  https://doi.org/10.1007/s40593-019-00177-z
  * Ramnath et al. (2020) Sahana Ramnath, Preksha Nema, Deep Sahni, and Mitesh M. Khapra. 2020.  Towards Interpreting BERT for Reading Comprehension Based QA. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 3236–3242.  https://doi.org/10.18653/v1/2020.emnlp-main.261
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.  https://arxiv.org/abs/1908.10084
  * Rivers (2017) Kelly Rivers. 2017\.  Automated Data-Driven Hint Generation for Learning Programming.  (7 2017).  https://doi.org/10.1184/R1/6714911.v1
  * Rivers et al. (2016) Kelly Rivers, Erik Harpstead, and Ken Koedinger. 2016\.  Learning Curve Analysis for Programming: Which Concepts do Students Struggle With?. In _Proceedings of the 2016 ACM Conference on International Computing Education Research_ (Melbourne, VIC, Australia) _(ICER ’16)_. Association for Computing Machinery, New York, NY, USA, 143–151.  https://doi.org/10.1145/2960310.2960333
  * Rogers et al. (2023) Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023\.  QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension.  _ACM Comput. Surv._ 55, 10, Article 197 (feb 2023), 45 pages.  https://doi.org/10.1145/3560260
  * Su et al. (2024) Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024\.  RoFormer: Enhanced transformer with Rotary Position Embedding.  _Neurocomputing_ 568 (2024), 127063.  https://doi.org/10.1016/j.neucom.2023.127063
  * Sun et al. (2023) Hong Sun, Xue Li, Yinchuan Xu, Youkow Homma, Qi Cao, Min Wu, Jian Jiao, and Denis Charles. 2023\.  Autohint: Automatic prompt optimization with hint generation.  _arXiv preprint arXiv:2307.07415_ (2023).
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023\.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ (2023).
  * Tolboom (2023) Ryan Tolboom. 2023\.  Computer Systems Security.
  * Tong et al. (2023) Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, and Jingbo Shang. 2023\.  Eliminating Reasoning via Inferring with Planning: A New Framework to Guide LLMs’ Non-linear Thinking.  _arXiv preprint arXiv:2310.12342_ (2023).
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023\.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ (2023).
  * Usher and Pajares (2006) Ellen L. Usher and Frank Pajares. 2006.  Sources of academic and self-regulatory efficacy beliefs of entering middle school students.  _Contemporary Educational Psychology_ 31, 2 (2006), 125–141.  https://doi.org/10.1016/j.cedpsych.2005.03.002
  * Wang et al. (2022a) Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022a.  ArchivalQA: A Large-scale Benchmark Dataset for Open-Domain Question Answering over Historical News Collections. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (¡conf-loc¿, ¡city¿Madrid¡/city¿, ¡country¿Spain¡/country¿, ¡/conf-loc¿) _(SIGIR ’22)_. Association for Computing Machinery, New York, NY, USA, 3025–3035.  https://doi.org/10.1145/3477495.3531734
  * Wang et al. (2023) Liang Wang, Nan Yang, and Furu Wei. 2023.  Query2doc: Query Expansion with Large Language Models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9414–9423.  https://doi.org/10.18653/v1/2023.emnlp-main.585
  * Wang et al. (2022b) Luqi Wang, Kaiwen Zheng, Liyin Qian, and Sheng Li. 2022b.  A Survey of Extractive Question Answering. In _2022 International Conference on High Performance Big Data and Intelligent Systems (HDIS)_. 147–153.  https://doi.org/10.1109/HDIS56859.2022.9991478
  * Whaley III (2005) Dewey Lonzo Whaley III. 2005\.  _The interquartile range: Theory and estimation_.  Ph. D. Dissertation. East Tennessee State University.
  * Workshop et al. (2022) BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022\.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_ (2022).
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023\.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_ (2023).
  * Yu and Liu (2021) Puning Yu and Yunyi Liu. 2021.  Roberta-based Encoder-decoder Model for Question Answering System. In _2021 International Conference on Intelligent Computing, Automation and Applications (ICAA)_. 344–349.  https://doi.org/10.1109/ICAA53760.2021.00070
  * Yu et al. (2023) Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023.  Generate rather than Retrieve: Large Language Models are Strong Context Generators. In _The Eleventh International Conference on Learning Representations_.  https://openreview.net/forum?id=fB0hRu9GZUS
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big Bird: Transformers for Longer Sequences. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 17283–17297.  https://proceedings.neurips.cc/paper_files/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  BERTScore: Evaluating Text Generation with BERT. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=SkeHuCVFDr
  * Zhong et al. (2020) Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020\.  JEC-QA: A Legal-Domain Question Answering Dataset.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 05 (Apr. 2020), 9701–9708.  https://doi.org/10.1609/aaai.v34i05.6519