DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels
Zhe Xu Jiasheng Ye Xiangyang Liu Tianxiang Sun Xiaoran Liu Qipeng Guo Linlin Li Qun Liu Xuanjing Huang Xipeng Qiu

UAV-Mounted Movable Antenna: Joint Optimization of UAV Placement and Antenna Configuration
    Xiao-Wei Tang, Yunmei Shi, Yi Huang, and Qingqing Wu
Xiao-Wei Tang, Yunmei Shi, Yi Huang ({xwtang, ymshi, and huangyi718b}@tongji.edu.cn) are with the Department of Information and Communication Engineering, Tongji University, Shanghai, China.
Qingqing Wu (qingqingwu@sjtu.edu.cn) is with the Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China.

    3 September 2024
======================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 





With the rapid advancement of Large Language Models (LLMs), long-context information understanding and processing have become a hot topic in academia and industry. However, benchmarks for evaluating the ability of LLMs to handle long-context information do not seem to have kept pace with the development of LLMs. Despite the emergence of various long-context evaluation benchmarks, the types of capability assessed are still limited, without new capability dimensions. In this paper, we introduce DetectiveQA, a narrative reasoning benchmark featured with an average context length of over 100K tokens. DetectiveQA focuses on evaluating the long-context reasoning ability of LLMs, which not only requires a full understanding of context but also requires extracting important evidences from the context and reasoning according to extracted evidences to answer the given questions. This is a new dimension of capability evaluation, which is more in line with the current intelligence level of LLMs. We use detective novels as data sources, which naturally have various reasoning elements. Finally, we manually annotated 600 questions in Chinese and then also provided an English edition of the context information and questions. We evaluate many long-context LLMs on DetectiveQA, including commercial and open-sourced models,  and the results indicate that existing long-context LLMs still require significant advancements to effectively process true long-context dependency questions.  





§ INTRODUCTION








The development of Large Language Models (LLMs) <cit.> has had a remarkable surge in recent years. The ability to long-context understanding and processing is important in the development of LLMs <cit.>. This capability is essential for some basic tasks that require a deep understanding of lengthy documents, such as information extraction from long documents, summarization for long documents, or translation for long documents. With this capability, LLMs can be applied to a wider range of scenarios (e.g., legal cases, academic field, or financial field). In addition to the rapid application of LLMs to these basic tasks, the development of LLMs has also allowed researchers to see the dawn of AGI (artificial general intelligence). Both academia and industry are exploring the application of LLMs in broader fields, such as advanced intelligent agents, robotics, and so on <cit.>. This kind of application requires LLMs to have a stronger understanding and reasoning ability over the long documents. This type of reasoning ability is the key to inferring the next action from the context information, rather than the basic capability of extracting the action directly from the context information. Just like humans, they can deduce from all their previous experiences what actions they are going to perform next. This is a more advanced capability, a new dimension of capability that needs to be evaluated for rapidly evolving LLMs.

Many long-context evaluation benchmarks for long-context LLMs have emerged with the development of long-context LLMs. To quickly fill in the gaps in long-context evaluation, some new benchmarks choose to integrate and transform large sets of existing datasets <cit.>. However, the context length of these datasets is basically less than 20K, which makes it difficult to meet the evaluation requirements of the current long-context LLMs. Besides, to extend the length of context, some benchmarks carry out data source re-selection and new question annotation <cit.>. However, these newly annotated datasets tend to focus on a few basic evaluation dimensions of the old benchmarks, such as information retrieval, summarization, code completion, and multi-hop reasoning. Although these benchmarks cover many application scenarios of long-context LLMs, it is difficult to measure the performance of LLMs in more advanced intelligent applications, such as advanced intelligent agents or robotics. 

To fill this gap, we introduce DetectiveQA, a novel benchmark that evaluates the performance of long-context LLMs on long-context questions. Unlike existing benchmarks, DetectiveQA takes care of both longer context information and longer context dependencies. Moreover, it introduces the assessment of narrative reasoning capability, a new dimension of assessment. We choose  orthodox school novels as our data source, which not only have long context lengths, but also have complex plots and character relationships. In order to construct DetectiveQA, we have deeply analysed and annotated these novels, extracted a large number of long contextual questions, and provided detailed answers. These questions cover a variety of genres, including character relationships, plot development, motivation analysis, etc., aiming to comprehensively assess the capability of long context language models in narrative reasoning.

The proposal of DetectiveQA aims to provide a new evaluation tool for the research and application of long context language models, to help researchers gain a deeper understanding of the performance and limitations of long context language models, as well as to support the advancement of long context language models in the field of more advanced intelligent applications.

In summary, our contributions are threefold.

    
  * We present the first long context narrative reasoning dataset, which helps to better evaluate the model's reasoning ability for complex problems in narrative contexts.
    
  * We have designed rich evaluation metrics that take into account data contamination issues and decoupling of long text capabilities, which helps to better analyse the performance of the model.
    
  * We have extensively evaluated the long-context reasoning capabilities of current large-scale language models, and have clarified the challenges that the key capability of narrative reasoning faces in the development of current large-scale language models.





§ RELATED WORK




  
Long-Context Evaluation Benchmark
Regarding datasets, NarrativeQA <cit.>, QuALITY <cit.>, and TriviaQA <cit.> provide datasets for document information retrieval, a longer dataset was constructed by providing document information. These datasets are useful for evaluating the information retrieval capabilities of large language models.QuALITY <cit.>'s evaluation criterion provides a unique evaluation perspective by using multiple-choice accuracy instead of BLEU or ROUGE-L. Meanwhile, HotpotQA <cit.> evaluates the model's multi-hop inference ability by using multiple related progressively inferenceable facts, while ELI5 <cit.> constructs datasets with longer responses by collecting quizzes and answers that are understandable to children as young as 5 years old and in which the model is required to respond according to the corresponding attempts to make inferences.
There are many benchmarks that also focus on measuring long text, Long Range Arena <cit.> designed six classification tasks on longer input, CAB <cit.> designed seven tasks for a more comprehensive measurement of skills. SCROLLS <cit.> and its extension ZEROSCROLLS <cit.> include documents from a variety of domains and also propose a variety of tasks, including query-based summarization, multi-hop problem analysis, sentiment aggregation, and sorting of book chapter summaries. LongBench <cit.> covers multilingual measures and focuses mainly on long text comprehension in the large language model. L-EVAL <cit.>  contains both open and closed tasks and comes in at 30K in length.
The datasets and Benchmarks used in these evaluations were relatively long at the time and placed some demands on the models' inference capabilities. However, the input lengths were still relatively short, no more than 50K.
To identify measurement issues in lengthy texts, datasets such as Needle in a Haystack <cit.> and the bABILONG <cit.> dataset have reached orders of magnitude of 100K pages. Needle in a Haystack <cit.> evaluates model performance at various lengths and insertion depths, while bABILONG <cit.> creates a multi-hop inference problem by inserting more facts into the data in the presence of very long contextualized inputs.
Meanwhile, InfiniteBench <cit.> has focused on the need for longer text reviews, building a set of over 100k reviews aimed at testing the model's five key capabilities for longer text: retrieval, maths, code, QA, and summaries.
 

Nowadays, the need for input support for very long text and model inference capabilities is becoming more prominent due to the development of large language models. In the past, datasets often lacked in terms of length, and inference was more of an information extraction task, where the answer would appear in the relevant document.
 
Therefore, we have constructed the only dataset with an input length of 100K that requires complex narrative reasoning to obtain answers that are not explicitly stated in the text. Additionally, we propose a more comprehensive evaluation criterion to assess the model's narrative reasoning ability.








§ DETECTIVEQA


In this section, we introduce DetectiveQA, a benchmark dataset to test the long-context reasoning ability of language models.




 §.§ Data Sources


A representative data to study language models' ability to handle long contexts is books, among which detective novels are a category that contains intensive reasoning-related content.
Therefore, we consider detective novels as promising candidates to be data sources of our benchmark.
Nevertheless, we find that a large volume of detective novels take the attractiveness of storytelling in the first place at the sacrifice of the strictness of reasoning processes.
Fortunately, we find a group of detective novels categorized as orthodox school <cit.>.
These novels are dedicated to entertaining readers keen on solving puzzles by ensuring that the reader has the same number of evidences as the detective in the novel, being ideal data sources that satisfy our need for rigorous reasoning.
Therefore, we collect orthodox detective novels as sources of long context and use questions related to the puzzles in the novels to test the language models.

Other consideration on data sources are their lengths and languages. 
A smooth gradient of difficulty helps to differentiate models at varying levels of proficiency.
Therefore, we collect orthodox detective novels with lengths ranging from 100K to 250K words.
Additionally, we only collect the Chinese and English versions given the language background of the researchers and data annotators.









 §.§ Desiderata of Data Annotation

Our data are largely question-answer pairs, as shown in Figure <ref>.
To ensure the efficacy of our data in evaluating the long-context reasoning ability of language models, we make several design decisions on the annotations on both questions and answers.



  
Questions. 
Each question is composed of a long context from the detective novels we collect and a multiple-choice question about the context.
For the context, we truncate the novel till the paragraph that writes the answer to the question in order to avoid answer leakage.
And we allow multiple questions for a novel to improve the utilization of the books.
For the questions, we design them as multiple-choice questions to ease extracting answers from model outputs, similar to the practice in many prominent benchmarks for large language models <cit.>.
We also require the question to center around the detective's reasoning about the cases.
In this way, we exclude questions that are too trivial to require understanding or extracting evidences from a long context[Here is an example of overly trivial questions: a novel mentions that a character is six years old when her sister is born, and then the question is how many years she is older than her sister.]. 






  
Answers. 
We require the answers to contain reference solutions with decomposed steps.
This helps fine-grained evaluation of the stepwise correctness of the model outputs.
Typically, the reasoning steps are either evidences drawn from the context or inferences from these evidences.
The two kinds of steps correspond to the ability of language models to understand long contexts and perform reasoning, respectively.
Therefore, we further differentiate them in our data annotations to facilitate disentangled evaluations of the two aspects. 


An exemplary data sample is in Figure <ref>. In summary, a data entry contains the following items:

    
    
  * A multiple-choice question with four candidate options, attaching a long novel as its context. 
    
  * The answer option and the reasoning process in the form of a list containing evidences and inferences.
    
  * Evidence position corresponding to reasoning. Each indicates the passage in which each evidence appears in the text, and if the corresponding position is an inference rather than a evidence then it is labeled -1. 
    
  * Answer position representing where the answer appears in the novel. It is used to truncate the text to ask the question.




 §.§ Annotation Procedure



  
Human Annotations. A naive approach to annotating the data is to employ human labor.
We can hire workers to read the novels, write down the questions, and reference answers. 
However, such a process is intolerably cumbersome due to the requirement of reading long novels <cit.>. 
As recorded It takes a median of around 3.5 hours for our annotators to complete reading a 100K-long novel.

There is a relatively large drain on annotators' time and mental energy, making it difficult to scale up the size of the datasets.
As a consequence, we seek an alternative approach to enable efficient data annotations.



  
Annotation with AI Assistance.
Our solution is to leverage existing language models that have the eminent long-context capability to assist the human annotator.
Our key insight is that, with the full detective novels, finding the reasoning questions as well as their answers can be treated as an information extraction problem, a much simpler task that large language models have achieved plausible performance <cit.>. 
Hereby, we design a workflow to decompose the data annotation procedure into steps of information extraction tasks and employ Claude 2, a leading long-context large language model, for assistance.

    
  * First, we enter the full novel and use the model the extract inferences drawn by the detectives in the novels. This forms the reasoning chain in our data. To help human annotators check the extraction later, we prepend indices to the paragraphs of the novel and require the model to output where the drawn inferences are located.
    
  * Then, with the extracted inferences, we require the model to seek the positions where the evidences mentioned in the inferences lie.
    
  * Finally, we use models to synthesize questions for each of the extracted reasoning chains.

Given the 100k limit of Claude 2, for novels exceeding the lengths, we decompose them into chunks.
We draw reasoning chains (the first step) and ask questions (the third step) in each chunk, respectively.
For gathering the mentioned evidences (the second step), we query all the chunks for each reasoning chain.
After these model calls, human annotators only need to make certain revisions by checking the model outputs with the related paragraphs[To help annotators gain a coherent understanding of the novel content, which we find helpful for their efficiency in checking the AI annotations, we also use the models to provide summarization of the novels.], without reading the full novels and thinking about the reasoning progress.Due to human calibration,we  ensuring the precision and rationality of the annotation. Although human annotators are still required to proofread the content, this somewhat mitigates the larger overhead of purely manual annotation.






 §.§ Statistics

In total, DetectiveQA contains 1200 question-answer pairs, among which 308 are from human annotations and 892 are annotated with the assistance of AI.
We detailed the statistics of the data samples as follows.




Table <ref> presents data statistics for part of novels annotated through both human annotation and AI-assisted annotation. While the AI-assisted annotations increased in question length, the manual annotations produced more detailed evidences and reasoning sections and slightly less coverage than the AI-assisted annotations. Overall, the difference in quality between manual and AI-assisted annotations was minimal, suggesting that the use of AI-assisted manual annotation is feasible.



  
Context lengths.

We show the distribution of context lengths in Figure <ref>.
The length of our queries ranges from 4K to over 250K words, with an average of 96K words.
The average length approaches the supported context length of most competent large language models:100k words.

Such a scale of context length makes our approach sufficient to cover the length of the context window provided by most models, thus providing a sufficiently long measurement scheme for long text capabilities.



  
Context coverage. 

We introduce a coverage factor to quantify the “global” nature of a question. Formally, the coverage factor is defined as the length of the context from the earliest evidence to the answer location as a percentage of the total contextual input length.
Depicted in Figure <ref>, our question coverage can be broadly categorized into three bands: 10 percent, 50 percent, and 100 percent. 
The substantial contextual coverage poses a significant challenge to the model's comprehension and information-gathering abilities from lengthy articles, rendering our dataset an effective assessment of the model's proficiency in reading and comprehending extensive texts.








  
Statistics of the reasoning.
Table <ref> reveals that answers in our dataset contain substantial information in the narrative reasoning . Additionally, the labeled questions exhibit a high demand for a broad span of evidences, necessitating the exploration of a relatively lengthy contextual span in the text to derive the answer.











§ EXPERIMENTS


In this section, we evaluate prevailing large language models supporting long context on DetectiveQA to benchmark their capabilities in long-context narrative reasoning. 





 §.§ Experimental Setups







  §.§.§ Experimental Settings

To explore the key elements for long-context reasoning, namely understanding long documents, extracting cue information, and reasoning about actions or responses based on cue information, our experiments employ three distinct settings for both human manual annotation and AI-assisted annotation.



  
Question+Context.
This fundamental setup includes a multiple-choice question for the model to answer, requiring the model to provide its response and reasoning process for the novel content until the answer is found in the text. This tests the model's abilities in long text comprehension, cue information extraction, and reasoning simultaneously.



  
Question-Only.
In this setup, we investigate whether the model, during pre-training, has been exposed to the corresponding novel content. We present the model with a question-only query, providing only the name and author of the novel along with single-choice questions. The model is then expected to output both the answer and the corresponding reasoning process.



  
Question+evidence.
In this configuration, we input the cue part of the human annotation into the model along with a multiple-choice problem. This cue section is akin to the result of a gold search of the article for the given problem. This setting isolates the model's ability to comprehend long articles and extract information, essentially testing the model's reasoning ability alone.



  §.§.§ Metrics

Our methodology for the evaluation contains two aspects of evaluation metrics.



  
Answer accuracy.
Similar to previous evaluation based on multiple-choice questions <cit.>, we provide the model a question with four annotated options and require the model to output a letter corresponding to the selected option.
At this point, we calculate the percentage of correctly answered questions as the score.[Since we are using data annotated on detective novels whose content the model may have seen during pre-training, resulting in high model scores, we discuss the influence of potential data contamination in Appendix <ref>.]



  
Reasoning metric.
To support the reliability of the model's answer decisively, it is imperative that the output not only provides an answer but also includes an explanation supporting that answer.
To this end, we examine how many of the annotated evidences are included in the model's output reasoning process, and then score the question based on the percentage of annotated evidences out of the total number of evidences. The average score across all questions represents the model's reasoning evaluation score on the dataset.
For this containment relationship we will use GPT4 review, ask GPT4 to give the contained evidences, and count the number.The specific prompt we used can be found in the Appendix <ref>.



























  



  §.§.§ Models







We conducted experiments using both open-source and closed-source models, focusing on selecting LLMs that support long text inputs that are capable of supporting input lengths of 100K or more.

Our choice of models prioritises dialogue-enabled LLMs that can process at least 100K long texts[We also did quite a lot experiments on the model with input limmited to 32K or less, more results can be found in the Appendix<ref>.] in order to extract meaningful information from the text. Our selection covers two broad categories: closed-source models, including GPT-4, Claude3, and Kimi, which are known for their robustness and long text support; and open-source models, such as chatGLM3-6B <cit.>, and IntermLM2-7B <cit.>.Model-specific information will be displayed in the Table <ref>.




 §.§ Main Results



















We present the final experimental results in Table <ref>. It can be seen that the current closed-source models generally score higher than the open-source models. It can also be seen that the long text review of narrative reasoning generally has room for improvement for the current models.
Secondly,by comparing the model's scores in the question-only setting with those in the Question + Context setting, we can measure the degree of data contamination by analyzing the model's win rate for responses. Based on the data in Table <ref>, most models have win rates of over 60% or higher, suggesting that data contamination is not a significant problem for these models.




 §.§ Analysis





We then did some analytical studies on the data.








  
GPT4 rubric reasoning is valid
Validity by manually evaluating the 100 reviews output from GPT4, see this task as a judgment question of whether the evidences are contained or not, to classify the evidences into two types of contained/not contained, and finally for all the results to calculate the Kappa coefficient, this coefficient and judgment both have an accuracy of 92% or more, so this should be valid. 




  
The dataset's problem is Challenging
Models such as InternLM2 for finding a needle in a haystack full of pairs and chatGLM3 perform well on infinite-bench retrieval tasks but still fall short of the leading long-document models (GPT4, Kimi, etc.) on our dataset.



  
Decoupling long text capabilities
By analysing the performance of the model in the given context and with the given cues we can analyse the following Table <ref>, where both correct refers to the percentage of all questions answered correctly in both the question+context setting and the question+evidence setting, and only with context refers to the percentage of all questions answered correctly in the question+context setting but incorrectly in the question+evidence setting, and only with evidence refers to the percentage of questions answered incorrectly in the question+context setting but correctly in the question+evidence setting.
Models with high scores in both correct indicate a strong ability to combine both narrative reasoning and long text comprehension. If BOTH CORRECT is not high and correct only in quesiont+evidence is high, the model is still deficient in long text comprehension but strong in narrative reasoning.



























We can see that in most of the models, the accuracy of question+evidence settings is much higher, and we will make some case study in the Appendix <ref> for the cases where giving the context can be done correctly, but giving the evidence can't be done correctly.





§ CONCLUSION

We introduced DetectiveQA to test the models' ability to reason narratively over long contexts, the first benchmark for narrative reasoning with an average context length of 100k. We challenged the models' ability to reason over long texts as well as narrative reasoning using detective novels, the real-world texts. For each model, our test gives two scores (answer accuracy and reasoning score) in three settings. With a rich experimental setup, we can deeply analyze the performance of the model and find that the current model still faces challenges in long text comprehension, information extraction and narrative reasoning.
We hope that our dataset will facilitate future improvements in model reasoning ability, leading to more robust AI applications and the highest machine intelligence.




§ LIMITATIONS

Our dataset only serves as an evaluation benchmark on long-context reasoning ability, while how to improve the model capability remains an open question. Meanwhile, our benchmark contains only data from detective novels and mainly serves narrative reasoning. More diverse scenarios can be included in the future. 



§ ETHICS STATEMENT

We are committed to ensuring that DetectiveQA is used only for academic and scientific purposes, and therefore we have rigorously copyright-checked all of the reasoning novels used in Detective's annotations to ensure that the individual novels are not designed to create copyright problems in non-commercial areas. Through these screening tools, we aim to respect the principle of ‘fair use’ under copyright protection and ensure that our project navigates within legal and ethical boundaries in a responsible manner.








§ PROMPT TEMPLATE


In our Reasoning Metric test, we mentioned the use of GPT4 for the number of contained leads, and we used the following prompt template <ref>

 
This template asks enough questions to get usable responses without adding additional samples for a few shots to help answer.



























































§ IMPLEMENTATION DETAILS

The actual experiments also require the processing of the inputs and outputs, such as text truncation, answer alignment, or a special two-step answering method to obtain the final answer. Below we describe the processing details in the following three experiments.



  
Text Truncation

We adopt a tail truncation approach for text handling in each model, wherein we truncate input questions from the end. Differing from the approach in InfiniteBench <cit.>, our dataset's nature leads us to believe that evidences pertinent to inference problems are more likely to be found near the end of the text rather than in the initial chapters of the novel. Thus, we employ a tail truncation method, retaining the longest relevant text at the end of the string as input.



  
Answer Alignment

As our output consists of two components – the answer and the reasoning process, we require the model's output to be in the format "answer": "x", "reasoning": "xxx" for easy storage as a dictionary using JSON. However, not all models consistently follow these instructions, leading to difficulties in loading as a dictionary. In response, we identified a specific pattern and developed a standardized response alignment script. This script enables converting responses in a particular format to dictionary form, ensuring the validity of the responses. The impact of this approach will be discussed in Appendix <ref>.



  
Special Two-stage Answer

In the case of the Qwen-7B model, we observed significant challenges in adhering to instructions for answer output. However, as our testing focuses on assessing reasoning abilities for a dataset where instruction adherence is not a primary criterion, we implemented a unique two-phase answering method. Initially, the model is tasked with providing an answer to the question, followed by a separate prompt for generating the corresponding reasoning process.
In LWM, the situation is different, perhaps because we did not train multiple rounds of dialogue, we still can not get the answer using two rounds of dialogue, so in order to test we first let the model to give the inference process and then directly after the model's output, we add ‘So the answer is’ and get the next output's logits of the next output, and look for the one with the highest probability among the ABCDs as the multiple choice answer. This setup allows models that cannot follow the instructions to complete the evaluation of this dataset, increasing the usability of the dataset and improving the range of models that can be evaluated.






§ ANSWER ALIGNMENT




 §.§ Examples of non-compliance with Output Rules

Here are some examples of error output formats from a wide variety of models in Figure <ref>



 §.§ Changing Invalid Rate

After our alignment work, the inefficiency of the answer is much reduced, and the exact change is shown in Figure <ref>.




 





§ EXPERIMENT ON 32K MODELS


We have done quite a lot experiments under the 32K input length model, and we can find that the experimental results of the 32K model are all relatively unsatisfactory.
The experimental results are shown in Table <ref>

  


But a larger number of parameters would allow the model to make full use of the information in the 32K text and its own reasoning power to mitigate the problem.










































§ CASE STUDY


Such cases are usually the result of incorrect output from the model, where the model suggests the correct answer in the analysis but gives the wrong conclusion or option at the end.The Figure <ref> shows the given sample.We can see that instability in the model's reasoning ability can lead to errors in the final result.







§ INDIVIDUAL MODEL PERFORMANCE


We derived nine metrics each for each of the different models based on the two types of annotation types, which are the answer accuracy (Acc) reasoning metric (RM) and answer invalid rate(IR) for each of the three questioning settings: Question-Only(simp), Question+Context(deta) and Question+evidence(clue), and the results are shown in Figure <ref>

From this figure, you can clearly see the various scores of the model in each setting, which can help you to analyze data contamination <ref>, comprehensible decoupling, and so on. At the same time, the inefficiency of the answers also has some reference value, you can modify the answer method <ref> or use the answer alignment <ref> to reduce the inefficiency so that the test is more effective.








§ NO OPTIONS SETTING


Since our questions are presented in the form of multiple choice questions, and the requirement for model answers is to provide only one of the letters of ABCD, there is a possibility that the answer will be correct by randomly outputting one of the four letters, which affects the reliability of assessment.

Therefore, we add a setting, the question in this setting contains the author of the novel, the name of the novel, and the question, but does not give the corresponding options, and then in the output let the model directly give the text answer instead of ABCD, and at the same time give the reasoning process.

Observation of the responses obtained for these questions shows that the responses obtained by the model without options have some correct answers, but most of them are still incorrect and even hallucinate in cases where the novel content is not given.



