Diversify-verify-adapt: Efficient and Robust Retrieval-Augmented Ambiguous Question Answering
Yeonjun In Sungchul Kim Ryan A. Rossi Md Mehrab Tanjim Tong Yu Ritwik Sinha Chanyoung Park

Pluralistic Salient Object Detection
    Xuelu Feng, Yunsheng Li, Dongdong Chen, Chunming Qiao, Fellow, IEEE, Junsong Yuan, Fellow, IEEE, Lu Yuan, and Gang Hua, Fellow, IEEE
        
Xuelu Feng, Chunming Qiao, Junsong Yuan are with the Department of Computer Science and Engineering, University at Buffalo, USA (e-mail: xuelufen@buffalo.edu; qiao@buffalo.edu; jsyuan@buffalo.edu).
Yunsheng Li, Dongdong Chen, Lu Yuan are with Microsoft GenAI, USA (e-mail: yunshengli@microsoft.com; dongdong.chen@microsoft.com; luyuan@microsoft.com)
Gang Hua is with Dolby Laboratories, USA (e-mail: ganghua@gmail.com).

    September 9, 2024
======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
The retrieval augmented generation (RAG) framework addresses an ambiguity in user queries in QA systems by retrieving passages that cover all plausible interpretations and generating comprehensive responses based on the passages. However, our preliminary studies reveal that a single retrieval process often suffers from low-quality results, as the retrieved passages frequently fail to capture all plausible interpretations. Although the iterative RAG approach has been proposed to address this problem, it comes at the cost of significantly reduced efficiency.
 
To address these issues, we propose the diversify-verify-adapt () framework.  first diversifies the retrieved passages to encompass diverse interpretations. Subsequently,  verifies the quality of the passages and adapts the most suitable approach tailored to their quality.
This approach improves the QA systems' accuracy and robustness by handling low quality retrieval issue in ambiguous questions, while enhancing efficiency.






§ INTRODUCTION



Open-domain question answering (QA) systems are designed to provide users with factually accurate responses across various domains. In these systems, user queries often pose an ambiguity issue, requiring the response encompassing answers from multiple interpretations; in fact, over 50% of Google search queries fall into this category <cit.>. Such ambiguous questions pose a significant challenge to QA systems as they must accurately determine user intentions, which critically affects the user experience. It is therefore crucial for QA systems to offer answers that encompass all potential interpretations of ambiguous questions. 




Despite the crucial importance of addressing ambiguous questions in real-world applications, this area remains insufficiently explored compared to the extensive research focused on unambiguous questions <cit.>. In this work, we address this gap by delving into the more complex scenarios of ambiguous QA to better handle such questions.







Retrieval-augmented generation (RAG) framework has made significant progress in open-domain QA tasks <cit.> and also proven to be an effective solution for addressing ambiguous questions <cit.>. Specifically, these approaches first retrieve passages on the given question and prompt the LLM to extract plausible interpretations and answers relying on the passages (c.f. Fig <ref>(a)).

Despite the success of the RAG framework on the ambiguous QA task, we should rethink:
Is a single retrieval process sufficient to retrieve passages encompassing all plausible interpretations? To answer this question, we conduct preliminary experiments (c.f. Sec <ref>) about the quality of the retrieved passages used in the  RAG framework. We observe that the passages obtained from the single retrieval process often pose a low quality issue with respect to addressing ambiguous questions. In other words, the retrieved passages often partially or completely failed to cover all plausible interpretations, leading to significant performance degradation in terms of factual accuracy. 

To address this issue, the iterative RAG approach, ToC <cit.>, has been introduced (c.f. Fig <ref>(b)) to further explore other interpretations that can not be covered by the single retrieval process. Specifically, 
to further explore missing interpretations, the interpretations extracted in the previous iteration are utilized as queries to retrieve new passages, additional interpretations are then extracted. This exploration process is repeated in multiple times, leading to encompassing more diverse interpretations and corresponding answers. However, we argue that this effectiveness comes with a significant increase in computational overheads due to the iterative passage retrieval and LLM reasoning. In our experiments, this method requires an average of 5.5 exploration steps per query. As shown in Figure <ref>, Iterative RAG (i.e., ToC) significantly outperforms the vanilla RAG approach in terms of factual accuracy but at the cost of greatly reduced efficiency, with notable increases in both inference time and API call costs.

To this end, we introduce an efficient and robust RAG framework for ambiguous QA, referred to as diversify-verify-adapt ().  comprises two key components efficiently addressing the low quality retrieval issue: 1) Retrieval Diversification (RD) and 2) Adaptive Generation (AG). The key idea of RD is to infer pseudo-interpretations of a question, using them to retrieve a set of passages that broadly cover these interpretations, thus enhancing retrieval quality without any iterative interpretation exploration process. To further enhance the robustness of this framework, we propose an adaptive generation (AG) method. The key idea of AG is to carefully verify the overall quality of the passages retrieved from RD before indiscriminately incorporating them. More specifically, we define a new criterion of quality levels tailored to ambiguous questions: {, , }. Subsequently, AG adapts the most suitable approach between relying on the retrieved passages and LLM's internal knowledge, each of which is tailored to the specific quality level of the passages.





Experiments demonstrate that the proposed RD method efficiently diversifies the retrieval process to obtain passages covering diverse interpretations, thereby enhancing both QA and retrieval accuracy. Additionally, the proposed AG method successfully discriminate low quality passages, leading to the improvement of the QA performance. Consequently,  outperforms existing baselines on the ASQA <cit.> and SituatedQA <cit.> across various    LLM backbones in a few-shot setup, achieving superior accuracy and efficiency. The key contributions of this work are as follows: 











    
    
  * To the best of our knowledge, this paper is the first attempt to investigate the practical limitations of the existing RAG frameworks when applied to ambiguous QA task: low quality retrieval and inefficiency.
    
    
    
  * We propose , an efficient and robust RAG framework that efficiently retrieves diverse passages, verifies their quality, and adapts the most suitable approach tailored to each retrieval quality.
    
  *  consistently outperforms state-of-the-art RAG approaches in ambiguous QA task, while significantly more efficient (nearly 1.5 - 3 times faster response generation). 
    





§ PRELIMINARY EXPERIMENTS




We investigate the quality of retrieved passages and their impact on the performance of the RAG framework (as in Fig <ref>(a)) in ambiguous QA task.

Experimental Details. 
We utilize the most recent ambiguous QA dataset, ASQA <cit.>. We classify the quality of retrieved passages into three labels: 1) Fully Cover, 2) Partially Cover, and 3) Not Cover. Fully Cover indicates that the retrieved passages encompass all plausible interpretations, Not Cover does that the retrieved passages do not contain any of them, and otherwise Partially Cover.  We obtain these labels for each question by computing a string exact match between a set of retrieved passages and all plausible answers provided in ASQA as ground-truth answers. For implementation details of retrieving passages, see Appendix <ref>.

Results. 
In Fig <ref>(a), we observe that for only 34.6% of questions (i.e., Fully Cover) the retriever successfully retrieves passages that cover all plausible interpretations. Additionally, for 15.7% of questions (i.e., Not Cover) the retriever fails to retrieve any relevant passages. 
More critically, as shown in Fig <ref>(b), the performance of the RAG framework (i.e., RAG in the figure)

significantly deteriorates in terms of the factual accuracy (i.e., D-F1) when the retrieved passages pose a low quality issue (i.e., Partial Cover and Not Cover), indicating that it is highly susceptible to noise and irrelevant information in the ambiguous QA. 



This observation raises a follow-up question: How can we handle cases where the retrieved passages do not fully cover the plausible answers? To address this issue, we conducted another experiment that compares the effectiveness of LLM's internal knowledge and provided passages for different cases, respectively. We observe that when the retrieved passages do not contain any of the plausible interpretations (i.e., Not Cover), the closed-book LLM (i.e., LLM in the figure) significantly outperforms the RAG framework. This suggests that QA performance benefits more from relying on the LLMs' internal knowledge rather than on external passages containing entirely irrelevant information. 





In short, while the quality of retrieval is crucial for the performance of the RAG framework in ambiguous QA, existing works have largely overlooked this critical issue, which notably diminishes their practical applicability.



§ PROPOSED METHOD: 





Based on the findings, we propose an efficient and robust RAG framework for ambiguous QA, diversify-verify-adapt ().

This framework comprises two key components: Retrieval Diversification (Sec <ref>) and Adaptive Generation (Sec <ref>). The retrieval diversification method aims to efficiently diversify the retrieved passages to encompass diverse interpretations. Subsequently, the adaptive generation method aims to verify the quality of the passages and adapt the most suitable approach tailored to their quality. Fig <ref>(c) and Algorithm <ref> show the overview and inference algorithm of , respectively.





 §.§ Problem Formulation



Given an ambiguous question q_i, the goal of the proposed RAG framework is to generate a comprehensive response r_i that encompasses all plausible answers 𝒜_i= { a_i,1,...,a_i,M} of the interpretations 𝒬_i={ q_i,1,...,q_i,M} based on the retrieved passages 𝒫_i = { p_i,1, ..., p_i, K}, where M and K indicate the number of plausible answers and passages, respectively. Specifically, given the 𝒫_i ideally contains all 𝒬_i and 𝒜_i, an LLM is first prompted with the question and the relevant passages to extract all plausible interpretations and their corresponding answers, formally represented as follows:




    𝒬_i, 𝒜_i ←(q_i, 𝒫_i, I_e),




where I_e is a text prompt for extracting 𝒬_i and 𝒜_i from the 𝒫_i. Subsequently, based on the 𝒬_i and 𝒜_i, the LLM is prompted to consolidate them with q_i and 𝒫_i to generate a response r_i, formally represented as follows:




    r_i ←(𝒬_i, 𝒜_i, 𝒫_i, q_i, I_g).




For the prompts I_e and I_g, we start with that of <cit.> and modify it for our setup (see Table <ref> and Table <ref> in Appendix <ref>).












 §.§ Retrieval Diversification (RD)



In this section, we propose a novel retrieval diversification (RD) method aiming to efficiently identify passages 𝒫_i encompassing all plausible answers 𝒜_i of the interpretations 𝒬_i. The key idea of RD is to infer pseudo-interpretations of a question, using them to retrieve a set of passages that maximally cover these interpretations. This approach guarantees the retrieved passages encompass diverse interpretations, without any iterative interpretation exploration process of <cit.>, leading to the generated response r_i covering all 𝒜_i.












Inferring Pseudo-Interpretations.  

To infer pseudo-interpretations 𝒬̂_i={q̂_i, 1, q̂_i, 2, ... }, each of which related to a true plausible answer of 𝒜_i, we draw inspiration from a human's reasoning chain inferring multiple interpretations of a question. Given an ambiguous question, a human would first identify the ambiguous part of the question and then determine the reason for the ambiguity, followed by inferring multiple interpretations of the question. For example, given the question "Who played the Weasley brothers in Harry Potter?", the ambiguous part is the object of the question, "Weasley brothers," and the corresponding reason is that "It can refer to multiple characters such as Ron, Percy, and so on." Consequently, a human would generate "Who played Ron Weasley in Harry Potter?", "Who played Percy Weasley in Harry Potter?", etc. 

To mimic this reasoning chain, we leverage the LLM's powerful reasoning ability to identify the ambiguous part of the question and the reason for the ambiguity, subsequently, to infer the pseudo-interpretations 𝒬̂_i from the results, formally represented as follows:




    𝒬̂_i ←(q_i, I_p, (q_i, I_a)),




where I_p and I_a are carefully designed instructions for each step, respectively. We present the conceptual example of I_a and I_p in Fig <ref> and full instructions in Table <ref> and <ref> of Appendix <ref>. For (·), we consider GPT-3.5 <cit.> and GPT-4 <cit.>.

Retrieving Relevant and Diverse Passages.   As a first stage retrieval, we obtain the candidate passages 𝒞_i generally relevant to the given question q_i from Wikipedia[We use ColBERT <cit.> and Bing search API as retrievers.].

From the 𝒞_i, we select a set of multiple passages 𝒫_i with maximal coverage of all distinct pseudo-interpretations 𝒬̂_i. 

Retrieval for unambiguous questions involves scoring a single passage individually based on their relevance to a single interpretation. Whereas, when it comes to the ambiguous questions, we should retrieve a set of passages encompassing multiple interpretations, which makes this problem more challenging. To obtain such set of passages, we explicitly employ our inferred pseudo-interpretations 𝒬̂_i to retrieve the set of passages 𝒫̃_i that maximally cover these interpretations, formally represented as follows:




    𝒫̃_i ←⋃_j=1^|𝒬̂_i|ℛ(𝒞_i, q̂_i, j; K),




where ℛ is a retriever yielding top-K passages from the 𝒞_i by relevance scores to each pseudo interpretation q̂_i, j. 


Pruning Noisy Passages.  Although this process explicitly enables 𝒫̃_i to encompass all pseudo interpretations, there could be some noisy and irrelevant passages due to the absence of perfect retriever and the noise of the inferred pseudo-interpretations 𝒬̂_i.  To this end, we find and prune the passages that are highly likely to be irrelevant and noisy. Our intuition is that 1) noisy passages caused by the imperfect retriever tend to be irrelevant to all pseudo-interpretations and 2) noisy passages caused by noisy pseudo-interpretations tend to be irrelevant to most of the pseudo-interpretations. Based upon this intuition, we measure an averaged relevance of the passage to determine if it is noisy or not. The averaged relevance of a passage 𝒮(p) is calculated as follows:






    𝒮(p) ←1/|𝒬̂_i|∑_j=1^|𝒬̂_i|(q̂_j) ·(p)/||(q̂_j)|| · ||(p)||,




where (·) encodes sentences to a dense vector and p ∈𝒫̃_i. We then select the top-K passages from the 𝒫̃_i based on these averaged scores as the final passage set 𝒫_i. 



Our approach is generic, allowing for the use of various sentence embedding models for calculating relevance scores. In line with the sota baseline <cit.>, we employ the frozen SentenceBERT <cit.> in ℛ(·) and (·) in our implementation.























 §.§ Adaptive Generation (AG)





Despite the effectiveness of the proposed RD method, there may be the low quality of 𝒫_i. To further enhance the robustness of , in this section, we propose an adaptive generation method. The key idea of AG is to carefully verify the overall quality of the passages retrieved from RD before indiscriminately incorporating them. 

From the findings in Section <ref>, if 𝒫_i does not encompass all plausible interpretations, 𝒬_i and 𝒜_i, the response generated by the RAG framework is highly likely to be inaccurate. To this end, we introduce an adaptive generation (AG) method that dynamically adjust the response generation strategy among the RAG framework and closed-book LLM, which is achieved by verifying the quality of 𝒫_i before attempting a solution.


Retrieval Verification (RV)  To verify the quality of 𝒫_i, we exploit the LLM’s strong natural language understanding ability. The existing works <cit.> verify whether 𝒫_i can sufficiently support answering q_i by prompting or training the LLM to give a proper label V_i (e.g., Yes / No): 






    V_i ←(q_i, 𝒫_i, I_v),





where I_v is the corresponding instruction. However, the retrieval quality in terms of ambiguous questions should be graded according to how many interpretations are encompassed by the retrieved passages, which can not be achieved by the existing approaches tailored to unambiguous questions. To this end, we newly define a criterion of quality levels tailored to ambiguous questions: {, , }.  indicates the 𝒫_i encompasses all 𝒬_i and 𝒜_i,  indicates the 𝒫_i does not contain any of them, otherwise . To determine these grades, we estimate how many interpretations are encompassed by the 𝒫_i by explicitly utilizing the pseudo-interpretations 𝒬̂_i:






    V_i,1←(q̂_i,1, 𝒫_i, I_v) 
                 ⋮
        V_i,|𝒬̂_i|←(q̂_i,|𝒬̂_i|, 𝒫_i, I_v),




where each V_i,j consists of a binary label (i.e., Yes or No). For instance, if all V_i,* are determined "Yes" the grade is . We present the full prompt of I_v in Table <ref> in Appendix <ref>. For  (·), we consider GPT-3.5 and GPT-4. 







Adaptive Generation  Once we get the verification results from Eq <ref>, if the 𝒫_i is classified to  or , we decide to utilize the retrieved passages 𝒫_i to generate a response by Eq <ref> and <ref>. If 𝒫_i is classified to , we decide to only utilize the LLM's internal knowledge to generate a response: (q_i, I_l). The full prompt of I_l is presented in Table <ref> in Appendix <ref>. This process enables the utilization of the most suitable approach tailored to each retrieval quality, which is beneficial to both accuracy and efficiency. 




 §.§ Discussion on Efficiency







We examine the factors contributing to 's strong efficiency in Figure <ref>, which illustrates the average number of input and output tokens per query when using the GPT-4 backbone. First, 's strong efficiency is largely due to the RD method. Unlike Iterative RAG, which involves an average of 5.5 exploration steps per query and requires more than 12,000 tokens for input and 1,200 tokens for output, the RD method significantly reduces the number of tokens needed. This reduction leads to improved efficiency in inference time and API costs. Second, although the RV method introduces some additional costs, these are acceptable compared to the complexity of Iterative RAG. Moreover, RV enables the adaptive generation (AG) strategy, where the faster closed-book LLM is selectively used instead of RAG, further enhancing efficiency. As a result, , which combines RD, RV, and AG, requires substantially less inference time and API costs.





§ EXPERIMENTAL SETUPS





 §.§ Datasets

Our proposed method and all baseline models are assessed using the ASQA <cit.> and SituatedQA <cit.> datasets. ASQA is a long-form QA dataset featuring ambiguous questions. SituatedQA is a short-form QA dataset featuring questions that specifically highlight ambiguities related to temporal and geographical contexts. We give these questions to the QA systems and assess how comprehensively the responses cover the provided possible interpretations of questions. Further details about the datasets are provided in the Appendix <ref>.



 §.§ Evaluation Metrics

Metrics for QA.  Following <cit.>, we mainly adopt F1-based metrics. For the short-form QA dataset (SituatedQA) we utilize F1 score. Given ASQA is the long-form QA dataset, following <cit.>, we use Disambig-F1 (D-F1) score instead of F1. We further leverage ROUGE-L (R-L) to measure correctness of the long-form responses. Finally, Disambiguation-ROUGE (DR), combines R-L and D-F1 scores for overall performance.

Metrics for Passage Retrieval.  Following <cit.>, we use MRecall@k to evaluate the quality of retrieved passages. 

For more details of the evaluation metrics, please refer to Appendix <ref>.




 §.§ Baselines


We compare our  against relevant models, including fully-supervised LMs, few-shot closed book LLMs, LLMs w/ RAG, and the adaptive generation. Specifically, fully-supervised LMs include the 1) T5 closed-book <cit.>, 2) T5 w/ JPR <cit.>, and 3) PaLM <cit.> w/ Soft Prompt Tuning.

Few-shot closed book LLMs include 3) Vanilla LLAMA3, GPT-3.5-turbo, and GPT-4  and 4) Query refinement <cit.>. Few-shot LLMs w/ RAG include 5) Vanilla RAG where we use RAC prompt in <cit.>, for 6) Iterative RAG we use the sota method ToC <cit.>, and for adaptive generation 7) Self-RAG <cit.>. For more details of the baselines, please refer to Appendix <ref>. 










































 §.§ Implementation Details

In , the LLM is employed across three modules: retrieval diversification (Eqn <ref>), retrieval verification (Eqn <ref>), and adaptive response generation (Eqn <ref>, <ref>, and closed-book LLM). For adaptive response generation, we use the same LLM backbones as the other baselines. For the retrieval diversification and verification modules, we assess the performance of GPT-3.5 (gpt-3.5-turbo) and GPT-4 (gpt-4) across them, ultimately opting to use GPT-4 for both modules in the ASQA dataset and GPT-3.5 for both modules in the SituatedQA dataset in all experiments. However, as demonstrated in Section <ref>, other LLMs also perform effectively in these modules. For other implementation details, please refer to Appendix <ref>.









§ EXPERIMENTAL RESULTS AND ANALYSES




 §.§ Main Results



Table <ref> presents the long-form ambiguous QA performance of baselines and  on the development set of ASQA. 








First,  outperforms the sota baseline, Iterative RAG, in terms of both accuracy and efficiency of response generation. 

Our method enhances Vanilla RAG framework by incorporating retrieval diversification and adaptive generation strategies that address low-quality retrieval and improve performance. It is also more efficient, requiring significantly less computational overhead and achieving 1.5x - 3x greater efficiency in inference time across various LLM backbones compared to Iterative RAG. Overall, our method produces more accurate and diverse interpretations without the cumbersome iterative exploration process.


Second,  demonstrates good adaptability in switching out the underlying LLM backbones. 
 consistently enhances Vanilla RAG with its RD and AG modules across different LLM backbones, demonstrating its adaptability and wide applicability. This suggests that  can easily integrate with more advanced LLMs in the future.




Additionally, Fig <ref> shows the performance and efficiency of baselines and  on the SituatedQA test set for short-form ambiguous QA tasks. All experimental results align with those seen in Table <ref>, demonstrating strong generalizability of  across different types of ambiguous questions.







 
















        









        
        



























        




 


 






    

    




    

        





















 

 


 §.§ Ablation Studies













To evaluate the importance of each component of , namely retrieval diversification (RD) and adaptive generation (AG), we incrementally add them to Vanilla RAG (row 2 in Table <ref>). Table <ref> reveals the following insights: 
1) RAG (row 2) with the closed-book LLM (row 1) significantly enhances the ability to handle ambiguity in questions. 
2) Implementing the RD module (row 3) enhances all performance metrics, demonstrating that RD effectively diversifies and improves the quality of retrieved passages, thereby enhancing the RAG framework. 3) Incorporating the AG module (row 4) also boosts all metrics, showing that the retrieval verification method accurately identifies  passages. Additionally, this supports our finding in Sec <ref> that when retrieved passages are of extremely low quality, the internal knowledge of LLMs proves more advantageous than RAG.




 §.§ Retrieval Analysis


We evaluate the effectiveness of our proposed RD method in Table <ref> using MRecall@k <cit.>. Vanilla RAG (row 1) involves basic retrieval of passages using a given question q_i. "+ RD" (row 3) applies the RD method to row 1, using pseudo-interpretations generated by our proposed instructions (i.e., I_p and I_a). Row 2 uses the RD method with pseudo-interpretations generated by the LLM query rewriter as described in <cit.> using simple instructions. "+ Oracle" (row 4) applies RD to Vanilla RAG using ground-truth interpretations from the ASQA dataset.

We observe that 1) adding RD leads to significant improvements of MRecall and D-F1 score compared to Vanilla RAG, demonstrating RD effectively addresses low-quality retrieval issue and then improve the QA performance. 2) "+ RD" outperforms "+ <cit.>" showing the superiority of our carefully designed instruction in inferring pseudo-interpretations. 3) "+ Oracle" (row 4) significantly outperforms RD, indicating that when more advanced LLMs are available in the future there is potential for RD to improve in accurately inferring pseudo-interpretations.























        























        























































 §.§ Sensitivity Analysis



For the retrieval diversification (RD) and retrieval verification (RV) modules, we explore how their performance is affected by the choice of LLM. We evaluate the impact of using GPT-3.5 and GPT-4 across both modules, comparing the overall QA performance against the sota baseline, ToC <cit.>, on the ASQA and SituatedQA datasets. Fig <ref>(a) and (b) represent using GPT-3.5 and GPT-4 as the response generation models on the ASQA dataset, respectively. Fig <ref>(c) represents using GPT-3.5 as the response generation model on the SituatedQA dataset.

In Fig <ref>, we observe the following: 1)  consistently outperforms ToC, regardless of the LLM model used in each module. 2) While the RD module shows very stable results, the RV module appears relatively sensitive to the choice of LLM. This highlights that verifying the quality of retrieved passages for ambiguous questions requires more powerful natural language understanding ability, underscoring the need for future work to alleviate the dependency on the choice of LLM. Based on these results, we argue that  is a general framework that is robust across different LLM models.










 §.§ Case Studies



We conduct a case study to qualitatively compare the reasoning chains of Iterative RAG, ToC <cit.>, and . Due to space limits, please refer to Appendix <ref> for detailed case studies.








§ RELATED WORK





RAG for Handling Ambiguous Question.  
To tackle the ambiguity inherent in certain questions, earlier studies such as those by <cit.> necessitated the fine-tuning of models using extensive training datasets. For example, <cit.> introduced JPR designed to identify multiple passages with maximal coverage of all plausible answers, and language model (LM) is trained to generate a comprehensive answer from them. AmbigPrompt <cit.> iteratively generates prompts based on previous answers and uses LM to produce new answers, ensuring coverage of all plausible responses. Recently, some studies have leveraged LLMs to generate comprehensive responses through few-shot in-context learning. For example, RAC <cit.> instructs LLM to extract plausible interpretations and answers from provided passages. <cit.> developed a query rewriter that clarifies ambiguous questions, enabling the retrieval of specific passages. Black-box LLMs then generate answers from these passages. However, they overlook the problem of low-quality retrieval, where the retrieved passages frequently fail to cover all plausible interpretations. This often results in significant performance drops in factual accuracy. To tackle this issue, ToC <cit.> explores missing interpretations by iteratively using previously extracted interpretations as queries to retrieve new passages, from which further interpretations are then extracted. This exploratory process is repeated multiple times. However, this approach incurs significant computational overhead due to the iterative passage retrieval and LLM reasoning. 













Retrieval Quality Verification.  
Many studies have noted that low-quality retrieval introduces significant irrelevant information to the RAG framework and have proposed various solutions. Self-RAG <cit.> fine-tunes LLM to generate a reflection token that assesses the relevance of a passage to the question at hand. Llatrieval <cit.> employs LLM to check if retrieved passages sufficiently support the answer, updating them if they are of low quality. Meanwhile, CRAG <cit.> trains a lightweight verifier to evaluate the quality of retrieved passages, making corrections if they fall below a set threshold. 







Adaptive Generation.  
Numerous studies have examined adaptive strategies that dynamically determine the need for retrieval, utilizing only the internal knowledge of LLMs when unnecessary <cit.>. <cit.> used an empirical method to retrieval, activating relying on the frequency of entity. 

AdaptiveRAG <cit.> dynamically chooses the optimal response generation strategy tailored to the complexity of the query. TA-ARE <cit.> uses in-context learning to assess whether a query necessitates retrieval. 

Compared with recent studies that either overlook or inefficiently address the issue of low-quality retrieval in ambiguous questions, we introduce the retrieval diversification method efficiently retrieves higher quality passages without relying on cumbersome iterative processes. Additionally, we propose retrieval verification and adaptive generation strategies specifically designed for ambiguous questions. To the best of our knowledge, this paper is the first effort to thoroughly analyze and address the problem of low-quality retrieval in the context of ambiguous questions and its potential solutions.






































§ CONCLUSION




In this study, we examined the shortcomings of the current RAG-based method in dealing with ambiguous questions, specifically its low-quality retrieval and inefficiency. Our proposed framework,  , effectively diversifies the retrieved passages to capture various interpretations, verifies their quality, and adapts the most appropriate approach based on that quality. This strategy improves QA performance while minimizing inefficiency.



§ LIMITATIONS



While  demonstrates clear advantages in effectiveness and efficiency through retrieval diversification and adaptive generation, its design is specifically tailored for ambiguous questions. In real-world QA systems, where queries can be a mix of ambiguous and unambiguous, the applicability of  may be limited. However, recent work has introduced methods to classify whether a query is ambiguous <cit.>, which leads to utilizing the suitable approach according to its ambiguity. Although <cit.> proposed simple approaches, there is still significant potential to enhance these methods using advanced techniques like in-context learning and RAG. Future research could focus on developing systematic approaches for classifying the ambiguity of queries. 
Furthermore, the performance of our proposed retrieval verification module is somewhat sensitive to the choice of LLM. Specifically, it tends to work better with GPT-4 than with GPT-3.5, though this may negatively impact the efficiency of . Therefore, future work should focus on developing a more efficient and robust retrieval quality verifier LLM, tailored to handling ambiguous questions, to enhance both effectiveness and efficiency.







§ ETHICS STATEMENT

Given that  is built on the RAG framework of QA systems, it is important to consider the following points: (1) the retrieved passages may contain offensive or harmful content, which could result in similarly harmful responses, and (2) user queries themselves may be offensive or harmful. Therefore, developing methods to detect harmful user queries and selectively retrieve passages that are free from harmful content could be a crucial focus for future research.






















§ EXPERIMENTAL DETAILS




 §.§ Datasets


Our proposed method and all baseline models are assessed using the ASQA <cit.> and SituatedQA <cit.> datasets. ASQA is a long-form QA dataset derived from a subset of ambiguous questions in the AmbigNQ dataset <cit.>. The ASQA dataset contains 6,316 ambiguous questions and their corresponding comprehensive long-form answers that contain all plausible answers, split into 4,353 for training, 948 for development, and 1,015 for testing. SituatedQA is a short-form QA dataset featuring questions that specifically highlight ambiguities related to temporal and geographical contexts. In this dataset, each question is subject to multiple interpretations, with corresponding answers varying by context. We give these questions to the QA systems and assess how comprehensively the responses cover the possible interpretations.




 §.§ Evaluation Metrics



Metrics for QA.  For both datasets, following previous studies on ambiguous QA <cit.>, we
mainly adopt F1-based metric. Specifically, for the short-form QA dataset (SituatedQA) we measure F1 based on the precision and recall between the ground-truth answers and the generated responses. Given ASQA is the long-form QA dataset, following <cit.>, we use Disambig-F1 (D-F1), which assesses the factual accuracy of long-form responses, instead of F1. Using a RoBERTa model <cit.> trained on SQuAD2.0, we extract short answers from the generated long-form responses and compare them to the ground-truth disambiguation questions (DQs). The F1 score of these extracted answers indicates whether the long-form answers contain correct information. We further leverage ROUGE-L (R-L) to measure correctness of the generated long-form responses to the ground-truth long-form answers. Finally, Disambiguation-ROUGE (DR), combines R-L and D-F1 scores as a geometric mean for overall performance.

Metrics for Passage Retrieval.  Following <cit.>, we use MRecall@k to evaluate the quality of retrieved passages by considering retrieval to be successful if all answers or at least k answers in the plausible answer set are recovered by the retrieved passages. 





 §.§ Baselines



We describe the details of models as follows:

1) T5 closed-book.  <cit.> fine-tuned T5-large <cit.> to generate long-form response on the whole train set.

2) T5 w/ JPR.  <cit.> fine-tuned T5-large <cit.> with JPR <cit.>, fully trained dense retriever for ambiguous QA, to generate long-form response on the whole train set. 

3) PaLM w/ Soft Prompt Tuning.  <cit.> employed a prompt engineering method to PaLM <cit.> that learn the soft prompts in the closed-book setup. 

4) Closed-book LLM.   Closed-book LLM indicates the traditional few-shot prompting method used in <cit.>. We consider the backbone LLM as LLAMA3-70B-Instruct, GPT-3.5, and GPT-4.

5) Query refinement.  Inspired by <cit.>, we developed an in-context learning method within a closed-book setup. First, we prompt the LLM to refine ambiguous questions into multiple possible interpretations. These interpretations are then used as in-context examples for the LLM to generate a response that addresses all potential interpretations. We consider the backbone LLM as LLAMA3-70B-Instruct, GPT-3.5, and GPT-4.

6) Vanilla RAG.  In this method, we begin by retrieving the top 5 relevant passages based on the frozen SentenceBERT similarity between the given query and candidate passages from Wikipedia. We then use the RAC prompt from <cit.> to extract interpretations and generate corresponding answers. We consider the backbone LLM as LLAMA3-70B-Instruct, GPT-3.5, and GPT-4.

7) Iterative RAG.  For this approach, we employ the state-of-the-art method ToC <cit.> for handling ambiguous QA. Specifically, ToC iteratively constructs a tree of possible interpretations for the ambiguous question using few-shot prompting that leverages external knowledge, and then uses this tree to generate a long-form response. Following the authors' implementation, we set the tree's maximum depth to 3 and the maximum number of nodes to 10. It is important to note that we do not use the tree pruning method in our implementation, as we observe that adding this method notably degrades the QA performance. The retrieval settings are identical to those used in Vanilla RAG. We consider the backbone LLM as LLAMA3-70B-Instruct <cit.>, GPT-3.5, and GPT-4.

8) Self-RAG.  The LLM is trained to adaptively manage retrieval and generation, initiating retrieval when a special token is predicted above a certain threshold, followed by generating the answer. We consider the model trained on LLAMA2-13B.





 §.§ Implementations Details





Since  utilizes few-shot prompting, we dynamically select k-shot examples through nearest neighbor search and incorporate them into the prompt, following the approach in <cit.> using dsp package <cit.>. For the retrieved passages 𝒫_i, we set the number of passages |𝒫_i| to 5. We use GPT-4 for both the retrieval diversification and verification steps. For adaptive response generation, we use the same LLM backbones as the other baselines. For the retrieval diversification and verification modules, we assess the performance of GPT-3.5 (gpt-3.5-turbo) and GPT-4 (gpt-4) across them, ultimately opting to use GPT-4 for both modules in the ASQA dataset and GPT-3.5 for both modules in the SituatedQA dataset for all experiments.  The APIs provided by OpenAI[https://openai.com/index/openai-api/] are employed for GPT-3.5-turbo and GPT-4, with the following settings: max tokens set to 300, top-p to 1.0, and temperature to 0.3. 




  §.§.§ Retrieval Process


To retrieve relevant passages for the given question, we follow the method utilized by <cit.>. Specifically, we first gather relevant Wikipedia documents for the question using two retrieval systems: ColBERT <cit.> and the Bing search engine[https://www.microsoft.com/bing]. After compiling a set of passages, we rerank and select the top-k passages. For reranking, we utilize SentenceBERT <cit.>, pre-trained on MS-Marco, as the backbone. 





§ ADDITIONAL EXPERIMENTS





 §.§ Case Study



Figure <ref> illustrates the reasoning chains of Iterative RAG, ToC <cit.>, and  using the ASQA question, "The movement of food in the food pipe is called?". In panel (a), the answer "Peristalsis" is easily covered during the first exploration, whereas "Swallowing" requires six steps of passage retrieval and LLM reasoning for exploration. In contrast, panel (b) shows that our pseudo-interpretations include both interpretations, with the RD retrieving passages that encompass all necessary information. Consequently, the LLM efficiently extracts all plausible interpretations from the retrieved passages without the need for the cumbersome iterative exploration process.




§ PROMPTS



Table <ref> and Table <ref> show an example of text prompt for inferring pseudo-interpretations (i.e., I_a and I_p in Eqn <ref>). Table <ref> shows an example of text prompt for verifying the retrieved passages (i.e., I_v in Eqn <ref>). Table <ref> and <ref> show an exmple of text prompt for response generation in vanilla RAG framework (i.e., I_e in Eqn <ref> and I_g in Eqn <ref>)




























