1. The method presented relies on extracting multiple responses from the LLM. For the variant with optimal performance, LLM prompting, 20 samples are needed to achieve the best reported results. Assuming a response contains 5 sentences, this requires 100 API calls to obtain a passage-level score (if I understand correctly), which is cost heavy and ineffective.
2. It remains unclear whether the proposed approach is suitable for detecting hallucinations in responses from other LLMs and across various application scenarios beyond WikiBio. This uncertainty arises because the experiment dataset exclusively encompasses WikiBio responses drawn from text-davinci-003.
3. The proposed method might struggle to detect hallucinations in open-ended responses, for example, the prompt "introduce a sports celebrity to me". In this case, the sampled responses could pertain to different individuals, making it challenging to identify shared information for consistency checking.
3. The proposed method might struggle to detect hallucinations in open-ended responses, for example, the prompt "introduce a sports celebrity to me". In this case, the sampled responses could pertain to different individuals, making it challenging to identify shared information for consistency checking.