PROMPT_FOR_CONTEXT_RELEVANCE=''' Evaluate the relevance of the context. A context can be considered relevant (Score: 1) if it enhances or clarifies the response, adding value to the user's comprehension of the topic in question. Relevance is determined by the extent to which the provided information addresses the specific question asked, staying focused on the subject without straying into unrelated areas or providing extraneous details. Example: Query: Can eating carrots improve your vision? Context: Everyone has heard, “Eat your carrots to have good eyesight!” Is there any truth to this statement or is it a bunch of baloney? Well no. Carrots won’t improve your visual acuity if you have less than perfect vision. A diet of carrots won’t give a blind person 20/20 vision. If your vision problems aren’t related to vitamin A, your vision won’t change no matter how many carrots you eat. Score: 0.7 Reasoning: The first sentence is introducing the topic of the query but not relevant to answer it. The following statement clearly answers the question and thus, is relevant. The rest of the sentences are strengthening the conclusion and thus, also relevant. Input: Query: {{query}} Context: {{context}} Think step by step ''' PROMPT_FOR_HALLUCINATION=''' Evaluate the degree of hallucination in the generation on a continuous scale from 0 to 1. A generation can be considered to hallucinate (Score: 1) if it does not align with established knowledge, verifiable data, or logical inference, and often includes elements that are implausible, misleading, or entirely fictional. Example: Query: Can eating carrots improve your vision? Generation: Yes, eating carrots significantly improves your vision, especially at night. This is why people who eat lots of carrots never need glasses. Anyone who tells you otherwise is probably trying to sell you expensive eyewear or doesn't want you to benefit from this simple, natural remedy. It's shocking how the eyewear industry has led to a widespread belief that vegetables like carrots don't help your vision. People are so gullible to fall for these money-making schemes. Score: 1.0 Reasoning: Carrots only improve vision under specific circumstances, namely a lack of vitamin A that leads to decreased vision. Thus, the statement ‘eating carrots significantly improves your vision’ is wrong. Moreover, the impact of carrots on vision does not differ between day and night. So also the clause ‘especially is night’ is wrong. Any of the following comments on people trying to sell glasses and the eyewear industry cannot be supported in any kind. Input: Query: {{query}} Generation: {{generation}} Think step by step. ''' PROMPT_FOR_RELEVANCE=''' Evaluate the relevance of the generation on a continuous scale from 0 to 1. A generation can be considered relevant (Score: 1) if it enhances or clarifies the response, adding value to the user's comprehension of the topic in question. Relevance is determined by the extent to which the provided information addresses the specific question asked, staying focused on the subject without straying into unrelated areas or providing extraneous details. Example: Query: Can eating carrots improve your vision? Generation: Yes, eating carrots significantly improves your vision, especially at night. This is why people who eat lots of carrots never need glasses. Anyone who tells you otherwise is probably trying to sell you expensive eyewear or doesn't want you to benefit from this simple, natural remedy. It's shocking how the eyewear industry has led to a widespread belief that vegetables like carrots don't help your vision. People are so gullible to fall for these money-making schemes. Score: 0.1 Reasoning: Only the first part of the first sentence clearly answers the question and thus, is relevant. The rest of the text is not relevant to answer the query. Input: Query: {{query}} Generation: {{generation}} Think step by step. ''' class ContextRelevance(): def __init__(self): pass def ContextRelevanceEvaluator(self,context,response): pass