 use case.

Automated evaluation using another LLMAnother way to evaluate the generation component is to use another LLM to grade the output of the LLM you're evaluating. This is a more sophisticated approach and requires a bit more setup. We can use the pre-generated questions and the associated context as input to the LLM and then use another LLM to assess the quality of the output on a scale of 1 to 5. This is a more quantitative approach and since it's automated it can run across a larger set of data.

LLMs don't always do well on this kind of evaluation where numbers are involved. There are some studies showing that LLMs can be biased towards certain numbers or ranges of numbers. This is something to keep in mind when using this approach. Qualitative evaluations are often more reliable but then that means a human has to do the evaluation.

We can start by setting up a Pydantic model to hold the data we need. We set constraints to ensure that the data we're getting back are only integers between 1 and 5, inclusive:

class LLMJudgedTestResult(BaseModel):

toxicity: conint(ge=1, le=5)

faithfulness: conint(ge=1, le=5)

helpfulness: conint(ge=1, le=5)

relevance: conint(ge=1, le=5)

We can use this in a test function that:

takes a question and a context as inputs

generates an answer using the LLM we're evaluating

makes a call to an (optionally different) LLM we're using to judge the quality of the answer getting back a score for each of the four categories in JSON format

parses the JSON and returns the result of the evaluation as our Pydantic model instance

Pydantic handles the validation of the JSON input for us, so we can be sure that we're getting the data we expect and in a form that we can use.

def llm_judged_test_e2e(

question: str,

context: str,

n_items_retrieved: int = 5,

) -> LLMJudgedTestResult:

"""E2E tests judged by an LLM.

Args:

item (dict): The item to test.

n_items_retrieved (int): The number of items to retrieve.

Returns:

LLMJudgedTestResult: The result of the test.

"""