te = (failures / total_tests) * 100

logging.info(f"Total tests: {total_tests}. Failures: {failures}. Failure rate: {failure_rate}%"

return round(failure_rate, 2)

Our end-to-end evaluation of the generation component is then a combination of these tests:

@step

def e2e_evaluation() -> (

Annotated[float, "failure_rate_bad_answers"],

Annotated[float, "failure_rate_bad_immediate_responses"],

Annotated[float, "failure_rate_good_responses"],

):

logging.info("Testing bad answers...")

failure_rate_bad_answers = run_tests(

bad_answers, test_content_for_bad_words

logging.info(f"Bad answers failure rate: {failure_rate_bad_answers}%")

logging.info("Testing bad immediate responses...")

failure_rate_bad_immediate_responses = run_tests(

bad_immediate_responses, test_response_starts_with_bad_words

logging.info(

f"Bad immediate responses failure rate: {failure_rate_bad_immediate_responses}%"

logging.info("Testing good responses...")

failure_rate_good_responses = run_tests(

good_responses, test_content_contains_good_words

logging.info(

f"Good responses failure rate: {failure_rate_good_responses}%"

return (

failure_rate_bad_answers,

failure_rate_bad_immediate_responses,

failure_rate_good_responses,

Running the tests using different LLMs will give different results. Here our Ollama Mixtral did worse than GPT 3.5, for example, but there were still some failures with GPT 3.5. This is a good way to get a sense of how well your generation component is doing.

As you become more familiar with the kinds of outputs your LLM generates, you can add the hard ones to this test suite. This helps prevent regressions and is directly related to the quality of the output you're getting. This way you can optimize for your specific use case.

Automated evaluation using another LLM