 total_tests) * 100

return round(failure_rate, 2)This function takes a sample size and a flag indicating whether to use reranking and evaluates the retrieval performance based on the generated questions and relevant documents. It queries similar documents for each question and checks whether the expected URL ending is present in the retrieved URLs. The failure rate is calculated as the percentage of failed tests over the total number of tests.

This function is then called in two separate evaluation steps: one for the retrieval system without reranking and one for the retrieval system with reranking.

@step

def retrieval_evaluation_full(

sample_size: int = 100,

) -> Annotated[float, "full_failure_rate_retrieval"]:

"""Executes the retrieval evaluation step without reranking."""

failure_rate = perform_retrieval_evaluation(

sample_size, use_reranking=False

logging.info(f"Retrieval failure rate: {failure_rate}%")

return failure_rate

@step

def retrieval_evaluation_full_with_reranking(

sample_size: int = 100,

) -> Annotated[float, "full_failure_rate_retrieval_reranking"]:

"""Executes the retrieval evaluation step with reranking."""

failure_rate = perform_retrieval_evaluation(

sample_size, use_reranking=True

logging.info(f"Retrieval failure rate with reranking: {failure_rate}%")

return failure_rate

Both of these steps return the failure rate of the respective retrieval systems. If we want, we can look into the logs of those steps (either on the dashboard or in the terminal) to see specific examples that failed. For example:

...

Loading default flashrank model for language en

Default Model: ms-marco-MiniLM-L-12-v2

Loading FlashRankRanker model ms-marco-MiniLM-L-12-v2

Loading model FlashRank model ms-marco-MiniLM-L-12-v2...

Running pairwise ranking..

Failed for question:  Based on the provided ZenML documentation text, here's a question

that can be asked: "How do I develop a custom alerter as described on the Feast page,