ns, especially using the basic setup we have here.To give you an indication of how long this process takes, generating 1800+ questions from an equivalent number of documentation chunks took a little over 45 minutes using the local model on a GPU-enabled machine with Ollama.

You can view the generated dataset on the Hugging Face Hub here. This dataset contains the original document chunks, the generated questions, and the URL reference for the original document.

Once we have the generated questions, we can then pass them to the retrieval component and check the results. For convenience we load the data from the Hugging Face Hub and then pass it to the retrieval component for evaluation. We shuffle the data and select a subset of it to speed up the evaluation process, but for a more thorough evaluation you could use the entire dataset. (The best practice of keeping a separate set of data for evaluation purposes is also recommended here, though we're not doing that in this example.)

@step

def retrieval_evaluation_full(

sample_size: int = 50,

) -> Annotated[float, "full_failure_rate_retrieval"]:

dataset = load_dataset("zenml/rag_qa_embedding_questions", split="train")

sampled_dataset = dataset.shuffle(seed=42).select(range(sample_size))

total_tests = len(sampled_dataset)

failures = 0

for item in sampled_dataset:

generated_questions = item["generated_questions"]

question = generated_questions[

]  # Assuming only one question per item

url_ending = item["filename"].split("/")[

1

]  # Extract the URL ending from the filename

_, _, urls = query_similar_docs(question, url_ending)

if all(url_ending not in url for url in urls):

logging.error(

f"Failed for question: {question}. Expected URL ending: {url_ending}. Got: {urls}"

failures += 1

logging.info(f"Total tests: {total_tests}. Failures: {failures}")

failure_rate = (failures / total_tests) * 100

return round(failure_rate, 2)