orming as expected.

When and how much to evaluateThe frequency and depth of evaluation will depend on your specific use case and the constraints of your project. In an ideal world, you would evaluate the performance of your embeddings and the RAG system as a whole as often as possible, but in practice, you'll need to balance the cost of running the evaluation with the need to iterate quickly.

Some tests can be run quickly and cheaply (notably the tests of the retrieval system) while others (like the LLM judge) are more expensive and time-consuming. You should structure your RAG tests and evaluation to reflect this, with some tests running frequently and others running less often, just as you would in any other software project.

There's more we could improve our evaluation system, but for now we can continue onwards to adding a reranker to improve our retrieval. This will allow us to improve the performance of our retrieval system without needing to retrain the embeddings. We'll cover this in the next section.

Try it out!

To see how this works in practice, you can run the evaluation pipeline using the project code. This will give you a sense of how the evaluation process works in practice and you can of course then play with and modify the evaluation code.

To run the evaluation pipeline, first clone the project repository:

git clone https://github.com/zenml-io/zenml-projects.git

Then navigate to the llm-complete-guide directory and follow the instructions in the README.md file to run the evaluation pipeline. (You'll have to have first run the main pipeline to generate the embeddings.)

To run the evaluation pipeline, you can use the following command:

python run.py --evaluation

This will run the evaluation pipeline and output the results to the console. You can then inspect the progress, logs and results in the dashboard!

PreviousGeneration evaluation

NextReranking for better retrieval

Last updated 15 days ago