Data ingestion and preprocessing

Understand how to ingest and preprocess data for RAG pipelines with ZenML.

The first step in setting up a RAG pipeline is to ingest the data that will be used to train and evaluate the retriever and generator models. This data can include a large corpus of documents, as well as any relevant metadata or annotations that can be used to train the retriever and generator.

In the interests of keeping things simple, we'll implement the bulk of what we need ourselves. However, it's worth noting that there are a number of tools and frameworks that can help you manage the data ingestion process, including downloading, preprocessing, and indexing large corpora of documents. ZenML integrates with a number of these tools and frameworks, making it easy to set up and manage RAG pipelines.

You can view all the code referenced in this guide in the associated project repository. Please visit the llm-complete-guide project inside the ZenML projects repository if you want to dive deeper.

You can add a ZenML step that scrapes a series of URLs and outputs the URLs quite easily. Here we assemble a step that scrapes URLs related to ZenML from its documentation. We leverage some simple helper utilities that we have created for this purpose:

from typing import List

from typing_extensions import Annotated

from zenml import log_artifact_metadata, step

from steps.url_scraping_utils import get_all_pages

@step

def url_scraper(

docs_url: str = "https://docs.zenml.io",

repo_url: str = "https://github.com/zenml-io/zenml",

website_url: str = "https://zenml.io",

) -> Annotated[List[str], "urls"]:

"""Generates a list of relevant URLs to scrape."""

docs_urls = get_all_pages(docs_url)

log_artifact_metadata(

metadata={

"count": len(docs_urls),

},

return docs_urls