r based, then you might want a smaller chunk size.For our purposes, given that we're working with web pages that are written as documentation for a software library, we're going to use a chunk size of 500 and we'll make sure that the chunks overlap by 50 characters. This means that we'll have a lot of overlap between our chunks, which can be useful for ensuring that we don't miss any important information when we're splitting up our text.

Again, depending on your data and use case, there is more you might want to do with your data. You might want to clean the text, remove code snippets or make sure that code snippets were not split across chunks, or even extract metadata from the text. This is a good starting point, but you can always add more complexity as needed.

Next up, generating embeddings so that we can use them to retrieve relevant documents...

Code Example

To explore the full code, visit the Complete Guide repository and particularly the code for the steps in this section. Note, too, that a lot of the logic is encapsulated in utility functions inside url_scraping_utils.py.

PreviousUnderstanding Retrieval-Augmented Generation (RAG)

NextEmbeddings generation

Last updated 15 days ago