

embeddings_2d[mask, 0],

embeddings_2d[mask, 1],c=[section_color_dict[section]],

label=section,

plt.title("t-SNE Visualization")

plt.legend()

plt.show()

# Dimensionality reduction using UMAP

def umap_visualization(embeddings, parent_sections):

umap_2d = umap.UMAP(n_components=2, random_state=42)

embeddings_2d = umap_2d.fit_transform(embeddings)

plt.figure(figsize=(8, 8))

for section in unique_parent_sections:

if section in section_color_dict:

mask = [section == ps for ps in parent_sections]

plt.scatter(

embeddings_2d[mask, 0],

embeddings_2d[mask, 1],

c=[section_color_dict[section]],

label=section,

plt.title("UMAP Visualization")

plt.legend()

plt.show()

In this stage, we have utilized the 'parent directory', which we had previously stored in the vector store as an additional attribute, as a means to color the values. This approach allows us to gain some insight into the semantic space inherent in our data. It demonstrates that you can visualize the embeddings and observe how similar chunks are grouped together based on their semantic meaning and context.

So this step iterates through all the chunks and generates embeddings representing each piece of text. These embeddings are then stored as an artifact in the ZenML artifact store as a NumPy array. We separate this generation from the point where we upload those embeddings to the vector database to keep the pipeline modular and flexible; in the future we might want to use a different vector database so we can just swap out the upload step without having to re-generate the embeddings.

In the next section, we'll explore how to store these embeddings in a vector database to enable fast and efficient retrieval of relevant chunks at inference time.

Code Example

To explore the full code, visit the Complete Guide repository. The embeddings generation step can be found here.

PreviousData ingestion and preprocessing

NextStoring embeddings in a vector database

Last updated 15 days ago