Manage artifacts

Understand and adjust how ZenML versions your data.

Data sits at the heart of every machine learning workflow. Managing and versioning this data correctly is essential for reproducibility and traceability within your ML pipelines. ZenML takes a proactive approach to data versioning, ensuring that every artifact—be it data, models, or evaluations—is automatically tracked and versioned upon pipeline execution.

This guide will delve into artifact versioning and management, showing you how to efficiently name, organize, and utilize your data with the ZenML framework.

Managing artifacts produced by ZenML pipelines

Artifacts, the outputs of your steps and pipelines, are automatically versioned and stored in the artifact store. Configuring these artifacts is pivotal for transparent and efficient pipeline development.

Giving names to your artifacts

Assigning custom names to your artifacts can greatly enhance their discoverability and manageability. As best practice, utilize the Annotated object within your steps to give precise, human-readable names to outputs:

from typing_extensions import Annotated

import pandas as pd

from sklearn.datasets import load_iris

from zenml import pipeline, step

# Using Annotated to name our dataset

@step

def training_data_loader() -> Annotated[pd.DataFrame, "iris_dataset"]:

"""Load the iris dataset as pandas dataframe."""

iris = load_iris(as_frame=True)

return iris.get("frame")

@pipeline

def feature_engineering_pipeline():

training_data_loader()

if __name__ == "__main__":

feature_engineering_pipeline()

Unspecified artifact outputs default to a naming pattern of {pipeline_name}::{step_name}::output. For visual exploration in the ZenML dashboard, it's best practice to give significant outputs clear custom names.

Artifacts named iris_dataset can then be found swiftly using various ZenML interfaces:

To list artifacts: zenml artifact list