ader() -> Tuple[

Annotated[pd.DataFrame, "data"],Annotated[DatasetProfileView, "profile"]

]:

"""Load the diabetes dataset."""

X, y = datasets.load_diabetes(return_X_y=True, as_frame=True)

# merge X and y together

df = pd.merge(X, y, left_index=True, right_index=True)

profile = why.log(pandas=df).profile().view()

return df, profile

How do you use it?

Whylogs's profiling functions take in a pandas.DataFrame dataset generate a DatasetProfileView object containing all the relevant information extracted from the dataset.

There are three ways you can use whylogs in your ZenML pipelines that allow different levels of flexibility:

instantiate, configure and insert the standard WhylogsProfilerStep shipped with ZenML into your pipelines. This is the easiest way and the recommended approach, but can only be customized through the supported step configuration parameters.

call the data validation methods provided by the whylogs Data Validator in your custom step implementation. This method allows for more flexibility concerning what can happen in the pipeline step, but you are still limited to the functionality implemented in the Data Validator.

use the whylogs library directly in your custom step implementation. This gives you complete freedom in how you are using whylogs's features.

You can visualize whylogs profiles in Jupyter notebooks or view them directly in the ZenML dashboard.

The whylogs standard step

ZenML wraps the whylogs/WhyLabs functionality in the form of a standard WhylogsProfilerStep step. The only field in the step config is a dataset_timestamp attribute which is only relevant when you upload the profiles to WhyLabs that uses this field to group and merge together profiles belonging to the same dataset. The helper function get_whylogs_profiler_step used to create an instance of this standard step takes in an optional dataset_id parameter that is also used only in the context of WhyLabs upload to identify the model in the context of which the profile is uploaded, e.g.: