Best practices

Recommended repository structure and best practices.

While it doesn't matter how you structure your ZenML project, here is a recommended project structure the core team often uses:

├── .dockerignore

├── Dockerfile

├── steps

│   ├── loader_step

│   │   ├── .dockerignore (optional)

│   │   ├── Dockerfile (optional)

│   │   ├── loader_step.py

│   │   └── requirements.txt (optional)

│   └── training_step

│       └── ...

├── pipelines

│   ├── training_pipeline

│   │   ├── .dockerignore (optional)

│   │   ├── config.yaml (optional)

│   │   ├── Dockerfile (optional)

│   │   ├── training_pipeline.py

│   │   └── requirements.txt (optional)

│   └── deployment_pipeline

│       └── ...

├── notebooks

│   └── *.ipynb

├── requirements.txt

├── .zen

└── run.py

All ZenML Project templates are modeled around this basic structure.

Steps

Keep your steps in separate Python files. This allows you to optionally keep their utils, dependencies, and Dockerfiles separate.

Logging

ZenML records the root python logging handler's output into the artifact store as a side-effect of running a step. Therefore, when writing steps, use the logging module to record logs, to ensure that these logs then show up in the ZenML dashboard.

# Use ZenML handler

from zenml.logger import get_logger

logger = get_logger(__name__)

...

@step

def training_data_loader():

# This will show up in the dashboard

logger.info("My logs")

Pipelines

Just like steps, keep your pipelines in separate Python files. This allows you to optionally keep their utils, dependencies, and Dockerfiles separate.

It is recommended that you separate the pipeline execution from the pipeline definition so that importing the pipeline does not immediately run it.

Do not give pipelines or pipeline instances the name "pipeline". Doing this will overwrite the imported pipeline and decorator and lead to failures at later stages if more pipelines are decorated there.