Configure your pipeline to add compute

Add more resources to your pipeline configuration.

Now that we have our pipeline up and running in the cloud, you might be wondering how ZenML figured out what sort of dependencies to install in the Docker image that we just ran on the VM. The answer lies in the runner script we executed (i.e. run.py), in particular, these lines:

pipeline_args["config_path"] = os.path.join(

config_folder, "training_rf.yaml"

# Configure the pipeline

training_pipeline_configured = training_pipeline.with_options(**pipeline_args)

# Create a run

training_pipeline_configured()

The above commands configure our training pipeline with a YAML configuration called training_rf.yaml (found here in the source code). Let's learn more about this configuration file.

The with_options command that points to a YAML config is only one way to configure a pipeline. We can also directly configure a pipeline or a step in the decorator:

@pipeline(settings=...)

However, it is best to not mix configuration from code to ensure separation of concerns in our codebase.

Breaking down our configuration YAML

The YAML configuration of a ZenML pipeline can be very simple, as in this case. Let's break it down and go through each section one by one:

The Docker settings

settings:

docker:

required_integrations:

sklearn

requirements:

pyarrow

containerization process. Here, we are simply telling ZenML that we need

DockerSettings class in the Python SDK.

Associating a ZenML Model

The next section is about associating a ZenML Model with this pipeline.

# Configuration of the Model Control Plane

model:

name: breast_cancer_classifier

version: rf

license: Apache 2.0

description: A breast cancer classifier

tags: ["breast_cancer", "classifier"]

You will see that this configuration lines up with the model created after executing these pipelines:

# List all versions of the breast_cancer_classifier

zenml model version list breast_cancer_classifier