Docker settings on a pipeline

Using Docker images to run your pipeline.

When a pipeline is run with a remote orchestrator a Dockerfile is dynamically generated at runtime. It is then used to build the Docker image using the image builder component of your stack. The Dockerfile consists of the following steps:

Starts from a parent image that has ZenML installed. By default, this will use the official ZenML image for the Python and ZenML version that you're using in the active Python environment. If you want to use a different image as the base for the following steps, check out this guide.

Installs additional pip dependencies. ZenML will automatically detect which integrations are used in your stack and install the required dependencies. If your pipeline needs any additional requirements, check out our guide on including custom dependencies.

Optionally copies your source files. Your source files need to be available inside the Docker container so ZenML can execute your step code. Check out this section for more information on how you can customize how ZenML handles your source files in Docker images.

Sets user-defined environment variables.

The process described above is automated by ZenML and covers the most basic use cases. This section covers various ways to customize the Docker build process to fit your needs.

For a full list of configuration options, check out the DockerSettings object on the SDKDocs.

How to configure settings for a pipeline

Customizing the Docker builds for your pipelines and steps is done using the DockerSettings class which you can import like this:

from zenml.config import DockerSettings

There are many ways in which you can supply these settings:

Configuring them on a pipeline applies the settings to all steps of that pipeline:

from zenml.config import DockerSettings

docker_settings = DockerSettings()

# Either add it to the decorator

@pipeline(settings={"docker": docker_settings})

def my_pipeline() -> None:

my_step()