view.

2. Add ZenML as an explicit pip requirementZenML requires that ZenML itself be installed for the containers running your pipelines and steps. Therefore, you need to explicitly state that ZenML should be installed. There are several ways to specify this, but as an example, you can update the code from above as follows:

from zenml.config import DockerSettings

from zenml import pipeline

docker_settings = DockerSettings(

parent_image="pytorch/pytorch:1.12.1-cuda11.3-cudnn8-runtime",

requirements=["zenml==0.39.1", "torchvision"]

@pipeline(settings={"docker": docker_settings})

def my_pipeline(...):

...

Adding these two extra settings options will ensure that CUDA is enabled for the specific steps that require GPU acceleration. Be cautious when choosing the image to avoid confusion when switching between local and remote environments. For example, you might have one version of PyTorch installed locally with a particular CUDA version, but when you switch to your remote stack or environment, you might be forced to use a different CUDA version.

The core cloud operators offer prebuilt Docker images that fit with their hardware. You can find more information on them here:

AWS

GCP

Azure

Not all of these images are available on DockerHub, so ensure that the orchestrator environment your pipeline runs in has sufficient permissions to pull images from registries if you are using one of those.

Reset the CUDA cache in between steps

Your use case will determine whether this is necessary or makes sense to do, but we have seen that resetting the CUDA cache in between steps can help avoid issues with the GPU cache. This is particularly necessary if your training jobs are pushing the boundaries of the GPU cache. Doing so is simple; just use a helper function to reset the cache at the beginning of any GPU-enabled steps. For example, something as simple as this might suffice:

import gc

import torch

def cleanup_memory() -> None:

while gc.collect():

torch.cuda.empty_cache()