 <ORCHESTRATOR_NAME> ... --set

Running a PipelineOnce configured, you can run any ZenML pipeline using the SkyPilot VM Orchestrator. Each step will run in a Docker container on a provisioned VM.

Additional Configuration

You can further configure the orchestrator using cloud-specific Settings objects:

from zenml.integrations.skypilot_<PROVIDER>.flavors.skypilot_orchestrator_<PROVIDER>_vm_flavor import Skypilot<PROVIDER>OrchestratorSettings

skypilot_settings = Skypilot<PROVIDER>OrchestratorSettings(

cpus="2",

memory="16",

accelerators="V100:2",

use_spot=True,

region=<REGION>,

...

@pipeline(

settings={

"orchestrator.vm_<PROVIDER>": skypilot_settings

This allows specifying VM size, spot usage, region, and more.

You can also configure resources per step:

high_resource_settings = Skypilot<PROVIDER>OrchestratorSettings(...)

@step(settings={"orchestrator.vm_<PROVIDER>": high_resource_settings})

def resource_intensive_step():

...

For more details and advanced options, see the full SkyPilot VM Orchestrator documentation.

PreviousMLflow

NextConnect services (AWS, GCP, Azure, K8s etc)

Last updated 15 days ago