Kubeflow

Run your ML pipelines on Kubeflow Pipelines.

The ZenML Kubeflow Orchestrator allows you to run your ML pipelines on Kubeflow Pipelines without writing Kubeflow code.

Prerequisites

To use the Kubeflow Orchestrator, you'll need:

ZenML kubeflow integration installed (zenml integration install kubeflow)

Docker installed and running

kubectl installed (optional, see below)

A Kubernetes cluster with Kubeflow Pipelines installed (see deployment guide for your cloud provider)

A remote artifact store and container registry in your ZenML stack

A remote ZenML server deployed to the cloud

The name of your Kubernetes context pointing to the remote cluster (optional, see below)

Configuring the Orchestrator

There are two ways to configure the orchestrator:

Using a Service Connector to connect to the remote cluster (recommended for cloud-managed clusters). No local kubectl context needed.

zenml orchestrator register <ORCHESTRATOR_NAME> --flavor kubeflow

zenml service-connector list-resources --resource-type kubernetes-cluster -e

zenml orchestrator connect <ORCHESTRATOR_NAME> --connector <CONNECTOR_NAME>

zenml stack update -o <ORCHESTRATOR_NAME>

Configuring kubectl with a context pointing to the remote cluster and setting kubernetes_context in the orchestrator config:

zenml orchestrator register <ORCHESTRATOR_NAME> \

--flavor=kubeflow \

--kubernetes_context=<KUBERNETES_CONTEXT>

zenml stack update -o <ORCHESTRATOR_NAME>

Running a Pipeline

Once configured, you can run any ZenML pipeline using the Kubeflow Orchestrator:

python your_pipeline.py

This will create a Kubernetes pod for each step in your pipeline. You can view pipeline runs in the Kubeflow UI.

Additional Configuration

You can further configure the orchestrator using KubeflowOrchestratorSettings:

from zenml.integrations.kubeflow.flavors.kubeflow_orchestrator_flavor import KubeflowOrchestratorSettings

kubeflow_settings = KubeflowOrchestratorSettings(

client_args={},