configure (among others) the following attributes:pod_settings: Node selectors, affinity, and tolerations to apply to the Kubernetes Pods running the steps of your pipeline. These can be either specified using the Kubernetes model objects or as dictionaries.

orchestrator_pod_settings: Node selectors, affinity, and tolerations to apply to the Kubernetes Pod that is responsible for orchestrating the pipeline and starting the other Pods. These can be either specified using the Kubernetes model objects or as dictionaries.

from zenml.integrations.kubernetes.flavors.kubernetes_orchestrator_flavor import KubernetesOrchestratorSettings

from kubernetes.client.models import V1Toleration

kubernetes_settings = KubernetesOrchestratorSettings(

pod_settings={

"affinity": {

"nodeAffinity": {

"requiredDuringSchedulingIgnoredDuringExecution": {

"nodeSelectorTerms": [

"matchExpressions": [

"key": "node.kubernetes.io/name",

"operator": "In",

"values": ["my_powerful_node_group"],

},

"tolerations": [

V1Toleration(

key="node.kubernetes.io/name",

operator="Equal",

value="",

effect="NoSchedule"

@pipeline(

settings={

"orchestrator.kubernetes": kubernetes_settings

...

Check out the SDK docs for a full list of available attributes and this docs page for more information on how to specify settings.

For more information and a full list of configurable attributes of the Kubernetes orchestrator, check out the SDK Docs .

Enabling CUDA for GPU-backed hardware

Note that if you wish to use this orchestrator to run steps on a GPU, you will need to follow the instructions on this page to ensure that it works. It requires adding some extra settings customization and is essential to enable CUDA for the GPU to give its full acceleration.

PreviousKubeflow Orchestrator

NextGoogle Cloud VertexAI Orchestrator

Last updated 15 days ago