 easy and secure access to the remote GKE cluster.Have an existing AKS cluster set up.

Make sure you have the az CLI set up first.

Download and install kubectl and ensure that it talks to your AKS cluster using the following command:Copyaz aks get-credentials --resource-group RESOURCE_GROUP --name CLUSTER_NAME

Install Kubeflow Pipelines onto your cluster.

Since Kubernetes v1.19, AKS has shifted

to containerd

. However, the workflow controller installed with the Kubeflow installation has Docker set as the default runtime. In order to make your pipelines work, you have to change the value to one of the options

listed here

, preferably k8sapi.

This change has to be made by editing the containerRuntimeExecutor property of the ConfigMap corresponding to the workflow controller. Run the following commands to first know what config map to change and then to edit it to reflect your new value.

kubectl get configmap -n kubeflow

kubectl edit configmap CONFIGMAP_NAME -n kubeflow

# This opens up an editor that can be used to make the change.

If one or more of the deployments are not in the Running state, try increasing the number of nodes in your cluster.

If you're installing Kubeflow Pipelines manually, make sure the Kubernetes service is called exactly ml-pipeline. This is a requirement for ZenML to connect to your Kubeflow Pipelines deployment.

Infrastructure Deployment

A Kubeflow orchestrator can be deployed directly from the ZenML CLI:

zenml orchestrator deploy kubeflow_orchestrator --flavor=kubeflow --provider=<YOUR_PROVIDER> ...

You can pass other configurations specific to the stack components as key-value arguments. If you don't provide a name, a random one is generated for you. For more information about how to work use the CLI for this, please refer to the dedicated documentation section.

How to use it

To use the Kubeflow orchestrator, we need:

The ZenML kubeflow integration installed. If you haven't done so, runCopyzenml integration install kubeflow

Docker installed and running.