Orchestrate on the cloud

Orchestrate using cloud resources.

Until now, we've only run pipelines locally. The next step is to get free from our local machines and transition our pipelines to execute on the cloud. This will enable you to run your MLOps pipelines in a cloud environment, leveraging the scalability and robustness that cloud platforms offer.

In order to do this, we need to get familiar with two more stack components:

The orchestrator manages the workflow and execution of your pipelines.

The container registry is a storage and content delivery system that holds your Docker container images.

These, along with remote storage, complete a basic cloud stack where our pipeline is entirely running on the cloud.

Starting with a basic cloud stack

The easiest cloud orchestrator to start with is the Skypilot orchestrator running on a public cloud. The advantage of Skypilot is that it simply provisions a VM to execute the pipeline on your cloud provider.

Coupled with Skypilot, we need a mechanism to package your code and ship it to the cloud for Skypilot to do its thing. ZenML uses Docker to achieve this. Every time you run a pipeline with a remote orchestrator, ZenML builds an image for the entire pipeline (and optionally each step of a pipeline depending on your configuration). This image contains the code, requirements, and everything else needed to run the steps of the pipeline in any environment. ZenML then pushes this image to the container registry configured in your stack, and the orchestrator pulls the image when it's ready to execute a step.

To summarize, here is the broad sequence of events that happen when you run a pipeline with such a cloud stack:

The user runs a pipeline on the client machine. This executes the run.py script where ZenML reads the @pipeline function and understands what steps need to be executed.

The client asks the server for the stack info, which returns it with the configuration of the cloud stack.