 Using the operator for all steps in your pipeline@pipeline(settings={"orchestrator.airflow": airflow_settings})

def my_pipeline(...):

Custom operators

If you want to use any other operator to run your steps, you can specify the operator in your AirflowSettings as a path to the python operator class:

from zenml.integrations.airflow.flavors.airflow_orchestrator_flavor import AirflowOrchestratorSettings

airflow_settings = AirflowOrchestratorSettings(

# This could also be a reference to one of your custom classes.

# e.g. `my_module.MyCustomOperatorClass` as long as the class

# is importable in your Airflow server environment

operator="airflow.providers.docker.operators.docker.DockerOperator",

# Dictionary of arguments to pass to the operator __init__ method

operator_args={}

Custom DAG generator file

To run a pipeline in Airflow, ZenML creates a Zip archive that contains two files:

A JSON configuration file that the orchestrator creates. This file contains all the information required to create the Airflow DAG to run the pipeline.

A Python file that reads this configuration file and actually creates the Airflow DAG. We call this file the DAG generator and you can find the implementation here .

original module . For this reason, we suggest starting by copying the original and modifying it according to your needs.

Check out our docs on how to apply settings to your pipelines here.

For more information and a full list of configurable attributes of the Airflow orchestrator, check out the SDK Docs .

PreviousTekton Orchestrator

NextSkypilot VM Orchestrator

Last updated 8 days ago