 use it

To use the Airflow orchestrator, we need:The ZenML airflow integration installed. If you haven't done so, runCopyzenml integration install airflow

Docker installed and running.

The orchestrator registered and part of our active stack:

zenml orchestrator register <ORCHESTRATOR_NAME> \

--flavor=airflow \

--local=True  # set this to `False` if using a remote Airflow deployment

# Register and activate a stack with the new orchestrator

zenml stack register <STACK_NAME> -o <ORCHESTRATOR_NAME> ... --set

Due to dependency conflicts, we need to install the Python packages to start a local Airflow server in a separate Python environment.

# Create a fresh virtual environment in which we install the Airflow server dependencies

python -m venv airflow_server_environment

source airflow_server_environment/bin/activate

# Install the Airflow server dependencies

pip install "apache-airflow==2.4.0" "apache-airflow-providers-docker<3.8.0" "pydantic~=2.7.1"

Before starting the local Airflow server, we can set a few environment variables to configure it:

AIRFLOW_HOME: This variable defines the location where the Airflow server stores its database and configuration files. The default value is ~/airflow.

AIRFLOW__CORE__DAGS_FOLDER: This variable defines the location where the Airflow server looks for DAG files. The default value is <AIRFLOW_HOME>/dags.

AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: This variable controls how often the Airflow scheduler checks for new or updated DAGs. By default, the scheduler will check for new DAGs every 30 seconds. This variable can be used to increase or decrease the frequency of the checks, depending on the specific needs of your pipeline.

When running this on MacOS, you might need to set the no_proxy environment variable to prevent crashes due to a bug in Airflow (see this page for more information):

export no_proxy=*

We can now start the local Airflow server by running the following command: