<h1>Objective</h1>

<p>I want to deploy Airflow on Kubernetes where pods have access to the same DAGs, in a Shared Persistent Volume.
According to the documentation (<a href="https://github.com/helm/charts/tree/master/stable/airflow#using-one-volume-for-both-logs-and-dags" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/stable/airflow#using-one-volume-for-both-logs-and-dags</a>), it seems I have to set and pass these values to Helm: <code>extraVolume</code>, <code>extraVolumeMount</code>, <code>persistence.enabled</code>, <code>logsPersistence.enabled</code>, <code>dags.path</code>, <code>logs.path</code>.</p>

<h1>Problem</h1>

<p>Any custom values I pass when installing the official Helm chart results in errors similar to:</p>

<pre><code>Error: YAML parse error on airflow/templates/deployments-web.yaml: error converting YAML to JSON: yaml: line 69: could not find expected ':'
</code></pre>

<ul>
<li>Works fine: <code>microk8s.helm install --namespace "airflow" --name
"airflow" stable/airflow</code></li>
<li><strong>Not working</strong>:</li>
</ul>

<pre><code>microk8s.helm install --namespace "airflow" --name "airflow" stable/airflow \
--set airflow.extraVolumes=/home/*user*/github/airflowDAGs \
--set airflow.extraVolumeMounts=/home/*user*/github/airflowDAGs \
--set dags.path=/home/*user*/github/airflowDAGs/dags \
--set logs.path=/home/*user*/github/airflowDAGs/logs \
--set persistence.enabled=false \
--set logsPersistence.enabled=false
</code></pre>

<ul>
<li><strong>Also not working</strong>: <code>microk8s.helm install --namespace "airflow" --name "airflow" stable/airflow --values=values_pv.yaml</code>, with <code>values_pv.yaml</code>: <a href="https://pastebin.com/PryCgKnC" rel="nofollow noreferrer">https://pastebin.com/PryCgKnC</a>

<ul>
<li>Edit: Please change <code>/home/*user*/github/airflowDAGs</code> to a path on your machine to replicate the error.</li>
</ul></li>
</ul>

<h1>Concerns</h1>

<ol>
<li>Maybe it is going wrong because of these lines in the default <code>values.yaml</code>:</li>
</ol>

<pre><code>## Configure DAGs deployment and update
dags:
  ##
  ## mount path for persistent volume.
  ## Note that this location is referred to in airflow.cfg, so if you change it, you must update airflow.cfg accordingly.
  path: /home/*user*/github/airflowDAGs/dags
</code></pre>

<p>How do I configure <code>airflow.cfg</code> in a Kubernetes deployement? In a non-containerized deployment of Airflow, this file can be found in <code>~/airflow/airflow.cfg</code>.</p>

<ol start="2">
<li>Line 69 in <code>airflow.cfg</code> refers to: <a href="https://github.com/helm/charts/blob/master/stable/airflow/templates/deployments-web.yaml#L69" rel="nofollow noreferrer">https://github.com/helm/charts/blob/master/stable/airflow/templates/deployments-web.yaml#L69</a></li>
</ol>

<p>Which contains <code>git</code>. Are the <code>.yaml</code> wrongly configured, and it falsely is trying to use <code>git pull</code>, but since no git path is specified, this fails?</p>

<h1>System</h1>

<ul>
<li>OS: Ubuntu 18.04 (single machine)</li>
<li>MicroK8s: v1.15.4 Rev:876</li>
<li><code>microk8s.kubectl version</code>: v1.15.4</li>
<li><code>microk8s.helm version</code>: v2.14.3</li>
</ul>

<h1>Question</h1>

<p>How do I correctly pass the right values to the Airflow Helm chart to be able to deploy Airflow on Kubernetes with Pods having access to the same DAGs and logs on a Shared Persistent Volume?</p>

<p>Not sure if you have this solved yet, but if you haven't I think there is a pretty simple way close to what you are doing.</p>

<p>All of the Deployments, Services, Pods need the persistent volume information - where it lives locally and where it should go within each kube kind. It looks like the values.yaml for the chart provides a way to do this. I'll only show this with dags below, but I think it should be roughly the same process for logs as well.</p>

<p>So the basic steps are, 1) tell kube where the 'volume' (directory) lives on your computer, 2) tell kube where to put that in your containers, and 3) tell airflow where to look for the dags. So, you can copy the values.yaml file from the helm repo and alter it with the following.</p>

<ol>
<li>The <code>airflow</code> section</li>
</ol>

<p>First, you need to create a volume containing the items in your local directory (this is the <code>extraVolumes</code> below). Then, that needs to be mounted - luckily putting it here will template it into all kube files. Once that volume is created, then you should tell it to mount <code>dags</code>. So basically, <code>extraVolumes</code> creates the volume, and <code>extraVolumeMounts</code> mounts the volume.</p>

<pre><code>airflow:
  extraVolumeMounts: # this will get the volume and mount it to that path in the container                                                                                                                                                               
  - name: dags
    mountPath: /usr/local/airflow/dags  # location in the container it will put the directory mentioned below.

  extraVolumes: # this will create the volume from the directory
  - name: dags
    hostPath:
      path: "path/to/local/directory"  # For you this is something like /home/*user*/github/airflowDAGs/dags

</code></pre>

<ol start="2">
<li>Tell the airflow config where the dags live in the container (same yaml section as above).</li>
</ol>

<pre><code>airflow:
  config:
    AIRFLOW__CORE__DAGS_FOLDER: "/usr/local/airflow/dags"  # this needs to match the mountPath in the extraVolumeMounts section
</code></pre>

<ol start="3">
<li>Install with helm and your new <code>values.yaml</code> file.</li>
</ol>

<pre><code>helm install --namespace "airflow" --name "airflow" -f local/path/to/values.yaml stable/airflow
</code></pre>

<p>In the end, this should allow airflow to see your local directory in the dags folder. If you add a new file, it should show up in the container - though it may take a minute to show up in the UI - I don't think the dagbag process is constantly running? Anyway, hope this helps!</p>
