<p>I am currently trying to setup Airflow to work in a Kubernetes like environment. For airflow to be useful, I need to be able to use the Git-Sync features so that the DAGs can be stored seperatly from the Pod, thus not being reset when the Pod downscales or restarts. I am trying to set it up with ssh.</p>

<p>I have been searching for good documentation on the Airflow config or tutorials on how to set this up properly, but this has been to no avail. I would very much appreciate some help here, as I have been struggling with this for a while. </p>

<p>Here is how i set the relevant config, please note I have some stand ins for links and some information due to security reasons:</p>

<pre><code>git_repo = https://&lt;git-host&gt;/scm/&lt;project-name&gt;/airflow
git_branch = develop
git_subpath = dags
git_sync_root = /usr/local/airflow
git_sync_dest = dags
git_sync_depth = 1
git_sync_ssh = true
git_dags_folder_mount_point = /usr/local/airflow/dags
git_ssh_key_secret_name = airflow-secrets
git_ssh_known_hosts_configmap_name = airflow-configmap
dags_folder = /usr/local/airflow/
executor = KubernetesExecutor
dags_in_image = False
</code></pre>

<p>Here is how I have setup my origin/config repo:</p>

<pre><code>-root
 |-configmaps/airflow
   |-airflow.cfg
   |-airflow-configmap.yaml
 |-environment
   |-&lt;environment specific stuff&gt;
 |-secrets
   |-airflow-secrets.yaml
 |-ssh
   |-id_rsa
   |-id_rsa.pub
 |-README.md
</code></pre>

<p>The airflow-conifgmap and secrets look like this:</p>

<pre><code>apiVersion: v1
kind: Secret
metadata:
  name: airflow-secrets
data:
  # key needs to be gitSshKey
  gitSshKey: &lt;base64 encoded private sshKey&gt;
</code></pre>

<p>and </p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: airflow-configmap
data:
  known_hosts: |
      https://&lt;git-host&gt;/ ssh-rsa &lt;base64 encoded public sshKey&gt;

</code></pre>

<p>The repo that I am trying to sync to has the Public key set as an access key and is just a folder named dags with 1 dag inside.</p>

<p>My issue is that I do not know what my issue is at this point. I have no way of knowing what part of my config has been set correctly and what part of it is set incorrectly and documentation on the subject is very lackluster. </p>

<p>If there is more information that is required I will be happy to provide it. </p>

<p>Thank you for your time</p>

NobiliChili
<p>Whats the error you're seeing on doing this ?</p>
<p>Couple of things you need to consider:</p>
<ul>
<li><p>Create an SSH key locally using this <a href="https://docs.github.com/en/github/authenticating-to-github/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent" rel="nofollow noreferrer">link</a> and:</p>
<ol>
<li><p>Repository Name &gt; Settings &gt; Deploy Keys &gt; Value of ssh_key.pub</p>
</li>
<li><p>Ensure &quot;write access&quot; is checked</p>
</li>
</ol>
</li>
<li><p>My <code>Dockerfile</code> I'm using looks like:</p>
<pre><code>FROM apache/airflow:2.1.2

COPY requirements.txt .

RUN python -m pip install --upgrade pip
RUN pip install -r requirements.txt
</code></pre>
</li>
<li><p>The <code>values.yaml</code> from the official Airflow Helm repository (<code>helm repo add apache-airflow https://airflow.apache.org</code>) needs the following values updated under <code>gitSync</code>:</p>
<ul>
<li><p><code>enabled: true</code></p>
</li>
<li><p><code>repo: ssh://git@github.com/username/repository-name.git</code></p>
</li>
<li><p><code>branch: master</code></p>
</li>
<li><p><code>subPaths: &quot;&quot;</code> (if DAGs are in repository root)</p>
</li>
<li><p><code>sshKeySecret: airflow-ssh-git-secret</code></p>
</li>
<li><p><code>credentialsSecret: git-credentials</code></p>
</li>
</ul>
</li>
<li><p>Export SSH key and <code>known_hosts</code> to Kubernetes secret for accessing the private repository</p>
<pre><code>kubectl create secret generic airflow-ssh-git-secret \
  --from-file=gitSshKey=/path/to/.ssh/id_ed25519 \
  --from-file=known_hosts=/path/to/.ssh/known_hosts \
  --from-file=id_ed25519.pub=/path/to/.ssh/id_ed25519.pub \
  -n airflow
</code></pre>
</li>
<li><p>Create and apply manifests:</p>
<pre><code>apiVersion: v1
kind: Secret
metadata:
  namespace: airflow
  name: airflow-ssh-git-secret
data:
  gitSshKey: &lt;base64_encoded_private_key_id_ed25519_in_one_line&gt;
</code></pre>
<pre><code>apiVersion: v1
kind: Secret
metadata:
  name: git-credentials
data:
  GIT_SYNC_USERNAME: base64_encoded_git_username
  GIT_SYNC_PASSWORD: base64_encoded_git_password
</code></pre>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  namespace: airflow
  name: known-hosts
data:
  known_hosts: |
    line 1 of known_host file
    line 2 of known_host file
    line 3 of known_host file
    ...
</code></pre>
</li>
<li><p>Update Airflow release</p>
<p><code>helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug</code></p>
</li>
<li><p>Get pods in the <em>airflow</em> namespace</p>
<p><code>kubectl get pods -n airflow</code></p>
</li>
<li><p>The <code>airflow-scheduler-SOME-STRING</code> pod is going to have 3 containers running. View the logs of container <code>git-sync-init</code> if you dont see the pods in Running state</p>
</li>
</ul>
