<p>Im attempting to incorporate git-sync sidecar container into my Airflow deployment yaml so my private Github repo gets synced to my Airflow Kubernetes env every time I make a change in the repo.</p>
<p>So far, it successfully creates a git-sync container along with our scheduler, worker, and web server pods, each in their respective pod (ex: scheduler pod contains a scheduler container and gitsync container).
 </p>
<p>I looked at the git-sync container logs and it looks like it successfully connects with my private repo (using a personal access token) and prints success logs every time I make a change to my repo.</p>
<pre><code>INFO: detected pid 1, running init handler
I0411 20:50:31.009097      12 main.go:401] &quot;level&quot;=0 &quot;msg&quot;=&quot;starting up&quot; &quot;pid&quot;=12 &quot;args&quot;=[&quot;/git-sync&quot;,&quot;-wait=60&quot;,&quot;-repo=https://github.com/jorgeavelar98/AirflowProject.git&quot;,&quot;-branch=master&quot;,&quot;-root=/opt/airflow/dags&quot;,&quot;-username=jorgeavelar98&quot;,&quot;-password-file=/etc/git-secret/token&quot;]
I0411 20:50:31.029064      12 main.go:950] &quot;level&quot;=0 &quot;msg&quot;=&quot;cloning repo&quot; &quot;origin&quot;=&quot;https://github.com/jorgeavelar98/AirflowProject.git&quot; &quot;path&quot;=&quot;/opt/airflow/dags&quot;
I0411 20:50:31.031728      12 main.go:956] &quot;level&quot;=0 &quot;msg&quot;=&quot;git root exists and is not empty (previous crash?), cleaning up&quot; &quot;path&quot;=&quot;/opt/airflow/dags&quot;
I0411 20:50:31.894074      12 main.go:760] &quot;level&quot;=0 &quot;msg&quot;=&quot;syncing git&quot; &quot;rev&quot;=&quot;HEAD&quot; &quot;hash&quot;=&quot;18d3c8e19fb9049b7bfca9cfd8fbadc032507e03&quot;
I0411 20:50:31.907256      12 main.go:800] &quot;level&quot;=0 &quot;msg&quot;=&quot;adding worktree&quot; &quot;path&quot;=&quot;/opt/airflow/dags/18d3c8e19fb9049b7bfca9cfd8fbadc032507e03&quot; &quot;branch&quot;=&quot;origin/master&quot;
I0411 20:50:31.911039      12 main.go:860] &quot;level&quot;=0 &quot;msg&quot;=&quot;reset worktree to hash&quot; &quot;path&quot;=&quot;/opt/airflow/dags/18d3c8e19fb9049b7bfca9cfd8fbadc032507e03&quot; &quot;hash&quot;=&quot;18d3c8e19fb9049b7bfca9cfd8fbadc032507e03&quot;
I0411 20:50:31.911065      12 main.go:865] &quot;level&quot;=0 &quot;msg&quot;=&quot;updating submodules&quot;
</code></pre>
<p> </p>
<p><strong>However, despite their being no error logs in my git-sync container logs, I could not find any of the files in the destination directory where my repo is supposed to be synced into (/opt/airflow/dags). Therefore, no DAGs are appearing in the Airflow UI</strong></p>
<p>This is our scheduler containers/volumes yaml definition for reference. We have something similar for workers and webserver</p>
<pre><code>      containers:
        - name: airflow-scheduler
          image: &lt;redacted&gt;
          imagePullPolicy: IfNotPresent
          envFrom:
            - configMapRef:
                name: &quot;AIRFLOW_SERVICE_NAME-env&quot;
          env:            
            &lt;redacted&gt;
          resources: 
            requests:
              memory: RESOURCE_MEMORY
              cpu: RESOURCE_CPU
          volumeMounts:
            - name: scripts
              mountPath: /home/airflow/scripts
            - name: dags-data
              mountPath: /opt/airflow/dags
              subPath: dags
            - name: dags-data
              mountPath: /opt/airflow/plugins
              subPath: plugins
            - name: variables-pools
              mountPath: /home/airflow/variables-pools/
            - name: airflow-log-config
              mountPath: /opt/airflow/config
          command:
            - &quot;/usr/bin/dumb-init&quot;
            - &quot;--&quot;
          args:
            &lt;redacted&gt;
        - name: git-sync
          image: registry.k8s.io/git-sync/git-sync:v3.6.5
          args:
            - &quot;-wait=60&quot;
            - &quot;-repo=&lt;repo&gt;&quot;
            - &quot;-branch=master&quot;
            - &quot;-root=/opt/airflow/dags&quot;
            - &quot;-username=&lt;redacted&gt;&quot;
            - &quot;-password-file=/etc/git-secret/token&quot;
          volumeMounts:
            - name: git-secret
              mountPath: /etc/git-secret
              readOnly: true
            - name: dags-data
              mountPath: /opt/airflow/dags
      volumes:
        - name: scripts
          configMap:
            name: AIRFLOW_SERVICE_NAME-scripts
            defaultMode: 493
        - name: dags-data
          emptyDir: {}
        - name: variables-pools
          configMap:
            name: AIRFLOW_SERVICE_NAME-variables-pools
            defaultMode: 493
        - name: airflow-log-config
          configMap:
            name: airflow-log-configmap
            defaultMode: 493
        - name: git-secret
          secret:
            secretName: github-token
</code></pre>
<p>What can be the issue? I couldn't find much documentation that could help me further investigate. Any help and guidance would be greatly appreciated!</p>

jorgeavelar98
<p>Your problem could be probably related to the directory structure you are defining across the different containers.</p>
<p>It is unclear in your question but, according to your containers definitions, your git repository should contain at least <code>dags</code> and <code>plugins</code> as top level directories:</p>
<pre><code>/
├─ dags/
├─ plugins/
</code></pre>
<p>This structure resembles a typical <code>airflow</code> folder structure: I assume, that is the one you configured.</p>
<p>Then, please, try using this slightly modified version of your Kubernetes configuration:</p>
<pre class="lang-yaml prettyprint-override"><code>      containers:
        - name: airflow-scheduler
          image: &lt;redacted&gt;
          imagePullPolicy: IfNotPresent
          envFrom:
            - configMapRef:
                name: &quot;AIRFLOW_SERVICE_NAME-env&quot;
          env:            
            &lt;redacted&gt;
          resources: 
            requests:
              memory: RESOURCE_MEMORY
              cpu: RESOURCE_CPU
          volumeMounts:
            - name: scripts
              mountPath: /home/airflow/scripts
            - name: dags-data
              mountPath: /opt/airflow/dags
              subPath: dags
            - name: dags-data
              mountPath: /opt/airflow/plugins
              subPath: plugins
            - name: variables-pools
              mountPath: /home/airflow/variables-pools/
            - name: airflow-log-config
              mountPath: /opt/airflow/config
          command:
            - &quot;/usr/bin/dumb-init&quot;
            - &quot;--&quot;
          args:
            &lt;redacted&gt;
        - name: git-sync
          image: registry.k8s.io/git-sync/git-sync:v3.6.5
          args:
            - &quot;-wait=60&quot;
            - &quot;-repo=&lt;repo&gt;&quot;
            - &quot;-branch=master&quot;
            - &quot;-root=/opt/airflow&quot;
            - &quot;-username=&lt;redacted&gt;&quot;
            - &quot;-password-file=/etc/git-secret/token&quot;
          volumeMounts:
            - name: git-secret
              mountPath: /etc/git-secret
              readOnly: true
            - name: dags-data
              mountPath: /opt
      volumes:
        - name: scripts
          configMap:
            name: AIRFLOW_SERVICE_NAME-scripts
            defaultMode: 493
        - name: dags-data
          emptyDir: {}
        - name: variables-pools
          configMap:
            name: AIRFLOW_SERVICE_NAME-variables-pools
            defaultMode: 493
        - name: airflow-log-config
          configMap:
            name: airflow-log-configmap
            defaultMode: 493
        - name: git-secret
          secret:
            secretName: github-token
</code></pre>
<p>Note that we basically changed the <code>root</code> argument of the <code>git-sync</code> container removing <code>/dags</code>.</p>
<p>If it doesn't work, please, try including and tweaking the value of the <a href="https://github.com/kubernetes/git-sync/tree/release-3.x#primary-flags" rel="nofollow noreferrer"><code>--dest</code></a> <code>git-sync</code> flag, I think it could be of help as well.</p>
