<p>I have a Google Cloud Composer 1 environment (Airflow 2.1.2) where I want to run an Airflow DAG that utilizes the <a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html" rel="nofollow noreferrer">KubernetesPodOperator</a>.</p>
<p>Cloud Composer <a href="https://cloud.google.com/composer/docs/concepts/cloud-storage#folders_in_the_bucket" rel="nofollow noreferrer">makes available</a> to all DAGs a shared file directory for storing application data. The files in the directory reside in a Google Cloud Storage bucket managed by Composer. Composer uses FUSE to map the directory to the path <code>/home/airflow/gcs/data</code> on all of its Airflow worker pods.</p>
<p>In my DAG I run several Kubernetes pods like so:</p>
<pre class="lang-py prettyprint-override"><code>    from airflow.contrib.operators import kubernetes_pod_operator
    
    # ...

    splitter = kubernetes_pod_operator.KubernetesPodOperator(
        task_id='splitter',
        name='splitter',
        namespace='default',
        image='europe-west1-docker.pkg.dev/redacted/splitter:2.3',
        cmds=[&quot;dotnet&quot;, &quot;splitter.dll&quot;],
    )
</code></pre>
<p>The application code in all the pods that I run needs to read from and write to the <code>/home/airflow/gcs/data</code> directory. But when I run the DAG my application code is unable to access the directory. Likely this is because Composer has mapped the directory into the worker pods but does not extend this courtesy to my pods.</p>
<p>What do I need to do to give my pods r/w access to the <code>/home/airflow/gcs/data</code> directory?</p>

urig
<p>Cloud Composer uses FUSE to mount certain directories from Cloud Storage into Airflow worker pods running in Kubernetes. It mounts these with default permissions that cannot be overwritten, because that metadata is not tracked by Google Cloud Storage. A possible solution is to use a bash operator that runs at the beginning of your DAG to copy files to a new directory. Another possible solution can be to use a non-Google Cloud Storage path like a <code>/pod</code> path.</p>
