<p>I have a container with the following configuration:</p>
<pre><code>spec:
  template:
    spec:
      restartPolicy: OnFailure
      volumes:
        - name: local-src
          hostPath:
            path: /src/analysis/src
           type: DirectoryOrCreate
      containers:
          securityContext:
            privileged: true
            capabilities:
              add:
                - SYS_ADMIN
</code></pre>
<ul>
<li>Note that I'm intentionally omitting some other configuration parameters to keep the question short</li>
</ul>
<p>However, when I deploy it to my cluster on kubernetes on gcloud, I see the following error:</p>
<pre><code>Error: failed to start container &quot;market-state&quot;: Error response from daemon: error while creating mount source path '/src/analysis/src': mkdir /src: read-only file system
</code></pre>
<p>I have tried deploying the exact same job locally with minikube and it works fine.</p>
<p>My guess is that this has to do with the pod's permissions relative to the host, but I expected it to work given the <code>SYS_ADMIN</code> permissions that I'm setting. When creating my cluster, I gave it a <code>devstorage.read_write</code> scope for other reason, but am wondering if there are other scopes I need as well?</p>
<pre><code>gcloud container clusters create my_cluster \
    --zone us-west1-a \
    --node-locations us-west1-a \
    --scopes=https://www.googleapis.com/auth/devstorage.read_write
</code></pre>
<p>DirectoryOrCreate</p>

<p>As pointed by user @DazWilkin:</p>
<blockquote>
<p>IIUC, if your cluster is using Container-Optimized VMs, you'll need to be aware of the structure of the file system for these instances.</p>
<p>See <a href="https://cloud.google.com/container-optimized-os/docs/concepts/disks-and-filesystem" rel="nofollow noreferrer">https://cloud.google.com/container-optimized-os/docs/concepts/disks-and-filesystem</a></p>
</blockquote>
<p>This is correct understanding. You can't write to readonly location like: <code>/</code>  (even with the <code>SYS_ADMIN</code> and <code>privileged</code> parameters) because of the:</p>
<blockquote>
<p><strong>The root filesystem is mounted as read-only to protect system integrity</strong>. However, home directories and /mnt/stateful_partition are persistent and writable.</p>
<p>-- <em><a href="https://cloud.google.com/container-optimized-os/docs/concepts/disks-and-filesystem#filesystem" rel="nofollow noreferrer">Cloud.google.com: Container optimized OS: Docs: Concepts: Disk and filesystem: Filesystem</a></em></p>
</blockquote>
<p>As for a <strong>workaround</strong> solution you can change the location of your <code>hostPath</code> on the node or use <code>GKE</code> with nodes that uses <code>Ubuntu</code> images instead of <code>Container Optimized OS</code> images. You will be able to use <code>hostPath</code> volumes with paths as specified in your question. You can read more about available node images by following official documentation:</p>
<ul>
<li><em><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/node-images" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Node images</a></em></li>
</ul>
<hr />
<p>If your workload/use case allows using <code>Persistent Volumes</code>, I encourage you to do so.</p>
<blockquote>
<p>PersistentVolume resources are used to manage durable storage in a cluster. In GKE, PersistentVolumes are typically backed by Compute Engine persistent disks.</p>
<p>&lt;---&gt;</p>
<p>PersistentVolumes are cluster resources that exist independently of <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/pod" rel="nofollow noreferrer">Pods</a>. This means that the disk and data represented by a PersistentVolume continue to exist as the cluster changes and as Pods are deleted and recreated. PersistentVolume resources can be provisioned dynamically through PersistentVolumeClaims, or they can be explicitly created by a cluster administrator.</p>
<p>-- <em><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Persistent Volumes</a></em></p>
</blockquote>
<p>You can also consider looking on <code>Local SSD</code> solution which can use <code>hostPath</code> type of <code>Volume</code>:</p>
<ul>
<li><em><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/local-ssd" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Persistent Volumes: Local SSD</a></em></li>
</ul>
<hr />
<blockquote>
<p>When creating my cluster, I gave it a <code>devstorage.read_write</code> scope for other reason, but am wondering if there are other scopes I need as well?</p>
</blockquote>
<p>You can create <code>GKE</code> cluster without adding any additional scopes like:</p>
<ul>
<li><code>$ gcloud container clusters create  --zone=ZONE</code></li>
</ul>
<p>The: <code>--scopes=SCOPE</code> will depend on the workload you are intending to run on it. You can assign scopes that will grant you access to specific Cloud Platform services (like Cloud Storage for example).</p>
<p>You can read more about it by following <code>gcloud</code> online manual:</p>
<ul>
<li><em><a href="https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--scopes" rel="nofollow noreferrer">Cloud.google.com: SDK: Gcloud: Container: Clusters: Create: Scopes</a></em></li>
</ul>
<p>To add to the topic of authentication to Cloud Platform services:</p>
<blockquote>
<p>There are three ways to authenticate to Google Cloud services using service accounts from within GKE:</p>
<ol>
<li>Use Workload Identity</li>
</ol>
<p>Workload Identity is the recommended way to authenticate to Google Cloud services from GKE. Workload Identity allows you to configure Google Cloud service accounts using Kubernetes resources. If this fits your use case, it should be your first option. This example is meant to cover use cases where Workload Identity is not a good fit.</p>
<ol start="2">
<li>Use the default Compute Engine Service Account</li>
</ol>
<p>Each node in a GKE cluster is a Compute Engine instance. Therefore, applications running on a GKE cluster by default will attempt to authenticate using the &quot;Compute Engine default service account&quot;, and inherit the associated scopes.</p>
<p>This default service account may or may not have permissions to use the Google Cloud services you need. It is possible to expand the scopes for the default service account, but that can create security risks and is not recommended.</p>
<ol start="3">
<li>Manage Service Account credentials using Secrets</li>
</ol>
<p>Your final option is to create a service account for your application, and inject the authentication key as a Kubernetes secret. This will be the focus of this tutorial.</p>
<p>-- <em><a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/authenticating-to-cloud-platform" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Authenticating to Cloud Platform</a></em></p>
</blockquote>
