<p>I currently have a ROM, RWO persistent volume claim that I regularly use as a read only volume in a deployment that sporadically gets repopulated by some job using it as a read write volume while the deployment is scaled down to 0. However, since in-tree plugins will be deprecated in future versions of kubernetes, I'm planning to migrate this process to volumes using csi drivers.</p>
<p>In order to clarify my current use of this kind of volumes, I'll put a sample yaml configuration file using the basic idea:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test
spec:
  storageClassName: standard
  accessModes:
  - ReadOnlyMany
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: batch/v1
kind: Job
metadata:
  name: test
spec:
  template:
    spec:
      containers:
      - name: test
        image: busybox
        # Populate the volume
        command:
        - touch
        - /foo/bar
        volumeMounts:
        - name: test
          mountPath: /foo/
          subPath: foo
      volumes:
      - name: test
        persistentVolumeClaim:
          claimName: test
      restartPolicy: Never
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: test
  name: test
spec:
  replicas: 0
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: test
        image: busybox
        command:
        - sh
        - '-c'
        - |
          # Check the volume has been populated
          ls /foo/
          # Prevent the pod from exiting for a while
          sleep 3600
        volumeMounts:
        - name: test
          mountPath: /foo/
          subPath: foo
      volumes:
      - name: test
        persistentVolumeClaim:
          claimName: test
          readOnly: true
</code></pre>
<p>so the job populates the the volume and later the deployment is scaled up. However, replacing the <code>storageClassName</code> field <code>standard</code> in the persistent volume claim by <code>singlewriter-standard</code> does not even allow the job to run.</p>
<p>Is this some kind of bug? Is there some workaround to this using volumes using the csi driver?</p>
<p>If this is a bug, I'd plan to migrate to using sci drivers later; however, if this is not a bug, how should I migrate my current workflow since in-tree plugins will eventually be deprecated?</p>
<p><strong>Edit:</strong></p>
<p>The version of the kubernetes server is <code>1.17.9-gke.1504</code>. As for the storage classes, they are the <code>standard</code> and <code>singlewriter-standard</code> default storage classes:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
    kubernetes.io/cluster-service: &quot;true&quot;
  name: standard
parameters:
  type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    components.gke.io/component-name: pdcsi-addon
    components.gke.io/component-version: 0.5.1
    storageclass.kubernetes.io/is-default-class: &quot;true&quot;
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: singlewriter-standard
parameters:
  type: pd-standard
provisioner: pd.csi.storage.gke.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
</code></pre>
<p>While the error is not shown in the job but in the pod itself (this is just for the <code>singlewriter-standard</code> storage class):</p>
<p><code>Warning FailedAttachVolume attachdetach-controller AttachVolume.Attach failed for volume &quot;...&quot; : CSI does not support ReadOnlyMany and ReadWriteOnce on the same PersistentVolume</code></p>

Raúl Arturo Chávez Sarmiento
<p>The message you encountered:</p>
<pre class="lang-sh prettyprint-override"><code>Warning FailedAttachVolume attachdetach-controller AttachVolume.Attach failed for volume &quot;...&quot; : CSI does not support ReadOnlyMany and ReadWriteOnce on the same PersistentVolume
</code></pre>
<p>is not a bug. The <code>attachdetach-controller</code> is showing this error as it doesn't know in which <code>accessMode</code> it should mount the volume:</p>
<blockquote>
<p>For [ReadOnlyMany, ReadWriteOnce] PV, the external attacher simply does not know if the attachment is going to be consumed as read-only(-many) or as read-write(-once)</p>
<p>-- <em><a href="https://github.com/kubernetes-csi/external-attacher/issues/153#issuecomment-500347886" rel="nofollow noreferrer">Github.com: Kubernetes CSI: External attacher: Issues: 153</a></em></p>
</blockquote>
<p>I encourage you to check the link above for a full explanation.</p>
<hr />
<blockquote>
<p>I currently have a ROM, RWO persistent volume claim that I regularly use as a read only volume in a deployment that sporadically gets repopulated by some job using it as a read write volume</p>
</blockquote>
<p>You can combine the steps from below guides:</p>
<ul>
<li>Turn on the <code>CSI</code> Persistent disk driver in GKE
<ul>
<li><em><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Gce-pd-csi-driver</a></em></li>
</ul>
</li>
<li>Create a <code>PVC</code> with <code>pd.csi.storage.gke.io</code> provisioner (you will need to modify <code>YAML</code> definitions with <code>storageClassName: singlewriter-standard</code>):
<ul>
<li><em><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/readonlymany-disks" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Readonlymany disks</a></em></li>
</ul>
</li>
</ul>
<p>Citing the documentation on steps to take (from <code>ReadOnlyMany</code> guide) that should fulfill the setup you've shown:</p>
<blockquote>
<p>Before using a persistent disk in read-only mode, you must format it.</p>
<p>To format your persistent disk:</p>
<ul>
<li>Create a persistent disk <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/preexisting-pd" rel="nofollow noreferrer">manually</a> or by using <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#dynamic_provisioning" rel="nofollow noreferrer">dynamic provisioning</a>.</li>
<li>Format the disk and populate it with data. To format the disk, you can:
<ul>
<li>Reference the disk as a <code>ReadWriteOnce</code> volume in a Pod. Doing this results in GKE automatically formatting the disk, and enables the Pod to pre-populate the disk with data. When the Pod starts, make sure the Pod writes data to the disk.</li>
<li>Manually mount the disk to a VM and format it. Write any data to the disk that you want. For details, see <a href="https://cloud.google.com/compute/docs/disks/add-persistent-disk#formatting" rel="nofollow noreferrer">Persistent disk formatting</a>.</li>
</ul>
</li>
<li>Unmount and detach the disk:
<ul>
<li>If you referenced the disk in a Pod, delete the Pod, wait for it to terminate, and wait for the disk to automatically detach from the node.</li>
<li>If you mounted the disk to a VM, detach the disk using <code>gcloud compute instances detach-disk</code>.</li>
</ul>
</li>
<li>Create Pods that access the volume as <code>ReadOnlyMany</code> as shown in the following section.</li>
</ul>
<p>-- <em><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/readonlymany-disks" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Readonlymany disks</a></em></p>
</blockquote>
<hr />
<p>Additional resources:</p>
<ul>
<li><em><a href="https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/container-storage-interface.md" rel="nofollow noreferrer">Github.com: Kubernetes: Design proposals: Storage: CSI</a></em></li>
<li><em><a href="https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/" rel="nofollow noreferrer">Kubernetes.io: Blog: Container storage interface</a></em></li>
<li><em><a href="https://kubernetes-csi.github.io/docs/drivers.html" rel="nofollow noreferrer">Kubernetes-csi.github.io: Docs: Drivers</a></em></li>
</ul>
<hr />
<h3>EDIT</h3>
<p>Following the official documentation:</p>
<ul>
<li><em><a href="https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/readonlymany-disks" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: How to: Persistent volumes: Readonlymany disks</a></em></li>
</ul>
<p>Please treat it as an example.</p>
<p>Dynamically create a <code>PVC</code> that will be used with <code>ReadWriteOnce</code> accessMode:</p>
<p><code>pvc.yaml</code></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-rwo
spec:
  storageClassName: singlewriter-standard
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 81Gi
</code></pre>
<p>Run a <code>Pod</code> with a <code>PVC</code> mounted to it:</p>
<p><code>pod.yaml</code></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: busybox-pvc
spec:
  containers:
  - image: k8s.gcr.io/busybox
    name: busybox
    command:
      - &quot;sleep&quot;
      - &quot;36000&quot;
    volumeMounts:
    - mountPath: /test-mnt
      name: my-volume
  volumes:
  - name: my-volume
    persistentVolumeClaim:
      claimName: pvc-rwo
</code></pre>
<p>Run following commands:</p>
<ul>
<li><code>$ kubectl exec -it busybox-pvc -- /bin/sh</code></li>
<li><code>$ echo &quot;Hello there!&quot; &gt; /test-mnt/hello.txt</code></li>
</ul>
<p>Delete the <code>Pod</code> and wait for the drive to be unmounted. Please do not delete <code>PVC</code> as deleting it:</p>
<blockquote>
<p>When you delete a claim, the corresponding PersistentVolume object and the provisioned Compute Engine persistent disk are also deleted.</p>
<p>-- <em><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#dynamic_provisioning" rel="nofollow noreferrer">Cloud.google.com: Kubernetes Engine: Persistent Volumes: Dynamic provisioning</a></em></p>
</blockquote>
<hr />
<p>Get the name (it's in <code>VOLUME</code> column) of the earlier created disk by running:</p>
<ul>
<li><code>$ kubectl get pvc</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS            AGE
pvc-rwo   Bound    pvc-11111111-2222-3333-4444-555555555555   81Gi       RWO            singlewriter-standard   52m
</code></pre>
<p>Create a <code>PV</code> and <code>PVC</code> with following definition:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-rox
spec:
  storageClassName: singlewriter-standard
  capacity:
    storage: 81Gi
  accessModes:
    - ReadOnlyMany
  claimRef:
    namespace: default
    name: pvc-rox # &lt;-- important
  gcePersistentDisk:
    pdName: &lt;INSERT HERE THE DISK NAME FROM EARLIER COMMAND&gt; 
    # pdName: pvc-11111111-2222-3333-4444-555555555555 &lt;- example
    fsType: ext4
    readOnly: true
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-rox # &lt;-- important
spec:
  storageClassName: singlewriter-standard
  accessModes:
    - ReadOnlyMany
  resources:
    requests:
      storage: 81Gi
</code></pre>
<p>You can test if your disk is in <code>ROX</code> accessMode when the spawned <code>Pods</code> were scheduled on multiple nodes and all of them have the <code>PVC</code> mounted:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 15
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - mountPath: /test-mnt
          name: volume-ro
          readOnly: true
      volumes:
      - name: volume-ro
        persistentVolumeClaim:
          claimName: pvc-rox
          readOnly: true
</code></pre>
<ul>
<li><code>$ kubectl get deployment nginx</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   15/15   15           15          3m1s
</code></pre>
<ul>
<li><code>$ kubectl exec -it nginx-6c77b8bf66-njhpm  -- cat /test-mnt/hello.txt</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>Hello there!
</code></pre>
