<p>I have a three node GCE cluster and a single-pod GKE deployment with three replicas. I created the PV and PVC like so:</p>

<pre><code># Create a persistent volume for web content
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nginx-content
  labels:
    type: local
spec:
  capacity:
    storage: 5Gi
  accessModes:
   - ReadOnlyMany
  hostPath:
    path: "/usr/share/nginx/html"
--
# Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-content-claim
  annotations:
    volume.alpha.kubernetes.io/storage-class: default
spec:
  accessModes: [ReadOnlyMany]
  resources:
    requests:
      storage: 5Gi
</code></pre>

<p>They are referenced in the container spec like so:</p>

<pre><code>    spec:
      containers:
      - image: launcher.gcr.io/google/nginx1
        name: nginx-container
        volumeMounts:
          - name: nginx-content
            mountPath: /usr/share/nginx/html
        ports:
          - containerPort: 80
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim
</code></pre>

<p>Even though I created the volumes as ReadOnlyMany, only one pod can mount the volume at any given time. The rest give "Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE". How can I make it so all three replicas read the same web content from the same volume?</p>

asdfaewefgav
<p>First I'd like to point out one fundamental discrapency in your configuration. Note that when you use your <code>PersistentVolumeClaim</code> defined as in your example, you don't use your <code>nginx-content</code> <code>PersistentVolume</code> at all. You can easily verify it by running:</p>
<pre><code>kubectl get pv
</code></pre>
<p>on your <strong>GKE cluster</strong>. You'll notice that apart from your manually created <code>nginx-content</code> <code>PV</code>, there is another one, which was automatically provisioned based on the <code>PVC</code> that you applied.</p>
<p>Note that in your <code>PersistentVolumeClaim</code> definition you're explicitely referring the <code>default</code> storage class which has nothing to do with your manually created <code>PV</code>. Actually even if you completely omit the annotation:</p>
<pre><code>annotations:
        volume.alpha.kubernetes.io/storage-class: default
</code></pre>
<p>it will work exactly the same way, namely the <code>default</code> storage class will be used anyway. Using the default storage class on <strong>GKE</strong> means that <strong>GCE Persistent Disk</strong> will be used as your volume provisioner. You can read more about it <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#storageclasses" rel="noreferrer">here</a>:</p>
<blockquote>
<p>Volume implementations such as gcePersistentDisk are configured
through StorageClass resources. GKE creates a default StorageClass for
you which uses the standard persistent disk type (ext4). The default
StorageClass is used when a PersistentVolumeClaim doesn't specify a
StorageClassName. You can replace the provided default StorageClass
with your own.</p>
</blockquote>
<p>But let's move on to the solution of the problem you're facing.</p>
<h3>Solution:</h3>
<p>First, I'd like to emphasize <strong>you don't have to use any NFS-like filesystems to achive your goal</strong>.</p>
<p>If you need your <code>PersistentVolume</code> to be available in <code>ReadOnlyMany</code> mode, <strong>GCE Persistent Disk</strong> is a perfect solution that entirely meets your requirements.</p>
<p>It can be mounted in <code>ro</code> mode by many <code>Pods</code> at the same time and what is even more important by many <code>Pods</code>, scheduled on different <strong>GKE</strong> <code>nodes</code>. Furthermore it's really simple to configure and it works on <strong>GKE</strong> out of the box.</p>
<p>In case you want to use your storage in <code>ReadWriteMany</code> mode, I agree that something like NFS may be the only solution as <strong>GCE Persistent Disk</strong> doesn't provide such capability.</p>
<p>Let's take a closer look how we can configure it.</p>
<p>We need to start from defining our <code>PVC</code>. This step was actually already done by yourself but you got lost a bit in further steps. Let me explain how it works.</p>
<p>The following configuration is correct (as I mentioned <code>annotations</code> section can be omitted):</p>
<pre><code># Request a persistent volume for web content
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: nginx-content-claim
spec:
  accessModes: [ReadOnlyMany]
  resources:
    requests:
      storage: 5Gi
</code></pre>
<p>However I'd like to add one important comment to this. You said:</p>
<blockquote>
<p>Even though I created the volumes as ReadOnlyMany, only one pod can
mount the volume at any given time.</p>
</blockquote>
<p>Well, actually <strong>you didn't</strong>. I know it may seem a bit tricky and somewhat surprising but this is not the way how defining <code>accessModes</code> really works. In fact it's a widely misunderstood concept. First of all <strong>you cannot define access modes in <code>PVC</code></strong> in a sense of putting there the constraints you want. Supported <strong>access modes</strong> are inherent feature of a particular storage type. They are already defined by the storage provider.</p>
<p>What you actually do in <code>PVC</code> definition is requesting a <code>PV</code> that supports the particular access mode or access modes. Note that it's in a form of <strong>a list</strong> which means you may provide many different access modes that you want your <code>PV</code> to support.</p>
<p>Basically it's like saying: <em>&quot;Hey! Storage provider! Give me a volume that supports <code>ReadOnlyMany</code> mode.&quot;</em> You're asking this way for a storage that will satisfy your requirements. Keep in mind however that you can be given more than you ask. And this is also our scenario when asking for a <code>PV</code> that supports <code>ReadOnlyMany</code> mode in <strong>GCP</strong>. It creates for us a <code>PersistentVolume</code> which meets our requirements we listed in <code>accessModes</code> section but it also supports <code>ReadWriteOnce</code> mode. Although we didn't ask for something that also supports <code>ReadWriteOnce</code> you will probably agree with me that storage which has a built-in support for those two modes fully satisfies our request for something that supports <code>ReadOnlyMany</code>. So basically this is the way it works.</p>
<p>Your <code>PV</code> that was automatically provisioned by GCP in response for your <code>PVC</code> supports those two <code>accessModes</code> and if you don't specify explicitely in <code>Pod</code> or <code>Deployment</code> definition that you want to mount it in <strong>read-only</strong> mode, by default it is mounted in <strong>read-write</strong> mode.</p>
<p>You can easily verify it by attaching to the <code>Pod</code> that was able to successfully mount the <code>PersistentVolume</code>:</p>
<pre><code>kubectl exec -ti pod-name -- /bin/bash
</code></pre>
<p>and trying to write something on the mounted filesystem.</p>
<p>The error message you get:</p>
<pre><code>&quot;Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE&quot;
</code></pre>
<p>concerns specifically <strong>GCE Persistent Disk</strong> that is already mounted by one <strong>GKE</strong> <code>node</code> in <code>ReadWriteOnce</code> mode and it cannot be mounted by another <code>node</code> on which the rest of your <code>Pods</code> were scheduled.</p>
<p>If you want it to be mounted in <code>ReadOnlyMany</code> mode, you need to specify it explicitely in your <code>Deployment</code> definition by adding <code>readOnly: true</code> statement in the <code>volumes</code> section under <code>Pod's</code> template specification like below:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: &quot;/usr/share/nginx/html&quot;
          name: nginx-content
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim
          readOnly: true
</code></pre>
<p>Keep in mind however that to be able to mount it in <code>readOnly</code> mode, first we need to pre-populate such volume with data. Otherwise you'll see another error message, saying that unformatted volume cannot be mounted in read only mode.</p>
<p>The easiest way to do it is by creating a single <code>Pod</code> which will serve only for copying data which was already uploaded to one of our <strong>GKE nodes</strong> to our destination <code>PV</code>.</p>
<p>Note that pre-populating <code>PersistentVolume</code> with data can be done in many different ways. You can mount in such <code>Pod</code> only your <code>PersistentVolume</code> that you will be using in your <code>Deployment</code> and get your data using <code>curl</code> or <code>wget</code> from some external location saving it directly on your destination <code>PV</code>. It's up to you.</p>
<p>In my example I'm showing how to do it using additional <a href="https://kubernetes.io/docs/concepts/storage/volumes/#local" rel="noreferrer">local</a> volume that allows us to mount into our <code>Pod</code> a <code>directory</code>, <code>partition</code> or <code>disk</code> (in my example I use a directory <code>/var/tmp/test</code> located on one of my GKE nodes) available on one of our kubernetes nodes. It's much more flexible solution than <code>hostPath</code> as we don't have to care about scheduling such <code>Pod</code> to particular node, that contains the data. Specific <strong>node affinity</strong> rule is already defined in <code>PersistentVolume</code> and <code>Pod</code> is automatically scheduled on specific node.</p>
<p>To create it we need 3 things:</p>
<p><code>StorageClass</code>:</p>
<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
</code></pre>
<p><code>PersistentVolume</code> definition:</p>
<pre><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-pv
spec:
  capacity:
    storage: 10Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  storageClassName: local-storage
  local:
    path: /var/tmp/test
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - &lt;gke-node-name&gt;
</code></pre>
<p>and finally <code>PersistentVolumeClaim</code>:</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myclaim
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-storage
</code></pre>
<p>Then we can create our temporary <code>Pod</code> which will serve only for copying data from our <strong>GKE node</strong> to our <strong>GCE Persistent Disk</strong>.</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: mypod
spec:
  containers:
    - name: myfrontend
      image: nginx
      volumeMounts:
      - mountPath: &quot;/mnt/source&quot;
        name: mypd
      - mountPath: &quot;/mnt/destination&quot;
        name: nginx-content
  volumes:
    - name: mypd
      persistentVolumeClaim:
        claimName: myclaim
    - name: nginx-content
      persistentVolumeClaim:
        claimName: nginx-content-claim
</code></pre>
<p>Paths you can see above are not really important. The task of this <code>Pod</code> is only to allow us to copy our data to the destination <code>PV</code>. Eventually our <code>PV</code> will be mounted in completely different path.</p>
<p>Once the <code>Pod</code> is created and both volumes are successfully mounted, we can attach to it by running:</p>
<pre><code>kubectl exec -ti my-pod -- /bin/bash
</code></pre>
<p>Withing the <code>Pod</code> simply run:</p>
<pre><code>cp /mnt/source/* /mnt/destination/
</code></pre>
<p>That's all. Now we can <code>exit</code> and delete our temporary <code>Pod</code>:</p>
<pre><code>kubectl delete pod mypod
</code></pre>
<p>Once it is gone, we can apply our <code>Deployment</code> and our <code>PersistentVolume</code> finally can be mounted in <code>readOnly</code> mode by all the <code>Pods</code> located on various <strong>GKE nodes</strong>:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - mountPath: &quot;/usr/share/nginx/html&quot;
          name: nginx-content
      volumes:
      - name: nginx-content
        persistentVolumeClaim:
          claimName: nginx-content-claim
          readOnly: true
</code></pre>
<p>Btw. if you are ok with the fact that your <code>Pods</code> will be scheduled only on one particular node, you can give up on using <strong>GCE Persistent Disk</strong> at all and switch to the above mentioned <a href="https://kubernetes.io/docs/concepts/storage/volumes/#local" rel="noreferrer">local</a> volume. This way all your <code>Pods</code> will be able not only to read from it but also to write to it at the same time. The only caveat is that all those <code>Pods</code> will be running on a single node.</p>
