<p>Kubernetes version:
V1.22.2</p>
<p>Cloud Provider Vsphere version 6.7</p>
<p>Architecture:</p>
<ul>
<li>3 Masters</li>
<li>15 Workers</li>
</ul>
<p>What happened:
One of the pods for some &quot;unknown&quot; reason went down, and when we try to lift him up, it couldn't attach the existing PVC.
This only happened to a specific pod, all the others didn't have any kind of problem.</p>
<p>What did you expect to happen:
Pods should dynamically assume PVCs</p>
<p>Validation:
First step: The connection to Vsphere has been validated, and we have confirmed that the PVC exists.
Second step: The Pod was restarted (Statefulset 1/1 replicas) to see if the pod would rise again and assume the pvc, but without success.
Third step: Made a restart to the services (kube-controller, kube-apiserve, etc)
Last step: All workers and masters were rebooted but without success, each time the pod was launched it had the same error &quot;&quot;Multi-Attach error for volume &quot;pvc......&quot; Volume is already exclusively attached to one node and can't be attached to another&quot;&quot;</p>
<p>When I delete a pod and try to recreate it, I get this warning:
Multi-Attach error for volume &quot;pvc-xxxxx&quot; The volume is already exclusively attached to a node
and cannot be attached to another</p>
<p>Anything else we need to know:
I have a cluster (3 master and 15 nodes)</p>
<p>Temporary resolution:
Erase the existing PVC and launch the pod again to recreate the PVC.
Since this is data, it is not the best solution to delete the existing PVC.</p>
<blockquote>
<p><strong>Multi-Attach error for volume &quot;pvc-xxx&quot; Volume is already
exclusively attached to one node and can't be attached to another</strong></p>
</blockquote>
<p><a href="https://i.stack.imgur.com/Q68qn.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Q68qn.png" alt="ERROR" /></a></p>

<p>A longer term solution is referring to 2 facts:</p>
<ol>
<li><p>You're using <code>ReadWriteOnce</code> <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" rel="nofollow noreferrer">access mode</a> where the volume can be mounted as read-write by a single node.</p>
</li>
<li><p>Pods might be schedule by K8S Scheduler on a different node for multiple reason.</p>
</li>
</ol>
<p>Consider switching to <code>ReadWriteMany</code> where the volume can be mounted as read-write by many nodes.</p>
