<p>In a Azure AKS kubernetes cluster, after a cluster version upgrade the nodepool nodes, I have a PV that has this node affinity:</p>
<pre><code>Node Affinity:
  Required Terms:
    Term 0:        failure-domain.beta.kubernetes.io/region in [westeurope]
</code></pre>
<p>The nodes don't have the label so the Deployment creates a Pod that cannot be scheduled for the corresponding PVC for this PV. The Pod is never started:</p>
<pre><code>  Warning  FailedScheduling  15m    default-scheduler  0/3 nodes are available: 3 node(s) had volume node affinity conflict. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
</code></pre>
<p>How can I add the label to the node or remove the label from the PV? I have tried to add the label to the node but I get:</p>
<pre><code>Error from server: admission webhook &quot;aks-node-validating-webhook.azmk8s.io&quot; denied the request: (UID: 931bf139-1579-4e96-b164-e4e6e2fdae65) User is trying to add or update a non-changeable system label (failure-domain.beta.kubernetes.io/region:westeurope). This action is denied..
</code></pre>
<p>Is the only solution to backup and restore the PV into a new one that does not have that deprecated label? What would the best process to do it (or any alternative solution)</p>

<p>We had the same problem. How we resolved it:</p>
<ol>
<li>Copy output from &quot;kubectl get pvc&quot; to get the link between the pvc and the pv.</li>
<li>Locate the disk in azure portal and create a snapshot of the disk. (In the MC_ resource group to the aks</li>
<li>Edit deployment in kubernetes and set replicacout to 0. Save and see that the pod are stopping ang removed.</li>
<li>Delete the PVC for this pod.</li>
<li>Edit deployment in kubernetes and set replicacout to 1. Save and see that there is a new PVC and a new PV created.</li>
<li>Edit deployment again and set replicacount to 0.</li>
<li>Locate the new disk in azure portal. Use &quot;kubectl get pvc&quot; to locate.</li>
<li>Delete new disk in azure portal</li>
<li>Locate snapshot created in pt 2.</li>
<li>Create a new disk based on the snapshot. New disk should have the same name as the disk deleted in pt 6.</li>
<li>Edit deployment in kubernetes and set replicacount to 1.
It should now start using the old disk with the new pvc and pv.</li>
</ol>
<p>Take backup of what ever referances and disk you can before starting</p>
