<h3>Question:</h3>
<p>I have an elastic cluster using static NFS PV. How I can move those data to another server has more space <code>(already did, just copy it)</code> and <strong>keep the application apply new backup data without reinstalling everything</strong>.
I use Rancher on Centos 7 (without VM Sphere).</p>
<h3>What I Try:</h3>
<ul>
<li>Update <strong>PV</strong> <code>NFS path</code> and <code>limitStorage</code> to new backup server but k8s don't allow it.</li>
<li>Update <strong>existing PVC</strong> using new <strong>PV</strong> on a new backup server but k8s still don't allow.</li>
</ul>
<p>error message:</p>
<blockquote>
<p># persistentvolumeclaims &quot;data-core-container-log-elasticsearch-data-0&quot; was not valid:<br />
# * spec: Forbidden: is immutable after creation except resources.requests for bound claims</p>
</blockquote>

Đinh Anh Huy
<p>There might be several solutions, this is what worked out for me taking everything that you mentioned in the above scenario.</p>

<p>Taken a stateful set with nfs volume claim at <code>10.20.4.101</code>, assuming nfs drive is filled up. I relocated and copied all the data to another vm <code>10.20.4.102</code>.</p>

<p>Now while I keep my old configurations alive, I created a new PV with <code>10.20.4.102</code> in the same namespace with a different label than the original one something like this </p>

<pre><code>New Settings
</code></pre>

<pre><code>metadata:
  name: my-pv-1
  namespace: pv-test
  labels:
    volume-type: pv-1
</code></pre>

<pre><code>Old Settings
</code></pre>

<pre><code>metadata:
  name: my-pv
  namespace: pv-test
  labels:
    volume-type: pv
</code></pre>

<p>This creates a new PV in your namespace you can see in the <code>kubectl get pv</code> with status <code>Available</code> while other as Bound</p>

<p>Now update your statefulset yaml file fields <code>volume-type</code> to the new one same as the label in new <code>pv</code> and also the change the name in <code>volumeClaimTemplates</code> to something new. Do not apply this settings now.</p>

<pre><code>New
</code></pre>

<pre><code>volumeClaimTemplates:
  - metadata:
      name: pv-data-1
      namespace: pv-test
    selector:
       matchLabels:
       volume-type: pv-1
</code></pre>

<pre><code>Old
</code></pre>

<pre><code>volumeClaimTemplates:
  - metadata:
      name: pv-data
      namespace: pv-test
    selector:
       matchLabels:
       volume-type: pv
</code></pre>

<p>As you can surely cannot apply directly which would throw an error something like <code>Forbidden: updates to statefulset spec for fields other than `replicas`, `template` and `updateStrategy` are forbidden.</code>. </p>

<p>Now either you can delete and recreate the whole <code>statefulset</code> with a slight downtime. Or you can do this small trick with <code>--cascade=false</code> flag</p>

<pre><code>kubectl delete statefulset mystatefulset -n pv-test --cascade=false
</code></pre>

<p>This will just delete the statefulset not the pods within the stateful set, if you keep a watch on all the resources in the namespace it will delete the statefulset but not the pods or anyresources. Keeping all the applications still accessing to the applications running.</p>

<p>Now <code>apply</code> the updated statefulset this will create the new statefulset with a different pvc. But your pods will be still referring to the old pvc and pv.</p>

<p>Now just delete the pod using <code>kubectl delete pod mypod-0 -n pv-test</code></p>

<p>This will delete the pod, but statefulset in the background creates a new pod with new pvc as this gets deleted. Now if you kubectl pv and pvc you will observe there will be an additional pvc and pv with <code>Available</code> will be turned into <code>Bound</code> and will be claimed by the pvc. </p>

<p>Manually delete all the pods, and the statefulset takes care of recreation. After everything is done, manually delete the old pvc first and then old pv later. </p>

<p>You might have tried all of this and know all of them, just to keep it clear I wrote all the steps here instead of vaguely explaining.</p>

<p>Hope this is helpful.</p>
