<p>Here is my overall goal:</p>

<ul>
<li><p>Have a MongoDB running</p></li>
<li><p>Persist the data through pod failures / updates etc</p></li>
</ul>

<p>The approach I’ve taken:</p>

<ul>
<li><p>K8S Provider: Digital Ocean</p></li>
<li><p>Nodes: 3</p></li>
<li><p>Create a PVC</p></li>
<li><p>Create a headless Service</p></li>
<li><p>Create a StatefulSet</p></li>
</ul>

<p>Here’s a dumbed down version of the config:</p>

<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: some-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: do-block-storage
---
apiVersion: v1
kind: Service
metadata:
  name: some-headless-service
  labels:
    app: my-app
spec:
  ports:
  - port: 27017
    name: my-app-database
  clusterIP: None
  selector:
    app: my-app
    tier: database
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-app-database
  labels:
    app: my-app
    tier: database
spec:
  serviceName: some-headless-service
  replicas: 1
  selector:
    matchLabels:
      app: my-app
      tier: database
  template:
    metadata:
      labels:
        app: my-app
        tier: database
    spec:
      containers:
      - name: my-app-database
        image: mongo:latest
        volumeMounts:
        - name: some-volume
          mountPath: /data
        ports:
        - containerPort: 27017
          name: my-app-database
      volumes:
      - name: some-volume
        persistentVolumeClaim:
          claimName: some-pvc
</code></pre>

<p>This is working as expected. I can spin down the replicas to 0:</p>

<p><code>kubectl scale —replicas=0 statefulset/my-app-database</code></p>

<p>Spin it back up:</p>

<p><code>kubectl scale —replicas=1 statefulset/my-app-database</code></p>

<p>And the data will persist..</p>

<p>But one time, as I was messing around by scaling the statefulset up and down, I was met with this error:</p>

<pre><code>Volume is already exclusively attached to one node and can't be attached to another
</code></pre>

<p>Being new to k8s, I deleted the PVC and “recreated” the same one:</p>

<pre><code>kubectl delete pvc some-pvc
kubectl apply -f persistent-volume-claims/
</code></pre>

<p>The <code>statefulset</code> spun back up with a new PV and the old PV was deleted as the <code>persistentVolumeReclaimPolicy</code> was set to <code>Delete</code> by default.</p>

<p>I set this new PV <code>persistentVolumeReclaimPolicy</code> to <code>Retain</code> to ensure that the data would not be automatically removed.. and I realized: I’m not sure how I’d reclaim that PV. Earlier to get through the “volume attachment” error, I deleted the PVC, which will just create another new PV with the setup I have, and now I’m left with my data in that <code>Released</code> PV.</p>

<p>My main questions are:</p>

<ul>
<li><p>Does this overall sound like the right approach for my goal?</p></li>
<li><p>Should I look into adding a <code>claimRef</code> to the dynamically created PV and then recreating a new PVC with that claimRef, as mentioned here: <a href="https://stackoverflow.com/questions/34282704/can-a-pvc-be-bound-to-a-specific-pv/34323691#34323691">Can a PVC be bound to a specific PV?</a></p></li>
<li><p>Should I be trying to get that fresh <code>statefulset</code> PVC to actually use that old PV?</p></li>
<li><p>Would it make sense to try to reattach the old PV to the correct node, and how would I do that?</p></li>
</ul>

<p>If your want to use <code>StatefulSet</code> with scalability, your storage should also support this, there are two way to handle this:</p>

<ul>
<li><p>If <code>do-block-storage</code> storage class supprt <code>ReadWriteMany</code>, then put all pod's data in single volume. </p></li>
<li><p>Each pod use a different volume, add <code>volumeClaimTemplate</code> to your <code>StatefulSet.spec</code>, 
then k8s will create PVC like <code>some-pvc-{statefulset_name}-{idx}</code> automatically:</p></li>
</ul>

<pre><code>spec:
  volumeClaimTemplates:
  - metadata:
      name: some-pvc
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      storageClassName: do-block-storage
</code></pre>

<p><strong>Update:</strong></p>

<p><code>StatefulSet</code> replicas <strong>Must</strong> deploy with <a href="https://docs.mongodb.com/manual/reference/replica-configuration/" rel="nofollow noreferrer">mongodb replication</a>, then each pod in <code>StatefulSet</code> will has same data storage. </p>

<p>So when container run <code>mongod</code> command, you must add option <code>--replSet={name}</code>. when all pods up, execute command <code>rs.initiate()</code> to tell mongodb how to handle data replication. When you scale up or down <code>StatefulSet</code>, execute command <code>rs.add()</code> or <code>rs.remove()</code> to tell mongodb members has changed.</p>
