<p>I am working on setting up a victoria metrics cluster in minikube to monitor my servers.</p>
<p>In this cluster I am using prometheus to send data to vminsert which further sends it to vmstorage and i have used vmselect to fetch the data and show it on grafana.</p>
<p>Data flow is like =&gt;&gt;&gt; prometheus---&gt; vminsert ---&gt; vmstorage &lt;--- vmselect &lt;--- grafana</p>
<p>Vminsert and vmselect is stateless while vmstorage is stateful.
Initially I have 5 replicas of vmstorage.
When I apply all the files everything is working fine and I am able to see the data on grafana.</p>
<p>I have created a storageclass for this purpose.</p>
<pre><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: vmstorage-sc
provisioner: k8s.io/minikube-hostpath
reclaimPolicy: Retain

</code></pre>
<p>After that I have also created a pvc for this. This pvc is using the above storageclass. I have used this pvc in my vmstorage deployment.</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-vms
  namespace: vms
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Filesystem
  resources:
    requests:
      storage: 50Mi
  storageClassName: vmstorage-sc
</code></pre>
<p>This is my vmstorage deployment file</p>
<pre><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vmstorage
  namespace: vms
spec:
  serviceName: vmstorage
  replicas: 5
  selector:
    matchLabels:
      app: vmstorage
  template:
    metadata:
      labels:
        app: vmstorage
    spec:
      containers:
      - name: vmstorage
        image: victoriametrics/vmstorage
        imagePullPolicy: &quot;IfNotPresent&quot;
        args:
        - -retentionPeriod=1 
        ports:
        - containerPort: 8400
          name: vminsert
        - containerPort: 8401
          name: vmselect
        volumeMounts:
        - name: mypvc
          mountPath: /victoria-metrics-data
      volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: pvc-vms
---
apiVersion: v1
kind: Service
metadata:
  name: vmstorage
  namespace: vms
spec:
  ports:
  - name: vminsert
    port: 8400
    targetPort: 8400
  - name: vmselect
    port: 8401
    targetPort: 8401
  selector:
    app: vmstorage

</code></pre>
<p>I am adding other files as well
this is my vminsert file</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: vminsert
  namespace: vms
spec:
  replicas: 4
  selector:
    matchLabels:
      app: vminsert
  template:
    metadata:
      labels:
        app: vminsert
    spec:
      containers:
      - name: vminsert
        image: victoriametrics/vminsert
        args:
        - &quot;-maxConcurrentInserts=64&quot;
        - &quot;-insert.maxQueueDuration=5m&quot;
        - &quot;-replicationFactor=5&quot;
#       - &quot;-dedup.minScrapeInterval=1ms&quot;
        - -storageNode=vmstorage-0.vmstorage.vms.svc.cluster.local:8400
        - -storageNode=vmstorage-1.vmstorage.vms.svc.cluster.local:8400
        - -storageNode=vmstorage-2.vmstorage.vms.svc.cluster.local:8400
        - -storageNode=vmstorage-3.vmstorage.vms.svc.cluster.local:8400
        - -storageNode=vmstorage-4.vmstorage.vms.svc.cluster.local:8400
        ports:
        - containerPort: 8480
          name: http-insert
---
apiVersion: v1
kind: Service
metadata:
  name: vminsert
  namespace: vms
spec:
  ports:
  - name: http
    port: 8480
    targetPort: 8480
  selector:
    app: vminsert
</code></pre>
<p>And this is my vmselect file</p>
<pre><code>apiVersion : apps/v1
kind : Deployment
metadata :
   name : vmselect
   namespace : vms
spec :
   replicas : 2
   selector :
     matchLabels :
       app : vmselect
   template :
     metadata :
       labels :
         app : vmselect
     spec :
       containers :
       - name : vmselect
         image : victoriametrics/vmselect
         args :
         - -search.maxUniqueTimeseries=1000000
         - -storageNode=vmstorage-2.vmstorage.vms.svc.cluster.local:8401
         - -storageNode=vmstorage-1.vmstorage.vms.svc.cluster.local:8401
#         - -storageNode=vmstorage-3.vmstorage.vms.svc.cluster.local:8401
         ports :
         - containerPort : 8481
           name : http-select
---
apiVersion: v1
kind: Service
metadata:
  name: vmselect
  namespace: vms
spec:
  ports:
  - name: http
    port: 8481
    targetPort: 8481
  selector:
    app: vmselect   
</code></pre>
<p><strong>All the pods are having same data.</strong>
Because of the replication factor I mentioned in vminsert file.</p>
<p>But the problem arises <strong>when I increase the number of replicas of vmstorage. When I make it 6, a new pod is created but it has no data</strong>, it is just a fresh pod without any data.
Or <strong>if I delete a pod, a new pod automatically creates but this pod also have no data.</strong></p>
<p>I want to achieve two things</p>
<ol>
<li>Whenever I increase the number of replica, the new pod should have same data as other pods.</li>
<li>If somehow a pod gets restarted then that pod should also have same data as others.</li>
</ol>
<p>In this setup I need to store the data of at least 1 month. So it is very important for me that if any of pod restarts or I scale up the number of pods then data should exists .</p>

Akash Prajapati
<p>Base on your vminsert file, a recommendation on increasing your</p>
<pre><code>&quot;-replicationFactor=5&quot;
</code></pre>
<p>to 6 and then adding another</p>
<pre><code>-storageNode=vmstorage-x.vmstorage.vms.svc.cluster.local:8400
</code></pre>
<p>This may fix the concern when a new pod is created without data.</p>
