title: Kubernetes persistent volume data corrupted after multiple pod deletions
body: <p>I am stuggling with a simple one replica deployment of the <a href="https://hub.docker.com/r/eventstore/eventstore/" rel="nofollow noreferrer">official event store image</a> on a Kubernetes cluster. I am using a persistent volume for the data storage. </p>
  
  <pre><code>apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: my-eventstore
  spec:
    strategy:
      type: Recreate
    replicas: 1
    template:
      metadata:
        labels:
          app: my-eventstore
      spec:
        imagePullSecrets:
          - name: runner-gitlab-account
        containers:
          - name: eventstore
            image: eventstore/eventstore
        env:
          - name: EVENTSTORE_DB
            value: "/usr/data/eventstore/data"
          - name: EVENTSTORE_LOG
            value: "/usr/data/eventstore/log"
        ports:
          - containerPort: 2113
          - containerPort: 2114
          - containerPort: 1111
          - containerPort: 1112
        volumeMounts:
          - name: eventstore-storage
            mountPath: /usr/data/eventstore
    volumes:
      - name: eventstore-storage
        persistentVolumeClaim:
          claimName: eventstore-pv-claim
  </code></pre>
  
  <p>And this is the yaml for my persistent volume claim:</p>
  
  <pre><code>apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: eventstore-pv-claim
  spec:
    accessModes:
      - ReadWriteOnce
   resources:
      requests:
        storage: 1Gi
  </code></pre>
  
  <p>The deployments work fine. It's when I tested for durability that I started to encounter a problem. I delete a pod to force actual state from desired state and see how Kubernetes reacts.</p>
  
  <p>It immediately launched a new pod to replace the deleted one. And the admin UI was still showing the same data. But after deleting a pod for the second time, the new pod did not come up. I got an error message that said "record too large" that indicated corrupted data according to this discussion. <a href="https://groups.google.com/forum/#!topic/event-store/gUKLaxZj4gw" rel="nofollow noreferrer">https://groups.google.com/forum/#!topic/event-store/gUKLaxZj4gw</a></p>
  
  <p>I tried again for a couple of times. Same result every time. After deleting the pod for the second time the data is corrupted. This has me worried that an actual failure will cause similar result.</p>
  
  <p>However, when deploying new versions of the image or scaling the pods in the deployment to zero and back to one no data corruption occurs. After several tries everything is fine. Which is odd since that also completely replaces pods (I checked the pod id's and they changed).</p>
  
  <p>This has me wondering if deleting a pod using kubectl delete is somehow more forcefull in the way that a pod is terminated. Do any of you have similar experience? Of insights on if/how delete is different? Thanks in advance for your input.</p>
  
  <p>Regards,</p>
  
  <p>Oskar</p>