<p>I've been debugging a 10min downtime of our service for some hours now, and I seem to have found the cause, but not the reason for it. Our redis deployment in kubernetes was down for quite a while, causing neither django nor redis to be able to reach it. This caused a bunch of jobs to be lost.</p>
<p>There are no events for the redis deployment, but here are the first logs before and after the reboot:</p>
<p>before:
<a href="https://i.stack.imgur.com/9CGeJ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9CGeJ.png" alt="enter image description here" /></a></p>
<p>after:
<a href="https://i.stack.imgur.com/l6iYA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/l6iYA.png" alt="enter image description here" /></a></p>
<p>I'm also attaching the complete redis yml at the bottom. We're using GKE Autopilot, so I guess something caused the pod to reboot? Resource usage is a lot lower than requested, at about 1% for both CPU and memory. Not sure what's going on here. I also couldn't find an annotation to tell Autopilot to leave a specific deployment alone</p>
<p>redis.yml:</p>
<pre class="lang-yaml prettyprint-override"><code>---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-disk
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: gce-ssd
  resources:
    requests:
      storage: &quot;2Gi&quot;
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
    - port: 6379
      name: redis
  clusterIP: None
  selector:
    app: redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      volumes:
        - name: redis-volume
          persistentVolumeClaim:
            claimName: redis-disk
            readOnly: false
      terminationGracePeriodSeconds: 5
      containers:
        - name: redis
          image: redis:6-alpine
          command: [&quot;sh&quot;]
          args: [&quot;-c&quot;, 'exec redis-server --requirepass &quot;$REDIS_PASSWORD&quot;']
          resources:
            requests:
              memory: &quot;512Mi&quot;
              cpu: &quot;500m&quot;
              ephemeral-storage: &quot;1Gi&quot;
          envFrom:
            - secretRef:
                name: env-secrets
          volumeMounts:
            - name: redis-volume
              mountPath: /data
              subPath: data
</code></pre>

<p><code>PersistentVolumeClaim</code> is an object in <em>kubernetes</em> allowing to decouple storage resource requests from actual resource provisioning done by its associated <code>PersistentVolume</code> part.</p>
<p>Given:</p>
<ul>
<li>no declared <code>PersistentVolume</code> object</li>
<li><a href="https://kubernetes.io/docs/concepts/storage/dynamic-provisioning/#using-dynamic-provisioning" rel="nofollow noreferrer">and Dynamic Provisioning being enabled on your cluster</a></li>
</ul>
<p><em>kubernetes</em> will try to <strong>dynamically provision</strong> a suitable persistent disk for you suitable for the underlying infrastructure being a <a href="https://kubernetes.io/docs/concepts/storage/volumes/#gcepersistentdisk" rel="nofollow noreferrer"><em>Google Compute Engine Persistent Disk</em></a> in you case based on the requested <em>storage class</em> (<em>gce-ssd</em>).</p>
<p>The claim will result then in an SSD-like Persistent Disk to be automatically provisioned for you and <strong>once the claim is deleted</strong> (the requesting pod is deleted due to downscale), <strong>the volume is destroyed</strong>.</p>
<p>To overcome this issue and avoid precious data loss, you should have two alternatives:</p>
<h2>At the PersistentVolumeClaim level</h2>
<p>To avoid data loss once the Pod and its PVC are deleted, you can set the <code>persistentVolumeReclaimPolicy</code> parameter to <code>Retain</code> on the PVC object:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redis-disk
spec:
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: gce-ssd
  resources:
    requests:
      storage: &quot;2Gi&quot;
</code></pre>
<p>This allows for the persistent volume to go back to the <code>Released</code> state and the underlying data can be <strong>manually</strong> backed up.</p>
<h2>At the StorageClass level</h2>
<p>As a general recommendation, you should set the <code>reclaimPolicy</code> parameter to <code>Retain</code> (default is <code>Delete</code>) for your used <code>StorageClass</code>:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  reclaimPolicy: Retain
  replication-type: regional-pd
  volumeBindingMode: WaitForFirstConsumer
</code></pre>
<p>Additional parameters are recommended:</p>
<ul>
<li><code>replication-type</code>: should be set to <code>regional-pd</code> to allow zonal replication</li>
<li><code>volumeBindingMode</code>: set to <code>WaitForFirstConsumer</code> to allow for first consumer dictating the zonal replication topology</li>
</ul>
<p>You can read more on all above <a href="https://kubernetes.io/docs/concepts/storage/storage-classes/" rel="nofollow noreferrer"><code>StorageClass</code> parameters in the <em>kubernetes</em> documentation</a>.</p>
<p>A <code>PersistentVolume</code> with same storage class name is then declared:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolume
metadata:
  name: ssd-volume
spec:
  storageClassName: &quot;ssd&quot;
  capacity:
    storage: 2G
  accessModes:
    - ReadWriteOnce
  gcePersistentDisk:
    pdName: redis-disk
</code></pre>
<p>And the <code>PersistentVolumeClaim</code> would only declare the requested <code>StorageClass</code> name:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ssd-volume-claim
spec:
  storageClassName: &quot;ssd&quot;
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: &quot;2Gi&quot;

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  labels:
    app: redis
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      volumes:
        - name: redis-volume
          persistentVolumeClaim:
            claimName: ssd-volume-claim
            readOnly: false
</code></pre>
<p>This objects declaration would prevent any failures or scale down operations from destroying the created PV either created manually by cluster administrators or dynamically using <em>Dynamic Provisioning</em>.</p>
