<p>I know there have been already a lot of questions about this, and I read already most of them, but my problem does not seem to fit them.</p>
<p>I am running a postgresql from bitnami using a helm chart as described below. A clean setup is no problem and everything starts fine. But after some time, until now I could not find any pattern, the pod goes into CrashLoopBackOff and I cannot recover it whatever I try!</p>
<p>Helm uninstall/install does not fix the problem. The PVs seem to be the problem, but I do not know why. <em>And</em> I do not get any error message, which is the weird and scary part of it.</p>
<p>I use a minikube to run the k8s and helm v3.</p>
<ul>
<li>Helm chart: <a href="https://artifacthub.io/packages/helm/bitnami/postgresql/10.9.5" rel="nofollow noreferrer">https://artifacthub.io/packages/helm/bitnami/postgresql/10.9.5</a></li>
</ul>
<p>Here are the definitions and logs:</p>
<pre><code># Source: aposphere/charts/sessiondb/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: sessiondb
  labels:
    app.kubernetes.io/name: sessiondb
    helm.sh/chart: sessiondb-10.9.6
    app.kubernetes.io/instance: asdf
    app.kubernetes.io/managed-by: Helm
  annotations:
  namespace: default
spec:
  type: ClusterIP
  ports:
    - name: tcp-postgresql
      port: 5432
      targetPort: tcp-postgresql
  selector:
    app.kubernetes.io/name: sessiondb
    app.kubernetes.io/instance: asdf
    role: primary
---
# Source: aposphere/charts/sessiondb/templates/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: sessiondb
  labels:
    app.kubernetes.io/name: sessiondb
    helm.sh/chart: sessiondb-10.9.6
    app.kubernetes.io/instance: asdf
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: primary
  annotations:
  namespace: default
spec:
  serviceName: sessiondb-headless
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app.kubernetes.io/name: sessiondb
      app.kubernetes.io/instance: asdf
      role: primary
  template:
    metadata:
      name: sessiondb
      labels:
        app.kubernetes.io/name: sessiondb
        helm.sh/chart: sessiondb-10.9.6
        app.kubernetes.io/instance: asdf
        app.kubernetes.io/managed-by: Helm
        role: primary
        app.kubernetes.io/component: primary
    spec:      
      affinity:
        podAffinity:
          
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: sessiondb
                    app.kubernetes.io/instance: asdf
                    app.kubernetes.io/component: primary
                namespaces:
                  - &quot;default&quot;
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:
          
      securityContext:
        fsGroup: 1001
      automountServiceAccountToken: false
      containers:
        - name: sessiondb
          image: docker.io/bitnami/postgresql:11.13.0-debian-10-r33
          imagePullPolicy: &quot;IfNotPresent&quot;
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
          securityContext:
            runAsUser: 1001
          env:
            - name: BITNAMI_DEBUG
              value: &quot;false&quot;
            - name: POSTGRESQL_PORT_NUMBER
              value: &quot;5432&quot;
            - name: POSTGRESQL_VOLUME_DIR
              value: &quot;/bitnami/postgresql&quot;
            - name: PGDATA
              value: &quot;/bitnami/postgresql/data&quot;
            - name: POSTGRES_USER
              value: &quot;postgres&quot;
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgresql-root-password
                  key: postgresql-password
            - name: POSTGRES_DB
              value: &quot;session&quot;
            - name: POSTGRESQL_ENABLE_LDAP
              value: &quot;no&quot;
            - name: POSTGRESQL_ENABLE_TLS
              value: &quot;no&quot;
            - name: POSTGRESQL_LOG_HOSTNAME
              value: &quot;false&quot;
            - name: POSTGRESQL_LOG_CONNECTIONS
              value: &quot;false&quot;
            - name: POSTGRESQL_LOG_DISCONNECTIONS
              value: &quot;false&quot;
            - name: POSTGRESQL_PGAUDIT_LOG_CATALOG
              value: &quot;off&quot;
            - name: POSTGRESQL_CLIENT_MIN_MESSAGES
              value: &quot;error&quot;
            - name: POSTGRESQL_SHARED_PRELOAD_LIBRARIES
              value: &quot;pgaudit&quot;
          ports:
            - name: tcp-postgresql
              containerPort: 5432
          livenessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - exec pg_isready -U &quot;postgres&quot; -d &quot;dbname=session&quot; -h 127.0.0.1 -p 5432
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - -e
                - |
                  exec pg_isready -U &quot;postgres&quot; -d &quot;dbname=session&quot; -h 127.0.0.1 -p 5432
                  [ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
            initialDelaySeconds: 5
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          volumeMounts:
            - name: custom-init-scripts
              mountPath: /docker-entrypoint-initdb.d/
            - name: dshm
              mountPath: /dev/shm
            - name: data
              mountPath: /bitnami/postgresql
              subPath: 
      volumes:
        - name: custom-init-scripts
          configMap:
            name: sessiondb-scheme
        - name: dshm
          emptyDir:
            medium: Memory
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - &quot;ReadWriteOnce&quot;
        resources:
          requests:
            storage: &quot;8Gi&quot;
</code></pre>
<p>Logs of the container:</p>
<pre><code>% kubectl logs sessiondb-0
postgresql 10:09:01.48 
postgresql 10:09:01.48 Welcome to the Bitnami postgresql container
postgresql 10:09:01.49 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
postgresql 10:09:01.49 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues
postgresql 10:09:01.49 
postgresql 10:09:01.50 DEBUG ==&gt; Configuring libnss_wrapper...
postgresql 10:09:01.51 INFO  ==&gt; ** Starting PostgreSQL setup **
postgresql 10:09:01.54 INFO  ==&gt; Validating settings in POSTGRESQL_* env vars..
postgresql 10:09:01.55 INFO  ==&gt; Loading custom pre-init scripts...
postgresql 10:09:01.55 INFO  ==&gt; Initializing PostgreSQL database...
postgresql 10:09:01.56 DEBUG ==&gt; Ensuring expected directories/files exist...
postgresql 10:09:01.57 INFO  ==&gt; pg_hba.conf file not detected. Generating it...
postgresql 10:09:01.58 INFO  ==&gt; Generating local authentication configuration
postgresql 10:09:01.58 INFO  ==&gt; Deploying PostgreSQL with persisted data...
postgresql 10:09:01.60 INFO  ==&gt; Configuring replication parameters
postgresql 10:09:01.65 INFO  ==&gt; Configuring fsync
postgresql 10:09:01.71 INFO  ==&gt; Loading custom scripts...
postgresql 10:09:01.72 INFO  ==&gt; Loading user's custom files from /docker-entrypoint-initdb.d ...
postgresql 10:09:01.72 INFO  ==&gt; Starting PostgreSQL in background...
pg_ctl: directory &quot;/bitnami/postgresql/data&quot; is not a database cluster directory
</code></pre>
<p><em>Afterwards the container terminates, no more logs!</em></p>
<p>Logs of the init container:</p>
<pre><code>% kubectl logs sessiondb-0 init-chmod-data
+ chown 1001:1001 /bitnami/postgresql
+ mkdir -p /bitnami/postgresql/data
+ chmod 700 /bitnami/postgresql/data
+ find /bitnami/postgresql -mindepth 1 -maxdepth 1 -not -name conf -not -name .snapshot -not -name lost+found
+ xargs chown -R 1001:1001
+ chmod -R 777 /dev/shm
</code></pre>
<p>Permissions:</p>
<pre><code>I have no name!@sessiondb-0:/$ stat /bitnami/postgresql/data
  File: /bitnami/postgresql/data
  Size: 207         Blocks: 0          IO Block: 4096   directory
Device: 10301h/66305d   Inode: 712929      Links: 12
Access: (0700/drwx------)  Uid: ( 1001/ UNKNOWN)   Gid: ( 1001/ UNKNOWN)
Access: 2021-11-10 15:16:13.958633094 +0000
Modify: 2021-11-26 08:40:42.621884636 +0000
Change: 2021-11-26 10:37:47.844490933 +0000
 Birth: -
</code></pre>
<p>Describe the resources:</p>
<pre><code>Name:         sessiondb-0
Namespace:    default
Priority:     0
Node:         ip-10-0-1-112.eu-central-1.compute.internal/10.0.1.112
Start Time:   Fri, 26 Nov 2021 10:40:02 +0100
Labels:       app.kubernetes.io/component=primary
              app.kubernetes.io/instance=asdf
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=sessiondb
              controller-revision-hash=sessiondb-578ddf476b
              helm.sh/chart=sessiondb-10.9.6
              role=primary
              statefulset.kubernetes.io/pod-name=sessiondb-0
Annotations:  &lt;none&gt;
Status:       Running
IP:           172.17.0.4
IPs:
  IP:           172.17.0.4
Controlled By:  StatefulSet/sessiondb
Containers:
  sessiondb:
    Container ID:   docker://a94f894687f0813196a94afe88f64723f238eb7d2cb061e4c7ef17354f27dee8
    Image:          docker.io/bitnami/postgresql:11.13.0-debian-10-r33
    Image ID:       docker-pullable://bitnami/postgresql@sha256:205e1c5a1d4b56d0d63f6579557652f958e321006c4cb5325d031d40313e4ea2
    Port:           5432/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 26 Nov 2021 10:50:45 +0100
      Finished:     Fri, 26 Nov 2021 10:50:46 +0100
    Ready:          False
    Restart Count:  7
    Requests:
      cpu:      250m
      memory:   256Mi
    Liveness:   exec [/bin/sh -c exec pg_isready -U &quot;postgres&quot; -d &quot;dbname=session&quot; -h 127.0.0.1 -p 5432] delay=30s timeout=5s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/sh -c -e exec pg_isready -U &quot;postgres&quot; -d &quot;dbname=session&quot; -h 127.0.0.1 -p 5432
[ -f /opt/bitnami/postgresql/tmp/.initialized ] || [ -f /bitnami/postgresql/.initialized ]
] delay=5s timeout=5s period=10s #success=1 #failure=6
    Environment:
      BITNAMI_DEBUG:                        false
      POSTGRESQL_PORT_NUMBER:               5432
      POSTGRESQL_VOLUME_DIR:                /bitnami/postgresql
      PGDATA:                               /bitnami/postgresql/data
      POSTGRES_USER:                        postgres
      POSTGRES_PASSWORD:                    &lt;set to the key 'postgresql-password' in secret 'postgresql-root-password'&gt;  Optional: false
      POSTGRES_DB:                          session
      POSTGRESQL_ENABLE_LDAP:               no
      POSTGRESQL_ENABLE_TLS:                no
      POSTGRESQL_LOG_HOSTNAME:              false
      POSTGRESQL_LOG_CONNECTIONS:           false
      POSTGRESQL_LOG_DISCONNECTIONS:        false
      POSTGRESQL_PGAUDIT_LOG_CATALOG:       off
      POSTGRESQL_CLIENT_MIN_MESSAGES:       error
      POSTGRESQL_SHARED_PRELOAD_LIBRARIES:  pgaudit
    Mounts:
      /bitnami/postgresql from data (rw)
      /dev/shm from dshm (rw)
      /docker-entrypoint-initdb.d/ from custom-init-scripts (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-sessiondb-0
    ReadOnly:   false
  custom-init-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sessiondb-scheme
    Optional:  false
  dshm:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      Memory
    SizeLimit:   &lt;unset&gt;
QoS Class:       Burstable
Node-Selectors:  &lt;none&gt;
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  11m                  default-scheduler  Successfully assigned default/sessiondb-0 to ip-10-0-1-112.eu-central-1.compute.internal
  Normal   Created    11m (x4 over 11m)    kubelet            Created container sessiondb
  Normal   Started    11m (x4 over 11m)    kubelet            Started container sessiondb
  Normal   Pulled     10m (x5 over 11m)    kubelet            Container image &quot;docker.io/bitnami/postgresql:11.13.0-debian-10-r33&quot; already present on machine
  Warning  BackOff    110s (x57 over 11m)  kubelet            Back-off restarting failed container

---

% kubectl describe pvc data-sessiondb-0
Name:          data-sessiondb-0
Namespace:     default
StorageClass:  standard
Status:        Bound
Volume:        pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels:        app.kubernetes.io/instance=asdf
               app.kubernetes.io/name=sessiondb
               role=primary
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      8Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       sessiondb-0
Events:        &lt;none&gt;

---

% kubectl describe pvc data-sessiondb-0
Name:          data-sessiondb-0
Namespace:     default
StorageClass:  standard
Status:        Bound
Volume:        pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels:        app.kubernetes.io/instance=asdf
               app.kubernetes.io/name=sessiondb
               role=primary
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: k8s.io/minikube-hostpath
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      8Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       sessiondb-0
Events:        &lt;none&gt;
cyrill@CyrillsMBP4380 core % kubectl describe pv pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Name:            pvc-6b56b20c-3e56-4a92-9278-794bf6cda4de
Labels:          &lt;none&gt;
Annotations:     hostPathProvisionerIdentity: 10bfa079-1086-4a77-849e-7d00de8e34dc
                 pv.kubernetes.io/provisioned-by: k8s.io/minikube-hostpath
Finalizers:      [kubernetes.io/pv-protection]
StorageClass:    standard
Status:          Bound
Claim:           default/data-sessiondb-0
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        8Gi
Node Affinity:   &lt;none&gt;
Message:         
Source:
    Type:          HostPath (bare host directory volume)
    Path:          /tmp/hostpath-provisioner/default/data-sessiondb-0
    HostPathType:  
Events:            &lt;none&gt;
</code></pre>
<p>EDIT: Add logs with DEBUG level
--&gt; Looking to fix: <code>directory &quot;/bitnami/postgresql/data&quot; is not a database cluster directory</code></p>
<p>EDIT2: Add logs of the init container <code>volumePermissions.enabled</code> and permissions</p>
<p>EDIT3: Ok, so I created a fresh version to compare the old and the new one. I am wondering, why there is such a difference in the files as both were working until one didn't anymore. (During regular business, no upgrades, nothing.)</p>
<p><a href="https://i.stack.imgur.com/eMZTO.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/eMZTO.jpg" alt="Logs" /></a></p>

<p>I really hope nobody else runs across this, but finally I found the problem and for once it was not only between the chair and the monitor, but also RTFM was involved.</p>
<p>As mentioned I am using minikube to run my k8s cluster which provides PVs stored on the host disk. Where it is stored you may ask? Exaclty, here: <code>/tmp/hostpath-provisioner/default/data-sessiondb-0/data/</code>. You find the problem? No, I also took some time to figure it out. WHY ON EARTH does minikube use the <code>tmp</code> folder to store <strong>persistant</strong> volume claims?</p>
<p><a href="https://i.stack.imgur.com/4L5oh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4L5oh.png" alt="tmp folder" /></a></p>
<p>This folder gets autom. cleared every now and so on.</p>
<p>SOLUTION: Change the path and DO NOT STORE PVs IN
<code>tmp</code> FOLDERS.</p>
<p>They mention this here: <a href="https://minikube.sigs.k8s.io/docs/handbook/persistent_volumes/#a-note-on-mounts-persistence-and-minikube-hosts" rel="nofollow noreferrer">https://minikube.sigs.k8s.io/docs/handbook/persistent_volumes/#a-note-on-mounts-persistence-and-minikube-hosts</a> and give an example.</p>
<p>But why use the &quot;dangerous&quot; <code>tmp</code> path per default and not, let's say, <code>data</code> without putting a <em>Warning</em> banner there?</p>
<p>Sigh. Closing this question ^^</p>
<p>--&gt; Workaround: <a href="https://github.com/kubernetes/minikube/issues/7511#issuecomment-612099413" rel="nofollow noreferrer">https://github.com/kubernetes/minikube/issues/7511#issuecomment-612099413</a></p>
<hr />
<p>Github issues to this topic:</p>
<ul>
<li><a href="https://github.com/kubernetes/minikube/issues/7511" rel="nofollow noreferrer">https://github.com/kubernetes/minikube/issues/7511</a></li>
<li><a href="https://github.com/kubernetes/minikube/issues/13038" rel="nofollow noreferrer">https://github.com/kubernetes/minikube/issues/13038</a></li>
<li><a href="https://github.com/kubernetes/minikube/issues/3318" rel="nofollow noreferrer">https://github.com/kubernetes/minikube/issues/3318</a></li>
<li><a href="https://github.com/kubernetes/minikube/issues/5144" rel="nofollow noreferrer">https://github.com/kubernetes/minikube/issues/5144</a></li>
</ul>
<p>My Github issue for clarification in the docs: <a href="https://github.com/kubernetes/minikube/issues/13038#issuecomment-981821696" rel="nofollow noreferrer">https://github.com/kubernetes/minikube/issues/13038#issuecomment-981821696</a></p>
