<p>I am building postgresql on kubernetes, but I cannot persist postgres data.
The host is a GCP instance with Ubuntu-20.4. The volumes for disk are using GCP volumes, which are mounted after attaching to the instance.</p>
<pre><code>Filesystem      1K-blocks    Used Available Use% Mounted on
/dev/root         9982728 4252592   5713752  43% /
devtmpfs          4038244       0   4038244   0% /dev
tmpfs             4059524       0   4059524   0% /dev/shm
tmpfs              811908    1568    810340   1% /run
tmpfs                5120       0      5120   0% /run/lock
tmpfs             4059524       0   4059524   0% /sys/fs/cgroup
/dev/loop0          50304   50304         0 100% /snap/core18/2671
/dev/loop2         217856  217856         0 100% /snap/google-cloud-cli/98
/dev/loop3          94208   94208         0 100% /snap/lxd/24065
/dev/loop1          60544   60544         0 100% /snap/core20/1782
/dev/loop4          44032   44032         0 100% /snap/snapd/17885
/dev/nvme0n1p15     99801    6004     93797   7% /boot/efi
tmpfs              811904       0    811904   0% /run/user/1001
shm                 65536       0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/49ebcf7b449f4b13d52aab6f52c28f139c551f83070f6f21207dbf52315dc264/shm
shm                 65536       0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/e46f0c6b19e5ccff9bb51fa3f7669a9a6a2e7cfccf54681e316a9cd58183dce4/shm
shm                 65536       0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/495c80e87521bfdda55827df64cdb84cddad149fb502ac7ee12f3607badd4649/shm
shm                 65536       0     65536   0% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/443e7b254d02c88873c59edc6d5b0a71e80da382ea81105e8b312ad4122d694a/shm
/dev/nvme0n3     10218772      12   9678088   1% /var/lib/postgresql ※ disk for postgres
/dev/nvme0n2      3021608      24   2847916   1% /var/lib/pgadmin ※ disk for pgadmin
shm                 65536    1052     64484   2% /run/containerd/io.containerd.grpc.v1.cri/sandboxes/bd83982e91b6a3bce7853416d72878d5473174e884c15578c47a8d8952f4e718/shm
</code></pre>
<p>Also, the pod volume is allocated using persistent volume and persistent volume claim.</p>
<pre><code>kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume  # Sets PV's name
  labels:
    app: postgres
spec:
  # storageClassName: local-storage
  capacity:
    storage: 10Gi # Sets PV Volume
  accessModes:
    - ReadWriteMany
  hostPath:
    path: &quot;/var/lib/postgresql&quot;
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pgadmin-pv-volume  # Sets PV's name
  labels:
    app: pgadmin
spec:
  # storageClassName: local-storage
  capacity:
    storage: 3Gi # Sets PV Volume
  accessModes:
    - ReadWriteMany
  hostPath:
    path: &quot;/var/lib/pgadmin&quot;
</code></pre>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgresql
  labels:
    app: postgresql
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgresql
  template:
    metadata:
      labels:
        app: postgresql
    spec:
      containers:
        - image: docker.io/yamamuratkr/postgres
          name: postgresql
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: postgres
                  key: database_user
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres
                  key: database_password
            - name: PGDATA
              value: &quot;/var/lib/postgresql/data/pgdata&quot;
          ports:
            - containerPort: 5432
              name: postgresql
          volumeMounts:
            - name: postgredb
              mountPath: /var/lib/postgresql
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim
</code></pre>
<p>However, when the pod is deleted, the postgres data also disappears with it, and I can confirm that it is not persistent.</p>
<p>If you know the cause of this problem, please let me know.
Thank you in advance.</p>
<p>None of the following worked</p>
<ul>
<li>Using hostpath for pod volumes</li>
<li>Use default PGDATA</li>
</ul>

山村武
<p>The source of your problem is here:</p>
<pre><code>volumeMounts:
  - name: postgredb
    mountPath: /var/lib/postgresql
</code></pre>
<p>The <code>postgres</code> image itself mounts a volume on <code>/var/lib/postgresql/data</code>. We can see that if we inspect the image:</p>
<pre><code>$ docker image inspect docker.io/postgres:14 | jq '.[0].ContainerConfig.Volumes'
{
  &quot;/var/lib/postgresql/data&quot;: {}
}
</code></pre>
<p>Your mount on <code>/var/lib/postgresql</code> is effectively a no-op. An ephemeral volume is created on <code>/var/lib/postgresql/data</code> each time the container starts, and since that's the default <code>PGDATA</code> location, your data is effectively discarded when the container exits.</p>
<p>I've put together an example in my local environment to demonstrate this behavior. I've made a few minor changes from your example that shouldn't operationally impact anything.</p>
<p>I've created the following Secret with the postgres credentials; by naming the keys like this we can use a single <code>envFrom</code> block instead of multiple <code>env</code> entries (see the Deployment for details):</p>
<pre><code>apiVersion: v1
kind: Secret
metadata:
  name: postgres-env
type: Opaque
stringData:
  POSTGRES_USER: example
  POSTGRES_PASSWORD: secret
</code></pre>
<p>And in line with my comment I'm injecting a file into <code>/docker-entrypoint-initdb.d</code> from this ConfigMap:</p>
<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
data:
  initdb.sql: |
    CREATE TABLE people (
      id SERIAL,
      name VARCHAR(40),
      favorite_color VARCHAR(10)
    );

    INSERT INTO people (name, favorite_color) VALUES ('bob', 'red');
    INSERT INTO people (name, favorite_color) VALUES ('alice', 'blue');
    INSERT INTO people (name, favorite_color) VALUES ('mallory', 'purple');
</code></pre>
<p>I'm using this PersistentVolumeClaim:</p>
<pre><code>apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pv-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
</code></pre>
<p>And finally I'm tying it all together in this Deployment:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: postgres
  name: postgresql
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - envFrom:
        - secretRef:
            name: postgres-env
        image: docker.io/postgres:14
        name: postgresql
        ports:
        - containerPort: 5432
          name: postgresql
        volumeMounts:
        - mountPath: /docker-entrypoint-initdb.d
          name: postgres-config
        - mountPath: /var/lib/postgresql
          name: postgres-data
      volumes:
      - name: postgres-data
        persistentVolumeClaim:
          claimName: postgres-pv-claim
      - configMap:
          name: postgres-config
        name: postgres-config

</code></pre>
<p>The significant changes here are:</p>
<ol>
<li><p>I'm using a single <code>envFrom</code> block to set environment variables from the keys in the <code>postgres-env</code> secret.</p>
</li>
<li><p>I'm using the upstream <code>docker.io/postgres:14</code> image rather than building my own custom image.</p>
</li>
<li><p>I'm injecting the contents of <code>/docker-entrypoint-initdb.d</code> from the postgres-config ConfigMap.</p>
</li>
</ol>
<p>Note that this deployment is using the same <code>mountPath</code> as in your example`.</p>
<hr />
<p>If I bring up this environment, I can see that the database initialization script was executed correctly. The <code>people</code> table exists and has the expected data:</p>
<pre><code>$ kubtectl exec -it deploy/postgresql -- psql -U example -c 'select * from people'
id |  name   | favorite_color
----+---------+----------------
1 | bob     | red
2 | alice   | blue
3 | mallory | purple
(3 rows)
</code></pre>
<p>Let's make a change to the database and see what happens when we restart the pod. First, we add a new row to the table and view the updated table:</p>
<pre><code>$ kubectl exec -it deploy/postgresql -- psql -U example -c &quot;insert into people (name, favorite_color) values ('carol', 'green')&quot;
INSERT 0 1
$ kubectl exec -it deploy/postgresql -- psql -U example -c 'select * from people'
id |  name   | favorite_color
----+---------+----------------
1 | bob     | red
2 | alice   | blue
3 | mallory | purple
4 | carol   | green
(4 rows)
</code></pre>
<p>Now we restart the database pod:</p>
<pre><code>$ kubectl rollout restart deployment/postgresql
</code></pre>
<p>And finally check if our changes survived the restart:</p>
<pre><code>$ kubectl exec -it deploy/postgresql -- psql -U example -c 'select * from people' id |  name   | favorite_color
----+---------+----------------
1 | bob     | red
2 | alice   | blue
3 | mallory | purple
(3 rows)
</code></pre>
<p>As expected, they did not! Let's change the <code>mountPath</code> in the Deployment so that it looks like this instead:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: postgres
  name: postgresql
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - envFrom:
        - secretRef:
            name: postgres-env
        image: docker.io/postgres:14
        name: postgresql
        ports:
        - containerPort: 5432
          name: postgresql
        volumeMounts:
        - mountPath: /docker-entrypoint-initdb.d
          name: postgres-config
        - mountPath: /var/lib/postgresql/data
          name: postgres-data
      volumes:
      - name: postgres-data
        persistentVolumeClaim:
          claimName: postgres-pv-claim
      - configMap:
          name: postgres-config
        name: postgres-config
</code></pre>
<p>Using this Deployment, with no other changes, we can re-run the previous test and see that our data persists as desired:</p>
<pre><code>$ kubectl exec -it deploy/postgresql -- psql -U example -c &quot;insert into people (name, favorite_color) values ('carol', 'green')&quot;
INSERT 0 1
$ k rollout restart deployment/postgresql
deployment.apps/postgresql restarted
$ k exec -it deploy/postgresql -- psql -U example -c 'select * from people' id |  name   | favorite_color
----+---------+----------------
  1 | bob     | red
  2 | alice   | blue
  3 | mallory | purple
  4 | carol   | green
(4 rows)
</code></pre>
<hr />
<p>An alternative solution would be to mount your volume in a completely different location and then set <code>PGDATA</code> appropriately. E.g.,</p>
<pre><code>          ...
          env:
            - name: PGDATA
              value: /data
          ...
          volumeMounts:
            - name: postgres-data
              mountPath: /data
          ...
</code></pre>
