<p>I think I'm going crazy. I had kubernetes setup to deploy a postgres database. Worked fine. I wanted to add an init script so I made a custom image, deployed to private docker hub repo, and then updated the deployment to pull new image. But no matter what I do kubernetes keeps deploying the old database.... I updated the user, password, and database name to confirm that its not getting upgraded. Here's the kubectl file:</p>
<pre><code># PostgreSQL StatefulSet Service
apiVersion: v1
kind: Service
metadata:
  name: postgres-db-lb
spec:
  selector:
    app: postgres-stockticker
  ports:
    - name: &quot;5432&quot;
      port: 5432
      targetPort: 5432
  type: ClusterIP

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-stockticker
spec:
  selector:
    matchLabels:
      app: postgres-stockticker
  replicas: 1
  template:
    metadata:
      labels:
        app: postgres-stockticker
    spec:
      containers:
      - name: postgres-stockticker
        image: goodwinmcd/postgres:1.0
        imagePullPolicy: Always
        ports:
        - containerPort: 5432
      # Data Volume
        volumeMounts:
        - name: postgresql-db-disk
          mountPath: /var/lib/postgresql/data
      volumes:
        - name: postgresql-db-disk
          persistentVolumeClaim:
            claimName: postgres-pv-claim
      imagePullSecrets:
        - name: docker-hub-repo-key
---

kind: PersistentVolume
apiVersion: v1
metadata:
  name: postgres-pv-volume  # Sets PV's name
  labels:
    type: local  # Sets PV's type to local
    app: postgres-stockticker
spec:
  storageClassName: manual
  capacity:
    storage: 5Gi # Sets PV Volume
  accessModes:
    - ReadWriteMany
  hostPath:
    path: &quot;/mnt/data&quot;

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: postgres-pv-claim  # Sets name of PVC
  labels:
    app: postgres-stockticker
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany  # Sets read and write access
  resources:
    requests:
      storage: 5Gi  # Sets volume size
</code></pre>
<ol>
<li>I've verified the following: Pulled down the image in a local docker image and confirmed that it has the correct user, password, database, and tables in it.</li>
<li>Described the pod to make sure it's pulling and using the correct image</li>
<li>Logged onto the pod and confirmed that the new username, password, and database are present on the pod with the correct values.</li>
<li>I tried running psql to connect to the database with the new username and password but no luck. It succeeded with the old credentials</li>
<li>Recreated and mounted new pv and new pvc to the pod</li>
<li>I deleted the service, the deployment, the pv, and the pvc and recreated them with the apply command</li>
<li>I tried restarting the postgres service (service postgres restart) and get the following error: <code>No PostgreSQL clusters exist; see &quot;man pg_createcluster&quot; ... (warning).</code></li>
</ol>
<p>So somehow kubernetes is pulling down this container that has the new username, password, and database baked into the image but then starting up the old postgres server on the cluster. It makes no sense to me. Someone please help. I'm losing my sanity making sure I'm not missing something obvious.</p>
<p>Let me know if you need any other information.</p>

<p>Figured it out. The kubernetes retention policy of the volume was set to retain. I tried setting it to delete but the data was still persisting. I ended up logging on to the instance and blowing away everything where I mounted the PV. This forced the pod to restart, and when it did it actually went through the correct postgres init process.</p>
