<h3>What I'm trying to achieve</h3>
<p>I'm setting up a microk8s cluster consisting of a postgres database and an elixir application that communicates with the database.</p>
<h3>The problem I'm encountering</h3>
<p>I always get an error from the database pod when attempting to connect:</p>
<pre><code>2022-01-05 18:54:05.179 UTC [216] DETAIL:  Password does not match for user &quot;phoenix_db_username
        &quot;.
        Connection matched pg_hba.conf line 99: &quot;host all all all md5&quot;
</code></pre>
<p>The connection between the database and app is clearly working since the database logs the error.</p>
<h3>What I've tried</h3>
<p>Others with the problem have suggested deleting the PV and PVC. See this github issue:
<a href="https://github.com/helm/charts/issues/16251#issuecomment-577560984" rel="nofollow noreferrer">https://github.com/helm/charts/issues/16251#issuecomment-577560984</a></p>
<ul>
<li>I've tried deleting the pvc and pv, I can confirm that the pv was removed as I checked <code>/var/snap/microk8s/common/default-storage</code> before and after removing it.</li>
<li>I've tried permanently deleting the storage by running <code>microk8s.disable storage</code> and then enabling storage again with <code>microk8s.enable storage</code>.</li>
</ul>
<p>Output from <code>microk8s.disable storage</code>:</p>
<pre><code>Disabling default storage
[...]
Storage removed
Remove PVC storage at /var/snap/microk8s/common/default-storage ? (Y/N): y
Storage space reclaimed
</code></pre>
<ul>
<li>I've checked the environment of the database pod with printenv and I see the correct values for POSTGRES_USER and POSTGRES_PASSWORD (phoenix_db_username, phoenix_db_password)</li>
<li>I've checked the environment of the application pod with printenv and I see the correct values for DB_USERNAME and DB_PASSWORD (phoenix_db_username, phoenix_db_password)</li>
<li><strong>I checked the pq_hba.conf file and it did not contain any user</strong></li>
</ul>
<p>According to postgres docker documentation this should create a user, however I don't think it is creating one. <a href="https://hub.docker.com/_/postgres" rel="nofollow noreferrer">https://hub.docker.com/_/postgres</a></p>
<h3>Elixir app yml resources</h3>
<p><strong>The configMap of the elixir app</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: phoenix-app-config
  labels:
    app: phoenix-app
data:
  APP_NAME: &quot;phoenix-app&quot;
  APP_HOST: &quot;0.0.0.0&quot;
  APP_PORT: &quot;4000&quot;
  DB_NAME: &quot;prod_db&quot;
  DB_HOSTNAME: &quot;phoenix-app-database&quot;
  NAMESPACE: &quot;production&quot;
</code></pre>
<p><strong>The secrets of the elixir app</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Secret
metadata:
  name: phoenix-app-secrets
  labels:
    app: phoenix-app
data:
  SECRET_KEY_BASE: KtpxCV3OY8KnRiC5yVo7Be+GRVeND+NxAsyk+FASDFasdfadffhNS804MLk
  DB_PASSWORD: cGhvZW5peC1kYi1wYXNzd29yZAo= # phoenix_db_username
  DB_USERNAME: cGhvZW5peC1kYi11c2VybmFtZQo= # phoenix_db_password
</code></pre>
<p><strong>The deployment of the elixir app</strong></p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: phoenix-app
  labels:
    app: phoenix-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: phoenix-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: phoenix-app
    spec:
      containers:
      - name: phoenix-app
        image: REDACTED
        imagePullPolicy: Always
        command: [&quot;./bin/hello&quot;, &quot;start&quot;]
        lifecycle:
          preStop:
            exec:
              command: [&quot;./bin/hello&quot;, &quot;stop&quot;]
        ports:
        - containerPort: 4000
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        envFrom:
        - configMapRef:
            name: phoenix-app-config
        - secretRef:
            name: phoenix-app-secrets
      imagePullSecrets:
      - name: gitlab-pull-secret
</code></pre>
<h3>Database yml resources</h3>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: phoenix-app-database
  labels:
    app: phoenix-app-database
spec:
  serviceName: phoenix-app-database
  replicas: 1
  selector:
    matchLabels:
      app: phoenix-app-database
  template:
    metadata:
      labels:
        app: phoenix-app-database
    spec:
      containers:
      - name: phoenix-app-database
        image: postgres:12-alpine
        envFrom:
        - configMapRef:
            name: phoenix-app-database-config
        - secretRef:
            name: phoenix-app-database-secrets
        ports:
        - containerPort: 5432
          name: postgresdb
        volumeMounts:
        - name: phoenix-app-database
          mountPath: /var/lib/postgresql/data
      volumes:
      - name: phoenix-app-database
        persistentVolumeClaim:
          claimName: phoenix-app-db-pvc
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: phoenix-app-db-pvc
spec:
  storageClassName: microk8s-hostpath
  capacity:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 250Mi
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: phoenix-app-database-config
  labels:
    app: phoenix-app-database
data:
  POSTGRES_DB: &quot;prod_db&quot;
---
apiVersion: v1
kind: Secret
metadata:
  name: phoenix-app-database-secrets
  labels:
    app: phoenix-app-database
data:
  POSTGRES_USER: cGhvZW5peF9kYl91c2VybmFtZQo= # phoenix_db_username
  POSTGRES_PASSWORD: cGhvZW5peF9kYl9wYXNzd29yZAo= # phoenix_db_password
---
apiVersion: v1
kind: Service
metadata:
  name: phoenix-app-database
  labels:
    app: phoenix-app-database
spec:
  ports:
  - port: 5432
    name: phoenix-app-database
  type: NodePort 
  selector:
    app: phoenix-app-database
---
</code></pre>
<h3>Logs from database pod creation</h3>
<pre class="lang-sh prettyprint-override"><code>me@me:~/Documents/kubernetes-test$ kubectl logs phoenix-app-database-0 -n production
The files belonging to this database system will be owned by user &quot;postgres&quot;.
This user must also own the server process.

The database cluster will be initialized with locale &quot;en_US.utf8&quot;.
The default database encoding has accordingly been set to &quot;UTF8&quot;.
The default text search configuration will be set to &quot;english&quot;.

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
sh: locale: not found
2022-01-05 20:47:02.013 UTC [30] WARNING:  no usable system locales were found
performing post-bootstrap initialization ... ok
syncing data to disk ... ok


Success. You can now start the database server using:

    pg_ctl -D /var/lib/postgresql/data -l logfile start

initdb: warning: enabling &quot;trust&quot; authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
waiting for server to start....2022-01-05 20:47:02.621 UTC [36] LOG:  starting PostgreSQL 12.9 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
2022-01-05 20:47:02.623 UTC [36] LOG:  listening on Unix socket &quot;/var/run/postgresql/.s.PGSQL.5432&quot;
2022-01-05 20:47:02.641 UTC [37] LOG:  database system was shut down at 2022-01-05 20:47:02 UTC
2022-01-05 20:47:02.645 UTC [36] LOG:  database system is ready to accept connections
 done
server started
CREATE DATABASE


/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down....2022-01-05 20:47:02.794 UTC [36] LOG:  received fast shutdown request
2022-01-05 20:47:02.795 UTC [36] LOG:  aborting any active transactions
2022-01-05 20:47:02.797 UTC [36] LOG:  background worker &quot;logical replication launcher&quot; (PID 43) exited with exit code 1
2022-01-05 20:47:02.797 UTC [38] LOG:  shutting down
2022-01-05 20:47:02.808 UTC [36] LOG:  database system is shut down
 done
server stopped

PostgreSQL init process complete; ready for start up.

2022-01-05 20:47:02.904 UTC [1] LOG:  starting PostgreSQL 12.9 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20211027) 10.3.1 20211027, 64-bit
2022-01-05 20:47:02.904 UTC [1] LOG:  listening on IPv4 address &quot;0.0.0.0&quot;, port 5432
2022-01-05 20:47:02.905 UTC [1] LOG:  listening on IPv6 address &quot;::&quot;, port 5432
2022-01-05 20:47:02.909 UTC [1] LOG:  listening on Unix socket &quot;/var/run/postgresql/.s.PGSQL.5432&quot;
2022-01-05 20:47:02.925 UTC [50] LOG:  database system was shut down at 2022-01-05 20:47:02 UTC
2022-01-05 20:47:02.929 UTC [1] LOG:  database system is ready to accept connections
</code></pre>

Beefcake
<p>Okay I have solved the issue and it was rather simple to fix, yet incredibly hard to notice.</p>
<p>I am creating the yml files via templates, such as this:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Secret
metadata:
  name: {{APP_NAME}}-database-secrets
  labels:
    app: {{APP_NAME}}-database
data:
  POSTGRES_USER: {{DB_USERNAME_B64}}
  POSTGRES_PASSWORD: {{DB_PASSWORD_B64}}
</code></pre>
<p>I then merge these templates together and replace all <code>{{ }}</code> enclosed declarations with a value from a specific environment. The ones that end with _B64 I encode into base64 format before inserting.</p>
<p>I was doing it like this which seemed to be working fine:</p>
<pre class="lang-bash prettyprint-override"><code>if [[ &quot;${variable_key}&quot; == *_B64 ]]; then
    variable_value=&quot;$(echo &quot;${variable_value}&quot; | base64)&quot;
fi
</code></pre>
<p>HOWEVER, this is NOT OK, because when I <code>echo</code> the variable here I am <strong>appending a newline</strong> to the variable <strong>which makes the database name and username illegal for postgres</strong>. I realized this when inspecting the decoded value on base64decode.org and saw that there was two lines.</p>
<p>I fixed it by changing the bash script to not print a newline (<code>-n</code>):</p>
<pre class="lang-bash prettyprint-override"><code>if [[ &quot;${variable_key}&quot; == *_B64 ]]; then
    variable_value=&quot;$(echo -n &quot;${variable_value}&quot; | base64 -w 0 )&quot;
fi
</code></pre>
<p>I hope this can help someone debug this problem in the future!</p>
