<p>I'm not an expert when it comes to Kubernetes so, for learning, I started building a simple Spring Boot web application (called meal-planner) which reads and writes to a PostgreSQL database.</p>
<p>PostgreSQL and Spring Boot app are deployed in Kubernetes which is running on (currently) two <a href="https://en.wikipedia.org/wiki/Raspberry_Pi" rel="nofollow noreferrer">Raspberry Pis</a> (model 3 and 3B+).</p>
<p>I work in the namespace <code>home</code> and have created a Deployment and Service for each (Postgres and Spring Boot application.).</p>
<p>I have created a secret defining credentials which I use to create the database and as database credentials in the Spring Boot application.
Productively, I wouldn't put the database in the Kubernetes cluster, but with the Raspberry Pi, I don't care since the data is worthless.</p>
<p>Describing the secret yields:</p>
<pre><code>Name:         dev-db-secret
Namespace:    home
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;

Type:  Opaque

Data
====
password:  20 bytes
username:  7 bytes
</code></pre>
<p>The <code>application.yaml</code> is the following:</p>
<pre><code>spring:
  jpa:
    database-platform: org.hibernate.dialect.PostgreSQLDialect
    open-in-view: false
    hibernate:
      ddl-auto: update
  datasource:
    url: &quot;jdbc:postgresql://postgres:5432/home&quot;
    username: ${DB_USER}
    password: ${DB_PASSWORD}
</code></pre>
<p>I used the following Dockerfile for building the Spring Boot application:</p>
<pre><code>FROM arm32v7/openjdk:11-jdk
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} app.jar
ENTRYPOINT [&quot;java&quot;,&quot;-jar&quot;,&quot;/app.jar&quot;]
</code></pre>
<p>And I build it on a Windows 10 PC using <code>docker buildx build --platform linux/arm/v7 -t ... --push</code>.
Kubernetes finds and deploys it just fine.</p>
<p>The following is the Service and Deployment for the PostgreSQL DB:</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: postgres
  namespace: home
  labels:
    app: postgres
spec:
  type: NodePort
  selector:
    app: postgres
  ports:
    - port: 5432
      targetPort: 5432
      name: postgres
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: postgres
  namespace: home
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:13.2
          imagePullPolicy: IfNotPresent
          env:
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  name: dev-db-secret
                  key: username
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: dev-db-secret
                  key: password
            - name: POSTGRES_DB
              value: home
          ports:
            - containerPort: 5432
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgres-data
      volumes:
        - name: postgres-data
          persistentVolumeClaim:
            claimName: postgres-pv-claim
---
</code></pre>
<p>The deployment works fine and I can access the database using an SQL client (like DBeaver) using the credentials in the <code>dev-db-secret</code> secret (when I port-forward the service using <code>kubectl port-forward service/postgres -n home 5432:5432</code>).</p>
<p>I use the following deployment descriptor for the Spring-Boot application.</p>
<pre><code>kind: Service
apiVersion: v1
metadata:
  name: meal-planner
  namespace: home
  labels:
    app: meal-planner
spec:
  type: NodePort
  selector:
    app: meal-planner
  ports:
    - port: 8080
      targetPort: 8080
      name: meal-planner
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: meal-planner
  namespace: home
spec:
  replicas: 1
  selector:
    matchLabels:
      app: meal-planner
  template:
    metadata:
      labels:
        app: meal-planner
    spec:
      containers:
        - name: meal-planner
          image: 08021986/meal-planner:v1
          imagePullPolicy: Always
          env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: dev-db-secret
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: dev-db-secret
                  key: password
          ports:
            - containerPort: 8080
---
</code></pre>
<p>The application deploys and starts just fine. However, when I <code>kubectl port-forward service/meal-planner -n home 8080:8080</code> and call the &quot;Read all data from DB&quot;-Endpoint, I get the following error:</p>
<pre><code>...
Caused by: org.hibernate.exception.GenericJDBCException: Unable to open JDBC Connection for DDL execution
...
Caused by: org.postgresql.util.PSQLException: FATAL: password authentication failed for user &quot;&lt;&lt;username from secret&gt;&gt;&quot;
</code></pre>
<p>When I start the application locally, port-forward the DB-Service (and replace the datasource URL in the <em>application.yaml</em> file with <code>jdbc:localhost://postgres:5432/home</code>), everything works perfectly.</p>
<p><strong>I don't understand why the PostgreSQL uses the secret to create the credentials - which are clearly working, since I can access the database using a client or with the application running locally, but the authentication fails for the Spring Boot service (which still seems to be able to fetch the username from the secret).
What am I doing wrong?</strong></p>
<p>For completeness of the deployment descriptors, here the namespace, persistent volume and claim descriptor:</p>
<pre><code>apiVersion: v1
kind: Namespace
metadata:
  name: home
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-volume
  namespace: home
  labels:
    type: local
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: &quot;/mnt/data&quot;
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pv-claim
  namespace: home
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
</code></pre>
<p>Code can be found on <a href="https://github.com/Urr4/deployments" rel="nofollow noreferrer">https://github.com/Urr4/deployments</a> and <a href="https://github.com/Urr4/meal-planner" rel="nofollow noreferrer">https://github.com/Urr4/meal-planner</a></p>
<p>Update:</p>
<p>After completely removing the secret and adding the credentials by hand to the <em>application.yaml</em> file, I still get the same error.
I'm now assuming, that I can’t reach the PostgreSQL service from the meal-planner pod, meaning the service can’t connect to the database and thus fails authentication, not because of credentials.</p>
<p>I tried <code>kubectl exec &lt;&lt;meal-planner-pod-id&gt;&gt; -n home -- ping postgres:5432</code> and got <code>ping: postgres:5432: Name or service not known</code>.</p>
<p>I also tried <code>postgres.default:5432</code> and <code>postgres.home:5432</code> without success.</p>
<p>Running <code>k exec &lt;&lt;meal-planner-pod-id&gt;&gt; -n home -- wget -O- postgres</code> yields <code>Connecting to postgres (postgres)|10.43.62.32|:80... failed: Network is unreachable.</code> which sounds like an error, which I don't know how to fix.</p>
<p><strong>I still don't know what the problem is</strong>.</p>
<p>Update:</p>
<p>I've started a <a href="https://en.wikipedia.org/wiki/BusyBox" rel="nofollow noreferrer">BusyBox</a>, because I can’t prevent the meal-planer from crashing and <code>kubectl exec -it</code> into the BusyBox pod and execute <code>telnet postgres.home 5432</code> which yields <code>telnet connected</code>. <code>wget -O- postgres.home</code> still yields <code>Network is unreachable</code>.
I tried telnet on the meal-planer pod, but it just tells me <code>telnet: not found</code>.</p>
<p>I think <code>postgres</code> (or <code>postgres.home</code>) is the right DNS name, since in that case I get the <code>Network is unreachable</code>. For non-existent services I get <code>Name or service not known</code>.</p>
<p><strong>I currently think there is some kind of network issue which prevents pods from communicating, but I don't know how to find and fix this.</strong></p>
<p><sub>After debugging in the completely wrong direction, I'm going to close this question in favor of asking it again, but more precisely and smaller. The new question can be found <a href="https://stackoverflow.com/questions/68409769/kubernetes-service-unreachable">here</a>.</sub></p>

Urr4
<p>The service created for PostgreSQL should be of type <em>ClusterIP</em>. If you are using NodePort, you will be able to access PostgreSQL using NodeIP:NodePort. Since both the instances are within the namespace, you can change type to <em>ClusterIP</em>.</p>
<p>Also, you can change the environment variable name in the deployment meal-planner.</p>
<p>DB_USER -&gt; username</p>
<p>DB_PASSWORD -&gt; password</p>
