<p>I'm a noob with Azure deployment, kubernetes and HA implementation. When I implement health probes as part of my app deployment, the health probes fail and I end up with either 503 (internal server error) or 502 (bad gateway) error when I try accessing the app via the URL. When I remove the health probes, I can successfully access the app using its URL.</p>
<p>I use the following yaml deployment configuration when implementing the health probes, which is utilised by an Azure devops pipeline. The app takes under 5 mins to become available, so I set the <code>initialDelaySeconds</code> for the health probes to <code>300s</code>.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
   name: myApp
spec:
   ... 
   template:
     metadata:
       labels:
         app: myApp
     spec:
        ...
        containers:
          - name: myApp
            ...
            ports:
              - containerPort: 5000          
            ...
            readinessProbe:
              tcpSocket:
                  port: 5000
              initialDelaySeconds: 300
              periodSeconds: 5
              successThreshold: 1
              failureThreshold: 3
            livenessProbe:
               tcpSocket:
                  port: 5000
               periodSeconds: 30 
               initialDelaySeconds: 300
               successThreshold: 1
               failureThreshold: 3

...
</code></pre>
<p>When I perform the deployment and describe the pod, I see the following listed under 'Events' at the bottom of the output:</p>
<pre><code>  Type     Reason     Age                   From                             Message
  ----     ------     ----                  ----                             -------
  Warning  Unhealthy  2m1s (x288 over 86m)  kubelet, aks-vm-id-appears-here  Readiness probe failed: dial tcp 10.123.1.23:5000: connect: connection refused
</code></pre>
<p>(this is confusing as it states the age as 2m1s - but the <code>initialDelaySeconds</code> is greater than this - so I'm not sure why it reports this as the age)</p>
<p>The readiness probe subsequently fails with the same error. The IP number matches the IP of my pod and I see this under <code>Containers</code> in the pod description:</p>
<pre><code>Containers:
....
Port:           5000/TCP
</code></pre>
<p>The failure of the liveness and readiness probes results in the pod being continually terminated and restarted.</p>
<p>The app has a default <code>index.html</code> page, so I <em>believe</em> the health probe should receive a 200 response if it's able to connect.</p>
<p>Because the health probe is failing, the pod IP doesn't get assigned to the endpoints object and therefore isn't assigned against the service.</p>
<p>If I comment out the <code>readinessProbe</code> and <code>livenessProbe</code> from the deployment, the app runs successfully when I use the URL via the browser, and the pod IP gets successfully assigned as an endpoint that the service can communicate with. The endpoint address is in the form 10.123.1.23:5000 - i.e. port 5000 seems to be the correct port for the pod.</p>
<p>I don't understand why the health probe would be failing to connect? It looks correct to me that it should be trying to connect on an IP that looks like 10.123.1.23:5000.</p>
<p>It's possible that the port is taking a long time than 300s to become open, but I don't know of a way I can check that. If I enter a bash session on the pod, <code>watch</code> isn't available (I read that <code>watch ss -lnt</code> can be used to examine port availability).</p>
<p>The following answer suggests increasing <code>initialDelaySeconds</code> but I already tried that - <a href="https://stackoverflow.com/a/51932875/1549918">https://stackoverflow.com/a/51932875/1549918</a></p>
<p>I saw this question - but resource utilisation (e.g. CPU/RAM) is not the issue
<a href="https://stackoverflow.com/questions/63904845/liveness-and-readiness-probe-connection-refused">Liveness and readiness probe connection refused</a></p>
<p><strong>UPDATE</strong></p>
<p>If I curl from a replica of the pod to <a href="https://10.123.1.23:5000" rel="nofollow noreferrer">https://10.123.1.23:5000</a>, I get a similar error (<code>Failed to connect to ...the IP.. port 5000: Connection refused</code>). Why could this be failing? I read something that suggests that attempting this connection from another pod may indicate reachability for the health probes also.</p>

Chris Halcrow
<p>If you are unsure if your application is starting correctly then replace it with a known good image. e.g. <a href="https://hub.docker.com/_/httpd/" rel="nofollow noreferrer">httpd</a></p>
<p>change the ports to 80, the image to httpd.</p>
<p>You might also want to increase the timeout for the health check as it defaults to 1 second to timeoutSeconds=5</p>
<p>in addition, if your image is a web application then it would be better to use <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-liveness-http-request" rel="nofollow noreferrer">a http probe</a></p>
