<p>I'm having a hard time understanding why a pods readiness probe is failing.</p>

<pre><code>  Warning  Unhealthy  21m (x2 over 21m)  kubelet, REDACTED  Readiness probe failed: Get http://192.168.209.74:8081/actuator/health: dial tcp 192.168.209.74:8081: connect: connection refused
</code></pre>

<p>If I exec into this pod (or in fact into any other I have for that application), I can run a curl against that very URL without issue:</p>

<pre><code>kubectl exec -it REDACTED-l2z5w /bin/bash
$ curl -v http://192.168.209.74:8081/actuator/health
$ curl -v http://192.168.209.74:8081/actuator/health
* Expire in 0 ms for 6 (transfer 0x5611b949ff50)
*   Trying 192.168.209.74...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x5611b949ff50)
* Connected to 192.168.209.74 (192.168.209.74) port 8081 (#0)
&gt; GET /actuator/health HTTP/1.1
&gt; Host: 192.168.209.74:8081
&gt; User-Agent: curl/7.64.0
&gt; Accept: */*
&gt; 
&lt; HTTP/1.1 200 
&lt; Set-Cookie: CM_SESSIONID=E62390F0FF8C26D51C767835988AC690; Path=/; HttpOnly
&lt; X-Content-Type-Options: nosniff
&lt; X-XSS-Protection: 1; mode=block
&lt; Cache-Control: no-cache, no-store, max-age=0, must-revalidate
&lt; Pragma: no-cache
&lt; Expires: 0
&lt; X-Frame-Options: DENY
&lt; Content-Type: application/vnd.spring-boot.actuator.v3+json
&lt; Transfer-Encoding: chunked
&lt; Date: Tue, 02 Jun 2020 15:07:21 GMT
&lt; 
* Connection #0 to host 192.168.209.74 left intact
{"status":"UP",...REDACTED..}
</code></pre>

<p>I'm getting this behavior from both a Docker-for-Desktop k8s cluster on my Mac as well as an OpenShift cluster.</p>

<p>The readiness probe is shown like this in kubectl describe:</p>

<pre><code>    Readiness:  http-get http://:8081/actuator/health delay=20s timeout=3s period=5s #success=1 #failure=10
</code></pre>

<p>The helm chart has this to configure it:</p>

<pre><code>    readinessProbe:
      failureThreshold: 10
      httpGet:
        path: /actuator/health
        port: 8081
        scheme: HTTP
      initialDelaySeconds: 20
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 3
</code></pre>

<p>I cannot fully rule out that HTTP proxy settings are to blame, but the k8s docs say that HTTP_PROXY is ignored for checks since v1.13, so it shouldn't happen locally.</p>

<p>The OpenShift k8s version is 1.11, my local one is 1.16.</p>

<p>Describing events always show the last event on the resource you are checking. The thing is that the last event logged was an error while checking the <code>readinessProbe</code>. </p>

<p>I tested it in my lab with the following pod manifest: </p>

<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: readiness-exec
spec:
  containers:
  - name: readiness
    image: k8s.gcr.io/busybox
    args:
    - /bin/sh
    - -c
    - sleep 30; touch /tmp/healthy; sleep 600
    readinessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5
</code></pre>

<p>As can be seen, a file <code>/tmp/healthy</code> will be created in the pod after 30 seconds and the <code>readinessProbe</code> will check if the file exists after 5 seconds and repeat the check after every 5 seconds.</p>

<p>Describing this pod will give me that: </p>

<pre><code>Events:
  Type     Reason     Age                    From                 Message
  ----     ------     ----                   ----                 -------
  Normal   Scheduled  7m56s                  default-scheduler    Successfully assigned default/readiness-exec to yaki-118-2
  Normal   Pulling    7m55s                  kubelet, yaki-118-2  Pulling image "k8s.gcr.io/busybox"
  Normal   Pulled     7m55s                  kubelet, yaki-118-2  Successfully pulled image "k8s.gcr.io/busybox"
  Normal   Created    7m55s                  kubelet, yaki-118-2  Created container readiness
  Normal   Started    7m55s                  kubelet, yaki-118-2  Started container readiness
  Warning  Unhealthy  7m25s (x6 over 7m50s)  kubelet, yaki-118-2  Readiness probe failed: cat: can't open '/tmp/healthy': No such file or directory
</code></pre>

<p>The <code>readinessProbe</code> looked for the file 6 times with no success and it's completely right as I configured it to check every 5 seconds and the file was created after 30 seconds. </p>

<p>What you think is a problem is actually the expected behavior. Your Events is telling you that the <code>readinessProbe</code> failed to check 21 minutes ago. It means actually that your pod is healthy since 21 minutes ago. </p>
