<p>Currently, I am facing an issue with my application: it does not become healthy due to the kubelet not being able to perform a successful health check.</p>
<p>From pod describe:</p>
<pre><code>  Warning  Unhealthy       84s                kubelet            Startup probe failed: Get &quot;http://10.128.0.208:7777/healthz/start&quot;: dial tcp 10.128.0.208:7777: connect: connection refused
  Warning  Unhealthy       68s (x3 over 78s)  kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
</code></pre>
<p>Now, I find this strange as I can run the health check fine from the worker node where the kubelet is running? So I am wondering what is the difference between running the health check from the worker node via curl, or the kubelet doing it.</p>
<p>Example:</p>
<pre><code>From worker node where the kubelet is running:
sh-4.4# curl -v http://10.128.0.208:7777/healthz/readiness
*   Trying 10.128.0.208...
* TCP_NODELAY set
* Connected to 10.128.0.208 (10.128.0.208) port 7777 (#0)
&gt; GET /healthz/readiness HTTP/1.1
&gt; Host: 10.128.0.208:7777
&gt; User-Agent: curl/7.61.1
&gt; Accept: */*
&gt; 
&lt; HTTP/1.1 200 OK
&lt; Content-Length: 0
&lt; Connection: close
&lt; 
* Closing connection 0
sh-4.4#
</code></pre>
<p>Can I somehow trace when the kubelet is sending the health probe check? Or maybe get into the kubelet and send it myself from there?</p>
<p>There is an extra thing to be told: my pod has got an istio-proxy container inside. It looks like the traffic from the kubelet gets blocked by this istio-proxy.</p>
<p>Setting the following annotation in my deployement:</p>
<pre><code> &quot;rewriteAppHTTPProbe&quot;: true
</code></pre>
<p>does not help for the kubelet. It did help to get a 200 OK when running the curl command from the worker node.</p>
<p>Maybe also to note: we are using the istio-cni plugin to ingest the istio sidecar. Not sure whether that makes a difference when using the former approach when injecting using istio-init ...</p>
<p>Any suggestions are welcome :).
Thanks.</p>

opstalj
<p>Issue looks to be that the istio-cni plugin changes the iptables and a re-direct of the health probe check happens towards the application.
However, the re-direct happens to the localhost: and the application is not listening there for the health probes ... only on the : ...</p>
<p>After changing the iptables to a more proper re-direct, the health probe could get responded fine with a 200 OK and the pod became healthy.</p>
