<p>I was testing my kubernetes services recently. And I found it's very unreliable. Here are the situation:<br>
1. The test service 'A' which receives HTTP requests at port 80 has five pods deployed on three nodes.<br>
2. An nginx ingress was set to route traffic outside onto the service 'A'.<br>
3. The ingress was set like this:</p>

<pre><code>apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-A
  annotations:    
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "1s"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "error timeout invalid_header http_502 http_503 http_504"
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
spec:
  rules:
  - host: &lt;test-url&gt;
    http:
      paths:
      - path: /
        backend:
          serviceName: A
          servicePort: 80
</code></pre>

<ol start="3">
<li>http_load was started on an client host and kept sending request to the ingress nginx at a speed of 1000 per-seconds. All the request were routed to the service 'A' in k8s and eveything goes well.</li>
</ol>

<p>When I restarted one of the nodes manually, things went wrong:<br>
<strong>In the next 3 minutes, about 20% requests were timeout, which is unacceptable in product environment.</strong>  </p>

<p>I don't know why k8s reacts so slow and is there a way to solve this problem?</p>

George Liang
<p>Thanks for @VAS's answer!<br>
Liveness probe is a way to solve this problem.<br>
But I finally figured out that what I want was passive health check, which the k8s dosen't surpport.<br>
And I solved this problem by introducing istio into my cluster. </p>
