<p>having a local Kubernetes cluster (Master and 2 Worker Nodes) on my VirtualBox deployed with kubeadm:</p>
<pre><code>NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   2d21h   v1.19.2
node1    Ready    &lt;none&gt;   2d21h   v1.19.2
node2    Ready    &lt;none&gt;   2d21h   v1.19.2
</code></pre>
<p>the containerized application is a trivial node.js that displays &quot;HelloWorld&quot; from the URL http://ExternalIP:8080</p>
<pre><code>NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
helloworld-service   LoadBalancer   10.111.237.33   192.168.1.163   8080:30317/TCP   124m
kubernetes           ClusterIP      10.96.0.1       &lt;none&gt;          443/TCP          2d21h
</code></pre>
<p>The Deployment configured has replicas=2. Supposing that one Pod is running on node1 and the other on node2, if i run the command:</p>
<pre><code> curl http://ExternalIP:8080
</code></pre>
<p>in a for loop, from an external machine outside the cluster, the application gives the &quot;HelloWorld&quot; for every received request, as expected. Here the describe Service:</p>
<pre><code> Name:                     helloworld-service
 Namespace:                default
 Labels:                   &lt;none&gt;
 Annotations:              &lt;none&gt;
 Selector:                 app=hello-world-app
 Type:                     LoadBalancer
 IP:                       10.111.237.33
 External IPs:             192.168.1.163
 Port:                     &lt;unset&gt;  8080/TCP
 TargetPort:               8081/TCP
 NodePort:                 &lt;unset&gt;  30317/TCP
 Endpoints:                10.36.0.1:8081,10.44.0.1:8081
 Session Affinity:         None
 External Traffic Policy:  Cluster
 Events:                   &lt;none&gt;
</code></pre>
<p>Now, having the for loop active from the external machine, i want to test downtime of the application turning off one of the Worker Node. The result is:</p>
<pre><code> - about 30 seconds having no response and 503 service unavailable from application
 - the application starts working fine only after the Master declares the turned-off Node as NotRunning 
</code></pre>
<p>at this point the output of the cluster is:</p>
<pre><code>NAME     STATUS      ROLES    AGE     VERSION
master   Ready       master   2d21h   v1.19.2
node1    Ready       &lt;none&gt;   2d21h   v1.19.2
node2    NotReady    &lt;none&gt;   2d21h   v1.19.2
</code></pre>
<p>and application replies to every request.</p>
<p>From kubernetes.io documentation:</p>
<p><em>The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A Pod is considered ready when all of its containers are ready. One use of this signal is to control which Pods are used as backends for Services. When a Pod is not ready, it is removed from Service load balancers.</em></p>
<p>I changed my yaml configuration in the Deployment object, but still getting those 30 seconds of application downtime:</p>
<pre><code>  apiVersion: v1
  kind: Service
  metadata:
    name: helloworld-service

  spec:
    selector:
      app: hello-world-app

    ports:
    - protocol: TCP
      port: 8080
      targetPort: 8081

    type: LoadBalancer
    clusterIP: 10.111.237.33

    externalIPs:
    - 192.168.1.163

  status:
    loadBalancer:
      ingress:
      - ip: 192.168.1.224
</code></pre>
<hr />
<pre><code>  apiVersion: apps/v1

  kind: Deployment

  metadata:
    name: helloworld-deployment

  spec:
    selector:
      matchLabels:
        app: hello-world-app
    
    replicas: 2

    template:
      metadata:
        labels:
          app: hello-world-app

      spec:
        containers:
        - name: hello-world-container
          image: marcoif81/testmarcoif:latest

          ports:
          - containerPort: 8081

          livenessProbe:
            tcpSocket:
              port: 8081
            initialDelaySeconds: 15
            periodSeconds: 2
            failureThreshold: 2
          readinessProbe:
            tcpSocket:
              port: 8081
            initialDelaySeconds: 15
            periodSeconds: 2
            failureThreshold: 2
</code></pre>
<p>from the command here, execute during application downtime:</p>
<pre><code>  # kubectl describe pod &quot;pod_name&quot;
</code></pre>
<p>in the Events section no output is showned about probing:</p>
<pre><code>Events:
Type     Reason        Age   From               Message
----     ------        ----  ----               -------
Normal   Scheduled     89s   default-scheduler  Successfully assigned default/helloworld-deployment-7456bb6569-qrfnj to node2
Warning  FailedMount   88s   kubelet            MountVolume.SetUp failed for volume &quot;default-token-hjq76&quot; : failed to sync secret cache: timed out waiting for the condition
Normal   Pulling       87s   kubelet            Pulling image &quot;marcoif81/testmarcoif:latest&quot;
Normal   Pulled        86s   kubelet            Successfully pulled image &quot;marcoif81/testmarcoif:latest&quot; in 1.581369887s
Normal   Created       86s   kubelet            Created container hello-world-container
Normal   Started       85s   kubelet            Started container hello-world-container
Warning  NodeNotReady  1s    node-controller    Node is not ready
</code></pre>

Marco_81
<p>Reading the suggested doc k8s from the comment above by Matt, it's clear that readiness and liveness probing are not the features to monitor the worker nodes' health. As document says:</p>
<p><em>Monitoring the nodes' health: The node controller is responsible for updating the NodeReady condition of NodeStatus to ConditionUnknown when a node becomes unreachable (i.e. the node controller stops receiving heartbeats for some reason, for example due to the node being down), and then later evicting all the pods from the node (using graceful termination) if the node continues to be unreachable. (The default timeouts are 40s to start reporting ConditionUnknown and 5m after that to start evicting pods.) The node controller checks the state of each node every --node-monitor-period seconds</em></p>
<p>For testing purposes i changed the following timers:</p>
<ul>
<li><p>On Master Node: --node-monitor-grace-period        from Default: 40s --&gt; to 6s</p>
</li>
<li><p>On Worker Nodes: --node-status-update-frequency    3s</p>
</li>
</ul>
<p>more about timers can be found here:</p>
<p><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/" rel="nofollow noreferrer">https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/</a></p>
