<h2>Question</h2>

<p><a href="https://github.com/kubernetes/node-problem-detector" rel="nofollow noreferrer">node-problem-detector</a> is mentioned in <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/monitor-node-health/#node-problem-detector" rel="nofollow noreferrer">Monitor Node Health</a> documentation if K8S. How do we use it if it is not in GCE? Does it feed information to Dashboard or provide API metrics?</p>

<p><em>"This tool aims to make various node problems visible to the upstream layers in cluster management stack. It is a daemon which runs on each node, detects node problems and reports them to apiserver."</em></p>

<p>Err Ok but... What does that actually mean? How can I tell if it went to the api server? 
<br><strong>What does the before and after look like? Knowing that would help me understand what it's doing.</strong> </p>

<p><strong>Before installing Node Problem Detector I see:</strong><br></p>

<pre><code>Bash# kubectl describe node ip-10-40-22-166.ec2.internal | grep -i condition -A 20 | grep Ready -B 20
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Thu, 20 Jun 2019 12:30:05 -0400   Thu, 20 Jun 2019 12:30:05 -0400   WeaveIsUp                    Weave pod has set this
  OutOfDisk            False   Thu, 20 Jun 2019 18:27:39 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 20 Jun 2019 18:27:39 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 20 Jun 2019 18:27:39 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 20 Jun 2019 18:27:39 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 20 Jun 2019 18:27:39 -0400   Thu, 20 Jun 2019 12:30:14 -0400   KubeletReady                 kubelet is posting ready status
</code></pre>

<p><strong>After installing Node Problem Detector I see:</strong><br></p>

<pre><code>Bash# helm upgrade --install npd stable/node-problem-detector -f node-problem-detector.values.yaml 
Bash# kubectl rollout status daemonset npd-node-problem-detector #(wait for up) 
Bash# kubectl describe node ip-10-40-22-166.ec2.internal | grep -i condition -A 20 | grep Ready -B 20 
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  DockerDaemon         False   Thu, 20 Jun 2019 22:06:17 -0400   Thu, 20 Jun 2019 22:04:14 -0400   DockerDaemonHealthy          Docker daemon is healthy
  EBSHealth            False   Thu, 20 Jun 2019 22:06:17 -0400   Thu, 20 Jun 2019 22:04:14 -0400   NoVolumeErrors               Volumes are attaching successfully
  KernelDeadlock       False   Thu, 20 Jun 2019 22:06:17 -0400   Thu, 20 Jun 2019 22:04:14 -0400   KernelHasNoDeadlock          kernel has no deadlock
  ReadonlyFilesystem   False   Thu, 20 Jun 2019 22:06:17 -0400   Thu, 20 Jun 2019 22:04:14 -0400   FilesystemIsNotReadOnly      Filesystem is not read-only
  NetworkUnavailable   False   Thu, 20 Jun 2019 12:30:05 -0400   Thu, 20 Jun 2019 12:30:05 -0400   WeaveIsUp                    Weave pod has set this
  OutOfDisk            False   Thu, 20 Jun 2019 22:07:10 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure       False   Thu, 20 Jun 2019 22:07:10 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Thu, 20 Jun 2019 22:07:10 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Thu, 20 Jun 2019 22:07:10 -0400   Thu, 20 Jun 2019 12:29:44 -0400   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Thu, 20 Jun 2019 22:07:10 -0400   Thu, 20 Jun 2019 12:30:14 -0400   KubeletReady                 kubelet is posting ready status
</code></pre>

<p>Note I asked for help coming up with a way to see this for all nodes, Kenna Ofoegbu came up with this super useful and readable gem: </p>

<pre><code>zsh# nodes=$(kubectl get nodes | sed '1d' | awk '{print $1}') &amp;&amp; for node in $nodes; do;  kubectl describe node | sed -n '/Conditions/,/Ready/p' ; done
Bash# (same command, gives errors)
</code></pre>

<p><br>
Ok so now I know what Node Problem Detector does but... what good is adding a condition to the node, how do I use the condition to do something useful? <br></p>

<p><strong>Question: How to use Kubernetes Node Problem Detector?</strong> 
<br><strong>Use Case #1: Auto heal borked nodes</strong> 
<br>Step 1.) Install Node Problem Detector, so it can attach new condition metadata to nodes.
<br>Step 2.) Leverage Planetlabs/draino to cordon and drain nodes with bad conditions. 
<br>Step 3.) Leverage <a href="https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler" rel="noreferrer">https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler</a> to auto heal. (When the node is cordon and drained it'll be marked unscheduleable, this will trigger a new node to be provisioned, and then the bad node's resource utilization will be super low which will cause the bad node to get deprovisioned)</p>

<p>Source: <a href="https://github.com/kubernetes/node-problem-detector#remedy-systems" rel="noreferrer">https://github.com/kubernetes/node-problem-detector#remedy-systems</a></p>

<p><br><strong>Use Case #2: Surface the unhealthy node event so that it can be detected by Kubernetes, and then injested into your monitoring stack so you have an auditable historic record that the event occurred and when.</strong> 
<br>These unhealthy node events are logged somewhere on the host node, but usually, the host node is generating so much noisy/useless log data that these events aren't usually collected by default.
<br>Node Problem Detector knows where to look for these events on the host node and filters out the noise when it sees the signal of a negative outcome it'll post it to its pod log, which isn't noisy. 
<br>The pod log is likely getting ingested into an ELK and Prometheus Operator stack, where it can be detected, alerted on, stored, and graphed. </p>

<p>Also, note that nothing is stopping you from implementing both use cases. </p>

<hr>

<p><strong>Update, added a snippet of node-problem-detector.helm-values.yaml file per request in comment:</strong> </p>

<pre><code>  log_monitors:
#https://github.com/kubernetes/node-problem-detector/tree/master/config contains the full list, you can exec into the pod and ls /config/ to see these as well.
    - /config/abrt-adaptor.json #Adds ABRT Node Events (ABRT: automatic bug reporting tool), exceptions will show up under "kubectl describe node $NODENAME | grep Events -A 20"
    - /config/kernel-monitor.json #Adds 2 new Node Health Condition Checks "KernelDeadlock" and "ReadonlyFilesystem"
    - /config/docker-monitor.json  #Adds new Node Health Condition Check "DockerDaemon" (Checks if Docker is unhealthy as a result of corrupt image)
#    - /config/docker-monitor-filelog.json #Error: "/var/log/docker.log: no such file or directory", doesn't exist on pod, I think you'd have to mount node hostpath to get it to work, gain doesn't sound worth effort.
#    - /config/kernel-monitor-filelog.json #Should add to existing Node Health Check "KernelDeadlock", more thorough detection, but silently fails in NPD pod logs for me.   

  custom_plugin_monitors: #[]
# Someone said all *-counter plugins are custom plugins, if you put them under log_monitors, you'll get #Error: "Failed to unmarshal configuration file "/config/kernel-monitor-counter.json""
    - /config/kernel-monitor-counter.json #Adds new Node Health Condition Check "FrequentUnregisteredNetDevice"
    - /config/docker-monitor-counter.json #Adds new Node Health Condition Check "CorruptDockerOverlay2"
    - /config/systemd-monitor-counter.json #Adds 3 new Node Health Condition Checks "FrequentKubeletRestart", "FrequentDockerRestart", and "FrequentContainerdRestart"    
</code></pre>
