<p>Recently we have created a cluster on Kubernetes Engine (GCP) and we started to notice a strange behavior on it. Every day the nodes are getting stopped and recreated automatically in a certain time of day, making applications unavailable for a few minutes.</p>

<p><strong>How the incidents are displayed in Stackdriver dashboard:</strong></p>

<p><a href="https://i.stack.imgur.com/PqgLq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PqgLq.png" alt="enter image description here"></a></p>

<p>In order to understand the root cause of the problem, I analyzed the logs in Stackdriver, taking as a reference the incident that happened today (<strong>2017-12-19</strong> <strong>12:22pm</strong>).</p>

<p><strong>Cluster log:</strong></p>

<p>The closest entry that exists related to the incident is just at <strong>12:26pm</strong> (probably the moment that the cluster was coming back).</p>

<p><a href="https://i.stack.imgur.com/ntAda.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ntAda.png" alt="enter image description here"></a></p>

<p><strong>Node log:</strong> </p>

<p>The instance log also doesn't seem to help too much. The records closest to the incident just appears at <strong>12:23pm</strong> (also after the instance start to come back).</p>

<p><a href="https://i.stack.imgur.com/DW5Zf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/DW5Zf.png" alt="enter image description here"></a></p>

<p>Has anyone ever been through this situation before or have any idea how can we debug it better and discover what is causing this behavior?</p>

<p>The cause of the incident apparently is not been shown in Stackdriver logs.</p>

<p>The described behavior is very similar to how the <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/preemptible-vm" rel="nofollow noreferrer">preemptible nodes in GKE</a> behave (they live a maximum of 24 hours).</p>

<p>If you're unsure if your nodes are preemptible, check the GCP WebUI (my sample<a href="https://i.stack.imgur.com/vI9ap.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/vI9ap.png" alt="enter image description here"></a> below, check the "Preemptible nodes" line), or via CLI:</p>

<pre><code>$ gcloud compute instances list | grep gke | awk '{print $4}'
</code></pre>

<p>If the CLI command will return "true", that means that the nodes are preemptible (see below):</p>

<pre><code>$ gcloud compute instances list | grep gke | awk '{print $4}'
true
true
true
</code></pre>

<p>Note: if you have multiple GKE clusters under the same project, after <code>grep</code> command add your GKE cluster name.</p>
