<p>I must surely be missing something obvious. GCP provides me with all sorts of visible indications when a container has failed to start. For example:</p>
<p><a href="https://i.stack.imgur.com/Iylay.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Iylay.png" alt="Failed deployment" /></a></p>
<p><a href="https://i.stack.imgur.com/YmLDq.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/YmLDq.png" alt="Container Status" /></a></p>
<p>But I cannot for the life of me figure out how to make it issue an alert when the container status is not OK.</p>
<p>How is it done?</p>

<ul>
<li>
<blockquote>
<p><a href="https://cloud.google.com/kubernetes-engine/docs/troubleshooting?&amp;_ga=2.52678688.-945193803.1584444246#CrashLoopBackOff" rel="nofollow noreferrer"><code>CrashLoopBackOff</code></a> indicates that a container is repeatedly crashing after restarting. A container might crash for many reasons, and checking a Pod's logs might aid in troubleshooting the root cause.</p>
</blockquote>
</li>
</ul>
<p>Apart from the error text message <code>Does not have minimum availability</code>, there could be other error text messages such as <code>Failed to pull image</code>. However, I recommend you to identify error text messages  which are appropriate for your environment. You can check with <code>kubectl logs &lt;pod_name&gt;</code> or on Log Viewer.</p>
<p>For your reference, here are explanations for <a href="https://cloud.google.com/kubernetes-engine/docs/troubleshooting#workload_issues" rel="nofollow noreferrer">pod issues</a>:</p>
<ol>
<li><strong>CrashLoopBackOff</strong> means the container was downloaded but failed to run</li>
<li><strong>ImagePullBackOff</strong> means the image was not downloaded</li>
<li><strong>&quot;Does not have minimum availability&quot;</strong> means that there are no resources available on cluster but not specific to a lack of resources. For instance there maybe nodes available but the pod is not scheduleable on them per the deployment.</li>
<li><strong>&quot;Insufficient cpu&quot;</strong> means there is insufficient cpu on the nodes.</li>
<li><strong>&quot;Unschedulable&quot;</strong> indicates that your Pod cannot be scheduled because of insufficient resources or some configuration error.</li>
</ol>
<hr />
<p>With that in mind, Here is the step-by-step for creating a Log based Metric for later creating an alert based on it.</p>
<ol>
<li><p>Setup a <a href="https://cloud.google.com/logging/docs/logs-based-metrics#getting_started" rel="nofollow noreferrer">Logs-based Metric</a> using the parameters:</p>
<pre><code>resource.type=&quot;k8s_pod&quot;
severity&gt;=WARNING
unschedulable
</code></pre>
<p>You can replace the filter to something that is more appropriate for your case.</p>
</li>
<li><p>Create a label in the metric that will allow you to identify the pod that was <code>unschedulable</code> (or other status). This will also help with grouping when you create the alert for a failing pod.</p>
</li>
<li><p>In Stackdriver Monitoring, <a href="https://cloud.google.com/monitoring/alerts/using-alerting-ui" rel="nofollow noreferrer">create an alert</a> with the following parameters.</p>
<ul>
<li>Set the resource type to <code>k8s_pod</code></li>
<li>Set the metric to the one you created in step 1</li>
<li>Set <code>Group By</code> to the <code>pod_name</code> (also created in step 1)</li>
<li>In the advanced aggregation section set the aligner to <code>sum</code> and the Alignment Period to <code>5m</code> (or what you thinks is more appropriate).</li>
<li>Configure the condition triggers <code>For</code> to more than 1 minute to prevent the alert from firing over and over. This can also be configured per your requirement.</li>
</ul>
</li>
</ol>
<p>I hope this information is helpful, If you have any questions let me know in the comments.</p>
