<p>We have an application deployed on GKE with a total of 10 pods running and serving the application.
I am trying to find the metrics using which I can create an alert when my Pod goes down or is there a way to check the status of Pods so that I can set up an alert based on that condition?</p>
<p>I explored GCP and looked into their documentation but couldn't find anything. What I could find is one metric below but I don't know what it measures.
To me it looks like a number of times Kubernetes thinks a pod has died and it restarts the pod.</p>
<pre><code>Metric: kubernetes.io/container/restart_count
Resource type: k8s_container
</code></pre>
<p>Any advice on this is highly appreciated as we can improve our monitoring based on this metric</p>
<p><a href="https://i.stack.imgur.com/rbPiw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/rbPiw.png" alt="GCP alerting policy creation" /></a></p>

<p>That metric is the same you are right it will the count of POD restart.</p>
<blockquote>
<p>Number of times the container has restarted. Sampled every 60 seconds.
After sampling, data is not visible for up to 120 seconds.</p>
</blockquote>
<p>Read more at : <a href="https://cloud.google.com/monitoring/api/metrics_kubernetes" rel="nofollow noreferrer">https://cloud.google.com/monitoring/api/metrics_kubernetes</a></p>
<p><strong>Or</strong></p>
<p>You can use Prometheus to get the metrics and monitor with <strong>Grafana</strong></p>
<pre><code>sum(kube_pod_container_status_restarts_total{cluster=&quot;$cluster&quot;,namespace=&quot;$namespace&quot;,pod=~&quot;$service.*&quot;})
</code></pre>
<p>This will give the value of the POD restart count.</p>
<p><strong>OR</strong></p>
<p>You can also use the <strong>BotKube</strong> : <a href="https://www.botkube.io/installation/" rel="nofollow noreferrer">https://www.botkube.io/installation/</a></p>
<p>You can set to notify when your readiness liveness fails to slack notification etc..</p>
<p><strong>Or</strong></p>
<p>You write your own script and run it on Kubernetes to monitor and notify when any POD restart in cluster.</p>
<p>Example github : <a href="https://github.com/harsh4870/Slack-Post-On-POD-Ready-State" rel="nofollow noreferrer">https://github.com/harsh4870/Slack-Post-On-POD-Ready-State</a></p>
<p>This script notifies in slack when POD becomes ready after deployment, you can change it to monitor the restart count.</p>
<p>i would recommend using <strong>Prometheus, Grafana</strong> option, however, <strong>stackdriver</strong> is Good but i am not Google employee.</p>
