<p>I have a stackdriver log based metric tracking GKE pod restarts. </p>

<p>I'd like to alert via email if the number of alerts breaches a predefined threshold. </p>

<p>I'm unsure as what thresholds I need to set inroder to trigger the alert via stackdriver. I have three pods via deployed service.</p>

<p>You should use the Logs Viewer and create a filter:</p>

<p>As a resource you should choose <code>GKE Cluster Operations</code> and add a filter.</p>

<p>Filter might look like this:</p>

<pre><code>resource.type="k8s_cluster"
resource.labels.cluster_name="&lt;CLUSTER_NAME&gt;"
resource.labels.location="&lt;CLUSTR_LOCATION&gt;"
jsonPayload.reason="Killing"
</code></pre>

<p>After that create a custom metric by clicking on <code>Create metric</code> button.</p>

<p>Then you can <code>Create alert from metric</code> by clicking on created metric in <code>Logs-based metrics</code>.</p>

<p>Then setting up a Configuration for triggers and conditions and threshold.</p>

<p>As for the correct Threshold, I would take the average amount of restarts from past time period and make it a bit more for alerting.</p>
