<p>We are running as hosted Kubernetes cluster on Google Cloud (GKE) and scraping it with Prometheus.</p>

<p>My Question is similar to <a href="https://stackoverflow.com/questions/39349744/kubernetes-prometheus-metrics-for-running-pods-and-nodes">this</a> one, but I'd like to know what are the most important metrics to look out for in the K8s Cluster and possibly alert on?</p>

<p>This is rather a K8s then a Prometheus question, but I'd really appreciate some hints. Please let me know if my question is to vague, so I can refine it.</p>

<p>etcd is the foundation of Kubernetes. So having a good set of alerts for it is important.
We wrote <a href="https://coreos.com/blog/developing-prometheus-alerts-for-etcd.html" rel="noreferrer">this blog post</a> and creating alerting rules for it and provided a base set at the end.</p>

<p>Further sources of important metrics in the Prometheus format are the Kubelet and cAdvisor, API servers, and  the fairly new <a href="https://github.com/kubernetes/kube-state-metrics" rel="noreferrer">kube-state-metrics</a>.
For those, I'm not aware of any public alerting rule sets as for etcd, unfortunately.</p>

<p>Generally, you want to ensure that the components as applications work flawlessly, e.g:</p>

<ul>
<li>Are my kubelets/API servers running/reachable? (<code>up</code> metric)</li>
<li>Are their response latency and error rates within bounds?</li>
<li>Can the API servers reach etcd?</li>
</ul>

<p>Then there's the Kubernetes business logic aspect, e.g:</p>

<ul>
<li>Are there pods that have been in non-ready/crashloop state forever?</li>
<li>Do I have enough CPU/memory capacity in my cluster?</li>
<li>Are my deployment replica expectations fulfilled?</li>
</ul>

<p>That's no drop-in solution unfortunately, but writing alerting rules roughly covering the scope of the above examples should get you quite far.</p>
