<p>Currently I have setup kubernetes, with kube-state-metrics, prometheus and loki. For most things this work really well, but the one thing I am struggling with is finding the exact reason why containers might've restarted.</p>
<p>For running pods, it's quite easy to see, i.e. with <code>kubectl describe pod</code>, for example, I get the events below:</p>
<pre><code>kubectl describe pod pod-name
.....
Events:
  Type     Reason     Age                  From     Message
  ----     ------     ----                 ----     -------
  Warning  Unhealthy  19m (x9 over 3h29m)  kubelet  message-here
</code></pre>
<p>This is very useful for troubleshooting exactly why a container have a lot of restarts, especially if using probes. But there are also other useful events.</p>
<p>However I cannot see any way to save these kind of events, in either loki or prometheus. But maybe I am missing something. I had expected <code>kube-state-metrics</code> to include such information but it seems not to be the case, I also do not see it anywhere in Loki.</p>
<p>Any tips on how I can save such events?</p>

<p>Whilst Prometheus and KSM are more focussed around metrics, eg: 'N number of pod restarts', Loki can be used for capturing events.</p>
<p>A good overview guide is <a href="https://grafana.com/blog/2019/08/21/how-grafana-labs-effectively-pairs-loki-and-kubernetes-events/" rel="nofollow noreferrer">here</a> which uses <a href="https://github.com/heptiolabs/eventrouter" rel="nofollow noreferrer">eventrouter</a> to push events into a backend (Loki or Elasticsearch for example)</p>
