<p>I would like to be able to see all of the various things that happened to a kube cluster on a timeline, including when nodes were found to be dead, when new nodes were added, when pods crashed and when they were restarted.</p>

<p>So far the best that we have found is <code>kubectl get event</code> but that seems to have a few limitations:</p>

<ul>
<li>it doesn't go back in time that far (I'm not sure how far it goes back. A day?)</li>
<li>it combines similar events and orders the resulting list by the time of the latest event in each group. This makes it impossible to know what happened during some time range since events in that range may have been combined with later events outside the range.</li>
</ul>

<p>One idea that I have is to write a pod that will use the API to watch the stream of events and log them to a file. This would let us control retention and it seems that events that occur while we are watching will not be combined, solving the second problem as well.</p>

<p>What are other people doing about this?</p>

<ul>
<li><p>My understanding is that Kubernetes itself dedups events, documented here:
<a href="https://github.com/kubernetes/kubernetes/blob/master/docs/design/event_compression.md" rel="noreferrer">https://github.com/kubernetes/kubernetes/blob/master/docs/design/event_compression.md</a> 
Once that happens, there is no way to get the individual events back.</p>

<p>See <a href="https://github.com/kubernetes/kubernetes/issues/36304" rel="noreferrer">https://github.com/kubernetes/kubernetes/issues/36304</a> for complaints how that loses info.  <a href="https://github.com/kubernetes/kubernetes/pull/46034" rel="noreferrer">https://github.com/kubernetes/kubernetes/pull/46034</a> at least improved the message.  See also <a href="https://github.com/kubernetes/enhancements/pull/1291" rel="noreferrer">https://github.com/kubernetes/enhancements/pull/1291</a> KEP for recent discussion and proposal to improve usability in kubectl.</p></li>
<li><p>How long events are retained?  Their "time-to-live" is apparently controlled by kube-apiserver <code>--event-ttl</code> option, defaults to 1 hour:
<a href="https://github.com/kubernetes/kubernetes/blob/da53a247633/cmd/kube-apiserver/app/options/options.go#L71-L72" rel="noreferrer">https://github.com/kubernetes/kubernetes/blob/da53a247633/cmd/kube-apiserver/app/options/options.go#L71-L72</a></p>

<p>You can raise this.  Might require more resources for <code>etcd</code> — from what I saw in some 2015 github discussions, event TTL used to be 2 days, and events were the main thing stressing <code>etcd</code>...</p></li>
</ul>

<p>In a pinch, it might be possible to figure out what happened earlier from various log, especially the kubelet logs?</p>

<h3>Saving events</h3>

<ul>
<li><p>Running <code>kubectl get event -o yaml --watch</code> into a persistent file sounds like a simple thing to do.  I <em>think</em> when you watch events as they arrive, you see them pre-dedup.</p></li>
<li><p>Heapster can send events to some of the supported sinks:
<a href="https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md" rel="noreferrer">https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md</a></p></li>
<li><p><a href="https://github.com/heptiolabs/eventrouter" rel="noreferrer">Eventrouter</a> can send events to various sinks: <a href="https://github.com/heptiolabs/eventrouter/tree/master/sinks" rel="noreferrer">https://github.com/heptiolabs/eventrouter/tree/master/sinks</a></p></li>
</ul>
