title: How to persist pod lifecycle states and events in Google Kubernetes Engine?
body: <p>We are using Google Kubernetes Engine on 1.9.6-gke.1, and have a cluster with several nodepools for which we enable auto-scaling because the nodes in them contain attached GPUs (p100s).</p>
  
  <p>Sometimes we run jobs overnight via a Kubernetes Pod on a node that was brought up due to a triggered auto-scaling event, and many hours later return to find that the pod has disappeared because the pod has terminated, in some unknown state, and since no other pod is scheduled to the node for 10 minutes, the node it ran on has been drained and removed. </p>
  
  <p>That is, once the node is gone the pod disappears from the perspective of the Kubernetes logs and control plane ie.running things like <code>kubectl get pods</code> and <code>kubectl describe pod</code>. We would like to be able to know the status of these pods at the time of termination, eg. 'Completed, Error, OOM'. Is there a way to have this pod lifecycle information logged in Google Cloud Platform, perhaps via Stackdriver or other? If it's already available where would we find it?</p>
  
  <p>Note this is for pods for which the node the pod ran on is no longer in the cluster.</p>
  
  <p>Thanks in advance!</p>