<p>We are trying a test setup with Kubernetes version 1.0.6 on AWS. </p>

<p>This setup involves pods for Cassandra (2-nodes), Spark (master, 2-workers, driver), RabbitMQ(1-node). Some the pods this setup die after a day or so</p>

<p>Is there way to get logs from Kubernetes on how/why they died?</p>

<p>When you try to restart died pods manually, you get some pods status as ''category/spark-worker is ready, container is creating' and the pod start never completes.</p>

<p>Only option in the scenario is to "kube-down.sh and then kube-up.sh" and go through entire setup from scratch. </p>

<p><code>kubectl describe ${POD_NAME}</code> or <code>kubectl logs ${POD_NAME} ${CONTAINER_NAME}</code> should give you more information to debug.</p>

<p>Please also see <a href="https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/application-troubleshooting.md#debugging-pods" rel="nofollow">https://github.com/kubernetes/kubernetes/blob/master/docs/user-guide/application-troubleshooting.md#debugging-pods</a> for general troubleshooting instructions.</p>

<p>EDIT: </p>

<p>After discussing in the comments, I think the problem with your node is that the node was unresponsive for >5 minutes (potentially due to high memory usage of influxdb). Node controller then deemed the node not ready and evicted all pods on the node. Note that pods managed by replication controllers would be re-created (with a different name), but pods created manually would not be recreated.</p>

<p>If you suspect influxdb memory usage is the root cause, you can try not running this pod to see if the problem resolves itself. Alternatively, you can change the memory limit of <a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/cluster-monitoring/influxdb/influxdb-grafana-controller.yaml#L28" rel="nofollow">influxdb container</a> to a smaller value.</p>

<p>EDIT2:</p>

<p>Some tips for finding out what happened to the node:</p>

<ol>
<li><p>Check <code>/var/log/kubelet.log</code>. This is the easiest approach.</p></li>
<li><p><code>kubectl describe nodes</code> or <code>kubectl get events | grep &lt;node_name&gt;</code> (for older version of kubernetes)</p></li>
</ol>

<p>This command would give you the events associated with the node status. However, the events are flushed every two hours, so you would need to run this command within the window of time after your node encounters the problem.</p>

<ol start="3">
<li><code>kubectl get node &lt;node_name&gt; -o yaml --watch</code> lets you monitor the node object, including its status in yaml. This would be updated periodically.</li>
</ol>
