<p>I'm experimenting with Cassandra and Redis on Kubernetes, using <a href="https://github.com/kubernetes/kubernetes/tree/v1.5.1/examples/storage/cassandra#step-2-use-a-statefulset-to-create-cassandra-ring" rel="nofollow noreferrer">the examples for v1.5.1</a>.</p>

<ul>
<li>With a Cassandra StatefulSet, if I shutdown a node without draining or deleting it via <code>kubectl</code>, that node's Pod stays around forever (at least over a week, anyway), without being moved to another node.</li>
<li>With Redis, even though the pod sticks around like with Cassandra, <a href="https://github.com/kubernetes/kubernetes/tree/v1.5.1/examples/storage/redis#conclusion" rel="nofollow noreferrer">the sentinel service</a> starts a new pod, so the number of functional pods is always maintained.</li>
</ul>

<p>Is there a way to automatically move the Cassandra pod to another node, if a node goes down? Or do I have to drain or delete the node manually?</p>

<p>Please refer to the documentation <a href="http://kubernetes.io/docs/tasks/manage-stateful-set/delete-pods/#deleting-pods" rel="nofollow noreferrer">here</a>.</p>

<blockquote>
  <p>Kubernetes (versions 1.5 or newer) will not delete Pods just because a
  Node is unreachable. The Pods running on an unreachable Node enter the
  ‘Terminating’ or ‘Unknown’ state after a timeout. Pods may also enter
  these states when the user attempts graceful deletion of a Pod on an
  unreachable Node. The only ways in which a Pod in such a state can be
  removed from the apiserver are as follows: </p>
  
  <ul>
  <li>The Node object is deleted (either by you, or by the Node Controller).</li>
  <li>The kubelet on the unresponsive Node starts responding,
  kills the Pod and removes the entry from the apiserver. </li>
  <li>Force deletion of the Pod by the user.</li>
  </ul>
</blockquote>

<p>This was a behavioral change introduced in kubernetes 1.5, which  allows StatefulSet to prioritize safety. </p>

<p>There is no way to differentiate between the following cases:</p>

<ol>
<li>The instance being shut down without the Node object being deleted.</li>
<li>A network partition is introduced between the Node in question and the kubernetes-master. </li>
</ol>

<p>Both these cases are seen as the kubelet on a Node being unresponsive by the Kubernetes master. If in the second case, we were to quickly create a replacement pod on a different Node, we may violate the at-most-one semantics guaranteed by StatefulSet, and have multiple pods with the same identity running on different nodes. At worst, this could even lead to split brain and data loss when running Stateful applications. </p>

<p>On most cloud providers, when an instance is deleted, Kubernetes can figure out that the Node is also deleted, and hence let the StatefulSet pod be recreated elsewhere. </p>

<p><strong>However, if you're running on-prem, this may not happen. It is recommended that you delete the Node object from kubernetes as you power it down, or have a reconciliation loop keeping the Kubernetes idea of Nodes in sync with the the actual nodes available.</strong></p>

<p>Some more context is in the <a href="https://github.com/kubernetes/kubernetes/issues/35145" rel="nofollow noreferrer">github issue</a>.</p>
