<p>The example of <a href="https://kubernetes.io/docs/tutorials/stateful-application/zookeeper/" rel="nofollow noreferrer">running ZooKeeper in Kubernetes</a> shows how pods can be rescheduled on different nodes following a node being drained from connections, typically because maintenance has to be performed on it.</p>

<p>The pods are rescheduled with the same identity that they have before, in this case the <code>myid</code> ZooKeeper servers, corresponding to the incremental number of each pod <code>zk-0</code>, <code>zk-1</code> and so on.</p>

<p>If a node is not responding (possibly due to overload or a network problem), is it possible for Kubernetes to reschedule a pod on another node while the original pod is still running?</p>

<p>It seems <a href="https://github.com/kubernetes/kubernetes/issues/27427" rel="nofollow noreferrer">some of this behavior has been specified explicitly</a>, but I don't know how to verify it short of setting up multiple nodes on a cloud provider and trying it.</p>

<p>If a node is unresponsive, Kubernetes >=1.5 will <strong>not</strong> reschedule a pod with the same identity until it is confirmed that it has been terminated. The behavior with respect to StatefulSet is detailed <a href="https://kubernetes.io/docs/tasks/manage-stateful-set/delete-pods/#statefulset-considerations" rel="nofollow noreferrer">here</a>.</p>

<blockquote>
  <p>Kubernetes (versions 1.5 or newer) will not delete Pods just because a
  Node is unreachable. The Pods running on an unreachable Node enter the
  ‘Terminating’ or ‘Unknown’ state after a timeout. Pods may also enter
  these states when the user attempts graceful deletion of a Pod on an
  unreachable Node. The only ways in which a Pod in such a state can be
  removed from the apiserver are as follows: </p>
  
  <ul>
  <li>The Node object is
  deleted (either by you, or by the Node Controller). </li>
  <li>The kubelet on
  the unresponsive Node starts responding, kills the Pod and removes the
  entry from the apiserver. </li>
  <li>Force deletion of the Pod by the user.</li>
  </ul>
</blockquote>

<p>Since the name of the pod is a lock, we never creating two pods with the same identity, giving us 'at-most-one' semantics for StatefulSets. The user can override this behavior by performing force-deletions (and manually guaranteeing that the pod is fenced) but there is no automation that can lead to two pods with the same identity.</p>

<p>The changes in Kubernetes 1.5 ensure that we prioritize safety in case of StatefulSets. <a href="https://kubernetes.io/docs/admin/node/#condition" rel="nofollow noreferrer">Node Controller documentation</a> has some details about the change. You can read the full proposal <a href="https://github.com/kubernetes/kubernetes/pull/34160/files" rel="nofollow noreferrer">here</a>.</p>
