<p>Consider a Statefulset (Cassandra using offical K8S example) across 3 Availability zones:</p>

<ul>
<li>cassandra-0 -> zone a</li>
<li>cassandra-1 -> zone b</li>
<li>cassandra-2 -> zone c</li>
</ul>

<p>Each Cassandra pod uses an EBS volume. So there is automatically an affinity. For instance, cassandra-0 cannot move to "zone-b" because its volume is in "zone-a". All good.</p>

<p>If some Kubernetes nodes/workers fail, they will be replaced. The pods will start again on the new node and be re-attached their EBS volume. Looking like nothing happened.</p>

<p>Now if the entire AZ "zone-a" goes down and is unavailable for some time (meaning cassandra-0 cannot start anymore due to affinity for EBS in the same zone). You are left with:</p>

<ul>
<li>cassandra-1 -> zone b</li>
<li>cassandra-2 -> zone c</li>
</ul>

<p>Kubernetes will never be able to start cassandra-0 for as long as "zone-a" is unavailable. That's all good because cassandra-1 and cassandra-2 can serve requests.</p>

<p>Now if on top of that, another K8S node goes down or you have setup auto-scaling of your infrastructure, you could end up with cassandra-1 or cassandra-2 needed to move to another K8S node.
It shouldn't be a problem.</p>

<p><strong>However</strong> from my testing, K8S will not do that because the pod cassandra-0 is offline. It will never self-heal cassandra-1 or cassandra-2 (or any cassandra-X) because it wants  cassandra-0 back first. And  cassandra-0 cannot start because it's volume is in a zone which is down and not recovering.</p>

<p>So if you use <strong>Statefulset + VolumeClaim + across zones</strong> 
AND you experience an <strong>entire AZ failure</strong>
AND you experience an EC2 failure in another AZ or have auto-scaling of your infrastructure</p>

<p>=> then you will loose all your Cassandra pods. Up until zone-a is back online</p>

<p>This seems like a dangerous situation. Is there a way for a stateful set to not care about the order and still self-heal or start more pod on cassandra-3, 4, 5, X?</p>

<p>Starting with Kubernetes 1.7 you can tell Kubernetes to relax the StatefulSet ordering guarantees using the <code>podManagementPolicy</code> option (<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.9/#statefulsetspec-v1-apps" rel="nofollow noreferrer">documentation</a>). By setting that option to <code>Parallel</code> Kubernetes will no longer guarantee any ordering when starting or stopping pods and start pods in parallel. This can have an impact on your service discovery, but should resolve the issue you're talking about.</p>
