<p>Consider a Statefulset (Cassandra using offical K8S example) across 3 Availability zones:</p>

<ul>
<li>cassandra-0 -> zone a</li>
<li>cassandra-1 -> zone b</li>
<li>cassandra-2 -> zone c</li>
</ul>

<p>Each Cassandra pod uses an EBS volume. So there is automatically an affinity. For instance, cassandra-0 cannot move to "zone-b" because its volume is in "zone-a". All good.</p>

<p>If some Kubernetes nodes/workers fail, they will be replaced. The pods will start again on the new node and be re-attached their EBS volume. Looking like nothing happened.</p>

<p>Now if the entire AZ "zone-a" goes down and is unavailable for some time (meaning cassandra-0 cannot start anymore due to affinity for EBS in the same zone). You are left with:</p>

<ul>
<li>cassandra-1 -> zone b</li>
<li>cassandra-2 -> zone c</li>
</ul>

<p>Kubernetes will never be able to start cassandra-0 for as long as "zone-a" is unavailable. That's all good because cassandra-1 and cassandra-2 can serve requests.</p>

<p>Now if on top of that, another K8S node goes down or you have setup auto-scaling of your infrastructure, you could end up with cassandra-1 or cassandra-2 needed to move to another K8S node.
It shouldn't be a problem.</p>

<p><strong>However</strong> from my testing, K8S will not do that because the pod cassandra-0 is offline. It will never self-heal cassandra-1 or cassandra-2 (or any cassandra-X) because it wants  cassandra-0 back first. And  cassandra-0 cannot start because it's volume is in a zone which is down and not recovering.</p>

<p>So if you use <strong>Statefulset + VolumeClaim + across zones</strong> 
AND you experience an <strong>entire AZ failure</strong>
AND you experience an EC2 failure in another AZ or have auto-scaling of your infrastructure</p>

<p>=> then you will loose all your Cassandra pods. Up until zone-a is back online</p>

<p>This seems like a dangerous situation. Is there a way for a stateful set to not care about the order and still self-heal or start more pod on cassandra-3, 4, 5, X?</p>

<p>Two options: </p>

<p>Option 1: <strong>use podManagementPolicy</strong> and set it to Parallel. 
The pod-1 and pod-2 will crash a few times until the seed node (pod-0) is available. This happens when creating the statefulset the first time.
Also note that Cassandra documentation used to recommend NOT creating multiple nodes in parallel but it seems recent updates makes this not true. Multiple nodes can be added to the cluster at the same time</p>

<p>Issue found: if using 2 seed nodes, you will get a split brain scenario. Each seed node will be created at the same time and create 2 separate logical Cassandra clusters</p>

<p>Option 1 b: <strong>use podManagementPolicy</strong> and set it to Parallel and use ContainerInit.
Same as option 1 but use an initContainer <a href="https://kubernetes.io/docs/concepts/workloads/pods/init-containers/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/workloads/pods/init-containers/</a>.
The init container is a short lived container which has for role to check that the seed node is available before starting the actual container. This is not required if we are happy for the pod to crash until the seed node is available again
The problem is that Init Container will always run which is not required. We want to ensure the Cassandra cluster was well formed the first time it was created. After that it does not matter</p>

<p>Option 2: <strong>create 3 different statefulets</strong>.</p>

<p>1 statefulset per AZ/Rack. Each statefulset has constraints so it can run only on nodes in the specific AZ. I've also got 3 storage classes (again constraint to a particular zone), to make sure the statefulset does not provision EBS in the wrong zone (statefulset does not handle that dynamically yet)
In each statefulset I've got a Cassandra seed node (defined as environment variable CASSANDRA_SEEDS which populates SEED_PROVIDER at run time). That makes 3 seeds which is plenty.
My setup can survive a complete zone outage thanks to replication-factor=3</p>

<p>Tips:</p>

<ul>
<li>the list of seed node contains all 3 nodes separated by commas:
"cassandra-a-0.cassandra.MYNAMESPACE.svc.cluster.local, cassandra-b-0.cassandra.MYNAMESPACE.svc.cluster.local, cassandra-c-0.cassandra.MYNAMESPACE.svc.cluster.local"</li>
<li>Wait until the first seed (cassandra-a-0) is ready before creating the other 2 statefulsets. Otherwise you get a split brain. This is only an issue when you create the cluster. After that, you can loose one or two seed nodes without impact as the third one is aware of all the others.</li>
</ul>
