<p>Cannot upgrade at the moment and I am stuck with Elasticsearch 5.6.</p>
<p>Trying to get cluster-mode working on GKE with success, but one major problem. Whenever I do a rolling restart (<code>StatefulSet</code>) of the Elasticsearch cluster I can get the cluster into a red state from which it cannot recover. Doesn't happen all the time though and works flawlessly if I only kill one node (out of three).</p>
<p>For some reason the restarted node is not able to recover their primary/replica shards on startup. Or the next node is terminated to early before shard allocation has been finished?</p>
<p>In any way I was looking for an option where I just keep every index fully on every node (mix of primary and replica shards). Not sure how to configure this properly.</p>
<p>How do I configure a Elasticsearch cluster with <strong>three</strong> nodes and two indices to survive an outage of <strong>two</strong> nodes?</p>
<p>EDIT: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/delayed-allocation.html" rel="nofollow noreferrer">https://www.elastic.co/guide/en/elasticsearch/reference/5.6/delayed-allocation.html</a> is probably what I need, isn't it?</p>

Moritz Schmitz v. Hülst
<p>A <a href="https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/" rel="nofollow noreferrer">StatefulSet</a> is a way to create stateful components within a distributed systems. Thus, Pods are intended execute independently with its own state. In fact a StatefulSet have a unique ordinal index, a stable network identity, and if stops it's data are not visible from the running nodes.
Here the block diagram to access the PV/PVC:</p>
<pre><code> PV - PVC - POD1 \
 PV - PVC - POD2 - StatefulSet  
 PV - PVC - POD3 /
</code></pre>
<p>What you cannot reach with StatefulSet (differently than Deployment) is to access the POD's data if that POD stops. This means that if you have 3 PODs: if 1 POD stops your application may continue to work, if 2 PODs stops your application likely doesn't work.</p>
<p>If you need to continue to access the data if some PODs stop, you probably need a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="nofollow noreferrer">Deployment</a>, instead, where every pod access the same data on PV-PVC:</p>
<pre><code>          / POD1 \
 PV - PVC - POD2 - Deployment
          \ POD3 /
</code></pre>
<p>So, in order to meet your requirements, try to use Deployments instead of StatefulSets.</p>
<p>Then you can distribute PODs on each node, following <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/" rel="nofollow noreferrer">the official docs</a>, in order to enable application availability in case 2 out of 3 nodes will not be reachable.
To do that you can use the following yaml example:</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
spec:
  replicas: 3
template:
  spec:
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: foo
        matchLabelKeys:
          - pod-template-hash
</code></pre>
