<p>I'm looking to perform an automatic rolling update of a Kubernetes cluster in a way that's ignorant of the details of the applications running on the cluster. In principle, PodDisruptionBudget should facilitate this.</p>

<p>Here's the snag: there's an Elasticsearch cluster running on this Kubernetes cluster, and I can't find a way to correctly express the "OK to evict an ES Pod" signal. Specifically, this seems to be a case where <strong>the "this Pod can receive traffic" and "this Pod can be evicted" signals can't both be represented by <code>readinessProbe</code>.</strong></p>

<p>This ES cluster's indices have <code>number_of_replicas: 1</code>, and there's a PDB with <code>maxUnavailable: 1</code>. The ES Pods each specify a readiness probe that requests <code>/_cluster/health?wait_for_status=yellow</code>.</p>

<p>As-is, if we evict an ES Pod, the replacement Pod will join the ES cluster, start up, and return to ready status while the ES cluster <em>as a whole</em> is still yellow and replicating shards (and thus it's still unsafe to evict any additional ES Pods).</p>

<p>Has anyone worked around this successfully? Am I misunderstanding the semantics of probes/PDBs?</p>

<hr>

<p>Some options we've considered:</p>

<ul>
<li>Using <code>wait_for_status=green</code> in the readiness probe would mean that <em>all</em> ES Pods become unready when the ES cluster health is yellow.</li>
<li>Increasing the ES indices' <code>number_of_replicas</code> to <code>2</code> only slightly reduces the probability of a rolling update damaging the ES cluster (assume these shards are slow to replicate).</li>
<li>Ditto with setting a large <code>initialDelaySeconds</code> on the <code>readinessProbe</code>. It's possible for that to undershoot the time for shard replication to complete.</li>
<li>Ditto with using a <code>preStop</code> hook (<a href="https://github.com/helm/charts/blob/0a5ddac35d7e1a8820f3edb5198a648f706146ef/incubator/elasticsearch/templates/configmap.yaml#L122-L143" rel="nofollow noreferrer">this is the approach the community Helm chart appears to take</a>) and a long grace period.</li>
<li>Decreasing the PDB's <code>maxUnavailable</code> to <code>0</code> means that the rolling update has to be run by a human who can remove the PDB, evaluate the status of the ES cluster, etc.</li>
<li>A hypothetical, um, <code>evictablenessProbe</code> that checked <code>wait_for_status=green</code> would work, but no such API exists.</li>
</ul>

Jesse Dubay
<p>First of all, by all means, saves yourself a ton of time and trouble and use the Helm chart:
<a href="https://github.com/helm/charts/tree/master/incubator/elasticsearch" rel="nofollow noreferrer">https://github.com/helm/charts/tree/master/incubator/elasticsearch</a></p>

<p>But just in case you can't, or in case it helps others, I think what you're looking for is <code>/_cluster/health?local=true</code>, e.g:</p>

<pre><code>    readinessProbe:
      httpGet:
        path: /_cluster/health?local=true
        port: 9200
</code></pre>

<p>Hope this helps!</p>
