<p>I am using Google Cloud and I am doing RnD whether we can apply <code>HPA</code> (Horizontal Pod Auto scaling) on elasticsearch in Kubernetes.</p>
<p>I did <code>elasticsearch</code> set up on <code>Kubernetes</code> : <a href="https://github.com/elastic/helm-charts/tree/master/elasticsearch" rel="nofollow noreferrer">https://github.com/elastic/helm-charts/tree/master/elasticsearch</a></p>
<p>But I found one of the post on forum they say <code>elasticsearch</code> HPA is hard</p>
<p><a href="https://discuss.elastic.co/t/how-to-scale-up-and-down-nodes-automatically/224089/2" rel="nofollow noreferrer">https://discuss.elastic.co/t/how-to-scale-up-and-down-nodes-automatically/224089/2</a></p>
<p>So is it possible to do <code>HPA</code> on elasticsearch or not ?</p>

<p>I don't think it would work well and you'd be at risk of losing data.  HPA tends to respond to load changes on a scale of a minute or so, and can occasionally make large changes (scaling from 5 replicas to 2, for example).  For Elasticsearch you need to scale in one node at a time, monitor the state of the cluster before you can proceed, and it could take a long time before you can move from one node to the next.</p>
<hr />
<p>Say you're running Elasticsearch in a StatefulSet.  Remember that each ES index is made up of <em>shards</em>; you will have some number of copies of the shards spread out across the nodes in the ES cluster.  So also let's say your index has 10 shards with 2 replicas each.</p>
<p>Scaling out is easy.  Increase the size of the StatefulSet as much as you want; configure each node to talk to <code>es-0</code> for discovery.  ES will see that the cluster has grown and start moving shards on to the new nodes on its own.</p>
<p>Scaling in is hard.  You can only delete one node at a time.  Once that node is shut down and the cluster realizes it's missing, then the shards that used to be on that node will be under-replicated.  ES will create new replicas on the remaining nodes and copy the shard data there.  You can observe this happening through things like the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-shards.html" rel="noreferrer"><code>/_cat/shards</code> API</a>.  The index status will be &quot;yellow&quot; as long as there are under-replicated shards, and then will switch back to &quot;green&quot; once the replication sequence finishes.</p>
<p>So say you currently have 8 nodes, and you want to scale down to 6.  There's some possibility the only two replicas of a given shard will be on <code>es-6</code> and <code>es-7</code> so you can't turn those both off together; you have to turn off <code>es-7</code> first, wait for replication to catch up, then turn off <code>es-6</code>.  There's also some possibility that, when you turn off <code>es-7</code>, the new replicas will be created on the doomed node <code>es-6</code>.</p>
<p>You can also <a href="https://stackoverflow.com/questions/17268495/how-to-remove-node-from-elasticsearch-cluster-on-runtime-without-down-time">tell Elasticsearch to move shards off of a replica</a> before removing it.  This avoids the cluster going into the &quot;yellow&quot; state, but it's harder to monitor.</p>
<p>Finally, the re-replication can take a long time depending on how much data is actually in the cluster.  (In a past cluster I maintained with a badly-designed index structure, it took multiple hours to shut down a node.)  That's a much slower operation than HPA is prepared for.</p>
