<h3>Background</h3>
<p>I have app running in kubernetes cluster using sharded mongodb and elasticsearch statefulsets. I setup horizontal pod autoscalers for deployment components in my app and everything works well.</p>
<h3>Problems</h3>
<p>Problems arise when the traffic goes up. My server deployment scales out just fine, but mongodb shards and elasticsearch nodes cannot handle this much traffic and throttle overall response time.</p>
<p>Simple solution is to configure those statefulset with more shards, more replicas. What bugs me is that traffic spike happens like 3-4 hours a day, thus it's kinda wasteful to let all those boys sitting idly for the rest of the day.</p>
<p>I did some research and looks like database in general is not supposed to scale out/in dynamically as it will consume a lot of network and disk io just to do replication between them. Also there is potential of data loss and inconsistency during scaling up, scaling down.</p>
<h3>Questions</h3>
<p>If possible, what is proper way to handle dynamic scaling in mongodb, elasticsearch... and database in general?<br />
If not, what can I do to save some cents off my cloud bill as we only need the maximum power from database pods for a short period per day.</p>

<p>You should read about Kubernetes autoscaling - <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">HPA</a>.</p>
<blockquote>
<p>The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics). Note that Horizontal Pod Autoscaling does not apply to objects that can't be scaled, for example, DaemonSets.</p>
<p>The Horizontal Pod Autoscaler is implemented as a Kubernetes API resource and a controller. The resource determines the behavior of the controller. The controller periodically adjusts the number of replicas in a replication controller or deployment to match the observed average CPU utilization to the target specified by user.</p>
</blockquote>
<p>With HPA you should have to also take care about the volume mounting and data latency.</p>
<hr />
<p>As @Serge mentioned in comments, I would  suggest to check the native scaling cluster option provided by the MongoDB and Elasticsearch itself.</p>
<p>Take a look at</p>
<ul>
<li>MongoDB operator <a href="https://docs.mongodb.com/kubernetes-operator/master/tutorial/install-k8s-operator/" rel="nofollow noreferrer">documentation</a></li>
<li>Elasticsearch operator <a href="https://www.elastic.co/blog/introducing-elastic-cloud-on-kubernetes-the-elasticsearch-operator-and-beyond" rel="nofollow noreferrer">documentation</a></li>
<li>Elasticsearch future release <a href="https://www.elastic.co/guide/en/elasticsearch/reference/master/xpack-autoscaling.html" rel="nofollow noreferrer">autoscaling</a></li>
</ul>
<p>I am not very familiar with MongoDB and Elasticsearch with Kubernetes, but maybe those tutorials help you:</p>
<ul>
<li><a href="https://medium.com/faun/scaling-mongodb-on-kubernetes-32e446c16b82" rel="nofollow noreferrer">https://medium.com/faun/scaling-mongodb-on-kubernetes-32e446c16b82</a></li>
<li><a href="https://www.youtube.com/watch?v=J7h0F34iBx0" rel="nofollow noreferrer">https://www.youtube.com/watch?v=J7h0F34iBx0</a></li>
<li><a href="https://kubernetes.io/blog/2017/01/running-mongodb-on-kubernetes-with-statefulsets/" rel="nofollow noreferrer">https://kubernetes.io/blog/2017/01/running-mongodb-on-kubernetes-with-statefulsets/</a></li>
<li><a href="https://sematext.com/blog/elasticsearch-operator-on-kubernetes/#toc-what-is-the-elasticsearch-operator-1" rel="nofollow noreferrer">https://sematext.com/blog/elasticsearch-operator-on-kubernetes/#toc-what-is-the-elasticsearch-operator-1</a></li>
</ul>
<hr />
<p>If you use <a href="https://helm.sh/" rel="nofollow noreferrer">helm</a> take a look at banzaicloud <a href="https://github.com/banzaicloud/hpa-operator" rel="nofollow noreferrer">Horizontal Pod Autoscaler operator</a></p>
<blockquote>
<p>You may not want nor can edit a Helm chart just to add an autoscaling feature. Nearly all charts supports custom annotations so we believe that it would be a good idea to be able to setup autoscaling just by adding some simple annotations to your deployment.</p>
<p>We have open sourced a Horizontal Pod Autoscaler operator. This operator watches for your Deployment or StatefulSet and automatically creates an HorizontalPodAutoscaler resource, should you provide the correct autoscale annotations.</p>
</blockquote>
<hr />
<p>Hope you find this useful.</p>
