<p>is there a way to scale dynamically the memory size of Pod based on size of data job (my use case)?  </p>

<p>Currently we have Job and Pods that are defined with memory amounts, but we wouldn't know how big the data will be for a given time-slice (sometimes 1000 rows, sometimes 100,000 rows).<br>
So it will break if the data is bigger than the memory we have allocated beforehand.  </p>

<p>I have thought of using slices by data volume, i.e. cut by every 10,000 rows, we will know memory requirement of processing a fixed amount of rows. But we are trying to aggregate by time hence the need for time-slice.  </p>

<p>Or any other solutions, like Spark on kubernetes?</p>

<p>Another way of looking at it:<br>
How can we do an implementation of Cloud Dataflow in Kubernetes on AWS</p>

cryanbhu
<p>If you don’t know the memory requirement for your pod a priori for a given time-slice, then it is difficult for Kubernete Cluster Autoscaler to automatically scale node pool for you as per this documentation [1]. Therefore for both of your suggestions like running either Cloud Dataflow or Spark on Kubernete with Kubernete Cluster Autoscaler, may not work for your case.</p>

<p>However, you can use custom scaling as a workaround. For example, you can export memory related metrics of the pod to Stackdriver, then deploy HorizontalPodAutoscaler (HPA) resource to scale your application as [2].</p>

<p>[1] <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#how_cluster_autoscaler_works" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler#how_cluster_autoscaler_works</a></p>

<p>[2] <a href="https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/tutorials/custom-metrics-autoscaling</a></p>
