<p>is there a way to scale dynamically the memory size of Pod based on size of data job (my use case)?  </p>

<p>Currently we have Job and Pods that are defined with memory amounts, but we wouldn't know how big the data will be for a given time-slice (sometimes 1000 rows, sometimes 100,000 rows).<br>
So it will break if the data is bigger than the memory we have allocated beforehand.  </p>

<p>I have thought of using slices by data volume, i.e. cut by every 10,000 rows, we will know memory requirement of processing a fixed amount of rows. But we are trying to aggregate by time hence the need for time-slice.  </p>

<p>Or any other solutions, like Spark on kubernetes?</p>

<p>Another way of looking at it:<br>
How can we do an implementation of Cloud Dataflow in Kubernetes on AWS</p>

cryanbhu
<p>It's a best practice always define <code>resources</code> in your container definition, in particular: </p>

<ul>
<li><code>limits</code>:the upper level of CPU and memory</li>
<li><code>requests</code>: the minimal level of CPU and memory</li>
</ul>

<p>This allows the scheduler to take a better decision and it eases the assignment of <strong>Quality of Service (QoS)</strong> for each pod (<a href="https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/</a>) which falls into three possible classes:   </p>

<ul>
<li><strong>Guaranteed</strong> (highest priority): when <strong><em>requests = limits</em></strong></li>
<li><strong>Burstable</strong>:  when <strong><em>requests &lt; limits</em></strong></li>
<li><strong>BestEffort</strong> (lowest priority): when <strong>requests and limits are not set</strong>.</li>
</ul>

<p>The QoS enables a criterion for killing pods when the system is overcommited.</p>
