<p>I have a Google Cloud Composer environment set up that has 3 nodes in the worker pool. Each node has memory of 16GB (using n1-standard-4) instance. I have tasks inside a DAG that takes around 7-8GB of memory. The allocable memory for the worker nodes is roughly 12GB and hence, these tasks should run without encountering a OOM error.</p>
<p>B
My intuition is that each worker node has some (variable)number of pods and when airflow scheduler queues tasks, the tasks run inside a worker pod and not the node itself. And there might be memory limits set for the pods that you can have inside worker nodes. Note that the worker node are in kubernetes node pool exclusively used by my composer instance.</p>
<p>How can I make sure that all my tasks run without encountering OOM given that my nodes have enough memory? Is there a way to set the pod memory limits to be higher?</p>
<p>I looked at the workloads for the worker inside Kubernetes workloads and I can see that memory limit is 3.7GB which I guess is the limit for the pods.</p>
<p>What should I do?</p>
<p>Thanks in advance.</p>

<p>It's certainly good practice to pre-assess the resources available in your node-pool and know in advance how &quot;big&quot; your Pods will be.</p>
<p>Once you know how many resources you have available, you can do 2 things:</p>
<p>1.set up resources/limits for all your Pods, making sure you never reach the maximum available in your node-pool;</p>
<p><a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/</a></p>
<p>2.try to understand, in case of malfunction or deployment of other NOT estimated resources in your K8s cluster, which are the Pods you would be willing to sacrifice and in which order to do it.</p>
<p>This will allow you to shutdown critical services ONLY after all other services have already been shut down.</p>
<p><a href="https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/</a></p>
<p>Obviously there is a 3 and 4 point in case you want to scale horizontally and vertically, with autoscaling setup for the node-pool and for your Deployments, but they are not strictly related to your question.</p>
