<p>I am trying to find an elegant way to resolve the following scenario.</p>
<p>We have a ASW Kubernetes cluster with 6 nodes of 16G RAM each one,
The cluster has various of pods with different resources requirements between 1G to 6G minimum requested memory.</p>
<p>There is a scenario when we get a pod pending due to insufficient memory.
It happens when we need to upgrade a few pods with different memory requirements. The pod with 6G is pending since no node has 6G available.</p>
<p>What I would expect from the Kubernetes to rearrange pods between nodes in order to free 6G on a specific node rather than hold 5G free on two diff notes (in total 10G) and returns insufficient memory.</p>
<p>Is there a way to instruct the Kubernetes to initialise the memory better and handle this automatically.</p>
<p>I was thinking about the pod priorities capability. The less memory request the low prioritised. Wondering if basing on this setting the Kubernetes will be able to restart the less important (the small) pod once the bigger is deployed, in this way to rearrange them between nodes.</p>
<p>Any idea will be appreciated</p>

<p>There is no silver bullet solution but there are combination things you can do maybe using <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity" rel="nofollow noreferrer">Pod Affinity/Pod AntiAffinity</a>, <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/" rel="nofollow noreferrer">Node Affinity</a>, and <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/" rel="nofollow noreferrer">Pod Topology Spread Constraints</a>. It also depends on your workload priorities.</p>
<p>If you have 6 nodes you can have something like this:</p>
<pre class="lang-sh prettyprint-override"><code>NAME    STATUS   ROLES    AGE     VERSION   LABELS
node1   Ready    &lt;none&gt;   4m26s   v1.16.0   node=node1,type=heavy
node2   Ready    &lt;none&gt;   3m58s   v1.16.0   node=node2,type=heavy
node3   Ready    &lt;none&gt;   3m17s   v1.16.0   node=node3,type=heavy
node4   Ready    &lt;none&gt;   2m43s   v1.16.0   node=node4,type=light
node5   Ready    &lt;none&gt;   3m17s   v1.16.0   node=node5,type=light
node6   Ready    &lt;none&gt;   2m43s   v1.16.0   node=node6,type=light
</code></pre>
<p>Then in your 6G Pod spec, which will schedule on node1-node6 with a skew of 3 on the heavy nodes based on the heavy pods having PodAffinity.</p>
<pre class="lang-yaml prettyprint-override"><code>kind: Pod
apiVersion: v1
metadata:
  name: mypod
  labels:
    workload: heavy
spec:
  topologySpreadConstraints:
  - maxSkew: 3
    topologyKey: type
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        workload: heavy
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: workload
            operator: In
            values:
            - heavy
        topologyKey: type
  containers:
  - name: myheavyapp
    image: myapp:latest
  ...
</code></pre>
<p>Then you can use NodeAffinity just to schedule your light 1G pods on the light nodes only.</p>
<pre class="lang-yaml prettyprint-override"><code>kind: Pod
apiVersion: v1
metadata:
  name: mylightpod
  labels:
    workload: light
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: type
            operator: In
            values:
            - light
    ...
</code></pre>
<p>This is just an example, you can change labels, and skews to fit whatever your use case is.</p>
<p>Additionally, to prevent downtime you can configure a <a href="https://kubernetes.io/docs/tasks/run-application/configure-pdb/" rel="nofollow noreferrer">PodDisruptionBudget</a></p>
