<p>I am just curious to know how k8s master/scheduler will handle this.</p>

<p>Lets consider I have a k8s master with 2 nodes. Assume that each node has 8GB RAM and each node running a pod which consumes 3GB RAM.</p>

<pre><code>node A - 8GB
   - pod A - 3GB
node B - 8GB
   - pod B - 3GB
</code></pre>

<p>Now I would like to schedule another pod, say pod C, which requires 6GB RAM. </p>

<p><strong>Question:</strong></p>

<ol>
<li>Will the k8s master shift pod A or B to other node to accommodate the pod C in the cluster or will the pod C be in the pending status? </li>
<li>If the pod C is going to be in pending status, how to use the resources efficiently with k8s?</li>
</ol>

<p>Unfortunately I could not try this with my minikube. If you know how k8s scheduler assigns the nodes, please clarify.</p>

<p>Most of the Kubernetes components are split by responsibility and workload assignment is no different. We could define the workload assignment process as <strong><em>Scheduling</em></strong> and <strong><em>Execution</em></strong>.</p>

<p>The <strong>Scheduler</strong> as the name suggests will be responsible for the <strong><em>Scheduling</em></strong> step, The process can be briefly described as, "<em>get a list of pods, if it is not scheduled to a node, assign it to one node with capacity to run the pod</em>". There is a nice blog post from Julia Evan <a href="https://jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work/" rel="nofollow noreferrer">here</a> explaining Schedulers.</p>

<p>And <strong>Kubelet</strong> is responsible for the <strong><em>Execution</em></strong> of pods scheduled to it's node. It will get a list of POD Definitions allocated to it's node, make sure they are running with the right configuration, if not running start then.</p>

<p>With that in mind, the scenario you described will have the behavior expected, the POD will not be scheduled, because you don't have a node with capacity available for the POD.</p>

<p>Resource Balancing is mainly decided at scheduling level, a nice way to see it is when you add a new node to the cluster, if there are no PODs pending allocation, the node will not receive any pods. A brief of the logic used to Resource balancing can be seen on <a href="https://github.com/kubernetes/kubernetes/pull/6150" rel="nofollow noreferrer">this PR</a></p>

<p>The solutions,</p>

<p>Kubernetes ships with a default scheduler. If the default scheduler does not suit your needs you can implement your own scheduler as described <a href="https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/" rel="nofollow noreferrer">here</a>. The idea would be implement and extension for the Scheduler to ReSchedule PODs already running when the cluster has capacity but not well distributed to allocated the new load.</p>

<p>Another option is use tools created for scenarios like this, the <a href="https://github.com/kubernetes-incubator/descheduler" rel="nofollow noreferrer">Descheduler</a> is one, it will monitor the cluster and evict pods from nodes to make the scheduler re-allocate the PODs with a better balance. There is a nice blog post <a href="https://itnext.io/keep-you-kubernetes-cluster-balanced-the-secret-to-high-availability-17edf60d9cb7" rel="nofollow noreferrer">here</a> describing these scenarios.</p>

<p>PS:
Keep in mind that the total memory of a node is not allocatable, depending on which provider you use, the capacity allocatable will be much lower than the total, take a look on this SO: <a href="https://stackoverflow.com/questions/54786341/cannot-create-a-deployment-that-requests-more-than-2gi-memory/54786781#54786781">Cannot create a deployment that requests more than 2Gi memory</a></p>
