title: Why is kubernetes reporting scheduling error because of Memory resource limits?
body: <p>I'm trying to define a job with a resource limit, here is a basic pod definition:</p>
  
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
      name: busybox
      labels:
          app: busybox
  spec:
      containers:
          - name: busybox
            image: busybox
            args: [/bin/sh, -c, 'sleep 600']
            resources:
              limits:
                  memory: "1Gi"
                  cpu: "1"
              requests:
                  cpu: "0.1"
                  memory: "400Mi"
      nodeSelector:
          gitlab: "true"
  </code></pre>
  
  <p>When declaring a job with a resource request of "300Mi" this works, but with a request of "400Mi" this fails with:</p>
  
  <blockquote>
    <p>52m     29s     183 default-scheduler           Warning     FailedScheduling    No nodes are available that match all of the following predicates:: Insufficient memory (3), MatchNodeSelector (2).</p>
  </blockquote>
  
  <p>Getting the relevant information from <code>kubectl describe node &lt;relevant node&gt;</code> shows me the following under resources:</p>
  
  <blockquote>
    <p>Capacity:<br>
     cpu:     2<br>
     memory:  2052872Ki<br>
     pods:        110<br>
    Allocatable:<br>
     cpu:     2<br>
     memory:  1950472Ki<br>
     pods:        110  </p>
  </blockquote>
  
  <p>Further down, past the system info I get the following:</p>
  
  <blockquote>
  <pre><code>Allocated resources:
    (Total limits may be over 100 percent, i.e., overcommitted.)
    CPU Requests    CPU Limits  Memory Requests Memory Limits
    ------------    ----------  --------------- -------------
    150m (7%)   1 (50%)     12Mi (0%)   128Mi (6%)
  Events:       &lt;none&gt;
  </code></pre>
  </blockquote>
  
  <p>Now, <em>why</em> is this job refusing to schedule with a request of 400Mi, but works fine when requesting 300Mi?  Nothing I add or subtract can show me that the 2GiB accessible memory ( as per "Allocatable" above)  would cause this job to not schedule.</p>
  
  <p>I only have a single container (busybox for demo sake).</p>
  
  <p>There is no resource quota for the namespace.</p>
  
  <p>Restarting the apiserver appears to allow pods to schedule, but only for a while.</p>