<p>So I wish to limit resources used by pod running for each of my namespace, and therefor want to use resource quota.
I am following this <a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/" rel="nofollow noreferrer">tutorial</a>.
It works well, but I wish something a little different.
When trying to schedule a pod which will go over the limit of my quota, I am getting a <code>403</code> error.
What I wish is the request to be scheduled, but waiting in a pending state until one of the other pod end and free some resources.</p>
<p>Any advice?</p>

Djoby
<p>Instead of using straight pod definitions (<code>kind: Pod</code>) use <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="nofollow noreferrer">deployment</a>.</p>
<p><em>Why?</em></p>
<p>Pods in Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/pods/#working-with-pods" rel="nofollow noreferrer">are designed as relatively ephemeral, disposable entities</a>:</p>
<blockquote>
<p>You'll rarely create individual Pods directly in Kubernetes—even singleton Pods. This is because Pods are designed as relatively ephemeral, disposable entities. When a Pod gets created (directly by you, or indirectly by a <a href="https://kubernetes.io/docs/concepts/architecture/controller/" rel="nofollow noreferrer">controller</a>), the new Pod is scheduled to run on a <a href="https://kubernetes.io/docs/concepts/architecture/nodes/" rel="nofollow noreferrer">Node</a> in your cluster. The Pod remains on that node until the Pod finishes execution, the Pod object is deleted, the Pod is <em>evicted</em> for lack of resources, or the node fails.</p>
</blockquote>
<p><a href="https://kubernetes.io/docs/concepts/workloads/pods/#using-pods" rel="nofollow noreferrer">Kubernetes assumes that for managing pods you should</a> a <a href="https://kubernetes.io/docs/concepts/workloads/" rel="nofollow noreferrer">workload resources</a> instead of creating pods directly:</p>
<blockquote>
<p>Pods are generally not created directly and are created using workload resources. See  <a href="https://kubernetes.io/docs/concepts/workloads/pods/#working-with-pods" rel="nofollow noreferrer">Working with Pods</a>  for more information on how Pods are used with workload resources.
Here are some examples of workload resources that manage one or more Pods:</p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/" rel="nofollow noreferrer">Deployment</a></li>
<li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/" rel="nofollow noreferrer">StatefulSet</a></li>
<li><a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset" rel="nofollow noreferrer">DaemonSet</a></li>
</ul>
</blockquote>
<p>By using deployment you will get very similar behaviour to the one you want.</p>
<p>Example below:</p>
<p>Let's suppose that I created <a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-pod-namespace/" rel="nofollow noreferrer">pod quota</a> for a custom namespace, set to &quot;2&quot; as <a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-pod-namespace/#create-a-resourcequota" rel="nofollow noreferrer">in this example</a> and I have two pods running in this namespace:</p>
<pre><code>kubectl get pods -n quota-demo
NAME           READY   STATUS    RESTARTS   AGE
quota-demo-1   1/1     Running   0          75s
quota-demo-2   1/1     Running   0          6s
</code></pre>
<p>Third pod definition:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
kind: Pod
metadata:
  name: quota-demo-3
spec:
  containers:
  - name: quota-demo-3
    image: nginx
    ports:
    - containerPort: 80
</code></pre>
<p>Now I will try to apply this third pod in this namespace:</p>
<pre><code>kubectl apply -f pod.yaml -n quota-demo
Error from server (Forbidden): error when creating &quot;pod.yaml&quot;: pods &quot;quota-demo-3&quot; is forbidden: exceeded quota: pod-demo, requested: pods=1, used: pods=2, limited: pods=2
</code></pre>
<p>Not working as expected.</p>
<p>Now I will change pod definition into deployment definition:</p>
<pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: quota-demo-3-deployment
  labels:
    app: quota-demo-3
spec:
  selector:
    matchLabels:
      app: quota-demo-3
  template:
    metadata:
      labels:
        app: quota-demo-3
    spec:
      containers:
      - name: quota-demo-3
        image: nginx
        ports:
        - containerPort: 80
</code></pre>
<p>I will apply this deployment:</p>
<pre><code>kubectl apply -f deployment-v3.yaml -n quota-demo
deployment.apps/quota-demo-3-deployment created
</code></pre>
<p>Deployment is created successfully, but there is no new pod, Let's check this deployment:</p>
<pre><code>kubectl get deploy -n quota-demo
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
quota-demo-3-deployment   0/1     0            0           12s
</code></pre>
<p>We can see that a pod quota is working, deployment is monitoring resources and waiting for the possibility to create a new pod.</p>
<p>Let's now delete one of the pod and check deployment again:</p>
<pre><code>kubectl delete pod quota-demo-2 -n quota-demo
pod &quot;quota-demo-2&quot; deleted
kubectl get deploy -n quota-demo
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
quota-demo-3-deployment   1/1     1            1           2m50s
</code></pre>
<p>The pod from the deployment is created automatically after deletion of the pod:</p>
<pre><code>kubectl get pods -n quota-demo
NAME                                       READY   STATUS    RESTARTS   AGE
quota-demo-1                               1/1     Running   0          5m51s
quota-demo-3-deployment-7fd6ddcb69-nfmdj   1/1     Running   0          29s
</code></pre>
<p>It works the same way for <a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/quota-memory-cpu-namespace/" rel="nofollow noreferrer">memory and CPU quotas for namespace</a> - when the resources are free, deployment will automatically create new pods.</p>
