<h2>Background</h2>

<p>I'm running a Kubernetes cluster on Google Cloud Platform. I have 2 Node-Pools in my cluster: <code>A</code> and <code>B</code>. <code>B</code> is cheaper (depends on hardware). I prefer that my deployment will run on <code>B</code>. Unless no free resources in <code>B</code>. In that case, new pods will deploy to <code>A</code>. </p>

<p>So I added this section to deployment YAML:</p>

<pre><code>  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: B
            operator: Exists
        weight: 100
</code></pre>

<p>So I giving more weight to node-pool B.</p>

<p>At start, it's working good. I came back after 24 hours and found that some pods are deployed to node-pool A while I have free resources (un-allocated machines) in node B. This is wast of money. </p>

<h2>So, how its happen?</h2>

<p>I sure that the property <code>nodeAffinity</code> is working currectly. I suspect that at same point, node pool B was running without any FREE resources. At this point, the cluster want to grow... The new pod was deployed to node pool A. Until here, everything is fine...</p>

<h2>What I want to achieve?</h2>

<p>Lets say that after an hour, from lack of node-pool <code>B</code> resources time, There are plany of resources free to alocation. I want that Kubernetes will move the existing pods from A to their new house in node pool B. </p>

<p>I looking for something like <code>preferredDuringSchedulingPreferedDuringExecution</code>.</p>

<h2>Question</h2>

<p>Is this possible?</p>

<h2>Update</h2>

<p>Based on @Hitobat answer, I tried to use this code:</p>

<pre><code> spec:
   tolerations:
    - key: A
      operator: "Exists"
      effect: "NoExecute"
      tolerationSeconds: 60
</code></pre>

<p>Unfortunately, After waiting enough time, I still see pods on my <code>A</code> nodepool. I did something wrong?</p>

<p>You can taint pool A. Then configure <em>all</em> your pods to tolerate the taint, but with a tolerationSeconds for the duration you want. This is in addition to the config you already did for pool B.</p>

<p>The effect will be that the pod is scheduled to A if it won't fit on B, but then after a while will be evicted (and hopefully rescheduled onto B again).</p>

<p>See: <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions</a></p>
