<p>I'm wondering the graceful way to reduce nodes in a Kubernetes cluster on GKE.</p>

<p>I have some nodes each of which has some pods watching a shared job queue and executing a job. I also have the script which monitors the length of the job queue and increase the number of instances when the length exceeds a threshold by executing <code>gcloud compute instance-groups managed resize</code> command and it works ok.</p>

<p>But I don't know the graceful way to reduce the number of instances when the length falls below the threshold.</p>

<p>Is there any good way to stop the pods working on the terminating instance before the instance gets terminated? or any other good practice?</p>

<p>Note</p>

<ul>
<li>Each job can take around between 30m and 1h</li>
<li>It is acceptable if a job gets executed more than once (in the worst case...)</li>
</ul>

<p>Before resizing the cluster, let's set the project context in the cloud shell by running the below commands:</p>

<pre><code>gcloud config set project [PROJECT_ID]
gcloud config set compute/zone [COMPUTE_ZONE]
gcloud config set compute/region [COMPUTE_REGION]
gcloud components update 
</code></pre>

<p><strong>Note:</strong> You can also set project, compute zone &amp; region as flags in the below command using --project, --zone, and --region operational flags</p>

<pre><code>gcloud container clusters resize [CLUSTER_NAME] --node-pool [POOL_NAME] --num-nodes [NUM_NODES]
</code></pre>

<p>Run the above command for each node pool. You can omit the --node-pool flag if you have only one node pool.</p>

<p>Reference: <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/resizing-a-cluster" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/resizing-a-cluster</a></p>
