<p>i am running k8s cluster on GKE</p>
<p>it has 4 node pool with different configuration</p>
<p><strong>Node pool : 1</strong> (Single node coroned status)</p>
<p>Running <strong>Redis &amp; RabbitMQ</strong></p>
<p><strong>Node pool : 2</strong> (Single node coroned status)</p>
<p>Running <strong>Monitoring &amp; Prometheus</strong></p>
<p><strong>Node pool : 3</strong> (Big large single node)</p>
<p>Application pods</p>
<p><strong>Node pool : 4</strong> (Single node with auto-scaling enabled)</p>
<p>Application pods</p>
<p>currently, i am running single replicas for each service on GKE</p>
<p>however 3 replicas of the main service which mostly manages everything.</p>
<p>when scaling this main service with HPA sometime seen the issue of Node getting crashed or <code>kubelet frequent restart</code> PODs goes to Unkown state.</p>
<p>How to handle this scenario ? If the node gets crashed GKE taking time to auto repair and which cause service down time.</p>
<p><strong>Question : 2</strong></p>
<p>Node pool : 3 -4 running application PODs. Inside the application, there are 3-4 memory-intensive micro services i am also thinking same to use <strong>Node selector</strong> and fix it on one Node.</p>
<p>while only small node pool will run main service which has HPA and node auto scaling auto work for that node pool.</p>
<p>however i feel like it's not best way to it with Node selector.</p>
<p>it's always best to run more than one replicas of each service but currently, we are running single replicas only of each service so please suggest considering that part.</p>

chagan
<p>As <a href="https://stackoverflow.com/users/9231144/patrick-w">Patrick W</a> rightly suggested in his comment:</p>
<blockquote>
<p><strong>if you have a single node, you leave yourself with a single point of
failure</strong>. Also keep in mind that <strong>autoscaling takes time to kick in and
is based on resource requests</strong>. <strong>If your node suffers OOM because of
memory intensive workloads, you need to readjust your memory requests
and limits</strong> – Patrick W Oct 10 at</p>
</blockquote>
<p>you may need to redesign a bit your infrastructure so you have more than a single node in every nodepool as well as <a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/" rel="nofollow noreferrer">readjust mamory requests and limits</a></p>
<p>You may want to take a look at the following sections in the <strong>official kubernetes docs</strong> and <strong>Google Cloud blog</strong>:</p>
<ul>
<li><a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/" rel="nofollow noreferrer">Managing Resources for Containers</a></li>
<li><a href="https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/" rel="nofollow noreferrer">Assign CPU Resources to Containers and Pods</a></li>
<li><a href="https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/" rel="nofollow noreferrer">Configure Default Memory Requests and Limits for a Namespace</a></li>
<li><a href="https://kubernetes.io/docs/concepts/policy/resource-quotas/" rel="nofollow noreferrer">Resource Quotas</a></li>
<li><a href="https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-resource-requests-and-limits" rel="nofollow noreferrer">Kubernetes best practices: Resource requests and limits</a></li>
</ul>
<blockquote>
<p>How to handle this scenario ? If the node gets crashed GKE taking time
to auto repair and which cause service down time.</p>
</blockquote>
<p>That's why having more than just one node for a single node pool can be much better option. It greatly reduces the likelihood that you'll end up in the situation described above. <strong>GKE</strong> <strong>autorapair</strong> feature needs to take its time (usually a few minutes) and if this is your only node, you cannot do much about it and need to accept possible downtimes.</p>
<blockquote>
<p>Node pool : 3 -4 running application PODs. Inside the application,
there are 3-4 memory-intensive micro services i am also thinking same
to use Node selector and fix it on one Node.</p>
<p>while only small node pool will run main service which has HPA and
node auto scaling auto work for that node pool.</p>
<p>however i feel like it's not best way to it with Node selector.</p>
</blockquote>
<p>You may also take a loot at <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity" rel="nofollow noreferrer">node affinity and anti-affinity</a> as well as <a href="https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/" rel="nofollow noreferrer">taints and tolerations</a></p>
