<p>I have many services. In a day, a few services are busy for about ten hours, while most other services are idle or use a small amount of cpu.</p>
<p>In the past, I put all services in a virtual machine with two cpus, and scale by cpu usage, there are two virtual machine at the busiest time, but most of the time there is only one.</p>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>services</th>
<th>instances</th>
<th>busy time in a day</th>
<th>cpu when busy<br/>(core/service)</th>
<th>cpu when idle<br/>(core/service)</th>
</tr>
</thead>
<tbody>
<tr>
<td>busy services</td>
<td>2</td>
<td>8~12 hours</td>
<td>0.5~1</td>
<td>0.1~0.5</td>
</tr>
<tr>
<td>busy services</td>
<td>2</td>
<td>8~12 hours</td>
<td>0.3~0.8</td>
<td>0.1~0.3</td>
</tr>
<tr>
<td>inactive services</td>
<td>30</td>
<td>0~1 hours</td>
<td>0.1~0.3</td>
<td>&lt; 0.1</td>
</tr>
</tbody>
</table>
</div>
<p>Now, I want to put them in kubernetes, each node has two CPUs, and use node autoscaling and HPA, in order to make the node autoscaling, I must set requests CPU for all services, which is exactly the difficulty I encountered.</p>
<p>This is my setting.</p>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>services</th>
<th>instances</th>
<th>busy time</th>
<th>requests cpu<br/> (cpu/service)</th>
<th>total requests cpu</th>
</tr>
</thead>
<tbody>
<tr>
<td>busy services</td>
<td>2</td>
<td>8~12 hours</td>
<td>300m</td>
<td>600m</td>
</tr>
<tr>
<td>busy services</td>
<td>2</td>
<td>8~12 hours</td>
<td>300m</td>
<td>600m</td>
</tr>
<tr>
<td>inactive services</td>
<td>30</td>
<td>0~1 hours</td>
<td>100m</td>
<td>3000m</td>
</tr>
</tbody>
</table>
</div>
<p>Note: The inactive service requests CPU is set to 100m because it will not work well if it is less than 100m when it is busy.</p>
<p>With this setting, the number of nodes will always be greater than three, which is too costly. I think the problem is that although these services require 100m of CPU to work properly, they are mostly idle.</p>
<p>I really hope that all services can autoscaling, I think this is the benefit of kubernetes, which can help me assign pods more flexibly. Is my idea wrong? Shouldn't I set a request CPU for an inactive service?</p>
<p>Even if I ignore inactive services. I find that kubernetes more often has more than two nodes. If I have more active services, even in off-peak hours, the requests CPU will exceed 2000m. Is there any solution?</p>

Fulo Lin
<blockquote>
<p>I put all services in a virtual machine with two cpus, and scale by cpu usage, there are two virtual machine at the busiest time, but most of the time there is only one.</p>
</blockquote>
<p>First, if you have any availability requirements, I would recommend to always have at least <strong>two</strong> nodes. If you have only one node and that one crash (e.g. hardware failure or kernel panic) it will take some minutes before this is detected and it will take some minutes before a new node is up.</p>
<blockquote>
<p>The inactive service requests cpu is set to 100m because it will not work well if it is less than 100m when it is busy.</p>
</blockquote>
<blockquote>
<p>I think the problem is that although these services require 100m of cpu to work properly, they are mostly idle.</p>
</blockquote>
<p>The CPU <em>request</em> is a guaranteed reserved resource amount. Here you reserve too much resources for your almost idling services. Set the CPU request lower, maybe as low as <code>20m</code> or even <code>5m</code>? But since these services will need more resources during busy periods, set a higher <em>limit</em> so that the container can &quot;burst&quot; and also use <a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/" rel="nofollow noreferrer">Horizontal Pod Autoscaler</a> for these. When using the Horizontal Pod Autoscaler more replicas will be created and the traffic will be load balanced across all replicas. Also see <a href="https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/" rel="nofollow noreferrer">Managing Resources for Containers</a>.</p>
<p>This is also true for your &quot;busy services&quot;, reserve less CPU resources and use Horizontal Pod Autoscaling more actively so that the traffic is spread to more nodes during high load, but can scale down and save cost when the traffic is low.</p>
<blockquote>
<p>I really hope that all services can autoscaling, I think this is the benefit of kubernetes, which can help me assign pods more flexibly. Is my idea wrong?</p>
</blockquote>
<p>Yes, I agree with you.</p>
<blockquote>
<p>Shouldn't I set a request cpu for an inactive service?</p>
</blockquote>
<p>It is a good practice to always set some value for <em>request</em> and <em>limit</em>, at least for a production environment. The scheduling and autoscaling will not work well without <em>resource requests</em>.</p>
<blockquote>
<p>If I have more active services, even in off-peak hours, the requests cpu will exceed 2000m. Is there any solution?</p>
</blockquote>
<p>In general, try to use lower <em>resource requests</em> and use Horizontal Pod Autoscaling more actively. This is true for both your &quot;busy services&quot; and your &quot;inactive services&quot;.</p>
<blockquote>
<p>I find that kubernetes more often has more than two nodes.</p>
</blockquote>
<p>Yes, there are two aspects of this.</p>
<p>If you only use two nodes, your environment probably is small and the Kubernetes control plane probably consists of more nodes and is the majority of the cost. For very small environments, Kubernetes may be expensive and it would be more attractive to use e.g. a serverless alternative like <a href="https://cloud.google.com/run" rel="nofollow noreferrer">Google Cloud Run</a></p>
<p>Second, for availability. It is good to have at least two nodes in case of an abrupt crash e.g. hardware failure or a kernel panic, so that your &quot;service&quot; is still available meanwhile the node autoscaler scales up a new node. This is also true for the number of <em>replicas</em> for a <code>Deployment</code>, if availability is important, use at least two replicas. When you e.g. <a href="https://kubernetes.io/docs/concepts/workloads/pods/disruptions/" rel="nofollow noreferrer">drain a node</a> for maintenance or node upgrade, the pods will be evicted - but not created on a different node first. The control plane will detect that the <code>Deployment</code> (technically ReplicaSet) has less than the desired number of replicas and create a new pod. But when a new Pod is created on a new node, the container image will first be pulled before the Pod is running. To avoid downtime during these events, use at least two replicas for your <code>Deployment</code> and <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/" rel="nofollow noreferrer">Pod Topology Spread Constraints</a> to make sure that those two replicas run on different nodes.</p>
<hr />
<p>Note: You might run into the same problem as <a href="https://stackoverflow.com/questions/66879191/how-to-use-k8s-hpa-and-autoscaler-when-pods-normally-need-low-cpu-but-periodical">How to use K8S HPA and autoscaler when Pods normally need low CPU but periodically scale</a> and that should be mitigated by an upcoming Kubernetes feature: <a href="https://github.com/kubernetes-sigs/scheduler-plugins/tree/6b7e77af527d8db82afb5060e5474ed524bdc0d6/kep/61-Trimaran-real-load-aware-scheduling" rel="nofollow noreferrer">KEP - Trimaran: Real Load Aware Scheduling</a></p>
