<p>We are working with Akka and have an application which uses Akka Cluster that we deploy to AWS EKS environment. When we run a load test against the application we observe heartbeat intervals between the components growing large and pods getting restarted. Akka documentation mentions not to use CPU resource limit, and removing the resource limit solves the issue.</p>
<p>But is there any other way of getting around this, we are not sure if removing the resource limits is a good practice when deploying application.</p>
<p><a href="https://doc.akka.io/docs/akka/current/additional/deploying.html?_ga=2.206580627.186827534.1646216705-1504733962.1642433119#resource-limits" rel="nofollow noreferrer">Resource limits</a></p>
<blockquote>
<p>To avoid CFS scheduler limits, it is best not to use resources.limits.cpu limits, but use resources.requests.cpu configuration instead.</p>
</blockquote>

<p>So, I know the docs do make this recommendation to not use limits so the following is just  &quot;unofficial&quot; advice you are getting from StackOverflow. (And thus, if you want support from Lightbend you will have to debate this with them.)</p>
<p>Firstly, I agree with you. For many reasons, you absolutely should have resource limits. For example, if you don't have CPU limits your process ends up being designated as &quot;best effort&quot; as far as the CFS scheduler is concerned and that can actually have bad consequences.</p>
<p>As I understand the history of this recommendation from Lightbend, it comes from a situation similar to yours where the CFS scheduler was preventing Akka from getting the resources it needed. Plus the broader problem that, especially when the garbage collector kicks in, it's definitely possible to consume all of your CFS quota very quickly and end up with long GC pause times. The Lightbend position has been if you use CPU resource limits, then the CFS scheduler will limit your CPU consumption and that could cause problems.</p>
<p>But my position is that limiting your CPU consumption is the entire point, and is actually a good thing. Kubernetes is a shared environment and limiting CPU consumption is how the system is designed to prevent &quot;noisy neighbors&quot;, fair sharing, and often cost chargebacks. My position is that the CPU limits themselves aren't bad, the problem is only when your limits are too low.</p>
<p>While I hate to make generic advice, as there may be some situations where I might make different recommendations, I would generally recommend setting CPU limits, but having them be significantly higher than your CPU requests. (2-3 times as a rule of thumb.) This type of setting will classify your Pods as &quot;Burstable&quot;. Which is important so that your Akka node can burst up to handle high load situations such as GC, handling failover, etc.</p>
<p>Ideally you then use something like HBA such that your Akka cluster will autoscale so that it can handle its normal load with its &quot;request&quot; allocation of CPU and only uses that burst allocation during these occasional circumstances. (Because if you are always operating past the request allocation and close to burst allocation, then you really aren't getting the benefit of bursting.)</p>
<p>For example, in your specific case, you say you have problems with heartbeats when you set CPU limits but that goes away when you turn off the CPU limits. This tells me that your CPU limits were <em>way</em> too low. If Akka isn't able get enough CPU to do heartbeats, even under normal circumstances, then there is no way that it would be able to get enough CPU to handle peaks.</p>
<p>(Also, since we are we assuming that the system will be normally running at the request CPU allocation, but potentially running at limit CPU allocation during GC, I would typically tune my Akka threads as if it had &quot;request&quot; CPU and tune my GC threads as if it had &quot;limit&quot; CPU allocation.)</p>
