<p>We are working with Akka and have an application which uses Akka Cluster that we deploy to AWS EKS environment. When we run a load test against the application we observe heartbeat intervals between the components growing large and pods getting restarted. Akka documentation mentions not to use CPU resource limit, and removing the resource limit solves the issue.</p>
<p>But is there any other way of getting around this, we are not sure if removing the resource limits is a good practice when deploying application.</p>
<p><a href="https://doc.akka.io/docs/akka/current/additional/deploying.html?_ga=2.206580627.186827534.1646216705-1504733962.1642433119#resource-limits" rel="nofollow noreferrer">Resource limits</a></p>
<blockquote>
<p>To avoid CFS scheduler limits, it is best not to use resources.limits.cpu limits, but use resources.requests.cpu configuration instead.</p>
</blockquote>

<p>There are two basic flavors of CPU limiting that Kubernetes supports: CFS and shares.</p>
<p>CFS places a hard limit on the host CPU time that a give container (or more properly the cgroup associated with the container) can consume: typically X milliseconds in every 100 milliseconds.  Once the container has had X milliseconds it cannot run for the remainder of the 100 millisecond slice.</p>
<p>Shares instead proportionally allocates CPU limits.  If the containers on a host have a total share of 4000, a container with a share of 1500 will get 15/40 = 37.5% of the total CPU capacity.  Since Kubernetes typically determines the share value based on millicores (1.0 CPU =&gt; 1000 share), in our 4000 total share example, if the host has 8 cores the container with a 1500 share (1.5 CPU) would actually get the equivalent number of clock cycles as 3 CPU (or more if other containers aren't using their allocations).  If the total shares were 8000, that container would get the equivalent of 1.5 CPU (Kubernetes in turn will ensure that the total share doesn't go above 8000).</p>
<p>So under full load, there's not really a difference between the CFS limit approach and the share approach.  When the host isn't fully used, CFS will have idle CPU while shares will allow containers to have more CPU.  If the workloads running on the host are bursty (especially at shorter timescales), the share approach will be very effective.</p>
<p>For steadier workloads (think IoT more than online retailing), that you're seeing this is indicative of an issue:</p>
<ul>
<li>You might be underprovisioning CPU.</li>
<li>It's possible that you're seeing thread starvation, especially in the dispatchers related to clustering/remoting.  A thread starvation detector (e.g. the one available with a Lightbend subscription (disclaimer: I am employed by Lightbend, but can endorse its utility from when I was a client)) can help diagnose this.  In Akka 2.6, clustering actors run in a separate dispatcher from application actors, so this is less likely to be an issue; if running pre-2.6, you can have clustering use a separate dispatcher by adjusting <code>akka.cluster.use-dispatcher</code>.  It is also probably worth looking for actors in your application which take a long time to process messages and moving those actors to other dispatchers.</li>
<li>JVM garbage collectors all have situations where they can become stop-the-world and typically the duration of the pause is correlated with the size of the heap: more instances with smaller heaps can help here.</li>
<li>Finally, if the constraints of the application and how you're deploying it are such that heartbeat arrivals are going to be volatile, you can adjust the failure detector to be less eager to declare a node failed.  The primary knobs to twiddle here are <code>akka.cluster.failure-detector.threshold</code> and <code>akka.cluster.failure-detector.acceptable-heartbeat-pause</code>.  Note that the former defaults to 8.0 which is considered appropriate for a LAN environment with physical hardware.  Every layer of virtualization and sharing of network/compute infrastructure with workloads you don't control warrants increasing the <code>threshold</code>: I've seen 12.0 recommended for EC2 deploys and given that EKS and such gives you even less control, I'd consider something like 13.0.  Likewise, my experience has been that <code>acceptable-heartbeat-pause</code> should be about 500ms per GB of Java heap.  Note that doing anything to make the failure detector less eager implies being slower to respond to a node being down.</li>
</ul>
