<p>I'm running an ASP.NET Core API on Linux, on Kubernetes in the Google Cloud.</p>

<p>This is an API with high load, and on every request it's executing a library doing a long (1-5 seconds), CPU-intensive operation.</p>

<p>What I see is that after deployment the API works properly for a while, but after 10-20 minutes it becomes unresponsive, and even the health check endpoint (which just returns a hardcoded <code>200 OK</code>) stops working and times out. (This makes Kubernetes kill the pods.)</p>

<p>Sometimes I'm also seeing the infamous <code>Heartbeat took longer than "00:00:01"</code> error message in the logs.</p>

<p>Googling these phenomenons points me to "Thread starvation", so that there are too many thread pool threads started, or too many threads are blocking waiting on something, so that there are no threads left in the pool which could pick up ASP.NET Core requests (hence the timeout of even the health check endpoint).</p>

<p>What is the best way to troubleshoot this issue? I started monitoring the numbers returned by <code>ThreadPool.GetMaxThreads</code> and <code>ThreadPool.GetAvailableThreads</code>, but they stayed constant (the completion port is always <code>1000</code> both for max and available, and the worker is always <code>32767</code>).<br>
Is there any other property I should monitor?</p>

<p>Generally speaking, long-running work is anathema for web applications. You want sub-second response times for a healthy web app. This is particularly true if the work you need to do is synchronous or CPU-bound. Async can at least free up threads during the process, but with CPU-bound work, the thread is hog-tied.</p>

<p>You should off-load whatever you're doing to a different process and then monitor the progress. For an API, the typical approach here is to schedule the work on a different process and then immediately return a <a href="https://httpstatuses.com/202" rel="noreferrer">202 Accepted</a>, with an endpoint in the response body the client can utilize to monitor the progress/get the eventual completed result. You could also implement a webhook, which the client may register to receive notification that the process has completed, without having to constantly check on it.</p>

<p>Your only other option is to throw more resources at the problem. For example, you could stage multiple instances behind a load balancer, divvying requests between each instance to reduce the overall load on each.</p>

<p>It's also entirely possible that there's some inefficiency or issue in your code that could be corrected to either reduce the amount of time the process takes and/or the resources being consumed. As a trivial example, say if you're using something like <code>Task.Run</code>, you could potentially free up a ton of threads by <em>not</em> doing that. <code>Task.Run</code> should pretty much never be used within the context of a web application. However, you have not posted any code, so it's impossible to give you exact guidance there.</p>
