<p>I have a kubernetes cluster into which I'm intending to implement a service in a pod - the service will accept a grpc request, start a long running process but return to the caller indicating the process has started. Investigation suggests that <code>IHostedService</code> (<code>BackgroundService</code>) is the way to go for this.</p>
<p>My question is, will use of <code>BackgroundService</code> behave nicely with various neat features of asp.net and k8s:</p>
<ul>
<li>Will horizontal scaling understand that a service is getting overloaded and spin up a new instance even though the service will appear to have no pending grpc requests because all the work is background (I appreciate there's probably hooks that can be implemented, I'm wondering what's default behaviour)</li>
<li>Will the notion of <code>await</code>ing allowing the current process to be swapped out and another run work okay with background services (I've only experienced it where one message received hits an <code>await</code> so allows another message to be processed, but backround services are not a messaging context)</li>
<li>I think asp.net will normally manage throttling too many requests, backing off if the server is too busy, but will any of that still work if the 'busy' is background processes</li>
<li>What's the best method to mitigate against overloading the service (if horizontal scaling is not an option) - I can have the grpc call reutrn 'too busy' but would need to detect it (not quite sure if that's cpu bound, memory or just number of background services)</li>
<li>Should I be considering something other than <code>BackgroundService</code> for this task</li>
</ul>
<p>I'm hoping the answer is that &quot;it all just works&quot; but feel it's better to have that confirmed than to just hope...</p>

<blockquote>
<p>Investigation suggests that IHostedService (BackgroundService) is the way to go for this.</p>
</blockquote>
<p>I <a href="https://blog.stephencleary.com/2021/01/asynchronous-messaging-1-basic-distributed-architecture.html" rel="nofollow noreferrer">strongly recommend</a> using a durable queue with a separate background service. It's not that difficult to split into two images, one running ASP.NET GRPC requests, and the other processing the durable queue (this can be a console app - see the Service Worker template in VS). Note that solutions using <strong>non</strong>-durable queues are not reliable (i.e., work may be lost whenever a pod restarts or is scaled down). This includes in-memory queues, which are commonly suggested as a &quot;solution&quot;.</p>
<p>If you do make your own background service in a console app, I recommend applying a <a href="https://blog.stephencleary.com/2020/05/backgroundservice-gotcha-startup.html" rel="nofollow noreferrer">few tweaks</a> (noted on my blog):</p>
<ul>
<li>Wrap <code>ExecuteAsync</code> in <code>Task.Run</code>.</li>
<li>Always have a top-level <code>try</code>/<code>catch</code> in <code>ExecuteAsync</code>.</li>
<li>Call <code>IHostApplicationLifetime.StopApplication</code> when the background service stops for any reason.</li>
</ul>
<blockquote>
<p>Will horizontal scaling understand that a service is getting overloaded and spin up a new instance even though the service will appear to have no pending grpc requests because all the work is background (I appreciate there's probably hooks that can be implemented, I'm wondering what's default behaviour)</p>
</blockquote>
<p>One reason I prefer using two different images is that they can scale on different triggers: GRPC requests for the API and queued messages for the worker. Depending on your queue, using &quot;queued messages&quot; as the trigger may require a custom metric provider. I do prefer using &quot;queued messages&quot; because it's a natural scaling mechanism for the worker image; out-of-the-box solutions like CPU usage don't always work well - in particular for asynchronous processors, which you mention you are using.</p>
<blockquote>
<p>Will the notion of awaiting allowing the current process to be swapped out and another run work okay with background services (I've only experienced it where one message received hits an await so allows another message to be processed, but backround services are not a messaging context)</p>
</blockquote>
<p>Background services can be asynchronous without any problems. In fact, it's not uncommon to grab messages in batches and process them all concurrently.</p>
<blockquote>
<p>I think asp.net will normally manage throttling too many requests, backing off if the server is too busy, but will any of that still work if the 'busy' is background processes</p>
</blockquote>
<p>No. ASP.NET only throttles requests. Background services do register with ASP.NET, but that is <em>only</em> to provide a best-effort at graceful shutdown. ASP.NET has no idea how busy the background services are, in terms of pending queue items, CPU usage, or outgoing requests.</p>
<blockquote>
<p>What's the best method to mitigate against overloading the service (if horizontal scaling is not an option) - I can have the grpc call reutrn 'too busy' but would need to detect it (not quite sure if that's cpu bound, memory or just number of background services)</p>
</blockquote>
<p>Not a problem if you use the durable queue + independent worker image solution. GRPC calls can pretty much always stick another message in the queue (very simple and fast), and K8 can autoscale based on your (possibly custom) metric of &quot;outstanding queue messages&quot;.</p>
