title: C# Producer/Consumer queue with delays
body: <p>The problem is the classic N producers (with N possibly large) and X consumers with limited resources (X is currently 4). The producers messages come in (say via MQTT) and get queued to be processed by consumers on a FIFO basis. The important part of the processing is that each consumer may need to send back to the producers one or more &quot;replies&quot; and such replies should be at least some time apart (the exact delay is not important). The classic solution where one starts X tasks that wait on the message queue, process and loop is easy to implement using, for example, <code>System.Threading.Channels</code>:</p>
  <pre><code>while (!cancellationToken.IsCancellationRequested &amp;&amp; await queue.Reader.WaitToReadAsync()) {
      while (queue.Reader.TryRead(out IncomingMessage item)) {
          // Do some processing.
          SendResponse(1);
          // Do some more processing.
          if (needsToSend2Response) {
              await Task.Delay(500);
              SendResponse(2);
          }
      }
  }
  </code></pre>
  <p>This works and works well <em>except</em> that if a task needs to delay it can't process any more messages and that's obviously bad.</p>
  <p>Possible solutions I thought of:</p>
  <ol>
  <li>Use an outbound queue that process the messages and makes sure there is at least a minimum delay between messages sent to the same producer.</li>
  <li>Don't use a queue. Just start a new Task every time a new message comes in and arbitrate the limited resources using a semaphore: it works but I don't see how to guarantee the FIFO requirement (some times messages from the same producer are processed in the wrong order).</li>
  </ol>
  <p>Any other ideas?</p>