title: Queue and Dequeue in multiple threads
body: <p>I'm creating an indexer which is Enqueueing items which needs to be processed. The indexer will add items to its processor. It will for example add 100 items, then doesn't add items for 3 minutes and add another 50 items.</p>
  
  <pre><code>public class Processer
  {
      private ConcurrentQueue&lt;Item&gt; items;
  
      public void AddItem(Item item)
      {
          this.items.Enqueue(item);
      }
  }
  </code></pre>
  
  <p>The items will come in at random intervals, so I will create a separate thread to dequeue and process these items.</p>
  
  <p>What would be the best option to use?</p>
  
  <ol>
  <li><p>Don't use a Collection, but use the ThreadPool:</p>
  
  <pre><code>public void AddItem(Item item)
  {
      ThreadPool.QueueUserWorkItem(function, item);
  }
  </code></pre>
  
  <p>This will automatically create a Queue, and process the items, but I have less control, when 20 items are found, they will almost stop my indexer to run and first finish this thread pool</p></li>
  <li><p>Use a long running Task:</p>
  
  <pre><code>public Processer()
  {
      this.task = Task.Factory.StartNew(() =&gt; DequeueItems(),
          CancellationToken.None,
          TaskCreationOptions.LongRunning,
          TaskScheduler.Default);
  }
  
  public DequeueItems()
  {
      while(true)
      {
          Item item = null;
          while(this.items.TryDequeue(out item)
          {
              this.store.ExecuteIndex((AbstractIndexCreationTask)item);
          }
  
          Thread.Sleep(100); 
      }
  }
  </code></pre>
  
  <p>But I hate the while() and thread.sleep I've got to use, since the enumerable will dry up after some time, and it will need recheck if there are new items.</p></li>
  <li><p>Use a short running task:</p>
  
  <pre><code>public Processer()
  {
  
  }
  private void Run()
  {
      this.task = Task.Factory.StartNew(() =&gt; DequeueItems(),
          CancellationToken.None,
          TaskCreationOptions.PreferFairness,
          TaskScheduler.Default);
  }
  public void AddItem(Item item)
  {
      this.items.Add(item);
      if(this.task == null || this.task.isCompleted)
          this.Run();
  }
  public DequeueItems()
  {
      Item item = null;
      while(this.items.TryDequeue(out item)
      {
          this.store.ExecuteIndex((AbstractIndexCreationTask)item);
      }
  }
  </code></pre>
  
  <p>This might be nicer? But starting a thread is a "expensive" operation, and I don't know if I can miss items since I check IsCompleted, which can be in the process of ending the while loop and this way missing 1 item. But it doesn't sleep, and use a dirty while loop.</p></li>
  <li><p>Your option, since MSDN recommends to use the TPL, I thought not to use Threads, but maybe there are better ways dealing with this problem</p></li>
  </ol>
  
  <h2>Changelog</h2>
  
  <ol>
  <li>Changed to BlockingCollection</li>
  <li>Changed back to ConcurrentQueue</li>
  </ol>
  
  <h2>Some things I've checked:</h2>
  
  <ul>
  <li><a href="https://stackoverflow.com/questions/15456499/de-queue-items-with-worker-threads">De-queue Items with worker threads</a> (using ThreadPool)</li>
  <li><a href="https://stackoverflow.com/questions/795842/thread-queues-for-dummies?rq=1">Thread queues for dummies</a> (only using Thread solutions, not the TPL)</li>
  <li><a href="https://stackoverflow.com/questions/4744630/unexpected-behaviour-for-threadpool-queueuserworkitem?rq=1">Unexpected behaviour for ThreadPool.QueueUserWorkItem</a> (same as above)</li>
  <li><a href="https://stackoverflow.com/questions/1774670/c-sharp-threadpool-vs-tasks">C# - ThreadPool vs Tasks</a> stating TPL is preferred when using multi threaded code</li>
  </ul>