title: C# add to queue in parallel and listen for queue propagation
body: <p>I am new to multithreaded programming. I have a program that needs to query a database and then perform some data manipulation on the data returned. Due to the structure of my org I have to make an individual call to the database to retrieve a single user's account info. My task involves collecting data over thousands of accounts. </p>
  
  <p>Currently, I am using Parallel.ForEach() to query the database and adding all elements into a ConcurrentList. Once all data has been returned from the database I then perform my manipulation in a synchronous fashion.</p>
  
  <p>Besides any glaring issues the one thing I don't like is keeping a large list in memory and essentially being blocked until the lengthy database process is finished. I would like to be able to push the data into a queue and then begin processing the data as soon as data is added. The consuming process does not need to be parallel or async. I just need it to be able to listen for when something is added to the queue or that the queue is not empty.</p>
  
  <p><strong>Parallel Process:</strong></p>
  
  <pre><code>public static ConcurrentBag&lt;CombinedAccountInfo&gt; GetAllAccountInfo(List&lt;AccountInfo&gt; accountList, string dbConnName)
      {
          logger.Info("Fetching Data");
          var concurrentCombinedData = new ConcurrentBag&lt;CombinedAccountInfo&gt;();
          Parallel.ForEach(accountList, new ParallelOptions { MaxDegreeOfParallelism = 5 }, r =&gt;
          {
              try
              {
                  var userPrefs = new List&lt;UserPreference&gt;().queryData(Queries.UserPrefQuery, dbConnName);
  
                  concurrentCombinedData.Add(new CombinedAccountInfo()
                  {
                      AccountName = r.AccountName,
                      AccountId = r.AccountId,
                      LastLoginDate = r.LastLoginDate,
                      AccountHandle = r.AccountHandle,
                      UserPreferences = userPrefs 
                  });
              }
              catch (Exception e)
              {
                  logger.Error(e);
              }
          });
  
          return concurrentCombinedTransaction;
      }
  </code></pre>
  
  <p>I have done some reading on Dataflow and saw a few articles about Reactive Extensions. However I can seem to find any simpler examples of multiple producers feeding into a single consumer. Any suggestions or ideas how to better reach the end goal would be greatly appreciated.</p>
  
  <p><strong>SOLVED</strong></p>
  
  <p>I'll be using the answer that <a href="https://stackoverflow.com/users/5101046/scott-hannen">Scott Hannen</a> provided. Because the manipulation is small and not very intensive each process can handle it rather than trying to tie everything back into a list.</p>