title: C# high speed data logging data handling
body: <p>I've written an application that logs trace data from an embedded system via UDP. Currently I receive datagrams and parse out the variable length records and store them in a list. The front end can access the list and present the data (graphs and text lists etc).</p>
  
  <p>The problem I'm running into is that sometimes I need to log an exceptional amount of data. So much that my list implementation causes an out of memory exception.</p>
  
  <p>My requirements are:</p>
  
  <ul>
  <li>Allow multithreaded reading and writing of the data (can't just post process)</li>
  <li>Handle large amounts of data (worst case ~2MB/s ... 7.2GB/hr of logging)</li>
  <li>Allow storage of data set</li>
  <li>Random read, index based, access</li>
  </ul>
  
  <p>Does anyone have some suggestions about how to attack this? Here were a few thoughts that I had:</p>
  
  <ul>
  <li>I'd like a nifty disk backed, memory cached List. It seems like something that would exist but I haven't found one.</li>
  <li>Local database? I don't know too much about databases but it seems like overkill.</li>
  <li>Store the data to a file right away. Keep a list in memory that holds the byte offset for each record index. Can my reader thread access this simultaneously?</li>
  </ul>