title: Storing Real Time data into 1000 files
body: <p>I have a program that receives real time data on 1000 topics.  It receives -- on average -- 5000 messages per second.  Each message consists of a two strings, a topic, and a message value. I'd like to save these strings along with a timestamp indicating the message arrival time.</p>
  
  <p>I'm using 32 bit Windows XP on 'Core 2' hardware and programming in C#.</p>
  
  <p>I'd like to save this data into 1000 files -- one for each topic.  I know many people will want to tell me to save the data into a database, but I don't want to go down that road.</p>
  
  <p>I've considered a few approaches:</p>
  
  <p>1) Open up 1000 files and write into each one as the data arrives.  I have two concerns about this.  I don't know if it is possible to open up 1000 files simultaneously, and I don't know what effect this will have on disk fragmentation.</p>
  
  <p>2) Write into one file and -- somehow -- process it later to produce 1000 files.</p>
  
  <p>3) Keep it all in RAM until the end of the day and then write one file at a time.  I think this would work well if I have enough ram although I might need to move to 64 bit to get over the 2 GB limit.</p>
  
  <p>How would you approach this problem?</p>