title: Storage of many log files
body: <p>I have a system which is receiving log files from different places through http (>10k producers, 10 logs per day, ~100 lines of text each). </p>
  
  <p>I would like to store them to be able to compute misc. statistics over them nightly , export them (ordered by date of arrival or first line content) ...   </p>
  
  <p>My question is : what's the best way to store them ?</p>
  
  <ul>
  <li>Flat text files (with proper locking), one file per uploaded file, one directory per day/producer</li>
  <li>Flat text files, one (big) file per day for all producers (problem here will be indexing and locking)</li>
  <li>Database Table with text (MySQL is preferred for internal reasons) (pb with DB purge as delete can be very long !)</li>
  <li>Database Table with one record per line of text</li>
  <li>Database with sharding (one table per day), allowing simple data purge. (this is partitioning. However the version of mysql I have access to (ie supported internally) does not support it)</li>
  <li>Document based DB à la couchdb or mongodb (problem could be with indexing / maturity / speed of ingestion)</li>
  </ul>
  
  <p>Any advice ?</p>