title: Deciding selecting a nosql db for log analysis
body: <p>I need to analyze gigabytes of log files everyday. I am trying to determine the right nosql DB to use. Here is an overview of the characteristics of the log files and the required analysis:</p>
  
  <ul>
  <li>These log files contain a mix of structured data (JSON or XML request response with various vendors), as well as some debug lines generated from the code.</li>
  <li>Almost all the lines(Except the case where a line break has been inserted by vendors/developers in the log statements.) contain a timestamp and a sessionID.</li>
  <li>Analysis requires querying/aggregating based on the contents of the JSON/XML documents. These documents can have different formats based on the type of documents and vendors. A lot of JSON field values are arrays.</li>
  <li>Analysis requires co-relating different documents based on the sessionID, like successful responses for a certain kind of requests, average number of results returned from vendor for a certain type of queries.</li>
  </ul>
  
  <p>I started with LogStash but I had to specify the schema for each kind of request and response to be able to query. What if the vendor changes response or a new vendor is added. Logstash seems to work well with a fixed formatted log, like apache server log.</p>
  
  <p>MongoDB doesn't look very helpful, as the performance for aggregation isn't very impressive per other users' blogs. In mongoDB, I need to explode the fields with array values to be able to query the values in those arrays. That would significantly increase the DB size.</p>
  
  <p>Anybody has any other ideas ?</p>