title: Storing records in a byte array vs using an array of structs
body: <p>I have 200 million records, some of which have variable sized fields (string, variable length array etc.). I need to perform some filters, aggregations etc. on them (analytics oriented queries). </p>
  
  <p>I want to just lay them all out in memory (enough to fit in a big box) and then do linear scans on them. There are two approach I can take, and I want to hear your opinions on which approach is better, for maximizing speed:</p>
  
  <ol>
  <li>Using an array of structs with <code>char*</code> and <code>int*</code> etc. to deal with variable length fields</li>
  <li>Use a large byte array, scan the byte array like a binary stream, and then parse the records</li>
  </ol>
  
  <p>Which approach would you recommend?</p>
  
  <p><strong>Update:</strong> Using C.</p>