title: Fastest and efficient way of parsing raw data from file
body: <p>I'm working on some project and I'm wondering which way is the most efficient to read a huge amount of data off a file(I'm speaking of file of 100 lines up to 3 billions lines approx., can be more thought). Once read, data will be stored in a structured data set (<code>vector&lt;entry&gt;</code> where "entry" defines a structured line).</p>
  
  <p>A structured line of this file may look like : 
    <code>string int int int string string</code>
  which also ends with the appropriate platform <code>EOL</code> and is TAB delimited </p>
  
  <p>What I wish to accomplish is :</p>
  
  <ol>
  <li>Read file into memory (<code>string</code>) or <code>vector&lt;char&gt;</code></li>
  <li>Read raw data from my buffer and format it into my data set.</li>
  </ol>
  
  <p>I need to consider memory footprint and have a fast parsing rate.
  I'm already avoiding usage of <code>stringstream</code> as they seems too slow.</p>
  
  <p>I'm also avoiding multiple I/O call to my file by using :</p>
  
  <pre><code>// open the stream
  std::ifstream is(filename);
  
  // determine the file length
  is.seekg(0, ios_base::end);
  std::size_t size = is.tellg();
  is.seekg(0, std::ios_base::beg);
  
  // "out" can be a std::string or vector&lt;char&gt;
  out.reserve(size / sizeof (char));
  out.resize(size / sizeof (char), 0);
  
  // load the data
  is.read((char *) &amp;out[0], size);
  
  // close the file
  is.close();
  </code></pre>
  
  <p>I've thought of taking this huge <code>std::string</code> and then looping line by line, I would extract line information (string and integer parts) into my data set row. Is there a better way of doing this? </p>
  
  <p>EDIT : This application may run on a 32bit, 64bit computer, or on a super computer for bigger files.</p>
  
  <p>Any suggestions are very welcome.</p>
  
  <p>Thank you</p>