title: Parsing huge logfiles in Node.js - read in line-by-line
body: <p>I need to do some parsing of large (5-10 Gb)logfiles in Javascript/Node.js (I'm using Cube).</p>
  
  <p>The logline looks something like:</p>
  
  <pre><code>10:00:43.343423 I'm a friendly log message. There are 5 cats, and 7 dogs. We are in state "SUCCESS".
  </code></pre>
  
  <p>We need to read each line, do some parsing (e.g. strip out <code>5</code>, <code>7</code> and <code>SUCCESS</code>), then pump this data into Cube (<a href="https://github.com/square/cube" rel="noreferrer">https://github.com/square/cube</a>) using their JS client.</p>
  
  <p>Firstly, what is the canonical way in Node to read in a file, line by line?</p>
  
  <p>It seems to be fairly common question online:</p>
  
  <ul>
  <li><a href="http://www.quora.com/What-is-the-best-way-to-read-a-file-line-by-line-in-node-js" rel="noreferrer">http://www.quora.com/What-is-the-best-way-to-read-a-file-line-by-line-in-node-js</a></li>
  <li><a href="https://stackoverflow.com/questions/6156501/read-a-file-one-line-at-a-time-in-node-js">Read a file one line at a time in node.js?</a></li>
  </ul>
  
  <p>A lot of the answers seem to point to a bunch of third-party modules:</p>
  
  <ul>
  <li><a href="https://github.com/nickewing/line-reader" rel="noreferrer">https://github.com/nickewing/line-reader</a></li>
  <li><a href="https://github.com/jahewson/node-byline" rel="noreferrer">https://github.com/jahewson/node-byline</a></li>
  <li><a href="https://github.com/pkrumins/node-lazy" rel="noreferrer">https://github.com/pkrumins/node-lazy</a></li>
  <li><a href="https://github.com/Gagle/Node-BufferedReader" rel="noreferrer">https://github.com/Gagle/Node-BufferedReader</a></li>
  </ul>
  
  <p>However, this seems like a fairly basic task - surely, there's a simple way within the stdlib to read in a textfile, line-by-line?</p>
  
  <p>Secondly, I then need to process each line (e.g. convert the timestamp into a Date object, and extract useful fields).</p>
  
  <p>What's the best way to do this, maximising throughput? Is there some way that won't block on either reading in each line, or on sending it to Cube?</p>
  
  <p>Thirdly - I'm guessing using string splits, and the JS equivalent of contains (IndexOf != -1?) will be a lot faster than regexes? Has anybody had much experience in parsing massive amounts of text data in Node.js?</p>
  
  <p>Cheers,
  Victor</p>