title: Nodejs Read very large file(~10GB), Process line by line then write to other file
body: <p>I have a <strong>10 GB</strong> log file in a particular format, I want to <strong>process this file line by line</strong> and then <strong>write the output to other file</strong> after applying some <strong>transformations</strong>. I am using node for this operation. </p>
  
  <p>Though this method is fine but it takes a hell lot of time to do this. I was able to do this within 30-45 mins in JAVA, but in node it is taking more than 160 minutes to do the same job. Following is the code:</p>
  
  <blockquote>
    <p>Following is the initiation code which reads each line from the input.</p>
  </blockquote>
  
  <pre><code>var path = '../10GB_input_file.txt';
  var output_file = '../output.txt';
  
  function fileopsmain(){
  
      fs.exists(output_file, function(exists){
          if(exists) {
              fs.unlink(output_file, function (err) {
                  if (err) throw err;
                  console.log('successfully deleted ' + output_file);
              });
          }
      });
  
      new lazy(fs.createReadStream(path, {bufferSize: 128 * 4096}))
          .lines
          .forEach(function(line){
              var line_arr = line.toString().split(';');
              perform_line_ops(line_arr, line_arr[6], line_arr[7], line_arr[10]);
          }
      );
  
  }
  </code></pre>
  
  <blockquote>
    <p>This is the method that performs some operation over that line and
    passes the input to write method to write it into the output file.</p>
  </blockquote>
  
  <pre><code>function perform_line_ops(line_arr, range_start, range_end, daynums){
  
      var _new_lines = '';
      for(var i=0; i&lt;days; i++){
          //perform some operation to modify line pass it to print
      }
  
      write_line_ops(_new_lines);
  }
  </code></pre>
  
  <blockquote>
    <p>Following method is used to write data into a new file.</p>
  </blockquote>
  
  <pre><code>function write_line_ops(line) {
      if(line != null &amp;&amp; line != ''){
          fs.appendFileSync(output_file, line);
      }
  }
  </code></pre>
  
  <p>I want to bring this time down to 15-20 mins. Is it possible to do so. </p>
  
  <p>Also for the record I'm trying this on a intel <strong>i7 processor</strong> with <strong>8 GB</strong> of RAM.</p>