title: Parsing huge file in Node.js faster, reading the file line-by-line, currently running too slow
body: <p>I have got a very big text file with over 15m records. It's a vocabulary file which shows the word on the left and the occurrence of the word on the right. The end result I am looking for is to filter out all these words and any words that occur less than 200 times are classed as difficult words.</p>
  
  <p>For example, this is what the text file will have inside: </p>
  
  <pre><code>of  12765289150
  and 12522922536
  </code></pre>
  
  <p>The word on the left and the occurrence on the right separated by a tab.</p>
  
  <p>Currently, the system I have in place returns me the words which occur <code>&lt;=</code> 200 times which is great but it takes over 15 seconds to read through the file and return these values. How could I speed this process up?</p>
  
  <p>This is the code that I am running: </p>
  
  <pre><code>exports.readText = (req, res, next) =&gt; {
    const fs = require('fs'),
      es = require('event-stream'),
      path = require("path"),
      filePath = path.join(__dirname, "../documents/vocab_cs");
  
    const infrequentWords = [];
  
    let s = fs.createReadStream(filePath)
      .pipe(es.split())
      .pipe(es.mapSync((line) =&gt; {
          const lines = line.split('\t');
          const freq = Number(lines[1]);
          if (freq &lt;= 200) {
            infrequentWords.push(lines[0]);
          }
          // pause the readstream
          s.pause();
  
          // process line here and call s.resume() when rdy
  
          // resume the readstream, possibly from a callback
          s.resume();
        })
        .on('error', (err) =&gt; {
          console.log('Error while reading file.', err);
        })
        .on('end', () =&gt; {
          infrequentWordsString = infrequentWords.join(' ');
          res.status(200).json(infrequentWordsString);
          console.log('Read entire file.')
        })
      );
  }
  </code></pre>
  
  <p>Any advice would be highly appreciated as I do not have much experience in this field and I am completely stuck on this issue! </p>
  
  <p>Just need this whole process to run much faster and return me these rare words <code>&lt;= 200</code>, because they get displayed on the frontend to help readers understand the text better, but with this delay, the user would have to wait over 10/15 seconds.</p>
  
  <p>Thanks in advance!</p>