title: How to remove stop words from a large collection files with more efficient way?
body: <p>I have 200,000 files for which I've to process and extract tokens for each file. The size of all files is 1.5GB. When I wrote the code for extracting tokens from each file, it works well. Over all execution time is 10mins.</p>
  
  <p>After that, I tried to remove <code>stopwords</code> Performance went down badly. It's taking 25 to 30 mins.</p>
  
  <p>I'm using stop words from the website <a href="http://jmlr.org/papers/volume5/lewis04a/a11-smart-stop-list/english.stop" rel="nofollow noreferrer">here</a> There are around 571 stop words. The general procedure is to extract each stop word from a text file at once and compare with each token in the file.</p>
  
  <p>This is the stub of code</p>
  
  <pre><code>StringBuilder sb = new StringBuilder();
  for(String s : tokens)
      Scanner sc=new Scanner(new File("stopwords.txt"));
      while(sc.hasNext())
      {
          if(sc.next().equals(s)){
              flag = true;
              break;
          }
      }
      if(flag)
          sb.append(s + "\n" );
      flag = false;
  }
  String str = sb.toString()
  </code></pre>
  
  <p>**Ignore errors.</p>
  
  <p>The performance of above code is at least 10 times less than below code. It takes 50 to 60 mins to execute.</p>
  
  <pre><code>StringBuilder sb = new StringBuilder();
  String s = tokens.toString();
  String str = s.replaceAll("StopWord1|Stopword2|Stopword3|........|LastStopWord"," ");
  </code></pre>
  
  <p>Performance is far good. This takes 20 to 25 mins.</p>
  
  <p>Is there any better procedure?</p>