title: How to count occurrences of a substring within string fast with Ruby
body: <p>I have a text file sized 300MB, I want to count the occurrences of each 10,000 substrings in the file. I want to know how to do it fast.</p>
  
  <p>Now, I use the following code:</p>
  
  <pre><code>
  content = IO.read("path/to/mytextfile")
  Word.each do |w|
    w.occurrence = content.scan(w.name).size
    w.save
  end
  </code>
  </pre>
  
  <p>Word is an ActiveRecord class.</p>
  
  <p>It took me almost 1 day to finish the counting. Is there anyway to do it faster? Thanks.</p>
  
  <p>Edit1:
  Thank you again. I am running rails 2.3.9. The <code>name</code> filed of words table contains what I am searching for, and it contains only unique values. Instead of using <code>Word.each</code>, I use batch(1000 rows a time) load. It should help. </p>
  
  <p>I rewrited the whole code with the idea from bpaulon. Now it only took a few hours to finish the counting. </p>
  
  <p>I profiled the new version code, now the largest time costing methods are utf8 encode supported string truncating code</p>
  
  <pre><code>def truncate(n)
    self.slice(/\A.{0,#{n}}/m)
  end
  </code></pre>
  
  <p>and characters counting code </p>
  
  <pre><code>def utf8_length
    self.unpack('U*').size
  end
  </code></pre>
  
  <p>Any other faster methods to replace them?</p>