title: Techniques for finding near duplicate records
body: <p>I'm attempting to clean up a database that, over the years, had acquired many duplicate records, with slightly different names.  For example, in the companies table, there are names like "Some Company Limited" and "SOME COMPANY LTD!".</p>
  
  <p>My plan was to export the offending tables into R, convert names to lower case, replace common synonyms (like "limited" -> "ltd"), strip out non-alphabetic characters and then use <code>agrep</code> to see what looks similar.</p>
  
  <p>My first problem is that <code>agrep</code> only accepts a single pattern to match, and looping over every company name to match against the others is slow.  (Some tables to be cleaned will have tens, possibly hundreds of thousands of names to check.)</p>
  
  <p>I've very briefly looked at the <code>tm</code> package (<a href="http://www.jstatsoft.org/v25/i05/">JSS article</a>), and it seems very powerful but geared towards analysing big chunks of text, rather than just names.</p>
  
  <p>I have a few related questions:</p>
  
  <ol>
  <li><p>Is the <code>tm</code> package appropriate for this sort of task?  </p></li>
  <li><p>Is there a faster alternative to <code>agrep</code>?  (Said function uses the
  Levenshtein edit distance which is anecdotally slow.)</p></li>
  <li><p>Are there other suitable tools in R, apart from <code>agrep</code> and <code>tm</code>?</p></li>
  <li><p>Should I even be doing this in R, or should this sort of thing be
  done directly in the database?  (It's an Access database, so I'd
  rather avoid touching it if possible.)</p></li>
  </ol>