title: SQL - Similarity between two strings of varying length
body: <p>I have a SQL Server table of products, and each product has a description that is publicly available on our website. I want to prevent, or at least warn our users when, a description is too similar to another product's description. Each product's description length can greatly vary. </p>
  
  <p>I'd like query for products with descriptions that include duplicate/similar paragraphs/blocks of text between one another. i.e. String A has a bunch of unique content, but shares a similar/identical paragraph w/ string B. However, I'm not sure which similarity algorithm is best to use:</p>
  
  <ul>
  <li><p>The <a href="http://en.wikipedia.org/wiki/Levenshtein_distance" rel="nofollow">Levenshtein distance</a> and <a href="http://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance" rel="nofollow">Jaro-Winler distance</a> algorithms appear to only work well with short strings.</p></li>
  <li><p>I'm not sure the <a href="http://en.wikipedia.org/wiki/Longest_common_subsequence_problem" rel="nofollow">longest common subsequence algorithm</a> takes into account large differences very well. i.e. it appears to ignore potential space between two characters, finding any similar combination sequence.</p></li>
  </ul>
  
  <p><a href="http://www.technologyreview.com/view/423446/the-unexpected-ubiquity-of-spam-detection-algorithms/" rel="nofollow">Fuzzy hashing</a> sort of sounds what I'm looking for, but I'm <em>not just</em> looking for duplicate content w/ subtle differences. I'm <em>also</em> looking for duplicate content w/ subtle differences injected <strong>within a unique block of text</strong>. And I'd have no idea how to implement fuzzy hashes in SQL. <a href="http://technet.microsoft.com/en-us/library/ms187384.aspx" rel="nofollow">SOUNDEX()</a> and <a href="http://technet.microsoft.com/en-us/library/ms188753.aspx" rel="nofollow">DIFFERENCE()</a> appear to use fuzzy hashing, but are quite imprecise for my use case.</p>
  
  <p>Ideally the similarity SQL function would be fast, but I could store cached similarity values in another table and schedule a job to occasionally update.</p>
  
  <p><strong>What is the best algorithm/SQL (or CLR integration) implementation to accomplish this?</strong></p>