<p>Dedupe should work fine for data of that size. </p>

<p>There has been some excellent work by <a href="https://people.cs.umass.edu/~mwick/MikeWeb/Publications_files/wick12hierarchical.pdf" rel="nofollow">Michael Wick</a> and <a href="http://arxiv.org/abs/1312.4645" rel="nofollow">Beka Steorts</a> that have better complexity than dedupe. </p>
Efficient Algorithm for Detecting Text Duplicates in Big Dataset
<p>I'm working on detecting duplicates in a list of around 5 million addresses, and was wondering if there was consensus on an efficient algorithm for such a purpose. I've looked at the Dedupe library on Gitbub (<a href="https://github.com/datamade/dedupe" rel="nofollow">https://github.com/datamade/dedupe</a>), but based on the documentation I'm not clear that this would scale to a large application well. </p>

<p>As an aside, I'm just looking to define duplicates based on textual similarity - have already done a lot of cleaning on the addresses. I've been using a crude method using Levenshtein distance, but was wondering if there's anything more efficient for large datasets.</p>

<p>Thanks,</p>