title: How to find and correct typos in a data frame in python pandas
body: <p>I have a data frame with names, surnames, birthdays and some random variables. Lets say it looks like this:</p>
  
  <pre><code>    BIRTH  NAME    SURNAME random_value institution
   1      1  Luke  Skywalker            1        1111
   2      1  Luke  Skywalker            2        1111
   4      2  Leia     Organa            3        1211
   5      3   Han       Solo            7        1342
   7      1   Ben       Solo            1        1342
   8      5 Lando Calrissian            3        1111
   9      3   Han       Solo            4        1111
   10     3   Ham       Solo            4        1342
   11     1  Luke  Wkywalker            9        1111
  </code></pre>
  
  <p>How can I figure out, if there is a typo in name or surname, based on <code>BIRTH</code>,<code>NAME</code>and <code>SURNAME</code>, and then replace the typo with the correct name or surname?</p>
  
  <p>For example, we see, that there are two <code>Han Solo</code>s with birthdays on <code>3</code>and then there is a <code>Ham Solo</code> with the same birthdate. What I would like this algorithm to do is figure out that <code>Ham</code>is wrong and replace it with <code>Han</code>.</p>
  
  <p>If there are two different spellings which have equal number of occurrences (for same <code>BIRTH</code>), it doesn't really matter, which one is chosen, as long that all the <code>NAME</code>or <code>SURNAME</code>for this group is the same (so always <code>Ham</code>or <code>Han</code>but not mixed for the same <code>BIRTH</code>).</p>
  
  <p>So the end result would be this:</p>
  
  <pre><code>    BIRTH  NAME    SURNAME random_value institution
   1      1  Luke  Skywalker            1        1111
   2      1  Luke  Skywalker            2        1111
   4      2  Leia     Organa            3        1211
   5      3   Han       Solo            7        1342
   7      1   Ben       Solo            1        1342
   8      5 Lando Calrissian            3        1111
   9      3   Han       Solo            4        1111
   10     3   Han       Solo            4        1342
   11     1  Luke  Skywalker            9        1111
  </code></pre>
  
  <p>Is there any automated way to do this? My data set is large (>3mill rows) and it would be impossible to check manually.</p>
  
  <p>I would imagine that we look for all the names and surnames with the same birth and then check, if there are some singular outliers that differ only by a letter or that the order of the letters is switched (<code>Luke</code>vs <code>Lkue</code>). When we find an outlier like that, we replace it.</p>
  
  <p>I already asked this in <code>R</code>forums (<a href="https://stackoverflow.com/questions/45990947/how-to-find-a-typo-in-a-data-frame-and-replace-it">How to find a typo in a data frame and replace it</a>) and got a reply. I tried to implement that method directly (it failed miserably, because of time complexity). I then modified it and used it on smaller sub data frames (grouped by <code>BIRTH</code>). But even then, I stoped the process when <code>R</code> estimated it will take more than 37 hours and that number was climbing.</p>
  
  <p>Is there any faster way to do this in <code>python</code>. I would be really grateful if you could give me any suggestion.</p>
  
  <p><strong>Edit:</strong> As pointed out in the comments, there is a possibility, that two persons with a similar name (Jon/John Smit(h)) will be born on the same date. In this case, we would either look at the institution column (each person should (but the reality of the data set can be a bit different) appear 8-9 times with one institution number and 3-4 times with another one). Also, more than one person share the same institution number. </p>
  
  <p>But because of possible faulty data with <code>institution</code>, we can also use the following reasoning:
  If the identical full name for the same birthday appears more than twice, we can conclude that it is in fact a new person, not a typo (as two (out of max 13) identical typos for the same person are unlikely), and leave it's name like it is.</p>