title: How to find closest string match in pandas dataframe column in another dataframe column
body: <p>I'm trying to match approximate movie titles in one dataframe to their closest match of actual movie titles in another dataframe. Here is the first dataframe:</p>
  
  <pre><code>Old ID  New ID      Movie1                  Movie2
  3101    771355141   this. This is Dogma     Shaun of the Dead.
  11903   18330       tale mise en scene      giallo thriller posing as
  16077   771225176   Evil Dead II            Brothers Grimm and Hawkeye
  NaN     381422014   'Requiem for a Dream'   'After Dark." If only
  4540    770676801   Ocean's Eleven          Saved By The Bell.
  9103    770673272   It's Godzilla           The Blair Witch Project,
  2473    49248746    day classic. Die Hard   Fellini? No. But maybe.
  </code></pre>
  
  <p>This is the second dataframe:</p>
  
  <pre><code>Old id  New id  Title
  NaN     21736.0 Peter Pan
  NaN 771471359.0 Dragonheart Battle for the Heartfire
  NaN 770725090.0 The Nude Vampire Vampire nue, La
  2281.0  19887.0 Beyond the Clouds
  10913.0 11286.0 Wild America
  NaN   17635.0   Sexual Dependency
  NaN 666370586.0 Body Slam
  709.0   771203994.0 Hatchet II
  NaN  11655.0    Lion of the Desert Omar Mukhtar
  15492.0 770681831.0 Imagine That
  </code></pre>
  
  <p>I'm trying to match the approximate Movie1 &amp; Movie2 titles in df1 to the actual titles in df2.  What I've done so far is to define a function that returns the closest movie title based on the string distance defined in difflib.  </p>
  
  <pre><code>from difflib import SequenceMatcher as SM    
  def find_closest(approx_title, movies):
          """Return the title in movies that is closest to the approximate title based on
          difflib distance
          """
          return max(movies, key = lambda title: SM(None, approx_title, title).ratio())
  </code></pre>
  
  <p>My idea was to apply this function to every element in Movie1 &amp; Movie2 and save it as a new column. </p>
  
  <pre><code>movie_lst = rt_info['Title'].tolist()
  movie_lst = [str(x) for x in movie_lst]
  movie1_match = meets_df['Movie1'].apply(lambda movie: 
      find_closest(movie, movie_lst))
  </code></pre>
  
  <p>However, this function runs very slowly (there are 24883 movies in movie_lst) and I think applying the function as a lambda may be inefficient (compounded with the lambda in the max function)  How can I optimize this process?  Thank you! </p>