title: Understanding another's text-mining function that removes similar strings
body: <p>I’m trying to replicate the methodology from this article, <a href="http://fivethirtyeight.com/features/these-are-the-phrases-each-gop-candidate-repeats-most/" rel="nofollow">538 Post about Most Repetitive Phrases</a>, in which the author mined US presidential debate transcripts to determine the most repetitive phrases for each candidate.</p>
  
  <p>I'm trying to implement this methodology with another dataset in R with the <code>tm</code> package.</p>
  
  <p>Most of the code (<a href="https://github.com/fivethirtyeight/data/blob/master/repeated-phrases-gop/robopol2.py" rel="nofollow">GitHub repository</a>) concerns mining the transcripts and assembling counts of each ngram, but I get lost at the <code>prune_substrings()</code> function code below:</p>
  
  <pre><code>def prune_substrings(tfidf_dicts, prune_thru=1000):
  
      pruned = tfidf_dicts
  
      for candidate in range(len(candidates)):
          # growing list of n-grams in list form
          so_far = []
  
          ngrams_sorted = sorted(tfidf_dicts[candidate].items(), key=operator.itemgetter(1), reverse=True)[:prune_thru]
          for ngram in ngrams_sorted:
              # contained in a previous aka 'better' phrase
              for better_ngram in so_far:
                  if overlap(list(better_ngram), list(ngram[0])):
                      #print "PRUNING!! "
                      #print list(better_ngram)
                      #print list(ngram[0])
  
                      pruned[candidate][ngram[0]] = 0
              # not contained, so add to so_far to prevent future subphrases
              else:
                  so_far += [list(ngram[0])]
  
      return pruned 
  </code></pre>
  
  <p>The input of the function, <code>tfidf_dicts</code>, is an array of dictionaries (one for each candidate) with ngrams as keys and tf-idf scores as values. For example, Trump's tf-idf dict begins like this:</p>
  
  <pre><code>trump.tfidf.dict = {'we don't win': 83.2, 'you have to': 72.8, ... }
  </code></pre>
  
  <p>so the structure of the input is like this:</p>
  
  <pre><code>tfidf_dicts = {trump.tfidf.dict, rubio.tfidf.dict, etc }
  </code></pre>
  
  <p>MY understanding is that <code>prune_substrings</code> does the following things, but I'm stuck on the <code>else if</code> clause, which is a pythonic thing I don't understand yet.</p>
  
  <blockquote>
    <p>A.    create list : pruned as tfidf_dicts; a list of tfidf dicts for each candidate</p>
    
    <p>B loop through each candidate:</p>
    
    <ol>
    <li>so_far = start an empty list of ngrams gone through so so_far</li>
    <li>ngrams_sorted = sorted member's tf-idf dict from smallest to biggest</li>
    <li>loop through each ngram in sorted
    
    <ul>
    <li>loop through each better_ngram in so_far
    
    <ol>
    <li>IF overlap b/w (below) == TRUE:
    
    <ul>
    <li>better_ngram (from so_far) and</li>
    <li>ngram (from ngrams_sorted)</li>
    <li>THEN zero out tf-idf for ngram</li>
    </ul></li>
    <li>ELSE if (WHAT?!?)
    
    <ul>
    <li>add ngram to list, so_far</li>
    </ul></li>
    </ol></li>
    </ul></li>
    </ol>
    
    <p>C. return pruned, i.e. list of unique ngrams sorted in order</p>
  </blockquote>
  
  <p>Any help at all is much appreciated!</p>