title: Cosine similarity using TFIDF
body: <p>There are several questions on SO and the web describing how to take the <code>cosine similarity</code> between two strings, and even between two strings with TFIDF as weights.  But the output of a function like scikit's <a href="http://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise.linear_kernel.html" rel="nofollow"><code>linear_kernel</code></a> confuses me a little.</p>
  
  <p>Consider the following code:</p>
  
  <pre><code>import pandas as pd
  from sklearn.feature_extraction.text import TfidfVectorizer
  
  a = ['hello world', 'my name is', 'what is your name?']
  b = ['my name is', 'hello world', 'my name is what?']
  
  df = pd.DataFrame(data={'a':a, 'b':b})
  df['ab'] = df.apply(lambda x : x['a'] + ' ' + x['b'], axis=1)
  print(df.head())
  
                      a                 b                                   ab
  0         hello world        my name is               hello world my name is
  1          my name is       hello world               my name is hello world
  2  what is your name?  my name is what?  what is your name? my name is what?
  </code></pre>
  
  <p><strong>Question</strong>:
  I'd like to have a column that is the cosine similarity between the strings in <code>a</code> and the strings in <code>b</code>.</p>
  
  <p><strong>What I tried</strong>:</p>
  
  <p>I trained a TFIDF classifier on <code>ab</code>, so as to include all the words:</p>
  
  <pre><code>clf = TfidfVectorizer(ngram_range=(1, 1), stop_words='english')
  clf.fit(df['ab'])
  </code></pre>
  
  <p>I then got the sparse TFIDF matrix of both <code>a</code> and <code>b</code> columns:</p>
  
  <pre><code>tfidf_a = clf.transform(df['a'])
  tfidf_b = clf.transform(df['b'])
  </code></pre>
  
  <p>Now, if I use scikit's <code>linear_kernel</code>, which is what others recommended, I get back a Gram matrix of (nfeatures,nfeatures), as mentioned in their docs.</p>
  
  <pre><code>from sklearn.metrics.pairwise import linear_kernel
  linear_kernel(tfidf_a,tfidf_b)
  
  array([[ 0.,  1.,  0.],
         [ 0.,  0.,  0.],
         [ 0.,  0.,  0.]])
  </code></pre>
  
  <p>But what I need is a simple vector, where the first element is the cosin_sim between the first row of <code>a</code> and the first row of <code>b</code>, the second element is the cos_sim(a[1],b[1]), and so forth.</p>
  
  <p>Using python3, scikit-learn 0.17. </p>