title: How to do vector operations on pyspark dataframe columns?
body: <p>I have a dataframe like so:</p>
  <pre><code>id | vector1 | id2 | vector2
  </code></pre>
  <p>where the ids are ints and the vectors are SparseVector types.</p>
  <p>For each row, I want to add on a column that is cosine similarity, which would be done by
  <code>vector1.dot(vector2)/(sqrt(vector1.dot(vector1)*sqrt(vector2.dot(vector2))</code>
  but I can't figure out how to use this to put it into a new column.  I've tried making a udf, but can't seem to figure it out</p>