title: Computing cosine similarity on two large lists containing Sparse vectors in PySpark
body: <p>I'm facing trouble computing cosine sim on two large lists in PySpark.
  There is list A containing 280M sparse vectors having features 262144.
  Similarly there is list B containing 56k sparse vectors having features 262144.</p>
  <p>When executing below statement cosine sim using scipy, pyspark is taking lot of time to compute and was never ending.</p>
  <p><strong>consine_sim= 1 - sp.distance.cdist(B, A, 'cosine')</strong></p>
  <p>Below are my spark conf</p>
  <ul>
  <li>driver memory - 35gb</li>
  <li>executor memory - 35gb</li>
  </ul>