<p>You can use the built-in <code>columnSimilarities()</code> method on a <code>RowMatrix</code>, that can both calculate the exact cosine similarities, or estimate it using the <a href="https://arxiv.org/abs/1304.1467" rel="noreferrer">DIMSUM</a> method, which will be considerably faster for larger datasets. The difference in usage is that for the latter, you'll have to specify a <code>threshold</code>. </p>

<p>Here's a small reproducible example:</p>

<pre><code>from pyspark.mllib.linalg.distributed import RowMatrix
rows = sc.parallelize([(1, 2, 3), (4, 5, 6), (7, 8, 9), (10, 11, 12)])

# Convert to RowMatrix
mat = RowMatrix(rows)

# Calculate exact and approximate similarities
exact = mat.columnSimilarities()
approx = mat.columnSimilarities(0.05)

# Output
exact.entries.collect()
[MatrixEntry(0, 2, 0.991935352214),
 MatrixEntry(1, 2, 0.998441152599),
 MatrixEntry(0, 1, 0.997463284056)]
</code></pre>
Apache Spark Python Cosine Similarity over DataFrames
<p>For a Recommender System, I need to compute the cosine similarity between all the columns of a whole Spark DataFrame.</p>

<p>In Pandas I used to do this:</p>

<pre><code>import sklearn.metrics as metrics
import pandas as pd

df= pd.DataFrame(...some dataframe over here :D ...)
metrics.pairwise.cosine_similarity(df.T,df.T)
</code></pre>

<p>That generates the Similarity Matrix between the columns (since I used the transposition)</p>

<p>Is there any way to do the same thing in Spark (Python)?</p>

<p>(I need to apply this to a matrix made of tens of millions of rows, and thousands of columns, so that's why I need to do it in Spark)</p>