title: Extremely slow pyspark filter
body: <p>I am performing a simple filter operation on a pyspark dataframe, that has a minhash jaccard similarity column.</p>
  <p>minhash_sig = ['123', '345']</p>
  <pre><code>minhash_sig = [str(x) for x in minhash.signature(doc)]
  
  
  df = spark.createDataFrame(....) # --dataframe with 100,000 rows
  # columns are id, and minhash_array(arrays of minhash signatures).
  df = df.withColumn('minhash_array0', array([lit(i) for i in minhash_sig]))
  intersect = size(array_intersect(&quot;minhash_array0&quot;, &quot;minhash_array&quot;))
  union = size(array_union(&quot;minhash_array0&quot;, &quot;minhash_array&quot;))
  df = df.withColumn('minhash_sim',intersect/union)
  
  df = df.filter(df.column &gt; .5)
  df.head()
  </code></pre>
  <p>I've tried <code> df.head() before the filter</code>
  and this only takes a few seconds to complete.</p>
  <p>This head after filter operation doesn't complete within 15 minutes of runtime. I've checked the number of partitions of the dataframe and it's only 4.</p>
  <p>Should I reduce the number of partitions? Is there any other solution to reduce compute time?</p>