title: Finding top words per kmeans cluster
body: <p>I have following section of code that maps the TFIDF for a collection of tweets onto original words, which are then used to find top words in each cluster:</p>
  
   <pre class="lang-py prettyprint-override"><code>#document = sc.textFile("&lt;text file path&gt;").map(lambda line: line.split(" "))
  #"tfidf" is an rdd of tweets contained in "document"
  #map tfidf to original tweets and cluster similar tweets
  clusterIds = clusters.predict(tfidf)
  mapped_value = clusterIds.zip(document)
  cluster_value = mapped_value.reduceByKey(lambda a,b: a+b).take(cluster_num)
  
  
  #Fetch the top 5 words from each cluster
  topics = []
  for i in cluster_value:
      word_count = sc.parallelize(i[1])
      topics.append(
          word_count.map(lambda x: (x,1))
              .reduceByKey(lambda x,y: x+y)
              .takeOrdered(5, key=lambda x: -x[1]))
  </code></pre>
  
  <p>Is there a better way to do this?
  I see on Spark UI that my code takes some 70 min when doing reduceByKey() operation on a cluster of 4 VMs with 20.5 Gb of executor memory and 2 gb of driver memory. Number of tweets is 500K. Text file size of 31 Mb post processing for stop words and junk characters.</p>