title: Speed up collaborative filtering for large dataset in Spark MLLib
body: <p>I'm using MLlib's matrix factorization to recommend items to users. I have about a big implicit interaction matrix of M=20 million users and N=50k items. After training the model I want to get a short list(e.g. 200) of recommendations for each user. I tried <code>recommendProductsForUsers</code> in <code>MatrixFactorizationModel</code> but it's very very slow (ran 9 hours but still far from finish. I'm testing with 50 executors, each with 8g memory). This might be expected since <code>recommendProductsForUsers</code> need to calculate all <code>M*N</code> user-item interactions and get top for each user.</p>
  
  <p>I'll try use more executors but from what I saw from the application detail on Spark UI, I doubt that it can finish in hours or a day even I have 1000 executors (after 9hours it's still in the <code>flatmap</code> here <a href="https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L279-L289" rel="noreferrer">https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/recommendation/MatrixFactorizationModel.scala#L279-L289</a>, 10000 total tasks and only ~200 finished)
  Are there any other things that I can tune to speed up the recommendation process beside increasing # of executors? </p>
  
  <p>Here is sample code:</p>
  
  <pre><code>val data = input.map(r =&gt; Rating(r.getString(0).toInt, r.getString(1).toInt, r.getLong(2))).cache
  val rank = 20
  val alpha = 40
  val maxIter = 10
  val lambda = 0.05
  val checkpointIterval = 5
  val als = new ALS()
      .setImplicitPrefs(true)
      .setCheckpointInterval(checkpointIterval)
      .setRank(rank)
      .setAlpha(alpha)
      .setIterations(maxIter)
      .setLambda(lambda)
  val model = als.run(ratings)
  val recommendations = model.recommendProductsForUsers(200)
  recommendations.saveAsTextFile(outdir)
  </code></pre>