title: PCA in Spark MLlib and Spark ML
body: <p>Spark now has two machine learning libraries - Spark MLlib and Spark ML. They do somewhat overlap in what is implemented, but as I understand (as a person new to the whole Spark ecosystem) Spark ML is the way to go and MLlib is still around mostly for backward compatibility.</p>
  
  <p>My question is very concrete and related to PCA. In <a href="http://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html#principal-component-analysis-pca">MLlib</a> implementation there seems to be a limitation of the number of columns </p>
  
  <blockquote>
    <p>spark.mllib supports PCA for tall-and-skinny matrices stored in row-oriented format and any Vectors.</p>
  </blockquote>
  
  <p>Also, if you look at the Java code example there is also this</p>
  
  <blockquote>
    <p>The number of columns should be small, e.g, less than 1000.</p>
  </blockquote>
  
  <p>On the other hand, if you look at <a href="http://spark.apache.org/docs/latest/ml-features.html#pca">ML</a> documentation, there are no limitations mentioned.</p>
  
  <p>So, my question is - does this limitation also exists in Spark ML? And if so, why the limitation and is there any workaround to be able to use this implementation even if the number of columns is large?</p>