title: How to extract variable weight from spark pipeline logistic model?
body: <p>I am currently trying to learn Spark Pipeline (Spark 1.6.0). I imported datasets (train and test) as oas.sql.DataFrame objects. After executing the following codes, the produced model is a <code>oas.ml.tuning.CrossValidatorModel</code>.</p>
  
  <p>You can use <code>model.transform</code> (test) to predict based on the test data in Spark. However, I would like to compare the weights that model used to predict with that from R. How to extract the weights of the predictors and intercept (if any) of model? The Scala codes are:</p>
  
  <pre><code>import sqlContext.implicits._
  import org.apache.spark.mllib.linalg.{Vectors, Vector}
  import org.apache.spark.SparkContext
  import org.apache.spark.mllib.regression.LabeledPoint
  import org.apache.spark.ml.Pipeline
  import org.apache.spark.ml.classification.{LogisticRegression, LogisticRegressionModel}
  import org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
  import org.apache.spark.ml.tuning.{ParamGridBuilder, CrossValidator}
  
  val conTrain = sc.textFile("AbsolutePath2Train.txt")
  val conTest = sc.textFile("AbsolutePath2Test.txt")
  
  // parse text and convert to sql.DataFrame
  val train = conTrain.map { line =&gt;
  val parts = line.split(",")
  LabeledPoint(parts(0).toDouble, Vectors.dense(parts(1).split(" +").map(_.toDouble)))
  }.toDF()
  val test =conTest.map{ line =&gt;
  val parts = line.split(",")
  LabeledPoint(parts(0).toDouble, Vectors.dense(parts(1).split(" +").map(_.toDouble)))
  }.toDF()
  
  // set parameter space and evaluation method
  val lr = new LogisticRegression().setMaxIter(400)
  val pipeline = new Pipeline().setStages(Array(lr))
  val paramGrid = new ParamGridBuilder().addGrid(lr.regParam, Array(0.1, 0.01)).addGrid(lr.fitIntercept).addGrid(lr.elasticNetParam, Array(0.0, 0.5, 1.0)).build()
  val cv = new CrossValidator().setEstimator(pipeline).setEvaluator(new BinaryClassificationEvaluator).setEstimatorParamMaps(paramGrid).setNumFolds(2)
  
  // fit logistic model
  val model = cv.fit(train)
  
  // If you want to predict with test
  val pred = model.transform(test)
  </code></pre>
  
  <p>My spark environment is not accessible. Thus, these codes are retyped and rechecked. I hope they are correct. So far, I have tried searching on webs, asking others. About my coding, welcome suggestions, and criticisms.</p>