title: How to use pyspark mllib RegressionMetrics with real predictions
body: <p>With pyspark 1.4 I am trying to use RegressionMetrics() for predictions
  generated by LinearRegressionWithSGD.</p>
  
  <p>All examples for RegressionMetrics() given in <a href="http://people.apache.org/~pwendell/spark-releases/latest/api/python/pyspark.mllib.html" rel="nofollow">pyspark mllib documentations</a> are for "artificial" predictions and observations
  like</p>
  
  <pre><code>predictionAndObservations = sc.parallelize([ (2.5, 3.0), (0.0, -0.5), (2.0, 2.0), (8.0, 7.0)])
  </code></pre>
  
  <p>For such "artificial" (generated with sc.parallelize) RDD everything works fine. However, when doing the same with another RDD generated in another way, I get</p>
  
  <pre><code>TypeError: DoubleType can not accept object in type &lt;type 'numpy.float64'&gt;
  </code></pre>
  
  <p>The short reproducible example is below. </p>
  
  <p>What can be the problem?</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.mllib.regression import LabeledPoint
  from pyspark.mllib.regression import LinearRegressionWithSGD, LinearRegressionModel
  from pyspark.mllib.evaluation import RegressionMetrics
  
  dataRDD = sc.parallelize([LabeledPoint(1, [1,1]), LabeledPoint(2, [2,2]), LabeledPoint(3, [3,3])])
  lrModel = LinearRegressionWithSGD.train(dataRDD)
  prediObserRDD = dataRDD.map(lambda p: (lrModel.predict(p.features), p.label)).cache()
  </code></pre>
  
  <p>Let us check that RDD is indeed of (prediction, observation) pairs</p>
  
  <pre><code>prediObserRDD.take(4) # looks OK
  </code></pre>
  
  <p>Now try to calculate metrics</p>
  
  <pre><code>metrics = RegressionMetrics(prediObserRDD)
  </code></pre>
  
  <p>It gives the following error</p>
  
  <pre><code>TypeError                                 Traceback (most recent call last)
  &lt;ipython-input-1-ca9ad8e9faf1&gt; in &lt;module&gt;()
        7 prediObserRDD = dataRDD.map(lambda p: (lrModel.predict(p.features), p.label)).cache()
        8 prediObserRDD.take(4)
  ----&gt; 9 metrics = RegressionMetrics(prediObserRDD)
       10 #metrics.explainedVariance
       11 #metrics.meanAbsoluteError
  
  /usr/local/spark-1.4.0-bin-hadoop2.6/python/pyspark/mllib/evaluation.py in __init__(self, predictionAndObservations)
       99         df = sql_ctx.createDataFrame(predictionAndObservations, schema=StructType([
      100             StructField("prediction", DoubleType(), nullable=False),
  --&gt; 101             StructField("observation", DoubleType(), nullable=False)]))
      102         java_class = sc._jvm.org.apache.spark.mllib.evaluation.RegressionMetrics
      103         java_model = java_class(df._jdf)
  
  /usr/local/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/context.py in createDataFrame(self, data, schema, samplingRatio)
      337 
      338         for row in rows:
  --&gt; 339             _verify_type(row, schema)
      340 
      341         # convert python objects to sql data
  
  /usr/local/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/types.py in _verify_type(obj, dataType)
     1027                              "length of fields (%d)" % (len(obj), len(dataType.fields)))
     1028         for v, f in zip(obj, dataType.fields):
  -&gt; 1029             _verify_type(v, f.dataType)
     1030 
     1031 _cached_cls = weakref.WeakValueDictionary()
  
  /usr/local/spark-1.4.0-bin-hadoop2.6/python/pyspark/sql/types.py in _verify_type(obj, dataType)
     1011     if type(obj) not in _acceptable_types[_type]:
     1012         raise TypeError("%s can not accept object in type %s"
  -&gt; 1013                         % (dataType, type(obj)))
     1014 
     1015     if isinstance(dataType, ArrayType):
  
  TypeError: DoubleType can not accept object in type &lt;type 'numpy.float64'&gt;
  </code></pre>
  
  <p>The same problem also appears (for another dataset and classification task) with BinaryClassificationMetrics.</p>