title: Pyspark py4j PickleException: "expected zero arguments for construction of ClassDict"
body: <p>This question is directed towards persons familiar with py4j - and can help to resolve a pickling error.  I am trying to add a method to the pyspark PythonMLLibAPI that accepts an RDD of a namedtuple, does some work, and returns a result in the form of an RDD.  </p>
  
  <p>This method is modeled after the PYthonMLLibAPI.trainALSModel() method, whose analogous <em>existing</em> relevant portions are:</p>
  
  <pre><code>  def trainALSModel(
      ratingsJRDD: JavaRDD[Rating],
      .. )
  </code></pre>
  
  <p>The <em>existing</em> python Rating class used to model the new code is:</p>
  
  <pre><code>class Rating(namedtuple("Rating", ["user", "product", "rating"])):
      def __reduce__(self):
          return Rating, (int(self.user), int(self.product), float(self.rating))
  </code></pre>
  
  <p>Here is the attempt So here are the relevant classes: </p>
  
  <p><em>New</em> python class pyspark.mllib.clustering.MatrixEntry:</p>
  
  <pre><code>from collections import namedtuple
  class MatrixEntry(namedtuple("MatrixEntry", ["x","y","weight"])):
      def __reduce__(self):
          return MatrixEntry, (long(self.x), long(self.y), float(self.weight))
  </code></pre>
  
  <p><em>New</em> method <em>foobarRDD</em> In PythonMLLibAPI: </p>
  
  <pre><code>  def foobarRdd(
      data: JavaRDD[MatrixEntry]): RDD[FooBarResult] = {
      val rdd = data.rdd.map { d =&gt; FooBarResult(d.i, d.j, d.value, d.i * 100 + d.j * 10 + d.value)}
      rdd
    }
  </code></pre>
  
  <p>Now let us try it out:</p>
  
  <pre><code>from pyspark.mllib.clustering import MatrixEntry
  
  def convert_to_MatrixEntry(tuple):
    return MatrixEntry(*tuple)
  
  from pyspark.mllib.clustering import *
  pic = PowerIterationClusteringModel(2)
  tups = [(1,2,3),(4,5,6),(12,13,14),(15,7,8),(16,17,16.5)]
  trdd = sc.parallelize(map(convert_to_MatrixEntry,tups))
  
  # print out the RDD on python side just for validation
  print "%s" %(repr(trdd.collect()))
  
  from pyspark.mllib.common import callMLlibFunc
  pic = callMLlibFunc("foobar", trdd)
  </code></pre>
  
  <p>Relevant portions of results:</p>
  
  <pre><code>[(1,2)=3.0, (4,5)=6.0, (12,13)=14.0, (15,7)=8.0, (16,17)=16.5]
  </code></pre>
  
  <p>which shows the input rdd is 'whole'.  However the pickling was unhappy:  </p>
  
  <pre><code>5/04/27 21:15:44 ERROR Executor: Exception in task 6.0 in stage 1.0 (TID 14)
  net.razorvine.pickle.PickleException: expected zero arguments for construction of ClassDict
  (for pyspark.mllib.clustering.MatrixEntry)
      at net.razorvine.pickle.objects.ClassDictConstructor.construct(ClassDictConstructor.java:23)
      at net.razorvine.pickle.Unpickler.load_reduce(Unpickler.java:617)
      at net.razorvine.pickle.Unpickler.dispatch(Unpickler.java:170)
      at net.razorvine.pickle.Unpickler.load(Unpickler.java:84)
      at net.razorvine.pickle.Unpickler.loads(Unpickler.java:97)
      at org.apache.spark.mllib.api.python.SerDe$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(PythonMLLibAPI.scala:1167)
      at org.apache.spark.mllib.api.python.SerDe$$anonfun$pythonToJava$1$$anonfun$apply$1.apply(PythonMLLibAPI.scala:1166)
      at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
      at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
      at scala.collection.Iterator$class.foreach(Iterator.scala:727)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
      at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
      at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
      at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
      at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
      at scala.collection.AbstractIterator.to(Iterator.scala:1157)
      at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
      at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
      at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
      at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
      at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:819)
      at org.apache.spark.rdd.RDD$$anonfun$17.apply(RDD.scala:819)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1523)
      at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1523)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
      at org.apache.spark.scheduler.Task.run(Task.scala:64)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:212)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:724)
  </code></pre>
  
  <p>Below is a visual of the python invocation stack trace:</p>
  
  <p><img src="https://i.stack.imgur.com/7crwS.jpg" alt="enter image description here"></p>