title: Spark: Writing to Avro file
body: <p>I am in Spark, I have an RDD from an Avro file. I now want to do some transformations on that RDD and save it back as an Avro file:</p>
  
  <pre><code>val job = new Job(new Configuration())
  AvroJob.setOutputKeySchema(job, getOutputSchema(inputSchema))
  
  rdd.map(elem =&gt; (new SparkAvroKey(doTransformation(elem._1)), elem._2))
     .saveAsNewAPIHadoopFile(outputPath, 
    classOf[AvroKey[GenericRecord]], 
    classOf[org.apache.hadoop.io.NullWritable], 
    classOf[AvroKeyOutputFormat[GenericRecord]], 
    job.getConfiguration)
  </code></pre>
  
  <p>When running this Spark complains that Schema$recordSchema is not serializable.</p>
  
  <p>If I uncomment the .map call (and just have rdd.saveAsNewAPIHadoopFile), the call succeeds.</p>
  
  <p>What am I doing wrong here?</p>
  
  <p>Any idea?</p>