title: Reading Xml file in PySpark via Hadoop Streaming
body: <p>I'm trying to adapt the code <a href="http://apache-spark-user-list.1001560.n3.nabble.com/Parsing-a-large-XML-file-using-Spark-td19239.html" rel="nofollow noreferrer">here</a> from a Scala version to a PySpark version. Here's the code I'm using:</p>
  
  <pre><code>    conf = SparkConf().setAppName("Parse Xml File")
      sc = SparkContext(conf = conf)
      sqlContext = HiveContext(sc)
  
      sc._jsc.hadoopConfiguration().set('stream.recordreader.class', 'org.apache.hadoop.streaming.StreamXmlRecordReader')
      sc._jsc.hadoopConfiguration().set('stream.recordreader.begin', '&lt;page&gt;')
      sc._jsc.hadoopConfiguration().set('stream.recordreader.end', '&lt;/page&gt;')
  
      xml_sdf = sc.newAPIHadoopFile(xml_data_path,
                                         'org.apache.hadoop.streaming.StreamInputFormat',
                                         'org.apache.hadoop.io.Text',
                                         'org.apache.hadoop.io.Text')
      print("Found {0} records.".format(wiki_xml_sdf.count()))
  
      sc.stop()
  </code></pre>
  
  <p>Error I'm getting is:</p>
  
  <pre><code>py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.newAPIHadoopFile.
  : java.lang.ClassCastException: org.apache.hadoop.streaming.StreamInputFormat cannot be cast to org.apache.hadoop.mapreduce.InputFormat
  </code></pre>
  
  <p>Is there a different input format / settings that I can use to make it work?</p>