title: create a spark dataframe from a nested json file in scala
body: <p>I have a json file that looks like this</p>
  
  <pre><code>{
  "group" : {},
  "lang" : [ 
      [ 1, "scala", "functional" ], 
      [ 2, "java","object" ], 
      [ 3, "py","interpreted" ]
  ]
  }
  </code></pre>
  
  <p>I tried to create a dataframe using</p>
  
  <pre><code>val path = "some/path/to/jsonFile.json"
  val df = sqlContext.read.json(path)
  df.show()
  </code></pre>
  
  <p>when I run this I get</p>
  
  <pre><code>df: org.apache.spark.sql.DataFrame = [_corrupt_record: string]
  </code></pre>
  
  <p>How do we create a df based on contents of "lang" key? I do not care about group{} all I need is, pull data out of "lang" and apply case class like this</p>
  
  <pre><code>case class ProgLang (id: Int, lang: String, type: String )
  </code></pre>
  
  <p>I have read this post <a href="https://stackoverflow.com/questions/38895057/reading-json-with-apache-spark-corrupt-record">Reading JSON with Apache Spark - `corrupt_record`</a> and understand that each record needs to be on a newline but in my case I cannot change the file structure </p>