title: Spark - How to parse a JSON-escaped String field as a JSON Object in DataFrames?
body: <p>I have as input a set of files formatted as a single JSON object per line. The problem, however, is that one field on these JSON objects is a JSON-escaped String. Example</p>
  
  <pre><code>{"clientAttributes":{"backfillId":null,"clientPrimaryKey":"abc"},"escapedJsonPayload":"{\"name\":\"Akash\",\"surname\":\"Patel\",\"items\":[{\"itemId\":\"abc\",\"itemName\":\"xyz\"}"}
  </code></pre>
  
  <p>As I create a data frame by reading json file, it is creating data frame as below</p>
  
  <pre><code>val df = spark.sqlContext.read.json("file:///home/akaspate/sample.json")
  df: org.apache.spark.sql.DataFrame = [clientAttributes: struct&lt;backfillId: string, clientPrimaryKey: string&gt;, escapedJsonPayload: string]
  </code></pre>
  
  <p>As we can see "<code>escapedJsonPayload</code>" is String and I need it to be Struct.</p>
  
  <p>Note: I got similar question in StackOverflow and followed it (<a href="https://stackoverflow.com/questions/37225414/how-to-let-spark-parse-a-json-escaped-string-field-as-a-json-object-to-infer-the">How to let Spark parse a JSON-escaped String field as a JSON Object to infer the proper structure in DataFrames?</a>) but it is giving me "[_corrupt_record: string]"</p>
  
  <p>I have tried below steps </p>
  
  <ol>
  <li><p><code>val df = spark.sqlContext.read.json("file:///home/akaspate/sample.json") (Work file)</code></p></li>
  <li><p><code>val escapedJsons: RDD[String] = sc.parallelize(Seq("""df"""))</code></p></li>
  <li><p><code>val unescapedJsons: RDD[String] = escapedJsons.map(_.replace("\"{", "{").replace("\"}", "}").replace("\\\"", "\""))</code></p></li>
  <li><p><code>val dfJsons: DataFrame = spark.sqlContext.read.json(unescapedJsons) (This results in [_corrupt_record: string])</code></p></li>
  </ol>
  
  <p>Any help would be appreciated</p>