title: Parse arbitrary JSON with Spark's from_json
body: <p>I have a dataset that looks like this:</p>
  
  <pre><code>~ ❯ head example.csv
  ix,value
  1,{"abc": {"name": "bob", "profession": "engineer"}}
  2,{"def": {"name": "sarah", "profession": "scientist"}, "ghi": {"name": "matt", "profession": "doctor"}}
  </code></pre>
  
  <p>The <code>value</code> column contains JSON blobs. As you can see, each JSON blob itself is of the form {A:B} where A is a random/arbitrary string, and B is a relatively well-formed JSON object.</p>
  
  <p>The result I wish to obtain from this is:</p>
  
  <pre><code>ix,names,professions
  1,[bob],[engineer]
  2,[sarah,matt],[scientist,doctor]
  </code></pre>
  
  <p>To then explode into this:</p>
  
  <pre><code>ix,name,profession
  1,bob,engineer
  2,sarah,scientist
  2,matt,doctor
  </code></pre>
  
  <p>Because I don't know the possible keys of A, I am having difficulty parsing the JSON blob into a StructType (I cannot enumerate all the possible keys) or MapType (not supported by from_json):</p>
  
  <pre><code>&gt;&gt;&gt; rdd.withColumn('parsed', F.from_json(F.col('value'), MapType(StringType(), MapType(StringType(), StringType(), False), False)))
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    File "/Users/gberger/Projects/spark/python/pyspark/sql/dataframe.py", line 1800, in withColumn
      return DataFrame(self._jdf.withColumn(colName, col._jc), self.sql_ctx)
    File "/Users/gberger/Projects/spark/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__
    File "/Users/gberger/Projects/spark/python/pyspark/sql/utils.py", line 69, in deco
      raise AnalysisException(s.split(': ', 1)[1], stackTrace)
  pyspark.sql.utils.AnalysisException: u"cannot resolve 'jsontostructs(`value`)' due to data type mismatch: Input schema map&lt;string,map&lt;string,string&gt;&gt; must be a struct or an array of structs.;;\n'Project [id#35, value#36, jsontostructs(MapType(StringType,MapType(StringType,StringType,false),false), value#36, Some(Europe/London)) AS parsed#46]\n+- Relation[id#35,value#36] csv\n"
  </code></pre>
  
  <p>I know I could use a UDF, but it would severely impact performance; I wish to keep within native Spark functions if possible.</p>