title: Spark from_json with dynamic schema
body: <p>I am trying to use Spark for processing JSON data with variable structure(nested JSON). Input JSON data could be very large with more than 1000 of keys per row and one batch could be more than 20 GB. 
  Entire batch has been generated from 30 data sources and 'key2' of each JSON can be used to identify the source and structure for each source is predefined.</p>
  
  <p>What would be the best approach for processing such data?
  I have tried using from_json like below but it works only with fixed schema and to use it first I need to group the data based on each source and then apply the schema. 
  Due to large data volume my preferred choice is to scan the data only once and extract required values from each source, based on predefined schema.</p>
  
  <pre><code>import org.apache.spark.sql.types._ 
  import spark.implicits._
  
  val data = sc.parallelize(
      """{"key1":"val1","key2":"source1","key3":{"key3_k1":"key3_v1"}}"""
      :: Nil)
  val df = data.toDF
  
  
  val schema = (new StructType)
      .add("key1", StringType)
      .add("key2", StringType)
      .add("key3", (new StructType)
      .add("key3_k1", StringType))
  
  
  df.select(from_json($"value",schema).as("json_str"))
    .select($"json_str.key3.key3_k1").collect
  res17: Array[org.apache.spark.sql.Row] = Array([xxx])
  </code></pre>