title: How to use schema from one of the columns of a dataset to parse another column and create a flattened dataset using Spark Streaming 2.2.0?
body: <p>How to use schema from one of the columns of a dataset to parse another column and create a flattened dataset using Spark Streaming 2.2.0? </p>
  
  <p>I have the following <strong>source data frame</strong> that I create from reading messages from Kafka</p>
  
  <pre><code>col1: string
  col2: json string
  
  
        col1    |   col2 
  ---------------------------------------------------------------------------
     schemaUri1 | "{"name": "foo", "zipcode": 11111}"
     schemaUri2 | "{"name": "bar", "zipcode": 11112, "id": 1234}"
     schemaUri1 | "{"name": "foobar", "zipcode": 11113}"
     schemaUri2 | "{"name": "barfoo", "zipcode": 11114, "id": 1235, "interest": "reading"}"
  </code></pre>
  
  <p><strong>My target data frame</strong></p>
  
  <pre><code>name   | zipcode | id  | interest
  -------------------------------- 
  foo    | 11111  | null | null
  bar    | 11112  | 1234 | null
  foobar | 11113  | null | null
  barfoo | 11114  | 1235 | reading
  </code></pre>
  
  <p><strong>Assume you have the following function</strong></p>
  
  <p>// This function returns a StructType that represents a schema for a given schemaUri</p>
  
  <pre><code>public StructType getSchema(String schemaUri)
  </code></pre>