title: How to convert a column of StringType (json string) to an array of StructType
body: <p>I realized that I may need to add a bit more details. Imagine that I have 2 columns in a dataframe. Both are strings, one is an ID, the other is a json string. </p>
  
  <p>This can be constructed below:</p>
  
  <pre><code>&gt;&gt;&gt; a1 = [{"a": 1, "b": "[{\"h\": 3, \"i\": 5} ,{\"h\": 4, \"i\": 6}]" },
  ...       {"a": 1, "b": "[{\"h\": 6, \"i\": 10},{\"h\": 8, \"i\": 12}]"}]
  &gt;&gt;&gt; df1 = sqlContext.read.json(sc.parallelize(a1))
  &gt;&gt;&gt; df1.show()
  +---+--------------------+
  |  a|                   b|
  +---+--------------------+
  |  1|[{"h": 3, "i": 5}...|
  |  1|[{"h": 6, "i": 10...|
  +---+--------------------+
  &gt;&gt;&gt; df1.printSchema()
  root
   |-- a: long (nullable = true)
   |-- b: string (nullable = true)
  </code></pre>
  
  <p>Note that the json code is <strong>StringType</strong>. I want to write a function that creates are new column which stores the data as a nested table, like below:</p>
  
  <pre><code>root
   |-- a: long (nullable = true)
   |-- b: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- h: long (nullable = true)
   |    |    |-- i: long (nullable = true)
  </code></pre>
  
  <p>I am using 1.6 therefore I don't have to_json cast function. I have tried to do this</p>
  
  <pre><code>&gt;&gt;&gt; df1.withColumn('new', get_json_object(df1.b,'$')).show()
  +---+--------------------+--------------------+
  |  a|                   b|                 new|
  +---+--------------------+--------------------+
  |  1|[{"h": 3, "i": 5}...|[{"h":3,"i":5},{"...|
  |  1|[{"h": 6, "i": 10...|[{"h":6,"i":10},{...|
  +---+--------------------+--------------------+
  </code></pre>
  
  <p>The issue is the new column created is still a string. :(</p>