title: How do I use a from_json() dataframe in Spark?
body: <p>I'm trying to create a dataset from a json-string within a dataframe in Databricks 3.5 (Spark 2.2.1). In the code block below 'jsonSchema' is a StructType with the correct layout for the json-string which is in the 'body' column of the dataframe.</p>
  
  <pre><code>val newDF = oldDF.select(from_json($"body".cast("string"), jsonSchema))
  </code></pre>
  
  <p>This returns a dataframe where the root object is</p>
  
  <pre><code>jsontostructs(CAST(body AS STRING)):struct
  </code></pre>
  
  <p>followed by the fields in the schema (looks correct). When I try another select on the newDF</p>
  
  <pre><code>val transform = newDF.select($"propertyNameInTheParsedJsonObject")
  </code></pre>
  
  <p>it throws the exception</p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: cannot resolve '`columnName`' given 
  input columns: [jsontostructs(CAST(body AS STRING))];;
  </code></pre>
  
  <p>I'm aparently missing something. I hoped from_json would return a dataframe I could manipulate further.</p>
  
  <p>My ultimate objective is to cast the json-string within the oldDF body-column to a dataset.</p>