title: How to explode get_json_object in Apache Spark
body: <p>I have the following string in one of my dataframe's column:</p>
  
  <pre><code>row1:[{"key":"foo"},{"key":"bar"},{"key":"baz"}]
  row2:[{"key":"foo"},{"key":"bar"}]
  row3:null
  etc
  </code></pre>
  
  <p>I found that Spark has "get_json_object" function. So if I want to use xpath to extract data I would use:</p>
  
  <pre><code> get_json_object($"json", s"$[0].key")
  </code></pre>
  
  <p>would returns:</p>
  
  <pre><code>"foo"
  "foo"
  null
  </code></pre>
  
  <p>but I need the equivalent of "explosion" function of Spark. </p>
  
  <p>I found that I can use the "*" symbol on my xpath.</p>
  
  <pre><code> get_json_object($"json", s"$[*].key")
  </code></pre>
  
  <p>Which don't do the as expected, it will create a string like:</p>
  
  <pre><code>[foo,bar,baz]
  [foo,baz]
  </code></pre>
  
  <p>I found the solution in another stackoverflow thread, </p>
  
  <pre><code>val jsonElements = (0 until 3).map(i =&gt; get_json_object($"json", s"$$[$i].key"))
  
  
  val jsonElements = .map(i =&gt; get_json_object($"json", s"$$[$i].key"))
  df.select($"id",explode(array(jsonElements: _*).alias("foo")))
  </code></pre>
  
  <p>This partially solve my issue, because this solution presumes that I know how maximum depth my array can be. The function "from_json" of Spark needs schema, I have huge complex JSON type would take an "infinity" amount of time to create the schema.</p>
  
  <p><strong>Disclaimer</strong></p>
  
  <p>I will not use any regular expression/substring/etc to parse JSON. The entire propose of use a parser is that.</p>