title: Taking an array within a JSON file and exploding it into rows using pyspark
body: <p>I have some JSON that is in the following format:</p>
  <pre><code>{&quot;items&quot;: [&quot;234&quot;, &quot;454&quot;, &quot;434&quot;, &quot;534&quot;], &quot;time&quot;: &quot;1574290618029&quot;, &quot;id&quot;: &quot;A1&quot;, &quot;user&quot;: &quot;Bob&quot;}
  {&quot;items&quot;: [&quot;432&quot;, &quot;123&quot;, &quot;765&quot;], &quot;time&quot;: &quot;1574200618021&quot;, &quot;id&quot;: &quot;B1&quot;, &quot;user&quot;: &quot;Tim&quot;}
  {&quot;items&quot;: [&quot;437&quot;], &quot;time&quot;: &quot;1274600618121&quot;, &quot;id&quot;: &quot;B1&quot;, &quot;user&quot;: &quot;Joe&quot;}
  </code></pre>
  <p>Each JSON is brought into a dataframe using</p>
  <pre><code>spark.read.json(path)
  </code></pre>
  <p>and looping through to union them all into a single dataframe.</p>
  <pre><code>df.show()
  </code></pre>
  <p>shows something like this:</p>
  <pre><code>|items| id| time| user|
  |[&quot;234&quot;, &quot;454&quot;, &quot;434&quot;, &quot;534&quot;] | &quot;1574290618029&quot; | &quot;A1&quot; | &quot;Bob&quot;|
  |[&quot;432&quot;, &quot;123&quot;, &quot;765&quot;] | &quot;1574200618021&quot; | &quot;B1&quot; | &quot;Tim&quot;|
  |[&quot;437&quot;] | &quot;1274600618121&quot; | &quot;B1&quot; | &quot;Joe&quot;|
  </code></pre>
  <p>Doing a df.explode(df.id, df.time, df.user, explode(df.items)).show() results in something very close, but not quite what I'm looking for.</p>
  <pre><code>|id| time| user| col|
  |A1| 1574290618029| Bob| 234|
  |A1| 1574290618029| Bob| 454|
  |A1| 1574290618029| Bob| 434|
  |A1| 1574290618029| Bob| 534|
  |B1| 1574200618021| Tim| 432|
  |B1| 1574200618021| Tim| 123|
  |B1| 1574200618021| Tim| 765|
  |B1| 1274600618121| Joe| 437|
  </code></pre>
  <p>What I am actually needing is the data to be in a format like this:</p>
  <pre><code>|id| time| user| item_num| col|
  |A1| 1574290618029| Bob| item1| 234|
  |A1| 1574290618029| Bob| item2| 454|
  |A1| 1574290618029| Bob| item3| 434|
  |A1| 1574290618029| Bob| item4| 534|
  |B1| 1574200618021| Tim| item1| 432|
  |B1| 1574200618021| Tim| item2| 123|
  |B1| 1574200618021| Tim| item3| 765|
  |B1| 1574200618021| Tim| item4| NA|
  |B1| 1274600618121| Joe| item1| 437|
  |B1| 1274600618121| Joe| item2| NA|
  |B1| 1274600618121| Joe| item3| NA|
  |B1| 1274600618121| Joe| item4| NA|
  </code></pre>
  <p>Is there a simple way to accomplish this utilizing explode that I'm not seeing? I'm also very new to spark coding so please excuse me if there is some very obvious answer...</p>