title: PySpark Nested Dataframe
body: <p>I need some Python Pyspark help. I have a source object that looks like:
  Edit: Both lists are numpy arrays of numpy data types</p>
  <pre><code>obj = [(&quot;thing1&quot;, ([1,2,3], [0.1,0.2,0.3]),
         (&quot;thing2&quot;, ([1,2,3], [0.1,0.2,0.3]),
         (&quot;thing3&quot;, ([1,2,3], [0.1,0.2,0.3]),
         (&quot;thing4&quot;, ([1,2,3], [0.1,0.2,0.3]),
         ...]
  </code></pre>
  <p>Is there a way to create spark dataframe so it looks like this. Where the object in &quot;column2&quot; is essentially another dataframe of two columns labeled &quot;label1&quot; and &quot;label2&quot;:</p>
  <pre><code>&quot;column1&quot; | &quot;column2&quot;
  ---------------------
  &quot;thing1&quot;  | [{&quot;label1&quot;: 1, &quot;label2&quot;: 0.1}, {&quot;label1&quot;: 2, &quot;label2&quot;: 0.2}, {&quot;label1&quot;: 3, &quot;label2&quot;: 0.3}]
  &quot;thing2&quot;  | [{&quot;label1&quot;: 1, &quot;label2&quot;: 0.1}, {&quot;label1&quot;: 2, &quot;label2&quot;: 0.2}}]
  ...
  </code></pre>
  <p>My end goal is to produce a JSON object that looks like this. I need the columns and labels:</p>
  <pre><code>{{&quot;column1&quot;:&quot;thing1&quot;, 
    &quot;column2&quot;:[{&quot;1abel1&quot;:1, &quot;label2&quot;:0.1},{&quot;1abel1&quot;:2, &quot;label2&quot;:0.2},{&quot;1abel1&quot;:3, &quot;label2&quot;:0.3},
   {&quot;column1&quot;:&quot;thing2&quot;, 
    &quot;column2&quot;:[{&quot;1abel1&quot;:1, &quot;label2&quot;:0.1},{&quot;1abel1&quot;:2, &quot;label2&quot;:0.2},{&quot;1abel1&quot;:3, &quot;label2&quot;:0.3},
   {&quot;column1&quot;:&quot;thing3&quot;, 
    &quot;column2&quot;:[{&quot;1abel1&quot;:1, &quot;label2&quot;:0.1},{&quot;1abel1&quot;:2, &quot;label2&quot;:0.2},{&quot;1abel1&quot;:3, &quot;label2&quot;:0.3},
   ...}
  </code></pre>
  <p>If this is possible to do relatively fast for about 1 million records, that would be great too</p>