title: json array split to multiple arrays
body: <p>I am reading the contents of an api ('https://myapi.com/allcolor') into a dataframe using the pyspark code below in azure databricks notebook. When I read the json content into dataframe somehow the json payload with one <strong>&quot;colors&quot;</strong> array is getting translated to multiple arrays.</p>
  <p><strong>Code:</strong></p>
  <pre><code>spark.sql(&quot;set spart.databricks.delta.preview.enabled=true&quot;)
  spark.sql(&quot;set spart.databricks.delta.retentionDutationCheck.preview.enabled=false&quot;)
  import json
  import requests
  from requests.auth import HTTPDigestAuth
  import pandas as pd
  user = &quot;username&quot;
  password = &quot;password&quot;
  myResponse = requests.get('https://myapi.com/allcolor', auth=(user, password))
  if(myResponse.ok):
    jData = json.loads(myResponse.content)
    s1 = json.dumps(jData)
    #load data from api
    x = json.loads(s1)
    data = pd.read_json(json.dumps(x))
    #create dataframe
    spark_df = spark.createDataFrame(data)
    spark_df.show()          
    spark.conf.set(&quot;fs.azure.account.key.&lt;your-storage-account-name&gt;.blob.core.windows.net&quot;,&quot;&lt;your-storage-account-access-key&gt;&quot;)
    spark_df.write.mode(&quot;overwrite&quot;).json(&quot;wasbs://&lt;container&gt;@&lt;storage-account-name&gt;.blob.core.windows.net/&lt;directory&gt;/&quot;)
  else:
    myResponse.raise_for_status()
  </code></pre>
  <p><strong>Source json payload:</strong></p>
  <pre><code>{
    &quot;colors&quot;: [
      {
        &quot;color&quot;: &quot;black&quot;,
        &quot;category&quot;: &quot;hue&quot;,
        &quot;type&quot;: &quot;primary&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            255,
            255,
            255,
            1
          ],
          &quot;hex&quot;: &quot;#000&quot;
        }
      },
      {
        &quot;color&quot;: &quot;white&quot;,
        &quot;category&quot;: &quot;value&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            0,
            0,
            0,
            1
          ],
          &quot;hex&quot;: &quot;#FFF&quot;
        }
      },
      {
        &quot;color&quot;: &quot;red&quot;,
        &quot;category&quot;: &quot;hue&quot;,
        &quot;type&quot;: &quot;primary&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            255,
            0,
            0,
            1
          ],
          &quot;hex&quot;: &quot;#FF0&quot;
        }
      },
      {
        &quot;color&quot;: &quot;blue&quot;,
        &quot;category&quot;: &quot;hue&quot;,
        &quot;type&quot;: &quot;primary&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            0,
            0,
            255,
            1
          ],
          &quot;hex&quot;: &quot;#00F&quot;
        }
      },
      {
        &quot;color&quot;: &quot;yellow&quot;,
        &quot;category&quot;: &quot;hue&quot;,
        &quot;type&quot;: &quot;primary&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            255,
            255,
            0,
            1
          ],
          &quot;hex&quot;: &quot;#FF0&quot;
        }
      },
      {
        &quot;color&quot;: &quot;green&quot;,
        &quot;category&quot;: &quot;hue&quot;,
        &quot;type&quot;: &quot;secondary&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            0,
            255,
            0,
            1
          ],
          &quot;hex&quot;: &quot;#0F0&quot;
        }
      }
    ]
  }
  </code></pre>
  <p><strong>Resultant dataframe:</strong></p>
  <pre><code>{
    &quot;colors&quot;: 
      {
        &quot;color&quot;: &quot;black&quot;,
        &quot;category&quot;: &quot;hue&quot;,
        &quot;type&quot;: &quot;primary&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            255,
            255,
            255,
            1
          ],
          &quot;hex&quot;: &quot;#000&quot;
        }
      }
      }
  {
    &quot;colors&quot;:     
      {
        &quot;color&quot;: &quot;white&quot;,
        &quot;category&quot;: &quot;value&quot;,
        &quot;code&quot;: {
          &quot;rgba&quot;: [
            0,
            0,
            0,
            1
          ],
          &quot;hex&quot;: &quot;#FFF&quot;
        }
      }
      }
  </code></pre>