title: UnFlatten Dataframe to a specific structure
body: <p>I have a flat dataframe (<code>df</code>) with the structure as below:</p>
  
  <pre><code>root
   |-- first_name: string (nullable = true)
   |-- middle_name: string (nullable = true)
   |-- last_name: string (nullable = true)
   |-- title: string (nullable = true)
   |-- start_date: string (nullable = true)
   |-- end_Date: string (nullable = true)
   |-- city: string (nullable = true)
   |-- zip_code: string (nullable = true)
   |-- state: string (nullable = true)
   |-- country: string (nullable = true)
   |-- email_name: string (nullable = true)
   |-- company: struct (nullable = true)
   |-- org_name: string (nullable = true)
   |-- company_phone: string (nullable = true)
   |-- partition_column: string (nullable = true)
  </code></pre>
  
  <p>And I need to convert this dataframe into a structure like (as my next data will be in this format):</p>
  
  <pre><code>root
   |-- firstName: string (nullable = true)
   |-- middleName: string (nullable = true)
   |-- lastName: string (nullable = true)
   |-- currentPosition: array (nullable = true)
   |    |-- element: struct (containsNull = true)
   |    |    |-- title: string (nullable = true)
   |    |    |-- startDate: string (nullable = true)
   |    |    |-- endDate: string (nullable = true)
   |    |    |-- address: struct (nullable = true)
   |    |    |    |-- city: string (nullable = true)
   |    |    |    |-- zipCode: string (nullable = true)
   |    |    |    |-- state: string (nullable = true)
   |    |    |    |-- country: string (nullable = true)
   |    |    |-- emailName: string (nullable = true)
   |    |    |-- company: struct (nullable = true)
   |    |    |    |-- orgName: string (nullable = true)
   |    |    |    |-- companyPhone: string (nullable = true)
   |-- partitionColumn: string (nullable = true)
  </code></pre>
  
  <p>So far I have implemented this:</p>
  
  <pre><code>case class IndividualCompany(orgName: String,
                               companyPhone: String)
  
  case class IndividualAddress(city: String,
                     zipCode: String,
                     state: String,
                     country: String)
  
  case class IndividualPosition(title: String,
                                startDate: String,
                                endDate: String,
                                address: IndividualAddress,
                                emailName: String,
                                company: IndividualCompany)
  
  case class Individual(firstName: String,
                       middleName: String,
                       lastName: String,
                       currentPosition: Seq[IndividualPosition],
                       partitionColumn: String)
  
  
  val makeCompany = udf((orgName: String, companyPhone: String) =&gt; IndividualCompany(orgName, companyPhone))
  val makeAddress = udf((city: String, zipCode: String, state: String, country: String) =&gt; IndividualAddress(city, zipCode, state, country))
  
  val makePosition = udf((title: String, startDate: String, endDate: String, address: IndividualAddress, emailName: String, company: IndividualCompany) 
                      =&gt; List(IndividualPosition(title, startDate, endDate, address, emailName, company)))
  
  
  val selectData = df.select(
        col("first_name").as("firstName"),
        col("middle_name).as("middleName"),
        col("last_name").as("lastName"),
        makePosition(col("job_title"),
          col("start_date"),
          col("end_Date"),
          makeAddress(col("city"),
            col("zip_code"),
            col("state"),
            col("country")),
          col("email_name"),
          makeCompany(col("org_name"),
            col("company_phone"))).as("currentPosition"),
        col("partition_column").as("partitionColumn")
      ).as[Individual]
  
  select_data.printSchema()
  select_data.show(10)
  </code></pre>
  
  <p>I can see a proper schema generated for <code>select_data</code>, but it gives an error on the last line where I am trying to get some actual data. I am getting an error saying failed to execute user defined function. </p>
  
  <pre><code> org.apache.spark.SparkException: Failed to execute user defined function(anonfun$4: (string, string, string, struct&lt;city:string,zipCode:string,state:string,country:string&gt;, string, struct&lt;orgName:string,companyPhone:string&gt;) =&gt; array&lt;struct&lt;title:string,startDate:string,endDate:string,address:struct&lt;city:string,zipCode:string,state:string,country:string&gt;,emailName:string,company:struct&lt;orgName:string,companyPhone:string&gt;&gt;&gt;)
  </code></pre>
  
  <p>Is there any better way to achieve this? </p>