title: Passing struct type to methods or UDFS in spark sql dataframes
body: <p>I have two dataframes and I have joined them and after joining in the joined dataframe , i have got two columns which are of type struct. Basically they are of Array[[String,Int]]. I need to derive a third column based on the elements of this struct type.</p>
  
  <p>My code looks like below.</p>
  
  <pre><code>val bdf = Seq(
   ("a",1,1,10)
  ,("a",1,2,10)
  ,("a",1,3,10)
  ,("a",1,4,10)
  ,("b",1,1,20)
  ,("b",1,2,10)
  ,("a",2,3,10)
  ,("a",2,4,20)
  ,("a",2,5,20)
  ,("c",2,1,10)
  ,("c",2,2,20)
  ,("c",2,3,20)
  ).toDF("contract_number","linenumber","monthdel","open_quant")
  
  val gbdf = bdf.withColumn("bmergedcol",struct(bdf("monthdel"),bdf("open_quant"))).groupBy("contract_number","linenumber").agg(collect_list("bmergedcol"))
  
  
  
  
  
  val pl = Seq(
  ("a",1,"FLAT",10)
  ,("a",1,"FLAT",30)
  ,("a",1,"NFE",10)
  ,("b",1,"FLAT",10)
  ,("b",1,"NFE",10)
  ,("c",2,"NFE",10)
  ,("a",3,"NFE",20)
  ,("c",2,"FLAT",20)).toDF("connum","linnum","type","qnt")
  
  import org.apache.spark.sql.functions._ 
  val gpl = pl.withColumn("mergedcol",struct(pl("type"),pl("qnt"))).groupBy("connum","linnum").agg(collect_list("mergedcol"))
  
  
  val jdf = gbdf.join(gpl,expr("((contract_number = connum) AND (linenumber = linnum ))"),"left_outer")
  </code></pre>
  
  <p>My output of jdf is like </p>
  
  <p><a href="https://i.stack.imgur.com/6rCuE.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6rCuE.png" alt="enter image description here"></a></p>
  
  <p>I need to understand how can i pass the two struct type fields to some method and derive a third one from it?</p>