title: Transform a collect_list column to other data type column using UDF in spark streaming
body: <p>The request is to union data from 2 streaming sources, and then reduce on the same key, and apply a function for the values to transform it to another UDO (user defined object type). I have no idea how to do it. So here I just created a similar problem as below:</p>
  
  <p>The 2 input streams have 3 columns, id, value and posttime. We first union it, and then reduce based on id and produce the final result with values (id, user-defined-function(values, posttimes)). How to achieve this?</p>
  
  <p>If I implement it as follows:</p>
  
  <pre><code>val dff = df.union(df2)
    .withWatermark("posttime", "15 minutes")
    .groupBy(window($"posttime", "10 minutes", "5 minutes"),$"id")
    .agg(collect_list(struct("value", "posttime")).as("data"))
    .withColumn("data", user-defined-function("data"))
  </code></pre>
  
  <p>How should I define the user-defined-function to input the column and output a column with different data type?</p>
  
  <pre><code>// How to tranform the Column (Type: A) to output Column (Type: B)
  def user-defined-function(columnName: String): Column = {
    val x = Column(columnName).cast(List&lt;struct&gt;)
    val ptime = if(x.posttime &lt; y.posttime) x.posttime else y.posttime
    val value = (x.value.toInt + y.value.toInt).toString
    return new Column(struct(value, ptime))
  }
  </code></pre>