title: Spark array struct and UDFs
body: <p>I have the following schema and I want to add a new column called distance. This colum compute the distance between the tow time series of each row: time_series1 and time_series2   </p>
  
  <pre><code>|-- websites: struct (nullable = true)
  |    |-- _1: integer (nullable = false)
  |    |-- _2: integer (nullable = false)
  |-- countryId1: integer (nullable = false)
  |-- countryId2: integer (nullable = false)
  |-- time_series1: array (nullable = true)
  |    |-- element: struct (containsNull = true)
  |    |    |-- _1: float (nullable = false)
  |    |    |-- _2: date (nullable = true)
  |-- time_series2: array (nullable = true)
  |    |-- element: struct (containsNull = true)
  |    |    |-- _1: float (nullable = false)
  |    |    |-- _2: date (nullable = true)
  </code></pre>
  
  <p>So I use the udf function to define this new column as:</p>
  
  <pre><code>val step2= step1
    .withColumn("distance",  distanceUDF(col("time_series1"),col("time_series2")))
    .select("websites","countryId1","countryId2","time_series1","time_series2","distance")
  </code></pre>
  
  <p>and the UDF :</p>
  
  <pre><code> val distanceUDF  = udf( (ts1:Seq[(Float,_)], ts2:Seq[(Float,_)])=&gt;
                              compute_distance( ts1.map(_._1) , ts2.map(_._1)))
  </code></pre>
  
  <p>but I have the problem on the mapping, I dont  know how to map the array (struct (float,date).to the scala.</p>
  
  <p>Is <code>Seq[(Float,Date)]</code> equivalent to array( struct (float,date)) ? 
  I have the following exeption: </p>
  
  <pre><code>java.lang.ClassCastException: .GenericRowWithSchema cannot be cast to scala.Tuple2
  </code></pre>
  
  <p>My problem is different thant the one exposed here <a href="https://stackoverflow.com/questions/38413040/spark-sql-udf-with-complex-input-parameter">Spark Sql UDF with complex input parameter</a> . I have an ordered time series withe date (i have an array and not only a struct type )</p>