title: How to calculate euclidean distance of each row in a dataframe to a constant reference array
body: <p>I have a dataframe which is created from parquet files that has 512 columns(all float values).</p>
  <p>I am trying to calculate euclidean distance of each row in my dataframe to a constant reference array.</p>
  <p>My development environment is Zeppelin 0.7.3 with spark 2.1 and Scala. Here is the zeppelin paragraphs I run:</p>
  <pre><code>import org.apache.spark.ml.feature.VectorAssembler
  import org.apache.spark.ml.linalg.Vectors
  
  //Create dataframe from parquet file
  val filePath = &quot;/tmp/vector.parquet/*.parquet&quot; 
  val df = spark.read.parquet(filePath)
  
  //Create assembler and vectorize df
  val assembler = new VectorAssembler()
    .setInputCols(df.columns)
    .setOutputCol(&quot;features&quot;)
  val training = assembler.transform(df)
  
  //Create udf
  val eucDisUdf = udf((features: Vector, 
  myvec:Vector)=&gt;Vectors.sqdist(features, myvec))
  
  //Cretae ref vector
  val myScalaVec = Vectors.dense( Array.fill(512)(25.44859))
  val distDF = 
  training2.withColumn(&quot;euc&quot;,eucDisUdf($&quot;features&quot;,myScalaVec))
  </code></pre>
  <p>This code gives the following error for eucDisUdf call:</p>
  <pre><code>error: type mismatch;  found   : org.apache.spark.ml.linalg.Vector 
  required: org.apache.spark.sql.Column
  </code></pre>
  <p>I appreciate any idea how to eliminate this error and compute distances properly in scala.</p>