title: Get value from Spark DenseVectors in DataFrame column into a new DataFrame column
body: <p>How can I get a specific value out of a Spark DenseVector that is stored in a DataFrame column into a new column in the same DataFrame without using a python user defined function (udf)?  </p>
  
  <p>More generally, how can I perform operations on vectors stored in a DataFrame column and put the results in a new column in the same DataFrame</p>
  
  <p>The following should be reproducible. </p>
  
  <pre><code>spark = pyspark.sql.SparkSession.builder.getOrCreate()
  from pyspark.mllib.linalg import DenseVector
  import pyspark.sql.types as T
  
  testdf = spark.createDataFrame([\
              (DenseVector([2, 3]),),\
              (DenseVector([4, 5]),),\
              (DenseVector([6, 7]),)],\
              ['DenseVectors'])
  </code></pre>
  
  <p>These work for single extractions.</p>
  
  <pre><code>testdf.collect()[0][0][1]
  3.0
  
  testdf.collect()[0][0].dot(DenseVector([0, 1]))   
  3.0
  </code></pre>
  
  <p>But I cannot get those to work to create a new column.</p>
  
  <pre><code>testdf \
    .withColumn('test', testdf.DenseVectors[0][0][1]) \
  
  &gt; AnalysisException: u"Can't extract value from DenseVectors#211: need struct type but got vector;"
  
  
  testdf \
    .withColumn('test', testdf.DenseVectors.dot(DenseVector([0, 1]))) \
  
  &gt; TypeError: 'Column' object is not callable
  </code></pre>