title: Pyspark Dataframe Apply function to two columns
body: <p>Say I have two PySpark DataFrames <code>df1</code> and <code>df2</code>. </p>
  
  <pre><code>df1=   'a' 
          1    
          2    
          5    
  
  df2=   'b'
          3
          6
  </code></pre>
  
  <p>And I want to find the closest <code>df2['b']</code> value for each <code>df1['a']</code>, and add the closest values as a new column in <code>df1</code>.</p>
  
  <p>In other words, for each value <code>x</code> in <code>df1['a']</code>, I want to find a <code>y</code> that achieves <code>min(abx(x-y))</code> for all <code>y in df2['b']</code>(note: can assume that there is only one <code>y</code> that can achieve the minimum distance), and the result would be</p>
  
  <pre><code>'a'    'b'
   1      3
   2      3
   5      6
  </code></pre>
  
  <p>I tried the following code to create a distance matrix first (before finding the values achieving the minimum distance):</p>
  
  <pre><code>from pyspark.sql.types import IntegerType
  from pyspark.sql.functions import udf
  
  def dict(x,y):
      return abs(x-y)
  udf_dict = udf(dict, IntegerType())
  
  sql_sc = SQLContext(sc)
  udf_dict(df1.a, df2.b)
  </code></pre>
  
  <p>which gives </p>
  
  <pre><code>Column&lt;PythonUDF#dist(a,b)&gt;
  </code></pre>
  
  <p>Then I tried </p>
  
  <pre><code>sql_sc.CreateDataFrame(udf_dict(df1.a, df2.b))
  </code></pre>
  
  <p>which runs forever without giving error/output.</p>
  
  <p>My questions are:</p>
  
  <ol>
  <li>As I'm new to Spark, is my way to construct the output DataFrame efficient? (My way would be creating a distance matrix for all the <code>a</code> and <code>b</code> values first, and then find the <code>min</code> one)</li>
  <li>What's wrong with the last line of my code and how to fix it?</li>
  </ol>