title: Apply custom function to cells of selected columns of a data frame in PySpark
body: <p>Let's say I have a data frame which looks like this:</p>
  
  <pre><code>+---+-----------+-----------+
  | id|   address1|   address2|
  +---+-----------+-----------+
  |  1|address 1.1|address 1.2|
  |  2|address 2.1|address 2.2|
  +---+-----------+-----------+
  </code></pre>
  
  <p>I would like to apply a custom function directly to the strings in the <em>address1</em> and <em>address2</em> columns, for example:</p>
  
  <pre><code>def example(string1, string2):
      name_1 = string1.lower().split(' ')
      name_2 = string2.lower().split(' ')
      intersection_count = len(set(name_1) &amp; set(name_2))
  
      return intersection_count
  </code></pre>
  
  <p>I want to store the result in a new column, so that my final data frame would look like:</p>
  
  <pre><code>+---+-----------+-----------+------+
  | id|   address1|   address2|result|
  +---+-----------+-----------+------+
  |  1|address 1.1|address 1.2|     2|
  |  2|address 2.1|address 2.2|     7|
  +---+-----------+-----------+------+
  </code></pre>
  
  <p>I've tried to execute it in a way I once applied a built-in function to the whole column, but I got an error:</p>
  
  <pre><code>&gt;&gt;&gt; df.withColumn('result', example(df.address1, df.address2))
  Traceback (most recent call last):
    File "&lt;stdin&gt;", line 1, in &lt;module&gt;
    File "&lt;stdin&gt;", line 2, in example
  TypeError: 'Column' object is not callable
  </code></pre>
  
  <p>What am I doing wrong and how I can apply a custom function to strings in selected columns?</p>