title: How can I use a function in dataframe withColumn function in Pyspark?
body: <p>I have the some dictionaries and a function defined:</p>
  
  <pre><code>dict_TEMPERATURE = {(0, 70): 'Low', (70.01, 73.99): 'Normal-Low',(74, 76): 'Normal', (76.01, 80): 'Normal-High', (80.01, 300): 'High'}
  ...
  hierarchy_dict = {'TEMP': dict_TEMPERATURE, 'PRESS': dict_PRESSURE, 'SH_SP': dict_SHAFT_SPEED, 'POI': dict_POI, 'TRIG': dict_TRIGGER}
  
  
  
  def function_definition(valor, atributo):
  
      dict_atributo = hierarchy_dict[atributo]
      valor_generalizado = None
  
      if isinstance(valor, (int, long, float, complex)):
  
          for key, value in dict_atributo.items():
  
              if(isinstance(key, tuple)):
                  lista = list(key)
  
                  if (valor &gt; key[0] and valor &lt; key[1]):
                      valor_generalizado = value
  
      else: # if it is not numeric
          valor_generalizado = dict_atributo.get(valor)
  
  
      return valor_generalizado
  </code></pre>
  
  <p>What this function basically do is: check the value which is passed as an argument to the "function_definition" function, and replace its value according to its dictionary's references.</p>
  
  <p>So, if I call "function_definition(60, 'TEMP')" it will return 'LOW'.</p>
  
  <p>On the other hand, I have a dataframe with the next structure (this is an example):</p>
  
  <pre><code>+----+-----+-----+---+----+
  |TEMP|SH_SP|PRESS|POI|TRIG|
  +----+-----+-----+---+----+
  |   0|    1|    2|  0|   0|
  |   0|    2|    3|  1|   1|
  |   0|    3|    4|  2|   1|
  |   0|    4|    5|  3|   1|
  |   0|    5|    6|  4|   1|
  |   0|    1|    2|  5|   1|
  +----+-----+-----+---+----+
  </code></pre>
  
  <p>What I want to do is to replace the values of one column of the dataframe based on the function defined above, so I have the next code-line:</p>
  
  <pre><code>dataframe_new = dataframe.withColumn(atribute_name, function_definition(dataframe[atribute_name], atribute_name))
  </code></pre>
  
  <p>But I get the next error message when executing it:</p>
  
  <pre><code>AssertionError: col should be Column
  </code></pre>
  
  <p>What is wrong in my code? How could do that?</p>