title: pyspark: user-defined function for determining NaN or Null is not working
body: <p>I am trying to write a user defined function in pyspark that determines whether a given entry in a dataframe is bad(Null or NaN).  I can't seem to figure out what I am doing wrong in this function:</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql.functions import UserDefinedFunction
  from pyspark.sql.types import *
  
  def is_bad(value):
     if (value != value | (value.isNull())):
         return True
     else:
         return False
  
  isBadEntry = UserDefinedFunction(lambda x: is_bad(x),BooleanType())
  
  df_test = sql.createDataFrame([(1,1,None ), (1,2, 5), (1,3, None), (1,4, None), (1,5, 10), (1,6,None )], ('session',"timestamp", "id"))
  df_test =df_test.withColumn("testing", isBadEntry(df_test.id)).show()
  </code></pre>
  
  <p>This is crashing with a cryptic error:</p>
  
  <pre><code>Py4JJavaErrorTraceback (most recent call last)
  &lt;ipython-input-379-b4109047ba40&gt; in &lt;module&gt;()
    1 df_test = sql.createDataFrame([(1,1,None ), (1,2, 5), (1,3, None), (1,4, None), (1,5, 10), (1,6,None )], ('session',"timestamp", "id"))
    2 #df_test.show()
  ----&gt; 3 df_test =df_test.withColumn("testing", isBadEntry(df_test.id)).show()
  
  /usr/local/spark/python/pyspark/sql/dataframe.py in show(self, n, truncate)
  285         +---+-----+
  286         """
  --&gt; 287         print(self._jdf.showString(n, truncate))
  288 
  289     def __repr__(self):
  
  /usr/local/spark/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py in __call__(self, *args)
     1131         answer = self.gateway_client.send_command(command)
     1132         return_value = get_return_value(
  -&gt; 1133             answer, self.gateway_client, self.target_id, self.name)
     1134 
     1135         for temp_arg in temp_args:
  
   /usr/local/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
   61     def deco(*a, **kw):
   62         try:
  ---&gt; 63             return f(*a, **kw)
   64         except py4j.protocol.Py4JJavaError as e:
   65             s = e.java_exception.toString()
  
  /usr/local/spark/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
  317                 raise Py4JJavaError(
  318                     "An error occurred while calling {0}{1}{2}.\n".
  --&gt; 319                     format(target_id, ".", name), value)
  320             else:
  321                 raise Py4JError(
  
  Py4JJavaError: An error occurred while calling o29884.showString.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 43.0 failed 4 times, most recent failure: Lost task 0.3 in stage 43.0 (TID 167, 172.16.193.79): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 172, in main
  process()
    File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 167, in process
  serializer.dump_stream(func(split_index, iterator), outfile)
    File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 106, in &lt;lambda&gt;
   func = lambda _, it: map(mapper, it)
    File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 92, in &lt;lambda&gt;
  mapper = lambda a: udf(*a)
    File "/spark/python/lib/pyspark.zip/pyspark/worker.py", line 70, in &lt;lambda&gt;
  return lambda *a: f(*a)
   File "&lt;ipython-input-378-2aac40340a6c&gt;", line 14, in &lt;lambda&gt;
   File "&lt;ipython-input-378-2aac40340a6c&gt;", line 9, in is_bad
  AttributeError: 'NoneType' object has no attribute 'isNull'
  </code></pre>
  
  <p>Could someone please help?</p>