title: AttributeError: 'NoneType' object has no attribute '_jvm' when passing sql function as a default parameter
body: <p>I'm aware that this same error has been asked about before, but in <a href="https://stackoverflow.com/questions/40297403/pyspark-error-attributeerror-nonetype-object-has-no-attribute-jvm">this example</a> and <a href="https://stackoverflow.com/questions/47552309/withcolumn-with-udf-yields-attributeerror-nonetype-object-has-no-attribute?rq=1">this other example</a>, the errors are caused by using <code>pyspark.sql</code> functions in a <code>udf</code>. This is not what I am doing.</p>
  
  <p>The offending code is the following function definition (if I remove the default parameter my code runs and passes all tests). </p>
  
  <pre><code>from pyspark.sql import functions as F
  
  def apply_filter(df, group=F.lit(True)):
      filtered_df = df.filter(group)
  </code></pre>
  
  <p>I am mainly looking for the reason that this code produces the same error that is found in the other examples. </p>
  
  <p>Edit: </p>
  
  <p>I cannot share the original code due to work, but if you run the previous code with <code>spark-submit --deploy-mode cluster &lt;filename&gt;</code> the following error is produced. </p>
  
  <pre><code>LogType:stdout
  Log Upload Time:Fri Mar 09 16:01:45 +0000 2018
  LogLength:343
  Log Contents:
  Traceback (most recent call last):
    File "temp.py", line 3, in &lt;module&gt;
      def apply_filter(df, group=F.lit(True)):
    File "/mnt/yarn/usercache/hadoop/appcache/application_1520603520946_0005/container_1520603520946_0005_01_000001/pyspark.zip/pyspark/sql/functions.py", line 40, in _
  AttributeError: 'NoneType' object has no attribute '_jvm'
  End of LogType:stdout
  </code></pre>
  
  <p>Interestingly enough, the error doesn't persist if it is ran locally. </p>