title: pyspark replace negative value to zero
body: <p>I may want to asking help about replacing the negative value from different between the timestamp to zero. Running on python3 on spark. Here is my code:</p>
  
  <p>code:</p>
  
  <pre><code>timeFmt = "yyyy-MM-dd HH:mm:ss"
  time_diff_1 = when((col("time1").isNotNull()) &amp;
                         (col("time2").isNotNull()),                      
                         (unix_timestamp('time2', format=timeFmt) - unix_timestamp('time1', format=timeFmt)) / 60
                        ).otherwise(lit(0))
  
  time_diff_2 = when((col("time2").isNotNull()) &amp; 
                         (col("time3").isNotNull()),                       
                         (unix_timestamp('time3', format=timeFmt) - unix_timestamp('time2', format=timeFmt)) / 60
                        ).otherwise(lit(0))
  
  time_diff_3 = when((col("time3").isNotNull()) &amp;                           
                         (col("time4").isNotNull()),                       
                         (unix_timestamp('time4', format=timeFmt) - unix_timestamp('time3', format=timeFmt)) / 60
                        ).otherwise(lit(0))
  
  df = (df      
        .withColumn('time_diff_1', time_diff_1)      
        .withColumn('time_diff_2', time_diff_2)
        .withColumn('time_diff_3', time_diff_3)
       )
  
  
  df = (df
        .withColumn('time_diff_1', when(col('time_diff_1') &lt; 0, 0).otherwise(col('time_diff_1')))
        .withColumn('time_diff_2', when(col('time_diff_2') &lt; 0, 0).otherwise(col('time_diff_2')))
        .withColumn('time_diff_3', when(col('time_diff_3') &lt; 0, 0).otherwise(col('time_diff_3')))
       )
  </code></pre>
  
  <p>when I run the above code, I get error.
  Here is the error:</p>
  
  <blockquote>
    <p>Py4JJavaError: An error occurred while calling o1083.showString. :
    org.apache.spark.SparkException: Job aborted due to stage failure:
    Task 0 in stage 56.0 failed 4 times, most recent failure: Lost task
    0.3 in stage 56.0 (TID 7246, fxhclxcdh8.dftz.local, executor 21): org.codehaus.janino.JaninoRuntimeException: failed to compile:
    org.codehaus.janino.JaninoRuntimeException: Code of method
    "apply_9$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificUnsafeProjection;Lorg/apache/spark/sql/catalyst/InternalRow;)V"
    of class
    "org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection"
    grows beyond 64 KB /* 001 <em>/ public java.lang.Object generate(Object[]
    references) { /</em> 002 <em>/   return new
    SpecificUnsafeProjection(references); /</em> 003 <em>/ } /</em> 004 <em>/ /</em> 005 <em>/
    class SpecificUnsafeProjection extends
    org.apache.spark.sql.catalyst.expressions.UnsafeProjection { /</em> 006 <em>/
    /</em> 007 <em>/   private Object[] references; /</em> 008 <em>/   private boolean
    evalExprIsNull; /</em> 009 <em>/   private boolean evalExprValue; /</em> 010 <em>/<br>
    private boolean evalExpr1IsNull; /</em> 011 <em>/   private boolean
    evalExpr1Value; /</em> 012 <em>/   private java.text.DateFormat formatter5;
    /</em> 013 <em>/   private java.text.DateFormat formatter8; /</em> 014 <em>/<br>
    private java.text.DateFormat formatter12; /</em> 015 <em>/   private
    java.text.DateFormat formatter13; /</em> 016 <em>/   private
    UTF8String.IntWrapper wrapper; /</em> 017 <em>/   private
    java.text.DateFormat formatter15; /</em> 018 <em>/   private
    java.text.DateFormat formatter18; /</em> 019 <em>/   private
    java.text.DateFormat formatter19; /</em> 020 <em>/   private
    java.text.DateFormat formatter23; /</em> 021 <em>/   private
    java.text.DateFormat formatter26; /</em> 022 <em>/   private
    java.text.DateFormat formatter27; /</em> 023 <em>/   private
    java.text.DateFormat formatter30; /</em> 024 */   private
    java.text.DateFormat formatter32; ........</p>
  </blockquote>
  
  <p>Anybody can help?</p>