title: pyspark sqlfunction expr function not working as expected?
body: <p>pyspark sqlfunction expr not working as expected.  </p>
  
  <p>my test1.txt contains</p>
  
  <pre><code>101|10|4
  101|12|1
  101|13|3
  101|14|2
  </code></pre>
  
  <p>my test2.txt contains</p>
  
  <pre><code>101|10|4
  101|11|1
  101|13|3
  101|14|2
  </code></pre>
  
  <p>I have created two dataframes using above data like below code. </p>
  
  <pre><code>df3 = spark.createDataFrame(sc.textFile("C://Users//cravi//Desktop//test1.txt").map( lambda x: x.split("|")[:3]),["cid","pid","pr"])
      df4 = spark.createDataFrame(sc.textFile("C://Users//cravi//Desktop//test2.txt").map( lambda x: x.split("|")[:3]),["cid","pid","p"])
      df5=df4.withColumnRenamed("p", "p")\
              .join(df3.withColumnRenamed("pr", "Pr")\
              , ["cid", "pid"], "outer")\
              .na.fill(0)
      tt=df5.withColumn('flag', sf.expr("case when p&gt;0 and pr=='null' then 'N'\
                                                          when p=0 and Pr&gt;0 then 'D'\
                                                          when p=Pr then 'R'\
                                                          else 'U' end"))
      tt.show()
  </code></pre>
  
  <p>I am getting  output like below</p>
  
  <pre><code>+---+---+----+----+----+
  |cid|pid|   p|  Pr|flag|
  +---+---+----+----+----+
  |101| 14|   2|   2|   R|
  |101| 10|   4|   4|   R|
  |101| 11|   1|null|   U|
  |101| 12|null|   1|   U|
  |101| 13|   3|   3|   R|
  +---+---+----+----+----+
  </code></pre>
  
  <p>pyspark sqlfunction expr not working as expected.  </p>
  
  <p>if <code>p</code> and <code>pr</code> is same then  my <code>falg</code> will be <code>'R'</code>.</p>
  
  <p>if <code>p</code> some value and <code>pr</code> is null my <code>flag</code> will be <code>'N'</code></p>
  
  <p>if <code>p</code> is null and <code>pr</code> is some value my flag is <code>'D'</code></p>
  
  <p>other case my flag is <code>'U'</code></p>
  
  <p>In this case expected output is  :</p>
  
  <pre><code>+---+---+----+----+----+
  |cid|pid|   p|  Pr|flag|
  +---+---+----+----+----+
  |101| 14|   2|   2|   R|
  |101| 10|   4|   4|   R|
  |101| 11|   1|null|   N|
  |101| 12|null|   1|   D|
  |101| 13|   3|   3|   R|
  +---+---+----+----+----+
  </code></pre>