title: String Concatenation issue in Spark SQL when using rtrim()
body: <p>I am facing a peculiar or unknown concatenation problem during PySpark SQL query</p>
  
  
  
  <pre class="lang-py prettyprint-override"><code>spark.sql("select *,rtrim(IncomeCat)+' '+IncomeCatDesc as trimcat from Dim_CMIncomeCat_handled").show() 
  </code></pre>
  
  <p>In this query both IncomeCat and IncomeCatDesc fields hold String type value so logically i thought it would concatenate but i get resultant field null</p>
  
  <p><a href="https://i.stack.imgur.com/u2TSv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/u2TSv.png" alt="enter image description here"></a></p>
  
  <p>where the achievable result will be '14100abcd' where 14100 is IncomeCat part and abcd is IncomeCatdesc part . i have tried explicit casting as well on IncomeCat field</p>
  
  <pre class="lang-py prettyprint-override"><code>spark.sql("select *,cast(rtrim(IncomeCat) as string)+' '+IncomeCatDesc as IncomeCatAndDesc from Dim_CMIncomeCat_handled").show() 
  </code></pre>
  
  <p>but I am getting same result. so  am i something missing here. kindly help me to solve this</p>