title: Conditional Transform on PySpark Array Column with Higher Order Functions
body: <p>I have a PySpark DF with an array column with data such as: [0,-1,0,0,1,1,1]</p>
  <p>I'd like to use higher order PySpark functions to convert all negative values to 0, for a result like: [0,0,0,0,1,1,1]</p>
  <p>I've tried:</p>
  <pre><code>sdf = (sdf
         .withColumn('ArrayCol',f.transform(f.col('ArrayCol'),lambda x: 0 if x &lt; 0 else x))
  )
  </code></pre>
  <p>but returns error &quot;Cannot convert column into bool: please use '&amp;' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.&quot;</p>
  <p>as well as:</p>
  <pre><code>sdf = (sdf
         .withColumn('ArrayCol',f.expr(&quot;transform(ArrayCol, x -&gt; CASE WHEN x &lt; 0 THEN 0 ELSE x &quot;))
  )
  </code></pre>
  <p>But returns error: extraneous input 'WHEN' expecting {')', ','}(line 1, pos 37)</p>
  <p>Any help would be greatly appreciated!</p>
  <p>EDIT:
  Looks like:</p>
  <p><code>.withColumn('ArrayCol',f.expr(&quot;transform(ArrayCol, x -&gt; CASE WHEN x &lt; 0 THEN 0 ELSE x END&quot;)) </code> woks.</p>
  <p>I'd still be interested to understand how you could do this with a lambda function. Thanks!</p>