title: How to auto increment for the specific condition in PySpark
body: <p>I have below PySpark dataframe</p>
  <pre><code>id      dept
  1       CSE
          ISE
          ECE
  2       EEE
  4       MCE
  </code></pre>
  <p>I am trying to increment the value by checking the condition i.e if id is null then add the value to the new column by incrementing the value by 1 from the defined value (max_value), if id is not null retain the same value.</p>
  <p>to achieve above scenario I am using below code.</p>
  <pre><code>max_value = 5
  
  df = df.withColumn(&quot;idx&quot;, monotonically_increasing_id())
  w = Window().orderBy(&quot;idx&quot;)
  df = df.withColumn(&quot;row_num&quot;, F.when(F.col(&quot;id&quot;).isNull() ,(max_value + row_number().over(w)).otherwise(F.col(&quot;id&quot;))))
  </code></pre>
  <p>But i am getting below error</p>
  <pre><code>IllegalArgumentException: otherwise() can only be applied on a Column previously generated by when()
  </code></pre>
  <p>Expected output:</p>
  <pre><code>id      dept   new_col
  1       CSE    1
          ISE    6
          ECE    7
  2       EEE    2
  4       MCE    4
  </code></pre>
  <p>Can anyone help me to resolve the issue. It will be great</p>