title: spark sql Ignore Null Values in partition by clause column
body: <p>Please find the below query. The partition column has Null Values and I want to ignore Null values while doing last_value in partition column too.</p>
  <pre><code>select * from (
  select col1, col2,state_time,
  coalesce (CASE WHEN ra.col2 ='' THEN NULL ELSE col2 end,
  last_value (col2) IGNORE NULLS OVER
   (partition by col1 order by state_time ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING )
  ) as job_icims_id_der
   from delta_source_db.recruiting_activity ra
   order by state_time desc) inq where col1 is null
   and col2 is null and job_icims_id_der is  not null order by state_time
    limit 10
  </code></pre>
  <p><a href="https://i.stack.imgur.com/QBM09.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/QBM09.png" alt="enter image description here" /></a></p>