title: Mark all data points between two dates in pyspark
body: <p>I am given a table with sales and data about promotions attached to it.</p>
  <p>When an entry has promo data filled in it means, that a promo campaign started this day for and item X. And it will end at <code>promo_end_date</code>.</p>
  <p>Here is an example:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>date</th>
  <th>promo_end_date</th>
  <th>sales</th>
  <th>item_id</th>
  <th>promo_id</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1.1.2020</td>
  <td>3.1.2020</td>
  <td>1</td>
  <td>1</td>
  <td>A</td>
  </tr>
  <tr>
  <td>2.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  </tr>
  <tr>
  <td>3.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  </tr>
  <tr>
  <td>4.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  </tr>
  <tr>
  <td>5.1.2020</td>
  <td>6.1.2020</td>
  <td>1</td>
  <td>1</td>
  <td>B</td>
  </tr>
  <tr>
  <td>6.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  </tr>
  <tr>
  <td>1.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  </tr>
  <tr>
  <td>2.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  </tr>
  <tr>
  <td>3.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  </tr>
  <tr>
  <td>4.1.2020</td>
  <td>6.1.2020</td>
  <td>1</td>
  <td>2</td>
  <td>C</td>
  </tr>
  <tr>
  <td>5.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  </tr>
  <tr>
  <td>6.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I want to create a binary column <code>on_promo</code>, which will be marking each day with promo campaigns.</p>
  <p>So it should look like this:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>date</th>
  <th>promo_end_date</th>
  <th>sales</th>
  <th>item_id</th>
  <th>promo_id</th>
  <th>on_promo</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1.1.2020</td>
  <td>3.1.2020</td>
  <td>1</td>
  <td>1</td>
  <td>A</td>
  <td>1</td>
  </tr>
  <tr>
  <td>2.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  <td>1</td>
  </tr>
  <tr>
  <td>3.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  <td>1</td>
  </tr>
  <tr>
  <td>4.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  <td>0</td>
  </tr>
  <tr>
  <td>5.1.2020</td>
  <td>6.1.2020</td>
  <td>1</td>
  <td>1</td>
  <td>B</td>
  <td>1</td>
  </tr>
  <tr>
  <td>6.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>1</td>
  <td>null</td>
  <td>1</td>
  </tr>
  <tr>
  <td>1.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  <td>0</td>
  </tr>
  <tr>
  <td>2.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  <td>0</td>
  </tr>
  <tr>
  <td>3.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  <td>0</td>
  </tr>
  <tr>
  <td>4.1.2020</td>
  <td>6.1.2020</td>
  <td>1</td>
  <td>2</td>
  <td>C</td>
  <td>1</td>
  </tr>
  <tr>
  <td>5.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  <td>1</td>
  </tr>
  <tr>
  <td>6.1.2020</td>
  <td>null</td>
  <td>1</td>
  <td>2</td>
  <td>null</td>
  <td>1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I thought it would be done with window function, where I would partition data by <code>item_id</code> and <code>promo_id</code> and have two conditions: start date and end date. However I can't think of a way to make pyspark take a  <code>promo_end_date</code> column as an end date condition.</p>