title: Daily forecast on a PySpark dataframe
body: <p>I have the following dataframe in PySpark:</p>
  <p><code>DT_BORD_REF</code>: Date column for the month<br />
  <code>REF_DATE</code>: A date reference for current day separating past and future<br />
  <code>PROD_ID</code>: Product ID<br />
  <code>COMPANY_CODE</code>: Company ID<br />
  <code>CUSTOMER_CODE</code>: Customer ID<br />
  <code>MTD_WD</code>: Month to Date count of working days (Date = DT_BORD_REF)<br />
  <code>QUANTITY</code>: Number of items sold<br />
  <code>QTE_MTD</code>: Number of items month to date</p>
  <pre><code>+-------------------+-------------------+-----------------+------------+-------------+-------------+------+--------+-------+
  |        DT_BORD_REF|           REF_DATE|          PROD_ID|COMPANY_CODE|CUSTOMER_CODE|COUNTRY_ALPHA|MTD_WD|QUANTITY|QTE_MTD|
  +-------------------+-------------------+-----------------+------------+-------------+-------------+------+--------+-------+
  |2020-11-02 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     1|     4.0|    4.0|
  |2020-11-05 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     3|    null|    4.0|
  |2020-11-06 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     4|    null|    4.0|
  |2020-11-09 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     5|    null|    4.0|
  |2020-11-10 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     6|    null|    4.0|
  |2020-11-11 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     7|    null|    4.0|
  |2020-11-12 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     8|    null|    4.0|
  |2020-11-13 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|     9|    null|    4.0|
  |2020-11-16 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    10|    null|    4.0|
  |2020-11-17 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    11|    null|    4.0|
  |2020-11-18 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    12|    null|    4.0|
  |2020-11-19 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    13|    null|    4.0|
  |2020-11-20 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    14|    null|    4.0|
  |2020-11-23 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    15|    null|    4.0|
  |2020-11-24 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    16|    null|    4.0|
  |2020-11-25 00:00:00|2020-11-04 00:00:00|          0000043|         503|     KDAI3982|          RUS|    17|    null|    4.0|
  </code></pre>
  <p>for <code>DT_BORD_REF &lt; REF_DATE</code> all rows are actual sales and do not necessarily occurs every working day. Sometimes happens in non working days too.</p>
  <p>for <code>DT_BORD_REF &gt;= REF_DATE</code> there are no sales (it's the future)</p>
  <p>The objective is to forecast the sales for all future rows using the formula: <code>QTE_MTD/MTD_WD</code> calculated on the <code>REF_DATE</code> for each product, customer and country.</p>
  <p>The QTE_MTD was calculated from QUANTITY column using a window function. I need to divide that for <code>MTD_WD</code> on <code>REF_DATE</code> which in this exemple is 3</p>
  <p>How can I add a column with <code>MTD_WD</code> on <code>REF_DATE</code> partitioning by product, customer and country?</p>
  <p>In other words, I need to add a column with the first occurrence of <code>MTD_WD</code> when the condition <code>DT_BORD_REF &gt; REF_DATE</code> is met (again, which is 3 in this exemple) for each product, customer and country.</p>
  <p>This dataset has millions of row for different products, customers and countries
  The working days are provided by country</p>
  <p>Hope it was clear :)</p>