title: calculating YTD and MTD using pyspark
body: <p>I have trying to implement this for about a month.
  Just using some sample data from other stack-overflow questions.</p>
  
  <pre><code>   FinancialYearStart  MonthOfFinancialYear  SalesTotal
              2015                     1          10
              2015                     2          10
              2015                     5          10
              2015                     6          50
              2016                     1          10
              2016                     3          20
              2016                     2          30
              2017                     6          70
              2017                     7          80
  </code></pre>
  
  <p>How can we use this to calculate YTD sales total for each month.</p>
  
  <pre><code>FinancialYearStart  MonthOfFinancialYear  SalesTotal  YTDTotal
              2015                     1          10        10
              2015                     2          10        20
              2015                     5          10        30
              2015                     6          50        50
              2016                     1          10        60
              2016                     3          20        80
              2016                     2          30       110
              2017                     6          70        70
              2017                     7          80       150
  </code></pre>
  
  <p>to be more specific how can we calculate these metrics using a group by or by using windowing functions.</p>
  
  <p>For example:</p>
  
  <pre><code>Year Month Customer TotalMonthlySales
  2015 1 Dog 10
  2015 2 Dog 10
  2015 3 Cat 20
  2015 4 Dog 30
  2015 5 Cat 10
  2015 7 Cat 20
  2015 7 Dog 10
  2016 1 Dog 40
  2016 2 Dog 20
  2016 3 Cat 70
  2016 4 Dog 30
  2016 5 Cat 10
  2016 6 Cat 20
  2016 7 Dog 10
  </code></pre>
  
  <p>Would give:</p>
  
  <pre><code>Year Month Customer TotalMonthlySales YTDSales
  2015 1 Dog 10 10
  2015 2 Dog 10 20
  2015 3 Cat 20 20
  2015 4 Dog 30 50
  2015 5 Cat 10 30
  2015 7 Cat 20 40
  2015 7 Dog 10 60
  2016 1 Dog 40 40
  2016 2 Dog 20 60
  2016 3 Cat 70 70
  2016 4 Dog 30 90
  2016 5 Cat 10 80
  2016 6 Cat 20 100
  2016 7 Dog 10 100
  </code></pre>
  
  <p>Now I am using rolling window functions to calculate last 4 weeks 13 weeks using the below approach.</p>
  
  <pre><code>w = (Window().partitionBy(col("number"),col("id")).orderBy(F.col("timestampGMT").cast('long')).rangeBetween(-days(27), 0))
  df1 = df.withColumn('lw_4weeksCY_SALES_DOLLARS', F.sum("CY_SALES_DOLLARS").over(w)).withColumn('lw_4weeksCY_SALES_UNITS', F.sum("CY_SALES_UNITS")
  </code></pre>