title: How can I disaggregate rows of a data frame in Spark?
body: <p>I have a Spark dataframe containing data similar to the following:</p>
  
  <pre><code>+----+---------------------+-------+----------+-------------+
  | ID |      Timestamp      | Value | Interval | Consumption |
  +----+---------------------+-------+----------+-------------+
  |  1 | 2012-05-02 12:30:00 |   550 |        1 |           5 |
  |  1 | 2012-05-02 12:45:00 |   551 |        1 |           1 |
  |  1 | 2012-05-02 13:00:00 |   554 |        1 |           3 |
  |  1 | 2012-05-02 14:00:00 |   578 |        4 |          24 |
  |  1 | 2012-05-02 14:15:00 |   578 |        1 |           0 |
  |  1 | 2012-05-02 14:30:00 |   584 |        1 |           6 |
  +----+---------------------+-------+----------+-------------+
  </code></pre>
  
  <p>I'm looking to turn this into something like the following:</p>
  
  <pre><code>+----+---------------------+-------+----------+-------------+------------+
  | ID |      Timestamp      | Value | Interval | Consumption | Estimation |
  +----+---------------------+-------+----------+-------------+------------+
  |  1 | 2012-05-02 12:30:00 |   550 |        1 |           5 | ?          |
  |  1 | 2012-05-02 12:45:00 |   551 |        1 |           1 | ?          |
  |  1 | 2012-05-02 13:00:00 |   554 |        1 |           3 | ?          |
  |  1 | 2012-05-02 13:15:00 |   560 |        1 |           6 | 4          |
  |  1 | 2012-05-02 13:30:00 |   566 |        1 |           6 | 4          |
  |  1 | 2012-05-02 13:45:00 |   572 |        1 |           6 | 4          |
  |  1 | 2012-05-02 14:00:00 |   578 |        1 |           6 | 4          |
  |  1 | 2012-05-02 14:15:00 |   578 |        1 |           0 | ?          |
  |  1 | 2012-05-02 14:30:00 |   584 |        1 |           6 | ?          |
  +----+---------------------+-------+----------+-------------+------------+
  </code></pre>
  
  <p>More specifically I want to turn this:</p>
  
  <pre><code>+----+---------------------+-------+----------+-------------+
  | ID |      Timestamp      | Value | Interval | Consumption |
  +----+---------------------+-------+----------+-------------+
  |  1 | 2012-05-02 14:00:00 |   578 |        4 |          24 |
  +----+---------------------+-------+----------+-------------+
  </code></pre>
  
  <p>Into this:</p>
  
  <pre><code>+----+---------------------+-------+----------+-------------+------------+
  | ID |      Timestamp      | Value | Interval | Consumption | Estimation |
  +----+---------------------+-------+----------+-------------+------------+
  |  1 | 2012-05-02 13:15:00 |   560 |        1 |           6 | 4          |
  |  1 | 2012-05-02 13:30:00 |   566 |        1 |           6 | 4          |
  |  1 | 2012-05-02 13:45:00 |   572 |        1 |           6 | 4          |
  |  1 | 2012-05-02 14:00:00 |   578 |        1 |           6 | 4          |
  +----+---------------------+-------+----------+-------------+------------+
  </code></pre>
  
  <p>I want to take the rows with more than 1 interval out of the original table, interpolate Values for missing intervals and reinsert the newly created rows into the initial table place of the original rows. I have ideas of how to achieve this (in PostgreSQL for example I would simply use the generate_series() function to create the required Timestamps and calculate new Values), but implementing these in Spark/Scala is proving troublesome.</p>
  
  <p>Assuming I've created a new dataframe containing only rows with Interval > 1, how could I replicate those rows 'n' times with 'n' being the value of Interval? I believe that would give me enough to get going using a Counter function partitioned by some row reference I can create.</p>
  
  <p>If there's a way to replicate the behavior of generate_series() that I've missed, even better.</p>