title: How to group a pyspark dataframe and use a shift operator as aggregation method?
body: <p>I have the following dataframe :</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>ride</th>
  <th>window_left</th>
  <th>window_right</th>
  <th>time</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>No</td>
  <td>No</td>
  <td>1</td>
  </tr>
  <tr>
  <td>1</td>
  <td>No</td>
  <td>Yes</td>
  <td>2</td>
  </tr>
  <tr>
  <td>1</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>3</td>
  </tr>
  <tr>
  <td>1</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>4</td>
  </tr>
  <tr>
  <td>2</td>
  <td>No</td>
  <td>No</td>
  <td>1</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>No</td>
  <td>2</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>3</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>4</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>5</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>And I would like to group this pyspark dataframe by the column <code>ride</code> and shift either <code>window_left</code> or <code>window_right</code> according to which one takes the value <code>Yes</code> first, like that :</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>ride</th>
  <th>window_left</th>
  <th>window_right</th>
  <th>time</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>No</td>
  <td>No</td>
  <td>1</td>
  </tr>
  <tr>
  <td>1</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>2</td>
  </tr>
  <tr>
  <td>1</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>3</td>
  </tr>
  <tr>
  <td>1</td>
  <td>None</td>
  <td>Yes</td>
  <td>4</td>
  </tr>
  <tr>
  <td>2</td>
  <td>No</td>
  <td>No</td>
  <td>1</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>2</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>3</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>Yes</td>
  <td>4</td>
  </tr>
  <tr>
  <td>2</td>
  <td>Yes</td>
  <td>None</td>
  <td>5</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>I would like to use pyspark only to perform this transformation. I cannot use pandas, and that's why I have some difficulties. The tricky part for me is to shift after grouping, knowing that the shift depends on the group.</p>
  <p>Any help would be appreciated, thanks !</p>