title: Creating indices for each group in Spark dataframe
body: <p>I have a dataframe in Spark with 2 columns, <code>group_id</code> and <code>value</code>, where <code>value</code> is a double. I would like to group the data based on the <code>group_id</code>, order each group by <code>value</code>, and then add a third column <code>index</code> that represents the position of <code>value</code> in the ordering of values for the group.</p>
  
  <p>For example, considering the following input data:</p>
  
  <pre><code>+--------+-----+
  |group_id|value|
  +--------+-----+
  |1       |1.3  |
  |2       |0.8  |
  |1       |3.4  |
  |1       |-1.7 |
  |2       |2.3  |
  |2       |5.9  |
  |1       |2.7  |
  |1       |0.0  |
  +--------+-----+
  </code></pre>
  
  <p>The output would then be something like</p>
  
  <pre><code>+--------+-----+-----+
  |group_id|value|index|
  +--------+-----+-----+
  |1       |-1.7 |1    |
  |1       |0.0  |2    |
  |1       |1.3  |3    |
  |1       |2.7  |4    |
  |1       |3.4  |5    |
  |2       |0.8  |1    |
  |2       |2.3  |2    |
  |2       |5.9  |3    |
  +--------+-----+-----+
  </code></pre>
  
  <p>It is unimportant if the index is 0-based and whether the sort is ascending or descending.</p>
  
  <p>As a follow-up, consider the case where there is a third column, <code>extra</code>, in the original data that takes on multiple values for some <code>(group_id, value)</code> combinations. An example is:</p>
  
  <pre><code>+--------+-----+-----+
  |group_id|value|extra|
  +--------+-----+-----+
  |1       |1.3  |1    |
  |1       |1.3  |2    |
  |2       |0.8  |1    |
  |1       |3.4  |1    |
  |1       |3.4  |2    |
  |1       |3.4  |3    |
  |1       |-1.7 |1    |
  |2       |2.3  |1    |
  |2       |5.9  |1    |
  |1       |2.7  |1    |
  |1       |0.0  |1    |
  +--------+-----+-----+
  </code></pre>
  
  <p>Is there a way to add an <code>index</code> column such that the <code>extra</code> column is not considered but still kept? The output in this case would be</p>
  
  <pre><code>+--------+-----+-----+-----+
  |group_id|value|extra|index|
  +--------+-----+-----+-----+
  |1       |-1.7 |1    |1    |
  |1       |0.0  |1    |2    |
  |1       |1.3  |1    |3    |
  |1       |1.3  |2    |3    |
  |1       |2.7  |1    |4    |
  |1       |3.4  |1    |5    |
  |1       |3.4  |2    |5    |
  |1       |3.4  |3    |5    |
  |2       |0.8  |1    |1    |
  |2       |2.3  |1    |2    |
  |2       |5.9  |1    |3    |
  +--------+-----+-----+-----+
  </code></pre>
  
  <p>I know that it is possible to do this by duplicating the data, dropping the <code>extra</code> column</p>
  
  <ol>
  <li>Duplicating the data</li>
  <li>Dropping the <code>extra</code> column</li>
  <li>Performing a  <code>distinct</code> operation, which would result in data in the original example</li>
  <li>Compute the <code>index</code> column using the original solution</li>
  <li>Join the result with the data from the second example</li>
  </ol>
  
  <p>However, this would involve a lot of extra computation and overhead. </p>