title: Apache Spark calculating column value on the basis of distinct value of columns
body: <p>I am processing the following tables and I would like to compute a new column (outcome) based on the distinct value of 2 other columns. </p>
  
  <pre><code>| id1  | id2 | outcome
  |  1   |  1  |  1
  |  1   |  1  |  1
  |  1   |  3  |  2
  |  2   |  5  |  1 
  |  3   |  1  |  1  
  |  3   |  2  |  2
  |  3   |  3  |  3
  </code></pre>
  
  <p>The outcome should begin in incremental order starting from 1 based on the combined value of <code>id1</code> and <code>id2</code>. Any hints how this can be accomplished in Scala. <code>row_number</code> doesn't seem to be useful here in this case.</p>
  
  <p>The logic here is that for each unique value of <code>id1</code> we will start numbering the outcome with min(<code>id2</code>) for corresponding <code>id1</code> being assigned a value of 1.</p>