title: Spark dataframe transform multiple rows to column
body: <p>I am a novice to spark, and I want to <strong>transform</strong> below <strong>source dataframe</strong> (load from JSON file):</p>
  
  <pre><code>+--+-----+-----+
  |A |count|major|
  +--+-----+-----+
  | a|    1|   m1|
  | a|    1|   m2|
  | a|    2|   m3|
  | a|    3|   m4|
  | b|    4|   m1|
  | b|    1|   m2|
  | b|    2|   m3|
  | c|    3|   m1|
  | c|    4|   m3|
  | c|    5|   m4|
  | d|    6|   m1|
  | d|    1|   m2|
  | d|    2|   m3|
  | d|    3|   m4|
  | d|    4|   m5|
  | e|    4|   m1|
  | e|    5|   m2|
  | e|    1|   m3|
  | e|    1|   m4|
  | e|    1|   m5|
  +--+-----+-----+
  </code></pre>
  
  <p><strong>Into</strong> below <strong>result dataframe</strong>:</p>
  
  <pre><code>+--+--+--+--+--+--+
  |A |m1|m2|m3|m4|m5|
  +--+--+--+--+--+--+
  | a| 1| 1| 2| 3| 0|
  | b| 4| 2| 1| 0| 0|
  | c| 3| 0| 4| 5| 0|
  | d| 6| 1| 2| 3| 4|
  | e| 4| 5| 1| 1| 1|
  +--+--+--+--+--+--+
  </code></pre>
  
  <p>Here is the <strong>Transformation Rule</strong>:</p>
  
  <ol>
  <li><p>The result dataframe is consisted with <code>A + (n major columns)</code> where the <code>major</code> columns names are specified by:</p>
  
  <pre><code>sorted(src_df.map(lambda x: x[2]).distinct().collect())
  </code></pre></li>
  <li><p>The result dataframe contains <code>m</code> rows where the values for <code>A</code> column are provided by:</p>
  
  <pre><code>sorted(src_df.map(lambda x: x[0]).distinct().collect())
  </code></pre></li>
  <li><p>The value for each major column in result dataframe is the value from source dataframe on the corresponding <code>A</code> and major
  (e.g. the count in Row 1 in source dataframe is mapped to the <code>box</code> where <code>A</code> is <code>a</code> and column <code>m1</code>)</p></li>
  <li><p>The combinations of <code>A</code> and <code>major</code> in source dataframe has no duplication (please consider it a primary key on the two columns in SQL)</p></li>
  </ol>