title: Seeking efficient way to categorize values in Spark
body: <p>Looking to improve my current approach (improve speed) to go through an RDD and quickly categorize the row values. For example, let's say I have four columns and five rows, and want to count the number of values in each category per column. My final result would look like:</p>
  
  <pre><code>Column Name  Category 1  Category 2  Category 3
  Col 1           2           3           0
  Col 2           0           4           1
  Col 3           0           0           5
  Col 4           2           2           1
  </code></pre>
  
  <p>I've been testing two approaches:</p>
  
  <p>Approach 1</p>
  
  <ul>
  <li>Map each row to a list of row count tuples. Before data could look like ['val1', 2.0, '0001231', True] -> afterwards it would look like [(1, 0, 0), (0, 1, 0), (0, 1, 0), (0, 0, 1)]</li>
  <li>Reduce each row by adding the tuples together</li>
  </ul>
  
  <blockquote>
    <p>my_rdd.map("function to categorize").reduce("add tuples")</p>
  </blockquote>
  
  <p>Approach 2</p>
  
  <ul>
  <li>Flat map each value to its own row as a key-value pair. Similar to first approach, but result would look like ("col1", (1, 0, 0), ("col2", (0, 1, 0)), ("col3", (0, 1, 0)), ("col4", (0, 0, 1)), where each tuple becomes a new row.</li>
  <li>Reduce by key</li>
  </ul>
  
  <blockquote>
    <p>my_rdd.flatMap("function to categorize and create row for each column value").reduceByKey("add tuples for each column")</p>
  </blockquote>
  
  <p>Is there a more efficient way to do this for a generic RDD? I'm also looking to count distinct values while I'm doing the mapping, but realize that would require a separate discussion.</p>