title: Spark/Hive UDAF for distinct values by key
body: <p>In big data processing it is often desirable to "sketch" summaries of groups <em>without</em> changing the existing query structure (grouping, subqueries, etc.). In Spark SQL (and HiveQL) <code>collect_set</code> is an example of a UDAF that does this. It builds an array of the unique values of a column per group. </p>
  
  <p>I am looking for a UDAF that builds a map of the unique values from column A for each unique value of column B. For example, given</p>
  
  <pre><code>date user_id category revenue  
  1/1  1       a        1  
  1/1  2       b        0  
  1/1  3       a        0  
  1/2  2       b        10  
  1/2  3       a        0  
  1/2  4       b        1  
  </code></pre>
  
  <p>the query </p>
  
  <pre class="lang-sql prettyprint-override"><code>select date, collect_set_by_key(category, user_id) as distinct_user_ids_by_category
  from tbl
  group by date
  order by date
  </code></pre>
  
  <p>should generate</p>
  
  <pre><code>date distinct_user_ids_by_category
  1/1  {a: [1, 3], b: [2]}
  1/2  {a: [3], b: [2, 4]}
  </code></pre>
  
  <p><a href="https://stackoverflow.com/questions/24826530/collect-to-a-map-in-hive">Collect to a Map in Hive</a> asks for the same outcome but does not require the solution to be via a single UDAF without changes to query structure, which makes this question substantially different.</p>
  
  <p>My deployment environment is Spark 2.1 so a Hive UDAF or a native Spark UDAF would work equally well. I've checked brickhouse, hivemall and a few other libraries with Hive UDAFs but have not found a function that does this.</p>