title: COLLECT_SET() in Hive (Hadoop)
body: <p>I just learned about the collect_set() function in Hive, and I started a job on a development 3-node cluster.</p>
  
  <p>I only have about 10 GB to process. The job, however, is literally taking forever. I think there might be either a bug in the implementation of collect_set(), a bug in my code, or the collect_set() function is really resource intensive.</p>
  
  <p>Here's My SQL for Hive (no pun intended):</p>
  
  <pre><code>INSERT OVERWRITE TABLE sequence_result_1
  SELECT sess.session_key as session_key,
         sess.remote_address as remote_address,
         sess.hit_count as hit_count,
         COLLECT_SET(evt.event_id) as event_set,
         hit.rsp_timestamp as hit_timestamp,
         sess.site_link as site_link
      FROM site_session sess 
          JOIN (SELECT * FROM site_event 
                  WHERE event_id = 274 OR event_id = 284 OR event_id = 55 OR event_id = 151) evt 
              ON (sess.session_key = evt.session_key)
          JOIN site_hit hit ON (sess.session_key = evt.session_key)
  GROUP BY sess.session_key, sess.remote_address, sess.hit_count, hit.rsp_timestamp, sess.site_link
  ORDER BY hit_timestamp;
  </code></pre>
  
  <p>There are 4 MR passes. The first took about 30 seconds. The second Map took about 1 minute. And most of the second reduce took about 2 minutes. In the last two hours, it's been incrementing from 97.71% to 97.73%. Is this right? I think there must be some issue. I took a look at the log, and I can't tell whether it's normal.</p>
  
  <p>[Sample of log]</p>
  
  <pre><code>2011-06-21 16:32:22,715 WARN org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 120894
  2011-06-21 16:32:22,758 WARN org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size = 108804
  2011-06-21 16:32:23,003 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 4 forwarding 5142000000 rows
  2011-06-21 16:32:23,003 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 5142000000 rows
  2011-06-21 16:32:24,138 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 4 forwarding 5143000000 rows
  2011-06-21 16:32:24,138 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 5143000000 rows
  2011-06-21 16:32:24,725 WARN org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Tbl flush: #hash table = 120894
  2011-06-21 16:32:24,768 INFO org.apache.hadoop.hive.ql.exec.GroupByOperator: 6 forwarding 42000000 rows
  2011-06-21 16:32:24,771 WARN org.apache.hadoop.hive.ql.exec.GroupByOperator: Hash Table flushed: new size = 108804
  2011-06-21 16:32:25,338 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 4 forwarding 5144000000 rows
  2011-06-21 16:32:25,338 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 5144000000 rows
  2011-06-21 16:32:26,467 INFO org.apache.hadoop.hive.ql.exec.JoinOperator: 4 forwarding 5145000000 rows
  2011-06-21 16:32:26,468 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 5 forwarding 5145000000 rows
  </code></pre>
  
  <p>I'm pretty new at this, and trying to work with collect_set() and Hive Array is driving me off the deep end.</p>
  
  <p>Thanks in advance :)</p>