title: Querying large amount of data processed by Hive
body: <p>Say I have around 10-20GB of data in HDFS as a Hive table. This has been obtained after several Map-Reduce jobs and JOIN over two separate datasets. I need to make this Queryable to the user. What options do I have?</p>
  
  <ol>
  <li>Use <a href="http://sqoop.apache.org/" rel="nofollow">Sqoop</a> to transfer data from HDFS to an RDS like Postgresql. But I want to avoid spending so much time on data transfer. I just tested HDFS->RDS in the same AWS region using Sqoop, and 800mb of data takes 4-8 minutes. So you can imagine ~60gb of data would be pretty unmanagable. This would be my last resort. </li>
  <li>Query Hive directly from my Webserver as per user request. I haven't ever head of Hive being used like this so I'm skeptical about this. This struck me because I just found out you can query hive tables remotely after some port forwarding on the EMR cluster. But being new to big(ish) data I'm not quite sure about the risks associated with this. Is it commonplace to do this?</li>
  <li>Some other solution - How do people usually do this kind of thing? Seems like a pretty common task.</li>
  </ol>
  
  <p>Just for completeness sake, my data looks like this: </p>
  
  <pre><code>id     time             cat1    cat2    cat3    metrics[200]
  A123   1234212133        12     ABC      24      4,55,231,34,556,123....(~200)
  .
  .
  .
  </code></pre>
  
  <p>(time is epoch)</p>
  
  <p>And my Queries look like this: </p>
  
  <p><code>select cat1, corr(metrics[2],metrics[3]),corr(metrics[2],metrics[4]),corr(metrics[2],metrics[5]),corr(metrics[2],metrics[6]) from tablename group by cat1;</code></p>
  
  <p>I <em>need</em> the correlation function, which is why I've chosen postgresql over MySQL.</p>