title: Hadoop-Hive-HBase Advice for Web Analytics
body: <p>The team I work on is fortunate enough to have management that recognizes the need to enhance our skills and learn new technologies.  As a result, whenever we have a little downtime between major projects, we are encouraged to use that time to stretch our minds a bit and learn something new.  We often tackle a large research project as a team so that everyone benefits from the knowledge.  For example, we built a spec-compliant Kerberos authentication server to get familiar with the ins and outs of the protocol.  We wrote our own webserver to learn about efficient design strategies for networked applications.</p>
  
  <p>Recently, we've been very curious about Map-Reduce, specifically Hadoop and the various supporting components (HBase, HDFS, Pig, Hive, etc.).  To learn a bit more about it, we would like to write a web analytics service.  It will use Javascript page tagging to gather the metrics, and Hadoop and <em>something</em> to make analytics and reports available via a web interface.</p>
  
  <p>The non-Hadoop side of the architecture is easy.  A Java servlet will parse the parameters from a Javascript tag (easy enough -- we're a Java shop).  The servlet will then send out a JMS message for asynchronous processing (again, easy).</p>
  
  <p>My question is...  What next?  We've researched things like Hive a bit, and it sounds like a great fit for querying the datastore for the various metrics we're looking for.  But, it's high latency.  We're fortunate enough to be able to drop this onto a website that gets a few million hits per month.  We'd really like to get relatively quick metrics using the web interface for our analytics tool.  Latency is not our friend.  So, what is the best way to accomplish this?  Would it be to run the queries as a scheduled job and then store the results somewhere with lower latency (PostgreSQL, etc.) and retrieve them from there?  If that's the case, where should the component listening for the JMS messages store the data?  Can Hive get its data from HBase directly?  Should we store it in HDFS somewhere and read it in Hive?</p>
  
  <p>Like I said, we're a very technical team and love learning new technologies.  This, though, is way different from anything we've learned before, so we'd like to get a sense of what the "best practices" would be here.  Any advice or opinions you can give are GREATLY appreciated!</p>
  
  <p><strong>EDIT :</strong>  I thought I'd add some clarification as to what I'm looking for.  I'm seeking advice on architecture and design for a solution such as this.  We'll collect 20-30 different metrics on a site that gets several million page views per month.  This will be a lot of data, and we'd like to be able to get metrics in as close to realtime as possible.  I'm looking for best practices and advice on the architecture of such a solution, because I don't want us to come up with something on our own that is really bad that will leave us thinking we're "Hadoop experts" just because it works.</p>