title: Implementing lambda architecture using Hadoop, Apache Spark, HBase
body: <p>I want to implement lambda architecture with a simple example. I am not able to fit my technology stack into each layer of lambda architecture.</p>
  
  <p>I want to find out top 10 popular hashtags in twitter tweets in real time. I am listing the purpose of each layer hare from <a href="http://lambda-architecture.net/" rel="nofollow">http://lambda-architecture.net/</a>.</p>
  
  <ul>
  <li>All <strong>data</strong> entering the system is dispatched to both the batch layer
  and the speed layer for processing.</li>
  <li>The <strong>batch layer</strong> has two functions: (i) managing the master dataset
  (an immutable, append-only set of raw data), and (ii) to pre-compute
  the batch views.</li>
  <li>The <strong>serving layer</strong> indexes the batch views so that they can be queried
  in low-latency, ad-hoc way.</li>
  <li>The <strong>speed layer</strong> compensates for the high latency of updates to the
  serving layer and deals with recent data only.</li>
  <li>Any incoming <strong>query</strong> can be answered by merging results from batch
  views and real-time views.</li>
  </ul>
  
  <p>What part of my problem I can solve with each layer. I am working with Apache Spark &amp; Hadoop HDFS technology stack.</p>