Showing posts from October, 2013

Jobs Page

Here you will get the job updates related to Hadoop Technology, in JOBS , also it is
recommended to follow in LinkedIn . I suggest you to update your jobs in hadoop by following this page hadoop jobs in facebook.

When to use HBase and when MapReduce?

Thanks to Hadoop Tips



MapReduce Patterns




Why does Hadoop uses KV (Key/Value) pairs?

Hadoop implements the MapReduce paradigm as mentioned in the original Google MapReduce Paper in terms of key/value pairs. The output types of the Map should match the input types of the Reduce as shown below.
(K1,V1) -> Map -> (K2,V2)
(K2, V2) -> Reduce -> (K3,V3)

The big question is why use key/value pairs?
MapReduce is derived from the concepts of functional programming. Here is a tutorial on the map/fold primitives in the functional programming which are used in the MapReduce paradigm and there is no mention of key/value pairs anywhere.
According to the Google MapReduce Paper

We realized that most of our computations involved applying a map operation to each logical “record” in our input in order to compute a set of intermediate key/value pairs, and then applying a reduce operation to all the values that shared the same key, in order to combine the derived data appropriately.

The only reason I could see why Hadoop used key/value pairs is that the Google Paper had …

BSP (Hama) vs MR (Hadoop)

Thanks to Hadoop Tips
BSP (Bulk Synchronous Parallel) is a distributed computing model similar to MR (MapReduce). For some class of problems BSP performs better than MR and other way. This paper compares MR and BSP.

While Hadoop implements MR, Hama implements BSP. The ecosystem/interest/tools/$$$/articles around Hadoop is much more when compared to Hama. Recently Hama has released 0.4.0 and includes examples for page rank, graph exploration, shortest path and others. Hama has many similarities (API, command line, design etc) with Hadoop and it shouldn't take much time to get started with Hama for those who are familiar with Hadoop.

As of now Hama is in incubator stage and is looking for contributors. Hama is still in the early stage and there is a lot of scope for improvement (performance, testing, documentation etc) . For someone who wants to start with Apache, Hama is a choice. Thomas and Edward had been actively blogging about Hama and are very responsive for any clari…


(input) -> map -> -> reduce -> (output)
(input) -> map -> -> combine* -> -> reduce -> (output)




Passing parameters to Mappers and Reducers


Setting the parameter:


2. Before launching the job using the old MR API

? 1 2 JobConf job = (JobConf) getConf(); job.set("test", "123");
3. Before launching the job using the new MR API

? 1 2 3 Configuration conf = newConfiguration(); conf.set("test", "123"); Job job = newJob(conf);
Getting the parameter:


? 1 2 3 4 privatestaticLong N; publicvoidconfigure(JobConf job) { N =…


Some thing really simple ....
A) This is due to Type Erasure :
Nice to learn new things !!!

Hadoop MapReduce challenges in the Enterprise




Platform) The current Hadoop MapReduce implementation does…

Hadoop and MapReduce Algorithms Academic Papers


Hadoop and JavaScript


Is Java a prerequisite for getting started with Hadoop?


Why No?

According to the Cloudera blog entry `What Do Real-Life Hadoop Workloads…

Seamless access to the Java SE API documentation

For seamless interaction between the two API, reference to
in the Hadoop API should be replaced with
This enables for seamless offline access and better productivity.

Hadoop Jar Hell




Installation and configuration of Apache Oozie


Introduction to Apache Hive and Pig





What is the relation between Hibernate and Swap Memory?


HDFS Facts and Fiction

Sometimes even the popular blogs/sites don't get the facts straight, not sure if articles are reviewed by others or not. As I mentioned earlier, changes in Hadoop are happening at a very rapid pace and it's very difficult to keep updated with the latest.

Here is a snippet from ZDNet article


HDFS supports appends and that's the core of HBase. Without HDFS append functionality HBase doesn't work. There had been some challenges to get appends work in HDFS, but they have been sorted out. Also, HBase supports random/real-time read/write access of the data on top of HDFS.

Agree that NameNode is a SPOF in HDFS. But, HDFS High Availability will include two NameNodes and for now switchover from the active to the standby NameNode is…

Getting started with HDFS client side mount table

With HDFS federation it's possible to have multiple NameNode in a HDFS cluster. While this is good from a NameNode scalability and isolation perspective, it's difficult to manage multiple name spaces from a client application perspective. HDFS client mount table makes multiple names spaces transparent to the client. ViewFs more details on how to use the HDFS client mount table.

Earlier blog entry detailed how to setup HDFS federation. Let's assume the two NameNodes have been setup successfully on namenode1 and namenode2.
Lets map ? 1 /NN1Home to hdfs://namenode1:9001/home and? 1 /NN2Home to hdfs://namenode2:9001/home Add the following to the core-site.xml? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 xmlversion="1.0"?> <configuration> <property> <name>hadoop.tmp.dir</name> <value>/home/praveensripati/Installations/hadoop-0.23.0/tmp</value> </property> <property> <name></name> <value>viewfs:///</value>…

How to start CheckPoint NameNode in Hadoop 0.23 release?


HDFS explained as comics


HDFS Name Node High Availability




Edit (11th March, 2012) : Detail…

Getting started with HDFS Federation

HDFS Federation enables multiple NameNodes in a cluster for horizontal scalability of NameNode. All these NameNodes work independently and don't require any co-ordination. A DataNode can register with multiple NameNodes in the cluster and can store the data blocks for multiple NameNodes.
Here are the instructions to set HDFS Federation on a cluster.

1) Download the Hadoop 0.23 release from here.

2) Extract it to a folder (let's call it HADOOP_HOME) on all the NameNodes (namenode1, namenode2) and on all the slaves (slave1, slave2, slave3)

3) Add the association between the hostnames and the ip address for the NameNodes and the slaves on all the nodes in the /etc/hosts file. Make sure that the all the nodes in the cluster are able to ping to each other.

4) Make sure that the NameNodes is able to do a password-less ssh to all the slaves. Here are theinstructions for the same.

5) Add the following to .bashrc in the home folder for the NameNodes and all the slaves.? 1 2

Big Data Trendz