From ArchWiki
Jump to navigation Jump to search



Install the hadoopAUR package.


By default, hadoop is already configured for pseudo-distributed operation. Some environment variables are set in /etc/profile.d/hadoop.sh with different values than traditional hadoop.

ENV Value Description Permission
HADOOP_CONF_DIR /etc/hadoop Where configuration files are stored. Read
HADOOP_LOG_DIR /tmp/hadoop/log Where log files are stored. Read and Write
HADOOP_WORKERS /etc/hadoop/workers File naming remote worker hosts. Read
HADOOP_PID_DIR /tmp/hadoop/run Where pid files are stored. Read and Write

You also should set up the following files correctly.


You need to tell hadoop your JAVA_HOME in /etc/hadoop/hadoop-env.sh because it doesn't assume the location where it's installed to in Arch Linux by itself:


Check installation with:

hadoop version

The HADOOP_WORKERS option was previously called HADOOP_SLAVES. If you get warning message "WARNING: HADOOP_SLAVES has been replaced by HADOOP_WORKERS. Using value of HADOOP_SLAVES." Then replace export HADOOP_SLAVES=/etc/hadoop/slaves in /etc/profile.d/hadoop.sh with:

export HADOOP_WORKERS=/etc/hadoop/workers

Single Node Setup

Note: This section is based on the Hadoop Official Documentation

Standalone Operation

By default, Hadoop is configured to run in a non-distributed mode, as a single Java process. This is useful for debugging.

The following example copies the unpacked conf directory to use as input and then finds and displays every match of the given regular expression. Output is written to the given output directory.

$ HADOOP_CONF_DIR=/usr/lib/hadoop/orig_etc/hadoop/
$ mkdir input
$ cp /etc/hadoop/*.xml input
$ hadoop jar /usr/lib/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0.jar grep input output 'dfs[a-z.]+'
$ cat output/*

Pseudo-Distributed Operation

Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

By default, Hadoop will run as the user root. You can change the user in /etc/conf.d/hadoop:

HADOOP_USERNAME="<your user name>"

Set up passphraseless ssh

Make sure you have sshd enabled. Now check that you can connect to localhost without a passphrase:

$ ssh localhost

If you cannot ssh to localhost without a passphrase, execute the following commands:

$ ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa
$ chmod 0600 ~/.ssh/authorized_keys

Also make sure this line is commented in /etc/ssh/sshd_config

#AuthorizedKeysFile .ssh/authorized_keys


Format a new distributed-filesystem:

$ hadoop namenode -format

Edit /etc/hadoop/core-site.xml and add below configuration:


Start the hadoop namenode:

$ hadoop namenode

You may access the web GUI via http://localhost:9870/


Reason: Instructions for older versions of Hadoop (Discuss in Talk:Hadoop#)

Start the following hadoop systemd units: hadoop-datanode, hadoop-jobtracker, hadoop-namenode, hadoop-secondarynamenode, hadoop-tasktracker.

The hadoop daemon log output is written to the ${HADOOP_LOG_DIR} directory (defaults to /var/log/hadoop).

Browse the web interface for the NameNode and the JobTracker; by default they are available at:

Copy the input files into the distributed filesystem:

$ hadoop fs -put /etc/hadoop input

Run some of the examples provided:

$ hadoop jar /usr/lib/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep input output 'dfs[a-z.]+'

Examine the output files:


$ hadoop fs -get output output
$ cat output/*


View the output files on the distributed filesystem:

$ hadoop fs -cat output/*

When you're done, stop the daemons hadoop-datanode, hadoop-jobtracker, hadoop-namenode, hadoop-secondarynamenode, hadoop-tasktracker.