title: Hbase MapReduce: how to use custom class as value for the mapper and/or reducer?
body: <p>I am trying to familiarize myself with Hadoop/Hbase MapReduce jobs to be able to properly write them. Right now I have an Hbase instance with a table called dns with some DNS records. I tried to make a simple unique domains counter that outputs a file and it worked. Right now, I only use <code>IntWritable</code> or <code>Text</code> and I was wondering if it's possible to use custom objects for my Mapper/Reducer. I tried to do it myself, but I'm getting</p>
  
  <pre><code>Error: java.io.IOException: Initialization of all the collectors failed. Error in last collector was :null
      at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:415)
      at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
      at org.apache.hadoop.mapred.MapTask$NewOutputCollector.&lt;init&gt;(MapTask.java:698)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:422)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
  Caused by: java.lang.NullPointerException
      at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:1011)
      at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
      ... 9 more
  </code></pre>
  
  <p>Since I'm new to this, I don't actually know what to do. I'm guessing I have to implement one or more interfaces or to extend an abstract class, but I can't find <a href="http://hbase.apache.org/book.html" rel="nofollow noreferrer">here</a> or on the internet a proper example.</p>
  
  <p>I tried to make a simple domains counter from my dns table, but using a class as a wrapper over an integer (for didactic purposes only). My <strong>Map class</strong> looks like this:</p>
  
  <pre><code>public class Map extends TableMapper&lt;Text, MapperOutputValue&gt; {
      private static byte[] columnName = "fqdn".getBytes();
      private static byte[] columnFamily = "d".getBytes();
  
      public void map(ImmutableBytesWritable row, Result value, Context context)
              throws InterruptedException, IOException {
  
          String fqdn = new String(value.getValue(columnFamily, columnName));
          Text key = new Text();
          key.set(fqdn);
          context.write(key, new MapperOutputValue(1));
  
      }
  }
  </code></pre>
  
  <p><strong>The Reducer</strong>:</p>
  
  <pre><code>public class Reduce extends Reducer&lt;Text, MapperOutputValue, Text, IntWritable&gt; {
      @Override
      public void reduce(Text key, Iterable&lt;MapperOutputValue&gt; values, Context context)
              throws IOException, InterruptedException {
  
          int i = 0;
          for (MapperOutputValue val : values) {
              i += val.getCount();
          }
  
          context.write(key, new IntWritable(i));
      }
  }
  </code></pre>
  
  <p>And a part of my <strong>Driver/Main</strong> function:</p>
  
  <pre><code> TableMapReduceUtil.initTableMapperJob(
                  "dns",
                  scan,
                  Map.class,
                  Text.class,
                  MapperOutputValue.class,
                  job);
  
  /* Set output parameters */
  job.setOutputKeyClass(Text.class);
  job.setOutputValueClass(IntWritable.class);
  job.setOutputFormatClass(TextOutputFormat.class);
  </code></pre>
  
  <p>As I said, <strong>MapperOutputValue</strong> is just a simple class that contains a private Integer, a constructor with a parameter, a getter and a setter. I also tried adding a <code>toString</code> method but it still doesn't work.</p>
  
  <p>So my question is: what's the best way to use custom classes as an output of the mapper/input for the reducer? Also, let's say I want to use a class with multiple fields as an final output of the reducer. What should this class implement/extends? Is it a good idea or I should stick to using "primitives" as IntWritable or Text?</p>
  
  <p>Thank!</p>