title: Can't run Nutch2 on Hadoop2 (Nutch 2.x + Hadoop 2.4.0 + HBase 0.94.18 + Gora 0.5 + Avro 1.7.6)
body: <p>I need to install Nutch 2.3 for EMR in above configuration (subj).</p>
  
  <p>Done on local computer:</p>
  
  <ol>
  <li>Nutch 2.x</li>
  </ol>
  
  <p>1.1 svn current 2.x version</p>
  
  <p>1.2. prepared scripts:</p>
  
  <p>1.2.1 ivy:</p>
  
  <pre>
      dependency org="org.apache.hadoop" name="hadoop-common" rev="2.4.0"
      dependency org="org.apache.hadoop" name="hadoop-mapreduce-client-core" rev="2.4.0"
      dependency org="org.apache.gora" name="gora" rev="0.5"  
      dependency org="org.apache.gora" name="gora-hbase" rev="0.5"
  </pre>
  
  <p>1.2.2 default.properties:</p>
  
  <pre><code>hadoop.version=2.4.0
  version=2.3-SNAPSHOT
  </code></pre>
  
  <p>1.3. added</p>
  
  <pre><code>public int getFieldsCount() { return Field.values().length; }
  </code></pre>
  
  <p>to ProtocolStatus.java, ParseStatus.java, Host.java, WebPage.java.</p>
  
  <ol start="2">
  <li>HBase</li>
  </ol>
  
  <p>2.1 svn HBase 0.94.18</p>
  
  <p>2.2 prepared for Protobuf 2.5.0, also thanks to Dobromyslov [ <a href="https://github.com/dobromyslov" rel="nofollow">https://github.com/dobromyslov</a> ]</p>
  
  <p>2.3 also generated hbase-0.94.18-hadoop-2.4.0.jar</p>
  
  <ol start="3">
  <li><p>Gora 0.5 (also was tested for versions 0.4, 0.6-SNAPSHOT, and 0.5.3 from com.argonio.gora)</p></li>
  <li><p>Avro 1.7.6 (also played with versions 1.7.4, 1.7.7)</p></li>
  </ol>
  
  <p>4.1 svn</p>
  
  <p>4.2 patched for AVRO-813</p>
  
  <p>4.3 patched for AVRO-882 and rollbacked</p>
  
  <p>4.4 patched as in [1] - commented throwing EOFException against</p>
  
  <pre><code>org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473),
  </code></pre>
  
  <p>etc.</p>
  
  <p>After numerous exceptions, some changes have been made in Nutch 2.x and Avro 1.7.6.</p>
  
  <p>Nutch looks like a bit of running, but is unstable and incorrect.</p>
  
  <p>Cycle (inject, generate, fetch, parse, updatedb) passed but some functionalities are broken and ignored.</p>
  
  <p>It seems that i broke the normal data exchange between Nutch and HBase (also with gora and avro). Some fields (and/or some of the data formats) read and write incorrectly. F.e. many markers are lost (temporary emulated in code); data in batchId field are lost; scoring is broken also.</p>
  
  <p>Please help! I'm ready to publish all my diffs and exception traces.</p>
  
  <p>[1] <a href="http://mail-archives.apache.org/mod_mbox/nutch-user/201409.mbox/%3cCAEmTxX9HrRM00SxerFAdRdZy=wVAd9xCchDTuLaxPQ=wi0QEsw@mail.gmail.com%3e" rel="nofollow">http://mail-archives.apache.org/mod_mbox/nutch-user/201409.mbox/%3cCAEmTxX9HrRM00SxerFAdRdZy=wVAd9xCchDTuLaxPQ=wi0QEsw@mail.gmail.com%3e</a></p>