title: ElasticSearch and Nutch integration
body: <p>I have a question about ElasticSearch and Apache Nutch integration. </p>
  
  <p>I've tried to follow the Nutch+ES guides listed here</p>
  
  <p><a href="https://gist.github.com/xrstf/b48a970098a8e76943b9" rel="nofollow">https://gist.github.com/xrstf/b48a970098a8e76943b9</a></p>
  
  <p><a href="https://qbox.io/blog/scraping-the-web-with-nutch-for-elasticsearch" rel="nofollow">https://qbox.io/blog/scraping-the-web-with-nutch-for-elasticsearch</a></p>
  
  <p>and</p>
  
  <p><a href="http://www.aossama.com/search-engine-with-apache-nutch-mongodb-and-elasticsearch/" rel="nofollow">http://www.aossama.com/search-engine-with-apache-nutch-mongodb-and-elasticsearch/</a></p>
  
  <p>Specifically, I was able to set up HDFS+HBase+Nutch and crawl a list of pages (I can confirm that the data is indeed crawled and stored in HBase's webpage table).</p>
  
  <p>I can also get ElasticSearch and Kibana to work with no problem (successfully created a new index and insert some records, confirmed working with Kibana)</p>
  
  <p>However, I couldn't get their <strong>combination</strong> to work. Basically, I after I did the following command on Nutch:</p>
  
  <pre><code>nutch inject &lt;file_containing_url&gt;
  nutch generate -topN 1
  nutch fetch -all
  nutch parse -all
  nutch updatedb -all
  </code></pre>
  
  <p>Now, I want to index the fetched data into ES, and I followed the guide:</p>
  
  <pre><code>nutch index elasticsearch -all
  </code></pre>
  
  <p>However, after this command has finished execution, nothing is changed in ElasticSearch.  The log under runtime/local/logs shows:</p>
  
  <pre><code>elastic.ElasticIndexWriter - Processing remaining requests [docs = 0, length = 0, total docs = 0]
  </code></pre>
  
  <p>Which makes me believe ES is not asked to index anything at all by nutch.</p>
  
  <p>Where did I do wrong in the entire process? Do I have to specify something about ES to nutch?</p>
  
  <p>FYI: here are some OS/tools details:
  CentOS 6.7, HBase 1.1.3 ElasticSearch 1.7.5, Nutch 2.3.1.
  I have modified nutch-site.xml to include ES's properties:</p>
  
  <pre><code>    &lt;property&gt;
          &lt;name&gt;plugin.includes&lt;/name&gt;
          &lt;!-- do **NOT** enable the parse-html plugin, if you want proper HTML parsing. Use something like parse-tika! --&gt;
          &lt;value&gt;protocol-httpclient|urlfilter-regex|parse-(text|tika|js)|index-(basic|anchor|more)|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)|indexer-elastic&lt;/value&gt;
      &lt;/property&gt;
      &lt;property&gt;
          &lt;name&gt;elastic.host&lt;/name&gt;
          &lt;value&gt;10.5.140.112&lt;/value&gt; &lt;!-- where is ElasticSearch listening --&gt;
      &lt;/property&gt;
        &lt;property&gt;
          &lt;name&gt;elastic.cluster&lt;/name&gt;
          &lt;value&gt;nutch&lt;/value&gt;
        &lt;/property&gt;
  
        &lt;property&gt;
          &lt;name&gt;elastic.index&lt;/name&gt;
          &lt;value&gt;nutch&lt;/value&gt;
        &lt;/property&gt;
  
        &lt;property&gt;
          &lt;name&gt;elastic.port&lt;/name&gt;
          &lt;value&gt;9300&lt;/value&gt;
        &lt;/property&gt;
  </code></pre>
  
  <p>Thank you all so much for the help!</p>