title: Why does Nutch (v2.3) crawl only the seed URL, instead of crawling an entire website?
body: <p>I am trying to crawl an entire, specific website (ignoring external links) using <strong>Nutch 2.3</strong> with <strong>HBase 0.94.14</strong>.</p>
  
  <p>I have followed a step-by-step tutorial (can find it <a href="https://anil.io/blog/apache/nutch/apache-nutch-2-3-hbase-0-94-14-and-solr-5-2-1-tutorial/" rel="nofollow noreferrer">here</a>) on how to set up and use these tools. However, I haven't been able to achieve my goal. Instead of crawling the entire website whose URL I've written in the seed.txt file, Nutch only retrieves that base URL in the first round. I need to run further crawls in order for Nutch to retrieve more URLs.</p>
  
  <p>The problem is I don't know how many rounds I need in order to crawl the entire website, so I need a way to tell Nutch to "keep crawling until the entire website has been crawled" (in other words, "crawl the entire website in a single round").</p>
  
  <p>Here are the key steps and settings I have followed so far:</p>
  
  <ol>
  <li><p>Put base URL in the <strong>seed.txt</strong> file.</p>
  
  <p><code>http://www.whads.com/</code></p></li>
  </ol>
  
  <hr>
  
  <ol start="2">
  <li><p>Set up Nutch's <strong>nutch-site.xml</strong> configuration file. After finishing the tutorial, I added a few more properties following suggestions on other StackOverflow questions (none of them, however, seem to have solved the problem for me).</p>
  
  <pre><code>&lt;?xml version="1.0"?&gt;
  &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt;
  
  &lt;!-- Put site-specific property overrides in this file. --&gt;
  &lt;configuration&gt;
          &lt;property&gt;
              &lt;name&gt;http.agent.name&lt;/name&gt;
              &lt;value&gt;test-crawler&lt;/value&gt;
          &lt;/property&gt;
          &lt;property&gt;
              &lt;name&gt;storage.data.store.class&lt;/name&gt;
              &lt;value&gt;org.apache.gora.hbase.store.HBaseStore&lt;/value&gt;
          &lt;/property&gt;
          &lt;property&gt;
              &lt;name&gt;plugin.includes&lt;/name&gt;
              &lt;value&gt;protocol-httpclient|urlfilter-regex|index-(basic|more)|query-(basic|site|url|lang)|indexer-solr|nutch-extensionpoints|protocol-httpclient|urlfilter-regex|parse-(text|html|msexcel|msword|mspowerpoint|pdf)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)protocol-http|urlfilter-regex|parse-(html|tika|metatags)|index-(basic|anchor|more|metadata)&lt;/value&gt;
          &lt;/property&gt;
          &lt;property&gt;
              &lt;name&gt;db.ignore.external.links&lt;/name&gt;
              &lt;value&gt;true&lt;/value&gt;
          &lt;/property&gt;
          &lt;property&gt;
              &lt;name&gt;db.ignore.internal.links&lt;/name&gt;
              &lt;value&gt;false&lt;/value&gt;
          &lt;/property&gt;
          &lt;property&gt;
              &lt;name&gt;fetcher.max.crawl.delay&lt;/name&gt;
              &lt;value&gt;-1&lt;/value&gt;
          &lt;/property&gt;
          &lt;property&gt;
              &lt;name&gt;fetcher.threads.per.queue&lt;/name&gt;
              &lt;value&gt;50&lt;/value&gt;
              &lt;description&gt;&lt;/description&gt;
          &lt;/property&gt;
          &lt;property&gt; 
              &lt;name&gt;generate.count.mode&lt;/name&gt; 
              &lt;value&gt;host&lt;/value&gt;
          &lt;/property&gt;
          &lt;property&gt; 
              &lt;name&gt;generate.max.count&lt;/name&gt; 
              &lt;value&gt;-1&lt;/value&gt;
          &lt;/property&gt;
  &lt;/configuration&gt;
  </code></pre></li>
  </ol>
  
  <hr>
  
  <ol start="3">
  <li><p>Added "accept anything else" rule to Nutch's <strong>regex-urlfilter.txt</strong> configuration file, following suggestions on StackOverflow and Nutch's mailing list.</p>
  
  <pre><code># Already tried these two filters (one at a time, 
  # and each one combined with the 'anything else' one)
  #+^http://www.whads.com
  #+^http://([a-z0-9]*.)*whads.com/
  
  # accept anything else
  +.
  </code></pre></li>
  </ol>
  
  <hr>
  
  <ol start="4">
  <li><p><strong>Crawling</strong>: I have tried using two different approaches (both yielding the same result, with only one URL generated and fetched on the first round):</p>
  
  <ul>
  <li><p>Using <code>bin/nutch</code> (following the tutorial):</p>
  
  <pre><code>bin/nutch inject urls
  bin/nutch generate -topN 50000
  bin/nutch fetch -all
  bin/nutch parse -all
  bin/nutch updatedb -all
  </code></pre></li>
  <li><p>Using <code>bin/crawl</code>:</p>
  
  <pre><code>bin/crawl urls whads 1
  </code></pre></li>
  </ul></li>
  </ol>
  
  <hr>
  
  <p>Am I still missing something? Am I doing something wrong? Or is it that Nutch can't crawl an entire website in one go?</p>
  
  <p>Thank you so much in advance!</p>