title: Nutch : Crawl Broken Links & Index it in Solr
body: <p>My purpose is to find how many URLs in an HTML page are invalid (404, 500, HostNotFound). So in Nutch is there a config change that we can do through which the web crawler crawls through broken links and indexes it in solr.</p>
  
  <p>Once all the broken links &amp; valid links are indexed in Solr I can just check the URLs which are invalid and can remove it from my HTML page.</p>
  
  <p>Any help will be highly appreciated.</p>
  
  <p>Thanks in advance.</p>