title: How to handle 500 Internal Server Error and 404 Page Not Found with Anemone, Boilerpipe and Nokigiri
body: <p>I'm implementing a tool that needs to crawl a website. I'm using anemone to crawl and on each anemone's page I'm using boilerpipe and Nokogiri to manage HTML format, etc.</p>
  
  <p>My problem is: if I get 500 Internal Server Error, it makes Nokogiri fail because there is no page.</p>
  
  <pre><code>Anemone.crawl(name) do |anemone|
     anemone.on_every_page do |page|
         if not (page.nil? &amp;&amp; page.not_found?)
                result = Boilerpipe.extract(page.url, {:output =&gt; :htmlFragment, :extractor =&gt; :ArticleExtractor})
                doc = Nokogiri::HTML.parse(result)
  
         end
      end
  end
  </code></pre>
  
  <p>In the case above, if there is a 500 Internal Server Error, the application will give an error on Nokogiri::HTML.parse(). I want to avoid this problem. If the server gives an error I want to continue computation ignoring this page.</p>
  
  <p>There is any way to handle 500 Internal Server Error and 404 Page Not Found with these tools?</p>
  
  <p>Kind regards,
  Hugo</p>