title: Web Crawler - Ignore Robots.txt file?
body: <p>Some servers have a robots.txt file in order to stop web crawlers from crawling through their websites. Is there a way to make a web crawler ignore the robots.txt file? I am using Mechanize for python.</p>