title: What's a good Web Crawler tool
body: <p>I need to index a whole lot of webpages, what good webcrawler utilities are there?  I'm preferably after something that .NET can talk to, but that's not a showstopper.</p>
  
  <p>What I really need is something that I can give a site url to &amp; it will follow every link and store the content for indexing.</p>