title: guide on crawling the entire web?
body: <p>i just had this thought, and was wondering if it's possible to crawl the entire web (just like the big boys!) on a single dedicated server (like Core2Duo, 8gig ram, 750gb disk 100mbps)
  .</p>
  
  <p>I've come across a paper where this was done....but i cannot recall this paper's title. it was like about crawling the entire web on a single dedicated server using some statistical model.</p>
  
  <p>Anyways, imagine starting with just around 10,000 seed URLs, and doing exhaustive crawl....</p>
  
  <p>is it possible ? </p>
  
  <p>I am in need of crawling the web but limited to a dedicated server. how can i do this, is there an open source solution out there already ?</p>
  
  <p>for example see this real time search engine. <a href="http://crawlrapidshare.com" rel="noreferrer">http://crawlrapidshare.com</a> the results are exteremely good and freshly updated....how are they doing this ?</p>