title: Is there a list of known web crawlers?
body: <p>I'm trying to get accurate download numbers for some files on a web server. I look at the user agents and some are clearly bots or web crawlers, but many for many I'm not sure, they may or may not be a web crawler and they are causing many downloads so it's important for me to know.</p>
  
  <p>Is there somewhere a list of know web crawlers with some documentation like user agent, IPs, behavior, etc?</p>
  
  <p>I'm not interested in the official ones, like Google's, Yahoo's, or Microsoft's. Those are generally well behaved and self-indentified.</p>