title: Bot Traffic Identification Strategies
body: <p>Bots. Hate 'em.</p>
  
  <p><strong>What I need to accomplish</strong>
  Like everyone else, I want to count up the number of times pieces of content get featured or are displayed as links without those numbers being artificially inflated by web bots.</p>
  
  <p><strong>Why we can't just rely on Google Analytics</strong>
  GA does a nice job of validating the numbers they report, however they only report the main url, and not "related items" since "related items" is different per page-view/per user, we need to track those ourselves.</p>
  
  <p>GA is a good standard against which we can hold our numbers against, but that's it.</p>
  
  <p><strong>What I've done so far</strong></p>
  
  <ul>
  <li>Authenticated users are never hassled Firewall maintains IP Address
  blacklist </li>
  <li>Applications keep track of known bots </li>
  <li><strong>Nightly roll up jobs trawl our logs looking for the following signals:</strong></li>
  <li>Sustained bursts of requests  (high pages per second for more than x second) </li>
  <li>Blocks of requests blocks of IP addresses (x.y.z.245, .246, .247, .248, etc. cannot coincidentally be traipsing through our content at the same
  time) </li>
  <li>Pattern of landing page + requests for every page, in order, in rapid succession (humans rarely read every article, and not that quickly)</li>
  </ul>
  
  <p><strong>What I am looking for</strong>
  Not vague advice, but actionable algorithms, or best practices, or articles which describe how approaches were actually implemented, with at least some pseudo-code snippets. I don't expect to get a silver bullet, but I know there are ways to approach this problem that I have not seen. I just need to see a good white paper or something.</p>
  
  <p><strong>What I have seen a million times</strong></p>
  
  <ul>
  <li>"our company implements a multi-tiered approach with browser
  challenges and backend analytics, and blah blah blah" Sounds great, i'm sure the investors loved it, how about an actual example?</li>
  <li>So tired of CIO-Speak.  "we put hidden fields on our forms, it's
  called a honeypot!"</li>
  </ul>
  
  <p><strong>Anything actually actionable would be most gratefully appreciated!!!</strong></p>