title: Interview question: Honeypots and web crawlers
body: <p>I was recently reading a book as prep for an interview and came across the following question:</p>
  
  <p>What will you do when your crawler runs into a honey pot that generates an infinite subgraph for you to wander about?</p>
  
  <p>I wanted to get some solutions to this qn. Personally, I would some form of depth limited search to prevent traversing continuously. Or perhaps use some form of machine learning to detect patterns. Thoughts?</p>