title: How to prevent abusive crawlers from crawling a rails app deployed on Heroku?
body: <p>I want to restrict the crawler access to my rails app running on Heroku. This would have been a straight forward task if I was using Apache OR nginX. Since the app is deployed on Heroku I am not sure how I can restrict access at the HTTP server level.</p>
  
  <p>I have tried to use robots.txt file, but the offending crawlers don't honor robot.txt.</p>
  
  <p>These are the solutions I am considering:</p>
  
  <p>1) A <code>before_filter</code> in the rails layer to restrict access.</p>
  
  <p>2) Rack based solution to restrict access</p>
  
  <p>I am wondering if there are any better ways to deal with this problem.</p>