title: Nutch can crawl all website.Is there any rule to crawl specific site.Does need permission before start crawling from specific site.?
body: <p>Using crawler we can search any specific site but this will slow the bandwidth of that site.Is there any rule before crawl specific site or using nutch we can search any site without an issue.I would like  to create vertical search using Nutch.Can anyone help me out from this question on the basis of above question section?</p>
  
  <p>If specific site have not allowed Nutch bot in robots.txt then how can search that site using Nutch?Do we need to take permission before.</p>