title: Googlebot and Bingbot crawling DNN site
body: <p>I have a DNN site with over 20,000 pages.  The Googlebot and Bingbot are consistently crawling my website.</p>
  
  <p>When I look at my sitelog I can see that google and bing are crawling my site via the pageid (ex: www.url.com/Default.aspx?TabID=5000)</p>
  
  <p>The bots are hitting my website every minute.  When I add new page, I am expecting the bots to crawl the new added page, instead I see the bots re-crawling very old pages and will take a couple of hours before it recognizes the newly added page.</p>
  
  <p>I have robot.txt file with over 10,000 entries that have the following defenitions:</p>
  
  <p>Disallow:/Default.aspx?TabID=5000<br>
  Disallow:/Default.aspx?TabID=5001<br>
  Disallow:/Default.aspx?TabID=5002</p>
  
  <p>and so forth.  </p>
  
  <p>So I am noticing a couple of issues:</p>
  
  <p>1 - Googlebot and Bingbot are ignoring my disallows and are recrawling pages that I have defined in the robots.txt - how does the bot know to go back and recrawl old pages, using the TabID?</p>
  
  <p>2 - I still notice that when I add a new page, both bots are busy crawling old content, and do not immediately read my new content, is there a way to force Google and Bing bots to always read newly added pages first?</p>
  
  <p>thank you in advance for any suggestions.</p>