title: Robots.txt file in MVC.NET 4
body: <p>I have read an article about ignoring the robots from some url in my ASP MVC.NET project. In his article author said that we should add some action in some off controllers like this. In this example he adds the action to the Home Controller:</p>
  
  <pre><code>#region -- Robots() Method --
  public ActionResult Robots()
  {
      Response.ContentType = "text/plain";
      return View();
  }
  #endregion
  </code></pre>
  
  <p>then we should add a Robots.cshtml file in our project with this body</p>
  
  <pre><code>@{
      Layout = null;
  }
  # robots.txt for @this.Request.Url.Host
  
  User-agent: *
  Disallow: /Administration/
  Disallow: /Account/
  </code></pre>
  
  <p>and finally we should add this line of code to the <code>Gloabal.asax</code></p>
  
  <pre><code>routes.MapRoute("Robots.txt",
                  "robots.txt",
                  new { controller = "Home", action = "Robots" });
  </code></pre>
  
  <p>my question is that do robots crawl the controllers which has [Authorization] attribute like <code>Administration</code>?</p>