title: Can domains be used in robots.txt?
body: <p>We have a development server at dev.example.com that is being indexed by Google. We are using AWS Lightsail to duplicate the development server to our production environment in totality — the same robots.txt file is used on both dev.example.com and example.com.</p>
  <p><a href="https://developers.google.com/search/docs/advanced/robots/create-robots-txt#create_rules" rel="nofollow noreferrer">Google's robots.txt documentation</a> doesn't explicitly state whether root domains can be defined. Can I implement domain specific rules to the robots.txt file? For example, is this acceptable:</p>
  <pre><code>User-agent: *
  Disallow: https://dev.example.com/
  
  User-agent: *
  Allow: https://example.com/
  
  Sitemap: https://example.com/sitemap.xml
  </code></pre>
  <p>To add, this can be resolved through .htaccess rewrite engine — my question is specifically about robots.txt.</p>