title: How to exclude all robots except Googlebot and Bingbot with both robots.txt and X-Robots-Tag
body: <p>I have 2 questions regarding crawlers and robots.</p>
  
  <h3>Background info</h3>
  
  <p>I only want Google and Bing to be excluded from the “disallow” and “noindex” limitations. In other words, I want ALL search engines except Google and Bing to follow the “disallow” and “noindex” rules. In addition, I would also like a “nosnippet” function for the search engines I mentioned (which all support “nosnippet”). Which code do I use to do this (using both robots.txt and X-Robots-Tag)?</p>
  
  <p>I want to have it in both the robots.txt file as well as the htacess file as an X-Robots-Tag. I understand that robots.txt may be outdated, but I would like clear instructions to crawlers even if they’re considered “ineffective” and “outdated” unless you think otherwise.</p>
  
  <h3>Question 1</h3>
  
  <p>Did I get the following code right to only allow Google and Bing to index (to prevent other search engines from showing in their results), and, furthermore, prevent Bing and Google from showing snippets in their search results?</p>
  
  <p><strong>X-Robots-Tag code</strong> (Is this correct? Don't think I need to add "index" to googlebot and bingbot due to "index" being a default value, but not sure.)</p>
  
  
  
  <pre class="lang-none prettyprint-override"><code>X-Robots-Tag: googlebot: nosnippet
  X-Robots-Tag: bingbot: nosnippet
  X-Robots-Tag: otherbot: noindex
  </code></pre>
  
  <p><strong>robots.txt code</strong> (Is this correct? I think the 1st one is, but not sure.)</p>
  
  <pre class="lang-none prettyprint-override"><code>    User-agent: Googlebot
      Disallow:
      User-agent: Bingbot
      Disallow:
      User-agent: *
      Disallow: /
  </code></pre>
  
  <p>or</p>
  
  <pre class="lang-none prettyprint-override"><code>    User-agent: *
      Disallow: /
      User-agent: Googlebot
      Disallow:
      User-agent: Bingbot
      Disallow:
  </code></pre>
  
  <h3>Question 2: Conflicts between robots.txt and X-Robots-Tag</h3>
  
  <p>I anticipate conflicts between the robots.txt and the X-Robots-Tag due to the disallow function and the noindex functions not being allowed to work in conjunction (<a href="https://stackoverflow.com/questions/35639040/is-there-any-advantage-of-using-x-robot-tag-instead-of-robots-txt">Is there any advantage of using X-Robot-Tag instead of robots.txt?</a> ). How do I get around this, and what is your recommendation? </p>
  
  <h3>End goal</h3>
  
  <p>As mentioned, the main goal of this is to explicitly tell all older robots (still using the robots.txt) and all the newer ones except Google and Bing (using X-Robots-Tag) to not show any of my pages in their search results (which I'm assuming is summed up in the noindex function). I understand they may not all follow it, but I want them ALL to know except Google and Bing to not show my pages in search results. To this end, I am looking to find the right codes for both the robots.txt code and X-Robots-Tag code that will work without conflict for this function for the HTML sites I am trying to build.</p>