title: Correctly redirect bot requests to static version of a website
body: <p>I'm having problems getting my website to index correctly by Google.</p>
  
  <h1>My folder structure looks like this:</h1>
  
  <pre><code>root
   - cms
   - www
  </code></pre>
  
  <p>example.com points to the root where a .htaccess routes all requests to /www:</p>
  
  <pre><code>RewriteEngine on
  RewriteRule ^(.*)$ /www/$1 [L]
  </code></pre>
  
  <h1>Front end</h1>
  
  <p>The Angular front end inside /www gets data from /cms via REST api. So far so good.</p>
  
  <p>What I want to achieve is that bots don't crawl inside my ajaxified /www page but instead inside /cms where I print out static contents corresponding to the URL structure in /www.</p>
  
  <h1>URL for static content:</h1>
  
  <p>/www/test1 -> Outputs nice content via REST</p>
  
  <p>/cms/test1 -> Outputs text-only content for the crawler</p>
  
  <h1>Bot redirect</h1>
  
  <p>I'm redirecting the bots coming to example.com/www to /cms like this:</p>
  
  <pre><code>RewriteCond %{HTTP_USER_AGENT} (googlebot|yahoo|bingbot|baiduspider) [NC]
  RewriteRule ^(.*)$ http://www.example.com/cms/$1 [R=301,L]
  </code></pre>
  
  <h1>Site map</h1>
  
  <p>I also registered a sitemap with Google with the following contents:</p>
  
  <pre><code>http://www.example/test1
  http://www.example/test2
  and so on...
  </code></pre>
  
  <h1>The problem</h1>
  
  <p>This all works fine BUT: Google is also crawling the static contents inside /cms without being redirected there by me. I only want this static subdomain to be fed through the redirect but not when Google's bot is searching for it itself. Kind of "disallowing" the bot to crawl here - but in the other hand I NEED it to crawl it. A catch 22 in my opinion.</p>
  
  <h1>Edit: complete .htaccess file</h1>
  
  <pre><code>RewriteEngine On
  
  # Sitemap
  RewriteRule ^sitemap(-+([a-zA-Z0-9_-]+))?\.xml(\.gz)?$ /cms/sitemap$1.xml$2 [L]
  RewriteRule ^sitemap(-+([a-zA-Z0-9_-]+))?\.html(\.gz)?$ /cms/sitemap$1.xml$2 [L]
  
  # Redirect bots to static pages
  RewriteCond %{HTTP_USER_AGENT} (googlebot|yahoo|bingbot|baiduspider) [NC]
  RewriteRule ^(.*)$ http://www.example.com/cms/$1 [R=301,L]
  
  # Angular HTML5 mode: Don't rewrite files or directories
  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !index
  
  # Angular HTML5 mode: Rewrite everything else to index.html to allow html5 state links
  RewriteRule (.*) /www/index.html [L]
  </code></pre>
  
  <h1>Edit 2</h1>
  
  <p>I have added this tag to the www page</p>
  
  <pre><code>&lt;meta name="fragment" content="!"&gt; 
  </code></pre>
  
  <p>to let the crawler know there's AJAX being used on the page. And I'm using the rewrite suggest by @Croises but in reaction to Google's _escaped_fragment_ re-request. Let's wait a few days...</p>
  
  <pre><code>RewriteCond %{HTTP_USER_AGENT} (googlebot|yahoo|bingbot|baiduspider) [NC]
  RewriteCond %{QUERY_STRING} _escaped_fragment_
  RewriteCond %{REQUEST_URI} !^/cms/
  RewriteRule ^(.*)$ cms/$1 [L]
  </code></pre>