title: Specifying variables in robots.txt
body: <p>My URL structure is set up in two parallels (both lead to the same place ):</p>
  
  <ul>
  <li><code>www.example.com/subname</code></li>
  <li><code>www.example.com/123</code></li>
  </ul>
  
  <p>The trouble is is that, the spiders are crawling into things like:</p>
  
  <ul>
  <li><code>www.example.com/subname/default_media_function</code></li>
  <li><code>www.example.com/subname/map_function</code></li>
  </ul>
  
  <p>Note that the name "subname" represents thousands of different pages on my site that all have that same function.</p>
  
  <p>And they are throwing out errors because those links are strictly for JSON or AJAX purposes and not actual links. I would like to block them from accessing those pages, but how would I do that if the URL contains a variable?</p>
  
  <p>Would this work in robots.txt?</p>
  
  <pre><code>Disallow: /map_function
  </code></pre>