title: Regular expression to parse robots.txt
body: <p>I have the following robots.txt as an example -</p>
  
  <pre>
  User-agent: googlebot
  User-agent: slurp
  User-agent: msnbot
  User-agent: teoma
  User-agent: W3C-checklink
  User-agent: WDG_SiteValidator
  Disallow: /
  Disallow: /js/
  Disallow: /Web_References/
  Disallow: /webresource.axd
  Disallow: /scriptresource.axd
  
  User-agent: Mediapartners-Google*
  Disallow:
  
  User-agent: *
  Disallow: /webresource.axd
  Disallow: /scriptresource.axd
  Disallow: /js/
  Disallow: /Web_References/
  </pre>
  
  <p>I may be asking too much of regex but I'm wanting to write an expression which will return matches in the following grouped and ordered fashion -</p>
  
  <pre>
  Matches
   - [0]
     - [UserAgents]
        - "googlebot"
        - "slurp"
        - "msnbot"
        - "teoma"
        - "W3C-checklink"
        - "WDG_SiteValidator"
      - [Routes]
        - [0]
          - [Permission] "Allow"
          - [Url] "/"
        - [1]
          - [Permission] "Disallow"
          - [Url] "/js/"
        - [2]
          - [Permission] "Disallow"
          - [Url] "/Web_References/"
  
  ...
  
  etc
  
  ...
  </pre>
  
  <p>I've written individual expressions to match elements of the document, however I can't get them to work when pieced together. Maybe someone can point out where I'm going wrong?</p>
  
  <p><strong>Patterns</strong></p>
  
  <p>User agents: <code>(?:user-agent:\s*)(?&lt;UserAgent&gt;[a-z_0-9-*]*)</code></p>
  
  <p>Permissions: <code>(?&lt;Permission&gt;(?:allow|disallow))(?:\s*:\s*)(?&lt;Url&gt;[/0-9_a-z.]*)</code></p>
  
  <p><strong>My attempt</strong></p>
  
  <p><code>((?&lt;UserAgents&gt;(?:user-agent:\s*)(?&lt;UserAgent&gt;[a-z_0-9-*]*))+(?&lt;Routes&gt;(?&lt;Permission&gt;(?:allow|disallow))(?:\s*:\s*)(?&lt;Url&gt;[/0-9_a-z.]*))+)+</code></p>
  
  <p>FYI, I'm using Expresso to debug these scripts and have the following checked - Multiline, Compiled and Ignore Case</p>