title: funcparserlib: unexpected result
body: <p>I'm using <code>funcparserlib</code> for parsing.</p>
  <p>I cannot ignore comments and thus have to check for comments after each
  token. This is why I added a comment parser to each parser created with
  <code>make_parser()</code>. To keep this example easy, the comment is mandatory and not
  optional.</p>
  <p>Concerning the example below I have two questions:</p>
  <ol>
  <li>The output is</li>
  </ol>
  <pre><code>Token(&lt;TOK.foo: 1&gt;, 'foo')
  Token(&lt;TOK.comment: 2&gt;, '// comment')
  (Token(&lt;TOK.foo: 1&gt;, 'foo'), Token(&lt;TOK.comment: 2&gt;, '// comment'))
  (Token(&lt;TOK.foo: 1&gt;, 'foo'), Token(&lt;TOK.comment: 2&gt;, '// comment'))
  </code></pre>
  <p>whereas I expected it to be either</p>
  <pre><code>(Token(&lt;TOK.foo: 1&gt;, 'foo'), Token(&lt;TOK.comment: 2&gt;, '// comment'))
  (Token(&lt;TOK.foo: 1&gt;, 'foo'), Token(&lt;TOK.comment: 2&gt;, '// comment'))
  (Token(&lt;TOK.foo: 1&gt;, 'foo'), Token(&lt;TOK.comment: 2&gt;, '// comment'))
  </code></pre>
  <p>or</p>
  <pre><code>Token(&lt;TOK.foo: 1&gt;, 'foo')
  Token(&lt;TOK.comment: 2&gt;, '// comment')
  Token(&lt;TOK.foo: 1&gt;, 'foo')
  Token(&lt;TOK.comment: 2&gt;, '// comment')
  Token(&lt;TOK.foo: 1&gt;, 'foo')
  Token(&lt;TOK.comment: 2&gt;, '// comment')
  </code></pre>
  <p>How do I get the expected output?</p>
  <ol start="2">
  <li>Is there is a better way to check for comments after each token?</li>
  </ol>
  <p>Here is the example source code:</p>
  <pre class="lang-py prettyprint-override"><code>from enum import Enum, auto
  from funcparserlib.lexer import make_tokenizer
  from funcparserlib.parser import Parser, some
  
  #
  # lexer
  #
  class TOK (Enum):
      foo = auto()
      comment = auto()
      whitespace = auto()
  
  tokenizer = make_tokenizer([
      (TOK.whitespace, (r'[ \n\r\t]+', )),
      (TOK.foo, (r'\bfoo\b', )),
      (TOK.comment, (r'//.*', )),
  ])
  
  #
  # parser
  #
  def make_parser (tok):
      @Parser
      def func (toks, state):
          p = (some (lambda t:t.type == tok) +
               some (lambda t:t.type == TOK.comment))
          return p.run(toks, state)
      return func
  
  foo = make_parser (TOK.foo)
  
  @Parser
  def content_parser (toks, state):
      p = foo + foo + foo
      return p.run (toks, state)
  
  #
  # main
  #
  content = &quot;&quot;&quot;
  foo // comment
  foo // comment
  foo // comment
  &quot;&quot;&quot;
  
  token_stream = [tok for tok in tokenizer(content)
                  if tok.type != TOK.whitespace]
  result = content_parser.parse (token_stream)
  
  for i in result:
      print(i.__repr__())
  </code></pre>