title: JavaCC lexer doesn't work as expected (whitespace not ignored)
body: <p>I'm trying to implement a parser for the example file listed below. I'd like to recognize quoted strings with <code>'+'</code> between them as a single token. So I created a jj file, but it doesn't match such strings. I was under the impression that JavaCC is supposed to match the longest possible match for each token spec. But that doesn't seem to be case for me.</p>
  
  <p>What am I doing wrong here? Why isn't my <code>&lt;STRING&gt;</code> token matching the <code>'+'</code> even though it's specified in there? Why is whitespace not being ignored?</p>
  
  <pre><code>options {
    TOKEN_FACTORY = "Token";
  }
  
  PARSER_BEGIN(Parser)
  
  package com.example.parser;
  
  public class Parser {
  
    public static void main(String args[]) throws ParseException {
  
        ParserTokenManager manager = new ParserTokenManager(new SimpleCharStream(Parser.class.getResourceAsStream("example")));
        Token token = manager.getNextToken();
        while (token != null &amp;&amp; token.kind != ParserConstants.EOF) {
            System.out.println(token.toString() + "[" + token.kind + "]");
            token = manager.getNextToken();
        }
  
        Parser parser = new Parser(Parser.class.getResourceAsStream("example"));
        parser.start();
    }
  
  }
  
  PARSER_END(Parser)
  
  // WHITE SPACE
  &lt;DEFAULT, IN_STRING_KEYWORD&gt;
  SKIP :
  {
    " " // &lt;-- skipping spaces
  | "\t"
  | "\n"
  | "\r"
  | "\f"
  }
  
  // TOKENS
  TOKEN :
  {
  &lt; KEYWORD1 : "keyword1" &gt; : IN_STRING_KEYWORD
  }
  
  &lt;IN_STRING_KEYWORD&gt;
  TOKEN : {&lt;STRING : &lt;CONCAT_STRING&gt; | &lt;UNQUOTED_STRING&gt; &gt; : DEFAULT 
  | &lt;#CONCAT_STRING : &lt;QUOTED_STRING&gt; ("+" &lt;QUOTED_STRING&gt;)+ &gt;
  // &lt;-- CONCAT_STRING never matches   "+" part when input is "'smth' +", because whitespace is not ignored!?
  | &lt;#QUOTED_STRING : &lt;SINGLEQUOTED_STRING&gt; | &lt;DOUBLEQUOTED_STRING&gt; &gt;
  | &lt;#SINGLEQUOTED_STRING : "'" (~["'"])* "'" &gt;
  | &lt;#DOUBLEQUOTED_STRING : 
      "\""
        (
          (~["\"", "\\"]) |
          ("\\" ["n", "t", "\"", "\\"])
        )* 
      "\""
    &gt;
  | &lt;#UNQUOTED_STRING : (~[" ","\t", ";", "{", "}", "/", "*", "'", "\"", "\n", "\r"] | "/" ~["/", "*"] | "*" ~["/"])+ &gt;
  }
  
  void start() :
  {}
  {
    (&lt;KEYWORD1&gt;&lt;STRING&gt;";")+ &lt;EOF&gt;
  }
  </code></pre>
  
  <p>Here's an example file that should get parsed:</p>
  
  <pre><code>keyword1 "foo" + ' bar';
  </code></pre>
  
  <p>I'd like to match the argument of the first <code>keyword1</code> as a single <code>&lt;STRING&gt;</code> token.</p>
  
  <p>Current output:</p>
  
  <pre><code>keyword1[6]
  Exception in thread "main" com.example.parser.TokenMgrError: Lexical error at line 1, column 15.  Encountered: " " (32), after : "\"foo\""
      at com.example.parser.ParserTokenManager.getNextToken(ParserTokenManager.java:616)
      at com.example.parser.Parser.main(Parser.java:12)
  </code></pre>
  
  <p>I'm using JavaCC 5.0.</p>