title: Inconsistent token handling in ANTLR4
body: <p>The ANTLR4 book references a multi-mode example</p>
  
  <p><a href="https://github.com/stfairy/learn-antlr4/blob/master/tpantlr2-code/lexmagic/ModeTagsLexer.g4" rel="nofollow">https://github.com/stfairy/learn-antlr4/blob/master/tpantlr2-code/lexmagic/ModeTagsLexer.g4</a></p>
  
  <pre><code>lexer grammar ModeTagsLexer;
  
  // Default mode rules (the SEA)
  OPEN  : '&lt;'     -&gt; mode(ISLAND) ;       // switch to ISLAND mode
  TEXT  : ~'&lt;'+ ;                         // clump all text together
  
  mode ISLAND;
  CLOSE : '&gt;'     -&gt; mode(DEFAULT_MODE) ; // back to SEA mode 
  SLASH : '/' ;
  ID    : [a-zA-Z]+ ;                     // match/send ID in tag to parser
  </code></pre>
  
  <p><a href="https://github.com/stfairy/learn-antlr4/blob/master/tpantlr2-code/lexmagic/ModeTagsParser.g4" rel="nofollow">https://github.com/stfairy/learn-antlr4/blob/master/tpantlr2-code/lexmagic/ModeTagsParser.g4</a></p>
  
  <pre><code>parser grammar ModeTagsParser;
  
  options { tokenVocab=ModeTagsLexer; } // use tokens from ModeTagsLexer.g4
  
  file: (tag | TEXT)* ;
  
  tag : '&lt;' ID '&gt;'
      | '&lt;' '/' ID '&gt;'
      ;
  </code></pre>
  
  <p>I'm trying to build on this example, but using the <code>«</code> and <code>»</code> characters for delimiters. If I simply substitute I'm getting error 126</p>
  
  <p><code>cannot create implicit token for string literal in non-combined grammar: '«'</code></p>
  
  <p>In fact, this seems to occur as soon as I have the <code>«</code> character in the parser <code>tag</code> rule.</p>
  
  <pre><code>tag : '«' ID '&gt;';
  </code></pre>
  
  <p>with</p>
  
  <pre><code>OPEN    : '«'   -&gt; pushMode(ISLAND);
  TEXT    : ~'«'+;
  </code></pre>
  
  <p>Is there some antlr foo I'm missing? This is using <code>antlr4-maven-plugin</code> <code>4.2</code>.</p>
  
  <p>The wiki mentions something along these lines, but the way I read it that's contradicting the example on github and anecdotal experience when using <code>&lt;</code>. See "Redundant String Literals" at <a href="https://theantlrguy.atlassian.net/wiki/display/ANTLR4/Lexer+Rules" rel="nofollow">https://theantlrguy.atlassian.net/wiki/display/ANTLR4/Lexer+Rules</a></p>