title: ANTLR3: match exactly one character in token
body: <p>I am a newbie on ANTLR and working on a parser using ANTLR3, but having trouble with the following situation. In the text we parse there can be multiple situation where the ^-character occurs. However, there is one special case where '^' is followed by exactly one character. This occurs in strings:</p>
  
  <ol>
  <li>'MyText'^M</li>
  <li>^MyValue</li>
  </ol>
  
  <p>In the first situation '^M' is part of a string where ^M indicates 13 hex, but in the second it is not; there it is a Pointer indicator. The second situation is captured in the grammar rules (the ^-character is used in multiple rules).</p>
  
  <p>If I solve it with the following tokens, it will fail, because '^MyValue' is tokenized in '^M' and 'yValue'. However, I want the token ControlChar only be used if there is exactly one character following ^. Otherwise it should be ignored and not tokenized so it can be used in the grammar.</p>
  
  <pre><code>Pointer                 : '^'
                          ;
  QuotedString            : '\'' ('\'\'' | ~('\''))* '\''
                          ;
  TkIdentifier            : (Alpha | '_') (Alpha | Digit | '_')*
                          ;
  ControlString           : Controlchar (Controlchar)*
                          ;
  fragment
  Controlchar             : '#' Digitseq
                          | '#' '$' Hexdigitseq
                          | '^' Alpha
                          ;
  fragment
  Alpha                   : 'a'..'z'
                          | 'A'..'Z'
                          ;
  fragment
  Digit                   : '0'..'9'
                          ;
  </code></pre>
  
  <p>So, my question is. How can I instruct ANTLR that <code>'^' Alpha</code> is only matched if there is exactly one Alpha following this character and otherwise leave '^' in the text and tokenize the Alpha, Digits or '_' as a TkIdentifier token?</p>
  
  <p>For example, the lexer should create the following tokens:</p>
  
  <pre><code>^Foo -&gt; Pointer TkIdentifier
  ^F oo -&gt; ControlChar TkIdentifier
  ^ F oo -&gt; Pointer TkIdentifier TkIdentifier
  Foo^M -&gt; TkIdentifier ControlChar
  Foo ^ M -&gt; TkIdentifier Pointer TkIdentifier
  Foo ^M -&gt; TkIdentifier ControlChar
  Foo^ M -&gt; TkIdentifier Pointer TkIdentifier
  
  'Text'^M -&gt; QuotedString ControlChar
  'Text' ^M -&gt; QuotedString ControlChar
  'Text' ^ M -&gt; QuotedString Pointer TkIdentifier
  ^M'Text' -&gt; ControlChar QuotedString
  ^M 'Text' -&gt; ControlChar QuotedString
  ^ M'Text' -&gt; Pointer TkIdentifier QuotedString
  </code></pre>