title: Matching lexeme variants with Antlr3
body: <p>I'm trying to match measurements in English input text, using Antlr 3.2 and Java1.6. I've got lexical rules like the following:</p>
  
  <pre><code>fragment
  MILLIMETRE
      :   'millimetre' | 'millimetres'
      |   'millimeter' | 'millimeters'
      |   'mm'
      ;
  
  MEASUREMENT
      :   MILLIMETRE | CENTIMETRE | ... ;
  </code></pre>
  
  <p>I'd like to be able to accept any combination of upper- and lowercase input and - more importantly - just return a single lexical token for all the variants of MILLIMETRE. But at the moment, my AST contains 'millimetre', 'millimeters', 'mm' etc. just as in the input text.</p>
  
  <p>After reading <a href="http://www.antlr.org/wiki/pages/viewpage.action?pageId=1802308" rel="nofollow">http://www.antlr.org/wiki/pages/viewpage.action?pageId=1802308</a>, I think I need to do something like the following:</p>
  
  <pre><code>tokens {
      T_MILLIMETRE;
  }
  
  fragment
  MILLIMETRE
      :   ('millimetre' | 'millimetres'
      |   'millimeter' | 'millimeters'
      |   'mm') { $type = T_MILLIMETRE; }
      ;
  </code></pre>
  
  <p>However, when I do this, I get the following compiler errors in the Java code generated by Antlr:</p>
  
  <pre><code>cannot find symbol
  _type = T_MILLIMETRE;
  </code></pre>
  
  <p>I tried the following instead:</p>
  
  <pre><code>MEASUREMENT
      :   MILLIMETRE  { $type = T_MILLIMETRE; }
      |   ...
  </code></pre>
  
  <p>but then MEASUREMENT is not matched anymore.</p>
  
  <p>The more obvious solution with a rewrite rule:</p>
  
  <pre><code>MEASUREMENT
      :   MILLIMETRE  -&gt; ^(T_MILLIMETRE MILLIMETRE)
      |   ...
  </code></pre>
  
  <p>causes an NPE:</p>
  
  <pre><code>java.lang.NullPointerException at org.antlr.grammar.v2.DefineGrammarItemsWalker.alternative(DefineGrammarItemsWalker.java:1555).
  </code></pre>
  
  <p>Making MEASUREMENT into a parser rule gives me the dreaded "The following token definitions can never be matched because prior tokens match the same input" error.</p>
  
  <p>By creating a parser rule</p>
  
  <pre><code>measurement :  T_MILLIMETRE | ...
  </code></pre>
  
  <p>I get the warning "no lexer rule corresponding to token: T_MILLIMETRE". Antlr runs though, but it still gives me the input text in the AST and not T_MILLIMETRE.</p>
  
  <p>I'm obviously not yet seeing the world the way Antlr does. Can anyone give me any hints or advice please?</p>
  
  <p>Steve</p>