title: Does a priority declaration disambiguate between alternative lexicals?
body: <p>In my <a href="https://stackoverflow.com/questions/51048128/rascal-ambiguity-not-resolved-by-disambiguation-rules">previous question</a>, there was a priority <code>&gt;</code> declaration in the example. It turned out not to matter because the solution there did not actually invoke priority but rather avoided it by making the alternatives disjoint. In this question, I'm asking whether priority can be used to select one lexical production over another. In the example below, the language of the production <code>WordInitialDigit</code> is intentionally a subset of that of <code>WordAny</code>. The production <code>Word</code> looks like it should disambiguate between the two properly, but the resulting parse tree has an ambiguity node at the top. Is a priority declaration able to decide between different lexical reductions, or does it require there to be a basis of common lexical elements? Or something else?</p>
  
  <p>The example is contrived (there are no actions in the grammar), but the situations it arises from are not. For example, I'd like to use something like this for error recovery, where I can recognize a natural boundary for a unit of syntax and write a production for it. This generic production would be the last element in a priority chain; if it reduces, it means that there was no valid parse. More generally, I need to be able to select lexical elements based on syntactic context. I had hoped, since Rascal is scannerless, that this would be seamless. Perhaps it is, though I don't see it at the moment.</p>
  
  <p>I'm on the unstable branch, version 0.10.0.201807050853.</p>
  
  <p>EDIT: This question is not about <code>&gt;</code> for defining an expression grammar. The <a href="http://tutor.rascal-mpl.org/Rascal/Declarations/SyntaxDefinition/Disambiguation/Priority/Priority.html" rel="nofollow noreferrer">documentation for priority declarations</a> talks mostly about expressions, but the very first sentence provides what looks like a perfectly clear definition:</p>
  
  <blockquote>
    <p>Priority declarations define a partial ordering between the productions <em>within a single non-terminal</em>.</p>
  </blockquote>
  
  <p>So the example has two productions, an ordering declared between them, and yet the parser is still generating an ambiguity node in the clear presence of a disambiguation rule. So to put a finer point on my question, it looks like I don't know which of two situations pertains. Either (1) if this isn't supposed to work, then there's a defect in the language definition as documented, a deficiency in error reporting of the compiler, and a language design decision that's somewhere between counter-intuitive and user-hostile. Or (2) if this is supposed to work, there's a defect in the compiler and/or parser (presumably because the focus was initially on expressions) and at some point the example will pass its tests.</p>
  
  <pre><code>module ssce
  
  import analysis::grammars::Ambiguity;
  import ParseTree;
  import IO;
  import String;
  
  lexical WordChar = [0-9A-Za-z] ;
  lexical Digit = [0-9] ;
  lexical WordInitialDigit = Digit WordChar* !&gt;&gt; WordChar;
  lexical WordAny = WordChar+ !&gt;&gt; WordChar;
  syntax Word =
      WordInitialDigit
      &gt; WordAny
      ;
  
  test bool WordInitialDigit_0() = parseAccept( #Word, "4foo" );
  test bool WordInitialDigit_1() = parseAccept( #WordInitialDigit, "4foo" );
  test bool WordInitialDigit_2() = parseAccept( #WordAny, "4foo" );
  
  bool verbose = false;
  
  bool parseAccept( type[&amp;T&lt;:Tree] begin, str input )
  {
      try
      {
          parse(begin, input, allowAmbiguity=false);
      }
      catch ParseError(loc _):
      {
          return false;
      }
      catch Ambiguity(loc l, str a, str b):
      {
          if (verbose)
          {
              println("[Ambiguity] #&lt;a&gt;, \"&lt;b&gt;\"");
              Tree tt = parse(begin, input, allowAmbiguity=true) ;
              iprintln(tt);
              list[Message] m = diagnose(tt) ;
              println( ToString(m) );
          }
          fail;
      }
      return true;
  }
  
  bool parseReject( type[&amp;T&lt;:Tree] begin, str input )
  {
      try
      {
          parse(begin, input, allowAmbiguity=false);
      }
      catch ParseError(loc _):
      {
          return true;
      }
      return false;
  }
  
  str ToString( list[Message] msgs ) =
      ( ToString( msgs[0] ) | it + "\n" + ToString(m) | m &lt;- msgs[1..]  );
  
  str ToString( Message msg)
  {
      switch(msg)
      {
          case error(str s, loc _): return "error: " + s;
          case warning(str s, loc _): return "warning: " + s;
          case info(str s, loc _): return "info: " + s;
      }
      return "";
  }
  </code></pre>