title: lex/yacc: why do lexers have to include a parser's header file?
body: <p>I'm trying to learn a little more about compiler construction, so I've been toying with flexc++ and bisonc++; for this question I'll refer to flex/bison however.</p>
  
  <p>In bison, one uses the <code>%token</code> declaration to define token names, for example</p>
  
  <pre><code>%token INTEGER
  %token VARIABLE
  </code></pre>
  
  <p>and so forth. When bison is used on the grammar specification file, a header file <code>y.tab.h</code> is generated which has some define directives for each token:</p>
  
  <pre><code>#define INTEGER 258
  #define VARIABLE 259
  </code></pre>
  
  <p>Finally, the lexer includes the header file <code>y.tab.h</code> by returning the right code for each token:</p>
  
  <pre><code>%{
  #include "y.tab.h"
  %}
  
  %%
  
  [a-z]   { 
              // some code
              return VARIABLE;
          }
  
  [1-9][0-9]* {
              // some code
              return INTEGER;
          }
  </code></pre>
  
  <p>So the parser defines the tokens, then the lexer has to use that information to know which integer codes to return for each token.</p>
  
  <p><strong>Is this not totally bizarre?</strong> Normally, the compiler pipeline goes lexer -> parser -> code generator. Why on earth should the lexer have to include information from the parser? The lexer should define the tokens, then flex creates a header file with all the integer codes. The parser then includes that header file. These dependencies would reflect the usual order of the compiler pipeline. What am I missing?</p>