title: Does each call to `yylex()` generate a token or all the tokens for the input?
body: <p>I am trying to understand how flex works under the hood. </p>
  
  <ul>
  <li><p>In the following first example,  it seems that <code>main()</code> calls <code>yylex()</code> only once, and <code>yylex()</code>  generates all the tokens for the entire input.</p></li>
  <li><p>In the second example, it seems that <code>main()</code> calls <code>yylex()</code>  once per token generated, and <code>yylex()</code>  generates a token per call. </p></li>
  </ul>
  
  <p>Does each call to <code>yylex()</code>  generate a token or all the tokens for the input?</p>
  
  <p>Why is <code>yylex()</code> called different number of times in the two examples?</p>
  
  <p>I heard that <code>yylex()</code> is like a coroutine, and each call to it will resume with the rest of the input left from last call and generate a token. In that sense, how does the first example calls <code>yylex()</code> just once and generate all the tokens in the input?</p>
  
  <pre><code>/* just like Unix wc */
  %{
  int chars = 0;
  int words = 0;
  int lines = 0;
  %}
  %%
  [a-zA-Z]+  { words++; chars += strlen(yytext); }
  \n         { chars++; lines++; }
  .          { chars++; }
  %%
  main(int argc, char **argv)
  {
    yylex();
    printf("%8d%8d%8d\n", lines, words, chars);
  }
  
  
  $ ./a.out
  The boy stood on the burning deck
  shelling peanuts by the peck
  ^D
  2 12 63
  $
  </code></pre>
  
  <p>and </p>
  
  <pre><code>/* recognize tokens for the calculator and print them out */
  %{
     enum yytokentype {
       NUMBER = 258,
       ADD = 259,
       SUB = 260,
       MUL = 261,
       DIV = 262,
       ABS = 263,
       EOL = 264
     };
     int yylval;
  %}
  %%
  "+"    { return ADD; }
  "-"    { return SUB; }
  "*"    { return MUL; }
  "/"    { return DIV; }
  "|"    { return ABS; }
  [0-9]+ { yylval = atoi(yytext); return NUMBER; }
  \n     { return EOL; }
  [ \t]  { /* ignore whitespace */ }
  .      { printf("Mystery character %c\n", *yytext); }
  %%
  main(int argc, char **argv)
  {
    int tok;
    while(tok = yylex()) {
      printf("%d", tok);
      if(tok == NUMBER) printf(" = %d\n", yylval);
      else printf("\n");
    }
  }
  
  $ ./a.out
  a / 34 + |45
  Mystery character a
  262
  258 = 34
  259
  263
  258 = 45
  264
  ^D
  $
  </code></pre>