title: C lexer: token concatenation of unterminated string literals
body: <p>Consider the following c code:</p>
  <pre class="lang-c prettyprint-override"><code>#include &lt;stdio.h&gt;
  
  #define PRE(a) &quot; ## a
  
  int main() {
      printf(&quot;%s\n&quot;, PRE(&quot;));
      return 0;
  }
  </code></pre>
  <p>If we adhere strictly to the tokenization rules of c99, I would expect it to break up as:</p>
  <pre><code>...
  [#] [define] [PRE] [(] [a] [)] [&quot;]* [##] [a]
  ...
  [printf] [(] [&quot;%s\n&quot;] [,] [PRE] [(] [&quot;]* [)] [)] [;]
  ...
  </code></pre>
  <p>* A single non-whitespace character that does not match any <code>preprocessing-token</code> pattern</p>
  <p>And thus, after running preprocessing directives, the printf line should become:</p>
  <pre class="lang-c prettyprint-override"><code>printf(&quot;%s\n&quot;, &quot;&quot;);
  </code></pre>
  <p>And parse normally. But instead, <strong>it throws an error</strong> when compiled with gcc, even when using the flags <code>-std=c99 -pedantic</code>. What am I missing?</p>