title: Can't get strtol() to detect overflow
body: <p>This program uses a lexical scanner to classify tokens as symbol, string, decimal number, hex number,… When a "number" is detected, it is handed over to <code>strtol()</code> to convert it to the internal 32-bit binary value. However I can't get <code>strtol()</code> to reliably return error on overflow.</p>
  
  <p>Part of conversion code is:</p>
  
  <pre><code>    errno = 0;      // erase any previous error in errno
      switch (constType) {
  …
  case lxHex:         // hexadecimal number X'1234567890ABCDEF' (X-string)
  fprintf(stderr,"** FindConstantFromString - converting %s\n",constBuffer);
          newDictEntry-&gt;dcValue = strtol(constBuffer+2, NULL, 16);
  int myerr = errno;
  fprintf(stderr,"     value %x errno %d\n",newDictEntry-&gt;dcValue, myerr);
          newDictEntry-&gt;dcType = syNumber;
          newDictEntry-&gt;dcSubType = 4;    // hexadecimal
          if  (   EINVAL == errno
              ||  ERANGE == errno
              ) {
              ErrDict = newDictEntry;
              AnaError (ConstMsg+2);
              newDictEntry-&gt;dcType = sySLit;
          };
          result.cstClass = newDictEntry-&gt;dcType;
          return result;
  …
  </code></pre>
  
  <p>When this code is tested with wrong input, it detects overflow only if the first hex digit is >= 8 (potentially giving negative value), as remonstrated by:</p>
  
  <pre><code>    29            | declare v;
      30            | v = x'fedcba9876543210'
  ** FindConstantFromString - meeting x'fedcba9876543210' as 11
  ** FindConstantFromString - converting x'fedcba9876543210'
       value ffffffff errno 34
  *Error 32: Candidate number x'fedcba9876543210' too large or could not be converted
  *Error 20: Unrecognisable lexical unit x'fedcba9876543210' at 30.5
      31            |     + x'123456789abcdef'
  ** FindConstantFromString - meeting x'123456789abcdef' as 11
  ** FindConstantFromString - converting x'123456789abcdef'
       value 89abcdef errno 0
      32            |     + 9876543210
  ** FindConstantFromString - meeting x'fedcba9876543210' as 8
                              symbol already known
  ** FindConstantFromString - converting x'fedcba9876543210'
       value 0 errno 0
  *Error 32: Candidate number x'fedcba9876543210' too large or could not be converted
  ** FindConstantFromString - meeting 9876543210 as 8
  ** FindConstantFromString - converting 9876543210
       value 4cb016ea errno 0
      33            |     + '12345a'
  ** FindConstantFromString - meeting 12345a as 3
      34            |     + '';
  ** FindConstantFromString - meeting 12345a as 8
                              symbol already known
  ** FindConstantFromString - converting 12345a
       value 3039 errno 0
  *Error 32: Candidate number 12345a too large or could not be converted
  ** FindConstantFromString - meeting  as 3
  ** FindConstantFromString - meeting  as 8
                              symbol already known
  *Error 33: Empty string cannot be converted to number
  </code></pre>
  
  <p>At line 30, the lexical scanner recognized a hex number and requested conversion from this hex form (11 = lxHex). <code>strtol()</code> correctly sets <code>errno</code> to <code>ERANGE</code> and error message is issued. Overflowed hex number is then kept in the dictionary as a string.</p>
  
  <p><em>Note that the returned value is -1, not <code>LONG_MAX</code>.</em></p>
  
  <p>At line 31, we again have another overflowing hex number, but it does not start with 8-9a-f. It is again detected as an hex number. Conversion is attempted but errno is not set at all. The value correspond to the lower 32 bits of the number. Since this considered success, the truncated value is kept as the result.</p>
  
  <p>When <code>+</code> is applied to "x'fed…'" and 89abcdef, another conversion is attempted on the <strong>string</strong> "x'fed…'" supposed to be a decimal number (denoted by the 8-request) and conversion fails because "x" cannot begin a decimal number.</p>
  
  <p>At line 32, we have an overflowing decimal 987654321. Once again, overflow is not detected (code not shown but similar to the one for hex numbers with the addition of a test on "endptr" since the strings may not be filtered by the lexical scanner and contain illegal characters). The returned value contains the least 32 bits of the number.</p>
  
  <p>If I change <code>strtol()</code> to <code>strtoul()</code>, the first <code>ERANGE</code> error disappears and I get the least 32 bits of the number.</p>
  
  <p>What am I doing wrong?</p>
  
  <p>System: Fedora Linux 29
  glibc: 2.27</p>