title: Using awk to detect UTF-8 multibyte character
body: <p>I am using awk (symlinked to gawk on my machine) to read through a file and get a character count per line to test if a file is fixed width.  I can then re-use the following script with the <code>-b --characters-as-bytes</code> option to see if the file is fixed width by byte.</p>
  
  <pre><code>#!/usr/bin/awk -f
  
  BEGIN {
      width = -1;
  }
  
  {
      len = length($0);
  
      if (width == -1) {
          width = len;
      } else if (len != 0 &amp;&amp; len != width) {
          exit 1;
      }
  }
  </code></pre>
  
  <p>I want to do something similar to test whether each line in a file has the same amount of bytes and characters to assume all characters are a single byte (I do realize this is subject false negatives).  The challenge is that I would like to run through the file one time and break out at first mismatch.  Is there a way to set the <code>-b</code> option from within an awk script similar to how you can adjust FS.  If this isn't possible, I'm open to options outside of awk.  I can always just write this in <code>C</code> if I have to, but I wanted to make sure there isn't something already available.</p>
  
  <p>Efficiency is what I am aiming for here.  Having this information will help me skip a costly process, so I don't this in itself to be costly.  I'm dealing with files that can be over 100 million lines long.</p>
  
  <p><strong>Clarification</strong></p>
  
  <p>I want something like the above.  Something like this </p>
  
  <pre><code>#!/usr/bin/awk -f
  {
      if (length($0) != bytelength($0))
          exit 1;
  }
  </code></pre>
  
  <p>I don't need any output.  I will just trigger off the return code (<code>$?</code> in bash).  So exit 1 if this fails.  Obviously bytelength is not a function.  I'm just looking for a way to achieve this without running awk twice.</p>
  
  <p><strong>UPDATE</strong></p>
  
  <p>sundeep's solution works for what I have described above:</p>
  
  <pre><code>awk -F '' -l ordchr '{for(i=1;i&lt;=NF;i++) if(ord($i)&lt;0) {exit 1;}}'
  </code></pre>
  
  <p>I was operating under the assumption that <code>awk</code> would count a higher-end character with a Windows single-byte encoding above 0x7F as a single character, but it actually doesn't count it at all.  So byte length would still not be the same as length.  I guess I will need to write this in C for something that specific.</p>
  
  <p><strong>Conclusion</strong></p>
  
  <p>So I think I did a poor job of explaining my problem.  I receive data that is encoded in either UTF-8 or Windows' style single-byte encoding like CP1252.  I wanted to check if there are any multibyte characters in the file and exit if found.  I originally wanted to do this in awk, but I playing with files that may have a different encoding has proven difficult.  </p>
  
  <p>So in a nutshell if we assume a file with a single character in it:</p>
  
  <pre class="lang-none prettyprint-override"><code>CHARACTER  FILE_ENCODING     ALL_SINGLE_BYTE   IN_HEX
  á          UTF-8             false             0xC3 0xA1
  á          CP1252            true              0xE1
  a          ANY               true              0x61
  </code></pre>