title: Correct way to unpack a 32 bit vector in Perl to read a uint32 written in C
body: <p>I am parsing a Photoshop raw, 16 bit/channel, RGB file in C and trying to keep a log of exceptional data points. I need a very fast C analysis of up to 36 MPix images with 16 bit quanta or 216 MB Photoshop .RAW files. </p>
  
  <p>&lt;1% of the points have weird skin tones and I want to graph them with PerlMagick or Perl GD to see where they are coming from.</p>
  
  <p>The first 4 bytes of the C data file contain the unsigned image width as a uint32_t. In Perl, I read the whole file in binary mode and extract the first 32 bits:</p>
  
  <pre><code>Xres=1779105792l = 0x6a0b0000
  </code></pre>
  
  <p>It looks a lot like the C log file:</p>
  
  <pre><code>DA: Color anomalies=14177=0.229%:
  DA: II=1) raw PIDX=0x10000b25,  XCols=[0]=0x00000b6a
  </code></pre>
  
  <p>Dec(0x00000b6a) = 2922, the Exact X_Columns_Width of a small test file. </p>
  
  <p>Clearly a case of intel's 1972 8008 NUXI architecture. How hard could it possibly be to translate 0x6a0b0000 to 0x6a0b0000; swap 2 bytes and 2 nibbles and you're done. Slicing the 8 characters and rearranging them could be done but that is the kind of ugly hack I am trying to avoid. </p>
  
  <p>Grab the same 32 bit vector from file offset zero and unpack it as "VAX" unsigned long. </p>
  
  <pre><code>$xres = vec($bdat, 0, 32);  # vec EXPR,OFFSET,BITS
  $vul   = unpack("V", vec($bdat, 0, 32));
  printf("Length (\$bdat)=%d, xres=0x%08x, Vax ulong=%ul=0x%08x\n",
      length($bdat), $xres, $vul, $vul);
  Length ($bdat) = 56712, xres=0x6a0b0000, Vax ulong=959919921l=0x39373731
  </code></pre>
  
  <p>Every single hex character is mangled. Obviously wrong Endian, it is not VAX. The "Other" one is Network Big-endian</p>
  
  <pre><code>http://perldoc.perl.org/functions/pack.html
  N  An unsigned long (32-bit) in "network" (big-endian) order.
  V  An unsigned long (32-bit) in "VAX" (little-endian) order.
  $nul = unpack("N", vec($bdat, 0, 32));  # Network Unsigned Long 32b
  printf("Xres=0x%08x, NET ulong=%ul=0x%08x\n", $xres, $nul, $nul);
  Xres=0x6a0b0000, NET ulong=825702201l=0x31373739
  </code></pre>
  
  <p>The $XRES still shows the right hex in the wrong order. The "NETWORK" long 32 bit uint extracted from the same bits is unrecognizable. Try Binary</p>
  
  <pre><code>$bits = unpack("b*", vec($bdat, 0, 32));
  printf("bits=$bits, len=%d\n", length $bits);
     bits=10001100111011001110110010011100100011000000110010101100111011001001110001001100, len=80
  </code></pre>
  
  <p>I clearly asked for 32 bits and got 80 bits. What gives? </p>
  
  <p>Try for 4, unsigned, 8bit bytes which can NOT be swapped:</p>
  
  <pre><code>for($ii = 0; $ii &lt; 4; $ii++)  {
      $bit_off=$ii*8;  # Bit offset
      $uc = unpack("C", vec($bdat, $bit_off, 8));  # C  An unsigned char 
      printf("II $ii, bo $bit_off, d=%d, u=%u, x=0x%x\n", 
         $uc,$uc, $uc);
  }
  II 0, bo 0, d=49, u=49, x=0x31
  II 1, bo 8, d=51, u=51, x=0x33
  II 2, bo 16, d=49, u=49, x=0x31
  II 3, bo 24, d=49, u=49, x=0x31
  </code></pre>
  
  <p>I am looking for hex 0, 6, a or b. There are no "3"s or "1"s in the right answer. Try pirating from a C file:</p>
  
  <pre><code>http://cpansearch.perl.org/src/MHX/Convert-Binary-C-0.76/tests/include/include/bits/byteswap.h
  $x = $xres;
  $x= (((($x) &amp; 0xff000000) &gt;&gt; 24) | ((($x) &amp; 0x00ff0000) &gt;&gt;  8) |     ((($x) &amp; 0x0000ff00) &lt;&lt;  8) | ((($x) &amp; 0x000000ff) &lt;&lt; 24));
  printf("\$xres=0x%08x -&gt; \$x=0x%08x = %u\n", $xres, $x, $x);
  $xres=0x6a0b0000 -&gt; $x=0x00000b6a = 2922
  </code></pre>
  
  <p>It WORKS! But, this is uglier than converting the original, wrong order hex number to a string to untangle it:</p>
  
  <pre><code>$stupid_str = sprintf("%08x", $xres);
  $stupid_num = join('', reverse ($stupid_str =~ m/../g));
  printf("Stupid_num '%s'-&gt;0x%08x=%d\n", $stupid_num, $dec=hex $stupid_num, $dec);
  Stupid_num '00000b6a'-&gt;0x00000b6a=2922
  </code></pre>
  
  <p>It's like judging the Ugliest Dog contest, but I would still rather have to maintain the text version than the even more abominable C version. </p>
  
  <p>I know there are ways to do this in Java/Python/Go/Ruby/.....</p>
  
  <p>I know there are command line utilities that do exactly this.</p>
  
  <p>I must figure out how I am misusing either VEC or Unpack, both of which I have used a zillion times. It is the Brain Teasing aspect which is driving me nuts!  EndianNess == EndianMess!!!</p>
  
  <p>TYVM!</p>
  
  <p>=================================================</p>
  
  <p>Borodin,</p>
  
  <p>Thanks for lookin' at this. </p>
  
  <p>My intel processor is little-endian. When I read it back, it was trans-mutilated by vec to the "correct" big-endian, network format. </p>
  
  <p>I just tried reading it VERBATIM from a BINARY file read and it works fine:</p>
  
  <pre><code>($b4 = $bdat) =~ s/^(....).*$/$1/msg;   # Give me my 4 bytes back without mutilation!
  printf("B4='%s'=&gt;0x%08x=&lt;0x%08x\n", $b4, unpack("L&gt;", $b4), unpack("L&lt;", $b4));
  B4='j...' = &gt;0x6a0b0000 = &lt;0x00000b6a   &lt;&lt;&lt;  THE RIGHT ANSWER!!!
  </code></pre>
  
  <blockquote>
    <blockquote>
      <p>If you try unpack 'V', $bdat then you will find that it works</p>
    </blockquote>
  </blockquote>
  
  <p>That was my first attempt:
      $vul   = unpack("V", vec($bdat, 0, 32));  #  UNPACK V!<br>
      printf("Length (\$bdat)=%d, xres=0x%08x, Vax ulong=%ul=0x%08x\n",
          length($bdat), $xres, $vul, $vul);
      Length ($bdat) = 56712, xres=0x6a0b0000, Vax ulong=959919921l=0x39373731   &lt;&lt;&lt;&lt; TOTALLY WRONG!</p>
  
  <p>I had already verified that the $BDAT info was the right data in the wrong format. It just needed some rearrangement. </p>
  
  <p>I just used vec() to generate 1 bit and 4 bit graphics files and it worked faithfully, returning the exact bits I wrote. It must have mistaken my Intel i7 for my IBM System/370. I7/37??? Easy mistake to make. :)</p>
  
  <p>I read the [confusing] part about "converted to a number as with pack ...". That's why my number was backward. The >>unpack("V", vec($bdat"&lt;&lt; ... was my ill-fated attempt to byte-swap the backward number in $BDAT from the WRONG VEC()-preferred FORMAT to the native format supported by my architecture.</p>
  
  <p>Now I understand why I saw so many examples of people extracting by the byte, to avoid Big Brother's helping hand! </p>
  
  <pre><code>Data::BitStream::Vec "uses a Perl vec to store the data. The vector is accessed in 1-bit units"
  </code></pre>
  
  <p>Thanks 1E6,</p>
  
  <pre><code>B
  </code></pre>