title: unpacking an unknown serialised format with variable length
body: <p>I'm using Perl (5.8.8, don't ask) and I'm looking at a serialised binary file that I want to parse and snaffle information from.</p>
  
  <p>The format is as follows:</p>
  
  <ul>
  <li>7 bytes whose meaning I do not know (DB DB 00 00 7A 03 00)</li>
  <li>null (0x00)</li>
  <li>7-byte string with user id</li>
  <li>null (0x00)</li>
  <li>12-byte string to be discarded</li>
  <li>null (0x00)</li>
  <li>3-byte number specifying number of items to follow</li>
  <li>null (0x00)</li>
  <li>variable-length string for first item</li>
  <li>newline (0x0a)</li>
  <li>variable-length string for second item</li>
  <li>newline (0x0a)</li>
  <li>etc ...</li>
  <li>null (0x00)</li>
  <li>7-byte string with user id</li>
  <li>etc ...</li>
  </ul>
  
  <p>My current code somewhat naively skips the first 8 bytes, then reads byte per byte until it hits a null and then does very specific parsing. </p>
  
  <pre><code>sub readGroupsFile {
      my %index;
  
      open (my $fh, "&lt;:raw", "groupsfile");
      seek($fh, 8, 0);
      while (read($fh, my $userID, 7)) {
          $index{$userID} = ();
          seek($fh, 18, 1);
          my $groups = "";
          while (read($fh, my $byte, 1)) {
              last if (ord($byte) == 0);
              $groups .= $byte;
          }
          my @grouplist = split("\n", $groups);
          $index{$userID} = \@grouplist;
      }
      close($fh);
  
      return \%index;
  }
  </code></pre>
  
  <p>Good news? It works.</p>
  
  <p>However, I think it's not very elegant, and wonder if I can use the 2-byte number that specifies the amount of items to follow to my advantage to speed up the parsing. I have no idea why else it would be there.</p>
  
  <p>I think <code>unpack()</code> and its templates may provide an answer, but I can't figure out how it can work with variable-length arrays of strings with their own variable lengths.</p>