title: How can I open a Unicode file with Perl?
body: <p>I'm using osql to run several sql scripts against a database and then I need to look at the results file to check if any errors occurred.  The problem is that Perl doesn't seem to like the fact that the results files are Unicode.</p>
  
  <p>I wrote a little test script to test it and the output comes out all warbled:</p>
  
  <pre><code>$file = shift;
  
  open OUTPUT, $file or die "Can't open $file: $!\n";
  while (&lt;OUTPUT&gt;) {
      print $_;
      if (/Invalid|invalid|Cannot|cannot/) {
          push(@invalids, $file);
          print "invalid file - $inputfile - schedule for retry\n";
          last;
      }            
  }
  </code></pre>
  
  <p>Any ideas?  I've tried decoding using <code>decode_utf8</code> but it makes no difference.  I've also tried to set the encoding when opening the file.</p>
  
  <p>I think the problem <em>might</em> be that osql puts the result file in UTF-16 format, but I'm not sure.  When I open the file in textpad it just tells me 'Unicode'.  </p>
  
  <p>Edit: Using perl v5.8.8
  Edit: Hex dump:</p>
  
  <pre><code>file name: Admin_CI.User.sql.results
  mime type: 
  
  0000-0010:  ff fe 31 00-3e 00 20 00-32 00 3e 00-20 00 4d 00  ..1.&gt;... 2.&gt;...M.
  0000-0020:  73 00 67 00-20 00 31 00-35 00 30 00-30 00 37 00  s.g...1. 5.0.0.7.
  0000-0030:  2c 00 20 00-4c 00 65 00-76 00 65 00-6c 00 20 00  ,...L.e. v.e.l...
  0000-0032:  31 00                                            1.
  </code></pre>