title: Resolving Out of Memory error when executing Perl script
body: <p>I'm attempting to build a n-gram language model based on the top 100K words found in the english language wikipedia dump.  I've already extracted out the plain text with a modified XML parser written in Java, but need to convert it to a vocab file.  </p>
  
  <p>In order to do this, I found a perl script that is said to do the job, but lacks instructions on how to execute.  Needless to say, I'm a complete newbie to Perl and this is the first time I've encountered a need for its usage.</p>
  
  <p>When I run this script, I'm getting an Out of Memory Error when using this on a 7.2GB text file on two separate dual core machines with 4GB RAM and runnung Ubuntu 10.04 and 10.10.</p>
  
  <p>When I contacted the author, he said this script ran fine on a MacBook Pro with 4GB RAM, and the total in-memory usage was about 78 MB when executed on a 6.6GB text file with perl 5.12.  The author also said that the script reads the input file line by line and creates a hashmap in memory.</p>
  
  <p>The script is:</p>
  
  <pre><code>#! /usr/bin/perl
  
  use FindBin;
  use lib "$FindBin::Bin";
  
  use strict;
  require 'english-utils.pl';
  
  ## Create a list of words and their frequencies from an input corpus document
  ## (format: plain text, words separated by spaces, no sentence separators)
  
  ## TODO should words with hyphens be expanded? (e.g. three-dimensional)
  
  my %dict;
  my $min_len = 3;
  my $min_freq = 1;
  
  while (&lt;&gt;) {
  
      chomp($_);
      my @words = split(" ", $_);
  
      foreach my $word (@words) {
  
          # Check validity against regexp and acceptable use of apostrophe
  
          if ((length($word) &gt;= $min_len) &amp;&amp; ($word =~ /^[A-Z][A-Z\'-]+$/) 
          &amp;&amp; (index($word,"'") &lt; 0 || allow_apostrophe($word))) {
              $dict{$word}++;
          }
      }
  
  }
  
  # Output words which occur with the $min_freq or more often
  
  foreach my $dictword (keys %dict) {
      if ( $dict{$dictword} &gt;= $min_freq ) {
          print $dictword . "\t" . $dict{$dictword} . "\n";
      }
  }
  </code></pre>
  
  <p>I'm executing this script from the command line via <code>mkvocab.pl corpus.txt</code></p>
  
  <p>The included extra script is simply a regex script to test the placement of apostrophe's and whether they match English grammar rules.</p>
  
  <p>I thought the memory leak was due to the different versions, as 5.10 was installed on my machine.  So I upgraded to 5.14, but the error still persists.  According to <code>free -m</code>, I have approximately 1.5GB free memory on my system.</p>
  
  <p>As I am completely unfamiliar with the syntax and structure of language, can you point out the problem areas along with why the issue exists and how to fix it.</p>