title: Process a file with 1.5 billion lines in perl
body: <p>I need to process a file with 1.5 billion entries with 11 columns, 300GB in size. I need to extract several information from each of those lines.
  I have divided the problem into two parts. First I read the file and try to reduce the file to the columns I need and some filtering. I write out this so called reduced file ~ 100GB with 700mil. entries.</p>
  <p>Then I read the reduced file and process each entry. Here is my code segment:</p>
  <pre><code>while (my $line = &lt;$fh_file_vss_rlrp_inst&gt;) {
    chomp $line;
    my @columns_line = split('\s+', $line);
    if( scalar @columns_line == 10 &amp;&amp; $columns_line[-1] !~ /X|Y|Z|K/ ) {
      print $fh_reduced &quot;$columns_line[1] $columns_line[7] $columns_line[-1]\n&quot;;
    } 
    $inst_count++;
     if($inst_count % 10000000 == 0) { 
       $date = `date`;
       print &quot;Processed $inst_count ... time: $date&quot;;
     }
  }
  </code></pre>
  <p>Now reading the reduced file for further processing and I have the <code>%h_kI_vB</code> in memory with 100,000 entries:</p>
  <pre><code>while (my $line = &lt;$fh_file_vss_rlrp_inst&gt;) {
  chomp $line;
  my @columns_line = split('\s+', $line);
    my $wip = $columns_line[0];
    my $this_inst = $columns_line[0];
    my @loc_xyz;
    my @loc_ijk;
    while (grep(/\//, $wip)){
        if (exists($h_kI_vB{$wip})){ 
            push(@loc_xyz, $h_kI_vB{$wip});                         
        }
        $wip =~ s/\/[^\/]+$//;
    }
    if (exists($h_kI_vB{$wip})){
        push(@loc_xyz, $h_kI_vB{$wip});
    }
    
    if (@loc_xyz){
        my $loc_block_string;
        foreach my $loc ( @loc_xyz){
            if (not defined $loc_block_string){
                $loc_block_string = &quot;$loc&quot;;
            } else {
                $loc_block_string = $loc_block_string.&quot;:$loc&quot;;
            }
        }
        $hierarchical_blocks{$this_inst}=$loc_block_string;
        
    }
    if(exists $hierarchical_blocks{$this_inst}) {
      my @x_y_layer = split(',', $columns_line[1]);
      $x_y_layer[0] =~ s/\(//;
      if(! defined $H_instBlock_coordinates{ $hierarchical_blocks{$this_inst} } ) { 
        my @initial_coordinate = ($x_y_layer[0], $x_y_layer[1], $x_y_layer[0], $x_y_layer[1]);
        print $fh_log &quot;For inst:$this_inst $hierarchical_blocks{$this_inst}: @initial_coordinate\n&quot; ;
        @{ $H_instBlock_coordinates{$hierarchical_blocks{$this_inst}} } = @initial_coordinate; 
      } else {
        my @old_x1_y1_x2_y2 =  @{ $H_instBlock_coordinates{$hierarchical_blocks{$this_inst}} };
        my $x1 = $old_x1_y1_x2_y2[0];
        my $y1 = $old_x1_y1_x2_y2[1];
        my $x2 = $old_x1_y1_x2_y2[2];
        my $y2 = $old_x1_y1_x2_y2[3];
        if($x_y_layer[0] &lt; $x1) {
          $x1 = $x_y_layer[0];
        } elsif ($x_y_layer[0] &gt; $x2) {
          $x2 = $x_y_layer[0];
        }
        
        if($x_y_layer[1] &lt; $y1) {
          $y1 = $x_y_layer[1];
        } elsif ($x_y_layer[1] &gt; $y2) {
          $y2 = $x_y_layer[1];
        }
        
        my @new_x1_y1_x2_y2 = ($x1, $y1, $x2, $y2);
        
        print $fh_log &quot;For inst:$this_inst Changing coordinate to block $hierarchical_blocks{$this_inst}: @new_x1_y1_x2_y2\n&quot; ;
        @{ $H_instBlock_coordinates{$hierarchical_blocks{$this_inst}} } = @new_x1_y1_x2_y2; 
        
      }
   }
    
   $inst_count++;
   if($inst_count % 1000000 == 0) {
     $date = `date`;
     print &quot;Processed $inst_count ...time: $date\n&quot;;
   }  
   }
  </code></pre>
  <p>This is horribly slow. I dispatch the job to a remote server with 450GB memory and it runs for 24 hours.
  I need to optimize the code so that it's finished within 1 hour (worst case).</p>
  <p>Thanks in advance.</p>