Take the 2-minute tour ×

I asked a question a couple of days ago about stripping HTML from files with PERL. I am a n00b and I've searched the site for answers to my question...but unfortunately I couldn't find anything...this is probably because I'm a n00b and I didn't see the answer when I was looking at it.

So, here is the situation. I have a directory with around 20 gb of text files. I want to strip the HTML from each file and output each file to a unique text file. I've written the program below, which seems to do the trick for the first 12 text files in the directory (there are about 12,000 text files in total)...however...I run into a couple of snags. The first snag is that after the 12th text file has been parsed, then I start getting warnings about deep recursion...and then shortly after this the program quits because I've run out of memory. I imagine that my programming is extremely inefficient. So, I'm wondering if any of you see any obvious errors with my code below that would case me to run out of memory. ...once I figure things out then hopefully I'll be able to contribute.

#!/usr/bin/perl -w
#use strict;
use Benchmark;
#get the HTML-Format package from the package manager.
use HTML::Formatter;
#get the HTML-TREE from the package manager
use HTML::TreeBuilder;
use HTML::FormatText;
$startTime = new Benchmark;
my $direct="C:\\Directory";
my $slash='\\';

opendir(DIR1,"$direct")||die "Can't open directory";
my @New1=readdir(DIR1);

foreach $file(@New1)

#Initialize the variable names.
my $HTML=0;
my $tree="Empty";
my $data="";
#Open the file and put the file in variable called $data

    local $/;
    open (SLURP, "$direct$slash"."$file") or die "can't open $file: $!"; 
    #read the contents into data
    $data = <SLURP>; 

    #close the filehandle called SLURP
    close SLURP or die "cannot close $file: $!";
            #the following steps strip out any HTML tags, etc.
my $outfile = "out_".$file;
open (FOUT, "> $direct\\$outfile");
print FOUT "file: $file\nHTML: $HTML\n$Alldata\n","*" x 40, "\n" ;


$endTime = new Benchmark;
$runTime = timediff($endTime, $startTime);
print ("Processing files took ", timestr($runTime));
share|improve this question
add comment

3 Answers

up vote 2 down vote accepted

You are using up a lot of space with the list of files in @New1.

In addition, if you are using an older version of HTML::TreeBuilder then your objects of this class may need explcitly deleting, as they used to be immune to automatic Perl garbage collection.

Here is a program that avoids both of these problems, by reading the directory incrementallly, and by using HTML::FormatText->format_string to format the text, which implicitly deletes any HTML::TreeBuilder objects that it creates.

In addition, File::Spec makes a tidier job of building absolute file paths, and it is a core module so will not need installing on your system

use strict;
use warnings;

use File::Spec;
use HTML::FormatText;

my $direct = 'C:\Directory';

opendir my $dh, $direct or die "Can't open directory";

while ( readdir $dh ) {

  next if /^\./;

  my $file = File::Spec->catfile($direct, $_);
  my $outfile = File::Spec->catfile($direct, "out_$_");
  next unless -f $file;

  my $html = do {
    open my $fh, '<', $file or die qq(Unable to open "$file" for reading: $!);
    local $/;

  next unless $html =~ /<html/i;

  my $formatted = HTML::FormatText->format_string(
      $html, leftmargin => 0, rightmargin => 60);

  open my $fh, '>', $outfile or die qq(Unable to open "$outfile" for writing: $!);

  print $fh "File: $file\n\n";
  print $fh "$formatted\n";
  print $fh "*" x 40, "\n" ;

  close $fh or die qq(Unable to close "$outfile" after writing: $!);
share|improve this answer
Thank you--all of the responses in this thread were excellent. –  TaterTots Aug 6 '12 at 2:12
Uhh, I just ran this code on a couple of "bigger" text files and got the "out of memory" warning. Are there other steps I can take to optimize this code? Thanks again for your help! –  TaterTots Aug 6 '12 at 15:41
@TaterTots: How big are the biggest of your HTML files? Please show your code. Use pastebin if necessary –  Borodin Aug 6 '12 at 16:07
add comment

What was wrong with the answer to your previous question?

Your opening files for writing without checking the return code. Are you sure the succeed? And in which directory do you thing the files are created?

A better approach would be to:

  • read files 1 by 1
  • strip the HTML
  • write out the new file in the correct directory and checking the return code

something like:

while ( my $file = readdir DIR ) {

    ....process file

    open my $newfile, '>', "$direct/out_$outfile" or die "cannot open $outfile: $!\n";

   ... etc
share|improve this answer
Thank you for this response. In relation to the other question, I made some significant changes to the code and I wanted some additional feedback. I'm sorry if posting this new question was considered 'spamming' the board. Thanks again for your help! –  TaterTots Aug 6 '12 at 2:11
add comment

How to reduce the memory footprint of this application:

Does the problem persist when you add $tree = $tree->delete to the end of your loop?

The perl garbage collector cannot resolve circular references; so you have to destroy the tree manually so you don't run out of memory.

(See the first example in the module documentation at http://search.cpan.org/~cjm/HTML-Tree-5.02/lib/HTML/TreeBuilder.pm)

You should put the readdir inside the loop. The way you coded it, you first read in this gigantic list of files. When you say

my $file;
while (defined($file = readdir DIR1)) {..}

only one entry is actually read at a time. Should save some extra memory.

A few other comments on style:

default values

You give $tree the default value of "Empty". That is completely unneccessary. If you want to show how undefined a variable is, set it to undef, which it is by default. Perl guarantees this initialization.


You use backslashes as a directory seperator? Stop worrying and just use normal slashes. Unless you are on DOS you can use normal slashes as well, Windows isn't that dumb.

statement modifiers

This line


can be written far more readable as

next if $file =~ /^\./;

consequent use of parens

Your use of parens for function argument lists is inconsequent. You can omit the parens for all built-in functions unless there is ambiguity. I prefer avoiding them, others may find them easier to read. But please stick to a style!

better regex

You test for the existence of /<HTML>/i. What if I told you the html tag can have attributes? You should rather consider testing for /<html/i.

simplification (removes another bug)

Your test

if($HTML==1) {...}

can be written as

$HTML = $data =~ /<html/i;

can be written as

$HTML = $data =~ /<html/i
if ($HTML) {...}

can be folded into

if ($data =~ /<html/i) {...}

The way you implemented it, the $HTML variable was never reset to a false value. So once a file contained html, all subsequent files would have been treated as html as well. You can counteract such problems by defining your vars in the innermost sensible scope.

use HTML::FormatText, tribute to @pavel

Use the modules you use to the fullest. Look what I found in the example for HTML::FormatText:

my $string = HTML::FormatText->format_file(
           leftmargin => 0, rightmargin => 50

You can easily adapt that to circumvent building the tree manually. Why hadn't you tried this approach, as @pavel told you to in your other post? Would have saved you the memory problem...

use strict

Why did you comment out use strict? Getting as much fatal warnings as possible is important when learning a language. Or when writing solid code. That would force you to declare all your variables like $file sensibly. And rather use warnings than the -w switch, which is a bit outdated.

well done

But a very big "well done" on checking the return value of close ;-) That is very un-n00bish!

share|improve this answer
The status checking on close isn't so great as it is pretty much impossible to get an error on closing a file opened for read only, and the close on the output file isn't checked –  Borodin Aug 5 '12 at 23:52
Wow--thanks for the time you put into this response. I really learned a lot--I appreciate it! –  TaterTots Aug 6 '12 at 2:12
add comment

Your Answer


