Ted Pedersen > Text-NSP > huge-merge.pl


Annotate this POD


Open  0
View/Report Bugs


huge-merge.pl - Merge the results of multiple huge-sort generated files into a single sorted file.


huge-merge.pl output-directory


Combine the sorted bigram files generated by huge-sort.pl efficiently.

This program is used internally by huge-count.pl.


huge-merge.pl [OPTIONS] SOURCEDIR


Required Arguments:


Input to huge-merge.pl should be a single flat directory containing multiple plain text files generated by huge-sort.pl. The result file, merge.* (* is a number, the final result file has the maximum number), is in the source directory.

Optional Arguments:


Switches ON the --keep option will keep all the intermediate merging files.

Other Options:


Displays the help information.


Displays the version information.


There is a limitation in huge-merge.pl. When the size of the corpus is very large (>16G) and the some of the terms of the bigrams is very long (>30 chars), the program could run out of memory at huge-merge.pl step. This is because huge-merge use two hashes to count the frequencies of the first and second term of the bigrams. These two hashes could use up the memory with the increase of the length of the terms and the increase of the number of the terms. If just for normal text, terms are within limited length and numbers, the software won't use up the memory.


Ying Liu, University of Minnesota, Twin Cities. liux0395 at umn.edu

Ted Pedersen, University of Minnesota, Duluth. tpederse at umn.edu


Copyright (C) 2009-2011, Ying Liu and Ted Pedersen



syntax highlighting: