Take the 2-minute tour ×

I have data like this:

1 10
1 30
1 40
1 10
2 20
2 20
2 30
3 50
3 10
3 10
3 10
4 20
4 10

I would like to sum all the values up IF the value of first column matches, the result would be like this,

1 90
2 70
3 80
4 30

I have my code here,

while (<DATA>) 
my ($a, $b) = split;
$hash{$a}  += $b;

foreach $a (sort keys %hash) 
$b = $hash{$a};
print OUT "$a $b\n";

It works with sample data (around 100MB) but it seems to take ages to deal with my real data (around 100G). Are there any ways to optimize my codes?

Appreciate any advises in advance!

share|improve this question
sounds like a good candidate for MapReduce. You could also look into using Threads. –  Hunter McMillen Jul 12 '13 at 20:18
Define "ages". Where is this data coming from? If it's from a hard disk, 100GB is going to take many many minutes to run, regardless of the processing that you're doing. –  Oliver Charlesworth Jul 12 '13 at 20:18
@OliCharlesworth it is from a hard disk.. –  Sam Jul 12 '13 at 20:22
try the simple time cat datafile >/dev/null will get the minimum time for reading the file sequentially... –  jm666 Jul 12 '13 at 20:24
@HunterMcMillen: It looks like the problem here is IO. I don't think threads or MapReduce are going to help (unless the data source is split across the nodes). –  Oliver Charlesworth Jul 12 '13 at 20:38

3 Answers 3

If your data looks like you show us it seems you have it sorted by key so hash is not necessary at all.

perl -anE'if($k!=$F[0]){say"$k $s"if$.>1;$k=$F[$s=0]}$s+=$F[1]}{say"$k $s"'

will do the trick. I doubt it will be slow.

share|improve this answer

Hashes are quite efficient. They are probably the best solution to your problem. However, there could be exceptions, depending on your data:

  • If all keys are integers in a (more or less) continuous range, then you can use an array instead, which is even more efficient than a hash:

    while (<DATA>) {
      my ($k, $v) = split;
      $array[$k] += $v;
    for my $i (grep defined $array[$_], 0 .. $#array) {
      print "$i $array[$i]\n";
  • If the keys are already sorted, we don't need any intermediate data structure. Just accumulate the sum into a scalar. When the key changes, output the sum of the last key.

  • If you have multiple files, you can apply your algorithm for each of these files in parallel and combine the results. This lets your code run in logarithmic time instead of linear time (aka. a big win). Either split the large file into smaller chunks, our do some magic with seek and tell to partition the file. The more busy processors you have, the faster your file will be summarized. With one caveat: It might very well be that I/O is your bottleneck. If this task has to be done regulary, using a SSD (instead of a HDD) might drastically improve performance.

share|improve this answer
Many thanks for your comment! –  Sam Jul 12 '13 at 21:12
thanks! using an array is more efficiency when the keys are in a continuous range. Anyway if the keys are not continuous, any efficient ideas using hash? –  Sam Jul 12 '13 at 21:29
@Sam The “problem” with arrays is, that they are compact: If you have a key 1 and a key 1000, there will be 999 allocated, but empty fields (all indices in between, plus zero). If the possible keys are low enough (where low depends on how important memory is for you), then an array is OK. For this use case, I'd get uncomfortable if any key exceeds 2E6 (two million). –  amon Jul 12 '13 at 21:36

As others stated, your most likely bottleneck isn't hashes or Perl, but disk access.

Split up the file into smaller chunks. (using standard Unix utils if you can).

Store them on SEPARATE IO sources (different disks ideally on different controllers, ideally on different PCs).

  • If you have only a few keys (e.g. >100-1000 rows per key), simply run chunks separately, then concatenate them all into 100x smaller file, and process that one file as a whole.

  • Otherwise, synchronize the processing using a database to store sums.

share|improve this answer
thanks and i think i will try to use a database due to the number of keys. Anyway splitting up the file is a good try! –  Sam Jul 12 '13 at 21:21
Trust DVK! Sort the chunks (files) with unix tools and sum the values until the key changes. Write the result to a new file. You can mangage tons of data with it. (also works when you have multiple keys) –  smartmeta Jul 12 '13 at 22:11

Your Answer


