title: AWK performance while processing big files
body: <p>I have an awk script that I use for calculate how much time some transactions takes to complete. The script gets the unique ID of each transaction and stores the minimum and maximum timestamp of each one. Then it calculates the difference and at the end it shows those results that are over 60 seconds.</p>
  
  <p>It works very well when used with some thousand (200k) but it takes more time when used in real world. I tested it several times and it takes about 15 minutes to process about 28 million of lines. Can I consider this good performance or it is possible to improve it?</p>
  
  <p>I'm open to any kind of suggestion.</p>
  
  <p>Here you have the complete code</p>
  
  <pre><code>zgrep -E "\(([a-z0-9]){15,}:" /path/to/very/big/log |  awk '{
  gsub("[()]|:.*","",$4); #just removing ugly chars
  ++cont
  min=$4"min" #name for maximun value of current transaction
  max=$4"max" #same as previous, just for readability 
  split($2,secs,/[:,]/) #split hours,minutes and seconds
  seconds = 3600*secs[1] + 60*secs[2] + secs[3] #turn everything into seconds
  if(arr[min] &gt; seconds || arr[min] == 0)
    arr[min]=seconds
  if(arr[max] &lt; seconds)
     arr[max]=seconds
  dif=arr[max] - arr[min]
  if(dif &gt; 60)
    result[$4] = dif
  }
  END{
  for(x in result)
     print x" - "result[x]
  print ":Processed "cont" lines"
  }'
  </code></pre>