title: sed / perl regex extremly slow
body: <p>So, I've got a file called
  cracked.txt, which contains thousands(80million+) lines of this:</p>
  <pre><code>dafaa15bec90fba537638998a5fa5085:_BD:zzzzzz12
  a8c2e774d406b319e33aca8b38540063:2JB:zzzzzz999
  d6d24dfcef852729d10391f186da5b08:WNb:zzzzzzzss
  2f1c72ccc940828b5daf4ab98e0f8731:@]9:zzzzzzzz
  3b7633b6c19d79e5ab76bdb9cce4fd42:#A9:zzzzzzzz
  a3dc9c03ff845776b485fa8337c9625a:yQ,:zzzzzzzz
  ade1d43b29674814a16e96098365f956:FZ-:zzzzzzzz
  ba93090dfa64d964889f521788aca889:/.g:zzzzzzzz
  c3bd6861732affa3a437df46a6295810:m}Z:zzzzzzzz
  b31d9f86c28bd1245819817e353ceeb1:&gt;)L:zzzzzzzzzzzz
  </code></pre>
  <p>and in my output.txt 80 million lines like this:</p>
  <pre><code>('chen123','45a36afe044ff58c09dc3cd2ee287164','','','','f+P',''),
  ('chen1234','45a36afe044ff58c09dc3cd2ee287164','','','','f+P',''),
  ('chen125','45a36afe044ff58c09dc3cd2ee287164','','','','f+P',''),
  </code></pre>
  <p>(45a36afe044ff58c09dc3cd2ee287164 and f+P change every line)</p>
  <p>What I've done is created a simple bash script to match the cracked.txt to output.txt and join them.</p>
  <pre><code>cat './cracked.txt' | while read LINE; do
  pwd=$(echo &quot;${LINE}&quot; | awk -F &quot;:&quot; '{print $NF}' | sed -e 's/\x27/\\\\\\\x27/g' -e 's/\//\\\x2f/g' -e 's/\x22/\\\\\\\x22/g' )
  hash=$(echo &quot;${LINE}&quot; | awk -F &quot;:&quot; '{print $1}')
  lines=$((lines+1))
  echo &quot;${lines} ${pwd}&quot;
  perl -p -i -e &quot;s/${hash}/${hash} ( ${pwd} ) /g&quot; output.txt
  #sed -u -i &quot;s/${hash}/${hash} ( ${pwd} ) /g&quot; output.txt
  done
  </code></pre>
  <p>As you can see by the comment, I've tried sed, and perl.
  perl seems to be a tad faster than sed
  I'm getting something like one line per second.</p>
  <p>I've never used perl, so I've got no idea how to use that to my advantage (multi threading, etc)</p>
  <p>What would the best way to speed up this process?</p>
  <p>Thanks</p>
  <p>edit:
  I got a suggestion that it would be better to use something like this:</p>
  <pre><code>while IFS=: read pwd seed hash; do 
     ...
  done &lt; cracked.txt
  </code></pre>
  <p>But because inbetween the first and last occurance of : (awk '{print $1}' awk '{print $NF}', : could appear inbetween there, it would make it bad(corrupt it)
  I could use it just for the &quot;hash&quot;, but not for the &quot;pwd&quot;.
  edit again;
  The above wouldn't work, because I would have to name all the other data, which ofc will be a problem.</p>