Take the tour ×

I need to create an output text file by deleting the first two lines of the input file.

At the moment I'm using sed "1,2d" input.txt > output.txt

I need to do this for thousands of files, so am using python:

import os
for filename in somelist:
  os.system('sed "1,2d" %s-in.txt > %s-out.txt'%(filename,filename))

but this is quite slow.

I need to keep the original file, so I can't sed in place.

Is there a way to do this faster? Using something other than sed?Perhaps using some other scripting language than python? Is it worth writing a short C program, or is the file-writing disk-access likely to be the bottleneck?

share|improve this question
One issue with using os.system() is that it spawns a new shell every loop which adds a bit of overhead. Running directly in a loop in e.g. bash would reduce the overhead a little bit (but the performance increase would probably only be theoretical, since disk I/O is the limiting factor as mentioned by others) –  pafcu Aug 19 '10 at 14:04
@pafcu: It might not be so theoretical if the files were all four lines. But you could get pretty much the same improvement by using exec instead of system, I'd imagine. Don't really need a shell for this. –  Jefromi Aug 19 '10 at 14:08
add comment

3 Answers

up vote 9 down vote accepted

Use tail. Doubt anything could be significantly faster:

tail -n +3 input.txt > output.txt

Wrap it in your loop of choice. But I really doubt sed is a whole ton slower - as you say, disk i/o is usually the ultimate bottleneck.

share|improve this answer
Thanks, I've just tried and it's essentially indistinguishable from the original in how long it takes to run (just like ghostdog's sed -i.bak), so I suspect it's an i/o bottleneck. –  Gyppo Aug 19 '10 at 13:58
I know there's no way I'll get an answer but... why'd this get downvoted? It's the canonical way to do this operation in *nix. –  Jefromi Aug 19 '10 at 18:39
add comment

I think this will be faster than launching sed:

import os
import shutil

path = '/some/path/to/files/'
for filename in os.listdir(path):
    basename, ext = os.path.splitext(filename)
    fullname = os.path.join(path, filename)
    newname = os.path.join(path, basename + '-out' + ext)
    with open(fullname) as read:
        #skip first two lines
        for n in xrange(2):
        # hand the rest to shutil.copyfileobj
        with open(newname, 'w') as write:
            shutil.copyfileobj(read, write)
share|improve this answer
add comment
for file in *.ext
    sed -i.bak -n '3,$p' $file 

or just

sed -i.bak -n '3,$p' *.ext
share|improve this answer
That's very nice, thanks, but unfortunately it appears that i/o is the bottleneck. –  Gyppo Aug 19 '10 at 13:57
add comment

Your Answer


