Take the 2-minute tour ×

What is the most efficient (fastest) way to simultaneously read in two large files and do some processing?

I have two files; a.txt and b.txt, each containing about a hundred thousand corresponding lines. My goal is to read in the two files and then do some processing on each line pair

def kernel:
    b_file=open('b.txt', 'r')
    a_line = a_file.readline()
    b_line = b_file.readline()
    while a_line:
        process(a_spl,b_spl) #process requiring both corresponding file lines

I looked in to xreadlines and readlines but i'm wondering if i can do better. speed is of paramount importance for this task.

thank you.

share|improve this question
Python isn't great for speed. C or C++ is recommended. Try: stackoverflow.com/questions/5164538/… –  Alvin K. Nov 8 '11 at 4:07
@Alvin K.: Profile first: Python is still fast enough to be able to outstrip most forms of I/O, especially if said I/O is hitting disk or network. –  Thanatos Nov 9 '11 at 4:13
@Thanatos: profiling is also mentioned in the link above, which claims that I/O isn't the main bottleneck. Thanks for highlighting it. –  Alvin K. Nov 12 '11 at 2:57
add comment

4 Answers

up vote 1 down vote accepted

The below code does not accumulate data from the input files in memory, unless the process function does that by itself.

def process(line1, line2):
  # process a line from each input

for a, b in izip(open(file1, 'r'), open(file2, 'r')):
  process(a, b)

If the process function is efficient, this code should run quickly enough for most purposes. The for loop will terminate when the end of one of the files is reached. If either file contains an extraordinarily long line (i.e. XML, JSON), or if the files are not text, this code may not work well.

share|improve this answer
You forgot to mention that your file requires itertools module. Without the following it won't work: from itertools import izip. –  Tadeck Nov 9 '11 at 2:37
Yes thanks. I also neglected to point out that in Jython, PyPy etc. this may leave the files open until the next gc. But cPython will close them immediately. –  wberry Nov 9 '11 at 15:51
so how can you explicitly close it if you don't give it a handle? –  Duke Nov 10 '11 at 2:51
You could assign the results of the open calls to variables f1 and f2 before the loop, and then call f1.close() and f2.close() after it's done. CPython does this for you as a courtesy whenever the refcount of file objects drops to zero. Jython and PyPy do not, as far as I know. –  wberry Nov 10 '11 at 14:36
add comment

You can use with statement to make sure your files are closed after the execution. From this blog entry:

to open a file, process its contents, and make sure to close it, you can simply do:

with open("x.txt") as f:
    data = f.read()
    do something with data
share|improve this answer
How is this relevant? –  Raymond Hettinger Nov 8 '11 at 4:33
add comment

String IO can be pretty fast -- probably your processing will be what slows things down. Consider a simple input loop to feed a queue like:

queue = multiprocessing.Queue(100)
a_file = open('a.txt')
b_file = open('b.txt')
for pair in itertools.izip(a_file, b_file):
     queue.put(pair) # blocks here on full queue

You can set up a pool of processes pulling items from the queue and taking action on each, assuming your problem can be parallelised this way.

share|improve this answer
In Python 2.x, be sure to use itertools.izip() so that the zip step doesn't happen all at once and pull both files completely into memory. –  Raymond Hettinger Nov 8 '11 at 4:35
maybe so, i'm reading in a list of tokens at each step and storing it in a list. maybe that's the bottleneck. program is hogging memory. –  Duke Nov 8 '11 at 13:44
Thanks Raymond for picking up that glaring mistake. Absolutely use izip in this scenario, not zip. I've fixed the example. –  Lars Yencken Nov 9 '11 at 4:10
thanks for the advice. got it to work using izip. also resolved my memory issue. did some optimizations on the algorithm and reduced the space consumption. –  Duke Nov 11 '11 at 20:59
add comment

I'd change your while condition to the following so that it doesn't fail when a has more lines than b.

while a_line and b_line

Otherwise, that looks good. You are reading in the two lines that you need, then processing. You could even multithread this by reading in N pairs of line and sending each pair off to a new thread or similar.

share|improve this answer
we are sure it will be the same line. goal is to reduce r.t. so avoided extra check –  Duke Nov 8 '11 at 4:04
Your program will crash horribly when you least expect it. If you have a GOOD REASON to be that worried about cycles, switch to C. –  ObscureRobot Nov 8 '11 at 4:09
zip will stop at the shorter of the two files –  Paul McGuire Nov 8 '11 at 4:45
add comment

Your Answer


